Lifelong Learning Problem Statement
Multi-Task Learning: Learn to solve a set of tasks.
Meta-Learning: Given i.i.d. task distribution, learn a new task efficiently, learn to learn tasks.
Some terminology:
Sequential learning settings: online learning, lifelong learning, continual learning, incremental learning, streaming data.
Distinct from sequence data and sequential decision-making.
What do you want from your lifelong learning algorithm?
Minimal regret (that grows slowly with t).
Regret: cumulative loss of learner - cumulative loss of best learner in hindsight
$$ Regret_T := \sum_{1}^{T}L_t(\theta_t) - \min_{\theta}\sum_{1}^{T}L_t(\theta_{best}) $$
Cannot be evaluated in practice, useful for analysis.
Regret that grows linearly in $t$ is trivial.
Basics Approaches to Lifelong Learning
Store all the data you’ve seen sof far, and train on it. -> Follow the leader algorithm
- Will achieve very strong performance
- Computation intensive -> Continuous fine-tuning can help
- Can be memory intensive -> Depends on the application
Take a gradient step on the data point you observe. -> Stochastic gradient descent
- Computationally cheap
- Requires 0 memory
- Subject to negative backward transfer - “forgetting”, sometimes referred to as catastrophic forgetting
- Slow learning
Using Meta-Learning to Accelerate Online Learning
See more from Deep online learning via meta-learning: Continual adaptation for model-based RL
Using Modified Vanilla SGD to Avoid Negative Backward Transfer
See more from Gradient Episodic Memory for Continual Learning
Online Learning Formulation
The online meta-learning setting
The follow the leader(FTL) algorithm:
- Store all the data you’ve seen so far, and train on it
- Deploy model on current task
The Follow the meta-leader(FTML) algorithm:
- Store all the data you’ve seen so far, and meta-train on it
- Run the update procedure on the current task
Summary
- Many flavors of lifelong learning, all under the same name
- Defining the problem statement is often the hardest part
Meta-learning can be viewed as a slice of the lifelong learning problem.
This is a very open area of research.
Note: Cover Picture