Learning a new task should preserve useful skills. Experience on an unfamiliar task should make the next attempt better. RL can address both goals, but they require different training signals.
This companion to Sample-Efficient Algorithms follows those two goals through policy optimization and RL-based self-adaptation.
Learning Across Windows
Suppose tasks arrive in windows $W_1, W_2, \ldots$. Each window brings prompts and a way to score responses. We can train on one window, pool the windows seen so far, or keep updating the same model as each window arrives.
Windows 1–3 have arrived; window 4 is still ahead.
Train on the current window only.
No earlier window contributes to this training run.
Pool all windows available so far.
Earlier prompts remain available for sampling.
Continue training as each window arrives.
Earlier learning survives through the weights; old-task performance can fall.
The continual setup creates the retention question: after learning $W_t$, how well does the model still perform on $W_1, \ldots, W_{t-1}$? Joint training keeps earlier prompts in the training pool, at the cost of retaining and revisiting that data. Neither setup guarantees that tasks will not interfere. A window here is a slice of arriving tasks or data, not a PPO minibatch.
Reducing Forgetting
A policy learns tasks in sequence. Each stage supplies new rewards, while earlier skills must survive the updates. The training objective needs to account for both acquisition and retention:
\[\text{new-task return}\; -\;\lambda\,\text{loss of earlier capabilities}.\]CPO approximates the retention term through a parameter constraint. TSRL controls where expert supervision enters RL training.
Remember coordinates that moved on earlier tasks.
Penalize movement on the accumulated mask.
Optimize GSPO with the retention penalty.
Protect selected parameters while learning the next task.
Measure the group’s success rate.
Add supervision only on difficult states.
Reduce the supervision weight over training.
Use expert guidance where exploration is failing.
CPO: constrain changes that threaten earlier skills
A KL penalty on current-task prompts does not directly constrain behavior on old tasks. CPO starts from behavioral KL on the historical task distribution, then approximates it without replaying old data.
After each task, it adds the parameter coordinates with the largest changes to a protected set $S$. On the next task, it minimizes
\[L(\theta)=L_{\mathrm{GSPO}}(\theta) +\frac{\lambda}{|S|}\sum_{j\in S}|\theta_j-\theta_j^{\mathrm{prev}}|.\]The penalty is omitted when $S$ is empty. Parameter movement serves as a proxy for sensitivity; the mask accumulates across tasks. This changes the update constraint, while task rewards remain the learning signal.
Evidence: sequential multimodal tasks with Qwen3-VL. Its results show why retention must be measured on earlier tasks, rather than inferred from a small KL on new data.
TSRL: ask for supervision when exploration fails
Sparse rewards can leave an agent stuck. TSRL combines group-based RL with expert supervision triggered by a low group success rate $q(s)$:
\[\begin{aligned} L(\theta)&=L_{\mathrm{RL}}(\theta)+\alpha_t L_{\mathrm{help}}(\theta),\\ L_{\mathrm{help}}&=\mathbb E_s\!\left[ \mathbf 1[q(s)\leq\delta]\,\bigl(-\log\pi_\theta(a_E\mid s)\bigr)\right]. \end{aligned}\]Here $a_E$ is an expert action. The default threshold $\delta=0$ activates help when the whole group fails; $\alpha_t$ decreases over training. The method uses GRPO or DAPO underneath.
Evidence: agent-task transfer and retention of math ability after search training. Selective supervision improves the acquisition–retention trade-off in these settings; it does not imply that RL alone prevents forgetting.
Improving Adaptation
RL can also train a model to make better use of experience. Early attempts supply information; later attempts reveal whether the policy used it well. The training signal therefore reaches beyond the current answer.
ORBIT and LaMer learn to adapt through context. SEAL learns to produce effective weight updates. These are adaptation mechanisms, not guarantees of long-term retention.
Interact with an unfamiliar environment.
Carry observations into the next attempt.
Train on the total successes across the trial.
GRPO compares whole trials. At deployment, experience changes context; weights stay fixed.
Collect actions, observations and outcomes.
Write guidance for the next attempt.
Their rewards train earlier actions and reflections.
GiGPO trains a policy whose credit assignment crosses episode boundaries.
Generate training data or update instructions.
Update model weights using the proposed edit.
Use downstream results to select useful edits.
ReSTᵉᵐ trains self-edit generation. Adaptation itself updates weights.
ORBIT: reward a whole learning trial
A trial contains repeated attempts at the same environment instance. Each episode conditions on the previous interaction history $H_{k-1}$:
\[\tau_k\sim\pi_\theta(\cdot\mid H_{k-1}), \qquad R_{\mathrm{trial}}=\sum_{k=1}^{K}R(\tau_k).\]GRPO compares complete trials, using their total successful completions. Early exploration can receive credit through later success. At deployment, weights stay fixed while the context grows.
Evidence: three-episode trials, with evaluation on Maze and Mastermind, task families absent from training. The useful measurement is the performance curve across attempts, alongside a single-episode RL baseline.
LaMer: train reflection with later rewards
LaMer explicitly generates a reflection between episodes and trains it using the next episode’s reward. Action credit also extends across episodes. At the episode level, its discounted return has the form
\[G_k=R_k+\gamma_{\mathrm{traj}}R_{k+1} +\gamma_{\mathrm{traj}}^2R_{k+2}+\cdots.\]The cross-episode discount controls how strongly later outcomes influence earlier decisions. GiGPO is the default optimizer; history and reflections provide the adaptation state without test-time weight updates.
Evidence: Sokoban, Minesweeper and WebShop, plus ALFWorld transfer. Its matched sampling comparison uses eight three-episode trials versus 24 independent episodes. This helps separate learning from feedback from simply generating more attempts.
SEAL: reward an update by what it teaches
SEAL makes a self-edit $e$ the action: synthetic training data or instructions for fine-tuning. It applies the edit, evaluates the adapted model, and uses that outcome to train the edit generator:
\[e\sim\pi_\theta(\cdot\mid D_{\mathrm{new}}), \qquad \theta'_e=\operatorname{SFT}(\theta,e).\]The reward comes from downstream performance after adaptation. ReST$^{\mathrm{EM}}$ samples edits, selects successful or best-performing ones according to the task protocol, and fine-tunes on those edits. This is reward-selected self-training, rather than a PPO clipped update.
Evidence: knowledge incorporation and few-shot ARC tasks. SEAL demonstrates learned parameter adaptation, while its repeated-update experiments still exhibit forgetting.
For continual learning, I would want both: an adaptation curve that improves with experience, and a retention curve that stays strong as tasks accumulate. Better performance after three attempts does not establish retention across a long task stream.