We rebuilt RLPD.The ablation changed the story.
A three-person PyTorch reproduction of offline-to-online reinforcement learning—extended to Humanoid-v5, stress-tested across three seeds, and forced to explain a result the paper did not predict.
Can old experience accelerate a policy without destabilizing what it learns?
RLPD makes a deceptively simple promise: mix a fixed offline dataset into ordinary online SAC, then add enough structure to keep the critic honest when the policy leaves the data distribution.
We treated reproduction as an interrogation—not a screenshot of one benchmark. The framework changed from JAX to PyTorch, the data from D4RL to Minari v5, and the evaluation expanded to a body the paper never tested.
The method is small. The implementation details carry the weight.
One shared motion language connects all three: sample, bound, update. Each value below comes directly from the checked configuration.
Symmetric sampling
Every update draws 128 online and 128 offline transitions. The dataset enters through the sampler—there is no RLPD pretraining phase.
ratio = 0.5 · batch = 256LayerNorm critic
Normalization limits value extrapolation on actions the offline data never covered. Removing it on Humanoid sent mean Q to 8.9×10¹⁰.
layernorm = trueEnsemble + high UTD
Ten critics and twenty gradient updates per environment step trade compute for sample efficiency while reducing single-critic overestimation.
ensemble = 10 · utd = 20The reproduction held. Consistency was the stronger result.
Across Hopper, Walker2d, and HalfCheetah, RLPD was the only method to finish between 88 and 90 on every task. IQL reached similar peaks but carried much wider seed variance; SACfD fell behind sharply.
These are final evaluations from the honest three-seed medium-data matrix. The chart leads; the policy footage follows.
Minari v5 normalization: random policy = 0, measured expert dataset = 100. Every row uses three complete runs at 245k environment steps.
policy replay
policy replay
policy replayThe failure was not subtle.
On Walker2d, SACfD's mean Q climbed two orders of magnitude to 85,300 while RLPD settled near 545 on the same data and three-seed budget.
The unbounded critic coincides with a final return of 8.1 ± 2.1. The later no-LayerNorm ablation reproduces the same explosion inside the RLPD architecture, strengthening the mechanism without pretending one curve proves every causal step.
A harder body exposed a different kind of advantage.
Humanoid-v5 expands the problem to 348 observations and 17 actuators. We ran every method to one million environment steps and kept divergent runs in the record.
IQL looks dominant on the environment-step axis, but it arrives with one million offline gradient steps already completed. Its 47.9-point start is a pretraining head start, not free learning.
last-five normalized at 1M environment steps
bounded critic, but little locomotion emerged
critic divergence was recorded, not retried
The moving policy is a best-seed visualization. The 70.1 ± 16.2 aggregate is the result.
We turned off the offline data. The policy got better.
Keeping LayerNorm, the critic ensemble, and high UTD—but setting the sampling ratio to online-only—outperformed the full 50/50 method at the matched 500k horizon.
The architecture survived. The prior data became the constraint.
This does not invalidate offline-to-online RL. It says the premise is conditional: data quality, policy coverage, tuning, and the handoff between learning regimes can matter more than the presence of a dataset alone. On this body, forcing half of every batch toward the offline distribution pulled learning away from where the live policy was going.
Every conclusion carries its sample count.
Incomplete runs stay visible. Best seeds never masquerade as aggregates.
The single complete expert-data run stayed near 4.0 after one million steps. That is evidence against a simple “the medium data was bad” explanation, but n=1 is not a population claim.
Reproduction is not imitation. It is a pressure test.
Variance is part of the result.
A peak policy can look convincing while the three-seed distribution tells a weaker story.
Implementation choices become research claims.
LayerNorm, ensemble size, and update ratio determined whether the critic stayed meaningful.
Ablations changed the question.
Once online-only won, the study shifted from “does RLPD reproduce?” to “when does prior data help?”
Visual behavior needs numeric context.
Rollouts explain what a score feels like; they do not replace aggregate evaluation.