Changelog¶

All notable changes to rlox are documented here.

[Unreleased]¶

Added¶

rlox-sandbox crate — unprivileged Linux sandbox for executing untrusted code with hard isolation (Component 1 of agentic-benchmark MVP). Combines Linux user+pid+net+mnt namespaces, seccomp-BPF allowlist filter (blocks network, CLONE_NEWUSER, ptrace, etc.), and cgroup v2 resource limits (memory.max, pids.max, cpu.weight) with kill-safety via cgroup freeze + cgroup.kill (Linux 5.14+). Memory bombs are contained at the cap (no swap-thrash) via memory.swap.max=0 with authoritative OOM detection through memory.events. Includes 26 regression tests covering benign execution, timeouts, and adversarial containment (fork bombs, memory bombs, stdout flooding, nested-userns denial, cgroup-base validation).
Adversarial corpus v1 (benchmarks/agentic/corpus/) — fixed, versioned, SHA-256-integrity-checked stimulus for the benchmark's P3 reliability claim. Covers all six categories (infinite loop, fork bomb, memory bomb, unkillable thread, blocking network, fd exhaustion); every sample is verified contained by rlox-sandbox (bounded time-to-contain, no survivors).
Rollout server (rlox-sandbox::server) — async axum /rollout service (Component 2 of agentic-benchmark MVP): calls a vLLM /v1/completions endpoint, runs each completion through the sandbox for a verifiable reward, computes group-relative advantages via the canonical rlox-core op, and returns trajectories plus first-class BackendStats telemetry (P1 throughput + P3 containment counters: adversarial_contained, contagion_events, setup_error_events, time_to_contain_secs, cgroup/oom events). vLLM failures surface as HTTP 502 (no silent zero-reward degradation); sandbox concurrency is bounded; cgroup_base is validated at startup (server must run inside a systemd-delegated scope). Python-side rlox.agentic.stats.BackendStats dataclass mirrors the wire contract.
Verification endpoint (rlox-sandbox::server POST /verify) — sandbox-only seam for reward-level hosts (prime-rl/verifiers): takes already-generated {code, tests, is_adversarial}, runs one sandbox execution (no vLLM), returns {reward, backend_stats}. Complements /rollout (full generate+verify, for AgentLoop-style hosts).
Verifiers adapter (rlox.agentic.verifiers_adapter, Component 4 of agentic-benchmark MVP) — implements prime-rl's verifiers load_environment seam with a one-key Baseline↔Treatment swap (rollout_backend): "in_loop" runs code in-process (the unsafe Baseline), "rlox" POSTs to the sandbox /verify endpoint (the isolated Treatment); identical config keys and identical adversarial injection in both. Wires group_size → env.sampling_args["n"].
Adversarial injector (rlox.agentic.adversarial_corpus) — loads + SHA-256-integrity-checks the corpus and injects adversarial samples at a configurable fraction via an instance-local seeded PRNG (deterministic, backend-independent). Canonical digest convention is consistent across Rust (serde_json) and Python (json.dumps).
Benchmark harness (rlox.agentic, Component 6/7/8 of agentic-benchmark MVP) — MetricCollector (injectable GPU sampler, per-step JSONL, warm-up-excluded summary; AC-4), ContagionDetector (VmRSS-spike / FD-leak / server-reported rules with rolling baselines; AC-5 P3 hard line), BenchmarkConfig + validate_config (refuses to run on unset locked constants or version-pin mismatch; AC-2), and the run_sweep driver (2 conditions × seeds × adversarial fractions, per-run survival + metric-store persistence, exception-resilient). All TDD-tested on a light env (no GPU).
OQ-3 pilot (benchmarks/agentic/oq3_pilot.py) — validates that the unprotected Baseline code-exec path degrades on the adversarial corpus (6/6 categories stall or exhaust), gating the pre-registered sweep; runs each sample in a resource-capped scope so it cannot wedge the host.
Reporting + go/no-go (rlox.agentic.reporting, Step 7 of agentic-benchmark MVP) — numpy percentile-bootstrap_ci (deterministic), ci_overlap_check quality-parity guardrail (≥2/3 seed pairs; AC-9), write_summary (machine-readable JSON + CSV; AC-7), and assess_go_no_go encoding the pre-registered P1/P3/guardrail thresholds verbatim.
repro.sh (benchmarks/agentic/repro.sh, AC-8) — one-command, idempotent reproduction: Rust toolchain + uv venv with pinned deps + release build of rlox-sandbox + adversarial-corpus SHA-256 integrity check + launch the sweep under a resource-capped systemd scope. --setup-only / --dry-run modes; the full 24-run grid (2×3 seeds×4 fractions) previews via run_benchmark.py --dry-run.
rlox-rl-ops crate — extracted GRPO/token-KL ops from rlox-core into a slim, zero-dependency-on-core crate (estimator-agnostic AdvantageEstimator trait for GRPO↔DAPO pluggability, GroupRelativeEstimator z-score normalisation with rayon parallel dispatch ≥4096 elems, single-group convenience functions, token-KL ops f32+f64 exact and Schulman 2020 approximate). rlox-core re-exports for back-compat.
rlox_agent Python package — the standalone, torch-free agentic harness (adversarial_corpus, verifiers_adapter, stats, metric_collector, contagion_detector, config, reporting) split out of rlox.agentic, which now re-exports as thin shims. Import as from rlox_agent.<module> (canonical); rlox.agentic.<module> remains for backward-compat but is deprecated.
/verify reward integrity — nonce-authenticated trusted-runner protocol in rlox-sandbox::server so model code cannot forge reward via sys.exit(0)/os._exit()/monkeypatch; runner seals return value before child process sees it.
Security hardening — seccomp fail-closed (denies by default), clone3 blocked, CLONE_NEWUSER masked (conditional BPF rule), no persistent /tmp per sandbox, proc-scoped environment. Regression-tested in crates/rlox-sandbox/tests/security_isolation.rs. Honest caveat: filesystem//proc info-isolation is best-effort under AppArmor unprivileged_userns (enforced at the Python-runtime level; a raw-syscall adversary could bypass it — a kernel-level guarantee requires the AppArmor profile to be relaxed).
TRL single-GPU GRPO runner (benchmarks/agentic/trl_grpo_run.py) — Qwen3-4B-Instruct-2507 + LoRA on unit-test-verified coding task; --backend {in_loop,rlox} and --adversarial-fraction flags for Baseline↔Treatment A/B study. 30-step canonical sweep: Treatment (rlox) survives 3/3 runs at every injection fraction (0/1/5/10 %) vs Baseline 3/3·3/3·2/3·1/3; quality-parity guardrail MET (final reward 0.917 = 0.917). P1 (multi-GPU throughput) deferred; rlox's novel contribution is P3 (containment) + reward integrity.
Test coverage — rlox-rl-ops 30 tests, rlox-core 472, rlox-sandbox 60+doctest, rlox_verify 11, benchmarks/agentic 567.
Per-algorithm maturity status — rlox.trainer.ALGORITHM_STATUS maps every registered algorithm to "validated" (convergence-tested with SB3 parity: PPO, SAC, TD3, DQN, A2C) or "experimental" (implemented but not convergence-validated: the other 13 registered algorithms). Surfaced via the new read-only Trainer.status property, included in repr(Trainer(...)), and a UserWarning is emitted when an experimental algorithm is constructed by name. Helper algorithm_status(name) does a case-insensitive lookup. A completeness invariant (set(ALGORITHM_STATUS) == set(ALGORITHM_REGISTRY)) forces every newly registered algorithm to declare a status.
rlox-grpc end-to-end test coverage — first integration tests for the distributed env-worker gRPC layer (previously zero Rust tests): in-process server↔client round-trips for reset_batch/step_batch shape correctness and terminal-observation transmission.
TQC algorithm (rlox.algorithms.TQC, Truncated Quantile Critics — Kuznetsov et al., ICML 2020, arXiv:2005.04269) — SAC with distributional critics: an ensemble of n_critics (default 5) quantile networks (each outputs n_quantiles = 25 quantiles); the target pools all n_critics × n_quantiles next-quantiles, sorts them, and drops the top top_quantiles_to_drop_per_net × n_critics (truncation) to control overestimation, trained with quantile-Huber regression. Reuses SAC's actor/replay/entropy. Solves Pendulum-v1 (greedy eval −155.67, textbook defaults, no tuning). Trainer("tqc", ...), status experimental.
Recurrent PPO (rlox.algorithms.RecurrentPPO) — an LSTM actor-critic PPO for partially-observable / memory tasks: hidden state carried across timesteps and reset at episode boundaries, BPTT over per-episode sequence segments with masked loss (gradients never cross a done), reusing the Rust GAE op and PPO's clipped-surrogate loss. Discrete action spaces. Fully solves CartPole-v1 (reward 500.0). Trainer.evaluate() now resets recurrent state between eval episodes (via reset_predict_state()). Trainer("recurrent_ppo", ...), status experimental.
CrossQ algorithm (rlox.algorithms.CrossQ, ICLR 2024 — Bhatt, Palenicek et al., arXiv:1902.05605) — a drop-in SAC upgrade for continuous control that removes target networks entirely and stabilises TD learning with Batch Renormalization critics (networks.BatchRenorm1d + BNQNetwork) and a joint forward pass (current and next state-actions go through the live critic together so BatchNorm statistics stay consistent between Q(s,a) and the bootstrap Q(s',a')). Reuses SAC's replay buffer, squashed-Gaussian actor, and automatic entropy tuning; actor updates are delayed by policy_delay (default 3). Uses Adam betas=(0.5, 0.999) (paper-specified — β1=0.5 is load-bearing; the torch default 0.9 is unstable with BatchNorm). Solves Pendulum-v1 reliably (greedy eval −134 ± 0.5 across 4 seeds, deterministic within a seed). Discrete action spaces raise. Registered as Trainer("crossq", ...), status experimental (MuJoCo sample-efficiency parity is the tracked follow-up).
TRPO promoted to validated — added a convergence config (benchmarks/convergence/configs/trpo_cartpole.yaml) and ran a 5-seed sweep: CartPole-v1 IQM = 500.0 (4/5 seeds ≥ 494.9, threshold 475), and confirmed learning on continuous control (Hopper-v4, 223 vs ~15 random at 100k). TRPO needs a large rollout per update (n_steps=2048) for a stable Fisher/KL estimate; smaller batches collapse convergence. ALGORITHM_STATUS["trpo"] is now "validated"; a full MuJoCo multi-seed parity sweep is the tracked follow-up.
PQN algorithm (rlox.algorithms.PQN, Parallelised Q-Network — Gallici et al., arXiv:2407.04811) — value-based RL with no replay buffer and no target network: a LayerNorm-regularised Q-network (LayerNormQNetwork) plus ℓ² regularisation stabilise TD learning, with λ-returns computed over parallel rollouts. Reuses the existing Rust compute_gae_batched op for the Q(λ) targets (no new Rust) and rlox's Rayon VecEnv for the parallel env loop. Discrete action spaces; correct truncation bootstrapping (adds γ·max_a Q(terminal_obs,a) on time-limit truncation, mirroring RolloutCollector). Registered as Trainer("pqn", ...), status experimental (learns CartPole; multi-seed validation pending). PQNConfig with validated ε schedule.

Added¶

scripts/check-ci-local.sh — reproduces every CI gate locally in one command, closing the three gaps that let CI failures land unseen: it lints rlox-sandbox against x86_64-unknown-linux-gnu (the only way to see lints inside #[cfg(target_os = "linux")] from macOS), runs pytest --collect-only as a fast standalone gate before the suite, and passes --no-fail-fast so no test binary masks another. Also sets PYO3_PYTHON to the repo venv, since pyo3 0.23 supports Python ≤ 3.13 and a newer system python3 hard-fails the pyo3-ffi build script. And it runs tests/agentic/ (the stdlib-only subset) on Python 3.10 in a throwaway uv venv — CI covers a 3.10–3.13 matrix while the local venv is one version, so version-gated code otherwise fails only on CI. Takes rust / python to scope, and WK=1 to include the Linux sandbox suite on wk-system.
SECURITY.md — the project had no security policy despite executing untrusted code by design. Defines a private reporting channel (GitHub advisories, no email exposure), what is in scope for rlox-sandbox (escape, resource escape, reward forgery, silent loss of containment) and what is not (checkpoint deserialization, the training APIs — only the sandbox is a security boundary), restates the known AppArmor unprivileged_userns limitation as a documented non-finding, and warns that a skipped containment test counts as passed, so a green run on an undelegated host proves nothing.
Contributing, Changelog, Security and the sandbox architecture on the docs site — surfaced under a new Project nav section. Implemented as symlinks to the repo-root files (and to crates/rlox-sandbox/README.md), so the site and the repo cannot drift apart the way a copy would.
SLOW=1 opt-in gate in scripts/check-ci-local.sh — runs the convergence tests locally. CI runs them on pushes to main only, so a regression there is structurally invisible to a PR and lands on main; this is the only way to check before merging. Off by default at ~20–40 min.
scripts/install-git-hooks.sh — installs a pre-push hook running the fast, high-yield gates (rustfmt, both clippy passes, pytest collection). Deliberately excludes the full test suites: a pre-push hook that takes minutes gets bypassed with --no-verify, which defeats the point.
rust-toolchain.toml — pins the toolchain (1.97.1) so local clippy enforces exactly the lint set CI does. Clippy gains lints every stable release, so an unpinned stable meant CI could fail on lints an older local toolchain never reported — which is precisely how doc_overindented_list_items reached main.
tests/test_repo_hygiene.py — guards the repo-layout mistakes that reached CI: a conftest.py that defines no fixtures or hooks (i.e. a shared-helper module wearing a pytest-reserved filename), any import/from of the ambiguous top-level conftest module, an unguarded import of a 3.11+ stdlib module such as tomllib (which breaks only the oldest supported Python, so no single-interpreter run catches it), and workflow-referenced extras missing from pyproject.toml. Static AST checks, so they hold whichever interpreter runs them.
python-collect CI job — a ~30 s pytest --collect-only gate on the oldest supported Python (3.10), which python-tests and slow-tests now depend on. A collection error means pytest runs nothing at all, so gating loses no signal while naming the broken module immediately instead of four matrix jobs each spending ~4 min rediscovering the same bad import.

Fixed¶

CI green again — the entire Python suite had been silently skipped. A single module-name collision aborted pytest collection (Interrupted: N errors during collection), so all ~2150 tests were deselected on every Python version rather than run. benchmarks/conftest.py was a plain helper library (BenchmarkResult, ComparisonResult, timed_run) wearing pytest's reserved filename: pytest imports every conftest.py as the single top-level module conftest, so tests/agentic/conftest.py won sys.modules["conftest"] and five tests/python/test_bench_*.py modules died on from conftest import BenchmarkResult. The helpers now live in benchmarks/harness.py. Also fixed in the same pass: tests/agentic/test_primerl_runner.py imported tomllib unguarded (stdlib only from 3.11, so it broke the 3.10 job — now falls back to the tomli backport, matching python/rlox/config.py); tests/agentic/test_verifiers_adapter.py imported the optional, vLLM-pulling verifiers package unguarded (now pytest.importorskip); and pyproject.toml gained the all extra that CI's wheel-smoke job already installed (pip install -e ".[all]" only warns on an unknown extra, so it was silently installing nothing) plus pytest-timeout/tomli/tomli-w.
Flaky rlox-burn gradient-flow tests — test_td3_multiple_steps_reduce_negative_q failed CI on a docs-only PR. BurnDeterministicPolicy::new takes no seed (unlike BurnStochasticPolicy), so with unseeded init an unlucky draw makes a training step move parameters by less than the 1e-8 threshold. Its sibling test_td3_actor_step_changes_params had already been quarantined with #[ignore] for the same cause. Seeding alone was not sufficient: Backend::seed sets process-global RNG state and cargo runs a test binary multi-threaded, so concurrent tests consume each other's RNG — measured 2 failures in 25 runs seeded-but-unlocked, 0 in 15 with --test-threads=1. Both halves are now in place via a seeded() guard (mutex + seed, poison-recovering), scoped to these four tests rather than forcing --test-threads=1 on the workspace or adding a serial_test dependency. 0 failures in 60 runs of the configuration that previously failed. The quarantined test is un-ignored, restoring the autograd-through-trait-boundary regression coverage: seeded margins are 0.075–1.46 against 1e-8/1e-7 thresholds, seven orders of magnitude, so it is also insensitive to cross-platform float differences.
Internal planning documents were published on the public docs site — mkdocs builds every .md under docs_dir regardless of nav, so 50 pages unreachable from the navigation were still served and search-indexable (verified: …/python/docs-overhaul-plan/ and …/python/plans/agentic-ops-and-package-refactor/ both returned HTTP 200). An exclude_docs rule now keeps plans/, the PRDs and the improvement plans in the repo but off the site. Listed explicitly rather than by a *-plan.md glob, because hybrid-collection-plan.md and rust-optimization-plan.md are in the nav and a glob would have silently dropped them.
Two dead links in crates/rlox-sandbox/README.md — both pointed at .wf/design.md, which is untracked and no longer exists, so they were broken for every reader. Repointed at the agentic benchmark guide and benchmarks/agentic/README.md.
README understated the Python suite by ~2× — claimed "~1100+ Python tests"; the measured total is 2233 (2142 non-slow + 75 slow + 16 skipped). Rust's "~670" was accurate (675). The old figure was plausibly right back when only tests/python/ was counted — the same narrow-path assumption that hid the conftest collision.
README had no contributor, changelog or security entry point, and no platform statement — zero mentions of CONTRIBUTING.md or CHANGELOG.md, so the verification workflow was undiscoverable from the front page; and while it stated Python 3.10–3.13 it never said which OSes, leaving rlox-sandbox being Linux-only to be inferred from test commands. All now in the Documentation table and the requirements list.
Slow tests (convergence) broke main after merge — test_tqc_greedy_eval_solves_pendulum hit the job's global --timeout=600, 40 minutes into the run. That job runs on pushes to main only, so no PR could have caught it; and before collection was repaired these tests had not run at all. The test is correct, the budget was wrong: TQC trains an ensemble of 5 quantile critics (25 quantiles each), so ~19k gradient steps cost far more than the single critic pair the 600 s was calibrated for. Measured 351 s locally and passing, and GitHub runners are ~2x slower on these envs. Fixed with a targeted @pytest.mark.timeout(1200) (~3.4x local, leaving headroom for runner variance) rather than shrinking total_timesteps — the -250 threshold is calibrated to 20k steps, so a smaller budget would weaken the assertion instead of fixing the timeout. Verified that a per-test marker really does override the CLI --timeout, since the fix depends on it. The job also gains timeout-minutes: 90; it had no cap while legitimately running ~40 min, so a genuine hang could have burned the 6 h default.
The documented sandbox_verify example did not work — examples/rust/README.md said scripts/wk-sync-test.sh "handles delegation automatically", but the helper only auto-wraps commands containing cargo test in the delegated rlox.slice scope. A cargo run therefore landed in a plain SSH session and died with SetupError("child could not write to cgroup.procs (cgroup migration failed)") — confirmed by running the documented command. The helper now accepts WK_DELEGATE=1 to force the scope for any command (additive; cargo test auto-detection unchanged), and the README uses it. Both the helper and the manual systemd-run path are verified end to end: benign → Clean, fork bomb → OomKilled.
Docs recommended pytest tests/python/, which hides a third of the suite — that path collects 1583 of 2233 tests, omitting tests/agentic/ and the repo-hygiene guards. It is also why the conftest collision between tests/agentic/ and tests/python/ stayed invisible locally while aborting collection for the whole suite in CI. CONTRIBUTING.md, README.md, PROJECT_QUICK_REFERENCE.md, docs/getting-started.md and docs/python-guide.md now say pytest tests/, matching CI; the stale "900+ tests" figure is replaced with measured counts.
crates/rlox-sandbox/README.md was materially inaccurate — it claimed "Total: 22 tests", listed five test files with wrong per-file counts, and omitted six others (corpus_containment, security_isolation, server_contract, rollout_pipeline, verify_endpoint, backend_stats_serde). Actual: 61 tests across 11 binaries plus a doctest, from a verified delegated run. It also documented the wk-sync-test backstops as TasksMax=100 / MemoryMax=2 GiB when the script defaults are 4096 / 24G, and did not mention the capability gates at all. All corrected — including the non-obvious point that a gated-out test returns early and is counted by cargo as passed, so 61 passed on CI is not evidence that containment was exercised.
Stale docs URL in a user-facing warning — the PPOTrainer/SACTrainer/DQNTrainer deprecation warning pointed at riserally.github.io/rlox/..., predating the repo transfer. Now the canonical host from mkdocs.yml (wojciechkpl.github.io).
tomllib imported unguarded in shipped code, breaking Python 3.10 — requires-python is >=3.10 but tomllib is stdlib only from 3.11. python/rlox/__main__.py crashed rlox train --config x.toml on 3.10, and benchmarks/agentic/run_benchmark.py::_render_toml raised ModuleNotFoundError inside the prime-rl launcher, where the caller's broad except Exception converted it into a silent "run failed, mean_reward_last=0.0" — so the launcher was dead on 3.10 without ever reporting why. Both now use the tomli-backport fallback already implemented in rlox.config._load_toml (the CLI reuses that helper directly; run_benchmark.py inlines it, since it is stdlib-only by contract and cannot import torch). This surfaced as 58 test failures on the 3.10 job the moment collection was repaired and those tests ran for the first time; reproduced on a real 3.10 interpreter (58 failed → 84 passed).
The target-cpu=native SIGILL guard in CI was a silent no-op — .cargo/config.toml sets build.rustflags = ["-C", "target-cpu=native"], and ci.yml tried to neutralise it for CI with CARGO_BUILD_RUSTFLAGS: "". Cargo treats an empty value for that key as unset and falls back to the config file, so -C target-cpu=native still reached every rustc invocation and the intermittent SIGILL: illegal instruction it was meant to prevent kept recurring (it takes down Rust tests whenever Swatinem/rust-cache serves a proc-macro dylib built on a runner with different CPU features). Verified empirically by running cargo build -v under each spelling: CARGO_BUILD_RUSTFLAGS="" leaves the flag in place; CARGO_ENCODED_RUSTFLAGS="" and RUSTFLAGS="" remove it. All Rust-building workflows now use CARGO_ENCODED_RUSTFLAGS: "", and tests/test_repo_hygiene.py rejects the ineffective spelling plus any Rust-building workflow that sets no effective override.
Published wheels were built with target-cpu=native — wheels.yml had no rustflags override, so PyPI wheels were compiled for the build runner's exact CPU and then distributed as generic manylinux/macosx/win wheels. A wheel built on a runner with, say, AVX-512 crashes with SIGILL on import rlox for any user whose CPU lacks it. Now overridden, with docker-options: -e CARGO_ENCODED_RUSTFLAGS so the setting also reaches the containerised manylinux builds. publish-crates.yml's verification build is covered too. Local builds keep native tuning via .cargo/config.toml — only CI and release artifacts are made portable.
Clippy -D warnings failures in rlox-sandbox — doc_overindented_list_items (×4 across corpus_containment.rs / seccomp_tests.rs), bool_assert_comparison (server_contract.rs), format_in_format_args and useless_vec (backend_stats_serde.rs). These persisted because cargo clippy --workspace cannot compile on macOS at all — rlox-sandbox is Linux-only and its seccompiler dependency fails against macOS libc — so every lint inside #[cfg(target_os = "linux")] was structurally invisible to local runs.
12 sandbox tests failed on CI runners and were masked — cargo test stops at the first failing test binary, so failures in integration_run_sandboxed, rollout_pipeline, security_isolation, server_contract, and verify_endpoint sat hidden behind an earlier one and never appeared in any CI log. All 12 need cgroup v2 self-migration, which shared runners cannot do: run_sandboxed returns SetupError("child could not write to cgroup.procs …"), making every Clean/Timeout/OomKilled assertion vacuous. They are now gated by two documented capability opt-ins in crates/rlox-sandbox/tests/common/mod.rs — RLOX_SANDBOX_CGROUP_TESTS (anything calling run_sandboxed) and RLOX_SANDBOX_ADVERSARIAL_TESTS (real fork/memory/pids bombs) — both exported by scripts/wk-sync-test.sh. Verified on a delegated Linux host: 61 passed / 0 failed with the gates enabled, and 61 passed / 0 failed with 16 explicit skips under CI-like conditions — no coverage lost.
test_torch_not_imported_at_module_level was order-dependent — it asserted on process-global sys.modules, so it reported whether any earlier test had imported torch, not what trl_grpo_run pulls in. Latent while collection was broken; exposed once the suite actually ran. Now probes a fresh interpreter in a subprocess, which is the only way to verify the intended contract.
Docs deploy reported a false failure on feature branches — the github-pages environment only permits deploys from the default branch, so a workflow_dispatch from a branch failed the run after build had already succeeded. The deploy job is now conditional on main; dispatching from a branch still exercises build as a docs smoke test.
Systemic seed no-op across 8 off-policy algorithms — SAC, TD3, MPO, AWR, DecisionTransformer, Cal-QL, QMIX, and DiffusionPolicy accepted a seed parameter but never applied it to any RNG, so runs were non-reproducible (and, for SAC/TD3, the multi-env collector path was silently hardcoded to seed 42 via getattr(self, "seed", 42) regardless of the requested seed — undermining the multi-seed validation methodology). All 8 now seed torch/numpy/env (action-space + first reset) in __init__ before network construction, matching the pqn/crossq precedent. Reproducibility regression tests added for SAC and TD3.
TOML config serialization crashed on None fields — tomli_w (the real TOML writer) raises on None-valued config fields (e.g. SACConfig.target_entropy=None); None values are now stripped before dumping (TOML has no null type; from_dict restores them from dataclass defaults on read). Fixes 4 CI test failures.
AWR predict() — raised AttributeError on every inference call (self.policy.actor referenced a non-existent attribute; the class holds self.actor/self.critic). Now branches on self.discrete and returns an int action (discrete) or a numpy.ndarray (continuous), matching the train() path. Regression tests added.
rlox-grpc dropped terminal observations — RemoteEnvClient hardcoded terminal_obs: vec![None; num_envs], so truncation-bootstrap observations were never transmitted, silently corrupting value targets for distributed IMPALA on truncating environments (e.g. all MuJoCo tasks). The StepResponse now carries terminal_obs (flat, zero-padded) plus a has_terminal_obs mask; the server errors on an obs-dim mismatch instead of silently truncating, and the client reconstructs Vec<Option<Vec<f32>>> (with a backward-compatible fallback against older servers).

[1.2.0] - 2026-05-05¶

Added¶

Trainer.evaluate(n_episodes, seed, render) -- deterministic evaluation returning mean/std/min/max reward and episode lengths
Trainer.enjoy(n_episodes, seed) -- render the trained policy for visual inspection
VideoRecordingCallback -- records evaluation episodes to mp4 at configurable intervals during training
AsymmetricPolicy -- actor-critic with separate observation spaces (actor sees deployment obs, critic sees privileged state). Supports both discrete and continuous actions
Episode statistics tracking -- RolloutCollector and GymVecEnv now expose episode_rewards and episode_lengths properties for completed episodes
RecordEpisodeStatistics auto-wrapping in GymVecEnv
Score normalization -- normalize_score(), normalize_scores(), and SCORE_BASELINES dict for mapping raw returns to [0, 1] using random/expert baselines (14 environments)
Bootstrap CI bands on learning curves in multi_seed_runner.py via --eval-freq flag
EmaRunningStats (Rust + PyO3) -- exponential moving average mean/variance for non-stationary signals. Constructors: EmaRunningStats(alpha), .from_window(N), .from_halflife(h)
CusumDetector (Rust + PyO3) -- two-sided CUSUM change-point detection with optional burn-in period for automatic reference level estimation
PageHinkleyDetector (Rust only) -- Page-Hinkley change-point detection
NonStationaryCartPole (Rust only) -- CartPole with configurable parameter drift (gravity, pole length, cart mass, force magnitude) via DriftMode (None, Linear, Sinusoidal, Step)
ReplayBuffer.sample_recent(batch_size, window_size, seed) -- sliding window replay for non-stationary RL, sampling only from recent transitions
Dynamic regret metrics in evaluation.py: dynamic_regret(), adaptation_latency(), forgetting_ratio() for non-stationary RL evaluation

Fixed¶

CI: override target-cpu=native to prevent SIGILL from stale cache on different hardware

[1.1.0] - 2026-03-29¶

Added¶

VPG algorithm -- Vanilla Policy Gradient with GAE support
Plugin ecosystem -- ENV_REGISTRY, BUFFER_REGISTRY, REWARD_REGISTRY for registering custom components; discover_plugins() for auto-discovery via Python entry points
Model zoo -- ModelZoo.register, ModelZoo.load, ModelCard for sharing and reusing pretrained agents
Visual RL wrappers -- FrameStack, ImagePreprocess, AtariWrapper, DMControlWrapper for pixel-based RL
Language RL wrappers -- LanguageWrapper, GoalConditionedWrapper for language-grounded tasks
Cloud deploy -- generate_dockerfile, generate_k8s_job, generate_sagemaker_config for deployment artifact generation
predict() method added to TRPO, IMPALA, MAPPO, A2C, VPG (all algorithms now support predict())
22 algorithm documentation pages completed

Changed¶

safe_torch_load() -- all checkpoint loading now uses weights_only=True by default for security
VecEnv::new now returns Result<VecEnv, RloxError> instead of panicking on invalid input
Transition.info changed from HashMap<String, f64> to Option<HashMap<String, f64>> (None when no metadata)
PBT (Population-Based Training) is now fully reproducible with seeded RNG
Docker deploy module validates all inputs (checkpoint paths, image names, resource specs) before generating artifacts
12+ core types now derive Debug and Clone for better ergonomics and debuggability

Fixed¶

Checkpoint security: prevented potential arbitrary code execution from untrusted checkpoints via weights_only=True

Test Suite¶

444 Rust tests (was 409)
~1094 Python tests (was 869)

[1.0.0] - 2026-03-29¶

API 1.0 Freeze¶

This release marks the first stable API. All public exports are frozen and covered by stability tests. Semver guarantees apply from this version onward.

Added¶

A2CTrainer -- high-level trainer wrapping A2C with callback/logger integration
TD3Trainer -- high-level trainer wrapping TD3 with callback/logger integration
MAPPOTrainer -- high-level trainer wrapping MAPPO for multi-agent environments
DreamerV3Trainer -- high-level trainer wrapping DreamerV3 world-model-based RL
IMPALATrainer -- high-level trainer wrapping IMPALA actor-learner architecture
All trainers expose train(total_timesteps), save(path), and from_checkpoint(path)
train_from_config dispatch extended to all 8 algorithms: ppo, sac, dqn, a2c, td3, mappo, dreamer, impala
Complete __all__ exports: trainers, configs, protocols, exploration, builders, losses, distributed, dashboard
Distributed components exported at top level: MultiGPUTrainer, RemoteEnvPool, launch_elastic
API stability test suite expanded: all 8 trainers, all 8 configs, distributed symbols, runner dispatch

Changed¶

_VALID_ALGORITHMS in config now includes mappo, dreamer, impala
Runner dispatch uses Trainer wrappers for all algorithms (no more raw algo class fallback)
pyproject.toml classifier updated to Development Status :: 5 - Production/Stable
Version bumped to 1.0.0 in __init__.py and pyproject.toml

[0.3.0] - 2026-03-29¶

Added¶

VecNormalize environment wrapper — obs/reward normalization at the environment boundary (SB3 architecture), replacing collector-level normalization
RunningStatsVec — per-dimension Welford statistics in Rust (PyO3 exposed)
Native Pendulum-v1 — Rust environment with continuous action space
Polymorphic VecEnv.step_all — accepts discrete (Vec) and continuous (ndarray float32) actions
VecEnv.action_space property — typed dict for Python-side detection
VecEnv protocol — formal protocol in protocols.py
A2CConfig, TD3Config — dataclass configs with validation and YAML support
Offline RL: TD3+BC, IQL, CQL, BC algorithms with OfflineDatasetBuffer (Rust)
Candle Hybrid Collection: CandleCollector (180K SPS on CartPole), HybridPPO trainer
OffPolicyCollector: Reusable multi-env collection for SAC, TD3, DQN (n_envs parameter)
OfflineAlgorithm base class with OfflineDataset protocol for extensible offline RL
SharedPolicy + weight sync for Candle/PyTorch interop
RolloutBatch extended with log_probs and values fields
SB3 migration guide at docs/tutorials/migration-sb3.md
API reference pages with mkdocstrings autodoc
CONTRIBUTING.md with development setup and guidelines
Cross-navigation header across all documentation components
Python 3.13 added to CI test matrix

Fixed¶

Truncation bootstrap — truncated episodes now bootstrap V(terminal_obs) instead of treating as deaths (value=0). Critical for MuJoCo time limits.
Per-dimension obs normalization — replaced scalar mean/std with per-dim tracking, preserving observation structure across different scales
Return-based reward normalization — std of discounted returns (SB3 convention) instead of std of raw rewards
Train/collect obs mismatch — consistent normalization during collection and training
A2C advantage normalization default — changed to False, preventing gradient explosion with small batches (n_steps=5, batch=40)
log_std init — 0.0 (std=1.0) matching SB3, was -0.5
GCS upload path — absolute paths in convergence benchmark scripts
IMPALA: V-trace now uses computed bootstrap value instead of hardcoded 0.0
IMPALA: Auto-detects continuous envs, falls back to GymVecEnv for non-CartPole
DreamerV3: World model frozen during actor-critic training (prevents gradient leakage)
DreamerV3: Gradient clipping added to both world model and actor-critic updates
MAPPO: NotImplementedError for n_agents > 1 (prevents silent dimension mismatch)
MAPPO: Simplified critic input for single-agent case

Changed¶

Normalization moved from RolloutCollector to VecNormalize wrapper
PPO auto-wraps env with VecNormalize when normalize flags set
EvalCallback freezes normalization stats during evaluation

Improved¶

Landing page redesigned with quickstart, benchmarks, comparison table, algorithm grid
Rust crate descriptions and lib.rs doc comments updated
80+ new Python tests (convergence fixes, VecNormalize, Pendulum, offline RL)
30+ new Rust tests (RunningStatsVec, Pendulum, OfflineDatasetBuffer)

[0.2.0] - 2026-03-16¶

Added¶

Phase 7: Algorithm Completeness
GymVecEnv wrapper for arbitrary Gymnasium environments (AutoresetMode.SAME_STEP)
ContinuousPolicy (Gaussian, orthogonal init) for on-policy continuous control
BatchSteppable trait for environment abstraction
Auto env detection: PPO/A2C auto-select Discrete/Continuous policy from action space
reward_fn parameter on RolloutCollector for reward shaping
Callbacks wired into all 7 algorithms (PPO, SAC, DQN, TD3, A2C, GRPO, DPO)
save()/from_checkpoint() on PPO, SAC, DQN, TD3, GRPO, DPO
from_yaml()/to_yaml() on PPOConfig, SACConfig, DQNConfig
GRPO batched advantages (eliminates Python loop, uses compute_batch_group_advantages)
Phase 8: Production Hardening
Statistical evaluation toolkit: IQM, bootstrap CI, performance profiles, P(improvement)
TrainingDiagnostics callback: entropy collapse, KL spike, gradient explosion detection
Memory-mapped replay buffer (MmapReplayBuffer) for hot/cold architecture
CI workflows: GitHub Actions for tests + maturin wheel builds (4 platforms)
Experiment metadata capture + save_experiment()
Phase 9: Distributed & Scale
Decoupled collection/training pipeline (crossbeam channels, AsyncCollector)
gRPC distributed env workers (rlox-grpc crate with tonic)
Multi-GPU training composition (PyTorch DDP wrapper)
vLLM, TGI, SGLang inference backends with factory
RemoteEnvPool Python client for gRPC workers
Transition provenance (TransitionMeta with serialize/deserialize)
MAPPO, DreamerV3, IMPALA algorithms with env auto-detection
API 1.0 freeze: comprehensive __all__, stability tests
Buffer Extensions
Typed extra columns (register_column/push_extra) with O(1) ColumnHandle access
Dict observation space (Observation::Dict, ObsSpace::Dict)
BatchDictBuilder for deduplicated PyO3 dict construction
Infrastructure
MIT OR Apache-2.0 dual license
Published to crates.io: rlox-core, rlox-nn, rlox-burn, rlox-candle
Tutorial: custom rewards and training loops (1,480 lines)
Logo and citation info (CITATION.cff)

Fixed¶

Critical: PyVecEnv silently fell back to CartPole for unknown env_ids — now raises ValueError
Critical: Replay buffer missing next_obs — off-policy algorithms (SAC, TD3, DQN) computed wrong Bellman targets
SAC: action scaling now multiplies by act_high (was only clipping)
TD3: critic target updates moved outside policy_delay gate
DQN: n-step flush uses actual termination flags (was hardcoded terminated=True)
Config consolidation: single validated PPOConfig (was duplicated)

Test Suite¶

313 Rust tests at v0.2.0 (was 255)
382 Python tests at v0.2.0 (was 85)
Zero benchmark regressions

[0.1.0] - 2026-03-14¶

Added¶

Phases 0-6: core Rust engine, environment stepping, buffers, GAE, V-trace
LLM post-training: GRPO, DPO, token KL, sequence packing
NN backend abstraction: rlox-nn traits, rlox-burn, rlox-candle
Three-framework benchmark suite (rlox vs TorchRL vs SB3)
Convergence benchmarks (rlox vs SB3 on Classic Control)
255 Rust tests, 85 Python tests