Config API

Configuration dataclasses for all algorithms. Every config class provides from_dict(), merge(), and to_dict() helpers, plus TOML/YAML loading via TrainingConfig.

Base

rlox.config.ConfigMixin

Shared serialization/deserialization for all config dataclasses.

Provides from_dict, from_yaml, from_toml, merge, to_dict, to_yaml, and to_toml so individual config classes only need to declare their fields and __post_init__ validation.

Unified

rlox.config.TrainingConfig dataclass

Top-level config for YAML/TOML-driven training.

Combines algorithm selection, environment, seed, hyperparameters, callbacks, and logging into a single serializable config.

from_dict(d: dict[str, Any]) -> TrainingConfig classmethod

Construct from a dict, ignoring unknown keys.

from_yaml(path: str | Path) -> TrainingConfig classmethod

Load from a YAML file.

from_toml(path: str | Path) -> TrainingConfig classmethod

Load from a TOML file.

to_yaml(path: str | Path) -> None

Save to a YAML file.

to_toml(path: str | Path) -> None

Save to a TOML file.

On-Policy

rlox.config.PPOConfig dataclass

Bases: ConfigMixin

Configuration for PPO training.

Defaults match CleanRL's PPO implementation for CartPole-v1.

Attributes

n_envs : int Number of parallel environments (default 8). n_steps : int Rollout length per environment per update (default 128). n_epochs : int Number of SGD passes over each rollout (default 4). batch_size : int Minibatch size for SGD (default 256). learning_rate : float Adam learning rate (default 2.5e-4). clip_eps : float PPO clipping range for the probability ratio (default 0.2). vf_coef : float Value loss coefficient (default 0.5). ent_coef : float Entropy bonus coefficient (default 0.01). max_grad_norm : float Maximum gradient norm for clipping (default 0.5). gamma : float Discount factor (default 0.99). gae_lambda : float GAE lambda (default 0.95). normalize_advantages : bool Whether to normalise advantages per minibatch (default True). clip_vloss : bool Whether to clip the value loss (default True). anneal_lr : bool Whether to linearly anneal the learning rate (default True).

rlox.config.A2CConfig dataclass

Bases: ConfigMixin

Configuration for A2C (Advantage Actor-Critic) training.

A2C uses a single gradient step per rollout (no clipping, no epochs). Typically paired with short rollouts (n_steps=5) and RMSprop.

Attributes

learning_rate : float RMSprop learning rate (default 7e-4). n_steps : int Rollout length per update (default 5). gamma : float Discount factor (default 0.99). gae_lambda : float GAE lambda (default 1.0, equivalent to full n-step returns). vf_coef : float Value function loss coefficient (default 0.5). ent_coef : float Entropy bonus coefficient (default 0.01). max_grad_norm : float Gradient clipping threshold (default 0.5). normalize_advantages : bool Normalize advantages per batch (default False). n_envs : int Number of parallel environments (default 8). hidden : int Hidden layer width (default 64).

rlox.config.VPGConfig dataclass

Bases: ConfigMixin

Configuration for Vanilla Policy Gradient (REINFORCE with baseline).

Attributes

learning_rate : float Policy optimizer learning rate (default 3e-4). vf_lr : float Value function learning rate (default 1e-3). n_envs : int Number of parallel environments (default 8). n_steps : int Rollout length per environment per update (default 2048). gamma : float Discount factor (default 0.99). gae_lambda : float GAE lambda for bias-variance tradeoff (default 0.97). vf_epochs : int Value function SGD epochs per rollout (default 5). max_grad_norm : float Maximum gradient norm for clipping (default 0.5). ent_coef : float Entropy bonus coefficient (default 0.0 -- no entropy bonus). hidden : int Hidden layer size for auto-created policies (default 64).

rlox.config.TRPOConfig dataclass

Bases: ConfigMixin

Configuration for Trust Region Policy Optimization.

Attributes

max_kl : float Maximum KL divergence per update (default 0.01). damping : float Damping coefficient for Fisher vector product (default 0.1). cg_iters : int Conjugate gradient iterations (default 10). line_search_steps : int Backtracking line search steps (default 10). n_envs : int Number of parallel environments (default 8). n_steps : int Rollout length per environment per update (default 2048). gamma : float Discount factor (default 0.99). gae_lambda : float GAE lambda (default 0.97). vf_lr : float Value function learning rate (default 1e-3). vf_epochs : int Value function SGD epochs per update (default 5).

Off-Policy

rlox.config.SACConfig dataclass

Bases: ConfigMixin

Configuration for SAC training.

Defaults match rl-zoo3 SAC hyperparameters.

Attributes

learning_rate : float Learning rate for all optimisers (default 3e-4). buffer_size : int Replay buffer capacity (default 1M). batch_size : int Minibatch size for critic/actor updates (default 256). tau : float Polyak averaging coefficient for target networks (default 0.005). gamma : float Discount factor (default 0.99). target_entropy : float or None Target entropy for auto-tuning. None = -dim(action_space). auto_entropy : bool Whether to automatically tune the entropy coefficient (default True). learning_starts : int Number of random exploration steps before training (default 1000). hidden : int Hidden layer width for actor and critic networks (default 256).

from_toml(path: str | Path) -> SACConfig classmethod

Load config from a TOML file, ignoring unknown keys.

rlox.config.TD3Config dataclass

Bases: ConfigMixin

Configuration for TD3 (Twin Delayed DDPG) training.

Deterministic policy with target policy smoothing and delayed updates.

Attributes

learning_rate : float Adam learning rate for both actor and critic (default 3e-4). buffer_size : int Replay buffer capacity (default 1M). batch_size : int Minibatch size (default 256). tau : float Polyak averaging coefficient for target networks (default 0.005). gamma : float Discount factor (default 0.99). learning_starts : int Random exploration steps before training (default 1000). policy_delay : int Actor update frequency relative to critic (default 2). target_noise : float Noise added to target actions for smoothing (default 0.2). noise_clip : float Clipping range for target noise (default 0.5). exploration_noise : float Std of Gaussian exploration noise (default 0.1). hidden : int Hidden layer width (default 256). n_envs : int Number of parallel environments (default 1).

rlox.config.DQNConfig dataclass

Bases: ConfigMixin

Configuration for DQN training with Rainbow extensions.

Supports Double DQN, Dueling architecture, N-step returns, and Prioritized Experience Replay (PER).

Attributes

learning_rate : float Adam learning rate (default 1e-4). buffer_size : int Replay buffer capacity (default 1M). batch_size : int Minibatch size (default 64). gamma : float Discount factor (default 0.99). target_update_freq : int Steps between hard target network updates (default 1000). exploration_fraction : float Fraction of training for epsilon decay (default 0.1). exploration_initial_eps : float Starting epsilon for exploration (default 1.0). exploration_final_eps : float Final epsilon after decay (default 0.05). learning_starts : int Random exploration steps before training (default 1000). double_dqn : bool Use Double DQN action selection (default True). dueling : bool Use Dueling network architecture (default False). n_step : int N-step return horizon (default 1). prioritized : bool Use Prioritized Experience Replay (default False). alpha : float PER priority exponent (default 0.6). beta_start : float PER initial importance-sampling exponent (default 0.4). hidden : int Hidden layer width (default 256).

rlox.config.MPOConfig dataclass

Bases: ConfigMixin

Configuration for Maximum a Posteriori Policy Optimization.

Attributes

learning_rate : float Adam learning rate for actor and critic (default 3e-4). buffer_size : int Replay buffer capacity (default 1_000_000). batch_size : int Minibatch size (default 256). gamma : float Discount factor (default 0.99). tau : float Polyak averaging coefficient for target networks (default 0.005). n_action_samples : int Number of action samples for the E-step (default 20). epsilon : float KL constraint for the M-step (default 0.1). epsilon_penalty : float KL penalty coefficient (default 0.001). dual_lr : float Learning rate for dual variables (default 1e-2). hidden : int Hidden layer width (default 256). learning_starts : int Random exploration steps before training (default 1000).

Multi-Agent

rlox.config.MAPPOConfig dataclass

Bases: ConfigMixin

Configuration for Multi-Agent PPO (MAPPO) training.

Centralized training with decentralized execution (CTDE).

Attributes

n_agents : int Number of agents (default 2). learning_rate : float Adam learning rate (default 5e-4). n_steps : int Rollout length per environment per update (default 128). n_epochs : int Number of SGD passes per rollout (default 5). clip_range : float PPO clipping range (default 0.2). gamma : float Discount factor (default 0.99). gae_lambda : float GAE lambda (default 0.95). vf_coef : float Value loss coefficient (default 0.5). ent_coef : float Entropy bonus coefficient (default 0.01). max_grad_norm : float Maximum gradient norm for clipping (default 10.0). share_parameters : bool Whether agents share actor parameters (default False). hidden : int Hidden layer width (default 64). n_envs : int Number of parallel environments (default 8).

rlox.config.QMIXConfig dataclass

Bases: ConfigMixin

Configuration for QMIX training.

Attributes

n_agents : int Number of agents (default 3). hidden_dim : int Hidden dimension for agent Q-networks (default 64). mixing_embed_dim : int Mixing network hidden dimension (default 32). lr : float Adam learning rate (default 5e-4). buffer_size : int Replay buffer capacity (default 50_000). gamma : float Discount factor (default 0.99). target_update_freq : int Steps between target network updates (default 200). batch_size : int Minibatch size (default 32).

Model-Based

rlox.config.DreamerV3Config dataclass

Bases: ConfigMixin

Configuration for DreamerV3 (world-model-based RL) training.

Attributes

learning_rate : float Learning rate for all optimisers (default 1e-4). buffer_size : int Replay buffer capacity (default 1M). batch_size : int Number of sequences per training batch (default 16). seq_len : int Sequence length for training (default 50). gamma : float Discount factor (default 0.997). lambda_ : float Lambda for lambda-returns (default 0.95). deter_dim : int Deterministic state dimension in RSSM (default 512). stoch_dim : int Stochastic state dimension (number of categoricals, default 32). stoch_classes : int Number of classes per categorical (default 32). hidden : int Hidden layer width (default 512). imagination_horizon : int Steps to imagine ahead for actor-critic (default 15). kl_balance : float KL balancing coefficient (default 0.8). free_nats : float Free nats for KL loss (default 1.0).

Distributed

rlox.config.IMPALAConfig dataclass

Bases: ConfigMixin

Configuration for IMPALA (Importance Weighted Actor-Learner Architecture).

Attributes

learning_rate : float RMSprop learning rate (default 4e-4). n_actors : int Number of actor threads (default 4). n_steps : int Rollout length per actor per batch (default 20). gamma : float Discount factor (default 0.99). vf_coef : float Value loss coefficient (default 0.5). ent_coef : float Entropy bonus coefficient (default 0.01). max_grad_norm : float Maximum gradient norm for clipping (default 40.0). rho_clip : float V-trace truncation for importance weights (default 1.0). c_clip : float V-trace truncation for trace coefficients (default 1.0). queue_size : int Maximum experience queue size (default 16). hidden : int Hidden layer width (default 256). n_envs_per_actor : int Number of environments per actor thread (default 1).

Offline RL

rlox.config.DecisionTransformerConfig dataclass

Bases: ConfigMixin

Configuration for Decision Transformer training.

Attributes

context_length : int Number of timesteps in the context window (default 20). n_heads : int Number of attention heads (default 4). n_layers : int Number of transformer layers (default 3). embed_dim : int Embedding dimension (default 128). learning_rate : float Adam learning rate (default 1e-4). batch_size : int Minibatch size for offline training (default 64). dropout : float Dropout rate (default 0.1). target_return : float Desired return for evaluation (default 200.0). warmup_steps : int Data collection steps before training (default 500).

rlox.config.CalQLConfig dataclass

Bases: ConfigMixin

Configuration for Cal-QL (Calibrated Conservative Q-Learning).

Attributes

learning_rate : float Learning rate for all optimisers (default 3e-4). buffer_size : int Replay buffer capacity (default 100_000). batch_size : int Minibatch size (default 256). gamma : float Discount factor (default 0.99). tau : float Polyak averaging coefficient (default 0.005). cql_alpha : float CQL penalty weight (default 5.0). calibration_tau : float Quantile for calibration threshold (default 0.5). auto_alpha : bool Whether to auto-tune cql_alpha (default False).

Exploration & Meta

rlox.config.DiffusionPolicyConfig dataclass

Bases: ConfigMixin

Configuration for Diffusion Policy training.

Attributes

n_diffusion_steps : int Number of diffusion timesteps T (default 50). action_horizon : int Number of future actions to predict (default 8). obs_horizon : int Number of past observations to condition on (default 2). hidden_dim : int Hidden layer width (default 256). learning_rate : float Adam learning rate (default 1e-4). batch_size : int Minibatch size (default 256). noise_schedule : str "cosine" or "linear" (default "cosine"). beta_start : float Starting beta for linear schedule (default 0.0001). beta_end : float Ending beta for linear schedule (default 0.02). buffer_size : int Replay buffer capacity (default 1_000_000). n_inference_steps : int Denoising steps at inference (default 10).

rlox.config.DTPConfig dataclass

Bases: ConfigMixin

Configuration for Decision Tree Policies (RWDTP / RCDTP).

Implements Koirala & Fleming, "Solving Offline Reinforcement Learning with Decision Tree Regression," arXiv:2401.11630, 2024.

Attributes

method : str Which framework to use: "rwdtp" (return-weighted) or "rcdtp" (return-conditioned). Default "rwdtp". gamma : float Discount factor for return computation (default 1.0). return_power : float Exponent p applied to normalised returns in RWDTP (default 1.0). n_trees : int Number of boosting rounds / weak policies (default 500). max_depth : int Maximum depth per tree (default 6). learning_rate_xgb : float XGBoost shrinkage / step-size (default 0.1). target_return : float or None RCDTP runtime target return. If None, uses the dataset maximum. subsample : float Row subsampling ratio per boosting round (default 1.0). colsample_bytree : float Column subsampling ratio per tree (default 1.0). reg_alpha : float L1 regularization on leaf weights (default 0.0). reg_lambda : float L2 regularization on leaf weights (default 1.0).

rlox.config.SelfPlayConfig dataclass

Bases: ConfigMixin

Configuration for Self-Play training.

Attributes

pool_size : int Maximum number of historical policy snapshots (default 20). snapshot_freq : int Steps between policy snapshots (default 10_000). matchmaking : str Opponent selection strategy: "uniform", "latest", or "elo" (default "uniform"). initial_elo : float Starting Elo rating for new pool entries (default 1000.0). elo_k : float Elo K-factor controlling rating volatility (default 32.0). opponent_fraction : float Fraction of games played against pool opponents vs self (default 0.8).

rlox.config.GoExploreConfig dataclass

Bases: ConfigMixin

Configuration for Go-Explore archive-based exploration.

Attributes

archive_size : int Maximum number of cells in the archive (default 100_000). cell_resolution : int Discretization granularity for observations (default 8). exploration_steps : int Random exploration steps from each archived state (default 100). score_weight : float Weight for score when selecting cells (default 0.3). novelty_weight : float Weight for novelty (inverse visit count) when selecting cells (default 0.7).

rlox.config.PBTConfig dataclass

Bases: ConfigMixin

Configuration for Population-Based Training.

Attributes

population_size : int Number of agents in the population (default 8). interval : int Training timesteps between exploit/explore cycles (default 10_000). n_iterations : int Number of PBT iterations (default 20). exploit_fraction : float Bottom fraction of population replaced each cycle (default 0.2). perturb_factor : float Hyperparameter perturbation range (default 0.2).