Training Gym

Modal Training Gym is an open-source Python SDK for RL post-training on Modal.

The library takes care of infrastructure concerns such as cluster topology, Ray/NCCL bring-up, volume mounts, checkpointing, and serving for eval and rollouts, so you (or your agent!) can focus on training models.

To get started, you can go through the Quickstart section below, or paste the prompt below into your favorite agent:

Install the `training-gym` library. Then, install the skill bundle into the local project with `training-gym skills install`. Finally, tell the user that they can ask you to do something like: "Walk through tutorials/rl_basics.py and train Qwen3.5-4B to write 5-7-5 haikus from statworx/haiku."

Quickstart

Requirements:

• Python 3.12

Install the package directly:

uv pip install -q git+https://github.com/modal-projects/training-gym.git@main

Or pin it in pyproject.toml:

training-gym = { git = "https://github.com/modal-projects/training-gym.git", branch = "main" }

Authenticate with Modal:

modal setup

Set up the dashboard:

training-gym setup

And empower your agents with the Gym’s skill bundle:

training-gym skills install

Then, it’s as easy as:

import re

from modal_training_gym import (
    HuggingFaceDataset,
    Qwen3_5_4B,
    Qwen3_5_4B_Recipe,
    TrainConfig,
)

model = Qwen3_5_4B()


async def gsm8k_rm(args, sample, **kwargs) -> float:
    text = model.parse_response(sample.response or "").content
    boxed = re.findall(r"\\boxed\{([^}]+)\}", text)
    pred = boxed[-1] if boxed else (re.findall(r"-?[\d,]+(?:\.\d+)?", text) or [""])[-1]
    try:
        return float(float(pred.replace(",", "")) == float(sample.label))
    except ValueError:
        return 0.0


config = TrainConfig(
    model=model,
    dataset=HuggingFaceDataset(
        hf_repo="skrishna/gsm8k_only_answer",
        hf_split="train[:120]",
        input_column="text",
        output_column="label",
        input_format="text",
    ),
    recipe=Qwen3_5_4B_Recipe(
        custom_rm_function=gsm8k_rm,
    ),
)
run = config.launch()
print(run.training_run_id)

For a step-by-step walkthrough, see the Getting started tutorial.

Supported models

Family	Models
DeepSeek	DeepSeek-V4.1-Flash
GLM	GLM-4.7
Gemma	gemma-4-26B-A4B-it
Inkling	Inkling-Small
Moonlight	Moonlight-16B-A3B-Instruct
Qwen3	Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-ASR-1.7B Qwen3-VL-8B-Instruct
Qwen3.5	Qwen3.5-0.8B Qwen3.5-2B Qwen3.5-4B Qwen3.5-9B
Qwen3.6	Qwen3.6-27B Qwen3.6-35B-A3B
Qwen3.8	Qwen3.8-27B