View on GitHub
"```python\nfrom modal_training_gym.common.eval import EvalConfig\n```\n\nEvaluate a deployed model on a dataset config.\n\nThe dataset must expose `load()` and return iterable dict examples.\n\n## Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `dataset` | `'DatasetConfig'` | | |\n| `eval_fn` | `EvalFn \\| None` | `None` | |\n| `eval_response_fn` | `EvalResponseFn \\| None` | `None` | |\n| `prompt_column` | `str \\| None` | `None` | |\n| `eval_config_id` | `str \\| None` | `None` | |\n| `generate_kwargs` | `dict[str, Any]` | `{}` | |\n\n## Methods\n\n### `build_prompt(self, row: 'DatasetRow') -> 'str'`\n\n### `evaluate(self, deployment: \"'ModelDeployment'\", debug: 'bool' = False, max_concurrency: 'int' = 1, ready_timeout: 'int' = 3000) -> 'EvalResult'`\n\n### `save(self) -> 'EvalConfigDurable'`\n\n### `to_durable(self) -> 'EvalConfigDurable'`\n\n## Related Tutorials\n\n- [Qwen3-4B haiku evaluation with verifiable rewards — serve, evaluate, train, compare](/tutorials/rl/000_rl_basics/)\n- [Multi-turn number-guessing RL with custom generate and reward functions](/tutorials/rl/002_multiturn/)\n- [On-policy distillation on math — Qwen3-8B teacher, Qwen3-4B student](/tutorials/rl/003_on_policy_distillation/)\n- [DAPO on math with Qwen3-4B](/tutorials/rl/005_dapo/)\n- [Audio GRPO on Qwen3-ASR-1.7B — transcribe LibriSpeech, reward −WER](/tutorials/rl/006_audio_asr/)\n- [GUI grounding with Qwen3-VL-8B — predict click coordinates from screenshots](/tutorials/rl/008_computer_use/)\n- [Cross-tokenizer agentic distillation on BFCL v3 multi-turn with live, execution-grounded rewards — DeepSeek V4 Flash teacher, Qwen3.6-35B-A3B student](/tutorials/rl/009_cross_tokenizer_distillation/)\n\n**Source:** [`modal_training_gym/common/eval.py`](https://github.com/modal-projects/training-gym/blob/main/modal_training_gym/common/eval.py)\n"
EvalConfig
Evaluate a deployed model on a dataset config.
from modal_training_gym.common.eval import EvalConfigEvaluate a deployed model on a dataset config.
The dataset must expose load() and return iterable dict examples.
Fields
Section titled “Fields”| Field | Type | Default | Description |
|---|---|---|---|
dataset | 'DatasetConfig' | ||
eval_fn | EvalFn | None | None | |
eval_response_fn | EvalResponseFn | None | None | |
prompt_column | str | None | None | |
eval_config_id | str | None | None | |
generate_kwargs | dict[str, Any] | {} |
Methods
Section titled “Methods”build_prompt(self, row: 'DatasetRow') -> 'str'
Section titled “build_prompt(self, row: 'DatasetRow') -> 'str'”evaluate(self, deployment: "'ModelDeployment'", debug: 'bool' = False, max_concurrency: 'int' = 1, ready_timeout: 'int' = 3000) -> 'EvalResult'
Section titled “evaluate(self, deployment: "'ModelDeployment'", debug: 'bool' = False, max_concurrency: 'int' = 1, ready_timeout: 'int' = 3000) -> 'EvalResult'”save(self) -> 'EvalConfigDurable'
Section titled “save(self) -> 'EvalConfigDurable'”to_durable(self) -> 'EvalConfigDurable'
Section titled “to_durable(self) -> 'EvalConfigDurable'”Related Tutorials
Section titled “Related Tutorials”- Qwen3-4B haiku evaluation with verifiable rewards — serve, evaluate, train, compare
- Multi-turn number-guessing RL with custom generate and reward functions
- On-policy distillation on math — Qwen3-8B teacher, Qwen3-4B student
- DAPO on math with Qwen3-4B
- Audio GRPO on Qwen3-ASR-1.7B — transcribe LibriSpeech, reward −WER
- GUI grounding with Qwen3-VL-8B — predict click coordinates from screenshots
- Cross-tokenizer agentic distillation on BFCL v3 multi-turn with live, execution-grounded rewards — DeepSeek V4 Flash teacher, Qwen3.6-35B-A3B student