"```python\nfrom modal_training_gym.common.eval import HarborEval\n```\n\nEvaluate a deployed model on a Harbor dataset using sandbox execution.\n\nAutomates the common pattern of generating code from a Harbor task,\nextracting it from the LLM response, running it in a Modal sandbox,\nand comparing stdout against expected test-case outputs.\n\nWhen neither `eval_fn` nor `eval_response_fn` is provided, a\ndefault sandbox-backed scorer is used automatically. Pass\n`extract_code_fn` to override how code is pulled from the model\nresponse, or supply your own `eval_fn` to take full control.\n\n**Inherits from:** `EvalConfig`\n\n## Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `dataset` | `'DatasetConfig'` | | |\n| `eval_fn` | `EvalFn \\| None` | `None` | |\n| `eval_response_fn` | `EvalResponseFn \\| None` | `None` | |\n| `prompt_column` | `str \\| None` | `None` | |\n| `eval_config_id` | `str \\| None` | `None` | |\n| `generate_kwargs` | `dict[str, Any]` | `{}` | |\n| `model` | `'ModelConfig \\| None'` | `None` | |\n| `test_cases` | `list[dict[str, str]] \\| None` | `None` | |\n| `sandbox_timeout` | `int` | `60` | |\n| `sandbox_cpu` | `float` | `1.0` | |\n| `sandbox_memory` | `int` | `1024` | |\n| `sandbox_cpu_policy` | `str` | `\"limit\"` | |\n| `sandbox_memory_policy` | `str` | `\"limit\"` | |\n| `sandbox_python_version` | `str` | `\"3.11\"` | |\n| `extract_code_fn` | `Callable[[str], str] \\| None` | `None` | |\n\n## Methods\n\n### `build_prompt(self, row: 'DatasetRow') -> 'str'`\n\n### `evaluate(self, deployment: \"'ModelDeployment'\", debug: 'bool' = False, max_concurrency: 'int' = 1, ready_timeout: 'int' = 3000) -> 'EvalResult'`\n\n### `save(self) -> 'EvalConfigDurable'`\n\n### `to_durable(self) -> 'EvalConfigDurable'`\n\n## Related Tutorials\n\n- [Code RL with Harbor hello-world and sandboxed verification](/tutorials/rl/001_sandboxes/)\n\n**Source:** [`modal_training_gym/common/eval.py`](https://github.com/modal-projects/training-gym/blob/main/modal_training_gym/common/eval.py)\n"

HarborEval

Evaluate a deployed model on a Harbor dataset using sandbox execution.

from modal_training_gym.common.eval import HarborEval

Evaluate a deployed model on a Harbor dataset using sandbox execution.

Automates the common pattern of generating code from a Harbor task, extracting it from the LLM response, running it in a Modal sandbox, and comparing stdout against expected test-case outputs.

When neither eval_fn nor eval_response_fn is provided, a default sandbox-backed scorer is used automatically. Pass extract_code_fn to override how code is pulled from the model response, or supply your own eval_fn to take full control.

Inherits from: EvalConfig

Fields

Field	Type	Default
`dataset`	`'DatasetConfig'`
`eval_fn`	`EvalFn \| None`	`None`
`eval_response_fn`	`EvalResponseFn \| None`	`None`
`prompt_column`	`str \| None`	`None`
`eval_config_id`	`str \| None`	`None`
`generate_kwargs`	`dict[str, Any]`	`{}`
`model`	`'ModelConfig \| None'`	`None`
`test_cases`	`list[dict[str, str]] \| None`	`None`
`sandbox_timeout`	`int`	`60`
`sandbox_cpu`	`float`	`1.0`
`sandbox_memory`	`int`	`1024`
`sandbox_cpu_policy`	`str`	`"limit"`
`sandbox_memory_policy`	`str`	`"limit"`
`sandbox_python_version`	`str`	`"3.11"`
`extract_code_fn`	`Callable[[str], str] \| None`	`None`

Methods

`build_prompt(self, row: 'DatasetRow') -> 'str'`

`evaluate(self, deployment: "'ModelDeployment'", debug: 'bool' = False, max_concurrency: 'int' = 1, ready_timeout: 'int' = 3000) -> 'EvalResult'`

`save(self) -> 'EvalConfigDurable'`

`to_durable(self) -> 'EvalConfigDurable'`

Code RL with Harbor hello-world and sandboxed verification

Source: modal_training_gym/common/eval.py