HuggingFaceDataset

from modal_training_gym.common.dataset import HuggingFaceDataset

A dataset loaded from a Hugging Face datasets repository.

Attributes

dataset_id str

Dataset ID. Default: ""

input_key str

Prompt column name. Default: ""

label_key str

Ground-truth column name. Default: "label"

output_format str

On-disk format written by prepare(), either parquet or jsonl. Default: "parquet"

apply_chat_template bool

Apply the model's chat template. Default: True

always_prepare bool

Rerun prepare() when the output path exists. Default: False

writes_eval_paths bool

Whether prepare() must materialize every eval_paths entry. Default: True

hf_repo str

Hugging Face dataset repository ID. Default: ""

hf_split str

Source dataset split. Default: "train"

hf_config str | None

Source dataset configuration name.

input_column str

Source prompt column. Default: ""

output_column str

Source answer column. Default: ""

system_prompt str

System message added to formatted examples. Default: ""

prompt_template str

Template applied to each source prompt. Default: "{input}"

n_rows int

Maximum number of source rows to load; zero loads all rows. Default: 0

load

load(split: Literal['all', 'train', 'eval'] = 'all') -> Any

Load raw examples, optionally filtered by split.

Returns

Raw examples for split.

name

name: str

prepare

prepare(path: str, eval_paths: dict[str, str] | None = None) -> None

Materialize training data at path and evaluation data at eval_paths.

to_pandas

to_pandas(*, formatted: bool = False)

validate_prepared

validate_prepared(path: str) -> None

Validate the prepared file format and required columns.