CustomDeployment

from modal_training_gym.common.deployment import CustomDeployment

A model deployed with an SGLang or vLLM recipe.

Attributes

deployment_id str

Stable deployment ID.

model ModelConfig

Served model configuration.

recipe VllmRecipe | SglangRecipe | None

Serving recipe.

app_name str

Modal app name.

served_model_name str

Model name exposed to clients.

unauthenticated bool

Whether the endpoint accepts unauthenticated requests. Default: True

modal_app_id str

Modal app ID. Default: ""

modal_app_url str

Dashboard URL. Default: ""

url str

Endpoint URL.

chat

chat(messages: list[dict], timeout: int = 120, max_attempts: int = 4, **extra) -> dict

Send one OpenAI-compatible chat-completion request.

Parameters

messages list[dict]

OpenAI-compatible chat messages.

timeout int

Timeout in seconds for each request. Default: 120

max_attempts int

Maximum request attempts for transient failures. Default: 4

**extra

Additional Chat Completions request fields.

Returns

The assistant message dict.

generate

generate(prompt: str | list[dict], **kwargs) -> str

Send a chat-completion request.

Returns

The assistant text, or reasoning_content when content is None.

launch

launch(model: ModelConfig | str, checkpoint: Checkpoint | None = None, *, recipe: VllmRecipe | SglangRecipe | None = None, app_name: str | None = None, served_model_name: str | None = None, unauthenticated: bool = True) -> CustomDeployment

Deploy model with an SGLang or vLLM serving recipe.

Parameters

model ModelConfig | str

Model configuration or Hugging Face model name.

checkpoint Checkpoint | None

Training checkpoint to convert and serve.

recipe VllmRecipe | SglangRecipe | None

SGLang or vLLM serving recipe.

app_name str | None

Modal app name derived from model when omitted.

served_model_name str | None

Model name exposed to clients.

unauthenticated bool

Whether the endpoint accepts requests without proxy credentials. Default: True

Returns

The deployed CustomDeployment handle.

wait_until_ready

wait_until_ready(timeout: int = 600) -> None

Wait until the deployment can serve traffic.

Parameters

timeout int

Maximum number of seconds to wait. Default: 600