Validate model limits

When integrating LLM providers, don’t assume the provider’s model spec equals what the specific API endpoint will actually accept. Enforce two checks in your model configuration: (1) set `context_limit` to a safe upper bound that matches the endpoint’s effective cap, and (2) restrict model selection to a curated allowlist of chat-capable models instead of...

copy reviewer prompt

Prompt

Reviewer Prompt
Deploy Agent

When integrating LLM providers, don’t assume the provider’s model spec equals what the specific API endpoint will actually accept. Enforce two checks in your model configuration: (1) set context_limit to a safe upper bound that matches the endpoint’s effective cap, and (2) restrict model selection to a curated allowlist of chat-capable models instead of auto-loading the full /v1/models catalog (which can include unsupported modalities).

Practical application:

  • For each provider/model entry, verify the endpoint-enforced context cap (not just vendor docs/specs). If there’s evidence the endpoint rejects above (e.g., 200k vs. 1M), set context_limit accordingly or keep a conservative cap until confirmed otherwise.
  • For OpenAI-compatible gateways, set dynamic_models to false and maintain an explicit list of supported chat models.

Example (pattern):

{
  "api_key_env": "PROVIDER_API_KEY",
  "base_url": "https://api.provider.com/v1/chat/completions",
  "dynamic_models": false,
  "models": [
    { "name": "Chat-Model-A", "context_limit": 200000 },
    { "name": "Chat-Model-B", "context_limit": 64000 }
  ]
}

Source discussions