When integrating LLM providers, don’t assume the provider’s model spec equals what the specific API endpoint will actually accept. Enforce two checks in your model configuration: (1) set context_limit to a safe upper bound that matches the endpoint’s effective cap, and (2) restrict model selection to a curated allowlist of chat-capable models instead of auto-loading the full /v1/models catalog (which can include unsupported modalities).
Practical application:
context_limit accordingly or keep a conservative cap until confirmed otherwise.dynamic_models to false and maintain an explicit list of supported chat models.Example (pattern):
{
"api_key_env": "PROVIDER_API_KEY",
"base_url": "https://api.provider.com/v1/chat/completions",
"dynamic_models": false,
"models": [
{ "name": "Chat-Model-A", "context_limit": 200000 },
{ "name": "Chat-Model-B", "context_limit": 64000 }
]
}