When integrating LLM providers/models, derive capability flags and limits from canonical model metadata (catalog) and apply deterministic compatibility rules with explicit fallbacks—avoid brittle name-based heuristics and keep user overrides authoritative.
Apply these practices:
context_limit/max_tokens from a catalog, override only when the user/config limit is unset or the documented sentinel; cap max_tokens to the effective context (user override wins).output_tokens where needed) so downstream logic and cost accounting remain consistent.Example pattern (resolve limits with canonical fallback while honoring user overrides):
pub fn with_catalog_provider_fallback(mut self, catalog_provider_id: &str) -> Self {
let canonical = maybe_get_canonical_model(catalog_provider_id, &self.model_name);
if let Some(c) = canonical {
// only override when user didn’t set a real limit
if self.context_limit.is_none() || self.context_limit == Some(0) {
self.context_limit = Some(c.limit.context);
}
// cap to effective context (not the raw catalog context)
if self.max_tokens.is_none() {
let effective_ctx = self.context_limit.unwrap_or(c.limit.context);
self.max_tokens = c.limit.output
.filter(|&out| out < effective_ctx)
.map(|out| out as i32);
}
}
self
}
Enter the URL of a public GitHub repository