Awesome Reviewers

LLM Serving & Gateways

Inference servers, model gateways and routers, KV caching, retrieval and ingestion pipelines.

335 instructions from 15 repositories. Last updated 2026-07-29.

Consistent Authorization and Auth

When changing or adding security-sensitive functionality, keep authorization semantics stable and use project-wide shared validation/auth plumbing.

Apply this standard: 1) Preserve existing access contracts

Example pattern (authorization contract + validation) inspired by the discussions:

// Preserve legacy behavior by only validating/applying role when role_id exists.
roleID := invitationRow.RoleID
if roleID != nil {
    if _, err := client.Role.Query().Where(
        role.IDEQ(*roleID),
        role.ProjectIDEQ(invitationRow.ProjectID),
    ).Only(ctx); err != nil {
        return fmt.Errorf("invitation role is no longer available")
    }
}
// Otherwise: keep null/legacy semantics to avoid privilege escalation.

Net effect: fewer privilege-escalation surprises, fewer auth-header mismatches, and consistent security behavior across the codebase.

Immutable Snapshots And Gates

When concurrency crosses subsystem boundaries (HTTP handlers, streaming, background workers, plugin hooks), treat shared state as immutable and coordinate lifecycle explicitly.

Coding standards 1) Snapshot/clone before publishing

2) Hold locks only for state access; release before blocking/calling out

3) Make streaming/background lifecycle coordination explicit

Example pattern (immutable config swap)

type CorsMiddleware struct {
  cfg atomic.Pointer[MyConfigSnapshot]
}

type MyConfigSnapshot struct {
  AllowedOrigins []string
  AllowedHeaders []string
}

func NewCorsMiddleware(cfg *MyConfig) *CorsMiddleware {
  m := &CorsMiddleware{}
  snap := &MyConfigSnapshot{
    AllowedOrigins: slices.Clone(cfg.AllowedOrigins),
    AllowedHeaders: slices.Clone(cfg.AllowedHeaders),
  }
  m.cfg.Store(snap)
  return m
}

func (m *CorsMiddleware) UpdateConfig(cfg *MyConfig) {
  snap := &MyConfigSnapshot{
    AllowedOrigins: slices.Clone(cfg.AllowedOrigins),
    AllowedHeaders: slices.Clone(cfg.AllowedHeaders),
  }
  m.cfg.Store(snap)
}

Review checklist

Deterministic DB Semantics

Make database changes (queries, migrations, and tests) explicit about shape and semantics: don’t rely on driver defaults, cached plan inference, or “looks right” reads when the DB can mask the truth.

Apply this as a checklist:

1) Prove correctness with real row effects

2) Migrations must use explicit projections and live-schema guards

3) Index/DDL changes should be concurrency- and transaction-safe

4) Update paths must match the monotonic invariants

5) Deterministic batching

Example (deterministic batch delete with correct counting):

func (s *ClickHouseLogStore) DeleteLogsBatch(ctx context.Context, cutoff time.Time, batchSize int) (int64, error) {
  var ids []string
  if err := s.db.WithContext(ctx).
    Model(&Log{}).
    Select("id").
    Where("created_at < ?", cutoff).
    Order("created_at ASC").
    Limit(batchSize).
    Pluck("id", &ids).Error; err != nil {
    return 0, err
  }

  res := s.db.WithContext(ctx).Where("id IN (?)", ids).Delete(&Log{})
  _ = res // affected rows may be driver-rewritten; rely on len(ids)
  return int64(len(ids)), res.Error
}

Outcome: DB code and migrations become robust to schema drift, driver quirks, and boundary/dedup semantics—so correctness is both provable and repeatable across environments.

Null semantics and safety

Define and enforce null/optional semantics consistently: (1) treat nil/zero/empty/absent as distinct states, (2) clear stale state when nil means “remove”, (3) never rely on plain interfaceValue != nil when the underlying value may be a typed-nil pointer, and (4) when using omitempty, only set pointer fields when the value is actually present.

Practical rules:

Example patterns:

1) Clear context key on nil

func setCatalogOnContext(ctx *BifrostContext, catalog ModelInfoProvider) {
  if ctx == nil { return }
  if catalog == nil {
    ctx.ClearValue(BifrostContextKeyModelCatalog) // nil means remove
    return
  }
  ctx.SetValue(BifrostContextKeyModelCatalog, catalog)
}

2) Reject typed-nil in interface{}

// ctx.Value(...) returns interface{}
if ctx != nil {
  if ov, ok := ctx.Value(BifrostContextKeyProviderOverride).(*ProviderOverride); ok && ov != nil {
    if ov.Key != nil { /* bypass key-pool */ }
  }
}

3) omitempty pointer only when non-empty

// server_label omitempty should omit when unset
var serverLabelPtr *string
if action.ServerLabel != "" {
  serverLabelPtr = &action.ServerLabel
}
aux := struct{ ServerLabel *string `json:"server_label,omitempty"` }{ServerLabel: serverLabelPtr}

4) Reset receiver before custom unmarshal

func (m *ResponsesMessage) UnmarshalJSON(data []byte) error {
  *m = ResponsesMessage{} // prevent stale preserved fields
  // ... decode; preserve rawToolSearch only for relevant types
  return nil
}

Adopting these rules prevents the common failure modes shown above: stale pointers persisting across requests, incorrect nil checks with typed-nil interfaces, JSON fields that accidentally appear despite omitempty, and reused-struct decode bugs.

Error Handling Consistency

When handling failures, make error semantics explicit and consistent across the stack:

Example pattern (retry classification):

func isRetryableTransportError(err error) bool {
	if err == nil {
		return false
	}
	if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
		return true
	}
	var netErr net.Error
	return errors.As(err, &netErr) && (netErr.Timeout() || netErr.Temporary())
}

And ensure callers only retry when the failure occurs before output is committed, with tests covering Timeout, Temporary, and the “neither set” negative case.

Prevent Secret Leakage

Apply strict trust boundaries to authentication/credential state:

Example (masked placeholder preservation rule):

// incoming may be a UI placeholder like "****"; treat it as non-secret.
if incomingVal.IsRedacted() && !incomingVal.IsFromSecret() {
    if storedVal == nil || !storedVal.IsSet() {
        return fmt.Errorf("masked preview requires stored value") // reject
    }
    incomingVal = *storedVal // preserve real secret
}

Also ensure pooled objects are cleaned:

// before putting request back into a pool or before reusing it
req.ProviderOverride = nil // clear per-request overrides

These practices reduce the risk of credential corruption, SSRF, and cross-request authentication bypass—key failure modes in the security domain.

Deterministic Test Hygiene

Adopt a consistent testing standard that prevents flaky behavior, leaked goroutines, and false positives.

1) Always register cleanup for started resources

bf, err := Init(ctx, cfg)
require.NoError(t, err)
t.Cleanup(func() { bf.Shutdown() })

2) Make concurrency/time-sensitive tests deterministic

deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
  if body.closeCount.Load() > 0 { break }
  time.Sleep(2 * time.Millisecond)
}
stop(); cleanup()

3) Ensure assertions are non-vacuous and validate invariants

attachBilledUsageFromContext(ctx, bifrostErr)
// Mutate original after attach; snapshot must not change.
usage.PromptTokens = 999
require.Equal(t, 10, bifrostErr.ExtraFields.BilledUsage.PromptTokens)

4) Keep mocks faithful to production side effects/commit semantics

candidate := budgets[i]          // copy
candidate.IsCalendarAligned = calendarAligned
if err := candidate.SetOverrideAt(...); err != nil { return nil, err }
budgets[i] = candidate          // commit only on success

5) Fail fast during test setup

if ok := a.GateSend(traceID, chunk0, false, false, ch, ctx); !ok {
  t.Fatalf("setup chunk 0: GateSend returned false")
}

Null and Omission Safety

When a config field is optional/nullable (or has multiple shapes), never rely on “default” or presence quirks. Make absence vs presence explicit, align schema rules with runtime parsing/normalization, and validate the structural inputs that runtime can actually interpret.

Apply these rules:

Example (absence vs explicit empty mode):

Example (defaults annotation vs runtime normalization):

Explicit React UI Behavior

When building React components, make user interaction and lifecycle behavior explicit—particularly for accessibility and ref/state initialization.

Example (tooltip focus + conditional tabIndex):

<Tooltip>
  <TooltipTrigger asChild>
    <span tabIndex={!isEnabled ? 0 : undefined}>
      <button
        disabled={!isEnabled || isRefreshing}
        onClick={(e) => {
          e.stopPropagation();
          handleRefresh();
        }}
      >
        Refresh
      </button>
    </span>
  </TooltipTrigger>
  <TooltipContent>
    {isEnabled ? "Refresh list" : "Enable to refresh"}
  </TooltipContent>
</Tooltip>

Example (explicit ref init):

const popupRef = useRef(initialPopup ?? null);
// Avoid useEffect(() => { popupRef.current = initialPopup ?? null }, [initialPopup])

Semantic Naming Consistency

Adopt naming rules that accurately reflect meaning, routing behavior, and established codebase conventions.

Apply these checks when adding/changing code:

Example (compound ID duplication guard):

first, modelOrName, found := strings.Cut(rawID, "/")
if found && isKnownInferenceProviderOrPolicy(first) {
	// rawID already includes an inference-provider/policy segment; do not prepend again
} else {
	// safe to prepend providerKey/inferenceProvider
	m.ID = fmt.Sprintf("%s/%s/%s", providerKey, inferenceProvider, rawID)
}
_ = modelOrName

Concurrency-safe retries

When concurrency is involved, ensure two things: (1) your tests/operations can’t accidentally contend on shared resources, and (2) any concurrency-handling you add (e.g., retrying on 409) is only applied to errors that are truly reachable from the concurrent scenario you’re mitigating.

Apply this standard as follows:

Example (test isolation + no blind retry):

import { randomUUID } from "crypto";

// Run-scoped suffix prevents cross-run collisions on shared backends.
const runId = randomUUID().slice(0, 8);
const keyName = `Refresh-Key-${runId}`;

// Prefer waiting for the UI/request to settle rather than retrying on errors
// you can’t actually produce from this code path.
await page.click(refreshBtn);
await expect(refreshBtn).toBeEnabled({ timeout: 15000 });

If you’re considering retries for a specific error code, first verify (via server logic/guards) that the code is reachable under the scenario you’re trying to handle; otherwise avoid swallowing it and fix the scenario instead.

Deterministic test setup/teardown

E2E tests should be deterministic by (1) registering mocks/fixtures before any action that triggers network calls (e.g., goto()), and (2) handling cleanup failures in a way that doesn’t skip subsequent tests/hooks.

Apply these rules:

Example pattern:

const created: { provider: string; keyName: string }[] = [];
const leaked: typeof created = [];

test.describe('Some e2e flow', () => {
  test.describe.configure({ mode: 'serial' });

  test.beforeEach(async ({ page }) => {
    // Register mocks BEFORE navigation triggers API calls
    await mockSomeApis(page);
    await providersPage.goto();
  });

  test.afterEach(async ({ providersPage }) => {
    for (const { provider, keyName } of [...created]) {
      try {
        await providersPage.selectProvider(provider);
        const exists = await providersPage.keyExists(keyName, 2000);
        if (exists) await providersPage.deleteKey(keyName);
      } catch (e) {
        leaked.push({ provider, keyName });
      }
    }
    created.length = 0;
  });

  test.afterAll(() => {
    if (leaked.length) {
      throw new Error(`Leaked resources: ${leaked.map(k => `${k.provider}/${k.keyName}`).join(', ')}`);
    }
  });
});

This prevents cascading skips, ensures mocks reliably intercept navigation-time calls, and still surfaces real cleanup problems loudly at the end of the suite.

Prefer safe tag invalidation

When a server mutation (e.g., model discovery refresh) changes data that feeds multiple cached selectors/queries, use RTK Query tag invalidation to refresh all dependent caches—especially indirect ones.

Guideline

Example (pattern)

const refreshProviderModels = builder.mutation<ModelProviderKey[], string>({
  query: (provider) => ({
    url: `/providers/${encodeURIComponent(provider)}/refresh-models`,
    method: 'POST',
  }),
  invalidatesTags: [
    'Models',
    'DBKeys',
    { type: 'Providers' as const, id: provider },
  ],
});

Application checklist

Explicit Undefined Handling

When a value is intentionally absent (e.g., an optional callback prop or required classification fields), represent that absence explicitly and don’t let missing/nullable data silently pass as “valid”.

Rules of thumb

  1. Optional callbacks: If there’s no behavior to run, prefer passing undefined (or omitting the prop) over a noop lambda—unless the component specifically requires a non-null function.
  2. Strict filtering: When narrowing collections based on optional fields, require the expected properties. Don’t write predicates that treat missing fields as matching (e.g., !m.type || ...). This prevents unrelated items (with absent fields) from being included.

Examples

// 1) Optional callback: use undefined instead of noop
<Toggle
  size="xs"
  checked={!allDisabled}
  onChange={undefined} // or simply omit onChange
/>

// 2) Strict null/undefined-aware filtering
const sttModels = models.filter(
  (m) => m.type === "audio" && m.subtype === "transcription"
);
// Avoid: (m) => !m.type || (m.type === "audio" && ...)

This approach reduces accidental behavior (no-op handlers) and avoids UI/data contamination caused by missing optional fields being treated as matches.

Bounded, Targeted Performance

When optimizing performance, avoid whole-object work and unbounded resource growth. Apply these standards:

1) Targeted extraction/rewrites (no full parsing)

Example (safe “model” rewrite):

func rewriteTopLevelModel(body []byte, alias, modelID string) []byte {
  trimmed := bytes.TrimLeft(body, " \t\r\n")
  if len(trimmed) == 0 || trimmed[0] != '{' { // multipart/form-data etc.
    return body
  }
  cur := gjson.GetBytes(body, "model")
  if cur.Type != gjson.String || !strings.EqualFold(cur.Str, alias) {
    return body
  }
  out, err := sjson.SetBytes(body, "model", modelID)
  if err != nil { return body }
  return out
}

2) Bound buffering and “perpetual due” logic

3) Coalesce duplicate concurrent work

4) Pooling hygiene (prevent allocation retention)

These practices together reduce CPU spent on unnecessary parsing/scanning, prevent memory blowups, and eliminate wasted concurrent work—directly improving throughput and latency.

Preserve Aggregation Invariants

When transforming/aggregating streamed or ranged data, explicitly define the invariants your algorithm depends on (index meaning, ownership/copy semantics, boundary semantics, and ordering), then implement collision- and edge-case-safe logic with targeted tests.

Practical rules: 1) Specify index/span semantics and keep them consistent across paths.

Collision-safe terminal-chunk handling (adapted from the discussed fix):

if isFinalChunk && chunk.StreamResponse != nil {
    if acc.TerminalResponseChunkIndex >= 0 {
        chunk.ChunkIndex = acc.TerminalResponseChunkIndex
    } else {
        if chunk.ChunkIndex <= acc.MaxResponsesChunkIndex {
            acc.MaxResponsesChunkIndex++
            chunk.ChunkIndex = acc.MaxResponsesChunkIndex
        }
        acc.TerminalResponseChunkIndex = chunk.ChunkIndex
    }
}

Test guidance:

Stable Backward-Compatible Migrations

When adding schema/config fields, design migrations to be stable across upgrade/downgrade boundaries:

Example pattern for config hash stability (Go):

// In GenerateClientConfigHash
if streamKeepAliveInterval != 0 { // only fold explicit values
    h.WriteString(fmt.Sprintf("stream_keepalive_interval=%d", streamKeepAliveInterval))
}

// In getter (or load/default layer)
if storedInterval == 0 {
    return 15 // default
}

Example pattern for reversible additive migrations:

Migrate: func(tx *gorm.DB) error {
    return addColumnIfNotExists(tx, logger, &tables.TableKey{}, "bedrock_project_id")
},
Rollback: func(tx *gorm.DB) error {
    return dropColumnIfExists(tx, logger, &tables.TableKey{}, "bedrock_project_id")
},

Apply this standard anytime you change config-store schema, add new columns used by runtime behavior, or touch migration rollback behavior—aim for upgrades that don’t break existing rows or cause unintended hash/config regeneration, while keeping rollback aligned with what the migration actually changes.

Semantic Merge And Deep Copy

When your code incrementally merges or inherits structured data (JSON payloads, rule trees, settings), correctness should be enforced by (a) semantic, lossless reconciliation and (b) deep-copying every nested mutable component.

Apply this standard: 1) For delta/stream/state reconciliation

Example (Go sketch):

func reconcileDoneArguments(forwarded, done string) (string, error) {
  // If done is empty, never overwrite accumulated deltas.
  if done == "" { return "", nil }

  // Prefer semantic equivalence over string equality.
  eq, err := semanticJSONEqual(forwarded, done)
  if err != nil { return "", err }
  if eq { return "", nil } // nothing missing

  // Otherwise, forward only missing suffix when done starts with forwarded.
  if forwarded != "" && strings.HasPrefix(done, forwarded) {
    return strings.TrimPrefix(done, forwarded), nil
  }

  return "", fmt.Errorf("divergent arguments")
}

func semanticJSONEqual(a, b string) (bool, error) {
  var va, vb any
  da := json.NewDecoder(strings.NewReader(a)); da.UseNumber()
  db := json.NewDecoder(strings.NewReader(b)); db.UseNumber()
  if err := da.Decode(&va); err != nil { return false, err }
  if err := db.Decode(&vb); err != nil { return false, err }
  return reflect.DeepEqual(va, vb), nil
}

2) For inheritance/transformations via cloning

Checklist before merging PRs:

Preserve API Contracts

Maintain deterministic, provider-compatible API contracts by (1) applying overrides explicitly and attempt-scoped (no hidden side effects), and (2) preserving exact request/response serialization expectations—especially where providers are sensitive to JSON schema structure/order or where passthrough/raw modes must not be rewritten.

How to apply:

Example (attempt-scoped override wiring):

preReq := resultFromHooks() // may contain ProviderOverride
ctx.ClearValue(schemas.BifrostContextKeyProviderOverride)
if preReq != nil && preReq.ProviderOverride != nil {
    ctx.SetValue(schemas.BifrostContextKeyProviderOverride, preReq.ProviderOverride)
}

Example (schema contract preserving order):

type ResponsesTextConfigFormatJSONSchema struct {
    Name   *string          `json:"name,omitempty"`
    Schema *OrderedMap     `json:"schema,omitempty"` // preserves JSON key order
}

Backend Enforcement, UI Validation

Treat the frontend as an UX layer, not an authorization or confidentiality enforcement boundary. Any access control (licensing, revoke/re-auth, hidden-content guarantees, RBAC) must be enforced by the backend/data layer; the UI may be redundant or fail-open without granting access. Separately, whenever the UI must render untrusted values into security-sensitive contexts (e.g., hyperlinks), validate/allow-list before using them.

Apply this as a standard:

Example pattern (safe URL rendering):

const uri = deviceState.verificationUri;
const safe = /^https:\/\/([a-z0-9-]+\.)*github\.com(\/|$)/i.test(uri);

return safe ? (
  <a href={uri} target="_blank" rel="noopener noreferrer">Open link</a>
) : (
  <code>{uri}</code>
);

If you’re tempted to add a UI-only if (shouldHide) return null, confirm the backend/storage already prevents the underlying data/action; otherwise, move or add the enforcement where it can’t be bypassed.

Explicit Config Overrides

Implement configuration in a way that is predictable: define defaults centrally, keep early rollout scope narrow (prefer deployment-level defaults), and make override semantics explicit.

Apply this standard: 1) Set defaults in the conf package (single source of truth). Avoid scattering “fallback” values across business logic. 2) Use deployment-level defaults first for new config knobs; defer per-channel/UI/schema expansion until there’s product/runtime backing. 3) For any “override” configuration, choose and document semantics clearly.

Example (full-replacement override pattern + deploy-level default):

// conf package should provide defaults, not business logic.
// e.g., conf.DefaultXXX

func codexImageMainModel() string {
    // deployment-level default (via env) only; keep scope limited initially
    if v := strings.TrimSpace(os.Getenv("AXONHUB_CODEX_IMAGE_MAIN_MODEL")); v != "" {
        return v
    }
    return defaultImageMainModel
}

func effectiveItems(now time.Time, schedule *Schedule, base Items) Items {
    if schedule == nil {
        return base
    }
    if override := findMatchingOverride(now, schedule); override != nil {
        // full replacement: override.Items is the complete effective set
        return override.Items
    }
    return base
}

Outcome: fewer surprises during runtime, easier debugging, and safer incremental rollout of configuration-driven features.

Graceful config validation

When configuration has conditional dependencies, make the system fail fast only for cases that break invariants (correctness/safety), but allow secure graceful degradation for “optional feature” toggles when prerequisites may be missing.

Apply this rule: 1) Enforce bidirectional relationships in schema

Example (JSON Schema conditional enforcement):

{
  "allOf": [
    {
      "if": {"properties": {"auth_type": {"const": "per_user_headers"}}, "required": ["auth_type"]},
      "then": {"required": ["per_user_header_keys"]}
    }
  ]
}

2) Don’t turn “safe no-ops” into hard failures

Decision heuristic:

Strict auth schema validation

Security-related Helm/config schemas should fail fast on invalid or unintended authentication settings.

Apply:

Example pattern (ServiceMonitor-style strict auth):

{
  "authorization": {
    "type": "object",
    "description": "Bearer-token auth for scrape",
    "additionalProperties": false,
    "properties": {
      "type": {"type": "string"},
      "credentials": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "name": {"type": "string"},
          "key": {"type": "string"}
        },
        "required": ["name", "key"]
      }
    },
    "required": ["type", "credentials"]
  }
}

Outcome: invalid auth configs (typos, missing required credential fields, or flags applied to the wrong provider) are rejected early, preventing insecure or broken authentication behavior at runtime.

Stability-First Caching

When caching affects externally visible payloads (prompts, auth/JWT inputs, token summaries), ensure the serialized representation used for cache identity is deterministic and the cache has an explicit validity boundary.

Practical rules:

Example: deterministic merge without map-key-order instability

func mergeDeterministic(base map[string]any, fallback map[string]any) map[string]any {
	if len(base) == 0 && len(fallback) == 0 {
		return nil
	}
	merged := make(map[string]any, len(base)+len(fallback))

	keys := make([]string, 0, len(base)+len(fallback))
	seen := make(map[string]struct{}, len(base)+len(fallback))
	for k := range base {
		if _, ok := seen[k]; !ok {
			seen[k] = struct{}{}
			keys = append(keys, k)
		}
	}
	for k := range fallback {
		if _, ok := seen[k]; !ok {
			seen[k] = struct{}{}
			keys = append(keys, k)
		}
	}
	sort.Strings(keys)
	for _, k := range keys {
		if v, ok := fallback[k]; ok {
			merged[k] = v
		} else {
			merged[k] = base[k]
		}
	}
	return merged
}

If the cached payload must preserve ordering semantics (e.g., output_config), prefer an ordered representation (OrderedMap) instead of converting to map[string]any during merges.

Follow test constraints

When adding UI logic or tests, align with the repository’s existing CI/test guardrails—especially global checks and test discovery conventions.

Example (i18n guardrail pattern):

// Prefer keeping the diff focused when full i18n key rollout is out of scope.
{condition && (
  // Inline callout instead of adding en-only keys that break locale-parity tests.
  <div className="text-xs">...</div>
)}
// Then open a follow-up PR to properly add i18n keys across required locales.

Example (test structure pattern):

// Keep dashboard component tests co-located under the component's __tests__
// so existing Vitest include globs discover them consistently.
// Put your new *.test.tsx next to the component tests rather than in a different folder
// unless the repo-wide convention is being intentionally changed.

Fail Safe Error Propagation

Establish an error-handling standard that prevents crashes/hangs, preserves root errors, and makes recovery bounded and verifiable.

Guidelines:

Example (SSE transform fail propagation + streaming semantics):

return new TransformStream({
  transform(chunk, controller) {
    try {
      // parse/process...
    } catch (err) {
      // Propagate and abort the SSE pipeline
      controller.error(err);
    }
  }
});

Example (guarded retry with non-replayable body check):

const bodyUnknown = options.body as unknown;
const isBodyStream =
  bodyUnknown && typeof bodyUnknown === 'object' &&
  (typeof (bodyUnknown as any).getReader === 'function');
const maxAttempts = isBodyStream ? 1 : 2;

for (let attempt = 0; attempt < maxAttempts; attempt++) {
  try {
    return await undiciDirect(input, { ...options, dispatcher });
  } catch (e) {
    const code = (e as { code?: string }).code;
    if (code === 'ECONNREFUSED' || String(e).includes('fetch failed')) continue;
    throw e;
  }
}

Deterministic Cache Keys

Cache keys must be built only from inputs that truly change model output, and must be deterministic, complete, and cheap to compute.

Rules

  1. Derive keys from a stable message “anchor”. If using conversation head hashing, explicitly define which roles and how many bytes you consider (e.g., contiguous leading system/developer, plus first user). Keep later turns from changing the head anchor when that’s intended.
  2. Include all message modalities in the fingerprint. Don’t hash only text; include non-text parts (image/video/document/audio/etc.) using a consistent scheme. When hashing variable-sized units, include length prefixes (or other unambiguous framing) so different payloads that share a prefix don’t collide.
  3. Do not let unrelated metadata affect cache hits. If you inject IDs (user/session/order IDs, fake IDs, routing metadata), ensure caching is driven solely by the message content (and any explicitly required rule/settings signatures).
  4. Keep cache-key work lightweight on hot paths. Avoid full JSON serialization during cache lookup. For invalidation due to developer-rule/settings changes, add a compact per-field signature (e.g., FNV or similar) computed from only the fields that affect candidate/routing behavior.

Sketch (Go-like pseudocode)

// Cache key inputs: (1) stable message anchor, (2) cheap settings/rules signature.
func cacheKey(messages []Message, rulesSig uint64, sessionID string) string {
    anchor := conversationAnchor(messages) // includes text + non-text parts with framing
    return fmt.Sprintf("%s|%d|%s", anchor, rulesSig, sessionID)
}

func conversationAnchor(messages []Message) string {
    // Hash only: leading system/developer + first user (bounded by anchorMaxBytes)
    // For each message: role + delimiter + all content parts.
    // For each part:
    //   - if text: write length prefix + text
    //   - if non-text: write part kind + length prefix + URL/data bytes
    // Ensure deterministic ordering and framing.
}

Outcome

Low-noise, instance-safe logs

When adding or modifying logging, follow these rules:

1) Attribute logs to the right component/instance

2) Pick the right level and avoid noise

3) Never silently fall back

4) Use the logging API’s formatting instead of fmt.Sprintf

5) Don’t bloat request/context just to move logs

Example patterns:

// 1+3) Instance-safe warning on fallback
if err != nil {
    logger.Warn("oauth: url parse failed, falling back to direct client: %v", err)
}

// 2) Correct level + log once (stream-aware)
if isStream && !isFinalChunk {
    // skip warning
} else {
    logger.Debug("cache: skipping write (namespace=%s, id=%s): no embedding available", ns, id)
}

// 4) Prefer format args
logger.Warn("logstore: skipping index maintenance: could not acquire index lock: %v", err)

// 5) Avoid reading logs back from fasthttp context; use dedicated slot/completer
// e.g., middleware publishes into a known slot, handler drains it after stream completes.

Applying this consistently reduces production log spam, preserves accurate attribution, improves diagnostics when fallbacks occur, and keeps logging implementation clean and maintainable.

Enforce API Contracts

When designing or extending APIs, ensure the request/response schema and behavior strictly preserve invariants and provider compatibility.

Apply these rules: 1) Don’t expose unsafe state transitions

2) Use explicit, strongly-typed API models for configuration/mappings

Example (schema shape for typed mappings):

input ReasoningEffortMappingInput {
  from: String!
  to: String!
}

type ReasoningEffortMapping {
  from: String!
  to: String!
}

input UpdateChatSettingsInput {
  reasoningEffortMapping: [ReasoningEffortMappingInput!]
}

3) Preserve the intended API lifecycle across fallback paths

4) Match third-party spec requirements exactly and guard provider-specific behaviors

Config-Aware Startup Failures

When adding startup preflight checks (permissions, filesystem writeability, required directories), treat failure as part of error-handling policy: be precise, actionable, and configuration-aware.

Standard

  1. Fail fast only when truly required. If the app is configured to use external stores or otherwise won’t write to APP_DIR, don’t crash-loop just because the directory is read-only.
  2. Provide an escape hatch for intentional exceptions. If you perform a writeability probe, allow an explicit override (e.g., BIFROST_SKIP_WRITE_CHECK=1) to downgrade exit 1 to a warning in known-safe scenarios.
  3. Make errors actionable. When exiting, print what was checked (owned UID:GID, mode) and concrete fixes (own the UID, group-writable for supplemental group like 0, set podSecurityContext.fsGroup).
  4. Avoid pathological control flow. For shell argument parsing/loops, ensure conditions and iteration always make progress (don’t keep $# constant while “looping” via set -- ...; shift).

Example pattern (self-contained)

APP_DIR=${APP_DIR:-/app/data}
SKIP_WRITE_CHECK=${BIFROST_SKIP_WRITE_CHECK:-0}

app_dir_writable() {
  probe="$APP_DIR/.write-test.$$"
  if [ -e "$probe" ]; then probe="$probe.$(date +%s)"; fi
  mkdir "$probe" 2>/dev/null && rmdir "$probe" 2>/dev/null
}

ensure_app_dir() {
  mkdir -p "$APP_DIR" 2>/dev/null || true
  [ -d "$APP_DIR" ] || { echo "Error: cannot create $APP_DIR"; exit 1; }

  if [ "$SKIP_WRITE_CHECK" = "1" ]; then
    echo "Warning: skipping $APP_DIR write check (BIFROST_SKIP_WRITE_CHECK=1)"
    return 0
  fi

  if ! app_dir_writable; then
    uid=$(id -u); gid=$(id -g)
    data_uid=$(stat -c '%u' "$APP_DIR" 2>/dev/null || echo 0)
    data_gid=$(stat -c '%g' "$APP_DIR" 2>/dev/null || echo 0)

    echo "Error: $APP_DIR not writable by UID:GID $uid:$gid (owned by $data_uid:$data_gid)"
    echo "Fix: make it owned by this UID or group-writable for a supplemental group (e.g., GID 0), or set podSecurityContext.fsGroup"
    exit 1
  fi
}

Applying this standard reduces crash-loops, makes failures diagnosable, and prevents “strict checks” from blocking valid deployments where writes are unnecessary.

Document behavior precisely

When updating documentation, ensure every statement about requirements, capabilities, and behavior is (1) correct for the current implementation, (2) explicit about when the requirement matters (e.g., render-time vs apply-time vs run-time), (3) consistent across related doc artifacts, and (4) does not imply unsupported configuration paths. Follow house style for headings/navigation (e.g., tab names).

How to apply:

Example (pattern to follow when prerequisites only matter at apply-time):

Auth config documentation

When Helm (or other config) controls authentication for a protected endpoint, treat credential source, key names, and auth enablement paths as security-sensitive behavior. Ensure both the implementation and the accompanying docs/changelog precisely describe:

Example validation approach:

helm template rel . \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.basicAuth.enabled=true \
  --set serviceMonitor.basicAuth.existingSecret=scrape-creds \
  --set bifrost.authConfig.existingSecret=bifrost-secrets \
  --set bifrost.authConfig.usernameKey=admin-username \
  --set bifrost.authConfig.passwordKey=admin-password

# Confirm rendered basicAuth uses scrape-creds with key names
# (e.g., username/password), not inherited admin-username/admin-password.

This prevents misleading security documentation that can cause operators to scrape with wrong credentials, misunderstand failure modes, or break credential rotation assumptions.

Validate Config Inputs

Any configuration derived from environment variables or external config must be (1) wired to the correct runtime fields, (2) normalized/validated with safe defaults, and (3) applied with deterministic precedence.

Practical standards:

Example (env validation + deterministic deny/allow precedence):

// 1) Env numeric validation with fallback
const parsed = parseInt(process.env.PRICING_SYNC_INTERVAL ?? "86400", 10);
const intervalMs = Number.isFinite(parsed) && parsed > 0
  ? parsed * 1000
  : 86400 * 1000; // 24h fallback

// 2) Snapshot-based allowlist restoration
function applyFilters({ provider, model, body, snapshot, cfg }: {
  provider: string; model: string; body: Record<string, unknown>; snapshot: Record<string, unknown>; cfg: any;
}) {
  // denylist: provider
  for (const k of cfg.block ?? []) delete body[k];
  // denylist: model
  for (const k of (cfg.models?.[model]?.block ?? [])) delete body[k];
  // allowlist: provider (restore only keys that were present originally)
  for (const k of (cfg.allow ?? [])) if (k in snapshot) body[k] = snapshot[k];
  // allowlist: model (most specific wins by applying last)
  for (const k of (cfg.models?.[model]?.allow ?? [])) if (k in snapshot) body[k] = snapshot[k];
}

Adopt this pattern whenever config impacts runtime behavior (HTTP targets, auth payloads, request parameter shaping, sync intervals/feature flags): fail safely, default predictably, and keep behavior order explicit.

Group-aware Pagination

When the client needs a logical unit (run/history group, full resource state), the API must paginate and update in a way that preserves that unit across requests.

Apply this standard: 1) Pagination must align to domain groups

Example contract shape (conceptual):

2) Update semantics must be explicit + concurrency-safe

3) Don’t export incorrect invariants to the UI

Explicit Null Safety

When templating, explicitly distinguish three states: missing, explicit false, and null. Relying on truthiness can silently drop intended configuration.

Apply two patterns: 1) Preserve explicit false (boolean fields)

{{/* BAD: with treats false as empty */}}
{{- with .Values.serviceMonitor.honorLabels }}
honorLabels: {{ . }}
{{- end }}

{{/* GOOD: preserves explicit false */}}
{{- if hasKey .Values.serviceMonitor "honorLabels" }}
honorLabels: {{ .Values.serviceMonitor.honorLabels }}
{{- end }}

2) Omit fields via real null (Helm values)

# values.yaml override example
podSecurityContext:
  runAsUser: null
  fsGroup: null
securityContext:
  runAsUser: null

Rule of thumb:

Efficient hot-path discipline

Adopt a performance-safety standard: prevent unbounded state growth in monitoring, and keep critical-path CPU/memory work linear and avoidable.

Apply these rules: 1) Treat performance instrumentation as state with lifecycle

2) Keep streaming/decoding parsers linear

3) Precompute repeated work in bounded loops

4) Add cheap guards before expensive transforms

5) Gate unnecessary upstream calls

Example patterns (self-contained):

// 1) Bounded performance marks performance.mark(“omni-request-body-size”, { detail: bodySize }); // … read/observe in your monitoring pipeline … performance.clearMarks(“omni-request-body-size”);

// 2) Rolling buffer (linear-time scanning) const chunks: Buffer[] = []; let processed = 0; req.on(“data”, (chunk) => { chunks.push(Buffer.from(chunk)); const buffer = Buffer.concat(chunks); // process from processed.. and advance processed cursor // then drop processed prefix to keep future work bounded });

// 3) Precomputed lookup const unsupportedParamsByModel = new Map<string, readonly string[]>(); // module init fills map function getUnsupportedParams(modelId: string) { return unsupportedParamsByModel.get(modelId) ?? []; }

// 4) Cheap guard before expensive work if (payload.length > MAX_BYTES * 2) throw new Error(“too large”); // only then do regex/strip/decode

// 5) Latency gate if (!hasAnyEnforcementCondition) return; // skip upstream preflight

Lifecycle-aligned Cancellation

When handling concurrent async work (streaming, tees, semaphores/limits, watchdogs), ensure cancellation, timeouts, and accounting are tied to the actual consumer lifecycle—release/decrement/teardown at the point where the work is truly finished, and don’t block on cancellation promises that can’t resolve until another branch drains.

Practical rules:

Example pattern (release on flush, not header arrival):

const slot = await acquireSemaphore();
let released = false;
const releaseOnce = () => {
  if (released) return;
  released = true;
  slot();
};

return new TransformStream<Uint8Array, Uint8Array>({
  transform(chunk, controller) {
    controller.enqueue(chunk);
  },
  flush() {
    // stream fully consumed (client drained)
    releaseOnce();
  }
});

These practices reduce race conditions, deadlocks, leaked capacity, and spurious failures in highly concurrent streaming systems.

Null-Safe Validation Standard

Always validate null/undefined (and malformed shapes) before dereferencing, filtering, or merging—treat null as an explicit, documented input rather than “missing”.

Apply this consistently:

Example patterns:

// 1) Filtering: guard nullish elements + explicit null meaning
function filterActiveConnections<T extends { isActive?: boolean }>(connections: (T | null | undefined)[]) {
  return connections.filter((c) => !!c && c.isActive !== false);
}

// 2) Parsing: tolerate malformed upstream arrays
function extractFirstVideo(response: unknown): { base64?: string; url?: string } | null {
  if (!response || typeof response !== 'object') return null;
  const videos = (response as any).videos;
  if (!Array.isArray(videos) || videos.length === 0) return null;
  const v = videos[0] as any;
  if (typeof v?.bytesBase64Encoded === 'string') return { base64: v.bytesBase64Encoded };
  if (typeof v?.gcsUri === 'string') return { url: v.gcsUri };
  return null;
}

// 3) Options default: tolerate explicit null
export function filterToOpenAIFormat(body: unknown, opts: { preserveCacheControl?: boolean } | null = {}) {
  const preserve = opts?.preserveCacheControl === true;
  // ...
}

// 4) PATCH semantics: explicit null clears; empty object is no-op
function applyPatch(existing: any, incomingWindowThresholds: any) {
  if (incomingWindowThresholds === null) return { ...existing, quotaWindowThresholds: null };
  if (incomingWindowThresholds && typeof incomingWindowThresholds === 'object') {
    const next = { ...(existing.quotaWindowThresholds ?? {}) };
    for (const [k, v] of Object.entries(incomingWindowThresholds)) {
      next[k] = v === null ? null : v;
      if (v === null) delete next[k]; // optional: choose your clear behavior per spec
    }
    return { ...existing, quotaWindowThresholds: next };
  }
  return existing; // {} should be no-op; missing should be no-op
}

If a function can receive null/undefined from upstream, config, or user input, make the “what happens” path explicit and test the nullish/empty cases.

Contract-First API Validation

Treat every API endpoint as a contract:

Example (pagination with schema validation):

export async function GET(request: Request) {
  const url = new URL(request.url);
  const offsetLimitRaw = {
    offset: url.searchParams.get("offset"),
    limit: url.searchParams.get("limit"),
  };

  // paginationSchema: offset int >= 0, limit int within an agreed range
  const validation = validateBody(paginationSchema, offsetLimitRaw);
  if (isValidationFailure(validation)) {
    return NextResponse.json({ error: validation.error }, { status: 400 });
  }

  const { offset, limit } = validation.data;
  const all = await getCombos();
  const combos = all.slice(offset, offset + limit);

  return NextResponse.json({ combos });
}

Team standard: no manual parseInt/fallback defaults for externally supplied API parameters; any transformation layer must preserve/emit the documented output shape.

Real Regression Tests

When you fix behavior (especially edge/boundary cases), add regression tests that are:

Apply like this:

Example (boundary regression + meaningful failure):

// Arrange
const LIMIT = 10;
// Restoring a buggy form should make this test fail.
function fixedAppend(current: string, next: string) {
  if (!next) return current;
  if (current.length >= LIMIT) {
    const keep = LIMIT > next.length ? LIMIT - next.length : 0;
    if (keep <= 0) return next.slice(-LIMIT);
    return current.slice(-keep) + next;
  }
  return current + next;
}

// Assert (exact boundary)
import test from "node:test";
import assert from "node:assert/strict";

test("appendBoundedText clamps at limit", () => {
  const cur = "x".repeat(LIMIT);
  const next = "y".repeat(LIMIT);
  assert.equal(fixedAppend(cur, next), next); // should be clamped
});

Also add guard assertions when you have rule systems/transform pipelines so that “no-op” rules (e.g., patterns that never change anything) can’t be introduced unnoticed.

Validate security context

Define and enforce security-context behavior as a clear contract, then implement CI/runtime logic to match it without brittle assumptions.

Apply this in two places:

1) Helm/manifest security-context checks (CI)

Example pattern:

# contract: default render must not set runAsUser
helm template bifrost ./helm-charts/bifrost \
  --set image.tag=v1.0.0 \
  > /tmp/out.yaml

if grep -Eq '^[[:space:]]+runAsUser:[[:space:]]*' /tmp/out.yaml; then
  echo "Fail: runAsUser must not be pinned in default render"
  exit 1
fi

2) Startup permissions (runtime)

Net effect: fewer admission/runtime surprises and fewer CI “green but wrong” security-context failures.

Streaming error invariants

Adopt a single, consistent error-handling strategy for streaming (and other multi-step flows): define retry semantics, tolerate isolated decode/parse errors, enforce terminal/completion invariants, and preserve the failure context needed by downstream billing/logging.

Guidelines: 1) Classify failures at the I/O boundary

2) Streaming: log-and-continue for local frame issues

3) Streaming: enforce completion invariants

4) Preserve context on ALL error paths

5) Async safety: prevent unrecoverable panics in teardown goroutines

Example pattern (Go, streaming):

// Local helper: attach billed usage accumulated in-context to the error.
func attachBilledUsageFromContext(ctx *schemas.BifrostContext, bifrostErr *schemas.BifrostError) {
  usage, ok := ctx.Value(schemas.BifrostContextKeyStreamAccumulatedUsage).(*schemas.BifrostLLMUsage)
  if !ok || usage == nil { return }
  if usage.PromptTokens == 0 && usage.CompletionTokens == 0 && usage.TotalTokens == 0 { return }
  // Important: copy or snapshot if usage will be mutated later.
  u := *usage
  bifrostErr.ExtraFields.BilledUsage = &u
}

// Inside stream read loop:
var sawDone bool
for {
  data, err := sseReader.ReadDataLine()
  if err != nil {
    if errors.Is(err, io.EOF) {
      break
    }
    // pre-first-byte / upstream classification happens at the Do() boundary,
    // but for mid-stream read errors, enrich and send.
    e := providerUtils.NewBifrostOperationError("stream read failed", err)
    attachBilledUsageFromContext(ctx, e)
    providerUtils.ProcessAndSendBifrostError(ctx, postHookRunner, e, responseChan, logger, postHookSpanFinalizer)
    return
  }

  var evt CartesiaSSEEvent
  if err := sonic.Unmarshal(data, &evt); err != nil {
    logger.Warn("Failed to parse SSE event: %v", err)
    continue // log-and-continue
  }

  switch evt.Type {
  case "done":
    sawDone = true
    // optionally process done payload
  case "chunk":
    if evt.Data == "" { continue }
    audio, decErr := base64.StdEncoding.DecodeString(evt.Data)
    if decErr != nil {
      logger.Warn("Failed to decode chunk: %v", decErr)
      continue // keep stream alive on single chunk decode errors
    }
    // send delta chunk...
  case "error":
    // surface terminal provider error
    e := providerUtils.NewBifrostOperationError("cartesia stream error: "+evt.Error, nil)
    attachBilledUsageFromContext(ctx, e)
    providerUtils.ProcessAndSendBifrostError(ctx, postHookRunner, e, responseChan, logger, postHookSpanFinalizer)
    return
  }

  if sawDone { break }
}

// Terminal-invariant guard: EOF without terminal done is truncated.
if !sawDone {
  ctx.SetValue(schemas.BifrostContextKeyStreamEndIndicator, true)
  e := providerUtils.NewBifrostOperationError("stream ended before terminal done", io.ErrUnexpectedEOF)
  attachBilledUsageFromContext(ctx, e)
  providerUtils.ProcessAndSendBifrostError(ctx, postHookRunner, e, responseChan, logger, postHookSpanFinalizer)
  return
}

This standard prevents: (a) “empty success” on truncated streams, (b) missing bills/trace data on cancellation/timeout, (c) inconsistent retry behavior, and (d) process crashes from panics in goroutines used for stream teardown/timeouts.

Secret-Aware Schema Contracts

When updating configuration schemas (and related env/test fixtures), treat the schema as a contract with runtime unmarshalling and deployment timing.

Apply these rules: 1) Mirror secret-aware runtime types in the schema

Example pattern (JSON Schema):

{
  "anyOf": [
    { "type": "string" },
    {
      "type": "object",
      "properties": {
        "value": { "type": "string" },
        "env_var": { "type": "string" },
        "from_env": { "type": "boolean" }
      },
      "required": ["value"],
      "additionalProperties": false
    }
  ]
}

2) Verify the config effect end-to-end (release coupling)

3) Avoid out-of-scope “partial tightening”

4) Keep test/env files suite-scoped

Outcome: schema validation should align with real runtime parsing, and “valid configs” should produce the intended behavior in the correct release.

Config Contract Validation

Ensure API configuration endpoints (e.g., PUT /api/config) define and enforce a precise request/response contract across schema shape, merge semantics, validation behavior, and persistence.

Apply this standard:

Example (server-side merge rule pattern):

// Only update each field when explicitly provided so partial PUTs do not clear stored values.
if payload.ClientConfig.MCPServerAuthMode != "" {
    updatedConfig.MCPServerAuthMode = payload.ClientConfig.MCPServerAuthMode
}
// otherwise keep currentConfig.MCPServerAuthMode

Example (E2E assertion pattern):

Graceful Degradation Policy

For external dependencies that are additive (e.g., mirrors/archives) rather than required for correctness, prefer fail-open behavior: log errors and continue operating using the primary source of truth. Reserve fail-fast for dependencies that are hard requirements for correctness or API functionality.

Practical rules:

Example (non-blocking additive sink):

onStartup():
  try:
    objectStoreClient = connect(config.objectStorage)
    archivalEnabled = true
  catch err:
    log.error("object storage unreachable; continuing database-only", err)
    archivalEnabled = false

writeAuditEvent(event):
  dbWrite(event) // must always succeed for correctness

flushBatch(batch):
  if not archivalEnabled: return
  try:
    objectStore.put(renderKey(batch), gzip(jsonl(batch)))
  catch err:
    log.error("archival upload failed; audit DB write already committed", err)
    // do not fail request handling

Consistent UI Formatting

Apply consistent, presentation-focused code style to UI components:

Example pattern:

const statCards = useMemo(() => {
  const display = stats
    ? stats.success_rate.toFixed(2) + "%"
    : undefined;
  const tooltip = stats
    ? stats.success_rate.toLocaleString(undefined, { maximumFractionDigits: 6 })
    : undefined;

  return [{
    title: "Success Rate",
    value: fetchingStats ? <Skeleton /> : display ?? "-",
    tooltip: fetchingStats ? undefined : tooltip,
    testId: "logs-stat-value-success-rate",
  }];
}, [fetchingStats, stats]);

This keeps UI layout stable, prevents truncation/tooltip bugs, improves readability, and ensures consistent formatting across the app.

Database-safe atomic updates

When writing/reading sensitive or destructive data, make the database do the work safely and atomically:

Example pattern (atomic read + update + safe predicates):

// Atomic read-modify-write even without db.transaction()
const nextScopes = Array.isArray(scopesUpdate)
  ? scopesUpdate.filter((s): s is string => typeof s === "string")
  : [];

const result = db.exec(`
  BEGIN IMMEDIATE;
`);

try {
  const prevRow = db
    .prepare<{ scopes: string | null }>(
      "SELECT scopes FROM api_keys WHERE id = ?"
    )
    .get(id);
  const previousScopes = parseStringList(prevRow?.scopes ?? null);

  db.prepare("UPDATE api_keys SET scopes = ? WHERE id = ?")
    .run(JSON.stringify(nextScopes), id);

  db.exec("COMMIT;");

  // emit audit using previousScopes -> nextScopes
} catch (e) {
  db.exec("ROLLBACK;");
  throw e;
}

-- Safe destructive delete by identifier prefix (no LIKE)
DELETE FROM key_value
WHERE namespace = 'syncedAvailableModels'
  AND substr(key, 1, ?) = ?;

Applying this consistently reduces correctness bugs, prevents expensive over-fetching, and keeps audit trails reliable across different SQLite/driver backends.

Cohesive Converters Organization

Keep Go files cohesive by responsibility, and make branching/conversion logic easy to extend.

1) Put types where they belong

2) Keep conversion logic readable

Example pattern:

if chunk.Index == nil || chunk.ContentBlock == nil {
  return nil, nil, false
}

switch chunk.ContentBlock.Type {
case AnthropicContentBlockTypeToolUse:
  // structured handling
case AnthropicContentBlockTypeRedactedThinking:
  // structured handling
default:
  return nil, nil, false
}

3) DRY repeated utilities safely

4) Centralize “magic” values

5) Prefer strong typing / reuse helpers

This reduces churn when adding new providers/types, makes reviews faster, and prevents subtle behavior drift from copy/pasted logic.

Network payload safety

When implementing HTTP/SSE adapters or transforming network responses/streams, enforce three rules: (1) preserve required response headers/trace metadata while normalizing body-related headers, (2) respect protocol framing so streamed events can’t merge, and (3) prevent hanging outbound calls with timeouts.

Apply this as a checklist:

Example patterns:

// 1) HTTP response transformation: preserve trace/version headers
function buildTransformedResponse(upstream: Response, data: any) {
  const newHeaders = new Headers(upstream.headers);
  // normalize payload headers for the transformed body
  newHeaders.set("Content-Type", "application/json");
  newHeaders.delete("Content-Length");
  return new Response(JSON.stringify(data), {
    status: upstream.status,
    headers: newHeaders,
  });
}

// 2) SSE framing: always end events with a delimiter
function enqueueSseEvent(controller: TransformStreamDefaultController, prefix: string, payload: string) {
  controller.enqueue(new TextEncoder().encode(`${prefix}${payload}\n\n`));
}

// 3) Outbound request timeout
async function fetchWithTimeout(url: string, init: RequestInit = {}, timeoutMs = 7000) {
  const ac = new AbortController();
  const t = setTimeout(() => ac.abort(), timeoutMs);
  try {
    return await fetch(url, { ...init, signal: ac.signal });
  } finally {
    clearTimeout(t);
  }
}

Teams should require these checks in code review for any networking code that adapts/streams/transforms data.

Isolate errors, avoid cascades

When a feature depends on multiple probes/loads, ensure that failures in secondary/best-effort steps don’t corrupt primary state or cause cascading failures.

Apply this in two ways: 1) Runtime/state handling: keep authoritative flags separate from enrichment fields.

Example pattern:

   export async function getStatus(url) {
     const installed = Boolean(pathFromCliProbe);

     // Best-effort enrichment; should NOT affect `installed`.
     let extras = { version: null, extras: { code: false, ml: false } };
     try {
       if (installed) extras = await getInstalledHeadroomExtras();
     } catch {
       // swallow or log; keep primary state stable
     }

     return { installed, ...extras };
   }

2) Test structure: fail fast in setup.

Example pattern:

   import { before, it } from "node:test";
   import assert from "node:assert/strict";

   let entry;
   before(async () => {
     entry = (await import("../../path/to/module.js")).default;
   });

   it("has expected id", () => {
     assert.equal(entry.id, "kimchi");
   });

This prevents both incorrect UI/business logic (wrong state derived from a failing secondary probe) and noisy, cascading test failures that obscure the real cause.

Secure configs across environments

When making security-related changes, ensure the configuration is (1) least-privilege/scoped, (2) actually enforced by the platform, and (3) consistent across code/docs.

Apply this standard:

Example (numeric UID to satisfy kubelet verification):

# Ensure kubelet can verify non-root when runAsNonRoot: true
USER 1000:0

Example (scope secrets to test jobs):

# Only provide sensitive tokens where they are required (e.g., PR test jobs)
# Avoid injecting tokens into broad release job env blocks.
env:
  # GITHUB_COPILOT_TOKEN: ${{ secrets.GITHUB_COPILOT_TOKEN }}  # keep out of release jobs

Harden build-time paths

When Dockerfile (or build scripts) use build-args/variables to choose target directories for chown/chmod, validate those paths at build time and fail if they target sensitive directories. This prevents overly-permissive ownership (e.g., group 0 write access) from becoming reachable via undocumented/override knobs.

Apply this rule by adding a guard or assertion before any recursive ownership/permission changes. For example:

ARG APP_DIR=/app/data

# Fail build if a dangerous directory is selected (adjust allowlist/prefix as needed)
RUN case "$APP_DIR" in \
  /app|/app/*) echo "Refusing insecure APP_DIR=$APP_DIR" >&2; exit 1 ;; \
esac

RUN chown -R appuser:0 "$APP_DIR" && chmod -R g=rwX "$APP_DIR"

Also prefer explicit allowlists (or “path must start with /app/data”) over implicit assumptions about defaults; don’t rely on the variable being “theoretical” or undocumented to avoid security regressions.

Cache correctness rules

Adopt a “cache correctness” checklist for all in-memory/DB caches:

1) Explicit bypass must be honored

2) Never poison future caching on failure

3) Guard correctness against changing upstream state

Example patterns:

// (1) Honor ttlMs=0 bypass (also ensure defaulting doesn’t override 0) async function getOrCoalesce(ttlMs: number, fetchFn: () => Promise) { if (ttlMs <= 0) { const data = await fetchFn(); return { data, cached: false }; } // ... normal cache lookup + inflight coalescing }

// (2) Evict rather than stop/poison function onRateLimited429(/* … */) { // evict so future getLimiter() creates a new instance // DO NOT call stop() if it would permanently reject future scheduling }

// (3) Generation-guarded, bounded cache const MAX = 100; let gen = 0; const cache = new Map<string, { generation: number; value: string }>();

function onWriteThatAffectsCache() { gen++; cache.clear(); }

function get(key: string): string | null { const entry = cache.get(key); if (!entry || entry.generation !== gen) return null; return entry.value; }

function set(key: string, value: string) { if (cache.size >= MAX) cache.delete(cache.keys().next().value); cache.set(key, { generation: gen, value }); }

Rule of thumb: every cache needs (a) explicit bypass semantics, (b) safe invalidation/eviction, and (c) a correctness strategy for when upstream data changes (generation/TTL + bounded size + test reset).

Idempotent normalization pipelines

When code must normalize or classify structured data (messages, request bodies, model/provider selection, metrics), implement it as deterministic, idempotent, and non-mutating transformations.

Apply these rules:

Example (non-mutating consecutive-role merge + idempotent ops sketch):

type Content = { role: string; parts: Array<{ text: string }> };

function mergeConsecutiveSameRole(contents: Content[]): Content[] {
  const merged: Content[] = [];
  for (const entry of contents) {
    const last = merged[merged.length - 1];
    if (last && last.role === entry.role) {
      // mutate only the new output copy
      last.parts.push(...entry.parts);
    } else {
      // prevent caller mutation
      merged.push({ ...entry, parts: [...entry.parts] });
    }
  }
  return merged;
}

type Op = { kind: 'prepend_system_block'; text: string; idempotencyKey?: string };
function applyOp(blocks: Array<{ type: 'text'; text: string }>, op: Op) {
  const alreadyThere = blocks.some((b) => b.text === op.text || b.text.startsWith(op.text));
  return alreadyThere ? blocks : [{ type: 'text', text: op.text }, ...blocks];
}

Practical checklist:

Hot Path Cost Controls

In performance-sensitive paths, actively control hidden costs: cap buffering, reuse expensive compiled primitives, and avoid full serialization/large allocations.

Apply these rules: 1) Bound buffering/reads in streaming probes

Example pattern (mimicking the approach):

if !preReadUntilContent && len(buffered) >= maxPreReadEvents {
    break // stop buffering; avoid response delay/unbounded growth
}

2) Hoist heavy compiled expressions

3) Replace full serialization with cheap deterministic signatures

Example pattern (self-contained):

func associationSignature(a []*MyAssociation) uint64 {
    h := fnv.New64a()
    for _, x := range a {
        // Write only semantically relevant fields
        h.Write([]byte(x.TargetType))
        h.Write([]byte{byte(x.Enabled)})
        for _, c := range x.Conditions {
            h.Write([]byte(c.Key))
            h.Write([]byte(c.Operator))
            h.Write([]byte(c.Value))
        }
    }
    return h.Sum64()
}

4) Test what changed semantics

Sanitize Upstream Stream Errors

When handling upstream streaming responses (SSE/streaming APIs), treat any non-stream payload (e.g., HTML error pages) as untrusted input: do not pipe it through streaming transforms, and do not reflect raw upstream text/HTML into client errors.

Actionable standard:

Example pattern:

const upstreamContentType = (providerResponse.headers.get('content-type') || '').toLowerCase();
if (
  upstreamContentType &&
  !upstreamContentType.includes('text/event-stream') &&
  !upstreamContentType.includes('application/json')
) {
  const bodyText = await providerResponse.text().catch(() => '');
  const titleMatch = bodyText.match(/<title>([^<]+)<\/title>/i);

  // Sanitized + clamped message (never reflect raw upstream HTML)
  const shortMsg =
    titleMatch?.[1]?.trim() ||
    (bodyText.length < 200 ? bodyText.trim() : `Upstream returned ${upstreamContentType}`);
  const safeMsg = shortMsg.replace(/<[^>]*>/g, '').slice(0, 160);

  return json({ error: safeMsg }, { status: providerResponse.status || 502 });
}

Consistent Test ID Naming

When adding or modifying UI elements, ensure their test/automation identifiers follow the existing naming conventions and remain stable.

Apply these rules:

Example patterns:

// Prefer prefix props when the component generates IDs
<ContactUsView
  testIdPrefix="license"
  title="Unlock license management"
  // generated IDs like: license-read-more / license-book-demo
/>

// Follow the local convention when setting explicitly
<Switch
  id="dump-errors-in-console-logs"
  data-testid="client-settings-dump-errors-switch"
  // ...
/>

This keeps automated tests resilient and avoids one-off/unstable identifiers that break CI and slow down debugging.

Config precedence and canonical sources

Establish a single, explicit source-of-truth approach for configuration resolution: (1) honor user-supplied overrides first, (2) otherwise derive from the active context/environment, and (3) only then fall back to defaults; additionally, treat repository-level configuration/lock files as canonical entrypoints for developer tooling.

How to apply:

Example (pattern for precedence):

export async function runLogsCommand(opts = {}) {
  const baseUrl =
    opts.baseUrl ||
    opts["base-url"] ||
    getBaseUrl({ context: opts.context }); // derived from active context/env

  // ...use baseUrl to build requests
}

Await Connection Drains

When async work depends on a connection that may still be resolving, prevent startup race conditions by (1) deferring the work to an “early” queue instead of silently skipping, and (2) awaiting the connection setup promises before concluding/awaiting the deferred work.

Apply this by standardizing the pattern:

Example pattern:

const earlyCostingQueue = [];
const connectionPromises = [];
const pendingVerifications = [];

function verifyLaterOrQueue({ reqId, name }) {
  const expectCost = getHeader(/* request */ null, 'x-bf-expect-cost');
  if (expectCost && /^(1|true|yes)$/i.test(String(expectCost))) {
    if (!logsDbReady || !logsDb) {
      // Defer instead of silently skipping
      earlyCostingQueue.push({ reqId, name });
      return;
    }
    pendingVerifications.push(verifyCostingRequest(logsDb, reqId, name, results, silent));
  }
}

function drainCostingQueue(activeLogsDb) {
  while (earlyCostingQueue.length > 0) {
    const item = earlyCostingQueue.shift();
    pendingVerifications.push(verifyCostingRequest(activeLogsDb, item.reqId, item.name, results, silent));
  }
}

// Track connection resolution so drains happen before final wait
connectionPromises.push(connectLogsDb().then(activeLogsDb => {
  drainCostingQueue(activeLogsDb);
}));

// Finalization: await drains before awaiting pending work
Promise.allSettled(connectionPromises)
  .then(() => Promise.allSettled(pendingVerifications))
  .then(() => done());

Also ensure your core verification logic runs before any status-based early returns (e.g., before “2xx-skip”) so failure/cancelled cases aren’t accidentally bypassed.

Harden Client URL Fetches

Any client-controlled URL (including OpenAI image_url) must be treated as SSRF-/DoS-/parsing-risk input and validated at every transition before connecting.

Apply this standard:

Example pattern (server-side fetch):

async function safeFetchImageBytes(inputUrl: string) {
  // 1) Parse + enforce strict outbound policy
  let current = inputUrl;
  for (let hop = 0; hop <= 3; hop++) {
    const parsed = parseAndValidatePublicUrl(current); // strict scheme/host policy

    // 2) DNS rebinding defense (reject if any IP answer is private)
    // validateResolvedIps(parsed.hostname)

    // 3) Manual redirect handling
    const res = await fetch(parsed.toString(), { method: 'GET', redirect: 'manual' });
    if (res.status >= 300 && res.status < 400 && res.headers.get('location')) {
      current = new URL(res.headers.get('location')!, parsed).toString();
      continue; // re-validate next hop
    }

    // 4) Verify content-type and read with a size cap
    const ct = (res.headers.get('content-type') || '').toLowerCase();
    if (!ct.startsWith('image/')) throw new Error('Image content-type required');
    const data = await readCapped(res, 1 * 1024 * 1024);
    return data;
  }
  throw new Error('Too many redirects');
}

Secure postMessage handling

When handling cross-window postMessage, don’t rely on a single window.location.origin check—loopback flows (e.g., localhost vs 127.0.0.1) can be legitimate but have different origins. For security, combine: (1) strict message type checking, (2) a whitelist of allowed origins that includes known legitimate loopback variants, and (3) a correlation ID match (e.g., loginTraceId) to reject unsolicited/injected messages.

Example pattern:

const allowedOrigins = new Set([
  'http://localhost:3000',
  'http://127.0.0.1:3000',
]);
const expectedTraceId = traceIdRef.current;

function onMessage(ev: MessageEvent) {
  if (!allowedOrigins.has(ev.origin)) return;

  const m = ev.data as {
    type?: string;
    loginTraceId?: string;
    success?: boolean;
    error?: string;
  };

  if (m?.type !== 'trae-oauth-callback') return;
  if (expectedTraceId && m.loginTraceId !== expectedTraceId) return;

  // handle verified message
}

Apply this anywhere you parse postMessage for auth/callback flows: accept only known origins, verify the message shape/type, and require a per-attempt correlation token.

Regex Guardrails for Filters

When implementing CLI output filters (regex/pattern matchers), make the match logic algorithmically safe: add explicit bypass guards for structured/machine-readable output and use precise patterns for command variants, then lock behavior with include/exclude regression tests.

Apply this standard: 1) Add structured-output bypasses

2) Use explicit alternation for CLI syntax variants

3) Test both inclusion and exclusion

Example (pattern-bypass + explicit variants):

const kubectlMatch = {
  outputTypes: ['kubectl'],
  commands: [
    // match kubectl actions
    '^kubectl\s+(?:get|describe|logs|apply|delete|rollout|exec|top|events?)\b',
    // bypass if structured output is requested (illustrative)
    '^(?!.*-o\s+(?:json|yaml|jsonpath|go-template)\b).*kubectl\s+(?:get|describe)\b'
  ]
};

(Shape the bypass to each CLI’s actual flags/outputs.)

This prevents corrupted structured data, reduces over-matching, and makes the filter’s behavior predictable over time.

Disable install scripts

In security-sensitive build environments (e.g., Docker/CI), prevent dependency install/postinstall hooks from executing arbitrary code. Use npm ci with --ignore-scripts to reduce supply-chain risk, but explicitly rebuild and validate any known native/runtime dependency that truly must compile/bind for the target platform.

Example (pattern):

# Require reproducible installs
RUN test -f package-lock.json \
  || (echo "package-lock.json is required" >&2 && exit 1)

# Block broad dependency script execution
RUN npm ci --no-audit --no-fund --legacy-peer-deps --ignore-scripts \
  && npm rebuild better-sqlite3 \
  && node -e "require('better-sqlite3')(':memory:').close()"

Apply this standard by:

Type-safe query contracts

When a UI route consumes or produces URL search/query parameters, treat those parameters as a contract: validate and type them at the route boundary, update them through typed helpers (no as any), and normalize/omit default-derived fields so the resulting query set is consistent.

Apply:

Example (TypeScript):

// Route boundary: validate search
// (Exact API depends on your router, but the idea is: no implicit/unchecked shape)
// validateSearch: (s) => Record<string, unknown>

function updateRequestSearch(
  apply: (draft: Record<string, unknown>) => void,
) {
  const nextSearch = (prev: Record<string, unknown> | undefined) => {
    const draft: Record<string, unknown> = { ...(prev ?? {}) };
    apply(draft);
    return draft;
  };

  navigate({
    search: nextSearch as any, // only if your router typing requires; ideally remove this via proper validateSearch types
    replace: true,
  });
}

function onDateRangeChange(range?: DateTimeRangeValue) {
  updateRequestSearch((draft) => {
    const normalized = range ? normalizeDateTimeRangeValue(range) : undefined;

    if (!normalized || (!normalized.from && !normalized.to)) {
      delete draft.createdAtFrom;
      delete draft.createdAtTo;
      delete draft.createdAtStartTime;
      delete draft.createdAtEndTime;
      return;
    }

    if (normalized.from) draft.createdAtFrom = formatSearchDate(normalized.from);
    else delete draft.createdAtFrom;

    if (normalized.to) draft.createdAtTo = formatSearchDate(normalized.to);
    else delete draft.createdAtTo;

    // Only keep time fields when the corresponding date side exists,
    // and omit defaults (e.g., 00:00:00 / 23:59:59) in your formatter.
    draft.createdAtStartTime = formatSearchTime(normalized.startTime);
    draft.createdAtEndTime = formatSearchTime(normalized.endTime);
  });
}

Externalize config copy

Treat user-facing UI text (help text, labels, descriptions, toggle text) as configuration: do not hardcode it in components. Instead, source it from the appropriate translation/config namespace using the team’s i18n helpers. Keep only upstream/system-provided constants (e.g., IDs, template metadata) as raw values when they are not user copy.

Apply this when adding/modifying UI around configuration features (e.g., API key scopes, access descriptions, combo/catalog descriptions).

Example pattern:

// en.json (or similar)
{
  "apiManager": {
    "managementAccessDesc": "Allow this API key to manage OmniRoute configuration."
  },
  "combos": {
    "autoCatalogTitle": "Auto-routing catalog",
    "autoCatalogDescription": "Built-in auto/* combos resolved dynamically from connected providers."
  }
}

import { useTranslations } from "next-intl";

const tApi = useTranslations();
const tCombos = useTranslations("combos");

export function Example() {
  return (
    <p>
      {tApi("apiManager.managementAccessDesc")}
      {/* or {tApi("managementAccessDesc")} if already scoped consistently */}
    </p>
  );
}

export function AutoComboHeader() {
  return (
    <h2>{tCombos("autoCatalogTitle")}</h2>
  );
}

Checklist:

Local Clarity Standards

When updating code, optimize for local clarity and correctness, not “cleanup” by indirection.

Apply these rules: 1) Prefer locality over forced DRY

2) Improve readability with explicit intent

3) Type-safety as a style baseline

4) Preserve formatting in content transforms

Example (readability + type narrowing):

// Prefer guard-driven simplification over redundant casts
if (aliases && parsed.model) {
  const directTarget = aliases[parsed.model];
  if (typeof directTarget === "string" && directTarget.includes("/")) {
    const slashIdx = directTarget.indexOf("/');
    if (slashIdx !== -1) {
      const providerPart = directTarget.slice(0, slashIdx);
      const modelPart = directTarget.slice(slashIdx + 1);
      // ...
    }
  }
}

Example (format-preserving cleaning):

function cleanResponse(text: string, strip = true): string {
  let t = text;
  // Remove only known unwanted tags/markers
  t = t.replace(/<[?]xml[^?]*[?]>/g, "");
  // Intentionally DO NOT collapse spaces/newlines, to preserve markdown/code formatting
  return strip ? t.trim() : t;
}

HTTP Timeout Strategy

Standardize how you apply timeouts for outbound HTTP requests.

Example pattern (non-stream):

// Client: no Timeout when you need long-lived streaming elsewhere
client := &http.Client{Transport: transport} // transport.ResponseHeaderTimeout set

ctx, cancel := context.WithTimeout(context.Background(), nonStreamCallTimeout)
defer cancel()

req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewBuffer(jsonData))
resp, err := client.Do(req)
// handle resp

Example pattern (client Timeout already set):

// NewModel sets httpClient.Timeout (e.g., 120s) and you don't pass ctx through
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
resp, err := httpClient.Do(req) // no per-request context timeout wrapper

Apply this consistently across drivers so timeout behavior is predictable for both normal responses and SSE/streaming.

Strict External API Validation

When integrating with external AI/HTTP APIs, treat both transport errors and payload validity as first-class failures. Validate inputs, verify provider/HTTP status, and enforce response invariants—never silently accept partial or malformed data.

Apply these rules:

Example pattern (fail fast + invariant checks):

resp, err := client.Do(req)
if err != nil { return nil, fmt.Errorf("send request: %w", err) }
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil { return nil, fmt.Errorf("read response: %w", err) }

if resp.StatusCode != http.StatusOK {
  return nil, fmt.Errorf("API error: %s body=%s", resp.Status, string(body))
}

// provider-level status (if present)
// var base BaseResp; json.Unmarshal(body,&base); if base.StatusCode!=0 { return nil, ... }

// invariants
embeddings := make([][]float64, len(texts))
for _, item := range parsed.Data {
  if item.Index < 0 || item.Index >= len(texts) {
    return nil, fmt.Errorf("unexpected index %d for %d inputs", item.Index, len(texts))
  }
  if len(item.Embedding) == 0 {
    return nil, fmt.Errorf("empty embedding for index %d", item.Index)
  }
  // convert with type guards to float64
}

Goal: errors should be deterministic and informative, and incorrect provider payloads should never silently degrade results (e.g., via skipped indices or indistinguishable default zeros).

Shared API Contracts

Design API endpoints so they orchestrate transport only, and keep domain logic + contract identifiers centralized.

Apply these rules: 1) Don’t couple libraries to route handlers: HTTP-specific code (request/response/headers/params) stays in route.ts; core behavior (embeddings, memory indexing, etc.) lives in shared services used by both the API route and other server modules. 2) Single source of truth for backend-recognized identifiers: any client/UI option that toggles engine behavior (rule names, modes, etc.) must use backend canonical constants/types—never “fabricated” strings.

Example pattern (service extraction):

// src/lib/embeddings/service.ts
export async function generateEmbeddings(input: string) {
  // domain logic here (no fetch/Response/route concerns)
}

// src/app/api/v1/embeddings/route.ts
import { generateEmbeddings } from "@/lib/embeddings/service";

export async function POST(req: Request) {
  const { input } = await req.json();
  const embeddings = await generateEmbeddings(input);
  return Response.json({ embeddings });
}

Example pattern (contract identifiers):

// shared/shared-rules.ts (or import from backend constant module)
export const ALL_CAVEMAN_RULES = [
  /* values copied/imported from backend canonical list */
] as const;

// UI uses the canonical list (no guessed strings)
import { ALL_CAVEMAN_RULES } from "@/shared/shared-rules";

If an option fails to work (e.g., skip/toggle doesn’t take effect), treat it as an API contract mismatch and trace the identifier back to its backend source of truth.

API Contract Verification

When adding/updating model-to-endpoint configuration, verify it against the provider’s official API docs and enforce the expected request/response contract.

Apply this standard: 1) Correct endpoint/path per model

2) Confirm response shape/schema compatibility

3) Add defensive validations for upstream/proxy anomalies

Example pattern (Go-like pseudocode):

// 1) Endpoint routing (per-model, docs-backed)
func urlSuffix(providerCfg map[string]string, model string) string {
  return providerCfg[model] // e.g., rerank models can have different suffixes
}

// 2) Defensive response validation
type EmbeddingData struct {
  Index     int
  Embedding []float64
  Object    string
}

type EmbeddingResponse struct {
  Data  []EmbeddingData
  Model string
  Object string
}

func normalizeEmbeddings(resp EmbeddingResponse, expectedCount int) ([][]float64, error) {
  if len(resp.Data) != expectedCount {
    return nil, fmt.Errorf("embedding count mismatch")
  }

  // Guard against duplicates/out-of-range
  seen := map[int]bool{}
  out := make([][]float64, expectedCount)
  for _, d := range resp.Data {
    if d.Index < 0 || d.Index >= expectedCount {
      return nil, fmt.Errorf("embedding index out of range")
    }
    if seen[d.Index] {
      return nil, fmt.Errorf("duplicate embedding index")
    }
    seen[d.Index] = true
    out[d.Index] = d.Embedding
  }
  return out, nil
}

Keep the docs link/source close to the mapping so future changes remain traceable and reviewable.

Bound Work and Queries

When improving performance, make sure code neither (a) fetches too much data, nor (b) does the same expensive work repeatedly, nor (c) runs unbounded workloads.

Practical rules

  1. Target by primary keys / narrow filters: Prefer get_by_id-style lookups over broad tenant-scoped queries when you already have the identifier.
  2. Push filtering to the datastore: If you currently fetch large result sets and filter in Python (or run many queries), move the filter into the ES/DB/search criteria (e.g., include doc_id / source_id in the search condition).
  3. Bound user-controlled heavy parameters: Add upper bounds for things like top_k so a bad request can’t trigger massive retrieval.
  4. Batch/chunk heavy processing: For page/document loops, use page batches to control memory/CPU and keep progress reporting accurate.
  5. Avoid repeated expensive work: Hoist invariant computations out of inner loops, and run file-level expensive steps once (not per sheet/per iteration).
  6. Resource-heavy ops must be policy-driven: If you unload/reload models or enable tracing, ensure it’s safe under concurrency (don’t unload while tasks run) and avoid enabling heavy instrumentation by default.

Examples

Fail-Closed Security Invariants

Default to secure behavior when you can’t verify. If a security-relevant check (e.g., whether encrypted credentials exist) or its dependencies (e.g., DB access) fails, treat the state as unsafe and stop or require explicit recovery—never assume “nothing to decrypt.” Also, preserve cross-component encryption/decryption invariants (prefix/salt/format) so that verification/encryption remains compatible.

Apply it:

Example (fail-closed encrypted credential inspection):

function hasEncryptedCredentials(dataDir) {
  const dbPath = join(dataDir, 'storage.sqlite');
  if (!existsSync(dbPath)) return false;

  try {
    const Database = require('better-sqlite3');
    const db = new Database(dbPath, { readonly: true, fileMustExist: true });
    try {
      const row = db.prepare(`
        SELECT 1
        FROM provider_connections
        WHERE access_token LIKE 'enc:v1:%'
           OR refresh_token LIKE 'enc:v1:%'
           OR api_key LIKE 'enc:v1:%'
           OR id_token LIKE 'enc:v1:%'
        LIMIT 1
      `).get();
      return !!row;
    } finally {
      db.close();
    }
  } catch {
    // Fail closed: treat inspection failure as unsafe/unknown
    throw new Error('Failed to inspect encrypted credentials; aborting setup/bootstrap.');
  }
}

Centralized input guards

Require a shared, spec-aligned “guard” step for any untrusted input used in security-sensitive contexts (SQL construction, dynamic table names, outbound URL fetch). Fail closed (raise a safe exception) before interpolation/requests, and reuse a single utility to avoid security-policy drift.

Practical rules:

Example pattern (UUID guard):

import uuid
import logging
logger = logging.getLogger(__name__)

def assert_valid_uuid(value: str, *, label: str) -> str:
    try:
        # Validate; optionally canonicalize to the storage format your SQL expects
        return uuid.UUID(str(value)).hex  # e.g., 32-char form
    except (ValueError, TypeError):
        logger.warning("Rejected invalid %s (len=%d)", label, len(str(value)))
        raise ValueError(f"Invalid {label} format")

# Use only after guard
kb_hex = assert_valid_uuid(kb_ids[0], label="kb_id")
table_name = f"ragflow_{tenant_id}_{kb_hex}"

Example pattern (SSRF logging hygiene):

# Instead of logging full URL:
logger.warning("SSRF guard blocked URL (scheme=%r host=%r)", scheme, parsed.hostname)

Centralize input bounds

When validating user-controlled numeric inputs, define allowed min/max values in one place and reuse them across both the UI/component props and the validation schema. Prevent mismatches where the UI permits one range but the schema enforces another (or vice versa), since this breaks input validation guarantees.

Example:

const TOP_N_MAX = 200;

export const topnSchema = {
  top_n: z.number().max(TOP_N_MAX).optional(),
};

function TopNFormField({ max = TOP_N_MAX, ...props }: { max?: number }) {
  // Ensure any passed `max` defaults/constraints align with TOP_N_MAX.
  return /* render */ null;
}

Practical checklist:

Test isolation and tiers

When adding or changing behavior, require unit tests that are (a) comprehensive for the logic contract, (b) correctly prioritized/selected via pytest markers, and (c) isolated from global state to avoid cross-suite flakiness.

Apply it like this:

Example pattern for isolating sys.modules stubs:

import sys
import pytest
from types import ModuleType

@pytest.fixture(autouse=True)
def isolate_sys_modules():
    original = sys.modules.copy()
    try:
        yield
    finally:
        sys.modules.clear()
        sys.modules.update(original)

def test_something_with_stubbed_google_lib():
    sys.modules["googleapiclient"] = ModuleType("googleapiclient")
    # ... import/use code under test safely ...

Additionally, for refactors that change call paths, either rely on the expanded unit tests above or attach lightweight scenario-based verification to show behavior is unchanged.

Null-safe input handling

Apply null-safety consistently for request payloads, optional dict keys, and nullable fields. Specifically:

Example pattern:

# 1) Guard None before comparisons
batch_size = max(1, int(os.getenv("PDF_PARSER_PAGE_BATCH_SIZE", "50")))
if total_pages is None or total_pages <= batch_size:
    run_single_pass()

# 2) Safe dict key access
content = chunk.get("content")
if not content:
    content = chunk.get("content_with_content") or ""

# 3) Avoid serializing None -> "None"
text = str(item.get("text") or "").strip()
if text:
    lines.append({"text": text})

# 4) Safe list merge
node0_attrs.setdefault("source_id", [])
node0_attrs["source_id"].extend(node1_attrs.get("source_id", []) or [])
node0_attrs["source_id"] = sorted(set(node0_attrs["source_id"]))

This prevents null-reference crashes and stops incorrect downstream data (like literal “None” strings) from being stored or emitted.

Deterministic Mapping Pipeline

When implementing parsing/storage/retrieval logic (or any pipeline with downstream queries), design the algorithm so that representations, keys, and batching decisions are stable and consistent:

Example pattern (parallel token/raw + bounded, deterministic detection):

# 1) Parallel values: keep token fields tokenized
if role in ("vectorize", "both"):
    chunk[f"{typed_key}_tks"] = tokenize(value)
if role in ("metadata", "both"):
    chunk[f"_raw_{col}"] = str(value)  # human-readable for aggregation/UI

# 2) Bounded windowing
for page_from in range(0, total_pages, batch_size):
    page_to = min(page_from + batch_size, total_pages)
    load_pages(page_from=page_from, page_to=page_to)

# 3) Deterministic detector sampling
sample = page_chars[:200]  # not random.sample
is_garbled = detect(sample, threshold=0.3)

Adopting this standard prevents subtle retrieval/search regressions, reduces heuristic flakiness, and improves correctness/performance of chunked algorithms.

Unified Error Handling

Adopt a single error-handling standard across LLM/provider layers:

1) Retry only transient failures (including connection jitter) with bounded, classified backoff

2) Keep return/output contracts stable between success and failure

3) Standardize failure encoding and checking at boundaries

Example (type-stable normalization before setting output):

try:
    transcription = seq2txt_mdl.transcription(tmp_path)
    # success may be tuple(text, token_count)
    txt = transcription[0] if isinstance(transcription, tuple) else transcription
except Exception as e:
    logging.warning(f"Transcription failed: {e}")
    txt = ""

self.set_output("text", txt)

Example (standard retry rule for transient polling blips):

@retry
def _describe_task_status(self, req):
    return self.client.DescribeTaskStatus(req)

while retries < max_retries:
    resp = self._describe_task_status(req)  # transient 429/5xx/DNS blips survive

Result: fewer hidden failure modes, consistent caller behavior, and reliable recovery for transient LLM/API issues.

Use Clear Identifiers

Use semantically meaningful, convention-compliant names everywhere (parameters, variables, fields, classes, constants), and avoid ad-hoc renaming/mapping that can create ambiguity or incorrect field exposure.

Apply these rules:

Example (naming parameters by role):

def update_metadata_to(existing_metadata: dict, incoming_metadata: dict) -> dict:
    """Merge incoming metadata payload into an existing metadata mapping."""
    existing_metadata.update(incoming_metadata)
    return existing_metadata

Extract Shared Validation

Prefer small shared helpers for repeated request parsing/validation, and use modern, consistent type annotations to keep code readable and uniform.

Apply this when you see the same “get field → type-check → return error” logic duplicated across endpoints or functions.

Example (shared helper with validation):

def _parse_reference_metadata(req: dict):
    extra_body = req.get("extra_body") or {}
    if extra_body and not isinstance(extra_body, dict):
        return get_error_data_result("extra_body must be an object."), False, None

    reference_metadata = extra_body.get("reference_metadata") or {}
    if reference_metadata and not isinstance(reference_metadata, dict):
        return get_error_data_result("reference_metadata must be an object."), False, None

    metadata_fields = reference_metadata.get("fields")
    if metadata_fields is not None and not isinstance(metadata_fields, list):
        return get_error_data_result("reference_metadata.fields must be an array."), False, None

    return None, bool(reference_metadata.get("include", False)), metadata_fields


async def agents_completion_openai_compatibility(tenant_id, agent_id):
    req = await get_request_json()
    err, include_reference_metadata, metadata_fields = _parse_reference_metadata(req)
    if err:
        return err
    # ...use include_reference_metadata/metadata_fields...

Style rules to enforce with this approach:

Use Shared UI Primitives

When building UI, follow the project’s existing design-system components and tokens instead of creating bespoke patterns.

Standards

  1. Prefer shared form/UI building blocks: Use existing components like RAGFlowFormItem, SimilaritySliderFormField, and SearchInput rather than re-implementing the same UI logic/styling.
  2. Use standard shadcn components for complex layouts: For table-like lists, use the shadcn Table component (not a hand-rolled <table> with custom class logic).
  3. Extract complex dialogs into standalone components/files: If a component contains a full Dialog implementation (multi-field forms, footers, local state), move it to a dedicated file/component.
  4. Use global design tokens/colors: Prefer color variables from tailwind.css over ad-hoc inline style={{ backgroundColor: ... }} so theming stays consistent.

Example (extraction + token usage)

// skills/components/CreateSpaceDialog.tsx
import { Dialog, DialogContent, DialogFooter, DialogHeader, DialogTitle, DialogDescription } from '@/components/ui/dialog';
import { Button } from '@/components/ui/button';
import { Input } from '@/components/ui/input';

export function CreateSpaceDialog(props: {
  open: boolean;
  onOpenChange: (v: boolean) => void;
  value: string;
  onChange: (v: string) => void;
  onCreate: () => void;
  onCancel: () => void;
}) {
  return (
    <Dialog open={props.open} onOpenChange={props.onOpenChange}>
      <DialogContent className="sm:max-w-[425px]">
        <DialogHeader>
          <DialogTitle>Create New Skill Space</DialogTitle>
          <DialogDescription>Organize and manage your skills.</DialogDescription>
        </DialogHeader>
        <div className="py-4">
          <label className="text-sm font-medium mb-2 block">Space Name</label>
          <Input value={props.value} onChange={(e) => props.onChange(e.target.value)} />
        </div>
        <DialogFooter>
          <Button variant="outline" onClick={props.onCancel}>Cancel</Button>
          <Button onClick={props.onCreate} disabled={!props.value.trim()}>Create</Button>
        </DialogFooter>
      </DialogContent>
    </Dialog>
  );
}

Applying these keeps UI consistent, reduces duplicated styling/behavior, improves readability, and makes future theme/UI updates cheaper.

Centralize API Interfaces

Enforce consistent API client construction: (1) centralize all REST endpoint paths and interface definitions in a shared module (e.g., api.ts) to prevent scattered string literals, and (2) when reading/writing request inputs like query params, use the correct standard APIs and keep TypeScript types aligned.

Apply as follows:

Example:

// web/src/api.ts
export const API_PREFIX = '/api/v1';
export const skillSpaceRoutes = {
  list: () => `${API_PREFIX}/skills/spaces`,
  detail: (spaceId: string) => `${API_PREFIX}/skills/spaces/${spaceId}`,
};

// web/src/services/skill-space-service.ts
import request from '@/utils/request';
import { skillSpaceRoutes } from '@/api';

class SkillSpaceService {
  private async request<T>(method: string, url: string, data?: any, params?: any): Promise<T> {
    const response: any = await request(url, { method: method as any, data, params });
    const jsonData = response?.data ?? response;
    if (jsonData?.code !== 0) throw new Error(jsonData?.message || 'Request failed');
    return jsonData.data;
  }

  async listSpaces() {
    return this.request<{ spaces: any[]; total: number }>('GET', skillSpaceRoutes.list());
  }
}

And for query params:

const getParams = (searchParams: URLSearchParams) => {
  const out: Record<string, string> = {};
  for (const [key, value] of searchParams.entries()) {
    out[key] = value;
  }
  return out;
};

Typed API Contracts

For each API endpoint, treat the request as a strict contract: define a dedicated typed request model for the exact payload shape, validate required inputs early, and keep auth/tenant scoping and client URL construction consistent.

Apply:

Example (endpoint + Pydantic request validation):

from pydantic import BaseModel, Field
from flask import request

class UpdateMetadataSettingReq(BaseModel):
    metadata: dict
    enable_metadata: bool | None = None
    built_in_metadata: dict | None = None

@app.route('/document/update_metadata_setting', methods=['POST'])
@token_required
def update_metadata_setting(tenant_id):
    req = UpdateMetadataSettingReq.model_validate(request.json)

    update_payload = {"metadata": req.metadata}
    if req.enable_metadata is not None:
        update_payload["enable_metadata"] = req.enable_metadata
    if req.built_in_metadata is not None:
        update_payload["built_in_metadata"] = req.built_in_metadata

    # additionally enforce tenant ownership before updating
    DocumentService.update_parser_config(doc_id=req_id, update_payload=update_payload)
    return get_json_result(data=True)

This prevents payload mismatches, inconsistent behavior across endpoints, and security/ownership mistakes while keeping API clients predictable (stable URL/version and auth header conventions).

PascalCase constants

Use a consistent PascalCase naming convention for constants (especially exported ones). Avoid mixed styles like snake_case, and keep acronym casing consistent (e.g., API).

When you rename a constant to match the standard, update all references/usages together (including “lite”/variant constants) to stay DRY and prevent mismatches.

Example:

// ❌ Avoid snake_case
// const api_host = `/v1`;

// ✅ Prefer PascalCase for constants, consistent acronym casing
const webAPI = `/v1`;

export const MarkdownRemarkPlugins = [remarkGfm, remarkMath, remarkBreaks];
export const MarkdownRemarkPluginsLite = [remarkGfm, remarkBreaks];

Config Contracts And Defaults

Treat environment variables and configuration flags as a documented contract: the code must (1) read settings from the correct source with safe defaults, (2) make feature-flag behavior explicit (including what becomes a true no-op when disabled), and (3) validate any external runtime dependency that the integration relies on.

Apply this standard:

Safe configuration changes

When updating configuration/env logic, treat it as compatibility-sensitive: preserve existing default behavior and env var formats, avoid enabling config blocks that silently change runtime semantics, and ensure config typing and dependent settings stay consistent.

Apply these rules:

Example pattern (YAML typing):

# Good: numeric port
port: '${MYSQL_PORT:-3306}'

# Avoid patterns that can cause type issues depending on templating/YAML parsing rules
# e.g., port: '3306' (string) if the consumer expects an int

Nil-safe assignment

When fields/items are optional, treat nil/empty as a first-class state: guard against nil dereferences, sanitize outputs, and never overwrite meaningful non-nil data with nil.

Apply:

Example:

// Nil-aware mapping: don’t overwrite with nil
if len(m.ReasoningDetails) > 0 {
    // ... build from details ...
} else if m.Reasoning != nil {
    m.ReasoningContent = m.Reasoning
}

// Defensive fallback for empty payload
if len(responseBody) > 0 {
    persistAggregatedResponse(ctx, responseBody, meta)
} else {
    // aggregated “completed” doesn’t guarantee responseBody presence
    persistResponseChunks(ctx)
}

// Stream sanitization: filter nil items
for _, item := range items {
    if item == nil {
        continue
    }
    emit(item)
}

Non-Blocking Async Recovery

Concurrency standard: (1) keep async code non-blocking, (2) don’t change sync/async boundaries unless callers support it, and (3) make recovery/state transitions race-safe.

Rules 1) Never call blocking (sync) work from async tasks. If you must use a sync API (e.g., llm.chat()), offload it (or refactor to an async client). 2) Match sync/async signatures to callers. If a function is only called from sync code, keep it sync (don’t make it async just to “fit”); if callers are async, then the callee can be async. 3) Recovery/requeue logic must avoid TOCTOU. When detecting “orphaned” or incomplete tasks, use an atomic snapshot, transaction, or a re-check/lock so you don’t incorrectly downgrade DONE → FAIL. 4) For time-sensitive heartbeat/recovery, avoid blocking the event loop. If Redis operations or other work are blocking, run them in a dedicated worker thread/process or ensure the client calls are truly non-blocking. 5) Use explicit stop coordination for graceful shutdown (e.g., stop_event set by SIGTERM/SIGINT) so recovery loops exit cleanly.

Example: offload sync call from async

import asyncio

async def chat_async(chat_model, system_msg, hist, conf):
    # chat_model.chat is sync; offload so the event loop stays responsive
    response = await asyncio.to_thread(chat_model.chat, system_msg, hist, conf)
    return response

Example: avoid TOCTOU in requeue

Applying these rules prevents event-loop stalls, fixes incorrect concurrency boundaries, and eliminates correctness bugs during crash recovery and task requeue.

Consistent Accessible UI

When adding expandable/conditional UI in React/TSX, keep the structure consistent within the surrounding component and ensure accessibility attributes are applied uniformly.

Apply this standard:

Example (toggle + conditional section, with consistent ARIA):

function AdvancedToggle({ showAdvanced, setShowAdvanced }: {
  showAdvanced: boolean;
  setShowAdvanced: (v: boolean) => void;
}) {
  const id = "advanced-settings";

  return (
    <>
      <button
        type="button"
        className="text-sm text-text-muted hover:text-text-primary flex items-center gap-1"
        aria-expanded={showAdvanced}
        aria-controls={id}
        onClick={() => setShowAdvanced(!showAdvanced)}
      >
        <span
          className={`transition-transform ${showAdvanced ? "rotate-90" : ""}`}
          aria-hidden="true"
        >
          ▶
        </span>
        advanced settings
      </button>

      {showAdvanced && (
        <div id={id} className="flex flex-col gap-3 pl-2 border-l-2 border-border">
          {/* advanced inputs here */}
        </div>
      )}
    </>
  );
}

Fail-Closed Credential Checks

When validating security state (e.g., whether credentials are encrypted, authorized sessions exist, tokens exist, etc.), treat any inability to inspect sensitive data as a security failure—don’t silently default to a permissive/incorrect outcome. In practice:

Example pattern (fail closed on inspection errors):

function hasEncryptedCredentials(dbPath) {
  if (!fs.existsSync(dbPath)) return false;

  try {
    const Database = require("better-sqlite3");
    const db = new Database(dbPath, { readonly: true, fileMustExist: true });
    try {
      const row = db
        .prepare(`
          SELECT 1
          FROM provider_connections
          WHERE access_token LIKE 'enc:v1:%'
             OR refresh_token LIKE 'enc:v1:%'
             OR api_key LIKE 'enc:v1:%'
             OR id_token LIKE 'enc:v1:%'
          LIMIT 1
        `)
        .get();
      return !!row; // only true when confirmed
    } finally {
      db.close();
    }
  } catch (e) {
    // Fail closed: do not treat inspection failure as a safe state.
    throw new Error(`Unable to inspect existing database at ${dbPath}`);
    // Alternatively: return false only if callers interpret false as “block/require re-encryption”.
  }
}

Apply this standard anytime code makes a security decision based on reading/parsing sensitive material (DBs, token stores, config files, key material). If you’re changing security-critical logic, avoid bundling unrelated refactors that could increase runtime/module-loading risk during the fix.

Enforce Layered Reuse

When changing or adding code, keep logic and dependencies clean:

Example (centralize clamping):

// prefer: clamp inside Calculate (source of truth)
first, request, _ := ts.perf.Calculate() // already returns clamped request

Example (no reverse dependency):

Example (reuse utility):

v := lo.ToPtr(123) // instead of custom xptr.IntPtr(123)

Deep-copy cached state

When cached data (e.g., row.settings) is used to initialize mutable UI state (forms, editors, draft objects), do not pass cached object references directly. Always deep-clone cached objects on entry to editable state to prevent unintended cache mutations and ensure copied/duplicated initialization is isolated.

Example pattern:

const settings = currentRow.settings
  ? structuredClone(currentRow.settings)
  : undefined;

// use `settings` as initial/editable state
const initialValues = {
  ...,
  settings,
};

If settings is nested, prefer structuredClone (or an equivalent deep-clone) over shallow spreads to guarantee full isolation.

Document Stable API Contracts

All API reference docs (and examples) must clearly define the public contract vs temporary/hidden behavior, and keep endpoint paths/query params and response fields consistent.

Apply:

Example pattern (streaming caveat):

:::caution NOTE
`client.chat.completions.create(stream=True, ...)` does not return `reference`.
`reference` is only exposed in the non-stream response payload (or via raw non-stream mode).
Use non-stream to access `reference`.
:::

Scoped Provider Transformations

When implementing AI/LLM request transformers (especially for proxies), make provider-specific shaping:

1) Scoped and structured: Clone the incoming llm.Request and apply provider/CLI-specific changes via existing structured fields before delegating to the shared/base transformer whenever possible. Avoid raw []byte request-body manipulation unless the field truly can’t be represented structurally.

2) No shared-model pollution: If a change (e.g., tool-name prefixing) would break assumptions for other transformers, ensure the change is applied only within the cloned request or handled by the provider-specific transformer layer.

3) Idempotent system prompt injection: If your proxy adds a provider system message (e.g., Claude Code), first detect whether the request already contains that system prompt (or any system prompt) to prevent duplicates.

Example (idempotent system injection + scoped preprocessing):

func (t *ClaudeCodeTransformer) TransformRequest(ctx context.Context, llmReq *llm.Request) (*httpclient.Request, error) {
	if llmReq == nil { return nil, fmt.Errorf("request is nil") }

	// Always work on a copy to avoid cross-request/state issues.
	reqCopy := *llmReq

	// 1) Idempotent system message injection.
	const sys = "You are Claude Code, Anthropic's official CLI for Claude."
	if !hasSystemPrompt(reqCopy.Messages, sys) {
		systemMsg := llm.Message{
			Role: "system",
			Content: llm.MessageContent{Content: lo.ToPtr(sys)},
		}
		reqCopy.Messages = append([]llm.Message{systemMsg}, reqCopy.Messages...)
	}

	// 2) Provider-specific structured preprocessing (only what the model supports).
	// Apply tool prefixing / cache control / metadata shaping only to reqCopy,
	// and keep other transformers unaffected.
	applyClaudeToolPrefix(&reqCopy)
	// (If a needed field is not representable structurally, then do minimal byte surgery later.)

	return t.Outbound.TransformRequest(ctx, &reqCopy)
}

func hasSystemPrompt(msgs []llm.Message, sys string) bool {
	for _, m := range msgs {
		if m.Role == "system" && m.Content.Content != nil && *m.Content.Content == sys {
			return true
		}
	}
	return false
}

Adopting these rules improves correctness (no duplicated system instructions), maintainability (less brittle byte-level edits), and compatibility (provider specifics don’t unintentionally affect other LLM transformers).

Token and Prompt Fidelity

When implementing LLM/AI APIs (including streaming), treat three things as contract-critical: (1) compute token usage with a real tokenizer, (2) ensure the intended system/user context is actually passed through every planning/tool stage, and (3) emit only the events your client expects.

1) Token usage must use tokenizer-based counting Avoid len(text) for prompt_tokens/total_tokens; it’s character-based and will be wrong across languages/models. Use a tokenizer compatible with your target model.

import tiktoken

tokenizer = tiktoken.get_encoding("cl100k_base")

context_token_used = sum(len(tokenizer.encode(m["content"])) for m in messages)
# use similar encoding for prompt_tokens/completion_tokens

2) Prompt plumbing must be correct If an agent/planning function takes prompt/sys_prompt/history, pass the correct variables. Add a small assertion or unit test to ensure downstream components receive the system prompt intended for the run (not a different variable like history or an uninitialized prompt).

3) Streaming/compatibility endpoints must filter noise If the upstream emits multiple event types (e.g., content vs reference chunks), only forward the subset your client UI consumes. Example logic:

# keep only the LLM response and its related finalization/reference
if event not in ["message", "message_end"]:
    continue

How to apply

Transactional delete and upsert

For database writes that can partially succeed (multi-step deletes/updates, bulk upserts with conflict handling), enforce invariants with atomic transactions, explicit outcome validation, and engine-specific SQL encapsulation.

Apply this standard:

Example (pattern):

@classmethod
@DB.atomic()
def delete_chunks(cls, chunk_ids, doc_id, kb_id):
    # dedupe inputs to make expected counts deterministic
    unique_ids = list(dict.fromkeys(chunk_ids or []))
    if not unique_ids:
        return 0

    condition = {"id": unique_ids, "doc_id": doc_id}
    try:
        deleted_count = settings.docStoreConn.delete(
            condition,
            search.index_name(DocumentService.get_tenant_id(doc_id)),
            kb_id,
        )
    except Exception:
        return 0

    # If partial deletion happened, restore consistency deterministically.
    if deleted_count == 0:
        raise ValueError("chunk deleting failure")
    if deleted_count < len(unique_ids):
        return settings.docStoreConn.delete(
            {"doc_id": doc_id},
            search.index_name(DocumentService.get_tenant_id(doc_id)),
            kb_id,
        )

    return deleted_count

Also ensure bulk upsert/bulk insert logic handles DB-specific conflict semantics (e.g., Postgres vs MySQL) inside the DB utility layer, while still using the same atomicity and validation rules at the call site.

UseEffect Dependency Correctness

Side effects must be written with the correct React lifecycle mental model: useEffect runs after render/commit, not at the place the code is written. Always include a dependency array and make the effect’s inputs explicit so state resets (e.g., initializing defaults when a dialog opens) happen at the right time and without stale reads.

Apply this standard:

Example (dialog reset):

useEffect(() => {
  const initialDeleteValue = chunk_num > 0;
  setDefaultValues({ delete: initialDeleteValue, apply_kb: false });

  if (visible && formCallbackRef.current) {
    formCallbackRef.current.reset({ delete: initialDeleteValue, apply_kb: false });
  }
}, [visible, chunk_num, setDefaultValues]);

Also: extract repeated side-effect logic into hooks when appropriate (to avoid duplicated effect code), but keep each hook’s effect dependencies correct.

Maintain dependency lockfiles

When changing dependency declarations in configuration files like pyproject.toml, treat group placement and lockfile updates as part of the change.

Example (pattern):

[project.optional-dependencies]
# Keep in both groups if both are required
default = ["pycryptodomex==3.20.0"]
test = ["pycryptodomex==3.20.0", "websockets>=14.0"]

After editing, regenerate and commit the matching lockfile (e.g., uv.lock).

Semantic identifier naming

Use semantic, unambiguous names for every externally visible identifier (API parameters/endpoints, config/env vars, and build/version tags). Names must reflect meaning and supported behavior; if an operation is not supported, document that constraint next to the relevant field/behavior.

Application rules:

Example (Docker tag standard):

RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1
RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1-slim
RAGFLOW_IMAGE=infiniflow/ragflow:dev
RAGFLOW_IMAGE=infiniflow/ragflow:nightly

Example (API constraint wording):

POST /api/v1/file/rename
Body: { "file_id": "...", "name": "new_name.txt" }
Note: Changing file extensions is not supported.

Documentation Precision

All documentation must accurately reflect product/API behavior, use exact feature names, and be unambiguous about constraints, defaults, and unsupported scenarios. Also ensure Markdown renders cleanly.

Apply this when editing or adding docs:

Example (API param + constraint precision):

## List files
**GET** `/api/v1/dataset/{dataset_id}/documents?keywords={keyword}&page={page}&page_size={page_size}`

- `page`: 0-based page index.
- `page_size`: number of records returned per page.

## Page rank
- Value type: integer
- Non-integers are rounded down.
- Range: [0, 100]

Documentation Wiring Accuracy

Ensure technical documentation is both semantically correct and operationally precise: (1) use the exact product terms for what a setting applies to (scope/terminology), and (2) when describing component internals/graphs, specify the exact wiring rule for references (use the internal managing node, not an external component), aligned with the documented visibility constraints.

How to apply:

Example (Iteration internal wiring pattern):

:::danger IMPORTANT
To reference the created segments from an added internal component, add a **Reference** variable that equals **IterationItem** within the **Input** section of that internal component. There is no need to reference the corresponding external component, as the **IterationItem** component manages the loop of the workflow for all created segments.
:::

Example (setting scope/terminology):

The variables `MAX_CONTENT_LENGTH` in `/docker/.env` and `client_max_body_size` in `/docker/nginx/nginx.conf` set the file size limit for each upload to a dataset or **File**. These settings DO NOT apply in this scenario.

Use enum constants

When a value represents a named domain concept (e.g., Theme), use the corresponding enum type/constant (e.g., ThemeEnum) instead of duplicating raw string literals ('light', 'dark'). This keeps identifier naming consistent across schema definitions and UI/component logic.

Example pattern:

// Prefer a single source of truth
export enum ThemeEnum {
  Light = 'light',
  Dark = 'dark',
}

// Derive schema from the enum rather than retyping literals
import { z } from 'zod';

const FormSchema = z.object({
  theme: z.enum([ThemeEnum.Light, ThemeEnum.Dark] as [string, string]),
});

// And use enum values in UI/selectors
<RadioGroup
  value={field.value}
  onValueChange={field.onChange}
>
  <RadioGroupItem value={ThemeEnum.Light} id="light" />
  <RadioGroupItem value={ThemeEnum.Dark} id="dark" />
</RadioGroup>

Use exec-form commands

When configuring container commands (e.g., Docker Compose), pass commands in exec/array form for any arguments that include secrets or externally provided values. This prevents shell parsing/whitespace issues and reduces the risk of injection via malformed values (including empty credentials).

Example (safe):

services:
  redis:
    image: valkey/valkey:8
    command: [
      "redis-server",
      "--requirepass", "${REDIS_PASSWORD}",
      "--maxmemory", "128mb",
      "--maxmemory-policy", "allkeys-lru"
    ]

Avoid string-form commands that rely on implicit parsing:

# Not recommended for secrets/external values
command: redis-server --requirepass ${REDIS_PASSWORD} --maxmemory 128mb

Portable Config Management

When changing configuration, treat runtime configuration as the source of truth and avoid embedding/duplicating settings.

Apply these rules: 1) Prefer .env + env_file: .env and avoid duplicating the same variables in multiple places. 2) Do not hard-code environment-specific identifiers like container_name in shared compose files. 3) For frequently changed/overridden configuration (including templates), mount files via volumes rather than baking templates into images that would require rebuilds to take effect. 4) If one setting is coupled to another (e.g., application upload limits and nginx limits), update both together.

Example (compose):

services:
  app:
    env_file: .env
    # no container_name in shared files
    volumes:
      - ./service_conf.yaml:/ragflow/conf/service_conf.yaml

Config Documentation Constraints

When adding or updating configuration-related docs (env vars, config files, feature/transport flags, and deployment commands), ensure the guidance is (1) context-correct, (2) compatibility-checked, and (3) explicit about defaults/overrides.

Apply this checklist:

Example (flag compatibility + defaults):

If you set `mcp-mode` to `host`, you must add `--no-transport-streamable-http-enabled`, because streamable-HTTP transport is not supported in host mode.
The legacy SSE transport and streamable-HTTP (JSON responses) are enabled by default unless explicitly disabled with the corresponding `--no-<flag>` options.

Example (context-correct config guidance):

meaningful exception handling

Exception handling should provide meaningful feedback and use appropriate exception types rather than failing silently or relying on fragile string-based checks. Avoid catching exceptions without proper logging, user feedback, or recovery mechanisms. Use built-in Python exceptions when available instead of creating custom ones unnecessarily.

Key practices:

Example of problematic silent handling:

try:
    result = subprocess.run(libreoffice_command, capture_output=True, text=True)
    # ... process result
except Exception as _:
    # Silent failure - no logging or user feedback
    pass

Better approach with meaningful handling:

try:
    result = subprocess.run(libreoffice_command, capture_output=True, text=True)
    # ... process result  
except subprocess.SubprocessError as e:
    logger.warning(f"LibreOffice conversion failed: {e}")
    return None
except FileNotFoundError:  # Use built-in exception
    raise FileNotFoundError(f"File {path} does not exist")

Optimize data structures

Choose appropriate built-in data structures and collection types to improve code efficiency and readability. Python’s collections module and built-in data structures often provide more elegant and performant solutions than manual implementations.

Key guidelines:

Examples:

# Instead of manual dictionary building:
target_nodes: dict[str, list[GraphEdge]] = {}
for edge in edge_mappings:
    if edge.target_node_id not in target_nodes:
        target_nodes[edge.target_node_id] = []
    target_nodes[edge.target_node_id].append(edge)

# Use defaultdict:
from collections import defaultdict
target_nodes = defaultdict(list)
for edge in edge_mappings:
    target_nodes[edge.target_node_id].append(edge)

# For deduplication, instead of manual loop:
unique_documents = []
doc_ids = set()
for document in documents:
    if document.metadata["doc_id"] not in doc_ids:
        doc_ids.add(document.metadata["doc_id"])
        unique_documents.append(document)

# Use dictionary comprehension:
unique_documents = list({doc.metadata["doc_id"]: doc for doc in documents}.values())

These optimizations reduce code complexity, improve performance, and make intent clearer to other developers.

Use descriptive names

Choose names that clearly describe the purpose, behavior, or content of variables, methods, and classes. Avoid ambiguous or generic names that require additional context to understand.

Key principles:

Examples of improvements:

# Before: Ambiguous naming
def create_alias(user, timeout):
    doc_id = set()

# After: Descriptive naming  
def create_or_update_alias(creator_user_id, timeout_seconds):
    doc_ids = set()

This practice makes code self-documenting and reduces the cognitive load for developers reading and maintaining the codebase.

Agent Prompt Contracts

When building LLM Agents, treat the system prompt + framework prompt blocks as a contract with the agent’s inputs and tool/sub-agent configuration.

Apply these rules:

Example pattern (conceptual):

System prompt (with variables):
- Role + instructions
- Tool policy: “Use Tool A for classification/summarization; use Tool B for retrieval; otherwise ask a clarifying question.”

Framework blocks:
- task_analysis(agent_prompt, task, tool_desc, context) -> produces structured analysis
- plan_generation(...) -> creates the next steps based on that analysis

Keys:
- Provide all variables referenced by system prompt/framework blocks.
- Confirm required keys exist via the keys UI (/ or (x)).

Document for team readability

Ensure code includes appropriate documentation to improve readability and help team members understand and work with the codebase effectively. This includes adding JSDoc comments for new components and features, and preserving existing documentation patterns like internationalization keys that serve as format specifications.

For new components, add JSDoc comments explaining purpose and functionality:

/**
 * Audio configuration component that manages file upload settings
 * for audio files in the workflow features
 */
const ConfigAudio: FC = () => {
  // component implementation
}

When refactoring, maintain existing documentation patterns rather than removing them:

// Keep i18n keys that document expected formats
const timeFormat = needTimePicker ? t('time.dateFormats.displayWithTime') : t('time.dateFormats.display')
// Instead of hardcoded strings like 'YYYY-MM-DD HH:mm'

Well-documented code increases readability and makes it easier for other developers to pick up and work with unfamiliar code sections.

prefer nullish coalescing operator

Use the nullish coalescing operator (??) instead of logical OR (||) when you want to provide default values only for null or undefined, while preserving other falsy values like empty arrays, empty strings, or zero.

The || operator treats all falsy values (null, undefined, false, 0, “”, []) as requiring the default, while ?? only treats null and undefined as requiring the default. This distinction is crucial for preserving intentionally falsy but valid values.

Example:

// ❌ Problematic - loses empty arrays
...(draft.file?.allowed_file_types || [])

// ✅ Better - preserves empty arrays  
...(draft.file?.allowed_file_types ?? [])

// ❌ Problematic - loses falsy but valid values
const config = userConfig || defaultConfig

// ✅ Better - only defaults for null/undefined
const config = userConfig ?? defaultConfig

This pattern is especially important when working with arrays, objects, or other values where falsy states have semantic meaning that should be preserved.

Concurrent resource management

When implementing concurrent operations, properly manage shared resources to prevent race conditions and ensure system stability. Use database-level locking mechanisms like SELECT ... FOR UPDATE SKIP LOCKED to prevent concurrent access to the same data. Configure thread pools with appropriate limits and timeouts to avoid resource exhaustion. Separate different types of tasks into distinct queues to prevent blocking.

Key practices:

Example of proper concurrent resource management:

# Use database locking to prevent race conditions
with session_factory() as session:
    schedules = session.scalars(
        query.with_for_update(skip_locked=True).limit(batch_size)
    )
    
# Configure thread pools with appropriate limits and timeouts
with ThreadPoolExecutor(max_workers=3) as executor:
    futures = [executor.submit(process_item, item) for item in items]
    concurrent.futures.wait(futures, timeout=30)

Document thread safety requirements for shared components and avoid synchronous blocking patterns in async systems.

CI testing tool integration

When integrating new testing or analysis tools into CI pipelines, use graceful failure patterns during initial adoption phases to prevent breaking existing workflows. Add tools as development dependencies and allow them to fail without blocking builds using || true until they are stable and properly configured.

This approach allows teams to experiment with new testing tools without disrupting the development workflow. Once the tools are properly configured and any initial issues are resolved, the || true can be removed to enforce the checks.

Example implementation:

- name: Run new analysis tool
  run: |
    cd api
    uv add --dev new-tool
    uv run new-tool check || true

This pattern is particularly useful when introducing type checkers, linters, or other code analysis tools that may initially produce many findings on existing codebases.

Name by semantic purpose

Choose names that reflect the semantic purpose and meaning rather than implementation details or arbitrary values. Use enum constants instead of string literals, select precise identifiers that clearly convey intent, and name functions/variables based on their actual purpose.

Examples:

This approach improves code readability, reduces magic strings, and makes the codebase more maintainable by clearly expressing the developer’s intent.

Validate before accessing

Always validate preconditions before performing operations that could throw runtime errors. This includes checking object property existence, verifying context validity, and ensuring required data is available before proceeding.

When validation fails, handle the situation gracefully rather than allowing the operation to throw. Options include:

Example of defensive property access:

// Instead of direct access that could throw:
created_at__after: dayjs().subtract(TIME_PERIOD_MAPPING[period].value, 'day')

// Add validation first:
...((period !== '9' && TIME_PERIOD_MAPPING[period])
  ? {
    created_at__after: dayjs().subtract(TIME_PERIOD_MAPPING[period].value, 'day')
  }
  : {})

This approach prevents runtime crashes and provides better user experience by handling edge cases proactively rather than reactively.

Prefer exceptions over silent failures

Raise exceptions instead of returning None, logging warnings, or silently modifying values when encountering error conditions. This makes failures explicit and forces callers to handle error cases properly, improving code reliability and debugging experience.

Why this matters:

Apply this by:

Example:

# ❌ Avoid - silent failure
def update_schedule(session: Session, schedule_id: str, updates: dict) -> Optional[WorkflowSchedulePlan]:
    schedule = session.get(WorkflowSchedulePlan, schedule_id)
    if not schedule:
        return None  # Forces caller to check for None

# ✅ Prefer - explicit exception
def update_schedule(session: Session, schedule_id: str, updates: dict) -> WorkflowSchedulePlan:
    schedule = session.get(WorkflowSchedulePlan, schedule_id)
    if not schedule:
        raise ValueError(f"Schedule not found: {schedule_id}")
    # ... rest of implementation

Exception: Use None returns only when the absence of a value is a valid business case, not an error condition.

SQLAlchemy 2.0 patterns

Migrate from legacy SQLAlchemy 1.x query patterns to modern 2.0 style for better performance, clarity, and future compatibility. This involves several key transformations:

Query Construction:

Result Retrieval:

Example transformations:

# Legacy 1.x style
user = db.session.query(User).filter(User.id == user_id).first()
users = db.session.query(User).filter(User.active == True).all()

# Modern 2.0 style  
stmt = select(User).where(User.id == user_id)
user = db.session.scalar(stmt)

stmt = select(User).where(User.active.is_(True))
users = db.session.scalars(stmt).all()

# Column-specific queries (note: use execute, not scalars)
stmt = select(User.id, User.name).where(User.active.is_(True))
results = db.session.execute(stmt).all()

Boolean Comparisons: Use .is_(True) or .is_(False) instead of == True/False for explicit boolean comparisons.

This migration improves code maintainability, leverages SQLAlchemy’s latest optimizations, and ensures compatibility with future versions.

Safe null handling

Handle null and undefined values safely by using appropriate fallback patterns, validating critical parameters early, and avoiding mutable default arguments.

Key Practices:

  1. Use safe fallback patterns for optional values: ```python

    Good: Safe fallback with or operator

    cpu_count = os.cpu_count() or 4 meta = meta or {}

Good: Safe attribute access with getattr

func_name = getattr(func, “name”, “Unknown”)

Avoid: Direct access that might fail

func_name = func.name # May raise AttributeError


2. **Validate critical parameters early** and fail fast:
```python
# Good: Early validation with clear error
def get_features(tenant_id: str):
    if not tenant_id:
        raise ValueError("tenant_id is required")
    # ... rest of function

# Avoid: Allowing empty/None values to propagate
def get_features(tenant_id: str):
    # ... function continues without validation
  1. Avoid mutable default arguments: ```python

    Good: Use None and assign in function body

    def create_blob_message(blob: bytes, meta: dict = None, save_as: str = “”): meta = meta or {}

Good: Use default_factory for Pydantic models

conditions: list[SubCondition] = Field(default_factory=list)

Avoid: Mutable default argument

def create_blob_message(blob: bytes, meta: dict = {}, save_as: str = “”):


4. **Use dict.get() with explicit defaults** for safe dictionary access:
```python
# Good: Explicit default value
for tool in agent_mode.get("tools", []):

# Less clear: Relying on or operator
for tool in agent_mode.get("tools") or []:
  1. Be explicit about parameter validation: ```python

    Good: Check parameter before use

    if keyword and isinstance(keyword, str): keyword_like_val = f”%{keyword[:30]}%”

Avoid: Using parameter without validation

keyword_like_val = f”%{keyword[:30]}%” # May fail if keyword is None


This approach prevents runtime errors, makes code more predictable, and clearly communicates intent about required vs optional values.

---

## Mock tests in tests/litellm

<!-- source: BerriAI/litellm | topic: Testing | language: Python | updated: 2025-08-30 -->

All new functionality must include mock tests placed in the `tests/litellm/` directory structure. Mock external API calls and dependencies rather than requiring real credentials or external services. This ensures tests can run reliably in CI/CD pipelines and are accessible to all contributors.

**Key Requirements:**
- Place tests in `tests/litellm/` following the source code structure (e.g., `tests/litellm/llms/provider_name/`)
- Mock external API calls using `unittest.mock` or `pytest-mock` instead of making real API requests
- Avoid tests that require external credentials, API keys, or network dependencies
- Use `MagicMock` to simulate responses from external services

**Example:**
```python
# Good: Mock test in tests/litellm/
from unittest.mock import patch, MagicMock
import pytest
from litellm import completion

def test_provider_completion_mock():
    mock_response = MagicMock()
    mock_response.json.return_value = {
        "choices": [{"message": {"content": "Hello"}}]
    }
    
    with patch("requests.post", return_value=mock_response):
        response = completion(
            model="provider/model-name",
            messages=[{"role": "user", "content": "Hi"}]
        )
        assert response.choices[0].message.content == "Hello"

# Bad: Integration test requiring real credentials
def test_provider_completion_real():
    # Requires PROVIDER_API_KEY environment variable
    response = completion(
        model="provider/model-name", 
        messages=[{"role": "user", "content": "Hi"}]
    )

This approach prevents test failures due to missing credentials, network issues, or rate limits, while ensuring comprehensive test coverage that contributors can run locally and in automated pipelines.

Safe access patterns

Always use safe access methods when working with potentially null or undefined values. Use .get() for dictionary access, check array bounds before indexing, and validate existence before operations.

Key patterns to follow:

Example of safe patterns:

# Safe dictionary access
description = tool.get("description")  # Instead of tool["description"]
api_key = deployment.get("litellm_params", {}).get("api_key")

# Safe array access  
if len(results) > 0 and results[0].get("moderations"):
    # Process moderations

# Proper null checking logic
if all([batch_cost is not None, batch_usage is not None, batch_models is not None]):
    # Process batch data

# Safe fallback with explicit None check
_deployment_tpm = _deployment.get("tpm")
if _deployment_tpm is None:
    _deployment_tpm = _deployment.get("litellm_params", {}).get("tpm")
if _deployment_tpm is None:
    _deployment_tpm = float("inf")

This prevents KeyError exceptions, IndexError exceptions, and unexpected behavior from falsy values being treated as None.

Use configuration helper utilities

Always use centralized helper functions for configuration management instead of direct environment variable access or hardcoded values. This ensures consistency, maintainability, and proper error handling across the codebase.

Key practices:

Example of proper configuration handling:

# Bad - direct environment access and hardcoded values
api_key = os.environ.get("GEMINI_API_KEY")
MAX_STRING_LENGTH = 1000
if os.getenv("NO_REDOC", "False") == "True":

# Good - using helper utilities
api_key = get_secret_str("GEMINI_API_KEY") or get_secret_str("GOOGLE_API_KEY")
MAX_STRING_LENGTH = get_secret_str("MAX_STRING_LENGTH_PROMPT_IN_DB", "1000")
if str_to_bool(os.getenv("NO_REDOC", "False")):

This approach prevents configuration drift, reduces code duplication, and makes configuration behavior predictable across different parts of the system.

Prefer generic API patterns

Avoid creating provider-specific API implementations when generic, reusable solutions exist or can be developed. This principle improves code maintainability, reduces duplication, and ensures consistency across the API surface.

Key practices:

Example from the discussions:

# Instead of creating a new handler:
class HostedVLLMRerank(BaseLLM):
    # custom implementation...

# Reuse existing patterns:
# "please follow the cohere integration, and use the base_llm_http_handler"

# Instead of hardcoded endpoints:
if "gateway.ai.cloudflare.com" in api_base:
    # custom logic...

# Use generic configuration:
api_base = api_base or get_configurable_base_url()

This approach reduces technical debt, makes the codebase more predictable for developers, and ensures that improvements to core functionality benefit all providers rather than being siloed in provider-specific implementations.

Offload heavy operations

Move heavy operations, especially those involving large data processing or bulk deletions, from synchronous request handlers to asynchronous Celery tasks. This prevents request timeouts, improves user experience, and allows for better resource management.

Heavy operations should be identified by their potential to:

Example implementation:

# Instead of handling deletion directly in the controller
@setup_required
@login_required
def delete(self, app_model):
    # Move this to service layer with Celery
    pass

# Use Celery task in service layer
@classmethod
def delete(cls, app_model: App, conversation_id: str, user: Optional[Union[Account, EndUser]]):
    # Verify permissions first
    conversation = cls.get_conversation_for_deletion(app_model, conversation_id, user)
    
    # Queue the heavy deletion work as a Celery task
    delete_conversation_task.delay(conversation.id)

This pattern ensures that web requests remain responsive while heavy operations are processed in the background by Celery workers.

Improve code organization

Prioritize clean code structure and readability in AI/ML codebases by avoiding inline conditionals, using descriptive expressions, and maintaining proper separation of concerns. This is especially important in machine learning systems where complex logic can quickly become unmaintainable.

Key practices:

  1. Separate platform-specific logic: Instead of inline conditionals, create separate functions for different backends or platforms
  2. Use readable expressions: Replace complex boolean logic with clearer alternatives
  3. Eliminate code duplication: Extract common patterns into reusable components
  4. Use named constants: Replace magic numbers with descriptive constants

Example of improved organization:

# Instead of inline conditionals:
if _is_npu:
    # NPU-specific logic here
else:
    # Default logic here

# Use separate functions:
def apply_rotary_pos_emb():
    ...

def apply_rotary_pos_emb_npu():
    ...

if _is_npu:
    apply_rotary_pos_emb = apply_rotary_pos_emb_npu

# Instead of complex conditions:
len(weight_block_size) == 2 or len(weight_block_size) == 1

# Use readable alternatives:
len(weight_block_size) in [1, 2]

# Instead of magic numbers:
TILE_SIZE = 128
BLOCK_SIZE = 4

This approach reduces cognitive load, improves maintainability, and makes AI/ML codebases more accessible to team members working across different components.

Eliminate redundant operations

Avoid duplicate function calls, repeated tensor operations, and redundant computations that can significantly impact performance. Implement caching mechanisms for expensive operations and pre-allocate tensors when possible.

Key optimization strategies:

  1. Cache expensive function results: Store results of costly operations like get_normalized_target_modules() or is_npu() to avoid repeated execution
  2. Pre-allocate tensors: Use pre-allocated buffers instead of creating new tensors with torch.arange() or torch.zeros() in hot paths
  3. Avoid duplicate initialization: Prevent initializing the same components multiple times (e.g., ragged wrappers) within a single forward pass
  4. Use lookup tables: Pre-compute valid cases in lookup tables rather than dynamic computation when the search space is bounded

Example of caching expensive operations:

# Before: Redundant calls
for lora_id, config in self.configs.items():
    user_normalized_modules = get_normalized_target_modules(config.target_modules)

# After: Cache and reuse
normalized_cache = {}
for lora_id, config in self.configs.items():
    if config.target_modules not in normalized_cache:
        normalized_cache[config.target_modules] = get_normalized_target_modules(config.target_modules)
    user_normalized_modules = normalized_cache[config.target_modules]

Example of pre-allocation:

# Before: Creating tensors in hot path
q_indptr = torch.arange(0, bs + 1, dtype=torch.int32, device=device)

# After: Pre-allocate during initialization
self.q_indptr_decode = torch.arange(0, max_bs + 1, dtype=torch.int32, device=device)
# Use pre-allocated buffer: q_indptr = self.q_indptr_decode[:bs + 1]

Performance impact can be substantial - caching reduced execution time from 133μs to 5μs in one measured case. Always profile critical paths and eliminate redundant work through strategic caching and pre-allocation.

eliminate code duplication

Actively identify and eliminate code duplication by extracting common patterns into reusable components. When you notice repeated code blocks, similar conditional logic, or duplicate class definitions, consolidate them through extraction techniques.

Key strategies:

Example of pattern extraction:

# Before: Repeated conditional logic
global_num_tokens_gpu=(
    self.prefill_global_num_tokens_gpu
    if hasattr(self, "capture_forward_mode")
    and self.capture_forward_mode == ForwardMode.EXTEND
    else self.global_num_tokens_gpu
),
global_num_tokens_for_logprob_gpu=(
    self.prefill_global_num_tokens_for_logprob_gpu
    if hasattr(self, "capture_forward_mode")
    and self.capture_forward_mode == ForwardMode.EXTEND
    else self.global_num_tokens_for_logprob_gpu
),

# After: Extract pattern into data structure
@dataclass
class TokenConfig:
    global_num_tokens_gpu: Tensor
    global_num_tokens_for_logprob_gpu: Tensor

config = prefill_config if self.capture_forward_mode == ForwardMode.EXTEND else default_config

This approach improves maintainability, reduces the chance of inconsistencies, and makes the codebase more readable and easier to modify.

Verify algorithmic correctness

Ensure algorithmic implementations are correct before applying optimizations or complex conditional logic. This is especially critical for performance optimizations, computational complexity considerations, and batch processing logic where errors can have cascading effects.

Key practices:

  1. Validate optimization conditions: Before applying algorithmic optimizations, ensure all prerequisite conditions are met
  2. Analyze computational complexity: Consider the algorithmic complexity (e.g., O(bs·seq_len^2)) and precision requirements when implementing performance-critical code
  3. Verify batch processing logic: Ensure state management and control flow are correct in algorithmic processing pipelines
  4. Use precise arithmetic: Choose appropriate operators (/ vs //) based on the mathematical requirements of the algorithm

Example from the discussions:

# Incorrect: Optimization applied without proper condition checking
if (_is_flashinfer_available and global_server_args_dict["enable_flashinfer_allreduce"]):
    final_hidden_states = flashinfer_allreduce(final_hidden_states)

# Correct: Verify all conditions before applying optimization
if (self.reduce_results and (self.tp_size > 1 or self.ep_size > 1) and
    _is_flashinfer_available and global_server_args_dict["enable_flashinfer_allreduce"]):
    final_hidden_states = flashinfer_allreduce(final_hidden_states)

# Computational complexity consideration with precise arithmetic
current_workload = sum_seq_lens / forward_batch.batch_size * sum_seq_lens  # Use / for mean

This approach prevents algorithmic bugs that can lead to incorrect results, performance degradation, or system instability.

Use Optional types safely

Always use proper nullable type annotations and safe access patterns to prevent runtime errors from null/undefined values. Use Optional[Type] or Union[None, Type] for nullable parameters, employ safe dictionary access with .get(), and handle None values explicitly in operations.

Examples of good practices:

# Use Optional type annotations
def process_data(encoder_lens: Optional[torch.Tensor] = None):
    pass

# Safe dictionary access
draft_url = params.get("draft_url", None)  # Instead of params["draft_url"]

# Null-safe assertions
assert (topk_weights is None) or (topk_weights.shape == topk_ids_.shape)

This prevents KeyError exceptions, improves type safety, and makes null handling explicit rather than implicit. Always consider whether a parameter or variable can be None and handle it appropriately in both type annotations and runtime logic.

Avoid code duplication

Eliminate redundant code by leveraging existing utilities, extracting reusable helper functions, and organizing logic in appropriate modules. Before implementing new functionality, check if similar logic already exists in the codebase that can be reused or extended.

Key practices:

Example of refactoring duplicate logic:

# Before: Duplicate SSL verification logic
ssl_verify = (os.getenv("SSL_VERIFY") != "False") if os.getenv("SSL_VERIFY") is not None else litellm.ssl_verify

# After: Extract to helper method
def get_ssl_verify_setting():
    return (os.getenv("SSL_VERIFY") != "False") if os.getenv("SSL_VERIFY") is not None else litellm.ssl_verify

ssl_verify = get_ssl_verify_setting()

This approach reduces maintenance burden, improves consistency, and makes the codebase more maintainable by centralizing common logic.

Optimize cache performance

Prioritize cache performance by leveraging proven optimization techniques and avoiding performance anti-patterns. Use standard caching decorators like @lru_cache for functions that benefit from memoization, implement zero-copy operations when moving data between cache layers, and avoid unnecessary operations that degrade performance.

Key practices:

Example:

from functools import lru_cache

@lru_cache(maxsize=128)
def should_use_flashinfer_cutlass_moe_fp4_allgather():
    # Expensive computation cached automatically
    return compute_moe_config()

# Zero-copy backup for Mooncake backend
if self.is_mooncake_backend():
    # Use zero-copy operations instead of generic page backup
    self.zerocopy_page_backup(operation)

This approach ensures cache operations remain performant under load while leveraging battle-tested optimization patterns that reduce computational overhead and memory copying.

Configuration defaults appropriateness

Ensure configuration parameters have appropriate default values and avoid hardcoded or overly specific settings. Default values should be conservative and safe for production environments, while avoiding personal assignments or environment-specific assumptions.

Examples of good practices:

This ensures configurations are maintainable, secure by default, and appropriate for team environments rather than individual setups.

Environment variable best practices

Avoid reading environment variables directly in performance-critical code paths, especially during model forward passes or hot loops. Instead, read and cache environment variables during initialization or use dedicated helper functions for consistent handling.

Use standardized helper functions like get_bool_env_var() instead of direct os.getenv() calls for better type safety and consistency. Follow clear naming conventions for environment variables, using descriptive prefixes like SGLANG_ for project-specific settings.

Example of what to avoid:

def forward_decode(self):
    # BAD: Reading env var in forward pass
    if os.getenv("USE_W4A8") != "1" and deep_gemm_wrapper.ENABLE_JIT_DEEPGEMM:
        # performance-critical code

Better approach:

def __init__(self):
    # GOOD: Cache env var during initialization
    self.use_w4a8 = get_bool_env_var("SGLANG_USE_W4A8")
    
def forward_decode(self):
    # GOOD: Use cached value
    if not self.use_w4a8 and deep_gemm_wrapper.ENABLE_JIT_DEEPGEMM:
        # performance-critical code

This practice improves performance by avoiding repeated system calls and ensures consistent configuration handling across the codebase.

API payload structure

Design API request and response payloads to match their specific purpose rather than forcing them to conform to existing application types. When API payloads require different structures or optional fields compared to internal data models, create dedicated API-specific types instead of adding placeholder values or compromising type safety.

This prevents situations where developers must add dummy values like empty strings to satisfy type requirements that don’t align with the API’s actual needs. It also enables cleaner API interfaces with appropriate optional parameters and nested data structures.

Example of the problem:

// Problematic: Forcing API payload to match internal type
environment_variables: environmentVariables.map((item: EnvironmentVariable) => {
  return {
    id: item.id,
    name: item.name,
    value_type: item.value_type,
    from_version: restoredSecretsInfo[item.id].from_version,
    // Adding dummy values to satisfy EnvironmentVariable type
    value: '',
    description: '',
  }
})

Better approach: Create API-specific types that reflect the actual payload structure and requirements, allowing for cleaner, more maintainable interfaces that don’t require workarounds or placeholder data.

Avoid broad exception catching

Avoid using overly broad exception handling like except Exception as it can mask specific errors and make debugging difficult. Instead, catch specific exception types when possible, or convert generic exceptions to more informative, specific error types with meaningful messages.

When you must catch broad exceptions, immediately convert them to specific error types that provide context about what went wrong and why. This helps with debugging and prevents silent failures.

Example of problematic broad catching:

try:
    speculative_algo = global_server_args_dict.get("speculative_algorithm", None)
    if speculative_algo is not None and hasattr(speculative_algo, "is_eagle"):
        return speculative_algo.is_eagle()
except Exception:  # Too broad - what could fail here?
    return False

Better approach with specific error handling:

try:
    ip_str, port_str = distributed_init_method.split(":")[-2:]
    ip = ip_str.split("/")[-1]
    port = int(port_str)
except (ValueError, IndexError) as e:  # Specific exceptions
    raise RuntimeError(
        f"Cannot parse host and port from {distributed_init_method}: {str(e)}"
    )

This approach makes errors more actionable for developers and prevents masking of unexpected issues that should be addressed.

Configuration value types

Ensure configuration values use appropriate types, avoid unnecessary Optional annotations, and maintain consistency in data formats and units.

Key principles:

  1. Use explicit boolean types instead of string representations when the configuration is consumed as a boolean
  2. Avoid Optional for fields with defaults - if a default value is provided, Optional is redundant
  3. Choose consistent time units across related configurations (prefer seconds for precision, minutes for readability)
  4. Use appropriate data formats - prefer unix timestamps over ISO strings for simplicity and timezone safety

Example of proper configuration typing:

class MyConfig(BaseSettings):
    # Good: explicit boolean with default
    SSL_VERIFY: bool = Field(default=True)
    
    # Bad: unnecessary Optional with default
    # SSL_VERIFY: Optional[bool] = Field(default=True)
    
    # Good: consistent time unit choice
    ACCESS_TOKEN_EXPIRE_MINUTES: int = Field(default=30)
    
    # Good: unix timestamp for cache keys
    def get_cache_timestamp(self) -> int:
        return int(datetime.now().timestamp())

This approach reduces configuration complexity, improves type safety, and ensures consistent behavior across different deployment environments.

Use appropriate log levels

Choose the correct logging level based on the importance and purpose of the message. Use DEBUG for debugging information that’s only useful during development, INFO for general operational information, WARNING for concerning conditions that don’t prevent operation, and ERROR for actual errors. Remove meaningless debug prints and consider using conditional logging for expensive operations.

Examples:

Avoid leaving temporary debug logs in production code and ensure log levels match the actual significance of the information being logged.

Safe operations with fallbacks

Implement defensive programming by wrapping potentially failing operations in safe methods that provide fallback behavior. Critical operations should never cause the main functionality to fail due to secondary concerns like formatting, logging, or auxiliary processing.

Key principles:

Example implementation:

# Instead of direct operation that could fail
"x-litellm-response-cost": str(round(response_cost, 6))

# Use safe helper method with fallback
def safe_round_cost(cost: float) -> str:
    try:
        return str(round(cost, 6))
    except Exception:
        return str(cost)  # fallback to original value

"x-litellm-response-cost": safe_round_cost(response_cost)

This approach prevents secondary failures from cascading and breaking primary functionality, while still attempting the preferred operation when possible.

Choose descriptive, unambiguous names

Select variable, method, and property names that clearly communicate their purpose and avoid ambiguity. Prioritize semantic accuracy over brevity, and consider the context in which names will be used to prevent confusion.

Key principles:

Example:

// Poor: ambiguous and uses magic strings
const effectNodes = findNodes(selector, nodes)
if (permission === 'only_me') { ... }

// Better: clear semantic meaning and structured types
const affectedNodes = findNodes(selector, nodes)
if (permission === DatasetPermission.ONLY_ME) { ... }

This approach reduces cognitive load, prevents misunderstandings, and makes code more maintainable by ensuring names accurately represent their underlying concepts.

Use descriptive names

Choose variable, function, and parameter names that accurately reflect their content, type, and purpose. Avoid misleading names that don’t match the actual data or functionality they represent.

Key principles:

Examples of improvements:

# Misleading - suggests it's a list but contains length
holding_tokens_list = len(tokens)  # Bad

# Clear and accurate
holding_tokens_count = len(tokens)  # Good

# Informal terminology
onfly_info = get_dispatch_info()  # Bad

# Standard terminology  
in_flight_info = get_dispatch_info()  # Good

# Overly verbose
should_fuse_allreduce_residual_rmsnorm = True  # Bad

# Concise but clear
should_allreduce_fusion = True  # Good

This practice prevents confusion, reduces debugging time, and makes code more maintainable by ensuring names serve as accurate documentation of the code’s intent.

Conservative configuration management

Be selective and intentional with configuration choices to avoid bloat and unnecessary complexity. This applies to build configurations, development environment settings, and dependency management.

Key practices:

Example from TypeScript configuration:

"exclude": [
  "**/node_modules/**",
  "**/.next/**"
]

This approach reduces build times, keeps development environments lean, and minimizes maintenance overhead while ensuring configurations remain focused on actual project needs.

Explicit API parameters

API functions should use explicit parameters rather than passing large opaque objects. When a function needs specific data from a complex object, extract and pass only the required fields as individual parameters. This makes the function’s dependencies clear, improves testability, and reduces coupling.

Avoid passing entire objects when only specific fields are needed:

# Bad: Opaque object passing
def forward(
    self,
    hidden_states: torch.Tensor,
    topk_output: StandardTopKOutput,
    forward_batch: Optional[ForwardBatch] = None,  # Unclear what fields are used
):

# Good: Explicit parameters
def forward(
    self,
    hidden_states: torch.Tensor,
    topk_output: StandardTopKOutput,
    dp_padding_mode: bool,
    batch_size: int,
    gathered_buffer: torch.Tensor,
    global_num_tokens: int,
):

This approach makes function interfaces self-documenting and prevents unnecessary dependencies on large data structures. If many parameters are needed, consider if the function has too many responsibilities and should be refactored.

optimize tensor operations

When working with PyTorch tensors, prioritize operations that avoid unnecessary memory allocations and copies to improve performance. Choose tensor operations carefully based on whether data will be immediately overwritten or needs preservation.

Key guidelines:

Example:

# Avoid unnecessary copy
if tensor.dtype != target_dtype:
    tensor = tensor.view(target_dtype)  # No copy if compatible
    # instead of: tensor = tensor.to(target_dtype)  # May cause copy

# Avoid unnecessary initialization
kv_indices = torch.empty(size, dtype=torch.int32)  # Will be filled next
# instead of: kv_indices = torch.zeros(size, dtype=torch.int32)  # Extra kernel launch

This approach reduces memory overhead and kernel launches, leading to better performance in tensor-heavy operations.

AI provider documentation completeness

Ensure comprehensive documentation for AI/ML provider integrations that covers all usage patterns, uses precise terminology, and provides helpful context for developers.

AI provider documentation should include:

  1. Complete usage examples: Cover SDK usage, proxy configuration, streaming, and advanced features like function calling or image input
  2. Precise terminology: Use accurate provider-specific terms (e.g., <hf_org_or_user>/<hf_model> instead of generic <model_id>)
  3. Configuration consistency: Show both direct API usage and proxy configuration patterns
  4. Provider-specific context: Explain unique features, billing models, and authentication methods
  5. Missing examples: Add proxy usage examples when only SDK examples exist

Example of comprehensive documentation structure:

# Basic SDK usage
response = completion(
    model="huggingface/together/deepseek-ai/DeepSeek-R1",
    messages=[{"content": "Hello", "role": "user"}]
)

# Proxy configuration
# config.yaml
model_list:
  - model_name: my-model
    litellm_params:
      model: huggingface/together/deepseek-ai/DeepSeek-R1
      api_key: os.environ/HF_TOKEN
      web_search_options: {}  # Provider-specific options

# Advanced features (when supported)
response = completion(
    model="huggingface/sambanova/Qwen/Qwen2.5-72B-Instruct",
    messages=[{
        "role": "user", 
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "..."}}
        ]
    }]
)

This ensures developers have complete guidance for integrating AI providers across different usage patterns and deployment scenarios.

API parameter consistency

Ensure consistent parameter naming, format, and usage across all API interfaces, documentation, and code examples. This includes maintaining uniform parameter names between SDK examples and proxy configurations, using correct environment variable names as specified in official documentation, and applying proper naming conventions for model identifiers and API routes.

Key areas to verify:

Example of consistent parameter usage:

# SDK example
response = completion(
    model="heroku/claude-3-5-haiku",
    api_base="https://us.inference.heroku.com",
    api_key="fake-heroku-key"
)

# Corresponding proxy config should use same parameter names
model_list:
  - model_name: claude-haiku
    litellm_params:
        model: heroku/claude-3-5-haiku
        api_base: https://us.inference.heroku.com
        api_key: fake-heroku-key

This consistency prevents confusion, reduces integration errors, and ensures predictable behavior across different usage contexts.

Prevent race conditions

Ensure atomic operations and proper synchronization when multiple threads or processes access shared state. Race conditions occur when the timing of operations affects correctness, leading to data corruption, memory leaks, or inconsistent behavior.

Common patterns to watch for:

Example of a race condition:

# Thread A checks condition
if operation.is_done():  # Returns False
    # Thread B calls operation.mark_done() here
    # Thread A continues with stale information
    operation.completed_tokens += self.page_size
    self.mem_pool_host.free(operation.host_indices[operation.completed_tokens:])

Solutions:

When implementing distributed operations, ensure all ranks maintain consistent state to prevent divergent behavior across the system.

Document configuration purpose clearly

When documenting configuration features, always provide clear explanations of their purpose, use cases, and how they map to underlying parameters. Users should understand not just how to use a configuration option, but why they would want to use it and what problem it solves.

For environment variables, explicitly document the mapping between environment variables and configuration parameters:

## Environment Variables

The Heroku provider is aware of the following config variables:

- `HEROKU_API_KEY`: This value corresponds to the [`api_key` param](https://docs.litellm.ai/docs/set_keys#litellmapi_key). Set this to the value of Heroku's `INFERENCE_KEY` config variable.
- `HEROKU_API_BASE`: This value corresponds to the [`api_base` param](https://docs.litellm.ai/docs/set_keys#litellmapi_base). Set this to the value of Heroku's `INFERENCE_URL` config variable.

For configuration features that may have non-obvious use cases, include practical scenarios:

#### Import Models

Import models from a YAML file. This is useful when migrating models from a self-hosted instance to a cloud instance, allowing you to gradually port your existing configuration.

This prevents confusion and reduces the need for clarifying questions about the purpose and intended workflow of configuration features.

AI model data validation

All AI model configuration parameters must be verified against official provider documentation before merging. This includes token limits, pricing values, provider assignments, model names, and supported features. Incorrect configuration data leads to runtime errors, cost miscalculations, and integration failures.

Key validation requirements:

Example of proper validation:

"gpt-5-2025-08-07": {
    "max_input_tokens": 272000,  // Verified against OpenAI docs
    "input_cost_per_token": 1.25e-06,  // Scientific notation, verified pricing
    "litellm_provider": "openai",  // Matches actual API provider
    "supports_function_calling": true  // JSON boolean, not Python
}

Always include source citations when making configuration changes to enable future verification and maintain data integrity.

Document AI limitations

When implementing AI model support or hardware backends, clearly document known limitations, compatibility constraints, and version requirements to prevent integration issues and set proper expectations.

AI systems often have specific constraints that can cause runtime failures or unexpected behavior if not properly communicated. This includes quantization limitations, hardware-specific requirements, version dependencies, and feature compatibility matrices.

Key areas to document:

Example documentation pattern:

## Known Issues

Several limitations currently affect offline quantized model loading:

1. Mixed-bit Quantization Limitations
   Mixed-bit quantization is not fully supported. Due to vLLM's layer fusion 
   (e.g., QKV fusion), applying different bit-widths to components within the 
   same fused layer can lead to compatibility issues.

2. Limited Support for Quantized MoE Models
   Most quantized MoE models may encounter inference issues due to 
   kernel-related limitations.

This prevents users from encountering undocumented failures and helps them choose appropriate alternatives or workarounds.

consolidate duplicated logic

When you identify duplicated code patterns across different branches or functions, consolidate the shared logic into a unified implementation to improve maintainability and reduce code bloat. This applies whether the duplication occurs within conditional branches, across similar functions, or in separate files.

For substantial duplication, extract the common logic into a shared function or kernel. For minimal implementations (a dozen lines or less), consider inlining the logic using C++ templates and constexpr to avoid overhead while maintaining clarity.

Example of consolidation:

// Before: Duplicated logic in branches
if (params.VPT <= 32) {
    // Complex processing logic A
    // ... 50 lines of similar code
} else {
    // Complex processing logic B  
    // ... 50 lines of nearly identical code
}

// After: Consolidated into unified kernel
template<bool small_vpt>
void unified_processing_kernel(...) {
    // Shared logic with template specialization
    if constexpr (small_vpt) {
        // Small VPT specific optimizations
    } else {
        // Large VPT specific optimizations  
    }
}

This approach reduces maintenance burden, minimizes the risk of inconsistent updates, and keeps the codebase clean and organized.

AI accuracy documentation

Ensure accurate representation of AI model capabilities, proper attribution, and honest claims about functionality. When documenting AI systems, focus on the underlying model capabilities rather than just pipeline descriptions, provide clear explanations of model variants (distilled vs full models), attribute models to their actual creators, and avoid overstating what has been achieved.

Key practices:

Example model.json with proper attribution:

{
  "metadata": {
    "author": "Meta", // Actual model creator, not generic name
    "description": "Llama 2 integration with Jan - our implementation of chat functionality"
  }
}

This prevents misleading users about AI capabilities while ensuring proper credit and technical accuracy.

Eliminate filler language

Remove unnecessary filler words and repetitive phrases that dilute clarity and assume reader incompetence. Avoid words like “essentially”, “basically”, “just”, “simple”, “easy”, and repetitive phrases like “at a high level” appearing multiple times in close proximity.

Write directly and respect your reader’s intelligence. Instead of hedging with filler words, state facts clearly and concisely.

Bad example:

Deep Research, at a high level, is essentially a pipeline / tool chain through which a user would go through. The inner workings of this pipeline might vary from provider to provider, but we've done up a table for you to compare the results below.

At a high level, the pipeline would look something like this:

Good example:

Deep Research is a pipeline or tool chain through which a user would go through. The inner workings of this pipeline might vary from provider to provider, but we've done up a table for you to compare the results below.

For example, a straightforward pipeline might look as follows:

This approach makes documentation more professional, easier to read, and shows respect for the reader’s technical competence. Focus on delivering information efficiently rather than cushioning it with unnecessary qualifiers.

avoid expensive operations

Identify and optimize resource-intensive operations in frequently executed code paths. This includes avoiding memory-doubling operations like copy.deepcopy() on large objects, preventing blocking I/O in async contexts, and eliminating expensive debug logging operations.

Key areas to review:

Example from the discussions:

# Problematic - doubles memory usage
redacted_response = copy.deepcopy(response_json)

# Better - consider in-place modification or streaming for large objects
# Or move expensive operations out of request path

# Problematic - blocking I/O in async context
session = ClientSession(trust_env=True)  # Reads files synchronously

# Better - read environment at startup
proxy_settings = get_environment_proxies()  # At startup
session = ClientSession(proxies=proxy_settings)  # At request time

Always profile and measure the impact of operations in performance-critical paths, especially when dealing with large data or high-frequency execution.

avoid hardcoded credentials

Never embed sensitive credentials, passwords, API keys, or other secrets directly in source code. Hardcoded credentials create security vulnerabilities by exposing sensitive data in version control systems and making it accessible to anyone with code access.

Instead, use secure alternatives:

Example of what to avoid:

cache_params:
  type: "redis"
  host: "redis-18438.c277.us-east-1-3.ec2.redns.redis-cloud.com"
  port: 18438
  password: "hB44ThYlB4W4m7wpCUwrSzteHqvDKnDV"  # ❌ Hardcoded password

Better approach:

cache_params:
  type: "redis"
  host: "redis-18438.c277.us-east-1-3.ec2.redns.redis-cloud.com"
  port: 18438
  password: ${REDIS_PASSWORD}  # ✅ Environment variable

This practice protects against credential leaks and ensures sensitive data remains secure across different deployment environments.

Implement graceful error fallbacks

Always implement fallback mechanisms when primary operations can fail, especially for critical operations like resource cleanup, async operations, and user-facing functionality. When the primary approach fails, provide a secondary method to handle the situation gracefully rather than letting errors propagate unchecked.

For resource cleanup, attempt graceful shutdown first, then force termination:

function killSubprocess(): Promise<void> {
  return fetch(NITRO_HTTP_KILL_URL, {
    method: "DELETE",
  }).catch((err) => {
    console.error(err);
    subprocess.kill();
    subprocess = null;
    return kill(PORT, "tcp").then(console.log).catch(console.log);
  });
}

For user-facing operations, validate inputs and provide meaningful error handling:

const convoId = currentConvo?.id
if (!convoId) {
  console.error('No conversation id')
  // TODO: Display toast notification to user
  return
}

This approach prevents silent failures, ensures resources are properly cleaned up even when primary methods fail, and maintains application stability by providing alternative paths when operations don’t succeed as expected.

HTTP standards compliance

Ensure API endpoints follow HTTP standards for status codes and method selection. Delete operations should return 204 (No Content) without a response body, while update operations should use PUT method instead of GET. This improves API consistency and follows REST conventions.

For delete operations, use:

def delete(self, member_id):
    # Delete logic here
    return "", 204  # No content, no body

For update operations that retrieve and update data:

class ToolMCPUpdateApi(Resource):
    def put(self, provider_id):  # Use PUT instead of GET
        # Update logic here
        return {"result": "success"}, 200

Avoid returning response bodies with DELETE methods as per HTTP specifications. Use appropriate status codes: 200 for successful operations with content, 204 for successful operations without content, and proper HTTP methods (GET for retrieval, POST for creation, PUT for updates, DELETE for deletion).

simplify error handling

Write concise and appropriate error handling code by avoiding unnecessary complexity and verbose constructions. Remove try-catch blocks when they serve no purpose, and use streamlined approaches for error message construction.

For example, instead of verbose error message handling:

let errMessage = 'Unexpected Error'
if (err instanceof Error) {
  errMessage = err.message
}
toaster({
  title: 'Failed to get Hugging Face models',
  description: errMessage,
  type: 'error',
})

Use a more concise approach:

toaster({
  title: 'Failed to get Hugging Face models',
  description: err instanceof Error ? err.message : 'Unexpected Error',
  type: 'error',
})

Similarly, avoid wrapping function calls in try-catch blocks when error handling isn’t actually needed or when the function doesn’t throw exceptions that require handling at that level.

Simplify for readability

Improve code readability by simplifying complex logic, using Pythonic idioms, and removing unnecessary code. Extract helper functions for complex conditions, avoid code duplication, and leverage Python’s built-in features for cleaner code.

Key practices:

Example:

# Instead of:
if self.save_only_first_rank and not self.metadata.is_first_rank():
    return

# Use a helper function:
def is_passive(self):
    return self.save_only_first_rank and not self.metadata.is_first_rank()

if self.is_passive():
    return

# Instead of:
if memory_objs == []:
    return

# Use Pythonic style:
if not memory_objs:
    return

minimize core dependencies

Keep core dependencies minimal in configuration files like pyproject.toml. Non-essential packages should be added as optional dependencies or placed in extras sections rather than as required core dependencies. This prevents package bloat and allows users to install only what they need.

When adding new dependencies, first determine if they are truly required for core functionality. If the dependency is only needed for specific features, testing, or optional functionality, use the appropriate configuration section:

# Core dependencies - only essential packages
[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.28.0"

# Optional dependencies for specific features
[tool.poetry.extras]
rich-output = ["rich>=13.0.0"]
anthropic-support = ["anthropic^0.50.0"]

# Development/testing dependencies
[tool.poetry.group.dev.dependencies]
pytest = "^7.0.0"

This approach maintains a lean core package while providing flexibility for users who need additional functionality.

Ensure database consistency

Maintain data consistency by using appropriate database operations and ensuring transactional integrity when updating related fields. Choose the right ORM methods (upsert vs create) to handle potential duplicates, and ensure that updates to related fields across multiple tables happen atomically to prevent data drift.

Key practices:

  1. Use upsert() instead of create() when records might already exist to avoid UniqueViolationError
  2. When updating related fields across multiple tables (like user.teams and team.members_with_roles), ensure both updates succeed or fail together
  3. Use correct schema field names in ORM include/select operations

Example of proper upsert usage:

# Instead of create() which can fail on duplicates
await self.prisma_client.db.litellm_managedobjecttable.upsert(
    where={"object_id": object_id},
    data=data,
    create=data
)

# Correct schema field names in includes
include={"end_users": True}  # Not "litellm_endusertable"

This prevents UniqueViolationError exceptions and maintains referential integrity across related database entities.

Eliminate redundant operations

Avoid unnecessary function calls, I/O operations, and redundant checks that can significantly impact performance. Look for opportunities to consolidate operations, remove duplicate work, and streamline execution paths.

Common patterns to watch for:

Example improvements:

# Instead of repeated append() calls and redundant checks:
if location not in key_mapping:
    key_mapping[location] = [key]
    start_mapping[location] = [start]
    end_mapping[location] = [end]
    continue
key_mapping[location].append(key)
start_mapping[location].append(start) 
end_mapping[location].append(end)

# Use efficient unpacking to avoid repeated operations:
if reordered_blocks:
    _, memory_objs, starts, ends = zip(*reordered_blocks)
    self.gpu_connector.batched_to_gpu(list(memory_objs), list(starts), list(ends), **kwargs)

# Remove unnecessary I/O operations:
# Instead of: if self.exists_in_put_tasks(key) or self.contains(key):
# Just use: if self.exists_in_put_tasks(key):
# The contains() call adds unnecessary RPC or I/O overhead

Always question whether each operation is truly necessary and look for opportunities to batch, cache, or eliminate redundant work entirely.

network service identification

Use descriptive string identifiers instead of numeric ports or generic IDs when designing network services to avoid conflicts and improve maintainability. Different network services should have clearly distinguishable communication channels.

When multiple network services operate within the same system, relying solely on numeric ports can lead to conflicts and configuration complexity. Instead, incorporate descriptive service names and contextual information into network identifiers.

Example of problematic approach:

# Unclear what each port is for, potential conflicts
rpc_port = 0  # lookup service
rpc_port = 100  # offload service
socket_path = f"ipc://{base_url}/lmcache_rpc_port_{rpc_port}"

Improved approach with descriptive identifiers:

# Clear service identification, no port conflicts
def get_zmq_rpc_path_lmcache(engine_id, service_name, tp_rank):
    return f"ipc://{base_url}/engine_{engine_id}_service_{service_name}_tp_rank_{tp_rank}"

# Usage:
lookup_path = get_zmq_rpc_path_lmcache(engine_id, "lookup", tp_rank)
offload_path = get_zmq_rpc_path_lmcache(engine_id, "offload", tp_rank)

This approach eliminates the need for users to configure ports manually, reduces the risk of service conflicts, and makes the system’s network architecture self-documenting. The descriptive identifiers make it immediately clear which service each communication channel serves, improving debugging and maintenance.

consolidate algorithmic patterns

Avoid duplicating similar algorithmic logic across multiple functions by consolidating them into generic, templated, or trait-based implementations. This reduces maintenance burden and improves code consistency.

Instead of creating separate functions for each data type or operation combination, use templates or generic approaches:

// Avoid this - separate functions for each type combination
static __device__ void convert_f32_f32(const float * src, float * dst) { *dst = *src; }
static __device__ void convert_f16_f32(const half * src, float * dst) { *dst = *src; }
static __device__ void convert_bf16_f32(const nv_bfloat16 * src, float * dst) { *dst = *src; }

// Prefer this - generic template approach
template<typename src_t, typename dst_t>
static __device__ __forceinline__ void convert_to_flt(const src_t * src, dst_t * dst) {
    *dst = float(*src);  // Convert through float as common intermediate
}

Similarly, for algorithmic logic that varies only by parameters, use templates instead of switch statements or multiple similar functions:

// Instead of separate functions for each granularity calculation
static int mmq_get_granularity_host(ggml_type type, const int mmq_x, const int cc) {
    // Use trait-based approach or template specialization
}

// Consider using trait structs for type-specific behavior
template<ggml_type T> struct mmq_type_traits {
    static constexpr int granularity(int mmq_x) { /* type-specific logic */ }
};

This approach is particularly important for computational kernels where similar patterns appear across different data types or operation sizes. Look for opportunities to extract common algorithmic patterns into reusable, parameterized implementations.

eliminate code duplication

Avoid duplicating code blocks by extracting common functionality into reusable functions, using templates for type-generic operations, and moving shared utilities to common modules. Code duplication makes maintenance difficult, increases the likelihood of bugs, and violates the DRY (Don’t Repeat Yourself) principle.

When you find similar code patterns:

  1. Extract common logic into helper functions
  2. Use templates for type-generic operations instead of copying code for different types
  3. Move shared utilities to common modules (like common.cpp) for reuse across files
  4. Refactor similar functions to share implementation details

Example of extracting a common string utility:

// Instead of duplicating string suffix logic:
static bool str_has_suffix(const std::string & str, const std::string & suffix) {
    return str.size() >= suffix.size() && str.compare(str.size() - suffix.size(), str.size(), suffix) == 0;
}

// Move to common.cpp as string_ends_with() for reuse

Example of using templates instead of duplication:

// Instead of separate functions for different types:
template<typename T>
static void ggml_compute_forward_repeat(const ggml_compute_params * params, ggml_tensor * dst) {
    // Generic implementation
}

This approach improves maintainability, reduces bugs, and makes the codebase more consistent and easier to understand.

leverage existing framework functionality

Before implementing custom solutions in AI model conversion code, thoroughly investigate whether the framework already provides the needed functionality. This applies to inheritance patterns, configuration handling, tensor operations, and mapping conventions.

Common anti-patterns to avoid:

Example of proper approach:

# Instead of copying LlamaModel code:
@ModelBase.register("MyModelForCausalLM")
class MyModel(TextModel):  # ❌ Copying everything
    # ... copied methods from LlamaModel

# Use proper inheritance:
@ModelBase.register("MyModelForCausalLM") 
class MyModel(LlamaModel):  # ✅ Inherit from specific model
    model_arch = gguf.MODEL_ARCH.MYMODEL
    # Only override what's actually different

# Instead of manual tensor operations:
# ❌ Manual splitting
gate_tensor = data_torch[:split_size, :]
up_tensor = data_torch[split_size:, :]

# ✅ Use framework functionality
# Use LLM_FFN_SWIGLU in model definition

Always check the framework’s existing model implementations, utility functions, and configuration mechanisms before writing custom code. This reduces maintenance burden, improves consistency, and leverages tested functionality.

Choose appropriate error mechanism

Use GGML_ASSERT for early validation and programming errors that indicate misconfigurations, but implement graceful error handling with logging for runtime conditions where fallback options exist. Assertions should catch problems early in development, while runtime errors should provide informative feedback and allow system recovery when possible.

For configuration validation and programming errors:

// Good: Catch misconfiguration early
GGML_ASSERT(n_expert > 0 && "n_expert must be > 0 for SMALLTHINKER");

For runtime conditions with fallback options:

// Good: Log and allow graceful fallback
if (!kernels) {
    GGML_LOG_DEBUG("%s: No suitable KleidiAI kernel available, falling back to standard CPU implementation\n", __func__);
    return false;  // Let the system fallback to standard CPU implementation
}

Avoid silent failures that provide no indication of what went wrong or that will cause assertions to fail later in the call chain. Always provide meaningful error messages or logging to aid debugging and system monitoring.

Choose efficient data structures

Select appropriate built-in data structures and algorithmic patterns that eliminate manual checks and reduce code complexity. This improves both performance and maintainability by leveraging Python’s optimized implementations.

Key principles:

  1. Use defaultdict instead of manual key existence checks - Replace patterns like if key not in dict: dict[key] = [] with defaultdict(list)
  2. Try direct operations before fallback logic - Attempt the primary algorithm first, then handle edge cases, rather than pre-checking conditions
  3. Leverage built-in data structure behaviors - Use data structures that naturally handle your use case rather than implementing the logic manually

Example from the codebase:

# Instead of manual key checking:
if location not in key_mapping:
    key_mapping[location] = [key]
    start_mapping[location] = [start] 
    end_mapping[location] = [end]

# Use defaultdict to eliminate the check:
from collections import defaultdict
block_mapping: defaultdict = defaultdict(list)
block_mapping[location].append((key, start, end))

For memory management algorithms, try allocation first before implementing eviction logic:

# Try direct allocation first
memory_obj = self.memory_allocator.allocate(shape, dtype)
if memory_obj is None:
    # Then implement eviction logic
    self._evict_and_retry_allocation(shape, dtype)

This approach reduces branching, leverages optimized implementations, and makes code more readable and less error-prone.

maintain naming consistency

Follow established naming patterns and conventions consistently throughout the codebase. This includes using consistent formatting (dashes vs underscores), proper capitalization, descriptive parameter names, and uniform data type representations.

Key principles:

Example of inconsistent naming:

// Inconsistent - mixing byte and element offsets
params[1] = src_misalignment;           // byte offset
params[3] = src->nb[0]/ggml_type_size(src->type);  // element offset

// Consistent - all element offsets
params[1] = src_misalignment_elements;
params[3] = src_stride_elements;

Before introducing new names, check existing codebase patterns and follow the established conventions. This reduces cognitive load and makes the code more maintainable.

AI documentation precision

Ensure documentation for AI/ML tools and libraries is precise, complete, and properly formatted. This includes being specific about tool references, providing complete command-line option formats, and using proper markdown formatting.

Key practices:

Example improvements:

# Before
* `--imatrix` uses data in file as importance matrix
* `--chunk` to skip the first n chunks
* More information: https://github.com/example/repo

# After  
* `--imatrix` uses data in file generated by `llama-imatrix` as importance matrix
* `--chunk | --from-chunk` to skip the first n chunks
* More information is [available here](https://github.com/example/repo)

This ensures users have clear, actionable information when working with AI/ML tools and can easily understand tool relationships and usage patterns.

AI model documentation consistency

Ensure consistent parameter names and configuration instructions across AI model documentation. Inconsistent documentation about model parsers, parameters, and configuration options leads to developer confusion and implementation errors.

When documenting AI model configuration options, maintain consistency in:

Example of inconsistent documentation to avoid:

# In one place:
Use `--reasoning-parser qwen3/qwen3-thinking`

# In another place:
Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3`

Instead, establish and follow a single standard format across all documentation, clearly specifying which parsers work with which model types and their exact parameter syntax.

use model metadata

Leverage model metadata from GGUF format instead of hardcoded configuration values or filename-based logic. Model metadata provides authoritative information about model behavior and should be the primary source for configuration decisions.

This approach improves maintainability by centralizing configuration in the model file, reduces the risk of incorrect assumptions, and aligns with the project’s migration toward structured GGUF format.

Examples of preferred patterns:

// Instead of hardcoded logic:
if (arch == "llada") {
    llama_token bos_token = llama_vocab_bos(vocab);
    if (bos_token != LLAMA_TOKEN_NULL && (input_tokens.empty() || input_tokens[0] != bos_token)) {
        input_tokens.insert(input_tokens.begin(), bos_token);
    }
}

// Use model metadata:
// This should be handled by the metadata in the GGUF model. 
// There is a boolean field for when BOS is needed or not.

// Instead of filename-based format detection:
if (!string_ends_with(fname, ".gguf")) {
    LOG_WRN("saving to legacy format because output suffix is not .gguf");
    save_legacy_format();
}

// Use GGUF format by default:
// The format should not be decided by the output filename

Always check for relevant metadata fields in the GGUF model before falling back to hardcoded values or heuristics. This ensures model-specific behavior is correctly handled and reduces maintenance burden when supporting new model variants.

Cache state consistency

Ensure cache operations maintain consistent state and handle shared resources properly throughout the cache lifecycle. This includes proper key type management, safe eviction of shared entries, and accurate state tracking across different operational phases.

Key practices:

  1. Type-safe cache keys: Always use proper key types rather than strings to prevent cache misses due to type mismatches
  2. Reference-aware eviction: Check reference counts before evicting cache entries that may be shared between multiple requests
  3. Consistent state tracking: Maintain accurate phase detection and state information to prevent incorrect cache behavior

Example of proper reference checking before eviction:

# Check ref_count before evicting shared cache entries
if self.memory_allocator.get_ref_count(self.hot_cache[evict_key]) > 1:
    continue  # Skip eviction if still referenced

Example of proper key type handling:

# Convert string keys to proper types before cache operations
for idx, key_str in enumerate(alloc_request.keys):
    key = CacheEngineKey.from_string(key_str)  # Ensure proper type
    if self._backend.contains(key, pin=True):  # Now works correctly

This prevents cache leaks, ensures proper cache hits, and maintains system reliability when cache entries are shared across multiple operations or requests.

AI model architecture flexibility

Avoid hardcoding AI model-specific assumptions and instead design systems to be configurable across different model architectures, data types, and configurations. This ensures your AI systems can adapt to evolving model designs and support multiple model families without requiring code changes.

Key areas to make flexible:

  1. Data types: Support multiple tensor types (bfloat16, float16, float32) rather than assuming a single type
  2. Model architecture parameters: Use dynamic configuration based on model metadata rather than hardcoded layer counts or dimensions
  3. Model-specific features: Design abstractions that can handle different model formats (e.g., MLA vs non-MLA architectures)

Example of inflexible code:

# Bad: Hardcoded for specific models
if model_name in ["llama-7b", "llama-8b"]:
    return CacheGenConfig(
        key_first_layers=10,
        key_second_layers=20, 
        key_third_layers=32,  # hardcoded total layers
        key_first_bins=32,
        # ...
    )

# Bad: Assumes specific data type
assert t.dtype == torch.float16  # Only supports fp16

Example of flexible code:

# Good: Dynamic configuration based on model metadata
return CacheGenConfig(
    nlayers=model_config.num_layers,  # Use actual layer count
    kspecs=[
        QuantizationSpec(start_layer=0, end_layer=model_config.num_layers//3, bins=32),
        QuantizationSpec(start_layer=model_config.num_layers//3, end_layer=model_config.num_layers, bins=16),
    ],
    # ...
)

# Good: Support multiple data types
DTYPE_TO_TAG = {
    torch.bfloat16: 0,
    torch.float16: 1, 
    torch.float32: 2,
}
tag = DTYPE_TO_TAG[t.dtype]  # Handle any supported type

This approach ensures your AI systems remain maintainable and can easily support new model architectures as they emerge.

Raise exceptions properly

Always ensure exceptions are properly raised and propagated rather than being logged and swallowed. Avoid patterns where errors are caught, logged, and then execution continues normally, as this can hide critical failures from calling code.

Key principles:

  1. Re-raise after logging: When catching exceptions for logging purposes, always re-raise them so calling code can handle the failure appropriately
  2. Avoid returning error strings: Don’t return error messages as strings from functions - raise exceptions instead so they can be properly handled by the caller
  3. Use specific exception handling: Avoid overly broad except Exception: blocks that might hide unexpected errors. When you must use broad handlers, ensure error details are preserved and re-raised
  4. Control exception chaining: Use from None when re-raising exceptions to prevent sensitive data (like file contents) from being captured in exception chains

Example of proper error handling:

# Bad - logs error but continues execution
try:
    result = risky_operation()
except Exception as e:
    logger.error(f"Error occurred: {e}")
    # Execution continues, error is hidden

# Good - logs error and re-raises
try:
    result = risky_operation()
except Exception as e:
    logger.error(f"Error occurred: {e}")
    raise  # or raise specific exception type

# Good - prevents data leakage in exception chains
try:
    file_text = byte_data.decode(encoding)
except (UnicodeDecodeError, UnicodeError):
    raise UnprocessableEntityError(
        f"File encoding detection failed: detected '{encoding}' but decode failed."
    ) from None  # Prevents original exception with file data from being chained

This ensures that failures are communicated to calling code and can be handled appropriately, rather than being silently ignored.

systematic test coverage

When writing tests that need to cover multiple scenarios (different data types, backend capabilities, etc.), structure them systematically rather than duplicating code or hardcoding conditional logic.

For multiple data types, use loops through type arrays instead of duplicating test cases:

// Instead of duplicating test cases for each type
test_cases.emplace_back(new test_conv_2d(..., GGML_TYPE_F32, ...));
test_cases.emplace_back(new test_conv_2d(..., GGML_TYPE_F16, ...));

// Use a loop through types
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16}) {
    test_cases.emplace_back(new test_conv_2d(..., type, ...));
}

For backend-specific capabilities, let the backend report what it supports rather than hardcoding conditional test logic. This allows tests to automatically skip unsupported operations and remain maintainable as backend capabilities evolve.

This approach ensures comprehensive test coverage while keeping the test code clean, maintainable, and adaptable to different environments and configurations.

Metal shared memory sizing

When allocating shared memory for Metal compute shaders, ensure proper sizing that accounts for both SIMD group requirements and Metal platform constraints. Metal requires shared memory buffer sizes to be multiples of 16 bytes. Consider using simplified allocation strategies that meet platform requirements while maintaining correctness.

For SIMD-based operations, you can often allocate a fixed amount of shared memory for simplicity rather than complex calculations:

// Simple approach - ensures Metal alignment requirements
const int64_t shmem_size = 32; // 32*sizeof(float) bytes
[encoder setThreadgroupMemoryLength:shmem_size*sizeof(float) atIndex:0];

This approach handles Metal’s alignment requirements automatically while providing sufficient memory for typical SIMD operations. When precise sizing is needed, ensure the calculated size accounts for the number of SIMD groups in the threadgroup (typically d_state / simd_size) but always verify the result meets Metal’s 16-byte alignment constraint.

optimize memory access patterns

Ensure CUDA kernels use optimal memory access patterns to maximize performance. This involves several key practices:

  1. Use __restrict__ qualifiers on pointer parameters to inform the compiler that pointers don’t alias, enabling better optimizations especially on older GPUs.

  2. Organize threads for coalesced memory access by ensuring adjacent threads access adjacent memory locations. Map thread indices to tensor data as if the tensor was flattened, avoiding fractional warps.

  3. Use typed pointers instead of byte offsets when possible, as explicit byte offset patterns perform poorly on GPUs compared to CPU code.

  4. Ensure proper thread-to-data mapping to avoid uncoalesced accesses.

Example of proper thread organization for coalesced access:

// Good: Adjacent threads access adjacent memory
const int i01 = blockIdx.x;
const int i00 = blockIdx.y*blockDim.x + threadIdx.x;

// Bad: Non-coalesced access pattern
const int i01 = blockDim.x * blockIdx.x + threadIdx.x;
const int i00 = blockIdx.y;

Example of using __restrict__ qualifiers:

static __global__ void kernel(
    const void * __restrict__ src0,
    void * __restrict__ dst,
    // ... other parameters
) {
    // kernel implementation
}

These optimizations are critical for GPU performance as memory bandwidth is often the limiting factor, and proper access patterns can significantly improve throughput.

configurable security settings

Security configurations should be made configurable through environment variables or runtime settings rather than hardcoded in the application logic. This allows teams to adapt security measures to different deployment environments and regulatory requirements without code changes.

Hardcoded security settings create inflexibility and force unnecessary security overhead in environments where it’s not needed. Instead, implement conditional security logic that can be toggled based on configuration.

Example of configurable security headers:

@app.middleware("http")
async def add_headers(request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
    # Make security headers configurable via environment variables
    if os.getenv("ENABLE_SECURITY_HEADERS", "false").lower() == "true":
        response.headers["Content-Security-Policy"] = "default-src 'none'"
        response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
    
    # Conditional validation based on environment requirements
    if os.getenv("STRICT_USER_VALIDATION", "false").lower() == "true":
        if "user_id" in metadata and not _valid_user_id(metadata["user_id"]):
            # Skip invalid user_id rather than failing
            pass

This approach enables compliance with regulatory standards when needed while maintaining flexibility for different deployment scenarios.

AI dependency justification

When modifying AI library dependencies, always provide clear justification for version choices, especially when using non-standard sources like git commits or release candidates. This is critical for AI projects where library compatibility and stability directly impact model performance and inference reliability.

For standard version updates, explain the motivation (bug fixes, new features, compatibility requirements). For non-standard sources like git commits, document:

Example of good justification:

# Using git commit due to bug in official v4.49.0 that affects vision models
"transformers @ git+https://github.com/huggingface/transformers.git@84f0186",
# TODO: Migrate back to official release when v4.49.1+ is available

This practice prevents confusion during code reviews, helps with dependency auditing, and ensures the team understands the rationale behind critical AI library choices that could affect model behavior or performance.

Incremental dependency updates

When updating dependencies in configuration files like requirements.txt, break large changes into smaller, focused pull requests that can be individually tested and validated. Each dependency update should include clear justification for why the change is necessary, especially when it triggers cascading updates to other packages.

Before proposing dependency bumps, verify they are actually required by attempting to revert the changes and running the full test suite. If tests pass without the update, consider whether the dependency change is truly needed.

For example, when updating multiple packages:

# Instead of updating all at once:
# openai==1.40.0 -> 1.45.0
# google-cloud-aiplatform==1.50.0 -> 1.61.0  
# protobuf==4.25.0 -> 5.0.0

# Break into separate PRs:
# PR 1: Update openai to 1.45.0
# PR 2: Update protobuf to 5.0.0 (with justification)
# PR 3: Update google-cloud-aiplatform to 1.61.0 (if still needed)

This approach allows load testing and CI/CD validation of each change independently, making it easier to identify which specific update might cause issues and reducing the blast radius of potential problems.

consistent clear naming

Ensure variable and identifier names are both consistent and unambiguous within their context. When multiple equivalent naming options exist (such as tile_B::I vs tile_C::J for the same value), choose one approach and apply it consistently throughout the codebase. Additionally, select names that clearly distinguish their purpose and avoid confusion with similar concepts in the same scope.

For consistency: If tile_C::J is chosen over tile_B::I, use tile_C::J consistently in all related code locations.

For clarity: Instead of potentially confusing names like ne01 that might be mistaken for byte offsets, use more descriptive alternatives like s01 that clearly indicate their distinct purpose.

This approach prevents both inconsistent naming patterns and semantic confusion that can lead to bugs and reduced code maintainability.

Use precise documentation details

Documentation should provide specific, actionable information rather than vague or generic references. This applies to both technical specifications and descriptive text.

For technical details, use exact package names, commands, or identifiers instead of approximate terms:

# Vague - avoid this
- Install curl-dev package

# Precise - do this  
- Debian/Ubuntu: `sudo apt-get install libcurl4-openssl-dev`
- Fedora/RHEL: `sudo dnf install libcurl-devel`
- Arch: `sudo pacman -S curl`

For links, use descriptive text or show identifiers rather than generic phrases like “available here” or “click here”. This improves accessibility and makes the purpose of the link immediately clear to readers.

# Vague - avoid this
More information is [available here](https://github.com/org/repo/pull/4861)

# Precise - do this
More information is available in <https://github.com/org/repo/pull/4861>

This approach ensures documentation is immediately useful and reduces ambiguity for users trying to follow instructions or understand references.

eliminate code redundancy

Remove duplicate imports, unused legacy code, and repetitive patterns to maintain clean, organized codebases. Extract commonly used functionality into utilities and avoid code duplication through proper abstractions. Use early returns to reduce nesting and improve readability.

Examples of redundancy to eliminate:

Configuration validation standards

Ensure configuration parameters follow consistent validation, naming, and default value standards to improve code safety and clarity.

Key Requirements:

  1. Use None for optional configurations instead of empty strings or other placeholder values
  2. Validate configuration combinations when multiple related settings must be checked together
  3. Use clear, unambiguous parameter names that accurately describe their purpose
  4. Implement proper type checking for configuration values, especially when they can be strings or booleans

Example:

# Bad: Ambiguous naming and empty string default
disk_url: str = ""

# Good: Clear naming and None default
disk_manager_url: Optional[str] = None

# Bad: Single condition check
self.remove_after_retrieve = config.nixl_role == "receiver"

# Good: Validate related conditions together
self.remove_after_retrieve = config.enable_nixl and config.nixl_role == "receiver"

# Good: Proper type validation for config values
if config.extra_config is not None:
    use_cufile = config.extra_config.get("use_cufile", None)
    if use_cufile is not None:
        if isinstance(use_cufile, str):
            use_cufile = use_cufile.lower() == "true"
        elif use_cufile in [False, True]:
            # Handle boolean values
            pass

This approach prevents configuration-related bugs, makes the codebase more maintainable, and provides clearer error messages when invalid configurations are provided.

prefer const variables

Add the const qualifier to variables that are not modified after initialization. This improves code clarity, prevents accidental modifications, and makes the code’s intent more explicit to both compilers and other developers.

Apply const to:

Example transformations:

// Before
float2 dsB;
dsB = __half22float2(y_dm[j*MMQ_TILE_Y_K + k01/QI8_1]);

int global_idx = blockIdx.x * blockDim.x + threadIdx.x;
int total_elements = batches * channels * out_h * out_w;

// After  
const float2 dsB = __half22float2(y_dm[j*MMQ_TILE_Y_K + k01/QI8_1]);

const int global_idx = blockIdx.x * blockDim.x + threadIdx.x;
const int total_elements = batches * channels * out_h * out_w;

This practice is especially important in CUDA kernels and performance-critical code where clarity about data mutability helps with optimization and debugging.

Improve code modularity

Organize code into well-structured, reusable components by extracting specific functionality into separate functions, using flexible parameter patterns, and consolidating duplicate logic. This improves readability, maintainability, and extensibility.

Key practices:

  1. Extract validation logic: When you have complex validation checks, create dedicated validation functions rather than embedding them inline
  2. Use parameter forwarding: When passing parameters to nested functions, consider using **kwargs to create a flexible foundation for future extensions
  3. Consolidate duplicate functionality: When similar logic appears in multiple places, refactor it into a shared, reusable function

Example of good modularity:

# Instead of inline validation
def process_file(file_path):
    if not file_path.exists():
        _exit_with_error(f"File does not exist: {file_path}")
    if not file_path.is_file():
        _exit_with_error(f"Path is not a file: {file_path}")
    # ... processing logic

# Extract validation into separate function
def _validate_file_path(file_path):
    if not file_path.exists():
        _exit_with_error(f"File does not exist: {file_path}")
    if not file_path.is_file():
        _exit_with_error(f"Path is not a file: {file_path}")

def process_file(file_path):
    _validate_file_path(file_path)
    # ... processing logic

# Use parameter forwarding for flexibility
def create_converter(**kwargs):
    return CustomConverter(**kwargs)  # Instead of passing individual params

CUDA compatibility verification

When modifying CUDA versions or GPU architecture configurations in AI/ML projects, verify compatibility across three critical dimensions: CUDA version support, GPU architecture coverage, and dependent library requirements. Check official compatibility matrices, test with target hardware, and validate that AI libraries maintain their functionality.

For GPU architectures, prefer inclusive configurations that support multiple generations rather than restrictive single-architecture builds. For example, use CMAKE_CUDA_ARCHITECTURES=90;100;120 and TORCH_CUDA_ARCH_LIST="9.0 10.0 12.0" to support Hopper and Blackwell families together.

Before upgrading CUDA versions, consult framework release notes (e.g., PyTorch dropped CUDA 12.4 support in v2.7.0) and verify that critical dependencies like specialized AI libraries won’t break. Some libraries have strict version requirements that may disable features if not met exactly.

Example verification process:

# Check base image CUDA version
FROM nvcr.io/nvidia/tritonserver:25.03-py3-min  # Contains CUDA 12.8

# Configure for multiple GPU generations
ARG CMAKE_CUDA_ARCHITECTURES=90;100;120
ARG TORCH_CUDA_ARCH_LIST="9.0 10.0 12.0"

# Validate compatibility with AI dependencies
RUN python -c "import torch; print(f'CUDA: {torch.version.cuda}, Architectures: {torch.cuda.get_arch_list()}')"

explicit None handling

Always handle potential None values explicitly through defensive programming patterns, proper type annotations, and explicit checks to prevent runtime errors and improve code reliability.

Use defensive patterns like value or default for safe fallbacks:

# Good: Explicit None handling with fallback
e.metadata.languages = detect_languages(e.text or "")
full_text = " ".join(e.text or "" for e in elements if hasattr(e, "text"))

# Good: Explicit None checks with proper logic
if encoding is None or (confidence is not None and confidence < ENCODE_REC_THRESHOLD):
    # handle the case

Use proper Optional type annotations when values can be None:

# Good: Correct Optional typing
max_pages: Optional[int] = None
customer_id: Optional[str] = None

# Bad: Missing Optional annotation
customer_id: str = None  # Type mismatch

Handle functions that may return None explicitly:

# Good: Check for None return value
nltk_data_dir = get_nltk_data_dir()
if nltk_data_dir is None:
    raise ValueError("Could not find a default download directory")

# Good: Use assertions when None is guaranteed not to occur
assert self._file is not None  # -- assured by ._validate() --

Avoid unnecessary None-to-string conversions and prefer removing optionality at interface boundaries when the None case adds no value. When designing APIs, consider whether Optional parameters truly need to be optional or if a sensible default can be provided instead.

Use descriptive consistent names

Choose names that clearly convey their purpose, role, and meaning while maintaining consistency with established patterns in the codebase. Names should be self-documenting and unambiguous to future maintainers.

Key principles:

  1. Semantic clarity: Names should reflect their actual purpose and role in the system architecture. For example, use receiver_host instead of peer_host_name when the architecture has a clear receiver-sender relationship, or num_skip_prefix_chunk instead of num_skip_chunk when the meaning is “number of prefix chunks to skip”.

  2. Consistency with patterns: Follow established naming conventions in the codebase. If other batch operations use batched_xxx pattern, use batched_contains instead of batch_contains. Similarly, maintain consistency between related functions like using batched instead of mixing layerwise and batched.

  3. Avoid ambiguous abbreviations: Use clear variable names instead of cryptic abbreviations. Replace unclear names like anw and anws with descriptive names like cache_exists_results.

  4. Distinguish similar entities: When multiple files or classes serve similar but distinct purposes, use names that clearly differentiate them. Instead of disagg_proxy_server.py and disagg_proxy_server_original.py, use names that describe their specific roles.

Example of good naming:

# Instead of:
def batch_contains(self, keys): pass
anws = self.engine_.batched_contains(keys)
for anw in anws:
    if not anw:

# Use:
def batched_contains(self, keys): pass  # Consistent with other batched_xxx methods
cache_exists_results = self.engine_.batched_contains(keys)
for cache_exists in cache_exists_results:
    if not cache_exists:

Follow naming conventions

Maintain consistency with established naming patterns and conventions throughout the codebase. This includes matching enum names with their value prefixes, following existing API naming patterns, using consistent variable naming within similar contexts, and applying appropriate prefixes to avoid naming conflicts.

Key principles:

Example:

// Good: Enum name matches value prefix
enum diffusion_alg {
    DIFFUSION_ALG_ORIGIN       = 0,
    DIFFUSION_ALG_MASKGIT_PLUS = 1,
};

// Bad: Enum name doesn't match value prefix  
enum diffusion_algorithm {
    DIFFUSION_ALG_ORIGIN       = 0,
    DIFFUSION_ALG_MASKGIT_PLUS = 1,
};

measure before optimizing

Always profile and measure performance impact before implementing optimizations, especially micro-optimizations. Many seemingly beneficial optimizations provide no measurable improvement and can hurt code maintainability.

Before adding performance optimizations:

  1. Use proper profiling tools to identify actual bottlenecks
  2. Measure the current performance with realistic workloads
  3. Implement the optimization
  4. Measure again to verify the improvement is significant
  5. Consider the maintenance cost vs. performance gain

Example from optimizer parameter pre-computation:

// Before: Micro-optimization attempt
const float keep = 1.0f - opt_pars.adamw.alpha * opt_pars.adamw.wd;

// After measurement: "the micro-optimization has no visible benefit in this case"
// Reverted to simpler, more maintainable code

Common scenarios where measurement reveals optimizations are ineffective:

Use profiling-enabled tools when available (e.g., enqueue_nd_range for kernel timing) and prefer steady_clock over high_resolution_clock for accurate timing measurements. Remember that maintainability and code clarity often outweigh minimal performance gains.

Use strongly typed configurations

Prefer strongly typed configuration values over primitive types to improve code safety and clarity. Use enum classes instead of raw strings for categorical options, and proper boolean types instead of integer flags for binary settings.

For categorical configurations, replace string parameters with enum classes:

// Instead of:
params.dataset_format = format; // string

// Use:
enum class DatasetFormat { AUTO, TEXT, PARQUET, GGUF };
params.dataset_format = DatasetFormat::AUTO;

For boolean configurations, use explicit boolean conversion and values:

// Instead of:
supports_set_rows = LLAMA_SET_ROWS ? atoi(LLAMA_SET_ROWS) : 0;

// Use:
supports_set_rows = LLAMA_SET_ROWS ? atoi(LLAMA_SET_ROWS) != 0 : false;

This approach prevents invalid configuration values, makes code more self-documenting, and enables better IDE support with auto-completion and type checking.

Ensure operation completion safety

In concurrent code, ensure that operations complete safely before proceeding to prevent race conditions and data corruption. This applies to several scenarios:

Lock Management: Use context managers instead of manual lock acquisition to guarantee proper cleanup:

# Good - automatic cleanup
with self.manager_lock:
    prefetch_task = self.prefetch_tasks.pop(key)

# Avoid - manual lock management
self.manager_lock.acquire()
prefetch_task = self.prefetch_tasks.pop(key)
self.manager_lock.release()

Async Operation Safety: When data integrity is critical, ensure async operations complete before proceeding:

# Good - synchronous when safety is needed
memory_obj.tensor.copy_(hidden_states)

# Risky - async copy without synchronization
memory_obj.tensor.copy_(hidden_states, non_blocking=True)

Atomic File Operations: Prevent race conditions in file operations using temporary files and atomic renames:

# Good - atomic operation
tmp_path = path + ".tmp"
with open(tmp_path, 'wb') as f:
    f.write(data)
os.rename(tmp_path, path)  # Atomic on most filesystems

Reference Counting Timing: Ensure reference counting operations happen at the right time relative to object usage:

# Ensure backend consumes object before counting down
backend.submit_put_task(key, obj)
# Backend should call ref_count_down() after consumption

Stream Synchronization: Use proper stream synchronization for CUDA operations:

put_stream = torch.cuda.Stream()
if kv_chunk.device != torch.cpu:
    put_stream.wait_stream(torch.cuda.default_stream(kv_chunk.device))

The key principle is to identify critical sections where partial completion could cause issues and ensure these operations are atomic or properly synchronized.

Use proper logging mechanisms

Always use the logging library instead of print statements and maintain consistent logging patterns throughout the codebase. Print statements should be avoided as they bypass proper log level controls, formatting, and routing. Instead, use appropriate logging methods like verbose_logger.info(), verbose_logger.debug(), or verbose_logger.warning().

Centralize logging logic to avoid scattered conditional blocks that can lead to bugs and inconsistent behavior. When implementing logging functionality, encapsulate the logic in dedicated functions or classes rather than duplicating if/else patterns across the codebase.

For logging integrations, use standardized logging objects and interfaces to ensure compatibility and prevent downstream issues.

Example of what to avoid:

print("error", e)  # Don't use print statements
print(file=sys.stderr)  # Even with stderr redirection

Example of proper approach:

verbose_logger.info("Processing completed successfully")
verbose_logger.debug("Bedrock AI: make_bedrock_api_request response: %s", redacted_response)
verbose_logger.warning("PostHog API Key not found in environment variables")

When dealing with sensitive data, always redact PII and confidential information before logging to prevent security leaks.

explicit control flow logic

Use explicit control flow structures like switch statements instead of complex boolean logic or implicit reasoning when handling algorithmic decisions. This improves code clarity, maintainability, and reduces the likelihood of logical errors in computational paths.

Key principles:

Example of preferred approach:

// Instead of implicit boolean reasoning:
struct ggml_tensor * opt_step = 
    m ? ggml_opt_step_adamw(ctx, node, grad, m, v, params) :
        ggml_opt_step_sgd(ctx, node, grad, params);

// Use explicit switch statement:
struct ggml_tensor * opt_step;
switch (opt_ctx->optimizer_type) {
    case GGML_OPT_OPTIMIZER_ADAMW:
        opt_step = ggml_opt_step_adamw(ctx, node, grad, m, v, params);
        break;
    case GGML_OPT_OPTIMIZER_SGD:
        opt_step = ggml_opt_step_sgd(ctx, node, grad, params);
        break;
    default:
        GGML_ABORT("fatal error");
        break;
}

This approach makes algorithmic decisions explicit, improves debugging, and ensures all code paths are properly handled. It also prevents mixing of unrelated algorithmic concerns and maintains clearer separation of computational logic.

AI parameter organization

Ensure proper organization and placement of AI model parameters, hyperparameters, and configuration options. Avoid parameter duplication by reusing existing values, and place parameters in their logically appropriate structures.

Key principles:

Example of good practice:

// Instead of adding a new parameter:
struct common_params_diffusion {
    int32_t max_length = 512;  // DON'T: create new parameter
};

// Use existing parameter:
// max_length should be removed and the existing n_ubatch parameter should be used instead

// Instead of duplicating available data:
const int n_swa;  // DON'T: duplicate from hparams
// This is already available from the hparams - no need to duplicate it here

This practice is especially important in AI model implementations where parameter proliferation can lead to configuration inconsistencies, maintenance overhead, and potential bugs in model behavior. Always check if the functionality you need already exists before introducing new parameters.

Validate access controls

Always implement comprehensive validation for access control mechanisms to prevent unauthorized access and privilege escalation. This includes validating user permissions, enforcing tenant isolation, and preventing bypass through direct API calls.

Key validation points:

  1. Tenant isolation: Validate tenant_id early in methods to prevent cross-tenant access
  2. Input validation: Add validation for user inputs that could bypass restrictions (e.g., email uniqueness checks)
  3. Privilege escalation prevention: When handling IDs from untrusted sources, always include tenant context
  4. Proper HTTP responses: Return 403 for authorization failures, not 404

Example implementation:

def get_execution_by_id(self, execution_id: str, tenant_id: Optional[str] = None):
    # When tenant_id is None, it's the caller's responsibility to ensure proper data isolation
    # If execution_id comes from untrusted sources (e.g., API request), 
    # set tenant_id to prevent horizontal privilege escalation
    if tenant_id is None:
        # Log warning or require tenant_id for security
        pass
    
    # Early tenant validation
    if dataset.tenant_id != user.current_tenant_id:
        raise NoPermissionError("Access denied")
    
    # Additional permission validation
    if not self._has_permission(user, execution_id):
        abort(403)  # Not 404 - proper HTTP semantics

This approach ensures that security boundaries are maintained at multiple layers and prevents common authorization vulnerabilities.

maintain code consistency

Always follow the existing code style patterns and conventions found in the surrounding codebase rather than introducing new formatting approaches. This includes matching brace placement, function organization, and parameter formatting styles.

For brace placement, match the surrounding code:

// If surrounding code uses this style:
enum ggml_opt_optimizer {
    GGML_OPT_ADAM = 0,
    GGML_OPT_SGD  = 1,
};

For function organization, group related declarations together:

// Keep getters together
GGML_API ggml_opt_context_t ggml_opt_context_init(void);
GGML_API enum ggml_opt_optimizer_type ggml_opt_context_optimizer_type(ggml_opt_context_t);
GGML_API void ggml_opt_context_free(ggml_opt_context_t);

For long parameter lists, improve readability by placing each parameter on a new line:

void diffusion_generate(
    llama_context * ctx,
    const llama_token * input_tokens,
    llama_token * output_tokens,
    int32_t n_input,
    int32_t max_length,
    struct diffusion_params params,
    int32_t * n_generated);

When established patterns exist in the codebase, follow them even if they require some code duplication. Consistency across the codebase is more valuable than eliminating all duplication, as it makes the code more predictable and maintainable for all contributors.

validate bounds before access

Always validate array indices, buffer sizes, and container bounds before accessing elements to prevent out-of-bounds errors and undefined behavior. This includes checking that indices are within valid ranges and that containers have sufficient size before element access.

Use explicit bounds checking with assertions or conditional statements before array/buffer operations:

// Check array bounds before access
GGML_ASSERT(seq_id_src < seq_to_stream.size());
GGML_ASSERT(seq_id_dst < seq_to_stream.size());
const auto s0 = seq_to_stream[seq_id_src];
const auto s1 = seq_to_stream[seq_id_dst];

// Validate string length before character access
if (nextArg.empty() || (nextArg.size() >= 2 && nextArg[0] == '-' && !std::isdigit(nextArg[1]))) {
    // Handle invalid case
}

// Use reserve() and push_back() instead of direct indexing for dynamic arrays
res.idxs.reserve(n_tokens);
// Use push_back() instead of res.idxs[i] = value

Consider using safer alternatives like the .at() method for containers, which throws exceptions on out-of-bounds access, or implement comprehensive bounds checking at function entry points. This practice prevents memory corruption, crashes, and security vulnerabilities that can arise from accessing invalid memory locations.

measure algorithm performance impact

Choose algorithms based on measurable performance benefits rather than theoretical complexity advantages. Start with simpler implementations and only adopt complex algorithms when they demonstrate clear performance improvements over basic approaches.

When evaluating algorithmic choices:

For example, when considering the Aho-Corasick algorithm for multi-pattern string matching: “If we can show that the algorithm improves the performance in a measurable way, then it’s ok. If not, we might want to fallback to some simpler brute-force approach.” Similarly, when implementing matrix operations, focus on achieving good performance with scalar kernels before adding cooperative matrix optimizations.

This approach ensures that algorithmic complexity is justified by real performance gains rather than theoretical benefits, leading to more maintainable and predictably performing code.

prioritize compile-time optimizations

In performance-critical code, favor compile-time optimizations over runtime flexibility to enable better compiler optimizations and reduce execution overhead. This principle applies to several key areas:

Use compile-time constants instead of runtime values: Replace runtime function calls with compile-time constants when the values are known at compile time. For example, use #define WG_M 16 instead of get_local_size() in GPU kernels, as compile-time constants allow the compiler to fully unroll loops and pre-calculate memory address offsets.

Enable loop unrolling with compiler hints: Add [[unroll]] annotations to loops with constant trip counts, especially in nested loop scenarios where the compiler might not automatically unroll:

[[unroll]] for (uint j = 0; j < NUM_COLS; ++j) {
    [[unroll]] for (uint n = 0; n < num_rows; ++n) {
        temp[j][n] = subgroupClusteredAdd(temp[j][n], GROUP_SIZE);
    }
}

Eliminate branches through constant folding: Structure code so the compiler can inline functions and fold constants, rather than using runtime boolean expressions in hot loops. Write separate functions for different code paths and let the compiler optimize each path independently.

Prefer build-time compilation: When possible, compile shaders and other performance-critical code at build time rather than runtime to avoid compilation overhead and enable better optimization opportunities.

These techniques are particularly important in GPU kernels, SIMD code, and other performance-critical paths where even small optimizations can yield significant performance improvements (10-17% gains are common).

API minimalism principle

Keep APIs minimal by avoiding redundant interfaces and preferring simple solutions over complex ones. When adding functionality, first consider extending existing functions rather than creating new ones, especially when there’s only a single use case. Remove unnecessary options and flags to maintain interface simplicity.

Key principles:

Example of extending existing API instead of adding new one:

// Instead of adding llama_sampler_accept_str() for one use case:
// AVOID: llama_sampler_accept_str(gsmpl->grmr, trigger.c_str());

// Extend existing function to accept optional initial string:
gsmpl->grmr = llama_sampler_init_grammar(model, grammar_str, "root", trigger.c_str());

Example of interface simplification:

// Instead of multiple preview flags:
// AVOID: --preview, --preview-count, --detokenize-preview

// Use single flag with sensible defaults:
// PREFER: --preview (shows fixed number of sequences as both tokens and text)

This principle ensures APIs remain maintainable, reduce cognitive load for users, and prevent feature bloat while still providing necessary functionality.

Maintain consistent naming patterns

Follow established naming conventions in the codebase and align with original source patterns while using modern, non-deprecated alternatives. Avoid creating confusing duplication in naming structures.

When adding new components, maintain consistency with existing patterns rather than creating new naming schemes. For example, if existing tensors use blk.{bid}.component format, continue that pattern rather than introducing component.{bid}.component which creates duplication.

Use modern language features and avoid deprecated imports. Replace deprecated typing constructs like Dict with built-in alternatives like dict or more appropriate types like Mapping.

When naming classes or components based on external models, align with the original naming conventions including version suffixes to maintain traceability and consistency.

Example:

# Good: Consistent with existing pattern, avoids duplication
MODEL_TENSOR.SHORTCONV_CONV: "blk.{bid}.shortconv.conv"

# Avoid: Creates confusing duplication
MODEL_TENSOR.SHORTCONV_CONV: "shortconv.{bid}.shortconv.conv"

# Good: Modern typing
from typing import Callable, Iterator, dict

# Avoid: Deprecated typing
from typing import Dict

Specify naming formats explicitly

When defining naming conventions in documentation, specifications, or APIs, explicitly specify the exact format and patterns to be used. Avoid ambiguous descriptions that could lead to inconsistent implementations.

This prevents confusion about whether to use leading zeros, specific data types, or formatting patterns. Clear specifications ensure all developers implement naming consistently.

Examples of improvement:

Instead of:

training.format.version: string (e.g., "1.0") - Specification version
training.tensor.{index} (e.g., training.tensor.0, training.tensor.1, ...)

Specify explicitly:

training.format.version: uint32 (e.g., 10034 for v1.0.34) - Specification version
training.tensor.{index} (e.g., training.tensor.0, training.tensor.1, ...). No leading zeros.

This applies to version numbering schemes, identifier patterns, file naming conventions, and any other naming standards where ambiguity could lead to inconsistent implementation.

Prefer lightweight observability integrations

Observability integrations should minimize external dependencies and avoid patching LiteLLM’s core functions. Instead of requiring users to install additional SDKs, prefer lightweight approaches using custom loggers with direct HTTP calls (like httpx) or implement CustomBatchLogger interfaces. Avoid monkey-patching completion functions as this can cause unexpected errors and conflicts with other integrations.

The preferred pattern follows langsmith’s approach - use pure httpx for sending telemetry data rather than requiring SDK installations. This reduces dependency bloat, prevents version conflicts, and maintains better isolation between different observability tools.

Example of preferred approach:

# Good: Lightweight custom logger
class MyObservabilityLogger(CustomBatchLogger):
    def __init__(self, api_key):
        self.api_key = api_key
        
    def log_success_event(self, kwargs, response_obj, start_time, end_time):
        # Use httpx to send data directly
        httpx.post("https://api.example.com/logs", 
                  json={"request": kwargs, "response": response_obj})

# Avoid: Requiring SDK installation and patching
# weave.init()  # This patches litellm functions

This approach ensures observability works reliably across all LiteLLM providers without introducing additional complexity or potential conflicts.

implement or fail explicitly

Algorithmic functions should either correctly implement their intended behavior or explicitly fail with an error, rather than silently returning potentially incorrect or unchanged results. This prevents subtle bugs where callers assume the algorithm was applied when it wasn’t.

When implementing algorithmic functions, avoid “do nothing” fallback behavior that could mask errors. Instead, either:

  1. Implement the complete algorithm for all supported input types
  2. Throw an explicit error for unsupported cases

For example, in a quantization function:

def quantize(data: np.ndarray, qtype: GGMLQuantizationType) -> np.ndarray:
    if qtype == GGMLQuantizationType.F32:
        return data.astype(np.float32, copy=False)
    elif qtype == GGMLQuantizationType.F16:
        return data.astype(np.float16, copy=False)
    elif (q := _type_traits.get(qtype)) is not None:
        return q.quantize(data)
    # BAD: Silent no-op that defeats the function's purpose
    # elif is_tmac_dtype(qtype):
    #     return data
    # GOOD: Explicit error for unsupported types
    else:
        raise QuantError(f"Quantization not implemented for type {qtype}")

This principle also applies to detection algorithms - they should clearly indicate when they cannot provide meaningful results rather than producing potentially misleading output.

validate configuration options explicitly

When handling configuration options or tool types, use explicit conditional statements with proper error handling for unknown values instead of implicit fallbacks or else clauses. This prevents silent failures and makes it clear which configurations are supported.

Use elif statements for each known configuration option and include an assert False or explicit error for unhandled cases:

if self.tool == "llama-bench":
    self.check_keys = set(LLAMA_BENCH_KEY_PROPERTIES + ["build_commit", "test_time", "avg_ts"])
elif self.tool == "test-backend-ops":
    self.check_keys = set(TEST_BACKEND_OPS_KEY_PROPERTIES + ["build_commit", "test_time"])
else:
    assert False, f"Unknown tool type: {self.tool}"

This approach helps catch configuration errors early, makes the supported options explicit in the code, and prevents issues like duplicate settings or silent overwrites that can occur with implicit handling. When adding new configuration options, developers must explicitly update the validation logic, ensuring all cases are properly considered.

optimize code placement

Ensure code is placed in its most logical and efficient location to improve maintainability and performance. This includes consolidating duplicate functionality into shared headers or utility files, and moving loop-invariant conditions outside of loops.

When you encounter duplicate functions across multiple files, consider moving them to a shared header as inline functions. For example, instead of having ggml_are_same_layout duplicated in ggml-alloc.c and ggml-backend.cpp, move it to ggml.h as an inline function.

Similarly, extract conditions that don’t change within loops to outside the loop:

// Instead of:
for (size_t i = 0, n = 3; i < n; ++i) {
    if (op->src[i] != NULL && (op->src[i]->type == GGML_TYPE_BF16 || op->type == GGML_TYPE_BF16)) {
        // ...
    }
}

// Extract the loop-invariant condition:
bool op_is_bf16 = (op->type == GGML_TYPE_BF16);
for (size_t i = 0, n = 3; i < n; ++i) {
    if (op->src[i] != NULL && (op->src[i]->type == GGML_TYPE_BF16 || op_is_bf16)) {
        // ...
    }
}

This approach reduces code duplication, improves performance by avoiding redundant checks, and enhances code readability by making the logic structure clearer.

Catch specific exception types

Avoid using broad exception handlers like except Exception: or bare except:. Instead, catch specific exception types that you expect and can handle meaningfully. This makes code more maintainable by:

  1. Preventing bugs from being silently masked
  2. Making error handling intentions clear
  3. Improving debugging by preserving stack traces

Example:

# Bad - catches and hides all errors
try:
    result = json.loads(args.compilation_config)
except Exception as e:
    logger.error(f"Error: {e}")
    return None

# Good - handles specific expected error
try:
    result = json.loads(args.compilation_config)
except json.JSONDecodeError as e:
    logger.error(f"Invalid JSON configuration: {e}")
    raise ValueError(f"Configuration must be valid JSON: {e}") from e

When catching multiple exceptions, order them from most specific to most general. If you must catch a broad exception (e.g., for API boundaries), log the full exception details and consider re-raising or wrapping in a domain-specific error.

Vectorize over Python loops

Replace Python loops and list comprehensions with vectorized operations when processing tensors or performing repeated computations. This optimization reduces interpreter overhead and enables efficient parallel execution on modern hardware.

Key practices:

  1. Use torch.einsum/matmul for batched matrix operations
  2. Leverage torch.gather for indexed tensor operations
  3. Pre-compute and cache frequently accessed values
  4. Process data in batches rather than element-by-element

Example - Converting nested loops to vectorized operations:

# Before - Inefficient nested loops
def disentangled_attention_bias(query_layer, key_layer, relative_pos, rel_embeddings):
    content_to_position = torch.zeros_like(content_to_content)
    for i in range(seq_len):
        q_i = query_layer[:, :, i, :]
        for j in range(seq_len):
            rel_idx = relative_pos[i, j].item()
            content_to_position[:, :, i, j] = torch.einsum("bnh,rnh->bnr", 
                                                         q_i, rel_k)[:, :, rel_idx]

# After - Vectorized implementation 
def disentangled_attention_bias(query_layer, key_layer, relative_pos, rel_embeddings):
    c2p_scores = torch.einsum("bnqh,rnh->bnqr", query_layer, rel_k)
    rel_pos_expanded = relative_pos.unsqueeze(0).unsqueeze(0)
    content_to_position = torch.gather(c2p_scores, 3, rel_pos_expanded)

This optimization can significantly improve performance by:

Follow logging best practices

Maintain high-quality logging throughout the codebase by following these best practices:

  1. Use appropriate log levels: Reserve debug for detailed troubleshooting, info for general operational information, warning for potential issues, and error/critical for actual problems. Avoid excessive logging at higher levels. 
    # For large object dumps or detailed diagnostics
logger.debug("Initialized config %s", config)
# Not: logger.info("Initialized config %s", config)

  2. Remove temporary debugging logs: Debug statements with developer identifiers (e.g., "[Kourosh]") should be removed before merging code.

  3. Optimize logging performance: Guard expensive parameter computations to avoid unnecessary work when the log level is disabled: 
    if logger.isEnabledFor(logging.DEBUG):
    logger.debug("Complex calculation result: %s", 
                 ",".join(map(str, complex_calculation())))

  4. Avoid logging in loops unless each iteration provides unique valuable information. Consider logging summaries before/after the loop instead.

  5. Use standard logging methods: Use logger.warning() (not the deprecated logger.warn()), and use logger.exception() for exception logging to automatically include tracebacks: 
    try:
    # Some operation
except Exception:
    logger.exception("Failed to perform operation")
    # Not: logger.error("Failed: %s", str(e))
  6. Initialize loggers consistently: Use the project’s standard logger initialization pattern: 
    from vllm.logger import init_logger
logger = init_logger(__name__)
  7. Make log messages clear and actionable: Include relevant context and avoid ambiguous messages.

Use self-documenting names

Variable, function, and parameter names should accurately reflect their purpose, behavior, and content. Choosing descriptive names improves code readability and reduces bugs caused by naming confusion.

When naming variables and parameters:

  1. Use names that precisely describe what the entity represents
  2. Maintain consistent naming across related components (config parameters, CLI args, classes)
  3. Avoid misleading names that imply incorrect functionality or types

Examples of issues to avoid:

# INCORRECT: Using "width" for height is misleading
img_height = image_processor.size.get("width", 224)

# CORRECT: Name accurately reflects the value
img_height = image_processor.size.get("height", 224)

# INCORRECT: CLI flag name doesn't match config parameter
scheduler_group.add_argument("--max-waiting-queue-length", 
                             **scheduler_kwargs["limit_queue_length"])

# CORRECT: Consistent naming between CLI and config
scheduler_group.add_argument("--limit-queue-length", 
                             **scheduler_kwargs["limit_queue_length"])

# INCORRECT: Name in __all__ doesn't match actual class name
__all__ = ["MoEConfig"]  # But actual class is FusedMoEConfig

# CORRECT: Name in __all__ matches the actual implementation
__all__ = ["FusedMoEConfig"]

Descriptive and consistent naming serves as implicit documentation, making code more maintainable and reducing the likelihood of errors during development.

Document AI model capabilities

Provide clear, comprehensive documentation for AI model capabilities, especially for multimodal features and deployment scenarios. Include:

  1. Explicit documentation of supported input types and formats
  2. Working code examples for common deployment patterns
  3. Clear parameter descriptions with their effects

Example for documenting multimodal capabilities:

# Clear documentation of input types and formats
"""
The Score API supports:
- Text inputs: Plain text strings for standard NLP tasks
- Multimodal inputs: 
  - Images: Supported formats: PNG, JPEG
  - Audio: Supported formats: WAV, MP3
  
Example usage:
"""
from vllm import LLM
llm = LLM(
    model="example-multimodal-model",
    # Document key parameters
    tensor_parallel_size=4,  # Uses 4 GPUs for parallel processing
    max_model_len=2048,     # Maximum sequence length
)

# Include complete, working example
outputs = llm.generate(
    prompt="Describe this image",
    multi_modal_data={
        "image": image_data,  # Numpy array or PIL Image
    },
    sampling_params=SamplingParams(
        temperature=0.2,
        max_tokens=64,
    ),
)

Environment-aware configuration values

When creating configuration files or defining configuration constants, ensure they properly adapt to different environments (Python versions, hardware architectures, etc.) with clear documentation. This prevents compatibility issues and helps other developers understand configuration choices.

For dependency specifications:

# Example: Properly constrained dependency with explanation
mistral_common[opencv] >= 1.6.2; python_version<="3.12" # Not compatible with Python 3.13 (see issue #XYZ)

For compile-time configuration constants:

// Example: Architecture-aware constant definition
#if defined(USE_ROCM)
  #if defined(__AMDGPU_WAVEFRONT_SIZE__)
    // Using compiler-provided macro for wavefront size
    #define WARP_SIZE __AMDGPU_WAVEFRONT_SIZE__
  #elif defined(__GFX9__)
    // Fallback for specific architecture
    #define WARP_SIZE 64
  #else
    // Default fallback
    #define WARP_SIZE 32
  #endif
#else // CUDA
  #define WARP_SIZE 32
#endif

Following these practices ensures configurations work correctly across diverse environments and remain maintainable as systems evolve.

Eliminate code redundancy

Keep your codebase maintainable by eliminating both unnecessary and duplicated code:

  1. Remove debugging artifacts: Delete commented-out code, debug print statements, and unused imports before merging. 
    # Remove this:
# print("kv_cache.shape = {}".format(kv_cache.shape))
   
# And this:
from vllm.model_executor.layers.activation import get_act_fn  # unused
  2. Extract duplicated logic into helper functions or shared classes: 
    # Instead of this:
def save_configs(num_experts, ...):
    if enable_expert_parallel:
        filename = get_config_file_name(
            local_num_experts, shard_intermediate_size // 2, 
            dtype_str, block_quant_shape)
    else:
        filename = get_config_file_name(
            num_experts, shard_intermediate_size // 2, 
            dtype_str, block_quant_shape)
               
# Do this:
def save_configs(num_experts, ...):
    num_experts_for_filename = (num_experts // ep_size
                               if enable_expert_parallel else num_experts)
    filename = get_config_file_name(
        num_experts_for_filename, shard_intermediate_size // 2,
        dtype_str, block_quant_shape)
  3. DRY up similar functionality when implementing related methods: 
    # Instead of repeating logic across methods:
def calculate_metrics(...):
    # Duplicated calculation logic
       
# Use intermediate variables or helper functions:
def _calculate_common_part(data):
    # Shared logic extracted here
       
def calculate_metrics(...):
    result = _calculate_common_part(data)
    # Unique logic continues...

Eliminate redundancy in all forms - be it duplicate code blocks, methods, imports, or commented-out experimental code that’s no longer needed. This makes your code more maintainable, reduces the chance of bugs when changes are needed, and improves readability for other developers.

Document code thoroughly

Always include comprehensive documentation for your code through both docstrings and explanatory comments.

For classes, functions, and methods, add docstrings that explain:

For non-obvious implementation details, add comments explaining:

def uniform_random_select_experts(
    hidden_states: torch.Tensor,
    router_logits: torch.Tensor,
    top_k: int,
    indices_type: Optional[torch.dtype] = None,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Selects experts randomly with uniform distribution instead of based on router scores.
    
    Args:
        hidden_states: Input tensor of shape [batch_size, sequence_length, hidden_size]
        router_logits: Router scores from which only the shape is used
        top_k: Number of experts to select for each token
        indices_type: Optional dtype for the output indices tensor
        
    Returns:
        Tuple containing (routing_weights, expert_indices)
    """
    # CPU only supports V1 architecture due to specialized optimizations
    # that aren't available in the regular implementation
    if current_platform.is_cpu() and os.environ.get("VLLM_USE_V1", "0") == "0":
        pytest.skip("CPU only supports V1")

Well-documented code improves maintainability, enables easier onboarding of new team members, and reduces the time needed to understand and modify existing functionality.

Avoid magic numbers

Replace hardcoded values and magic numbers in AI model code with named constants or configuration parameters. This improves maintainability, reduces errors, and makes code more adaptable to different model architectures.

Common issues include hardcoded head dimensions, tensor shapes, and data paths:

# Problematic: Hardcoded head dimensions
prefill_wrapper.plan(
    qo_indptr,
    kv_indptr,
    num_qo_heads,
    num_kv_heads,
    192,  # Magic number for head_dim_qk
    causal=True,
    head_dim_vo=128,  # Another magic number
)

# Better: Use variables from configuration
prefill_wrapper.plan(
    qo_indptr,
    kv_indptr,
    num_qo_heads,
    num_kv_heads,
    head_dim_qk,  
    causal=True,
    head_dim_vo=self.kv_cache_spec.head_size,
)

For model parameters, always derive values from configuration objects rather than embedding assumptions in code. When testing, avoid hardcoded paths and dimensions that may not be portable across environments. In multimodal models, be especially careful with tokenization and embedding logic, which should adapt to the model architecture rather than assuming specific token structures.

Check before access

Always verify that objects, attributes, and variables are not None before accessing their properties, calling their methods, or using them in operations. This prevents common runtime errors like TypeError, AttributeError, or UnboundLocalError when working with optional values.

For optional attributes or dictionary keys:

# Before - may raise AttributeError or TypeError if token_type_ids is None
token_type_embeddings = self.token_type_embeddings(token_type_ids)

# After - safely checks before access
if self.token_type_ids is not None:
    model_kwargs["token_type_ids"] = cast(torch.Tensor, self.token_type_ids)[:num_scheduled_tokens]

For conditional variables:

# Before - may cause UnboundLocalError if condition is false
if is_v1_kv_transfer_group():
    invalid_block_ids = connector.get_block_ids_with_load_errors()
# Later code uses invalid_block_ids regardless of condition

# After - initialize before conditional assignment
invalid_block_ids: Optional[set[int]] = None
if is_v1_kv_transfer_group():
    invalid_block_ids = connector.get_block_ids_with_load_errors()

For optional containers:

# Before - raises TypeError if finished_sending is None
for req_id in kv_connector_metadata.finished_sending:
    self._done_sending_count[req_id] += 1

# After - safely handles None case
for req_id in kv_connector_metadata.finished_sending or []:
    self._done_sending_count[req_id] += 1

Always match return values with declared types:

# Before - implicitly returns None instead of declared tuple type
if not model_runner_output.kv_connector_metadata:
    return

# After - explicitly returns correct type
if not model_runner_output.kv_connector_metadata:
    return None, None

Process configurations consistently

Ensure that configuration data is processed consistently and correctly throughout the codebase. This includes:

  1. Use proper serialization methods for configuration dictionaries. Prefer json.dumps() over str() when converting dictionaries to strings:
# Wrong: Using str() produces invalid JSON
if isinstance(self.ep_config, dict):
    self.ep_config = EPConfig.from_cli(str(self.ep_config))

# Correct: Use json.dumps() for proper serialization
if isinstance(self.ep_config, dict):
    self.ep_config = EPConfig.from_cli(json.dumps(self.ep_config))
  1. Process configurations consistently in all code paths. Don’t skip processing steps conditionally unless absolutely necessary:
# Wrong: Only calling adapt_config_dict in one branch
if max_position_embeddings is None:
    max_position_embeddings = _maybe_retrieve_max_pos_from_hf()
    config_dict["max_position_embeddings"] = max_position_embeddings
    config = adapt_config_dict(config_dict)

# Correct: Call processing function in all cases
if max_position_embeddings is None:
    max_position_embeddings = _maybe_retrieve_max_pos_from_hf()
    config_dict["max_position_embeddings"] = max_position_embeddings
config = adapt_config_dict(config_dict)
  1. Handle nested configurations recursively when transforming configuration structures:
# Process nested dictionaries recursively
def _recursive_remap(elem: Any) -> Any:
    if not isinstance(elem, dict):
        return elem
    
    new_dict = {}
    for key, value in elem.items():
        new_key = config_mapping.get(key, key)
        new_dict[new_key] = _recursive_remap(value)
    return new_dict

  1. Verify that default values are sensible and documented correctly in docstrings, ensuring the implementation matches the documentation.

  2. Avoid changing default configurations that can break backward compatibility. When defaults must change, document the change and provide a migration path.

Ensure all documentation links are stable, consistent, and correctly targeted to improve navigation and long-term maintainability:

  1. Use stable references: Always link to stable release tags or specific commit hashes rather than volatile branches like master or main: ```diff
    • kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/refs/heads/master/ray-operator/config/samples/vllm/ray-service.vllm.yaml
    • kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/vX.Y.Z/ray-operator/config/samples/vllm/ray-service.vllm.yaml ```
  2. Maintain consistent file extensions: Use consistent file extensions (.md or .html) throughout documentation links, preferring .md for markdown source files: ```diff

  3. Verify relative paths: Ensure relative paths resolve correctly within the documentation structure by testing links in the built documentation.

  4. Use standardized link syntax: Follow project conventions for link formatting, such as using autolink syntax: ```diff
    • While ongoing efforts like #17419
    • While ongoing efforts like <gh-issue:17419> ```
  5. Target specific sections: When linking to pages with tabs or complex navigation, link directly to the specific section or tab to improve user experience.

Preserve API compatibility

When modifying API interfaces, parameters, or argument behavior, ensure backward compatibility is maintained to prevent breaking existing user code. If a parameter needs to be renamed or enhanced, continue supporting the old parameter name alongside the new one.

For example, when expanding API parameters to support new functionality:

# GOOD: Maintain backward compatibility
class ScoreRequest(OpenAIBaseModel):
    model: Optional[str] = None
    # Keep old parameter names to maintain compatibility
    text_1: Optional[Union[list[str], str]] = None
    text_2: Optional[Union[list[str], str]] = None
    # Add new parameters with enhanced functionality
    data_1: Optional[Union[list[str], str, ScoreMultiModalParam]] = None
    data_2: Optional[Union[list[str], str, ScoreMultiModalParam]] = None
    
    def __post_init__(self):
        # Use old parameters if new ones aren't provided
        if self.data_1 is None and self.text_1 is not None:
            self.data_1 = self.text_1
        if self.data_2 is None and self.text_2 is not None:
            self.data_2 = self.text_2

When removing or changing interface methods, first mark them as deprecated and provide clear migration guidance. For public APIs, follow a formal deprecation policy with adequate notice before removal. For internal APIs with multiple consumers, coordinate changes to ensure dependent systems aren’t broken.

Always document your API changes thoroughly, including:

  1. What changed and why
  2. How existing users should migrate their code
  3. When deprecated features will be removed (if applicable)

Validate algorithmic operations carefully

Mathematical and logical operations require careful validation to ensure correctness. Common issues include:

  1. Operator precedence in complex conditions
  2. Incorrect scaling/normalization calculations
  3. Type mismatches in comparisons
  4. Edge case handling

Example of a problematic implementation:

def _is_fp8_w8a8_sm90_or_sm100(weight_quant, input_quant):
    return (self._check_scheme_supported(90) or
            self._check_scheme_supported(100) and
            self._is_fp8_w8a8(weight_quant, input_quant))

Corrected version with proper operator precedence:

def _is_fp8_w8a8_sm90_or_sm100(weight_quant, input_quant):
    return ((self._check_scheme_supported(90) or
             self._check_scheme_supported(100)) and
            self._is_fp8_w8a8(weight_quant, input_quant))

Key validation practices:

explicit performance handling

When writing performance benchmarking and measurement code, always be explicit about all possible cases and organize related constants clearly. Avoid catch-all else clauses that could mask bugs or unexpected values in performance-critical code paths.

This approach prevents silent failures in benchmark parsing and makes performance code more maintainable by clearly documenting all expected scenarios.

Example of explicit case handling:

# Instead of a catch-all else clause
if unit == "TFLOPS":
    gflops = value * 1000
elif unit == "MFLOPS":
    gflops = value / 1000
elif unit == "GFLOPS":
    gflops = value
else:
    assert False  # Explicit failure for unexpected units

Example of organized performance constants:

# Clearly separate constants by benchmark type
LLAMA_BENCH_BOOL_PROPERTIES = ["embeddings", "cpu_strict", "use_mmap", "no_kv_offload", "flash_attn"]
TEST_BACKEND_OPS_BOOL_PROPERTIES = ["supported", "passed"]

Combine identical CSS

When multiple CSS selectors share identical styling properties, combine them using comma separation rather than duplicating the same rules. This reduces code duplication, improves maintainability, and makes stylesheets more concise.

Example - Instead of this:

.md-typeset .admonition.code,
.md-typeset details.code {
  border-color: #64dd17
}
.md-typeset .admonition.console,
.md-typeset details.console {
  border-color: #64dd17
}

Do this:

.md-typeset .admonition.code,
.md-typeset details.code,
.md-typeset .admonition.console,
.md-typeset details.console {
  border-color: #64dd17;
}

This practice applies to all CSS rules with identical properties and values. By consolidating duplicate rules, you’ll create more maintainable stylesheets that are easier to update and less prone to inconsistencies when making changes.

Optimize GPU execution

Ensure GPU code is optimized for both proper thread utilization and correct architecture dispatching:

  1. Maximize thread parallelism - Design CUDA kernels to fully utilize available threads. When appropriate, use multi-dimensional grid configurations to parallelize across all relevant dimensions of your problem.
// Instead of TODO comments like:
// TODO utilize more CUDA threads
// this will probably need some extra padding for warps

// Consider implementing a 2D grid approach:
dim3 block_dim(32, 8);  // Thread block dimensions
dim3 grid_dim((n + block_dim.x - 1) / block_dim.x, 
             (padded_m + block_dim.y - 1) / block_dim.y);
kernel<<<grid_dim, block_dim>>>(...);
  1. Implement proper architecture dispatching - When supporting multiple GPU architectures, combine compile-time preprocessor directives with runtime architecture detection:
// Instead of compile-time only checks:
#if defined ENABLE_CUTLASS_MOE_SM100 && ENABLE_CUTLASS_MOE_SM100
  if (version_num >= 100) {  // Use runtime version check
    cutlass_moe_mm_sm100(out_tensors, a_tensors, b_tensors, a_scales, b_scales,
                       expert_offsets, problem_sizes, a_strides, b_strides,
                       c_strides, per_act_token, per_out_ch);
    return;
  }
#endif

These approaches prevent performance bottlenecks and ensure code works correctly across different GPU hardware generations.

Secure before deployment

Remove or secure development-specific code before deploying to production environments. Development artifacts like debug print statements, weak validation methods, and development endpoints can introduce significant security vulnerabilities.

Specifically:

  1. Remove all debugging print statements that might leak sensitive request data, credentials, or tokens
  2. Implement thorough validation for security-critical functions, considering bypass techniques (e.g., symbolic links in command validation)
  3. Ensure development features have proper warnings and automated tests to verify they’re not accidentally enabled in production

Example of insecure code:

@router.post("/v1/responses")
async def create_responses(request: ResponsesRequest, raw_request: Request):
    print(request, raw_request)  # INSECURE: Leaks sensitive data to logs
    
def is_dangerous_cmd(cmd):
    cmd_base = os.path.basename(cmd)  # INSECURE: Can be bypassed with symlinks
    return cmd_base in COMMAND_BLACKLIST

Example of secure code:

@router.post("/v1/responses")
async def create_responses(request: ResponsesRequest, raw_request: Request):
    # Debug statements removed for production
    
def is_dangerous_cmd(cmd):
    # Resolve any symlinks to get the real path
    real_path = os.path.realpath(cmd)
    cmd_base = os.path.basename(real_path)
    return cmd_base in COMMAND_BLACKLIST
    
# When enabling development features:
if envs.VLLM_SERVER_DEV_MODE:
    logger.warning("SECURITY WARNING: Development endpoints are enabled!")
    # Accompany with tests to verify this warning is present

Remove unnecessary code elements

Keep code clean and maintainable by removing unnecessary elements that add complexity without value:

  1. Remove unused header includes that increase compile times
  2. Delete commented-out code blocks - use version control to track alternatives
  3. Eliminate redundant checks and validations
  4. Clean up duplicate definitions

Example of code to avoid:

#include <iostream>  // Unnecessary include

// Commented out alternative implementation
// void alternativeFunction() {
//   ...
// }

// Redundant checks
TORCH_CHECK(a_tensors.dtype() == torch::kFloat8_e4m3fn,
           "A tensors must be of type float8_e4m3fn.");
TORCH_CHECK(a_tensors.dtype() == torch::kFloat8_e4m3fn);  // Duplicate check

Instead, keep only the essential, active code elements and rely on version control for tracking alternatives.

Standardize environment versions

Ensure consistent versions of languages, tools, and dependencies across all CI/CD environments (GitHub Actions, CircleCI, local development, etc.). Inconsistent versions between environments make it difficult to reproduce issues and debug failures, as seen when using “Python 3.12 in GitHub & 3.9 elsewhere - makes it hard to figure out how to fix the broken unit tests.”

Define a single source of truth for version specifications and reference it across all pipeline configurations. This includes:

Example approach:

# .tool-versions or similar
python 3.12.0
node 18.17.0
docker 24.0.0

# Reference in GitHub Actions
- uses: actions/setup-python@v4
  with:
    python-version: '3.12.0'

# Reference in CircleCI  
- image: cimg/python:3.12.0

This prevents environment drift, reduces debugging complexity, and ensures consistent behavior across all stages of your CI/CD pipeline.

Match reference names

Ensure that filenames and paths referenced in scripts, commands, or configuration files exactly match the actual names of the files they reference. Name mismatches between references and actual files can cause commands to fail during execution.

For example, in a Justfile, the reference:

python {{vllm-directory}}benchmarks/benchmark_one_concurrent_req.py

should be changed to:

python {{vllm-directory}}benchmarks/benchmark_one_concurrent.py

to match the actual filename.

Always verify references against the actual filesystem structure before committing code to prevent runtime failures. This is especially important in build scripts, makefiles, and configuration files where a simple typo can break the entire workflow.

Check before using values

Always validate that values are not null, undefined, or empty before using them in operations that depend on their content. This prevents downstream errors and unexpected behavior when collections are empty or strings are null.

When working with collections, check if they contain elements before proceeding with operations that assume non-empty data:

# Before: Risk of empty collection causing issues downstream
for item in some_list:
    keys.append(item.key)
    memory_objs.append(item.memory_obj)
# subsequent logic assumes memory_objs is not empty

# After: Check collection before proceeding
if memory_objs:  # handles both None and empty cases
    # execute subsequent logic

For string values, use concise truthiness checks that handle both null and empty cases:

# Before: Only checks for null
if chunk_message is not None:
    server_message.append(chunk_message)

# After: Handles both null and empty
if chunk_message:  # handles None, "", and other falsy values
    server_message.append(chunk_message)

This pattern prevents runtime errors and makes code more robust by ensuring operations only proceed when they have valid data to work with.

Enable callback chaining

When designing APIs that accept callbacks or configuration functions, always return the previous callback/configuration to enable chaining and composition. This allows multiple components to layer their functionality without losing existing behavior.

For callback setters, return the previous callback so callers can chain multiple handlers:

// Good: Returns previous callback for chaining
GGML_API ggml_abort_callback_t ggml_set_abort_callback(ggml_abort_callback_t callback);

// Usage allows chaining:
ggml_abort_callback_t prev_callback = ggml_set_abort_callback(my_callback);
// Can later restore or chain: my_callback can call prev_callback

For initialization functions, prefer passing callbacks as parameters rather than extending the API with many specific functions:

// Preferred: Callback-based approach
llama_sampler_init_grammar(grammar_str, grammar_root, is_empty_callback, accept_str_callback);

// Avoid: Function proliferation
llama_sampler_is_grammar_empty(gsmpl);  // Separate function for each operation

This approach promotes API composability, reduces function proliferation, and enables multiple components to work together seamlessly.

AI model persistence

When containerizing AI applications, ensure proper model persistence by mounting volumes to the default cache locations used by AI frameworks and setting appropriate environment variables. This prevents redundant downloads of large models and improves development efficiency.

Example for Hugging Face models in Docker:

volumes:
  - models:/root/.cache/huggingface
environment:
  HF_HOME: /root/.cache/huggingface

Follow consistent naming patterns

Maintain consistent naming conventions across all aspects of the codebase, including file structures, folder organization, environment variables, and identifiers. This ensures predictability and reduces cognitive load for developers.

Key principles:

Example:

# Good: Consistent with framework patterns
NVIDIA_API_BASE = os.getenv("NVIDIA_API_BASE") or os.getenv("NVIDIA_BASE_URL")  # Support both

# Good: File structure matches provider
# litellm/llms/nvidia/chat/transformation.py
# tests/litellm/llms/nvidia/chat/test_transformation.py

# Good: Concise enum naming
class SupportedGuardrailIntegrations(Enum):
    PRISMA_AIRS = "prisma_airs"  # Not "panw_prisma_airs"

optimize algorithmic complexity

When implementing or refactoring algorithms, prioritize efficient data structures and computational approaches over simpler but slower alternatives. This is especially critical in performance-sensitive code where operations may be called frequently.

Key considerations:

Example from the codebase:

// Instead of linear search through nodes:
int j = cgraph->n_nodes - 1;
for (; j >= 0; --j) {
    if (s == cgraph->nodes[j]) {
        break;
    }
}

// Use hash table lookup:
// Allocate use_counts array with hash_size and use ggml_hash_find
// to find the slot for the node, avoiding O(n) search complexity

This principle applies broadly - from choosing vectorized implementations over scalar ones, to using hash tables instead of linear searches, to preallocating command buffers rather than creating them on-demand. The goal is to minimize computational overhead, especially in hot code paths where these optimizations compound significantly.

Protect shared state

When mutable state (like dictionaries, lists, or counters) is accessed from multiple threads or concurrent tasks, use appropriate synchronization mechanisms to prevent race conditions. Unprotected access to shared state can lead to data corruption, inconsistent behavior, and hard-to-debug issues.

For example, wrap dictionary operations with locks:

# Unsafe - race condition if called concurrently:
self._reqs_to_send[req_id] = some_value
del self._reqs_to_send[req_id]

# Safe with lock protection:
self._reqs_to_send_lock = threading.Lock()  # Initialize in constructor

# Later in code:
with self._reqs_to_send_lock:
    self._reqs_to_send[req_id] = some_value
    del self._reqs_to_send[req_id]

In asynchronous code, also be careful with shared variables that might be modified by interleaved tasks. Instead of relying on shared context, pass necessary context through the call stack:

# Unsafe with shared state in async code:
self._set_tokenizer(tokenizer)  # Could be modified by another task before use

# Safe - pass explicitly through call stack:
result = await self._process_with_tokenizer(tokenizer, input_text)

Follow documentation standards

Ensure all documentation follows established patterns and standards for components, formatting, and code examples. This includes using proper imports for documentation components (like Image imports instead of markdown syntax) and maintaining consistent formatting for code samples with standardized attributes.

For images, use the proper component import:

import Image from '@theme/IdealImage';
// Use: <Image src="path/to/image.png" alt="Description" />
// Instead of: ![Description](path/to/image.png)

For code examples, include standard formatting attributes:

# Code samples should have showLineNumbers and title

Always reference existing documentation examples as templates to maintain consistency across the codebase. This ensures a professional appearance and better user experience while preventing technical rendering issues.

Optimize tensor memory operations

When working with PyTorch tensors, use memory-efficient operations that avoid unnecessary copies. Specify memory formats directly during tensor creation instead of applying operations like .t().contiguous(). For C++/CUDA kernel interfacing, use .data_ptr() instead of .view(dtype) to ensure safe memory access and maintain compatibility with future PyTorch versions.

# Inefficient approach with unnecessary copy:
tensor = torch.empty((n, m), device="cuda", dtype=torch.bfloat16).t().contiguous()

# Efficient approach:
tensor = torch.empty((n, m), device="cuda", dtype=torch.bfloat16, 
                     memory_format=torch.contiguous_format).t()

# Unsafe C++ kernel interfacing:
cutlass_function(w1_scale.view(torch.int32), w1.view(torch.long))

# Safe approach with explicit pointer access:
cutlass_function(w1_scale.data_ptr(), w1.data_ptr())

optimize CI/CD pipelines

CI/CD pipelines should be optimized for both performance and maintainability through strategic caching, script consolidation, and compilation optimization. This involves three key practices:

  1. Implement proper caching strategies: Use comprehensive cache keys that include build identifiers and dependency checksums to ensure cache invalidation works correctly. For environment caching, include unique build IDs to prevent stale cache reuse.

  2. Consolidate related scripts: Minimize the number of separate bash scripts by merging related functionality. This reduces maintenance overhead and simplifies pipeline complexity as the project grows.

  3. Optimize compilation performance: Implement compilation caching (like ccache) for projects with significant build times to improve pipeline speed.

Example cache configuration:

plugins:
  - cache#v1.5.2:
      key: "venv-{{ BUILDKITE_BUILD_ID }}-{{ checksum \"requirements/common.txt\" }}-{{ checksum \"requirements/test.txt\" }}"
      path: ".venv"
      save: "pipeline"
      force: true

These optimizations ensure CI/CD pipelines remain fast, reliable, and maintainable as projects scale.

consistent localhost addressing

Use 127.0.0.1 instead of ‘localhost’ for network requests and implement broad localhost detection rather than specific port-based checks. This ensures consistent behavior across different environments and avoids issues with DNS resolution or host file configurations.

For network requests, prefer the loopback IP address:

// Preferred
await fetch(`http://127.0.0.1:${cortexJsPort}/v1/system`, {

// Avoid
await fetch('http://localhost:1337/v1/system', {

For localhost detection, check the hostname broadly rather than enumerating specific ports:

const isLocalHost = urlObj.hostname === 'localhost' ||
                   urlObj.hostname === '127.0.0.1' ||
                   urlObj.hostname === '0.0.0.0'

This approach is more reliable and covers all local providers rather than maintaining a list of specific ports that may become outdated as new local AI providers emerge.

Ensure configuration consistency

Maintain consistency across configuration variables to prevent setup confusion and deployment issues. Avoid duplicate or conflicting environment variables, ensure related configuration values align, and use appropriate defaults for the deployment context.

Key practices:

Example of inconsistent configuration to avoid:

# Bad: Misaligned passwords cause login issues
OPENSEARCH_PASSWORD=admin
OPENSEARCH_INITIAL_ADMIN_PASSWORD=Qazwsxedc!@#123

# Bad: Duplicate variables for same purpose
PLUGIN_DAEMON_KEY=lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi
PLUGIN_API_KEY=lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi

This prevents initial setup difficulties and reduces confusion during development and deployment.

Maintain consistent organization

Ensure proper file organization and consistent import conventions throughout the codebase. Assets should be placed in designated folders, utility functions should be moved to appropriate shared locations, and import paths should follow established patterns.

Key practices:

Example of proper organization:

// Instead of: bg-[url('~@/app/components/tools/add-tool-modal/empty.png')]
// Use: bg-[url('/assets/empty.png')]

// Instead of: import useAvailableVarList from '../../_base/hooks/use-available-var-list'
// Use: import useAvailableVarList from '@/app/components/workflow/nodes/_base/hooks/use-available-var-list'

// Move utility functions from component files to shared locations:
// From: web/app/components/base/chat/embedded-chatbot/hooks.tsx
// To: web/app/components/base/chat/utils/index.ts

This approach improves code maintainability, makes dependencies clearer, and ensures consistent project structure across the team.

Use environment variables

Prefer environment variables over hardcoded paths and configuration values in build scripts and configuration files. This approach improves portability, flexibility, and maintainability by allowing different environments to specify their own paths without modifying the codebase.

Instead of hardcoding paths like:

set(oneCCL_DIR "/opt/intel/oneapi/ccl/latest/lib/cmake/oneCCL")
set(MPI_INCLUDE_PATH "/opt/intel/oneapi/mpi/latest/include")

Use environment variables:

set(oneCCL_DIR "$ENV{ONEAPI_ROOT}/ccl/latest/lib/cmake/oneCCL")
set(MPI_INCLUDE_PATH "$ENV{ONEAPI_ROOT}/mpi/latest/include")

For debug options and feature flags, check environment variables instead of adding compile-time options. This allows runtime configuration without rebuilding and follows established patterns like GGML_SCHED_DEBUG. Environment variables make configurations more flexible across different deployment environments and reduce the need for environment-specific build modifications.

Maintain comprehensive error handling

Always preserve and implement thorough error handling that includes proper resource cleanup, meaningful logging, and appropriate return values. Don’t remove existing error handling during code refactoring or simplification.

Key practices:

  1. Preserve error handling during refactoring - When simplifying code, maintain existing try-catch blocks and error recovery logic
  2. Clean up resources on failure - Always free allocated memory, close connections, or reset state when operations fail
  3. Choose appropriate error responses - Return sensible defaults (like False) instead of throwing exceptions when callers can handle the failure gracefully
  4. Log exceptions with context - Include meaningful error messages and context when catching exceptions

Example of proper error handling preservation:

# Good - maintains error handling even when simplifying
try:
    return self.connection.exists_sync(key)
except Exception as e:
    with self.lock:
        self.connection = None
        self.failure_time = time.time()
    logger.warning(f"Remote connection failed in contains: {e}")
    logger.warning("Returning False")
    return False

# Bad - removes error handling during simplification
return self.connection.exists_sync(key)

This approach prevents silent failures, resource leaks, and provides better debugging information when issues occur.

Use OAuth Authorization Header

When authenticating requests with bearer tokens or OAuth-derived credentials, transmit them using the standards-based Authorization request header as required by the relevant OAuth specification. Avoid sending secrets via custom headers like api_key, and ensure documentation/examples match the OAuth flow requirements.

Example (adapted to a typical async client):

# Preferred: OAuth-compliant Authorization header
headers = {
    "Authorization": "Bearer YOUR_ACCESS_TOKEN"  # or the OAuth-defined scheme/value
}

async with sse_client(
    "http://localhost:9382/sse",
    headers=headers,
) as streams:
    # Rest of your code...

Apply this to code, SDK wrappers, and docs: if the auth method is OAuth 2.1 (or similar), use Authorization and the required formatting/scheme, rather than custom credential headers.

semantic configuration validation

When designing configuration systems, prefer semantic parameters over generic key-value approaches for common features, and implement runtime validation to detect incompatible configuration combinations.

For common functionality, use dedicated semantic variables rather than generic context variables. This maintains a unified API surface and prevents users from needing to learn template-specific idiosyncrasies. When features are incompatible, validate configurations at runtime and provide clear error messages.

Example:

// Good: Semantic parameter for common feature
struct config {
    bool enable_thinking = true;
    bool assistant_prefill = false;
};

// Validation with clear error message
if ((!inputs.enable_thinking) || inputs.chat_template_kwargs.find("enable_thinking") != inputs.chat_template_kwargs.end()) {
    throw std::runtime_error("Assistant response prefill is incompatible with enable_thinking.");
}

This approach reduces complexity for users while maintaining flexibility for advanced use cases, and prevents runtime failures through proactive validation.

background task coordination

When working with background tasks in async applications, ensure proper task lifecycle management and coordination to prevent memory leaks, race conditions, and resource conflicts.

Key practices:

  1. Use asyncio.create_task for fire-and-forget operations to avoid blocking the main execution flow
  2. Implement proper task cleanup by having tasks remove themselves from tracking collections when complete
  3. Use controlled loop conditions instead of while True: to allow graceful termination
  4. Prevent overlapping executions by checking if a task is already running before starting a new one
  5. Always release resources in finally blocks to ensure cleanup even when exceptions occur

Example implementation:

# Global task tracking with automatic cleanup
BACKGROUND_TASKS = set()

async def start_background_operation():
    # Use create_task for non-blocking execution
    task = asyncio.create_task(background_health_check())
    # Add to tracking set and auto-remove when done
    BACKGROUND_TASKS.add(task)
    task.add_done_callback(BACKGROUND_TASKS.discard)

async def background_health_check():
    # Use controlled condition instead of while True
    while use_background_health_checks:
        try:
            # Perform work
            await perform_health_check()
        finally:
            # Always cleanup resources
            await cleanup_resources()

This approach prevents memory leaks from orphaned tasks, allows controlled shutdown of background operations, and ensures resources are properly released even when errors occur.

eliminate unnecessary code

Remove redundant operations, unnecessary code blocks, and verbose constructs to improve code clarity and maintainability. Focus on eliminating dead code, avoiding redundant loops, removing unnecessary conditional checks, and using more concise expressions where appropriate.

Key practices:

Example of improvement:

# Before: Unnecessary loop variable
CSS_CLASS_TO_ELEMENT_TYPE_MAP = {
    element_type().css_class_name: element_type
    for element_type in ALL_ONTOLOGY_ELEMENT_TYPES
    for tag in element_type().allowed_tags  # 'tag' is unused
}

# After: Remove unused loop variable
CSS_CLASS_TO_ELEMENT_TYPE_MAP = {
    element_type().css_class_name: element_type 
    for element_type in ALL_ONTOLOGY_ELEMENT_TYPES
}

# Before: Verbose list building
lines = []
for each in obj:
    lines.append(json.dumps(each, **kwargs))
return "\n".join(lines)

# After: Concise generator expression
return "\n".join(json.dumps(each, **kwargs) for each in obj)

This approach reduces cognitive load, improves performance, and makes code more maintainable by eliminating distractions and focusing on essential logic.

prefer OpenAI compatibility

When integrating new LLM providers or AI services, prioritize OpenAI-compatible patterns and reuse existing utilities rather than creating separate implementations. This approach reduces code duplication, leverages battle-tested functionality, and maintains consistency across the codebase.

Key practices:

Example of preferred approach:

# Instead of creating a separate provider block:
elif custom_llm_provider == "new_provider":
    # Custom implementation...

# Prefer adding to openai_compatible_providers:
openai_compatible_providers = [..., "new_provider"]

# And use existing handlers:
response = base_llm_http_handler.completion(
    model=model,
    messages=messages,
    custom_llm_provider=custom_llm_provider,
    api_base=api_base,
    api_key=api_key,
    # ...
)

This pattern ensures new AI integrations benefit from existing error handling, parameter validation, streaming support, and other features while minimizing maintenance overhead.

Descriptive balanced naming

Choose identifier names that clearly communicate their purpose while maintaining an appropriate balance between brevity and clarity.

Guidelines:

  1. Use descriptive names that reflect types and purpose: 
    // Poor: Using abbreviated or generic names
cap, err := GetModel(n.String())
   
// Good: Name reflects the actual type
model, err := GetModel(n.String())
  2. Avoid abbreviations unless universally understood: 
    // Poor: Unnecessary abbreviation
type RopeOpts struct {}
   
// Good: Full word for clarity
type RopeOptions struct {}
  3. Balance verbosity - aim for 1-2 word components: 
    // Too verbose
var templateToolPrefix, templateToolPrefixFound := ToolPrefix(model.Template.Template)
   
// Better balance
prefix, found := toolPrefix(model.Template.Template)
  4. Be consistent with naming patterns: 
    // Inconsistent with other handlers
func (s *Server) version(w http.ResponseWriter, r *http.Request)
   
// Consistent with other handler naming
func (s *Server) VersionHandler(c *gin.Context)

  5. Avoid single-letter variables outside of short-lived scopes like simple loops.

  6. In environment variables and constants, spell words completely: 
    // Poor: Ambiguous abbreviation
OLLAMA_GPU_DEVS
   
// Good: Clear, fully spelled out
OLLAMA_GPU_DEVICES

When choosing names, prioritize clarity for future readers over saving a few keystrokes now.

Follow GoDoc conventions

Document code following Go’s official documentation style guide (https://tip.golang.org/doc/comment). All exported functions, types, interfaces, and struct fields should have descriptive comments.

Function documentation should start with the function name and a verb describing what it does:

// addContent returns the thinking content and the normal content that should be
// immediately sent to the user. It will internally buffer if it needs to see
// more content to disambiguate
func (s *thinkingParser) addContent(content string) (string, string) {
    // Implementation
}

For struct fields, place comments on the line above each field:

type ErrorResponse struct {
    // Err is the error from the server. It helps with debugging the code-path
    Err  string `json:"error"`

    // Hint is a user-friendly message about what went wrong, with suggested troubleshooting
    Hint string `json:"hint"`
}

Document interfaces thoroughly, especially when they serve as implementation guides for different backends:

// ScaledDotProductAttention defines operations for attention computation
// that must be implemented by backend providers
type ScaledDotProductAttention interface {
    // Methods...
}

Include parameter descriptions when their purpose isn’t obvious:

// parseJSONToolCalls attempts to parse a JSON string into a slice of ToolCalls.
// It first checks for balanced braces before attempting to parse.
// Parameters:
//   - s: The JSON string to parse
//   - name: The field name representing the tool name in the JSON
//   - arguments: The field name representing the tool arguments in the JSON
// Returns:
//   - []api.ToolCall: The parsed tool calls if successful
//   - error: ErrAccumulateMore if braces unbalanced, ErrInvalidToolCall if invalid, or nil if successful
func parseJSONToolCalls(s string, name, arguments string) ([]api.ToolCall, error) {

Always explain non-obvious behaviors or design decisions:

// topK limits the number of tokens considered to the k highest logits.
// The slice is passed by pointer because its length will be modified to k.
func topK(ts *[]token, k int) {
    // Implementation
}

Use idiomatic Go flow

Follow Go’s idiomatic control flow patterns to improve code readability and maintainability. Key practices include:

  1. Prefer early returns over else blocks
  2. Return errors on the right in function signatures
  3. Use if statements instead of switches for single cases

Example - Before:

func inferThinkingOption(caps *[]model.Capability, runOpts *runOptions, explicitlySetByUser bool) (error, *bool) {
    if condition {
        // success case
    } else {
        return errors.New("error"), nil
    }
}

After:

func inferThinkingOption(caps *[]model.Capability, runOpts *runOptions, explicitlySetByUser bool) (*bool, error) {
    if !condition {
        return nil, errors.New("error")
    }
    // success case
}

This approach:

Ensure metadata consistency

Always persist metadata alongside data and implement proper checking mechanisms to prevent duplicates and ensure consistency in distributed storage scenarios. When writing data to storage backends, metadata should be written atomically with the data itself. Additionally, implement separate checking logic for existence vs. active operations to prevent race conditions and duplicate storage.

For distributed systems, implement fallback checks when local caches miss, as other nodes may have added data not present in the local hot cache. Use separate read/write clients when possible to optimize for different access patterns and enable role-based separation.

Example implementation pattern:

# Always write metadata with data
async def store_data(self, key, data):
    metadata = pack_metadata(data.shape, data.dtype, data.nbytes)
    # Write both atomically
    await self._write_data_and_metadata(key, data, metadata)
    # Update local cache
    with self.hot_lock:
        self.hot_cache[key] = metadata

# Separate existence check from active operations check
def contains(self, key: CacheEngineKey) -> bool:
    # Check local cache first
    with self.hot_lock:
        if key in self.hot_cache:
            return True
    # Fallback to filesystem check for distributed scenarios
    return self._try_to_read_metadata(key)

def store(self, key: CacheEngineKey, data):
    # Check active operations to prevent duplicates
    with self.put_lock:
        if key in self.put_tasks:
            return  # Already being stored
        self.put_tasks.add(key)
    # Proceed with storage...

This pattern ensures data integrity, prevents duplicate operations, and maintains consistency across distributed storage backends.

Complete null checks

Always perform thorough null checks before dereferencing pointers, and ensure all operations on potentially null objects are contained within the null check block. This prevents null pointer dereferences and undefined behavior.

For example, when freeing resources, place all cleanup operations inside the null check:

// Bad - operations outside null check
if (g != nullptr) {
    delete g->vocab;
    // Other operations...
}
g->someOperation(); // Potential null dereference!

// Good - all operations inside null check
if (g != nullptr) {
    delete g->vocab;
    g->someOperation(); 
    // All operations safely inside the check
}

When working with references or pointers that may become invalid, consider returning objects by value instead of by reference to prevent dangling references. This applies especially when the object lifetime cannot be guaranteed:

// Risky - caller might use reference after source is deallocated
const std::string & token_to_piece(uint32_t token);

// Safer - returns a copy that remains valid
const std::string token_to_piece(uint32_t token);

Optimize memory allocations

Be strategic about memory allocations to improve performance in your C++ code. Consider two key optimization patterns:

  1. Pre-reserve container capacity: When using containers like vectors that will grow to a predictable size, use reserve() with a reasonable initial capacity to reduce allocation overhead. This prevents multiple small reallocations as the container grows.
// Before - May cause multiple reallocations during population
vec_rules[i].resize(n_rules);  

// After - Pre-allocate with reasonable capacity
vec_rules[i].reserve(16);  // Start with reasonable capacity
  1. Avoid unnecessary preallocation: When designing functions that return variable-sized data, prefer return values over output parameters when the required size is difficult to predict in advance. This eliminates the need for callers to pre-allocate buffers of the correct size.
// Before - Requires caller to pre-allocate buffer of correct size
int schema_to_grammar(const char *json_schema, char *grammar, size_t max_len);

// After - Return value approach avoids pre-allocation issues
std::string schema_to_grammar(const char *json_schema);

Both strategies help minimize memory churn, reduce fragmentation, and improve overall application performance, particularly in performance-critical code paths.

Document configuration choices

Configuration scripts and files should clearly document non-obvious choices, especially numeric IDs, special values, and environment-specific commands. Include comments explaining the purpose and reasoning behind configuration decisions to help future maintainers understand why specific values or approaches were chosen.

When using numeric user/group IDs, always document which user they represent:

# Set ownership to proxy user (UID 13) required for Squid operation
chown -R 13:13 /var/log/squid

Additionally, verify that configuration commands work reliably across environments. Avoid commands with known compatibility issues and prefer more explicit alternatives:

# Use explicit directory change instead of -C flag due to poetry issue #9415
cd api && poetry install

This practice prevents configuration failures and reduces debugging time when scripts don’t work as expected in different environments.

Avoid hardcoded configuration values

Configuration files and scripts should use placeholders, environment variables, or parameterized values instead of hardcoding specific configuration values. Hardcoded values make configurations inflexible and difficult to adapt across different environments or deployments.

Use placeholder formats that clearly indicate what type of value should be provided:

# Good - uses placeholder format
IMAGE=<IMAGE_NAME>:<TAG>

# Avoid - hardcoded specific values
IMAGE=lmcache/vllm-openai:2025-05-17-v1

This approach ensures configurations remain flexible and can be easily customized for different environments, deployments, or use cases without requiring code changes.

Remove redundant AI commands

Eliminate duplicate or redundant commands when setting up AI infrastructure, model serving, or ML library installations. Redundant commands create confusion, increase maintenance overhead, and can lead to unexpected behavior in AI workflows.

Common scenarios to watch for:

Example of redundancy removal:

# Before: Redundant vllm installation
uv pip install vllm
uv pip install --upgrade vllm

# After: Single command achieves the same result
uv pip install --upgrade vllm

# Before: Redundant Docker entrypoint
docker run --entrypoint "/usr/local/bin/vllm" $IMAGE serve $MODEL_NAME

# After: Use existing entrypoint (when ENTRYPOINT is already ["vllm", "serve"])
docker run $IMAGE $MODEL_NAME

Always review AI setup scripts and configurations to identify and eliminate unnecessary command duplication, ensuring cleaner and more maintainable infrastructure code.

Clear recoverable error messages

Error messages should be clear, actionable, and indicate whether recovery is possible. When designing error handling:

  1. Use descriptive messages that explain what went wrong and how to fix it
  2. Indicate if the error is expected/recoverable
  3. Provide appropriate fallbacks when safe
  4. Include relevant context in error messages

Example:

// Instead of
if err != nil {
    return fmt.Errorf("puller failed: %w", err)
}

// Do this
if err != nil {
    if errors.Is(err, ErrModelNotFound) {
        // Clear message with actionable hint
        return fmt.Errorf("model %q not found - check available models at: https://ollama.com/models", name)
    }
    // Indicate if retry may help
    if isTemporary(err) {
        return fmt.Errorf("temporary error fetching model %q, retry may resolve: %w", name, err)
    }
    return fmt.Errorf("failed to pull model %q: %w", name, err)
}

// Consider fallbacks for non-critical errors
if req.Options == nil {
    // Use safe defaults instead of error
    req.Options = api.DefaultOptions()
}

This approach helps users understand and recover from errors while maintaining system stability through appropriate fallbacks.

Prioritize naming clarity

Choose names that are immediately self-explanatory and unambiguous, even if they are more verbose. Names should convey their full meaning and context without requiring additional documentation or guesswork.

When naming variables, configuration settings, or API patterns, prioritize precision over brevity. A longer, descriptive name is preferable to a shorter, ambiguous one.

Examples:

The goal is that any developer reading the code should immediately understand what a name refers to without needing to consult documentation or make assumptions about its scope or purpose.

Guard against nil

Always check for nil values and successful type assertions before accessing or dereferencing objects, especially when working with interfaces, type conversions, or functions that may return nil values.

For type assertions, use the two-return form to check if the assertion succeeded:

// Good: Check if type assertion succeeded before using
if tp, ok := s.model.(model.TextProcessor); ok {
  vocab = tp.Vocabulary()
}

// Bad: May panic if assertion fails
vocab = s.model.(model.TextProcessor).Vocabulary()

For values that may be nil, verify before dereferencing:

// Good: Check if grammar is nil after initialization
grammar := llama.InitGrammarChain(grammarStr)
if grammar == nil {
  return nil, errors.New("failed to initialize grammar")
}

// Bad: May cause nil pointer dereference
grammar := llama.InitGrammarChain(grammarStr)
grammar.AddSymbol(s, uint32(id))  // Panic if grammar is nil

Consider explicitly initializing structures that may be nil:

// Good: Initialize and check configuration
if c.config == nil {
  var config ml.CacheConfig
  if cc, ok := backend.(ml.BackendCacheConfig); ok {
    // Do something with config
  }
}

These practices prevent runtime panics and make your code more robust and maintainable.

Design extensible permission systems

When implementing authorization features, design permission systems that can accommodate future security requirements rather than adding permissions directly to existing tables. Create dedicated permission tables with proper relationships to entity tables to ensure authorization can be properly enforced.

Instead of adding permission arrays directly to existing tables:

model LiteLLM_TeamTable {
    // ... other fields
    mcp_servers String[] @default([])  // Avoid this approach
}

Create extensible permission tables with proper relationships:

model LiteLLM_ObjectPermissionTable {
  permission_id  String         @id @default(uuid())
  mcp_servers    String[]       @default([])
  // Future permissions can be added here
}

model LiteLLM_VerificationToken {
  // ... other fields
  permission_id     String?
  object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [permission_id], references: [permission_id])
}

This approach ensures that permission enforcement logic has a clear structure to work with and can be extended as new authorization requirements emerge without requiring schema migrations to core entity tables.

Reuse buffers strategically

For frequently called methods or hot code paths, reuse allocations instead of creating new buffers on each call. This significantly reduces GC pressure and improves performance in memory-intensive operations.

To implement this pattern:

  1. Add reusable buffer fields to structs that handle data processing
  2. Check the capacity of existing buffers before allocating new ones
  3. Resize only when necessary based on input requirements

Example implementation:

// Before: Allocating a new buffer on each call
func (s *weighted) Sample(logits []float32) (int32, error) {
    tokens := make([]tokenInfo, len(logits))
    // Process logits...
}

// After: Reusing a buffer across calls
type weighted struct {
    rng        *rand.Rand
    transforms []transform
    buf        []tokenInfo // reusable buffer field
}

func (s *weighted) Sample(logits []float32) (int32, error) {
    // Resize only when necessary
    if cap(s.buf) < len(logits) {
        s.buf = make([]tokenInfo, len(logits))
    }
    tokens := s.buf[:len(logits)]
    // Process logits...
}

This pattern is especially important for functions in critical processing paths like sampling, data copying, or quantization operations. For data copying operations that don’t require transformation, consider using io.Copy or similar approaches that avoid loading everything into memory.

Comprehensive test coverage

Test functions should provide comprehensive coverage of both expected and edge cases. Include tests for:

  1. Boundary conditions: Test zero, minimum, maximum values, and other special cases. 
    func TestTemperature(t *testing.T) {
  // Test normal case
  logits, err := Temperature(0.5).Apply([]float64{-3, -2, -1, 0, 1, 2, 4})
  if err != nil {
    t.Error(err)
  }
     
  // Test boundary case (zero)
  logits, err = Temperature(0).Apply([]float64{-3, -2, -1, 0, 1, 2, 4})
  if err != nil {
    t.Error(err)
  }
}
  2. Varied input patterns: Test unsorted, sorted, empty, and different input formats. 
    // Test with unsorted inputs
logits, err := Temperature(0.5).Apply([]float64{2, -1, 4, -3, 1, -2, 0})
  3. Multiple levels of granularity: Test intermediate steps as well as end-to-end functionality. 
    // Don't just test ParseToolCalls, but also intermediate steps
func TestFindTemplatePrefix(t *testing.T) { /* ... */ }
func TestExtractToolTokens(t *testing.T) { /* ... */ }
func TestParseToolCalls(t *testing.T) { /* ... */ }
  4. Both valid and invalid cases: Each test function should validate both expected successful cases and error conditions. 
    tests := []struct {
  name    string
  input   string
  valid   bool
  want    Result
  wantErr error
}{
  {
    name:  "valid input",
    input: "valid data",
    valid: true,
    want:  expectedResult,
  },
  {
    name:    "invalid input",
    input:   "invalid data",
    valid:   false,
    wantErr: ErrInvalidData,
  },
}

Thorough test coverage helps catch edge cases and ensures code robustness, particularly for functionality that may be used across different parts of the system.

Document synchronization intent

Always clearly document the purpose and scope of synchronization primitives (mutexes, read-write locks) by:

  1. Placing them directly adjacent to the data they protect
  2. Adding comments explaining what’s being protected when not obvious
  3. Making locking requirements explicit in method/function documentation

This practice prevents race conditions and helps future developers understand the concurrent access patterns in your code.

Bad example:

var mu sync.Mutex
// ... many lines later or in another function ...
func doSomething() {
    mu.Lock()
    // use some shared data
    mu.Unlock()
}

Good example:

// UserCache guards access to the user data map
var (
    userCache map[string]UserData
    userCacheMu sync.RWMutex  // guards userCache
)

// GetUser returns user data, safe for concurrent access
// Locking: Acquires userCacheMu read lock
func GetUser(id string) (UserData, bool) {
    userCacheMu.RLock()
    defer userCacheMu.RUnlock()
    data, found := userCache[id]
    return data, found
}

When synchronizing access to maps or slices, consider using a dedicated synchronized wrapper type that enforces the locking protocol rather than managing locks separately. For synchronized data structures, ensure methods that return internal state (like Maps) return copies rather than references to prevent race conditions from external modifications.

Optimize AI implementation patterns

When implementing AI systems, prioritize established patterns and optimizations rather than creating new implementations for existing problems. This applies to all aspects of AI development:

  1. For model operations, prefer reusing existing functions over implementing new ones when they serve the same purpose. For example, use ggml_scale(ctx, cur, -1) instead of implementing a separate ggml_neg operation, as it achieves the same result with equivalent performance and memory usage.

  2. When designing data structures for AI models (like vocabulary handling), ensure they account for all possible variations and edge cases. For instance, when handling end-of-generation tokens, remember there may be multiple token types (EOS, EOT) rather than assuming a single one.

  3. Maintain precision in documentation of AI tools to ensure clarity for users with different experience levels.

// Instead of implementing new operations:
// ggml_tensor * result = ggml_neg(ctx, tensor);

// Prefer reusing existing operations:
ggml_tensor * result = ggml_scale(ctx, tensor, -1);

// Instead of:
struct vocab {
    uint32_t eog_token; // Single end token
};

// Prefer:
struct vocab {
    std::vector<uint32_t> eog_tokens; // Multiple possible end tokens (EOS, EOT)
};

These optimizations improve code maintainability, reduce potential bugs, and often lead to better performance in AI systems that already have high computational demands.

Balance CI/CD trade-offs

When configuring CI/CD pipelines, balance comprehensive coverage and security with practical constraints like resource usage, tool compatibility, and project context. Avoid blindly applying best practices without considering their impact on functionality and efficiency.

Consider these trade-offs:

Example from a GitHub workflow:

strategy:
  matrix:
    python-version:
      - "3.12"  # Primary version for most users
      # - "3.11"  # Commented out to save CI resources

Document your reasoning when making these trade-offs so future maintainers understand the decisions and can revisit them as constraints change.

Separate AI instructions

Always separate AI model instructions from user input to prevent prompt injection vulnerabilities and maintain clear architectural boundaries. Use system messages for AI instructions and user messages for actual user content, rather than embedding user input directly into instruction templates.

This approach provides two key benefits:

  1. Security: Prevents prompt injection attacks where malicious user input could override or manipulate the AI’s instructions
  2. Clarity: Maintains clear separation of concerns between what the AI should do (instructions) and what data it should process (user input)

Example of the recommended approach:

# Good: Separate system instructions from user input
prompt_messages = [
    SystemPromptMessage(content="Generate JSON output following this schema: {...}"),
    UserPromptMessage(content=user_provided_content)
]

# Avoid: Embedding user input in instructions
prompt_template = "Generate JSON for: {user_input}"  # Vulnerable to injection

Be particularly careful when implementing structured output features, as system prompts may be modified or discarded depending on the model’s native capabilities. Always validate that the intended instruction-user separation is preserved throughout the processing pipeline.

simplify complex logic

Improve code readability by reducing complexity and nesting through early returns, extracting complex logic into dedicated functions, and using clear parameter naming.

Key practices:

  1. Use early returns to reduce nested if statements and simplify logic flow
  2. Extract complex nested logic into dedicated local functions with descriptive names
  3. Use named parameters instead of positional boolean parameters for better readability
  4. Simplify conditional expressions by using early continue/break statements when appropriate

Examples:

# Instead of deeply nested conditions:
if condition1:
    if condition2:
        if condition3:
            # complex logic here
            pass

# Use early returns:
if not condition1:
    return self
if not condition2:
    return self
if not condition3:
    return self
# complex logic here

# Instead of positional boolean parameters:
vn.ask(prompt, False, True, False, allow_llm_to_see_data)

# Use named parameters:
vn.ask(prompt, print_results=False, auto_train=True, visualize=False, allow_llm_to_see_data=allow_llm_to_see_data)

# Instead of complex nested logic in main function:
if self.node_data.structured_output_enabled and self.node_data.structured_output:
    # 20+ lines of complex logic

# Extract to dedicated function:
def process_structured_output(self):
    if not self.node_data.structured_output_enabled:
        return
    if not self.node_data.structured_output:
        return
    # complex logic here

# Then call it:
process_structured_output()

This approach makes code more maintainable, easier to test, and significantly improves readability by reducing cognitive load.

Use environment variables

Use environment variables instead of hardcoding configuration values such as file paths, port numbers, or system-specific locations. This makes the code more flexible across different platforms, installation methods, and user preferences.

When implementing configuration:

  1. Use environment variables for values that might differ between environments
  2. Provide sensible defaults when environment variables are not set
  3. Support platform-specific path conventions (like ~ expansion) across operating systems
  4. Handle architecture differences (x86_64, aarch64, etc.) appropriately

Example:

// Bad: Hardcoded path that won't work across environments
func RegisterService() {
    server, err = zeroconf.Register(
        "OllamaInstance",    // Service Name
        "_ollama._tcp",      // Service Type
        "local.",            // Domain
        11434,               // Port - hardcoded!
    )
}

// Good: Use environment variable with fallback
func RegisterService() {
    port := 11434 // Default
    if hostEnv := os.Getenv("OLLAMA_HOST"); hostEnv != "" {
        if _, portStr, err := net.SplitHostPort(hostEnv); err == nil && portStr != "" {
            if p, err := strconv.Atoi(portStr); err == nil {
                port = p
            }
        }
    }
    
    server, err = zeroconf.Register(
        "OllamaInstance",    // Service Name
        "_ollama._tcp",      // Service Type
        "local.",            // Domain
        port,                // Port from environment or default
    )
}

For path handling, support platform-specific conventions:

func getConfigPath() string {
    if path, exists := os.LookupEnv("OLLAMA_CONFIG"); exists {
        // Handle home directory expansion
        if strings.HasPrefix(path, "~/") {
            homeDir, err := os.UserHomeDir()
            if err == nil {
                path = filepath.Join(homeDir, path[2:])
            }
        }
        return path
    }
    // Default fallback
    homeDir, _ := os.UserHomeDir()
    return filepath.Join(homeDir, ".ollama", "config")
}

Platform-aware configuration documentation

Always document platform-specific configuration differences and requirements thoroughly. When writing documentation or implementing configuration-related code, explicitly address variations across different operating systems and environments.

Key practices:

  1. Specify environment variable differences by platform (e.g., temp directories use TMPDIR on Linux, but TMP/TEMP on Windows)
  2. Document installation and configuration paths that may vary between systems
  3. Provide appropriate command syntax for each platform, noting differences in flags or parameters
  4. Include platform-specific prerequisites and setup requirements

Example:

# For temporary directory configuration:
# On Linux/macOS:
export TMPDIR=/path/with/sufficient/space

# On Windows:
set TMP=D:\path\with\sufficient\space

# When removing configuration directories:
# On Linux/macOS:
sudo rm -rf /etc/ollama  # Use -r flag for directories

# For cross-platform command examples:
ollama completion [bash|zsh|fish] > /path/to/completion/file  # Standard syntax pattern

Following these practices ensures configurations work correctly across different environments and helps users avoid platform-specific issues.

Loose API coupling

Maintain a clear separation between API definitions and their implementations by avoiding direct passing of API structures to implementation layers. Extract only the necessary values from API structures when passing data downstream.

This approach has several benefits:

Example of problematic coupling:

// Avoid: Passing entire API structure downstream
sampler := sample.NewSampler(req.Options, grammar)

Better approach:

// Better: Extract only needed values
sampler := sample.NewSampler(
    req.Options.Temperature,
    req.Options.TopK,
    req.Options.TopP,
    req.Options.MinP,
    req.Options.Seed,
    grammar,
)

Similarly, for transformation pipelines, prefer direct function calls with clear parameters over passing abstract data structures:

// Instead of: Using lists of transforms
transforms := []Transform{topK, topP, softmax}
Weighted(rng, transforms...)

// Better: Use specific function calls in a clear sequence
result := topK(topP(softmax(values)))

This principle extends to all layers of your API implementation - keep interfaces clean and pass only what’s needed.

Extract duplicated code

Identify repeated code segments and extract them into reusable functions or variables. This improves maintainability, reduces the risk of inconsistencies, and makes the code more concise.

Example from shader code:

// Before
float theta_base = (float) pos[i2];
// Code that uses theta_base
// ...
// Later in the code, theta_base is calculated again
float theta_base = (float) pos[i2];

// After
float calculate_theta_base(device const int32_t* pos, int i2) {
    return (float) pos[i2];
}
// ...
float theta_base = calculate_theta_base(pos, i2);
// ...
// Later in the code
float theta_base = calculate_theta_base(pos, i2);

Even for small code segments, applying the DRY (Don’t Repeat Yourself) principle leads to more maintainable code. When you see the same calculation, transformation, or logic repeated, consider extracting it into a function or storing it in a well-named variable. This also makes code more resistant to bugs when changes are needed, as you’ll only need to update the logic in one place.

Optimize with standard library

When implementing algorithms, prefer using Go’s standard library over custom implementations or external dependencies for common operations. The standard library offers well-tested, efficient implementations for many algorithmic needs.

Why this matters:

  1. Reduces code complexity and maintenance burden
  2. Leverages battle-tested implementations
  3. Eliminates external dependencies when possible
  4. Often provides better performance

Examples:

Instead of implementing custom sorting or search functions:

// Don't reinvent binary search
for low, high := 0, len(arr); low < high; {
    mid := (low + high) / 2
    if arr[mid] < target {
        low = mid + 1
    } else {
        high = mid
    }
}

// Better: Use standard library
idx := slices.BinarySearchFunc(arr, target, func(a, b YourType) int {
    return cmp.Compare(a, b)
})

Instead of custom data structures like priority queues:

// Don't use external dependencies for basic data structures
import pq "github.com/emirpasic/gods/queues/priorityqueue"
queue := pq.New()  // External dependency

// Better: Use container/heap from standard library
type tokenHeap []token
// Implement heap.Interface methods: Len, Less, Swap, Push, Pop
// Then use with standard heap package:
h := &tokenHeap{}
heap.Init(h)
heap.Push(h, item)

For string operations:

// Avoid manual string manipulation when possible
if strings.HasPrefix(s, prefix) {
    s = strings.TrimSpace(s[len(prefix):])
}

// Better: Use strings.Cut
if after, found := strings.CutPrefix(s, prefix); found {
    s = strings.TrimSpace(after)
}

For numeric operations:

// Avoid manual min/max checks
if temperature < 1e-7 {
    temperature = 1e-7
}

// Better: Use built-in min/max functions
temperature = max(temperature, 1e-7)

Improve documentation discoverability

Ensure that important information about code purpose, behavior, and context is easily discoverable through proper documentation placement and completeness. Add comprehensive docstrings to modules, classes, and functions that explain their purpose and usage. Place critical implementation details in docstrings rather than inline comments to improve discoverability. Use specific type annotations with clear explanations rather than generic types like Any.

For example, instead of:

# Note: We have tight control over the order of built-in converters, but
def __init__(self, markitdown: Any):
    pass

Use:

def __init__(self, markitdown: MarkItDown):
    """Initialize converter with MarkItDown instance.
    
    Args:
        markitdown: MarkItDown instance reference to allow recursive 
                   conversions of nested files. We maintain tight control 
                   over the order of built-in converters.
    """
    pass

This ensures developers can quickly understand the code’s purpose and important implementation details without hunting through comments or guessing from generic type hints.

AI memory management

Document and implement proper memory management strategies for AI model inference to prevent out-of-memory errors and optimize performance. When developing AI applications or documentation:

  1. Specify hardware memory requirements
    • Document minimum and recommended memory configurations
    • For GPU-accelerated models, specify VRAM requirements
    • Example: “Set UMA for iGPU in BIOS (at least >1GB, recommend >8GB for Llama3:8b q4_0 model size is 4.7GB)”
  2. Configure context window appropriately
    • Set context window size based on available memory and use case requirements
    • Example: OLLAMA_CONTEXT_LENGTH=8192 ollama serve
  3. Implement memory optimization techniques
    • Use environment variables to control memory usage: 
      # Reserve additional GPU memory buffer
OLLAMA_GPU_OVERHEAD=536870912 ollama serve
     
# Enable more efficient memory usage
OLLAMA_FLASH_ATTENTION=1 ollama serve
     
# Allow GPU to use CPU memory (Linux only)
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ollama serve
     
# Control parallel processing
OLLAMA_NUM_PARALLEL=1 ollama serve
  4. Document fallback strategies
    • Provide clear instructions for reducing memory requirements when needed
    • Include troubleshooting steps for memory-related errors

By explicitly documenting memory requirements and optimization strategies, you ensure reliable operation of AI models across different environments and hardware configurations.

AI dependency decoupling strategy

Implement clear boundaries between your core AI system and external machine learning libraries through abstraction layers. This facilitates future library updates or replacements as AI technology evolves. When integrating external libraries (like llama.cpp), establish documented synchronization processes and gradually work toward independence, especially for critical components like tokenizers.

For example, instead of directly depending on an external tokenizer:

// Avoid direct dependency
struct llama_vocab * vocab = llama_load_vocab_from_file(fname);

// Instead, create an abstraction layer
struct tokenizer * tokenizer = create_tokenizer_from_file(fname);
// Where create_tokenizer_from_file might initially use llama underneath
// but could later be replaced with a different implementation

This approach reduces technical debt and makes it easier to adopt new AI model architectures or optimizations as they become available.

validate before processing

Always validate input state and types before performing operations to prevent runtime errors and unexpected behavior. This includes checking data types before applying type-specific operations and preserving object state when performing validation checks.

When working with streams, restore the original position after reading during validation to avoid side effects:

# Good: Preserve stream position
cur_pos = file_stream.tell()
data = file_stream.read()
file_stream.seek(cur_pos)

When processing data, validate the type before applying operations:

# Good: Check type before text operations
if isinstance(content, str):
    content = self._strip_leading_blanks(content)

This defensive approach prevents errors from assumptions about input state or type, similar to how null checks prevent null reference exceptions.

explicit API parameters

Make API parameters explicit and named rather than hiding them in **kwargs. This improves clarity, enables better validation, and provides clearer interfaces for callers.

When designing APIs, avoid burying important parameters in generic **kwargs dictionaries. Instead, define them as explicit named parameters with proper types, defaults, and validation. This makes the API more discoverable, self-documenting, and less prone to errors.

For example, instead of:

def convert(self, local_path, **kwargs):
    pdf_engine = kwargs.get("pdf_engine", "pdfminer")
    # Hidden parameter, no validation

Use explicit parameters with validation:

def convert(self, local_path, pdf_engine: Literal['pdfminer', 'pymupdf4llm'] = 'pdfminer', **kwargs):
    _engines = {"pdfminer": pdfminer, "pymupdf4llm": pymupdf4llm}
    if pdf_engine not in _engines:
        raise ValueError(f"Invalid pdf_engine. Choose from {list(_engines.keys())}")

This approach also supports instance-based configuration where converters can be configured with parameters like API keys, while still maintaining clear method signatures. Avoid hardcoding assumptions (like content types) and instead derive them dynamically from the actual data when possible.

Complete HTTP protocol handling

When implementing HTTP client functionality, ensure comprehensive support for all standard HTTP status codes and secure connection scenarios:

  1. Support all standard redirect status codes (301, 302, 303, 307, 308), not just a subset:
switch resp.StatusCode {
case http.StatusOK:
    return requestURL, nil
case http.StatusMovedPermanently, // 301
     http.StatusFound,            // 302
     http.StatusSeeOther,         // 303
     http.StatusTemporaryRedirect, // 307
     http.StatusPermanentRedirect: // 308
    // Handle redirects
}
  1. For HTTPS connections with self-signed or internal certificates, provide explicit mechanisms rather than general “insecure” flags. Consider using special URL schemes like https+insecure:// for connections where certificate validation can be bypassed:
if strings.HasPrefix(url, "https+insecure://") {
    // Configure TLS client to skip certificate validation
    tlsConfig.InsecureSkipVerify = true
    // Strip prefix for actual connection
    url = "https://" + url[16:]
}
  1. Always document security implications when certificate validation is bypassed, with clear scope limitations (e.g., only for local development, internal networks).

Model precision matters

When deploying AI models on specialized hardware accelerators (GPUs, NPUs), ensure you’re using compatible model precision formats. Different hardware platforms have different optimal precision requirements that significantly impact performance and compatibility.

For example, when working with Ascend NPUs, fp16 models often work better than higher precision formats:

# When compiling AI frameworks for Ascend NPUs
export CUSTOM_CPU_FLAGS=cann
make --no-print-directory -f make/Makefile.cann

# When running models, verify hardware is properly detected
./ollama serve  # Should show detected NPUs in logs
# If models run on CPU despite hardware detection, try fp16 model variants
./ollama run model-name-f16  # Explicitly use fp16 version

Even when hardware is properly detected, models may still default to CPU execution if precision formats are incompatible with the acceleration hardware. Always test multiple precision variants (fp16, fp32, int8) to determine the optimal configuration for your target hardware.

Path traversal prevention

Implement multiple validation layers to prevent path traversal attacks. File paths provided by users or external systems must be validated before use in filesystem operations:

  1. Validate paths using both general and specialized checks: 
    // First validate with fs.ValidPath
if !fs.ValidPath(fp) {
    return nil, fmt.Errorf("%w: %s", errFilePath, fp)
}
   
// Then use filepath.Clean to normalize
fp = filepath.Clean(fp)
   
// Finally validate containment with os.Root
root, err := os.OpenRoot(safeDirectory)
if err != nil {
    return nil, err
}
defer root.Close()
// All file operations should go through this root
  2. Enforce a consistent path policy, such as:
    • Paths must be relative (no leading “/”)
    • Paths must not contain traversal components (“../”, “./”)
    • Paths must remain within the designated directory
  3. Implement validation at every entry point rather than relying on validation in a single place - this defense-in-depth approach ensures security even if the code is refactored or one validation is bypassed.

Abstract model operations cleanly

When implementing AI model operations, create clean abstractions through interfaces that separate core mathematical operations from specific implementations. This enables:

Example:

// Define interface for core operation
type ScaledDotProductAttention interface {
    // Document mathematical operation
    // Attention(Q,K,V) = softmax(QK^T/√d_k)V
    Forward(ctx Context, key, value, mask Tensor, scale float64) Tensor
}

// Implementation with proper validation and documentation
func Attention(ctx Context, query, key, value, mask Tensor, scale float64) Tensor {
    // Validate shapes match
    if !ValidateShapes(query, key, value, mask) {
        return nil, fmt.Errorf("invalid tensor shapes for attention")
    }
    
    // Use optimized implementation if available
    if sdpa, ok := query.(ScaledDotProductAttention); ok {
        return sdpa.Forward(ctx, key, value, mask, scale)
    }
    
    // Fallback to manual implementation
    kq := key.MulmatFullPrec(ctx, query)
    kq = kq.Scale(ctx, scale)
    kq = kq.Add(ctx, mask)
    return kq.Softmax(ctx)
}

Documentation language precision

Ensure documentation uses precise, accurate language that clearly describes functionality and highlights important behaviors users need to understand. Avoid vague or misleading terms that could confuse users about what the code actually does.

Key practices:

Example improvement:

// Instead of:
"This extension allows you to convert multiple files..."

// Use:
"This example shows how to convert multiple files..."

// And add clarifying details:
"Note that original files will remain unchanged and new markdown files are created with the same base name."

This ensures users have accurate expectations about functionality and understand important implementation details that affect their workflow.

Keep container images current

Always use up-to-date and supported container images in Dockerfiles and configuration files. Before specifying a base image or version:

  1. Verify the image hasn’t been deprecated
  2. Confirm the specified version is available in repositories
  3. Consider cross-platform compatibility requirements

For example, instead of:

FROM cosdt/cann:${ASCEND_VERSION} AS ascend-build-arm64

Use the recommended current alternative:

FROM ascendai/cann:${ASCEND_VERSION} AS ascend-build-arm64

Similarly, when updating dependency versions like ROCm, verify availability and compatibility with your base OS image:

# Check if this version exists with your base image before specifying
ARG ROCM_VERSION=6.3.2

When the desired version isn’t available with your current base image, consider updating the base image itself (e.g., moving from centos-7 to almalinux-8 for newer ROCm versions).

Use semantically clear names

Choose variable, method, and class names that clearly express their purpose and functionality to avoid confusion and improve code readability. Names should accurately reflect what the code does rather than how it’s implemented.

Key principles:

Examples:

# ❌ Misleading - sounds like it creates an index
def create_index(self) -> "PineconeIndex":
    return existing_index

# ✅ Clear - indicates it retrieves an existing index  
def get_index(self) -> "PineconeIndex":
    return existing_index

# ❌ Ambiguous abbreviation
def group_text_extraction_acc():
    pass

# ✅ Explicit and clear
def group_text_extraction_accuracy():
    pass

# ❌ Generic and unclear purpose
def write_raw_dict(self, elements_dict):
    self.normalize_data(elements_dict)
    self.write_to_db(elements_dict)

# ✅ Describes what the method actually does
def normalize_and_write_dict(self, elements_dict):
    self.normalize_data(elements_dict) 
    self.write_to_db(elements_dict)

# ❌ Predicate named like a function
skip_outside_ci = os.getenv("CI", "").lower() in {"", "false", "f", "0"}

# ✅ Boolean variable with clear intent
is_in_ci = os.getenv("CI", "").lower() not in {"", "false", "f", "0"}

This approach reduces cognitive load for developers, makes code self-documenting, and prevents misunderstandings about functionality.

optimize algorithmic complexity

When implementing algorithms, prioritize performance by avoiding unnecessary computations and choosing efficient approaches. Consider algorithmic complexity early in design and look for opportunities to eliminate redundant operations or expensive calculations when simpler alternatives exist.

Key strategies include:

Example from the codebase:

# Instead of always running expensive Levenshtein distance:
if 0.5 < len(output_cct.encode()) / len(source_cct.encode()) < 2.0:
    accuracy = round(calculate_accuracy(output_cct, source_cct, weights), 3)
else:
    accuracy = 0.01  # Skip expensive calculation when files differ wildly

This approach prevents performance bottlenecks and ensures algorithms scale appropriately with input size.

Use portable path configurations

Build and configuration scripts should use portable path handling techniques to ensure they work correctly across different environments and installations.

Key practices:

  1. Use wildcards when copying library files to include all version suffixes: 
    # Correct - captures all version suffixes
cp ${VULKAN_ROOT}/libvulkan.so* "${BUILD_DIR}/bin/"
   
# Incorrect - only copies the base library file
cp "${VULKAN_ROOT}/libvulkan.so" "${BUILD_DIR}/bin/"
  2. Avoid hardcoded absolute paths in configuration variables. When default paths are needed, make them overridable: 
    # Better approach - allows overriding with environment variables
if [ -z "${VULKAN_ROOT}" ]; then
    # Default only used if not explicitly set
    VULKAN_ROOT=/usr/lib/
fi
  3. Use relative paths with $ORIGIN for runtime library paths instead of absolute paths that assume specific installation locations: 
    # Better - uses relative paths from binary location
EXTRA_LIBS="-Wl,-rpath,'$ORIGIN'"
   
# Avoid - assumes specific installation location
EXTRA_LIBS="-Wl,-rpath,/opt/intel/oneapi/compiler/latest/lib"
  4. Be careful with string formatting in configuration variables to prevent command execution errors: 
    # Correct - no space after equals
DNF_COMPATIBILITY_FMT="addrepo --from-repofile="
   
# Incorrect - space will cause command parsing issues
DNF_COMPATIBILITY_FMT="addrepo --from-repofile= "

Following these practices ensures build scripts and configurations work reliably across different environments, making software more portable and easier to install for end users.

Responsible AI management

Ensure AI models are managed responsibly throughout their lifecycle and have appropriate boundaries on their decision-making authority. When deprecating AI models, maintain backward compatibility by marking old models as deprecated rather than removing them entirely. Additionally, prevent AI models from controlling sensitive parameters that could lead to security risks or unintended consequences.

For model deprecation, create new model files while preserving old ones:

# Old model file: cohere.command-r-16k.yaml
model: cohere.command-r-16k
deprecated: true

# New model file: cohere.command-r-08-2024.yaml  
model: cohere.command-r-08-2024

For tool parameters, ensure AI models don’t control sensitive values:

# Dangerous - AI decides the key
parameters:
  - name: key
    form: llm

# Safe - User provides the key
parameters:
  - name: key
    form: form

This approach prevents breaking existing configurations while maintaining security boundaries around AI decision-making capabilities.

Purpose-reflecting file names

File names should clearly communicate their purpose and functionality. Avoid generic, ambiguous, or numbered names (like “Makefile2”) that don’t convey the file’s actual role or lead to recurring questions.

Choose names that:

  1. Reflect the file’s specific purpose (e.g., “vendor.make” for vendoring operations)
  2. Avoid conflicts with convention-based names that carry specific expectations
  3. Prevent confusion with automatically generated files

For example, instead of:

Makefile2  # What does the "2" mean? What is this for?

Use:

vendor.make  # Clearly indicates this handles vendoring operations

This practice enhances code maintainability and reduces onboarding time for new developers, who can understand a file’s purpose at a glance rather than having to investigate its contents or ask questions.

Handle async concurrency properly

When writing async endpoints, ensure that long-running CPU-bound operations don’t block the event loop and that concurrent requests don’t conflict over shared resources. CPU-bound operations should be offloaded to thread pools using asyncio.run_in_executor() or similar mechanisms. Additionally, avoid using predictable file paths or shared resources that can cause race conditions between concurrent requests.

Example of problematic code:

@app.post("/convert")
async def convert(file: UploadFile = File(...)):
    # Problem 1: Blocking CPU-bound operation
    result = markitdown.convert(temp_file_path)  # Blocks event loop
    
    # Problem 2: Race condition with shared file path
    temp_file_path = f"/tmp/{file.filename}"  # Conflicts with concurrent requests

Better approach:

@app.post("/convert")
async def convert(file: UploadFile = File(...)):
    # Solution 1: Use thread pool for CPU-bound work
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(None, markitdown.convert, temp_file_path)
    
    # Solution 2: Use unique file paths
    import uuid
    temp_file_path = f"/tmp/{uuid.uuid4()}_{file.filename}"

This prevents blocking the async event loop and eliminates race conditions between concurrent requests.

PEP 257 docstring compliance

All public modules, classes, methods, and functions must have docstrings that follow PEP 257 formatting standards. This ensures consistent, professional documentation across the codebase.

Required PEP 257 Format:

Examples:

Single-line docstring:

def get_nltk_data_dir() -> str | None:
    """The directory where `nltk` resources are located."""

Multi-line docstring:

def clean_pdfminer_inner_elements(document: "DocumentLayout") -> "DocumentLayout":
    """Move pdfminer elements from inside tables to the extra_info dictionary.

    Each element appears in the `extra_info` dictionary using its table id as the key.
    """

Module docstring:

"""NDJSON file utilities for document processing.

This module provides functionality for reading and writing newline-delimited JSON files.
Used primarily for batch processing of document elements during partitioning workflows.
"""

Coverage Requirements:

This standard improves code discoverability, reduces onboarding time, and maintains professional documentation quality throughout the codebase.

Simplify configuration interfaces

Design configuration interfaces that prioritize user experience by providing sensible defaults, using clear naming conventions, and logically grouping related options. Complex tools with many configuration parameters should minimize the cognitive load on users through thoughtful interface design.

Key principles:

  1. Provide sensible defaults: Allow users to skip commonly-used parameters by implementing reasonable default values that work for typical use cases
  2. Use clear, unambiguous naming: Avoid parameter names that could be misinterpreted (e.g., --model-api-name instead of --model when referring to API endpoint naming)
  3. Group related configurations: Consolidate related parameters into higher-level options (e.g., --enable-lmcache instead of multiple LMCache-specific flags)
  4. Improve example readability: Format complex command examples with line breaks and remove verbose default values

Example of improvement:

# Before: Complex command with many explicit parameters
python3 rag.py --qps 3.5 --model mistralai/Mistral-7B-Instruct-v0.2 --dataset ~/CacheBlend/inputs/musique_s.json --system-prompt "You will be asked a question..." --query-prompt "Answer the question..." --separator "" --prompt-build-method QA --base-url "http://localhost:8000/v1" --kv-storage-size 30GB --max-tokens 32 --output summary.csv

# After: Simplified with defaults and clear grouping
python3 rag.py \
    --qps 3.5 \
    --model mistralai/Mistral-7B-Instruct-v0.2 \
    --dataset ~/CacheBlend/inputs/musique_s.json \
    --enable-lmcache \
    --output summary.csv

This approach reduces user friction while maintaining the flexibility needed for advanced use cases.

Cross-platform configuration simplification

When writing configuration files and scripts, prioritize cross-platform compatibility while keeping configurations simple and maintainable. Use platform-aware tools like run-script-os for npm scripts that need platform-specific behavior, but prefer built-in solutions and global defaults over complex custom configurations.

For npm scripts, combine similar platform commands when possible:

{
  "scripts": {
    "version-restore": "run-script-os",
    "version-restore:darwin:linux": "jq --arg ver $(cat .version.bak) '.version = $ver' package.json > package.tmp && mv package.tmp package.json && rm .version.bak",
    "version-restore:win32": "node -e \"/* Windows-specific implementation */\""
  }
}

Avoid explicitly setting configuration properties that can use sensible defaults. Consider using native tools (like node --watch) instead of additional dependencies when they provide equivalent functionality. This reduces complexity, improves maintainability, and ensures consistent behavior across different development environments.

prefer efficient operations

Choose efficient operations and algorithms over expensive alternatives, particularly when processing data or performing file operations. Avoid operations that create unnecessary intermediate objects or make repeated system calls when vectorized or batch operations are available.

Key principles:

Examples:

# Instead of computing values in a loop:
for item in items:
    item['width'] = item['right'] - item['left']
    item['height'] = item['bottom'] - item['top']

# Use vectorized operations:
df['width'] = df['right'] - df['left'] 
df['height'] = df['bottom'] - df['top']

# Instead of expensive string splitting:
filename = (doc.split("/")[-1]).split(f".{output_type}")[0]

# Use efficient slicing:
filename = os.path.basename(doc)[:-len(output_type)-1]

# Instead of repeated system calls:
while os.path.exists(os.path.join(dir, filename)):
    # check and increment

# Use single directory listing:
existing_files = os.listdir(dir)
# then filter and find max counter

This approach reduces computational overhead, memory allocation, and system call frequency, leading to better overall performance.

validate configuration defaults

Configuration parameters should be properly validated with appropriate defaults and clear handling of edge cases. Avoid using arbitrary defaults when values should be explicitly required, and ensure configuration validation distinguishes between None, empty, and zero values.

Key practices:

  1. Validate parameter types and ranges - Check that configuration values meet expected constraints
  2. Use meaningful defaults - Set defaults that make sense for the use case, not just convenient values
  3. Handle None vs zero distinction - Be explicit about whether 0 is a valid value or should trigger default behavior
  4. Filter unsupported parameters - Remove configuration options not supported by the current version
  5. Make critical configs required - Don’t provide defaults for parameters that users should explicitly set

Example of proper configuration validation:

@property
def TESSERACT_CHARACTER_CONFIDENCE_THRESHOLD(self) -> float:
    """Tesseract predictions with confidence below this threshold are ignored"""
    return self._get_float("TESSERACT_CHARACTER_CONFIDENCE_THRESHOLD", 0.0)

def get_retries_config(self, retries_initial_interval: Optional[int]) -> Optional[retries.RetryConfig]:
    # Check for None explicitly, not just falsy values
    if retries_initial_interval is not None:
        return retries_initial_interval
    return DEFAULT_RETRIES_INITIAL_INTERVAL_SEC

# Filter configuration to supported fields only
possible_fields = [f.name for f in fields(PartitionParameters)]
filtered_config = {k: v for k, v in config.items() if k in possible_fields}

This prevents configuration errors, improves user experience, and ensures consistent behavior across different environments and versions.

Parameterize hardcoded configuration values

Replace hardcoded values in configuration files with parameterized variables to improve flexibility and maintainability. Use build arguments (ARG) and environment variables (ENV) in Dockerfiles instead of embedding specific versions, paths, or other configuration values directly in the code.

This approach makes configurations more adaptable to different environments and reduces the need to modify multiple locations when values change. For example, instead of hardcoding tool versions and paths throughout a Dockerfile:

# Instead of hardcoded values:
RUN pip3.11 install --user -r requirements.txt
COPY ./nltk_data /home/notebook-user/nltk_data

# Use parameterized approach:
ARG PYTHON_VERSION=3.11
ARG NLTK_DATA_SOURCE=./nltk_data
ENV PIP=pip${PYTHON_VERSION}

RUN ${PIP} install --user -r requirements.txt
COPY ${NLTK_DATA_SOURCE} /home/notebook-user/nltk_data

This pattern applies to any configuration file where values might need to change across different environments or deployments.

Follow configuration best practices

Always adhere to official guidelines and proper validation when setting up configurations, whether for dependencies, file formats, or environment setup. This prevents subtle issues that can cause tests to pass incorrectly or create environment conflicts.

For dependency installation, use the recommended method from official documentation rather than generic approaches:

# Instead of:
install:
	pip install hatch

# Use the recommended approach:
install:
	pipx install hatch

For file configurations, validate that files have the correct format and MIME type, not just the right extension. A .doc file should have MIME type application/msword, not application/octet-stream. Simply renaming a .docx file to .doc doesn’t change its underlying format and can lead to false test results.

This practice ensures reliable, conflict-free environments and prevents configuration-related bugs from propagating through your system.

Validate configuration assumptions

Before adding or modifying configuration settings, understand the underlying tool capabilities and test your assumptions. Many configuration issues arise from assumptions about what’s needed without investigating whether the tool already provides the desired functionality or has specific constraints.

For example, when setting up development containers, research whether build tools like Hatch already provide features you’re trying to configure manually:

// Instead of assuming you need postCreateCommand for editable installs
"postCreateCommand": "pip install -e .",

// Understand that Hatch already handles editable mode for testing
"features": {
    "ghcr.io/devcontainers-extra/features/hatch:2": {}
}

Similarly, when configuration choices seem to conflict with best practices (like using root user), document the practical constraints that necessitate the decision rather than just following defaults. Test alternative approaches when possible, but accept necessary trade-offs when tools have specific requirements for proper functionality.

Manage side effects properly

Avoid performing side effects directly in component render functions. Use useEffect for side effects like API calls, DOM manipulations, or state changes that should occur after render. Wrap functions with useCallback when they need to be optimized or used as dependencies, and use async/await for proper asynchronous handling.

Example of incorrect usage:

const I18n: FC<II18nProps> = ({ locale, children }) => {
  // ❌ Side effect in render
  locale && changeLanguage(locale)
  
  return <I18NContext.Provider>...</I18NContext.Provider>
}

Example of correct usage:

const I18n: FC<II18nProps> = ({ locale, children }) => {
  // ✅ Side effect in useEffect
  useEffect(() => {
    locale && changeLanguage(locale)
  }, [locale])

  // ✅ Function wrapped with useCallback and async/await
  const handleRender = useCallback(async () => {
    await renderFlowchart(code)
  }, [code])
  
  return <I18NContext.Provider>...</I18NContext.Provider>
}

This ensures predictable component behavior, prevents unnecessary re-renders, and maintains React’s rendering principles.

extensible parameter design

Design API parameters to support future extension without breaking changes. Use string enums instead of boolean flags when the parameter might need additional values later, implement comprehensive parameter forwarding with **kwargs, and avoid overly specific parameter designs that limit future flexibility.

Key patterns to follow:

  1. String enums over booleans: Instead of contains_ontology_schema: bool = False, use html_parser: str = "v1" to allow future parser versions without new parameters.

  2. Comprehensive parameter forwarding: Use **kwargs to forward all parameters to delegate functions rather than explicitly listing each one: ```python

    Preferred approach

    elements = _partition_doc(filename, file=file, **kwargs)

Avoid this maintenance-heavy pattern

elements = _partition_doc( filename=filename, file=file, infer_table_structure=infer_table_structure, languages=languages, strategy=strategy, **kwargs, )


3. **Flexible interface design**: Accept the most general form of input (e.g., always expect text rather than having separate file/text/stdin branches) to reduce conditional complexity.

This approach ensures APIs can evolve gracefully, reduces maintenance overhead when adding new parameters, and provides a consistent experience for API consumers. All partitioners automatically receive new parameters through kwargs without requiring explicit updates to forwarding logic.

---

## maintain established naming conventions

<!-- source: Unstructured-IO/unstructured | topic: Naming Conventions | language: Json | updated: 2024-10-22 -->

Ensure all naming follows established patterns and conventions already present in the codebase. When introducing new identifiers or modifying existing ones, they must conform to the team's built-in expectations and standard patterns to maintain consistency across the system.

This prevents violations of established conventions that can break compatibility or create confusion. For example, if element IDs are expected to be formatted as hashes without dashes (like "782cf07be8b3ab8f05188e479edb7f61"), then all element IDs should follow this pattern. Similarly, if certain class names violate standard output patterns used throughout the system, they should be removed or renamed to maintain consistency.

Example of applying this standard:
```json
// Bad - violates hash convention with dashes
"element_id": "897a8a47-377c-4ad6-aab8-39a929879537"

// Good - follows established hash format
"element_id": "897a8a47377c4ad6aab839a929879537"

Before introducing any new naming patterns, verify they align with existing conventions in the codebase and don’t break established expectations that other parts of the system depend on.

Enhance documentation detail

Documentation should provide comprehensive detail that enables users to understand impacts and developers to understand implementation. For breaking changes, include migration guidance, deprecated features, and specific functional changes. For technical documentation, provide granular step-by-step explanations that trace execution flow.

For breaking changes in changelogs:

For developer guides:

Example from changelog:

## 0.16.0

### Breaking Changes
* **Remove ingest implementation**
  - Migration: Use new unstructured-ingest library (link to installation guide)
  - Deprecated extras: s3, local-inference, github (see migration guide)
  - Removed submodules: unstructured.ingest.v1.*, unstructured.ingest.connectors.*
  - Related: embed submodule updated with SecretStr for secret handling

Avoid external URLs

Links to external repositories or domains in documentation and code can create security vulnerabilities through URL hijacking attacks. Malicious actors could potentially gain control of external domains and redirect users to harmful content or credential harvesting sites.

Always use official, internal, or trusted repository URLs instead of external ones. When linking to documentation, resources, or code examples, ensure the URLs point to your organization’s official repositories or well-established, trusted sources.

Example of the security risk:

<!-- Vulnerable - external repository -->
- <a href="https://github.com/NHPT/jan/blob/dev/README_CN.md">简体中文</a>

<!-- Secure - official repository -->
- <a href="https://github.com/janhq/jan/blob/dev/README_CN.md">简体中文</a>

This practice is especially critical in README files, documentation, and any user-facing content where links could be used to redirect users to potentially malicious sites.

Environment variable patterns

Use consistent patterns for environment variable handling in shell scripts. Always provide sensible defaults using the ${VAR:-default} syntax, validate required variables early with clear error messages, and preserve the ability to override configuration values.

For variables with defaults:

strategies=${STRATEGIES:=fast,hi_res,ocr_only}
RUN_SCRIPT=${RUN_SCRIPT:-unstructured-ingest}
PYTHONPATH=${PYTHONPATH:-.}

For required variables, validate early and provide helpful error messages:

if [ -z "$VECTARA_OAUTH_CLIENT_ID" ]; then
  echo "Skipping VECTARA ingest test because VECTARA_OAUTH_CLIENT_ID env var is not set."
  exit 0
fi

Document configurable environment variables in script comments or usage instructions to make them discoverable for users who aren’t familiar with shell scripting. This approach allows scripts to work out-of-the-box with reasonable defaults while remaining flexible for different environments and use cases.

Use centralized model state

Always access AI model information, parameters, and configuration through centralized state management rather than direct access or hardcoded values. This ensures consistency, maintainability, and proper separation of concerns when working with AI models.

For model access, use established state management patterns:

// Good: Use centralized state
const model = currentActiveModel

// Avoid: Direct model access
const model = message.model

For model parameters (like llama.cpp settings), define constraints centrally rather than scattered throughout components:

// Good: Centralized parameter definitions
const llamaParams = {
  temperature: { min: 0, max: 2, step: 0.1 },
  maxTokens: { min: 1, max: 4096, step: 1 }
}

// Avoid: Hardcoded values in components
<SliderRightPanel min={0} max={100} step={1} />

This approach prevents inconsistencies, makes parameter updates easier, and provides a single source of truth for AI model configuration across the application.

Document configuration parameters clearly

When documenting configuration objects and their parameters, provide clear explanations of each parameter’s purpose, whether it’s required or optional, and why optional parameters are useful for development. Use backticks to format code symbols, parameter names, class names, and error types for better readability.

For example, when documenting configuration like:

context=ProcessorConfig(
    verbose=True,        # Optional: enables detailed logging for debugging
    work_dir="local-working-dir",  # Required: working directory for pipeline
    reprocess=True,      # Optional: forces reprocessing of existing files
    re_download=True,    # Optional: forces re-downloading of source files
)

Explain that verbose, reprocess, and re_download are optional parameters that make development easier by providing debugging information and forcing fresh processing during iterative development.

In documentation text, wrap configuration-related terms in backticks: “Sets minimum version of unstructured-client to avoid raising a TypeError when passing api_key_auth to UnstructuredClient”

Use descriptive specific names

Choose names that clearly indicate their context and purpose rather than generic terms. Generic names can be ambiguous and make code harder to understand, especially when the same codebase handles multiple contexts or domains.

When naming variables, parameters, methods, or classes, prefer specific descriptive names that include relevant context over broad generic terms. This makes the code self-documenting and reduces the cognitive load for other developers.

Examples of improvements:

The goal is to make names immediately clear about what they represent without requiring additional context or documentation.

intentional naming patterns

Recognize when seemingly redundant naming patterns serve specific technical purposes rather than being mistakes or oversights. Some naming conventions that appear duplicative are actually intentional and necessary for technical reasons.

Common examples include:

Before suggesting changes to apparently redundant naming, verify whether the pattern serves a technical requirement such as type checking, import management, or API design. Understanding the underlying purpose helps distinguish between genuine redundancy and intentional technical patterns.

# Intentional - republishing import in type stub
from ._element import HtmlElement as HtmlElement

# Intentional - same logical function, different signatures  
@overload
def strip_elements(tree: _ElementOrTree, tags: Collection[_TagSelector]) -> None: ...
@overload  
def strip_elements(tree: _ElementOrTree, *tags: _TagSelector) -> None: ...

Use configuration constants

Always use named constants, enums, or well-defined configuration objects instead of magic strings, numbers, or inline literals for configuration values. This improves maintainability, reduces typos, enables better IDE support, and makes configuration changes easier to track.

Examples of good practices:

This approach makes configuration management more robust and prevents runtime errors from typos in configuration keys or values.

Document configuration reasoning

Always document the reasoning behind configuration decisions, especially for temporary settings like dependency pins, parameter choices, and version constraints. Include explanations of why the configuration was needed, what problem it solves, and when it might be safe to remove or update.

For dependency pins, explain the specific compatibility issue or version conflict being resolved:

# tenacity 9.0.0 is being installed via chroma, but other dependencies (langchain) restrict tenacity
# to <=9.0.0 in other dependency extras and resolve to 8.5.0.
# The original langchain pin: https://github.com/langchain-ai/langchain/pull/849/
tenacity==8.5.0

For API parameters, document when parameters conflict or when one supersedes another to avoid confusion. Remove deprecated or conflicting configuration options rather than documenting both. This practice enables future maintainers to understand when temporary configurations can be safely removed and helps prevent accumulation of unnecessary constraints.

Complete documentation coverage

Documentation should comprehensively cover all supported options, variations, and potential future use cases. When creating documentation:

  1. Use generic titles and structures that can accommodate future additions
    • Example: Instead of “AMD iGPU 780m Guide”, use “AMD iGPU Guide” to allow for documenting additional GPU models
  2. Include examples for all supported options and environments
    • Example: When documenting shell commands, provide examples for all supported shells: ```

      For bash

      echo “source <(ollama completion bash)” » ~/.bashrc

    For zsh

    echo “source <(ollama completion zsh)” » ~/.zshrc

    For fish

    ollama completion fish > ~/.config/fish/completions/ollama.fish ```

  3. Document complete workflows from beginning to end

  4. Verify that all links are current and functional

Comprehensive documentation reduces the need for clarifications, helps users across different environments, and ensures documentation remains relevant as software evolves.

Use descriptive names

Choose variable and function names that clearly communicate their purpose, type, and behavior. Names should be self-documenting and immediately understandable to other developers without requiring additional context.

For boolean variables and functions, use names that clearly indicate they return true/false values:

For properties and variables, prefer generic, maintainable names over product-specific ones, especially in open source projects where the codebase may be forked or renamed.

Examples:

// Poor naming - unclear purpose
const SOME_API_KEY_ADDED = 'someApiKeyAdded'
const checkFileExists = (path: string) => boolean

// Better naming - clear purpose and type
const IS_ANY_REMOTE_MODEL_CONFIGURED = 'isAnyRemoteModelConfigured'
const exists = (path: string) => boolean

// Poor naming - product-specific
const properties = { JanVersion: VERSION }

// Better naming - generic and maintainable  
const properties = { appVersion: VERSION }

This approach improves code readability, reduces the need for comments, and makes the codebase more maintainable for both current and future developers.

Database schema compliance

Ensure database integrations use correct field names, data types, and schema conventions specific to each database system. Different databases have special field naming requirements and data type expectations that must be respected to avoid runtime errors and ensure proper functionality.

Key considerations:

Example from Astra DB integration:

def normalize_dict(self, element_dict: dict) -> dict:
    return {
        "_id": str(uuid.uuid4()),  # Astra DB expects _id, not id
        "$vector": element_dict.pop("embeddings", None),  # Special $vector field
        "content": element_dict.pop("text", None),
        "metadata": element_dict,
    }

This prevents issues like vector search failures due to incorrect field naming and ensures data is stored in the format expected by the target database system.

improve code readability

Format code to enhance readability and maintainability by breaking long lines and extracting repeated values into variables. Long command lines with multiple parameters should be split across multiple lines with proper line continuation, placing each parameter on its own line. When the same value (like URLs, paths, or configuration strings) appears multiple times, extract it into a clearly named variable at the top of the file.

Example of improved formatting:

# Extract repeated URL base
BASE_URL="https://ingest-test-azure-cognitive-search.search.windows.net"

# Break long command lines for readability
PYTHONPATH=${PYTHONPATH:-.} "$RUN_SCRIPT" \
  --reprocess \
  --input-path example-docs/fake-memo.pdf \
  --work-dir "$WORK_DIR" \
  --chunking-strategy by_title \
  --chunk-combine-text-under-n-chars 150 \
  --chunk-new-after-n-chars 1500 \
  --chunk-max-characters 2500 \
  --chunk-multipage-sections \
  --chunk-no-include-orig-elements

This approach reduces duplication, makes the code easier to modify, and significantly improves readability by avoiding overly long lines that are difficult to scan and understand.

avoid hardcoded secrets

Never include real API keys, passwords, or other sensitive credentials directly in source code, including example files and configuration. This creates security vulnerabilities and risks credential exposure in version control systems.

For example code and documentation, use clear placeholder text that indicates where credentials should be provided:

# Bad - real API key hardcoded
access_config=PineconeAccessConfig(api_key="0e8555a2-b944-4da4-837e-03b1202e00c7")

# Good - clear placeholder
access_config=PineconeAccessConfig(api_key="<YOUR_PINECONE_API_KEY_HERE>")

# Better - environment variable
access_config=PineconeAccessConfig(api_key=os.getenv("PINECONE_API_KEY"))

When handling credentials in configuration classes, mark sensitive fields appropriately to ensure they are handled securely:

@dataclass
class SqlAccessConfig(AccessConfig):
    username: str  # Not sensitive, needed for access
    password: t.Optional[str] = enhanced_field(sensitive=True)  # Mark as sensitive

Use environment variables, secure configuration files, or credential management systems for real deployments.

Python version standardization

Establish and document a consistent Python version standard for dependency compilation across the project. Team confusion about which Python version to use for pip-compile leads to build inconsistencies, compatibility issues, and wasted development time.

The discussions reveal conflicting approaches: some team members believe dependencies should be compiled with the minimum supported Python version (3.9) for maximum compatibility, while others use different versions (3.10). This inconsistency causes build failures and requires rework.

Implementation:

  1. Document the official Python version for pip-compile in your project’s README or contributing guidelines
  2. Add version checks to your Makefile or build scripts to prevent compilation with incorrect Python versions
  3. Include the Python version requirement in CI/CD pipeline documentation

Example from the discussions:

# Error when wrong Python version is used:
AssertionError
python version not equal to expected 3.9: Python 3.10.13
make: *** [pip-compile] Error 1

Ensure all team members understand and follow the same Python version standard to maintain build reproducibility and avoid compatibility issues across different environments.

Centralize configuration sources

Maintain configuration settings in a single authoritative source to avoid duplication and inconsistencies across multiple files. When possible, use pyproject.toml as the primary configuration file and remove redundant specifications from other files like .pre-commit-config.yaml and Makefile.

This approach follows the DRY (Don’t Repeat Yourself) principle and ensures that configuration changes only need to be made in one place, reducing maintenance overhead and preventing configuration drift.

Example:

# pyproject.toml - Single source of truth
[tool.ruff.lint]
select = [
    "UP018",
    "UP032", 
    "UP034",
    "W",        # -- Warnings, including invalid escape-sequence --
]
ignore = [
    "COM812",
    "PT005",    # -- flags mock fixtures with names intentionally matching private method name --
    "PT011",
]

Remove redundant ruff configurations from .pre-commit-config.yaml and Makefile so they automatically pick up settings from pyproject.toml.

Validate API parameter formats

Ensure API parameters match the expected format and structure of target APIs before runtime. Validate parameter types, required fields, and data structures during development rather than discovering mismatches through runtime errors.

When integrating with external APIs, explicitly validate that:

Example of problematic parameter passing:

# Incorrect - API expects array but single value provided
--requested-indexing-policy '{"deny": "metadata"}'

# Correct - API expects array format
--requested-indexing-policy '{"deny": ["metadata"]}'

Additionally, make authentication explicit through parameters:

# Better - explicit parameter
--embedding-api-key "$VERTEX_API_KEY"

# Rather than relying on implicit environment variables
# GOOGLE_APPLICATION_CREDENTIALS (less obvious to users)

This prevents runtime failures and makes API integrations more maintainable and user-friendly.

Write clear test cases

Structure test cases for maximum readability and maintainability by following these principles:

  1. Organize each test in three distinct parts separated by blank lines:
    • Setup (arrange test fixtures)
    • Execution (run unit under test)
    • Verification (assert expected results)
  2. Minimize intermediate variables - use direct assertions when possible:

Instead of:

matrix = [["cell1", "cell2"], ["cell3", "cell4"]]
expected_html = "<table><tr><td>cell1</td>...</table>"
assert utils.htmlify_matrix(matrix) == expected_html

Prefer:

assert utils.htmlify_matrix([["cell1", "cell2"], ["cell3", "cell4"]]) == (
    "<table><tr><td>cell1</td><td>cell2</td></tr>"
    "<tr><td>cell3</td><td>cell4</td></tr></table>"
)
  1. Use parameterization to avoid repeated test code: 
    @pytest.mark.parametrize(
 "date", ["1990-12-01", "2050-01-01T00:00:00", "2050-01-01+00:00:00"]
)
def test_validate_date_args_accepts_standard_formats(date):
 assert utils.validate_date_args(date)
  2. Write meaningful assertions that clearly indicate the behavior being tested: ```python

    Weak assertion:

    assert “error” in caplog.text

Better assertion:

assert “invalid date format, expected YYYY-MM-DD” in caplog.text


---

## model description accuracy

<!-- source: menloresearch/jan | topic: AI | language: Json | updated: 2024-04-22 -->

Ensure AI model descriptions are specific, capability-focused, and avoid marketing language or temporal references like "latest". Descriptions should clearly communicate what the model excels at and its intended use cases to help users make informed selection decisions.

Replace placeholder or generic text with meaningful descriptions that highlight actual capabilities. Avoid hyperbolic statements and focus on practical information.

Example of good description:
```json
{
  "description": "Meta's Llama 3 excels at general usage situations, including chat, general world knowledge, and coding."
}

Instead of:

{
  "description": "The latest model from MetaAI"
}

Be concise and pragmatic - “CodeNinja is finetuned on openchat/openchat-3.5-1210. Good for coding.” is better than “CodeNinja is an enhanced version of the renowned model… designed to be an indispensable tool for coders”.

embedding provider configuration

When configuring embedding providers in AI integrations, ensure proper dimension specification and use dedicated test files for different providers. Always specify the embedding dimension that matches your chosen provider (e.g., 384 for huggingface embedders) and avoid modifying existing configurations for testing purposes.

Instead of changing embedding providers in existing files for testing, create separate test files for each provider. This prevents disruption to working configurations and provides clearer test isolation.

Example of proper configuration:

PYTHONPATH=. ./unstructured/ingest/main.py \
  --embedding-provider "langchain-huggingface" \
  --embedding-dimension 384 \
  astra \
  --token "$ASTRA_DB_APPLICATION_TOKEN" \
  --collection-name "$COLLECTION_NAME"

For testing new providers, create dedicated test files like test_unstructured_ingest/src/local-embed-vertexai.sh rather than modifying existing provider configurations. This approach maintains stability while enabling comprehensive testing of different AI embedding services.

separate formatting changes

Keep formatting and style changes separate from functional code changes to maintain clear, reviewable diffs. When making functional changes, avoid simultaneously reordering imports, adjusting whitespace, or making other cosmetic modifications that obscure the actual logic changes. If formatting improvements are needed, submit them as standalone pull requests.

This practice ensures that code reviewers can focus on the core logic changes without being distracted by formatting noise. It also makes it easier to track the history of functional changes and reduces the likelihood of merge conflicts.

Example of what to avoid:

# Don't mix functional changes with formatting changes
@@ -1,19 +1,20 @@
 -c "constraints.in"
+backoff  # functional change
+# Also reordering other dependencies (formatting change)

Instead, submit formatting changes like dependency reordering as separate commits or pull requests. This keeps the diff focused on the actual functional change (adding the backoff dependency) rather than mixing it with unrelated organizational changes.

Use platform-specific endpoints

When configuring API integrations, always use the actual platform-specific endpoints and identifiers rather than generic placeholders. Generic configurations often lead to integration failures and confusion for users following documentation.

Ensure that:

For example, when integrating with Groq API:

{
  "full_url": "https://api.groq.com/openai/v1/chat/completions",
  "api_key": "<your-groq-api-key>"
}

And use actual model IDs from the platform:

{
  "id": "mixtral-8x7b-32768",
  // not generic "groq"
}

This practice prevents integration failures and ensures documentation accurately reflects working configurations that users can directly implement.

API interface consistency

Ensure that API function signatures, return types, and interface implementations are consistent and properly declared. When modifying functions to be asynchronous or changing their return types, update both the implementation and any related type declarations to maintain contract integrity.

Key principles:

Example of proper async function declaration:

// Good: Declaration matches implementation
async fileStat(path: string, outsideJanDataFolder?: boolean): Promise<FileStat | undefined> {
  // async implementation
}

// Good: Focused interface implementation
export default class JSONConversationalExtension
  implements ThreadExtension, MessageExtension {
  // implements specific feature contracts
}

This ensures API consumers can rely on consistent contracts and prevents runtime errors from signature mismatches.

Document credential formats

When documenting security-sensitive parameters such as private keys, certificates, API tokens, or other authentication credentials, always specify the expected format including any required headers, footers, and structural elements. This prevents configuration errors that could lead to authentication failures or security vulnerabilities.

For example, when documenting a private key parameter, clarify the format:

- ``Private Key (PEM)`` (*required*): Input the Private Key associated with the Consumer Key. 
  The key should begin with -----BEGIN RSA PRIVATE KEY----- and end with -----END RSA PRIVATE KEY-----.

This practice helps users correctly configure security credentials and reduces the risk of authentication issues or security misconfigurations that could compromise system security.

Ensure proper event sequencing

When working with event-driven systems, ensure that events are emitted at the appropriate time and in the correct context to prevent race conditions and incorrect behavior. Consider whether the function’s purpose aligns with event emission - single-response functions should not broadcast events meant for streaming contexts. Additionally, be mindful of timing when handling sequential operations to avoid race conditions.

For example, avoid emitting events in non-streaming contexts:

// Bad: Emitting events in single-response function
const message = {
  status: MessageStatus.Pending,
  // ... other properties
};
events.emit(MessageEvent.OnMessageResponse, message); // Inappropriate for single-response

// Good: Only emit events in appropriate streaming contexts
if (isStreamingMode) {
  events.emit(MessageEvent.OnMessageResponse, message);
}

When handling sequential operations, ensure proper timing:

// Bad: Potential race condition
if (messages.length == 1) {
  // Handle first message logic here - race condition possible
}

// Good: Handle after message response to avoid race
// Move to EventHandler > OnMessageResponse to ensure proper sequencing

API examples completeness

Ensure all API code examples include all required parameters with correct placeholder values. Incomplete examples lead to user confusion and implementation errors when developers copy-paste documentation snippets.

When documenting API usage, always include:

Example of complete API documentation:

elements = partition_via_api(
    filename=filename,
    api_key="<<REPLACE WITH YOUR API KEY>>",
    api_url="https://<<REPLACE WITH YOUR API URL>>/general/v0/general"
)

Rather than incomplete examples missing the api_key parameter or using incorrect placeholder values. Consider providing separate examples for different usage scenarios (SaaS vs self-hosted) when the parameter requirements differ significantly.

AI model configuration completeness

Ensure all necessary AI model parameters are properly configured and passed through the system. AI models require comprehensive configuration including hardware settings (CPU/GPU allocation, thread counts), model-specific parameters (context length, n_gpu_layers), and proper path resolution. Missing configuration can lead to runtime failures, suboptimal performance, or initialization errors.

Key requirements:

Example of complete model configuration:

const modelSettings = {
  llama_model_path: await joinPath([modelsDir, model.id]),
  ctx_len: model.settings.ctx_len || 2048,
  ngl: model.settings.ngl || 100,
  cpu_threads: nitroResourceProbe.numCpuPhysicalCore,
  prompt_template: model.settings.prompt_template,
  embedding: true
};

// Validate before initialization
if (!modelSettings.llama_model_path || !modelSettings.cpu_threads) {
  throw new Error('Incomplete model configuration');
}

This prevents critical issues like models failing to load due to missing parameters or processes spawning before proper configuration is established.

Use centralized logging framework

Always use the established logging infrastructure instead of creating custom logging solutions or multiple logging systems. This prevents fragmentation and ensures consistent log management across the codebase.

When you need logging functionality, leverage the existing logging framework (like @janhq/core/node log function) rather than implementing ad-hoc solutions. If additional features like log levels are needed, extend the central framework rather than creating parallel systems.

Avoid patterns like:

// Don't create custom debug wrappers
export const debugInspectorSync = process.env.DEBUG ? debugInspector(setBinPath) : setBinPath

// Don't implement separate file logging
subprocess.stderr.on("data", (data) => {
  fs.writeFileSync("nitro-err.txt", data); // Avoid this
});

Instead, use the centralized approach:

import { log } from "@janhq/core/node";

// Use the established logging system
log("Debug info", data);

This approach ensures all logging goes through a unified system that can handle log routing, formatting, and management consistently.

externalize configuration values

Configuration values, especially sensitive data like API keys and environment-specific settings, should be externalized to environment variables rather than hardcoded in source code. This prevents sensitive information from being permanently stored in git history and enables proper configuration management across different environments.

Hardcoded values create security risks and operational inflexibility. When values are committed to version control, they become part of the permanent history and may require credential rotation if exposed.

Example of the problem:

// Bad - hardcoded sensitive values
algolia: {
  appId: "Y8QU1SIVLP",
  apiKey: "484787878bcf6f4a26834105f0855fa3",
},
googleTagManager: {
  containerId: "GTM-59R6474K",
}

Better approach:

// Good - use environment variables
algolia: {
  appId: process.env.ALGOLIA_APP_ID || "default_value",
  apiKey: process.env.ALGOLIA_API_KEY || "default_value",
},
googleTagManager: {
  containerId: process.env.GTM_CONTAINER_ID,
}

This approach requires setting up proper environment variable management and updating deployment configurations, but provides better security and flexibility for different environments.

Concise documentation writing

Write documentation that respects user intelligence by being concise and avoiding over-explanation of obvious concepts. Assume users are “lazy but smart” - they don’t need explanations like “A thread title is a title of a thread” but do need clear directions to find what they need.

Avoid unnecessary step-by-step instructions like “click this, click that” unless referring to confusing external systems. Instead, provide the right directions and trust users’ ability to figure out basic interactions.

Adopt a tone that is:

Example of what to avoid:

### Setting Thread Title
A thread title acts as a name for your thread. It appears on the left side of the chat window, helping you easily navigate through your interactions.

Better approach:

### Setting Thread Title
Thread titles appear on the left sidebar for easy navigation between conversations.

When writing installation guides, focus on essential information rather than obvious steps. Provide clear directions to download locations and system requirements, but don’t walk users through basic file operations they already know.

Configuration structure clarity

Ensure configuration parameters are properly structured, accurately documented, and clearly indicate their scope and purpose. Configuration hierarchies should be explicit (global, assistant-level, thread-level), parameter categories should be well-defined (init vs runtime), and documented values must match actual system behavior.

Key practices:

Example structure:

{
  "parameters": {
    "init": {
      "model_path": "/path/to/model",
      "max_memory": "8GB"
    },
    "runtime": {
      "temperature": 0.7,
      "max_tokens": 2048
    }
  }
}

This prevents user confusion and ensures configurations work as documented across different scopes and contexts.

Externalize hardcoded configurations

Avoid hardcoding configuration values like ports, URLs, API endpoints, and environment-specific settings directly in source code. Instead, externalize these values to environment variables, configuration files, or dependency injection to improve maintainability and deployment flexibility.

Hardcoded configurations make applications difficult to deploy across different environments and create tight coupling between code and environment-specific values.

What to externalize:

Example:

// ❌ Avoid: Hardcoded in source
const PORT = 1337;
const LOCAL_HOST = "127.0.0.1";
const JAN_HTTP_SERVER_URL = `http://${LOCAL_HOST}:${PORT}`;

// ✅ Better: Use environment variables
const PORT = process.env.JAN_API_PORT || 1337;
const LOCAL_HOST = process.env.JAN_HOST || "127.0.0.1";
const JAN_HTTP_SERVER_URL = `http://${LOCAL_HOST}:${PORT}`;

Move configuration values to .env files, environment variables, or inject them from the application layer rather than defining them in core modules. This enables different configurations for development, testing, and production environments without code changes.

Document AI infrastructure requirements

When documenting AI applications that run local models or perform inference, always provide comprehensive infrastructure requirements including hardware specifications, platform-specific drivers, and setup instructions. This ensures users can successfully run AI models without encountering compatibility issues.

Include the following details:

Example documentation structure:

### Hardware Requirements
- **RAM/VRAM**: 8GB minimum (3B models), 16GB recommended (7B models)
- **CPU**: ARM, x86 architectures supported
- **GPU**: NVIDIA (via llama.cpp), AMD and Intel support coming soon

### Platform Setup
- **Windows**: Install NVIDIA drivers if GPU available
- **Linux**: Install CUDA Toolkit if GPU available

This prevents user frustration and ensures AI applications can run as intended across different hardware configurations.

consolidate build scripts

When creating multiple build variants (debug, publish, etc.), avoid duplicating build commands by reusing the base build script. This reduces maintenance overhead, ensures consistency across build targets, and shortens complex build commands.

Instead of duplicating the entire build process in each script:

{
  "build": "tsc -b . && webpack --config webpack.config.js",
  "build:publish": "tsc -b . && webpack --config webpack.config.js && rimraf *.tgz --glob && npm pack && cpx *.tgz ../../electron/core/pre-install"
}

Consolidate by referencing the base build command:

{
  "build": "tsc -b . && webpack --config webpack.config.js",
  "build:publish": "npm run build && rimraf *.tgz --glob && npm pack && cpx *.tgz ../../electron/core/pre-install"
}

This approach ensures that changes to the base build process automatically propagate to all build variants, reducing the risk of inconsistencies and making the build pipeline more maintainable.

leverage existing solutions

Avoid duplicating functionality by leveraging existing libraries and creating reusable components. Instead of redefining common utilities or creating similar components for each use case, use established libraries and build generic, reusable components that can be configured for different scenarios.

For example, rather than redefining a classNames utility:

// Avoid
function classNames(...classes: any) {
  // custom implementation
}

// Prefer
import classnames from 'classnames'

Similarly, create reusable components instead of duplicating similar functionality:

// Instead of separate DeleteModal, ConfirmModal, etc.
// Create one ConfirmationModal that accepts different props
const ConfirmationModal: React.FC<Props> = ({ 
  atom, title, description, onConfirm 
}) => {
  // Generic modal implementation
}

This approach reduces code duplication, improves maintainability, and ensures consistency across the codebase.

This site is open source. Improve this page.