# LLM Serving & Gateways
Inference servers, model gateways and routers, KV caching, retrieval and ingestion pipelines.
335 instructions from 15 repositories. Last updated 2026-07-29.
---
## Consistent Authorization and Auth
When changing or adding security-sensitive functionality, keep authorization semantics stable and use project-wide shared validation/auth plumbing.
Apply this standard:
1) Preserve existing access contracts
- If legacy data can’t map to new roles/permissions, keep the “null/legacy” behavior unless you intentionally migrate it with explicit tests for resulting scopes.
2) Validate authorization inputs during permission granting
- When a new permission relies on an external/related entity (e.g., Role tied to an invitation), confirm the role still exists and belongs to the expected project before granting.
3) Don’t introduce partial input-validation patterns
- For identity parameters (e.g., GUIDs), follow the existing project-wide convention; if type validation is desired, implement it once (e.g., in ConvertGUIDToInt or middleware/shared resolver wrapper), not ad-hoc in a subset of mutations.
4) Use shared token/auth providers and correct auth methods
- Prefer the shared OAuth TokenProvider abstraction; let the HttpClient derive headers from `httpReq.Auth` rather than manually mixing header logic. Ensure OAuth always uses the expected auth type (e.g., bearer) and API key auth uses the expected API-key mechanism.
5) Make security posture changes uniformly
- If you want to enforce HTTPS (or similar network security requirements), implement it across all external integrations/checkers consistently rather than patching one client.
Example pattern (authorization contract + validation) inspired by the discussions:
```go
// Preserve legacy behavior by only validating/applying role when role_id exists.
roleID := invitationRow.RoleID
if roleID != nil {
if _, err := client.Role.Query().Where(
role.IDEQ(*roleID),
role.ProjectIDEQ(invitationRow.ProjectID),
).Only(ctx); err != nil {
return fmt.Errorf("invitation role is no longer available")
}
}
// Otherwise: keep null/legacy semantics to avoid privilege escalation.
```
Net effect: fewer privilege-escalation surprises, fewer auth-header mismatches, and consistent security behavior across the codebase.
---
## Immutable Snapshots And Gates
When concurrency crosses subsystem boundaries (HTTP handlers, streaming, background workers, plugin hooks), treat shared state as immutable and coordinate lifecycle explicitly.
**Coding standards**
1) **Snapshot/clone before publishing**
- Never return or store pointers/maps/slices that remain aliases to internal catalog/store data that other goroutines may mutate.
- Prefer immutable snapshots and atomic swaps for hot-reload config (and clone any slice/map contents).
2) **Hold locks only for state access; release before blocking/calling out**
- Lock to take a consistent snapshot, then unlock before invoking callbacks, restarting goroutines, or doing any potentially blocking work.
- Design around lock ordering and avoid re-entrant lock reads.
3) **Make streaming/background lifecycle coordination explicit**
- Use dedicated gate/flusher lifecycle state and ensure cleanup ordering can’t delay client-visible release.
- Always wire cancellation into streaming fan-out and flusher drain paths; ensure paused/end/cleanup transitions are deterministic.
**Example pattern (immutable config swap)**
```go
type CorsMiddleware struct {
cfg atomic.Pointer[MyConfigSnapshot]
}
type MyConfigSnapshot struct {
AllowedOrigins []string
AllowedHeaders []string
}
func NewCorsMiddleware(cfg *MyConfig) *CorsMiddleware {
m := &CorsMiddleware{}
snap := &MyConfigSnapshot{
AllowedOrigins: slices.Clone(cfg.AllowedOrigins),
AllowedHeaders: slices.Clone(cfg.AllowedHeaders),
}
m.cfg.Store(snap)
return m
}
func (m *CorsMiddleware) UpdateConfig(cfg *MyConfig) {
snap := &MyConfigSnapshot{
AllowedOrigins: slices.Clone(cfg.AllowedOrigins),
AllowedHeaders: slices.Clone(cfg.AllowedHeaders),
}
m.cfg.Store(snap)
}
```
**Review checklist**
- Does this code publish (or store) any pointer to shared mutable structures across goroutines/requests?
- Are we cloning maps/slices/attribute maps that might be mutated later?
- Are we holding a mutex while calling out / restarting / doing I/O (risking deadlock or re-entrancy)?
- For streaming, do paused/end/cleanup transitions ensure deterministic flusher behavior and correct client release under cancellation?
- Do we have tests that actually exercise timing-sensitive paths (not only unit-level outcomes)?
---
## Deterministic DB Semantics
Make database changes (queries, migrations, and tests) explicit about shape and semantics: don’t rely on driver defaults, cached plan inference, or “looks right” reads when the DB can mask the truth.
Apply this as a checklist:
1) Prove correctness with real row effects
- If the DB has read-time transforms/dedup (e.g., ClickHouse ReplacingMergeTree + FINAL), a “exists” style read can be misleading.
- Prefer assertions that measure the actual stored cardinality for the scenario you’re testing.
2) Migrations must use explicit projections and live-schema guards
- Avoid `SELECT *` inside migrations where earlier DDL in the same run can change column sets/types.
- Build an explicit projection from schema, then intersect with physically present/live columns to avoid selecting missing columns.
- Never depend on cached plans’ result types; keep queries previously-uncached by using a fixed explicit column list.
3) Index/DDL changes should be concurrency- and transaction-safe
- If an index needs `CONCURRENTLY`, create it outside transactional migration blocks (or via a post-startup mechanism) to avoid blocking/invalid transaction constraints.
4) Update paths must match the monotonic invariants
- Guard predicates must be carefully chosen for each statement’s boundary semantics (e.g., `<` vs `<=` depending on whether unchanged boundaries must still persist).
- If order matters within a transaction (flush overrides before advancing a “last_reset/last_*” boundary), encode it and ensure the guards remain valid.
5) Deterministic batching
- When batching with `LIMIT`, add an explicit `ORDER BY` so each batch removes the intended rows.
- If the driver reports unusable affected-row counts, compute the metric from a deterministic pre-selection (e.g., pluck ids, then delete).
Example (deterministic batch delete with correct counting):
```go
func (s *ClickHouseLogStore) DeleteLogsBatch(ctx context.Context, cutoff time.Time, batchSize int) (int64, error) {
var ids []string
if err := s.db.WithContext(ctx).
Model(&Log{}).
Select("id").
Where("created_at < ?", cutoff).
Order("created_at ASC").
Limit(batchSize).
Pluck("id", &ids).Error; err != nil {
return 0, err
}
res := s.db.WithContext(ctx).Where("id IN (?)", ids).Delete(&Log{})
_ = res // affected rows may be driver-rewritten; rely on len(ids)
return int64(len(ids)), res.Error
}
```
Outcome: DB code and migrations become robust to schema drift, driver quirks, and boundary/dedup semantics—so correctness is both provable and repeatable across environments.
---
## Null semantics and safety
Define and enforce null/optional semantics consistently: (1) treat nil/zero/empty/absent as distinct states, (2) clear stale state when nil means “remove”, (3) never rely on plain `interfaceValue != nil` when the underlying value may be a typed-nil pointer, and (4) when using `omitempty`, only set pointer fields when the value is actually present.
Practical rules:
- **Clear, don’t skip:** if `nil` is documented as “clears”, ensure your code calls `ClearValue(...)` / resets pointers / zeroes receiver fields rather than leaving previous values intact (especially on reused long-lived contexts).
- **Safe nil detection:** if a value is stored as `interface{}` and may be a pointer, use comma-ok type assertions and/or safe extract helpers that reject typed-nil.
- Bad: `if x != nil { ... }` when `x` is `interface{}` holding a `(*T)(nil)`.
- Good: `v, ok := x.(*T); if ok && v != nil { ... }` or `SafeExtract...`.
- **omitempty correctness:** only take addresses / assign pointers when the underlying scalar is non-empty (so `omitempty` truly omits).
- **Preserve invariants and avoid redundant nil guards:** encode assumptions as comments and ensure dereferences are guarded by real construction-time invariants.
- **Reset reusable decoders/receivers:** when custom JSON unmarshalling preserves raw bytes, reset the receiver first to prevent stale preserved fields when decoding the same struct instance multiple times.
Example patterns:
**1) Clear context key on nil**
```go
func setCatalogOnContext(ctx *BifrostContext, catalog ModelInfoProvider) {
if ctx == nil { return }
if catalog == nil {
ctx.ClearValue(BifrostContextKeyModelCatalog) // nil means remove
return
}
ctx.SetValue(BifrostContextKeyModelCatalog, catalog)
}
```
**2) Reject typed-nil in interface{}**
```go
// ctx.Value(...) returns interface{}
if ctx != nil {
if ov, ok := ctx.Value(BifrostContextKeyProviderOverride).(*ProviderOverride); ok && ov != nil {
if ov.Key != nil { /* bypass key-pool */ }
}
}
```
**3) omitempty pointer only when non-empty**
```go
// server_label omitempty should omit when unset
var serverLabelPtr *string
if action.ServerLabel != "" {
serverLabelPtr = &action.ServerLabel
}
aux := struct{ ServerLabel *string `json:"server_label,omitempty"` }{ServerLabel: serverLabelPtr}
```
**4) Reset receiver before custom unmarshal**
```go
func (m *ResponsesMessage) UnmarshalJSON(data []byte) error {
*m = ResponsesMessage{} // prevent stale preserved fields
// ... decode; preserve rawToolSearch only for relevant types
return nil
}
```
Adopting these rules prevents the common failure modes shown above: stale pointers persisting across requests, incorrect nil checks with typed-nil interfaces, JSON fields that accidentally appear despite `omitempty`, and reused-struct decode bugs.
---
## Error Handling Consistency
When handling failures, make error semantics explicit and consistent across the stack:
- Define retryability precisely: only treat errors as retryable when you can guarantee it won’t duplicate already-delivered output (e.g., transport ended before a usable response). Add regression tests for each retry branch (positive + negative cases).
- Avoid redundant status checks: if your HTTP client wrapper already converts non-2xx/3xx to errors, don’t re-check `StatusCode` in the caller; reserve parsing/"could not parse" errors for successful responses.
- Keep error construction consistent with local conventions: use the project’s existing error formatting/wrapping approach (e.g., `fmt.Errorf`) rather than introducing new error packages/types without an established convention.
Example pattern (retry classification):
```go
func isRetryableTransportError(err error) bool {
if err == nil {
return false
}
if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
return true
}
var netErr net.Error
return errors.As(err, &netErr) && (netErr.Timeout() || netErr.Temporary())
}
```
And ensure callers only retry when the failure occurs before output is committed, with tests covering `Timeout`, `Temporary`, and the “neither set” negative case.
---
## Prevent Secret Leakage
Apply strict trust boundaries to authentication/credential state:
- **Reset per-request sensitive fields on reuse** (especially pooled request/contexts). Never let prior request overrides (keys, base URLs, auth headers) bleed into later requests.
- **Write-protect security-critical context keys.** Treat reserved/internal keys as framework-owned: plugins should not be able to set them via public setters. Use an internal-only bypass for framework methods.
- **Never persist redacted/masked placeholders as real secrets.** If an update payload contains a “masked preview” (redacted non-secret), require a non-empty stored counterpart; otherwise fail validation (400) instead of storing the placeholder.
- **Validate outbound targets that depend on credentials/config** to prevent SSRF (e.g., enforce HTTPS + allowlisted host patterns; validate again at dial time, not only at create time).
Example (masked placeholder preservation rule):
```go
// incoming may be a UI placeholder like "****"; treat it as non-secret.
if incomingVal.IsRedacted() && !incomingVal.IsFromSecret() {
if storedVal == nil || !storedVal.IsSet() {
return fmt.Errorf("masked preview requires stored value") // reject
}
incomingVal = *storedVal // preserve real secret
}
```
Also ensure pooled objects are cleaned:
```go
// before putting request back into a pool or before reusing it
req.ProviderOverride = nil // clear per-request overrides
```
These practices reduce the risk of credential corruption, SSRF, and cross-request authentication bypass—key failure modes in the security domain.
---
## Deterministic Test Hygiene
Adopt a consistent testing standard that prevents flaky behavior, leaked goroutines, and false positives.
**1) Always register cleanup for started resources**
- Prefer `t.Cleanup(...)` / `tb.Cleanup(...)` over scattered `defer`s.
- Use it for: clients, servers, accumulators/tickers, streaming helpers, and `Init(...)` results.
```go
bf, err := Init(ctx, cfg)
require.NoError(t, err)
t.Cleanup(func() { bf.Shutdown() })
```
**2) Make concurrency/time-sensitive tests deterministic**
- Avoid fixed sleeps like `time.Sleep(3 * time.Millisecond)` for synchronization.
- Use a bounded poll/wait with a deadline.
```go
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
if body.closeCount.Load() > 0 { break }
time.Sleep(2 * time.Millisecond)
}
stop(); cleanup()
```
**3) Ensure assertions are non-vacuous and validate invariants**
- Don’t rely on branches that may skip the assertion.
- Assert the exact behavior that should occur (e.g., “role dropped” or “snapshot deep-copied”).
```go
attachBilledUsageFromContext(ctx, bifrostErr)
// Mutate original after attach; snapshot must not change.
usage.PromptTokens = 999
require.Equal(t, 10, bifrostErr.ExtraFields.BilledUsage.PromptTokens)
```
**4) Keep mocks faithful to production side effects/commit semantics**
- If production “updates only on success,” mocks must not mutate global/shared state until validation passes.
- Avoid pointer-aliasing bugs where rollback logic doesn’t fully revert state.
```go
candidate := budgets[i] // copy
candidate.IsCalendarAligned = calendarAligned
if err := candidate.SetOverrideAt(...); err != nil { return nil, err }
budgets[i] = candidate // commit only on success
```
**5) Fail fast during test setup**
- When setup steps can fail (e.g., sending initial chunks), check return values immediately and `t.Fatalf(...)` with a setup-specific message.
- This prevents cascaded failures that obscure the root cause.
```go
if ok := a.GateSend(traceID, chunk0, false, false, ch, ctx); !ok {
t.Fatalf("setup chunk 0: GateSend returned false")
}
```
---
## Null and Omission Safety
When a config field is optional/nullable (or has multiple shapes), never rely on “default” or presence quirks. Make absence vs presence explicit, align schema rules with runtime parsing/normalization, and validate the structural inputs that runtime can actually interpret.
Apply these rules:
- Distinguish “key omitted” from “key present but empty/null”: if a branch should apply only when a field is explicitly empty, encode that with schema logic that won’t accidentally match omitted keys.
- Don’t assume JSON Schema `default` will populate runtime values: implement (and test) normalization during reconciliation/parsing so omitted fields resolve to the correct internal default, while explicit values like `0` are preserved.
- For object-typed variants, require the properties that runtime can’t resolve safely. If runtime parsing only succeeds inside a “value exists” gate, the schema should reject `{}`-like objects so you don’t fall through to unintended plaintext/empty literals.
- Keep “derived” fields out of the source-of-truth validation: if some fields are recomputed from anchors/other totals, enforce ordering/invariants at runtime where the derived value is computed.
Example (absence vs explicit empty mode):
- Avoid a plain `const: ""` if you want to reject cases where `override_mode` is omitted but other override fields are set.
- Instead, use explicit `if/then/else`/`not` logic that makes the empty-mode branch match only when the field is truly in the empty-state, not when it’s missing.
Example (defaults annotation vs runtime normalization):
- Treat schema `default` as documentation.
- Add reconciliation logic like:
- `if input == nil { resolved = 200 } else { resolved = input }`
- ensuring `0` stays `0` and omission becomes the intended default.
---
## Explicit React UI Behavior
When building React components, make user interaction and lifecycle behavior explicit—particularly for accessibility and ref/state initialization.
- Tooltips + disabled controls: If a disabled element can’t receive focus, put the focusable/hoverable semantics on a wrapper inside `TooltipTrigger asChild`, and use conditional `tabIndex` to avoid adding extra tab stops. Don’t duplicate ARIA text that the tooltip system already wires up.
- Ref/state initialization: Prefer initializing refs/state directly from props (or a stable initializer) rather than using “ref-seeding” `useEffect` that can depend on effect ordering.
Example (tooltip focus + conditional tabIndex):
```tsx
{isEnabled ? "Refresh list" : "Enable to refresh"}
```
Example (explicit ref init):
```tsx
const popupRef = useRef(initialPopup ?? null);
// Avoid useEffect(() => { popupRef.current = initialPopup ?? null }, [initialPopup])
```
---
## Semantic Naming Consistency
Adopt naming rules that accurately reflect meaning, routing behavior, and established codebase conventions.
Apply these checks when adding/changing code:
- Provider-specific vs provider-agnostic naming: If a helper operates on shared Bifrost schema types (e.g., `BifrostResponsesResponse`) and isn’t actually OpenAI-specific, avoid placing it in `openai/*` and avoid OpenAI-specific naming; move it to provider-agnostic utils and name it for the shared concept.
- Follow existing file/type conventions: For non-OpenAI providers, define the provider struct in the main `.go` file, and reserve `types.go` for provider-specific request/response DTO structs.
- Use the correct semantic field: Prefer the modern/non-deprecated field for provider attribution (e.g., routing/provider information) and ensure call sites use the populated source that matches the pipeline.
- Align sentinel names/docs with config: If you use a “disabled” sentinel, name it explicitly (e.g., `LiveModelsSyncDisabled`) and ensure docs and schema/config describe the same disable value semantics.
- Preserve identifier semantics: When building compound IDs used for routing, parse and detect already-present inference-provider/policy segments; don’t blindly prepend segments (prepending twice breaks routing).
- Type naming consistency: Name new types to match existing conventions (e.g., include the relevant parent/type prefix like `ChatAssistantMessageImage` for assistant-message image entries).
Example (compound ID duplication guard):
```go
first, modelOrName, found := strings.Cut(rawID, "/")
if found && isKnownInferenceProviderOrPolicy(first) {
// rawID already includes an inference-provider/policy segment; do not prepend again
} else {
// safe to prepend providerKey/inferenceProvider
m.ID = fmt.Sprintf("%s/%s/%s", providerKey, inferenceProvider, rawID)
}
_ = modelOrName
```
---
## Concurrency-safe retries
When concurrency is involved, ensure two things: (1) your tests/operations can’t accidentally contend on shared resources, and (2) any concurrency-handling you add (e.g., retrying on 409) is only applied to errors that are truly reachable from the concurrent scenario you’re mitigating.
Apply this standard as follows:
- Use run/worker-unique identifiers for any created resources (keys, records, filenames) so parallel CI runs can’t collide and corrupt each other’s cleanup.
- Don’t add “catch-and-retry” for lock/in-flight errors unless the backend guarantees that error is produced by the concurrent path you’re exercising. If the error guard is unreachable for that path, retrying will only mask real failures.
Example (test isolation + no blind retry):
```ts
import { randomUUID } from "crypto";
// Run-scoped suffix prevents cross-run collisions on shared backends.
const runId = randomUUID().slice(0, 8);
const keyName = `Refresh-Key-${runId}`;
// Prefer waiting for the UI/request to settle rather than retrying on errors
// you can’t actually produce from this code path.
await page.click(refreshBtn);
await expect(refreshBtn).toBeEnabled({ timeout: 15000 });
```
If you’re considering retries for a specific error code, first verify (via server logic/guards) that the code is reachable under the scenario you’re trying to handle; otherwise avoid swallowing it and fix the scenario instead.
---
## Deterministic test setup/teardown
E2E tests should be deterministic by (1) registering mocks/fixtures before any action that triggers network calls (e.g., `goto()`), and (2) handling cleanup failures in a way that doesn’t skip subsequent tests/hooks.
Apply these rules:
- Register mocks in a `beforeEach` that runs *before* navigation/side effects.
- Track created resources and attempt cleanup in `afterEach`.
- Avoid throwing inside `afterEach` when it can prevent later tests/hooks from running (especially with serial execution). Instead, record leaks and fail in `afterAll`.
Example pattern:
```ts
const created: { provider: string; keyName: string }[] = [];
const leaked: typeof created = [];
test.describe('Some e2e flow', () => {
test.describe.configure({ mode: 'serial' });
test.beforeEach(async ({ page }) => {
// Register mocks BEFORE navigation triggers API calls
await mockSomeApis(page);
await providersPage.goto();
});
test.afterEach(async ({ providersPage }) => {
for (const { provider, keyName } of [...created]) {
try {
await providersPage.selectProvider(provider);
const exists = await providersPage.keyExists(keyName, 2000);
if (exists) await providersPage.deleteKey(keyName);
} catch (e) {
leaked.push({ provider, keyName });
}
}
created.length = 0;
});
test.afterAll(() => {
if (leaked.length) {
throw new Error(`Leaked resources: ${leaked.map(k => `${k.provider}/${k.keyName}`).join(', ')}`);
}
});
});
```
This prevents cascading skips, ensures mocks reliably intercept navigation-time calls, and still surfaces real cleanup problems loudly at the end of the suite.
---
## Prefer safe tag invalidation
When a server mutation (e.g., model discovery refresh) changes data that feeds multiple cached selectors/queries, use RTK Query tag invalidation to refresh all dependent caches—especially indirect ones.
Guideline
- Invalidate every cache tag type that can become stale due to the mutation (not just the primary entity).
- If the mutation response can’t be cleanly mapped into the existing cached query shape(s), prefer invalidation over patching to avoid subtle shape-mismatch bugs.
- Invalidate provider-scoped entries as well, so list/detail views (and any key/model pickers) update consistently.
Example (pattern)
```ts
const refreshProviderModels = builder.mutation({
query: (provider) => ({
url: `/providers/${encodeURIComponent(provider)}/refresh-models`,
method: 'POST',
}),
invalidatesTags: [
'Models',
'DBKeys',
{ type: 'Providers' as const, id: provider },
],
});
```
Application checklist
- Identify what the refresh rewrites on the backend (e.g., per-key `models`) and which UI components depend on it.
- Add corresponding tag types to `invalidatesTags` for those dependencies.
- Avoid complex `updateQueryData`/patching unless the response can be mapped losslessly to the cached query’s existing item shape.
---
## Explicit Undefined Handling
When a value is intentionally absent (e.g., an optional callback prop or required classification fields), represent that absence explicitly and don’t let missing/nullable data silently pass as “valid”.
**Rules of thumb**
1. **Optional callbacks:** If there’s no behavior to run, prefer passing `undefined` (or omitting the prop) over a noop lambda—unless the component specifically requires a non-null function.
2. **Strict filtering:** When narrowing collections based on optional fields, require the expected properties. Don’t write predicates that treat missing fields as matching (e.g., `!m.type || ...`). This prevents unrelated items (with absent fields) from being included.
**Examples**
```tsx
// 1) Optional callback: use undefined instead of noop
// 2) Strict null/undefined-aware filtering
const sttModels = models.filter(
(m) => m.type === "audio" && m.subtype === "transcription"
);
// Avoid: (m) => !m.type || (m.type === "audio" && ...)
```
This approach reduces accidental behavior (no-op handlers) and avoids UI/data contamination caused by missing optional fields being treated as matches.
---
## Bounded, Targeted Performance
When optimizing performance, avoid whole-object work and unbounded resource growth. Apply these standards:
1) Targeted extraction/rewrites (no full parsing)
- If you only need one JSON field, read it with gjson/sonic.Get and rewrite with sjson, rather than unmarshalling the full body/request.
- Guard on JSON shape before using sjson.Set to avoid corrupting non-JSON/multipart bodies.
Example (safe “model” rewrite):
```go
func rewriteTopLevelModel(body []byte, alias, modelID string) []byte {
trimmed := bytes.TrimLeft(body, " \t\r\n")
if len(trimmed) == 0 || trimmed[0] != '{' { // multipart/form-data etc.
return body
}
cur := gjson.GetBytes(body, "model")
if cur.Type != gjson.String || !strings.EqualFold(cur.Str, alias) {
return body
}
out, err := sjson.SetBytes(body, "model", modelID)
if err != nil { return body }
return out
}
```
2) Bound buffering and “perpetual due” logic
- Any paused replay buffer, retry loop, or time-based scheduler must have hard limits (e.g., max bytes) and fail-closed handling for invalid/negative durations.
3) Coalesce duplicate concurrent work
- For expensive refresh/exchange operations, use singleflight (or equivalent) so only one goroutine performs the work per key while others wait.
4) Pooling hygiene (prevent allocation retention)
- If you use pooled objects, clear embedded request/state that retains large allocations before returning to the pool (do in-place zeroing; don’t rely on GC).
These practices together reduce CPU spent on unnecessary parsing/scanning, prevent memory blowups, and eliminate wasted concurrent work—directly improving throughput and latency.
---
## Preserve Aggregation Invariants
When transforming/aggregating streamed or ranged data, explicitly define the invariants your algorithm depends on (index meaning, ownership/copy semantics, boundary semantics, and ordering), then implement collision- and edge-case-safe logic with targeted tests.
Practical rules:
1) Specify index/span semantics and keep them consistent across paths.
- If downstream expects character offsets, don’t silently pass byte offsets; document and align conversions.
2) Make boundary/range aggregation match the “exact predicate” semantics of the raw path.
- If using pre-aggregated/materialized views, split the window into interior vs boundary buckets so first/last bars count only in-window rows.
3) Enforce collision-safe dedup/remapping in stream accumulators.
- Terminal/final markers are common collision points; reserve remapped indices deterministically on first final delivery and only remap on genuine collisions.
4) Prevent unintended aliasing in accumulators/transform layers.
- Clone slices/maps that might be mutated later; keep other fields shallow only when you can prove they’re immutable.
5) Preserve deterministic ordering when output bytes matter.
- Avoid Go map re-marshaling with sorted keys when providers expect literal key order; use ordered data structures and lock it with tests.
Collision-safe terminal-chunk handling (adapted from the discussed fix):
```go
if isFinalChunk && chunk.StreamResponse != nil {
if acc.TerminalResponseChunkIndex >= 0 {
chunk.ChunkIndex = acc.TerminalResponseChunkIndex
} else {
if chunk.ChunkIndex <= acc.MaxResponsesChunkIndex {
acc.MaxResponsesChunkIndex++
chunk.ChunkIndex = acc.MaxResponsesChunkIndex
}
acc.TerminalResponseChunkIndex = chunk.ChunkIndex
}
}
```
Test guidance:
- Add edge-case tests that reflect the violated invariants (e.g., monotonic streams where the final chunk arrives twice; mixed reasoning blocks where indices must not collapse; boundary histogram windows that split across the first/last bucket).
---
## Stable Backward-Compatible Migrations
When adding schema/config fields, design migrations to be stable across upgrade/downgrade boundaries:
- Handle “unset vs default” explicitly in config-store: store a sentinel/unset value (commonly `0`), ensure getters apply the real default (e.g., 15), and ensure config hashing only incorporates the field when it is explicitly set (non-zero). Add a test that proves no config churn for unset values.
- Use consistent migration patterns:
- Prefer helper patterns like `addColumnIfNotExists` / `dropColumnIfExists` and the repo’s column existence checks.
- Decide rollback policy based on intent:
- Additive-only schema changes (adding new columns, no data move/transform) should be reversible (rollback drops only the added columns).
- If a migration transforms/moves existing data (or intentionally relies on older binaries ignoring new columns), don’t pretend rollback can restore prior state—mark it non-rollbackable per existing precedent.
Example pattern for config hash stability (Go):
```go
// In GenerateClientConfigHash
if streamKeepAliveInterval != 0 { // only fold explicit values
h.WriteString(fmt.Sprintf("stream_keepalive_interval=%d", streamKeepAliveInterval))
}
// In getter (or load/default layer)
if storedInterval == 0 {
return 15 // default
}
```
Example pattern for reversible additive migrations:
```go
Migrate: func(tx *gorm.DB) error {
return addColumnIfNotExists(tx, logger, &tables.TableKey{}, "bedrock_project_id")
},
Rollback: func(tx *gorm.DB) error {
return dropColumnIfExists(tx, logger, &tables.TableKey{}, "bedrock_project_id")
},
```
Apply this standard anytime you change config-store schema, add new columns used by runtime behavior, or touch migration rollback behavior—aim for upgrades that don’t break existing rows or cause unintended hash/config regeneration, while keeping rollback aligned with what the migration actually changes.
---
## Semantic Merge And Deep Copy
When your code incrementally merges or inherits structured data (JSON payloads, rule trees, settings), correctness should be enforced by (a) semantic, lossless reconciliation and (b) deep-copying every nested mutable component.
Apply this standard:
1) For delta/stream/state reconciliation
- Treat the terminal value as canonical; allow equivalent representations (e.g., JSON key order/whitespace) but reject truly different semantics.
- Compute and forward only the missing suffix/delta to avoid overwriting accumulated content.
- Use lossless parsing for equality (e.g., preserve numeric precision with `json.Decoder.UseNumber()`), so equality checks don’t collapse distinct values.
- Fail explicitly on divergence to prevent emitting corrupted concatenated state.
Example (Go sketch):
```go
func reconcileDoneArguments(forwarded, done string) (string, error) {
// If done is empty, never overwrite accumulated deltas.
if done == "" { return "", nil }
// Prefer semantic equivalence over string equality.
eq, err := semanticJSONEqual(forwarded, done)
if err != nil { return "", err }
if eq { return "", nil } // nothing missing
// Otherwise, forward only missing suffix when done starts with forwarded.
if forwarded != "" && strings.HasPrefix(done, forwarded) {
return strings.TrimPrefix(done, forwarded), nil
}
return "", fmt.Errorf("divergent arguments")
}
func semanticJSONEqual(a, b string) (bool, error) {
var va, vb any
da := json.NewDecoder(strings.NewReader(a)); da.UseNumber()
db := json.NewDecoder(strings.NewReader(b)); db.UseNumber()
if err := da.Decode(&va); err != nil { return false, err }
if err := db.Decode(&vb); err != nil { return false, err }
return reflect.DeepEqual(va, vb), nil
}
```
2) For inheritance/transformations via cloning
- Any “rule” object you copy should be treated as a tree: deep-copy the entire nested structure (e.g., `When` conditions and any nested condition nodes), not only the top-level struct.
- Ensure nested slices/maps are copied (create new backing arrays/maps) before mutation.
- Avoid pointer aliasing between original and inherited objects; aliasing will cause future changes in one to silently affect the other.
Checklist before merging PRs:
- Merge/reconcile: Are you comparing semantics with lossless parsing? Are you computing deltas/suffixes instead of overwriting? Do you explicitly error on divergence?
- Clone/inherit: Are all nested nodes and collections deep-copied so inherited mutations can’t affect originals?
---
## Preserve API Contracts
Maintain deterministic, provider-compatible API contracts by (1) applying overrides explicitly and attempt-scoped (no hidden side effects), and (2) preserving exact request/response serialization expectations—especially where providers are sensitive to JSON schema structure/order or where passthrough/raw modes must not be rewritten.
How to apply:
- Overrides/routing: clear any previous attempt state before re-applying request-derived overrides; only allow plugins/hooks to inject routing via the supported Update* APIs (not by directly setting reserved context keys).
- Endpoint/route gating: only parse/extract request fields when you’re sure of the endpoint + content-type; skip admin routes, multipart, and other non-target shapes.
- Raw passthrough modes: when a request is passed through “raw body,” don’t rewrite or inline URLs based on typed structs.
- Serialization contracts: for structured outputs, don’t use plain map decoding/re-encoding when key order matters—use an OrderedMap (or equivalent) and normalize through the same ordered representation end-to-end.
Example (attempt-scoped override wiring):
```go
preReq := resultFromHooks() // may contain ProviderOverride
ctx.ClearValue(schemas.BifrostContextKeyProviderOverride)
if preReq != nil && preReq.ProviderOverride != nil {
ctx.SetValue(schemas.BifrostContextKeyProviderOverride, preReq.ProviderOverride)
}
```
Example (schema contract preserving order):
```go
type ResponsesTextConfigFormatJSONSchema struct {
Name *string `json:"name,omitempty"`
Schema *OrderedMap `json:"schema,omitempty"` // preserves JSON key order
}
```
---
## Backend Enforcement, UI Validation
Treat the frontend as an *UX layer*, not an authorization or confidentiality enforcement boundary. Any access control (licensing, revoke/re-auth, hidden-content guarantees, RBAC) must be enforced by the backend/data layer; the UI may be redundant or fail-open without granting access. Separately, whenever the UI must render untrusted values into security-sensitive contexts (e.g., hyperlinks), validate/allow-list before using them.
Apply this as a standard:
- **Authorization/confidentiality:** must be enforced in server middleware, RBAC/DAC checks, and/or storage/query logic. UI conditions are allowed for user guidance but never as the source of truth.
- **Destructive or identity-sensitive actions:** ensure the backend’s scope/authorization is correct; don’t rely on hiding the button or owner-gating in React.
- **Defense-in-depth:** it’s acceptable to repeat RBAC checks in nested routes (consistent with app style), as long as backend enforcement remains the ultimate gate.
- **Untrusted rendering:** before using any user-/server-provided string in `href` (or other executable/sensitive contexts), apply an allow-list policy and fall back to safe rendering (e.g., plain text).
Example pattern (safe URL rendering):
```ts
const uri = deviceState.verificationUri;
const safe = /^https:\/\/([a-z0-9-]+\.)*github\.com(\/|$)/i.test(uri);
return safe ? (
Open link
) : (
{uri}
);
```
If you’re tempted to add a UI-only `if (shouldHide) return null`, confirm the backend/storage already prevents the underlying data/action; otherwise, move or add the enforcement where it can’t be bypassed.
---
## Explicit Config Overrides
Implement configuration in a way that is predictable: define defaults centrally, keep early rollout scope narrow (prefer deployment-level defaults), and make override semantics explicit.
Apply this standard:
1) Set defaults in the conf package (single source of truth). Avoid scattering “fallback” values across business logic.
2) Use deployment-level defaults first for new config knobs; defer per-channel/UI/schema expansion until there’s product/runtime backing.
3) For any “override” configuration, choose and document semantics clearly.
- If the override is meant to fully take over for the period, treat it as full replacement: only the explicitly configured item codes apply; all others become 0 (or equivalent “unset” behavior), rather than silently merging.
Example (full-replacement override pattern + deploy-level default):
```go
// conf package should provide defaults, not business logic.
// e.g., conf.DefaultXXX
func codexImageMainModel() string {
// deployment-level default (via env) only; keep scope limited initially
if v := strings.TrimSpace(os.Getenv("AXONHUB_CODEX_IMAGE_MAIN_MODEL")); v != "" {
return v
}
return defaultImageMainModel
}
func effectiveItems(now time.Time, schedule *Schedule, base Items) Items {
if schedule == nil {
return base
}
if override := findMatchingOverride(now, schedule); override != nil {
// full replacement: override.Items is the complete effective set
return override.Items
}
return base
}
```
Outcome: fewer surprises during runtime, easier debugging, and safer incremental rollout of configuration-driven features.
---
## Graceful config validation
When configuration has conditional dependencies, make the system fail fast only for cases that break invariants (correctness/safety), but allow secure graceful degradation for “optional feature” toggles when prerequisites may be missing.
Apply this rule:
1) Enforce bidirectional relationships in schema
- If the meaning of one field depends on another (e.g., `auth_type`), use JSON Schema conditionals so invalid combinations are rejected rather than silently ignored.
- Prefer `allOf` with focused `if`/`then` blocks to avoid partial validation gaps.
Example (JSON Schema conditional enforcement):
```json
{
"allOf": [
{
"if": {"properties": {"auth_type": {"const": "per_user_headers"}}, "required": ["auth_type"]},
"then": {"required": ["per_user_header_keys"]}
}
]
}
```
2) Don’t turn “safe no-ops” into hard failures
- For runtime-dependent options (e.g., requires object storage), if the runtime behavior is explicitly safe (e.g., “drop content, never leak”), keep schema validation non-fatal.
- Implement a one-time warning and fall back to the secure default instead of failing startup/validation.
Decision heuristic:
- If misconfiguration could lead to incorrect/unsafe behavior → enforce in schema (fail validation).
- If misconfiguration only disables functionality and the system guarantees secure fallback → allow it, warn, and degrade gracefully.
---
## Strict auth schema validation
Security-related Helm/config schemas should fail fast on invalid or unintended authentication settings.
Apply:
- Close security-sensitive objects with `additionalProperties: false` so typos/mistyped fields are rejected at `helm template`/validation time (instead of silently producing insecure runtime behavior).
- When auth is enabled, model credentials as required structured fields (e.g., require `credentials.name` and `credentials.key` when auth is on).
- Reuse shared, closed definitions for repeated security sub-structures (e.g., relabeling blocks) so misspellings like `sourceLables` don’t slip through.
- Ensure provider-scoped security flags are only accepted for intended providers: don’t let permissive parent schema parts (e.g., `base_key`) implicitly allow unrelated security flags.
Example pattern (ServiceMonitor-style strict auth):
```json
{
"authorization": {
"type": "object",
"description": "Bearer-token auth for scrape",
"additionalProperties": false,
"properties": {
"type": {"type": "string"},
"credentials": {
"type": "object",
"additionalProperties": false,
"properties": {
"name": {"type": "string"},
"key": {"type": "string"}
},
"required": ["name", "key"]
}
},
"required": ["type", "credentials"]
}
}
```
Outcome: invalid auth configs (typos, missing required credential fields, or flags applied to the wrong provider) are rejected early, preventing insecure or broken authentication behavior at runtime.
---
## Stability-First Caching
When caching affects externally visible payloads (prompts, auth/JWT inputs, token summaries), ensure the *serialized representation used for cache identity is deterministic* and the cache has an *explicit validity boundary*.
Practical rules:
- **Never rely on Go map iteration order** for data that participates in caching (e.g., JSON used as cache key/prompt identity). Use a deterministic approach:
- either **preserve ordered structures** end-to-end (e.g., OrderedMap),
- or **deterministically merge** by inserting keys in sorted order.
- **Keep cache ownership and invalidation triggers consistent** with existing conventions:
- cache-first only when the in-memory/cache store is the intended hot-path source;
- evict only when the *downstream system* signals invalid state (e.g., 401/403), not on local token-acquisition errors.
- **Cache hot-path derivatives as immutable snapshots** where possible (e.g., fingerprint provider config paths to avoid per-request file reads) and document that changes at the same path require the designed reload/invalidation mechanism.
- **When reconstructing fields from cached/denormalized data, scope to what your query actually selected**—don’t “fill in” details that weren’t loaded.
Example: deterministic merge without map-key-order instability
```go
func mergeDeterministic(base map[string]any, fallback map[string]any) map[string]any {
if len(base) == 0 && len(fallback) == 0 {
return nil
}
merged := make(map[string]any, len(base)+len(fallback))
keys := make([]string, 0, len(base)+len(fallback))
seen := make(map[string]struct{}, len(base)+len(fallback))
for k := range base {
if _, ok := seen[k]; !ok {
seen[k] = struct{}{}
keys = append(keys, k)
}
}
for k := range fallback {
if _, ok := seen[k]; !ok {
seen[k] = struct{}{}
keys = append(keys, k)
}
}
sort.Strings(keys)
for _, k := range keys {
if v, ok := fallback[k]; ok {
merged[k] = v
} else {
merged[k] = base[k]
}
}
return merged
}
```
If the cached payload must preserve ordering semantics (e.g., `output_config`), prefer an ordered representation (OrderedMap) instead of converting to `map[string]any` during merges.
---
## Follow test constraints
When adding UI logic or tests, align with the repository’s existing CI/test guardrails—especially global checks and test discovery conventions.
- **Respect cross-cutting test rules (e.g., i18n parity):** Don’t introduce changes that require mass updates across all locales/fixtures unless you’re prepared to complete them. If full compliance is out of scope, keep the functional change minimal and defer complete i18n work to a focused follow-up.
- **Follow existing test organization/discovery patterns:** Place new tests where Vitest/CI already expects them (e.g., co-located `__tests__` for dashboard components if the repo’s `vitest.config.ts` includes that path). Avoid reorganizing test locations unless you also update imports and discovery configuration.
Example (i18n guardrail pattern):
```tsx
// Prefer keeping the diff focused when full i18n key rollout is out of scope.
{condition && (
// Inline callout instead of adding en-only keys that break locale-parity tests.
...
)}
// Then open a follow-up PR to properly add i18n keys across required locales.
```
Example (test structure pattern):
```ts
// Keep dashboard component tests co-located under the component's __tests__
// so existing Vitest include globs discover them consistently.
// Put your new *.test.tsx next to the component tests rather than in a different folder
// unless the repo-wide convention is being intentionally changed.
```
---
## Fail Safe Error Propagation
Establish an error-handling standard that prevents crashes/hangs, preserves root errors, and makes recovery bounded and verifiable.
Guidelines:
- Catch safely: prefer `catch (err: unknown)` and narrow before reading fields.
- Propagate failures to consumers: for streams/transforms, call `controller.error(err)` (not enqueue/continue) and perform deterministic teardown/abort.
- Fail safe on corrupted/unknown inputs: when runtime data may be corrupted, never throw from “read/execute” paths; fall back to a default and keep stricter validation only on writes.
- Bounded fallback/retry: only retry transient connection/dispatcher errors; cap attempts; respect remaining timeout budget.
- Retry guard for request bodies: if the body is non-replayable (stream/Blob), do not retry.
- Don’t shadow the root error: cleanup/rollback failures must not replace the original exception.
- Don’t silently misclassify: when a status code is ambiguous (e.g., 403), decide validity based on explicit evidence (message/body patterns) and cover with tests.
Example (SSE transform fail propagation + streaming semantics):
```ts
return new TransformStream({
transform(chunk, controller) {
try {
// parse/process...
} catch (err) {
// Propagate and abort the SSE pipeline
controller.error(err);
}
}
});
```
Example (guarded retry with non-replayable body check):
```ts
const bodyUnknown = options.body as unknown;
const isBodyStream =
bodyUnknown && typeof bodyUnknown === 'object' &&
(typeof (bodyUnknown as any).getReader === 'function');
const maxAttempts = isBodyStream ? 1 : 2;
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await undiciDirect(input, { ...options, dispatcher });
} catch (e) {
const code = (e as { code?: string }).code;
if (code === 'ECONNREFUSED' || String(e).includes('fetch failed')) continue;
throw e;
}
}
```
---
## Deterministic Cache Keys
Cache keys must be built only from inputs that truly change model output, and must be deterministic, complete, and cheap to compute.
**Rules**
1. **Derive keys from a stable message “anchor”.** If using conversation head hashing, explicitly define which roles and how many bytes you consider (e.g., contiguous leading `system`/`developer`, plus first `user`). Keep later turns from changing the head anchor when that’s intended.
2. **Include all message modalities in the fingerprint.** Don’t hash only `text`; include non-text parts (image/video/document/audio/etc.) using a consistent scheme. When hashing variable-sized units, include **length prefixes** (or other unambiguous framing) so different payloads that share a prefix don’t collide.
3. **Do not let unrelated metadata affect cache hits.** If you inject IDs (user/session/order IDs, fake IDs, routing metadata), ensure caching is driven solely by the message content (and any explicitly required rule/settings signatures).
4. **Keep cache-key work lightweight on hot paths.** Avoid full JSON serialization during cache lookup. For invalidation due to developer-rule/settings changes, add a **compact per-field signature** (e.g., FNV or similar) computed from only the fields that affect candidate/routing behavior.
**Sketch (Go-like pseudocode)**
```go
// Cache key inputs: (1) stable message anchor, (2) cheap settings/rules signature.
func cacheKey(messages []Message, rulesSig uint64, sessionID string) string {
anchor := conversationAnchor(messages) // includes text + non-text parts with framing
return fmt.Sprintf("%s|%d|%s", anchor, rulesSig, sessionID)
}
func conversationAnchor(messages []Message) string {
// Hash only: leading system/developer + first user (bounded by anchorMaxBytes)
// For each message: role + delimiter + all content parts.
// For each part:
// - if text: write length prefix + text
// - if non-text: write part kind + length prefix + URL/data bytes
// Ensure deterministic ordering and framing.
}
```
**Outcome**
- Correctness: different prompts (including non-text variations) don’t share the same cached response.
- Performance: key computation stays in the hot path without heavy serialization.
- Hit rate: unrelated metadata changes don’t bust caches unexpectedly.
---
## Low-noise, instance-safe logs
When adding or modifying logging, follow these rules:
1) **Attribute logs to the right component/instance**
- Avoid process-global logging hooks/state when a per-instance logger exists.
- Thread the instance-bound logger into call paths so warnings are correctly attributed.
2) **Pick the right level and avoid noise**
- Use `Debug` for informational-but-non-actionable signals that can be frequent.
- For streaming or chunked flows, **gate warnings** so they emit once per logical event (e.g., only on final chunk) rather than per chunk.
3) **Never silently fall back**
- If you fall back to a default behavior due to an error (e.g., URL parse failure), emit a warning with enough context to diagnose.
4) **Use the logging API’s formatting instead of `fmt.Sprintf`**
- Prefer `logger.Warn("...: %v", err)` over `logger.Warn(fmt.Sprintf("...: %s", err))`.
5) **Don’t bloat request/context just to move logs**
- Don’t stash ad-hoc log payloads in request context solely for later retrieval.
- Use existing streaming/middleware plumbing (e.g., dedicated completer slots) to move data explicitly.
Example patterns:
```go
// 1+3) Instance-safe warning on fallback
if err != nil {
logger.Warn("oauth: url parse failed, falling back to direct client: %v", err)
}
// 2) Correct level + log once (stream-aware)
if isStream && !isFinalChunk {
// skip warning
} else {
logger.Debug("cache: skipping write (namespace=%s, id=%s): no embedding available", ns, id)
}
// 4) Prefer format args
logger.Warn("logstore: skipping index maintenance: could not acquire index lock: %v", err)
// 5) Avoid reading logs back from fasthttp context; use dedicated slot/completer
// e.g., middleware publishes into a known slot, handler drains it after stream completes.
```
Applying this consistently reduces production log spam, preserves accurate attribution, improves diagnostics when fallbacks occur, and keeps logging implementation clean and maintainable.
---
## Enforce API Contracts
When designing or extending APIs, ensure the request/response schema and behavior strictly preserve invariants and provider compatibility.
Apply these rules:
1) Don’t expose unsafe state transitions
- If a field (e.g., status) must be controlled by an internal state machine, do not add GraphQL mutations/resolvers that allow clients to set it directly—even if the input object contains the field.
2) Use explicit, strongly-typed API models for configuration/mappings
- Prefer dedicated GraphQL types/inputs over generic JSON scalars for mapping objects.
- Validate invariants at the API boundary (e.g., duplicate keys) and ensure the field is reliably returned/persisted (avoid “saving others then dropping mapping” scenarios).
Example (schema shape for typed mappings):
```graphql
input ReasoningEffortMappingInput {
from: String!
to: String!
}
type ReasoningEffortMapping {
from: String!
to: String!
}
input UpdateChatSettingsInput {
reasoningEffortMapping: [ReasoningEffortMappingInput!]
}
```
3) Preserve the intended API lifecycle across fallback paths
- If you add auto-aggregation or fallback handling, do not accidentally bypass required middleware phases. Streaming-phase hooks should still execute when the stream is processed.
4) Match third-party spec requirements exactly and guard provider-specific behaviors
- Include fields even when they are zero if the provider spec requires them.
- Apply provider-specific selectors/decorators only when the negotiated API format supports them (use an explicit condition/guard).
---
## Config-Aware Startup Failures
When adding startup preflight checks (permissions, filesystem writeability, required directories), treat failure as part of error-handling policy: be precise, actionable, and configuration-aware.
**Standard**
1. **Fail fast only when truly required.** If the app is configured to use external stores or otherwise won’t write to `APP_DIR`, don’t crash-loop just because the directory is read-only.
2. **Provide an escape hatch for intentional exceptions.** If you perform a writeability probe, allow an explicit override (e.g., `BIFROST_SKIP_WRITE_CHECK=1`) to downgrade `exit 1` to a warning in known-safe scenarios.
3. **Make errors actionable.** When exiting, print what was checked (owned UID:GID, mode) and concrete fixes (own the UID, group-writable for supplemental group like 0, set `podSecurityContext.fsGroup`).
4. **Avoid pathological control flow.** For shell argument parsing/loops, ensure conditions and iteration always make progress (don’t keep `$#` constant while “looping” via `set -- ...; shift`).
**Example pattern (self-contained)**
```sh
APP_DIR=${APP_DIR:-/app/data}
SKIP_WRITE_CHECK=${BIFROST_SKIP_WRITE_CHECK:-0}
app_dir_writable() {
probe="$APP_DIR/.write-test.$$"
if [ -e "$probe" ]; then probe="$probe.$(date +%s)"; fi
mkdir "$probe" 2>/dev/null && rmdir "$probe" 2>/dev/null
}
ensure_app_dir() {
mkdir -p "$APP_DIR" 2>/dev/null || true
[ -d "$APP_DIR" ] || { echo "Error: cannot create $APP_DIR"; exit 1; }
if [ "$SKIP_WRITE_CHECK" = "1" ]; then
echo "Warning: skipping $APP_DIR write check (BIFROST_SKIP_WRITE_CHECK=1)"
return 0
fi
if ! app_dir_writable; then
uid=$(id -u); gid=$(id -g)
data_uid=$(stat -c '%u' "$APP_DIR" 2>/dev/null || echo 0)
data_gid=$(stat -c '%g' "$APP_DIR" 2>/dev/null || echo 0)
echo "Error: $APP_DIR not writable by UID:GID $uid:$gid (owned by $data_uid:$data_gid)"
echo "Fix: make it owned by this UID or group-writable for a supplemental group (e.g., GID 0), or set podSecurityContext.fsGroup"
exit 1
fi
}
```
Applying this standard reduces crash-loops, makes failures diagnosable, and prevents “strict checks” from blocking valid deployments where writes are unnecessary.
---
## Document behavior precisely
When updating documentation, ensure every statement about requirements, capabilities, and behavior is (1) correct for the current implementation, (2) explicit about when the requirement matters (e.g., render-time vs apply-time vs run-time), (3) consistent across related doc artifacts, and (4) does not imply unsupported configuration paths. Follow house style for headings/navigation (e.g., tab names).
How to apply:
- Check “point of impact” phrasing: state prerequisites at the moment they affect the user’s workflow.
- Remove stale/changelog/README prose that no longer matches current guards/behavior; reword to match actual enforcement.
- Keep duplicated sections in sync (if the repo uses byte-identical entries, update all copies identically).
- Don’t add documentation for features that the schema/product doesn’t support (avoid implying a config.json mechanism when it isn’t available).
- Match house style for UI labels and section/tab names.
Example (pattern to follow when prerequisites only matter at apply-time):
- Good:
“Set `serviceMonitor.enabled: true` to render a `ServiceMonitor` targeting the `http` service port. The `monitoring.coreos.com/v1` CRDs must be present when the manifest is applied; rendering itself does not require them.”
- Avoid:
“CRDs are required to render,” if rendering succeeds without them.
---
## Auth config documentation
When Helm (or other config) controls authentication for a protected endpoint, treat credential source, key names, and auth enablement paths as security-sensitive behavior. Ensure both the implementation and the accompanying docs/changelog precisely describe:
- **Auth enablement outcome**: what happens to unauthenticated requests (e.g., `401` leading to Prometheus `up=0`).
- **Credential inheritance vs explicit override**:
- Default behavior should state whether credentials and **their key names** are inherited from an auth configuration.
- Any `existingSecret`/override should state that it **fully overrides** inherited secret identity and key mapping (i.e., keys come from the override’s `usernameKey`/`passwordKey`, not from the auth config’s key names).
- **Validation step**: verify documentation claims by inspecting the *rendered output* (e.g., via `helm template`) and confirming the exact Secret names and key fields.
Example validation approach:
```bash
helm template rel . \
--set serviceMonitor.enabled=true \
--set serviceMonitor.basicAuth.enabled=true \
--set serviceMonitor.basicAuth.existingSecret=scrape-creds \
--set bifrost.authConfig.existingSecret=bifrost-secrets \
--set bifrost.authConfig.usernameKey=admin-username \
--set bifrost.authConfig.passwordKey=admin-password
# Confirm rendered basicAuth uses scrape-creds with key names
# (e.g., username/password), not inherited admin-username/admin-password.
```
This prevents misleading security documentation that can cause operators to scrape with wrong credentials, misunderstand failure modes, or break credential rotation assumptions.
---
## Validate Config Inputs
Any configuration derived from environment variables or external config must be (1) wired to the correct runtime fields, (2) normalized/validated with safe defaults, and (3) applied with deterministic precedence.
Practical standards:
- **Normalize env-derived values**: e.g., for hostnames, rely on URL parsing normalization but defensively normalize case.
- **Validate numeric env params**: parse with `Number.isFinite(...)` and enforce sane bounds; fall back to a documented default if invalid.
- **Avoid hardcoding runtime ports/URLs**: derive from env with a clear fallback chain.
- **Enforce config precedence deterministically**: when mixing hardcoded and DB-driven rules, define an explicit order (e.g., provider-level denylist → model-level denylist → allowlists) and ensure allowlist restoration only reintroduces keys that existed in the original request snapshot.
- **Prevent mismatched config shapes**: when config is “flattened” before reaching your class/function, assert the exact runtime fields you consume.
Example (env validation + deterministic deny/allow precedence):
```ts
// 1) Env numeric validation with fallback
const parsed = parseInt(process.env.PRICING_SYNC_INTERVAL ?? "86400", 10);
const intervalMs = Number.isFinite(parsed) && parsed > 0
? parsed * 1000
: 86400 * 1000; // 24h fallback
// 2) Snapshot-based allowlist restoration
function applyFilters({ provider, model, body, snapshot, cfg }: {
provider: string; model: string; body: Record; snapshot: Record; cfg: any;
}) {
// denylist: provider
for (const k of cfg.block ?? []) delete body[k];
// denylist: model
for (const k of (cfg.models?.[model]?.block ?? [])) delete body[k];
// allowlist: provider (restore only keys that were present originally)
for (const k of (cfg.allow ?? [])) if (k in snapshot) body[k] = snapshot[k];
// allowlist: model (most specific wins by applying last)
for (const k of (cfg.models?.[model]?.allow ?? [])) if (k in snapshot) body[k] = snapshot[k];
}
```
Adopt this pattern whenever config impacts runtime behavior (HTTP targets, auth payloads, request parameter shaping, sync intervals/feature flags): fail safely, default predictably, and keep behavior order explicit.
---
## Group-aware Pagination
When the client needs a logical unit (run/history group, full resource state), the API must paginate and update in a way that preserves that unit across requests.
Apply this standard:
1) **Pagination must align to domain groups**
- If records must be reconstructed into “runs” (or any parent/group), **paginate by the group key** (e.g., `webhook_id`/run) and return the group’s children together.
- Don’t rely on client-side heuristics to reassemble groups when groups can straddle page boundaries; that will be incorrect whenever boundaries split a group.
Example contract shape (conceptual):
- Instead of: `GET /deliveries?limit=25&offset=...` returning *attempts*
- Prefer: `GET /deliveryRuns?endpointId=...&limit=...&cursor=...` returning runs:
```ts
type DeliveryRun = { runId: string; webhookId: string; attempts: WebhookDelivery[] }
type DeliveryRunsResponse = { items: DeliveryRun[]; nextCursor?: string }
```
2) **Update semantics must be explicit + concurrency-safe**
- If the API uses full-state `PUT` without a revision/ETag, document that **it is overwrite-by-replace** and that clients must send the complete resource state.
- If concurrent edits/updates are possible, support **optimistic concurrency** (e.g., `ETag` + `If-Match`) so clients can avoid accidental lost updates.
3) **Don’t export incorrect invariants to the UI**
- The UI should be able to trust that what it receives is already consistent for the domain view (e.g., each run is whole), rather than attempting best-effort reconstruction across pagination and polling offsets.
---
## Explicit Null Safety
{% raw %}
When templating, explicitly distinguish three states: **missing**, **explicit false**, and **null**. Relying on truthiness can silently drop intended configuration.
**Apply two patterns:**
1) **Preserve explicit `false`** (boolean fields)
- Don’t use `with` / truthy checks for booleans if `false` must still render.
- Instead, render based on **key presence**.
```gotemplate
{{/* BAD: with treats false as empty */}}
{{- with .Values.serviceMonitor.honorLabels }}
honorLabels: {{ . }}
{{- end }}
{{/* GOOD: preserves explicit false */}}
{{- if hasKey .Values.serviceMonitor "honorLabels" }}
honorLabels: {{ .Values.serviceMonitor.honorLabels }}
{{- end }}
```
2) **Omit fields via real null (Helm values)**
- If you need to remove defaulted keys so downstream controllers (e.g., SCCs) can assign values, override with **`null`** (and confirm rendered output).
- Use `helm template` to verify that `runAsUser: null` / `fsGroup: null` (or similar) do **not** appear in manifests.
```yaml
# values.yaml override example
podSecurityContext:
runAsUser: null
fsGroup: null
securityContext:
runAsUser: null
```
**Rule of thumb:**
- If `false` must be honored → **check `hasKey`**.
- If a value must be removed from rendered YAML → **override with `null`** and verify with `helm template`.
{% endraw %}
---
## Efficient hot-path discipline
Adopt a performance-safety standard: prevent unbounded state growth in monitoring, and keep critical-path CPU/memory work linear and avoidable.
Apply these rules:
1) Treat performance instrumentation as state with lifecycle
- If you create marks/measures per request/stream, ensure they don’t accumulate indefinitely.
- If downstream consumers rely on marks being observable right after return, document and test that behavior; otherwise prefer “clear after creation/consumption” to bound memory.
2) Keep streaming/decoding parsers linear
- Avoid re-concatenating/rehashing the entire buffer on every `data` event.
- Use a rolling buffer + cursor offset / slicing so each byte is processed once.
3) Precompute repeated work in bounded loops
- Tokenization/scoring: compute invariant sets once per request (e.g., context tokens) and reuse for each candidate.
- Policy lookups: precompute maps at module load for O(1) repeated checks.
4) Add cheap guards before expensive transforms
- Before regex/whitespace stripping/base64 decode, do a fast size check to fail early and avoid CPU burn.
5) Gate unnecessary upstream calls
- Only do quota/usage preflights when at least one enforcement condition is active; otherwise skip the upstream fetch to reduce latency and load.
Example patterns (self-contained):
// 1) Bounded performance marks
performance.mark("omni-request-body-size", { detail: bodySize });
// ... read/observe in your monitoring pipeline ...
performance.clearMarks("omni-request-body-size");
// 2) Rolling buffer (linear-time scanning)
const chunks: Buffer[] = [];
let processed = 0;
req.on("data", (chunk) => {
chunks.push(Buffer.from(chunk));
const buffer = Buffer.concat(chunks);
// process from processed.. and advance processed cursor
// then drop processed prefix to keep future work bounded
});
// 3) Precomputed lookup
const unsupportedParamsByModel = new Map();
// module init fills map
function getUnsupportedParams(modelId: string) {
return unsupportedParamsByModel.get(modelId) ?? [];
}
// 4) Cheap guard before expensive work
if (payload.length > MAX_BYTES * 2) throw new Error("too large");
// only then do regex/strip/decode
// 5) Latency gate
if (!hasAnyEnforcementCondition) return; // skip upstream preflight
---
## Lifecycle-aligned Cancellation
When handling concurrent async work (streaming, tees, semaphores/limits, watchdogs), ensure cancellation, timeouts, and accounting are tied to the *actual* consumer lifecycle—release/decrement/teardown at the point where the work is truly finished, and don’t block on cancellation promises that can’t resolve until another branch drains.
Practical rules:
- **Don’t await cancellation of detached/tee branches** if it may remain pending until another consumer drains; cancel and proceed.
- **Release concurrency slots only after the stream is fully consumed** (e.g., `TransformStream.flush`/end-of-stream), not when headers/initial chunks arrive.
- **Prevent double accounting**: if you decrement pending counters at multiple phases (e.g., flush + handler), gate with a flag or ensure only one phase decrements.
- **Use abort timeouts for network calls** and merge with any provided AbortSignal so requests can’t hang indefinitely.
- **Guard watchdog math**: if the baseline timestamp/listener wiring can be missing, seed it or skip instead of computing stall against `undefined`.
Example pattern (release on flush, not header arrival):
```ts
const slot = await acquireSemaphore();
let released = false;
const releaseOnce = () => {
if (released) return;
released = true;
slot();
};
return new TransformStream({
transform(chunk, controller) {
controller.enqueue(chunk);
},
flush() {
// stream fully consumed (client drained)
releaseOnce();
}
});
```
These practices reduce race conditions, deadlocks, leaked capacity, and spurious failures in highly concurrent streaming systems.
---
## Null-Safe Validation Standard
Always validate `null`/`undefined` (and malformed shapes) before dereferencing, filtering, or merging—treat `null` as an explicit, documented input rather than “missing”.
Apply this consistently:
- **Filter/transform:** check the element exists before reading its fields; don’t rely on optional chaining alone.
- **Parse responses:** guard arrays and nested properties so malformed upstream payloads fall through to a safe result (not a throw).
- **Defaults:** handle callers passing `null` (not just omitting the argument) when reading options.
- **API semantics:** explicitly define how `null` vs `{}` vs partial objects affect state (e.g., PATCH “clear” must be explicit `null`).
- **Type narrowing:** prefer runtime guards (`typeof`, `in`, `Array.isArray`) over casts.
- **Tests:** add regression tests for `null`/empty/malformed shapes.
Example patterns:
```ts
// 1) Filtering: guard nullish elements + explicit null meaning
function filterActiveConnections(connections: (T | null | undefined)[]) {
return connections.filter((c) => !!c && c.isActive !== false);
}
// 2) Parsing: tolerate malformed upstream arrays
function extractFirstVideo(response: unknown): { base64?: string; url?: string } | null {
if (!response || typeof response !== 'object') return null;
const videos = (response as any).videos;
if (!Array.isArray(videos) || videos.length === 0) return null;
const v = videos[0] as any;
if (typeof v?.bytesBase64Encoded === 'string') return { base64: v.bytesBase64Encoded };
if (typeof v?.gcsUri === 'string') return { url: v.gcsUri };
return null;
}
// 3) Options default: tolerate explicit null
export function filterToOpenAIFormat(body: unknown, opts: { preserveCacheControl?: boolean } | null = {}) {
const preserve = opts?.preserveCacheControl === true;
// ...
}
// 4) PATCH semantics: explicit null clears; empty object is no-op
function applyPatch(existing: any, incomingWindowThresholds: any) {
if (incomingWindowThresholds === null) return { ...existing, quotaWindowThresholds: null };
if (incomingWindowThresholds && typeof incomingWindowThresholds === 'object') {
const next = { ...(existing.quotaWindowThresholds ?? {}) };
for (const [k, v] of Object.entries(incomingWindowThresholds)) {
next[k] = v === null ? null : v;
if (v === null) delete next[k]; // optional: choose your clear behavior per spec
}
return { ...existing, quotaWindowThresholds: next };
}
return existing; // {} should be no-op; missing should be no-op
}
```
If a function can receive `null`/`undefined` from upstream, config, or user input, make the “what happens” path explicit and test the nullish/empty cases.
---
## Contract-First API Validation
Treat every API endpoint as a contract:
- Validate all request inputs (query params, body fields) with shared schemas (e.g., Zod) instead of manual parsing.
- On validation failure, return a consistent 4xx (typically 400) rather than silently coercing/defaulting values.
- When transforming or wrapping responses (especially SSE/streaming), ensure the emitted wire format/shape matches what downstream clients parse.
- Normalize constructed upstream URLs/paths to avoid accidental double-appends/suffixes.
Example (pagination with schema validation):
```ts
export async function GET(request: Request) {
const url = new URL(request.url);
const offsetLimitRaw = {
offset: url.searchParams.get("offset"),
limit: url.searchParams.get("limit"),
};
// paginationSchema: offset int >= 0, limit int within an agreed range
const validation = validateBody(paginationSchema, offsetLimitRaw);
if (isValidationFailure(validation)) {
return NextResponse.json({ error: validation.error }, { status: 400 });
}
const { offset, limit } = validation.data;
const all = await getCombos();
const combos = all.slice(offset, offset + limit);
return NextResponse.json({ combos });
}
```
Team standard: no manual `parseInt`/fallback defaults for externally supplied API parameters; any transformation layer must preserve/emit the documented output shape.
---
## Real Regression Tests
When you fix behavior (especially edge/boundary cases), add regression tests that are:
- **Reachable** (not unexported/inaccessible without refactors that make them testable).
- **Meaningful** (tests should fail when the buggy logic is restored, not merely “exercise” code).
- **Aligned with the repo’s runner** (use the framework actually used by the unit test script).
- **Supported by testable design** (prefer dependency injection over `as any` to control external effects).
Apply like this:
- If you add/adjust internal logic, make it testable (e.g., extract helpers or inject dependencies) so unit tests can target the exact behavior.
- For boundary bugs, include exact-limit cases and verify tests catch regressions.
Example (boundary regression + meaningful failure):
```ts
// Arrange
const LIMIT = 10;
// Restoring a buggy form should make this test fail.
function fixedAppend(current: string, next: string) {
if (!next) return current;
if (current.length >= LIMIT) {
const keep = LIMIT > next.length ? LIMIT - next.length : 0;
if (keep <= 0) return next.slice(-LIMIT);
return current.slice(-keep) + next;
}
return current + next;
}
// Assert (exact boundary)
import test from "node:test";
import assert from "node:assert/strict";
test("appendBoundedText clamps at limit", () => {
const cur = "x".repeat(LIMIT);
const next = "y".repeat(LIMIT);
assert.equal(fixedAppend(cur, next), next); // should be clamped
});
```
Also add guard assertions when you have rule systems/transform pipelines so that “no-op” rules (e.g., patterns that never change anything) can’t be introduced unnoticed.
---
## Validate security context
Define and enforce security-context behavior as a clear contract, then implement CI/runtime logic to match it without brittle assumptions.
Apply this in two places:
1) Helm/manifest security-context checks (CI)
- Assert the contract (e.g., “default render does not pin `runAsUser`”), not a magic UID number.
- Prevent substring false positives by using anchored match patterns.
- Scope checks to the exact template(s) whose output you intend to validate, and repeat the same assertions for every render mode (sqlite/deployment/postgres/etc.).
Example pattern:
```sh
# contract: default render must not set runAsUser
helm template bifrost ./helm-charts/bifrost \
--set image.tag=v1.0.0 \
> /tmp/out.yaml
if grep -Eq '^[[:space:]]+runAsUser:[[:space:]]*' /tmp/out.yaml; then
echo "Fail: runAsUser must not be pinned in default render"
exit 1
fi
```
2) Startup permissions (runtime)
- Don’t assume the container has privilege to `chown`/fix ownership (avoid CAP_CHOWN-dependent correctness).
- Use an actual writability probe for the target data dir, create it when possible, and if it’s not writable, fail early with actionable, platform-aware guidance.
- Avoid advising securityContext values that may be rejected by the target platform’s admission controller (e.g., OpenShift `fsGroup` constraints).
Net effect: fewer admission/runtime surprises and fewer CI “green but wrong” security-context failures.
---
## Streaming error invariants
Adopt a single, consistent error-handling strategy for streaming (and other multi-step flows): define retry semantics, tolerate isolated decode/parse errors, enforce terminal/completion invariants, and preserve the failure context needed by downstream billing/logging.
Guidelines:
1) Classify failures at the I/O boundary
- Treat “request failed before the first response byte” as a retriable upstream connection error (so retry logic can kick in). Avoid treating it as a terminal provider/operation failure.
2) Streaming: log-and-continue for local frame issues
- If a single SSE event/chunk is malformed (unmarshal/base64 decode), log and skip that event instead of aborting the whole stream.
3) Streaming: enforce completion invariants
- Don’t assume EOF implies success.
- If the protocol defines a terminal marker (e.g., Cartesia’s `done` event), surface an error when the terminal marker is never observed.
4) Preserve context on ALL error paths
- Before returning errors, attach provider response headers and any accumulated usage/billing data to the error object so post-hooks can charge and trace correctly.
5) Async safety: prevent unrecoverable panics in teardown goroutines
- Any goroutine that can outlive the request (idle timers, readers, cancellation helpers) must be crash-safe (recover panics inside the goroutine if the panic would otherwise be unrecoverable).
Example pattern (Go, streaming):
```go
// Local helper: attach billed usage accumulated in-context to the error.
func attachBilledUsageFromContext(ctx *schemas.BifrostContext, bifrostErr *schemas.BifrostError) {
usage, ok := ctx.Value(schemas.BifrostContextKeyStreamAccumulatedUsage).(*schemas.BifrostLLMUsage)
if !ok || usage == nil { return }
if usage.PromptTokens == 0 && usage.CompletionTokens == 0 && usage.TotalTokens == 0 { return }
// Important: copy or snapshot if usage will be mutated later.
u := *usage
bifrostErr.ExtraFields.BilledUsage = &u
}
// Inside stream read loop:
var sawDone bool
for {
data, err := sseReader.ReadDataLine()
if err != nil {
if errors.Is(err, io.EOF) {
break
}
// pre-first-byte / upstream classification happens at the Do() boundary,
// but for mid-stream read errors, enrich and send.
e := providerUtils.NewBifrostOperationError("stream read failed", err)
attachBilledUsageFromContext(ctx, e)
providerUtils.ProcessAndSendBifrostError(ctx, postHookRunner, e, responseChan, logger, postHookSpanFinalizer)
return
}
var evt CartesiaSSEEvent
if err := sonic.Unmarshal(data, &evt); err != nil {
logger.Warn("Failed to parse SSE event: %v", err)
continue // log-and-continue
}
switch evt.Type {
case "done":
sawDone = true
// optionally process done payload
case "chunk":
if evt.Data == "" { continue }
audio, decErr := base64.StdEncoding.DecodeString(evt.Data)
if decErr != nil {
logger.Warn("Failed to decode chunk: %v", decErr)
continue // keep stream alive on single chunk decode errors
}
// send delta chunk...
case "error":
// surface terminal provider error
e := providerUtils.NewBifrostOperationError("cartesia stream error: "+evt.Error, nil)
attachBilledUsageFromContext(ctx, e)
providerUtils.ProcessAndSendBifrostError(ctx, postHookRunner, e, responseChan, logger, postHookSpanFinalizer)
return
}
if sawDone { break }
}
// Terminal-invariant guard: EOF without terminal done is truncated.
if !sawDone {
ctx.SetValue(schemas.BifrostContextKeyStreamEndIndicator, true)
e := providerUtils.NewBifrostOperationError("stream ended before terminal done", io.ErrUnexpectedEOF)
attachBilledUsageFromContext(ctx, e)
providerUtils.ProcessAndSendBifrostError(ctx, postHookRunner, e, responseChan, logger, postHookSpanFinalizer)
return
}
```
This standard prevents: (a) “empty success” on truncated streams, (b) missing bills/trace data on cancellation/timeout, (c) inconsistent retry behavior, and (d) process crashes from panics in goroutines used for stream teardown/timeouts.
---
## Secret-Aware Schema Contracts
When updating configuration schemas (and related env/test fixtures), treat the schema as a contract with runtime unmarshalling and deployment timing.
Apply these rules:
1) Mirror secret-aware runtime types in the schema
- If the runtime config uses secret variables that accept either a plain string or a structured secret object, model credential fields the same way (don’t restrict to string only).
Example pattern (JSON Schema):
```json
{
"anyOf": [
{ "type": "string" },
{
"type": "object",
"properties": {
"value": { "type": "string" },
"env_var": { "type": "string" },
"from_env": { "type": "boolean" }
},
"required": ["value"],
"additionalProperties": false
}
]
}
```
2) Verify the config effect end-to-end (release coupling)
- If the schema change introduces a new field consumed only by a different repo/service, ensure the corresponding runtime wiring will ship in the same release; otherwise the field may validate but behave like a no-op.
3) Avoid out-of-scope “partial tightening”
- Don’t change schema enums to match runtime allowlists unless the change is intentional and scoped; if runtime already enforces support via a separate validator, understand that schema passes may be an accepted pre-existing behavior.
4) Keep test/env files suite-scoped
- Only define keys that the specific test suite consumes. Don’t add placeholder vars from other suites to “silence” missing placeholders—this creates divergence and false confidence.
Outcome: schema validation should align with real runtime parsing, and “valid configs” should produce the intended behavior in the correct release.
---
## Config Contract Validation
Ensure API configuration endpoints (e.g., `PUT /api/config`) define and enforce a precise request/response contract across schema shape, merge semantics, validation behavior, and persistence.
Apply this standard:
- **Schema must match canonical payload serialization.** For complex/credential fields, your JSON Schema should validate the exact shape your server produces/round-trips (e.g., secret references). Validate with the same Draft/validator behavior used in production.
- **Strict validation with no state change on rejection.** If a payload is invalid (unknown enum, incompatible sub-objects, wrong auth-mode config), return **400** and guarantee the config is **not mutated**.
- **Explicit partial-merge vs reset semantics.** If the endpoint performs a partial merge, omitted fields must preserve existing stored values (do not “default reset”). Document this behavior and test it.
- **Test the contract end-to-end.** Add E2E tests that:
- PUT invalid config ⇒ 400 and assert GET round-trip shows no change.
- PUT valid config ⇒ GET confirms persistence.
- When behavior is mode-dependent, assert the correct branch (and that security flips only when intended).
Example (server-side merge rule pattern):
```go
// Only update each field when explicitly provided so partial PUTs do not clear stored values.
if payload.ClientConfig.MCPServerAuthMode != "" {
updatedConfig.MCPServerAuthMode = payload.ClientConfig.MCPServerAuthMode
}
// otherwise keep currentConfig.MCPServerAuthMode
```
Example (E2E assertion pattern):
- PUT invalid payload ⇒ expect `400` and response text includes the relevant field name.
- Then GET `/api/config` and assert the boot/auth mode equals the preexisting value (round-trip persistence).
---
## Graceful Degradation Policy
For external dependencies that are *additive* (e.g., mirrors/archives) rather than required for correctness, prefer **fail-open** behavior: log errors and continue operating using the primary source of truth. Reserve **fail-fast** for dependencies that are *hard requirements* for correctness or API functionality.
Practical rules:
- Classify each dependency/feature as either:
- **Critical (correctness-gating):** if unavailable at startup or runtime, fail fast (or reject requests) to avoid silently incorrect behavior.
- **Additive (mirror/archive/log sink):** if unavailable, continue core writes/operations; record the error for observability.
- Ensure the primary store remains the source of truth (e.g., DB-only audit logging still works).
- At startup and during batch flushes:
- Catch connection/unreachability errors.
- **Log + emit metrics**.
- Do not block startup or request handling.
- Document the policy difference explicitly so teams don’t “standardize” into the wrong behavior.
Example (non-blocking additive sink):
```pseudo
onStartup():
try:
objectStoreClient = connect(config.objectStorage)
archivalEnabled = true
catch err:
log.error("object storage unreachable; continuing database-only", err)
archivalEnabled = false
writeAuditEvent(event):
dbWrite(event) // must always succeed for correctness
flushBatch(batch):
if not archivalEnabled: return
try:
objectStore.put(renderKey(batch), gzip(jsonl(batch)))
catch err:
log.error("archival upload failed; audit DB write already committed", err)
// do not fail request handling
```
---
## Consistent UI Formatting
Apply consistent, presentation-focused code style to UI components:
- Preserve flex/height semantics on “fill” containers: when a layout depends on full-viewport panels, ensure the relevant containers keep the same flex height/overflow utilities (e.g., `flex`, `flex-col`, `h-full`, `min-h-0`, and the intended `overflow-*` behavior).
- Make truncation reliable (especially with tooltips): use the correct element display semantics so truncation actually applies—commonly `className="block truncate ..."` on the text node used by the tooltip trigger.
- Centralize shared formatting: any non-trivial formatting logic (currency/token/number conversions) should live in shared utilities (e.g., `ui/lib/utils/numbers.ts`) rather than being reimplemented in components.
- Separate derived presentation values: in `useMemo`, precompute display strings and tooltip strings separately (e.g., rounded display, full-precision tooltip) instead of mixing formatting logic inline.
Example pattern:
```ts
const statCards = useMemo(() => {
const display = stats
? stats.success_rate.toFixed(2) + "%"
: undefined;
const tooltip = stats
? stats.success_rate.toLocaleString(undefined, { maximumFractionDigits: 6 })
: undefined;
return [{
title: "Success Rate",
value: fetchingStats ? : display ?? "-",
tooltip: fetchingStats ? undefined : tooltip,
testId: "logs-stat-value-success-rate",
}];
}, [fetchingStats, stats]);
```
This keeps UI layout stable, prevents truncation/tooltip bugs, improves readability, and ensures consistent formatting across the app.
---
## Database-safe atomic updates
When writing/reading sensitive or destructive data, make the database do the work safely and atomically:
- Prefer SQL-side filtering to avoid broad in-memory fetch/processing (e.g., add `authType` filtering in the query rather than filtering afterward).
- For read-modify-write flows that require audit/comparison (e.g., capture previous scopes), wrap the previous-row SELECT and the UPDATE in a single atomic SQL block using explicit `BEGIN … COMMIT` (and `ROLLBACK` on error) rather than assuming a higher-level `db.transaction()` exists.
- For identifier-driven destructive queries, avoid `LIKE` with unescaped/untrusted identifiers (because `%`/`_` change semantics). Use deterministic predicates like exact match or parameterized prefix matching (e.g., `substr(col,1,?) = ?`).
Example pattern (atomic read + update + safe predicates):
```ts
// Atomic read-modify-write even without db.transaction()
const nextScopes = Array.isArray(scopesUpdate)
? scopesUpdate.filter((s): s is string => typeof s === "string")
: [];
const result = db.exec(`
BEGIN IMMEDIATE;
`);
try {
const prevRow = db
.prepare<{ scopes: string | null }>(
"SELECT scopes FROM api_keys WHERE id = ?"
)
.get(id);
const previousScopes = parseStringList(prevRow?.scopes ?? null);
db.prepare("UPDATE api_keys SET scopes = ? WHERE id = ?")
.run(JSON.stringify(nextScopes), id);
db.exec("COMMIT;");
// emit audit using previousScopes -> nextScopes
} catch (e) {
db.exec("ROLLBACK;");
throw e;
}
```
```sql
-- Safe destructive delete by identifier prefix (no LIKE)
DELETE FROM key_value
WHERE namespace = 'syncedAvailableModels'
AND substr(key, 1, ?) = ?;
```
Applying this consistently reduces correctness bugs, prevents expensive over-fetching, and keeps audit trails reliable across different SQLite/driver backends.
---
## Cohesive Converters Organization
Keep Go files cohesive by responsibility, and make branching/conversion logic easy to extend.
**1) Put types where they belong**
- Move request/response structs (and their `GetExtraParams` helpers) into `types.go`.
- Keep provider-specific files (e.g., `embedding.go`) for converter/transform functions only.
**2) Keep conversion logic readable**
- Prefer `guard + switch` over repeated inline `if x.Type == ...` checks.
- Use small, named helpers for steps like “convert content parts”, “apply extra params”, “encode/decode media”.
Example pattern:
```go
if chunk.Index == nil || chunk.ContentBlock == nil {
return nil, nil, false
}
switch chunk.ContentBlock.Type {
case AnthropicContentBlockTypeToolUse:
// structured handling
case AnthropicContentBlockTypeRedactedThinking:
// structured handling
default:
return nil, nil, false
}
```
**3) DRY repeated utilities safely**
- Extract near-verbatim duplication into shared helpers (e.g., reserving terminal chunk indexes), but keep fields separate when dedup semantics differ.
**4) Centralize “magic” values**
- Move status/reason strings and other repeated constants into `schemas` (or a shared constants module) and reuse them across paths.
**5) Prefer strong typing / reuse helpers**
- Avoid `map[string]any` when the project has a structured type (e.g., `schemas.OrderedMap`).
- Use generics where it meaningfully removes duplicated helper functions (e.g., pointer equality for multiple primitives).
This reduces churn when adding new providers/types, makes reviews faster, and prevents subtle behavior drift from copy/pasted logic.
---
## Network payload safety
When implementing HTTP/SSE adapters or transforming network responses/streams, enforce three rules: (1) preserve required response headers/trace metadata while normalizing body-related headers, (2) respect protocol framing so streamed events can’t merge, and (3) prevent hanging outbound calls with timeouts.
Apply this as a checklist:
- **After filtering/rewriting an HTTP response:** reuse upstream headers that clients depend on (e.g., request/trace/version), but set/reset **Content-Type** appropriately and avoid stale **Content-Length**.
- **For SSE/streaming transforms:** ensure every enqueued “event” ends with a proper SSE delimiter (`\n\n`) so later sentinel/stop messages don’t concatenate with buffered tail data.
- **For outbound network calls:** wrap `fetch` (or equivalent) with an `AbortController` timeout; treat timeouts as recoverable failures where possible.
Example patterns:
```ts
// 1) HTTP response transformation: preserve trace/version headers
function buildTransformedResponse(upstream: Response, data: any) {
const newHeaders = new Headers(upstream.headers);
// normalize payload headers for the transformed body
newHeaders.set("Content-Type", "application/json");
newHeaders.delete("Content-Length");
return new Response(JSON.stringify(data), {
status: upstream.status,
headers: newHeaders,
});
}
// 2) SSE framing: always end events with a delimiter
function enqueueSseEvent(controller: TransformStreamDefaultController, prefix: string, payload: string) {
controller.enqueue(new TextEncoder().encode(`${prefix}${payload}\n\n`));
}
// 3) Outbound request timeout
async function fetchWithTimeout(url: string, init: RequestInit = {}, timeoutMs = 7000) {
const ac = new AbortController();
const t = setTimeout(() => ac.abort(), timeoutMs);
try {
return await fetch(url, { ...init, signal: ac.signal });
} finally {
clearTimeout(t);
}
}
```
Teams should require these checks in code review for any networking code that adapts/streams/transforms data.
---
## Isolate errors, avoid cascades
When a feature depends on multiple probes/loads, ensure that failures in secondary/best-effort steps don’t corrupt primary state or cause cascading failures.
Apply this in two ways:
1) Runtime/state handling: keep authoritative flags separate from enrichment fields.
- Compute primary status from the most reliable probe (e.g., a CLI “installed” check).
- Treat secondary lookups (e.g., `pip list`, optional extras) as best-effort: on failure, return safe defaults for only the enrichment fields, not the primary flag.
Example pattern:
```js
export async function getStatus(url) {
const installed = Boolean(pathFromCliProbe);
// Best-effort enrichment; should NOT affect `installed`.
let extras = { version: null, extras: { code: false, ml: false } };
try {
if (installed) extras = await getInstalledHeadroomExtras();
} catch {
// swallow or log; keep primary state stable
}
return { installed, ...extras };
}
```
2) Test structure: fail fast in setup.
- Load/import prerequisites in a `before()`/setup hook.
- If loading fails, the test should fail once at setup and not continue with `undefined` values that produce many misleading downstream errors.
Example pattern:
```js
import { before, it } from "node:test";
import assert from "node:assert/strict";
let entry;
before(async () => {
entry = (await import("../../path/to/module.js")).default;
});
it("has expected id", () => {
assert.equal(entry.id, "kimchi");
});
```
This prevents both incorrect UI/business logic (wrong state derived from a failing secondary probe) and noisy, cascading test failures that obscure the real cause.
---
## Secure configs across environments
{% raw %}
When making security-related changes, ensure the configuration is (1) least-privilege/scoped, (2) actually enforced by the platform, and (3) consistent across code/docs.
Apply this standard:
- Scope secrets/tokens narrowly: don’t add sensitive tokens to release/runtime job env unless the release jobs truly need them. Prefer using them only in dedicated test/integration jobs.
- Make non-root enforcement verifiable: if you set `runAsNonRoot: true`, the kubelet must be able to verify the container’s non-root user. Avoid cases where the image uses a non-numeric/named `USER`.
- Use numeric UIDs for `USER` when `runAsNonRoot` is enabled (or ensure the platform can otherwise verify). If you need deterministic behavior across k8s distros, set a numeric UID in the Dockerfile, and align `podSecurityContext` with your storage-class behavior.
- Validate Helm rendering/merge behavior: changes like switching `podSecurityContext` to `{}` can behave unexpectedly due to Helm map merges—always confirm the rendered `securityContext` with `helm template`.
- Keep security defaults in schema/docs aligned with code: don’t adjust documented defaults without updating the code default, especially for auth/security parameters.
Example (numeric UID to satisfy kubelet verification):
```dockerfile
# Ensure kubelet can verify non-root when runAsNonRoot: true
USER 1000:0
```
Example (scope secrets to test jobs):
```yaml
# Only provide sensitive tokens where they are required (e.g., PR test jobs)
# Avoid injecting tokens into broad release job env blocks.
env:
# GITHUB_COPILOT_TOKEN: ${{ secrets.GITHUB_COPILOT_TOKEN }} # keep out of release jobs
```
{% endraw %}
---
## Harden build-time paths
When Dockerfile (or build scripts) use build-args/variables to choose target directories for `chown`/`chmod`, validate those paths at build time and fail if they target sensitive directories. This prevents overly-permissive ownership (e.g., group 0 write access) from becoming reachable via undocumented/override knobs.
Apply this rule by adding a guard or assertion before any recursive ownership/permission changes. For example:
```dockerfile
ARG APP_DIR=/app/data
# Fail build if a dangerous directory is selected (adjust allowlist/prefix as needed)
RUN case "$APP_DIR" in \
/app|/app/*) echo "Refusing insecure APP_DIR=$APP_DIR" >&2; exit 1 ;; \
esac
RUN chown -R appuser:0 "$APP_DIR" && chmod -R g=rwX "$APP_DIR"
```
Also prefer explicit allowlists (or “path must start with /app/data”) over implicit assumptions about defaults; don’t rely on the variable being “theoretical” or undocumented to avoid security regressions.
---
## Cache correctness rules
Adopt a “cache correctness” checklist for all in-memory/DB caches:
1) Explicit bypass must be honored
- If the caller passes a disable value (commonly `ttlMs === 0`), bypass both lookup and inflight coalescing. Don’t treat `0` as “use default TTL”.
2) Never poison future caching on failure
- When the cached mechanism becomes invalid (e.g., a limiter instance after repeated 429), evict/disconnect so future calls lazily create a fresh instance.
- Avoid permanently stopping/poisoning objects that remain referenced by in-flight work.
3) Guard correctness against changing upstream state
- Use generation/epoch guards and clear/bump generations on relevant writes.
- Keep caches bounded (max size) and add TTL/staleness policies aligned to refresh cadence.
- Provide test reset hooks for any module-level cache that depends on mutable external state.
Example patterns:
// (1) Honor ttlMs=0 bypass (also ensure defaulting doesn’t override 0)
async function getOrCoalesce(ttlMs: number, fetchFn: () => Promise) {
if (ttlMs <= 0) {
const data = await fetchFn();
return { data, cached: false };
}
// ... normal cache lookup + inflight coalescing
}
// (2) Evict rather than stop/poison
function onRateLimited429(/* ... */) {
// evict so future getLimiter() creates a new instance
// DO NOT call stop() if it would permanently reject future scheduling
}
// (3) Generation-guarded, bounded cache
const MAX = 100;
let gen = 0;
const cache = new Map();
function onWriteThatAffectsCache() {
gen++;
cache.clear();
}
function get(key: string): string | null {
const entry = cache.get(key);
if (!entry || entry.generation !== gen) return null;
return entry.value;
}
function set(key: string, value: string) {
if (cache.size >= MAX) cache.delete(cache.keys().next().value);
cache.set(key, { generation: gen, value });
}
Rule of thumb: every cache needs (a) explicit bypass semantics, (b) safe invalidation/eviction, and (c) a correctness strategy for when upstream data changes (generation/TTL + bounded size + test reset).
---
## Idempotent normalization pipelines
When code must normalize or classify structured data (messages, request bodies, model/provider selection, metrics), implement it as deterministic, *idempotent*, and *non-mutating* transformations.
Apply these rules:
- **Normalize shapes with explicit merge/update rules**: if an external API rejects a structure (e.g., consecutive same-role entries), implement a function that enforces the required invariant (e.g., merge adjacent same-role items).
- **Keep transformations idempotent**: prepends/appends should be guarded by an idempotency key or prefix/text detection so re-running the pipeline doesn’t duplicate content.
- **Avoid unintended mutation**: if you transform an array/object derived from caller input, copy the elements/arrays you modify.
- **Use precise matching for classification**: prefer boundary-aware regex or generalized capability checks over naive `includes()` where false positives are possible.
- **Ensure aggregation update rules are correct**: when computing totals, accumulate instead of overwriting per-group values.
Example (non-mutating consecutive-role merge + idempotent ops sketch):
```ts
type Content = { role: string; parts: Array<{ text: string }> };
function mergeConsecutiveSameRole(contents: Content[]): Content[] {
const merged: Content[] = [];
for (const entry of contents) {
const last = merged[merged.length - 1];
if (last && last.role === entry.role) {
// mutate only the new output copy
last.parts.push(...entry.parts);
} else {
// prevent caller mutation
merged.push({ ...entry, parts: [...entry.parts] });
}
}
return merged;
}
type Op = { kind: 'prepend_system_block'; text: string; idempotencyKey?: string };
function applyOp(blocks: Array<{ type: 'text'; text: string }>, op: Op) {
const alreadyThere = blocks.some((b) => b.text === op.text || b.text.startsWith(op.text));
return alreadyThere ? blocks : [{ type: 'text', text: op.text }, ...blocks];
}
```
Practical checklist:
- Add regression tests for (a) repeated execution, (b) edge input shapes (empty/missing fields), and (c) realistic provider/model strings to prevent matching errors.
- For metrics/analytics, add tests that verify accumulation (`sum += value`) rather than overwriting. If you follow these, you’ll prevent the common failure modes seen across the discussions: invalid request shapes, duplicated system blocks, false-positive classification, and incorrect aggregation totals.
---
## Hot Path Cost Controls
In performance-sensitive paths, actively control hidden costs: cap buffering, reuse expensive compiled primitives, and avoid full serialization/large allocations.
Apply these rules:
1) Bound buffering/reads in streaming probes
- Any “pre-read”, “lookahead”, or retry window must have a hard maximum so latency/memory can’t grow unbounded.
- Prefer explicit exit conditions (counts/time) and then hand control back to the main handler.
Example pattern (mimicking the approach):
```go
if !preReadUntilContent && len(buffered) >= maxPreReadEvents {
break // stop buffering; avoid response delay/unbounded growth
}
```
2) Hoist heavy compiled expressions
- Do not compile regexes inside functions that run frequently. Hoist `regexp.MustCompile(...)` to package/module scope and reuse.
3) Replace full serialization with cheap deterministic signatures
- On hot paths, avoid `json.Marshal` of large structs/slices when you only need routing/decision equivalence.
- Compute a field-level hash/signature (e.g., FNV) over only the fields that can affect behavior, and include nested-condition fields if they impact routing.
Example pattern (self-contained):
```go
func associationSignature(a []*MyAssociation) uint64 {
h := fnv.New64a()
for _, x := range a {
// Write only semantically relevant fields
h.Write([]byte(x.TargetType))
h.Write([]byte{byte(x.Enabled)})
for _, c := range x.Conditions {
h.Write([]byte(c.Key))
h.Write([]byte(c.Operator))
h.Write([]byte(c.Value))
}
}
return h.Sum64()
}
```
4) Test what changed semantics
- When you replace logic (bounded buffering, precompiled parsing, signature computation), add/keep regression tests that cover edge cases and nested variations so you don’t accidentally change the meaning while improving performance.
---
## Sanitize Upstream Stream Errors
When handling upstream streaming responses (SSE/streaming APIs), treat any non-stream payload (e.g., HTML error pages) as untrusted input: do not pipe it through streaming transforms, and do not reflect raw upstream text/HTML into client errors.
Actionable standard:
- Validate `Content-Type` before piping/transforming.
- If it’s not the expected streaming type, consume the body and produce a *safe* error message:
- extract a small, human-readable string (e.g., from ``),
- strip tags/avoid raw HTML,
- clamp length before including it in any client-visible response,
- return a clean JSON error payload.
- Ensure the error path is handled consistently (e.g., call stream error handlers / avoid router-crash scenarios).
Example pattern:
```js
const upstreamContentType = (providerResponse.headers.get('content-type') || '').toLowerCase();
if (
upstreamContentType &&
!upstreamContentType.includes('text/event-stream') &&
!upstreamContentType.includes('application/json')
) {
const bodyText = await providerResponse.text().catch(() => '');
const titleMatch = bodyText.match(/<title>([^<]+)<\/title>/i);
// Sanitized + clamped message (never reflect raw upstream HTML)
const shortMsg =
titleMatch?.[1]?.trim() ||
(bodyText.length < 200 ? bodyText.trim() : `Upstream returned ${upstreamContentType}`);
const safeMsg = shortMsg.replace(/<[^>]*>/g, '').slice(0, 160);
return json({ error: safeMsg }, { status: providerResponse.status || 502 });
}
```
---
## Consistent Test ID Naming
<!-- source: maximhq/bifrost | topic: Naming Conventions | language: TSX | updated: 2026-06-29 -->
When adding or modifying UI elements, ensure their test/automation identifiers follow the existing naming conventions and remain stable.
Apply these rules:
- Prefer component-supported prefixing (e.g., `testIdPrefix`) so generated IDs stay consistent across related CTAs.
- If you must set `data-testid` directly, follow the local established pattern for that area (for example, `client-settings-<name>-<element>` where `<element>` reflects the control type like `switch` / `input`).
- Keep identifiers predictable: use a scope/feature prefix, then the setting/feature name, then the UI element type.
Example patterns:
```tsx
// Prefer prefix props when the component generates IDs
<ContactUsView
testIdPrefix="license"
title="Unlock license management"
// generated IDs like: license-read-more / license-book-demo
/>
// Follow the local convention when setting explicitly
<Switch
id="dump-errors-in-console-logs"
data-testid="client-settings-dump-errors-switch"
// ...
/>
```
This keeps automated tests resilient and avoids one-off/unstable identifiers that break CI and slow down debugging.
---
## Config precedence and canonical sources
<!-- source: diegosouzapw/OmniRoute | topic: Configurations | language: Other | updated: 2026-06-22 -->
Establish a single, explicit source-of-truth approach for configuration resolution: (1) honor user-supplied overrides first, (2) otherwise derive from the active context/environment, and (3) only then fall back to defaults; additionally, treat repository-level configuration/lock files as canonical entrypoints for developer tooling.
How to apply:
- CLI/config resolution: implement a consistent precedence order and pass required context through the full call chain.
- Never ignore an available active context (or env) just because a command has a “convenient” default (e.g., localhost).
- Dev tooling: don’t duplicate or move canonical root config/lock files (e.g., flake.nix/devbox.json/associated locks); changes require coordinating the tooling that consumes them.
Example (pattern for precedence):
```js
export async function runLogsCommand(opts = {}) {
const baseUrl =
opts.baseUrl ||
opts["base-url"] ||
getBaseUrl({ context: opts.context }); // derived from active context/env
// ...use baseUrl to build requests
}
```
---
## Await Connection Drains
<!-- source: maximhq/bifrost | topic: Concurrency | language: JavaScript | updated: 2026-06-20 -->
When async work depends on a connection that may still be resolving, prevent startup race conditions by (1) deferring the work to an “early” queue instead of silently skipping, and (2) awaiting the connection setup promises before concluding/awaiting the deferred work.
Apply this by standardizing the pattern:
- If the required DB/connection isn’t ready yet, enqueue the verification (do not SKIP).
- Once the connection resolves, drain the queue and push the resulting promises into your main pending list.
- In the final “done”/completion handler, await ALL connectionPromises first, then await Promise.allSettled(pendingVerifications).
Example pattern:
```js
const earlyCostingQueue = [];
const connectionPromises = [];
const pendingVerifications = [];
function verifyLaterOrQueue({ reqId, name }) {
const expectCost = getHeader(/* request */ null, 'x-bf-expect-cost');
if (expectCost && /^(1|true|yes)$/i.test(String(expectCost))) {
if (!logsDbReady || !logsDb) {
// Defer instead of silently skipping
earlyCostingQueue.push({ reqId, name });
return;
}
pendingVerifications.push(verifyCostingRequest(logsDb, reqId, name, results, silent));
}
}
function drainCostingQueue(activeLogsDb) {
while (earlyCostingQueue.length > 0) {
const item = earlyCostingQueue.shift();
pendingVerifications.push(verifyCostingRequest(activeLogsDb, item.reqId, item.name, results, silent));
}
}
// Track connection resolution so drains happen before final wait
connectionPromises.push(connectLogsDb().then(activeLogsDb => {
drainCostingQueue(activeLogsDb);
}));
// Finalization: await drains before awaiting pending work
Promise.allSettled(connectionPromises)
.then(() => Promise.allSettled(pendingVerifications))
.then(() => done());
```
Also ensure your core verification logic runs before any status-based early returns (e.g., before “2xx-skip”) so failure/cancelled cases aren’t accidentally bypassed.
---
## Harden Client URL Fetches
<!-- source: diegosouzapw/OmniRoute | topic: Security | language: TypeScript | updated: 2026-06-03 -->
Any client-controlled URL (including OpenAI image_url) must be treated as SSRF-/DoS-/parsing-risk input and validated *at every transition* before connecting.
Apply this standard:
- **Strict outbound policy:** allow only http(s); reject embedded creds, localhost, link-local/private/CGNAT, and metadata hostnames.
- **Redirect safety:** use `redirect: "manual"`; for each hop, resolve + validate again and enforce a small maximum redirect count.
- **DNS rebinding safety:** after hostname validation, resolve IPs and reject if *any* resolved address is private/local/link-local/metadata.
- **Resource caps:** cap wall-clock fetch time and final decoded/received size (and ideally cheap pre-checks).
- **Content integrity:** verify `content-type` (and for `data:` URLs: enforce `image/*`, base64-only as required).
- **Fail closed with sanitized errors:** throw a 400-class error with a message that does not echo attacker-controlled URLs/hosts.
- **No “wildcard” targets for client-driven messaging:** when sending `postMessage`, explicitly target the intended loopback origins (no `*`).
Example pattern (server-side fetch):
```ts
async function safeFetchImageBytes(inputUrl: string) {
// 1) Parse + enforce strict outbound policy
let current = inputUrl;
for (let hop = 0; hop <= 3; hop++) {
const parsed = parseAndValidatePublicUrl(current); // strict scheme/host policy
// 2) DNS rebinding defense (reject if any IP answer is private)
// validateResolvedIps(parsed.hostname)
// 3) Manual redirect handling
const res = await fetch(parsed.toString(), { method: 'GET', redirect: 'manual' });
if (res.status >= 300 && res.status < 400 && res.headers.get('location')) {
current = new URL(res.headers.get('location')!, parsed).toString();
continue; // re-validate next hop
}
// 4) Verify content-type and read with a size cap
const ct = (res.headers.get('content-type') || '').toLowerCase();
if (!ct.startsWith('image/')) throw new Error('Image content-type required');
const data = await readCapped(res, 1 * 1024 * 1024);
return data;
}
throw new Error('Too many redirects');
}
```
---
## Secure postMessage handling
<!-- source: diegosouzapw/OmniRoute | topic: Security | language: TSX | updated: 2026-05-31 -->
When handling cross-window `postMessage`, don’t rely on a single `window.location.origin` check—loopback flows (e.g., `localhost` vs `127.0.0.1`) can be legitimate but have different origins. For security, combine: (1) strict message type checking, (2) a whitelist of allowed origins that includes known legitimate loopback variants, and (3) a correlation ID match (e.g., `loginTraceId`) to reject unsolicited/injected messages.
Example pattern:
```ts
const allowedOrigins = new Set([
'http://localhost:3000',
'http://127.0.0.1:3000',
]);
const expectedTraceId = traceIdRef.current;
function onMessage(ev: MessageEvent) {
if (!allowedOrigins.has(ev.origin)) return;
const m = ev.data as {
type?: string;
loginTraceId?: string;
success?: boolean;
error?: string;
};
if (m?.type !== 'trae-oauth-callback') return;
if (expectedTraceId && m.loginTraceId !== expectedTraceId) return;
// handle verified message
}
```
Apply this anywhere you parse `postMessage` for auth/callback flows: accept only known origins, verify the message shape/type, and require a per-attempt correlation token.
---
## Regex Guardrails for Filters
<!-- source: diegosouzapw/OmniRoute | topic: Algorithms | language: Json | updated: 2026-05-28 -->
When implementing CLI output filters (regex/pattern matchers), make the match logic algorithmically safe: add explicit bypass guards for structured/machine-readable output and use precise patterns for command variants, then lock behavior with include/exclude regression tests.
Apply this standard:
1) Add structured-output bypasses
- If a command supports JSON/YAML/template output, ensure the filter does nothing when those flags are present (e.g., by using negative lookahead like `(?!.*--json)` or `(?!.+-o\s+(?:json|yaml|jsonpath|go-template|template\/name)\b)` depending on your command style).
2) Use explicit alternation for CLI syntax variants
- Prefer patterns that directly encode known forms (e.g., `docker compose` vs `docker-compose`) rather than fuzzy modifiers that won’t match real whitespace-separated variants.
3) Test both inclusion and exclusion
- Add tests that assert:
- the filter matches the intended “noise-stripping” commands, and
- the filter does NOT match (or claims nothing) for structured-output invocations and for unrelated subcommands.
Example (pattern-bypass + explicit variants):
```js
const kubectlMatch = {
outputTypes: ['kubectl'],
commands: [
// match kubectl actions
'^kubectl\s+(?:get|describe|logs|apply|delete|rollout|exec|top|events?)\b',
// bypass if structured output is requested (illustrative)
'^(?!.*-o\s+(?:json|yaml|jsonpath|go-template)\b).*kubectl\s+(?:get|describe)\b'
]
};
```
(Shape the bypass to each CLI’s actual flags/outputs.)
This prevents corrupted structured data, reduces over-matching, and makes the filter’s behavior predictable over time.
---
## Disable install scripts
<!-- source: diegosouzapw/OmniRoute | topic: Security | language: Dockerfile | updated: 2026-05-27 -->
In security-sensitive build environments (e.g., Docker/CI), prevent dependency install/postinstall hooks from executing arbitrary code. Use `npm ci` with `--ignore-scripts` to reduce supply-chain risk, but explicitly rebuild and validate any known native/runtime dependency that truly must compile/bind for the target platform.
Example (pattern):
```dockerfile
# Require reproducible installs
RUN test -f package-lock.json \
|| (echo "package-lock.json is required" >&2 && exit 1)
# Block broad dependency script execution
RUN npm ci --no-audit --no-fund --legacy-peer-deps --ignore-scripts \
&& npm rebuild better-sqlite3 \
&& node -e "require('better-sqlite3')(':memory:').close()"
```
Apply this standard by:
- Defaulting to `--ignore-scripts` for `npm ci` in container builds.
- Adding narrow, explicit rebuild steps for only the dependencies that require native bindings for the target platform.
- Running a minimal smoke test that loads/uses the rebuilt module to fail fast if the native binding isn’t valid.
---
## Type-safe query contracts
<!-- source: looplj/axonhub | topic: API | language: TSX | updated: 2026-05-26 -->
When a UI route consumes or produces URL search/query parameters, treat those parameters as a contract: validate and type them at the route boundary, update them through typed helpers (no `as any`), and normalize/omit default-derived fields so the resulting query set is consistent.
Apply:
- Add a search validator/type for list/detail routes (or the routing layer) so `navigate({ search })` matches the route’s expected schema.
- Prefer a single typed helper that updates a `draft: Record<string, unknown>` (or a stronger type) and returns the next search object.
- Normalize composite params (e.g., date ranges): only write “time” keys when their corresponding date keys are present; omit default time values to avoid noisy or inconsistent URLs.
Example (TypeScript):
```ts
// Route boundary: validate search
// (Exact API depends on your router, but the idea is: no implicit/unchecked shape)
// validateSearch: (s) => Record<string, unknown>
function updateRequestSearch(
apply: (draft: Record<string, unknown>) => void,
) {
const nextSearch = (prev: Record<string, unknown> | undefined) => {
const draft: Record<string, unknown> = { ...(prev ?? {}) };
apply(draft);
return draft;
};
navigate({
search: nextSearch as any, // only if your router typing requires; ideally remove this via proper validateSearch types
replace: true,
});
}
function onDateRangeChange(range?: DateTimeRangeValue) {
updateRequestSearch((draft) => {
const normalized = range ? normalizeDateTimeRangeValue(range) : undefined;
if (!normalized || (!normalized.from && !normalized.to)) {
delete draft.createdAtFrom;
delete draft.createdAtTo;
delete draft.createdAtStartTime;
delete draft.createdAtEndTime;
return;
}
if (normalized.from) draft.createdAtFrom = formatSearchDate(normalized.from);
else delete draft.createdAtFrom;
if (normalized.to) draft.createdAtTo = formatSearchDate(normalized.to);
else delete draft.createdAtTo;
// Only keep time fields when the corresponding date side exists,
// and omit defaults (e.g., 00:00:00 / 23:59:59) in your formatter.
draft.createdAtStartTime = formatSearchTime(normalized.startTime);
draft.createdAtEndTime = formatSearchTime(normalized.endTime);
});
}
```
---
## Externalize config copy
<!-- source: diegosouzapw/OmniRoute | topic: Configurations | language: TSX | updated: 2026-05-21 -->
Treat user-facing UI text (help text, labels, descriptions, toggle text) as configuration: do not hardcode it in components. Instead, source it from the appropriate translation/config namespace using the team’s i18n helpers. Keep only upstream/system-provided constants (e.g., IDs, template metadata) as raw values when they are not user copy.
Apply this when adding/modifying UI around configuration features (e.g., API key scopes, access descriptions, combo/catalog descriptions).
Example pattern:
```ts
// en.json (or similar)
{
"apiManager": {
"managementAccessDesc": "Allow this API key to manage OmniRoute configuration."
},
"combos": {
"autoCatalogTitle": "Auto-routing catalog",
"autoCatalogDescription": "Built-in auto/* combos resolved dynamically from connected providers."
}
}
```
```tsx
import { useTranslations } from "next-intl";
const tApi = useTranslations();
const tCombos = useTranslations("combos");
export function Example() {
return (
<p>
{tApi("apiManager.managementAccessDesc")}
{/* or {tApi("managementAccessDesc")} if already scoped consistently */}
</p>
);
}
export function AutoComboHeader() {
return (
<h2>{tCombos("autoCatalogTitle")}</h2>
);
}
```
Checklist:
- New/edited user-facing strings → add to en.json (or the shared config) under the correct namespace.
- Components → useTranslations/use i18n helper instead of inline literals.
- Only translate user copy; leave upstream-defined constants/IDs/metadata as raw values.
---
## Local Clarity Standards
<!-- source: diegosouzapw/OmniRoute | topic: Code Style | language: TypeScript | updated: 2026-05-13 -->
When updating code, optimize for local clarity and correctness, not “cleanup” by indirection.
Apply these rules:
1) Prefer locality over forced DRY
- If a constant/abstraction is used once, keep it local. Don’t add module-scope indirection unless it materially improves maintainability.
- Keep configuration entries self-contained even if it duplicates an existing pattern; avoid helpers that create implicit coupling when different providers may evolve independently.
2) Improve readability with explicit intent
- Use clear derived booleans for control-flow (e.g., “passthrough” flags).
- Avoid mutation surprises: when a branch should be symmetric, assign the derived variable explicitly (e.g., `translatedBody = body`), rather than relying on implicit state.
3) Type-safety as a style baseline
- Let type guards do the work: remove redundant casts when `typeof`/guards already narrow.
- Avoid introducing `any` for new helpers; use focused interfaces and `Partial<...>` for recovery/state update parameters.
4) Preserve formatting in content transforms
- In response cleaning/translation paths, avoid broad whitespace normalization or tag stripping that can mangle markdown, code blocks, tables, or indentation. Only remove content you explicitly intend to remove.
Example (readability + type narrowing):
```ts
// Prefer guard-driven simplification over redundant casts
if (aliases && parsed.model) {
const directTarget = aliases[parsed.model];
if (typeof directTarget === "string" && directTarget.includes("/")) {
const slashIdx = directTarget.indexOf("/');
if (slashIdx !== -1) {
const providerPart = directTarget.slice(0, slashIdx);
const modelPart = directTarget.slice(slashIdx + 1);
// ...
}
}
}
```
Example (format-preserving cleaning):
```ts
function cleanResponse(text: string, strip = true): string {
let t = text;
// Remove only known unwanted tags/markers
t = t.replace(/<[?]xml[^?]*[?]>/g, "");
// Intentionally DO NOT collapse spaces/newlines, to preserve markdown/code formatting
return strip ? t.trim() : t;
}
```
---
## HTTP Timeout Strategy
<!-- source: infiniflow/ragflow | topic: Networking | language: Go | updated: 2026-05-09 -->
Standardize how you apply timeouts for outbound HTTP requests.
- If the shared `http.Client` already sets `Timeout`, do **not** also wrap each request with `context.WithTimeout` (it’s redundant and can make behavior harder to reason about).
- If the shared `http.Client` does **not** have a safe end-to-end timeout (common for streaming/SSE), avoid `http.Client.Timeout` because it can truncate long-lived bodies. Instead:
- Apply a request-scoped deadline with `context.WithTimeout` for **non-streaming** calls.
- For streaming calls, rely on transport-level limits for connection establishment/headers (e.g., `transport.ResponseHeaderTimeout`) and avoid client-level `Timeout`.
Example pattern (non-stream):
```go
// Client: no Timeout when you need long-lived streaming elsewhere
client := &http.Client{Transport: transport} // transport.ResponseHeaderTimeout set
ctx, cancel := context.WithTimeout(context.Background(), nonStreamCallTimeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewBuffer(jsonData))
resp, err := client.Do(req)
// handle resp
```
Example pattern (client Timeout already set):
```go
// NewModel sets httpClient.Timeout (e.g., 120s) and you don't pass ctx through
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
resp, err := httpClient.Do(req) // no per-request context timeout wrapper
```
Apply this consistently across drivers so timeout behavior is predictable for both normal responses and SSE/streaming.
---
## Strict External API Validation
<!-- source: infiniflow/ragflow | topic: Error Handling | language: Go | updated: 2026-05-09 -->
When integrating with external AI/HTTP APIs, treat both transport errors and payload validity as first-class failures. Validate inputs, verify provider/HTTP status, and enforce response invariants—never silently accept partial or malformed data.
Apply these rules:
- Preconditions: return clear errors when required config is missing (API key/model) and decide explicitly for empty inputs (e.g., return empty embeddings/rerank scores).
- Status checks:
- Check HTTP status (e.g., resp.StatusCode == http.StatusOK).
- Also check provider-level status fields (e.g., BaseResp/status_code) before using returned data.
- Payload invariants:
- Embeddings: for each returned item, validate index bounds and embedding presence; error if indices are out of range or embeddings are empty.
- Rerank: validate every result index is within [0, len(texts)); error on out-of-range indices.
- Count matching: if the API is expected to return one score per input (especially when you set top_n), error when result count doesn’t match.
- Defensive JSON decoding: when numeric fields may deserialize as different float types, guard and convert safely; add targeted tests for the guarded edge cases.
Example pattern (fail fast + invariant checks):
```go
resp, err := client.Do(req)
if err != nil { return nil, fmt.Errorf("send request: %w", err) }
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil { return nil, fmt.Errorf("read response: %w", err) }
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("API error: %s body=%s", resp.Status, string(body))
}
// provider-level status (if present)
// var base BaseResp; json.Unmarshal(body,&base); if base.StatusCode!=0 { return nil, ... }
// invariants
embeddings := make([][]float64, len(texts))
for _, item := range parsed.Data {
if item.Index < 0 || item.Index >= len(texts) {
return nil, fmt.Errorf("unexpected index %d for %d inputs", item.Index, len(texts))
}
if len(item.Embedding) == 0 {
return nil, fmt.Errorf("empty embedding for index %d", item.Index)
}
// convert with type guards to float64
}
```
Goal: errors should be deterministic and informative, and incorrect provider payloads should never silently degrade results (e.g., via skipped indices or indistinguishable default zeros).
---
## Shared API Contracts
<!-- source: diegosouzapw/OmniRoute | topic: API | language: TSX | updated: 2026-05-09 -->
Design API endpoints so they orchestrate transport only, and keep domain logic + contract identifiers centralized.
Apply these rules:
1) **Don’t couple libraries to route handlers**: HTTP-specific code (request/response/headers/params) stays in `route.ts`; core behavior (embeddings, memory indexing, etc.) lives in shared services used by both the API route and other server modules.
2) **Single source of truth for backend-recognized identifiers**: any client/UI option that toggles engine behavior (rule names, modes, etc.) must use backend canonical constants/types—never “fabricated” strings.
Example pattern (service extraction):
```ts
// src/lib/embeddings/service.ts
export async function generateEmbeddings(input: string) {
// domain logic here (no fetch/Response/route concerns)
}
// src/app/api/v1/embeddings/route.ts
import { generateEmbeddings } from "@/lib/embeddings/service";
export async function POST(req: Request) {
const { input } = await req.json();
const embeddings = await generateEmbeddings(input);
return Response.json({ embeddings });
}
```
Example pattern (contract identifiers):
```ts
// shared/shared-rules.ts (or import from backend constant module)
export const ALL_CAVEMAN_RULES = [
/* values copied/imported from backend canonical list */
] as const;
// UI uses the canonical list (no guessed strings)
import { ALL_CAVEMAN_RULES } from "@/shared/shared-rules";
```
If an option fails to work (e.g., skip/toggle doesn’t take effect), treat it as an API contract mismatch and trace the identifier back to its backend source of truth.
---
## API Contract Verification
<!-- source: infiniflow/ragflow | topic: API | language: Json | updated: 2026-05-09 -->
When adding/updating model-to-endpoint configuration, verify it against the provider’s official API docs and enforce the expected request/response contract.
Apply this standard:
1) Correct endpoint/path per model
- Don’t assume all models share the same URL suffix.
- Map each `model` to the exact endpoint/path described in the docs (e.g., rerank models may differ by provider).
2) Confirm response shape/schema compatibility
- Ensure your typed structs match the documented JSON fields (names, nesting, types).
- If the provider claims “OpenAI-compatible,” validate the key fields you depend on (e.g., `data[].index`, `data[].embedding`, `usage`, etc.).
3) Add defensive validations for upstream/proxy anomalies
- Validate lengths and indices before trusting arrays from responses.
- Reject or guard against duplicates, out-of-range indices, or count mismatches to prevent silent corruption.
Example pattern (Go-like pseudocode):
```go
// 1) Endpoint routing (per-model, docs-backed)
func urlSuffix(providerCfg map[string]string, model string) string {
return providerCfg[model] // e.g., rerank models can have different suffixes
}
// 2) Defensive response validation
type EmbeddingData struct {
Index int
Embedding []float64
Object string
}
type EmbeddingResponse struct {
Data []EmbeddingData
Model string
Object string
}
func normalizeEmbeddings(resp EmbeddingResponse, expectedCount int) ([][]float64, error) {
if len(resp.Data) != expectedCount {
return nil, fmt.Errorf("embedding count mismatch")
}
// Guard against duplicates/out-of-range
seen := map[int]bool{}
out := make([][]float64, expectedCount)
for _, d := range resp.Data {
if d.Index < 0 || d.Index >= expectedCount {
return nil, fmt.Errorf("embedding index out of range")
}
if seen[d.Index] {
return nil, fmt.Errorf("duplicate embedding index")
}
seen[d.Index] = true
out[d.Index] = d.Embedding
}
return out, nil
}
```
Keep the docs link/source close to the mapping so future changes remain traceable and reviewable.
---
## Bound Work and Queries
<!-- source: infiniflow/ragflow | topic: Performance Optimization | language: Python | updated: 2026-05-08 -->
When improving performance, make sure code neither (a) fetches too much data, nor (b) does the same expensive work repeatedly, nor (c) runs unbounded workloads.
**Practical rules**
1. **Target by primary keys / narrow filters**: Prefer `get_by_id`-style lookups over broad tenant-scoped queries when you already have the identifier.
2. **Push filtering to the datastore**: If you currently fetch large result sets and filter in Python (or run many queries), move the filter into the ES/DB/search criteria (e.g., include `doc_id` / `source_id` in the search condition).
3. **Bound user-controlled heavy parameters**: Add upper bounds for things like `top_k` so a bad request can’t trigger massive retrieval.
4. **Batch/chunk heavy processing**: For page/document loops, use page batches to control memory/CPU and keep progress reporting accurate.
5. **Avoid repeated expensive work**: Hoist invariant computations out of inner loops, and run file-level expensive steps once (not per sheet/per iteration).
6. **Resource-heavy ops must be policy-driven**: If you unload/reload models or enable tracing, ensure it’s safe under concurrency (don’t unload while tasks run) and avoid enabling heavy instrumentation by default.
**Examples**
- Bound retrieval:
```python
top = int(req.get("top_k", 1024))
top = min(top, 200) # cap user input
results = SearchService.search(..., top_k=top)
```
- Chunk heavy page processing:
```python
batch_size = max(1, int(os.getenv("PDF_PARSER_PAGE_BATCH_SIZE", "50")))
for page_from in range(from_page, to_page, batch_size):
page_to = min(page_from + batch_size, to_page)
__images__(fnm, page_from=page_from, page_to=page_to)
chunk_boxes = parse_window_into_boxes()
all_boxes.extend(to_global(chunk_boxes))
```
- Push doc_id/source_id filtering into search:
```python
# include source_id/doc_id in the search condition so the store filters
cond = {"source_id": [doc_id], **other_filters}
res = SearchService.search(dataset_id=dataset_id, condition=cond, limit=1)
```
---
## Fail-Closed Security Invariants
<!-- source: diegosouzapw/OmniRoute | topic: Security | language: Other | updated: 2026-05-08 -->
Default to secure behavior when you can’t verify. If a security-relevant check (e.g., whether encrypted credentials exist) or its dependencies (e.g., DB access) fails, treat the state as unsafe and stop or require explicit recovery—never assume “nothing to decrypt.” Also, preserve cross-component encryption/decryption invariants (prefix/salt/format) so that verification/encryption remains compatible.
Apply it:
- Encryption format compatibility: keep the same prefix/salt/parameters across CLI and server/app decrypt paths, or migrate both sides together.
- Fail closed on inspection errors: when you inspect for encrypted secrets and the inspection fails, don’t generate new keys based on a “no encrypted creds” assumption.
- Scope filesystem/path security checks to the proven threat to prevent both bypasses and excessive false positives.
Example (fail-closed encrypted credential inspection):
```js
function hasEncryptedCredentials(dataDir) {
const dbPath = join(dataDir, 'storage.sqlite');
if (!existsSync(dbPath)) return false;
try {
const Database = require('better-sqlite3');
const db = new Database(dbPath, { readonly: true, fileMustExist: true });
try {
const row = db.prepare(`
SELECT 1
FROM provider_connections
WHERE access_token LIKE 'enc:v1:%'
OR refresh_token LIKE 'enc:v1:%'
OR api_key LIKE 'enc:v1:%'
OR id_token LIKE 'enc:v1:%'
LIMIT 1
`).get();
return !!row;
} finally {
db.close();
}
} catch {
// Fail closed: treat inspection failure as unsafe/unknown
throw new Error('Failed to inspect encrypted credentials; aborting setup/bootstrap.');
}
}
```
---
## Centralized input guards
<!-- source: infiniflow/ragflow | topic: Security | language: Python | updated: 2026-04-30 -->
Require a shared, spec-aligned “guard” step for any untrusted input used in security-sensitive contexts (SQL construction, dynamic table names, outbound URL fetch). Fail closed (raise a safe exception) before interpolation/requests, and reuse a single utility to avoid security-policy drift.
Practical rules:
- SQL / identifier interpolation: validate identifiers (e.g., UUIDs) *before* using them in f-strings or dynamic table names; parameterize whenever possible.
- Reuse instead of duplicating: if an existing validation helper exists, ensure it matches the required constraints (e.g., don’t silently narrow “UUID version” rules) and raises the exception type your call site expects.
- Outbound fetch (SSRF): centralize URL safety checks in a shared utility; avoid per-endpoint ad-hoc rules.
- Logging hygiene: don’t log full user-supplied URLs (may contain credentials/tokens). Log minimal metadata (scheme/host or a redacted form).
Example pattern (UUID guard):
```python
import uuid
import logging
logger = logging.getLogger(__name__)
def assert_valid_uuid(value: str, *, label: str) -> str:
try:
# Validate; optionally canonicalize to the storage format your SQL expects
return uuid.UUID(str(value)).hex # e.g., 32-char form
except (ValueError, TypeError):
logger.warning("Rejected invalid %s (len=%d)", label, len(str(value)))
raise ValueError(f"Invalid {label} format")
# Use only after guard
kb_hex = assert_valid_uuid(kb_ids[0], label="kb_id")
table_name = f"ragflow_{tenant_id}_{kb_hex}"
```
Example pattern (SSRF logging hygiene):
```python
# Instead of logging full URL:
logger.warning("SSRF guard blocked URL (scheme=%r host=%r)", scheme, parsed.hostname)
```
---
## Centralize input bounds
<!-- source: infiniflow/ragflow | topic: Security | language: TSX | updated: 2026-04-28 -->
When validating user-controlled numeric inputs, define allowed min/max values in one place and reuse them across both the UI/component props and the validation schema. Prevent mismatches where the UI permits one range but the schema enforces another (or vice versa), since this breaks input validation guarantees.
Example:
```ts
const TOP_N_MAX = 200;
export const topnSchema = {
top_n: z.number().max(TOP_N_MAX).optional(),
};
function TopNFormField({ max = TOP_N_MAX, ...props }: { max?: number }) {
// Ensure any passed `max` defaults/constraints align with TOP_N_MAX.
return /* render */ null;
}
```
Practical checklist:
- Use a shared constant (or a single policy source) for allowed bounds.
- Derive component `max`/limits from that constant rather than duplicating numbers.
- Avoid hard-coded schema limits that can diverge from what the component consumer can pass.
---
## Test isolation and tiers
<!-- source: infiniflow/ragflow | topic: Testing | language: Python | updated: 2026-04-27 -->
When adding or changing behavior, require unit tests that are (a) comprehensive for the logic contract, (b) correctly prioritized/selected via pytest markers, and (c) isolated from global state to avoid cross-suite flakiness.
Apply it like this:
- Cover the real decision points and fallbacks: defaults, manual-vs-auto modes, empty inputs, type/probe fallbacks, and output shape/values.
- Use the right pytest marker level so the test actually runs in your expected pipeline (e.g., don’t leave important coverage at `p3` if it won’t execute by default). Ensure markers are registered in `pyproject.toml`.
- If tests stub imports or mutate `sys.modules`/env, isolate that mutation with `autouse` fixtures that restore the original state after each test.
Example pattern for isolating `sys.modules` stubs:
```python
import sys
import pytest
from types import ModuleType
@pytest.fixture(autouse=True)
def isolate_sys_modules():
original = sys.modules.copy()
try:
yield
finally:
sys.modules.clear()
sys.modules.update(original)
def test_something_with_stubbed_google_lib():
sys.modules["googleapiclient"] = ModuleType("googleapiclient")
# ... import/use code under test safely ...
```
Additionally, for refactors that change call paths, either rely on the expanded unit tests above or attach lightweight scenario-based verification to show behavior is unchanged.
---
## Null-safe input handling
<!-- source: infiniflow/ragflow | topic: Null Handling | language: Python | updated: 2026-04-27 -->
Apply null-safety consistently for request payloads, optional dict keys, and nullable fields. Specifically:
- Guard `None` before doing comparisons/logic that assumes a value exists.
- Don’t index dict keys that may be missing; use `dict.get(..., default)` (and prefer early returns).
- When building strings for output/storage, normalize nullable values to empty strings (avoid turning `None` into the literal "None").
- When merging/extending lists, extend with a default empty list if the source may be missing/nullable.
Example pattern:
```python
# 1) Guard None before comparisons
batch_size = max(1, int(os.getenv("PDF_PARSER_PAGE_BATCH_SIZE", "50")))
if total_pages is None or total_pages <= batch_size:
run_single_pass()
# 2) Safe dict key access
content = chunk.get("content")
if not content:
content = chunk.get("content_with_content") or ""
# 3) Avoid serializing None -> "None"
text = str(item.get("text") or "").strip()
if text:
lines.append({"text": text})
# 4) Safe list merge
node0_attrs.setdefault("source_id", [])
node0_attrs["source_id"].extend(node1_attrs.get("source_id", []) or [])
node0_attrs["source_id"] = sorted(set(node0_attrs["source_id"]))
```
This prevents null-reference crashes and stops incorrect downstream data (like literal "None" strings) from being stored or emitted.
---
## Deterministic Mapping Pipeline
<!-- source: infiniflow/ragflow | topic: Algorithms | language: Python | updated: 2026-04-27 -->
When implementing parsing/storage/retrieval logic (or any pipeline with downstream queries), design the algorithm so that *representations, keys, and batching decisions are stable and consistent*:
- **Preserve representation contracts:** If downstream search/filter expects tokenized fields (e.g., `*_tks`), never “simplify” stored values to raw strings for those fields.
- For mixed needs (human-readable + search tokens), **store parallel values** (e.g., tokenized for vector/search fields, raw for metadata display/aggregation).
- **Use authoritative mappings; avoid brittle inference:** If the KB/task snapshot already contains the correct `field_map`/column-to-typed-key mapping, load it rather than guessing via suffix/fuzzy matching (especially for non-ASCII/Pinyin-derived keys). Keep inference/probing only as a last-resort fallback when the authoritative data is unavailable.
- **Implement bounded windowing explicitly:** For batched/chunked processing, compute and pass explicit window bounds (e.g., `page_to = min(page_from + batch, total_pages)`) to avoid default truncation.
- **Make quality gates deterministic:** Any detector used to decide OCR vs text (e.g., garbled-page detection) must be deterministic—remove randomness from sampling or make it repeatable (e.g., `s[:N]` instead of `random.sample`). Prefer thresholding to reduce false positives.
Example pattern (parallel token/raw + bounded, deterministic detection):
```python
# 1) Parallel values: keep token fields tokenized
if role in ("vectorize", "both"):
chunk[f"{typed_key}_tks"] = tokenize(value)
if role in ("metadata", "both"):
chunk[f"_raw_{col}"] = str(value) # human-readable for aggregation/UI
# 2) Bounded windowing
for page_from in range(0, total_pages, batch_size):
page_to = min(page_from + batch_size, total_pages)
load_pages(page_from=page_from, page_to=page_to)
# 3) Deterministic detector sampling
sample = page_chars[:200] # not random.sample
is_garbled = detect(sample, threshold=0.3)
```
Adopting this standard prevents subtle retrieval/search regressions, reduces heuristic flakiness, and improves correctness/performance of chunked algorithms.
---
## Unified Error Handling
<!-- source: infiniflow/ragflow | topic: Error Handling | language: Python | updated: 2026-04-24 -->
Adopt a single error-handling standard across LLM/provider layers:
1) Retry only transient failures (including connection jitter) with bounded, classified backoff
- Centralize retry behavior in shared utilities (e.g., @retry / retry_or_fallback) using error classification.
- Treat connection/network/DNS and rate-limit/server errors as retryable.
- Ensure the last attempt returns/raises a clear max-retry outcome (so callers never “fall through” without an error).
- Don’t silently support unsupported patterns: fail fast for generator/streaming functions (retrying mid-stream is unsafe).
2) Keep return/output contracts stable between success and failure
- Never change output types on exception paths (e.g., if success returns tuple(text, token_count), failure must return the same shape or be normalized before setting outputs).
3) Standardize failure encoding and checking at boundaries
- If providers encode failure in return values (e.g., prefixed strings), require callers to use the shared prefix constant and an is_error_result-style helper instead of ad-hoc string checks.
Example (type-stable normalization before setting output):
```python
try:
transcription = seq2txt_mdl.transcription(tmp_path)
# success may be tuple(text, token_count)
txt = transcription[0] if isinstance(transcription, tuple) else transcription
except Exception as e:
logging.warning(f"Transcription failed: {e}")
txt = ""
self.set_output("text", txt)
```
Example (standard retry rule for transient polling blips):
```python
@retry
def _describe_task_status(self, req):
return self.client.DescribeTaskStatus(req)
while retries < max_retries:
resp = self._describe_task_status(req) # transient 429/5xx/DNS blips survive
```
Result: fewer hidden failure modes, consistent caller behavior, and reliable recovery for transient LLM/API issues.
---
## Use Clear Identifiers
<!-- source: infiniflow/ragflow | topic: Naming Conventions | language: Python | updated: 2026-04-23 -->
Use semantically meaningful, convention-compliant names everywhere (parameters, variables, fields, classes, constants), and avoid ad-hoc renaming/mapping that can create ambiguity or incorrect field exposure.
Apply these rules:
- Name parameters by role: prefer `existing_*` / `incoming_*` (or equivalent) over generic names like `metadata, meta`.
- Follow standard casing:
- variables/functions: `snake_case`
- classes: `PascalCase` (e.g., `GroqChat`)
- constants/settings: `UPPER_SNAKE_CASE` (e.g., `REGISTER_ENABLED`)
- Don’t use misleading string operations for identifier suffixes (e.g., prefer `removesuffix('_app')` over trimming characters with `rstrip`).
- Ensure response/request field mappings don’t duplicate or wrongly alias keys; verify one-to-one intended names.
- If you need derived uniqueness (e.g., appending an index), don’t blindly mutate identifiers at the call site—validate the format or normalize earlier so downstream code isn’t surprised.
Example (naming parameters by role):
```python
def update_metadata_to(existing_metadata: dict, incoming_metadata: dict) -> dict:
"""Merge incoming metadata payload into an existing metadata mapping."""
existing_metadata.update(incoming_metadata)
return existing_metadata
```
---
## Extract Shared Validation
<!-- source: infiniflow/ragflow | topic: Code Style | language: Python | updated: 2026-04-23 -->
Prefer small shared helpers for repeated request parsing/validation, and use modern, consistent type annotations to keep code readable and uniform.
Apply this when you see the same “get field → type-check → return error” logic duplicated across endpoints or functions.
Example (shared helper with validation):
```py
def _parse_reference_metadata(req: dict):
extra_body = req.get("extra_body") or {}
if extra_body and not isinstance(extra_body, dict):
return get_error_data_result("extra_body must be an object."), False, None
reference_metadata = extra_body.get("reference_metadata") or {}
if reference_metadata and not isinstance(reference_metadata, dict):
return get_error_data_result("reference_metadata must be an object."), False, None
metadata_fields = reference_metadata.get("fields")
if metadata_fields is not None and not isinstance(metadata_fields, list):
return get_error_data_result("reference_metadata.fields must be an array."), False, None
return None, bool(reference_metadata.get("include", False)), metadata_fields
async def agents_completion_openai_compatibility(tenant_id, agent_id):
req = await get_request_json()
err, include_reference_metadata, metadata_fields = _parse_reference_metadata(req)
if err:
return err
# ...use include_reference_metadata/metadata_fields...
```
Style rules to enforce with this approach:
- De-duplicate identical parsing/validation blocks by extracting helpers.
- Keep helpers small and single-purpose; avoid adding too many responsibilities to one function.
- Use modern typing syntax consistently (e.g., `tuple[...]`, `A | B` instead of `Tuple[...]`, `Union[...]`).
---
## Use Shared UI Primitives
<!-- source: infiniflow/ragflow | topic: Code Style | language: TSX | updated: 2026-04-22 -->
{% raw %}
When building UI, follow the project’s existing design-system components and tokens instead of creating bespoke patterns.
**Standards**
1. **Prefer shared form/UI building blocks**: Use existing components like `RAGFlowFormItem`, `SimilaritySliderFormField`, and `SearchInput` rather than re-implementing the same UI logic/styling.
2. **Use standard shadcn components for complex layouts**: For table-like lists, use the shadcn `Table` component (not a hand-rolled `<table>` with custom class logic).
3. **Extract complex dialogs into standalone components/files**: If a component contains a full `Dialog` implementation (multi-field forms, footers, local state), move it to a dedicated file/component.
4. **Use global design tokens/colors**: Prefer color variables from `tailwind.css` over ad-hoc inline `style={{ backgroundColor: ... }}` so theming stays consistent.
**Example (extraction + token usage)**
```tsx
// skills/components/CreateSpaceDialog.tsx
import { Dialog, DialogContent, DialogFooter, DialogHeader, DialogTitle, DialogDescription } from '@/components/ui/dialog';
import { Button } from '@/components/ui/button';
import { Input } from '@/components/ui/input';
export function CreateSpaceDialog(props: {
open: boolean;
onOpenChange: (v: boolean) => void;
value: string;
onChange: (v: string) => void;
onCreate: () => void;
onCancel: () => void;
}) {
return (
<Dialog open={props.open} onOpenChange={props.onOpenChange}>
<DialogContent className="sm:max-w-[425px]">
<DialogHeader>
<DialogTitle>Create New Skill Space</DialogTitle>
<DialogDescription>Organize and manage your skills.</DialogDescription>
</DialogHeader>
<div className="py-4">
<label className="text-sm font-medium mb-2 block">Space Name</label>
<Input value={props.value} onChange={(e) => props.onChange(e.target.value)} />
</div>
<DialogFooter>
<Button variant="outline" onClick={props.onCancel}>Cancel</Button>
<Button onClick={props.onCreate} disabled={!props.value.trim()}>Create</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
```
Applying these keeps UI consistent, reduces duplicated styling/behavior, improves readability, and makes future theme/UI updates cheaper.
{% endraw %}
---
## Centralize API Interfaces
<!-- source: infiniflow/ragflow | topic: API | language: TypeScript | updated: 2026-04-22 -->
Enforce consistent API client construction: (1) centralize all REST endpoint paths and interface definitions in a shared module (e.g., `api.ts`) to prevent scattered string literals, and (2) when reading/writing request inputs like query params, use the correct standard APIs and keep TypeScript types aligned.
Apply as follows:
- In services, import route builders/constants instead of embedding `'/api/v1/...` strings.
- For query params, don’t treat `URLSearchParams` as a plain object; iterate via `searchParams.entries()` and type-check values.
Example:
```ts
// web/src/api.ts
export const API_PREFIX = '/api/v1';
export const skillSpaceRoutes = {
list: () => `${API_PREFIX}/skills/spaces`,
detail: (spaceId: string) => `${API_PREFIX}/skills/spaces/${spaceId}`,
};
// web/src/services/skill-space-service.ts
import request from '@/utils/request';
import { skillSpaceRoutes } from '@/api';
class SkillSpaceService {
private async request<T>(method: string, url: string, data?: any, params?: any): Promise<T> {
const response: any = await request(url, { method: method as any, data, params });
const jsonData = response?.data ?? response;
if (jsonData?.code !== 0) throw new Error(jsonData?.message || 'Request failed');
return jsonData.data;
}
async listSpaces() {
return this.request<{ spaces: any[]; total: number }>('GET', skillSpaceRoutes.list());
}
}
```
And for query params:
```ts
const getParams = (searchParams: URLSearchParams) => {
const out: Record<string, string> = {};
for (const [key, value] of searchParams.entries()) {
out[key] = value;
}
return out;
};
```
---
## Typed API Contracts
<!-- source: infiniflow/ragflow | topic: API | language: Python | updated: 2026-04-21 -->
For each API endpoint, treat the request as a strict contract: define a dedicated typed request model for the exact payload shape, validate required inputs early, and keep auth/tenant scoping and client URL construction consistent.
Apply:
- Use a dedicated request schema per endpoint (don’t reuse a model if the payload shape differs).
- Validate required fields immediately (e.g., reject empty or missing required collections).
- Build/whitelist update payloads from allowed fields only.
- Use centralized auth (e.g., a decorator) rather than manual token parsing; enforce tenant ownership/scoping invariants for create/update/delete.
- Keep endpoint base URL/versioning and Authorization header formats consistent between client and server.
Example (endpoint + Pydantic request validation):
```python
from pydantic import BaseModel, Field
from flask import request
class UpdateMetadataSettingReq(BaseModel):
metadata: dict
enable_metadata: bool | None = None
built_in_metadata: dict | None = None
@app.route('/document/update_metadata_setting', methods=['POST'])
@token_required
def update_metadata_setting(tenant_id):
req = UpdateMetadataSettingReq.model_validate(request.json)
update_payload = {"metadata": req.metadata}
if req.enable_metadata is not None:
update_payload["enable_metadata"] = req.enable_metadata
if req.built_in_metadata is not None:
update_payload["built_in_metadata"] = req.built_in_metadata
# additionally enforce tenant ownership before updating
DocumentService.update_parser_config(doc_id=req_id, update_payload=update_payload)
return get_json_result(data=True)
```
This prevents payload mismatches, inconsistent behavior across endpoints, and security/ownership mistakes while keeping API clients predictable (stable URL/version and auth header conventions).
---
## PascalCase constants
<!-- source: infiniflow/ragflow | topic: Naming Conventions | language: TypeScript | updated: 2026-04-19 -->
Use a consistent PascalCase naming convention for constants (especially exported ones). Avoid mixed styles like snake_case, and keep acronym casing consistent (e.g., API).
When you rename a constant to match the standard, update all references/usages together (including “lite”/variant constants) to stay DRY and prevent mismatches.
Example:
```ts
// ❌ Avoid snake_case
// const api_host = `/v1`;
// ✅ Prefer PascalCase for constants, consistent acronym casing
const webAPI = `/v1`;
export const MarkdownRemarkPlugins = [remarkGfm, remarkMath, remarkBreaks];
export const MarkdownRemarkPluginsLite = [remarkGfm, remarkBreaks];
```
---
## Config Contracts And Defaults
<!-- source: infiniflow/ragflow | topic: Configurations | language: Python | updated: 2026-04-14 -->
Treat environment variables and configuration flags as a documented contract: the code must (1) read settings from the correct source with safe defaults, (2) make feature-flag behavior explicit (including what becomes a true no-op when disabled), and (3) validate any external runtime dependency that the integration relies on.
Apply this standard:
- Feature flags: if a toggle changes output structure, confirm the “disabled” behavior is intentional and harmless to the core payload.
- Example pattern:
```python
docs = {}
if include_document_metadata:
documents = DocumentService.get_by_ids([...])
docs = {d.id: DocMetadataService.get_document_metadata(d.id) or {} for d in documents}
# When false, metadata loop should no-op; chunk title/url/content still come from chunks.
```
- Config/env parsing: use the intended settings/env source and add a default to avoid breaking existing deployments.
- Example pattern:
```python
secure = str(settings.MINIO.get("secure", False)).lower() == "true"
client = Minio(host, access_key=..., secret_key=..., secure=secure)
```
- External dependency validation: if a library shells out (e.g., Java) or relies on host tools, detect and warn/fail early with the required version and where it must be installed.
- Environment variable semantics: document assumptions about env-var behavior (e.g., `CUDA_VISIBLE_DEVICES` remapping makes device indices continuous within the process), and ensure the code aligns with those semantics.
- Avoid config mistakes: keep imports consistent with the intended module (e.g., `DEBUG` from the correct settings namespace).
---
## Safe configuration changes
<!-- source: infiniflow/ragflow | topic: Configurations | language: Other | updated: 2026-04-11 -->
When updating configuration/env logic, treat it as compatibility-sensitive: preserve existing default behavior and env var formats, avoid enabling config blocks that silently change runtime semantics, and ensure config typing and dependent settings stay consistent.
Apply these rules:
- Preserve existing env var formats/semantics (or add a compatibility path). Example: keep ES_HOST in its prior format rather than switching it to a URL if templates still assume the old form.
- Don’t uncomment/enable template sections that change defaults. Prefer explicit flags controlled via the corresponding .env.
- Keep .env and config templates in sync; if versions can diverge for users, make the smallest possible change to reduce confusion/breakage.
- Ensure correct config types: in YAML templates, don’t accidentally turn numbers into strings (e.g., quoting values can cause type errors).
- Update paired dependent configs when you change limits/settings (e.g., if you change MAX_CONTENT_LENGTH, also update nginx client_max_body_size).
Example pattern (YAML typing):
```yaml
# Good: numeric port
port: '${MYSQL_PORT:-3306}'
# Avoid patterns that can cause type issues depending on templating/YAML parsing rules
# e.g., port: '3306' (string) if the consumer expects an int
```
---
## Nil-safe assignment
<!-- source: looplj/axonhub | topic: Null Handling | language: Go | updated: 2026-04-11 -->
When fields/items are optional, treat `nil`/empty as a first-class state: guard against nil dereferences, sanitize outputs, and never overwrite meaningful non-nil data with `nil`.
Apply:
- **Nil-aware assignment**: only assign optional pointers/values when they are non-nil (avoid forcing `XContent = nil`).
- **Defensive branching**: when logic depends on “completion”, don’t assume related payload fields are non-empty; keep/introduce fallbacks for `len(...)==0`.
- **Output sanitization**: filter `nil`/empty stream elements so downstream consumers see only valid items.
- **Normalize optional inputs**: if a downstream expects a non-pointer, convert/normalize with explicit nil checks rather than unsafe deref.
Example:
```go
// Nil-aware mapping: don’t overwrite with nil
if len(m.ReasoningDetails) > 0 {
// ... build from details ...
} else if m.Reasoning != nil {
m.ReasoningContent = m.Reasoning
}
// Defensive fallback for empty payload
if len(responseBody) > 0 {
persistAggregatedResponse(ctx, responseBody, meta)
} else {
// aggregated “completed” doesn’t guarantee responseBody presence
persistResponseChunks(ctx)
}
// Stream sanitization: filter nil items
for _, item := range items {
if item == nil {
continue
}
emit(item)
}
```
---
## Non-Blocking Async Recovery
<!-- source: infiniflow/ragflow | topic: Concurrency | language: Python | updated: 2026-04-10 -->
Concurrency standard: (1) keep async code non-blocking, (2) don’t change sync/async boundaries unless callers support it, and (3) make recovery/state transitions race-safe.
**Rules**
1) **Never call blocking (sync) work from async tasks**. If you must use a sync API (e.g., `llm.chat()`), offload it (or refactor to an async client).
2) **Match sync/async signatures to callers**. If a function is only called from sync code, keep it sync (don’t make it `async` just to “fit”); if callers are async, then the callee can be async.
3) **Recovery/requeue logic must avoid TOCTOU**. When detecting “orphaned” or incomplete tasks, use an atomic snapshot, transaction, or a re-check/lock so you don’t incorrectly downgrade DONE → FAIL.
4) **For time-sensitive heartbeat/recovery, avoid blocking the event loop**. If Redis operations or other work are blocking, run them in a dedicated worker thread/process or ensure the client calls are truly non-blocking.
5) **Use explicit stop coordination for graceful shutdown** (e.g., stop_event set by SIGTERM/SIGINT) so recovery loops exit cleanly.
**Example: offload sync call from async**
```python
import asyncio
async def chat_async(chat_model, system_msg, hist, conf):
# chat_model.chat is sync; offload so the event loop stays responsive
response = await asyncio.to_thread(chat_model.chat, system_msg, hist, conf)
return response
```
**Example: avoid TOCTOU in requeue**
- Perform the “orphaned docs” lookup and “incomplete tasks” lookup under a consistent transaction, or
- Re-check document/task state right before updating, and only mark FAIL if the state still indicates “no incomplete tasks.”
Applying these rules prevents event-loop stalls, fixes incorrect concurrency boundaries, and eliminates correctness bugs during crash recovery and task requeue.
---
## Consistent Accessible UI
<!-- source: diegosouzapw/OmniRoute | topic: Code Style | language: TSX | updated: 2026-03-16 -->
When adding expandable/conditional UI in React/TSX, keep the structure consistent within the surrounding component *and* ensure accessibility attributes are applied uniformly.
Apply this standard:
- If the same “advanced settings” pattern appears multiple times, consider extracting a reusable component (e.g., `AdvancedPathSettings`) to reduce duplication once it’s repeated.
- Regardless of whether you inline or extract, interactive toggles must include the correct ARIA wiring, and decorative indicators (like a chevron) must be marked as non-informative.
Example (toggle + conditional section, with consistent ARIA):
```tsx
function AdvancedToggle({ showAdvanced, setShowAdvanced }: {
showAdvanced: boolean;
setShowAdvanced: (v: boolean) => void;
}) {
const id = "advanced-settings";
return (
<>
<button
type="button"
className="text-sm text-text-muted hover:text-text-primary flex items-center gap-1"
aria-expanded={showAdvanced}
aria-controls={id}
onClick={() => setShowAdvanced(!showAdvanced)}
>
<span
className={`transition-transform ${showAdvanced ? "rotate-90" : ""}`}
aria-hidden="true"
>
▶
</span>
advanced settings
</button>
{showAdvanced && (
<div id={id} className="flex flex-col gap-3 pl-2 border-l-2 border-border">
{/* advanced inputs here */}
</div>
)}
</>
);
}
```
---
## Fail-Closed Credential Checks
<!-- source: diegosouzapw/OmniRoute | topic: Security | language: JavaScript | updated: 2026-03-14 -->
When validating security state (e.g., whether credentials are encrypted, authorized sessions exist, tokens exist, etc.), treat any inability to inspect sensitive data as a security failure—don’t silently default to a permissive/incorrect outcome. In practice:
- Do not swallow exceptions in security checks (no empty `catch {}` that turns “unknown” into a benign value).
- Prefer “fail closed”: if the inspection DB/read/parsing step errors, return a conservative result (often “encrypted not confirmed” / “credentials not trusted”) and/or throw so the caller can block the unsafe path.
- Keep the change narrowly scoped: security PRs should avoid unrelated refactors that expand module/runtime surface.
Example pattern (fail closed on inspection errors):
```js
function hasEncryptedCredentials(dbPath) {
if (!fs.existsSync(dbPath)) return false;
try {
const Database = require("better-sqlite3");
const db = new Database(dbPath, { readonly: true, fileMustExist: true });
try {
const row = db
.prepare(`
SELECT 1
FROM provider_connections
WHERE access_token LIKE 'enc:v1:%'
OR refresh_token LIKE 'enc:v1:%'
OR api_key LIKE 'enc:v1:%'
OR id_token LIKE 'enc:v1:%'
LIMIT 1
`)
.get();
return !!row; // only true when confirmed
} finally {
db.close();
}
} catch (e) {
// Fail closed: do not treat inspection failure as a safe state.
throw new Error(`Unable to inspect existing database at ${dbPath}`);
// Alternatively: return false only if callers interpret false as “block/require re-encryption”.
}
}
```
Apply this standard anytime code makes a security decision based on reading/parsing sensitive material (DBs, token stores, config files, key material). If you’re changing security-critical logic, avoid bundling unrelated refactors that could increase runtime/module-loading risk during the fix.
---
## Enforce Layered Reuse
<!-- source: looplj/axonhub | topic: Code Style | language: Go | updated: 2026-03-04 -->
When changing or adding code, keep logic and dependencies clean:
- Centralize invariants in the owning function: if you need to clamp/validate values, apply it inside the core method that produces/returns them (the “source of truth”) rather than at every call site.
- Respect package layering: lower-level packages must not import higher-level/business packages (no reverse dependencies). Define contracts in the appropriate layer so dependencies flow downward only.
- Reuse existing utilities: avoid adding tiny wrappers/helpers when an approved library/helper already exists.
Example (centralize clamping):
```go
// prefer: clamp inside Calculate (source of truth)
first, request, _ := ts.perf.Calculate() // already returns clamped request
```
Example (no reverse dependency):
- If `llm` is a bottom package, it should not `import objects` (business). Move shared types to a neutral/lower package or pass data via interfaces defined at the lower layer.
Example (reuse utility):
```go
v := lo.ToPtr(123) // instead of custom xptr.IntPtr(123)
```
---
## Deep-copy cached state
<!-- source: looplj/axonhub | topic: Caching | language: TSX | updated: 2026-02-07 -->
When cached data (e.g., `row.settings`) is used to initialize mutable UI state (forms, editors, draft objects), do not pass cached object references directly. Always deep-clone cached objects on entry to editable state to prevent unintended cache mutations and ensure copied/duplicated initialization is isolated.
Example pattern:
```ts
const settings = currentRow.settings
? structuredClone(currentRow.settings)
: undefined;
// use `settings` as initial/editable state
const initialValues = {
...,
settings,
};
```
If `settings` is nested, prefer `structuredClone` (or an equivalent deep-clone) over shallow spreads to guarantee full isolation.
---
## Document Stable API Contracts
<!-- source: infiniflow/ragflow | topic: API | language: Markdown | updated: 2026-02-05 -->
All API reference docs (and examples) must clearly define the *public* contract vs *temporary/hidden* behavior, and keep endpoint paths/query params and response fields consistent.
Apply:
- Add explicit caution notes for behavioral differences (e.g., streaming vs non-stream payload fields). If a field is unavailable, document it and how to obtain it (e.g., non-stream with raw response).
- Mark hidden/unstable parameters as such and state that they must not be treated as a stable public API.
- Ensure endpoint paths are formatted correctly (no accidental extra slashes) and query parameters in URLs match the documented schema.
- Provide practical request examples that cover common patterns, including multi-value/array query parameters.
- Keep response schema fields consistent across endpoints (e.g., don’t use `code` and `error_code` interchangeably without a documented rule).
Example pattern (streaming caveat):
```md
:::caution NOTE
`client.chat.completions.create(stream=True, ...)` does not return `reference`.
`reference` is only exposed in the non-stream response payload (or via raw non-stream mode).
Use non-stream to access `reference`.
:::
```
---
## Scoped Provider Transformations
<!-- source: looplj/axonhub | topic: AI | language: Go | updated: 2026-01-24 -->
When implementing AI/LLM request transformers (especially for proxies), make provider-specific shaping:
1) **Scoped and structured**: Clone the incoming `llm.Request` and apply provider/CLI-specific changes via existing structured fields **before** delegating to the shared/base transformer whenever possible. Avoid raw `[]byte` request-body manipulation unless the field truly can’t be represented structurally.
2) **No shared-model pollution**: If a change (e.g., tool-name prefixing) would break assumptions for other transformers, ensure the change is applied only within the cloned request or handled by the provider-specific transformer layer.
3) **Idempotent system prompt injection**: If your proxy adds a provider system message (e.g., Claude Code), first detect whether the request already contains that system prompt (or any system prompt) to prevent duplicates.
Example (idempotent system injection + scoped preprocessing):
```go
func (t *ClaudeCodeTransformer) TransformRequest(ctx context.Context, llmReq *llm.Request) (*httpclient.Request, error) {
if llmReq == nil { return nil, fmt.Errorf("request is nil") }
// Always work on a copy to avoid cross-request/state issues.
reqCopy := *llmReq
// 1) Idempotent system message injection.
const sys = "You are Claude Code, Anthropic's official CLI for Claude."
if !hasSystemPrompt(reqCopy.Messages, sys) {
systemMsg := llm.Message{
Role: "system",
Content: llm.MessageContent{Content: lo.ToPtr(sys)},
}
reqCopy.Messages = append([]llm.Message{systemMsg}, reqCopy.Messages...)
}
// 2) Provider-specific structured preprocessing (only what the model supports).
// Apply tool prefixing / cache control / metadata shaping only to reqCopy,
// and keep other transformers unaffected.
applyClaudeToolPrefix(&reqCopy)
// (If a needed field is not representable structurally, then do minimal byte surgery later.)
return t.Outbound.TransformRequest(ctx, &reqCopy)
}
func hasSystemPrompt(msgs []llm.Message, sys string) bool {
for _, m := range msgs {
if m.Role == "system" && m.Content.Content != nil && *m.Content.Content == sys {
return true
}
}
return false
}
```
Adopting these rules improves **correctness** (no duplicated system instructions), **maintainability** (less brittle byte-level edits), and **compatibility** (provider specifics don’t unintentionally affect other LLM transformers).
---
## Token and Prompt Fidelity
<!-- source: infiniflow/ragflow | topic: AI | language: Python | updated: 2026-01-22 -->
When implementing LLM/AI APIs (including streaming), treat three things as contract-critical: (1) compute token usage with a real tokenizer, (2) ensure the intended system/user context is actually passed through every planning/tool stage, and (3) emit only the events your client expects.
**1) Token usage must use tokenizer-based counting**
Avoid `len(text)` for `prompt_tokens/total_tokens`; it’s character-based and will be wrong across languages/models. Use a tokenizer compatible with your target model.
```python
import tiktoken
tokenizer = tiktoken.get_encoding("cl100k_base")
context_token_used = sum(len(tokenizer.encode(m["content"])) for m in messages)
# use similar encoding for prompt_tokens/completion_tokens
```
**2) Prompt plumbing must be correct**
If an agent/planning function takes `prompt`/`sys_prompt`/history, pass the correct variables. Add a small assertion or unit test to ensure downstream components receive the system prompt intended for the run (not a different variable like history or an uninitialized prompt).
**3) Streaming/compatibility endpoints must filter noise**
If the upstream emits multiple event types (e.g., content vs reference chunks), only forward the subset your client UI consumes. Example logic:
```python
# keep only the LLM response and its related finalization/reference
if event not in ["message", "message_end"]:
continue
```
**How to apply**
- For every LLM endpoint response, confirm token fields are derived from tokenizer counts.
- For every agent/tool/planning path, verify the system prompt and user history are the same objects used to build the actual prompt that reaches planning.
- For streaming, ensure you accumulate deltas consistently and only emit client-meaningful events.
---
## Transactional delete and upsert
<!-- source: infiniflow/ragflow | topic: Database | language: Python | updated: 2026-01-15 -->
For database writes that can partially succeed (multi-step deletes/updates, bulk upserts with conflict handling), enforce invariants with **atomic transactions**, **explicit outcome validation**, and **engine-specific SQL encapsulation**.
Apply this standard:
- Wrap helper methods that perform writes in `@DB.atomic()` (or an equivalent transaction boundary).
- After deletes/updates, validate affected-row counts against the expected scope (e.g., dedupe ids and check counts). If partial success breaks invariants, use a clearly justified compensating action.
- When supporting multiple DB engines, don’t assume identical upsert/conflict syntax—branch or adapterize the conflict resolution configuration, but keep the same invariant checks and error handling.
Example (pattern):
```python
@classmethod
@DB.atomic()
def delete_chunks(cls, chunk_ids, doc_id, kb_id):
# dedupe inputs to make expected counts deterministic
unique_ids = list(dict.fromkeys(chunk_ids or []))
if not unique_ids:
return 0
condition = {"id": unique_ids, "doc_id": doc_id}
try:
deleted_count = settings.docStoreConn.delete(
condition,
search.index_name(DocumentService.get_tenant_id(doc_id)),
kb_id,
)
except Exception:
return 0
# If partial deletion happened, restore consistency deterministically.
if deleted_count == 0:
raise ValueError("chunk deleting failure")
if deleted_count < len(unique_ids):
return settings.docStoreConn.delete(
{"doc_id": doc_id},
search.index_name(DocumentService.get_tenant_id(doc_id)),
kb_id,
)
return deleted_count
```
Also ensure bulk upsert/bulk insert logic handles DB-specific conflict semantics (e.g., Postgres vs MySQL) inside the DB utility layer, while still using the same atomicity and validation rules at the call site.
---
## UseEffect Dependency Correctness
<!-- source: infiniflow/ragflow | topic: React | language: TSX | updated: 2026-01-14 -->
Side effects must be written with the correct React lifecycle mental model: `useEffect` runs after render/commit, not at the place the code is written. Always include a dependency array and make the effect’s inputs explicit so state resets (e.g., initializing defaults when a dialog opens) happen at the right time and without stale reads.
Apply this standard:
- If logic depends on props/state (e.g., `visible`, `defaultValues`, `formCallbackRef`), include them in the `useEffect` dependency list.
- Perform “reset when dialog opens” inside an effect keyed to the open/visible flag rather than relying on render-order.
Example (dialog reset):
```ts
useEffect(() => {
const initialDeleteValue = chunk_num > 0;
setDefaultValues({ delete: initialDeleteValue, apply_kb: false });
if (visible && formCallbackRef.current) {
formCallbackRef.current.reset({ delete: initialDeleteValue, apply_kb: false });
}
}, [visible, chunk_num, setDefaultValues]);
```
Also: extract repeated side-effect logic into hooks when appropriate (to avoid duplicated effect code), but keep each hook’s effect dependencies correct.
---
## Maintain dependency lockfiles
<!-- source: infiniflow/ragflow | topic: Configurations | language: Toml | updated: 2025-12-19 -->
When changing dependency declarations in configuration files like `pyproject.toml`, treat group placement and lockfile updates as part of the change.
- **Update/submit the lockfile** whenever dependencies or version constraints are modified (e.g., commit the corresponding `uv.lock`).
- **Preserve required dependency scope**: if a dependency is needed by multiple groups (such as both `default` and `test`), keep it declared in each required group—don’t “move” it to another group just to reduce duplication.
Example (pattern):
```toml
[project.optional-dependencies]
# Keep in both groups if both are required
default = ["pycryptodomex==3.20.0"]
test = ["pycryptodomex==3.20.0", "websockets>=14.0"]
```
After editing, regenerate and commit the matching lockfile (e.g., `uv.lock`).
---
## Semantic identifier naming
<!-- source: infiniflow/ragflow | topic: Naming Conventions | language: Markdown | updated: 2025-12-09 -->
Use semantic, unambiguous names for every externally visible identifier (API parameters/endpoints, config/env vars, and build/version tags). Names must reflect meaning and supported behavior; if an operation is not supported, document that constraint next to the relevant field/behavior.
Application rules:
- API naming: prefer consistent, descriptive snake_case names for parameters (e.g., `parent_id`, `file_ids`, `page_size`, `orderby`, `desc`) and keep them aligned with the actual request/response contract.
- API constraints: when behavior is restricted, state it explicitly (e.g., “Changing file extensions is *not* supported.”) so users don’t infer unsupported semantics from the parameter/field naming.
- Build/release naming: standardize version tag labels so stability and feature set are obvious.
- Use `vX.Y.Z` for stable releases.
- Use `vX.Y.Z-slim` for stable “slim” variants.
- Use `dev` (not ambiguous labels like `dev1`) for development images.
- Use `nightly` for nightly builds.
Example (Docker tag standard):
```text
RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1
RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1-slim
RAGFLOW_IMAGE=infiniflow/ragflow:dev
RAGFLOW_IMAGE=infiniflow/ragflow:nightly
```
Example (API constraint wording):
```text
POST /api/v1/file/rename
Body: { "file_id": "...", "name": "new_name.txt" }
Note: Changing file extensions is not supported.
```
---
## Documentation Precision
<!-- source: infiniflow/ragflow | topic: Documentation | language: Markdown | updated: 2025-12-03 -->
All documentation must accurately reflect product/API behavior, use exact feature names, and be unambiguous about constraints, defaults, and unsupported scenarios. Also ensure Markdown renders cleanly.
Apply this when editing or adding docs:
- **Match exact product/UI terminology**: use the same feature names as the product (e.g., “File” vs “File Management”) everywhere.
- **State exact behavior for edge cases**: if the UI allows non-integers, document how values are handled (e.g., rounding).
- **Fully define parameters and semantics**:
- For API pagination, describe what `page` and `page_size` mean (offset/limit vs “records per page”) and keep parameter names typo-free.
- **Document defaults and constraints** (ranges, enable/disable conditions) and **include failure/unsupported cases** when a feature can’t be used as a retrieval source.
- **Keep Markdown clean and consistent**: remove unnecessary escaping (e.g., don’t escape `@` in normal Markdown) and standardize spacing/punctuation in READMEs.
Example (API param + constraint precision):
```md
## List files
**GET** `/api/v1/dataset/{dataset_id}/documents?keywords={keyword}&page={page}&page_size={page_size}`
- `page`: 0-based page index.
- `page_size`: number of records returned per page.
## Page rank
- Value type: integer
- Non-integers are rounded down.
- Range: [0, 100]
```
---
## Documentation Wiring Accuracy
<!-- source: infiniflow/ragflow | topic: Documentation | language: Other | updated: 2025-12-03 -->
Ensure technical documentation is both semantically correct and operationally precise: (1) use the exact product terms for what a setting applies to (scope/terminology), and (2) when describing component internals/graphs, specify the exact wiring rule for references (use the internal managing node, not an external component), aligned with the documented visibility constraints.
How to apply:
- Scope/terminology check: If a doc claims that a config limit “applies to X,” confirm the exact system feature name and what scenario it does/does not apply to.
- Graph/reference check: For internal workflows, document the concrete reference variable to use (e.g., the internal loop controller/seed node) and explicitly state when external references are unnecessary.
Example (Iteration internal wiring pattern):
```mdx
:::danger IMPORTANT
To reference the created segments from an added internal component, add a **Reference** variable that equals **IterationItem** within the **Input** section of that internal component. There is no need to reference the corresponding external component, as the **IterationItem** component manages the loop of the workflow for all created segments.
:::
```
Example (setting scope/terminology):
```mdx
The variables `MAX_CONTENT_LENGTH` in `/docker/.env` and `client_max_body_size` in `/docker/nginx/nginx.conf` set the file size limit for each upload to a dataset or **File**. These settings DO NOT apply in this scenario.
```
---
## Use enum constants
<!-- source: infiniflow/ragflow | topic: Naming Conventions | language: TSX | updated: 2025-11-28 -->
When a value represents a named domain concept (e.g., Theme), use the corresponding enum type/constant (e.g., `ThemeEnum`) instead of duplicating raw string literals (`'light'`, `'dark'`). This keeps identifier naming consistent across schema definitions and UI/component logic.
Example pattern:
```ts
// Prefer a single source of truth
export enum ThemeEnum {
Light = 'light',
Dark = 'dark',
}
// Derive schema from the enum rather than retyping literals
import { z } from 'zod';
const FormSchema = z.object({
theme: z.enum([ThemeEnum.Light, ThemeEnum.Dark] as [string, string]),
});
// And use enum values in UI/selectors
<RadioGroup
value={field.value}
onValueChange={field.onChange}
>
<RadioGroupItem value={ThemeEnum.Light} id="light" />
<RadioGroupItem value={ThemeEnum.Dark} id="dark" />
</RadioGroup>
```
---
## Use exec-form commands
<!-- source: infiniflow/ragflow | topic: Security | language: Yaml | updated: 2025-11-19 -->
When configuring container commands (e.g., Docker Compose), pass commands in exec/array form for any arguments that include secrets or externally provided values. This prevents shell parsing/whitespace issues and reduces the risk of injection via malformed values (including empty credentials).
Example (safe):
```yaml
services:
redis:
image: valkey/valkey:8
command: [
"redis-server",
"--requirepass", "${REDIS_PASSWORD}",
"--maxmemory", "128mb",
"--maxmemory-policy", "allkeys-lru"
]
```
Avoid string-form commands that rely on implicit parsing:
```yaml
# Not recommended for secrets/external values
command: redis-server --requirepass ${REDIS_PASSWORD} --maxmemory 128mb
```
---
## Portable Config Management
<!-- source: infiniflow/ragflow | topic: Configurations | language: Yaml | updated: 2025-11-13 -->
When changing configuration, treat runtime configuration as the source of truth and avoid embedding/duplicating settings.
Apply these rules:
1) Prefer `.env` + `env_file: .env` and avoid duplicating the same variables in multiple places.
2) Do not hard-code environment-specific identifiers like `container_name` in shared compose files.
3) For frequently changed/overridden configuration (including templates), mount files via volumes rather than baking templates into images that would require rebuilds to take effect.
4) If one setting is coupled to another (e.g., application upload limits and nginx limits), update both together.
Example (compose):
```yaml
services:
app:
env_file: .env
# no container_name in shared files
volumes:
- ./service_conf.yaml:/ragflow/conf/service_conf.yaml
```
---
## Config Documentation Constraints
<!-- source: infiniflow/ragflow | topic: Configurations | language: Markdown | updated: 2025-11-11 -->
When adding or updating configuration-related docs (env vars, config files, feature/transport flags, and deployment commands), ensure the guidance is (1) context-correct, (2) compatibility-checked, and (3) explicit about defaults/overrides.
Apply this checklist:
- Context-match every file/command: clearly distinguish “source launch” vs “dockerized launch” and reference the correct config targets (e.g., templates vs generated configs; soft links vs real files).
- Declare incompatibilities and required flag combinations: if a mode requires disabling a transport/feature, state it as a hard rule with the exact flag.
- Make defaults explicit: explain what is enabled by default and how to disable/override specific behaviors with the corresponding “no-<flag>” options.
- Provide precision in examples: include the exact env var names, file paths, and command syntax; avoid ambiguous phrasing.
- Add a short “why” for non-obvious constraints (e.g., version pinning to keep entrypoint/image behavior consistent).
Example (flag compatibility + defaults):
```md
If you set `mcp-mode` to `host`, you must add `--no-transport-streamable-http-enabled`, because streamable-HTTP transport is not supported in host mode.
The legacy SSE transport and streamable-HTTP (JSON responses) are enabled by default unless explicitly disabled with the corresponding `--no-<flag>` options.
```
Example (context-correct config guidance):
- For source launch, instruct changes to the source config (e.g., `conf/service_conf.yaml`).
- For docker launch, instruct changes to the docker template (e.g., `docker/service_conf.yaml.template`), and don’t mix them in the same step.
---
## meaningful exception handling
<!-- source: microsoft/markitdown | topic: Error Handling | language: Python | updated: 2025-09-17 -->
Exception handling should provide meaningful feedback and use appropriate exception types rather than failing silently or relying on fragile string-based checks. Avoid catching exceptions without proper logging, user feedback, or recovery mechanisms. Use built-in Python exceptions when available instead of creating custom ones unnecessarily.
Key practices:
- Never catch exceptions silently - always log warnings or provide user feedback
- Use built-in exception types (like FileNotFoundError) instead of custom exceptions when appropriate
- Avoid checking exception messages with string comparisons as they can be fragile across versions or locales
- Provide clear error context to help with debugging
Example of problematic silent handling:
```python
try:
result = subprocess.run(libreoffice_command, capture_output=True, text=True)
# ... process result
except Exception as _:
# Silent failure - no logging or user feedback
pass
```
Better approach with meaningful handling:
```python
try:
result = subprocess.run(libreoffice_command, capture_output=True, text=True)
# ... process result
except subprocess.SubprocessError as e:
logger.warning(f"LibreOffice conversion failed: {e}")
return None
except FileNotFoundError: # Use built-in exception
raise FileNotFoundError(f"File {path} does not exist")
```
---
## Optimize data structures
<!-- source: langgenius/dify | topic: Algorithms | language: Python | updated: 2025-09-13 -->
Choose appropriate built-in data structures and collection types to improve code efficiency and readability. Python's collections module and built-in data structures often provide more elegant and performant solutions than manual implementations.
Key guidelines:
- Use `collections.UserDict` instead of inheriting from `dict` when you need to override dictionary behavior
- Use `collections.defaultdict` to eliminate manual key existence checks and initialization
- Leverage dictionary comprehensions for deduplication and data transformation tasks
Examples:
```python
# Instead of manual dictionary building:
target_nodes: dict[str, list[GraphEdge]] = {}
for edge in edge_mappings:
if edge.target_node_id not in target_nodes:
target_nodes[edge.target_node_id] = []
target_nodes[edge.target_node_id].append(edge)
# Use defaultdict:
from collections import defaultdict
target_nodes = defaultdict(list)
for edge in edge_mappings:
target_nodes[edge.target_node_id].append(edge)
# For deduplication, instead of manual loop:
unique_documents = []
doc_ids = set()
for document in documents:
if document.metadata["doc_id"] not in doc_ids:
doc_ids.add(document.metadata["doc_id"])
unique_documents.append(document)
# Use dictionary comprehension:
unique_documents = list({doc.metadata["doc_id"]: doc for doc in documents}.values())
```
These optimizations reduce code complexity, improve performance, and make intent clearer to other developers.
---
## Use descriptive names
<!-- source: langgenius/dify | topic: Naming Conventions | language: Python | updated: 2025-09-11 -->
Choose names that clearly describe the purpose, behavior, or content of variables, methods, and classes. Avoid ambiguous or generic names that require additional context to understand.
Key principles:
- Include units or context in names when relevant (e.g., `timeout_seconds` instead of `timeout`)
- Use descriptive verbs for methods that indicate their action (e.g., `create_or_update` instead of `create_alias`)
- Specify the role or relationship in parameter names (e.g., `_creator_user_id` instead of `_created_by`)
- Use plural forms for collections to avoid ambiguity (e.g., `doc_ids` instead of `doc_id`)
- Choose method names that accurately reflect their behavior (e.g., `preserve_flask_contexts` instead of `flask_context_manager`)
Examples of improvements:
```python
# Before: Ambiguous naming
def create_alias(user, timeout):
doc_id = set()
# After: Descriptive naming
def create_or_update_alias(creator_user_id, timeout_seconds):
doc_ids = set()
```
This practice makes code self-documenting and reduces the cognitive load for developers reading and maintaining the codebase.
---
## Agent Prompt Contracts
<!-- source: infiniflow/ragflow | topic: AI | language: Other | updated: 2025-09-11 -->
When building LLM Agents, treat the system prompt + framework prompt blocks as a contract with the agent’s inputs and tool/sub-agent configuration.
Apply these rules:
- Bind required variables: an Agent’s data inputs come from keys (variables) that must be used together with the system prompt. In practice, open the key list (e.g., `/` or the **(x)** button) and ensure every variable referenced by the prompt/framework blocks is supplied.
- Use framework blocks with the expected inputs: for example, `task_analysis` should receive `agent_prompt`, `task`, `tool_desc`, and `context`, and `plan_generation` should generate the next execution plan from `task_analysis` results.
- Make tool usage deterministic: if tools are available, specify in the prompt what kinds of tasks should trigger which tools.
- Don’t assume “lead agent only” activation: framework prompt blocks should appear/operate when an Agent has Tools or Subagents configured (verify by checking the **Framework** dropdown after configuring Tools/Subagents), and ensure subagent delegation keeps the prompt contract intact.
Example pattern (conceptual):
```text
System prompt (with variables):
- Role + instructions
- Tool policy: “Use Tool A for classification/summarization; use Tool B for retrieval; otherwise ask a clarifying question.”
Framework blocks:
- task_analysis(agent_prompt, task, tool_desc, context) -> produces structured analysis
- plan_generation(...) -> creates the next steps based on that analysis
Keys:
- Provide all variables referenced by system prompt/framework blocks.
- Confirm required keys exist via the keys UI (/ or (x)).
```
---
## Document for team readability
<!-- source: langgenius/dify | topic: Documentation | language: TSX | updated: 2025-09-11 -->
Ensure code includes appropriate documentation to improve readability and help team members understand and work with the codebase effectively. This includes adding JSDoc comments for new components and features, and preserving existing documentation patterns like internationalization keys that serve as format specifications.
For new components, add JSDoc comments explaining purpose and functionality:
```typescript
/**
* Audio configuration component that manages file upload settings
* for audio files in the workflow features
*/
const ConfigAudio: FC = () => {
// component implementation
}
```
When refactoring, maintain existing documentation patterns rather than removing them:
```typescript
// Keep i18n keys that document expected formats
const timeFormat = needTimePicker ? t('time.dateFormats.displayWithTime') : t('time.dateFormats.display')
// Instead of hardcoded strings like 'YYYY-MM-DD HH:mm'
```
Well-documented code increases readability and makes it easier for other developers to pick up and work with unfamiliar code sections.
---
## prefer nullish coalescing operator
<!-- source: langgenius/dify | topic: Null Handling | language: TSX | updated: 2025-09-11 -->
Use the nullish coalescing operator (`??`) instead of logical OR (`||`) when you want to provide default values only for null or undefined, while preserving other falsy values like empty arrays, empty strings, or zero.
The `||` operator treats all falsy values (null, undefined, false, 0, "", []) as requiring the default, while `??` only treats null and undefined as requiring the default. This distinction is crucial for preserving intentionally falsy but valid values.
Example:
```typescript
// ❌ Problematic - loses empty arrays
...(draft.file?.allowed_file_types || [])
// ✅ Better - preserves empty arrays
...(draft.file?.allowed_file_types ?? [])
// ❌ Problematic - loses falsy but valid values
const config = userConfig || defaultConfig
// ✅ Better - only defaults for null/undefined
const config = userConfig ?? defaultConfig
```
This pattern is especially important when working with arrays, objects, or other values where falsy states have semantic meaning that should be preserved.
---
## Concurrent resource management
<!-- source: langgenius/dify | topic: Concurrency | language: Python | updated: 2025-09-10 -->
When implementing concurrent operations, properly manage shared resources to prevent race conditions and ensure system stability. Use database-level locking mechanisms like `SELECT ... FOR UPDATE SKIP LOCKED` to prevent concurrent access to the same data. Configure thread pools with appropriate limits and timeouts to avoid resource exhaustion. Separate different types of tasks into distinct queues to prevent blocking.
Key practices:
- Use database locks for concurrent data access: `query.with_for_update(skip_locked=True)`
- Set explicit worker limits: `ThreadPoolExecutor(max_workers=1)` when sequential processing is required
- Add timeouts to concurrent operations: `concurrent.futures.wait(futures, timeout=30)`
- Implement time-windowing for high-frequency updates to reduce contention
- Separate task types into different queues: `@shared_task(queue="schedule")` vs `@shared_task(queue="execution")`
Example of proper concurrent resource management:
```python
# Use database locking to prevent race conditions
with session_factory() as session:
schedules = session.scalars(
query.with_for_update(skip_locked=True).limit(batch_size)
)
# Configure thread pools with appropriate limits and timeouts
with ThreadPoolExecutor(max_workers=3) as executor:
futures = [executor.submit(process_item, item) for item in items]
concurrent.futures.wait(futures, timeout=30)
```
Document thread safety requirements for shared components and avoid synchronous blocking patterns in async systems.
---
## CI testing tool integration
<!-- source: langgenius/dify | topic: Testing | language: Yaml | updated: 2025-09-10 -->
When integrating new testing or analysis tools into CI pipelines, use graceful failure patterns during initial adoption phases to prevent breaking existing workflows. Add tools as development dependencies and allow them to fail without blocking builds using `|| true` until they are stable and properly configured.
This approach allows teams to experiment with new testing tools without disrupting the development workflow. Once the tools are properly configured and any initial issues are resolved, the `|| true` can be removed to enforce the checks.
Example implementation:
```yaml
- name: Run new analysis tool
run: |
cd api
uv add --dev new-tool
uv run new-tool check || true
```
This pattern is particularly useful when introducing type checkers, linters, or other code analysis tools that may initially produce many findings on existing codebases.
---
## Name by semantic purpose
<!-- source: langgenius/dify | topic: Naming Conventions | language: TSX | updated: 2025-09-09 -->
Choose names that reflect the semantic purpose and meaning rather than implementation details or arbitrary values. Use enum constants instead of string literals, select precise identifiers that clearly convey intent, and name functions/variables based on their actual purpose.
Examples:
- Use `operationName === DocumentActionType.delete` instead of `operationName === 'delete'`
- Use `provider_type === 'local_file'` instead of checking multiple properties when a single semantic property exists
- Name functions like `handleUpload` instead of `handleImageCropped` when the function's purpose extends beyond just cropping
- Choose descriptive names like `'sketch'` instead of technical terms like `'handDrawn'`
- Define enums for related constants: `enum LookType { Classic = 'classic', Sketch = 'sketch' }`
This approach improves code readability, reduces magic strings, and makes the codebase more maintainable by clearly expressing the developer's intent.
---
## Validate before accessing
<!-- source: langgenius/dify | topic: Error Handling | language: TSX | updated: 2025-09-07 -->
Always validate preconditions before performing operations that could throw runtime errors. This includes checking object property existence, verifying context validity, and ensuring required data is available before proceeding.
When validation fails, handle the situation gracefully rather than allowing the operation to throw. Options include:
- Providing user feedback or prompts when context is invalid
- Using conditional checks to prevent undefined property access
- Implementing fallback behavior for missing data
Example of defensive property access:
```javascript
// Instead of direct access that could throw:
created_at__after: dayjs().subtract(TIME_PERIOD_MAPPING[period].value, 'day')
// Add validation first:
...((period !== '9' && TIME_PERIOD_MAPPING[period])
? {
created_at__after: dayjs().subtract(TIME_PERIOD_MAPPING[period].value, 'day')
}
: {})
```
This approach prevents runtime crashes and provides better user experience by handling edge cases proactively rather than reactively.
---
## Prefer exceptions over silent failures
<!-- source: langgenius/dify | topic: Error Handling | language: Python | updated: 2025-09-01 -->
Raise exceptions instead of returning None, logging warnings, or silently modifying values when encountering error conditions. This makes failures explicit and forces callers to handle error cases properly, improving code reliability and debugging experience.
**Why this matters:**
- Returning None forces every caller to check for null values, similar to error handling in Go
- Silent failures or logging-only approaches hide problems until they cause issues downstream
- Exceptions provide clear stack traces and force explicit error handling
**Apply this by:**
- Replace `return None` for missing resources with specific exceptions
- Use exceptions instead of logging warnings for validation failures
- Avoid silently modifying invalid input values in validators
**Example:**
```python
# ❌ Avoid - silent failure
def update_schedule(session: Session, schedule_id: str, updates: dict) -> Optional[WorkflowSchedulePlan]:
schedule = session.get(WorkflowSchedulePlan, schedule_id)
if not schedule:
return None # Forces caller to check for None
# ✅ Prefer - explicit exception
def update_schedule(session: Session, schedule_id: str, updates: dict) -> WorkflowSchedulePlan:
schedule = session.get(WorkflowSchedulePlan, schedule_id)
if not schedule:
raise ValueError(f"Schedule not found: {schedule_id}")
# ... rest of implementation
```
**Exception:** Use None returns only when the absence of a value is a valid business case, not an error condition.
---
## SQLAlchemy 2.0 patterns
<!-- source: langgenius/dify | topic: Database | language: Python | updated: 2025-08-31 -->
Migrate from legacy SQLAlchemy 1.x query patterns to modern 2.0 style for better performance, clarity, and future compatibility. This involves several key transformations:
**Query Construction:**
- Use `select()` instead of `session.query()`
- Use `where()` instead of `filter()`
**Result Retrieval:**
- Use `session.scalar(stmt)` instead of `session.execute(stmt).scalars().first()`
- Use `session.scalars(stmt)` instead of `session.execute(stmt).scalars()`
- Be careful with column-specific queries - they require `session.execute(stmt)` not `session.scalars(stmt)`
**Example transformations:**
```python
# Legacy 1.x style
user = db.session.query(User).filter(User.id == user_id).first()
users = db.session.query(User).filter(User.active == True).all()
# Modern 2.0 style
stmt = select(User).where(User.id == user_id)
user = db.session.scalar(stmt)
stmt = select(User).where(User.active.is_(True))
users = db.session.scalars(stmt).all()
# Column-specific queries (note: use execute, not scalars)
stmt = select(User.id, User.name).where(User.active.is_(True))
results = db.session.execute(stmt).all()
```
**Boolean Comparisons:**
Use `.is_(True)` or `.is_(False)` instead of `== True/False` for explicit boolean comparisons.
This migration improves code maintainability, leverages SQLAlchemy's latest optimizations, and ensures compatibility with future versions.
---
## Safe null handling
<!-- source: langgenius/dify | topic: Null Handling | language: Python | updated: 2025-08-31 -->
Handle null and undefined values safely by using appropriate fallback patterns, validating critical parameters early, and avoiding mutable default arguments.
**Key Practices:**
1. **Use safe fallback patterns** for optional values:
```python
# Good: Safe fallback with or operator
cpu_count = os.cpu_count() or 4
meta = meta or {}
# Good: Safe attribute access with getattr
func_name = getattr(func, "__name__", "Unknown")
# Avoid: Direct access that might fail
func_name = func.__name__ # May raise AttributeError
```
2. **Validate critical parameters early** and fail fast:
```python
# Good: Early validation with clear error
def get_features(tenant_id: str):
if not tenant_id:
raise ValueError("tenant_id is required")
# ... rest of function
# Avoid: Allowing empty/None values to propagate
def get_features(tenant_id: str):
# ... function continues without validation
```
3. **Avoid mutable default arguments**:
```python
# Good: Use None and assign in function body
def create_blob_message(blob: bytes, meta: dict = None, save_as: str = ""):
meta = meta or {}
# Good: Use default_factory for Pydantic models
conditions: list[SubCondition] = Field(default_factory=list)
# Avoid: Mutable default argument
def create_blob_message(blob: bytes, meta: dict = {}, save_as: str = ""):
```
4. **Use dict.get() with explicit defaults** for safe dictionary access:
```python
# Good: Explicit default value
for tool in agent_mode.get("tools", []):
# Less clear: Relying on or operator
for tool in agent_mode.get("tools") or []:
```
5. **Be explicit about parameter validation**:
```python
# Good: Check parameter before use
if keyword and isinstance(keyword, str):
keyword_like_val = f"%{keyword[:30]}%"
# Avoid: Using parameter without validation
keyword_like_val = f"%{keyword[:30]}%" # May fail if keyword is None
```
This approach prevents runtime errors, makes code more predictable, and clearly communicates intent about required vs optional values.
---
## Mock tests in tests/litellm
<!-- source: BerriAI/litellm | topic: Testing | language: Python | updated: 2025-08-30 -->
All new functionality must include mock tests placed in the `tests/litellm/` directory structure. Mock external API calls and dependencies rather than requiring real credentials or external services. This ensures tests can run reliably in CI/CD pipelines and are accessible to all contributors.
**Key Requirements:**
- Place tests in `tests/litellm/` following the source code structure (e.g., `tests/litellm/llms/provider_name/`)
- Mock external API calls using `unittest.mock` or `pytest-mock` instead of making real API requests
- Avoid tests that require external credentials, API keys, or network dependencies
- Use `MagicMock` to simulate responses from external services
**Example:**
```python
# Good: Mock test in tests/litellm/
from unittest.mock import patch, MagicMock
import pytest
from litellm import completion
def test_provider_completion_mock():
mock_response = MagicMock()
mock_response.json.return_value = {
"choices": [{"message": {"content": "Hello"}}]
}
with patch("requests.post", return_value=mock_response):
response = completion(
model="provider/model-name",
messages=[{"role": "user", "content": "Hi"}]
)
assert response.choices[0].message.content == "Hello"
# Bad: Integration test requiring real credentials
def test_provider_completion_real():
# Requires PROVIDER_API_KEY environment variable
response = completion(
model="provider/model-name",
messages=[{"role": "user", "content": "Hi"}]
)
```
This approach prevents test failures due to missing credentials, network issues, or rate limits, while ensuring comprehensive test coverage that contributors can run locally and in automated pipelines.
---
## Safe access patterns
<!-- source: BerriAI/litellm | topic: Null Handling | language: Python | updated: 2025-08-30 -->
Always use safe access methods when working with potentially null or undefined values. Use .get() for dictionary access, check array bounds before indexing, and validate existence before operations.
Key patterns to follow:
- Use `dict.get("key")` instead of `dict["key"]` for optional fields
- Check `len(array) > 0` before accessing `array[0]`
- Use `if value is not None:` for explicit null checks
- Avoid using `or` operator for fallback values when 0 or empty strings are valid (use explicit None checks instead)
Example of safe patterns:
```python
# Safe dictionary access
description = tool.get("description") # Instead of tool["description"]
api_key = deployment.get("litellm_params", {}).get("api_key")
# Safe array access
if len(results) > 0 and results[0].get("moderations"):
# Process moderations
# Proper null checking logic
if all([batch_cost is not None, batch_usage is not None, batch_models is not None]):
# Process batch data
# Safe fallback with explicit None check
_deployment_tpm = _deployment.get("tpm")
if _deployment_tpm is None:
_deployment_tpm = _deployment.get("litellm_params", {}).get("tpm")
if _deployment_tpm is None:
_deployment_tpm = float("inf")
```
This prevents KeyError exceptions, IndexError exceptions, and unexpected behavior from falsy values being treated as None.
---
## Use configuration helper utilities
<!-- source: BerriAI/litellm | topic: Configurations | language: Python | updated: 2025-08-29 -->
Always use centralized helper functions for configuration management instead of direct environment variable access or hardcoded values. This ensures consistency, maintainability, and proper error handling across the codebase.
Key practices:
- Use `get_secret_str()` utility for sensitive configuration values instead of `os.environ.get()`
- Use `str_to_bool()` for boolean environment variables instead of manual string comparison
- Create helper functions in common utilities (e.g., `llms/gemini/common_utils.py`) for provider-specific configuration logic
- Move hardcoded constants to `constants.py` and control them through environment variables
Example of proper configuration handling:
```python
# Bad - direct environment access and hardcoded values
api_key = os.environ.get("GEMINI_API_KEY")
MAX_STRING_LENGTH = 1000
if os.getenv("NO_REDOC", "False") == "True":
# Good - using helper utilities
api_key = get_secret_str("GEMINI_API_KEY") or get_secret_str("GOOGLE_API_KEY")
MAX_STRING_LENGTH = get_secret_str("MAX_STRING_LENGTH_PROMPT_IN_DB", "1000")
if str_to_bool(os.getenv("NO_REDOC", "False")):
```
This approach prevents configuration drift, reduces code duplication, and makes configuration behavior predictable across different parts of the system.
---
## Prefer generic API patterns
<!-- source: BerriAI/litellm | topic: API | language: Python | updated: 2025-08-28 -->
Avoid creating provider-specific API implementations when generic, reusable solutions exist or can be developed. This principle improves code maintainability, reduces duplication, and ensures consistency across the API surface.
Key practices:
- Reuse existing handlers and base classes instead of creating new ones
- Avoid hardcoded API endpoints; use configurable base URLs
- Register providers in existing systems rather than adding custom routing logic
- Create generic parameter handling that works across multiple providers
Example from the discussions:
```python
# Instead of creating a new handler:
class HostedVLLMRerank(BaseLLM):
# custom implementation...
# Reuse existing patterns:
# "please follow the cohere integration, and use the base_llm_http_handler"
# Instead of hardcoded endpoints:
if "gateway.ai.cloudflare.com" in api_base:
# custom logic...
# Use generic configuration:
api_base = api_base or get_configurable_base_url()
```
This approach reduces technical debt, makes the codebase more predictable for developers, and ensures that improvements to core functionality benefit all providers rather than being siloed in provider-specific implementations.
---
## Offload heavy operations
<!-- source: langgenius/dify | topic: Celery | language: Python | updated: 2025-08-28 -->
Move heavy operations, especially those involving large data processing or bulk deletions, from synchronous request handlers to asynchronous Celery tasks. This prevents request timeouts, improves user experience, and allows for better resource management.
Heavy operations should be identified by their potential to:
- Process large amounts of data
- Perform bulk database operations (deletions, updates)
- Take significant time to complete
- Block request threads
Example implementation:
```python
# Instead of handling deletion directly in the controller
@setup_required
@login_required
def delete(self, app_model):
# Move this to service layer with Celery
pass
# Use Celery task in service layer
@classmethod
def delete(cls, app_model: App, conversation_id: str, user: Optional[Union[Account, EndUser]]):
# Verify permissions first
conversation = cls.get_conversation_for_deletion(app_model, conversation_id, user)
# Queue the heavy deletion work as a Celery task
delete_conversation_task.delay(conversation.id)
```
This pattern ensures that web requests remain responsive while heavy operations are processed in the background by Celery workers.
---
## Improve code organization
<!-- source: sgl-project/sglang | topic: AI | language: Python | updated: 2025-08-25 -->
Prioritize clean code structure and readability in AI/ML codebases by avoiding inline conditionals, using descriptive expressions, and maintaining proper separation of concerns. This is especially important in machine learning systems where complex logic can quickly become unmaintainable.
Key practices:
1. **Separate platform-specific logic**: Instead of inline conditionals, create separate functions for different backends or platforms
2. **Use readable expressions**: Replace complex boolean logic with clearer alternatives
3. **Eliminate code duplication**: Extract common patterns into reusable components
4. **Use named constants**: Replace magic numbers with descriptive constants
Example of improved organization:
```python
# Instead of inline conditionals:
if _is_npu:
# NPU-specific logic here
else:
# Default logic here
# Use separate functions:
def apply_rotary_pos_emb():
...
def apply_rotary_pos_emb_npu():
...
if _is_npu:
apply_rotary_pos_emb = apply_rotary_pos_emb_npu
# Instead of complex conditions:
len(weight_block_size) == 2 or len(weight_block_size) == 1
# Use readable alternatives:
len(weight_block_size) in [1, 2]
# Instead of magic numbers:
TILE_SIZE = 128
BLOCK_SIZE = 4
```
This approach reduces cognitive load, improves maintainability, and makes AI/ML codebases more accessible to team members working across different components.
---
## Eliminate redundant operations
<!-- source: sgl-project/sglang | topic: Performance Optimization | language: Python | updated: 2025-08-22 -->
Avoid duplicate function calls, repeated tensor operations, and redundant computations that can significantly impact performance. Implement caching mechanisms for expensive operations and pre-allocate tensors when possible.
Key optimization strategies:
1. **Cache expensive function results**: Store results of costly operations like `get_normalized_target_modules()` or `is_npu()` to avoid repeated execution
2. **Pre-allocate tensors**: Use pre-allocated buffers instead of creating new tensors with `torch.arange()` or `torch.zeros()` in hot paths
3. **Avoid duplicate initialization**: Prevent initializing the same components multiple times (e.g., ragged wrappers) within a single forward pass
4. **Use lookup tables**: Pre-compute valid cases in lookup tables rather than dynamic computation when the search space is bounded
Example of caching expensive operations:
```python
# Before: Redundant calls
for lora_id, config in self.configs.items():
user_normalized_modules = get_normalized_target_modules(config.target_modules)
# After: Cache and reuse
normalized_cache = {}
for lora_id, config in self.configs.items():
if config.target_modules not in normalized_cache:
normalized_cache[config.target_modules] = get_normalized_target_modules(config.target_modules)
user_normalized_modules = normalized_cache[config.target_modules]
```
Example of pre-allocation:
```python
# Before: Creating tensors in hot path
q_indptr = torch.arange(0, bs + 1, dtype=torch.int32, device=device)
# After: Pre-allocate during initialization
self.q_indptr_decode = torch.arange(0, max_bs + 1, dtype=torch.int32, device=device)
# Use pre-allocated buffer: q_indptr = self.q_indptr_decode[:bs + 1]
```
Performance impact can be substantial - caching reduced execution time from 133μs to 5μs in one measured case. Always profile critical paths and eliminate redundant work through strategic caching and pre-allocation.
---
## eliminate code duplication
<!-- source: sgl-project/sglang | topic: Code Style | language: Python | updated: 2025-08-21 -->
Actively identify and eliminate code duplication by extracting common patterns into reusable components. When you notice repeated code blocks, similar conditional logic, or duplicate class definitions, consolidate them through extraction techniques.
Key strategies:
- **Extract repeated conditional patterns**: When you see the same condition checked multiple times, create data structures or helper functions to eliminate repetition
- **Consolidate duplicate classes/functions**: Instead of creating separate classes with similar functionality, extend existing ones or create shared base implementations
- **Move utility functions to appropriate modules**: Place domain-specific utilities in their respective modules (e.g., vision utilities in `vision_utils.py`, general utilities in `utils.py`)
- **Break down complex functions**: Extract logical chunks from long functions into separate, focused functions for better modularity
Example of pattern extraction:
```python
# Before: Repeated conditional logic
global_num_tokens_gpu=(
self.prefill_global_num_tokens_gpu
if hasattr(self, "capture_forward_mode")
and self.capture_forward_mode == ForwardMode.EXTEND
else self.global_num_tokens_gpu
),
global_num_tokens_for_logprob_gpu=(
self.prefill_global_num_tokens_for_logprob_gpu
if hasattr(self, "capture_forward_mode")
and self.capture_forward_mode == ForwardMode.EXTEND
else self.global_num_tokens_for_logprob_gpu
),
# After: Extract pattern into data structure
@dataclass
class TokenConfig:
global_num_tokens_gpu: Tensor
global_num_tokens_for_logprob_gpu: Tensor
config = prefill_config if self.capture_forward_mode == ForwardMode.EXTEND else default_config
```
This approach improves maintainability, reduces the chance of inconsistencies, and makes the codebase more readable and easier to modify.
---
## Verify algorithmic correctness
<!-- source: sgl-project/sglang | topic: Algorithms | language: Python | updated: 2025-08-21 -->
Ensure algorithmic implementations are correct before applying optimizations or complex conditional logic. This is especially critical for performance optimizations, computational complexity considerations, and batch processing logic where errors can have cascading effects.
Key practices:
1. **Validate optimization conditions**: Before applying algorithmic optimizations, ensure all prerequisite conditions are met
2. **Analyze computational complexity**: Consider the algorithmic complexity (e.g., O(bs·seq_len^2)) and precision requirements when implementing performance-critical code
3. **Verify batch processing logic**: Ensure state management and control flow are correct in algorithmic processing pipelines
4. **Use precise arithmetic**: Choose appropriate operators (/ vs //) based on the mathematical requirements of the algorithm
Example from the discussions:
```python
# Incorrect: Optimization applied without proper condition checking
if (_is_flashinfer_available and global_server_args_dict["enable_flashinfer_allreduce"]):
final_hidden_states = flashinfer_allreduce(final_hidden_states)
# Correct: Verify all conditions before applying optimization
if (self.reduce_results and (self.tp_size > 1 or self.ep_size > 1) and
_is_flashinfer_available and global_server_args_dict["enable_flashinfer_allreduce"]):
final_hidden_states = flashinfer_allreduce(final_hidden_states)
# Computational complexity consideration with precise arithmetic
current_workload = sum_seq_lens / forward_batch.batch_size * sum_seq_lens # Use / for mean
```
This approach prevents algorithmic bugs that can lead to incorrect results, performance degradation, or system instability.
---
## Use Optional types safely
<!-- source: sgl-project/sglang | topic: Null Handling | language: Python | updated: 2025-08-20 -->
Always use proper nullable type annotations and safe access patterns to prevent runtime errors from null/undefined values. Use `Optional[Type]` or `Union[None, Type]` for nullable parameters, employ safe dictionary access with `.get()`, and handle None values explicitly in operations.
Examples of good practices:
```python
# Use Optional type annotations
def process_data(encoder_lens: Optional[torch.Tensor] = None):
pass
# Safe dictionary access
draft_url = params.get("draft_url", None) # Instead of params["draft_url"]
# Null-safe assertions
assert (topk_weights is None) or (topk_weights.shape == topk_ids_.shape)
```
This prevents KeyError exceptions, improves type safety, and makes null handling explicit rather than implicit. Always consider whether a parameter or variable can be None and handle it appropriately in both type annotations and runtime logic.
---
## Avoid code duplication
<!-- source: BerriAI/litellm | topic: Code Style | language: Python | updated: 2025-08-19 -->
Eliminate redundant code by leveraging existing utilities, extracting reusable helper functions, and organizing logic in appropriate modules. Before implementing new functionality, check if similar logic already exists in the codebase that can be reused or extended.
Key practices:
- Use existing utilities instead of reimplementing logic (e.g., use `get_deployment` in router.py rather than duplicating deployment lookup logic)
- Extract common logic into helper functions or static methods to avoid repetition across multiple locations
- Organize provider-specific logic in dedicated transformation files rather than mixing it in generic handlers
- Move utility functions to appropriate modules (e.g., proxy/utils.py) rather than defining them inline
- Create reusable helper utilities for common patterns that can be shared across endpoints
Example of refactoring duplicate logic:
```python
# Before: Duplicate SSL verification logic
ssl_verify = (os.getenv("SSL_VERIFY") != "False") if os.getenv("SSL_VERIFY") is not None else litellm.ssl_verify
# After: Extract to helper method
def get_ssl_verify_setting():
return (os.getenv("SSL_VERIFY") != "False") if os.getenv("SSL_VERIFY") is not None else litellm.ssl_verify
ssl_verify = get_ssl_verify_setting()
```
This approach reduces maintenance burden, improves consistency, and makes the codebase more maintainable by centralizing common logic.
---
## Optimize cache performance
<!-- source: sgl-project/sglang | topic: Caching | language: Python | updated: 2025-08-19 -->
Prioritize cache performance by leveraging proven optimization techniques and avoiding performance anti-patterns. Use standard caching decorators like @lru_cache for functions that benefit from memoization, implement zero-copy operations when moving data between cache layers, and avoid unnecessary operations that degrade performance.
Key practices:
- Apply @lru_cache decorator to functions with reusable results
- Implement zero-copy techniques for cache data transfers
- Avoid additional slice operations or data transformations in cache paths
- Choose cache architectures that minimize operational overhead
Example:
```python
from functools import lru_cache
@lru_cache(maxsize=128)
def should_use_flashinfer_cutlass_moe_fp4_allgather():
# Expensive computation cached automatically
return compute_moe_config()
# Zero-copy backup for Mooncake backend
if self.is_mooncake_backend():
# Use zero-copy operations instead of generic page backup
self.zerocopy_page_backup(operation)
```
This approach ensures cache operations remain performant under load while leveraging battle-tested optimization patterns that reduce computational overhead and memory copying.
---
## Configuration defaults appropriateness
<!-- source: langgenius/dify | topic: Configurations | language: Yaml | updated: 2025-08-19 -->
Ensure configuration parameters have appropriate default values and avoid hardcoded or overly specific settings. Default values should be conservative and safe for production environments, while avoiding personal assignments or environment-specific assumptions.
Examples of good practices:
- Use conservative defaults: `WORKFLOW_LOG_CLEANUP_ENABLED: ${WORKFLOW_LOG_CLEANUP_ENABLED:-false}` instead of `true`
- Provide sensible defaults for optional parameters like expiration times
- Remove inappropriate hardcoded assignments like single-user `assignees` in configuration files
- Avoid mixing deployment concerns by keeping third-party service configurations separate from main application config
This ensures configurations are maintainable, secure by default, and appropriate for team environments rather than individual setups.
---
## Environment variable best practices
<!-- source: sgl-project/sglang | topic: Configurations | language: Python | updated: 2025-08-19 -->
Avoid reading environment variables directly in performance-critical code paths, especially during model forward passes or hot loops. Instead, read and cache environment variables during initialization or use dedicated helper functions for consistent handling.
Use standardized helper functions like `get_bool_env_var()` instead of direct `os.getenv()` calls for better type safety and consistency. Follow clear naming conventions for environment variables, using descriptive prefixes like `SGLANG_` for project-specific settings.
Example of what to avoid:
```python
def forward_decode(self):
# BAD: Reading env var in forward pass
if os.getenv("USE_W4A8") != "1" and deep_gemm_wrapper.ENABLE_JIT_DEEPGEMM:
# performance-critical code
```
Better approach:
```python
def __init__(self):
# GOOD: Cache env var during initialization
self.use_w4a8 = get_bool_env_var("SGLANG_USE_W4A8")
def forward_decode(self):
# GOOD: Use cached value
if not self.use_w4a8 and deep_gemm_wrapper.ENABLE_JIT_DEEPGEMM:
# performance-critical code
```
This practice improves performance by avoiding repeated system calls and ensures consistent configuration handling across the codebase.
---
## API payload structure
<!-- source: langgenius/dify | topic: API | language: TypeScript | updated: 2025-08-19 -->
Design API request and response payloads to match their specific purpose rather than forcing them to conform to existing application types. When API payloads require different structures or optional fields compared to internal data models, create dedicated API-specific types instead of adding placeholder values or compromising type safety.
This prevents situations where developers must add dummy values like empty strings to satisfy type requirements that don't align with the API's actual needs. It also enables cleaner API interfaces with appropriate optional parameters and nested data structures.
Example of the problem:
```typescript
// Problematic: Forcing API payload to match internal type
environment_variables: environmentVariables.map((item: EnvironmentVariable) => {
return {
id: item.id,
name: item.name,
value_type: item.value_type,
from_version: restoredSecretsInfo[item.id].from_version,
// Adding dummy values to satisfy EnvironmentVariable type
value: '',
description: '',
}
})
```
Better approach: Create API-specific types that reflect the actual payload structure and requirements, allowing for cleaner, more maintainable interfaces that don't require workarounds or placeholder data.
---
## Avoid broad exception catching
<!-- source: sgl-project/sglang | topic: Error Handling | language: Python | updated: 2025-08-19 -->
Avoid using overly broad exception handling like `except Exception` as it can mask specific errors and make debugging difficult. Instead, catch specific exception types when possible, or convert generic exceptions to more informative, specific error types with meaningful messages.
When you must catch broad exceptions, immediately convert them to specific error types that provide context about what went wrong and why. This helps with debugging and prevents silent failures.
Example of problematic broad catching:
```python
try:
speculative_algo = global_server_args_dict.get("speculative_algorithm", None)
if speculative_algo is not None and hasattr(speculative_algo, "is_eagle"):
return speculative_algo.is_eagle()
except Exception: # Too broad - what could fail here?
return False
```
Better approach with specific error handling:
```python
try:
ip_str, port_str = distributed_init_method.split(":")[-2:]
ip = ip_str.split("/")[-1]
port = int(port_str)
except (ValueError, IndexError) as e: # Specific exceptions
raise RuntimeError(
f"Cannot parse host and port from {distributed_init_method}: {str(e)}"
)
```
This approach makes errors more actionable for developers and prevents masking of unexpected issues that should be addressed.
---
## Configuration value types
<!-- source: langgenius/dify | topic: Configurations | language: Python | updated: 2025-08-18 -->
Ensure configuration values use appropriate types, avoid unnecessary Optional annotations, and maintain consistency in data formats and units.
Key principles:
1. **Use explicit boolean types** instead of string representations when the configuration is consumed as a boolean
2. **Avoid Optional for fields with defaults** - if a default value is provided, Optional is redundant
3. **Choose consistent time units** across related configurations (prefer seconds for precision, minutes for readability)
4. **Use appropriate data formats** - prefer unix timestamps over ISO strings for simplicity and timezone safety
Example of proper configuration typing:
```python
class MyConfig(BaseSettings):
# Good: explicit boolean with default
SSL_VERIFY: bool = Field(default=True)
# Bad: unnecessary Optional with default
# SSL_VERIFY: Optional[bool] = Field(default=True)
# Good: consistent time unit choice
ACCESS_TOKEN_EXPIRE_MINUTES: int = Field(default=30)
# Good: unix timestamp for cache keys
def get_cache_timestamp(self) -> int:
return int(datetime.now().timestamp())
```
This approach reduces configuration complexity, improves type safety, and ensures consistent behavior across different deployment environments.
---
## Use appropriate log levels
<!-- source: langgenius/dify | topic: Logging | language: Python | updated: 2025-08-14 -->
Choose the correct logging level based on the importance and purpose of the message. Use DEBUG for debugging information that's only useful during development, INFO for general operational information, WARNING for concerning conditions that don't prevent operation, and ERROR for actual errors. Remove meaningless debug prints and consider using conditional logging for expensive operations.
Examples:
- Use DEBUG for detailed debugging: `if logger.isEnabledFor(logging.DEBUG): logger.debug("Detailed debug info")`
- Use WARNING for concerning conditions: `logger.warning("Document not found, skipping processing")`
- Remove debug prints: `logging.error(f"{dataset_id_str}")` should be removed or changed to appropriate level
- Use INFO for operational messages: `logger.info("Created workflow alias: %s for workflow %s", alias_name, workflow_id)`
Avoid leaving temporary debug logs in production code and ensure log levels match the actual significance of the information being logged.
---
## Safe operations with fallbacks
<!-- source: BerriAI/litellm | topic: Error Handling | language: Python | updated: 2025-08-14 -->
Implement defensive programming by wrapping potentially failing operations in safe methods that provide fallback behavior. Critical operations should never cause the main functionality to fail due to secondary concerns like formatting, logging, or auxiliary processing.
Key principles:
- Use try-catch blocks around operations that could fail but shouldn't break core functionality
- Provide sensible fallback values when operations fail
- Create helper functions for common failure-prone operations like JSON parsing or data formatting
- Ensure that auxiliary operations (like cost calculation, logging, or formatting) don't prevent the main response from being returned
Example implementation:
```python
# Instead of direct operation that could fail
"x-litellm-response-cost": str(round(response_cost, 6))
# Use safe helper method with fallback
def safe_round_cost(cost: float) -> str:
try:
return str(round(cost, 6))
except Exception:
return str(cost) # fallback to original value
"x-litellm-response-cost": safe_round_cost(response_cost)
```
This approach prevents secondary failures from cascading and breaking primary functionality, while still attempting the preferred operation when possible.
---
## Choose descriptive, unambiguous names
<!-- source: langgenius/dify | topic: Naming Conventions | language: TypeScript | updated: 2025-08-14 -->
Select variable, method, and property names that clearly communicate their purpose and avoid ambiguity. Prioritize semantic accuracy over brevity, and consider the context in which names will be used to prevent confusion.
Key principles:
- Use semantically accurate names that reflect the actual purpose (e.g., `affectedNodes` instead of `effectNodes` when referring to nodes that are affected by an operation)
- Avoid naming conflicts by choosing distinct terms when similar concepts coexist (e.g., use `inputs` instead of `variables` when a `variable` property already exists)
- Replace magic strings with enums or constants for better type safety and self-documentation
Example:
```typescript
// Poor: ambiguous and uses magic strings
const effectNodes = findNodes(selector, nodes)
if (permission === 'only_me') { ... }
// Better: clear semantic meaning and structured types
const affectedNodes = findNodes(selector, nodes)
if (permission === DatasetPermission.ONLY_ME) { ... }
```
This approach reduces cognitive load, prevents misunderstandings, and makes code more maintainable by ensuring names accurately represent their underlying concepts.
---
## Use descriptive names
<!-- source: sgl-project/sglang | topic: Naming Conventions | language: Python | updated: 2025-08-13 -->
Choose variable, function, and parameter names that accurately reflect their content, type, and purpose. Avoid misleading names that don't match the actual data or functionality they represent.
Key principles:
- Names should clearly indicate what the variable contains or what the function does
- Avoid names that suggest a different type than what's actually stored (e.g., don't use `_list` suffix for non-list variables)
- Use standard, well-understood terminology rather than abbreviations or informal terms
- Balance descriptiveness with conciseness - avoid excessively long names while maintaining clarity
Examples of improvements:
```python
# Misleading - suggests it's a list but contains length
holding_tokens_list = len(tokens) # Bad
# Clear and accurate
holding_tokens_count = len(tokens) # Good
# Informal terminology
onfly_info = get_dispatch_info() # Bad
# Standard terminology
in_flight_info = get_dispatch_info() # Good
# Overly verbose
should_fuse_allreduce_residual_rmsnorm = True # Bad
# Concise but clear
should_allreduce_fusion = True # Good
```
This practice prevents confusion, reduces debugging time, and makes code more maintainable by ensuring names serve as accurate documentation of the code's intent.
---
## Conservative configuration management
<!-- source: langgenius/dify | topic: Configurations | language: Json | updated: 2025-08-13 -->
Be selective and intentional with configuration choices to avoid bloat and unnecessary complexity. This applies to build configurations, development environment settings, and dependency management.
Key practices:
- Exclude unnecessary directories and files from build processes to improve performance
- Make optional development features configurable rather than enabled by default
- Avoid introducing new dependencies unless existing solutions cannot fulfill requirements
- Document optional configurations clearly so developers can enable them when needed
Example from TypeScript configuration:
```json
"exclude": [
"**/node_modules/**",
"**/.next/**"
]
```
This approach reduces build times, keeps development environments lean, and minimizes maintenance overhead while ensuring configurations remain focused on actual project needs.
---
## Explicit API parameters
<!-- source: sgl-project/sglang | topic: API | language: Python | updated: 2025-08-13 -->
API functions should use explicit parameters rather than passing large opaque objects. When a function needs specific data from a complex object, extract and pass only the required fields as individual parameters. This makes the function's dependencies clear, improves testability, and reduces coupling.
Avoid passing entire objects when only specific fields are needed:
```python
# Bad: Opaque object passing
def forward(
self,
hidden_states: torch.Tensor,
topk_output: StandardTopKOutput,
forward_batch: Optional[ForwardBatch] = None, # Unclear what fields are used
):
# Good: Explicit parameters
def forward(
self,
hidden_states: torch.Tensor,
topk_output: StandardTopKOutput,
dp_padding_mode: bool,
batch_size: int,
gathered_buffer: torch.Tensor,
global_num_tokens: int,
):
```
This approach makes function interfaces self-documenting and prevents unnecessary dependencies on large data structures. If many parameters are needed, consider if the function has too many responsibilities and should be refactored.
---
## optimize tensor operations
<!-- source: sgl-project/sglang | topic: Pytorch | language: Python | updated: 2025-08-13 -->
When working with PyTorch tensors, prioritize operations that avoid unnecessary memory allocations and copies to improve performance. Choose tensor operations carefully based on whether data will be immediately overwritten or needs preservation.
Key guidelines:
- Use `tensor.view(dtype)` instead of `tensor.to(dtype)` when possible to avoid copies
- Use `torch.empty()` instead of `torch.zeros()` when the tensor will be fully populated immediately after creation
- Understand memory allocation contexts (like symmetric memory pools) and their implications for tensor operations
Example:
```python
# Avoid unnecessary copy
if tensor.dtype != target_dtype:
tensor = tensor.view(target_dtype) # No copy if compatible
# instead of: tensor = tensor.to(target_dtype) # May cause copy
# Avoid unnecessary initialization
kv_indices = torch.empty(size, dtype=torch.int32) # Will be filled next
# instead of: kv_indices = torch.zeros(size, dtype=torch.int32) # Extra kernel launch
```
This approach reduces memory overhead and kernel launches, leading to better performance in tensor-heavy operations.
---
## AI provider documentation completeness
<!-- source: BerriAI/litellm | topic: AI | language: Markdown | updated: 2025-08-12 -->
Ensure comprehensive documentation for AI/ML provider integrations that covers all usage patterns, uses precise terminology, and provides helpful context for developers.
AI provider documentation should include:
1. **Complete usage examples**: Cover SDK usage, proxy configuration, streaming, and advanced features like function calling or image input
2. **Precise terminology**: Use accurate provider-specific terms (e.g., `<hf_org_or_user>/<hf_model>` instead of generic `<model_id>`)
3. **Configuration consistency**: Show both direct API usage and proxy configuration patterns
4. **Provider-specific context**: Explain unique features, billing models, and authentication methods
5. **Missing examples**: Add proxy usage examples when only SDK examples exist
Example of comprehensive documentation structure:
```python
# Basic SDK usage
response = completion(
model="huggingface/together/deepseek-ai/DeepSeek-R1",
messages=[{"content": "Hello", "role": "user"}]
)
# Proxy configuration
# config.yaml
model_list:
- model_name: my-model
litellm_params:
model: huggingface/together/deepseek-ai/DeepSeek-R1
api_key: os.environ/HF_TOKEN
web_search_options: {} # Provider-specific options
# Advanced features (when supported)
response = completion(
model="huggingface/sambanova/Qwen/Qwen2.5-72B-Instruct",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "..."}}
]
}]
)
```
This ensures developers have complete guidance for integrating AI providers across different usage patterns and deployment scenarios.
---
## API parameter consistency
<!-- source: BerriAI/litellm | topic: API | language: Markdown | updated: 2025-08-11 -->
Ensure consistent parameter naming, format, and usage across all API interfaces, documentation, and code examples. This includes maintaining uniform parameter names between SDK examples and proxy configurations, using correct environment variable names as specified in official documentation, and applying proper naming conventions for model identifiers and API routes.
Key areas to verify:
- Model names should include provider prefixes (e.g., `heroku/claude-3-5-haiku`) to enable proper provider identification
- Parameter names must match between SDK examples and proxy configurations (e.g., if SDK uses `api_base`, proxy config should use the same)
- Environment variables should follow official documentation (e.g., `ANTHROPIC_AUTH_TOKEN` and `ANTHROPIC_BASE_URL` for Anthropic)
- API routes should clearly indicate the underlying service being called (e.g., `openai/responses/<model>` instead of `chat/<model>` for responses API)
Example of consistent parameter usage:
```python
# SDK example
response = completion(
model="heroku/claude-3-5-haiku",
api_base="https://us.inference.heroku.com",
api_key="fake-heroku-key"
)
# Corresponding proxy config should use same parameter names
model_list:
- model_name: claude-haiku
litellm_params:
model: heroku/claude-3-5-haiku
api_base: https://us.inference.heroku.com
api_key: fake-heroku-key
```
This consistency prevents confusion, reduces integration errors, and ensures predictable behavior across different usage contexts.
---
## Prevent race conditions
<!-- source: sgl-project/sglang | topic: Concurrency | language: Python | updated: 2025-08-11 -->
Ensure atomic operations and proper synchronization when multiple threads or processes access shared state. Race conditions occur when the timing of operations affects correctness, leading to data corruption, memory leaks, or inconsistent behavior.
Common patterns to watch for:
- **Check-then-act sequences**: When checking a condition and acting on it are separate operations, the state can change between them
- **Shared counter updates**: Multiple threads modifying counters without proper locking
- **Cross-process state management**: Replacing managed objects instead of updating them in-place
Example of a race condition:
```python
# Thread A checks condition
if operation.is_done(): # Returns False
# Thread B calls operation.mark_done() here
# Thread A continues with stale information
operation.completed_tokens += self.page_size
self.mem_pool_host.free(operation.host_indices[operation.completed_tokens:])
```
Solutions:
- Use locks or atomic operations for shared state modifications
- Combine check-and-update operations into single atomic transactions
- For multiprocessing, update managed objects in-place rather than replacing them
- Consider if synchronization is necessary on critical paths and document the reasoning
When implementing distributed operations, ensure all ranks maintain consistent state to prevent divergent behavior across the system.
---
## Document configuration purpose clearly
<!-- source: BerriAI/litellm | topic: Configurations | language: Markdown | updated: 2025-08-11 -->
When documenting configuration features, always provide clear explanations of their purpose, use cases, and how they map to underlying parameters. Users should understand not just how to use a configuration option, but why they would want to use it and what problem it solves.
For environment variables, explicitly document the mapping between environment variables and configuration parameters:
```markdown
## Environment Variables
The Heroku provider is aware of the following config variables:
- `HEROKU_API_KEY`: This value corresponds to the [`api_key` param](https://docs.litellm.ai/docs/set_keys#litellmapi_key). Set this to the value of Heroku's `INFERENCE_KEY` config variable.
- `HEROKU_API_BASE`: This value corresponds to the [`api_base` param](https://docs.litellm.ai/docs/set_keys#litellmapi_base). Set this to the value of Heroku's `INFERENCE_URL` config variable.
```
For configuration features that may have non-obvious use cases, include practical scenarios:
```markdown
#### Import Models
Import models from a YAML file. This is useful when migrating models from a self-hosted instance to a cloud instance, allowing you to gradually port your existing configuration.
```
This prevents confusion and reduces the need for clarifying questions about the purpose and intended workflow of configuration features.
---
## AI model data validation
<!-- source: BerriAI/litellm | topic: AI | language: Json | updated: 2025-08-10 -->
All AI model configuration parameters must be verified against official provider documentation before merging. This includes token limits, pricing values, provider assignments, model names, and supported features. Incorrect configuration data leads to runtime errors, cost miscalculations, and integration failures.
Key validation requirements:
- Verify max_input_tokens, max_output_tokens against official docs (e.g., "Should this be 65535 per Google's documentation?")
- Validate pricing using scientific notation and correct per-token costs (e.g., "The pricing `0.00021` is incorrect. It's 0.21/1m tokens")
- Ensure provider field matches actual API provider (e.g., "provider should be `bedrock_converse`" not "bedrock")
- Confirm model names match those returned by provider APIs for accurate cost tracking
- Use proper JSON formatting (true/false, not True/False)
Example of proper validation:
```json
"gpt-5-2025-08-07": {
"max_input_tokens": 272000, // Verified against OpenAI docs
"input_cost_per_token": 1.25e-06, // Scientific notation, verified pricing
"litellm_provider": "openai", // Matches actual API provider
"supports_function_calling": true // JSON boolean, not Python
}
```
Always include source citations when making configuration changes to enable future verification and maintain data integrity.
---
## Document AI limitations
<!-- source: sgl-project/sglang | topic: AI | language: Markdown | updated: 2025-08-10 -->
When implementing AI model support or hardware backends, clearly document known limitations, compatibility constraints, and version requirements to prevent integration issues and set proper expectations.
AI systems often have specific constraints that can cause runtime failures or unexpected behavior if not properly communicated. This includes quantization limitations, hardware-specific requirements, version dependencies, and feature compatibility matrices.
Key areas to document:
- Model quantization constraints (e.g., "Mixed-bit quantization is not fully supported due to vLLM's layer fusion")
- Hardware backend limitations (e.g., "Wave path only works with page size = 1")
- Version-specific requirements (e.g., "Only python==3.11 is supported currently")
- Performance optimization settings and their rationale
Example documentation pattern:
```markdown
## Known Issues
Several limitations currently affect offline quantized model loading:
1. Mixed-bit Quantization Limitations
Mixed-bit quantization is not fully supported. Due to vLLM's layer fusion
(e.g., QKV fusion), applying different bit-widths to components within the
same fused layer can lead to compatibility issues.
2. Limited Support for Quantized MoE Models
Most quantized MoE models may encounter inference issues due to
kernel-related limitations.
```
This prevents users from encountering undocumented failures and helps them choose appropriate alternatives or workarounds.
---
## consolidate duplicated logic
<!-- source: sgl-project/sglang | topic: Code Style | language: CUDA | updated: 2025-08-10 -->
When you identify duplicated code patterns across different branches or functions, consolidate the shared logic into a unified implementation to improve maintainability and reduce code bloat. This applies whether the duplication occurs within conditional branches, across similar functions, or in separate files.
For substantial duplication, extract the common logic into a shared function or kernel. For minimal implementations (a dozen lines or less), consider inlining the logic using C++ templates and constexpr to avoid overhead while maintaining clarity.
Example of consolidation:
```cpp
// Before: Duplicated logic in branches
if (params.VPT <= 32) {
// Complex processing logic A
// ... 50 lines of similar code
} else {
// Complex processing logic B
// ... 50 lines of nearly identical code
}
// After: Consolidated into unified kernel
template<bool small_vpt>
void unified_processing_kernel(...) {
// Shared logic with template specialization
if constexpr (small_vpt) {
// Small VPT specific optimizations
} else {
// Large VPT specific optimizations
}
}
```
This approach reduces maintenance burden, minimizes the risk of inconsistent updates, and keeps the codebase clean and organized.
---
## AI accuracy documentation
<!-- source: menloresearch/jan | topic: AI | language: Other | updated: 2025-08-08 -->
Ensure accurate representation of AI model capabilities, proper attribution, and honest claims about functionality. When documenting AI systems, focus on the underlying model capabilities rather than just pipeline descriptions, provide clear explanations of model variants (distilled vs full models), attribute models to their actual creators, and avoid overstating what has been achieved.
Key practices:
- Explain that AI functionality depends on "the model and its capabilities" to use tools effectively, not just the pipeline architecture
- Clarify model relationships (e.g., "why these say qwen vs llama" when discussing model variants)
- Set proper attribution: "Meta for Llama2 and Mistral for mixtral-8x7b-32768" rather than generic authors
- Use honest language: "created our own flavor of it" rather than claiming to have "replicated" proprietary systems
- Explain technical distinctions like "distilled model versus the full one" so users understand what they're actually downloading
Example model.json with proper attribution:
```json
{
"metadata": {
"author": "Meta", // Actual model creator, not generic name
"description": "Llama 2 integration with Jan - our implementation of chat functionality"
}
}
```
This prevents misleading users about AI capabilities while ensuring proper credit and technical accuracy.
---
## Eliminate filler language
<!-- source: menloresearch/jan | topic: Documentation | language: Other | updated: 2025-08-08 -->
Remove unnecessary filler words and repetitive phrases that dilute clarity and assume reader incompetence. Avoid words like "essentially", "basically", "just", "simple", "easy", and repetitive phrases like "at a high level" appearing multiple times in close proximity.
Write directly and respect your reader's intelligence. Instead of hedging with filler words, state facts clearly and concisely.
**Bad example:**
```markdown
Deep Research, at a high level, is essentially a pipeline / tool chain through which a user would go through. The inner workings of this pipeline might vary from provider to provider, but we've done up a table for you to compare the results below.
At a high level, the pipeline would look something like this:
```
**Good example:**
```markdown
Deep Research is a pipeline or tool chain through which a user would go through. The inner workings of this pipeline might vary from provider to provider, but we've done up a table for you to compare the results below.
For example, a straightforward pipeline might look as follows:
```
This approach makes documentation more professional, easier to read, and shows respect for the reader's technical competence. Focus on delivering information efficiently rather than cushioning it with unnecessary qualifiers.
---
## avoid expensive operations
<!-- source: BerriAI/litellm | topic: Performance Optimization | language: Python | updated: 2025-08-07 -->
Identify and optimize resource-intensive operations in frequently executed code paths. This includes avoiding memory-doubling operations like `copy.deepcopy()` on large objects, preventing blocking I/O in async contexts, and eliminating expensive debug logging operations.
Key areas to review:
- **Memory operations**: Avoid deep copying large objects without considering alternatives like in-place modification or streaming
- **Async contexts**: Move blocking operations (file I/O, environment variable reads) to startup initialization rather than request-time execution
- **Debug logging**: Avoid expensive serialization operations, especially `json.dumps()` on large objects in hot paths
- **Unbounded growth**: Implement limits on data structures to prevent memory leaks
Example from the discussions:
```python
# Problematic - doubles memory usage
redacted_response = copy.deepcopy(response_json)
# Better - consider in-place modification or streaming for large objects
# Or move expensive operations out of request path
# Problematic - blocking I/O in async context
session = ClientSession(trust_env=True) # Reads files synchronously
# Better - read environment at startup
proxy_settings = get_environment_proxies() # At startup
session = ClientSession(proxies=proxy_settings) # At request time
```
Always profile and measure the impact of operations in performance-critical paths, especially when dealing with large data or high-frequency execution.
---
## avoid hardcoded credentials
<!-- source: BerriAI/litellm | topic: Security | language: Yaml | updated: 2025-08-07 -->
Never embed sensitive credentials, passwords, API keys, or other secrets directly in source code. Hardcoded credentials create security vulnerabilities by exposing sensitive data in version control systems and making it accessible to anyone with code access.
Instead, use secure alternatives:
- Environment variables for local development
- Secret management systems (GitHub Secrets, AWS Secrets Manager, etc.)
- Configuration files that are excluded from version control
Example of what to avoid:
```yaml
cache_params:
type: "redis"
host: "redis-18438.c277.us-east-1-3.ec2.redns.redis-cloud.com"
port: 18438
password: "hB44ThYlB4W4m7wpCUwrSzteHqvDKnDV" # ❌ Hardcoded password
```
Better approach:
```yaml
cache_params:
type: "redis"
host: "redis-18438.c277.us-east-1-3.ec2.redns.redis-cloud.com"
port: 18438
password: ${REDIS_PASSWORD} # ✅ Environment variable
```
This practice protects against credential leaks and ensures sensitive data remains secure across different deployment environments.
---
## Implement graceful error fallbacks
<!-- source: menloresearch/jan | topic: Error Handling | language: TypeScript | updated: 2025-08-05 -->
Always implement fallback mechanisms when primary operations can fail, especially for critical operations like resource cleanup, async operations, and user-facing functionality. When the primary approach fails, provide a secondary method to handle the situation gracefully rather than letting errors propagate unchecked.
For resource cleanup, attempt graceful shutdown first, then force termination:
```typescript
function killSubprocess(): Promise<void> {
return fetch(NITRO_HTTP_KILL_URL, {
method: "DELETE",
}).catch((err) => {
console.error(err);
subprocess.kill();
subprocess = null;
return kill(PORT, "tcp").then(console.log).catch(console.log);
});
}
```
For user-facing operations, validate inputs and provide meaningful error handling:
```typescript
const convoId = currentConvo?.id
if (!convoId) {
console.error('No conversation id')
// TODO: Display toast notification to user
return
}
```
This approach prevents silent failures, ensures resources are properly cleaned up even when primary methods fail, and maintains application stability by providing alternative paths when operations don't succeed as expected.
---
## HTTP standards compliance
<!-- source: langgenius/dify | topic: API | language: Python | updated: 2025-08-04 -->
Ensure API endpoints follow HTTP standards for status codes and method selection. Delete operations should return 204 (No Content) without a response body, while update operations should use PUT method instead of GET. This improves API consistency and follows REST conventions.
For delete operations, use:
```python
def delete(self, member_id):
# Delete logic here
return "", 204 # No content, no body
```
For update operations that retrieve and update data:
```python
class ToolMCPUpdateApi(Resource):
def put(self, provider_id): # Use PUT instead of GET
# Update logic here
return {"result": "success"}, 200
```
Avoid returning response bodies with DELETE methods as per HTTP specifications. Use appropriate status codes: 200 for successful operations with content, 204 for successful operations without content, and proper HTTP methods (GET for retrieval, POST for creation, PUT for updates, DELETE for deletion).
---
## simplify error handling
<!-- source: menloresearch/jan | topic: Error Handling | language: TSX | updated: 2025-08-04 -->
Write concise and appropriate error handling code by avoiding unnecessary complexity and verbose constructions. Remove try-catch blocks when they serve no purpose, and use streamlined approaches for error message construction.
For example, instead of verbose error message handling:
```javascript
let errMessage = 'Unexpected Error'
if (err instanceof Error) {
errMessage = err.message
}
toaster({
title: 'Failed to get Hugging Face models',
description: errMessage,
type: 'error',
})
```
Use a more concise approach:
```javascript
toaster({
title: 'Failed to get Hugging Face models',
description: err instanceof Error ? err.message : 'Unexpected Error',
type: 'error',
})
```
Similarly, avoid wrapping function calls in try-catch blocks when error handling isn't actually needed or when the function doesn't throw exceptions that require handling at that level.
---
## Simplify for readability
<!-- source: LMCache/LMCache | topic: Code Style | language: Python | updated: 2025-08-01 -->
Improve code readability by simplifying complex logic, using Pythonic idioms, and removing unnecessary code. Extract helper functions for complex conditions, avoid code duplication, and leverage Python's built-in features for cleaner code.
Key practices:
- Extract helper functions for complex boolean conditions instead of inline logic
- Use Pythonic idioms like `if not items:` instead of `items == []` (PEP 8)
- Leverage walrus operator `:=` to reduce redundant dictionary lookups
- Avoid code duplication by refactoring common patterns
- Remove commented-out code and unused imports - git tracks history
- Simplify repeated conditional checks with single variables
Example:
```python
# Instead of:
if self.save_only_first_rank and not self.metadata.is_first_rank():
return
# Use a helper function:
def is_passive(self):
return self.save_only_first_rank and not self.metadata.is_first_rank()
if self.is_passive():
return
# Instead of:
if memory_objs == []:
return
# Use Pythonic style:
if not memory_objs:
return
```
---
## minimize core dependencies
<!-- source: BerriAI/litellm | topic: Configurations | language: Toml | updated: 2025-07-31 -->
Keep core dependencies minimal in configuration files like pyproject.toml. Non-essential packages should be added as optional dependencies or placed in extras sections rather than as required core dependencies. This prevents package bloat and allows users to install only what they need.
When adding new dependencies, first determine if they are truly required for core functionality. If the dependency is only needed for specific features, testing, or optional functionality, use the appropriate configuration section:
```toml
# Core dependencies - only essential packages
[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.28.0"
# Optional dependencies for specific features
[tool.poetry.extras]
rich-output = ["rich>=13.0.0"]
anthropic-support = ["anthropic^0.50.0"]
# Development/testing dependencies
[tool.poetry.group.dev.dependencies]
pytest = "^7.0.0"
```
This approach maintains a lean core package while providing flexibility for users who need additional functionality.
---
## Ensure database consistency
<!-- source: BerriAI/litellm | topic: Database | language: Python | updated: 2025-07-30 -->
Maintain data consistency by using appropriate database operations and ensuring transactional integrity when updating related fields. Choose the right ORM methods (upsert vs create) to handle potential duplicates, and ensure that updates to related fields across multiple tables happen atomically to prevent data drift.
Key practices:
1. Use `upsert()` instead of `create()` when records might already exist to avoid UniqueViolationError
2. When updating related fields across multiple tables (like user.teams and team.members_with_roles), ensure both updates succeed or fail together
3. Use correct schema field names in ORM include/select operations
Example of proper upsert usage:
```python
# Instead of create() which can fail on duplicates
await self.prisma_client.db.litellm_managedobjecttable.upsert(
where={"object_id": object_id},
data=data,
create=data
)
# Correct schema field names in includes
include={"end_users": True} # Not "litellm_endusertable"
```
This prevents UniqueViolationError exceptions and maintains referential integrity across related database entities.
---
## Eliminate redundant operations
<!-- source: LMCache/LMCache | topic: Performance Optimization | language: Python | updated: 2025-07-29 -->
Avoid unnecessary function calls, I/O operations, and redundant checks that can significantly impact performance. Look for opportunities to consolidate operations, remove duplicate work, and streamline execution paths.
Common patterns to watch for:
- Redundant function calls that could be eliminated or cached
- Unnecessary I/O operations like extra RPC calls or database queries
- Duplicate checks or validations that serve the same purpose
- Inefficient loops that could be replaced with batch operations
Example improvements:
```python
# Instead of repeated append() calls and redundant checks:
if location not in key_mapping:
key_mapping[location] = [key]
start_mapping[location] = [start]
end_mapping[location] = [end]
continue
key_mapping[location].append(key)
start_mapping[location].append(start)
end_mapping[location].append(end)
# Use efficient unpacking to avoid repeated operations:
if reordered_blocks:
_, memory_objs, starts, ends = zip(*reordered_blocks)
self.gpu_connector.batched_to_gpu(list(memory_objs), list(starts), list(ends), **kwargs)
```
```python
# Remove unnecessary I/O operations:
# Instead of: if self.exists_in_put_tasks(key) or self.contains(key):
# Just use: if self.exists_in_put_tasks(key):
# The contains() call adds unnecessary RPC or I/O overhead
```
Always question whether each operation is truly necessary and look for opportunities to batch, cache, or eliminate redundant work entirely.
---
## network service identification
<!-- source: LMCache/LMCache | topic: Networking | language: Python | updated: 2025-07-29 -->
Use descriptive string identifiers instead of numeric ports or generic IDs when designing network services to avoid conflicts and improve maintainability. Different network services should have clearly distinguishable communication channels.
When multiple network services operate within the same system, relying solely on numeric ports can lead to conflicts and configuration complexity. Instead, incorporate descriptive service names and contextual information into network identifiers.
Example of problematic approach:
```python
# Unclear what each port is for, potential conflicts
rpc_port = 0 # lookup service
rpc_port = 100 # offload service
socket_path = f"ipc://{base_url}/lmcache_rpc_port_{rpc_port}"
```
Improved approach with descriptive identifiers:
```python
# Clear service identification, no port conflicts
def get_zmq_rpc_path_lmcache(engine_id, service_name, tp_rank):
return f"ipc://{base_url}/engine_{engine_id}_service_{service_name}_tp_rank_{tp_rank}"
# Usage:
lookup_path = get_zmq_rpc_path_lmcache(engine_id, "lookup", tp_rank)
offload_path = get_zmq_rpc_path_lmcache(engine_id, "offload", tp_rank)
```
This approach eliminates the need for users to configure ports manually, reduces the risk of service conflicts, and makes the system's network architecture self-documenting. The descriptive identifiers make it immediately clear which service each communication channel serves, improving debugging and maintenance.
---
## consolidate algorithmic patterns
<!-- source: ggml-org/llama.cpp | topic: Algorithms | language: CUDA | updated: 2025-07-28 -->
Avoid duplicating similar algorithmic logic across multiple functions by consolidating them into generic, templated, or trait-based implementations. This reduces maintenance burden and improves code consistency.
Instead of creating separate functions for each data type or operation combination, use templates or generic approaches:
```cpp
// Avoid this - separate functions for each type combination
static __device__ void convert_f32_f32(const float * src, float * dst) { *dst = *src; }
static __device__ void convert_f16_f32(const half * src, float * dst) { *dst = *src; }
static __device__ void convert_bf16_f32(const nv_bfloat16 * src, float * dst) { *dst = *src; }
// Prefer this - generic template approach
template<typename src_t, typename dst_t>
static __device__ __forceinline__ void convert_to_flt(const src_t * src, dst_t * dst) {
*dst = float(*src); // Convert through float as common intermediate
}
```
Similarly, for algorithmic logic that varies only by parameters, use templates instead of switch statements or multiple similar functions:
```cpp
// Instead of separate functions for each granularity calculation
static int mmq_get_granularity_host(ggml_type type, const int mmq_x, const int cc) {
// Use trait-based approach or template specialization
}
// Consider using trait structs for type-specific behavior
template<ggml_type T> struct mmq_type_traits {
static constexpr int granularity(int mmq_x) { /* type-specific logic */ }
};
```
This approach is particularly important for computational kernels where similar patterns appear across different data types or operation sizes. Look for opportunities to extract common algorithmic patterns into reusable, parameterized implementations.
---
## eliminate code duplication
<!-- source: ggml-org/llama.cpp | topic: Code Style | language: C++ | updated: 2025-07-28 -->
Avoid duplicating code blocks by extracting common functionality into reusable functions, using templates for type-generic operations, and moving shared utilities to common modules. Code duplication makes maintenance difficult, increases the likelihood of bugs, and violates the DRY (Don't Repeat Yourself) principle.
When you find similar code patterns:
1. Extract common logic into helper functions
2. Use templates for type-generic operations instead of copying code for different types
3. Move shared utilities to common modules (like `common.cpp`) for reuse across files
4. Refactor similar functions to share implementation details
Example of extracting a common string utility:
```cpp
// Instead of duplicating string suffix logic:
static bool str_has_suffix(const std::string & str, const std::string & suffix) {
return str.size() >= suffix.size() && str.compare(str.size() - suffix.size(), str.size(), suffix) == 0;
}
// Move to common.cpp as string_ends_with() for reuse
```
Example of using templates instead of duplication:
```cpp
// Instead of separate functions for different types:
template<typename T>
static void ggml_compute_forward_repeat(const ggml_compute_params * params, ggml_tensor * dst) {
// Generic implementation
}
```
This approach improves maintainability, reduces bugs, and makes the codebase more consistent and easier to understand.
---
## leverage existing framework functionality
<!-- source: ggml-org/llama.cpp | topic: AI | language: Python | updated: 2025-07-28 -->
Before implementing custom solutions in AI model conversion code, thoroughly investigate whether the framework already provides the needed functionality. This applies to inheritance patterns, configuration handling, tensor operations, and mapping conventions.
Common anti-patterns to avoid:
- Copying code from existing model classes instead of using proper inheritance
- Manually setting configuration values that are automatically handled by the framework
- Implementing custom tensor operations when built-in functions exist
- Using string matching or hardcoded fallbacks instead of proper architecture detection
Example of proper approach:
```python
# Instead of copying LlamaModel code:
@ModelBase.register("MyModelForCausalLM")
class MyModel(TextModel): # ❌ Copying everything
# ... copied methods from LlamaModel
# Use proper inheritance:
@ModelBase.register("MyModelForCausalLM")
class MyModel(LlamaModel): # ✅ Inherit from specific model
model_arch = gguf.MODEL_ARCH.MYMODEL
# Only override what's actually different
# Instead of manual tensor operations:
# ❌ Manual splitting
gate_tensor = data_torch[:split_size, :]
up_tensor = data_torch[split_size:, :]
# ✅ Use framework functionality
# Use LLM_FFN_SWIGLU in model definition
```
Always check the framework's existing model implementations, utility functions, and configuration mechanisms before writing custom code. This reduces maintenance burden, improves consistency, and leverages tested functionality.
---
## Choose appropriate error mechanism
<!-- source: ggml-org/llama.cpp | topic: Error Handling | language: C++ | updated: 2025-07-28 -->
Use GGML_ASSERT for early validation and programming errors that indicate misconfigurations, but implement graceful error handling with logging for runtime conditions where fallback options exist. Assertions should catch problems early in development, while runtime errors should provide informative feedback and allow system recovery when possible.
For configuration validation and programming errors:
```cpp
// Good: Catch misconfiguration early
GGML_ASSERT(n_expert > 0 && "n_expert must be > 0 for SMALLTHINKER");
```
For runtime conditions with fallback options:
```cpp
// Good: Log and allow graceful fallback
if (!kernels) {
GGML_LOG_DEBUG("%s: No suitable KleidiAI kernel available, falling back to standard CPU implementation\n", __func__);
return false; // Let the system fallback to standard CPU implementation
}
```
Avoid silent failures that provide no indication of what went wrong or that will cause assertions to fail later in the call chain. Always provide meaningful error messages or logging to aid debugging and system monitoring.
---
## Choose efficient data structures
<!-- source: LMCache/LMCache | topic: Algorithms | language: Python | updated: 2025-07-28 -->
Select appropriate built-in data structures and algorithmic patterns that eliminate manual checks and reduce code complexity. This improves both performance and maintainability by leveraging Python's optimized implementations.
Key principles:
1. **Use defaultdict instead of manual key existence checks** - Replace patterns like `if key not in dict: dict[key] = []` with `defaultdict(list)`
2. **Try direct operations before fallback logic** - Attempt the primary algorithm first, then handle edge cases, rather than pre-checking conditions
3. **Leverage built-in data structure behaviors** - Use data structures that naturally handle your use case rather than implementing the logic manually
Example from the codebase:
```python
# Instead of manual key checking:
if location not in key_mapping:
key_mapping[location] = [key]
start_mapping[location] = [start]
end_mapping[location] = [end]
# Use defaultdict to eliminate the check:
from collections import defaultdict
block_mapping: defaultdict = defaultdict(list)
block_mapping[location].append((key, start, end))
```
For memory management algorithms, try allocation first before implementing eviction logic:
```python
# Try direct allocation first
memory_obj = self.memory_allocator.allocate(shape, dtype)
if memory_obj is None:
# Then implement eviction logic
self._evict_and_retry_allocation(shape, dtype)
```
This approach reduces branching, leverages optimized implementations, and makes code more readable and less error-prone.
---
## maintain naming consistency
<!-- source: ggml-org/llama.cpp | topic: Naming Conventions | language: C++ | updated: 2025-07-27 -->
Follow established naming patterns and conventions consistently throughout the codebase. This includes using consistent formatting (dashes vs underscores), proper capitalization, descriptive parameter names, and uniform data type representations.
Key principles:
- Use consistent separators: prefer dashes over underscores in identifiers like `"falcon-h1"` instead of `"falcon_h1"`
- Follow established capitalization patterns: `"kFLOP"` not `"KFLOP"`
- Use descriptive parameter names: `type` instead of single letters like `T`
- Maintain consistency in data representation: use either byte offsets or element offsets throughout, not both
- Choose clear, semantic names that convey purpose: `send_progress` instead of `include_prompt_progress`
Example of inconsistent naming:
```cpp
// Inconsistent - mixing byte and element offsets
params[1] = src_misalignment; // byte offset
params[3] = src->nb[0]/ggml_type_size(src->type); // element offset
// Consistent - all element offsets
params[1] = src_misalignment_elements;
params[3] = src_stride_elements;
```
Before introducing new names, check existing codebase patterns and follow the established conventions. This reduces cognitive load and makes the code more maintainable.
---
## AI documentation precision
<!-- source: ggml-org/llama.cpp | topic: AI | language: Markdown | updated: 2025-07-27 -->
Ensure documentation for AI/ML tools and libraries is precise, complete, and properly formatted. This includes being specific about tool references, providing complete command-line option formats, and using proper markdown formatting.
Key practices:
- Be explicit about tool sources and relationships (e.g., "data generated by `llama-imatrix`" instead of just "data in file")
- Show both short and long forms of command-line options (e.g., `--chunk | --from-chunk` instead of just `--chunk`)
- Use proper markdown formatting for links and references
- Ensure all relevant options and methods are documented completely
Example improvements:
```markdown
# Before
* `--imatrix` uses data in file as importance matrix
* `--chunk` to skip the first n chunks
* More information: https://github.com/example/repo
# After
* `--imatrix` uses data in file generated by `llama-imatrix` as importance matrix
* `--chunk | --from-chunk` to skip the first n chunks
* More information is [available here](https://github.com/example/repo)
```
This ensures users have clear, actionable information when working with AI/ML tools and can easily understand tool relationships and usage patterns.
---
## AI model documentation consistency
<!-- source: sgl-project/sglang | topic: AI | language: Other | updated: 2025-07-27 -->
Ensure consistent parameter names and configuration instructions across AI model documentation. Inconsistent documentation about model parsers, parameters, and configuration options leads to developer confusion and implementation errors.
When documenting AI model configuration options, maintain consistency in:
- Parameter names and formats (e.g., `qwen3-thinking` vs `qwen3/qwen3-thinking`)
- Usage instructions and examples
- Supported configuration combinations
Example of inconsistent documentation to avoid:
```
# In one place:
Use `--reasoning-parser qwen3/qwen3-thinking`
# In another place:
Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3`
```
Instead, establish and follow a single standard format across all documentation, clearly specifying which parsers work with which model types and their exact parameter syntax.
---
## use model metadata
<!-- source: ggml-org/llama.cpp | topic: AI | language: C++ | updated: 2025-07-26 -->
Leverage model metadata from GGUF format instead of hardcoded configuration values or filename-based logic. Model metadata provides authoritative information about model behavior and should be the primary source for configuration decisions.
This approach improves maintainability by centralizing configuration in the model file, reduces the risk of incorrect assumptions, and aligns with the project's migration toward structured GGUF format.
Examples of preferred patterns:
```cpp
// Instead of hardcoded logic:
if (arch == "llada") {
llama_token bos_token = llama_vocab_bos(vocab);
if (bos_token != LLAMA_TOKEN_NULL && (input_tokens.empty() || input_tokens[0] != bos_token)) {
input_tokens.insert(input_tokens.begin(), bos_token);
}
}
// Use model metadata:
// This should be handled by the metadata in the GGUF model.
// There is a boolean field for when BOS is needed or not.
```
```cpp
// Instead of filename-based format detection:
if (!string_ends_with(fname, ".gguf")) {
LOG_WRN("saving to legacy format because output suffix is not .gguf");
save_legacy_format();
}
// Use GGUF format by default:
// The format should not be decided by the output filename
```
Always check for relevant metadata fields in the GGUF model before falling back to hardcoded values or heuristics. This ensures model-specific behavior is correctly handled and reduces maintenance burden when supporting new model variants.
---
## Cache state consistency
<!-- source: LMCache/LMCache | topic: Caching | language: Python | updated: 2025-07-26 -->
Ensure cache operations maintain consistent state and handle shared resources properly throughout the cache lifecycle. This includes proper key type management, safe eviction of shared entries, and accurate state tracking across different operational phases.
Key practices:
1. **Type-safe cache keys**: Always use proper key types rather than strings to prevent cache misses due to type mismatches
2. **Reference-aware eviction**: Check reference counts before evicting cache entries that may be shared between multiple requests
3. **Consistent state tracking**: Maintain accurate phase detection and state information to prevent incorrect cache behavior
Example of proper reference checking before eviction:
```python
# Check ref_count before evicting shared cache entries
if self.memory_allocator.get_ref_count(self.hot_cache[evict_key]) > 1:
continue # Skip eviction if still referenced
```
Example of proper key type handling:
```python
# Convert string keys to proper types before cache operations
for idx, key_str in enumerate(alloc_request.keys):
key = CacheEngineKey.from_string(key_str) # Ensure proper type
if self._backend.contains(key, pin=True): # Now works correctly
```
This prevents cache leaks, ensures proper cache hits, and maintains system reliability when cache entries are shared across multiple operations or requests.
---
## AI model architecture flexibility
<!-- source: LMCache/LMCache | topic: AI | language: Python | updated: 2025-07-25 -->
Avoid hardcoding AI model-specific assumptions and instead design systems to be configurable across different model architectures, data types, and configurations. This ensures your AI systems can adapt to evolving model designs and support multiple model families without requiring code changes.
Key areas to make flexible:
1. **Data types**: Support multiple tensor types (bfloat16, float16, float32) rather than assuming a single type
2. **Model architecture parameters**: Use dynamic configuration based on model metadata rather than hardcoded layer counts or dimensions
3. **Model-specific features**: Design abstractions that can handle different model formats (e.g., MLA vs non-MLA architectures)
Example of inflexible code:
```python
# Bad: Hardcoded for specific models
if model_name in ["llama-7b", "llama-8b"]:
return CacheGenConfig(
key_first_layers=10,
key_second_layers=20,
key_third_layers=32, # hardcoded total layers
key_first_bins=32,
# ...
)
# Bad: Assumes specific data type
assert t.dtype == torch.float16 # Only supports fp16
```
Example of flexible code:
```python
# Good: Dynamic configuration based on model metadata
return CacheGenConfig(
nlayers=model_config.num_layers, # Use actual layer count
kspecs=[
QuantizationSpec(start_layer=0, end_layer=model_config.num_layers//3, bins=32),
QuantizationSpec(start_layer=model_config.num_layers//3, end_layer=model_config.num_layers, bins=16),
],
# ...
)
# Good: Support multiple data types
DTYPE_TO_TAG = {
torch.bfloat16: 0,
torch.float16: 1,
torch.float32: 2,
}
tag = DTYPE_TO_TAG[t.dtype] # Handle any supported type
```
This approach ensures your AI systems remain maintainable and can easily support new model architectures as they emerge.
---
## Raise exceptions properly
<!-- source: Unstructured-IO/unstructured | topic: Error Handling | language: Python | updated: 2025-07-25 -->
Always ensure exceptions are properly raised and propagated rather than being logged and swallowed. Avoid patterns where errors are caught, logged, and then execution continues normally, as this can hide critical failures from calling code.
Key principles:
1. **Re-raise after logging**: When catching exceptions for logging purposes, always re-raise them so calling code can handle the failure appropriately
2. **Avoid returning error strings**: Don't return error messages as strings from functions - raise exceptions instead so they can be properly handled by the caller
3. **Use specific exception handling**: Avoid overly broad `except Exception:` blocks that might hide unexpected errors. When you must use broad handlers, ensure error details are preserved and re-raised
4. **Control exception chaining**: Use `from None` when re-raising exceptions to prevent sensitive data (like file contents) from being captured in exception chains
Example of proper error handling:
```python
# Bad - logs error but continues execution
try:
result = risky_operation()
except Exception as e:
logger.error(f"Error occurred: {e}")
# Execution continues, error is hidden
# Good - logs error and re-raises
try:
result = risky_operation()
except Exception as e:
logger.error(f"Error occurred: {e}")
raise # or raise specific exception type
# Good - prevents data leakage in exception chains
try:
file_text = byte_data.decode(encoding)
except (UnicodeDecodeError, UnicodeError):
raise UnprocessableEntityError(
f"File encoding detection failed: detected '{encoding}' but decode failed."
) from None # Prevents original exception with file data from being chained
```
This ensures that failures are communicated to calling code and can be handled appropriately, rather than being silently ignored.
---
## systematic test coverage
<!-- source: ggml-org/llama.cpp | topic: Testing | language: C++ | updated: 2025-07-25 -->
When writing tests that need to cover multiple scenarios (different data types, backend capabilities, etc.), structure them systematically rather than duplicating code or hardcoding conditional logic.
For multiple data types, use loops through type arrays instead of duplicating test cases:
```cpp
// Instead of duplicating test cases for each type
test_cases.emplace_back(new test_conv_2d(..., GGML_TYPE_F32, ...));
test_cases.emplace_back(new test_conv_2d(..., GGML_TYPE_F16, ...));
// Use a loop through types
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16}) {
test_cases.emplace_back(new test_conv_2d(..., type, ...));
}
```
For backend-specific capabilities, let the backend report what it supports rather than hardcoding conditional test logic. This allows tests to automatically skip unsupported operations and remain maintainable as backend capabilities evolve.
This approach ensures comprehensive test coverage while keeping the test code clean, maintainable, and adaptable to different environments and configurations.
---
## Metal shared memory sizing
<!-- source: ggml-org/llama.cpp | topic: Performance Optimization | language: Objective-C | updated: 2025-07-25 -->
When allocating shared memory for Metal compute shaders, ensure proper sizing that accounts for both SIMD group requirements and Metal platform constraints. Metal requires shared memory buffer sizes to be multiples of 16 bytes. Consider using simplified allocation strategies that meet platform requirements while maintaining correctness.
For SIMD-based operations, you can often allocate a fixed amount of shared memory for simplicity rather than complex calculations:
```objc
// Simple approach - ensures Metal alignment requirements
const int64_t shmem_size = 32; // 32*sizeof(float) bytes
[encoder setThreadgroupMemoryLength:shmem_size*sizeof(float) atIndex:0];
```
This approach handles Metal's alignment requirements automatically while providing sufficient memory for typical SIMD operations. When precise sizing is needed, ensure the calculated size accounts for the number of SIMD groups in the threadgroup (typically `d_state / simd_size`) but always verify the result meets Metal's 16-byte alignment constraint.
---
## optimize memory access patterns
<!-- source: ggml-org/llama.cpp | topic: Performance Optimization | language: CUDA | updated: 2025-07-24 -->
Ensure CUDA kernels use optimal memory access patterns to maximize performance. This involves several key practices:
1. **Use `__restrict__` qualifiers** on pointer parameters to inform the compiler that pointers don't alias, enabling better optimizations especially on older GPUs.
2. **Organize threads for coalesced memory access** by ensuring adjacent threads access adjacent memory locations. Map thread indices to tensor data as if the tensor was flattened, avoiding fractional warps.
3. **Use typed pointers instead of byte offsets** when possible, as explicit byte offset patterns perform poorly on GPUs compared to CPU code.
4. **Ensure proper thread-to-data mapping** to avoid uncoalesced accesses.
Example of proper thread organization for coalesced access:
```cuda
// Good: Adjacent threads access adjacent memory
const int i01 = blockIdx.x;
const int i00 = blockIdx.y*blockDim.x + threadIdx.x;
// Bad: Non-coalesced access pattern
const int i01 = blockDim.x * blockIdx.x + threadIdx.x;
const int i00 = blockIdx.y;
```
Example of using `__restrict__` qualifiers:
```cuda
static __global__ void kernel(
const void * __restrict__ src0,
void * __restrict__ dst,
// ... other parameters
) {
// kernel implementation
}
```
These optimizations are critical for GPU performance as memory bandwidth is often the limiting factor, and proper access patterns can significantly improve throughput.
---
## configurable security settings
<!-- source: BerriAI/litellm | topic: Security | language: Python | updated: 2025-07-24 -->
Security configurations should be made configurable through environment variables or runtime settings rather than hardcoded in the application logic. This allows teams to adapt security measures to different deployment environments and regulatory requirements without code changes.
Hardcoded security settings create inflexibility and force unnecessary security overhead in environments where it's not needed. Instead, implement conditional security logic that can be toggled based on configuration.
Example of configurable security headers:
```python
@app.middleware("http")
async def add_headers(request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
# Make security headers configurable via environment variables
if os.getenv("ENABLE_SECURITY_HEADERS", "false").lower() == "true":
response.headers["Content-Security-Policy"] = "default-src 'none'"
response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
# Conditional validation based on environment requirements
if os.getenv("STRICT_USER_VALIDATION", "false").lower() == "true":
if "user_id" in metadata and not _valid_user_id(metadata["user_id"]):
# Skip invalid user_id rather than failing
pass
```
This approach enables compliance with regulatory standards when needed while maintaining flexibility for different deployment scenarios.
---
## AI dependency justification
<!-- source: sgl-project/sglang | topic: AI | language: Toml | updated: 2025-07-24 -->
When modifying AI library dependencies, always provide clear justification for version choices, especially when using non-standard sources like git commits or release candidates. This is critical for AI projects where library compatibility and stability directly impact model performance and inference reliability.
For standard version updates, explain the motivation (bug fixes, new features, compatibility requirements). For non-standard sources like git commits, document:
- The specific issue being resolved
- Why the official release is insufficient
- Plans for migrating back to official releases
Example of good justification:
```python
# Using git commit due to bug in official v4.49.0 that affects vision models
"transformers @ git+https://github.com/huggingface/transformers.git@84f0186",
# TODO: Migrate back to official release when v4.49.1+ is available
```
This practice prevents confusion during code reviews, helps with dependency auditing, and ensures the team understands the rationale behind critical AI library choices that could affect model behavior or performance.
---
## Incremental dependency updates
<!-- source: BerriAI/litellm | topic: Configurations | language: Txt | updated: 2025-07-23 -->
When updating dependencies in configuration files like requirements.txt, break large changes into smaller, focused pull requests that can be individually tested and validated. Each dependency update should include clear justification for why the change is necessary, especially when it triggers cascading updates to other packages.
Before proposing dependency bumps, verify they are actually required by attempting to revert the changes and running the full test suite. If tests pass without the update, consider whether the dependency change is truly needed.
For example, when updating multiple packages:
```
# Instead of updating all at once:
# openai==1.40.0 -> 1.45.0
# google-cloud-aiplatform==1.50.0 -> 1.61.0
# protobuf==4.25.0 -> 5.0.0
# Break into separate PRs:
# PR 1: Update openai to 1.45.0
# PR 2: Update protobuf to 5.0.0 (with justification)
# PR 3: Update google-cloud-aiplatform to 1.61.0 (if still needed)
```
This approach allows load testing and CI/CD validation of each change independently, making it easier to identify which specific update might cause issues and reducing the blast radius of potential problems.
---
## consistent clear naming
<!-- source: ggml-org/llama.cpp | topic: Naming Conventions | language: CUDA | updated: 2025-07-23 -->
Ensure variable and identifier names are both consistent and unambiguous within their context. When multiple equivalent naming options exist (such as `tile_B::I` vs `tile_C::J` for the same value), choose one approach and apply it consistently throughout the codebase. Additionally, select names that clearly distinguish their purpose and avoid confusion with similar concepts in the same scope.
For consistency: If `tile_C::J` is chosen over `tile_B::I`, use `tile_C::J` consistently in all related code locations.
For clarity: Instead of potentially confusing names like `ne01` that might be mistaken for byte offsets, use more descriptive alternatives like `s01` that clearly indicate their distinct purpose.
This approach prevents both inconsistent naming patterns and semantic confusion that can lead to bugs and reduced code maintainability.
---
## Use precise documentation details
<!-- source: ggml-org/llama.cpp | topic: Documentation | language: Markdown | updated: 2025-07-23 -->
Documentation should provide specific, actionable information rather than vague or generic references. This applies to both technical specifications and descriptive text.
For technical details, use exact package names, commands, or identifiers instead of approximate terms:
```markdown
# Vague - avoid this
- Install curl-dev package
# Precise - do this
- Debian/Ubuntu: `sudo apt-get install libcurl4-openssl-dev`
- Fedora/RHEL: `sudo dnf install libcurl-devel`
- Arch: `sudo pacman -S curl`
```
For links, use descriptive text or show identifiers rather than generic phrases like "available here" or "click here". This improves accessibility and makes the purpose of the link immediately clear to readers.
```markdown
# Vague - avoid this
More information is [available here](https://github.com/org/repo/pull/4861)
# Precise - do this
More information is available in <https://github.com/org/repo/pull/4861>
```
This approach ensures documentation is immediately useful and reduces ambiguity for users trying to follow instructions or understand references.
---
## eliminate code redundancy
<!-- source: menloresearch/jan | topic: Code Style | language: TypeScript | updated: 2025-07-22 -->
Remove duplicate imports, unused legacy code, and repetitive patterns to maintain clean, organized codebases. Extract commonly used functionality into utilities and avoid code duplication through proper abstractions. Use early returns to reduce nesting and improve readability.
Examples of redundancy to eliminate:
- Duplicate imports: Remove repeated import statements like `import { basename } from '@tauri-apps/api/path'` appearing multiple times
- Legacy code cleanup: Remove unused variables and outdated code patterns that no longer serve a purpose
- Utility extraction: Move system-wide functionality like ID generation (`jan-${(Date.now() / 1000).toFixed(0)}`) into shared utilities
- Early returns: Replace nested conditionals with early returns (`if (condition) return` at the start of functions)
- Code duplication: Consolidate similar functionality into reusable abstractions rather than duplicating logic across multiple locations
---
## Configuration validation standards
<!-- source: LMCache/LMCache | topic: Configurations | language: Python | updated: 2025-07-22 -->
Ensure configuration parameters follow consistent validation, naming, and default value standards to improve code safety and clarity.
**Key Requirements:**
1. **Use None for optional configurations** instead of empty strings or other placeholder values
2. **Validate configuration combinations** when multiple related settings must be checked together
3. **Use clear, unambiguous parameter names** that accurately describe their purpose
4. **Implement proper type checking** for configuration values, especially when they can be strings or booleans
**Example:**
```python
# Bad: Ambiguous naming and empty string default
disk_url: str = ""
# Good: Clear naming and None default
disk_manager_url: Optional[str] = None
# Bad: Single condition check
self.remove_after_retrieve = config.nixl_role == "receiver"
# Good: Validate related conditions together
self.remove_after_retrieve = config.enable_nixl and config.nixl_role == "receiver"
# Good: Proper type validation for config values
if config.extra_config is not None:
use_cufile = config.extra_config.get("use_cufile", None)
if use_cufile is not None:
if isinstance(use_cufile, str):
use_cufile = use_cufile.lower() == "true"
elif use_cufile in [False, True]:
# Handle boolean values
pass
```
This approach prevents configuration-related bugs, makes the codebase more maintainable, and provides clearer error messages when invalid configurations are provided.
---
## prefer const variables
<!-- source: ggml-org/llama.cpp | topic: Code Style | language: CUDA | updated: 2025-07-22 -->
Add the `const` qualifier to variables that are not modified after initialization. This improves code clarity, prevents accidental modifications, and makes the code's intent more explicit to both compilers and other developers.
Apply `const` to:
- Local variables that are initialized once and never changed
- Function parameters that are not modified within the function
- Any variable where the value remains constant throughout its scope
Example transformations:
```cpp
// Before
float2 dsB;
dsB = __half22float2(y_dm[j*MMQ_TILE_Y_K + k01/QI8_1]);
int global_idx = blockIdx.x * blockDim.x + threadIdx.x;
int total_elements = batches * channels * out_h * out_w;
// After
const float2 dsB = __half22float2(y_dm[j*MMQ_TILE_Y_K + k01/QI8_1]);
const int global_idx = blockIdx.x * blockDim.x + threadIdx.x;
const int total_elements = batches * channels * out_h * out_w;
```
This practice is especially important in CUDA kernels and performance-critical code where clarity about data mutability helps with optimization and debugging.
---
## Improve code modularity
<!-- source: microsoft/markitdown | topic: Code Style | language: Python | updated: 2025-07-19 -->
Organize code into well-structured, reusable components by extracting specific functionality into separate functions, using flexible parameter patterns, and consolidating duplicate logic. This improves readability, maintainability, and extensibility.
Key practices:
1. **Extract validation logic**: When you have complex validation checks, create dedicated validation functions rather than embedding them inline
2. **Use parameter forwarding**: When passing parameters to nested functions, consider using `**kwargs` to create a flexible foundation for future extensions
3. **Consolidate duplicate functionality**: When similar logic appears in multiple places, refactor it into a shared, reusable function
Example of good modularity:
```python
# Instead of inline validation
def process_file(file_path):
if not file_path.exists():
_exit_with_error(f"File does not exist: {file_path}")
if not file_path.is_file():
_exit_with_error(f"Path is not a file: {file_path}")
# ... processing logic
# Extract validation into separate function
def _validate_file_path(file_path):
if not file_path.exists():
_exit_with_error(f"File does not exist: {file_path}")
if not file_path.is_file():
_exit_with_error(f"Path is not a file: {file_path}")
def process_file(file_path):
_validate_file_path(file_path)
# ... processing logic
# Use parameter forwarding for flexibility
def create_converter(**kwargs):
return CustomConverter(**kwargs) # Instead of passing individual params
```
---
## CUDA compatibility verification
<!-- source: sgl-project/sglang | topic: AI | language: Dockerfile | updated: 2025-07-18 -->
When modifying CUDA versions or GPU architecture configurations in AI/ML projects, verify compatibility across three critical dimensions: CUDA version support, GPU architecture coverage, and dependent library requirements. Check official compatibility matrices, test with target hardware, and validate that AI libraries maintain their functionality.
For GPU architectures, prefer inclusive configurations that support multiple generations rather than restrictive single-architecture builds. For example, use `CMAKE_CUDA_ARCHITECTURES=90;100;120` and `TORCH_CUDA_ARCH_LIST="9.0 10.0 12.0"` to support Hopper and Blackwell families together.
Before upgrading CUDA versions, consult framework release notes (e.g., PyTorch dropped CUDA 12.4 support in v2.7.0) and verify that critical dependencies like specialized AI libraries won't break. Some libraries have strict version requirements that may disable features if not met exactly.
Example verification process:
```dockerfile
# Check base image CUDA version
FROM nvcr.io/nvidia/tritonserver:25.03-py3-min # Contains CUDA 12.8
# Configure for multiple GPU generations
ARG CMAKE_CUDA_ARCHITECTURES=90;100;120
ARG TORCH_CUDA_ARCH_LIST="9.0 10.0 12.0"
# Validate compatibility with AI dependencies
RUN python -c "import torch; print(f'CUDA: {torch.version.cuda}, Architectures: {torch.cuda.get_arch_list()}')"
```
---
## explicit None handling
<!-- source: Unstructured-IO/unstructured | topic: Null Handling | language: Python | updated: 2025-07-16 -->
Always handle potential None values explicitly through defensive programming patterns, proper type annotations, and explicit checks to prevent runtime errors and improve code reliability.
Use defensive patterns like `value or default` for safe fallbacks:
```python
# Good: Explicit None handling with fallback
e.metadata.languages = detect_languages(e.text or "")
full_text = " ".join(e.text or "" for e in elements if hasattr(e, "text"))
# Good: Explicit None checks with proper logic
if encoding is None or (confidence is not None and confidence < ENCODE_REC_THRESHOLD):
# handle the case
```
Use proper Optional type annotations when values can be None:
```python
# Good: Correct Optional typing
max_pages: Optional[int] = None
customer_id: Optional[str] = None
# Bad: Missing Optional annotation
customer_id: str = None # Type mismatch
```
Handle functions that may return None explicitly:
```python
# Good: Check for None return value
nltk_data_dir = get_nltk_data_dir()
if nltk_data_dir is None:
raise ValueError("Could not find a default download directory")
# Good: Use assertions when None is guaranteed not to occur
assert self._file is not None # -- assured by ._validate() --
```
Avoid unnecessary None-to-string conversions and prefer removing optionality at interface boundaries when the None case adds no value. When designing APIs, consider whether Optional parameters truly need to be optional or if a sensible default can be provided instead.
---
## Use descriptive consistent names
<!-- source: LMCache/LMCache | topic: Naming Conventions | language: Python | updated: 2025-07-16 -->
Choose names that clearly convey their purpose, role, and meaning while maintaining consistency with established patterns in the codebase. Names should be self-documenting and unambiguous to future maintainers.
Key principles:
1. **Semantic clarity**: Names should reflect their actual purpose and role in the system architecture. For example, use `receiver_host` instead of `peer_host_name` when the architecture has a clear receiver-sender relationship, or `num_skip_prefix_chunk` instead of `num_skip_chunk` when the meaning is "number of prefix chunks to skip".
2. **Consistency with patterns**: Follow established naming conventions in the codebase. If other batch operations use `batched_xxx` pattern, use `batched_contains` instead of `batch_contains`. Similarly, maintain consistency between related functions like using `batched` instead of mixing `layerwise` and `batched`.
3. **Avoid ambiguous abbreviations**: Use clear variable names instead of cryptic abbreviations. Replace unclear names like `anw` and `anws` with descriptive names like `cache_exists_results`.
4. **Distinguish similar entities**: When multiple files or classes serve similar but distinct purposes, use names that clearly differentiate them. Instead of `disagg_proxy_server.py` and `disagg_proxy_server_original.py`, use names that describe their specific roles.
Example of good naming:
```python
# Instead of:
def batch_contains(self, keys): pass
anws = self.engine_.batched_contains(keys)
for anw in anws:
if not anw:
# Use:
def batched_contains(self, keys): pass # Consistent with other batched_xxx methods
cache_exists_results = self.engine_.batched_contains(keys)
for cache_exists in cache_exists_results:
if not cache_exists:
```
---
## Follow naming conventions
<!-- source: ggml-org/llama.cpp | topic: Naming Conventions | language: Other | updated: 2025-07-16 -->
Maintain consistency with established naming patterns and conventions throughout the codebase. This includes matching enum names with their value prefixes, following existing API naming patterns, using consistent variable naming within similar contexts, and applying appropriate prefixes to avoid naming conflicts.
Key principles:
- **Enum consistency**: Match enum names with their value prefixes (e.g., `diffusion_alg` for `DIFFUSION_ALG_*` values)
- **API pattern adherence**: Maintain established function naming patterns (e.g., `llama_sampler_init_*` family)
- **Variable consistency**: Use consistent naming within similar contexts (e.g., `api_prefix` instead of `server_prefix` when other variables use `api_*`)
- **Appropriate specificity**: Prefer generic names when functionality isn't specific to one implementation (`LLM_KV_MAMBA_RMS_NORM` instead of `LLM_KV_FALCON_H1_MAMBA_RMS_NORM`)
- **Conflict avoidance**: Add appropriate prefixes to prevent naming collisions (e.g., `GGML_CPU_` prefix for CPU-specific macros)
Example:
```c
// Good: Enum name matches value prefix
enum diffusion_alg {
DIFFUSION_ALG_ORIGIN = 0,
DIFFUSION_ALG_MASKGIT_PLUS = 1,
};
// Bad: Enum name doesn't match value prefix
enum diffusion_algorithm {
DIFFUSION_ALG_ORIGIN = 0,
DIFFUSION_ALG_MASKGIT_PLUS = 1,
};
```
---
## measure before optimizing
<!-- source: ggml-org/llama.cpp | topic: Performance Optimization | language: C++ | updated: 2025-07-16 -->
Always profile and measure performance impact before implementing optimizations, especially micro-optimizations. Many seemingly beneficial optimizations provide no measurable improvement and can hurt code maintainability.
Before adding performance optimizations:
1. Use proper profiling tools to identify actual bottlenecks
2. Measure the current performance with realistic workloads
3. Implement the optimization
4. Measure again to verify the improvement is significant
5. Consider the maintenance cost vs. performance gain
Example from optimizer parameter pre-computation:
```cpp
// Before: Micro-optimization attempt
const float keep = 1.0f - opt_pars.adamw.alpha * opt_pars.adamw.wd;
// After measurement: "the micro-optimization has no visible benefit in this case"
// Reverted to simpler, more maintainable code
```
Common scenarios where measurement reveals optimizations are ineffective:
- CPU-bound optimizations in I/O-bound operations
- Micro-optimizations in infrequently called code paths
- Complex optimizations with "microscopic" performance gains
- Operations that run once per request rather than per iteration
Use profiling-enabled tools when available (e.g., `enqueue_nd_range` for kernel timing) and prefer `steady_clock` over `high_resolution_clock` for accurate timing measurements. Remember that maintainability and code clarity often outweigh minimal performance gains.
---
## Use strongly typed configurations
<!-- source: ggml-org/llama.cpp | topic: Configurations | language: C++ | updated: 2025-07-16 -->
Prefer strongly typed configuration values over primitive types to improve code safety and clarity. Use enum classes instead of raw strings for categorical options, and proper boolean types instead of integer flags for binary settings.
For categorical configurations, replace string parameters with enum classes:
```cpp
// Instead of:
params.dataset_format = format; // string
// Use:
enum class DatasetFormat { AUTO, TEXT, PARQUET, GGUF };
params.dataset_format = DatasetFormat::AUTO;
```
For boolean configurations, use explicit boolean conversion and values:
```cpp
// Instead of:
supports_set_rows = LLAMA_SET_ROWS ? atoi(LLAMA_SET_ROWS) : 0;
// Use:
supports_set_rows = LLAMA_SET_ROWS ? atoi(LLAMA_SET_ROWS) != 0 : false;
```
This approach prevents invalid configuration values, makes code more self-documenting, and enables better IDE support with auto-completion and type checking.
---
## Ensure operation completion safety
<!-- source: LMCache/LMCache | topic: Concurrency | language: Python | updated: 2025-07-15 -->
In concurrent code, ensure that operations complete safely before proceeding to prevent race conditions and data corruption. This applies to several scenarios:
**Lock Management**: Use context managers instead of manual lock acquisition to guarantee proper cleanup:
```python
# Good - automatic cleanup
with self.manager_lock:
prefetch_task = self.prefetch_tasks.pop(key)
# Avoid - manual lock management
self.manager_lock.acquire()
prefetch_task = self.prefetch_tasks.pop(key)
self.manager_lock.release()
```
**Async Operation Safety**: When data integrity is critical, ensure async operations complete before proceeding:
```python
# Good - synchronous when safety is needed
memory_obj.tensor.copy_(hidden_states)
# Risky - async copy without synchronization
memory_obj.tensor.copy_(hidden_states, non_blocking=True)
```
**Atomic File Operations**: Prevent race conditions in file operations using temporary files and atomic renames:
```python
# Good - atomic operation
tmp_path = path + ".tmp"
with open(tmp_path, 'wb') as f:
f.write(data)
os.rename(tmp_path, path) # Atomic on most filesystems
```
**Reference Counting Timing**: Ensure reference counting operations happen at the right time relative to object usage:
```python
# Ensure backend consumes object before counting down
backend.submit_put_task(key, obj)
# Backend should call ref_count_down() after consumption
```
**Stream Synchronization**: Use proper stream synchronization for CUDA operations:
```python
put_stream = torch.cuda.Stream()
if kv_chunk.device != torch.cpu:
put_stream.wait_stream(torch.cuda.default_stream(kv_chunk.device))
```
The key principle is to identify critical sections where partial completion could cause issues and ensure these operations are atomic or properly synchronized.
---
## Use proper logging mechanisms
<!-- source: BerriAI/litellm | topic: Logging | language: Python | updated: 2025-07-15 -->
Always use the logging library instead of print statements and maintain consistent logging patterns throughout the codebase. Print statements should be avoided as they bypass proper log level controls, formatting, and routing. Instead, use appropriate logging methods like `verbose_logger.info()`, `verbose_logger.debug()`, or `verbose_logger.warning()`.
Centralize logging logic to avoid scattered conditional blocks that can lead to bugs and inconsistent behavior. When implementing logging functionality, encapsulate the logic in dedicated functions or classes rather than duplicating if/else patterns across the codebase.
For logging integrations, use standardized logging objects and interfaces to ensure compatibility and prevent downstream issues.
Example of what to avoid:
```python
print("error", e) # Don't use print statements
print(file=sys.stderr) # Even with stderr redirection
```
Example of proper approach:
```python
verbose_logger.info("Processing completed successfully")
verbose_logger.debug("Bedrock AI: make_bedrock_api_request response: %s", redacted_response)
verbose_logger.warning("PostHog API Key not found in environment variables")
```
When dealing with sensitive data, always redact PII and confidential information before logging to prevent security leaks.
---
## explicit control flow logic
<!-- source: ggml-org/llama.cpp | topic: Algorithms | language: C++ | updated: 2025-07-14 -->
Use explicit control flow structures like switch statements instead of complex boolean logic or implicit reasoning when handling algorithmic decisions. This improves code clarity, maintainability, and reduces the likelihood of logical errors in computational paths.
Key principles:
- Replace complex boolean conditions with explicit switch statements when dealing with enumerated types or distinct algorithmic paths
- Maintain consistent state throughout algorithm execution rather than allowing state changes mid-process
- Separate different algorithmic concerns (e.g., buffer allocation vs gradient accumulation logic)
- Ensure algorithmic compatibility constraints are explicitly validated (e.g., sequence independence in batch processing)
Example of preferred approach:
```cpp
// Instead of implicit boolean reasoning:
struct ggml_tensor * opt_step =
m ? ggml_opt_step_adamw(ctx, node, grad, m, v, params) :
ggml_opt_step_sgd(ctx, node, grad, params);
// Use explicit switch statement:
struct ggml_tensor * opt_step;
switch (opt_ctx->optimizer_type) {
case GGML_OPT_OPTIMIZER_ADAMW:
opt_step = ggml_opt_step_adamw(ctx, node, grad, m, v, params);
break;
case GGML_OPT_OPTIMIZER_SGD:
opt_step = ggml_opt_step_sgd(ctx, node, grad, params);
break;
default:
GGML_ABORT("fatal error");
break;
}
```
This approach makes algorithmic decisions explicit, improves debugging, and ensures all code paths are properly handled. It also prevents mixing of unrelated algorithmic concerns and maintains clearer separation of computational logic.
---
## AI parameter organization
<!-- source: ggml-org/llama.cpp | topic: AI | language: Other | updated: 2025-07-14 -->
Ensure proper organization and placement of AI model parameters, hyperparameters, and configuration options. Avoid parameter duplication by reusing existing values, and place parameters in their logically appropriate structures.
Key principles:
- Avoid duplicating parameters that are already available elsewhere in the codebase
- Use existing parameters instead of creating new ones when functionality overlaps
- Place parameters in their conceptually correct locations (e.g., epochs belong with training logic, not optimizer configuration)
- Follow established patterns for parameter organization within the codebase
Example of good practice:
```cpp
// Instead of adding a new parameter:
struct common_params_diffusion {
int32_t max_length = 512; // DON'T: create new parameter
};
// Use existing parameter:
// max_length should be removed and the existing n_ubatch parameter should be used instead
// Instead of duplicating available data:
const int n_swa; // DON'T: duplicate from hparams
// This is already available from the hparams - no need to duplicate it here
```
This practice is especially important in AI model implementations where parameter proliferation can lead to configuration inconsistencies, maintenance overhead, and potential bugs in model behavior. Always check if the functionality you need already exists before introducing new parameters.
---
## Validate access controls
<!-- source: langgenius/dify | topic: Security | language: Python | updated: 2025-07-14 -->
Always implement comprehensive validation for access control mechanisms to prevent unauthorized access and privilege escalation. This includes validating user permissions, enforcing tenant isolation, and preventing bypass through direct API calls.
Key validation points:
1. **Tenant isolation**: Validate tenant_id early in methods to prevent cross-tenant access
2. **Input validation**: Add validation for user inputs that could bypass restrictions (e.g., email uniqueness checks)
3. **Privilege escalation prevention**: When handling IDs from untrusted sources, always include tenant context
4. **Proper HTTP responses**: Return 403 for authorization failures, not 404
Example implementation:
```python
def get_execution_by_id(self, execution_id: str, tenant_id: Optional[str] = None):
# When tenant_id is None, it's the caller's responsibility to ensure proper data isolation
# If execution_id comes from untrusted sources (e.g., API request),
# set tenant_id to prevent horizontal privilege escalation
if tenant_id is None:
# Log warning or require tenant_id for security
pass
# Early tenant validation
if dataset.tenant_id != user.current_tenant_id:
raise NoPermissionError("Access denied")
# Additional permission validation
if not self._has_permission(user, execution_id):
abort(403) # Not 404 - proper HTTP semantics
```
This approach ensures that security boundaries are maintained at multiple layers and prevents common authorization vulnerabilities.
---
## maintain code consistency
<!-- source: ggml-org/llama.cpp | topic: Code Style | language: Other | updated: 2025-07-14 -->
Always follow the existing code style patterns and conventions found in the surrounding codebase rather than introducing new formatting approaches. This includes matching brace placement, function organization, and parameter formatting styles.
For brace placement, match the surrounding code:
```cpp
// If surrounding code uses this style:
enum ggml_opt_optimizer {
GGML_OPT_ADAM = 0,
GGML_OPT_SGD = 1,
};
```
For function organization, group related declarations together:
```cpp
// Keep getters together
GGML_API ggml_opt_context_t ggml_opt_context_init(void);
GGML_API enum ggml_opt_optimizer_type ggml_opt_context_optimizer_type(ggml_opt_context_t);
GGML_API void ggml_opt_context_free(ggml_opt_context_t);
```
For long parameter lists, improve readability by placing each parameter on a new line:
```cpp
void diffusion_generate(
llama_context * ctx,
const llama_token * input_tokens,
llama_token * output_tokens,
int32_t n_input,
int32_t max_length,
struct diffusion_params params,
int32_t * n_generated);
```
When established patterns exist in the codebase, follow them even if they require some code duplication. Consistency across the codebase is more valuable than eliminating all duplication, as it makes the code more predictable and maintainable for all contributors.
---
## validate bounds before access
<!-- source: ggml-org/llama.cpp | topic: Null Handling | language: C++ | updated: 2025-07-14 -->
Always validate array indices, buffer sizes, and container bounds before accessing elements to prevent out-of-bounds errors and undefined behavior. This includes checking that indices are within valid ranges and that containers have sufficient size before element access.
Use explicit bounds checking with assertions or conditional statements before array/buffer operations:
```cpp
// Check array bounds before access
GGML_ASSERT(seq_id_src < seq_to_stream.size());
GGML_ASSERT(seq_id_dst < seq_to_stream.size());
const auto s0 = seq_to_stream[seq_id_src];
const auto s1 = seq_to_stream[seq_id_dst];
// Validate string length before character access
if (nextArg.empty() || (nextArg.size() >= 2 && nextArg[0] == '-' && !std::isdigit(nextArg[1]))) {
// Handle invalid case
}
// Use reserve() and push_back() instead of direct indexing for dynamic arrays
res.idxs.reserve(n_tokens);
// Use push_back() instead of res.idxs[i] = value
```
Consider using safer alternatives like the `.at()` method for containers, which throws exceptions on out-of-bounds access, or implement comprehensive bounds checking at function entry points. This practice prevents memory corruption, crashes, and security vulnerabilities that can arise from accessing invalid memory locations.
---
## measure algorithm performance impact
<!-- source: ggml-org/llama.cpp | topic: Algorithms | language: Other | updated: 2025-07-14 -->
Choose algorithms based on measurable performance benefits rather than theoretical complexity advantages. Start with simpler implementations and only adopt complex algorithms when they demonstrate clear performance improvements over basic approaches.
When evaluating algorithmic choices:
- Implement and measure simpler solutions first before adding complexity
- Defer advanced optimizations until basic performance targets are met
- Question whether complex algorithms provide measurable benefits over brute-force approaches
- Prefer deterministic algorithms using integer arithmetic over floating-point approaches when precision and consistency matter
For example, when considering the Aho-Corasick algorithm for multi-pattern string matching: "If we can show that the algorithm improves the performance in a measurable way, then it's ok. If not, we might want to fallback to some simpler brute-force approach." Similarly, when implementing matrix operations, focus on achieving good performance with scalar kernels before adding cooperative matrix optimizations.
This approach ensures that algorithmic complexity is justified by real performance gains rather than theoretical benefits, leading to more maintainable and predictably performing code.
---
## prioritize compile-time optimizations
<!-- source: ggml-org/llama.cpp | topic: Performance Optimization | language: Other | updated: 2025-07-13 -->
In performance-critical code, favor compile-time optimizations over runtime flexibility to enable better compiler optimizations and reduce execution overhead. This principle applies to several key areas:
**Use compile-time constants instead of runtime values:** Replace runtime function calls with compile-time constants when the values are known at compile time. For example, use `#define WG_M 16` instead of `get_local_size()` in GPU kernels, as compile-time constants allow the compiler to fully unroll loops and pre-calculate memory address offsets.
**Enable loop unrolling with compiler hints:** Add `[[unroll]]` annotations to loops with constant trip counts, especially in nested loop scenarios where the compiler might not automatically unroll:
```cpp
[[unroll]] for (uint j = 0; j < NUM_COLS; ++j) {
[[unroll]] for (uint n = 0; n < num_rows; ++n) {
temp[j][n] = subgroupClusteredAdd(temp[j][n], GROUP_SIZE);
}
}
```
**Eliminate branches through constant folding:** Structure code so the compiler can inline functions and fold constants, rather than using runtime boolean expressions in hot loops. Write separate functions for different code paths and let the compiler optimize each path independently.
**Prefer build-time compilation:** When possible, compile shaders and other performance-critical code at build time rather than runtime to avoid compilation overhead and enable better optimization opportunities.
These techniques are particularly important in GPU kernels, SIMD code, and other performance-critical paths where even small optimizations can yield significant performance improvements (10-17% gains are common).
---
## API minimalism principle
<!-- source: ggml-org/llama.cpp | topic: API | language: C++ | updated: 2025-07-12 -->
Keep APIs minimal by avoiding redundant interfaces and preferring simple solutions over complex ones. When adding functionality, first consider extending existing functions rather than creating new ones, especially when there's only a single use case. Remove unnecessary options and flags to maintain interface simplicity.
Key principles:
- Extend existing functions instead of adding new APIs when there's only one use case
- Remove redundant command-line flags and options that complicate the interface
- Use simple string operations instead of complex formatting when appropriate
- Keep core library APIs minimal - implement convenience wrappers in user code or examples
Example of extending existing API instead of adding new one:
```c
// Instead of adding llama_sampler_accept_str() for one use case:
// AVOID: llama_sampler_accept_str(gsmpl->grmr, trigger.c_str());
// Extend existing function to accept optional initial string:
gsmpl->grmr = llama_sampler_init_grammar(model, grammar_str, "root", trigger.c_str());
```
Example of interface simplification:
```cpp
// Instead of multiple preview flags:
// AVOID: --preview, --preview-count, --detokenize-preview
// Use single flag with sensible defaults:
// PREFER: --preview (shows fixed number of sequences as both tokens and text)
```
This principle ensures APIs remain maintainable, reduce cognitive load for users, and prevent feature bloat while still providing necessary functionality.
---
## Maintain consistent naming patterns
<!-- source: ggml-org/llama.cpp | topic: Naming Conventions | language: Python | updated: 2025-07-11 -->
Follow established naming conventions in the codebase and align with original source patterns while using modern, non-deprecated alternatives. Avoid creating confusing duplication in naming structures.
When adding new components, maintain consistency with existing patterns rather than creating new naming schemes. For example, if existing tensors use `blk.{bid}.component` format, continue that pattern rather than introducing `component.{bid}.component` which creates duplication.
Use modern language features and avoid deprecated imports. Replace deprecated typing constructs like `Dict` with built-in alternatives like `dict` or more appropriate types like `Mapping`.
When naming classes or components based on external models, align with the original naming conventions including version suffixes to maintain traceability and consistency.
Example:
```python
# Good: Consistent with existing pattern, avoids duplication
MODEL_TENSOR.SHORTCONV_CONV: "blk.{bid}.shortconv.conv"
# Avoid: Creates confusing duplication
MODEL_TENSOR.SHORTCONV_CONV: "shortconv.{bid}.shortconv.conv"
# Good: Modern typing
from typing import Callable, Iterator, dict
# Avoid: Deprecated typing
from typing import Dict
```
---
## Specify naming formats explicitly
<!-- source: ggml-org/llama.cpp | topic: Naming Conventions | language: Markdown | updated: 2025-07-11 -->
When defining naming conventions in documentation, specifications, or APIs, explicitly specify the exact format and patterns to be used. Avoid ambiguous descriptions that could lead to inconsistent implementations.
This prevents confusion about whether to use leading zeros, specific data types, or formatting patterns. Clear specifications ensure all developers implement naming consistently.
Examples of improvement:
Instead of:
```
training.format.version: string (e.g., "1.0") - Specification version
training.tensor.{index} (e.g., training.tensor.0, training.tensor.1, ...)
```
Specify explicitly:
```
training.format.version: uint32 (e.g., 10034 for v1.0.34) - Specification version
training.tensor.{index} (e.g., training.tensor.0, training.tensor.1, ...). No leading zeros.
```
This applies to version numbering schemes, identifier patterns, file naming conventions, and any other naming standards where ambiguity could lead to inconsistent implementation.
---
## Prefer lightweight observability integrations
<!-- source: BerriAI/litellm | topic: Observability | language: Markdown | updated: 2025-07-10 -->
Observability integrations should minimize external dependencies and avoid patching LiteLLM's core functions. Instead of requiring users to install additional SDKs, prefer lightweight approaches using custom loggers with direct HTTP calls (like httpx) or implement CustomBatchLogger interfaces. Avoid monkey-patching completion functions as this can cause unexpected errors and conflicts with other integrations.
The preferred pattern follows langsmith's approach - use pure httpx for sending telemetry data rather than requiring SDK installations. This reduces dependency bloat, prevents version conflicts, and maintains better isolation between different observability tools.
Example of preferred approach:
```python
# Good: Lightweight custom logger
class MyObservabilityLogger(CustomBatchLogger):
def __init__(self, api_key):
self.api_key = api_key
def log_success_event(self, kwargs, response_obj, start_time, end_time):
# Use httpx to send data directly
httpx.post("https://api.example.com/logs",
json={"request": kwargs, "response": response_obj})
# Avoid: Requiring SDK installation and patching
# weave.init() # This patches litellm functions
```
This approach ensures observability works reliably across all LiteLLM providers without introducing additional complexity or potential conflicts.
---
## implement or fail explicitly
<!-- source: ggml-org/llama.cpp | topic: Algorithms | language: Python | updated: 2025-07-10 -->
Algorithmic functions should either correctly implement their intended behavior or explicitly fail with an error, rather than silently returning potentially incorrect or unchanged results. This prevents subtle bugs where callers assume the algorithm was applied when it wasn't.
When implementing algorithmic functions, avoid "do nothing" fallback behavior that could mask errors. Instead, either:
1. Implement the complete algorithm for all supported input types
2. Throw an explicit error for unsupported cases
For example, in a quantization function:
```python
def quantize(data: np.ndarray, qtype: GGMLQuantizationType) -> np.ndarray:
if qtype == GGMLQuantizationType.F32:
return data.astype(np.float32, copy=False)
elif qtype == GGMLQuantizationType.F16:
return data.astype(np.float16, copy=False)
elif (q := _type_traits.get(qtype)) is not None:
return q.quantize(data)
# BAD: Silent no-op that defeats the function's purpose
# elif is_tmac_dtype(qtype):
# return data
# GOOD: Explicit error for unsupported types
else:
raise QuantError(f"Quantization not implemented for type {qtype}")
```
This principle also applies to detection algorithms - they should clearly indicate when they cannot provide meaningful results rather than producing potentially misleading output.
---
## validate configuration options explicitly
<!-- source: ggml-org/llama.cpp | topic: Configurations | language: Python | updated: 2025-07-09 -->
When handling configuration options or tool types, use explicit conditional statements with proper error handling for unknown values instead of implicit fallbacks or else clauses. This prevents silent failures and makes it clear which configurations are supported.
Use `elif` statements for each known configuration option and include an `assert False` or explicit error for unhandled cases:
```python
if self.tool == "llama-bench":
self.check_keys = set(LLAMA_BENCH_KEY_PROPERTIES + ["build_commit", "test_time", "avg_ts"])
elif self.tool == "test-backend-ops":
self.check_keys = set(TEST_BACKEND_OPS_KEY_PROPERTIES + ["build_commit", "test_time"])
else:
assert False, f"Unknown tool type: {self.tool}"
```
This approach helps catch configuration errors early, makes the supported options explicit in the code, and prevents issues like duplicate settings or silent overwrites that can occur with implicit handling. When adding new configuration options, developers must explicitly update the validation logic, ensuring all cases are properly considered.
---
## optimize code placement
<!-- source: ggml-org/llama.cpp | topic: Code Style | language: Objective-C | updated: 2025-07-09 -->
Ensure code is placed in its most logical and efficient location to improve maintainability and performance. This includes consolidating duplicate functionality into shared headers or utility files, and moving loop-invariant conditions outside of loops.
When you encounter duplicate functions across multiple files, consider moving them to a shared header as inline functions. For example, instead of having `ggml_are_same_layout` duplicated in `ggml-alloc.c` and `ggml-backend.cpp`, move it to `ggml.h` as an inline function.
Similarly, extract conditions that don't change within loops to outside the loop:
```c
// Instead of:
for (size_t i = 0, n = 3; i < n; ++i) {
if (op->src[i] != NULL && (op->src[i]->type == GGML_TYPE_BF16 || op->type == GGML_TYPE_BF16)) {
// ...
}
}
// Extract the loop-invariant condition:
bool op_is_bf16 = (op->type == GGML_TYPE_BF16);
for (size_t i = 0, n = 3; i < n; ++i) {
if (op->src[i] != NULL && (op->src[i]->type == GGML_TYPE_BF16 || op_is_bf16)) {
// ...
}
}
```
This approach reduces code duplication, improves performance by avoiding redundant checks, and enhances code readability by making the logic structure clearer.
---
## Catch specific exception types
<!-- source: vllm-project/vllm | topic: Error Handling | language: Python | updated: 2025-07-08 -->
Avoid using broad exception handlers like `except Exception:` or bare `except:`. Instead, catch specific exception types that you expect and can handle meaningfully. This makes code more maintainable by:
1. Preventing bugs from being silently masked
2. Making error handling intentions clear
3. Improving debugging by preserving stack traces
Example:
```python
# Bad - catches and hides all errors
try:
result = json.loads(args.compilation_config)
except Exception as e:
logger.error(f"Error: {e}")
return None
# Good - handles specific expected error
try:
result = json.loads(args.compilation_config)
except json.JSONDecodeError as e:
logger.error(f"Invalid JSON configuration: {e}")
raise ValueError(f"Configuration must be valid JSON: {e}") from e
```
When catching multiple exceptions, order them from most specific to most general. If you must catch a broad exception (e.g., for API boundaries), log the full exception details and consider re-raising or wrapping in a domain-specific error.
---
## Vectorize over Python loops
<!-- source: vllm-project/vllm | topic: Performance Optimization | language: Python | updated: 2025-07-08 -->
Replace Python loops and list comprehensions with vectorized operations when processing tensors or performing repeated computations. This optimization reduces interpreter overhead and enables efficient parallel execution on modern hardware.
Key practices:
1. Use torch.einsum/matmul for batched matrix operations
2. Leverage torch.gather for indexed tensor operations
3. Pre-compute and cache frequently accessed values
4. Process data in batches rather than element-by-element
Example - Converting nested loops to vectorized operations:
```python
# Before - Inefficient nested loops
def disentangled_attention_bias(query_layer, key_layer, relative_pos, rel_embeddings):
content_to_position = torch.zeros_like(content_to_content)
for i in range(seq_len):
q_i = query_layer[:, :, i, :]
for j in range(seq_len):
rel_idx = relative_pos[i, j].item()
content_to_position[:, :, i, j] = torch.einsum("bnh,rnh->bnr",
q_i, rel_k)[:, :, rel_idx]
# After - Vectorized implementation
def disentangled_attention_bias(query_layer, key_layer, relative_pos, rel_embeddings):
c2p_scores = torch.einsum("bnqh,rnh->bnqr", query_layer, rel_k)
rel_pos_expanded = relative_pos.unsqueeze(0).unsqueeze(0)
content_to_position = torch.gather(c2p_scores, 3, rel_pos_expanded)
```
This optimization can significantly improve performance by:
- Reducing Python interpreter overhead
- Enabling parallel execution on GPU/TPU
- Minimizing memory allocations
- Leveraging optimized CUDA kernels
---
## Follow logging best practices
<!-- source: vllm-project/vllm | topic: Logging | language: Python | updated: 2025-07-08 -->
Maintain high-quality logging throughout the codebase by following these best practices:
1. **Use appropriate log levels**: Reserve `debug` for detailed troubleshooting, `info` for general operational information, `warning` for potential issues, and `error`/`critical` for actual problems. Avoid excessive logging at higher levels.
```python
# For large object dumps or detailed diagnostics
logger.debug("Initialized config %s", config)
# Not: logger.info("Initialized config %s", config)
```
2. **Remove temporary debugging logs**: Debug statements with developer identifiers (e.g., `"[Kourosh]"`) should be removed before merging code.
3. **Optimize logging performance**: Guard expensive parameter computations to avoid unnecessary work when the log level is disabled:
```python
if logger.isEnabledFor(logging.DEBUG):
logger.debug("Complex calculation result: %s",
",".join(map(str, complex_calculation())))
```
4. **Avoid logging in loops** unless each iteration provides unique valuable information. Consider logging summaries before/after the loop instead.
5. **Use standard logging methods**: Use `logger.warning()` (not the deprecated `logger.warn()`), and use `logger.exception()` for exception logging to automatically include tracebacks:
```python
try:
# Some operation
except Exception:
logger.exception("Failed to perform operation")
# Not: logger.error("Failed: %s", str(e))
```
6. **Initialize loggers consistently**: Use the project's standard logger initialization pattern:
```python
from vllm.logger import init_logger
logger = init_logger(__name__)
```
7. **Make log messages clear and actionable**: Include relevant context and avoid ambiguous messages.
---
## Use self-documenting names
<!-- source: vllm-project/vllm | topic: Naming Conventions | language: Python | updated: 2025-07-08 -->
Variable, function, and parameter names should accurately reflect their purpose, behavior, and content. Choosing descriptive names improves code readability and reduces bugs caused by naming confusion.
When naming variables and parameters:
1. Use names that precisely describe what the entity represents
2. Maintain consistent naming across related components (config parameters, CLI args, classes)
3. Avoid misleading names that imply incorrect functionality or types
Examples of issues to avoid:
```python
# INCORRECT: Using "width" for height is misleading
img_height = image_processor.size.get("width", 224)
# CORRECT: Name accurately reflects the value
img_height = image_processor.size.get("height", 224)
# INCORRECT: CLI flag name doesn't match config parameter
scheduler_group.add_argument("--max-waiting-queue-length",
**scheduler_kwargs["limit_queue_length"])
# CORRECT: Consistent naming between CLI and config
scheduler_group.add_argument("--limit-queue-length",
**scheduler_kwargs["limit_queue_length"])
# INCORRECT: Name in __all__ doesn't match actual class name
__all__ = ["MoEConfig"] # But actual class is FusedMoEConfig
# CORRECT: Name in __all__ matches the actual implementation
__all__ = ["FusedMoEConfig"]
```
Descriptive and consistent naming serves as implicit documentation, making code more maintainable and reducing the likelihood of errors during development.
---
## Document AI model capabilities
<!-- source: vllm-project/vllm | topic: AI | language: Markdown | updated: 2025-07-08 -->
Provide clear, comprehensive documentation for AI model capabilities, especially for multimodal features and deployment scenarios. Include:
1. Explicit documentation of supported input types and formats
2. Working code examples for common deployment patterns
3. Clear parameter descriptions with their effects
Example for documenting multimodal capabilities:
```python
# Clear documentation of input types and formats
"""
The Score API supports:
- Text inputs: Plain text strings for standard NLP tasks
- Multimodal inputs:
- Images: Supported formats: PNG, JPEG
- Audio: Supported formats: WAV, MP3
Example usage:
"""
from vllm import LLM
llm = LLM(
model="example-multimodal-model",
# Document key parameters
tensor_parallel_size=4, # Uses 4 GPUs for parallel processing
max_model_len=2048, # Maximum sequence length
)
# Include complete, working example
outputs = llm.generate(
prompt="Describe this image",
multi_modal_data={
"image": image_data, # Numpy array or PIL Image
},
sampling_params=SamplingParams(
temperature=0.2,
max_tokens=64,
),
)
```
---
## Environment-aware configuration values
<!-- source: vllm-project/vllm | topic: Configurations | language: Other | updated: 2025-07-08 -->
When creating configuration files or defining configuration constants, ensure they properly adapt to different environments (Python versions, hardware architectures, etc.) with clear documentation. This prevents compatibility issues and helps other developers understand configuration choices.
For dependency specifications:
- Use conditional dependencies with appropriate version constraints
- Document why constraints exist with specific references
```python
# Example: Properly constrained dependency with explanation
mistral_common[opencv] >= 1.6.2; python_version<="3.12" # Not compatible with Python 3.13 (see issue #XYZ)
```
For compile-time configuration constants:
- Use available detection mechanisms rather than hardcoding values
- Provide appropriate fallbacks with clear structure
```c++
// Example: Architecture-aware constant definition
#if defined(USE_ROCM)
#if defined(__AMDGPU_WAVEFRONT_SIZE__)
// Using compiler-provided macro for wavefront size
#define WARP_SIZE __AMDGPU_WAVEFRONT_SIZE__
#elif defined(__GFX9__)
// Fallback for specific architecture
#define WARP_SIZE 64
#else
// Default fallback
#define WARP_SIZE 32
#endif
#else // CUDA
#define WARP_SIZE 32
#endif
```
Following these practices ensures configurations work correctly across diverse environments and remain maintainable as systems evolve.
---
## Eliminate code redundancy
<!-- source: vllm-project/vllm | topic: Code Style | language: Python | updated: 2025-07-07 -->
Keep your codebase maintainable by eliminating both unnecessary and duplicated code:
1. **Remove debugging artifacts**: Delete commented-out code, debug print statements, and unused imports before merging.
```python
# Remove this:
# print("kv_cache.shape = {}".format(kv_cache.shape))
# And this:
from vllm.model_executor.layers.activation import get_act_fn # unused
```
2. **Extract duplicated logic** into helper functions or shared classes:
```python
# Instead of this:
def save_configs(num_experts, ...):
if enable_expert_parallel:
filename = get_config_file_name(
local_num_experts, shard_intermediate_size // 2,
dtype_str, block_quant_shape)
else:
filename = get_config_file_name(
num_experts, shard_intermediate_size // 2,
dtype_str, block_quant_shape)
# Do this:
def save_configs(num_experts, ...):
num_experts_for_filename = (num_experts // ep_size
if enable_expert_parallel else num_experts)
filename = get_config_file_name(
num_experts_for_filename, shard_intermediate_size // 2,
dtype_str, block_quant_shape)
```
3. **DRY up similar functionality** when implementing related methods:
```python
# Instead of repeating logic across methods:
def calculate_metrics(...):
# Duplicated calculation logic
# Use intermediate variables or helper functions:
def _calculate_common_part(data):
# Shared logic extracted here
def calculate_metrics(...):
result = _calculate_common_part(data)
# Unique logic continues...
```
Eliminate redundancy in all forms - be it duplicate code blocks, methods, imports, or commented-out experimental code that's no longer needed. This makes your code more maintainable, reduces the chance of bugs when changes are needed, and improves readability for other developers.
---
## Document code thoroughly
<!-- source: vllm-project/vllm | topic: Documentation | language: Python | updated: 2025-07-07 -->
Always include comprehensive documentation for your code through both docstrings and explanatory comments.
For classes, functions, and methods, add docstrings that explain:
- The purpose and responsibility of the component
- Parameters with their types, meanings, and constraints
- Return values and their significance
- Special cases or exceptions
For non-obvious implementation details, add comments explaining:
- Why certain approach was chosen
- The reasoning behind workarounds
- Platform-specific behaviors
- Historical context for future maintainers
```python
def uniform_random_select_experts(
hidden_states: torch.Tensor,
router_logits: torch.Tensor,
top_k: int,
indices_type: Optional[torch.dtype] = None,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Selects experts randomly with uniform distribution instead of based on router scores.
Args:
hidden_states: Input tensor of shape [batch_size, sequence_length, hidden_size]
router_logits: Router scores from which only the shape is used
top_k: Number of experts to select for each token
indices_type: Optional dtype for the output indices tensor
Returns:
Tuple containing (routing_weights, expert_indices)
"""
# CPU only supports V1 architecture due to specialized optimizations
# that aren't available in the regular implementation
if current_platform.is_cpu() and os.environ.get("VLLM_USE_V1", "0") == "0":
pytest.skip("CPU only supports V1")
```
Well-documented code improves maintainability, enables easier onboarding of new team members, and reduces the time needed to understand and modify existing functionality.
---
## Avoid magic numbers
<!-- source: vllm-project/vllm | topic: AI | language: Python | updated: 2025-07-07 -->
Replace hardcoded values and magic numbers in AI model code with named constants or configuration parameters. This improves maintainability, reduces errors, and makes code more adaptable to different model architectures.
Common issues include hardcoded head dimensions, tensor shapes, and data paths:
```python
# Problematic: Hardcoded head dimensions
prefill_wrapper.plan(
qo_indptr,
kv_indptr,
num_qo_heads,
num_kv_heads,
192, # Magic number for head_dim_qk
causal=True,
head_dim_vo=128, # Another magic number
)
# Better: Use variables from configuration
prefill_wrapper.plan(
qo_indptr,
kv_indptr,
num_qo_heads,
num_kv_heads,
head_dim_qk,
causal=True,
head_dim_vo=self.kv_cache_spec.head_size,
)
```
For model parameters, always derive values from configuration objects rather than embedding assumptions in code. When testing, avoid hardcoded paths and dimensions that may not be portable across environments. In multimodal models, be especially careful with tokenization and embedding logic, which should adapt to the model architecture rather than assuming specific token structures.
---
## Check before access
<!-- source: vllm-project/vllm | topic: Null Handling | language: Python | updated: 2025-07-07 -->
Always verify that objects, attributes, and variables are not None before accessing their properties, calling their methods, or using them in operations. This prevents common runtime errors like TypeError, AttributeError, or UnboundLocalError when working with optional values.
For optional attributes or dictionary keys:
```python
# Before - may raise AttributeError or TypeError if token_type_ids is None
token_type_embeddings = self.token_type_embeddings(token_type_ids)
# After - safely checks before access
if self.token_type_ids is not None:
model_kwargs["token_type_ids"] = cast(torch.Tensor, self.token_type_ids)[:num_scheduled_tokens]
```
For conditional variables:
```python
# Before - may cause UnboundLocalError if condition is false
if is_v1_kv_transfer_group():
invalid_block_ids = connector.get_block_ids_with_load_errors()
# Later code uses invalid_block_ids regardless of condition
# After - initialize before conditional assignment
invalid_block_ids: Optional[set[int]] = None
if is_v1_kv_transfer_group():
invalid_block_ids = connector.get_block_ids_with_load_errors()
```
For optional containers:
```python
# Before - raises TypeError if finished_sending is None
for req_id in kv_connector_metadata.finished_sending:
self._done_sending_count[req_id] += 1
# After - safely handles None case
for req_id in kv_connector_metadata.finished_sending or []:
self._done_sending_count[req_id] += 1
```
Always match return values with declared types:
```python
# Before - implicitly returns None instead of declared tuple type
if not model_runner_output.kv_connector_metadata:
return
# After - explicitly returns correct type
if not model_runner_output.kv_connector_metadata:
return None, None
```
---
## Process configurations consistently
<!-- source: vllm-project/vllm | topic: Configurations | language: Python | updated: 2025-07-07 -->
Ensure that configuration data is processed consistently and correctly throughout the codebase. This includes:
1. **Use proper serialization methods** for configuration dictionaries. Prefer `json.dumps()` over `str()` when converting dictionaries to strings:
```python
# Wrong: Using str() produces invalid JSON
if isinstance(self.ep_config, dict):
self.ep_config = EPConfig.from_cli(str(self.ep_config))
# Correct: Use json.dumps() for proper serialization
if isinstance(self.ep_config, dict):
self.ep_config = EPConfig.from_cli(json.dumps(self.ep_config))
```
2. **Process configurations consistently in all code paths**. Don't skip processing steps conditionally unless absolutely necessary:
```python
# Wrong: Only calling adapt_config_dict in one branch
if max_position_embeddings is None:
max_position_embeddings = _maybe_retrieve_max_pos_from_hf()
config_dict["max_position_embeddings"] = max_position_embeddings
config = adapt_config_dict(config_dict)
# Correct: Call processing function in all cases
if max_position_embeddings is None:
max_position_embeddings = _maybe_retrieve_max_pos_from_hf()
config_dict["max_position_embeddings"] = max_position_embeddings
config = adapt_config_dict(config_dict)
```
3. **Handle nested configurations recursively** when transforming configuration structures:
```python
# Process nested dictionaries recursively
def _recursive_remap(elem: Any) -> Any:
if not isinstance(elem, dict):
return elem
new_dict = {}
for key, value in elem.items():
new_key = config_mapping.get(key, key)
new_dict[new_key] = _recursive_remap(value)
return new_dict
```
4. **Verify that default values are sensible** and documented correctly in docstrings, ensuring the implementation matches the documentation.
5. **Avoid changing default configurations** that can break backward compatibility. When defaults must change, document the change and provide a migration path.
---
## Stable documentation links
<!-- source: vllm-project/vllm | topic: Documentation | language: Markdown | updated: 2025-07-07 -->
Ensure all documentation links are stable, consistent, and correctly targeted to improve navigation and long-term maintainability:
1. **Use stable references**: Always link to stable release tags or specific commit hashes rather than volatile branches like `master` or `main`:
```diff
- kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/refs/heads/master/ray-operator/config/samples/vllm/ray-service.vllm.yaml
+ kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/vX.Y.Z/ray-operator/config/samples/vllm/ray-service.vllm.yaml
```
2. **Maintain consistent file extensions**: Use consistent file extensions (.md or .html) throughout documentation links, preferring `.md` for markdown source files:
```diff
- See also: [full example](../examples/online_serving/structured_outputs.html)
+ See also: [full example](../examples/online_serving/structured_outputs.md)
```
3. **Verify relative paths**: Ensure relative paths resolve correctly within the documentation structure by testing links in the built documentation.
4. **Use standardized link syntax**: Follow project conventions for link formatting, such as using autolink syntax:
```diff
- While ongoing efforts like [#17419](https://github.com/vllm-project/vllm/issues/17419)
+ While ongoing efforts like <gh-issue:17419>
```
5. **Target specific sections**: When linking to pages with tabs or complex navigation, link directly to the specific section or tab to improve user experience.
---
## Preserve API compatibility
<!-- source: vllm-project/vllm | topic: API | language: Python | updated: 2025-07-07 -->
When modifying API interfaces, parameters, or argument behavior, ensure backward compatibility is maintained to prevent breaking existing user code. If a parameter needs to be renamed or enhanced, continue supporting the old parameter name alongside the new one.
For example, when expanding API parameters to support new functionality:
```python
# GOOD: Maintain backward compatibility
class ScoreRequest(OpenAIBaseModel):
model: Optional[str] = None
# Keep old parameter names to maintain compatibility
text_1: Optional[Union[list[str], str]] = None
text_2: Optional[Union[list[str], str]] = None
# Add new parameters with enhanced functionality
data_1: Optional[Union[list[str], str, ScoreMultiModalParam]] = None
data_2: Optional[Union[list[str], str, ScoreMultiModalParam]] = None
def __post_init__(self):
# Use old parameters if new ones aren't provided
if self.data_1 is None and self.text_1 is not None:
self.data_1 = self.text_1
if self.data_2 is None and self.text_2 is not None:
self.data_2 = self.text_2
```
When removing or changing interface methods, first mark them as deprecated and provide clear migration guidance. For public APIs, follow a formal deprecation policy with adequate notice before removal. For internal APIs with multiple consumers, coordinate changes to ensure dependent systems aren't broken.
Always document your API changes thoroughly, including:
1. What changed and why
2. How existing users should migrate their code
3. When deprecated features will be removed (if applicable)
---
## Validate algorithmic operations carefully
<!-- source: vllm-project/vllm | topic: Algorithms | language: Python | updated: 2025-07-07 -->
Mathematical and logical operations require careful validation to ensure correctness. Common issues include:
1. Operator precedence in complex conditions
2. Incorrect scaling/normalization calculations
3. Type mismatches in comparisons
4. Edge case handling
Example of a problematic implementation:
```python
def _is_fp8_w8a8_sm90_or_sm100(weight_quant, input_quant):
return (self._check_scheme_supported(90) or
self._check_scheme_supported(100) and
self._is_fp8_w8a8(weight_quant, input_quant))
```
Corrected version with proper operator precedence:
```python
def _is_fp8_w8a8_sm90_or_sm100(weight_quant, input_quant):
return ((self._check_scheme_supported(90) or
self._check_scheme_supported(100)) and
self._is_fp8_w8a8(weight_quant, input_quant))
```
Key validation practices:
- Use parentheses to make operator precedence explicit
- Validate numerical operations with edge cases (zero, negative, overflow)
- Ensure type consistency in comparisons
- Add assertions or validation checks for critical assumptions
- Test with boundary conditions and extreme values
---
## explicit performance handling
<!-- source: ggml-org/llama.cpp | topic: Performance Optimization | language: Python | updated: 2025-07-07 -->
When writing performance benchmarking and measurement code, always be explicit about all possible cases and organize related constants clearly. Avoid catch-all else clauses that could mask bugs or unexpected values in performance-critical code paths.
This approach prevents silent failures in benchmark parsing and makes performance code more maintainable by clearly documenting all expected scenarios.
Example of explicit case handling:
```python
# Instead of a catch-all else clause
if unit == "TFLOPS":
gflops = value * 1000
elif unit == "MFLOPS":
gflops = value / 1000
elif unit == "GFLOPS":
gflops = value
else:
assert False # Explicit failure for unexpected units
```
Example of organized performance constants:
```python
# Clearly separate constants by benchmark type
LLAMA_BENCH_BOOL_PROPERTIES = ["embeddings", "cpu_strict", "use_mmap", "no_kv_offload", "flash_attn"]
TEST_BACKEND_OPS_BOOL_PROPERTIES = ["supported", "passed"]
```
---
## Combine identical CSS
<!-- source: vllm-project/vllm | topic: Code Style | language: Css | updated: 2025-07-07 -->
When multiple CSS selectors share identical styling properties, combine them using comma separation rather than duplicating the same rules. This reduces code duplication, improves maintainability, and makes stylesheets more concise.
Example - Instead of this:
```css
.md-typeset .admonition.code,
.md-typeset details.code {
border-color: #64dd17
}
.md-typeset .admonition.console,
.md-typeset details.console {
border-color: #64dd17
}
```
Do this:
```css
.md-typeset .admonition.code,
.md-typeset details.code,
.md-typeset .admonition.console,
.md-typeset details.console {
border-color: #64dd17;
}
```
This practice applies to all CSS rules with identical properties and values. By consolidating duplicate rules, you'll create more maintainable stylesheets that are easier to update and less prone to inconsistencies when making changes.
---
## Optimize GPU execution
<!-- source: vllm-project/vllm | topic: Performance Optimization | language: CUDA | updated: 2025-07-07 -->
Ensure GPU code is optimized for both proper thread utilization and correct architecture dispatching:
1. **Maximize thread parallelism** - Design CUDA kernels to fully utilize available threads. When appropriate, use multi-dimensional grid configurations to parallelize across all relevant dimensions of your problem.
```cuda
// Instead of TODO comments like:
// TODO utilize more CUDA threads
// this will probably need some extra padding for warps
// Consider implementing a 2D grid approach:
dim3 block_dim(32, 8); // Thread block dimensions
dim3 grid_dim((n + block_dim.x - 1) / block_dim.x,
(padded_m + block_dim.y - 1) / block_dim.y);
kernel<<<grid_dim, block_dim>>>(...);
```
2. **Implement proper architecture dispatching** - When supporting multiple GPU architectures, combine compile-time preprocessor directives with runtime architecture detection:
```cuda
// Instead of compile-time only checks:
#if defined ENABLE_CUTLASS_MOE_SM100 && ENABLE_CUTLASS_MOE_SM100
if (version_num >= 100) { // Use runtime version check
cutlass_moe_mm_sm100(out_tensors, a_tensors, b_tensors, a_scales, b_scales,
expert_offsets, problem_sizes, a_strides, b_strides,
c_strides, per_act_token, per_out_ch);
return;
}
#endif
```
These approaches prevent performance bottlenecks and ensure code works correctly across different GPU hardware generations.
---
## Secure before deployment
<!-- source: vllm-project/vllm | topic: Security | language: Python | updated: 2025-07-05 -->
Remove or secure development-specific code before deploying to production environments. Development artifacts like debug print statements, weak validation methods, and development endpoints can introduce significant security vulnerabilities.
Specifically:
1. Remove all debugging print statements that might leak sensitive request data, credentials, or tokens
2. Implement thorough validation for security-critical functions, considering bypass techniques (e.g., symbolic links in command validation)
3. Ensure development features have proper warnings and automated tests to verify they're not accidentally enabled in production
Example of insecure code:
```python
@router.post("/v1/responses")
async def create_responses(request: ResponsesRequest, raw_request: Request):
print(request, raw_request) # INSECURE: Leaks sensitive data to logs
def is_dangerous_cmd(cmd):
cmd_base = os.path.basename(cmd) # INSECURE: Can be bypassed with symlinks
return cmd_base in COMMAND_BLACKLIST
```
Example of secure code:
```python
@router.post("/v1/responses")
async def create_responses(request: ResponsesRequest, raw_request: Request):
# Debug statements removed for production
def is_dangerous_cmd(cmd):
# Resolve any symlinks to get the real path
real_path = os.path.realpath(cmd)
cmd_base = os.path.basename(real_path)
return cmd_base in COMMAND_BLACKLIST
# When enabling development features:
if envs.VLLM_SERVER_DEV_MODE:
logger.warning("SECURITY WARNING: Development endpoints are enabled!")
# Accompany with tests to verify this warning is present
```
---
## Remove unnecessary code elements
<!-- source: vllm-project/vllm | topic: Code Style | language: CUDA | updated: 2025-07-03 -->
Keep code clean and maintainable by removing unnecessary elements that add complexity without value:
1. Remove unused header includes that increase compile times
2. Delete commented-out code blocks - use version control to track alternatives
3. Eliminate redundant checks and validations
4. Clean up duplicate definitions
Example of code to avoid:
```cpp
#include <iostream> // Unnecessary include
// Commented out alternative implementation
// void alternativeFunction() {
// ...
// }
// Redundant checks
TORCH_CHECK(a_tensors.dtype() == torch::kFloat8_e4m3fn,
"A tensors must be of type float8_e4m3fn.");
TORCH_CHECK(a_tensors.dtype() == torch::kFloat8_e4m3fn); // Duplicate check
```
Instead, keep only the essential, active code elements and rely on version control for tracking alternatives.
---
## Standardize environment versions
<!-- source: BerriAI/litellm | topic: CI/CD | language: Yaml | updated: 2025-07-03 -->
Ensure consistent versions of languages, tools, and dependencies across all CI/CD environments (GitHub Actions, CircleCI, local development, etc.). Inconsistent versions between environments make it difficult to reproduce issues and debug failures, as seen when using "Python 3.12 in GitHub & 3.9 elsewhere - makes it hard to figure out how to fix the broken unit tests."
Define a single source of truth for version specifications and reference it across all pipeline configurations. This includes:
- Runtime versions (Python, Node.js, etc.)
- Tool versions (Docker, kubectl, etc.)
- Action/orb versions in CI systems
Example approach:
```yaml
# .tool-versions or similar
python 3.12.0
node 18.17.0
docker 24.0.0
# Reference in GitHub Actions
- uses: actions/setup-python@v4
with:
python-version: '3.12.0'
# Reference in CircleCI
- image: cimg/python:3.12.0
```
This prevents environment drift, reduces debugging complexity, and ensures consistent behavior across all stages of your CI/CD pipeline.
---
## Match reference names
<!-- source: vllm-project/vllm | topic: Naming Conventions | language: Other | updated: 2025-07-03 -->
{% raw %}
Ensure that filenames and paths referenced in scripts, commands, or configuration files exactly match the actual names of the files they reference. Name mismatches between references and actual files can cause commands to fail during execution.
For example, in a Justfile, the reference:
```
python {{vllm-directory}}benchmarks/benchmark_one_concurrent_req.py
```
should be changed to:
```
python {{vllm-directory}}benchmarks/benchmark_one_concurrent.py
```
to match the actual filename.
Always verify references against the actual filesystem structure before committing code to prevent runtime failures. This is especially important in build scripts, makefiles, and configuration files where a simple typo can break the entire workflow.
{% endraw %}
---
## Check before using values
<!-- source: LMCache/LMCache | topic: Null Handling | language: Python | updated: 2025-07-02 -->
Always validate that values are not null, undefined, or empty before using them in operations that depend on their content. This prevents downstream errors and unexpected behavior when collections are empty or strings are null.
When working with collections, check if they contain elements before proceeding with operations that assume non-empty data:
```python
# Before: Risk of empty collection causing issues downstream
for item in some_list:
keys.append(item.key)
memory_objs.append(item.memory_obj)
# subsequent logic assumes memory_objs is not empty
# After: Check collection before proceeding
if memory_objs: # handles both None and empty cases
# execute subsequent logic
```
For string values, use concise truthiness checks that handle both null and empty cases:
```python
# Before: Only checks for null
if chunk_message is not None:
server_message.append(chunk_message)
# After: Handles both null and empty
if chunk_message: # handles None, "", and other falsy values
server_message.append(chunk_message)
```
This pattern prevents runtime errors and makes code more robust by ensuring operations only proceed when they have valid data to work with.
---
## Enable callback chaining
<!-- source: ggml-org/llama.cpp | topic: API | language: Other | updated: 2025-07-01 -->
When designing APIs that accept callbacks or configuration functions, always return the previous callback/configuration to enable chaining and composition. This allows multiple components to layer their functionality without losing existing behavior.
For callback setters, return the previous callback so callers can chain multiple handlers:
```c
// Good: Returns previous callback for chaining
GGML_API ggml_abort_callback_t ggml_set_abort_callback(ggml_abort_callback_t callback);
// Usage allows chaining:
ggml_abort_callback_t prev_callback = ggml_set_abort_callback(my_callback);
// Can later restore or chain: my_callback can call prev_callback
```
For initialization functions, prefer passing callbacks as parameters rather than extending the API with many specific functions:
```c
// Preferred: Callback-based approach
llama_sampler_init_grammar(grammar_str, grammar_root, is_empty_callback, accept_str_callback);
// Avoid: Function proliferation
llama_sampler_is_grammar_empty(gsmpl); // Separate function for each operation
```
This approach promotes API composability, reduces function proliferation, and enables multiple components to work together seamlessly.
---
## AI model persistence
<!-- source: vllm-project/vllm | topic: AI | language: Other | updated: 2025-06-28 -->
When containerizing AI applications, ensure proper model persistence by mounting volumes to the default cache locations used by AI frameworks and setting appropriate environment variables. This prevents redundant downloads of large models and improves development efficiency.
Example for Hugging Face models in Docker:
```yaml
volumes:
- models:/root/.cache/huggingface
environment:
HF_HOME: /root/.cache/huggingface
```
---
## Follow consistent naming patterns
<!-- source: BerriAI/litellm | topic: Naming Conventions | language: Python | updated: 2025-06-27 -->
Maintain consistent naming conventions across all aspects of the codebase, including file structures, folder organization, environment variables, and identifiers. This ensures predictability and reduces cognitive load for developers.
Key principles:
- **File/folder structure**: Match provider names exactly (e.g., `llms/nvidia/` not `llms/meta/` for `meta-llama`)
- **Test organization**: Mirror source structure (`tests/litellm/integrations/SlackAlerting/test_slack_alerting.py` matches `litellm/integrations/SlackAlerting/`)
- **Environment variables**: Follow framework conventions (`*_API_BASE` for litellm, while supporting `*_BASE_URL` for cross-framework compatibility)
- **Naming brevity**: Use concise but clear names (`prisma_airs` instead of `panw_prisma_airs`)
- **External API alignment**: Match external service naming conventions (`space_id` vs `space_key` based on API requirements)
Example:
```python
# Good: Consistent with framework patterns
NVIDIA_API_BASE = os.getenv("NVIDIA_API_BASE") or os.getenv("NVIDIA_BASE_URL") # Support both
# Good: File structure matches provider
# litellm/llms/nvidia/chat/transformation.py
# tests/litellm/llms/nvidia/chat/test_transformation.py
# Good: Concise enum naming
class SupportedGuardrailIntegrations(Enum):
PRISMA_AIRS = "prisma_airs" # Not "panw_prisma_airs"
```
---
## optimize algorithmic complexity
<!-- source: ggml-org/llama.cpp | topic: Algorithms | language: C | updated: 2025-06-27 -->
When implementing or refactoring algorithms, prioritize efficient data structures and computational approaches over simpler but slower alternatives. This is especially critical in performance-sensitive code where operations may be called frequently.
Key considerations:
- Replace linear searches with hash table lookups when dealing with node/element lookups
- Use platform-specific optimizations (like ARM NEON vectorization) when available, with proper feature detection
- Preallocate resources and precompute operations when possible to avoid runtime overhead
- Consider the computational complexity of your approach and choose algorithms that scale well
Example from the codebase:
```c
// Instead of linear search through nodes:
int j = cgraph->n_nodes - 1;
for (; j >= 0; --j) {
if (s == cgraph->nodes[j]) {
break;
}
}
// Use hash table lookup:
// Allocate use_counts array with hash_size and use ggml_hash_find
// to find the slot for the node, avoiding O(n) search complexity
```
This principle applies broadly - from choosing vectorized implementations over scalar ones, to using hash tables instead of linear searches, to preallocating command buffers rather than creating them on-demand. The goal is to minimize computational overhead, especially in hot code paths where these optimizations compound significantly.
---
## Protect shared state
<!-- source: vllm-project/vllm | topic: Concurrency | language: Python | updated: 2025-06-27 -->
When mutable state (like dictionaries, lists, or counters) is accessed from multiple threads or concurrent tasks, use appropriate synchronization mechanisms to prevent race conditions. Unprotected access to shared state can lead to data corruption, inconsistent behavior, and hard-to-debug issues.
For example, wrap dictionary operations with locks:
```python
# Unsafe - race condition if called concurrently:
self._reqs_to_send[req_id] = some_value
del self._reqs_to_send[req_id]
# Safe with lock protection:
self._reqs_to_send_lock = threading.Lock() # Initialize in constructor
# Later in code:
with self._reqs_to_send_lock:
self._reqs_to_send[req_id] = some_value
del self._reqs_to_send[req_id]
```
In asynchronous code, also be careful with shared variables that might be modified by interleaved tasks. Instead of relying on shared context, pass necessary context through the call stack:
```python
# Unsafe with shared state in async code:
self._set_tokenizer(tokenizer) # Could be modified by another task before use
# Safe - pass explicitly through call stack:
result = await self._process_with_tokenizer(tokenizer, input_text)
```
---
## Follow documentation standards
<!-- source: BerriAI/litellm | topic: Documentation | language: Markdown | updated: 2025-06-25 -->
Ensure all documentation follows established patterns and standards for components, formatting, and code examples. This includes using proper imports for documentation components (like Image imports instead of markdown syntax) and maintaining consistent formatting for code samples with standardized attributes.
For images, use the proper component import:
```javascript
import Image from '@theme/IdealImage';
// Use: <Image src="path/to/image.png" alt="Description" />
// Instead of: 
```
For code examples, include standard formatting attributes:
```python
# Code samples should have showLineNumbers and title
```
Always reference existing documentation examples as templates to maintain consistency across the codebase. This ensures a professional appearance and better user experience while preventing technical rendering issues.
---
## Optimize tensor memory operations
<!-- source: vllm-project/vllm | topic: Pytorch | language: Python | updated: 2025-06-25 -->
When working with PyTorch tensors, use memory-efficient operations that avoid unnecessary copies. Specify memory formats directly during tensor creation instead of applying operations like `.t().contiguous()`. For C++/CUDA kernel interfacing, use `.data_ptr()` instead of `.view(dtype)` to ensure safe memory access and maintain compatibility with future PyTorch versions.
```python
# Inefficient approach with unnecessary copy:
tensor = torch.empty((n, m), device="cuda", dtype=torch.bfloat16).t().contiguous()
# Efficient approach:
tensor = torch.empty((n, m), device="cuda", dtype=torch.bfloat16,
memory_format=torch.contiguous_format).t()
# Unsafe C++ kernel interfacing:
cutlass_function(w1_scale.view(torch.int32), w1.view(torch.long))
# Safe approach with explicit pointer access:
cutlass_function(w1_scale.data_ptr(), w1.data_ptr())
```
---
## optimize CI/CD pipelines
<!-- source: LMCache/LMCache | topic: CI/CD | language: Shell | updated: 2025-06-24 -->
{% raw %}
CI/CD pipelines should be optimized for both performance and maintainability through strategic caching, script consolidation, and compilation optimization. This involves three key practices:
1. **Implement proper caching strategies**: Use comprehensive cache keys that include build identifiers and dependency checksums to ensure cache invalidation works correctly. For environment caching, include unique build IDs to prevent stale cache reuse.
2. **Consolidate related scripts**: Minimize the number of separate bash scripts by merging related functionality. This reduces maintenance overhead and simplifies pipeline complexity as the project grows.
3. **Optimize compilation performance**: Implement compilation caching (like ccache) for projects with significant build times to improve pipeline speed.
Example cache configuration:
```yaml
plugins:
- cache#v1.5.2:
key: "venv-{{ BUILDKITE_BUILD_ID }}-{{ checksum \"requirements/common.txt\" }}-{{ checksum \"requirements/test.txt\" }}"
path: ".venv"
save: "pipeline"
force: true
```
These optimizations ensure CI/CD pipelines remain fast, reliable, and maintainable as projects scale.
{% endraw %}
---
## consistent localhost addressing
<!-- source: menloresearch/jan | topic: Networking | language: TypeScript | updated: 2025-06-24 -->
Use 127.0.0.1 instead of 'localhost' for network requests and implement broad localhost detection rather than specific port-based checks. This ensures consistent behavior across different environments and avoids issues with DNS resolution or host file configurations.
For network requests, prefer the loopback IP address:
```typescript
// Preferred
await fetch(`http://127.0.0.1:${cortexJsPort}/v1/system`, {
// Avoid
await fetch('http://localhost:1337/v1/system', {
```
For localhost detection, check the hostname broadly rather than enumerating specific ports:
```typescript
const isLocalHost = urlObj.hostname === 'localhost' ||
urlObj.hostname === '127.0.0.1' ||
urlObj.hostname === '0.0.0.0'
```
This approach is more reliable and covers all local providers rather than maintaining a list of specific ports that may become outdated as new local AI providers emerge.
---
## Ensure configuration consistency
<!-- source: langgenius/dify | topic: Configurations | language: Other | updated: 2025-06-18 -->
Maintain consistency across configuration variables to prevent setup confusion and deployment issues. Avoid duplicate or conflicting environment variables, ensure related configuration values align, and use appropriate defaults for the deployment context.
Key practices:
- Align related configuration variables (e.g., passwords, API keys) across different sections
- Avoid duplicate environment variables with different names for the same purpose
- Use context-appropriate defaults (e.g., reasonable token expiry times)
- Ensure configuration variables in example files match their intended usage
Example of inconsistent configuration to avoid:
```bash
# Bad: Misaligned passwords cause login issues
OPENSEARCH_PASSWORD=admin
OPENSEARCH_INITIAL_ADMIN_PASSWORD=Qazwsxedc!@#123
# Bad: Duplicate variables for same purpose
PLUGIN_DAEMON_KEY=lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi
PLUGIN_API_KEY=lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi
```
This prevents initial setup difficulties and reduces confusion during development and deployment.
---
## Maintain consistent organization
<!-- source: langgenius/dify | topic: Code Style | language: TSX | updated: 2025-06-18 -->
Ensure proper file organization and consistent import conventions throughout the codebase. Assets should be placed in designated folders, utility functions should be moved to appropriate shared locations, and import paths should follow established patterns.
Key practices:
- Move assets (images, fonts, etc.) to proper asset directories instead of keeping them in component folders
- Extract utility functions to shared locations like `utils/` directories rather than embedding them in component files
- Use consistent import path conventions (e.g., always use `@/app/` prefix for internal imports)
- Organize code by logical groupings and maintain clear separation of concerns
Example of proper organization:
```typescript
// Instead of: bg-[url('~@/app/components/tools/add-tool-modal/empty.png')]
// Use: bg-[url('/assets/empty.png')]
// Instead of: import useAvailableVarList from '../../_base/hooks/use-available-var-list'
// Use: import useAvailableVarList from '@/app/components/workflow/nodes/_base/hooks/use-available-var-list'
// Move utility functions from component files to shared locations:
// From: web/app/components/base/chat/embedded-chatbot/hooks.tsx
// To: web/app/components/base/chat/utils/index.ts
```
This approach improves code maintainability, makes dependencies clearer, and ensures consistent project structure across the team.
---
## Use environment variables
<!-- source: ggml-org/llama.cpp | topic: Configurations | language: Txt | updated: 2025-06-18 -->
Prefer environment variables over hardcoded paths and configuration values in build scripts and configuration files. This approach improves portability, flexibility, and maintainability by allowing different environments to specify their own paths without modifying the codebase.
Instead of hardcoding paths like:
```cmake
set(oneCCL_DIR "/opt/intel/oneapi/ccl/latest/lib/cmake/oneCCL")
set(MPI_INCLUDE_PATH "/opt/intel/oneapi/mpi/latest/include")
```
Use environment variables:
```cmake
set(oneCCL_DIR "$ENV{ONEAPI_ROOT}/ccl/latest/lib/cmake/oneCCL")
set(MPI_INCLUDE_PATH "$ENV{ONEAPI_ROOT}/mpi/latest/include")
```
For debug options and feature flags, check environment variables instead of adding compile-time options. This allows runtime configuration without rebuilding and follows established patterns like `GGML_SCHED_DEBUG`. Environment variables make configurations more flexible across different deployment environments and reduce the need for environment-specific build modifications.
---
## Maintain comprehensive error handling
<!-- source: LMCache/LMCache | topic: Error Handling | language: Python | updated: 2025-06-17 -->
Always preserve and implement thorough error handling that includes proper resource cleanup, meaningful logging, and appropriate return values. Don't remove existing error handling during code refactoring or simplification.
Key practices:
1. **Preserve error handling during refactoring** - When simplifying code, maintain existing try-catch blocks and error recovery logic
2. **Clean up resources on failure** - Always free allocated memory, close connections, or reset state when operations fail
3. **Choose appropriate error responses** - Return sensible defaults (like False) instead of throwing exceptions when callers can handle the failure gracefully
4. **Log exceptions with context** - Include meaningful error messages and context when catching exceptions
Example of proper error handling preservation:
```python
# Good - maintains error handling even when simplifying
try:
return self.connection.exists_sync(key)
except Exception as e:
with self.lock:
self.connection = None
self.failure_time = time.time()
logger.warning(f"Remote connection failed in contains: {e}")
logger.warning("Returning False")
return False
# Bad - removes error handling during simplification
return self.connection.exists_sync(key)
```
This approach prevents silent failures, resource leaks, and provides better debugging information when issues occur.
---
## Use OAuth Authorization Header
<!-- source: infiniflow/ragflow | topic: Security | language: Markdown | updated: 2025-06-16 -->
When authenticating requests with bearer tokens or OAuth-derived credentials, transmit them using the standards-based `Authorization` request header as required by the relevant OAuth specification. Avoid sending secrets via custom headers like `api_key`, and ensure documentation/examples match the OAuth flow requirements.
Example (adapted to a typical async client):
```python
# Preferred: OAuth-compliant Authorization header
headers = {
"Authorization": "Bearer YOUR_ACCESS_TOKEN" # or the OAuth-defined scheme/value
}
async with sse_client(
"http://localhost:9382/sse",
headers=headers,
) as streams:
# Rest of your code...
```
Apply this to code, SDK wrappers, and docs: if the auth method is OAuth 2.1 (or similar), use `Authorization` and the required formatting/scheme, rather than custom credential headers.
---
## semantic configuration validation
<!-- source: ggml-org/llama.cpp | topic: Configurations | language: Other | updated: 2025-06-14 -->
When designing configuration systems, prefer semantic parameters over generic key-value approaches for common features, and implement runtime validation to detect incompatible configuration combinations.
For common functionality, use dedicated semantic variables rather than generic context variables. This maintains a unified API surface and prevents users from needing to learn template-specific idiosyncrasies. When features are incompatible, validate configurations at runtime and provide clear error messages.
Example:
```cpp
// Good: Semantic parameter for common feature
struct config {
bool enable_thinking = true;
bool assistant_prefill = false;
};
// Validation with clear error message
if ((!inputs.enable_thinking) || inputs.chat_template_kwargs.find("enable_thinking") != inputs.chat_template_kwargs.end()) {
throw std::runtime_error("Assistant response prefill is incompatible with enable_thinking.");
}
```
This approach reduces complexity for users while maintaining flexibility for advanced use cases, and prevents runtime failures through proactive validation.
---
## background task coordination
<!-- source: BerriAI/litellm | topic: Concurrency | language: Python | updated: 2025-06-13 -->
When working with background tasks in async applications, ensure proper task lifecycle management and coordination to prevent memory leaks, race conditions, and resource conflicts.
Key practices:
1. **Use asyncio.create_task for fire-and-forget operations** to avoid blocking the main execution flow
2. **Implement proper task cleanup** by having tasks remove themselves from tracking collections when complete
3. **Use controlled loop conditions** instead of `while True:` to allow graceful termination
4. **Prevent overlapping executions** by checking if a task is already running before starting a new one
5. **Always release resources in finally blocks** to ensure cleanup even when exceptions occur
Example implementation:
```python
# Global task tracking with automatic cleanup
BACKGROUND_TASKS = set()
async def start_background_operation():
# Use create_task for non-blocking execution
task = asyncio.create_task(background_health_check())
# Add to tracking set and auto-remove when done
BACKGROUND_TASKS.add(task)
task.add_done_callback(BACKGROUND_TASKS.discard)
async def background_health_check():
# Use controlled condition instead of while True
while use_background_health_checks:
try:
# Perform work
await perform_health_check()
finally:
# Always cleanup resources
await cleanup_resources()
```
This approach prevents memory leaks from orphaned tasks, allows controlled shutdown of background operations, and ensures resources are properly released even when errors occur.
---
## eliminate unnecessary code
<!-- source: Unstructured-IO/unstructured | topic: Code Style | language: Python | updated: 2025-06-09 -->
Remove redundant operations, unnecessary code blocks, and verbose constructs to improve code clarity and maintainability. Focus on eliminating dead code, avoiding redundant loops, removing unnecessary conditional checks, and using more concise expressions where appropriate.
Key practices:
- Remove unnecessary loops in comprehensions when variables aren't used
- Eliminate redundant conditional checks (e.g., checking the same condition twice)
- Use generator expressions instead of building intermediate lists
- Inline short-lived variables that don't add clarity
- Remove unnecessary `pass` statements when docstrings are present
- Avoid unnecessary `if` blocks when the contained code is a no-op
- Use guard clauses to reduce nesting levels
Example of improvement:
```python
# Before: Unnecessary loop variable
CSS_CLASS_TO_ELEMENT_TYPE_MAP = {
element_type().css_class_name: element_type
for element_type in ALL_ONTOLOGY_ELEMENT_TYPES
for tag in element_type().allowed_tags # 'tag' is unused
}
# After: Remove unused loop variable
CSS_CLASS_TO_ELEMENT_TYPE_MAP = {
element_type().css_class_name: element_type
for element_type in ALL_ONTOLOGY_ELEMENT_TYPES
}
# Before: Verbose list building
lines = []
for each in obj:
lines.append(json.dumps(each, **kwargs))
return "\n".join(lines)
# After: Concise generator expression
return "\n".join(json.dumps(each, **kwargs) for each in obj)
```
This approach reduces cognitive load, improves performance, and makes code more maintainable by eliminating distractions and focusing on essential logic.
---
## prefer OpenAI compatibility
<!-- source: BerriAI/litellm | topic: AI | language: Python | updated: 2025-06-06 -->
When integrating new LLM providers or AI services, prioritize OpenAI-compatible patterns and reuse existing utilities rather than creating separate implementations. This approach reduces code duplication, leverages battle-tested functionality, and maintains consistency across the codebase.
Key practices:
- Add new providers to `openai_compatible_providers` instead of creating separate handler blocks
- Use `base_llm_http_handler` or `openai_like_chat_completion` for OpenAI-compatible APIs
- Inherit from `OpenAILikeConfig` or `OpenAIGPTConfig` for transformation classes
- Reuse existing utilities like `get_llm_provider`, `supports_function_calling`, and parameter mapping functions
- Avoid hardcoding provider-specific logic in main completion functions
Example of preferred approach:
```python
# Instead of creating a separate provider block:
elif custom_llm_provider == "new_provider":
# Custom implementation...
# Prefer adding to openai_compatible_providers:
openai_compatible_providers = [..., "new_provider"]
# And use existing handlers:
response = base_llm_http_handler.completion(
model=model,
messages=messages,
custom_llm_provider=custom_llm_provider,
api_base=api_base,
api_key=api_key,
# ...
)
```
This pattern ensures new AI integrations benefit from existing error handling, parameter validation, streaming support, and other features while minimizing maintenance overhead.
---
## Descriptive balanced naming
<!-- source: ollama/ollama | topic: Naming Conventions | language: Go | updated: 2025-06-03 -->
Choose identifier names that clearly communicate their purpose while maintaining an appropriate balance between brevity and clarity.
**Guidelines:**
1. **Use descriptive names that reflect types and purpose:**
```go
// Poor: Using abbreviated or generic names
cap, err := GetModel(n.String())
// Good: Name reflects the actual type
model, err := GetModel(n.String())
```
2. **Avoid abbreviations unless universally understood:**
```go
// Poor: Unnecessary abbreviation
type RopeOpts struct {}
// Good: Full word for clarity
type RopeOptions struct {}
```
3. **Balance verbosity - aim for 1-2 word components:**
```go
// Too verbose
var templateToolPrefix, templateToolPrefixFound := ToolPrefix(model.Template.Template)
// Better balance
prefix, found := toolPrefix(model.Template.Template)
```
4. **Be consistent with naming patterns:**
```go
// Inconsistent with other handlers
func (s *Server) version(w http.ResponseWriter, r *http.Request)
// Consistent with other handler naming
func (s *Server) VersionHandler(c *gin.Context)
```
5. **Avoid single-letter variables** outside of short-lived scopes like simple loops.
6. **In environment variables and constants**, spell words completely:
```go
// Poor: Ambiguous abbreviation
OLLAMA_GPU_DEVS
// Good: Clear, fully spelled out
OLLAMA_GPU_DEVICES
```
When choosing names, prioritize clarity for future readers over saving a few keystrokes now.
---
## Follow GoDoc conventions
<!-- source: ollama/ollama | topic: Documentation | language: Go | updated: 2025-05-27 -->
Document code following Go's official documentation style guide (https://tip.golang.org/doc/comment). All exported functions, types, interfaces, and struct fields should have descriptive comments.
Function documentation should start with the function name and a verb describing what it does:
```go
// addContent returns the thinking content and the normal content that should be
// immediately sent to the user. It will internally buffer if it needs to see
// more content to disambiguate
func (s *thinkingParser) addContent(content string) (string, string) {
// Implementation
}
```
For struct fields, place comments on the line above each field:
```go
type ErrorResponse struct {
// Err is the error from the server. It helps with debugging the code-path
Err string `json:"error"`
// Hint is a user-friendly message about what went wrong, with suggested troubleshooting
Hint string `json:"hint"`
}
```
Document interfaces thoroughly, especially when they serve as implementation guides for different backends:
```go
// ScaledDotProductAttention defines operations for attention computation
// that must be implemented by backend providers
type ScaledDotProductAttention interface {
// Methods...
}
```
Include parameter descriptions when their purpose isn't obvious:
```go
// parseJSONToolCalls attempts to parse a JSON string into a slice of ToolCalls.
// It first checks for balanced braces before attempting to parse.
// Parameters:
// - s: The JSON string to parse
// - name: The field name representing the tool name in the JSON
// - arguments: The field name representing the tool arguments in the JSON
// Returns:
// - []api.ToolCall: The parsed tool calls if successful
// - error: ErrAccumulateMore if braces unbalanced, ErrInvalidToolCall if invalid, or nil if successful
func parseJSONToolCalls(s string, name, arguments string) ([]api.ToolCall, error) {
```
Always explain non-obvious behaviors or design decisions:
```go
// topK limits the number of tokens considered to the k highest logits.
// The slice is passed by pointer because its length will be modified to k.
func topK(ts *[]token, k int) {
// Implementation
}
```
---
## Use idiomatic Go flow
<!-- source: ollama/ollama | topic: Code Style | language: Go | updated: 2025-05-27 -->
Follow Go's idiomatic control flow patterns to improve code readability and maintainability. Key practices include:
1. Prefer early returns over else blocks
2. Return errors on the right in function signatures
3. Use if statements instead of switches for single cases
Example - Before:
```go
func inferThinkingOption(caps *[]model.Capability, runOpts *runOptions, explicitlySetByUser bool) (error, *bool) {
if condition {
// success case
} else {
return errors.New("error"), nil
}
}
```
After:
```go
func inferThinkingOption(caps *[]model.Capability, runOpts *runOptions, explicitlySetByUser bool) (*bool, error) {
if !condition {
return nil, errors.New("error")
}
// success case
}
```
This approach:
- Makes code flow more predictable
- Reduces nesting and cognitive load
- Follows established Go conventions
- Makes error handling more consistent
---
## Ensure metadata consistency
<!-- source: LMCache/LMCache | topic: Database | language: Python | updated: 2025-05-27 -->
Always persist metadata alongside data and implement proper checking mechanisms to prevent duplicates and ensure consistency in distributed storage scenarios. When writing data to storage backends, metadata should be written atomically with the data itself. Additionally, implement separate checking logic for existence vs. active operations to prevent race conditions and duplicate storage.
For distributed systems, implement fallback checks when local caches miss, as other nodes may have added data not present in the local hot cache. Use separate read/write clients when possible to optimize for different access patterns and enable role-based separation.
Example implementation pattern:
```python
# Always write metadata with data
async def store_data(self, key, data):
metadata = pack_metadata(data.shape, data.dtype, data.nbytes)
# Write both atomically
await self._write_data_and_metadata(key, data, metadata)
# Update local cache
with self.hot_lock:
self.hot_cache[key] = metadata
# Separate existence check from active operations check
def contains(self, key: CacheEngineKey) -> bool:
# Check local cache first
with self.hot_lock:
if key in self.hot_cache:
return True
# Fallback to filesystem check for distributed scenarios
return self._try_to_read_metadata(key)
def store(self, key: CacheEngineKey, data):
# Check active operations to prevent duplicates
with self.put_lock:
if key in self.put_tasks:
return # Already being stored
self.put_tasks.add(key)
# Proceed with storage...
```
This pattern ensures data integrity, prevents duplicate operations, and maintains consistency across distributed storage backends.
---
## Complete null checks
<!-- source: ollama/ollama | topic: Null Handling | language: C++ | updated: 2025-05-26 -->
Always perform thorough null checks before dereferencing pointers, and ensure all operations on potentially null objects are contained within the null check block. This prevents null pointer dereferences and undefined behavior.
For example, when freeing resources, place all cleanup operations inside the null check:
```cpp
// Bad - operations outside null check
if (g != nullptr) {
delete g->vocab;
// Other operations...
}
g->someOperation(); // Potential null dereference!
// Good - all operations inside null check
if (g != nullptr) {
delete g->vocab;
g->someOperation();
// All operations safely inside the check
}
```
When working with references or pointers that may become invalid, consider returning objects by value instead of by reference to prevent dangling references. This applies especially when the object lifetime cannot be guaranteed:
```cpp
// Risky - caller might use reference after source is deallocated
const std::string & token_to_piece(uint32_t token);
// Safer - returns a copy that remains valid
const std::string token_to_piece(uint32_t token);
```
---
## Optimize memory allocations
<!-- source: ollama/ollama | topic: Performance Optimization | language: C++ | updated: 2025-05-26 -->
Be strategic about memory allocations to improve performance in your C++ code. Consider two key optimization patterns:
1. **Pre-reserve container capacity**: When using containers like vectors that will grow to a predictable size, use `reserve()` with a reasonable initial capacity to reduce allocation overhead. This prevents multiple small reallocations as the container grows.
```cpp
// Before - May cause multiple reallocations during population
vec_rules[i].resize(n_rules);
// After - Pre-allocate with reasonable capacity
vec_rules[i].reserve(16); // Start with reasonable capacity
```
2. **Avoid unnecessary preallocation**: When designing functions that return variable-sized data, prefer return values over output parameters when the required size is difficult to predict in advance. This eliminates the need for callers to pre-allocate buffers of the correct size.
```cpp
// Before - Requires caller to pre-allocate buffer of correct size
int schema_to_grammar(const char *json_schema, char *grammar, size_t max_len);
// After - Return value approach avoids pre-allocation issues
std::string schema_to_grammar(const char *json_schema);
```
Both strategies help minimize memory churn, reduce fragmentation, and improve overall application performance, particularly in performance-critical code paths.
---
## Document configuration choices
<!-- source: langgenius/dify | topic: Configurations | language: Shell | updated: 2025-05-23 -->
Configuration scripts and files should clearly document non-obvious choices, especially numeric IDs, special values, and environment-specific commands. Include comments explaining the purpose and reasoning behind configuration decisions to help future maintainers understand why specific values or approaches were chosen.
When using numeric user/group IDs, always document which user they represent:
```bash
# Set ownership to proxy user (UID 13) required for Squid operation
chown -R 13:13 /var/log/squid
```
Additionally, verify that configuration commands work reliably across environments. Avoid commands with known compatibility issues and prefer more explicit alternatives:
```bash
# Use explicit directory change instead of -C flag due to poetry issue #9415
cd api && poetry install
```
This practice prevents configuration failures and reduces debugging time when scripts don't work as expected in different environments.
---
## Avoid hardcoded configuration values
<!-- source: LMCache/LMCache | topic: Configurations | language: Shell | updated: 2025-05-20 -->
Configuration files and scripts should use placeholders, environment variables, or parameterized values instead of hardcoding specific configuration values. Hardcoded values make configurations inflexible and difficult to adapt across different environments or deployments.
Use placeholder formats that clearly indicate what type of value should be provided:
```bash
# Good - uses placeholder format
IMAGE=<IMAGE_NAME>:<TAG>
# Avoid - hardcoded specific values
IMAGE=lmcache/vllm-openai:2025-05-17-v1
```
This approach ensures configurations remain flexible and can be easily customized for different environments, deployments, or use cases without requiring code changes.
---
## Remove redundant AI commands
<!-- source: LMCache/LMCache | topic: AI | language: Shell | updated: 2025-05-19 -->
Eliminate duplicate or redundant commands when setting up AI infrastructure, model serving, or ML library installations. Redundant commands create confusion, increase maintenance overhead, and can lead to unexpected behavior in AI workflows.
Common scenarios to watch for:
- Installing the same AI library multiple times (e.g., both `uv pip install vllm` and `uv pip install --upgrade vllm`)
- Duplicating Docker entrypoint commands when the base image already defines them
- Repeating model configuration parameters across different setup stages
Example of redundancy removal:
```bash
# Before: Redundant vllm installation
uv pip install vllm
uv pip install --upgrade vllm
# After: Single command achieves the same result
uv pip install --upgrade vllm
# Before: Redundant Docker entrypoint
docker run --entrypoint "/usr/local/bin/vllm" $IMAGE serve $MODEL_NAME
# After: Use existing entrypoint (when ENTRYPOINT is already ["vllm", "serve"])
docker run $IMAGE $MODEL_NAME
```
Always review AI setup scripts and configurations to identify and eliminate unnecessary command duplication, ensuring cleaner and more maintainable infrastructure code.
---
## Clear recoverable error messages
<!-- source: ollama/ollama | topic: Error Handling | language: Go | updated: 2025-05-15 -->
Error messages should be clear, actionable, and indicate whether recovery is possible. When designing error handling:
1. Use descriptive messages that explain what went wrong and how to fix it
2. Indicate if the error is expected/recoverable
3. Provide appropriate fallbacks when safe
4. Include relevant context in error messages
Example:
```go
// Instead of
if err != nil {
return fmt.Errorf("puller failed: %w", err)
}
// Do this
if err != nil {
if errors.Is(err, ErrModelNotFound) {
// Clear message with actionable hint
return fmt.Errorf("model %q not found - check available models at: https://ollama.com/models", name)
}
// Indicate if retry may help
if isTemporary(err) {
return fmt.Errorf("temporary error fetching model %q, retry may resolve: %w", name, err)
}
return fmt.Errorf("failed to pull model %q: %w", name, err)
}
// Consider fallbacks for non-critical errors
if req.Options == nil {
// Use safe defaults instead of error
req.Options = api.DefaultOptions()
}
```
This approach helps users understand and recover from errors while maintaining system stability through appropriate fallbacks.
---
## Prioritize naming clarity
<!-- source: BerriAI/litellm | topic: Naming Conventions | language: Markdown | updated: 2025-05-14 -->
Choose names that are immediately self-explanatory and unambiguous, even if they are more verbose. Names should convey their full meaning and context without requiring additional documentation or guesswork.
When naming variables, configuration settings, or API patterns, prioritize precision over brevity. A longer, descriptive name is preferable to a shorter, ambiguous one.
Examples:
- Prefer `maximum_spend_logs_retention_period` over `maximum_retention_period` to clearly specify what type of retention period is being configured
- Use explicit path structures like `huggingface/<provider>/<hf_org_or_user>/<hf_model>` to make hierarchical relationships and namespacing clear
The goal is that any developer reading the code should immediately understand what a name refers to without needing to consult documentation or make assumptions about its scope or purpose.
---
## Guard against nil
<!-- source: ollama/ollama | topic: Null Handling | language: Go | updated: 2025-05-12 -->
Always check for nil values and successful type assertions before accessing or dereferencing objects, especially when working with interfaces, type conversions, or functions that may return nil values.
For type assertions, use the two-return form to check if the assertion succeeded:
```go
// Good: Check if type assertion succeeded before using
if tp, ok := s.model.(model.TextProcessor); ok {
vocab = tp.Vocabulary()
}
// Bad: May panic if assertion fails
vocab = s.model.(model.TextProcessor).Vocabulary()
```
For values that may be nil, verify before dereferencing:
```go
// Good: Check if grammar is nil after initialization
grammar := llama.InitGrammarChain(grammarStr)
if grammar == nil {
return nil, errors.New("failed to initialize grammar")
}
// Bad: May cause nil pointer dereference
grammar := llama.InitGrammarChain(grammarStr)
grammar.AddSymbol(s, uint32(id)) // Panic if grammar is nil
```
Consider explicitly initializing structures that may be nil:
```go
// Good: Initialize and check configuration
if c.config == nil {
var config ml.CacheConfig
if cc, ok := backend.(ml.BackendCacheConfig); ok {
// Do something with config
}
}
```
These practices prevent runtime panics and make your code more robust and maintainable.
---
## Design extensible permission systems
<!-- source: BerriAI/litellm | topic: Security | language: Prisma | updated: 2025-05-07 -->
When implementing authorization features, design permission systems that can accommodate future security requirements rather than adding permissions directly to existing tables. Create dedicated permission tables with proper relationships to entity tables to ensure authorization can be properly enforced.
Instead of adding permission arrays directly to existing tables:
```prisma
model LiteLLM_TeamTable {
// ... other fields
mcp_servers String[] @default([]) // Avoid this approach
}
```
Create extensible permission tables with proper relationships:
```prisma
model LiteLLM_ObjectPermissionTable {
permission_id String @id @default(uuid())
mcp_servers String[] @default([])
// Future permissions can be added here
}
model LiteLLM_VerificationToken {
// ... other fields
permission_id String?
object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [permission_id], references: [permission_id])
}
```
This approach ensures that permission enforcement logic has a clear structure to work with and can be extended as new authorization requirements emerge without requiring schema migrations to core entity tables.
---
## Reuse buffers strategically
<!-- source: ollama/ollama | topic: Performance Optimization | language: Go | updated: 2025-05-06 -->
For frequently called methods or hot code paths, reuse allocations instead of creating new buffers on each call. This significantly reduces GC pressure and improves performance in memory-intensive operations.
To implement this pattern:
1. Add reusable buffer fields to structs that handle data processing
2. Check the capacity of existing buffers before allocating new ones
3. Resize only when necessary based on input requirements
Example implementation:
```go
// Before: Allocating a new buffer on each call
func (s *weighted) Sample(logits []float32) (int32, error) {
tokens := make([]tokenInfo, len(logits))
// Process logits...
}
// After: Reusing a buffer across calls
type weighted struct {
rng *rand.Rand
transforms []transform
buf []tokenInfo // reusable buffer field
}
func (s *weighted) Sample(logits []float32) (int32, error) {
// Resize only when necessary
if cap(s.buf) < len(logits) {
s.buf = make([]tokenInfo, len(logits))
}
tokens := s.buf[:len(logits)]
// Process logits...
}
```
This pattern is especially important for functions in critical processing paths like sampling, data copying, or quantization operations. For data copying operations that don't require transformation, consider using io.Copy or similar approaches that avoid loading everything into memory.
---
## Comprehensive test coverage
<!-- source: ollama/ollama | topic: Testing | language: Go | updated: 2025-05-04 -->
Test functions should provide comprehensive coverage of both expected and edge cases. Include tests for:
1. **Boundary conditions**: Test zero, minimum, maximum values, and other special cases.
```go
func TestTemperature(t *testing.T) {
// Test normal case
logits, err := Temperature(0.5).Apply([]float64{-3, -2, -1, 0, 1, 2, 4})
if err != nil {
t.Error(err)
}
// Test boundary case (zero)
logits, err = Temperature(0).Apply([]float64{-3, -2, -1, 0, 1, 2, 4})
if err != nil {
t.Error(err)
}
}
```
2. **Varied input patterns**: Test unsorted, sorted, empty, and different input formats.
```go
// Test with unsorted inputs
logits, err := Temperature(0.5).Apply([]float64{2, -1, 4, -3, 1, -2, 0})
```
3. **Multiple levels of granularity**: Test intermediate steps as well as end-to-end functionality.
```go
// Don't just test ParseToolCalls, but also intermediate steps
func TestFindTemplatePrefix(t *testing.T) { /* ... */ }
func TestExtractToolTokens(t *testing.T) { /* ... */ }
func TestParseToolCalls(t *testing.T) { /* ... */ }
```
4. **Both valid and invalid cases**: Each test function should validate both expected successful cases and error conditions.
```go
tests := []struct {
name string
input string
valid bool
want Result
wantErr error
}{
{
name: "valid input",
input: "valid data",
valid: true,
want: expectedResult,
},
{
name: "invalid input",
input: "invalid data",
valid: false,
wantErr: ErrInvalidData,
},
}
```
Thorough test coverage helps catch edge cases and ensures code robustness, particularly for functionality that may be used across different parts of the system.
---
## Document synchronization intent
<!-- source: ollama/ollama | topic: Concurrency | language: Go | updated: 2025-05-01 -->
Always clearly document the purpose and scope of synchronization primitives (mutexes, read-write locks) by:
1. Placing them directly adjacent to the data they protect
2. Adding comments explaining what's being protected when not obvious
3. Making locking requirements explicit in method/function documentation
This practice prevents race conditions and helps future developers understand the concurrent access patterns in your code.
**Bad example:**
```go
var mu sync.Mutex
// ... many lines later or in another function ...
func doSomething() {
mu.Lock()
// use some shared data
mu.Unlock()
}
```
**Good example:**
```go
// UserCache guards access to the user data map
var (
userCache map[string]UserData
userCacheMu sync.RWMutex // guards userCache
)
// GetUser returns user data, safe for concurrent access
// Locking: Acquires userCacheMu read lock
func GetUser(id string) (UserData, bool) {
userCacheMu.RLock()
defer userCacheMu.RUnlock()
data, found := userCache[id]
return data, found
}
```
When synchronizing access to maps or slices, consider using a dedicated synchronized wrapper type that enforces the locking protocol rather than managing locks separately. For synchronized data structures, ensure methods that return internal state (like Maps) return copies rather than references to prevent race conditions from external modifications.
---
## Optimize AI implementation patterns
<!-- source: ollama/ollama | topic: AI | language: Other | updated: 2025-05-01 -->
When implementing AI systems, prioritize established patterns and optimizations rather than creating new implementations for existing problems. This applies to all aspects of AI development:
1. For model operations, prefer reusing existing functions over implementing new ones when they serve the same purpose. For example, use `ggml_scale(ctx, cur, -1)` instead of implementing a separate `ggml_neg` operation, as it achieves the same result with equivalent performance and memory usage.
2. When designing data structures for AI models (like vocabulary handling), ensure they account for all possible variations and edge cases. For instance, when handling end-of-generation tokens, remember there may be multiple token types (EOS, EOT) rather than assuming a single one.
3. Maintain precision in documentation of AI tools to ensure clarity for users with different experience levels.
```cpp
// Instead of implementing new operations:
// ggml_tensor * result = ggml_neg(ctx, tensor);
// Prefer reusing existing operations:
ggml_tensor * result = ggml_scale(ctx, tensor, -1);
// Instead of:
struct vocab {
uint32_t eog_token; // Single end token
};
// Prefer:
struct vocab {
std::vector<uint32_t> eog_tokens; // Multiple possible end tokens (EOS, EOT)
};
```
These optimizations improve code maintainability, reduce potential bugs, and often lead to better performance in AI systems that already have high computational demands.
---
## Balance CI/CD trade-offs
<!-- source: langgenius/dify | topic: CI/CD | language: Yaml | updated: 2025-04-29 -->
When configuring CI/CD pipelines, balance comprehensive coverage and security with practical constraints like resource usage, tool compatibility, and project context. Avoid blindly applying best practices without considering their impact on functionality and efficiency.
Consider these trade-offs:
- **Permissions**: Start with minimal permissions but adjust when tools require broader access to function correctly
- **Test matrices**: Include comprehensive version coverage when possible, but prioritize the most critical versions when resources are limited
- **Automation frequency**: Match update intervals to your project's actual needs rather than using default aggressive settings
Example from a GitHub workflow:
```yaml
strategy:
matrix:
python-version:
- "3.12" # Primary version for most users
# - "3.11" # Commented out to save CI resources
```
Document your reasoning when making these trade-offs so future maintainers understand the decisions and can revisit them as constraints change.
---
## Separate AI instructions
<!-- source: langgenius/dify | topic: AI | language: Python | updated: 2025-04-18 -->
Always separate AI model instructions from user input to prevent prompt injection vulnerabilities and maintain clear architectural boundaries. Use system messages for AI instructions and user messages for actual user content, rather than embedding user input directly into instruction templates.
This approach provides two key benefits:
1. **Security**: Prevents prompt injection attacks where malicious user input could override or manipulate the AI's instructions
2. **Clarity**: Maintains clear separation of concerns between what the AI should do (instructions) and what data it should process (user input)
Example of the recommended approach:
```python
# Good: Separate system instructions from user input
prompt_messages = [
SystemPromptMessage(content="Generate JSON output following this schema: {...}"),
UserPromptMessage(content=user_provided_content)
]
# Avoid: Embedding user input in instructions
prompt_template = "Generate JSON for: {user_input}" # Vulnerable to injection
```
Be particularly careful when implementing structured output features, as system prompts may be modified or discarded depending on the model's native capabilities. Always validate that the intended instruction-user separation is preserved throughout the processing pipeline.
---
## simplify complex logic
<!-- source: langgenius/dify | topic: Code Style | language: Python | updated: 2025-04-17 -->
Improve code readability by reducing complexity and nesting through early returns, extracting complex logic into dedicated functions, and using clear parameter naming.
**Key practices:**
1. **Use early returns** to reduce nested if statements and simplify logic flow
2. **Extract complex nested logic** into dedicated local functions with descriptive names
3. **Use named parameters** instead of positional boolean parameters for better readability
4. **Simplify conditional expressions** by using early continue/break statements when appropriate
**Examples:**
```python
# Instead of deeply nested conditions:
if condition1:
if condition2:
if condition3:
# complex logic here
pass
# Use early returns:
if not condition1:
return self
if not condition2:
return self
if not condition3:
return self
# complex logic here
# Instead of positional boolean parameters:
vn.ask(prompt, False, True, False, allow_llm_to_see_data)
# Use named parameters:
vn.ask(prompt, print_results=False, auto_train=True, visualize=False, allow_llm_to_see_data=allow_llm_to_see_data)
# Instead of complex nested logic in main function:
if self.node_data.structured_output_enabled and self.node_data.structured_output:
# 20+ lines of complex logic
# Extract to dedicated function:
def process_structured_output(self):
if not self.node_data.structured_output_enabled:
return
if not self.node_data.structured_output:
return
# complex logic here
# Then call it:
process_structured_output()
```
This approach makes code more maintainable, easier to test, and significantly improves readability by reducing cognitive load.
---
## Use environment variables
<!-- source: ollama/ollama | topic: Configurations | language: Go | updated: 2025-04-16 -->
Use environment variables instead of hardcoding configuration values such as file paths, port numbers, or system-specific locations. This makes the code more flexible across different platforms, installation methods, and user preferences.
When implementing configuration:
1. Use environment variables for values that might differ between environments
2. Provide sensible defaults when environment variables are not set
3. Support platform-specific path conventions (like `~` expansion) across operating systems
4. Handle architecture differences (x86_64, aarch64, etc.) appropriately
Example:
```go
// Bad: Hardcoded path that won't work across environments
func RegisterService() {
server, err = zeroconf.Register(
"OllamaInstance", // Service Name
"_ollama._tcp", // Service Type
"local.", // Domain
11434, // Port - hardcoded!
)
}
// Good: Use environment variable with fallback
func RegisterService() {
port := 11434 // Default
if hostEnv := os.Getenv("OLLAMA_HOST"); hostEnv != "" {
if _, portStr, err := net.SplitHostPort(hostEnv); err == nil && portStr != "" {
if p, err := strconv.Atoi(portStr); err == nil {
port = p
}
}
}
server, err = zeroconf.Register(
"OllamaInstance", // Service Name
"_ollama._tcp", // Service Type
"local.", // Domain
port, // Port from environment or default
)
}
```
For path handling, support platform-specific conventions:
```go
func getConfigPath() string {
if path, exists := os.LookupEnv("OLLAMA_CONFIG"); exists {
// Handle home directory expansion
if strings.HasPrefix(path, "~/") {
homeDir, err := os.UserHomeDir()
if err == nil {
path = filepath.Join(homeDir, path[2:])
}
}
return path
}
// Default fallback
homeDir, _ := os.UserHomeDir()
return filepath.Join(homeDir, ".ollama", "config")
}
```
---
## Platform-aware configuration documentation
<!-- source: ollama/ollama | topic: Configurations | language: Markdown | updated: 2025-04-11 -->
Always document platform-specific configuration differences and requirements thoroughly. When writing documentation or implementing configuration-related code, explicitly address variations across different operating systems and environments.
Key practices:
1. Specify environment variable differences by platform (e.g., temp directories use TMPDIR on Linux, but TMP/TEMP on Windows)
2. Document installation and configuration paths that may vary between systems
3. Provide appropriate command syntax for each platform, noting differences in flags or parameters
4. Include platform-specific prerequisites and setup requirements
Example:
```shell
# For temporary directory configuration:
# On Linux/macOS:
export TMPDIR=/path/with/sufficient/space
# On Windows:
set TMP=D:\path\with\sufficient\space
# When removing configuration directories:
# On Linux/macOS:
sudo rm -rf /etc/ollama # Use -r flag for directories
# For cross-platform command examples:
ollama completion [bash|zsh|fish] > /path/to/completion/file # Standard syntax pattern
```
Following these practices ensures configurations work correctly across different environments and helps users avoid platform-specific issues.
---
## Loose API coupling
<!-- source: ollama/ollama | topic: API | language: Go | updated: 2025-04-07 -->
Maintain a clear separation between API definitions and their implementations by avoiding direct passing of API structures to implementation layers. Extract only the necessary values from API structures when passing data downstream.
This approach has several benefits:
- Improves maintainability as implementation details can evolve independently
- Reduces tight coupling between API contracts and internal code
- Makes testing easier by simplifying dependencies
- Prevents implementation-specific requirements from leaking into the API design
Example of problematic coupling:
```go
// Avoid: Passing entire API structure downstream
sampler := sample.NewSampler(req.Options, grammar)
```
Better approach:
```go
// Better: Extract only needed values
sampler := sample.NewSampler(
req.Options.Temperature,
req.Options.TopK,
req.Options.TopP,
req.Options.MinP,
req.Options.Seed,
grammar,
)
```
Similarly, for transformation pipelines, prefer direct function calls with clear parameters over passing abstract data structures:
```go
// Instead of: Using lists of transforms
transforms := []Transform{topK, topP, softmax}
Weighted(rng, transforms...)
// Better: Use specific function calls in a clear sequence
result := topK(topP(softmax(values)))
```
This principle extends to all layers of your API implementation - keep interfaces clean and pass only what's needed.
---
## Extract duplicated code
<!-- source: ollama/ollama | topic: Code Style | language: Other | updated: 2025-03-19 -->
Identify repeated code segments and extract them into reusable functions or variables. This improves maintainability, reduces the risk of inconsistencies, and makes the code more concise.
Example from shader code:
```
// Before
float theta_base = (float) pos[i2];
// Code that uses theta_base
// ...
// Later in the code, theta_base is calculated again
float theta_base = (float) pos[i2];
// After
float calculate_theta_base(device const int32_t* pos, int i2) {
return (float) pos[i2];
}
// ...
float theta_base = calculate_theta_base(pos, i2);
// ...
// Later in the code
float theta_base = calculate_theta_base(pos, i2);
```
Even for small code segments, applying the DRY (Don't Repeat Yourself) principle leads to more maintainable code. When you see the same calculation, transformation, or logic repeated, consider extracting it into a function or storing it in a well-named variable. This also makes code more resistant to bugs when changes are needed, as you'll only need to update the logic in one place.
---
## Optimize with standard library
<!-- source: ollama/ollama | topic: Algorithms | language: Go | updated: 2025-03-12 -->
When implementing algorithms, prefer using Go's standard library over custom implementations or external dependencies for common operations. The standard library offers well-tested, efficient implementations for many algorithmic needs.
**Why this matters:**
1. Reduces code complexity and maintenance burden
2. Leverages battle-tested implementations
3. Eliminates external dependencies when possible
4. Often provides better performance
**Examples:**
Instead of implementing custom sorting or search functions:
```go
// Don't reinvent binary search
for low, high := 0, len(arr); low < high; {
mid := (low + high) / 2
if arr[mid] < target {
low = mid + 1
} else {
high = mid
}
}
// Better: Use standard library
idx := slices.BinarySearchFunc(arr, target, func(a, b YourType) int {
return cmp.Compare(a, b)
})
```
Instead of custom data structures like priority queues:
```go
// Don't use external dependencies for basic data structures
import pq "github.com/emirpasic/gods/queues/priorityqueue"
queue := pq.New() // External dependency
// Better: Use container/heap from standard library
type tokenHeap []token
// Implement heap.Interface methods: Len, Less, Swap, Push, Pop
// Then use with standard heap package:
h := &tokenHeap{}
heap.Init(h)
heap.Push(h, item)
```
For string operations:
```go
// Avoid manual string manipulation when possible
if strings.HasPrefix(s, prefix) {
s = strings.TrimSpace(s[len(prefix):])
}
// Better: Use strings.Cut
if after, found := strings.CutPrefix(s, prefix); found {
s = strings.TrimSpace(after)
}
```
For numeric operations:
```go
// Avoid manual min/max checks
if temperature < 1e-7 {
temperature = 1e-7
}
// Better: Use built-in min/max functions
temperature = max(temperature, 1e-7)
```
---
## Improve documentation discoverability
<!-- source: microsoft/markitdown | topic: Documentation | language: Python | updated: 2025-03-12 -->
Ensure that important information about code purpose, behavior, and context is easily discoverable through proper documentation placement and completeness. Add comprehensive docstrings to modules, classes, and functions that explain their purpose and usage. Place critical implementation details in docstrings rather than inline comments to improve discoverability. Use specific type annotations with clear explanations rather than generic types like `Any`.
For example, instead of:
```python
# Note: We have tight control over the order of built-in converters, but
def __init__(self, markitdown: Any):
pass
```
Use:
```python
def __init__(self, markitdown: MarkItDown):
"""Initialize converter with MarkItDown instance.
Args:
markitdown: MarkItDown instance reference to allow recursive
conversions of nested files. We maintain tight control
over the order of built-in converters.
"""
pass
```
This ensures developers can quickly understand the code's purpose and important implementation details without hunting through comments or guessing from generic type hints.
---
## AI memory management
<!-- source: ollama/ollama | topic: AI | language: Markdown | updated: 2025-03-10 -->
Document and implement proper memory management strategies for AI model inference to prevent out-of-memory errors and optimize performance. When developing AI applications or documentation:
1. **Specify hardware memory requirements**
- Document minimum and recommended memory configurations
- For GPU-accelerated models, specify VRAM requirements
- Example: "Set UMA for iGPU in BIOS (at least >1GB, recommend >8GB for Llama3:8b q4_0 model size is 4.7GB)"
2. **Configure context window appropriately**
- Set context window size based on available memory and use case requirements
- Example: `OLLAMA_CONTEXT_LENGTH=8192 ollama serve`
3. **Implement memory optimization techniques**
- Use environment variables to control memory usage:
```bash
# Reserve additional GPU memory buffer
OLLAMA_GPU_OVERHEAD=536870912 ollama serve
# Enable more efficient memory usage
OLLAMA_FLASH_ATTENTION=1 ollama serve
# Allow GPU to use CPU memory (Linux only)
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ollama serve
# Control parallel processing
OLLAMA_NUM_PARALLEL=1 ollama serve
```
4. **Document fallback strategies**
- Provide clear instructions for reducing memory requirements when needed
- Include troubleshooting steps for memory-related errors
By explicitly documenting memory requirements and optimization strategies, you ensure reliable operation of AI models across different environments and hardware configurations.
---
## AI dependency decoupling strategy
<!-- source: ollama/ollama | topic: AI | language: C++ | updated: 2025-03-10 -->
Implement clear boundaries between your core AI system and external machine learning libraries through abstraction layers. This facilitates future library updates or replacements as AI technology evolves. When integrating external libraries (like llama.cpp), establish documented synchronization processes and gradually work toward independence, especially for critical components like tokenizers.
For example, instead of directly depending on an external tokenizer:
```cpp
// Avoid direct dependency
struct llama_vocab * vocab = llama_load_vocab_from_file(fname);
// Instead, create an abstraction layer
struct tokenizer * tokenizer = create_tokenizer_from_file(fname);
// Where create_tokenizer_from_file might initially use llama underneath
// but could later be replaced with a different implementation
```
This approach reduces technical debt and makes it easier to adopt new AI model architectures or optimizations as they become available.
---
## validate before processing
<!-- source: microsoft/markitdown | topic: Null Handling | language: Python | updated: 2025-03-06 -->
Always validate input state and types before performing operations to prevent runtime errors and unexpected behavior. This includes checking data types before applying type-specific operations and preserving object state when performing validation checks.
When working with streams, restore the original position after reading during validation to avoid side effects:
```python
# Good: Preserve stream position
cur_pos = file_stream.tell()
data = file_stream.read()
file_stream.seek(cur_pos)
```
When processing data, validate the type before applying operations:
```python
# Good: Check type before text operations
if isinstance(content, str):
content = self._strip_leading_blanks(content)
```
This defensive approach prevents errors from assumptions about input state or type, similar to how null checks prevent null reference exceptions.
---
## explicit API parameters
<!-- source: microsoft/markitdown | topic: API | language: Python | updated: 2025-03-05 -->
Make API parameters explicit and named rather than hiding them in **kwargs. This improves clarity, enables better validation, and provides clearer interfaces for callers.
When designing APIs, avoid burying important parameters in generic **kwargs dictionaries. Instead, define them as explicit named parameters with proper types, defaults, and validation. This makes the API more discoverable, self-documenting, and less prone to errors.
For example, instead of:
```python
def convert(self, local_path, **kwargs):
pdf_engine = kwargs.get("pdf_engine", "pdfminer")
# Hidden parameter, no validation
```
Use explicit parameters with validation:
```python
def convert(self, local_path, pdf_engine: Literal['pdfminer', 'pymupdf4llm'] = 'pdfminer', **kwargs):
_engines = {"pdfminer": pdfminer, "pymupdf4llm": pymupdf4llm}
if pdf_engine not in _engines:
raise ValueError(f"Invalid pdf_engine. Choose from {list(_engines.keys())}")
```
This approach also supports instance-based configuration where converters can be configured with parameters like API keys, while still maintaining clear method signatures. Avoid hardcoding assumptions (like content types) and instead derive them dynamically from the actual data when possible.
---
## Complete HTTP protocol handling
<!-- source: ollama/ollama | topic: Networking | language: Go | updated: 2025-03-05 -->
When implementing HTTP client functionality, ensure comprehensive support for all standard HTTP status codes and secure connection scenarios:
1. Support all standard redirect status codes (301, 302, 303, 307, 308), not just a subset:
```go
switch resp.StatusCode {
case http.StatusOK:
return requestURL, nil
case http.StatusMovedPermanently, // 301
http.StatusFound, // 302
http.StatusSeeOther, // 303
http.StatusTemporaryRedirect, // 307
http.StatusPermanentRedirect: // 308
// Handle redirects
}
```
2. For HTTPS connections with self-signed or internal certificates, provide explicit mechanisms rather than general "insecure" flags. Consider using special URL schemes like `https+insecure://` for connections where certificate validation can be bypassed:
```go
if strings.HasPrefix(url, "https+insecure://") {
// Configure TLS client to skip certificate validation
tlsConfig.InsecureSkipVerify = true
// Strip prefix for actual connection
url = "https://" + url[16:]
}
```
3. Always document security implications when certificate validation is bypassed, with clear scope limitations (e.g., only for local development, internal networks).
---
## Model precision matters
<!-- source: ollama/ollama | topic: AI | language: Shell | updated: 2025-03-03 -->
When deploying AI models on specialized hardware accelerators (GPUs, NPUs), ensure you're using compatible model precision formats. Different hardware platforms have different optimal precision requirements that significantly impact performance and compatibility.
For example, when working with Ascend NPUs, fp16 models often work better than higher precision formats:
```bash
# When compiling AI frameworks for Ascend NPUs
export CUSTOM_CPU_FLAGS=cann
make --no-print-directory -f make/Makefile.cann
# When running models, verify hardware is properly detected
./ollama serve # Should show detected NPUs in logs
# If models run on CPU despite hardware detection, try fp16 model variants
./ollama run model-name-f16 # Explicitly use fp16 version
```
Even when hardware is properly detected, models may still default to CPU execution if precision formats are incompatible with the acceleration hardware. Always test multiple precision variants (fp16, fp32, int8) to determine the optimal configuration for your target hardware.
---
## Path traversal prevention
<!-- source: ollama/ollama | topic: Security | language: Go | updated: 2025-02-28 -->
Implement multiple validation layers to prevent path traversal attacks. File paths provided by users or external systems must be validated before use in filesystem operations:
1. Validate paths using both general and specialized checks:
```go
// First validate with fs.ValidPath
if !fs.ValidPath(fp) {
return nil, fmt.Errorf("%w: %s", errFilePath, fp)
}
// Then use filepath.Clean to normalize
fp = filepath.Clean(fp)
// Finally validate containment with os.Root
root, err := os.OpenRoot(safeDirectory)
if err != nil {
return nil, err
}
defer root.Close()
// All file operations should go through this root
```
2. Enforce a consistent path policy, such as:
- Paths must be relative (no leading "/")
- Paths must not contain traversal components ("../", "./")
- Paths must remain within the designated directory
3. Implement validation at every entry point rather than relying on validation in a single place - this defense-in-depth approach ensures security even if the code is refactored or one validation is bypassed.
---
## Abstract model operations cleanly
<!-- source: ollama/ollama | topic: AI | language: Go | updated: 2025-02-21 -->
When implementing AI model operations, create clean abstractions through interfaces that separate core mathematical operations from specific implementations. This enables:
- Swapping in optimized implementations
- Clear error handling boundaries
- Documentation of mathematical operations
- Easier testing and maintenance
Example:
```go
// Define interface for core operation
type ScaledDotProductAttention interface {
// Document mathematical operation
// Attention(Q,K,V) = softmax(QK^T/√d_k)V
Forward(ctx Context, key, value, mask Tensor, scale float64) Tensor
}
// Implementation with proper validation and documentation
func Attention(ctx Context, query, key, value, mask Tensor, scale float64) Tensor {
// Validate shapes match
if !ValidateShapes(query, key, value, mask) {
return nil, fmt.Errorf("invalid tensor shapes for attention")
}
// Use optimized implementation if available
if sdpa, ok := query.(ScaledDotProductAttention); ok {
return sdpa.Forward(ctx, key, value, mask, scale)
}
// Fallback to manual implementation
kq := key.MulmatFullPrec(ctx, query)
kq = kq.Scale(ctx, scale)
kq = kq.Add(ctx, mask)
return kq.Softmax(ctx)
}
```
---
## Documentation language precision
<!-- source: microsoft/markitdown | topic: Documentation | language: Markdown | updated: 2025-02-10 -->
Ensure documentation uses precise, accurate language that clearly describes functionality and highlights important behaviors users need to understand. Avoid vague or misleading terms that could confuse users about what the code actually does.
Key practices:
- Use specific, accurate verbs (e.g., "shows how to" instead of "allows you to" when describing examples)
- Explicitly state important behaviors and side effects (e.g., "original files remain unchanged")
- Tailor language to the documentation's specific purpose and audience
- Avoid unnecessary duplication while maintaining completeness where needed
Example improvement:
```markdown
// Instead of:
"This extension allows you to convert multiple files..."
// Use:
"This example shows how to convert multiple files..."
// And add clarifying details:
"Note that original files will remain unchanged and new markdown files are created with the same base name."
```
This ensures users have accurate expectations about functionality and understand important implementation details that affect their workflow.
---
## Keep container images current
<!-- source: ollama/ollama | topic: Configurations | language: Dockerfile | updated: 2025-02-09 -->
Always use up-to-date and supported container images in Dockerfiles and configuration files. Before specifying a base image or version:
1. Verify the image hasn't been deprecated
2. Confirm the specified version is available in repositories
3. Consider cross-platform compatibility requirements
For example, instead of:
```dockerfile
FROM cosdt/cann:${ASCEND_VERSION} AS ascend-build-arm64
```
Use the recommended current alternative:
```dockerfile
FROM ascendai/cann:${ASCEND_VERSION} AS ascend-build-arm64
```
Similarly, when updating dependency versions like ROCm, verify availability and compatibility with your base OS image:
```dockerfile
# Check if this version exists with your base image before specifying
ARG ROCM_VERSION=6.3.2
```
When the desired version isn't available with your current base image, consider updating the base image itself (e.g., moving from centos-7 to almalinux-8 for newer ROCm versions).
---
## Use semantically clear names
<!-- source: Unstructured-IO/unstructured | topic: Naming Conventions | language: Python | updated: 2025-02-07 -->
Choose variable, method, and class names that clearly express their purpose and functionality to avoid confusion and improve code readability. Names should accurately reflect what the code does rather than how it's implemented.
**Key principles:**
- **Avoid misleading names**: Don't use names that suggest different functionality than what they provide
- **Be descriptive**: Use specific, meaningful names rather than generic or abbreviated ones
- **Match intent**: Ensure the name reflects the actual purpose, not implementation details
**Examples:**
```python
# ❌ Misleading - sounds like it creates an index
def create_index(self) -> "PineconeIndex":
return existing_index
# ✅ Clear - indicates it retrieves an existing index
def get_index(self) -> "PineconeIndex":
return existing_index
# ❌ Ambiguous abbreviation
def group_text_extraction_acc():
pass
# ✅ Explicit and clear
def group_text_extraction_accuracy():
pass
# ❌ Generic and unclear purpose
def write_raw_dict(self, elements_dict):
self.normalize_data(elements_dict)
self.write_to_db(elements_dict)
# ✅ Describes what the method actually does
def normalize_and_write_dict(self, elements_dict):
self.normalize_data(elements_dict)
self.write_to_db(elements_dict)
# ❌ Predicate named like a function
skip_outside_ci = os.getenv("CI", "").lower() in {"", "false", "f", "0"}
# ✅ Boolean variable with clear intent
is_in_ci = os.getenv("CI", "").lower() not in {"", "false", "f", "0"}
```
This approach reduces cognitive load for developers, makes code self-documenting, and prevents misunderstandings about functionality.
---
## optimize algorithmic complexity
<!-- source: Unstructured-IO/unstructured | topic: Algorithms | language: Python | updated: 2025-02-07 -->
When implementing algorithms, prioritize performance by avoiding unnecessary computations and choosing efficient approaches. Consider algorithmic complexity early in design and look for opportunities to eliminate redundant operations or expensive calculations when simpler alternatives exist.
Key strategies include:
- **Early termination**: Skip expensive operations when conditions make them unnecessary, like checking file size ratios before running costly distance calculations
- **Prevent rather than filter**: Design algorithms to avoid creating unwanted data rather than filtering it out later
- **Eliminate duplicate work**: Avoid parsing or processing the same data multiple times in a pipeline
- **Profile when uncertain**: When multiple approaches have unclear performance trade-offs, measure and compare them
Example from the codebase:
```python
# Instead of always running expensive Levenshtein distance:
if 0.5 < len(output_cct.encode()) / len(source_cct.encode()) < 2.0:
accuracy = round(calculate_accuracy(output_cct, source_cct, weights), 3)
else:
accuracy = 0.01 # Skip expensive calculation when files differ wildly
```
This approach prevents performance bottlenecks and ensures algorithms scale appropriately with input size.
---
## Use portable path configurations
<!-- source: ollama/ollama | topic: Configurations | language: Shell | updated: 2025-02-05 -->
Build and configuration scripts should use portable path handling techniques to ensure they work correctly across different environments and installations.
Key practices:
1. **Use wildcards when copying library files** to include all version suffixes:
```bash
# Correct - captures all version suffixes
cp ${VULKAN_ROOT}/libvulkan.so* "${BUILD_DIR}/bin/"
# Incorrect - only copies the base library file
cp "${VULKAN_ROOT}/libvulkan.so" "${BUILD_DIR}/bin/"
```
2. **Avoid hardcoded absolute paths** in configuration variables. When default paths are needed, make them overridable:
```bash
# Better approach - allows overriding with environment variables
if [ -z "${VULKAN_ROOT}" ]; then
# Default only used if not explicitly set
VULKAN_ROOT=/usr/lib/
fi
```
3. **Use relative paths with `$ORIGIN` for runtime library paths** instead of absolute paths that assume specific installation locations:
```bash
# Better - uses relative paths from binary location
EXTRA_LIBS="-Wl,-rpath,'$ORIGIN'"
# Avoid - assumes specific installation location
EXTRA_LIBS="-Wl,-rpath,/opt/intel/oneapi/compiler/latest/lib"
```
4. **Be careful with string formatting** in configuration variables to prevent command execution errors:
```bash
# Correct - no space after equals
DNF_COMPATIBILITY_FMT="addrepo --from-repofile="
# Incorrect - space will cause command parsing issues
DNF_COMPATIBILITY_FMT="addrepo --from-repofile= "
```
Following these practices ensures build scripts and configurations work reliably across different environments, making software more portable and easier to install for end users.
---
## Responsible AI management
<!-- source: langgenius/dify | topic: AI | language: Yaml | updated: 2025-02-04 -->
Ensure AI models are managed responsibly throughout their lifecycle and have appropriate boundaries on their decision-making authority. When deprecating AI models, maintain backward compatibility by marking old models as deprecated rather than removing them entirely. Additionally, prevent AI models from controlling sensitive parameters that could lead to security risks or unintended consequences.
For model deprecation, create new model files while preserving old ones:
```yaml
# Old model file: cohere.command-r-16k.yaml
model: cohere.command-r-16k
deprecated: true
# New model file: cohere.command-r-08-2024.yaml
model: cohere.command-r-08-2024
```
For tool parameters, ensure AI models don't control sensitive values:
```yaml
# Dangerous - AI decides the key
parameters:
- name: key
form: llm
# Safe - User provides the key
parameters:
- name: key
form: form
```
This approach prevents breaking existing configurations while maintaining security boundaries around AI decision-making capabilities.
---
## Purpose-reflecting file names
<!-- source: ollama/ollama | topic: Naming Conventions | language: Other | updated: 2025-01-31 -->
File names should clearly communicate their purpose and functionality. Avoid generic, ambiguous, or numbered names (like "Makefile2") that don't convey the file's actual role or lead to recurring questions.
Choose names that:
1. Reflect the file's specific purpose (e.g., "vendor.make" for vendoring operations)
2. Avoid conflicts with convention-based names that carry specific expectations
3. Prevent confusion with automatically generated files
For example, instead of:
```
Makefile2 # What does the "2" mean? What is this for?
```
Use:
```
vendor.make # Clearly indicates this handles vendoring operations
```
This practice enhances code maintainability and reduces onboarding time for new developers, who can understand a file's purpose at a glance rather than having to investigate its contents or ask questions.
---
## Handle async concurrency properly
<!-- source: microsoft/markitdown | topic: Concurrency | language: Python | updated: 2025-01-28 -->
When writing async endpoints, ensure that long-running CPU-bound operations don't block the event loop and that concurrent requests don't conflict over shared resources. CPU-bound operations should be offloaded to thread pools using asyncio.run_in_executor() or similar mechanisms. Additionally, avoid using predictable file paths or shared resources that can cause race conditions between concurrent requests.
Example of problematic code:
```python
@app.post("/convert")
async def convert(file: UploadFile = File(...)):
# Problem 1: Blocking CPU-bound operation
result = markitdown.convert(temp_file_path) # Blocks event loop
# Problem 2: Race condition with shared file path
temp_file_path = f"/tmp/{file.filename}" # Conflicts with concurrent requests
```
Better approach:
```python
@app.post("/convert")
async def convert(file: UploadFile = File(...)):
# Solution 1: Use thread pool for CPU-bound work
loop = asyncio.get_event_loop()
result = await loop.run_in_executor(None, markitdown.convert, temp_file_path)
# Solution 2: Use unique file paths
import uuid
temp_file_path = f"/tmp/{uuid.uuid4()}_{file.filename}"
```
This prevents blocking the async event loop and eliminates race conditions between concurrent requests.
---
## PEP 257 docstring compliance
<!-- source: Unstructured-IO/unstructured | topic: Documentation | language: Python | updated: 2025-01-23 -->
All public modules, classes, methods, and functions must have docstrings that follow PEP 257 formatting standards. This ensures consistent, professional documentation across the codebase.
**Required PEP 257 Format:**
- Use triple double-quotes (`"""`)
- First line must be a complete sentence that fits on one line
- For multi-line docstrings, add a blank line after the summary
- Closing triple-quotes on separate line for multi-line docstrings
- Describe the contract/purpose, not implementation details
**Examples:**
Single-line docstring:
```python
def get_nltk_data_dir() -> str | None:
"""The directory where `nltk` resources are located."""
```
Multi-line docstring:
```python
def clean_pdfminer_inner_elements(document: "DocumentLayout") -> "DocumentLayout":
"""Move pdfminer elements from inside tables to the extra_info dictionary.
Each element appears in the `extra_info` dictionary using its table id as the key.
"""
```
Module docstring:
```python
"""NDJSON file utilities for document processing.
This module provides functionality for reading and writing newline-delimited JSON files.
Used primarily for batch processing of document elements during partitioning workflows.
"""
```
**Coverage Requirements:**
- All public modules should explain their purpose and usage
- All public classes should document their role and key methods
- All public methods and functions require docstrings
- Internal functions benefit from docstrings for maintainability
This standard improves code discoverability, reduces onboarding time, and maintains professional documentation quality throughout the codebase.
---
## Simplify configuration interfaces
<!-- source: LMCache/LMCache | topic: Configurations | language: Markdown | updated: 2025-01-17 -->
Design configuration interfaces that prioritize user experience by providing sensible defaults, using clear naming conventions, and logically grouping related options. Complex tools with many configuration parameters should minimize the cognitive load on users through thoughtful interface design.
Key principles:
1. **Provide sensible defaults**: Allow users to skip commonly-used parameters by implementing reasonable default values that work for typical use cases
2. **Use clear, unambiguous naming**: Avoid parameter names that could be misinterpreted (e.g., `--model-api-name` instead of `--model` when referring to API endpoint naming)
3. **Group related configurations**: Consolidate related parameters into higher-level options (e.g., `--enable-lmcache` instead of multiple LMCache-specific flags)
4. **Improve example readability**: Format complex command examples with line breaks and remove verbose default values
Example of improvement:
```bash
# Before: Complex command with many explicit parameters
python3 rag.py --qps 3.5 --model mistralai/Mistral-7B-Instruct-v0.2 --dataset ~/CacheBlend/inputs/musique_s.json --system-prompt "You will be asked a question..." --query-prompt "Answer the question..." --separator "" --prompt-build-method QA --base-url "http://localhost:8000/v1" --kv-storage-size 30GB --max-tokens 32 --output summary.csv
# After: Simplified with defaults and clear grouping
python3 rag.py \
--qps 3.5 \
--model mistralai/Mistral-7B-Instruct-v0.2 \
--dataset ~/CacheBlend/inputs/musique_s.json \
--enable-lmcache \
--output summary.csv
```
This approach reduces user friction while maintaining the flexibility needed for advanced use cases.
---
## Cross-platform configuration simplification
<!-- source: menloresearch/jan | topic: Configurations | language: Json | updated: 2025-01-13 -->
When writing configuration files and scripts, prioritize cross-platform compatibility while keeping configurations simple and maintainable. Use platform-aware tools like `run-script-os` for npm scripts that need platform-specific behavior, but prefer built-in solutions and global defaults over complex custom configurations.
For npm scripts, combine similar platform commands when possible:
```json
{
"scripts": {
"version-restore": "run-script-os",
"version-restore:darwin:linux": "jq --arg ver $(cat .version.bak) '.version = $ver' package.json > package.tmp && mv package.tmp package.json && rm .version.bak",
"version-restore:win32": "node -e \"/* Windows-specific implementation */\""
}
}
```
Avoid explicitly setting configuration properties that can use sensible defaults. Consider using native tools (like `node --watch`) instead of additional dependencies when they provide equivalent functionality. This reduces complexity, improves maintainability, and ensures consistent behavior across different development environments.
---
## prefer efficient operations
<!-- source: Unstructured-IO/unstructured | topic: Performance Optimization | language: Python | updated: 2025-01-13 -->
Choose efficient operations and algorithms over expensive alternatives, particularly when processing data or performing file operations. Avoid operations that create unnecessary intermediate objects or make repeated system calls when vectorized or batch operations are available.
Key principles:
- Use vectorized operations instead of loops when working with data structures like pandas DataFrames
- Avoid expensive string operations like `split()` when simple slicing suffices
- Minimize repeated system calls by batching operations or using single calls that return multiple results
Examples:
```python
# Instead of computing values in a loop:
for item in items:
item['width'] = item['right'] - item['left']
item['height'] = item['bottom'] - item['top']
# Use vectorized operations:
df['width'] = df['right'] - df['left']
df['height'] = df['bottom'] - df['top']
# Instead of expensive string splitting:
filename = (doc.split("/")[-1]).split(f".{output_type}")[0]
# Use efficient slicing:
filename = os.path.basename(doc)[:-len(output_type)-1]
# Instead of repeated system calls:
while os.path.exists(os.path.join(dir, filename)):
# check and increment
# Use single directory listing:
existing_files = os.listdir(dir)
# then filter and find max counter
```
This approach reduces computational overhead, memory allocation, and system call frequency, leading to better overall performance.
---
## validate configuration defaults
<!-- source: Unstructured-IO/unstructured | topic: Configurations | language: Python | updated: 2025-01-10 -->
Configuration parameters should be properly validated with appropriate defaults and clear handling of edge cases. Avoid using arbitrary defaults when values should be explicitly required, and ensure configuration validation distinguishes between None, empty, and zero values.
Key practices:
1. **Validate parameter types and ranges** - Check that configuration values meet expected constraints
2. **Use meaningful defaults** - Set defaults that make sense for the use case, not just convenient values
3. **Handle None vs zero distinction** - Be explicit about whether 0 is a valid value or should trigger default behavior
4. **Filter unsupported parameters** - Remove configuration options not supported by the current version
5. **Make critical configs required** - Don't provide defaults for parameters that users should explicitly set
Example of proper configuration validation:
```python
@property
def TESSERACT_CHARACTER_CONFIDENCE_THRESHOLD(self) -> float:
"""Tesseract predictions with confidence below this threshold are ignored"""
return self._get_float("TESSERACT_CHARACTER_CONFIDENCE_THRESHOLD", 0.0)
def get_retries_config(self, retries_initial_interval: Optional[int]) -> Optional[retries.RetryConfig]:
# Check for None explicitly, not just falsy values
if retries_initial_interval is not None:
return retries_initial_interval
return DEFAULT_RETRIES_INITIAL_INTERVAL_SEC
# Filter configuration to supported fields only
possible_fields = [f.name for f in fields(PartitionParameters)]
filtered_config = {k: v for k, v in config.items() if k in possible_fields}
```
This prevents configuration errors, improves user experience, and ensures consistent behavior across different environments and versions.
---
## Parameterize hardcoded configuration values
<!-- source: Unstructured-IO/unstructured | topic: Configurations | language: Dockerfile | updated: 2025-01-06 -->
Replace hardcoded values in configuration files with parameterized variables to improve flexibility and maintainability. Use build arguments (ARG) and environment variables (ENV) in Dockerfiles instead of embedding specific versions, paths, or other configuration values directly in the code.
This approach makes configurations more adaptable to different environments and reduces the need to modify multiple locations when values change. For example, instead of hardcoding tool versions and paths throughout a Dockerfile:
```dockerfile
# Instead of hardcoded values:
RUN pip3.11 install --user -r requirements.txt
COPY ./nltk_data /home/notebook-user/nltk_data
# Use parameterized approach:
ARG PYTHON_VERSION=3.11
ARG NLTK_DATA_SOURCE=./nltk_data
ENV PIP=pip${PYTHON_VERSION}
RUN ${PIP} install --user -r requirements.txt
COPY ${NLTK_DATA_SOURCE} /home/notebook-user/nltk_data
```
This pattern applies to any configuration file where values might need to change across different environments or deployments.
---
## Follow configuration best practices
<!-- source: microsoft/markitdown | topic: Configurations | language: Other | updated: 2024-12-19 -->
Always adhere to official guidelines and proper validation when setting up configurations, whether for dependencies, file formats, or environment setup. This prevents subtle issues that can cause tests to pass incorrectly or create environment conflicts.
For dependency installation, use the recommended method from official documentation rather than generic approaches:
```makefile
# Instead of:
install:
pip install hatch
# Use the recommended approach:
install:
pipx install hatch
```
For file configurations, validate that files have the correct format and MIME type, not just the right extension. A .doc file should have MIME type `application/msword`, not `application/octet-stream`. Simply renaming a .docx file to .doc doesn't change its underlying format and can lead to false test results.
This practice ensures reliable, conflict-free environments and prevents configuration-related bugs from propagating through your system.
---
## Validate configuration assumptions
<!-- source: microsoft/markitdown | topic: Configurations | language: Json | updated: 2024-12-17 -->
Before adding or modifying configuration settings, understand the underlying tool capabilities and test your assumptions. Many configuration issues arise from assumptions about what's needed without investigating whether the tool already provides the desired functionality or has specific constraints.
For example, when setting up development containers, research whether build tools like Hatch already provide features you're trying to configure manually:
```json
// Instead of assuming you need postCreateCommand for editable installs
"postCreateCommand": "pip install -e .",
// Understand that Hatch already handles editable mode for testing
"features": {
"ghcr.io/devcontainers-extra/features/hatch:2": {}
}
```
Similarly, when configuration choices seem to conflict with best practices (like using root user), document the practical constraints that necessitate the decision rather than just following defaults. Test alternative approaches when possible, but accept necessary trade-offs when tools have specific requirements for proper functionality.
---
## Manage side effects properly
<!-- source: langgenius/dify | topic: React | language: TSX | updated: 2024-12-05 -->
Avoid performing side effects directly in component render functions. Use useEffect for side effects like API calls, DOM manipulations, or state changes that should occur after render. Wrap functions with useCallback when they need to be optimized or used as dependencies, and use async/await for proper asynchronous handling.
Example of incorrect usage:
```tsx
const I18n: FC<II18nProps> = ({ locale, children }) => {
// ❌ Side effect in render
locale && changeLanguage(locale)
return <I18NContext.Provider>...</I18NContext.Provider>
}
```
Example of correct usage:
```tsx
const I18n: FC<II18nProps> = ({ locale, children }) => {
// ✅ Side effect in useEffect
useEffect(() => {
locale && changeLanguage(locale)
}, [locale])
// ✅ Function wrapped with useCallback and async/await
const handleRender = useCallback(async () => {
await renderFlowchart(code)
}, [code])
return <I18NContext.Provider>...</I18NContext.Provider>
}
```
This ensures predictable component behavior, prevents unnecessary re-renders, and maintains React's rendering principles.
---
## extensible parameter design
<!-- source: Unstructured-IO/unstructured | topic: API | language: Python | updated: 2024-10-23 -->
Design API parameters to support future extension without breaking changes. Use string enums instead of boolean flags when the parameter might need additional values later, implement comprehensive parameter forwarding with **kwargs, and avoid overly specific parameter designs that limit future flexibility.
Key patterns to follow:
1. **String enums over booleans**: Instead of `contains_ontology_schema: bool = False`, use `html_parser: str = "v1"` to allow future parser versions without new parameters.
2. **Comprehensive parameter forwarding**: Use `**kwargs` to forward all parameters to delegate functions rather than explicitly listing each one:
```python
# Preferred approach
elements = _partition_doc(filename, file=file, **kwargs)
# Avoid this maintenance-heavy pattern
elements = _partition_doc(
filename=filename,
file=file,
infer_table_structure=infer_table_structure,
languages=languages,
strategy=strategy,
**kwargs,
)
```
3. **Flexible interface design**: Accept the most general form of input (e.g., always expect text rather than having separate file/text/stdin branches) to reduce conditional complexity.
This approach ensures APIs can evolve gracefully, reduces maintenance overhead when adding new parameters, and provides a consistent experience for API consumers. All partitioners automatically receive new parameters through kwargs without requiring explicit updates to forwarding logic.
---
## maintain established naming conventions
<!-- source: Unstructured-IO/unstructured | topic: Naming Conventions | language: Json | updated: 2024-10-22 -->
Ensure all naming follows established patterns and conventions already present in the codebase. When introducing new identifiers or modifying existing ones, they must conform to the team's built-in expectations and standard patterns to maintain consistency across the system.
This prevents violations of established conventions that can break compatibility or create confusion. For example, if element IDs are expected to be formatted as hashes without dashes (like "782cf07be8b3ab8f05188e479edb7f61"), then all element IDs should follow this pattern. Similarly, if certain class names violate standard output patterns used throughout the system, they should be removed or renamed to maintain consistency.
Example of applying this standard:
```json
// Bad - violates hash convention with dashes
"element_id": "897a8a47-377c-4ad6-aab8-39a929879537"
// Good - follows established hash format
"element_id": "897a8a47377c4ad6aab839a929879537"
```
Before introducing any new naming patterns, verify they align with existing conventions in the codebase and don't break established expectations that other parts of the system depend on.
---
## Enhance documentation detail
<!-- source: Unstructured-IO/unstructured | topic: Documentation | language: Markdown | updated: 2024-10-11 -->
Documentation should provide comprehensive detail that enables users to understand impacts and developers to understand implementation. For breaking changes, include migration guidance, deprecated features, and specific functional changes. For technical documentation, provide granular step-by-step explanations that trace execution flow.
For breaking changes in changelogs:
- Advise where to find replacement functionality
- List all deprecated extras/features
- Specify which submodules or functionalities are removed
- Include related implementation updates
For developer guides:
- Diagram and explain the complete execution journey
- Trace granularly at function level (e.g., "incoming request -> constructing command -> building pipeline -> executing pipeline")
- Enable developers to understand where code fits and how to debug
Example from changelog:
```markdown
## 0.16.0
### Breaking Changes
* **Remove ingest implementation**
- Migration: Use new unstructured-ingest library (link to installation guide)
- Deprecated extras: s3, local-inference, github (see migration guide)
- Removed submodules: unstructured.ingest.v1.*, unstructured.ingest.connectors.*
- Related: embed submodule updated with SecretStr for secret handling
```
---
## Avoid external URLs
<!-- source: menloresearch/jan | topic: Security | language: Markdown | updated: 2024-09-16 -->
Links to external repositories or domains in documentation and code can create security vulnerabilities through URL hijacking attacks. Malicious actors could potentially gain control of external domains and redirect users to harmful content or credential harvesting sites.
Always use official, internal, or trusted repository URLs instead of external ones. When linking to documentation, resources, or code examples, ensure the URLs point to your organization's official repositories or well-established, trusted sources.
Example of the security risk:
```markdown
<!-- Vulnerable - external repository -->
- <a href="https://github.com/NHPT/jan/blob/dev/README_CN.md">简体中文</a>
<!-- Secure - official repository -->
- <a href="https://github.com/janhq/jan/blob/dev/README_CN.md">简体中文</a>
```
This practice is especially critical in README files, documentation, and any user-facing content where links could be used to redirect users to potentially malicious sites.
---
## Environment variable patterns
<!-- source: Unstructured-IO/unstructured | topic: Configurations | language: Shell | updated: 2024-09-13 -->
Use consistent patterns for environment variable handling in shell scripts. Always provide sensible defaults using the `${VAR:-default}` syntax, validate required variables early with clear error messages, and preserve the ability to override configuration values.
For variables with defaults:
```bash
strategies=${STRATEGIES:=fast,hi_res,ocr_only}
RUN_SCRIPT=${RUN_SCRIPT:-unstructured-ingest}
PYTHONPATH=${PYTHONPATH:-.}
```
For required variables, validate early and provide helpful error messages:
```bash
if [ -z "$VECTARA_OAUTH_CLIENT_ID" ]; then
echo "Skipping VECTARA ingest test because VECTARA_OAUTH_CLIENT_ID env var is not set."
exit 0
fi
```
Document configurable environment variables in script comments or usage instructions to make them discoverable for users who aren't familiar with shell scripting. This approach allows scripts to work out-of-the-box with reasonable defaults while remaining flexible for different environments and use cases.
---
## Use centralized model state
<!-- source: menloresearch/jan | topic: AI | language: TSX | updated: 2024-09-10 -->
Always access AI model information, parameters, and configuration through centralized state management rather than direct access or hardcoded values. This ensures consistency, maintainability, and proper separation of concerns when working with AI models.
For model access, use established state management patterns:
```typescript
// Good: Use centralized state
const model = currentActiveModel
// Avoid: Direct model access
const model = message.model
```
For model parameters (like llama.cpp settings), define constraints centrally rather than scattered throughout components:
```typescript
// Good: Centralized parameter definitions
const llamaParams = {
temperature: { min: 0, max: 2, step: 0.1 },
maxTokens: { min: 1, max: 4096, step: 1 }
}
// Avoid: Hardcoded values in components
<SliderRightPanel min={0} max={100} step={1} />
```
This approach prevents inconsistencies, makes parameter updates easier, and provides a single source of truth for AI model configuration across the application.
---
## Document configuration parameters clearly
<!-- source: Unstructured-IO/unstructured | topic: Configurations | language: Markdown | updated: 2024-08-13 -->
When documenting configuration objects and their parameters, provide clear explanations of each parameter's purpose, whether it's required or optional, and why optional parameters are useful for development. Use backticks to format code symbols, parameter names, class names, and error types for better readability.
For example, when documenting configuration like:
```python
context=ProcessorConfig(
verbose=True, # Optional: enables detailed logging for debugging
work_dir="local-working-dir", # Required: working directory for pipeline
reprocess=True, # Optional: forces reprocessing of existing files
re_download=True, # Optional: forces re-downloading of source files
)
```
Explain that `verbose`, `reprocess`, and `re_download` are optional parameters that make development easier by providing debugging information and forcing fresh processing during iterative development.
In documentation text, wrap configuration-related terms in backticks: "Sets minimum version of `unstructured-client` to avoid raising a `TypeError` when passing `api_key_auth` to `UnstructuredClient`"
---
## Use descriptive specific names
<!-- source: Unstructured-IO/unstructured | topic: Naming Conventions | language: Markdown | updated: 2024-08-13 -->
Choose names that clearly indicate their context and purpose rather than generic terms. Generic names can be ambiguous and make code harder to understand, especially when the same codebase handles multiple contexts or domains.
When naming variables, parameters, methods, or classes, prefer specific descriptive names that include relevant context over broad generic terms. This makes the code self-documenting and reduces the cognitive load for other developers.
Examples of improvements:
- Use `library` instead of `repo` when referring to a software library specifically
- Use `pdf_hi_res_max_pages` instead of `max_pages` when the parameter specifically limits PDF processing in high-resolution mode
- Use `source` and `destination` (lowercase) instead of capitalized `Source` and `Destination` for common technical terms
The goal is to make names immediately clear about what they represent without requiring additional context or documentation.
---
## intentional naming patterns
<!-- source: Unstructured-IO/unstructured | topic: Naming Conventions | language: Other | updated: 2024-08-12 -->
Recognize when seemingly redundant naming patterns serve specific technical purposes rather than being mistakes or oversights. Some naming conventions that appear duplicative are actually intentional and necessary for technical reasons.
Common examples include:
- Import aliases in type stubs: `from ._element import HtmlElement as HtmlElement` - This pattern avoids unused import errors when republishing imports while maintaining explicit exports
- Function overloads: Multiple functions with identical names but different signatures represent the same logical operation with different interfaces
- Alternative approaches like `__all__ = ["HtmlElement"]` exist but may be less compact or appropriate for the specific context
Before suggesting changes to apparently redundant naming, verify whether the pattern serves a technical requirement such as type checking, import management, or API design. Understanding the underlying purpose helps distinguish between genuine redundancy and intentional technical patterns.
```python
# Intentional - republishing import in type stub
from ._element import HtmlElement as HtmlElement
# Intentional - same logical function, different signatures
@overload
def strip_elements(tree: _ElementOrTree, tags: Collection[_TagSelector]) -> None: ...
@overload
def strip_elements(tree: _ElementOrTree, *tags: _TagSelector) -> None: ...
```
---
## Use configuration constants
<!-- source: menloresearch/jan | topic: Configurations | language: TSX | updated: 2024-08-07 -->
Always use named constants, enums, or well-defined configuration objects instead of magic strings, numbers, or inline literals for configuration values. This improves maintainability, reduces typos, enables better IDE support, and makes configuration changes easier to track.
Examples of good practices:
- Use enums for status values: `status === EngineStatus.ready` instead of `status === 'ready'`
- Define constants for localStorage keys: `const EXPERIMENTAL_FEATURE = 'experimentalFeature'` instead of inline strings
- Use declarative configuration arrays with proper defaults: `localStorage.getItem(HTTPS_PROXY_FEATURE) ?? ""`
- Avoid optional parameters for configuration that always has a value: `serverEnabled: boolean` instead of `serverEnabled?: boolean`
This approach makes configuration management more robust and prevents runtime errors from typos in configuration keys or values.
---
## Document configuration reasoning
<!-- source: Unstructured-IO/unstructured | topic: Configurations | language: Other | updated: 2024-08-07 -->
Always document the reasoning behind configuration decisions, especially for temporary settings like dependency pins, parameter choices, and version constraints. Include explanations of why the configuration was needed, what problem it solves, and when it might be safe to remove or update.
For dependency pins, explain the specific compatibility issue or version conflict being resolved:
```python
# tenacity 9.0.0 is being installed via chroma, but other dependencies (langchain) restrict tenacity
# to <=9.0.0 in other dependency extras and resolve to 8.5.0.
# The original langchain pin: https://github.com/langchain-ai/langchain/pull/849/
tenacity==8.5.0
```
For API parameters, document when parameters conflict or when one supersedes another to avoid confusion. Remove deprecated or conflicting configuration options rather than documenting both. This practice enables future maintainers to understand when temporary configurations can be safely removed and helps prevent accumulation of unnecessary constraints.
---
## Complete documentation coverage
<!-- source: ollama/ollama | topic: Documentation | language: Markdown | updated: 2024-08-07 -->
Documentation should comprehensively cover all supported options, variations, and potential future use cases. When creating documentation:
1. Use generic titles and structures that can accommodate future additions
- Example: Instead of "AMD iGPU 780m Guide", use "AMD iGPU Guide" to allow for documenting additional GPU models
2. Include examples for all supported options and environments
- Example: When documenting shell commands, provide examples for all supported shells:
```
# For bash
echo "source <(ollama completion bash)" >> ~/.bashrc
# For zsh
echo "source <(ollama completion zsh)" >> ~/.zshrc
# For fish
ollama completion fish > ~/.config/fish/completions/ollama.fish
```
3. Document complete workflows from beginning to end
4. Verify that all links are current and functional
Comprehensive documentation reduces the need for clarifications, helps users across different environments, and ensures documentation remains relevant as software evolves.
---
## Use descriptive names
<!-- source: menloresearch/jan | topic: Naming Conventions | language: TypeScript | updated: 2024-07-30 -->
Choose variable and function names that clearly communicate their purpose, type, and behavior. Names should be self-documenting and immediately understandable to other developers without requiring additional context.
For boolean variables and functions, use names that clearly indicate they return true/false values:
- Prefix with `is`, `has`, `can`, `should`, or similar indicators
- Use descriptive terms that make the boolean nature obvious
For properties and variables, prefer generic, maintainable names over product-specific ones, especially in open source projects where the codebase may be forked or renamed.
Examples:
```typescript
// Poor naming - unclear purpose
const SOME_API_KEY_ADDED = 'someApiKeyAdded'
const checkFileExists = (path: string) => boolean
// Better naming - clear purpose and type
const IS_ANY_REMOTE_MODEL_CONFIGURED = 'isAnyRemoteModelConfigured'
const exists = (path: string) => boolean
// Poor naming - product-specific
const properties = { JanVersion: VERSION }
// Better naming - generic and maintainable
const properties = { appVersion: VERSION }
```
This approach improves code readability, reduces the need for comments, and makes the codebase more maintainable for both current and future developers.
---
## Database schema compliance
<!-- source: Unstructured-IO/unstructured | topic: Database | language: Python | updated: 2024-07-03 -->
Ensure database integrations use correct field names, data types, and schema conventions specific to each database system. Different databases have special field naming requirements and data type expectations that must be respected to avoid runtime errors and ensure proper functionality.
Key considerations:
- Research database-specific field naming conventions (e.g., `$vector` for embeddings in Astra DB, `_id` for document identifiers)
- Convert data types appropriately for database storage (e.g., timestamps as integers for file modification times)
- Validate schema requirements during development and testing
- Use database-native field names in normalization methods
Example from Astra DB integration:
```python
def normalize_dict(self, element_dict: dict) -> dict:
return {
"_id": str(uuid.uuid4()), # Astra DB expects _id, not id
"$vector": element_dict.pop("embeddings", None), # Special $vector field
"content": element_dict.pop("text", None),
"metadata": element_dict,
}
```
This prevents issues like vector search failures due to incorrect field naming and ensures data is stored in the format expected by the target database system.
---
## improve code readability
<!-- source: Unstructured-IO/unstructured | topic: Code Style | language: Shell | updated: 2024-07-02 -->
Format code to enhance readability and maintainability by breaking long lines and extracting repeated values into variables. Long command lines with multiple parameters should be split across multiple lines with proper line continuation, placing each parameter on its own line. When the same value (like URLs, paths, or configuration strings) appears multiple times, extract it into a clearly named variable at the top of the file.
Example of improved formatting:
```bash
# Extract repeated URL base
BASE_URL="https://ingest-test-azure-cognitive-search.search.windows.net"
# Break long command lines for readability
PYTHONPATH=${PYTHONPATH:-.} "$RUN_SCRIPT" \
--reprocess \
--input-path example-docs/fake-memo.pdf \
--work-dir "$WORK_DIR" \
--chunking-strategy by_title \
--chunk-combine-text-under-n-chars 150 \
--chunk-new-after-n-chars 1500 \
--chunk-max-characters 2500 \
--chunk-multipage-sections \
--chunk-no-include-orig-elements
```
This approach reduces duplication, makes the code easier to modify, and significantly improves readability by avoiding overly long lines that are difficult to scan and understand.
---
## avoid hardcoded secrets
<!-- source: Unstructured-IO/unstructured | topic: Security | language: Python | updated: 2024-06-27 -->
Never include real API keys, passwords, or other sensitive credentials directly in source code, including example files and configuration. This creates security vulnerabilities and risks credential exposure in version control systems.
For example code and documentation, use clear placeholder text that indicates where credentials should be provided:
```python
# Bad - real API key hardcoded
access_config=PineconeAccessConfig(api_key="0e8555a2-b944-4da4-837e-03b1202e00c7")
# Good - clear placeholder
access_config=PineconeAccessConfig(api_key="<YOUR_PINECONE_API_KEY_HERE>")
# Better - environment variable
access_config=PineconeAccessConfig(api_key=os.getenv("PINECONE_API_KEY"))
```
When handling credentials in configuration classes, mark sensitive fields appropriately to ensure they are handled securely:
```python
@dataclass
class SqlAccessConfig(AccessConfig):
username: str # Not sensitive, needed for access
password: t.Optional[str] = enhanced_field(sensitive=True) # Mark as sensitive
```
Use environment variables, secure configuration files, or credential management systems for real deployments.
---
## Python version standardization
<!-- source: Unstructured-IO/unstructured | topic: Configurations | language: Txt | updated: 2024-05-22 -->
Establish and document a consistent Python version standard for dependency compilation across the project. Team confusion about which Python version to use for pip-compile leads to build inconsistencies, compatibility issues, and wasted development time.
The discussions reveal conflicting approaches: some team members believe dependencies should be compiled with the minimum supported Python version (3.9) for maximum compatibility, while others use different versions (3.10). This inconsistency causes build failures and requires rework.
**Implementation:**
1. Document the official Python version for pip-compile in your project's README or contributing guidelines
2. Add version checks to your Makefile or build scripts to prevent compilation with incorrect Python versions
3. Include the Python version requirement in CI/CD pipeline documentation
**Example from the discussions:**
```
# Error when wrong Python version is used:
AssertionError
python version not equal to expected 3.9: Python 3.10.13
make: *** [pip-compile] Error 1
```
Ensure all team members understand and follow the same Python version standard to maintain build reproducibility and avoid compatibility issues across different environments.
---
## Centralize configuration sources
<!-- source: Unstructured-IO/unstructured | topic: Configurations | language: Toml | updated: 2024-05-16 -->
Maintain configuration settings in a single authoritative source to avoid duplication and inconsistencies across multiple files. When possible, use `pyproject.toml` as the primary configuration file and remove redundant specifications from other files like `.pre-commit-config.yaml` and `Makefile`.
This approach follows the DRY (Don't Repeat Yourself) principle and ensures that configuration changes only need to be made in one place, reducing maintenance overhead and preventing configuration drift.
Example:
```toml
# pyproject.toml - Single source of truth
[tool.ruff.lint]
select = [
"UP018",
"UP032",
"UP034",
"W", # -- Warnings, including invalid escape-sequence --
]
ignore = [
"COM812",
"PT005", # -- flags mock fixtures with names intentionally matching private method name --
"PT011",
]
```
Remove redundant ruff configurations from `.pre-commit-config.yaml` and `Makefile` so they automatically pick up settings from `pyproject.toml`.
---
## Validate API parameter formats
<!-- source: Unstructured-IO/unstructured | topic: API | language: Shell | updated: 2024-05-15 -->
Ensure API parameters match the expected format and structure of target APIs before runtime. Validate parameter types, required fields, and data structures during development rather than discovering mismatches through runtime errors.
When integrating with external APIs, explicitly validate that:
- Parameter formats match API specifications (e.g., single values vs arrays)
- Authentication methods are clearly defined through parameters rather than implicit environment setup
- Required fields and data structures are properly formatted
Example of problematic parameter passing:
```bash
# Incorrect - API expects array but single value provided
--requested-indexing-policy '{"deny": "metadata"}'
# Correct - API expects array format
--requested-indexing-policy '{"deny": ["metadata"]}'
```
Additionally, make authentication explicit through parameters:
```bash
# Better - explicit parameter
--embedding-api-key "$VERTEX_API_KEY"
# Rather than relying on implicit environment variables
# GOOGLE_APPLICATION_CREDENTIALS (less obvious to users)
```
This prevents runtime failures and makes API integrations more maintainable and user-friendly.
---
## Write clear test cases
<!-- source: Unstructured-IO/unstructured | topic: Testing | language: Python | updated: 2024-04-24 -->
Structure test cases for maximum readability and maintainability by following these principles:
1. Organize each test in three distinct parts separated by blank lines:
- Setup (arrange test fixtures)
- Execution (run unit under test)
- Verification (assert expected results)
2. Minimize intermediate variables - use direct assertions when possible:
Instead of:
```python
matrix = [["cell1", "cell2"], ["cell3", "cell4"]]
expected_html = "<table><tr><td>cell1</td>...</table>"
assert utils.htmlify_matrix(matrix) == expected_html
```
Prefer:
```python
assert utils.htmlify_matrix([["cell1", "cell2"], ["cell3", "cell4"]]) == (
"<table><tr><td>cell1</td><td>cell2</td></tr>"
"<tr><td>cell3</td><td>cell4</td></tr></table>"
)
```
3. Use parameterization to avoid repeated test code:
```python
@pytest.mark.parametrize(
"date", ["1990-12-01", "2050-01-01T00:00:00", "2050-01-01+00:00:00"]
)
def test_validate_date_args_accepts_standard_formats(date):
assert utils.validate_date_args(date)
```
4. Write meaningful assertions that clearly indicate the behavior being tested:
```python
# Weak assertion:
assert "error" in caplog.text
# Better assertion:
assert "invalid date format, expected YYYY-MM-DD" in caplog.text
```
---
## model description accuracy
<!-- source: menloresearch/jan | topic: AI | language: Json | updated: 2024-04-22 -->
Ensure AI model descriptions are specific, capability-focused, and avoid marketing language or temporal references like "latest". Descriptions should clearly communicate what the model excels at and its intended use cases to help users make informed selection decisions.
Replace placeholder or generic text with meaningful descriptions that highlight actual capabilities. Avoid hyperbolic statements and focus on practical information.
Example of good description:
```json
{
"description": "Meta's Llama 3 excels at general usage situations, including chat, general world knowledge, and coding."
}
```
Instead of:
```json
{
"description": "The latest model from MetaAI"
}
```
Be concise and pragmatic - "CodeNinja is finetuned on openchat/openchat-3.5-1210. Good for coding." is better than "CodeNinja is an enhanced version of the renowned model... designed to be an indispensable tool for coders".
---
## embedding provider configuration
<!-- source: Unstructured-IO/unstructured | topic: AI | language: Shell | updated: 2024-03-26 -->
When configuring embedding providers in AI integrations, ensure proper dimension specification and use dedicated test files for different providers. Always specify the embedding dimension that matches your chosen provider (e.g., 384 for huggingface embedders) and avoid modifying existing configurations for testing purposes.
Instead of changing embedding providers in existing files for testing, create separate test files for each provider. This prevents disruption to working configurations and provides clearer test isolation.
Example of proper configuration:
```bash
PYTHONPATH=. ./unstructured/ingest/main.py \
--embedding-provider "langchain-huggingface" \
--embedding-dimension 384 \
astra \
--token "$ASTRA_DB_APPLICATION_TOKEN" \
--collection-name "$COLLECTION_NAME"
```
For testing new providers, create dedicated test files like `test_unstructured_ingest/src/local-embed-vertexai.sh` rather than modifying existing provider configurations. This approach maintains stability while enabling comprehensive testing of different AI embedding services.
---
## separate formatting changes
<!-- source: Unstructured-IO/unstructured | topic: Code Style | language: Other | updated: 2024-03-22 -->
Keep formatting and style changes separate from functional code changes to maintain clear, reviewable diffs. When making functional changes, avoid simultaneously reordering imports, adjusting whitespace, or making other cosmetic modifications that obscure the actual logic changes. If formatting improvements are needed, submit them as standalone pull requests.
This practice ensures that code reviewers can focus on the core logic changes without being distracted by formatting noise. It also makes it easier to track the history of functional changes and reduces the likelihood of merge conflicts.
Example of what to avoid:
```python
# Don't mix functional changes with formatting changes
@@ -1,19 +1,20 @@
-c "constraints.in"
+backoff # functional change
+# Also reordering other dependencies (formatting change)
```
Instead, submit formatting changes like dependency reordering as separate commits or pull requests. This keeps the diff focused on the actual functional change (adding the `backoff` dependency) rather than mixing it with unrelated organizational changes.
---
## Use platform-specific endpoints
<!-- source: menloresearch/jan | topic: API | language: Other | updated: 2024-03-07 -->
When configuring API integrations, always use the actual platform-specific endpoints and identifiers rather than generic placeholders. Generic configurations often lead to integration failures and confusion for users following documentation.
Ensure that:
- API endpoints point to the correct service-specific URLs
- Model IDs match the exact identifiers used by the target platform
- Configuration examples reflect real, working values
For example, when integrating with Groq API:
```json
{
"full_url": "https://api.groq.com/openai/v1/chat/completions",
"api_key": "<your-groq-api-key>"
}
```
And use actual model IDs from the platform:
```json
{
"id": "mixtral-8x7b-32768",
// not generic "groq"
}
```
This practice prevents integration failures and ensures documentation accurately reflects working configurations that users can directly implement.
---
## API interface consistency
<!-- source: menloresearch/jan | topic: API | language: TypeScript | updated: 2024-02-23 -->
Ensure that API function signatures, return types, and interface implementations are consistent and properly declared. When modifying functions to be asynchronous or changing their return types, update both the implementation and any related type declarations to maintain contract integrity.
Key principles:
- Function signatures must match their actual implementation (async functions should be declared as returning Promise)
- Interface implementations should be coherent and focused - prefer implementing multiple feature interfaces rather than extending multiple parent classes
- Runtime type checking considerations should be addressed when designing extensible API interfaces
Example of proper async function declaration:
```typescript
// Good: Declaration matches implementation
async fileStat(path: string, outsideJanDataFolder?: boolean): Promise<FileStat | undefined> {
// async implementation
}
// Good: Focused interface implementation
export default class JSONConversationalExtension
implements ThreadExtension, MessageExtension {
// implements specific feature contracts
}
```
This ensures API consumers can rely on consistent contracts and prevents runtime errors from signature mismatches.
---
## Document credential formats
<!-- source: Unstructured-IO/unstructured | topic: Security | language: Other | updated: 2024-02-14 -->
When documenting security-sensitive parameters such as private keys, certificates, API tokens, or other authentication credentials, always specify the expected format including any required headers, footers, and structural elements. This prevents configuration errors that could lead to authentication failures or security vulnerabilities.
For example, when documenting a private key parameter, clarify the format:
```
- ``Private Key (PEM)`` (*required*): Input the Private Key associated with the Consumer Key.
The key should begin with -----BEGIN RSA PRIVATE KEY----- and end with -----END RSA PRIVATE KEY-----.
```
This practice helps users correctly configure security credentials and reduces the risk of authentication issues or security misconfigurations that could compromise system security.
---
## Ensure proper event sequencing
<!-- source: menloresearch/jan | topic: Concurrency | language: TypeScript | updated: 2024-02-09 -->
When working with event-driven systems, ensure that events are emitted at the appropriate time and in the correct context to prevent race conditions and incorrect behavior. Consider whether the function's purpose aligns with event emission - single-response functions should not broadcast events meant for streaming contexts. Additionally, be mindful of timing when handling sequential operations to avoid race conditions.
For example, avoid emitting events in non-streaming contexts:
```typescript
// Bad: Emitting events in single-response function
const message = {
status: MessageStatus.Pending,
// ... other properties
};
events.emit(MessageEvent.OnMessageResponse, message); // Inappropriate for single-response
// Good: Only emit events in appropriate streaming contexts
if (isStreamingMode) {
events.emit(MessageEvent.OnMessageResponse, message);
}
```
When handling sequential operations, ensure proper timing:
```typescript
// Bad: Potential race condition
if (messages.length == 1) {
// Handle first message logic here - race condition possible
}
// Good: Handle after message response to avoid race
// Move to EventHandler > OnMessageResponse to ensure proper sequencing
```
---
## API examples completeness
<!-- source: Unstructured-IO/unstructured | topic: API | language: Other | updated: 2024-02-06 -->
Ensure all API code examples include all required parameters with correct placeholder values. Incomplete examples lead to user confusion and implementation errors when developers copy-paste documentation snippets.
When documenting API usage, always include:
- All required parameters, not just the most obvious ones
- Correct placeholder formats (e.g., use `<<REPLACE WITH YOUR API KEY>>` for keys, not URLs)
- Clear distinction between different parameter types (URLs vs keys vs other values)
Example of complete API documentation:
```python
elements = partition_via_api(
filename=filename,
api_key="<<REPLACE WITH YOUR API KEY>>",
api_url="https://<<REPLACE WITH YOUR API URL>>/general/v0/general"
)
```
Rather than incomplete examples missing the `api_key` parameter or using incorrect placeholder values. Consider providing separate examples for different usage scenarios (SaaS vs self-hosted) when the parameter requirements differ significantly.
---
## AI model configuration completeness
<!-- source: menloresearch/jan | topic: AI | language: TypeScript | updated: 2024-02-03 -->
Ensure all necessary AI model parameters are properly configured and passed through the system. AI models require comprehensive configuration including hardware settings (CPU/GPU allocation, thread counts), model-specific parameters (context length, n_gpu_layers), and proper path resolution. Missing configuration can lead to runtime failures, suboptimal performance, or initialization errors.
Key requirements:
- Include hardware settings: CPU threads, GPU layers (ngl), memory allocation
- Specify model parameters: context length (ctx_len), prompt templates, embedding settings
- Use proper path resolution instead of hardcoded paths
- Validate configuration completeness before model initialization
Example of complete model configuration:
```typescript
const modelSettings = {
llama_model_path: await joinPath([modelsDir, model.id]),
ctx_len: model.settings.ctx_len || 2048,
ngl: model.settings.ngl || 100,
cpu_threads: nitroResourceProbe.numCpuPhysicalCore,
prompt_template: model.settings.prompt_template,
embedding: true
};
// Validate before initialization
if (!modelSettings.llama_model_path || !modelSettings.cpu_threads) {
throw new Error('Incomplete model configuration');
}
```
This prevents critical issues like models failing to load due to missing parameters or processes spawning before proper configuration is established.
---
## Use centralized logging framework
<!-- source: menloresearch/jan | topic: Logging | language: TypeScript | updated: 2024-02-03 -->
Always use the established logging infrastructure instead of creating custom logging solutions or multiple logging systems. This prevents fragmentation and ensures consistent log management across the codebase.
When you need logging functionality, leverage the existing logging framework (like `@janhq/core/node` log function) rather than implementing ad-hoc solutions. If additional features like log levels are needed, extend the central framework rather than creating parallel systems.
Avoid patterns like:
```typescript
// Don't create custom debug wrappers
export const debugInspectorSync = process.env.DEBUG ? debugInspector(setBinPath) : setBinPath
// Don't implement separate file logging
subprocess.stderr.on("data", (data) => {
fs.writeFileSync("nitro-err.txt", data); // Avoid this
});
```
Instead, use the centralized approach:
```typescript
import { log } from "@janhq/core/node";
// Use the established logging system
log("Debug info", data);
```
This approach ensures all logging goes through a unified system that can handle log routing, formatting, and management consistently.
---
## externalize configuration values
<!-- source: menloresearch/jan | topic: Configurations | language: JavaScript | updated: 2024-01-11 -->
Configuration values, especially sensitive data like API keys and environment-specific settings, should be externalized to environment variables rather than hardcoded in source code. This prevents sensitive information from being permanently stored in git history and enables proper configuration management across different environments.
Hardcoded values create security risks and operational inflexibility. When values are committed to version control, they become part of the permanent history and may require credential rotation if exposed.
Example of the problem:
```javascript
// Bad - hardcoded sensitive values
algolia: {
appId: "Y8QU1SIVLP",
apiKey: "484787878bcf6f4a26834105f0855fa3",
},
googleTagManager: {
containerId: "GTM-59R6474K",
}
```
Better approach:
```javascript
// Good - use environment variables
algolia: {
appId: process.env.ALGOLIA_APP_ID || "default_value",
apiKey: process.env.ALGOLIA_API_KEY || "default_value",
},
googleTagManager: {
containerId: process.env.GTM_CONTAINER_ID,
}
```
This approach requires setting up proper environment variable management and updating deployment configurations, but provides better security and flexibility for different environments.
---
## Concise documentation writing
<!-- source: menloresearch/jan | topic: Documentation | language: Markdown | updated: 2023-12-29 -->
Write documentation that respects user intelligence by being concise and avoiding over-explanation of obvious concepts. Assume users are "lazy but smart" - they don't need explanations like "A thread title is a title of a thread" but do need clear directions to find what they need.
Avoid unnecessary step-by-step instructions like "click this, click that" unless referring to confusing external systems. Instead, provide the right directions and trust users' ability to figure out basic interactions.
Adopt a tone that is:
- Concise and matter-of-fact
- Not salesy or superfluous
- Show don't tell
Example of what to avoid:
```markdown
### Setting Thread Title
A thread title acts as a name for your thread. It appears on the left side of the chat window, helping you easily navigate through your interactions.
```
Better approach:
```markdown
### Setting Thread Title
Thread titles appear on the left sidebar for easy navigation between conversations.
```
When writing installation guides, focus on essential information rather than obvious steps. Provide clear directions to download locations and system requirements, but don't walk users through basic file operations they already know.
---
## Configuration structure clarity
<!-- source: menloresearch/jan | topic: Configurations | language: Markdown | updated: 2023-12-26 -->
Ensure configuration parameters are properly structured, accurately documented, and clearly indicate their scope and purpose. Configuration hierarchies should be explicit (global, assistant-level, thread-level), parameter categories should be well-defined (init vs runtime), and documented values must match actual system behavior.
Key practices:
- Categorize parameters by their lifecycle and mutability (e.g., `init` for initialization-time settings, `runtime` for dynamic updates)
- Document configuration scope clearly (global defaults, assistant-level overrides, thread-specific settings)
- Verify documented specifications match actual behavior and update when changes occur
- Provide clear examples and visual aids for complex configuration hierarchies
Example structure:
```json
{
"parameters": {
"init": {
"model_path": "/path/to/model",
"max_memory": "8GB"
},
"runtime": {
"temperature": 0.7,
"max_tokens": 2048
}
}
}
```
This prevents user confusion and ensures configurations work as documented across different scopes and contexts.
---
## Externalize hardcoded configurations
<!-- source: menloresearch/jan | topic: Configurations | language: TypeScript | updated: 2023-12-04 -->
Avoid hardcoding configuration values like ports, URLs, API endpoints, and environment-specific settings directly in source code. Instead, externalize these values to environment variables, configuration files, or dependency injection to improve maintainability and deployment flexibility.
Hardcoded configurations make applications difficult to deploy across different environments and create tight coupling between code and environment-specific values.
**What to externalize:**
- Server ports and host addresses
- API endpoints and URLs
- Database connection strings
- Feature flags and environment-specific settings
- API keys and secrets
**Example:**
```typescript
// ❌ Avoid: Hardcoded in source
const PORT = 1337;
const LOCAL_HOST = "127.0.0.1";
const JAN_HTTP_SERVER_URL = `http://${LOCAL_HOST}:${PORT}`;
// ✅ Better: Use environment variables
const PORT = process.env.JAN_API_PORT || 1337;
const LOCAL_HOST = process.env.JAN_HOST || "127.0.0.1";
const JAN_HTTP_SERVER_URL = `http://${LOCAL_HOST}:${PORT}`;
```
Move configuration values to `.env` files, environment variables, or inject them from the application layer rather than defining them in core modules. This enables different configurations for development, testing, and production environments without code changes.
---
## Document AI infrastructure requirements
<!-- source: menloresearch/jan | topic: AI | language: Markdown | updated: 2023-11-16 -->
When documenting AI applications that run local models or perform inference, always provide comprehensive infrastructure requirements including hardware specifications, platform-specific drivers, and setup instructions. This ensures users can successfully run AI models without encountering compatibility issues.
Include the following details:
- **Hardware requirements**: Specify RAM/VRAM needs for different model sizes (e.g., "8GB RAM/VRAM for 3B models, 16GB for 7B models")
- **CPU/GPU architecture support**: List supported architectures (ARM, x86 for CPU; NVIDIA, AMD, Intel for GPU)
- **Platform-specific drivers**: Include GPU driver requirements (NVIDIA drivers for Windows, CUDA Toolkit for Linux)
- **Model size relationships**: Explain how hardware capacity relates to usable model sizes
Example documentation structure:
```markdown
### Hardware Requirements
- **RAM/VRAM**: 8GB minimum (3B models), 16GB recommended (7B models)
- **CPU**: ARM, x86 architectures supported
- **GPU**: NVIDIA (via llama.cpp), AMD and Intel support coming soon
### Platform Setup
- **Windows**: Install NVIDIA drivers if GPU available
- **Linux**: Install CUDA Toolkit if GPU available
```
This prevents user frustration and ensures AI applications can run as intended across different hardware configurations.
---
## consolidate build scripts
<!-- source: menloresearch/jan | topic: CI/CD | language: Json | updated: 2023-11-14 -->
When creating multiple build variants (debug, publish, etc.), avoid duplicating build commands by reusing the base build script. This reduces maintenance overhead, ensures consistency across build targets, and shortens complex build commands.
Instead of duplicating the entire build process in each script:
```json
{
"build": "tsc -b . && webpack --config webpack.config.js",
"build:publish": "tsc -b . && webpack --config webpack.config.js && rimraf *.tgz --glob && npm pack && cpx *.tgz ../../electron/core/pre-install"
}
```
Consolidate by referencing the base build command:
```json
{
"build": "tsc -b . && webpack --config webpack.config.js",
"build:publish": "npm run build && rimraf *.tgz --glob && npm pack && cpx *.tgz ../../electron/core/pre-install"
}
```
This approach ensures that changes to the base build process automatically propagate to all build variants, reducing the risk of inconsistencies and making the build pipeline more maintainable.
---
## leverage existing solutions
<!-- source: menloresearch/jan | topic: Code Style | language: TSX | updated: 2023-11-02 -->
Avoid duplicating functionality by leveraging existing libraries and creating reusable components. Instead of redefining common utilities or creating similar components for each use case, use established libraries and build generic, reusable components that can be configured for different scenarios.
For example, rather than redefining a classNames utility:
```typescript
// Avoid
function classNames(...classes: any) {
// custom implementation
}
// Prefer
import classnames from 'classnames'
```
Similarly, create reusable components instead of duplicating similar functionality:
```typescript
// Instead of separate DeleteModal, ConfirmModal, etc.
// Create one ConfirmationModal that accepts different props
const ConfirmationModal: React.FC<Props> = ({
atom, title, description, onConfirm
}) => {
// Generic modal implementation
}
```
This approach reduces code duplication, improves maintainability, and ensures consistency across the codebase.