# Databases & Data Platforms Query engines, storage, replication, streaming, ORMs and analytics backends. 431 instructions from 15 repositories. Last updated 2026-05-09. --- ## Improve Readability Style Adopt a “review-first” style for complex C code: keep control flow and conditions small, make side effects explicit, and avoid duplication. Key rules: 1) Split overly long conditionals - Avoid putting multiple parsing/validation calls into a single large `if`. - Prefer sequential guarded `if` blocks. 2) Use guard clauses and shared cleanup - Reduce indentation by returning early on validation failures. - When there are multiple paths (e.g., fast-path vs fallback), share cleanup via a single label or helper, so the structure stays flat. 3) Prefer explicit branching for special modes - Don’t hide unusual behavior inside another branch (e.g., special aggregate handling mixed into SUM). Use `else if` with a clear name/comment. 4) Extract duplicated “finalization”/side-effect logic - If multiple commands repeat the same update/notification/keyspace-finalization logic, move it to a helper to keep behavior consistent (and reduce the chance of diverging implementations). 5) Enforce consistent formatting - Always use braces for multi-line `if/for/while` bodies. - For multi-line conditions, wrap the controlled block with `{}` for scan-ability. 6) Use named constants instead of magic numbers - Prefer existing shared constants (e.g., buffer sizing macros) over local ad-hoc values. Example (splitting long validation): ```c // Before (hard to read) if (parseA()!=C_OK || parseB()!=C_OK || parseC()!=C_OK) return; // After (reviewable) if (parseA() != C_OK) return; if (parseB() != C_OK) return; if (parseC() != C_OK) return; ``` Example (shared cleanup to reduce indentation): ```c if (fast_path_ok) { goto done; } // fallback path done: cleanup_common(); return; ``` Applying these consistently makes diffs easier to scan, reduces reviewer cognitive load, and prevents subtle behavioral drift across similar command implementations. --- ## Explicit Build Flags Ensure build configuration explicitly sets (1) include paths and (2) required language/toolchain modes to prevent environment-dependent header resolution and compiler-mode breakages. - Include paths: Add third-party include directories explicitly and in a way that selects the intended headers. If a dependency’s headers conflict by name, adjust `-I` order/paths to avoid accidental inclusion of the wrong header. Example: ```make # Prefer the dependency’s local includes to avoid header-name collisions CFLAGS += -I../deps/tre/local_includes ``` - Language standard/toolchain mode: Do not rely on compiler defaults. Set the language standard required by the code (especially for C/C++ standard library features and syntax supported only in newer modes). If the code uses C99-only syntax (e.g., loop variable declarations), configure `-std=c99` so older toolchains can compile successfully. Example: ```make # Required because the code uses C99 syntax (e.g., for (size_t i = ...)) STD ?= -std=c99 CFLAGS += $(STD) ``` Apply this rule whenever you change a Makefile/feature flag/build script that depends on external headers or compiler behavior—treat build flags and include paths as configuration that must be deterministic across environments. --- ## Correct Fast-Path Semantics When implementing algorithmic optimizations (fast paths, binary search vs linear, prefetch/slot routing, realloc/compaction shortcuts), require *semantic correctness* and *invariant preservation* before performance. Apply this standard: - **Guard fast paths with exactness conditions** (range, precision, rounding model). If the proof/assumption doesn’t hold, **fall back** to the reference/slow path. - **Match externally observable semantics exactly** (e.g., parse success vs failure, and `endptr`/“consumed input” behavior). Never accept partial parses just because a value (like `nan`) was detected. - **Use overflow checks that match the actual arithmetic** (multiplication vs addition). Validate the intermediate that can overflow, not just a later result. - **After possible realloc/relayout**, re-apply any derived indexes/maps that depend on the old address/layout (e.g., tree/rax nodes pointing at listpack memory). - **Defer micro-optimizations** if correctness is the priority and the difference is unlikely to matter at RC time; choose the simpler correct algorithm first. Example (parsing correctness guard): ```c // Rule: only accept nan(...) if the parse consumed the expected characters. if (parsed && isnan(result)) { if (endptr) *endptr = (char*)eptr; // But also ensure the parse covered the full grammar for the token. // If eptr != pend for an otherwise-invalid/partial nan(...), reject. } ``` Example (overflow check matching operation): ```c // If overflow risk is in multiplication, check multiplication, not addition. if (emission_interval_us > LLONG_MAX / num_requests) { addReplyError(c, "overflow"); return; } ``` --- ## Fail/Clamp Error Contracts When implementing commands with numeric failure modes (overflow, out-of-range, NaN/Inf) or multi-option parsing, define an explicit error contract and enforce it end-to-end: 1) Establish FAIL vs CLAMP semantics - Default to FAIL: reject the operation (return an error) and do not modify the key. - CLAMP (or equivalent): only under an explicit mode, saturate/cap to allowed bounds and proceed. - Apply the same policy consistently to both implicit failures (type limits) and explicit failures (LBOUND/UBOUND). 2) Validate inputs and option combinations early - Parse strictly and reject contradictory/incomplete options (e.g., STRICT/ONBOUND requires bounds). - Prevent duplicate/repeated flags (e.g., BYINT specified twice). - Resolve parsing ambiguity (e.g., if the stored value is float, don’t assume integer conversion can’t fail—fallback safely based on detected type/parse result). 3) Propagate errors immediately - Treat conversion/parsing failures and corrupted data as hard failures (early return). - For environment-dependent operations, prefer graceful fallback paths when syscalls fail. 4) Respect fatal subsystem state - Don’t call “cleanup” helpers in states where the subsystem documented it must not be called after a fatal error (e.g., SSL fatal error states). Example (numeric failure contract sketch): ```c bool clamp_mode = (flags & OBJ_ONBOUND_CLAMP); // Validate/parse first; no state mutation before checks. if (parse_value_or_reply(c, &value) != C_OK) return; if (parse_bounds_or_reply(c, &lb, &ub) != C_OK) return; value2 = value + incr; if (isnan(value2) || isinf(value2)) { addReplyError(c, "increment would produce NaN/Infinity"); return; // FAIL always } if (value2 < lb || value2 > ub) { if (!clamp_mode) { addReplyError(c, "value is out of bounds"); return; // do not modify key } value2 = (value2 < lb) ? lb : ub; // CLAMP } // Only now: perform the key update + reply. set_key_value(c, value2); ``` Adopting this standard prevents inconsistent edge-case behavior, avoids partial updates on failure, and makes error handling predictable for both client-facing semantics and internal defensive safety. --- ## Bounded Prefetch Batches When adding performance optimizations (prefetch/batching/fast paths), make them bounded, gated, and semantics-correct: - **Bound the work**: use fixed small batch sizes (e.g., 16) and stack-backed buffers with a safe default cap; avoid strategies where user-controlled sizes can force huge allocations (e.g., `vecReserve(count)` without a ceiling). - **Gate the fast path**: only prefetch/batch when it’s likely to help (e.g., skip when cross-command prefetch already warmed the cache, or when the dict/slot has nothing to look up). - **Preserve correctness**: fast paths must mirror the “real” iterator semantics (expired-field handling, `allow_access_expired`/skip behavior, and cluster/slot/dict lifetime across batches). If batch boundaries can change pointers (e.g., expire+free), re-fetch per batch. - **Avoid redundant expensive work**: don’t repeat lookups/scans that the caller or batch already computed; prefer passing/reusing computed state instead of re-deriving it. Example pattern (batch + gate + bounded stack, with a hard ceiling): ```c #define BATCH_SZ 16 #define STACK_CAP 256 void do_fast_prefetch_and_reply(client *c, dict *d, int use_prefetch) { int numkeys = c->argc - 1; addReplyArrayLen(c, numkeys); /* Gate: only prefetch if there is something to warm. */ if (!use_prefetch || d == NULL || dictSize(d) == 0) { for (int j = 1; j < c->argc; j++) { kvobj *o = lookupKeyRead(c->db, c->argv[j]); if (o == NULL || o->type != OBJ_STRING) addReplyNull(c); else addReplyBulk(c, o); } return; } /* Bounded batch work. */ int j = 1; while (j < c->argc) { int end = j + BATCH_SZ; if (end > c->argc) end = c->argc; int n = end - j; /* Re-fetch any pointer that may become invalid across batch boundaries. */ d = kvstoreGetDict(c->db->keys, /*slot*/0); if (d == NULL || dictSize(d) == 0) { for (int k = 0; k < n; k++) { lookupKeyRead(c->db, c->argv[j + k]); addReplyNull(c); } j = end; continue; } /* Prefetch within the bounded batch. */ for (int k = 0; k < n; k++) redis_prefetch_read(&d->ht_table[0][0]); /* placeholder */ for (int k = 0; k < n; k++) { kvobj *o = lookupKeyRead(c->db, c->argv[j + k]); if (o == NULL || o->type != OBJ_STRING) addReplyNull(c); else addReplyBulk(c, o); } j = end; } } ``` Apply this checklist in code reviews for any performance PR: if the optimization can turn into unbounded allocation, double-prefetch, or semantic drift across expiry/cluster boundaries, it should be reworked to add bounding + gating + semantic parity. --- ## Fail Fast Input Validation Apply a single security rule: validate arity, lengths, ranges, and invariants before using inputs to index arrays, dereference pointers, compute derived pointers, or persist/serialize into security-sensitive targets. Practical checklist: - **Before argv/key extraction:** verify command arity so helpers can’t read `argv` out of bounds. - **Before pointer math / derived references:** assert required preconditions (e.g., “the requested metadata bit is set”) before computing offsets. - **Before dereferencing/freeing-lifetime assumptions:** compute any needed properties (e.g., length) before freeing buffers; don’t call methods on freed objects. - **Before persisting user-controlled strings into config/protocol text:** either reject dangerous characters (e.g., control chars) or rely on a known escaping layer—don’t do “partial” safety. - **Before auth/security state changes:** validate security-sensitive parameters (e.g., password policy/length) and keep logging/redaction rules. Example (arity defense-in-depth): ```c // In an ACL permission path that inspects argv for keys/channels if (!commandCheckArity(cmd, argc, NULL)) { if (idxptr) *idxptr = 0; return ACL_DENIED_CMD; } // Also keep boundary checks in shared key-extraction helpers, // even if callers already validated. ``` Example (invariant assertion before derived pointer): ```c serverAssert(metaId < KEY_META_ID_MAX && (bits & (1u << metaId))); ``` Example (string control-char validation at persistence boundary): ```c if (sentinelStringContainsControlChars(val->ptr, sdslen(val->ptr))) { addReplyError(c, "value must not contain control characters"); goto bad; } ``` --- ## API Contracts And Compatibility When adding or evolving APIs (commands, module interfaces, persistence callbacks), make the contract explicit and stable: define lifecycle rules, invariants, grammar, and compatibility boundaries, and ensure implementations preserve existing behavior (including error/return types) across encodings and versions. Practical rules: - Define deterministic lifecycle constraints: e.g., “only allowed during OnLoad” for registration-type APIs; document what happens if called elsewhere. - Establish strong invariants for callbacks: document when callbacks run/are skipped (e.g., “only invoked when meta != reset_value”) so modules can rely on it. - Version persistence with clear dispatch: use an encoding version (metaver) and specify how load/save behaves for older/newer versions; require modules to handle incompatibility explicitly. - Keep user-visible semantics encoding-independent: if the API offers BYINT vs BYFLOAT, ensure validation and error/reply behavior matches the selected mode, not the underlying representation. - Preserve argument grammar and error behavior: if you introduce a parsing abstraction (table/descriptor), ensure it doesn’t change permissible argument order/placement, validation depth, or failure reply semantics. - Avoid leaking internal implementation details into user responses: internal encodings/representation should not change user-visible replies unless the API explicitly documents it. Example invariant pattern (persistence/module callbacks): ```c // Contract: core guarantees callback invocation only when meta != reset_value. // Modules can safely free/cleanup only under that condition. if (meta != reset_value) { myMeta_free_callback(keyname, meta); } ``` Apply the same discipline to command grammar (e.g., optional params like IDS/FORCE) and to event subscription APIs: define whether duplicates are allowed, whether deregistration is strict or best-effort, and what constitutes misuse vs supported behavior. --- ## Targeted, Non-Bloated Tests When adding/modifying tests, ensure they are (a) semantically aligned to the behavior under test, (b) precise enough to prove the intended code path was executed, and (c) not bloated or duplicated. Apply this checklist: 1) **One behavior per test/subcase**: the scenario should directly relate to the option/feature being tested (avoid mixing unrelated parameters). 2) **Assert what proves the branch**: verify specific, narrow indicators (exact error substrings, distinctive log messages, or exact state transitions). Don’t rely only on “command succeeded.” 3) **Avoid coverage-masking fallbacks**: don’t use broad `catch`/fallback logic that allows the test to pass when the targeted flow didn’t occur. 4) **Consolidate instead of multiplying**: prefer reusing setup helpers and merging similar checks; remove duplicates and don’t add “for every command” coverage when broader tooling already exists. 5) **Keep expectations robust**: if output may differ slightly, match stable substrings rather than brittle full-text matches. Example pattern (branch proof, not permissive fallback): ```tcl # Good: only accept success if the targeted log line is present. wait_for_log_messages -2 "*Disconnecting timedout replica (full sync)*" $loglines 100 100 # Later for the specific subcase: wait_for_log_messages -2 "*Diskless rdb transfer, last replica dropped, killing fork child*" $loglines 1 1 ``` If you need to reduce flakiness, do it by making the test deterministic (timing/config/input shaping), not by weakening the assertions that demonstrate the intended behavior was reached. --- ## Atomic Contracts Enforcement Define and enforce atomicity as an explicit contract: if correctness depends on thread/writer invariants, encode it in the API name/docs and validate it at usage sites; if a value can be concurrently updated, read/write it exclusively via atomic operations (no non-atomic peeking into parts of an atomic container). Practical rules: 1) Single-writer “atomic” helpers - If an operation is implemented with load+add+store (not true RMW), it must be correct only under a strict single-writer rule. - Make that rule unmissable in the name and comment (e.g., `...SingleWriter...`), and ensure no other thread can ever update the same variable. 2) Packed atomic bitfields - Treat the entire storage (e.g., `atomic_flags_refcount`) as atomic. - If only some bits are intended to be immutable after init, state that assumption explicitly and ensure code never violates it. - All reads of the atomic container must use `atomicGet` (or equivalent) before masking/shifting; don’t read the underlying integer non-atomically from another thread. 3) Document IO-thread vs main-thread invariants - When refcounts/pointers are updated in different thread phases, comments must explain what is safe to update when, and why—field names should reflect semantics (e.g., “current node” vs “start node”). Example (atomic-packed refcount access): ```c #define OBJ_REFCOUNT_MASK ((1U << OBJ_REFCOUNT_BITS) - 1) #define OBJ_EXPIRABLE_BIT 30 #define OBJ_ISKVOBJ_BIT 31 static inline unsigned robj_refcount(const robj *o) { uint32_t v; atomicGet(o->atomic_flags_refcount, v); // must be atomic return v & OBJ_REFCOUNT_MASK; } static inline unsigned robj_expirable(const robj *o) { uint32_t v; atomicGet(o->atomic_flags_refcount, v); // must be atomic return (v >> OBJ_EXPIRABLE_BIT) & 1; } ``` If you can’t clearly state the invariants and make every concurrent access atomic-safe, redesign the ownership model or synchronization rather than relying on “it never races” assumptions. --- ## Consistent TTL Pre-check When handling TTL/expiration, decide “already expired?” (and validate expiry inputs) before performing DB mutations or expensive work, and then keep DB changes, stats/counters, and keyspace notifications consistent with the real outcome. For conditional variants (NX/XX, ENX/FNX/FXX), short-circuit behavior must match the condition. Practical checklist: - Parse/validate expiration arguments early. - If TTL is already expired: - Do not insert/overwrite new data (unless semantics require it); delete only if an old key existed/was actually overwritten. - Update expired-key counters and keyspace notifications consistently with the effective DB outcome. - Honor conditional flags (e.g., only perform the “expired short-circuit” path when the operation would otherwise proceed per NX/XX rules). - If you use TTL-aware fast paths/optimizations (prefetch/batching/cached pointers), ensure cache/resource lifetimes are safe across mutations that can delete/free entries; refresh per batch when necessary. - Add tests for expire-then-insert across batch boundaries and for conditional short-circuiting. Pattern: ```c // Pseudocode pattern if (expire && checkAlreadyExpired(milliseconds)) { if (found_existing_key) { // delete with correct delete semantics (e.g., overwrite deletion vs expired deletion) dbDeleteOrExpireSemantics(db, key); emit_del_ksn_and_update_stats(); } else { // for “missing key” cases: short-circuit without creating state // but still update counters if spec requires it emit_required_counters_only(); } return; } // otherwise: proceed with normal insert/update and set TTL apply_set_or_increment_and_set_expire(); ``` --- ## CI/CD Workflow Guardrails {% raw %} When building or modifying CI/CD GitHub Actions workflows, structure them as an explicit DAG of artifact-driven jobs and guard execution to prevent fork/irrelevant-context failures. **Standards** 1) **Gate by repo once, with correct allowlist** - Prefer workflow/job-level `if: github.repository == ''` (or a clearly maintained allowlist) so jobs don’t run in forks where secrets/tokens or release logic would fail. 2) **Make ordering explicit with `needs`** - If a job uses outputs/artifacts from another job, declare `needs` and consume `needs..outputs`. - Ensure a “summary/finalization” job uses `if: always()` and has the needed `needs: [...]` so it can run even when earlier jobs are skipped. 3) **Use job outputs/env, not ad-hoc shell env unless needed** - Export values via `outputs` on the producing job (computed from steps) and reference them in downstream jobs. 4) **Add lightweight artifact validation** - For release workflows that create tarballs/binaries, include a basic sanity check (e.g., checksum and/or size range) to catch malformed or unexpectedly changed artifacts early. 5) **Keep CI dependencies aligned with what you actually run** - Don’t install or configure toolchains/flags (e.g., ASAN/cpp toolchain) unless that CI job truly exercises them. **Example pattern** ```yaml jobs: extract: if: github.repository == 'redis/redis' runs-on: ubuntu-latest outputs: tag: ${{ steps.info.outputs.tag }} steps: - id: info run: echo "tag=v1.2.3" >> "$GITHUB_OUTPUT" create-artifact: needs: extract runs-on: ubuntu-latest outputs: sha256: ${{ steps.checksum.outputs.sha256 }} steps: - run: ./utils/releasetools/01_create_tarball.sh "${{ needs.extract.outputs.tag }}" - id: checksum run: | TARBALL="/tmp/redis-${{ needs.extract.outputs.tag }}.tar.gz" SHA256=$(shasum -a 256 "$TARBALL" | cut -d' ' -f1) echo "sha256=$SHA256" >> "$GITHUB_OUTPUT" summary: needs: [extract, create-artifact] if: always() runs-on: ubuntu-latest steps: - run: echo "Pipeline completed (may be partially skipped)." ``` Applying these rules prevents cascading failures on forks, makes workflow behavior predictable, and improves reliability of release automation by validating what you produce before you publish it. {% endraw %} --- ## Doc Semantics Consistency Ensure comments and public-facing documentation precisely match the actual behavior and contract. Apply this standard by checking: - **No doc/behavior drift**: If you add/adjust command options (e.g., STRICT/PERSIST) or state transitions (e.g., TTL handling), update all descriptions (PR text, command docs, inline comments) to reflect the implemented semantics. - **Define exact meaning of outputs/fields**: For INFO/metrics and API parameters, state what is included/excluded and what the field represents (e.g., “count is number of field names, not values”; “shared vs unshared bytes”). - **Document semantic edge cases**: If duplicates are possible (e.g., subkey field names) or behavior depends on flags/invariants, explicitly document the expectation and any performance-vs-deduplication tradeoff. - **Clarify unit/size semantics**: When a function returns a *requested* value rather than an *allocated* one, document that explicitly and ensure call sites don’t rely on the wrong notion. - **Explain non-trivial logic**: For complex algorithms or rate-limiting behavior, include a short rationale/step summary in comments so maintainers can follow the intent. - **Justify magic constants**: For bounds like a specific upper limit, add the reason (e.g., overflow/representable range) and keep the check consistent. Example (magic constant rationale): ```c if (period <= 0 || period >= 1000000000000LL) { /* 1e12 chosen so that when converting to microseconds we stay within int64 range */ addReplyError(c, "period must be > 0 and < 1e12"); return; } ``` Rule of thumb: if someone can misinterpret the contract from the docs (TTL semantics, count semantics, shared-vs-unshared accounting, requested-vs-allocated), fix the documentation until the interpretation is unambiguous. --- ## Intentional Error Handling Handle errors based on intent: make operations idempotent for known conflicts, and explicitly classify/propagate errors for known transient conditions. Apply this standard in test scripts and helper utilities: - For deterministic conflicts (e.g., resource already exists), use the command’s recovery flag rather than letting the call fail. - For remote/cluster calls, wrap the call in `catch`, and: - Treat known transient routing issues (e.g., `MOVED*`, `ASK*`) as recoverable (retry/continue). - Re-raise anything else immediately so real bugs don’t get masked. Example patterns: ```tcl # Known conflict: make restore idempotent catch {r restore key 0 $crafted replace} err # Known transient cluster routing: ignore MOVED/ASK, fail fast otherwise if {$cluster_load == 1} { if {[catch {$r read} err]} { if {[string match {MOVED*} $err] || [string match {ASK*} $err]} { continue } error $err } } else { $r read } ``` --- ## Maintain algorithmic invariants When implementing algorithms that depend on “current state” (client context, cached slot/client, etc.) or on custom data structures (stack/heap modes), make the invariant explicit and ensure all computations are derived from the specific operand/object being processed—not from incidental outer execution context. Practical rules: - Operand over incidental context: in nested execution (e.g., MULTI/EXEC where commands run while other client state is active), compute shard slot / routing from the key/channel argument itself and do not rely on cached “current_client”/slot. - Encode ownership & storage mode: for DSs with multiple backing stores (stack vs heap), clearly define what is owned/freed, what is never freed, and when heap allocation happens. - Allocation discipline: in Redis core, use Redis allocator wrappers (zmalloc/zrealloc/zfree), not libc allocation. - Ergonomics that reduce misuse: prefer DS shapes that make correct usage easy (e.g., embed a small fixed buffer variant like SmallVector rather than forcing callers to manage extra stack buffers), and keep the API naming consistent with Redis style. Example (stack-embedded “small vector” idea): ```c // SmallVector-like: small fixed buffer embedded in the DS. typedef struct vec { size_t size; size_t cap; void **data; void *small[VEC_DEFAULT_INITCAP]; } vec; void vecInit(vec *v, size_t hint) { v->size = 0; if (hint <= VEC_DEFAULT_INITCAP) { v->data = v->small; v->cap = VEC_DEFAULT_INITCAP; } else { v->data = zmalloc(sizeof(void*) * hint); v->cap = hint; } } void vecDestroy(vec *v) { if (v->data != v->small) zfree(v->data); } ``` Apply this mindset to both algorithmic routing decisions and DS implementations: correctness comes from invariants tied to the actual data being acted upon, and API/structure choices that prevent accidental reliance on stale execution-context or ambiguous ownership. --- ## Clear Semantic Identifier Naming Adopt naming conventions that are (1) case-consistent by identifier kind and (2) semantically self-documenting (what it represents/includes/excludes, and what the function returns). **Rules** 1. **Identifier case consistency** - Use **camelCase** for **functions** and **global/server variables**. - Use **snake_case** for **local variables**. - Do not mix styles in the same function/scope. 2. **Names must encode meaning** - For metrics/fields/helpers, include/exclude behavior must be evident from the name (or be explicitly documented next to the declaration). - Prefer names like `*Shared*` / `*Unshared*` when inclusion varies. 3. **Function names must match behavior** - If a function returns a boolean/status (e.g., 0/1), the return meaning must be reflected in the name (e.g., `should...`, `try...`, `is...`) or the code should clearly define it in the comment immediately above the function. - Align with existing naming patterns (e.g., `clientsCronRunClient`-style). 4. **Avoid ambiguity for internal concepts** - For internal commands or domain terms, avoid unclear abbreviations unless the naming/context makes the value’s meaning obvious; prefer descriptive names or add a precise comment that explains the stored value and its unit. **Examples** - Prefer: - `mem_clients_ref` / `mem_clients_orphan_ref` (but ensure docs/state match inclusion). - `getClientMemoryUsage()` vs `getClientOutputBufferMemoryUsage()` should clearly state inclusion/exclusion; if needed, rename one to reflect it (e.g., “Size” vs “Usage”). - Prefer boolean-like naming: - `int shouldFreeClientAndHandleTimeout(...)` over a generic `cronHandleClients(...)` if the return value drives control flow. - Enforce case: - `uint64_t *kvObjMetaRef(...)` (function: camelCase). - `lower_mask` (locals: snake_case). --- ## Consistent Clear Naming Use naming that is (1) safe per language rules, (2) semantically explicit, and (3) consistent with existing symbol families. Practical checklist: - Avoid reserved/unsafe identifier patterns. In C/C++, don’t introduce names starting with `_[A-Z]` (rename instead). - Make intent clear in the identifier. If a field like `check_time` can represent multiple checks, use a precise prefix that states what it tracks (e.g., `last_cron_check_time`, `client_last_cron_check_time`, etc.). - Keep related APIs/handlers/constants consistent with the project’s naming “families.” If you already have `scanHashTable`, `scanListpack`, then a new counterpart should follow the same `scan*` pattern (e.g., `scanIntSet` when applicable). For cluster constants, align names with the established `CLUSTER_*` scheme (e.g., rename `GETSLOT_CROSSSLOT` → `CLUSTER_CROSSSLOT`). Example: ```c // Bad: reserved pattern in C11 REDISMODULE_API void (*_RedisModule_FreeString)(RedisModuleCtx *ctx, RedisModuleString *str); // Good: renamed to a non-reserved identifier REDISMODULE_API void (*RedisModule_FreeString_raw)(RedisModuleCtx *ctx, RedisModuleString *str); // Good: semantic clarity via explicit prefix // struct client { mstime_t last_cron_check_time; } // Good: naming family consistency void scanIntSet(client *c, robj *o, scanOptions *opts); // Good: cluster constant consistency #define CLUSTER_CROSSSLOT (-2) ``` --- ## Null-safe Free Wrappers When a free/cleanup API may receive NULL (or an invalid handle), don’t rely on ad-hoc checks at call sites. Instead, define a NULL-safe wrapper/helper that: - No-ops when the argument is NULL (mirrors `free(NULL)` semantics). - Avoids double-evaluation (store the argument in a temporary). - Preserves compile-time type checking (use a correctly typed temporary; avoid unnecessary casts). - Is safe to use across versions when needed (e.g., macro wrapper vs direct symbol). - Also guard cleanup operations (e.g., `close`) to run only when the handle was actually acquired. Example pattern (C): ```c /* NULL-safe wrapper: evaluates arg once, type-checked, no-op on NULL */ #define Module_FreeStringSafe(ctx, str) \ do { \ RedisModuleString *_ms = (str); \ if (_ms) RedisModule_FreeString(ctx, _ms); \ } while (0) ``` Example guard (Tcl): ```tcl if {$fd ne {}} { catch {close $fd} } ``` Apply this rule consistently to any code path that frees/finalizes memory or resources that might be absent due to conditional logic or partial failures. --- ## Nullable Contracts Enforcement Adopt explicit, consistent nullability contracts for pointer/optional-field parameters and honor them at every call site. Practical rules: - **Document/decide whether a parameter may be NULL**. If it can’t, keep the check out; if it can, guard and define behavior. - **Align new helpers with existing conventions** (e.g., `get*From*Object` patterns) so developers don’t guess whether NULL is legal. - **Guard before dereference** when an optional field can be absent (e.g., `if (obj->field) { use }`). - **Avoid UB from “meaningless” fields**: don’t read dict values when the dict is configured for `no_value`; pass/handle the correct canonical value (`NULL` for absent value). - **Prefer correctness via interface contracts over defensive checks inside low-level helpers** (unless the team has a specific safety requirement). If a helper would UB with bad inputs (e.g., `memcpy(dst,NULL,n>0)`), enforce the precondition at call sites. - **After invalidation/nullification, treat pointers as unusable** (prevent accidental use-after-invalidate by structuring code flow so dereferences are impossible). Example pattern: ```c /* Optional pointer is truly nullable; guard before use */ if (nack->consumer) { addReplyBulkCBuffer(c, nack->consumer->name, sdslen(nack->consumer->name)); } else { addReplyBulkCBuffer(c, "", 0); } /* Dict configured with no_value: never read dictGetVal() */ if (dictAdd(dst, key, NULL) == DICT_OK) { /* ... */ } ``` This standard prevents ambiguous NULL expectations, avoids undefined behavior, and keeps null-safety practical instead of scattered defensive code. --- ## Version Compatibility First When changing serialized/wire formats (RDB/DUMP/RESTORE or module metadata), treat it as a data migration: define compatibility behavior and make encoding placement deterministic. Apply these rules: 1) Version/format contract: if new RDB object types or metadata layouts are introduced, explicitly decide whether you will (a) bump the RDB/module format version, or (b) keep old encodings unchanged unless the new feature is used. Ensure the code and release process agree on the final version. 2) Upgrade/downgrade story: document what happens when a newer producer writes data that an older consumer might load/replicate. If the new encoding can appear only under certain runtime conditions (e.g., “only when XNACK was used”), spell out the exact condition and the resulting failure mode. 3) Deterministic metadata placement: avoid “lazy first occurrence” schemes that produce structurally different files depending on which keys appear first. - Prefer writing global metadata/class definitions at a deterministic location (e.g., beginning of the RDB/DUMP), or - Prefer embedding complete metadata context per key as an atomic unit (e.g., ``), so partial loading/streaming doesn’t require global state. 4) Don’t rely on “compat isn’t important”: even if you plan a version bump, compatibility impacts replication and user workflows; decide and implement the intended behavior deliberately. Illustrative pattern (eager header vs per-key embedding): ```c // Option A: eager, deterministic header (fixed location) write_rdb_header(); write_meta_class_definitions(); for (each key) { write_key_with_value_without_global_state(); } // Option B: per-key atomic embedding (metadata always complete) for (each key) { write(T_META); write(TYPE); write(KEY); write(VALUE); } ``` --- ## Secure Workflow Permissions {% raw %} Apply two security practices to GitHub Actions workflows: 1) Enforce least privilege - Set `permissions: {}` at the workflow level (or default deny) and only add the minimal scopes required for each job/step. 2) Safely pass dynamic workflow data into shell - Values coming from `needs.*.outputs` (e.g., tag/version strings) should not be interpolated repeatedly into shell snippets. - Instead, pass them through a controlled job-level environment variable (or a dedicated setup step) and always quote them when used in bash/script invocations. Example pattern: ```yml permissions: {} jobs: extract-release-info: # ... create-tarball: needs: extract-release-info runs-on: ubuntu-latest env: RELEASE_TAG: ${{ needs.extract-release-info.outputs.tag_name }} steps: - name: Create tarball run: | echo "Creating tarball for version ${RELEASE_TAG}..." # If using a script, pass quoted args: # ./utils/releasetools/01_create_tarball.sh "${RELEASE_TAG}" ``` This reduces blast radius (least privilege) and mitigates shell-injection risk when dynamic workflow outputs are consumed by bash. {% endraw %} --- ## Performance-first Structure When optimizing, treat performance as a design constraint: prevent contention/cache inefficiency, ensure server-side work is bounded per call, and keep tests from becoming slow bottlenecks. 1) Cache-line & contention hygiene (C structs) - Align/pad frequently-updated counters to avoid false sharing. - Avoid mixing rarely-used “cold” fields into the same cache footprint as hot ones; move them deeper in the struct. - Prefer per-thread stats to share less with other threads; if needed, split “hot counters” into their own cache-line-aligned struct. Example pattern: ```c typedef struct __attribute__((aligned(CACHE_LINE_SIZE))) { redisAtomic long long io_reads_processed; redisAtomic long long io_writes_processed; /* other hot, per-IO-thread counters */ } IOThreadStats; typedef struct { /* hot per-IO-thread fields */ IOThreadStats stats; /* each thread updates its own cache line */ /* cold / infrequently read fields after */ int unused_cold_field; } IOThread; ``` 2) Bound work for collection-style commands - For commands that iterate over IDs supplied by the client, ensure per-ID processing is O(1) with a small constant, and rely on the existing batching convention (client chooses how many IDs to send). - Only introduce server-side COUNT/batching knobs if they meaningfully prevent a real bottleneck or reduce complexity (not just to mimic client-side batching). 3) Keep tests fast and deterministic - Avoid long fixed sleeps for timing/cron behavior. - Use polling/`wait_for_condition`-style loops and reduce durations/timeouts so CI runtime stays low. --- ## Validate security-critical data When code consumes security-critical data—either to perform privileged state changes at runtime or to download/execute external artifacts—it must validate integrity/invariants before proceeding. Apply these checks: - Runtime guardrails for powerful flags: For any command that can force behavior (e.g., recovery/replay/creation), validate that referenced resources actually exist and that the operation will not create inconsistent/phantom state. If validation fails, skip or reject deterministically. - Supply-chain artifact integrity: For build/dependency downloads, verify the downloaded file using the correct checksum from the official source for the exact artifact you download (e.g., don’t mix tar.gz vs tar.xz checksums). Example (runtime invariant + safe skip): ```c // Pseudocode for a FORCE-like path if (force_enabled) { if (!stream_entry_exists(stream_id)) { // Prevent phantom state creation return OK_WITHOUT_MUTATION; // or skip this ID } create_pel_entry(...); // only after invariant holds } ``` Example (build checksum verification): ```sh RUST_VERSION=1.94.0 # Ensure checksum matches the exact downloaded artifact RUST_SHA256='9a358120ce1491a4d5b7f71a41e4e97b380b5db5d4ec31f7110f5b3090bd3d55' curl -sSLO "$URL" # URL must point to the same tarball as the checksum echo "$RUST_SHA256 $(basename "$URL")" | sha256sum -c - ``` --- ## Header Organization And Comments Keep header edits readable and durable: 1) Don’t write brittle comments - Avoid referencing exact line numbers or file offsets (they drift and become misleading). Prefer describing the behavior/why, not where it currently is. 2) Group declarations by concept in headers - Place new structs/functions near the related types/metadata they operate on (e.g., keep kvstore-related metadata together; place “context” structs beside the owning feature’s metadata blocks). - Don’t move unrelated APIs into headers just because they’re being touched—keep them in the header that matches their domain/ownership (e.g., robj/kvobj-related declarations in object.h; “server command/memory command”-style helpers remain in their original area). 3) Avoid unnecessary exposure - If a new symbol is only used by one module, consider removing it from broader headers or narrowing its scope instead of exporting it. Example (durable comment): ```c // Bad: “Fix for crash at file.c:1234” (will rot) // Good: Explain the invariant/ordering requirement that prevents stale pointers. // e.g., “Ensure notification happens after all kvobj accesses to avoid stale pointers // if the notification callback reallocates the kvobj.” ``` Example (grouping): - Put `asmTrimCtx` right after the kvstore metadata structs it supports, and keep `kvstoreMetadata` adjacent to `kvstoreDictMetadata` rather than splitting them across the file. --- ## Clear API Contracts Public APIs must make their guarantees and caller responsibilities explicit—especially around encapsulation, lifecycle/ownership, and callback semantics. Guidelines: - Encapsulate internals: expose only what modules/users need; hide implementation details behind typedefs/opaque structs when appropriate. - Define behavioral contracts in writing: if behavior depends on sentinels, ordering, or conditional invocation rules, document the exact meaning and required checks. - Specify ownership and lifecycle for every callback/input/output: - Who owns input structs (e.g., is `conf` copied?) and what is the scope of that ownership? - For `out` parameters like `*meta`, state what the API expects modules to do (e.g., whether to keep vs replace, and how to update pointers). - Clarify when free/unlink/copy callbacks may be invoked and on which threads. - Prefer typed, intention-revealing signatures over “generic hide-the-type” parameters when the concrete type is known. - Avoid unnecessary API complexity: if a simpler call pattern is safer and sufficiently performant, prefer it—and document any non-obvious semantic details. Example pattern (copy callback semantics): ```c /* Copy callback: update *meta and return whether metadata is kept. */ static int KeyMetaCopyCallback(RedisModuleKeyOptCtx *ctx, uint64_t *meta) { REDISMODULE_NOT_USED(ctx); char *str = (char *)*meta; if (str) { char *new_str = strdup(str); *meta = (uint64_t)new_str; /* API contract: update out-param */ } return 1; /* keep metadata */ } ``` If you introduce sentinel values or out-parameter replacement behavior, treat it as an API contract: make the sentinel meaning explicit, define when callbacks are skipped, and provide short examples in the docs or tests so module authors don’t guess. --- ## Thread Ownership Discipline When code can run on both main and IO threads, require an explicit ownership + synchronization rule for every shared piece of state (flags, counters, refcounts, pending lists, and memory frees). Concretely: 1) No racy reads/writes of shared fields - Avoid reading/writing client flags/state from the “wrong” thread unless protected (atomics) or guaranteed single-writer. - If IO thread must notify main, set a thread-safe “pending” flag/queue and return. 2) Main-thread only for shared sliding-window / non-thread-safe maintenance - Shared maintenance structures (e.g., active-clients windows) should be updated in one place (main thread) to avoid races; use atomics only for standalone counters. 3) Refcount changes must be atomic end-to-end - Don’t do atomicGet then a non-atomic decrement. Use atomic decrement operations that provide the resulting value, so concurrent decrRefCount cannot double-free. 4) If deallocation/free callbacks are not thread-safe: defer to the safe thread - If decrRefCount/free may run in background threads, ensure free callbacks are thread-safe or document that they may run off the main thread. - Prefer an IO->main deferred-free queue for objects that must be freed on main. - Keep queue state consistent (reset counters/size as needed) and consider splitting responsibilities (freeing vs unlinking from “pending reply” lists). Example patterns A) IO thread signals main using a pending flag instead of touching racy state ```c if (!(c->io_flags & CLIENT_IO_READ_ENABLED)) { /* No racy reads of c->flags; just signal via atomic pending flag */ atomicSetWithSync(c->pending_read, 1); return; } ``` B) Main-thread-only maintenance ```c atomicIncr(server.stat_total_client_process_input_buff_events, 1); if (c->running_tid == IOTHREAD_MAIN_THREAD_ID) statsUpdateActiveClients(c); ``` C) Safe atomic refcount decrement ```c unsigned short new_refcnt = atomicDecr(o->refcount, 1); if (new_refcnt == 0) { /* safe to free */ } ``` D) Deferred free from IO thread to main ```c /* IO thread */ ioDeferFreeRobj(c, obj); /* main thread when client is back */ freeClientIODeferredObjects(c, /*free_array=*/0); ``` Documentation requirement - Any module/type free callback that can run off-main must explicitly state its thread-safety assumptions (e.g., “may run in a background thread; not protected by GIL/GCIL”). --- ## Semantic Naming Accuracy Ensure names and naming-related metadata are semantically accurate and unambiguous. - Don’t reuse a helper/identifier whose name is tied to a specific command if it creates confusion. Prefer a dedicated, clearly named function per command. - Don’t use broad or incorrect category/type metadata fields. If a field conveys the key/type “group”, set it to the actual applicable value. Example (command-specific key helper): ```js // Avoid ambiguity: do not reuse sintercardGetKeys for SUNIONCARD function sintercardGetKeys(...) { /* keys for SINTERCARD */ } function sunioncardGetKeys(...) { /* keys for SUNIONCARD */ } ``` Example (correct semantic metadata): ```json { "GCRA": { "group": "string" } } ``` --- ## prioritize code readability Write code that prioritizes clarity and readability over cleverness or brevity. This includes several key practices: **Extract static functions**: Move functions that don't depend on component state outside the component to reduce complexity and make their static nature clear. ```tsx // Instead of defining inside component: export function MyComponent() { const validateFile = (file: File): string | null => { // validation logic }; // ... } // Extract to module level: function validateFile(file: File): string | null { // validation logic } export function MyComponent() { // component logic only } ``` **Separate concerns**: Extract styles to CSS modules rather than embedding them inline, and separate business logic from presentation logic. **Prefer simple procedural code**: Choose clear, step-by-step code over complex one-liners or overly clever patterns, even if the latter appears more "elegant." ```tsx // Prefer this clear approach: let q = z.query.issue.orderBy(sortField, sortDirection); if (open !== undefined) { q = q.where('open', open); } if (creator) { q = q.whereExists('creator', q => q.where('login', creator)); } // Over complex one-liners that are harder to follow ``` **Avoid unnecessary complexity**: Question whether complex patterns or abstractions truly add value, especially when simpler alternatives exist. Save complexity budget for features that genuinely require it. --- ## Optimize React performance patterns Prioritize React performance by avoiding expensive operations and using proper React patterns. Key practices include: 1. **Avoid unnecessary exception throwing**: Exception dispatch is a slow operation in React 18. Instead of throwing promises directly, create helper functions to minimize performance impact: ```ts // Good: Hoist React.use detection and create suspend helper const reactUse = (React as unknown as {use?: (p: Promise) => void}).use; const suspend: (p: Promise) => void = reactUse ? reactUse : p => { throw p; }; if (suspendUntil === 'complete' && !view.complete) { suspend(view.waitForComplete()); } ``` 2. **Use modern JavaScript syntax**: Prefer optional chaining over verbose conditional checks: ```ts // Good init?.(z); // Avoid if (init) { init(z); } ``` 3. **Use appropriate React components**: Always use proper React components instead of generic HTML elements for interactive elements: ```ts // Good // Avoid
setDisplayAllComments(true)}>Older
``` These practices improve both performance and code maintainability while following React best practices. --- ## API parameter design When designing API functions and types, prioritize maintainability and extensibility through proper parameter and type design patterns. For functions with multiple parameters, especially multiple boolean flags, consolidate them into a single options object to improve readability and future extensibility. This prevents parameter lists from becoming unwieldy and makes the API more self-documenting. For type definitions, extend existing types rather than duplicating their structure. Use intersection types (&) or interface extension to build upon established patterns, ensuring consistency and reducing maintenance overhead. Example of good parameter design: ```typescript // Instead of multiple boolean parameters: function query(clientID: string, query: Query, enabled: boolean, requireComplete: boolean) // Use an options object: function query(clientID: string, query: Query, options: { enabled?: boolean; requireComplete?: boolean; }) ``` Example of good type composition: ```typescript // Instead of duplicating structure: export type UseSuspenseQueryOptions = { ttl?: TTL | undefined; suspendUntil?: 'complete' | 'non-empty'; }; // Extend existing types: export type UseSuspenseQueryOptions = UseQueryOptions & { suspendUntil?: 'complete' | 'non-empty'; }; ``` This approach makes APIs more maintainable, self-documenting, and easier to extend without breaking changes. --- ## API consistency patterns Maintain consistent patterns across similar APIs to improve developer experience and reduce cognitive overhead. When designing related APIs, use factory patterns for shared contexts, consistent parameter structures, and uniform return types. Key principles: - Use factory patterns to eliminate repetitive context definitions - Prefer object parameters over multiple positional arguments for better extensibility - Structure return types consistently across similar functions - Place related methods on the same object for logical grouping Example of factory pattern for shared context: ```ts // Instead of repeating auth context everywhere const { syncedQueryWithContext } = createQueriesWithContextFactory() syncedQueryWithContext( 'user', validator.parse, // Automatically typed as AuthData | undefined (auth, userID) => {} ) ``` Example of consistent return structure: ```ts // Return structured data consistently type QueryResult = readonly [ Smash, QueryResultDetails, ]; // Instead of mixed return types function useQuery(): Accessor> ``` Example of object parameters: ```ts // Prefer object parameters for extensibility mutator(args: { arg1?: string; arg2?: string; }); // Over positional arguments that are rigid mutator(arg1?: string, arg2?: string) ``` --- ## Use descriptive names Names should be descriptive and unambiguous, clearly communicating their purpose and intent. Avoid abbreviations and ambiguous terms that could confuse readers about the actual functionality or data being represented. Key principles: - Expand abbreviated names for clarity (e.g., `additional_authenticated_data` instead of `aad`) - Use semantic names that reflect actual purpose (e.g., `MISSING` instead of `INVALID` when data is absent, `HivePartitioningExecutor` instead of `HivePartitioning` for execution logic) - Follow established naming conventions consistently (PascalCase for methods like `GetProgress()`, snake_case for variables like `ordinality_data`) - Choose names that indicate data structure (e.g., `extension_directories` when storing multiple values) - Use descriptive method names (e.g., `MicrosLength()` instead of generic `Length()`) Example of good descriptive naming: ```cpp // Bad: abbreviated and unclear bool aad = false; string extension_directory; static idx_t Length(int32_t time[], char micro_buffer[]); // Good: descriptive and clear bool additional_authenticated_data = false; vector extension_directories; static idx_t MicrosLength(int32_t micros, char micro_buffer[]); ``` Names should make code self-documenting and reduce the need for additional comments to understand functionality. --- ## validate before executing operations Always validate inputs and preconditions before executing database operations, and fail explicitly rather than silently ignoring invalid states or configurations. This prevents data corruption, ensures system consistency, and makes debugging easier. Key validation practices: - Check existence of required resources before operations (e.g., replicas, tables, columns) - Validate configuration compatibility (e.g., partition strategies with data lakes) - Throw explicit errors for invalid inputs rather than silently ignoring them - Verify schema consistency before schema changes - Ensure backward compatibility when making breaking changes Example from the discussions: ```cpp // Bad: Silently ignore column list if (open_bracket.ignore(pos, expected)) { // Column list parsing but then ignored } // Good: Validate and throw if invalid if (open_bracket.ignore(pos, expected)) { throw Exception(ErrorCodes::BAD_ARGUMENTS, "Column list specification is not supported in this context"); } // Good: Check existence before operations if (!zookeeper->exists(zookeeper_path + "/replicas/" + replica_name)) { // Skip operation as replica doesn't exist return; } ``` This approach prevents silent failures, makes systems more predictable, and helps maintain data integrity across complex database operations. --- ## Add missing test coverage Identify and address gaps in test coverage by requesting specific tests for untested functionality. When reviewing code changes, look for new features, complex logic, or critical interactions that lack corresponding tests. Focus on testing type inference, component interactions, edge cases, and different code paths. Examples of coverage gaps to watch for: - Type inference validation: "Can you add a test that the inference works? Should not need to provide type on `id` in query func." - Component interaction testing: "Would be good to add a basic test of the interaction with MutationTracker, probably using a mock." - Missing test cases for new functionality: "Please add a test case for push with a compound partition key." - Authentication and authorization logic: "we should really add tests for these" When identifying coverage gaps, be specific about what needs testing and suggest appropriate testing approaches (unit tests, integration tests, mocking, etc.). Prioritize testing critical business logic, error handling, and complex interactions between components. --- ## eliminate code duplication Actively identify and eliminate code duplication in all its forms to improve maintainability and reduce bugs. This includes removing redundant conditional checks, extracting common functionality into reusable functions, and avoiding component or logic duplication. Common patterns to watch for: - **Redundant conditionals**: Remove duplicate checks that are already handled by parent conditions - **Duplicated logic blocks**: Extract identical code (except for small variations like keys) into shared functions - **Component duplication**: Move or reuse existing components instead of creating duplicates - **Logic reuse**: Leverage existing logic instances rather than reimplementing the same functionality Example of redundant conditional elimination: ```tsx // Before: redundant check {!singleFilter && (
{!singleFilter && ( // ❌ Already checked above )}
)} // After: clean structure {!singleFilter && (
// ✅ No redundant check
)} ``` Example of extracting duplicated logic: ```tsx // Before: duplicated except for dismissKey const billingPeriodEnd = values.billing?.billing_period?.current_period_end?.format('YYYY-MM-DD') // ... identical logic with different dismissKey // After: extract to function const handleDismissal = (dismissKey: string) => { const billingPeriodEnd = values.billing?.billing_period?.current_period_end?.format('YYYY-MM-DD') // ... shared logic } ``` Always prefer reusing existing implementations over creating new ones, and extract common patterns into shared utilities when the same logic appears multiple times. --- ## API response standardization Ensure API responses follow established patterns and use proper typing. Always use standardized response types like `PaginatedResponse` for paginated endpoints, avoid hardcoded union types for dynamic values that should be strings, and return fresh data from API responses rather than stale local data. Key practices: - Use established response patterns: `PaginatedResponse` for paginated APIs - Prefer flexible string types over hardcoded unions for dynamic values (e.g., sync names like "Stripe") - Return updated data from API responses, not the original request data - Create reusable type definitions instead of complex inline types Example: ```typescript // Bad: Hardcoded union types and inline complex types async recentActivity(): Promise<{ activities: Array<{ type: 'external_data_sync' | 'materialized_view' // Too rigid }> }> // Good: Standardized response with flexible types type FunctionActionType = 'function' | 'function_email' | 'function_sms' | 'function_slack' | 'function_webhook'; async recentActivity(): Promise> { // Use string for dynamic sync names like "Stripe", "Salesforce" type: string // More flexible for dynamic values } // Always return fresh API data const updatedSource = await api.externalDataSources.update(source.id, source) return { ...values.dataWarehouseSources, results: values.dataWarehouseSources?.results.map((s) => s.id === updatedSource.id ? updatedSource : s // Use fresh data ) || [] } ``` --- ## Verify database state changes When testing database operations that modify schema, data, or metadata, always add comprehensive assertions to verify the expected state changes. Database operations often have side effects beyond the primary action, and tests should validate both the intended outcome and related system state. Key verification patterns: - **Schema changes**: After ALTER operations, verify the new schema with `SHOW CREATE TABLE` or similar introspection queries - **Data operations**: After DROP/DELETE operations, explicitly check data persistence or removal as intended - **Metadata cleanup**: After replica/node removal operations, verify that associated metadata (like ZooKeeper nodes) are properly cleaned up Example from schema evolution test: ```python # After schema modification instance.query(f"ALTER TABLE {TABLE_NAME} MODIFY COLUMN x Int64;") # Verify the schema change took effect result = instance.query(f"SHOW CREATE TABLE {TABLE_NAME}") assert "Int64" in result ``` Example from drop operations: ```python # After dropping table/replica drop_iceberg_table(instance, TABLE_NAME) # Verify data still exists if that's the expected behavior files = get_table_files(TABLE_NAME) assert len(files) > 0, "Drop should not delete user data" ``` This approach catches subtle bugs where operations appear to succeed but don't have the expected side effects, ensuring database tests provide comprehensive coverage of system behavior. --- ## Add monitoring metrics Critical code paths, especially error handling and exception scenarios, should include metrics or counters to enable monitoring and alerting. This prevents systems from failing silently and provides visibility into system health. When implementing error handling, background processes, or exception catching, add appropriate metrics that can be used for dashboards and alerts. This is particularly important for services that could degrade gracefully but cause issues over time if problems go unnoticed. Example: ```typescript // Before: Silent error handling (e) => { logger.error('Error refreshing team retention periods', e) } // After: Add metrics for monitoring (e) => { logger.error('Error refreshing team retention periods', e) statsd.increment('retention_service.refresh_error') } ``` Consider what dashboards and alerts you would want to create when the code is deployed. If you can't easily monitor whether a system is working correctly, add the necessary instrumentation. --- ## Configuration naming clarity Ensure configuration variable names, display labels, and feature flag names accurately reflect their actual purpose and behavior. Misleading or outdated names create confusion for both developers and users, making the codebase harder to maintain and understand. When configuration names don't match their function, it leads to situations where developers must add explanatory comments or use workarounds. For example, avoid using generic names like "New query engine" when the underlying flag is "WEB_ANALYTICS_API", and update variable names when their scope changes (like renaming MAX_SELECT_RETURNED_ROWS when it specifically applies to CSV exports). Example of good practice: ```typescript // Good: Name reflects actual purpose const MAX_CSV_EXPORT_ROWS = 300000 // Bad: Generic name doesn't indicate CSV-specific usage const MAX_SELECT_RETURNED_ROWS = 300000 ``` When introducing new features or changing existing ones, take time to review and update related configuration names to maintain clarity and prevent technical debt. --- ## leverage framework capabilities Structure workflows and activities to take full advantage of the orchestration framework's built-in capabilities for fault tolerance, retries, and durable execution rather than implementing custom solutions. Break down complex operations into separate, resumable steps that can leverage the framework's retry and restart mechanisms. Avoid duplicating functionality that the framework already provides automatically. For example, instead of implementing a single monolithic operation that handles multiple steps internally: ```python # Avoid: Single operation that can't resume individual steps @dagster.op def snapshot_all_project_data(context, project_id: int, s3: S3Resource): snapshot_postgres_data(...) # If this succeeds but next fails, snapshot_clickhouse_data(...) # we can't resume from here # Prefer: Separate operations that can be retried independently @dagster.op def snapshot_postgres_project_data(context, project_id: int, s3: S3Resource): # This can be retried/resumed independently @dagster.op def snapshot_clickhouse_project_data(context, project_id: int, s3: S3Resource): # This can be retried/resumed independently ``` Similarly, remove redundant code when the framework already provides the needed functionality automatically, such as context attributes for metrics or built-in retry policies. --- ## Optimize database query patterns Avoid N+1 query problems and overly complex conditional SQL construction. When loading related data, prefer batch operations or dedicated endpoints that can fetch all required data in fewer queries. For conditional SQL parameters, consider using null values with database defaults instead of constructing different query strings. Example of the problem: ```typescript // Avoid: Multiple individual requests (N+1 pattern) const results = await Promise.all( dataSources.map(async (source) => { const jobs = await api.externalDataSources.jobs(source.id, monthStartISO, null) return sumMTDRows(jobs, monthStartISO) }) ) // Avoid: Complex conditional SQL construction const query = forcedId ? `INSERT INTO posthog_person (id, created_at, ...) VALUES ($1, $2, ...)` : `INSERT INTO posthog_person (created_at, ...) VALUES ($1, $2, ...)` ``` Better approaches: ```typescript // Prefer: Single batch request const allJobsData = await api.externalDataSources.batchJobs(dataSourceIds, monthStartISO) // Prefer: Always include parameter, use null for auto-increment const query = `INSERT INTO posthog_person (id, created_at, ...) VALUES ($1, $2, ...)` // Pass null for id when auto-increment is desired ``` This reduces database load, improves performance, and simplifies code maintenance by eliminating conditional query construction. --- ## Test edge cases comprehensively Ensure your tests cover not just the happy path, but also edge cases, empty states, error conditions, and boundary scenarios. This includes testing with empty datasets, validating pagination logic, handling unusual input values, and verifying error handling behavior. When writing tests, systematically consider: - Empty or null inputs (empty models, missing data) - Boundary conditions (timezone edge cases like +5:30 offsets) - Error scenarios and exception handling - Pagination and query logic validation Example from the discussions: ```python def test_recent_activity_includes_external_jobs_and_modeling_jobs(self): # Don't just test the happy path - also test: # - Empty states (if either model is empty) # - Pagination works correctly # - Query logic handles edge cases properly ``` ```python def test_timezone_edge_cases(self): # Test unusual timezone offsets like India +5:30 # Even if it's to assert that something throws an error ``` Comprehensive edge case testing catches bugs early, validates assumptions about system behavior, and ensures robust handling of real-world scenarios that may not be immediately obvious during development. --- ## Use descriptive semantic names Names should accurately reflect their purpose, behavior, and semantic meaning to avoid confusion and improve code readability. Avoid misleading names that don't match the actual functionality or content. Key principles: - Function names should match their actual behavior (e.g., `projectCorrelatedColumns` when called with non-correlated data should be renamed to reflect what it actually does) - Variable names should be specific and unambiguous (e.g., `cached_reader` is misleading when it specifically refers to `CachedInMemoryReadBufferFromFile` - use `page_cache_reader` instead) - Parameter names should clearly indicate their role (e.g., `column_idx` should be `presence_column_idx` if it's specifically the index of a presence column) - Avoid abbreviations that sacrifice clarity (e.g., use `storage` or `table` instead of `tbl`) - Method names should indicate what they return or do (e.g., `getHivePart()` should be split into separate methods if it also performs normalization) Example of improvement: ```cpp // Before: misleading name void projectCorrelatedColumns(rhs_plan, ...); // but no correlation involved // After: accurate name void projectIdentifierColumns(rhs_plan, ...); // clearly indicates projecting from identifiers // Before: ambiguous variable auto * cached_reader = typeid_cast(&reader); // After: specific variable auto * page_cache_reader = typeid_cast(&reader); ``` This ensures that code is self-documenting and reduces cognitive load when reading and maintaining the codebase. --- ## Use descriptive names Choose names that clearly communicate purpose and accurately represent what they describe. Avoid ambiguous or misleading names that require additional context to understand. Key principles: - **Be specific**: Use precise terms that indicate the actual purpose or content - **Avoid generic terms**: Replace vague names with descriptive alternatives - **Match semantic meaning**: Ensure names accurately reflect what they represent Examples of improvements: ```python # Too generic/ambiguous property_type: str # What kind of property type? closure: bool # What does this boolean represent? key = name or "default" # Key for what? Too generic # More descriptive property_type: PropertyTypeEnum # Use enum for type safety is_factory: bool # Clearly indicates factory pattern registration_key = f"{module_name}#{function_name}" # Specific and unique # Misleading names class FeatureFlagToolkit: # Actually handles surveys pass class OriginProduct: # Not all values are products pass # Accurate names class SurveyToolkit: # Accurately describes purpose pass class Origin: # Broader, more accurate term pass ``` When naming is unclear, consider: "Would a new developer understand this name without additional context?" If not, choose a more descriptive alternative. --- ## semantic naming accuracy Ensure that method, variable, and class names accurately reflect their actual behavior, purpose, or content. Names should semantically match what they represent to prevent confusion and bugs. Key principles: - Method names should match their actual behavior: `detachX()` should perform `get + cancel`, not just `cancel` - Variable names should reflect their content: `patch_block_indices` not `patch_col_indices` when indexing blocks - Class names should accurately describe their function: `Serialization` not `Compression` when the class handles serialization - Method names should be descriptive of their purpose: `shouldSearchForPartsOnDisk()` not `lookOnDisk()` - Use consistent, accurate terminology throughout: `disk` not `drive` when referring to storage disks Example of problematic vs. corrected naming: ```cpp // Problematic - method name doesn't match behavior void detachParallelReadingExtension(); // actually just clears/cancels // Corrected - name matches actual behavior void clearParallelReadingExtension(); // or implement proper detach semantics // Problematic - variable name doesn't match content PaddedPODArray patch_col_indices; // actually indexes blocks // Corrected - name matches content PaddedPODArray patch_block_indices; ``` This prevents misunderstandings about code behavior and makes the codebase more maintainable by ensuring names serve as accurate documentation of functionality. --- ## Cache invalidation consistency Ensure comprehensive and consistent cache invalidation patterns across all models that affect cached data. Every model that can impact cached content must have proper invalidation signals, and cache keys should be designed to enable targeted busting without affecting unrelated data. Key requirements: 1. **Complete signal coverage**: Add both `post_save` and `post_delete` receivers for all models that affect cached data, even if soft deletes are primarily used 2. **Consistent invalidation patterns**: Don't rely on developers to remember to add cache busting - make it systematic and hard to miss 3. **Targeted cache keys**: Use Redis hashes with structured keys like `team_id:git_sha:model_name` to enable selective invalidation without clearing unrelated cache entries 4. **Handle related model changes**: Consider how foreign key and many-to-many relationships affect cached data and ensure those changes also trigger appropriate invalidation Example implementation: ```python # Bad - easy to forget invalidation for new models class ExternalDataSource(models.Model): objects: CacheManager = CacheManager() # Missing: no invalidation signals # Good - systematic invalidation pattern class ExternalDataSource(models.Model): objects: CacheManager = CacheManager() @receiver(post_save, sender=ExternalDataSource) @receiver(post_delete, sender=ExternalDataSource) # Even with soft deletes def invalidate_external_data_source_cache(sender, instance, **kwargs): ExternalDataSource.objects.invalidate_cache(instance.team_id) ``` This prevents the common issue where "ORM writes (including bulk updates, signals, admin edits, scripts) can bypass your invalidation path → stale cache" and ensures that cache invalidation is not an afterthought that developers can easily miss when adding new cached models. --- ## validate inputs comprehensively Ensure thorough input validation and comprehensive edge case handling in database operations. This includes checking parameter validity, handling null inputs, testing boundary conditions, and covering all comparison operators or function variants. Key practices: - Validate input parameters before processing (check for null pointers, out-of-bounds indices, type mismatches) - Verify logical preconditions (e.g., check if bound column reference matches expected indexed column before modifying) - Add comprehensive test coverage for edge cases, error conditions, and all supported operations - Handle missing or incomplete cases (e.g., add missing comparison operators like IS [NOT] DISTINCT) Example from column binding validation: ```cpp case ExpressionClass::BOUND_COLUMN_REF: { auto &bound_column_ref_expr = expr->Cast(); // Validate that the bound column actually matches the indexed column if (bound_column_ref_expr.binding.column_index == indexed_columns[0]) { for (idx_t i = 0; i < input_column_ids.size(); ++i) { if (input_column_ids[i] == indexed_columns[0]) { bound_column_ref_expr.binding.column_index = i; return; } } } } ``` This approach prevents logic errors, improves reliability, and ensures database operations handle all valid inputs and edge cases correctly. --- ## Split complex migrations incrementally Break complex schema changes into multiple, sequential migrations to ensure deployment safety and proper data handling. Each migration should represent a single, atomic change that can succeed or fail independently. **Why this matters:** - Multiple migrations in one PR can cause deployment failures if one succeeds and another fails, leaving the database out of sync with the codebase - Complex changes are safer when split into logical steps that can be rolled back individually - Incremental approach allows for safer data migration and validation at each step **Recommended approach:** 1. **Add nullable field** - Introduce new columns as nullable first 2. **Migrate data** - Populate the new field with appropriate values 3. **Add constraints** - Apply NOT NULL constraints or other restrictions after data is migrated 4. **Remove old fields** - Deprecate or remove old columns in separate migration **Example:** Instead of creating and altering in one migration: ```python # Avoid: Complex migration doing multiple operations operations = [ migrations.AddField(model_name="cohort", name="cohort_type", field=models.CharField(max_length=20, choices=CHOICES)), migrations.AlterField(model_name="cohort", name="cohort_type", field=models.CharField(max_length=20, choices=CHOICES, null=False)), ] ``` Prefer incremental steps across separate PRs: ```python # Step 1: Add nullable field field=models.CharField(max_length=20, choices=CHOICES, null=True, blank=True) # Step 2 (separate PR): Migrate existing data # Step 3 (separate PR): Drop null constraint ``` **One migration per PR** - Keep each PR focused on a single migration to avoid deployment synchronization issues. --- ## Documentation completeness standards Ensure all documentation meets completeness and formatting standards. This includes: (1) Adding language specifications to all fenced code blocks to satisfy linting requirements, (2) Including deprecation notes with clear migration guidance when removing or replacing features, and (3) Following established documentation patterns and embedding guidelines. For code blocks, always specify the language: ```sql DESC format(JSONEachRow, '{"arr" : [42, "hello", [1, 2, 3]]}'); ``` For deprecated features, provide clear migration paths: ``` system.latency_log was deprecated after version 25.x, instead system.histogram_metrics should be used ``` Consider embedding documentation directly in code when appropriate, as this keeps documentation close to implementation and reduces maintenance overhead. --- ## Comprehensive database testing Database tests should verify both query execution and optimization behavior through comprehensive scenario coverage. When testing query optimizations, include EXPLAIN PLAN verification to ensure the optimizer produces expected execution plans. For database features like row policies or security constraints, test multiple scenarios including edge cases and complex configurations. Example approach: ```sql -- Test the actual query execution SELECT 1 FROM t2 LEFT JOIN t1 ON t2.id = t1.fid LEFT JOIN t3 ON t1.tid = t3.id WHERE (t2.resource_id IS NOT NULL) AND (t2.status IN ('OPEN')); -- Also verify the execution plan EXPLAIN PLAN SELECT 1 FROM t2 LEFT JOIN t1 ON t2.id = t1.fid... -- Test multiple policy scenarios CREATE ROW POLICY permissive_policy ON table USING condition1; CREATE ROW POLICY restrictive_policy ON table USING condition2; -- Test cases where policy columns aren't selected SELECT other_column FROM table WHERE some_condition; ``` This ensures database functionality works correctly and performs optimally across various real-world scenarios, catching both functional bugs and performance regressions. --- ## Document implementation rationale When implementing networking or system-level functionality where multiple approaches exist, always document the rationale behind your implementation choice. This is especially critical when the chosen method may not be the most obvious one or when platform-specific considerations influence the decision. Include relevant documentation excerpts or specifications that support your choice, rather than just providing URLs. Explain why alternative approaches were rejected, particularly regarding reliability, portability, or system compatibility. Example: ```cpp /// Unlike s3 and azure, which are object storages, /// hdfs is a filesystem, so it cannot list files by partial prefix, /// only by directory. /// Therefore in the below methods we use allow_partial_prefix=false. ``` This practice prevents future confusion, reduces the likelihood of well-intentioned but incorrect "improvements," and helps maintainers understand the constraints and trade-offs that shaped the implementation. For networking code, this documentation is particularly valuable since different protocols, platforms, and network configurations may require different approaches. --- ## preserve existing environment variables When modifying environment variables in code, always preserve existing values by appending or merging rather than overwriting. This prevents breaking existing configurations that may have been set by the system, other tools, or previous code execution. Environment variables often contain multiple space-separated or colon-separated values that work together. Overwriting these variables can disable important functionality or cause unexpected behavior. Example of the problem: ```python # BAD: Overwrites existing TSAN_OPTIONS env["TSAN_OPTIONS"] = f"memory_limit_mb={tsan_memory_limit_mb}" ``` Example of the correct approach: ```python # GOOD: Preserves existing TSAN_OPTIONS and appends new values tsan_options = f"memory_limit_mb={tsan_memory_limit_mb}" env["TSAN_OPTIONS"] = env.get("TSAN_OPTIONS", "") + " " + tsan_options ``` This practice is especially important for debugging and profiling tools like TSAN, ASAN, and other environment-based configurations where multiple options need to coexist. --- ## Verify error handling paths When implementing error handling logic, ensure that both the behavior and reasoning are clear, and that error paths are properly tested at appropriate levels. For methods that handle missing data, the default behavior should be explicitly documented and the method name should clearly indicate what the return value represents. For exception handling, verify that upstream code properly catches and handles the exceptions to ensure graceful failure. Example from the discussions: ```typescript // Good: Clear method name and documented default behavior private async isRecipientOptedOutOfAction(invocation, action): Promise { // ... logic to find recipient if (!recipient) { // Default to opted-in if no preference exists (per TOS) return false; // false = not opted out = can send message } // ... rest of logic } // When throwing exceptions, ensure upstream handling exists if (this.isPropertiesSizeConstraintViolation(error)) { logger.warn('Rejecting person properties create/update, exceeds size limit', { team_id: teamId, violation_type: 'create_person_size_violation', }); throw new PersonPropertiesSizeViolationError(/* ... */); } ``` Always verify that exception handling is tested at the service level to ensure the application fails gracefully rather than crashing unexpectedly. --- ## Document authentication precedence When implementing systems that support multiple authentication methods, always clearly document which authentication method takes precedence when multiple options are provided. This prevents security misconfigurations and ensures predictable behavior. Failing to document authentication precedence can lead to: - Developers making incorrect assumptions about which credentials will be used - Potential security vulnerabilities if weaker authentication methods are unexpectedly prioritized - Inconsistent behavior across different environments Example from Azure Blob Storage documentation: ``` - `account_key` - if storage_account_url is used, then account key can be specified here - `extra_credentials` - Use `client_id` and `tenant_id` for authentication instead of `account_name` and `account_key`. If extra_credentials are provided, they are given priority over account name & account key. ``` Always explicitly state which authentication method will be used when multiple are configured, ensuring users understand the security implications of their configuration choices. --- ## Eliminate code duplication Actively identify and eliminate code duplication to improve maintainability and reduce the risk of inconsistent behavior. This includes several strategies: **Reuse existing utilities instead of reimplementing:** ```cpp // Instead of writing custom switch-case: switch (expr->GetExpressionClass()) { case ExpressionClass::BOUND_COLUMN_REF: // handle case break; // ... more cases } // Use existing utility: ExpressionIterator::EnumerateChildren(expr, [&](Expression &child) { // handle child }); ``` **Extract common functionality into shared methods:** ```cpp // Instead of copy-pasted functions: string FileHandle::ReadLine() { // duplicate implementation } string FileHandle::ReadLine(QueryContext context) { // same implementation with context } // Extract common logic: string FileHandle::ReadLine() { return ReadLine(QueryContext()); } ``` **Use offset variables instead of duplicating code blocks:** ```cpp // Instead of duplicating based on boolean flags: if (!lhs_first) { for (idx_t i = 0; i < right_projection_map.size(); i++) { // code block A } for (idx_t i = 0; i < left_projection_map.size(); i++) { // code block B } } else { for (idx_t i = 0; i < left_projection_map.size(); i++) { // code block B (duplicate) } for (idx_t i = 0; i < right_projection_map.size(); i++) { // code block A (duplicate) } } // Use offset approach: idx_t left_offset = lhs_first ? 0 : right_projection_map.size(); idx_t right_offset = lhs_first ? left_projection_map.size() : 0; // Single implementation using offsets ``` **Create dedicated classes for repeated patterns:** When the same pattern appears across multiple locations, extract it into a reusable class or struct. This is especially important for complex initialization patterns, common data structures, or frequently used utility functions. Always check if existing utilities can accomplish the same goal before implementing new functionality. --- ## Validate before unsafe operations Always validate values before performing operations that could result in undefined behavior, particularly dynamic casts and arithmetic on unsigned types that might underflow. For dynamic_cast operations, check the result for nullptr before use, even if you expect the cast to succeed: ```cpp DatabaseReplicated * replicated_database = dynamic_cast(database.get()); if (!replicated_database) { throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected replicated database"); } ``` For unsigned arithmetic that could underflow, validate inputs first: ```cpp // Before: UInt8 event_idx = event_type - 1; // Underflows if event_type == 0 // After: if (event_type == 0) { // Handle the no-match case appropriately continue; } UInt8 event_idx = event_type - 1; ``` This defensive approach prevents crashes and undefined behavior that can be difficult to debug, especially when the problematic values come from external data or user input. The small performance cost of validation is typically worthwhile compared to the cost of debugging production crashes. --- ## break down large functions Large functions that handle multiple responsibilities should be decomposed into smaller, focused functions to improve readability, maintainability, and facilitate code reviews. When a function becomes difficult to understand at a glance or handles multiple distinct operations, consider extracting logical chunks into separate methods. This is especially important when functions exceed ~50 lines or when reviewers comment that the code would benefit from splitting. **Example of improvement:** ```python # Before: Large function doing multiple things def _process_insight_for_evaluation(self, insight: Insight, query_executor: AssistantQueryExecutor) -> dict: insight_info = self._create_base_insight_info(insight) try: query_dict = self._parse_insight_query(insight) if query_dict: self._execute_and_update_info(insight_info, query_dict, query_executor) self._add_visualization_message(insight_info, insight) else: self._handle_no_query(insight_info, insight) except Exception as e: self._handle_evaluation_error(insight_info, insight, e) return insight_info # After: Broken into focused helper methods def _parse_insight_query(self, insight: Insight) -> dict | None: # Separate method for query parsing pass def _execute_and_update_info(self, insight_info: dict, query_dict: dict, executor): # Separate method for execution pass ``` This approach makes each function easier to test, understand, and modify independently. It also makes code reviews more focused since reviewers can evaluate each logical operation separately. --- ## Document non-obvious code Add explanatory comments for any code element that is not self-explanatory, including complex return types, method parameters, classes, and technical concepts that may not be familiar to all developers. Key areas requiring documentation: - **Complex return types**: Functions returning `std::pair` or other composite types should explain what each element represents - **Method parameters**: Non-obvious parameters should have comments explaining their purpose and when they are used - **Classes and structs**: All classes should have brief class-level comments explaining their purpose - **Technical concepts**: Specialized implementations or domain-specific concepts should be explained for developers unfamiliar with the domain - **Method names**: Methods with unclear names should have comments explaining their functionality Examples: ```cpp // Bad - no explanation of return values std::pair setupHivePartitioning(...); // Good - clear explanation /// Returns a pair of (partition_columns, regular_columns) extracted from the path std::pair setupHivePartitioning(...); // Bad - no explanation of parameter void prepareProcessedRequests(Coordination::Requests & requests, LastProcessedFileInfoMapPtr created_nodes = nullptr); // Good - parameter purpose explained /// Prepare keeper requests, required to set file as Processed. /// @param created_nodes - map of newly created nodes for tracking, passed when... void prepareProcessedRequests(Coordination::Requests & requests, LastProcessedFileInfoMapPtr created_nodes = nullptr); // Bad - no class documentation struct HiveStylePartitionStrategy : PartitionStrategy // Good - explains the concept /// Implements Hive-style partitioning where data is organized into directories /// based on partition key values (e.g., year=2023/month=01/data.parquet) struct HiveStylePartitionStrategy : PartitionStrategy ``` The goal is to make code self-documenting and reduce the cognitive load for future maintainers who may not be familiar with the specific domain or implementation details. --- ## Database schema consistency Ensure database operations maintain consistent behavior and ordering, especially in distributed systems. When working with data structures that affect schema representation or query results, prioritize deterministic ordering over performance optimizations to prevent schema mismatches and ensure reliable distributed operations. Key considerations: - Use ordered containers (like `std::map`) instead of unordered ones (`absl::flat_hash_map`) when column ordering affects schema equality or distributed consistency - Avoid ignoring important write modes or operations that could affect data integrity - Ensure virtual columns and metadata operations have consistent, well-defined semantics - Validate that query optimizations don't change correctness of results Example from the codebase: ```cpp // Before: Unordered map causing schema mismatches absl::flat_hash_map & map; // After: Ordered map ensuring consistent schema representation std::map & map; ``` This prevents distributed systems from encountering schema mismatch exceptions due to inconsistent column ordering in operations like `ColumnsDescription::toString()` equality checks. --- ## Extract common patterns Identify and extract repeated code patterns into reusable functions or methods to improve maintainability and reduce duplication. When you notice similar logic appearing in multiple places, create a common utility function or method that can be shared. Examples of patterns to extract: **Callback pattern extraction:** ```cpp // Instead of repeating this pattern everywhere: settings.path.push_back(substream); callback(settings.path); settings.path.pop_back(); // Extract to a common method: const auto call_callback_for_substream = [&](const auto substream) { settings.path.push_back(substream); callback(settings.path); settings.path.pop_back(); }; ``` **String joining patterns:** ```cpp // Instead of manual loops: for (const auto & column_name : copy_query->column_names) { // manual concatenation logic } // Use existing utilities: boost::algorithm::join(copy_query->column_names, ", "); ``` **Reuse existing functions:** ```cpp // Instead of duplicating similar logic: for (const auto & command : commands) { // duplicate partition ID extraction logic } // Reuse existing functionality: MergeTreeData::getPartitionIdsAffectedByCommands(commands); ``` This approach reduces maintenance burden, prevents bugs from inconsistent implementations, and makes the codebase more cohesive. Always check if similar functionality already exists before implementing new logic. --- ## Documentation precision standards Ensure API documentation uses precise type specifications, proper formatting, and clear language to improve developer understanding and usability. Key requirements: - Use specific generic types instead of generic placeholders (e.g., `Array(T)` instead of `Array`) - Apply backticks around technical terms, numbers, and variable names for proper formatting (e.g., "`1` to `N`, where `N` is number of capturing groups") - Remove confusing notation like square brackets in parameter names (use `max_substrings` not `[max_substrings]`) - Simplify overly complex or confusing descriptions with clear, direct language Example of improved documentation: ```cpp FunctionDocumentation::Arguments arguments = { {"arr", "The array to concatenate.", {"Array(T)"}}, // Specific type {"max_substrings", "Optional. When `max_substrings > 0`...", {"Int64"}} // Clear naming, backticks for technical terms }; ``` This ensures documentation serves as an effective reference for developers and reduces confusion about parameter types and usage. --- ## Document mutex responsibilities Use Thread Safety Analysis (TSA) annotations to explicitly document which mutexes protect which variables, making thread safety relationships clear to both static analyzers and code readers. Every mutex should have a clear, documented purpose with TSA annotations that specify what data it protects. This prevents issues like unused mutexes, inconsistent locking patterns, and unclear concurrent access semantics. Example using TSA annotations: ```cpp class XRayInstrumentationManager { private: mutable std::mutex functions_mutex; std::list instrumented_functions GUARDED_BY(functions_mutex); InstrumentedFunctions getInstrumentedFunctions() const REQUIRES(functions_mutex) { std::lock_guard lock(functions_mutex); return instrumented_functions; } }; ``` For read-only concurrent access, use const-correctness and shared locks with clear documentation: ```cpp // Functions can access this info in parallel for reading purposes only std::pair> getIndexInfo() const; ``` TSA macros like `GUARDED_BY`, `REQUIRES`, and `EXCLUDES` help catch threading issues early and make code self-documenting about its concurrency assumptions. --- ## Enrich telemetry context Always include relevant contextual metadata when capturing telemetry data (events, exceptions, logs, metrics) to improve debugging and operational visibility. This includes user information, team ownership, product areas, IP addresses, user agents, geographic data, and other relevant context that helps with troubleshooting and error assignment. When capturing exceptions, include contextual information like: ```python capture_exception( e, { "project_id": project_id, "user_id": request.user.id if request.user else None, "user_distinct_id": request.user.distinct_id if request.user else None, "ai_product": "wizard", "team": "growth" # Makes it easier for auto-assignment in error tracker } ) ``` When logging events or actions, include debugging properties: ```python report_user_action(user=user, event="login notification sent", properties={ "ip_address": ip_address, "user_agent": short_user_agent, "location": location, "login_time": login_time_str }) ``` Rich contextual data transforms raw telemetry into actionable insights, enabling faster debugging, better error routing, and more effective monitoring. --- ## optimize algorithmic complexity Always consider the algorithmic complexity and performance implications of data structure choices, memory allocation patterns, and container operations. This includes pre-allocating containers when sizes are known, choosing appropriate data structures based on usage patterns, and measuring performance impact of changes. Key practices: 1. **Pre-allocate containers**: Reserve memory when the final size is predictable to avoid repeated reallocations 2. **Choose containers wisely**: Consider trade-offs between different data structures (e.g., `std::vector` vs `std::unordered_set` vs `std::set`) based on access patterns, insertion frequency, and memory constraints 3. **Size buffers defensively**: When working with external libraries that require "big enough" buffers, add safety margins (e.g., +20%) to prevent overflows 4. **Benchmark performance changes**: Always measure performance impact when modifying algorithms, especially for hot paths Example of good practice: ```cpp // Pre-allocate when size is known structure_granule.all_paths.reserve(structure_granule.num_paths); // Choose appropriate container based on usage // For frequent lookups with few duplicates: std::unordered_set // For ordered iteration: std::set // For simple iteration: std::vector constexpr size_t initial_size_degree = 9; // Conservative default for hash tables ClearableHashSetWithStackMemory, initial_size_degree> set; // Size buffers defensively for external libraries PaddedPODArray compressed_buffer(uncompressed_size * 1.2); // +20% safety margin ``` This approach prevents performance regressions, reduces memory fragmentation, and ensures predictable behavior under different load conditions. --- ## Cache expensive operations Identify and eliminate redundant expensive operations by implementing caching, memoization, or conditional execution. Look for repeated database queries, complex calculations, object creation, or data processing that can be cached or avoided entirely when not needed. Common patterns to optimize: - **Repeated database queries**: Extract common query logic and cache results - **Expensive lookups**: Cache lookup results to avoid O(n²) complexity when searching through collections multiple times - **Conditional expensive operations**: Skip unnecessary work when results won't be used (e.g., timing calculations when debug mode is off) - **Object recreation**: Cache expensive-to-create objects like compiled regex patterns or metric meters Example of query optimization: ```python # Before: Duplicated query logic if cache_enabled: tables = list( DataWarehouseTable.objects.filter(team_id=team.pk) .exclude(deleted=True) .select_related("credential", "external_data_source") .fetch_cached(team_id=team_id or team.pk, key_prefix=CACHE_KEY_PREFIX) ) else: tables = list( DataWarehouseTable.objects.filter(team_id=team.pk) .exclude(deleted=True) .select_related("credential", "external_data_source") ) # After: Extract common logic, apply caching conditionally tables = DataWarehouseTable.objects.filter(team_id=team.pk) .exclude(deleted=True) .select_related("credential", "external_data_source") if cache_enabled: tables = tables.fetch_cached(team_id=team_id or team.pk, key_prefix=CACHE_KEY_PREFIX) ``` Example of lookup caching: ```python # Add caching to avoid O(n²) complexity def _get_all_loaded_insight_ids(self) -> set[int]: """Get all insight IDs from loaded pages.""" if not hasattr(self, '_cached_insight_ids'): all_ids = set() for page_insights in self._loaded_pages.values(): for insight in page_insights: all_ids.add(insight.id) self._cached_insight_ids = all_ids return self._cached_insight_ids ``` --- ## Maintain naming consistency Ensure consistent naming conventions, terminology, and identifiers across the entire codebase. Names should be uniform between frontend/backend, across different modules, and within the same domain. Key areas to check: - **Cross-system consistency**: Frontend and backend should use the same identifiers for the same concepts (e.g., `search_docs` vs `search_documentation`) - **Module consistency**: Related modules should use consistent naming patterns (e.g., `sdks` vs `onboarding` logic should align with their actual scope) - **Terminology consistency**: Use the same terms throughout the codebase for the same concepts (e.g., standardize on "identify respondents" rather than mixing "track/collect/identify responses") - **Type consistency**: Use the same enums/types across frontend and backend for shared concepts - **Import consistency**: Consistently use the same import sources (e.g., always use `urls` rather than mixing `urls` and `productUrls`) Example of inconsistent naming: ```typescript // Frontend uses different identifier than backend identifier: 'search_docs' as const, // Frontend // vs search_documentation // Backend API path // Mixed terminology in UI "Track responses to users" "Collect user responses" "Identify respondents" // Should standardize on one ``` Before merging, verify that new names align with existing patterns and that any changes maintain consistency across all related files and systems. --- ## Constructor configuration injection Prefer injecting configuration objects through constructors rather than passing them as ad-hoc parameters or using global instances. This approach improves extensibility, reduces error-prone context passing, and enables per-instance customization. Instead of passing configuration parameters to individual methods or relying on global singletons, inject configuration objects during construction: ```cpp // Avoid: Ad-hoc parameter passing void LoadExistingDatabase(optional_ptr encryption_key = nullptr); read_head.buffer_handle = file_handle.Read(QueryContext(), read_head.buffer_ptr, read_head.size, read_head.location); // Prefer: Constructor injection class ThriftFileTransport { ClientContext &context; public: ThriftFileTransport(ClientContext &ctx) : context(ctx) {} // Use injected context instead of creating new QueryContext() }; class SingleFileBlockManager { StorageManagerOptions options; public: SingleFileBlockManager(StorageManagerOptions opts) : options(opts) {} // Use options.encryption_key instead of method parameters }; ``` Avoid global configuration instances that prevent extension customization. Instead, store configuration in `DatabaseInstance` or `DBConfig` to support per-instance settings and extension registration. --- ## Check existence before operations Always verify that keys, IDs, indices, or other required values exist before performing operations that depend on them. This prevents runtime errors and unexpected behavior from null references or missing data. Key patterns to follow: - Check if dictionary keys exist before registration or access - Validate that database IDs exist before queries - Verify array indices are within bounds before access - Use early returns for cleaner null handling: `if value is None: continue` Example of the pattern: ```python # Before: Potential KeyError or duplicate registration REGISTERED_FUNCTIONS[key] = func # After: Check existence first if key in REGISTERED_FUNCTIONS: raise ValueError(f"Function {key} already registered") REGISTERED_FUNCTIONS[key] = func # Before: Potential IndexError summary_text_chunk = summary[0]["text"] # After: Check bounds and existence if summary and len(summary) > 0: summary_text_chunk = summary[0]["text"] ``` This proactive approach prevents silent failures and makes code more robust by catching potential issues at the point of access rather than allowing them to propagate. --- ## RESTful endpoint organization API endpoints should be properly organized around resources and follow RESTful principles. Avoid placing aggregation or utility endpoints in viewsets where they don't belong conceptually. Each viewset should represent a specific resource type, and endpoints within that viewset should operate on that resource. When you need aggregation endpoints or cross-resource operations, create dedicated API endpoints rather than forcing them into existing viewsets. Example of what to avoid: ```python # DON'T: Adding aggregation endpoint to external_data_source viewset class ExternalDataSourceViewSet(ModelViewSet): @action(methods=["GET"], detail=False) def dwh_scene_stats(self, request): # This doesn't belong here # Returns aggregated data warehouse statistics pass ``` Example of proper organization: ```python # DO: Create dedicated endpoint for aggregations class DataWarehouseViewSet(ModelViewSet): @action(methods=["GET"], detail=False) def scene_stats(self, request): # Returns aggregated data warehouse statistics pass ``` For individual vs collection resources, handle both cases properly: ```python # Handle both individual survey and survey collection def surveys(request: Request): if survey_id: # Return individual survey with proper error handling try: survey = Survey.objects.get(id=survey_id, team=team) return JsonResponse({"survey": serialized_survey}) except Survey.DoesNotExist: return JsonResponse({"error": "Survey not found"}, status=404) # Return collection of surveys return JsonResponse(get_surveys_response(team)) ``` This approach makes APIs more intuitive, maintainable, and follows REST conventions that frontend developers expect. --- ## optimize data loading Review data loading operations to ensure they are properly scoped, filtered, and batched to prevent performance issues. Large datasets should be handled with appropriate pagination, filtering by relevant criteria (like date ranges), and avoiding operations that could load excessive amounts of data into memory. Key areas to check: - Avoid spread operators with large arrays that could cause memory issues - Ensure pagination limits don't truncate important data - consider if limits like "last 200 jobs" could miss records in high-volume scenarios - Apply proper filtering before loading data rather than loading everything and filtering later - Question whether all requested data is actually needed for the use case Example of problematic pattern: ```typescript // Loads ALL jobs for ALL sources without filtering const allJobs = await Promise.all( dataSources.map(async (source) => { // This could return 283,914 items for large teams return await api.externalDataSources.jobs(source.id, null, null) }) ) ``` Better approach: ```typescript // Apply filtering and reasonable limits upfront const recentJobs = await Promise.all( dataSources.map(async (source) => { return await api.externalDataSources.jobs( source.id, cutoffDate, // Filter by date REASONABLE_LIMIT // Appropriate batch size ) }) ) ``` --- ## Test complex logic thoroughly When implementing complex business logic, state management, or algorithms with multiple edge cases, ensure comprehensive test coverage. Complex logic is prone to bugs and difficult to reason about during code review, making tests essential for maintainability and correctness. Focus on: - **Extract testable functions**: Break down complex logic into smaller, pure functions that can be easily unit tested - **Cover edge cases**: Test boundary conditions, error scenarios, and potential infinite loops - **Test state transitions**: For stateful logic, verify all possible state changes and their side effects Example from state management code: ```typescript // Instead of testing the entire complex state logic inline initializeMessageStates: ({ inputCount, outputCount }) => { // Complex state calculation logic here... } // Extract into testable utility functions const calculateMessageStates = (inputCount: number, outputCount: number) => { // Logic here - now easily testable } // Test the extracted function thoroughly describe('calculateMessageStates', () => { it('should handle edge case where counts exceed limits', () => { // Test potential infinite loop scenario }) }) ``` This approach makes code review easier by allowing reviewers to focus on the test cases to understand the expected behavior, rather than trying to mentally trace through complex logic paths. --- ## Write focused, clear tests Tests should be simple, focused on a single concern, and easy to understand. When a test combines multiple potential failure conditions or verification points, split it into separate, focused test cases. Avoid over-engineering tests with unnecessary complexity. For example, instead of testing multiple failure conditions in one test: ```sql -- Bad: Unclear which aspect causes failure CREATE TABLE codecTest (c0 LowCardinality(Nullable(Time)) CODEC(DoubleDelta)) ENGINE = MergeTree() ORDER BY tuple(); -- { serverError BAD_ARGUMENTS } ``` Split into focused tests: ```sql -- Good: Each test focuses on one specific failure condition CREATE TABLE codecTest (c0 Time CODEC(DoubleDelta)) ENGINE = MergeTree() ORDER BY tuple(); -- { serverError BAD_ARGUMENTS } CREATE TABLE codecTest (c0 LowCardinality(String) CODEC(DoubleDelta)) ENGINE = MergeTree() ORDER BY tuple(); -- { serverError BAD_ARGUMENTS } CREATE TABLE codecTest (c0 Nullable(Int32) CODEC(DoubleDelta)) ENGINE = MergeTree() ORDER BY tuple(); -- { serverError BAD_ARGUMENTS } ``` Similarly, simplify tests that verify basic functionality rather than adding unnecessary verification layers. If the goal is to test that queries don't crash, simple SELECT statements are sufficient without complex EXPLAIN PLAN analysis. --- ## Avoid unnecessary allocations Minimize memory allocations, data copying, and expensive operations by implementing early exits, using move semantics, and choosing appropriate buffer management strategies. Key optimization strategies: 1. **Early exits**: Return immediately when the result is known to avoid processing remaining data 2. **Move semantics**: Use std::move() instead of copying large objects like buffers 3. **Proper buffer sizing**: Use count() instead of offset() for total bytes, and resize() instead of repeated emplace_back() 4. **Avoid expensive conversions**: Use direct data insertion instead of converting to intermediate types like Field 5. **Cache sizing**: Limit cache sizes to prevent memory bloat in long-running threads Example of early exit optimization: ```cpp // Instead of checking condition in loop for (size_t row = 0; row < input_rows_count; ++row) { dst_data[row] = filter ? filter->find(value.data, value.size) : true; } // Return early when filter is not found if (!filter) { auto dst = ColumnVector::create(); dst->getData().resize_fill(input_rows_count, 1); return dst; } ``` Example of avoiding unnecessary copying: ```cpp // Instead of copying buffer before send message_transport->send(PostgreSQLProtocol::Messaging::CopyOutData(std::move(result_buf))); // Then explicitly reinitialize if needed result_buf = {}; ``` These optimizations are particularly important in performance-critical paths where small improvements can have significant cumulative impact. --- ## Capture broad exceptions When using broad exception handlers like `except Exception:`, always capture and log the exception to avoid silent failures that are difficult to debug in production. Broad exception handling without proper error capture masks underlying issues and makes troubleshooting nearly impossible. **Bad:** ```python try: verify_github_signature(request.body.decode("utf-8"), kid, sig) except Exception: # Silent failure - no way to debug what went wrong break ``` **Good:** ```python try: verify_github_signature(request.body.decode("utf-8"), kid, sig) except Exception as e: capture_exception(e) # or logger.exception() break ``` This practice is especially critical in complex functions with nested operations where multiple failure points exist. Even when you intend to handle errors gracefully, capturing the exception provides valuable debugging information without changing the error handling behavior. The goal is to maintain your intended error recovery while ensuring production issues can be diagnosed and fixed. --- ## Configuration constants management Extract configuration values into well-named constants instead of using magic numbers or inline values. Use consistent naming patterns across environments and organize configuration values in a maintainable way. **Why this matters:** - Magic numbers scattered throughout code are hard to maintain and understand - Inconsistent naming across environments leads to confusion and errors - Centralized configuration makes it easier to modify behavior without hunting through code **How to apply:** 1. Replace magic numbers with descriptive constants 2. Use consistent naming patterns across all environments 3. Group related configuration values together 4. Make non-parameterized values into module-level constants **Example:** ```python # Bad - magic numbers and inconsistent naming def paginate_results(self): self._page_size = 50 max_pages = 6 timeout = 180 if storage_policy == "s3": policy = "s3_policy" # Different naming in other envs # Good - named constants with consistent patterns DEFAULT_PAGE_SIZE = 50 MAX_PAGES_LIMIT = 6 REQUEST_TIMEOUT_SECONDS = 180 S3_STORAGE_POLICY = "s3_backed" # Same across all environments BASE_ERROR_INSTRUCTIONS = "Tell the user that you encountered an issue..." def paginate_results(self): self._page_size = DEFAULT_PAGE_SIZE max_pages = MAX_PAGES_LIMIT timeout = REQUEST_TIMEOUT_SECONDS ``` This approach makes configuration changes safer, more discoverable, and reduces the risk of environment-specific bugs. --- ## Add explanatory tooltips When UI elements have unclear functionality or purpose, add tooltips to provide immediate context and explanation. This is especially important for icons, checkboxes, and interactive elements where the behavior or consequences aren't immediately obvious to users. Tooltips should: - Explain what the element does in clear, user-friendly language - Describe any non-obvious consequences or behaviors - Use terminology consistent with the rest of the UI - Link to documentation when appropriate (though some tooltip implementations only support strings) Example from the discussions: ```tsx { updateFilterOptional({ ...filter, optionalInFunnel: checked, index, }) }} label="Optional step" tooltip="When checked, this step won't cause users to drop out of the funnel if they skip it" /> ``` This practice improves user experience by providing inline documentation that helps users understand functionality without needing to consult external documentation or guess at behavior. --- ## Keep state in Kea React components should focus on presentation and user interaction, not state management logic. All state logic should be contained within Kea stores, with components only pulling in actions and values. This separation improves testability, reusability, and maintainability. Components should avoid: - useState and useEffect for business logic - Complex state calculations - Direct API calls or data fetching Instead, delegate these responsibilities to Kea logic files and consume the results: ```tsx // ❌ Avoid: State logic in component export function DataWarehouseScene(): JSX.Element { const [recentActivity, setRecentActivity] = useState([]) const [totalRowsProcessed, setTotalRowsProcessed] = useState(0) useEffect(() => { const loadData = async (): Promise => { const [activities, totalRows] = await Promise.all([ fetchRecentActivity(dataWarehouseSources?.results || [], materializedViews), fetchTotalRowsProcessed(dataWarehouseSources?.results || [], materializedViews), ]) setRecentActivity(activities) setTotalRowsProcessed(totalRows) } loadData() }, [dataWarehouseSources?.results, materializedViews]) } // ✅ Preferred: State logic in Kea, component consumes values export function DataWarehouseScene(): JSX.Element { const { recentActivity, totalRowsProcessed } = useValues(dataWarehouseLogic) const { loadDashboardData } = useActions(dataWarehouseLogic) } ``` This pattern ensures components remain focused on rendering and user interactions while keeping business logic centralized and testable. --- ## Local configuration exclusion Exclude personal and local configuration files from version control while ensuring they are properly handled during environment setup. Personal configurations should use patterns like `*.local.*` or be placed in designated directories that are gitignored to prevent accidental commits of developer-specific settings. When setting up new environments or worktrees, be explicit about which configuration files to copy. Use specific patterns rather than wildcards to avoid copying unintended files. Example: ```bash # In .gitignore bin/mprocs*.local.yaml .env.local playground/ # In setup scripts - be specific about what to copy if [[ -f "${main_repo}/.env" ]]; then cp "${main_repo}/.env" "${worktree_path}/" fi # Rather than copying all .env* files indiscriminately ``` This prevents personal development configurations from polluting the shared codebase while ensuring consistent environment setup across different development contexts. --- ## two-phase filtering algorithms When working with large datasets or complex matching operations, implement algorithms that use a two-phase approach: first filter candidates using efficient broad criteria, then verify matches with precise logic. This pattern significantly improves performance by reducing the number of expensive operations. The approach is particularly effective when: - Database-level filtering can eliminate most irrelevant records - Verification logic is computationally expensive - Memory usage needs to be minimized Example implementation: ```python def get_dependent_cohorts_reverse(cohort: Cohort) -> list[Cohort]: # Phase 1: Database-level filtering using broad criteria filter_conditions = Q() filter_conditions |= Q(filters__icontains=f'"value": {cohort.id}') filter_conditions |= Q(filters__icontains=f'"value": "{str(cohort.id)}"') candidate_cohorts = ( Cohort.objects.filter(filter_conditions, team=cohort.team, deleted=False) .exclude(id=cohort.id) ) dependent_cohorts = [] # Phase 2: Precise verification of filtered candidates for candidate_cohort in candidate_cohorts: for prop in candidate_cohort.properties.flat: if prop.type == "cohort" and not isinstance(prop.value, list): try: if int(prop.value) == cohort.id: dependent_cohorts.append(candidate_cohort) break except (ValueError, TypeError): continue return dependent_cohorts ``` This pattern avoids loading all records into memory and performing expensive operations on irrelevant data, instead using the database's indexing and filtering capabilities first. --- ## optimize ORM queries Optimize Django ORM queries to prevent performance issues and unnecessary database load. Avoid N+1 query problems by using appropriate prefetch_related() and select_related() calls. Remove unnecessary select_related() when you're only filtering by foreign key IDs and not accessing the related object. Choose the correct database routing (read vs write replicas) based on the operation. For large tables, prefer database-level filtering over Python-level processing to avoid scanning entire tables. Example of unnecessary select_related(): ```python # Bad - unnecessary select_related when only filtering by team_id feature_flag = FeatureFlag.objects.select_related("team").get(key=flag_key, team_id=self._team.id) # Good - remove select_related when not using the related object feature_flag = FeatureFlag.objects.get(key=flag_key, team_id=self._team.id) ``` Example of preventing N+1 queries: ```python # Bad - causes N+1 queries for obj in Group.objects.all(): relationship = obj.notebook_relationships.first() # Good - use prefetch_related to avoid N+1 groups = Group.objects.prefetch_related("notebook_relationships__notebook") for obj in groups: relationship = obj.notebook_relationships.first() ``` For large datasets, use database-level operations instead of scanning entire tables in application code, especially in production environments where tables can be very large. --- ## Avoid duplicate HTTP headers Ensure HTTP headers are sent only once per response to prevent network protocol violations and connection errors. Sending headers multiple times violates HTTP protocol standards and can cause "Broken pipe" errors and connection failures. When working with HTTP responses, be careful not to call methods that send headers (like `response.send()`) before the final response is written. The framework typically handles header transmission automatically during the final write operation. Example of problematic code: ```cpp response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_NO_CONTENT); response.setChunkedTransferEncoding(false); response.send(); // ❌ Sends headers prematurely // ... later in code ... // Headers sent again automatically - causes protocol violation ``` Example of correct code: ```cpp response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_NO_CONTENT); response.setChunkedTransferEncoding(false); // ✅ Let the framework send headers during final write operation ``` This applies to all HTTP server implementations and is critical for maintaining stable network connections, especially under high load or in distributed environments where connection reliability is essential. --- ## consistent formatting rules Maintain consistent formatting throughout the codebase to improve readability and maintainability. This includes standardizing function declaration spacing, comment styles, pointer/reference formatting, and type naming conventions. Key formatting rules to follow: 1. **Function declarations**: Use consistent spacing around braces and parameters ```cpp // Preferred - consistent spacing const NATSConnectionPtr & getConnection() { return connection; } const std::vector & getSubjects() const { return subjects; } // Avoid - inconsistent spacing const NATSConnectionPtr & getConnection(){return connection;} const String & getQueueName() const{return queue_name;} ``` 2. **Comments**: Use triple slashes (///) for single-line comments instead of double slashes (//) ```cpp /// Implements `instrumentation` system table, which allows you to get information about functions instrumented by XRay. // Avoid: // Get a vector of indices. ``` 3. **Pointer and reference formatting**: Add spaces around `*` and `&` operators ```cpp // Preferred std::vector getIndices(const GinFilter * filter, const PostingsCacheForStore * cache_store) const; // Avoid std::vector getIndices(const GinFilter *filter, const PostingsCacheForStore *cache_store) const; ``` 4. **Type naming**: Follow project conventions (e.g., `UInt32` instead of `uint32_t` in ClickHouse codebase) Consistent formatting reduces cognitive load during code reviews and makes the codebase more professional and maintainable. --- ## Consistent naming conventions Maintain consistent naming conventions within your codebase, even when external specifications or APIs use different naming patterns. Internal consistency should take precedence over matching external naming styles to ensure code readability and maintainability. When faced with a choice between following an external API's naming convention and maintaining internal consistency, choose internal consistency. For example, if your system uses snake_case for column names, convert external PascalCase property names to snake_case rather than mixing conventions. Example from system table columns: ```sql -- Inconsistent (mixing conventions) code_point, code_point_value, Alphabetic, ASCII_Hex_Digit -- Consistent (all snake_case) code_point, code_point_value, alphabetic, ascii_hex_digit ``` If external naming alignment is crucial for user experience, consider providing aliases that maintain the external naming while keeping the primary implementation consistent with your internal standards. However, in most cases, prioritize internal consistency as it benefits long-term code maintainability more than preserving external naming patterns. --- ## prefer simple optimizations When implementing performance optimizations, favor simple, straightforward approaches over complex solutions unless the complexity is clearly justified by significant performance gains. Complex optimizations often introduce maintenance burden, reduce code readability, and may not provide proportional benefits. Before implementing a complex optimization: 1. Try the simplest approach first and measure its impact 2. Only add complexity if simple solutions are insufficient 3. Document why the complexity is necessary Example of preferred simple approach: ```cpp // Simple prefetching - preferred if (prefetch_ahead > 0) { __builtin_prefetch(current + prefetch_ahead); } // Instead of complex ring buffer implementation ``` Example of keeping codec selection simple: ```cpp // Prefer portable, simple codec selection static constexpr auto CODEC_NAME = "simdfastpfor128"; static constexpr size_t THRESHOLD = 4; // Instead of complex compile-time CPU feature detection ``` This approach reduces the risk of introducing bugs in performance-critical code while maintaining the primary performance benefits. Complex optimizations should be reserved for cases where profiling clearly demonstrates that simple approaches are insufficient and the performance gain justifies the added complexity. --- ## Validate inputs recursively Always implement recursive validation and sanitization for user inputs, especially when dealing with encoded content or external data sources. Single-pass validation can be bypassed through multiple encoding layers or nested attacks. When validating URLs, decode recursively until no further changes occur to prevent encoding bypass attacks like `javascript%253Aalert(1)` which could decode through multiple layers to become `javascript:alert(1)`. Similarly, when integrating with external services, never trust their validation - always re-validate on your end. Example of secure URL validation: ```python def _is_safe_url(self, url: str) -> bool: """Validate URL with recursive decoding to prevent bypass attacks.""" # Recursively decode until no changes to prevent encoding bypasses decoded = url while True: new_decoded = unquote(decoded) if new_decoded == decoded: break decoded = new_decoded # Now validate the fully decoded URL parsed = urlparse(decoded.lower()) return parsed.scheme in self.ALLOWED_SCHEMES ``` This approach prevents attackers from using multiple encoding layers to bypass validation and ensures that external data sources are not blindly trusted for security-critical decisions like email verification status. --- ## Recognize nullable type context Before adding nullable casts or special null handling logic, verify whether the type or context already provides the necessary nullability semantics. This prevents redundant operations and ensures appropriate handling of different type categories. When working with generic code, check if types are already nullable to avoid unnecessary casts: ```cpp // Instead of redundant casting: SELECT _CAST(__table1.`v.String`, 'Nullable(String)') AS `variantElement(v, 'String')` // Recognize when the subcolumn is already Nullable(String): SELECT __table1.`v.String` AS `variantElement(v, 'String')` ``` For generic functions handling different types, consider special cases for pointer types which have inherent null semantics: ```cpp // Add conditional logic for pointer types: if constexpr (!std::is_pointer_v) LOG_INFO(getLogger("Jemalloc"), "Value for {} set to {} (from {})", name, value, old_value); // Special handling needed for const char* and other pointer types ``` This approach reduces code complexity, improves performance by eliminating unnecessary operations, and ensures type-appropriate null handling across different contexts. --- ## optimize algorithm selection Choose algorithms and data structures based on actual performance characteristics rather than defaulting to standard library implementations. Consider lighter-weight alternatives when standard library functions are over-engineered for your use case. Key optimization strategies: 1. **Hash function selection**: Prefer faster hash functions like `UInt128HashCRC32` over STL's heavier default hash implementations when cryptographic security isn't required. 2. **Search algorithm heuristics**: Use linear search for small datasets (typically ≤16 elements) before falling back to binary search, as the overhead of binary search can exceed its benefits for small collections. 3. **Specialized parsing**: Implement template-based parsing functions for specific bases (2, 8, 10, 16) instead of using generic standard library functions like `strtol()` or `sscanf()` when performance is critical. 4. **Selection algorithms**: Use `std::nth_element` for top-k selection problems instead of full sorting when you only need the k smallest/largest elements. Example of algorithm selection based on size: ```cpp inline bool containsInPartitionIdsOrEmpty(const PartitionIds & haystack, const String & needle) { if (haystack.empty()) return true; // Use linear search for small collections, binary search for larger ones if (haystack.size() <= 16) return std::find(haystack.begin(), haystack.end(), needle) != haystack.end(); else return std::binary_search(haystack.cbegin(), haystack.cend(), needle); } ``` Always profile and measure the actual performance impact of algorithm choices in your specific use case, as theoretical complexity doesn't always translate to real-world performance gains. --- ## AI context efficiency When providing context to LLMs, choose the most efficient method based on the nature and size of the context data. For bounded, static context (like feature flag mappings or configuration options), inject the information directly into the system prompt rather than using tool calls. This approach is faster, more cost-effective, and reduces complexity. **Prefer prompt injection when:** - Context data is relatively small and bounded - Data doesn't change frequently during the conversation - You want to minimize API calls and latency **Use tool calls when:** - Context data is large or unbounded - Data needs to be fetched dynamically based on user input - You need the LLM to make decisions about what context to retrieve **Example:** ```python # Good: Inject bounded feature flag context into prompt enhanced_system_prompt = SURVEY_CREATION_SYSTEM_PROMPT if feature_flag_context: enhanced_system_prompt += f"\n\n## Available Feature Flags\n{feature_flag_context}" # Avoid: Using tool calls for static, bounded context # This adds unnecessary complexity and cost def retrieve_flag_id(feature_key): # Tool call - overkill for small static data return api_call_to_get_flag_id(feature_key) ``` Also ensure your prompts only reference capabilities the LLM actually has - avoid instructing LLMs to manipulate internal state variables they cannot access. --- ## consistent mutex protection Ensure consistent mutex usage across all access points to shared data structures and clearly document what each mutex protects. Inconsistent locking can lead to race conditions, data corruption, and crashes. Key principles: 1. **Use the same mutex** for all operations on a shared data structure - don't mix different mutexes for the same resource 2. **Document protection boundaries** clearly, ideally with Thread Safety Analysis (TSA) annotations 3. **Prefer `std::lock_guard`** over `std::scoped_lock` for single mutex scenarios 4. **Structure code to minimize lock scope** and avoid accessing potentially unsafe pointers outside critical sections Example of problematic inconsistent locking: ```cpp // BAD: Different mutexes protecting the same data std::mutex mutex; std::shared_mutex shared_mutex; std::unordered_map functionIdToHandlers; void setHandler() { std::lock_guard lock(mutex); // Uses mutex functionIdToHandlers[id] = handler; } void dispatchHandler() { std::shared_lock lock(shared_mutex); // Uses shared_mutex! auto it = functionIdToHandlers.find(id); // Race condition! } void processHandler() { // No lock at all - definitely unsafe! functionIdToHandlers[id].process(); } ``` Better approach with consistent protection: ```cpp // GOOD: Single mutex with clear scope std::shared_mutex handlers_mutex; // Clearly named std::unordered_map functionIdToHandlers GUARDED_BY(handlers_mutex); void setHandler() { std::lock_guard lock(handlers_mutex); functionIdToHandlers[id] = handler; } void dispatchHandler() { HandlerInfo handler_copy; { std::shared_lock lock(handlers_mutex); auto it = functionIdToHandlers.find(id); if (it != functionIdToHandlers.end()) { handler_copy = it->second; // Copy under lock } } // Use handler_copy safely outside lock if (handler_copy.isValid()) { handler_copy.process(); } } ``` This prevents race conditions where concurrent modifications can invalidate iterators or cause undefined behavior during map resizing. --- ## Use error chain iterators When traversing error chains in Rust, prefer using the `chain()` iterator method over manual source traversal with while loops. The `chain()` method provides a more idiomatic and readable approach to walking through error chains, eliminating the need for manual loop management and making the code more concise. Instead of manually iterating through error sources: ```rust let mut source = error.source(); while let Some(err) = source { if let Some(rl) = err.downcast_ref::() { return rl.retry_after; } source = err.source(); } ``` Use the iterator-based approach: ```rust error .chain() .find_map(|e| e.downcast_ref::()) .and_then(|rl| rl.retry_after) ``` This pattern works well with other iterator methods like `any()`, `find()`, and `filter_map()` to create more expressive and maintainable error handling code. --- ## avoid repeated expensive operations Identify and eliminate repeated expensive computations, especially in loops and frequently executed code paths. This optimization principle focuses on moving invariant calculations outside loops, caching results of expensive operations, and avoiding unnecessary data conversions or memory allocations. Key strategies include: 1. **Move loop-invariant operations outside loops**: Instead of performing the same expensive computation in every iteration, calculate it once before the loop starts. ```cpp // Instead of this: for (idx_t child_idx = 0; child_idx < entry.length; child_idx++) { if (actual_value == ((T *)value_data.data)[0]) { // Repeated casting // ... } } // Do this: auto values = (T *)value_data.data; // Cast once outside loop for (idx_t child_idx = 0; child_idx < entry.length; child_idx++) { if (actual_value == values[0]) { // ... } } ``` 2. **Cache expensive expression evaluations**: If an expression can be folded or has `IsFoldable()`, cache the result to avoid repeated evaluation in performance-critical paths. 3. **Avoid unnecessary string conversions and copies**: Use direct data access methods like `StringUtil::CIEquals` on `string_t` data instead of creating temporary string copies for comparisons. 4. **Optimize early exit conditions**: Move null checks and other early termination conditions to the beginning of loops to avoid wasting computation on data that will be discarded. This principle is particularly important in database query execution, aggregation functions, and data processing pipelines where the same operations may be performed millions of times. --- ## Setting declaration practices Always declare settings before using them and optimize access patterns for performance. Settings must be properly declared in the appropriate scope before being referenced in code. When accessing settings multiple times, cache the value in a local variable to avoid repeated lookups. For configuration parsing, avoid modifying AST nodes directly and use local variables instead. Example of proper setting declaration and usage: ```cpp // Declare setting before use DECLARE(UInt64, database_replicated_logs_to_keep, 1000, "Description", 0) // Cache setting value when used multiple times bool enable_optimization = planner_context->getQueryContext()->getSettingsRef()[Setting::enable_add_distinct_to_in_subqueries]; for (const auto & item : items) { if (enable_optimization) { // Use cached value instead of repeated setting access } } // Avoid modifying AST, use local variables auto key_value = evaluateConstantExpressionOrIdentifierAsLiteral(children[0], context); // Store in local variable instead of updating children[0] directly ``` This practice prevents runtime errors from undeclared settings, improves performance by reducing repeated setting lookups, and maintains cleaner code by avoiding direct AST modifications during configuration parsing. --- ## API parameter semantics Ensure API parameters have clear semantic meaning and avoid sending null values for optional fields. When designing API endpoints, use parameter names that accurately reflect their purpose and data type. For optional fields, omit them entirely from the payload rather than setting them to null, as this creates cleaner API contracts and prevents confusion about field requirements. Example of what to avoid: ```javascript // Bad: sending null values for optional fields setSurveyValue('conditions', { ...survey.conditions, linkedFlagVariant: null, // Don't send null for optional fields }); ``` Example of proper approach: ```javascript // Good: omit optional fields entirely const { linkedFlagVariant, ...conditions } = survey.conditions; setSurveyValue('conditions', { ...conditions, // Only include non-null values }); ``` Additionally, ensure parameter names match their expected data types. For example, use `email` parameter when expecting email addresses rather than overloading `distinct_id` with email values, as this maintains semantic clarity and prevents data attribution issues. --- ## minimize expensive operations Avoid triggering expensive operations (queries, API calls, computations) on every user input or state change. Instead, use appropriate triggers that balance responsiveness with performance. Key strategies: - Use `onBlur` instead of `onChange` for expensive operations that don't need immediate feedback - Implement save/submit buttons for complex forms to batch expensive operations - Move data fetching from `useEffect` to event handlers when the data is only needed in response to specific user actions - Consider the cost-benefit ratio: "breakdown queries are expensive as is" - defer expensive operations until truly necessary Example from the discussions: ```tsx // Instead of triggering expensive queries on every change: expensiveQuery(value)} // Causes UI jumpiness /> // Use onBlur or save buttons: expensiveQuery(value)} // Better performance /> // Or move from useEffect to event handlers: // Instead of: useEffect(() => { if (needsData) { fetchExpensiveData() } }, [dependency]) // Use: const handleUserAction = () => { if (needsData) { fetchExpensiveData() // Only when actually needed } } ``` This approach reduces unnecessary resource utilization and prevents performance bottlenecks that degrade user experience. --- ## Verify HTML escaping {% raw %} Always verify that user-controlled content in templates is properly HTML-escaped to prevent XSS attacks. Don't just assume framework defaults are working - actively test with potentially malicious input to confirm that HTML tags are rendered as text rather than executed. When displaying dynamic content in templates, test with HTML payloads like `` or `` to ensure they appear as literal text. For Django templates, confirm that the standard `{{ variable }}` syntax properly escapes HTML characters, converting `<` to `<`, `>` to `>`, etc. Example verification: ```html

API Key: {{ more_info }}

``` This practice helps catch cases where unsafe rendering methods might be accidentally used or where framework protections might not apply. {% endraw %} --- ## Explicit error handling Always handle error conditions explicitly rather than silently ignoring them or using generic error responses. Use specific error codes that reflect business logic rather than low-level system errors, and throw exceptions for unexpected states instead of using assertions or returning generic failure indicators. Key principles: 1. **Throw specific exceptions** for unexpected conditions rather than using assertions that can cause segfaults 2. **Use business-logic error codes** instead of generic system-level codes to provide better context 3. **Explicitly check expected error conditions** and throw exceptions for unexpected ones 4. **Avoid silent failures** - parse and validate inputs completely, throwing unsupported exceptions for unhandled cases Example of good explicit error handling: ```cpp // Instead of silent failure or generic error if (code != Coordination::Error::ZOK) return false; // BAD: silently ignores all error types // Be explicit about expected vs unexpected errors if (code == Coordination::Error::ZNONODE || code == Coordination::Error::ZNOAUTH) return false; // Expected errors else if (Coordination::isHardwareError(code)) throw Exception(ErrorCodes::ZOOKEEPER_EXCEPTION, "Hardware error: {}", code); // Unexpected errors // Use specific error codes that reflect business context throw Exception(ErrorCodes::UDF_EXECUTION_FAILED, "UDF execution failed: {}", details); // Instead of generic: ErrorCodes::CANNOT_WRITE_TO_FILE_DESCRIPTOR // Replace assertions with proper exceptions chassert(tuple_function->arguments->children.size() == partition_columns.size()); // BAD if (tuple_function->arguments->children.size() != partition_columns.size()) throw Exception(ErrorCodes::LOGICAL_ERROR, "Partition argument count mismatch"); // GOOD ``` This approach improves debugging, provides better user experience, and prevents silent corruption of program state. --- ## remove unnecessary code Eliminate unnecessary code elements that add complexity without providing value. This includes removing redundant function wrappers, unnecessary type annotations, singleton classes that could be simple objects, explicit type conversions where implicit ones suffice, and methods that duplicate existing functionality. Examples of unnecessary code to remove: - Wrapper functions that don't add logic: `const lock = async () => { ... }; const release = await lock();` → inline the logic directly - Redundant type annotations: `const BASE_TRANSFORM: any = {...}` → `const BASE_TRANSFORM = {...}` - Unnecessary type assertions: `satisfies CustomMutatorDefs` when type inference works - Explicit conversions: `String(shardNum)` → `shardNum` when concatenating with strings - Singleton classes: `class ContextManager` with single instance → plain object with functions Before adding new abstractions, consider if existing functionality can be reused. Before adding type annotations, verify they provide meaningful type safety beyond what TypeScript can infer. This reduces cognitive overhead and makes code more maintainable by focusing on essential logic rather than ceremonial code. --- ## Optimize algorithm complexity Choose efficient algorithms and data structures to avoid unnecessary computational complexity. Replace manual loops with optimized library functions, select appropriate containers based on access patterns, and use direct conversion paths when possible. Key optimizations to consider: - Use `memcmp()` instead of character-by-character loops for string comparisons to avoid quadratic complexity - Add pointer equality short-circuits before expensive comparisons: `if (this_data == other_data) return true;` - Replace `std::find_first_of()` with simple character checks when scanning for specific characters: `result_offset += c == '\\' || c == '\'';` - Choose `unordered_map` over `map` when ordering is not required for better lookup performance - Use direct conversion paths like `std::chrono::duration_cast(time_point.time_since_epoch()).count()` instead of intermediate conversions through `time_t` Example of string comparison optimization: ```cpp bool operator==(const String &other) const { if (this_size != other_size) return false; const char *this_data = GetData(); const char *other_data = other.GetData(); // Short-circuit if same pointer if (this_data == other_data) return true; // Use optimized library function return memcmp(this_data, other_data, this_size) == 0; } ``` Always profile performance-critical code paths and prefer established algorithms over custom implementations when equivalent functionality exists. --- ## validate before use Always validate that values are truthy or defined before using them, even when they are expected to exist. This prevents runtime errors and unexpected behavior when assumptions about data presence are violated. The pattern helps catch edge cases where expected values might be missing, null, or undefined, avoiding silent failures or unexpected default behaviors. Example: ```typescript // Instead of assuming properties exists let properties = data.event.properties if ('exception_props' in properties) { // use properties } // Validate it's truthy first let properties = data.event.properties if (properties && 'exception_props' in properties) { // use properties } // For expected values that might be missing if (event.now) { const capturedAtDateTime = DateTime.fromISO(event.now).toUTC() preIngestionEvent.capturedAt = capturedAtDateTime.isValid ? capturedAtDateTime : undefined } ``` This approach prevents scenarios like getting unexpected fallback values (e.g., "1970-01-01 00:00:00.000000" in databases) when validation fails silently. --- ## Guard expensive logging operations {% raw %} Avoid executing expensive operations in logging code paths when logging is disabled or not needed. Always check if logging should occur before performing costly operations like string construction, formatting, or data processing for log messages. The key principle is to separate the lightweight "should log" check from expensive log message preparation. This prevents performance overhead when logging is disabled and avoids unnecessary work like duplicate string construction or complex data processing. Example of the problem: ```cpp // BAD: Always executes expensive operation const auto event = state.offset_in_group == (idx_t)group.num_rows ? "SkipRowGroup" : "ReadRowGroup"; DUCKDB_LOG(context, PhysicalOperatorLogType, *state.op, "ParquetReader", event, {{"file", file.path}, {"row_group_id", to_string(state.group_idx_list[state.current_group])}}); ``` Better approach: ```cpp // GOOD: Guard expensive operations behind logging check if (DUCKDB_SHOULD_LOG(context, PhysicalOperatorLogType)) { const auto event = state.offset_in_group == (idx_t)group.num_rows ? "SkipRowGroup" : "ReadRowGroup"; vector> log_payload = { {"file", file.path}, {"row_group_id", to_string(state.group_idx_list[state.current_group])} }; DUCKDB_WRITE_LOG(context, PhysicalOperatorLogType, *state.op, "ParquetReader", event, log_payload); } ``` This pattern is especially important for high-frequency logging in performance-critical code paths, and helps avoid creating log messages multiple times for different output streams. {% endraw %} --- ## Environment-based configuration management Prefer environment variables over file mounting for configuration values, and avoid hardcoded environment-specific conditionals like `isCloud()`. Instead, rely purely on configuration environment variables that can be set differently across environments. When setting default configuration values, consider the constraints and limits of downstream systems to prevent runtime failures. For example, instead of: ```typescript sslOptions: isCloud() ? { ca: fs.readFileSync(join(__dirname, '../cassandra/ca.crt')), } : undefined ``` Use environment variables: ```typescript sslOptions: process.env.CASSANDRA_SSL_CA ? { ca: Buffer.from(process.env.CASSANDRA_SSL_CA, 'base64'), } : undefined ``` And when setting defaults, consider system limits: ```typescript // Consider Kafka's 1MB limit when setting defaults PERSON_PROPERTIES_SIZE_LIMIT: 512 * 1024, // 512KB default (safe margin under 1MB Kafka limit) ``` This approach makes configuration more flexible, testable, and avoids deployment complexities like file mounting while ensuring defaults don't cause system failures. --- ## maintain codebase consistency Ensure new code follows established patterns, conventions, and standards already present in the codebase. This includes adhering to parameter ordering conventions, maintaining existing code patterns, using consistent header formatting, and reusing existing utilities rather than creating duplicates. Key areas to check: - **Parameter ordering**: Follow established conventions for parameter order (e.g., allocators as first parameters) - **Code patterns**: Maintain existing patterns rather than introducing new approaches without justification - **Header formatting**: Use consistent header comment blocks across all files - **Constructor design**: Avoid unnecessary constructors when implicit conversions or default values suffice - **Code reuse**: Leverage existing utility functions instead of reimplementing similar functionality Example of good consistency: ```cpp // Follow established parameter ordering string_t(ArenaAllocator &arena, const char *data, const uint32_t len) // Good string_t(const char *data, const uint32_t len, ArenaAllocator &arena) // Inconsistent // Use standard header format //===----------------------------------------------------------------------===// // DuckDB // // duckdb/common/types/string.hpp // //===----------------------------------------------------------------------===// ``` This approach reduces cognitive load for developers, makes the codebase more predictable, and prevents the accumulation of inconsistent patterns over time. --- ## Environment variable handling When working with environment variables, follow consistent naming conventions, properly check for their presence, and preserve the environment when spawning subprocesses. Use standard naming patterns (e.g., `OPENSSL_VERSION_OVERRIDE` instead of `OPEN_SSL_VERSION_OVERRIDE`), check both command-line arguments and environment variables for configuration options, and always copy the current environment when passing custom environments to subprocess calls. Example of proper environment handling: ```python # Good: Standard naming convention openssl_version = os.getenv("OPENSSL_VERSION_OVERRIDE", "3.0.8") # Good: Check both argument and environment variable summarize_failures = args.summarize_failures or os.getenv("SUMMARIZE_FAILURES") == "1" # Good: Preserve current environment when spawning subprocess env = os.environ.copy() if list_of_tests or no_exit or tests_per_invocation: env['SUMMARIZE_FAILURES'] = '0' env['NO_DUPLICATING_HEADERS'] = '1' res = subprocess.run(test_cmd, stdout=unittest_stdout, stderr=unittest_stderr, timeout=timeout, env=env) ``` This ensures configuration consistency, prevents loss of important environment settings, and maintains predictable behavior across different execution contexts. --- ## Follow CSS naming patterns Maintain consistency with established CSS naming conventions already used in the codebase. For CSS classes, follow BEM methodology when that's the existing pattern. For custom properties, use semantic names that clearly indicate their purpose and work across themes. Examples: - Use BEM-style class names: `&.FunnelBarHorizontal--has-optional-steps` instead of `&.has-optional-steps` - Use semantic CSS custom properties: `--gray-1`, `--gray-2` for theme-agnostic color variables that automatically adapt to light/dark themes This ensures code maintainability and helps other developers quickly understand the naming system in use. --- ## validate schema decisions When reviewing database schema changes or data structure modifications, ensure that field inclusion/exclusion decisions are explicitly justified and documented. Question ambiguous schema choices and require clear rationale for data organization patterns. Schema decisions should address: - Why specific fields are included or omitted (e.g., "Do we need team_id in addition to person_id and event_name?") - How the chosen structure supports expected query patterns - Whether data access methods are intuitive for developers Example from schema design: ```sql -- Migration: Create person event occurrences table -- Question: Is team_id necessary alongside person_id and event_name? CREATE TABLE IF NOT EXISTS person_event_occurrences ( team_id INT, -- Rationale needed: Does this enable required queries? person_id TEXT, event_name TEXT ); ``` When data property references seem unclear or confusing (like choosing between `$groups: { key: value }` vs `$group_0`), document the reasoning behind the chosen approach and consider developer experience in accessing the data. --- ## avoid unnecessary computations Optimize algorithms by eliminating redundant work and intermediate data structures. Look for opportunities to use lazy evaluation, conditional execution, and more efficient operations. Key strategies: - Use lazy evaluation to defer expensive operations until actually needed - Avoid creating intermediate arrays or objects when direct operations are possible - Apply conditional logic to skip unnecessary function calls or computations - Choose more efficient built-in operations when available Examples: ```typescript // Instead of always creating intermediate arrays: const materialized = [...stream()].map(materializeNodeRelationships); // Use imperative style to avoid extra allocations: const materialized: Node[] = []; for (const n of stream()) { materialized.push(materializeNodeRelationships(n)); } // Conditional predicate application: const matchesConstraintAndFilters = predicate ? (row: Row) => matchesConstraint(row) && predicate(row) : matchesConstraint; // Avoid wrapping when unnecessary // Use efficient built-ins: yield* sorted.slice(0, k); // Instead of manual loop ``` This approach improves performance by reducing computational overhead, memory allocations, and unnecessary operations while maintaining code correctness. --- ## avoid global state Avoid global state in concurrent environments where multiple instances, workers, or clients may share the same JavaScript context. Global variables and singletons can lead to unpredictable behavior when accessed concurrently. Instead, ensure proper ownership by making state local to specific classes or function scopes. Each instance should own its resources rather than sharing global state. **Problematic pattern:** ```ts // Global state shared across instances const rwLocks = new Map(); export const ClientMetrics = { timeToConnect: new Gauge(), // ... other metrics }; ``` **Better approach:** ```ts // Each instance owns its state class SQLiteStore { private rwLock = new RWLock(); // ... } class ReflectClient { private metrics = { timeToConnect: new Gauge(), // ... other metrics }; // ... } ``` This prevents issues where "multiple Reflect clients running in a page at the same time... will share these constants leading to wackiness" and ensures that "everything has to be properly owned by one of the durable object classes, or local state owned by the worker fetch function." --- ## Use allowlists over blocklists When filtering data for security purposes, prefer allowlists (explicitly defining what is permitted) over blocklists (explicitly defining what is forbidden). Blocklists are inherently less secure because they can miss new threats, require constant maintenance as infrastructure changes, and operate on the assumption that anything not explicitly blocked is safe. Allowlists are more secure because they operate on the principle of least privilege - only explicitly permitted items are allowed, and everything else is automatically rejected. This approach is particularly important when handling user input, HTTP headers, API parameters, or any external data. Example of converting from blocklist to allowlist for HTTP headers: ```ts // Insecure: blocklist approach const DISALLOWED_HEADERS = ['x-forwarded-for', 'x-forwarded-host', 'x-forwarded-proto', 'cookie'] // Secure: allowlist approach const ALLOWED_HEADERS = ['Accept', 'Accept-Encoding', 'Accept-Language', 'Cache-Control', 'Pragma', 'Content-Type', 'Content-Length', 'Content-Encoding', 'Content-Language', 'User-Agent', 'Host', 'Date'] ``` Apply this principle when filtering file extensions, API endpoints, database fields, configuration options, or any scenario where you need to control what data is processed or passed through your system. --- ## maintain API backward compatibility When evolving APIs, prioritize backward compatibility by creating new methods or overloads rather than modifying existing function signatures. This prevents breaking changes for existing users and extensions. Key strategies include: - Add new methods with descriptive names instead of changing existing ones (e.g., `duckdb_vector_slice_dictionary` instead of modifying `duckdb_slice_vector`) - Provide overloaded versions when adding parameters (e.g., separate `FileHandle::Read` methods with and without `QueryContext`) - Use proper memory management patterns in C APIs - allocate with `duckdb_malloc` so users can free with `duckdb_free`, or encapsulate complex data in opaque objects with accessor methods - Avoid overly specialized helper functions that may become maintenance burdens; instead expose existing lower-level APIs that provide the same functionality Example of good practice: ```cpp // Instead of modifying existing signature: // DUCKDB_C_API void duckdb_slice_vector(duckdb_vector vector, duckdb_selection_vector selection, idx_t len); // Add a new method: DUCKDB_C_API void duckdb_vector_slice_dictionary(duckdb_vector vector, idx_t dict_size, duckdb_selection_vector selection, idx_t len); ``` This approach ensures existing code continues to work while providing new functionality through clearly named, purpose-built interfaces. --- ## Document API parameters inline When designing APIs, prioritize clarity and self-documentation by adding inline comments for non-obvious parameters and choosing descriptive method names. This reduces the need for external documentation and makes the code more maintainable. For boolean parameters or complex arguments, include inline comments that explain their purpose: ```cpp // Good: Clear parameter documentation void setRawPath(const Path & path_) override { path = {path_.path, /* allow_partial_prefix */false}; } // Good: Descriptive method names with explanatory comments /// Unlike s3 and azure, which are object storages, /// hdfs is a filesystem, so it cannot list files by partial prefix, /// only by directory. Path getRawPath() const override { return path; } ``` Additionally, consider feature detection over versioning when possible to maintain backward compatibility and reduce API complexity. Well-documented APIs reduce the cognitive load on developers and prevent misuse of parameters or methods. --- ## Use proper authorization attributes Avoid using Django framework attributes like `is_staff` and `is_impersonated` for application role checking, as these serve different purposes (admin panel access and user impersonation respectively). Instead, use application-specific role validation methods to ensure proper authorization. For role-based access control, import and use the appropriate organization logic: ```typescript const { isAdminOrOwner } = useValues(organizationLogic) // Use this instead of user?.is_staff if (isAdminOrOwner) { // Admin/owner specific logic } ``` This prevents potential authorization bypass vulnerabilities that could occur when framework attributes are misused for application-level access control decisions. --- ## Use descriptive naming Choose names that clearly communicate intent and purpose rather than being generic, abbreviated, or potentially misleading. Names should accurately reflect what the code does, not how it does it. Key principles: - Use domain-specific terminology (`column_idx` instead of `field_idx` for database operations) - Choose method names that describe the action (`SerializeToDisk` instead of `GetStorageInfo`, `LogFailure` instead of `LogBoth`) - Make boolean flags readable (`supports_ordinality` instead of `ordinality_implemented`) - Spell out full names instead of abbreviations (`"replace"` instead of `"r"` for error handling modes) - Use named constants instead of magic numbers (`match_actions.size()` instead of hardcoded `3`) - Avoid names that contradict behavior (don't name something `extension_loading = none` when it actually loads extensions) Example: ```cpp // Avoid generic or misleading names void GetStorageInfo(optional_ptr context, bool all_expr_inputs_valid); for (idx_t match_idx = 0; match_idx < 3; match_idx++) { // magic number // Use descriptive, intent-revealing names void SerializeToDisk(optional_ptr context, bool all_inputs_valid); for (idx_t match_idx = 0; match_idx < match_actions.size(); match_idx++) { // named constant ``` This approach makes code self-documenting and reduces the cognitive load for developers reading and maintaining the codebase. --- ## consistent null validation Ensure null checks and input validation are applied consistently across similar functions and code paths. When one function in an API performs null pointer checks or input validation, all similar functions should follow the same pattern to maintain predictable behavior and prevent runtime errors. Key practices: - Add null pointer checks consistently across similar API functions - Include input validation assertions (e.g., `D_ASSERT(!name.empty())`) for required parameters - Validate data integrity (e.g., UTF8 validation for string types) - Use comprehensive validation functions when available (e.g., `Value::IsFinite` instead of separate NaN and infinity checks) Example from the codebase: ```cpp // Before: Inconsistent null checking char *duckdb_to_sql_string(duckdb_value val) { auto v = UnwrapValue(val); // Missing null check // ... } // After: Consistent with other API functions char *duckdb_to_sql_string(duckdb_value val) { if (!val) { return nullptr; // Consistent null check like other functions } auto v = UnwrapValue(val); // ... } ``` This pattern prevents inconsistent behavior where some functions handle null inputs gracefully while others crash, and ensures that validation logic is applied uniformly across the codebase. --- ## Use semantically accurate names Choose names that accurately reflect the purpose, scope, and semantics of variables, methods, and classes. Names should be self-documenting and eliminate ambiguity about what the code element represents or does. Key principles: - **Match scope to name**: If functionality expands beyond the original name, update it accordingly (e.g., `updateVoterPeriodTimer` → `updateVoterSetPeriodTimer` when it handles add, remove, and update operations) - **Use precise terminology**: Avoid vague or ambiguous terms (e.g., `isMarkedArchived` → `isTerminalState` for a boolean indicating no further state transitions) - **Make purpose explicit**: Method names should clearly indicate their function (e.g., `getInternalTopics` → `getInternalTopicsToBeDeleted` when the intent is deletion) - **Eliminate confusion**: When similar concepts exist, use distinguishing names (e.g., `boolean isReconfigSupported` instead of generic parameter names) Example improvements: ```java // Before: Ambiguous about what "messages" vs "records" means int maxInFlightMessages; // After: Consistent with codebase terminology int maxInFlightRecords; // Before: Unclear what constitutes "marked archived" private boolean isMarkedArchived = false; // After: Clear semantic meaning - no further transitions allowed private boolean isTerminalState = false; // Before: Generic parameter name requiring context to understand public void testMethod(boolean withKip853Rpc) // After: Self-documenting parameter name public void testMethod(boolean isReconfigSupported) ``` Well-chosen names reduce the need for comments and make code intentions immediately clear to readers. --- ## Session-specific configuration access Always access configuration through the appropriate session context rather than using global configuration access. This ensures that session-specific settings are respected and maintains consistency across different execution contexts. **Why this matters:** - Different Spark sessions may have different configuration values - Global configuration access can ignore session-specific overrides - Proper session context ensures configuration consistency **Preferred patterns:** 1. **Pass SQLConf instances explicitly:** ```scala // Instead of accessing global config val maxStringLen = SQLConf.get.getConf(SQLConf.JSON_MAX_STRING_LENGTH) // Pass SQLConf instance to constructors class JSONOptions(parameters: Map[String, String], sqlConf: SQLConf) { private val maxStringLen = sqlConf.getConf(SQLConf.JSON_MAX_STRING_LENGTH) } ``` 2. **Access config through SparkSession:** ```scala // Instead of using global SQLConf.get val shuffleCleanupMode = determineShuffleCleanupMode(SQLConf.get) // Use session-specific configuration val shuffleCleanupMode = determineShuffleCleanupMode(sparkSession.sessionState.conf) ``` 3. **Use session timezone settings:** ```scala // Instead of hardcoded UTC or assumptions val zoneId = UTC // Use session-specific timezone val zoneId = DateTimeUtils.getZoneId(SQLConf.get.sessionLocalTimeZone) ``` This pattern follows established practices in components like `ParquetOptions` and ensures that configuration behavior is predictable and respects user-specified session settings. --- ## Validate configurations early Perform comprehensive configuration validation as early as possible in the execution flow, before any state modifications occur. This prevents invalid configurations from causing partial state changes and ensures cleaner error handling. Key principles: 1. **Validate before state changes**: Check all configuration constraints before modifying any formatter state, creating resources, or initializing components 2. **Account for all relevant flags**: When validating configurations, consider all related configuration flags that might affect validity (e.g., `unstable.feature.versions.enable` affecting which metadata versions are allowed) 3. **Provide context-aware error messages**: Include information about which configurations are actually supported based on current settings Example from StorageTool validation: ```scala // Validate configuration conflicts early, before formatter modifications if (!config.quorumConfig.voters().isEmpty && (namespace.getString("initial_controllers") != null || namespace.getBoolean("standalone"))) { // Fail fast with clear error message throw new InvalidConfigurationException("Cannot use --initial-controllers or --standalone with static quorum") } // Later, when handling version parsing, account for configuration flags try { formatter.setReleaseVersion(MetadataVersion.fromVersionString(releaseVersion)) } catch { case _: Throwable => val supportedVersions = if (config.unstableFeatureVersionsEnabled) { // Include unstable versions in error message metadataVersionsToString(MetadataVersion.MINIMUM_VERSION, MetadataVersion.latest()) } else { metadataVersionsToString(MetadataVersion.MINIMUM_VERSION, MetadataVersion.latestProduction()) } throw new TerseFailure(s"Unknown metadata.version $releaseVersion. Supported versions: $supportedVersions") } ``` This approach prevents partial failures, reduces debugging complexity, and provides users with actionable error messages that reflect their actual configuration context. --- ## context-independent schema design Database schema elements (views, constraints, tables) should be designed to be context-independent and self-contained. This means all identifiers should be fully qualified, and only information supported by the SQL parser should be included in DDL representations. Key principles: 1. **Fully qualify identifiers**: View text should contain fully qualified identifiers (catalog.schema.table) to avoid dependency on current context 2. **Parser-compatible DDL**: Only include information in DDL output that is supported by the SQL parser - remove unsupported elements like validation status 3. **Self-contained definitions**: Schema elements should not rely on external context like current catalog or namespace Example of context-independent view text: ```java // Bad - context-dependent "SELECT * FROM my_table" // Good - context-independent "SELECT * FROM my_catalog.my_schema.my_table" ``` Example of parser-compatible constraint DDL: ```java // Remove unsupported validation status from DDL return String.format( "CONSTRAINT %s %s %s %s", name, definition(), enforced ? "ENFORCED" : "NOT ENFORCED" // validationStatus removed - not supported by parser ); ``` This approach ensures schema portability, reduces ambiguity, and maintains compatibility with SQL standards. --- ## avoid unnecessary computations Prevent performance degradation by avoiding unnecessary expensive operations such as premature execution, redundant iterations, and costly data conversions. Use lazy evaluation, short-circuiting, and efficient data handling patterns to optimize execution paths. Key strategies: - Use `lazy val` for expensive computations that may be accessed multiple times during plan transformations to avoid repeated evaluation - Implement short-circuiting in conditional logic to skip expensive calculations when conditions aren't met - Avoid converting between data types unnecessarily, especially expensive operations like UTF8String ↔ String conversions - Prevent premature execution of child operations during plan analysis phases - Use single-pass iterations instead of multiple passes over collections Example of lazy evaluation: ```scala // Instead of: val orderExpressions: Seq[SortOrder] = orderChildExpressions.zipWithIndex.map { ... } // Use: private lazy val orderExpressions: Seq[SortOrder] = orderChildExpressions.zipWithIndex.map { ... } ``` Example of short-circuiting: ```scala // Instead of always computing both: if (left.maxRows.isDefined && right.maxRows.isDefined) { val leftMaxRows = left.maxRows.get val rightMaxRows = right.maxRows.get // ... } // Use short-circuiting: val leftMaxRowsOption = left.maxRows val rightMaxRowsOption = if (leftMaxRowsOption.isDefined) right.maxRows else None ``` This approach reduces computational overhead, improves response times, and prevents unnecessary resource consumption during query planning and execution. --- ## optimize data structures Choose appropriate data structures and algorithms to optimize computational complexity and performance. Consider the specific use case and access patterns when selecting between different approaches. Key optimization opportunities: 1. **Use efficient data structures for lookups**: Replace linear searches with constant-time operations where possible. For example, convert `ArrayContains` operations on literal arrays to `InSet` operations for O(1) lookup instead of O(n). 2. **Choose appropriate collection types**: Use `Set[String]` with `contains()` for simple string matching instead of `Set[Regex]` when regex functionality isn't needed. This avoids unnecessary regex compilation overhead. 3. **Leverage built-in collection methods**: Use `Array.tabulate(size)(constructor)` instead of manual loops for array initialization, and `collection.zip(other).toMap` for efficient map construction. 4. **Consider algorithmic properties**: Be aware of algorithm limitations like XOR checksums having issues with duplicate values. Consider alternatives like combining sum + XOR or using hash functions like Fowler–Noll–Vo when order-independence and duplicate-handling are required. 5. **Select semantic-appropriate data structures**: Use `ExpressionSet` for deduplicating expressions by semantics rather than object equality when the logical meaning matters more than object identity. Example transformation: ```scala // Instead of O(n) array search: case ArrayContains(arrayParam: Literal, col) => // Linear search through array elements // Use O(1) set lookup: case ArrayContains(arrayParam: Literal, col) if arrayParam.value != null => InSet(col, arrayParam.value.asInstanceOf[GenericArrayData].array.toSet) ``` This approach reduces computational complexity and improves performance, especially for frequently executed operations or large datasets. --- ## prefer simple APIs Design APIs with simplicity in mind by avoiding unnecessary method overloads, reducing configuration options, and preferring single well-designed methods that handle all cases rather than multiple variants. When faced with API design choices, favor approaches that make the developer's life easier by providing a single, clear way to accomplish a task. Instead of creating multiple overloads or adding numerous configuration knobs, design one method that can handle all scenarios elegantly. For example, instead of having two separate methods: ```scala batchWrite.commit(messages) batchWrite.commitWithOperationMetrics(messages, metrics) ``` Prefer a single method that always accepts the optional parameter: ```scala // Always call commit with metrics parameter // If no metrics, pass an empty Map batchWrite.commit(messages, metrics.getOrElse(Map.empty)) ``` This approach reduces the API surface area, eliminates the need for implementers to handle multiple code paths, and provides a consistent interface. Avoid creating "too many knobs" that complicate the API - instead, ask whether the functionality can be achieved through careful design of a single, flexible method. The same principle applies to streaming APIs and data structures - prefer simple, direct approaches over complex batching or grouping mechanisms unless there's a clear performance or functional requirement that justifies the added complexity. --- ## validate configuration dependencies Validate configuration dependencies and constraints at initialization time rather than allowing invalid combinations to cause runtime failures. When configurations have interdependencies, implement validation logic that checks these relationships early and provides clear error messages. Key practices: - Validate configuration constraints in the config class itself (e.g., `KafkaConfig` should validate that `quorum.auto.join.enable` is only true when `process.role` contains `controller`) - Use appropriate validation methods like `CommandLineUtils.checkRequiredArgs()` but delegate validation to specialized methods when possible - For test configurations, ensure resource requirements match the test scenario (e.g., use 2 brokers with appropriate replica settings rather than over-provisioning with 5 brokers) - When handling feature configurations, validate that finalized feature levels are consistent with the current image rather than making assumptions about missing features Example from the discussions: ```java // In KafkaConfig validation public static final String QUORUM_AUTO_JOIN_ENABLE = QUORUM_PREFIX + "auto.join.enable"; public static final boolean DEFAULT_QUORUM_AUTO_JOIN_ENABLE = false; // Validation should ensure this is only true when process.role contains controller if (autoJoinEnabled && !processRoles.contains("controller")) { throw new ConfigException("quorum.auto.join.enable can only be true when process.role contains controller"); } ``` This approach prevents configuration-related runtime errors and makes system requirements explicit and discoverable during setup rather than during operation. --- ## validate user inputs Always validate and properly escape user-controlled input before incorporating it into structured data formats like JSON, SQL, XML, or command strings. Unescaped input can lead to injection vulnerabilities that allow attackers to manipulate application logic or access unauthorized data. When building JSON strings programmatically, use proper JSON libraries instead of string concatenation. For example, instead of: ```cpp // VULNERABLE - direct string concatenation std::string request_body = "{\"hostname\": \"" + host_str + "\"}"; ``` Use a JSON library or proper escaping: ```cpp // SAFE - using JSON library Poco::JSON::Object json_obj; json_obj.set("hostname", host_str); std::ostringstream oss; json_obj.stringify(oss); std::string request_body = oss.str(); ``` This principle applies to all contexts where user input is incorporated into structured formats. Additionally, validate authentication parameters completely - reject incomplete credential sets with clear error messages rather than silently choosing defaults or ignoring missing values. --- ## optimize expensive operations Before executing computationally expensive operations, implement conditional checks to determine if the operation is actually necessary. This optimization technique can significantly improve performance by avoiding costly computations when simpler alternatives exist or when the operation can be skipped entirely. Key principles: 1. **Identify expensive operations**: Look for operations that involve data collection, type conversion, or complex computations 2. **Add necessity checks**: Implement conditional logic to determine if the expensive operation is required 3. **Provide efficient alternatives**: When the expensive operation isn't needed, use simpler approaches Example from Arrow conversion optimization: ```python # Check if expensive conversion is needed needs_conversion = any( LocalDataToArrowConversion._need_converter(field.dataType, field.nullable) for field in return_type.fields ) if not needs_conversion: try: # Use direct, efficient approach when conversion isn't needed return [pa.RecordBatch.from_pylist(data, schema=pa.schema(list(arrow_return_type)))] except: # Fall back to expensive conversion only if necessary pass ``` This pattern is particularly important in data processing pipelines where operations may be called frequently on large datasets. The upfront cost of the necessity check is typically much lower than the expensive operation itself. --- ## avoid unnecessary object creation Minimize object allocation in performance-critical code paths by reusing existing objects, caching expensive operations, and choosing efficient alternatives to object creation patterns. Key strategies include: 1. **Reuse existing objects instead of creating new ones**: When possible, duplicate or reuse existing objects rather than instantiating new ones. For example, use `partition.duplicate()` instead of creating a new `AlterShareGroupOffsetsResponsePartition`. 2. **Avoid Optional creation in hotspots**: In performance-critical paths, use direct null checks instead of `Optional.ofNullable()`. Replace: ```java Optional.ofNullable(image.cluster().broker(brokerId)).ifPresent(b -> { // process broker }); ``` With: ```java var broker = image.cluster().broker(brokerId); if (broker != null && !broker.fenced() && broker.listeners().containsKey(listenerName.value())) { res.add(brokerId); } ``` 3. **Cache expensive operations outside loops**: When performing expensive operations like `Errors.forException()` that scan lists, compute the result once and reuse it rather than calling it repeatedly in loops. 4. **Choose efficient collection operations**: Use `Collections.unmodifiableSet()` instead of `Set.copyOf()` to avoid deep copying, and prefer direct collection operations over creating intermediate temporary collections. 5. **Optimize conditional checks**: Order conditional checks to perform cheaper operations first, such as checking `partitionInfo.leader() == null` before creating a `TopicPartition` object. This approach is particularly important in frequently executed code paths, request processing hotspots, and tight loops where object allocation overhead can significantly impact performance. --- ## Ensure complete JavaDoc coverage All public classes, methods, and parameters must have comprehensive JavaDoc documentation. This includes: 1. **Class-level JavaDoc**: Every public class and interface requires JavaDoc explaining its purpose and usage 2. **Method JavaDoc**: All public methods need documentation describing their behavior, parameters, return values, and exceptions 3. **Parameter documentation**: When adding new parameters to existing methods, always include @param tags in the JavaDoc 4. **Deprecated elements**: When marking code as @Deprecated, use plain `@Deprecated` annotation without metadata and add corresponding `@deprecated` JavaDoc tag with version and replacement information Example for deprecated code: ```java /** * @deprecated Since 4.2 and should not be used any longer. * @see org.apache.kafka.streams.StreamsConfig */ @Deprecated public class BrokerNotFoundException { // implementation } ``` Example for new parameters: ```java /** * Creates a new RecordAccumulator instance. * @param logContext the log context * @param batchSize the batch size * @param kafka19012Instrumentation instrumentation for KAFKA-19012 metrics */ public RecordAccumulator(LogContext logContext, int batchSize, Kafka19012Instrumentation kafka19012Instrumentation) { // implementation } ``` This ensures consistent documentation standards and helps maintain code readability and API usability for both internal developers and external users. --- ## optimize data structures When working with data structures and collections, optimize for performance and correctness by using modern APIs, proper sizing, result filtering, and lazy initialization. Use `List.of()` and `Map.of()` instead of legacy `Collections.singletonList()` and `Collections.singletonMap()` for better readability and performance. Size collections appropriately - avoid initializing with zero capacity when you know elements will be added. Filter results to match input parameters when reusing cached data to prevent returning unrelated entries. Delay object creation until all required parameters are available to avoid unnecessary work and ensure data consistency. Example of modern collection usage: ```java // Instead of: Collections.singletonList(new StreamsGroupSubtopologyDescription(...)) Collections.singletonMap(groupId, future) // Use: List.of(new StreamsGroupSubtopologyDescription(...)) Map.of(groupId, future) ``` Example of proper collection sizing and lazy initialization: ```java // Instead of creating objects early with incomplete data: new FetchRequest.PartitionData(topicId, offset, 0, 0, ...) // Delay creation until maxBytes is known: // Collect required data first, then create objects with complete parameters ``` This approach improves performance, reduces memory allocation overhead, and ensures data consistency - critical factors for database operations and data access patterns. --- ## validate network state Always validate network connectivity and cluster state before attempting network operations. Check for leader availability, partition existence, and proper metadata state to avoid unnecessary network calls and provide better error handling. Key practices: - Validate that partitions have leaders before performing operations - Filter out partitions without leaders to avoid failed network requests - Request metadata updates and flag partitions as awaiting updates when encountering network errors like NOT_LEADER_OR_FOLLOWER or FENCED_LEADER_EPOCH - Separate validation of network state from the actual operation logic Example from partition validation: ```java // Check the partitions have leader List partitionsWithoutLeader = filterNoneLeaderPartitions(partitionsToReset); if (!partitionsWithoutLeader.isEmpty()) { String partitionStr = partitionsWithoutLeader.stream() .map(TopicPartition::toString) .collect(Collectors.joining(",")); throw new LeaderNotAvailableException("The partitions \"" + partitionStr + "\" have no leader"); } // Prepare data for partitions with leaders only topicPartitions.removeAll(partitionsWithoutLeader); ``` Example from fetch error handling: ```java if (partitionData.currentLeader().leaderId() != -1 && partitionData.currentLeader().leaderEpoch() != -1) { // Process with valid leader info partitionsWithUpdatedLeaderInfo.put(partition, new Metadata.LeaderIdAndEpoch(...)); } else { // Request metadata update for invalid leader state requestMetadataUpdate(metadata, subscriptions, partition); subscriptions.awaitUpdate(partition); } ``` This approach prevents wasted network calls, provides clearer error messages, and ensures operations only proceed when the network state is valid. --- ## comprehensive test coverage Ensure test suites provide comprehensive coverage by including edge cases, boundary conditions, and input variations while verifying that tests actually validate the intended functionality. Tests should cover different input formats (case sensitivity, precision levels), boundary values (empty inputs, maximum/minimum values), and error conditions. Remove test cases that don't actually test the target functionality. Example of comprehensive testing: ```sql -- Test different case variations SELECT time_trunc('HOUR', time'12:34:56'); SELECT time_trunc('hour', time'12:34:56'); SELECT time_trunc('Hour', time'12:34:56'); -- Test boundary conditions SELECT split('', '', -1); SELECT split('', '', 0); SELECT split('', '', 1); -- Test different precisions SELECT time_trunc('MICROSECOND', time'12:34:56.123456'); SELECT time_trunc('MICROSECOND', time'12:34:56.123456789'); ``` This approach ensures robust validation of functionality across all realistic usage scenarios and prevents regressions from unexpected input combinations. --- ## Document configuration constraints Configuration documentation must explicitly specify when settings should or should not be used, including conditional requirements and mutually exclusive options. This prevents misconfigurations and helps developers understand the proper context for each setting. When documenting configurations: - Clearly state when a configuration should NOT be used - Specify conditional requirements (e.g., "only applies when X is set") - Explain mutually exclusive options - Provide context about when settings are relevant Example: ``` controller.quorum.voters - Map of id/endpoint information for static voters. This should NOT be set if using dynamic quorums. See Static versus Dynamic KRaft Quorums for details. group.protocol=streams - Additional ACLs are required for the new Streams rebalance protocol only when group.protocol=streams is set in the configuration. ``` This approach ensures developers can quickly understand configuration boundaries and avoid common setup mistakes. --- ## optimize database operations When working with database operations, prioritize batching multiple statements and pushing operations down to the database level for optimal performance. Use batch statements instead of executing individual prepared statements, and implement proper compatibility checks before attempting query pushdown optimizations. For batching operations, group related database statements together: ```scala // Instead of individual statements conn.prepareStatement("CREATE TABLE test.people (name TEXT, id INTEGER)").executeUpdate() conn.prepareStatement("INSERT INTO test.people VALUES ('fred', 1)").executeUpdate() conn.prepareStatement("INSERT INTO test.people VALUES ('mary', 2)").executeUpdate() // Use batch operations val batchStmt = conn.createStatement() batchStmt.addBatch("CREATE TABLE test.people (name TEXT, id INTEGER)") batchStmt.addBatch("INSERT INTO test.people VALUES ('fred', 1)") batchStmt.addBatch("INSERT INTO test.people VALUES ('mary', 2)") batchStmt.executeBatch() ``` For join pushdown optimizations, ensure proper compatibility validation before attempting to push operations to the database: ```scala override def isOtherSideCompatibleForJoin(other: SupportsPushDownJoin): Boolean = { if (!jdbcOptions.pushDownJoin || !dialect.supportsJoin) return false other.isInstanceOf[JDBCScanBuilder] && jdbcOptions.url == other.asInstanceOf[JDBCScanBuilder].jdbcOptions.url } ``` This approach reduces network round trips, improves query execution performance, and leverages database-native optimizations. Always validate compatibility and feature support before implementing pushdown operations to avoid runtime failures. --- ## comprehensive test coverage Ensure tests cover not only the happy path but also edge cases, error scenarios, and complete workflows. Many code reviews reveal gaps where only basic functionality is tested while important edge cases, exception handling, and integration scenarios are missing. Key areas to address: **Add missing test scenarios**: When implementing new functionality, include tests for all supported operations and variations. For example, if adding support for `--to-earliest`, also add tests for `--to-latest` and `--to-datetime`. **Cover edge cases and error conditions**: Test boundary conditions, empty inputs, invalid parameters, and exception scenarios. Include tests for when external dependencies fail or return unexpected results. **Test complete workflows**: Rather than testing individual components in isolation, add integration-style tests that verify end-to-end functionality works correctly. **Use randomized testing for consistency**: For complex logic, consider adding tests that run multiple iterations with random inputs to ensure consistent behavior across different scenarios. **Verify exception handling**: When code can throw exceptions, test both the success and failure paths to ensure proper error handling and recovery. Example of comprehensive coverage: ```java @Test public void testAlterShareGroupToLatestSuccess() { // Test the happy path } @Test public void testAlterShareGroupWithInvalidInput() { // Test error scenarios } @Test public void testAlterShareGroupEndToEndWorkflow() { // Test complete integration workflow } @Test public void testAlterShareGroupRandomizedInputs() { // Test with varied random inputs for consistency } ``` This approach catches bugs early, improves code reliability, and provides confidence that the implementation handles real-world usage patterns correctly. --- ## Consistent clear naming Use consistent terminology across similar concepts and choose names that clearly indicate their purpose. Avoid overloaded terms that cause confusion in your domain context, and ensure naming conventions align with established patterns. Key principles: - **Consistency across similar concepts**: Use the same naming pattern for related functionality (e.g., "Driver Adapters" not "Prisma Adapters") - **Avoid overloaded terms**: When a term has multiple meanings in your domain, use aliases or more specific names (e.g., alias `Schema` import as `Shape` to avoid confusion with domain Schema) - **Clear purpose indication**: Choose names that make functionality obvious and discoverable to users (e.g., `ignoreSpanNames` instead of `ignoreLayersTypes` when filtering spans) - **Domain-appropriate terminology**: Use terminology that matches the domain or external standards you're working with Example of good naming consistency: ```typescript // Good - consistent naming across adapters export class DriverAdapterError extends Error { name = 'DriverAdapterError' } // Bad - inconsistent terminology export class PrismaAdapterError extends Error { name = 'PrismaAdapterError' } ``` Example of avoiding overloaded terms: ```typescript // Good - avoids confusion with domain Schema import { Schema as Shape } from 'effect' // Potentially confusing in Prisma context import { Schema } from 'effect' ``` --- ## API completeness validation Ensure APIs are complete by validating that all necessary cases are handled, all required arguments are properly validated, and response structures are consistent across similar operations. Key areas to check: 1. **Switch statement completeness**: When adding new enum values or operation types, ensure all switch statements that handle them include the new cases 2. **Argument validation coverage**: Test all invalid argument combinations, not just the happy path scenarios 3. **Response structure consistency**: Maintain consistent field presence across similar API responses (e.g., if one response has top-level error codes, related responses should too) 4. **Parameter naming consistency**: Follow established naming conventions from KIPs and existing APIs Example from the discussions: ```java // Incomplete - missing new RPC cases case UPDATE_VOTER: handledSuccessfully = handleUpdateVoterResponse(response, currentTimeMs); break; // Need to add: case ADD_RAFT_VOTER: handledSuccessfully = handleAddVoterResponse(response, currentTimeMs); break; case REMOVE_RAFT_VOTER: handledSuccessfully = handleRemoveVoterResponse(response, currentTimeMs); break; ``` This prevents runtime failures from unhandled cases and ensures consistent developer experience across the API surface. --- ## Structure components with clarity Maintain clean and logical component organization by following these guidelines: 1. Keep related files together - place test files beside their components rather than in separate folders 2. Avoid declaring components within other components - this creates unnecessary coupling and complicates state management 3. Use PropsWithChildren instead of React.FC for typing components 4. Minimize prop drilling by accessing state directly where needed Example of proper component organization: ```tsx // ✅ Good: Test file next to component // components/MyComponent/ // - MyComponent.tsx // - MyComponent.test.tsx // ✅ Good: Components defined at module level export const ParentComponent = () => { return } export const ChildComponent = () => { return
Child
} // ❌ Bad: Component defined within another const ParentComponent = () => { const NestedComponent = () => { // Avoid this return
Nested
} return } ``` This organization improves code maintainability, reduces coupling, and makes the codebase easier to navigate. --- ## catch specific exceptions Avoid catching overly broad exception types like `Throwable` or `Exception` when you can be more specific about the expected failure modes. Catching `Throwable` is particularly dangerous as it captures JVM-level errors like `OutOfMemoryError` that should typically cause the application to exit. Instead of catching broad exception types, identify the specific exceptions that can occur and handle them appropriately. This makes error handling more precise and prevents masking serious system-level problems. **Example of problematic code:** ```java try { compressedPayload = ClientTelemetryUtils.compress(payload, compressionType); } catch (Throwable e) { log.debug("Failed to compress telemetry payload for compression: {}, sending uncompressed data", compressionType); // This catches OutOfMemoryError, which should probably crash the JVM } ``` **Improved approach:** ```java try { compressedPayload = ClientTelemetryUtils.compress(payload, compressionType); } catch (IOException | NoClassDefFoundError e) { log.debug("Failed to compress telemetry payload for compression: {}, sending uncompressed data", compressionType, e); // Only catches the specific exceptions we can recover from } ``` When you must catch broader exception types, document why and consider whether the error handling is appropriate for all possible exception types that could be caught. Remember that catching `Throwable` would catch things like `OutOfMemoryException` and various other exceptions that indicate the JVM should exit. --- ## Externalize configuration values Avoid hardcoding configuration values directly in code. Instead, make them externally configurable through appropriate mechanisms based on their usage patterns. Use environment variables for values that may need to change between deployments or runs, move configuration setup to appropriate lifecycle methods (like setupClass), and choose the right location based on variability expectations. For values that vary across runs of the same pipeline, use CLI arguments. For static pipeline configurations, use configuration files or specs. When possible, leverage built-in configuration mechanisms rather than implementing custom parsing. Example of improvement: ```python # Before: hardcoded retry count exec_sbt(profiles_and_goals, retry=3) # After: configurable via environment variable retry_count = int(os.environ.get('SBT_RETRY_COUNT', '3')) exec_sbt(profiles_and_goals, retry=retry_count) ``` This approach improves maintainability, reduces the need for code changes when configuration requirements change, and follows the principle that configuration should be external to code. --- ## parameterize configuration values Replace hardcoded configuration values with parameterized variables in build files and configuration management. This enables flexible environment-specific settings and improves maintainability. Hardcoded values in configuration files create inflexibility and make it difficult to adapt builds for different environments or requirements. Instead, use variables that can be overridden through system properties or environment-specific configuration. Example of improvement: ```xml 17 1.11.655 ${maven.compiler.release} ${aws.sdk.version} ``` For dependency management, centralize version definitions in parent POM files using properties that can be easily updated and maintained. This approach allows teams to specify different versions through build parameters like `-Djava.version=21` while maintaining compatibility and consistency across the project. --- ## Complete method documentation All public methods, especially in interfaces and APIs, must have comprehensive JavaDoc documentation that clearly describes their purpose, return values, and behavior. Avoid vague or ambiguous descriptions that leave developers guessing about the method's functionality. Method documentation should be particularly thorough for public APIs since users may only have access to compiled JARs without source code comments. Each method should explicitly state what it returns, under what conditions, and any important behavioral details. Example of unclear documentation: ```java /** * Similar to {@link #toDDL()}, but returns the description of this constraint for describing */ ``` Example of clear documentation: ```java /** * Returns the constraint description for DESCRIBE TABLE output, excluding the constraint name (shown separately). */ ``` For interface methods, always include JavaDoc even if the method signature seems self-explanatory, as this documentation becomes the primary reference for implementers and users of the API. --- ## prefer modern collection APIs Use modern collection creation methods instead of legacy alternatives for improved readability and conciseness. Replace older patterns with their modern equivalents: - Use `List.of()` instead of `Arrays.asList()` or `Collections.singletonList()` - Use `Set.of()` instead of `new HashSet<>(Arrays.asList(...))` - Use `.toList()` instead of `.collect(Collectors.toList())` - Use `Collections.emptyList()` → `List.of()` when creating empty immutable lists Example transformations: ```java // Before List items = Arrays.asList("a", "b", "c"); Set permissions = new HashSet<>(Arrays.asList("READ", "WRITE")); List results = stream.collect(Collectors.toList()); // After List items = List.of("a", "b", "c"); Set permissions = Set.of("READ", "WRITE"); List results = stream.toList(); ``` Modern collection APIs are more concise, expressive, and create immutable collections by default, which helps prevent accidental modifications and improves code safety. They also eliminate the need for intermediate collection creation steps, making the code more direct and readable. --- ## Include contextual information Log messages should include comprehensive contextual information to support debugging and system monitoring. This includes relevant identifiers, operation details, and state information that help developers understand what happened and why. Key practices: - Include relevant IDs (task ID, provider ID, query run ID, etc.) - Explain the purpose or context of operations, especially for non-obvious cases like "version + 1" representing the current operating version - Add quantitative information when available (e.g., "how many were closed and how many were readded") - Use structured formatting with field names for clarity: `StateStoreProviderId[ storeId=$storeId, queryRunId=$queryRunId ]` - Provide context for temporary or placeholder states to avoid confusion Example of good contextual logging: ```scala logInfo(log"Task thread trigger maintenance to close provider " + log"${MDC(TASK_ID, taskId)} for ${MDC(STATE_STORE_PROVIDER_ID, providerId)} " + log"- provider removed from loadedProviders") ``` Balance completeness with readability - include essential context but avoid excessive noise that makes logs harder to parse. Focus on information that would be valuable for debugging, monitoring, or understanding system behavior. --- ## Protect shared state Always protect shared mutable state with appropriate synchronization mechanisms to prevent race conditions and data corruption in multi-threaded environments. Use `std::call_once` for thread-safe initialization, mutexes for protecting shared resources, and carefully consider whether state belongs in global vs local thread contexts. Key practices: - Use `std::call_once` instead of simple boolean flags for one-time initialization - Wrap shared resources like static variables with mutex protection - Place ordinality counters and similar sequential data in global state with proper locking when multiple threads need coordinated access - Avoid placing thread-coordination data in local state when parallel execution is expected Example of proper initialization: ```cpp void KeywordHelper::InitializeKeywordMaps() { static std::once_flag initialized_flag; std::call_once(initialized_flag, []() { // initialization code here }); } ``` Example of protecting shared output: ```cpp struct FailureSummary { void AddSummary(const string &failure) { std::lock_guard guard(lock); summary << failure; } std::ostringstream summary; std::mutex lock; }; ``` When designing parallel operators, ensure that shared sequential state (like ordinality indices) is properly synchronized in global state rather than duplicated in local thread state, which would produce incorrect results across parallel execution paths. --- ## self-documenting code practices Write code that explains itself through clear structure and meaningful names, while adding targeted comments only where necessary for future maintainability. Extract complex logic into well-named functions with comprehensive documentation, but avoid redundant comments that duplicate what the code already expresses clearly. Key principles: - Extract complex logic into dedicated functions with descriptive names and comprehensive documentation - Add comments to explain non-obvious business logic or data structures for future readers - Remove comments that simply restate what the code does - let clear exception messages and function names speak for themselves - Organize code into logical groupings that make the overall structure self-evident Example of good practice: ```scala // Good: Extract complex logic with clear documentation /** * Creates table filters for dataflow operations. * Maps expected filter results depending on combination of full refresh and partial refresh. */ private def createTableFilters(fullRefresh: Boolean, partialRefresh: Boolean): TableFilters = { // Implementation here } // Good: Targeted comment for future maintainers // Providers that couldn't be processed now and need to be added back to the queue val providersToRequeue = new ArrayBuffer[(StateStoreProviderId, StateStoreProvider)]() // Avoid: Redundant comment that restates the obvious // throw exception if pipeline does not have table or persisted view throw new IllegalStateException("Pipeline must have table or persisted view") ``` This approach reduces maintenance burden by preventing stale comments while ensuring complex logic remains understandable to future developers. --- ## Use parameter-based paths When designing API routes and interfaces, always use parameter-based path syntax instead of hardcoded literals. This approach provides better flexibility, improved type safety, and allows for more reusable code. Additionally, always validate parameters before using them in routes or API calls to prevent undefined reference errors. Example of proper path parameterization: ```typescript // Incorrect - using literal values path: '/platform/projects/default/analytics/endpoints/logs.all' // Correct - using parameter syntax path: '/platform/projects/:ref/analytics/endpoints/logs.all' ``` Example of proper parameter validation: ```typescript // Incorrect - not validating parameter before use if (slug === 'last-visited-org') { router.replace(`/new/${lastVisitedOrganization}`) } // Correct - validating parameter existence if (slug === 'last-visited-org' && !!lastVisitedOrganization) { router.replace(`/new/${lastVisitedOrganization}`) } ``` This approach also enhances API flexibility by supporting optional parameters, allowing consumers to provide only what they need. When designing component APIs, consider parameterizing inputs to increase reusability across different contexts. --- ## avoid manual error handling Prefer centralized error handling mechanisms and specialized utilities over manual error handling in individual components. Instead of implementing error handling logic directly in each function or component, leverage global error handlers and purpose-built utilities. For centralized handling, use global `onError` handlers rather than manual error handling in each mutator: ```js // Avoid: Manual error handling in each component const handleSubmit = async () => { const result = z.mutate.issue.create({...}); const raceResult = await promiseRace([sleep(5000), result.server]); if (raceResult === 0) { // TODO show toast - manual error handling } }; ``` For specific error scenarios, use specialized utilities that provide both static and runtime safety: ```js // Avoid: Generic error throwing if (filter === 'crew') { // handle crew } else if (filter === 'creators') { // handle creators } else { throw new Error(`Unknown filter: ${filter}`); } // Prefer: Specialized error utilities if (filter === 'crew') { // handle crew } else if (filter === 'creators') { // handle creators } else { unreachable(filter); // Provides static and runtime assertion } ``` This approach reduces code duplication, improves consistency, and provides better type safety and debugging capabilities. --- ## ensure test isolation Tests must properly clean up resources and avoid side effects that can impact other tests, especially when running in parallel. This prevents non-deterministic failures and ensures reliable test execution. Key practices: - Clean up any created resources (tables, files, connections) after test completion - Use unique identifiers for test resources to avoid conflicts - Ensure tests don't leave persistent state that affects subsequent tests - Consider using test fixtures or teardown methods for consistent cleanup Example of proper cleanup: ```python def test_streaming_foreach_batch_external_column(self): table_name = "testTable_foreach_batch_external_column" try: # Test logic here pass finally: # Clean up the table to avoid affecting other tests self.spark.sql(f"DROP TABLE IF EXISTS {table_name}") ``` This approach prevents issues where tests affect catalog-related tests or cause hanging/flaky behavior when run in parallel environments. --- ## validate before data access Always validate for null or None values before accessing data elements, especially when working with collections or optional parameters. Avoid direct indexing or attribute access without first ensuring the data exists and is valid. When accessing elements from collections that may contain nulls, filter out null values first rather than assuming the first element is valid: ```python # Instead of direct access: # return pser.iloc[0].__UDT__ # Filter nulls first, then access: notnull = pser[pser.notnull()] if len(notnull) > 0 and hasattr(notnull.iloc[0], "__UDT__"): return notnull.iloc[0].__UDT__ ``` Additionally, maintain consistent patterns for handling None vs empty collections in function parameters to avoid confusion and potential bugs. If a function expects a list but receives None, establish a clear convention (either convert None to empty list, or handle None explicitly) and apply it consistently across the codebase. --- ## Simplify conditional structures Organize complex conditional logic using clear, sequential patterns rather than nested structures or multiple early returns. This improves code readability and maintainability by making the flow of logic easier to follow. Key practices: - **Avoid nested conditionals**: Use consecutive checks instead of deeply nested if-else structures - **Organize exit points early**: Move validation checks and early returns to the beginning of methods - **Use match statements for complex combinations**: When dealing with multiple related conditions, prefer pattern matching over sequential if-else chains - **Eliminate early returns in loops**: Structure conditional flow to return a single value at the end Example of improvement: ```scala // Instead of nested conditionals: windowExpression.windowFunction match { case AggregateExpression(_, _, true, _, _) => // nested validation logic } // Use consecutive checks with extracted methods: checkWindowFunction(windowExpression) checkWindowFunctionAndFrameMismatch(windowExpression) ``` For complex parameter combinations, prefer explicit pattern matching: ```scala // Instead of sequential validation: if (fullRefreshTables.nonEmpty && fullRefreshAll) { ... } if (refreshTables.nonEmpty && fullRefreshAll) { ... } // Use pattern matching: (fullRefreshTables, refreshTableNames) match { case (Nil, Nil) => ... case (fullRefreshTables, Nil) => ... case ... } ``` This approach reduces cognitive load and makes the code's intent clearer by explicitly handling each logical case. --- ## optimize algorithmic complexity Replace inefficient algorithms with more optimal data structures and approaches to improve computational complexity. Look for opportunities to use appropriate data structures (like priority queues) instead of linear searches, avoid redundant iterations, and choose efficient traversal patterns. Key optimization patterns to apply: 1. **Use priority queues for selection problems**: Instead of iterating through collections to find minimum/maximum elements, maintain a priority queue for O(log n) operations. ```java // Instead of linear search through all members final Member member = findMemberWithLeastLoad(allMembers, task, false); // Use priority queue for efficient selection PriorityQueue processByLoad = new PriorityQueue<>(Comparator.comparingDouble(ProcessState::load)); processByLoad.addAll(localState.processIdToState.values()); ProcessState processWithLeastLoad = processByLoad.poll(); // Remember to add back after state changes processByLoad.add(processWithLeastLoad); ``` 2. **Avoid redundant iterations**: Use iterators or single-pass algorithms when possible to prevent multiple traversals of the same data. ```java // Instead of filtering then iterating again finalizedFeatureLevels.keySet().stream().filter(predicate).forEach(action); // Use iterator for single-pass removal var iter = finalizedFeatureLevels.keySet().iterator(); while (iter.hasNext()) { var featureName = iter.next(); if (shouldRemove(featureName)) { removeFinalizedFeatureLevelMetric(featureName); iter.remove(); } } ``` 3. **Choose efficient loop constructs**: Prefer simple loops over streams when performance is critical, especially for primitive operations. ```java // Replace streams with loops for better performance // replaced the Java Streams based iteration with a loop, since it's more efficient for (Member member : prevMembers) { if (member.hasLeastLoad()) { return member; } } ``` 4. **Understand recursion bounds**: When using recursion, ensure it has predictable and bounded depth to avoid stack overflow while maintaining algorithmic clarity. This approach reduces time complexity from O(n) linear searches to O(log n) priority queue operations and eliminates unnecessary object creation during iteration. --- ## Clear, descriptive identifiers Choose variable, component, and parameter names that clearly describe their purpose and avoid ambiguity. Names should fully reflect functionality, be properly spelled, and avoid confusion with library terms or similar concepts. Good identifiers: - Are semantic rather than implementation-focused (e.g., use `ProtectedSchemaWarning` instead of `Alert_Shadcn_`) - Avoid ambiguity with library terminology (e.g., use `isHealthy` instead of `isSuccess` when not related to API request status) - Fully describe contained functionality (e.g., `BillingCustomerDataDialog` instead of `BillingAddressDialog` when handling both address and tax ID) - Use correct spelling (e.g., `referral` not `referal`) Example: ```typescript // Unclear naming, potential confusion with React Query const StatusMessage = ({ isSuccess, status }) => { if (isSuccess) return 'Healthy' // ... } // Clear, descriptive naming that avoids ambiguity const StatusMessage = ({ isHealthy, status }) => { if (isHealthy) return 'Healthy' // ... } ``` Clear naming significantly improves code readability, maintainability, and reduces the cognitive load for developers who need to understand the code. --- ## comprehensive database testing Database tests should execute actual queries and verify results comprehensively, not just check query plans or use hash comparisons. Always include edge cases, test data scenarios with both expected and unexpected data, and verify storage correctness through restart/reload cycles. Key practices: - Execute queries and verify actual results rather than only checking query plans - Use labeled results with `nosort` instead of hash comparisons for result verification - Add comprehensive test cases covering edge cases, null/non-null data, and boundary conditions - For storage tests, include restart/reload verification to ensure data persistence correctness - Test both positive cases (expected data) and negative cases (edge conditions) Example of proper test verification: ```sql query III nosort read_csv_result SELECT * FROM read_csv('test/data.csv') WITH ORDINALITY ORDER BY col1, col2, ordinality; query III nosort read_csv_result SELECT *, row_number() OVER () AS ordinality FROM read_csv('test/data.csv') ORDER BY col1, col2, ordinality; ``` For storage tests, always verify data integrity after database restart: ```sql statement ok CREATE TABLE test_table AS SELECT * FROM large_dataset; # Restart database restart query I SELECT COUNT(*) FROM test_table; ``` This approach ensures database functionality works correctly under real-world conditions and catches issues that plan-only testing might miss. --- ## API initialization side effects When initializing API clients, prefer bootstrap/configuration patterns over method calls that may trigger unintended side effects like billing events, data capture, or state changes. Method calls during initialization can have unexpected consequences that users may not anticipate or want to pay for. Instead of calling methods like `identify()`, `capture()`, or similar action-triggering functions during client setup, use configuration objects, bootstrap data, or initialization parameters to achieve the same result without side effects. Example of problematic initialization: ```javascript // This triggers an identify event that users get billed for posthog.init(token, config); posthog.identify(distinctId); // Captures billable event ``` Preferred approach using bootstrap configuration: ```javascript // This achieves the same result without capturing events const config = { api_host: projectConfig.api_host, bootstrap: { distinctId: distinctId // Set identity without triggering events } }; posthog.init(token, config); ``` This pattern ensures that client initialization only sets up the necessary state without triggering actions that have business implications or costs. Always consider whether initialization methods have side effects and prefer declarative configuration approaches when available. --- ## Use parameterized logging Use parameterized logging with placeholders (`{}`) instead of string concatenation for better performance and readability. When logging exceptions, pass the exception object as a separate parameter to the logger rather than including its message in the format string. **Why this matters:** - Parameterized logging avoids unnecessary string concatenation when the log level is disabled - Exception objects as separate parameters provide full stack traces and better debugging information - Cleaner, more readable code that follows logging framework best practices **Examples:** ❌ **Avoid string concatenation:** ```java logger.debug("Fail to read the clean shutdown file in " + cleanShutdownFile.toPath() + ":" + e); LOGGER.error(format("Encountered error while deleting %s", file.getAbsolutePath())); ``` ✅ **Use parameterized logging:** ```java logger.debug("Fail to read the clean shutdown file in {}:{}", cleanShutdownFile.toPath(), e); LOGGER.error("Encountered error while deleting {}", file.getAbsolutePath(), e); ``` ❌ **Avoid including exception message in format string:** ```java String msg = String.format("Deserializing record %s from %s failed due to: %s", record, tp, ex.getMessage()); LOG.error(msg); ``` ✅ **Pass exception as separate parameter:** ```java String msg = String.format("Deserializing record %s from %s failed", record, tp); LOG.error(msg, ex); ``` This approach ensures optimal performance, provides complete exception information, and maintains consistent logging patterns across the codebase. --- ## avoid null in Scala Avoid using null values in Scala code and prefer Option types for representing optional values. Null usage can lead to NullPointerException at runtime and makes code less safe and predictable. When working with optional values, use Option[T] and handle the None case explicitly. For Java interoperability where null is required, use `.orNull` on the Option to convert safely: ```scala // Avoid this val aliasName = if (rightSideRequiredColumnNames.contains(name)) { generateJoinOutputAlias(name) } else { null // Dangerous - can cause NPE } // Prefer this val aliasName = if (rightSideRequiredColumnNames.contains(name)) { Some(generateJoinOutputAlias(name)) } else { None } // When calling Java functions that expect null javaFunction(aliasName.orNull) ``` This pattern is especially important when dealing with serializable classes where transient fields can become null after deserialization, potentially causing unexpected NPEs. Make fields private when they could be null due to serialization to prevent accidental access to null values. --- ## Maintain code consistency Ensure consistent code organization, naming conventions, and structure throughout the codebase: 1. Use identical parameter names for similar components (e.g., `queryGroup="language"` instead of custom names) 2. Place utility functions at appropriate scope levels - if a function has no component dependencies, define it outside the component 3. Apply proper semantic HTML structure (e.g., wrapping form elements in `
` tags) 4. Scope linting disables narrowly with `disable-next-line` rather than disabling for entire files ```jsx // GOOD: Consistent parameter naming // GOOD: Function properly scoped outside component const generateNonce = async (): Promise => { // implementation } function MyComponent() { // component code using generateNonce } // GOOD: Proper semantic structure // GOOD: Narrowly scoped linting disable {/* supa-mdx-lint-disable-next-line Rule003Spelling */} ``` These consistent practices improve code readability, maintainability, and ensure predictable behavior across the application. --- ## Maintain consistent naming patterns Ensure all new functions, files, and identifiers follow established naming conventions within the codebase. This prevents naming conflicts, maintains API consistency, and improves code readability. For API functions, consistently use the established prefix pattern. For example, instead of `arrow_to_duckdb_schema`, use `duckdb_schema_from_arrow` to maintain the `duckdb_` prefix convention. Similarly, prefer descriptive names like `duckdb_value_to_string` over `duckdb_to_sql_string` to clearly indicate what the function operates on. File names should also reflect their contents accurately - rename files like `new_vector_types.json` to `new_vector_functions.json` when the content changes to include functions rather than just types. Deviating from established patterns can cause integration issues for clients that expect consistent naming conventions, especially when generating bindings that assume specific prefixes. --- ## thoughtful configuration design When designing configuration options, environment variables, and build settings, follow established patterns and ensure they serve a clear purpose. Consider cross-platform compatibility, use familiar conventions (like PATH-style lists), and align configuration choices with project requirements. Key principles: - Follow established patterns (e.g., colon/semicolon-separated lists like PATH) - Handle platform differences appropriately (`;` on Windows, `:` on Unix) - Ensure every configuration variable is actually used - avoid creating intermediate variables that serve no purpose - Align configuration granularity with project compatibility guarantees (e.g., version numbering should reflect actual compatibility boundaries) - Remove unused configuration variables rather than leaving them as dead code Example: ```cmake # Good: Uses platform-appropriate separators and follows PATH convention set(EXTENSION_DIRECTORIES "~/.duckdb/extensions" CACHE STRING "Extension directories (colon/semicolon separated)") # Good: Version reflects actual compatibility guarantees set_target_properties(duckdb PROPERTIES SOVERSION ${DUCKDB_MAJOR_VERSION}.${DUCKDB_MINOR_VERSION}) # Bad: Creates unused intermediate variable set(SUMMARIZE_FAILURES_ENV "$ENV{SUMMARIZE_FAILURES}") # Never actually used ``` --- ## Verify CI build consistency Ensure CI pipelines thoroughly verify build outputs and maintain consistency across different build tools and configurations. This prevents broken builds from reaching contributors and ensures reliable deployment artifacts. Key practices: 1. **Cross-tool verification**: When using multiple build tools (Maven, SBT), verify that both produce consistent, working artifacts 2. **Proper build phases**: Use correct Maven lifecycle phases (e.g., `package test` instead of just `test`) to ensure proper classpath resolution 3. **Rigorous output testing**: Test the actual distribution artifacts, not just intermediate build outputs Example from Maven workflow: ```yaml # Ensure proper Maven phase ordering for integration tests ./build/mvn $MAVEN_CLI_OPTS -pl sql/connect/client/jvm,sql/connect/client/integration-tests,sql/connect/common,sql/connect/server package test -fae ``` This approach protects contributors' daily development by catching build inconsistencies early and making it easier to identify offending commits when builds break. --- ## API response completeness Ensure API responses contain all necessary data fields and provide mechanisms for clients to verify operation results. When building API responses, successful operations should include complete metadata (such as topic IDs, timestamps, or other identifiers), while failed operations should contain appropriate error information. Additionally, provide corresponding read/list operations that allow clients to verify the results of write operations. For example, when implementing delete operations like `deleteShareGroupOffsets`, ensure that: 1. Successful responses include all relevant identifiers 2. A corresponding list operation (`listShareGroupOffsets`) exists for verification 3. Response builders properly merge successful and failed parts without losing critical data ```scala // Ensure topic IDs are preserved in successful responses val responseBuilder = new AlterShareGroupOffsetsResponse.Builder() // For successful topics, include topic ID in response responseBuilder.addPartition(topic.topicName(), partition.partitionIndex(), error.error) // Verify merge operation preserves all necessary fields requestHelper.sendMaybeThrottle(request, responseBuilder.merge(response).build()) ``` This practice enables clients to implement robust error handling and verification workflows, improving the overall reliability of API interactions. --- ## avoid stale ref values When using React refs in hooks like useLayoutEffect or useEffect, pass the ref object itself rather than ref.current to avoid stale values. The ref.current value captured at hook definition time may be outdated when the effect callback executes. Example of the problem: ```typescript // ❌ Problematic - elm will be stale export function useElementSize(elm: HTMLElement | null) { useLayoutEffect(() => { // elm here is the stale value from when hook was defined if (elm) { // This may operate on wrong/null element } }, [elm]); } ``` Correct approach: ```typescript // ✅ Better - pass the ref object export function useElementSize(elm: React.RefObject) { useLayoutEffect(() => { // elm.current will always be the current value if (elm.current) { // This operates on the correct element } }, [elm]); } ``` This pattern ensures your effects always work with the current DOM element rather than a potentially stale reference, preventing subtle bugs that might only surface under specific rendering conditions. --- ## prefer system properties directly When detecting operating system or environment characteristics, prefer direct access to system properties over external library dependencies for basic checks. Use `System.getProperty("os.name")` with `regionMatches()` for case-insensitive OS detection instead of relying on third-party utilities like Apache Commons Lang3's SystemUtils. This approach reduces external dependencies while maintaining equivalent functionality. The `regionMatches()` method provides robust, case-insensitive string matching that handles OS name variations effectively. Example implementation: ```java // Instead of: SystemUtils.IS_OS_WINDOWS boolean isWindows = System.getProperty("os.name").regionMatches(true, 0, "Windows", 0, 7); // Instead of: SystemUtils.IS_OS_UNIX String osName = System.getProperty("os.name"); String[] unixPrefixes = {"AIX", "HP-UX", "Linux", "Mac OS X", "Solaris", "FreeBSD"}; boolean isUnix = Arrays.stream(unixPrefixes) .anyMatch(prefix -> osName.regionMatches(true, 0, prefix, 0, prefix.length())); ``` This pattern is particularly valuable when the external library is used minimally and the custom implementation can be easily maintained and tested. --- ## explicit null handling Prefer explicit null and undefined handling over optional or nullable types. When possible, provide default values or objects instead of making parameters optional. When null/undefined checks are necessary, handle both values explicitly and use descriptive patterns. Instead of relying on optional parameters that create chains of conditional access: ```typescript // Avoid interface MigrateSetupInput { schemaFilter?: MigrateTypes.SchemaFilter } // Later: schemaFilter?.['prop'] access // Prefer const defaultSchemaFilter = { externalTables: [], exclude: [], } satisfies MigrateTypes.SchemaFilter ``` When checking for null/undefined values, be explicit about both: ```typescript // Handle both null and undefined if (obj.currentTimeframe == null) { return {} } // Or check explicitly if (response.body === null) { return reject(new Error('response.body is null')) } ``` This approach reduces conditional access chains, makes code more predictable, prevents null reference errors, and improves type safety by making the presence or absence of values explicit rather than implicit. --- ## Defensive null validation Always validate null parameters and dependencies early with proper ordering to prevent NullPointerExceptions and provide clear error messages. Use defensive programming practices including parameter validation, method overloading instead of null parameter handling, and careful ordering of null checks. Key practices: 1. **Early parameter validation**: Check for null parameters at method entry points using `Objects.requireNonNull()` with descriptive messages 2. **Proper validation ordering**: Perform null checks before operations that could mask the real cause of NPEs 3. **Method overloading over null handling**: Instead of accepting null parameters and checking them, provide separate method signatures 4. **Proactive dependency checking**: Validate that required dependencies exist before using them to avoid runtime NPEs Example of proper null validation ordering: ```java public record CommittedPartitionState(Set isr, LeaderRecoveryState leaderRecoveryState) { public CommittedPartitionState { Objects.requireNonNull(isr); // Check null first this.isr = Set.copyOf(isr); // Then perform operations Objects.requireNonNull(leaderRecoveryState); } } ``` Example of method overloading instead of null checks: ```java // Instead of: props(Properties extraProperties) with null check private Properties props() { /* base implementation */ } private Properties props(Properties extraProperties) { Properties config = props(); config.putAll(extraProperties); // No null check needed return config; } ``` This approach prevents NPEs, provides clearer stack traces when failures occur, and makes the code's null-handling contract explicit to callers. --- ## Improve code readability Write code that prioritizes readability through clear string formatting, descriptive method calls, and well-organized structure. Use string interpolation instead of concatenation for better readability, choose descriptive assertion methods that provide meaningful error messages, avoid code duplication by leveraging method overloading, and extract complex logic into focused methods. For string formatting, prefer interpolated strings: ```scala // Instead of: throw new TerseFailure("Unknown metadata.version " + releaseVersion + ". Supported metadata.version are " + metadataVersionsToString(...)) // Use: throw new TerseFailure(s"Unknown metadata.version $releaseVersion. Supported metadata.version are ${metadataVersionsToString(...)}") ``` For assertions, use descriptive methods: ```scala // Instead of: assertTrue(memberId != JoinGroupRequest.UNKNOWN_MEMBER_ID) // Use: assertNotEquals(JoinGroupRequest.UNKNOWN_MEMBER_ID, memberId) ``` Avoid code duplication by overloading existing methods rather than creating similar methods with slight variations. When methods become complex, extract the core logic into separate, focused methods that can be easily tested and understood. This approach reduces cognitive load, improves maintainability, and makes code reviews more effective. --- ## Centralize configuration values Extract and centralize configuration values instead of duplicating or hardcoding them throughout the codebase. This improves maintainability by ensuring changes only need to be made in one place and enforces consistency across the application. **Best practices:** - Use shared configuration objects for common settings (like query options) - Extract hardcoded values into named constants or environment variables - Create structured environment checks for environment-specific configurations **Example - Before:** ```typescript // Hardcoded values in multiple files Sentry.init({ dsn: 'https://3c27c63b42048231340b7d640767ad02@o398706.ingest.us.sentry.io/4508132895096832', }) // Duplicated configuration { enabled: enabled && typeof projectRef !== 'undefined', refetchOnWindowFocus: false, } // Simple environment check export const DEFAULT_PROVIDER: CloudProvider = process.env.NEXT_PUBLIC_ENVIRONMENT !== 'prod' ? 'AWS_K8S' : 'AWS' ``` **Example - After:** ```typescript // Constants file export const SENTRY_DSN = process.env.SENTRY_DSN || 'https://3c27c63b42048231340b7d640767ad02@o398706.ingest.us.sentry.io/4508132895096832' // Shared configuration object export const UNIFIED_LOGS_QUERY_OPTIONS = { enabled: enabled && typeof projectRef !== 'undefined', refetchOnWindowFocus: false, } // Structured environment check export const DEFAULT_PROVIDER: CloudProvider = process.env.NEXT_PUBLIC_ENVIRONMENT && ['staging', 'preview'].includes(process.env.NEXT_PUBLIC_ENVIRONMENT) ? 'AWS_K8S' : 'AWS' ``` --- ## Explicit CI configurations CI/CD workflows should use explicit, named configurations rather than wildcards, globs, or implicit behaviors to improve maintainability and debuggability. This makes it easier for developers to understand failures and reduces silent errors. Key practices: - Use explicit job names and dependencies instead of wildcards (e.g., individual config files with logical names like "encrypted_db.config" rather than globbing "test/configs/*") - Specify exact build targets instead of pattern matching (e.g., "cp310" instead of "cp310-*") - Add proper job dependencies with `needs:` to ensure correct execution order - Use explicit file handling that fails on missing files rather than silently continuing - Maintain lean, explicit version matrices with only necessary combinations Example of explicit vs implicit configuration: ```yaml # Avoid: Implicit glob pattern run: | for file in test/configs/*; do ./build/release/test/unittest --test-config "$file" done # Prefer: Explicit named jobs encrypted-db-test: name: Encrypted Database Test run: ./build/release/test/unittest --test-config encrypted_db.config performance-test: name: Performance Test run: ./build/release/test/unittest --test-config performance.config ``` This approach makes CI failures easier to diagnose and workflows more maintainable for the development team. --- ## Environment variable patterns Use consistent patterns for environment variable handling and configuration validation. Access environment variables directly where they're needed rather than loading them unnecessarily in build configurations. Provide clear fallback mechanisms and fail fast with descriptive error messages when required configuration is missing or invalid. Key practices: - Access `process.env` directly in the consuming module instead of loading environment variables in build/config files - Use environment variables instead of hard-coded paths: prefer `ZERO_SCHEMA_PATH=shared/schema.ts` over hard-coding paths - Implement proper fallback chains with validation: ```typescript const loadSchemaJson = () => { if (process.env.ZERO_SCHEMA_JSON) { return process.env.ZERO_SCHEMA_JSON; } try { const schema = readFileSync("zero-schema.json", "utf8"); return JSON.stringify(JSON.parse(schema)); } catch (error) { throw new Error( "Schema must be provided via ZERO_SCHEMA_JSON env var or zero-schema.json file" ); } }; ``` - Fail fast on configuration conflicts rather than issuing warnings that may be ignored - Add new environment variables to `.env.example` files for documentation - Validate configuration values early in the application lifecycle with clear error messages --- ## enforce database constraints properly Database schemas should use appropriate constraints to enforce business rules and prevent data inconsistencies. When designing tables, identify potential duplicate data scenarios and implement unique constraints to prevent them. Consider consolidating related tables when a single table with proper constraints can achieve the same goal more efficiently. For example, instead of allowing duplicate emoji assignments: ```sql -- Problem: Allows same user to assign same emoji multiple times CREATE TABLE emoji ( "id" VARCHAR PRIMARY KEY, "value" VARCHAR NOT NULL, "subjectID" VARCHAR NOT NULL, "creatorID" VARCHAR REFERENCES "user"(id) ); -- Solution: Add unique constraint to prevent duplicates CREATE TABLE emoji ( "id" VARCHAR PRIMARY KEY, "value" VARCHAR NOT NULL, "subjectID" VARCHAR NOT NULL, "creatorID" VARCHAR REFERENCES "user"(id), UNIQUE ("subjectID", "creatorID", "value") ); ``` Additionally, document non-obvious schema design decisions with comments, especially when columns might seem redundant but serve specific business logic purposes. Consider primary key ordering based on common query patterns - place the most frequently filtered column first for better performance. --- ## avoid overly specific examples When documenting configuration options, use generic examples that don't unnecessarily tie documentation to specific versions, environments, or exact default values. This keeps documentation maintainable and prevents examples from becoming outdated or environment-specific. Instead of specifying exact OS versions or precise default values: ```bash # Too specific image_name="ducker-ak-openjdk:17-buster" bash tests/docker/run_tests.sh # Better - generic version image_name="ducker-ak-openjdk:17" bash tests/docker/run_tests.sh ``` For default values, mention their existence without being overly specific: ``` You can customize the OpenJDK base image using the `-j` or `--jdk` parameter, otherwise a default value will be used. ``` This approach makes configuration documentation more resilient to changes and easier to maintain across different environments. --- ## Centralize configuration values Avoid duplicating configuration values across multiple files by maintaining a single source of truth for environment variables, build arguments, version specifications, and other configuration parameters. When the same value appears in build scripts, Dockerfiles, documentation, or configuration files, consolidate it to one authoritative location and reference it from other places using placeholders or variables. This prevents inconsistencies that arise when updating configuration values, reduces maintenance overhead, and eliminates inaccurate or stale configuration data that becomes outdated during rebuilds or updates. Example of the problem: ```dockerfile # In Dockerfile - duplicated default value ARG jdk_version=openjdk:17-bullseye # In build script - same default value duplicated DEFAULT_JDK=openjdk:17-bullseye ``` Better approach: ```dockerfile # In Dockerfile - no default, relies on build script ARG jdk_version # In README.md - use placeholder FROM ``` Also avoid configuration values that become inaccurate over time, such as hardcoded build dates in environment variables that don't reflect actual rebuild times. --- ## Eliminate unnecessary complexity Remove unnecessary default parameters and consolidate related conditional logic to improve code clarity and maintainability. When parameters are always provided in practice, avoid adding default values that create false optionality. Similarly, combine related conditions into single, more readable expressions. Examples of improvements: - Remove default parameters that are never used: Change `def run(spec_path: Path, full_refresh: Optional[Sequence[str]] = None)` to `def run(spec_path: Path, full_refresh: Optional[Sequence[str]])` when the parameter is always provided - Consolidate related conditions: Replace separate `if full_refresh:` and `if refresh:` checks with `if full_refresh or refresh:` when they serve the same logical purpose This approach reduces cognitive load for readers and eliminates misleading code patterns that suggest optional behavior when none exists. --- ## Use appropriate HTTP methods Choose HTTP methods that align with the actual operation being performed. Use GET for retrieving data with query parameters, and POST for operations that modify state or require complex request bodies. When designing API endpoints: - For data retrieval operations, prefer GET requests with query parameters: ```typescript // Better approach for data retrieval const res = await fetchHandler(`${BASE_PATH}/api/check-cname?domain=${domain}`, { headers, method: 'GET' }); ``` - For operations with complex data requirements, use POST requests: ```typescript const res = await fetchHandler(`${BASE_PATH}/api/resource`, { headers, method: 'POST', body: JSON.stringify(payload) }); ``` - Consider supporting both GET and POST methods for the same functionality when beneficial for client compatibility: ```typescript if (req.method === 'GET') { const payload = { ...toForward, project_tier } return retrieveData(name as string, payload) } else if (req.method === 'POST') { const payload = { ...req.body, project_tier } return retrieveData(name as string, payload) } ``` This practice ensures API endpoints follow RESTful conventions, improves API discoverability, and allows for proper caching mechanisms. --- ## explicit null handling Use explicit null and undefined checks with assertions to validate assumptions and maintain type safety. Prefer `!= null` for checking both null and undefined, and use assertions to validate non-null expectations rather than relying on implicit checks. Key patterns to follow: 1. **Use explicit undefined checks in type definitions:** ```ts // Preferred - allows explicitly passing undefined argv?: string[] | undefined; // Avoid - implicit undefined handling argv?: string[]; ``` 2. **Assert non-null assumptions with clear error messages:** ```ts // Preferred - explicit assertion assert(this.#aliveClientsManager !== undefined); // or const manager = must(this.#aliveClientsManager); ``` 3. **Use consistent null comparison patterns:** ```ts // Preferred for checking both null and undefined if (shortID != null) { ... } // Use when specifically checking undefined if (issue?.shortID !== undefined) { ... } ``` 4. **Handle optional values with fallback patterns:** ```ts // Preferred - explicit fallback const permissions = must(this.#permissions).permissions ?? DEFAULT_PERMISSIONS; // Alternative - short-circuit evaluation const result = getZeroData && await getZeroData('rebase', hash); ``` This approach prevents runtime null reference errors, makes null handling intentions explicit, and improves code maintainability by clearly documenting assumptions about null/undefined states. --- ## Prevent hardcoded secrets Never store sensitive information such as API keys, passwords, tokens, or credentials directly in your source code. These hardcoded secrets are easily exposed through version control systems, code sharing, or security breaches, creating significant security vulnerabilities. Instead, use: 1. Environment variables (process.env.SECRET_KEY) 2. Secret management services (AWS Secrets Manager, HashiCorp Vault) 3. Configuration files excluded from version control For client-side applications, consider using server-side proxies to make authenticated requests rather than exposing secrets to the client. Example of unsafe code: ```typescript // UNSAFE: Hardcoded secrets directly in code export const authConfig = { twilio_auth_token: "a9b8c7d6e5f4g3h2i1", aws_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" }; ``` Secure approach: ```typescript // BETTER: Using environment variables export const authConfig = { twilio_auth_token: process.env.TWILIO_AUTH_TOKEN, aws_secret_access_key: process.env.AWS_SECRET_ACCESS_KEY }; // For configuration schemas, use descriptive labels without values export const authFieldLabels = { sms_twilio_auth_token: "Twilio Auth Token", aws_secret_access_key: "AWS Secret Access Key" }; ``` --- ## minimize public API surface Only expose APIs that are truly necessary for external consumers and avoid creating public interfaces that may become maintenance burdens. Before adding public builders, helper utilities, or exposing internal types, critically evaluate whether they provide genuine value to API users. Key principles: - Question the necessity of public builders when objects are unlikely to be constructed by consumers - Avoid exposing internal implementation details (like catalyst package types) as public APIs - Consider whether simple helper utilities are actually needed or if they add unnecessary complexity - Prefer internal implementations with utility methods over public builders when appropriate Example from the codebase: ```java // Instead of exposing a public Builder that may not be used: public interface MergeMetrics { class Builder { /* public builder methods */ } } // Consider internal implementation: // Keep MergeMetrics as interface, add internal case class like LogicalWriteInfo // or make it a proper Java class with utility construction method ``` This approach reduces long-term API maintenance overhead while keeping the public interface clean and focused on genuine user needs. --- ## Abstract user-facing errors Error messages displayed to end users should abstract away implementation details while providing actionable information. Avoid exposing internal system components (like "containerd", "K8S") or raw error messages that might confuse users or reveal security-sensitive information. Instead, map technical errors to user-friendly messages that explain what happened and suggest possible next steps. Always implement proper validation and error handling for data from external sources by using try-catch blocks, and ensure error information is properly displayed regardless of how it's transmitted (URL parameters, hash fragments, etc.). For example, instead of: ```typescript // DON'T do this - exposes implementation details if (failedStatus.message?.startsWith('unable to retrieve container logs for containerd://')) { // show loading state } ``` Do this: ```typescript // Abstract error details with a function function isTransientLogLoadingError(message: string): boolean { // Check for known transient errors without exposing details to UI return message?.includes('retrieve container logs') || message?.includes('logs unavailable'); } try { // Attempt to display logs if (isTransientLogLoadingError(failedStatus.message)) { return ; } else { return ; } } catch (error) { // Fallback for any unexpected errors return ; } ``` --- ## Use configuration over hardcoding Always use configuration constants instead of hardcoding values directly in the code. This improves maintainability and reduces errors when values need to change across different environments or contexts. Key practices: 1. Define configuration values in a centralized location 2. Reference these values using constants or environment variables 3. Make text and numerical values dynamic when they represent configurable limits or settings Example: ```typescript // ❌ Bad const StorageSettings = () => { return (
Maximum size in bytes of a file that can be uploaded is 500 GB
); } // ✅ Good const StorageSettings = () => { return (
Maximum size in bytes of a file that can be uploaded is { formatBytes(STORAGE_FILE_SIZE_LIMIT_MAX_BYTES_UNCAPPED) }
); } ``` --- ## consistent null handling Maintain consistency in null handling patterns across the codebase. When multiple approaches exist for representing absent or disabled values, choose one approach and apply it uniformly throughout the project. Common inconsistencies to avoid: - Mixing sentinel values (like -1) with Optional types - prefer Optional types for better null safety - Mixing null checks with length/isEmpty checks - prefer length/isEmpty for collections and arrays Example of improving consistency: ```java // Instead of mixing approaches: private long numTargetRowsCopied = -1; // sentinel value if (rowBasedChecksums != null) { ... } // null check // Use consistent patterns: private OptionalLong numTargetRowsCopied = OptionalLong.empty(); // Optional type if (rowBasedChecksums.length > 0) { ... } // length check ``` This consistency improves code readability, reduces cognitive load for developers, and prevents bugs that arise from mixing different null handling strategies in the same codebase. --- ## Synchronization safety patterns Ensure proper synchronization mechanisms to prevent deadlocks and race conditions in concurrent code. When designing thread-safe components, carefully consider lock ordering, avoid nested synchronization blocks that could cause deadlocks, and make mutable shared state thread-safe. Key practices: 1. **Avoid nested locks**: Don't call methods that acquire locks while holding another lock, as this can lead to deadlock scenarios 2. **Make shared mutable state thread-safe**: Use proper synchronization for fields accessed by multiple threads 3. **Document concurrency assumptions**: Clearly specify which methods are thread-safe and which require external synchronization 4. **Use appropriate synchronization primitives**: Choose between synchronized blocks, ReentrantLock, or other mechanisms based on the specific use case Example of problematic code: ```java // Dangerous: calling acquisitionLockTimeoutTask.run() within synchronized block synchronized void completeStateTransition(boolean commit) { // ... other operations if (acquisitionLockTimeoutTask.hasExpired()) acquisitionLockTimeoutTask.run(); // Could cause deadlock } // Dangerous: mutable public field accessed by multiple threads public ClientInformation clientInformation; // Not thread-safe ``` Example of safer approach: ```java // Better: avoid nested locks, handle synchronization explicitly synchronized void completeStateTransition(boolean commit) { // ... state transition logic // Schedule task execution outside of synchronized block } // Better: provide controlled access to mutable state private volatile ClientInformation clientInformation; public synchronized void updateClientInformation(ClientInformation info) { // Controlled update with proper synchronization } ``` Always analyze the potential for deadlock when acquiring multiple locks and consider whether operations can be restructured to avoid nested synchronization. --- ## condition-based network synchronization When waiting for network state propagation or distributed system synchronization, avoid using fixed sleep times and instead implement condition-based waiting that verifies the actual desired state. Fixed delays are unreliable in network environments due to variable latency and don't guarantee the expected state has been reached. Instead of using arbitrary sleep durations: ```scala def sleepMillisToPropagateMetadata(durationMs: Long, partition: TopicPartition): Unit = { Thread.sleep(durationMs) // Unreliable - doesn't verify actual state } ``` Implement condition-based waiting that polls for the actual desired network state: ```scala def waitForMetadataPropagation(brokers: Seq[Broker], partition: TopicPartition): Unit = { TestUtils.waitUntilTrue(() => { brokers.forall(broker => getPartitionLeader(broker, partition) == expectedLeader ) }, "Metadata propagation across all brokers") } ``` This approach ensures reliable synchronization by verifying that all network nodes have reached the expected state, rather than assuming a fixed time duration is sufficient for network propagation. --- ## Use condition-based waiting Replace fixed-time delays with condition-based waiting mechanisms to ensure reliable synchronization and avoid timing-dependent race conditions. Fixed delays like `Thread.sleep()` are unreliable because they don't guarantee the expected state change has occurred, leading to flaky tests and potential production issues. Instead of using arbitrary sleep durations, use proper waiting utilities that check for specific conditions: ```scala // Avoid this - timing-based waiting Thread.sleep(1000) TimeUnit.MILLISECONDS.sleep(100) // Prefer this - condition-based waiting TestUtils.waitUntilTrue(() => { brokers.head.metadataCache.getPartitionLeaderEndpoint(partition.topic, partition.partition(), listenerName).isDefined }, "Metadata should be propagated") ``` This approach is more reliable because it waits for the actual condition to be met rather than assuming a fixed time is sufficient. It also makes tests faster when conditions are met quickly and prevents false failures when systems are under load. --- ## maintain naming consistency Ensure consistent naming conventions and parameter ordering throughout the codebase. This includes maintaining consistent parameter order within classes, using established import aliases consistently, and ensuring method names accurately reflect their expected parameters and behavior. Key practices: - Keep parameter ordering consistent within the same class or module (e.g., if other methods use `groupId` as the first parameter, maintain this pattern) - Use established import aliases consistently instead of mixing full type names with aliases - Ensure method names align with their actual parameters and usage context Example of inconsistent parameter ordering: ```scala // Other methods in class use groupId first def fetchOffset(groupId: String, topic: String, partition: Int) // This method breaks the pattern def fetchOffset(topic: String, partition: Int, groupId: String) // Should reorder for consistency ``` Example of inconsistent alias usage: ```scala // Import alias defined import java.util.{Map => JMap} // Inconsistent - mixing full name with available alias def callback(responses: java.util.Map[TopicIdPartition, PartitionResponse]) // Consistent - use the established alias def callback(responses: JMap[TopicIdPartition, PartitionResponse]) ``` This consistency reduces cognitive load for developers and makes the codebase more maintainable. --- ## sequence data state updates When working with distributed data systems, ensure that state updates are performed in the correct conceptual order to maintain data consistency and avoid race conditions. Operations that logically depend on each other should be sequenced appropriately, and components should maintain clear ownership of their data state. For example, when updating watermarks and offsets, perform the logical data update first, then update the watermark: ```scala // Update the last written offset first updateLastWrittenOffset(newOffset) // Then update the high watermark val currentHighWatermark = log.highWatermark if (currentHighWatermark > previousHighWatermark) { onHighWatermarkUpdated.accept(currentHighWatermark) previousHighWatermark = currentHighWatermark } ``` This principle also applies to maintaining consistency between different data views - ensure that components have clear ownership of their state and that cross-component operations respect the logical data flow. Avoid moving functionality between components unless the data ownership boundaries are also properly adjusted. --- ## maintain API backwards compatibility Never modify existing stable API versions in ways that could break backwards compatibility. When adding new functionality, always introduce it in unstable API versions first before stabilizing in new releases. For serialization formats, avoid renaming existing fields - instead use property mapping or add new fields alongside deprecated ones. Example violations: ```json // DON'T: Adding to existing stable version "apis/v1/v1.2/v1.2.0.json": { "duckdb_get_time_ns" // New function in stable version } // DON'T: Renaming serialization fields { "id": 201, "name": "new_field_name" // Breaking change } ``` Correct approach: ```json // DO: Add to unstable first "apis/v1/unstable/new_functions.json": { "duckdb_get_time_ns" // New function in unstable } // DO: Use property mapping for field changes { "id": 201, "name": "projections", "property": "projection_map" // Maps to new internal name } ``` This ensures existing client code continues to work while allowing evolution of the API through proper versioning channels. --- ## Resource cleanup responsibility Ensure proper resource management by clearly defining cleanup responsibilities and implementing robust cleanup patterns. Resources should be closed by their creator, and cleanup must occur even when exceptions happen during resource creation or usage. Key principles: 1. **Creator responsibility**: The method that creates a resource should be responsible for its cleanup 2. **Comprehensive try-catch coverage**: Resource creation should be within try-catch blocks to handle failures 3. **Guaranteed cleanup**: Use finally blocks or try-with-resources to ensure cleanup happens even during exceptions 4. **Separate cleanup logic**: Implement dedicated cleanup methods with their own exception handling Example pattern: ```scala def fromJDBCConnectionFactory(getConnection: Int => Connection): JDBCDatabaseMetadata = { var conn: Connection = null def closeConnection(): Unit = { try { if (null != conn) { conn.close() } } catch { case e: Exception => logWarning("Exception closing connection during metadata fetch", e) } } try { conn = getConnection(-1) // Resource creation in try block // ... use connection } catch { case NonFatal(e) => logWarning("Exception while getting database metadata", e) // Return default values } finally { closeConnection() // Guaranteed cleanup } } ``` This prevents resource leaks and ensures graceful degradation when resource operations fail. --- ## Handle external operations safely Always implement explicit error handling for external operations such as network requests, database queries, and API calls. When errors occur, either: 1. **Degrade functionality gracefully**: Implement a fallback behavior when appropriate. 2. **Propagate errors explicitly**: Use consistent error handling patterns to ensure errors bubble up properly. 3. **Use appropriate error types/codes**: Ensure error responses accurately reflect the nature of the failure. Avoid simply logging errors without proper handling or recovery strategy. **Example - Before:** ```typescript async function fetchDatabaseExtensions() { try { const dbExtensions = await getDatabaseExtensions( { projectRef, connectionString }, undefined, headers ) effectiveAiOptInLevel = checkNetworkExtensionsAndAdjustOptInLevel( dbExtensions, effectiveAiOptInLevel ) } catch (error) { console.error('Failed to fetch database extensions:', error) // No error handling strategy - neither degrading nor propagating } } ``` **Example - After:** ```typescript async function fetchDatabaseExtensions() { try { const dbExtensions = await getDatabaseExtensions( { projectRef, connectionString }, undefined, headers ) effectiveAiOptInLevel = checkNetworkExtensionsAndAdjustOptInLevel( dbExtensions, effectiveAiOptInLevel ) } catch (error) { console.error('Failed to fetch database extensions:', error) // Option 1: Degrade functionality gracefully effectiveAiOptInLevel = 'restricted' // Default to restricted access on error // Option 2: Propagate the error to fail explicitly // throw new Error(`Database extensions check failed: ${error.message}`) } } ``` --- ## Prevent re-render triggers Avoid creating new object/array references in component render functions and carefully manage state updates to prevent unnecessary re-renders. This improves performance by reducing computational overhead and DOM manipulations. Key practices include: 1. Move static arrays/objects outside component functions 2. Use lazy initialization for expensive computations in useState 3. Apply useMemo for computed values that depend on specific props/state 4. Access form field values directly instead of using form.watch() when possible ```jsx // Bad: Creates new references on every render const Component = () => { // New array created every render const defaultPrompts = ['Create a table', 'Write a query']; // Expensive calculation runs on every render const [wrapperTables, setWrapperTables] = useState(formatWrapperTables(wrapper, wrapperMeta)); // Watches entire form state for a single value const { max_bytes_per_second } = form.watch(); return ; } // Good: Prevents unnecessary re-renders // Move constants outside component const DEFAULT_PROMPTS = ['Create a table', 'Write a query']; const Component = () => { // Lazy initialization runs only once const [wrapperTables, setWrapperTables] = useState(() => formatWrapperTables(wrapper, wrapperMeta)); // In form field render functions, use the field value directly return ( { const { value, unit } = convertFromBytes(field.value ?? 0); // Rest of rendering logic }} /> ); } ``` --- ## Document precise security requirements Security documentation must specify exact permission requirements with clear scope and timing details. Vague or outdated security requirements can lead to insufficient ACL configurations, creating security gaps or deployment failures. When documenting security permissions: - Specify the exact operations required (e.g., DESCRIBE, READ, CREATE) - Clearly define the resource scope (e.g., "all topics in the application's topology" vs "topics included in the message") - Include timing context when permissions are needed (e.g., "when first joining") - Keep documentation synchronized with implementation changes Example of imprecise vs precise documentation: ``` // Imprecise - could lead to missing permissions Required for all topics included in the message // Precise - clear scope and timing Required for all topics used in the application's topology, when first joining ``` Regularly review security documentation against actual implementation requirements to ensure accuracy and completeness. --- ## type-safe database operations Implement proper type conversion and validation when working with different database systems to prevent runtime errors and data corruption. This is critical when handling data types that behave differently across database engines. Key practices: 1. **Explicit type casting**: When working with JSON data in PostgreSQL, use explicit casting to avoid driver confusion: ```typescript // Instead of relying on automatic type inference queryArgs: change.queryArgs === undefined ? null : JSON.stringify(change.queryArgs) // Use explicit casting queryArgs: ${change.queryArgs === undefined ? null : JSON.stringify(change.queryArgs)}::text::json ``` 2. **Database-specific type mapping**: Create conversion functions for each database system: ```typescript function toSQLiteType(v: unknown, type: ValueType): unknown { switch (type) { case 'boolean': return v === null ? null : v ? 1 : 0; case 'json': return JSON.stringify(v); default: return v; } } ``` 3. **Preserve data fidelity**: Choose data types that preserve original data structure. For example, use `JSON` instead of `JSONB` in PostgreSQL unless you specifically need JSONB features like indexing, since JSONB reorders fields and doesn't allow NULL bytes. 4. **Validate type assumptions**: Always validate boolean and enum conversions: ```typescript // Correct boolean check isEnum: row.enum.toLowerCase().startsWith('t') // Not: row.enum.toLowerCase().startsWith('f') ``` 5. **Handle timezone conversions carefully**: Use appropriate SQL functions for timestamp conversions: ```sql -- For timestamp with timezone EXTRACT(EPOCH FROM column_name) * 1000 -- For timestamp without timezone EXTRACT(EPOCH FROM column_name::timestamp AT TIME ZONE 'UTC') * 1000 ``` This approach prevents subtle bugs that can occur when data types are handled inconsistently across different database systems and ensures reliable data persistence and retrieval. --- ## Optimize collection conversions When converting between Java and Scala collections or performing set operations, choose methods that minimize temporary collection creation to improve performance and reduce memory overhead. Avoid creating unnecessary intermediate collections by: - Using `diff()` instead of converting to Set and using `--` operator - Leveraging iterators with filtering and mapping before final collection conversion - Using Java streams with collectors when working with Java collections - Choosing direct HashMap operations over collection conversions when appropriate Examples: ```scala // Inefficient - creates temporary Set val result = partitionState.isr.asScala.map(_.toInt).toSet -- outOfSyncReplicaIds // Better - uses diff to avoid temporary collection val result = partitionState.isr.asScala.map(_.toInt).diff(outOfSyncReplicaIds) // Alternative with iterator for complex operations val result = current.isr.asScala.iterator.map(_.toInt).filter(_ != localBrokerId).to(Set) // Java stream approach val result = current.isr.stream().filter(isr => isr != localBrokerId).collect(Collectors.toUnmodifiableSet).asScala ``` This optimization reduces algorithmic complexity by eliminating unnecessary O(n) collection creation steps and improves memory efficiency in data structure operations. --- ## Handle all error paths Ensure comprehensive error handling throughout the codebase by implementing proper error handling blocks, defensive validation, and thorough resource cleanup. Three critical aspects to implement: 1. Use proper error handling blocks (PG_TRY/PG_CATCH) to catch and handle exceptions, while ensuring critical cleanup code runs in all scenarios: ```c PG_TRY(); { before_shmem_exit(cleanup_function, data); // ... operation that might fail ... cancel_before_shmem_exit(cleanup_function, data); } PG_CATCH(); { cancel_before_shmem_exit(cleanup_function, data); // Critical cleanup that must happen regardless of error HOLD_INTERRUPTS(); page_server->disconnect(shard_no); RESUME_INTERRUPTS(); PG_RE_THROW(); } PG_END_TRY(); ``` 2. Add defensive validation with assertions to catch unexpected conditions early, especially in debug builds: ```c if (slot->response->tag == T_NeonErrorResponse) continue; // Add assertion to validate expected state Assert(slot->response->tag == T_NeonGetPageResponse); ``` 3. Always validate inputs thoroughly, generating proper errors instead of allowing potential crashes: ```c if (strncmp(safekeepers_list, "g#", 2) == 0) { errno = 0; *generation = strtoul(safekeepers_list + 2, &endptr, 10); if (errno != 0) { wp_log(FATAL, "failed to parse neon.safekeepers generation number: %m"); } // Validate we have the expected format before proceeding if (*endptr != ':') { wp_log(FATAL, "invalid format in neon.safekeepers, expected 'g#NUMBER:' format"); } return endptr + 1; } ``` Remember that graceful error handling is always preferable to crashes or undefined behavior. Even if certain conditions "should never happen" based on program logic, add appropriate validation and error handling. --- ## Meaningful consistent naming Use descriptive, semantically clear names that follow consistent patterns throughout the codebase. Names should convey purpose and follow established conventions. Key guidelines: 1. **Use descriptive names that reflect purpose** - Avoid ambiguous variable names that can cause confusion - Replace unclear abbreviations with complete words ```go // Bad lockCtx, killWhileRenamingCancel := context.WithTimeout(ctx, onlineDDL.CutOverThreshold) defer killWhileRenamingCancel() // Good lockCtx, lockTimeoutCancel := context.WithTimeout(ctx, onlineDDL.CutOverThreshold) defer lockTimeoutCancel() ``` 2. **Follow established naming patterns** - Use `Is` prefix for boolean getter methods - Follow Pascal case (PascalCase) for struct fields - Use existing patterns within the codebase ```go // Bad func (session *SafeSession) GetTxErrorBlockNextQueries() bool // Good func (session *SafeSession) IsTxErrorBlockNextQueries() bool ``` 3. **Avoid redundancy in names** - Don't prefix struct names with package names - Remove unnecessary qualifiers when context is clear ```go // Bad (in package discovery) type discoveryOptions struct {...} // Good type options struct {...} // Bad type myShardActionInfo interface {...} // Good type shardActionInfo interface {...} ``` 4. **Be consistent with delimiters and formatting** - Use hyphens for command-line flags - Name functions to accurately describe what they do ```go // Bad fs.StringVar(&flagMysqlBindAddress, "mycnf_mysql_bin_address", ...) // Good fs.StringVar(&flagMysqlBindAddress, "mycnf-mysql-bin-address", ...) // Bad - function splits SQL but is called "parse" func parseSQL(querySQL ...string) ([]string, error) // Good - name matches actual behavior func splitSQL(querySQL ...string) ([]string, error) ``` 5. **Prefer clarity over brevity** - Use complete words rather than unclear abbreviations ```go // Bad RetryNb int `json:"-"` // Good RetryCount int `json:"-"` ``` Following these naming conventions improves code readability, reduces cognitive load during reviews, and helps prevent bugs caused by confusion over variable or function purposes. --- ## Minimize unnecessary allocations Avoid allocations and cloning when they don't provide sufficient benefit relative to their performance cost. Balance optimization efforts against code readability, especially in rarely executed paths. Specifically: 1. Return static strings or use direct values instead of string formatting when possible 2. Avoid unnecessary `.clone()` calls on values that are already being moved 3. Prefer owned-type conversions over reference conversions to avoid implicit allocations 4. Use iterator combinators like `itertools::join` instead of collecting into intermediate collections 5. Avoid redundant cloning in asynchronous code paths However, don't sacrifice code readability for minor optimizations that don't meaningfully impact performance: ```rust // Prefer this when the code path is critical and frequently used: pub fn v_str(&self) -> &'static str { match self { PgMajorVersion::PG17 => "v17", PgMajorVersion::PG16 => "v16", PgMajorVersion::PG15 => "v15", PgMajorVersion::PG14 => "v14", } } // But this might be acceptable for rarely executed code where simplicity matters more: pub fn assemble_response(self) -> tonic::Result { // Using a Vec allocation here is fine if it keeps the code simpler // and this is a rare code path with larger I/O costs elsewhere let mut response = page_api::GetPageResponse { page_images: Vec::with_capacity(self.block_shards.len()), // ... other fields }; // ... } ``` Focus your optimization efforts on frequently executed code paths where the performance gains outweigh the cost to code clarity. --- ## Design metrics for insights Design metrics that provide actionable insights while maintaining system efficiency. Follow these key principles: 1. Track success and failure counts instead of just total counts to enable error rate calculations 2. Consider cardinality impact when adding dimension labels 3. Ensure exactly-once metric increments 4. Initialize metrics with meaningful default values Example: ```rust // Good: Track success/failure separately pub static OPERATION_SUCCESS: Counter = register_counter!( "operation_success_total", "Number of successful operations" ); pub static OPERATION_FAILURES: Counter = register_counter!( "operation_failure_total", "Number of failed operations" ); // Bad: High cardinality with too many labels pub static REQUESTS: CounterVec = register_counter_vec!( "requests_total", "Request count", &["endpoint", "user_id", "session_id"] // Too many dimensions ); // Good: Focused dimensions pub static REQUESTS: CounterVec = register_counter_vec!( "requests_total", "Request count", &["endpoint", "status"] // Key dimensions only ); ``` --- ## Connection pooling with pipelining Implement connection pooling with request pipelining for network services to optimize resource usage and improve throughput. Pool should manage three resource levels: 1. Channels (TCP connections): Share between multiple clients with HTTP/2 multiplexing 2. Clients: Acquire from channel pool, return when done 3. Streams: Reuse for multiple requests with controlled queue depth Example implementation: ```rust const CLIENTS_PER_CHANNEL: usize = 16; const STREAM_QUEUE_DEPTH: usize = 2; pub struct ChannelPool { channels: Mutex>, } pub struct ClientPool { channel_pool: Arc, idle: Mutex>, } pub struct StreamPool { client_pool: Arc, streams: Arc>>, } ``` Key principles: - Use HTTP/2 stream multiplexing to share TCP connections - Pipeline multiple requests (queue depth 2-3) to mask network latency - Return resources to pool promptly to enable reuse - Implement proper backpressure through queue depth limits - Consider cleanup of idle resources after timeout period --- ## Ensure algorithm robustness When implementing algorithms, ensure they handle all edge cases correctly and robustly. Code should gracefully manage exceptional conditions rather than producing incorrect results or panicking. Key practices: 1. **Test boundary conditions**: Verify your algorithm works correctly with empty collections, maximum/minimum values, and other edge cases. 2. **Handle hash functions comprehensively**: Include all relevant state in hash functions to prevent unintended collisions. ```rust // Suboptimal: Missing important fields impl Hash for ImportTask { fn hash(&self, state: &mut H) { self.key_range.hash(state); self.path.hash(state); } } // Better: Includes all relevant fields impl Hash for ImportTask { fn hash(&self, state: &mut H) { self.shard_identity.hash(state); // Include this for robustness self.key_range.hash(state); self.path.hash(state); } } ``` 3. **Be cautious with floating-point comparisons**: Don't assume exact equality or that values will sum to expected amounts due to precision errors. ```rust // Problematic: Assumes perfect precision if total_percentage == 100.0 { // ... } // Better: Use appropriate epsilon or handle the last case separately let last_variant = variants.last().unwrap(); if mapped_user_id <= total_percentage / 100.0 { return Some(last_variant.key.clone()); } ``` 4. **Verify logical conditions carefully**: Double-check complex conditionals to ensure they express exactly what you intend. ```rust // Incorrect: Complex condition with a logic error if !(plan_hash == progress.import_plan_hash || progress.jobs != plan.jobs.len()) { anyhow::bail!("Import plan does not match storcon metadata"); } // Correct: Simplified, clear conditions if plan_hash != progress.import_plan_hash { anyhow::bail!("Import plan hash does not match storcon metadata hash"); } if progress.jobs != plan.jobs.len() { anyhow::bail!("Import plan job length does not match storcon metadata"); } ``` 5. **Document known limitations**: When an algorithm doesn't handle certain cases, clearly document these limitations to prevent misuse. --- ## Document API specs completely When designing and implementing APIs, always provide comprehensive specifications that clearly document all endpoints, methods, parameters, and expected responses. This should include: 1. **Method and parameter definitions** - Document each endpoint with its HTTP method, required/optional parameters, and parameter types. Example: ``` PUT: /v1/storage/{tenant}/{timeline}/{endpoint}/{path} Parameters: - tenant: string (required) - The tenant identifier - timeline: string (required) - The timeline identifier - endpoint: string (required) - The endpoint identifier - path: string (required) - The relative file path - data: binary (required) - The file content Response: JSON object with status and operation details ``` 2. **Lifecycle states and transitions** - For resources with complex lifecycles, document the possible states (e.g., Active, Deleted) and the API operations that trigger transitions. 3. **Service integration patterns** - When an API is used for service-to-service communication, document which service is responsible for initiating specific operations and how services should interact. 4. **Error responses and handling** - Include possible error codes, their meanings, and recommended client handling strategies. Clear API specifications facilitate integration, prevent misunderstandings, and serve as essential documentation for both API producers and consumers. --- ## Environment-specific config defaults Define appropriate configuration defaults for different environments (development, testing, production) using dedicated configuration classes with sensible defaults. When test environments require different configurations than production, explicitly create separate config structures rather than hardcoding values throughout tests. When handling external dependencies in test configurations, vendor them rather than downloading at runtime to ensure test reliability and eliminate network dependencies. Example: ```python @dataclass class PageserverTestConfig: # Test-specific defaults that differ from production page_cache_size: int = 16384 max_file_descriptors: int = 500000 @staticmethod def default() -> PageserverTestConfig: return PageserverTestConfig( # Override with test-appropriate values ) # Usage neon_env_builder.pageserver_config_override = f"page_cache_size={config.page_cache_size}; max_file_descriptors={config.max_file_descriptors}" ``` This approach ensures configurations are consistent, properly defaulted, and environment-appropriate without spreading hardcoded values throughout the codebase. --- ## use modern Java syntax Prefer modern Java language features and constructs to write more concise, readable code. Since Apache Spark 4.0.0, the project recommends using contemporary Java syntax where applicable. Key practices: - Use switch expressions (JEP-361) instead of traditional switch statements for cleaner, more compact code - Use Java records for simple data structures instead of classes with boilerplate code - Leverage pattern matching and other modern language features when appropriate Examples: **Switch expressions** - Replace verbose switch statements: ```java // Instead of: BloomFilter result; switch (version) { case 1: result = BloomFilterImpl.readFrom(bin); break; case 2: result = BloomFilterImplV2.readFrom(bin); break; default: throw new IllegalArgumentException("Unknown BloomFilter version: " + version); } return result; // Use: return switch (version) { case 1 -> BloomFilterImpl.readFrom(bin); case 2 -> BloomFilterImplV2.readFrom(bin); default -> throw new IllegalArgumentException("Unknown BloomFilter version: " + version); }; ``` **Records for data structures** - Use records instead of classes for simple data holders: ```java // Instead of a class with boilerplate: public final class JoinColumn implements NamedReference { // constructor, getters, equals, hashCode, toString... } // Use a record: public record JoinColumn(...) implements NamedReference { // automatic constructor, getters, equals, hashCode, toString } ``` This approach reduces boilerplate code, improves readability, and takes advantage of language improvements that make code more maintainable. --- ## Explicit null handling Use explicit patterns when dealing with potentially null or undefined values to prevent runtime errors and improve code clarity: 1. **Mark optional properties with the `?` operator** in TypeScript interfaces when values might be undefined: ```typescript // Bad interface GuideArticleProps { content: string mdxOptions: SerializeOptions } // Good interface GuideArticleProps { content?: string mdxOptions?: SerializeOptions } ``` 2. **Prefer required properties** when a value should always be present. This reduces defensive coding and makes contract expectations clear: ```typescript // Instead of allowing properties to be undefined and checking later // Make them required in the interface/type definition { status: edgeFunctionsStatus?.healthy, isSuccess: edgeFunctionsStatus?.healthy, } ``` 3. **Add explicit null checks** before accessing properties of potentially undefined objects: ```typescript // Bad - may cause errors if currentOrg is undefined enabled: !['team', 'enterprise'].includes(currentOrg?.plan.id ?? ''), // Good - explicitly check existence first enabled: currentOrg !== undefined && !['team', 'enterprise'].includes(currentOrg?.plan.id ?? ''), ``` Consistently applying these practices helps prevent null reference errors and makes code behavior more predictable across the codebase. --- ## Handle network interrupts safely Network code must properly handle interrupts and maintain consistent connection state at all potential interruption points. When implementing network operations: 1. Check for interrupts at all blocking points, including connection attempts, I/O operations, and sleep/delay intervals 2. Reset all connection-specific state when disconnecting due to errors or timeouts 3. Implement robust retry logic that can survive interrupted operations 4. Ensure proper state cleanup even when operations are canceled mid-execution Example of robust network code with proper interrupt handling: ```c /* Ensure proper state cleanup during reconnection attempts */ while (us_since_last_attempt < shard->delay_us) { pg_usleep(shard->delay_us - us_since_last_attempt); /* Handle interrupts during backoff periods */ CHECK_FOR_INTERRUPTS(); now = GetCurrentTimestamp(); us_since_last_attempt = (int64) (now - shard->last_reconnect_time); } /* For network operations, ensure proper sequencing with error handling */ while (!page_server->send(shard_no, &request.hdr) || !page_server->flush(shard_no) || !consume_prefetch_responses()) { /* Loop until all network operations complete successfully */ /* Each step should properly handle connection failures */ } /* When disconnecting, reset all protocol-specific state */ void disconnect(int shard_no) { pageserver_disconnect_shard(shard_no); prefetch_on_ps_disconnect(shard_no); /* Reset protocol state */ } ``` Failing to properly handle interrupts in network code can lead to connection leaks, protocol inconsistencies, and hard-to-debug timeout issues. --- ## Document structure consistency Maintain consistent document structure and formatting in documentation to improve readability and user experience. Follow these key principles: 1. **Use sentence case for headings** - Only capitalize the first word and proper nouns: - ✅ "Rate limits" instead of "Rate Limits" - ✅ "Auth OTP/Magic link request limit" instead of "Auth OTP/Magic Link request limit" 2. **Follow proper heading hierarchy** - Avoid duplicate H1 headings when frontmatter already defines a title. Start content with H2: ```md --- title: 'Replication and Analytics with Supabase' --- ## Introduction Start your content here... ## Features ``` 3. **Organize content logically** - Use clear section names and consider breaking long documents into focused pages: ```md ## Quickstart guide ## Advanced development tips ``` 4. **Use standardized admonition types** - Stick to approved types: `note`, `tip`, `caution`, `deprecation`, `danger`: ```md Highlight important information or warnings using appropriate admonition types ``` 5. **Present step-by-step instructions clearly** - Structure multi-step processes with clear ordering and grouping: ```md ## Configuration ### Step 1: Set up credentials 1. First do this... 2. Then do that... ### Step 2: Configure settings ``` Following these standards ensures documentation remains consistent, accessible, and professional across the project. --- ## prefer settings over pragmas When implementing configuration options, prefer database settings over pragma functions to maintain consistency and better user experience. Settings provide a unified interface that's easier to discover, document, and manage compared to scattered pragma functions. Instead of creating pragma functions like: ```cpp set.AddFunction(PragmaFunction::PragmaStatement("enable_logging", PragmaEnableLogging)); ``` Use database settings: ```cpp // In settings registration void EnableLoggingSetting::SetGlobal(DatabaseInstance *db, DBConfig &config, const Value &input) { config.options.enable_logging = input.GetValue(); } ``` This approach consolidates configuration management, eliminates the need for environment variables scattered throughout the codebase, and provides a consistent `SET variable=value` interface. When multiple configuration sources exist (environment variables, pragma functions, attach options), refactor them into a unified settings system to reduce complexity and potential conflicts. Consider using table functions (`CALL enable_logging(...)`) for operations that require parameters or complex logic, but keep simple boolean/value configurations as settings. --- ## Avoid code duplication Maintain clean, maintainable code by avoiding duplication and following proper code organization principles: 1. Place utility functions in dedicated utility files rather than defining them inline within component files 2. Reuse existing constants, functions, and utilities through imports rather than duplicating code 3. Maintain a single source of truth for shared logic and data **Example - Before:** ```typescript // In components/SomeFeature.ts const getFileName = (path: string): string => { if (!path) return '' const cleanPath = path.split('?')[0].split('#')[0] const segments = cleanPath.split('/') return segments[segments.length - 1] || '' } // In another file export const CANCELLATION_REASONS = [ 'Pricing', "My project isn't getting traction", // ... more reasons ] ``` **Example - After:** ```typescript // In utils/fileUtils.ts export const getFileName = (path: string): string => { if (!path) return '' const cleanPath = path.split('?')[0].split('#')[0] const segments = cleanPath.split('/') return segments[segments.length - 1] || '' } // In components/SomeFeature.ts import { getFileName } from 'utils/fileUtils' // In various files that need cancellation reasons import { CANCELLATION_REASONS } from 'components/interfaces/Billing/Billing.constants' ``` This approach improves maintainability, reduces bugs from inconsistent implementations, and makes code easier to test and refactor. --- ## Use descriptive identifiers Choose clear, meaningful names for variables, parameters, and constants that convey their purpose without requiring additional documentation or context. Key practices: 1. **Use semantic names instead of indices**: Replace cryptic array indices with descriptive variable names. ```python # Instead of this: assert prewarm_info[0] > 0 and prewarm_info[1] > 0 # Do this: total, prewarmed, skipped = prewarm_info assert total > 0 and prewarmed > 0 ``` 2. **Apply consistent prefixes** for related parameters to indicate their domain: ```python # Instead of this: disable_kick_secondary_downloads: bool = False # Do this: storcon_kick_secondary_downloads: bool = True ``` 3. **Use standard naming conventions** like uppercase for constants: ```python # Instead of this: prewarm_label = "compute_ctl_lfc_prewarm_requests_total" # Do this: PREWARM_LABEL = "compute_ctl_lfc_prewarm_requests_total" ``` 4. **Avoid negatively named booleans** and consider enums for complex options: ```python # Instead of this: def test_lfc_prewarm(neon_simple_env: NeonEnv, with_compute_ctl: bool): """with_compute_ctl: Test compute ctl's methods instead of querying Postgres directly""" # Do this: class LfcQueryMethod(Enum): COMPUTE_CTL = "compute_ctl" POSTGRES = "postgres" def test_lfc_prewarm(neon_simple_env: NeonEnv, query: LfcQueryMethod): ``` Clear naming reduces cognitive load, minimizes the need for comments, and makes code more maintainable and easier to understand at first glance. --- ## dependency management practices Ensure comprehensive dependency management in package.json files by following these practices: 1. **Evaluate necessity**: Before adding dependencies, check if native Node.js APIs provide the functionality. For example, use `fs.mkdtemp(path.join(os.tmpdir(), 'prefix'))` instead of adding a temporary file library. 2. **Choose appropriate dependency types**: Carefully classify dependencies as `dependencies`, `devDependencies`, or `peerDependencies` based on their usage. When uncertain about peer dependencies in monorepos, start with `devDependencies` and adjust as needed. 3. **Use consistent versioning**: Establish a clear strategy for version specifications - either pin exact versions (`"esbuild": "0.25.0"`) or use ranges (`"esbuild": "^0.25.0"`) consistently across the project. 4. **Update all related files**: When modifying package.json, ensure package-lock.json is also updated and included in commits. Example of proper dependency evaluation: ```js // Instead of adding a dependency like 'tmp' "dependencies": { "tmp": "^0.2.3" } // Use native Node.js APIs fs.mkdtemp(path.join(os.tmpdir(), 'myapp-')); ``` This approach reduces dependency bloat, minimizes security vulnerabilities, and maintains cleaner, more maintainable projects. --- ## Add proactive null checks Always add null checks before accessing methods or properties on objects that can potentially be null, especially when dealing with Java APIs or exceptional conditions like RejectedExecutionException. When working with Java collections that return null (unlike Scala collections that return Option), use appropriate null safety patterns: ```scala // Before accessing methods on potentially null objects if (task != null && !task.isDone) { // safe to call methods on task } // When working with Java Map.get() which can return null val response = responses.get(partition) assertNotNull(response) result.fire(response) // Alternative approach using Optional val response = Optional.ofNullable(responses.get(partition)) assertTrue(response.isPresent) result.fire(response.get) ``` This prevents NullPointerException at runtime and makes the code more robust, particularly when integrating Scala code with Java APIs or handling exceptional conditions where objects might not be properly initialized. --- ## Clear consistent identifier names Choose clear, consistent, and non-redundant names for identifiers across the codebase. Follow these guidelines: 1. Use specific, descriptive names that clearly indicate purpose: ```rust // Bad type SegmentSize = u32; // Good type WalSegmentSize = u32; // Clearly indicates this is for WAL segments ``` 2. Maintain consistent naming patterns throughout the codebase: ```rust // Bad: Mixing naming patterns struct PostHogLocalEvaluationFlag {} struct LocalEvaluationResponse {} // Good: Consistent naming struct LocalEvaluationFlag {} struct LocalEvaluationResponse {} ``` 3. Avoid redundant prefixes when context is clear: ```rust // Bad: Redundant context pub http_client: reqwest::Client // In HTTP-specific module // Good: Clean, contextual naming pub client: reqwest::Client ``` 4. Use clear, specific variable names in method parameters: ```rust // Bad: Ambiguous naming async fn from_request_parts(_: &mut Parts, state: &Arc) // Good: Clear purpose async fn from_request_parts(_: &mut Parts, compute: &Arc) ``` This standard helps maintain code readability and reduces confusion during code review and maintenance. --- ## Manage output streams carefully Always consider the destination and lifecycle of output streams to prevent protocol interference, data loss, and unexpected behavior. Choose stdout for application output and stderr for debugging/logging. Avoid accidentally closing streams and be mindful of output pollution in protocol-sensitive contexts. Key practices: - Use stderr for debug/log output to avoid interfering with application protocols - When piping streams, use `{ end: false }` option to prevent closing the target stream - In protocol-sensitive contexts (like JSON-RPC), redirect console output to stderr to prevent stdout pollution - Prefer `process.stderr.write` over `console.warn` in tests to avoid mock interference - Use `console.error.bind(console)` instead of custom implementations to preserve formatting Example: ```typescript // Good: Prevent stdout closure when piping fileStream.pipe(process.stdout, { end: false }) // Good: Redirect console in protocol contexts if (process.argv.includes('mcp')) { console.log = console.error.bind(console) } // Good: Use stderr for debug output console.warn(`${namespace} ${format}`, ...rest) ``` --- ## Pin GitHub action versions Always pin GitHub Actions to specific commit hashes instead of using major/minor version tags (like @v4). This ensures reproducible builds and prevents supply chain attacks through compromised action versions. Example: ```yaml # Don't do this: - uses: actions/checkout@v4 - uses: actions/cache@v4 # Do this instead: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3 ``` Include the version number in a comment after the commit hash for better readability and version tracking. This practice should be applied consistently across all workflow files. Consider implementing a linter to enforce this standard automatically. --- ## Protect sensitive API keys Never expose keys with elevated privileges (such as `service_role` or secret keys) in client-side code. These keys can bypass Row Level Security (RLS) and provide unrestricted access to your data. Always use elevated privilege keys only in server-side environments like: - Backend servers - Edge Functions - Secure CI/CD pipelines For client-side applications, only use the publishable or `anon` key, and ensure Row Level Security is properly configured on all tables to protect your data. ```typescript // WRONG - Never do this in client-side code const supabase = createClient( 'https://your-project.supabase.co', 'SUPABASE_SERVICE_ROLE_KEY' // Dangerous! ) // CORRECT - Server-side code only (e.g., Edge Function) const supabaseAdmin = createClient( Deno.env.get('SUPABASE_URL') ?? '', Deno.env.get('SUPABASE_SERVICE_ROLE_KEY') ?? '' // Secure, from environment variables ) // CORRECT - Client-side code const supabase = createClient( 'https://your-project.supabase.co', 'SUPABASE_ANON_KEY' // Safe with proper RLS policies ) ``` Additionally, ensure sensitive files containing keys (like `.env` files or `.sentryclirc`) are added to `.gitignore` to prevent accidental exposure through version control. --- ## Avoid flaky tests Tests should be designed to be deterministic and reliable to prevent wasted developer time and false confidence. Two common causes of flakiness to avoid: 1. **Time-based waits**: Never rely on `sleep()` calls for synchronization. Instead, use condition-based waits: ```python # Bad practice sleep(offload_secs + 1) # May fail if processing takes longer # Good practice wait_until(lambda: check_if_lfc_content_appears_in_remote_storage(), timeout_seconds=10, message="LFC content did not appear in remote storage") ``` 2. **Shared testing resources**: Use isolated resources for tests instead of shared ones: ```python # Potentially flaky - using default database def test_something(neon_simple_env: NeonEnv): conn = endpoint.connect() # Connects to default 'postgres' database # More reliable - using dedicated test database def test_something(neon_simple_env: NeonEnv): conn = endpoint.connect(dbname=f"test_db_{uuid.uuid4()}") ``` Also, follow proper pytest patterns like using `@pytest.mark.skipif` decorators instead of early returns to clearly indicate test conditions and improve maintainability. --- ## Optimize data structures When implementing algorithms, prioritize data structure choices that minimize resource usage while maintaining functionality. Consider if a simpler approach with counters, efficient state tracking, or built-in functions can replace complex custom implementations. For hierarchical or nested data: 1. Track levels with counters instead of creating redundant objects 2. Store relationship information in fewer objects to reduce allocation costs 3. Use appropriate memory contexts for your allocation pattern For example, instead of creating one structure per level like this: ```c /* Creates one structure per level - O(n) space complexity */ while (SubtransDdlLevel != 0) { DdlHashTable *new_table = MemoryContextAlloc(TopTransactionContext, sizeof(DdlHashTable)); new_table->prev_table = CurrentDdlTable; CurrentDdlTable = new_table; SubtransDdlLevel -= 1; } ``` Consider storing level information in the structure itself: ```c /* Creates structures only when needed - O(1) space complexity for most cases */ if (need_new_table) { DdlHashTable *new_table = MemoryContextAlloc(TopTransactionContext, sizeof(DdlHashTable)); new_table->prev_table = CurrentDdlTable; new_table->subtrans_level = SubtransDdlLevel - 1; CurrentDdlTable = new_table; } ``` Similarly, for string parsing operations, prefer standard library functions over custom implementations: ```c /* More efficient and clearer than manual parsing */ int pos; if (sscanf(safekeepers_list, "g#%u:%n", generation, &pos) == 1) { return safekeepers_list + pos; } else { return safekeepers_list; } ``` --- ## comprehensive test assertions Write test assertions that are both comprehensive and maintainable. Ensure all relevant fields and behaviors are validated, while using generic matchers where appropriate to reduce brittleness. **Comprehensive Coverage**: Assert all relevant fields in response objects, not just error codes. For example, when testing fetch responses, verify highWatermark, logStartOffset, and other expected fields: ```scala // Instead of only checking error assertEquals(Errors.NONE, responses(topicIdPartition).error) // Also verify other relevant fields assertEquals(Errors.NONE, responses(topicIdPartition).error) assertEquals(highWatermark, responses(topicIdPartition).highWatermark) assertEquals(leaderLogStartOffset, responses(topicIdPartition).logStartOffset) ``` **Maintainable Assertions**: Use generic matchers like `anyLong()` instead of hardcoded values when the specific value isn't critical to the test logic: ```scala // Instead of brittle specific values verify(coordinator, times(0)).updateLastWrittenOffset(0) verify(coordinator, times(1)).updateLastWrittenOffset(2) verify(coordinator, times(1)).updateLastWrittenOffset(5) // Use generic matchers when appropriate verify(coordinator, times(0)).updateLastWrittenOffset(anyLong()) verify(coordinator, times(3)).updateLastWrittenOffset(anyLong()) ``` This approach ensures tests validate the complete behavior while remaining resilient to implementation changes that don't affect the core functionality being tested. --- ## Extract for clarity Extract complex or reused logic into focused, well-named methods with single responsibilities. This improves code readability, testability, and maintainability while reducing duplication. When refactoring: 1. Identify blocks of code that perform a distinct task or are used in multiple places 2. Extract them into appropriately named methods 3. Keep the scope of annotations like `@SuppressForbidden` minimal by extracting annotated code into dedicated helper methods 4. Consider creating separate classes for related logic sets For example, instead of placing complex logic inline: ```java @SuppressForbidden( reason = "TODO Deprecate any lenient usage of Boolean#parseBoolean https://github.com/elastic/elasticsearch/issues/128993" ) public static boolean parseBoolean(String value) { // complex implementation return Boolean.parseBoolean(value); } ``` Extract it to minimize annotation scope: ```java public static boolean parseBoolean(String value) { return parseBooleanInternal(value); } @SuppressForbidden( reason = "TODO Deprecate any lenient usage of Boolean#parseBoolean https://github.com/elastic/elasticsearch/issues/128993" ) private static boolean parseBooleanInternal(String value) { return Boolean.parseBoolean(value); } ``` For complex class functionality, consider isolating related methods: ```java // Comment from Discussion 6: // "A logistical suggestion would be to isolate all the InlineJoin logic into its own class // (could be nested), as there are a few methods here (plus this record) // that are dedicated just for this feature." ``` This approach makes code more maintainable, easier to understand, and simplifies future changes. --- ## Name reflects meaning Choose names that clearly communicate the intent, behavior, and semantics of code elements. Names should be accurate, consistent, and follow established patterns: 1. **Method names must reflect what they operate on**: If a method manipulates indices, name it accordingly (e.g., `moveIndicesToPreviouslyFailedStep` rather than `moveClusterStateToPreviouslyFailedStep`). Method names should express what they do, not how they do it. 2. **Variable names should precisely describe their content**: Use terms that clearly convey meaning, especially for metrics or measurements: ```java // Unclear - doesn't specify these are averages int percentWriteThreadPoolUtilization; long maxTaskTimeInWriteQueueMillis; // Clear - precisely describes the metrics int averageWriteThreadPoolUtilization; long averageWriteThreadPoolQueueLatency; ``` 3. **Use consistent naming patterns**: Follow established patterns within the codebase and respect external conventions: - For third-party products, match their official capitalization (e.g., "IBM watsonx" not "IBM WatsonX") - For test methods, follow JUnit conventions (begin with "test" using camelCase) - For task queues, ensure names match their purpose (e.g., "update-data-stream-mappings" for mappings-related tasks) 4. **Prefer constants over string literals**: Extract repeated or semantically meaningful strings into named constants: ```java // Hard to maintain - string literals scattered throughout code if (configs.group.equals("unsupported") || configs.group.equals("union-types")) { // Better - using named constants private static final String GROUP_UNSUPPORTED = "unsupported"; private static final String GROUP_UNION_TYPES = "union-types"; if (configs.group.equals(GROUP_UNSUPPORTED) || configs.group.equals(GROUP_UNION_TYPES)) { ``` --- ## Prevent redundant operations In distributed database systems, prevent redundant operations that can overload cluster resources. When implementing update operations that might be triggered simultaneously from multiple nodes: 1. Add conditional checks to avoid redundant processing when the state hasn't changed: ```java // Before applying mapping updates, check if they're identical to existing mappings if (existingMapper != null && sourceToMerge.equals(existingMapper.mappingSource())) { context.resetForNoopMappingUpdateRetry(initialMappingVersion); return true; } ``` 2. Consider sequencing operations when high concurrency is expected, especially for resource-intensive updates: - Execute updates sequentially rather than allowing parallel execution - For critical operations, consider coordinating through a single node (like the master node) - Use appropriate locking or versioning mechanisms to prevent conflicting updates 3. Use direct access methods to avoid constructing unnecessary intermediate objects: - Access metadata directly with methods like `getProjectMetadata()` instead of constructing full state objects - Preserve important properties like query boost and name parameters when transforming queries These optimizations are particularly important when multiple data nodes might be issuing similar updates (like dynamic mappings) or when working near capacity limits of the system. --- ## Clarity over uncertainty Technical documentation should use precise language that clearly differentiates between product behavior and user configuration options. Frame explanations from the user's perspective, emphasizing capabilities rather than limitations. When describing configurable functionality: - Replace phrases that suggest uncertainty (like "this is not guaranteed") with empowering language ("you may choose to adjust this behavior"). - Use specific examples to illustrate options rather than vague statements. - Structure information as actionable items when possible. For performance-related documentation: - Be explicit about cause-effect relationships. - Use bullet points for clarity when listing multiple scenarios. - Consider how language around limitations will be perceived by users. Example improvement: ```diff - Force merging will be performed by the nodes within the current phase of the index. - A forcemerge in the `hot` phase will use hot nodes with potentially faster nodes, - while impacting ingestion more. A forcemerge in the `warm` phase will use warm - nodes and potentially take longer to perform, but without impacting ingestion in - the `hot` tier. + Force merging will be performed by the node hosting the shard. Usually, the node's + role matches the data tier of the ILM phase that the index is in. For example: + * A force merge in the `hot` phase will use hot nodes. Merges may be faster on this + potentially higher performance hardware but may impact ingestion. + * A force merge in the `warm` phase will use warm nodes. Merges may take longer to + perform on potentially lower performance hardware but will avoid impacting + ingestion in the `hot` tier. ``` Remember that marketing/positioning around limitations is particularly sensitive - always emphasize what users can do rather than what they cannot do. --- ## Configure type serialization When working with databases that exchange data with other systems, ensure proper serialization and deserialization of data types like UUIDs, dates, or complex objects. System-specific representation differences can lead to data corruption, query failures, or incorrect results. For example, when working with MongoDB and UUIDs: ``` # Specify UUID representation in connection string mongodb+srv://username:password@cluster.mongodb.net/mydb?uuidRepresentation=standard # For legacy UUID representations, use appropriate parameters: # - C#: uuidRepresentation=csharpLegacy # - Java: uuidRepresentation=javaLegacy # - Python: uuidRepresentation=pythonLegacy ``` When joining data across different sources, ensure join keys have compatible data types: - Integer types (`byte`, `short`, `integer`) can typically be joined - Float types (`half_float`, `float`, `scaled_float`, `double`) are often interchangeable - Date types often require exact matching or explicit conversion - Text/string types may require specific formatting or normalization Always document data type compatibility requirements in your schema design and provide conversion functions where needed to ensure consistent data handling across systems. --- ## Maintain network controls Create maintainable and clearly documented network access controls. Instead of hardcoding network restrictions, use dedicated tables that can be updated without code changes. Always document networking concepts with plain language explanations for developers unfamiliar with networking terminology. For IP filtering: ```sql -- Create a dedicated table for IP blocklist CREATE TABLE private.auth_ip_blocklist ( blocked_range cidr, reason text, created_at timestamp DEFAULT now() ); -- Use the table in your filtering logic CREATE OR REPLACE FUNCTION public.check_ip_access(client_ip inet) RETURNS boolean AS $$ BEGIN RETURN NOT EXISTS ( SELECT 1 FROM private.auth_ip_blocklist WHERE client_ip <<= blocked_range ); END; $$ LANGUAGE plpgsql; ``` When implementing WebSocket authorization, clearly indicate when private channels are required: ```ts // Set up a private channel listener with proper authorization const channel = supabase.channel('topic:' + recordId, { config: { broadcast: { self: true }, private: true // Required for realtime.broadcast_changes } }) ``` Always include explanations of networking terms in documentation or code comments, especially for concepts like CIDR notation, WebSocket protocols, or authentication mechanisms. --- ## Design for evolution When designing APIs, prioritize flexibility and independent evolution of components. Avoid tightly coupling related services or wrapping existing components when they might evolve differently over time. Instead, design APIs with extension points and loose coupling to accommodate future changes. For example, when designing request structures that may evolve differently between local and remote contexts: ```java // Avoid tight coupling with wrapping public class RemoteClusterStateRequest extends ActionRequest { // Duplicating fields to allow independent evolution private final String[] indices; private final boolean local; // other fields... // Instead of: // private final ClusterStateRequest clusterStateRequest; } ``` For extensible APIs, consider using collections or generic structures that can be extended: ```json // More extensible API design "hybrid": { "fields": ["content", "content.semantic"], "query": "foo", "rank_modifiers": [ {"rule": { ... }}, {"rerank": { "inference_id": "my-reranker-service", "field": "content" }} ] } // Rather than fixed structures that are harder to extend: "hybrid": { "fields": ["content", "content.semantic"], "query": "foo", "rule": { ... }, "rerank": { "inference_id": "my-reranker-service", "field": "content" } } ``` This approach makes APIs more resilient to changing requirements and easier to extend without breaking compatibility. --- ## Specify explicit REST formats Always specify explicit request formats in REST API tests rather than relying on default behaviors. This includes: 1. Set appropriate Content-Type headers (typically application/json for REST APIs) 2. Define explicit query parameters that clearly test intended functionality Without proper format specifications, tests may fail unexpectedly or test incorrect behaviors. For example, missing Content-Type headers might cause an endpoint to default to different formats like cbor instead of JSON: ```yaml - do: headers: Content-Type: application/json # Explicitly specify format index: index: test id: 1 body: { "field": "value" } ``` Similarly, when testing query parameters, be explicit about the expected behavior rather than using generic queries that might not fully test the intended functionality: ```yaml retriever: standard: query: match_none: {} # Explicitly testing empty results case ``` This practice improves test reliability and clarity by making the expected request format visible and intentional rather than implicit. --- ## Measure before optimizing performance Before implementing performance optimizations, measure and validate the impact through benchmarks. This applies especially to: 1. Changes affecting common operations (string conversions, collections) 2. Modifications to critical paths (search, indexing) 3. Updates to shared resources or services Example benchmark approach: ```java @Fork(value = 1) @Warmup(iterations = 5) @Measurement(iterations = 5) @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.NANOSECONDS) public class PerformanceBenchmark { @Param({"10", "100", "1000"}) private int size; @Benchmark public void benchmarkOperation() { // Test both existing and new implementation // Compare results across different input sizes } } ``` Key practices: - Test with varying input sizes to understand scaling characteristics - Measure both average case and edge cases - Compare before/after metrics to quantify improvements - Consider system-wide impact, not just local optimizations This approach helps prevent premature optimization and ensures changes provide meaningful performance benefits. --- ## Use configuration access methods When accessing configuration settings, always use the appropriate type-safe accessor methods provided by the configuration framework rather than manual string parsing. This improves code readability, maintainability, and reduces potential bugs from string parsing errors. For boolean settings: - Use `getAsBoolean()` with explicit default values instead of `Boolean.parseBoolean()` - For system properties, use `Booleans.parseBoolean()` with default value For example, instead of: ```java boolean sourceOnly = Boolean.parseBoolean(indexSettings.getSettings().get("index.source_only")); boolean isFrozen = Boolean.parseBoolean(writeIndex.getSettings().get("index.frozen")); boolean enabledDebugLogs = Boolean.parseBoolean(System.getProperty(ENABLE_KERBEROS_DEBUG_LOGS_KEY)); ``` Prefer: ```java boolean sourceOnly = indexSettings.getAsBoolean("index.source_only", false); boolean isFrozen = writeIndex.getSettings().getAsBoolean("index.frozen", false); boolean enabledDebugLogs = Booleans.parseBoolean(System.getProperty(ENABLE_KERBEROS_DEBUG_LOGS_KEY), false); ``` Similarly, when exposing configuration parameters in APIs, avoid implicit defaults and require explicit parameters. When checking compatibility or validating configurations, use clear and readable expressions that make the intent obvious. This practice promotes consistency, makes default values explicit, and leverages the built-in validation provided by the configuration framework. --- ## Keep files focused small Maintain code organization by keeping files focused on a single responsibility and splitting large files into smaller, well-organized modules. This improves code navigation, readability, and maintainability. Key guidelines: 1. Move code to dedicated files when it represents a distinct functionality 2. Split large files (>500 lines) into logical modules 3. Place related functionality in the same module 4. Use clear, descriptive file names that reflect the content Example of good organization: ```rust // Instead of putting everything in mod.rs: // src/binary/mod.rs mod local_proxy; // Move to separate file pub use local_proxy::*; // src/binary/local_proxy.rs pub struct LocalProxy { // Implementation details } // Instead of one large compute.rs: // src/compute/mod.rs mod core; // Core compute functionality mod prewarm; // Prewarm-specific implementations mod helpers; // Helper functions pub use core::ComputeNode; ``` This approach: - Makes code easier to navigate and maintain - Reduces cognitive load when working with the codebase - Facilitates code review and testing - Improves module cohesion --- ## Hierarchical semantic naming Use hierarchical prefixes and clear descriptive names to indicate the domain, source, and purpose of code elements. This improves code organization, searchability, and helps maintain consistency across the codebase. For metrics: - Start with the component name (e.g., `compute_`) - Add the data source if applicable (e.g., `pg_` for PostgreSQL data) - Finish with a semantically accurate description (e.g., `oldest_frozen_xid_age`) For API methods: - Follow established style guides (e.g., Google's recommendation to use verbs in method names) - Use specific, descriptive names rather than generic ones (e.g., `GetDbSize` instead of `DbSize`) - Group related fields and use consistent prefixing (e.g., `request_id`, `request_class`) Example: ``` // Metrics: compute_pg_oldest_frozen_xid_age // Not: frozen_xid_age or compute_min_mxid_age // API Methods: rpc GetDbSize (DbSizeRequest) returns (DbSizeResponse); // Not: DbSize // Message fields: message GetPageRequest { uint64 request_id = 1; // Consistent prefixing for request metadata RequestLsn read_lsn = 2; // Specific, descriptive name vs. "common" } ``` This avoids ambiguity, improves code readability, and makes the codebase more maintainable. --- ## Proactive cache warming Implement proactive cache warming strategies to minimize performance degradation after system restarts or during cold starts. Rather than waiting for normal workload patterns to gradually populate caches, actively prefetch and load likely-needed data into memory ahead of time. For optimal performance: 1. Identify critical cache contents that impact application performance (like LFC state in compute nodes) 2. Implement mechanisms to periodically persist cache state with appropriate frequency (balancing cost vs performance) 3. Add functionality to load this state during startup or in background threads 4. Consider cost-performance tradeoffs when designing the cache dump frequency ```rust struct ComputeSpec { // Whether to do auto-prewarm at start or not pub lfc_auto_prewarm: bool, // Interval in seconds between automatic dumps of LFC state // Default to 300s (5 min) for reasonable cost/performance balance pub lfc_dump_interval_sec: Option } // When implementing prewarming: // 1. Store raw metrics for accurate measurement struct PrewarmState { pub status: PrewarmStatus, // Use absolute values instead of percentages pub pages_total: usize, pub pages_processed: usize, pub error: Option } ``` By proactively warming caches, you can reduce the performance impact of restarts and provide more consistent user experience, especially for large workloads where cold cache performance can be significantly degraded. --- ## Ensure concurrent resource cleanup Always ensure proper cleanup of concurrent resources like semaphores, transactions, and async operations, even when exceptions occur. Use finally blocks to guarantee resource release and Promise.allSettled instead of Promise.all when handling multiple concurrent operations that might fail. Key patterns to follow: - Wrap resource acquisition/release in try-finally blocks to prevent resource leaks - Use Promise.allSettled for concurrent operations where some failures are acceptable - Implement timeouts for operations that might hang indefinitely - Consider using AbortController for proper termination of long-running concurrent operations Example: ```typescript // Good: Guaranteed cleanup with finally const semaphore = new Semaphore(maxWorkers) await semaphore.acquire() const pendingJob = (async () => { try { // do work } finally { semaphore.release() // Always released, even on exception } })() // Good: Handle failures gracefully in concurrent operations await Promise.allSettled([...transactions.values()].map(tx => closeTransaction(tx))) // Good: Prevent hanging with timeout const timeout = setTimeout(() => resolve(''), 1000) ``` This prevents deadlocks, resource exhaustion, and hanging operations that can severely impact application reliability in concurrent scenarios. --- ## Adaptive cache expiration strategy Design cache expiration policies that align with actual workload patterns rather than arbitrary timeframes. For systems with varying access patterns (e.g., daily vs. monthly workloads), implement configurable TTL settings to prevent premature cache invalidation. ```rust struct CacheConfig { // Allow users to configure the cache TTL based on their workload patterns /// TTL in seconds for cache persistence. Default value balances /// memory usage with performance for typical workloads. /// Set to `None` for permanent persistence until explicit deletion. pub cache_ttl_sec: Option, /// Whether to perform automatic prewarming on restart pub auto_prewarm: bool, /// Custom prewarming strategy for different workload types pub prewarm_strategy: PrewarmStrategy, } enum PrewarmStrategy { /// Minimal prewarming for frequent access patterns Basic, /// Medium prewarming for daily/weekly workloads Standard, /// Full prewarming for monthly/infrequent workloads Complete, } ``` When implementing caching mechanisms, consider adding workload-aware expiration policies rather than fixed TTLs. For systems with infrequent but performance-critical operations (like monthly reporting), overly aggressive cache expiration can cause significant performance degradation. Cache policies should balance automatic management with user configuration to accommodate both typical use cases and edge cases. Consider adding monitoring to track actual cache hit rates and auto-adjust TTLs based on observed workload patterns. --- ## Database before memory When working with database systems that also maintain in-memory state, always update the persistent database state before updating the in-memory representation. This ensures that if a crash occurs between operations, the system can correctly rebuild its state from the database without inconsistencies or "time traveling" issues. For example: ```rust // Good practice self.persistence .set_tombstone(node_id) .await?; // Now update in-memory state let mut locked = self.inner.write().unwrap(); let (nodes, _, scheduler) = locked.parts_mut(); scheduler.node_remove(node_id); // ...rest of in-memory updates // Bad practice - DON'T DO THIS let mut locked = self.inner.write().unwrap(); let (nodes, _, scheduler) = locked.parts_mut(); scheduler.node_remove(node_id); // If we crash here, database is out of sync with what was in memory self.persistence .set_tombstone(node_id) .await?; ``` Additionally, be careful not to schedule redundant database operations. When deleting database objects, ensure you're not accidentally scheduling the same deletion twice through different mechanisms. Always handle edge cases where automatic cleanup might not occur, such as empty collections that won't be iterated over by reconciliation processes. --- ## Feature flag implementation clarity When implementing feature flags in the system, clearly document both the evaluation strategy and its performance implications. For HTTP evaluation vs. local evaluation, include expected request volumes and latency impacts. Always provide configuration examples that demonstrate proper usage. For example, when implementing a feature flag system: ``` // In your configuration file (e.g., local_proxy.json) { "feature_flags": { "evaluation_mode": "local", // Options: "local", "http" "refresh_interval_sec": 60, "flags": { "gc-compaction": { "default": "disabled", "variants": ["disabled", "stage-1", "stage-2", "fully-enabled"], "filters": { "plan_type": ["scale", "enterprise"], "min_resident_size_gb": 10, "max_resident_size_gb": 100 } } } } } ``` Consider performance optimizations when possible. For example, you could map percentage rollouts to tenant ID patterns (e.g., enabling for tenant IDs starting with 0x00 for 1/16th rollout) to allow for easier monitoring via filtering in observability tools. Document the trade-offs of your chosen approach. For HTTP evaluation, consider the request volume (e.g., "754X requests per month" as noted in one example). For local evaluation, document how often configuration is refreshed and how user properties are evaluated. --- ## Enforce least privilege Always assign the minimum permissions necessary for functionality when implementing role-based access controls. This fundamental security principle reduces the potential attack surface and minimizes the impact of compromised accounts. Key practices: 1. Start with read-only permissions by default and add specific write permissions only where justified 2. Question and verify any broad write access permissions during code reviews 3. Separate read and write privileges in role definitions to make access patterns explicit 4. Regularly audit existing role permissions against actual usage needs Example: ```java // Good: Explicit, minimal permissions RoleDescriptor.IndicesPrivileges.builder() .indices(ReservedRolesStore.ENTITY_STORE_V1_LATEST_INDEX) .privileges("read", "view_index_metadata") .build() // Avoid: Unnecessarily broad permissions RoleDescriptor.IndicesPrivileges.builder() .indices(ReservedRolesStore.ENTITY_STORE_V1_LATEST_INDEX) .privileges("read", "view_index_metadata", "write", "maintenance") .build() ``` When in doubt, start with more restrictive permissions and expand only when necessary based on functional requirements. Challenge assumptions about permission needs during code reviews, as shown in the discussion where write access was initially questioned and determined to be unnecessary. --- ## Optimize before implementing Before implementing algorithms, evaluate their efficiency implications, especially for operations that may be executed frequently or with large datasets. Consider: - Time and space complexity in typical and worst-case scenarios - Memory allocation patterns and potential object churn - Opportunities to avoid redundant computations or unnecessary iterations - Impact on existing optimization pipelines and future optimization opportunities Document algorithmic decisions and assumptions with clear comments. Flag potential future optimizations with TODO comments that explain the intended improvement. For iterators and query processing, implement early termination strategies where possible: ```java // Consider if we can skip entire ranges when we know no matches are possible if (value < minTimestamp && allValuesInCurrentPrimarySort) { // We know we will not match any more documents in this primary sort approximation.match = Match.NO_AND_SKIP; return false; } ``` For data structures, leverage existing infrastructure that handles memory tracking efficiently: ```java // Use BlockHash to efficiently manage memory and circuit breaking BlockHash blockHash = BlockHash.build(keys, blockFactory, breaker); ObjectArray queues = new ObjectArray<>(blockHash.capacity(), breaker); ``` When selecting nodes or distributing work, prefer approaches that minimize network traffic and object creation: ```java // Iterate over eligible nodes (probably fewer) rather than all candidate nodes return nodeIds.stream() .filter(nodeId -> candidateNodeIds.contains(nodeId)) .collect(Collectors.toSet()); ``` --- ## Dynamic configuration needs validation When implementing dynamic configuration options, validate that the system actually supports runtime changes for that setting. A configuration should only be marked as dynamic if: 1. The code actively monitors for changes to this value 2. The system can safely handle runtime updates 3. Changes are properly persisted according to configuration 4. Backwards compatibility is maintained Example of proper dynamic config implementation: ```go // Good: Dynamic config that's actually monitored allowRecovery = viperutil.Configure( "allow-recovery", viperutil.Options[bool]{ FlagName: "allow-recovery", Default: true, Dynamic: true, // System actively checks this value } ) // Bad: Misleading dynamic flag discoveryMaxConcurrency = viperutil.Configure( "discovery-max-concurrency", viperutil.Options[int]{ FlagName: "discovery-max-concurrency", Default: 300, Dynamic: true, // Wrong! Value only checked during initialization } ) ``` For backwards compatibility, carefully consider default values and document any runtime limitations. When in doubt, mark as static rather than potentially misleading users about dynamic capabilities. --- ## Robust test assertions Use precise, informative assertions in tests to provide clear feedback when tests fail and verify the correct behavior rather than implementation details. **Key practices:** 1. **Use assertion methods with descriptive messages**: Replace plain `assert` statements with proper test framework assertions that provide clear error messages. ```java // Instead of this: assert valueCount == 1; // Do this: assertEquals(1, valueCount, "Multi-values make chunking more complex, and it's not a real case yet"); ``` 2. **Use expectThrows for exception testing**: Replace try-catch blocks with more readable and concise expectThrows pattern. ```java // Instead of this: try { createComponents("my_analyzer", analyzerSettings, testAnalysis.tokenizer, testAnalysis.charFilter, testAnalysis.tokenFilter); fail("expected failure"); } catch (IllegalArgumentException e) { // assertions } // Do this: IllegalArgumentException e = expectThrows( IllegalArgumentException.class, () -> createComponents("my_analyzer", analyzerSettings, testAnalysis.tokenizer, testAnalysis.charFilter, testAnalysis.tokenFilter) ); // assertions on exception ``` 3. **Focus assertions on behavior, not implementation details**: Verify that the code does the right thing, not how it does it. ```java // Instead of verifying exact implementation details: verify(coordinatingIndexingPressure).increment(1, bytesUsed(doc0Source)); verify(coordinatingIndexingPressure).increment(1, bytesUsed(doc1Source)); // Verify the essential behavior: verify(coordinatingIndexingPressure, times(6)).increment(eq(1), longThat(l -> l > 0)); ``` 4. **Verify all relevant fields**: Assert on all important aspects of the test output, not just a subset. 5. **Randomize test values**: Use randomized values instead of hardcoded constants to ensure tests catch edge cases. ```java // Instead of hardcoded values: String apiKeyId = "test-id"; // Use randomization: String apiKeyId = randomAlphaOfLength(20); ``` 6. **Use appropriate matchers**: For values that may vary, use flexible assertions like `greaterThanOrEqualTo()` instead of strict equality. --- ## Log level appropriately Select the appropriate log level based on operational significance and ensure messages are clear, accurate, and formatted for human readability: 1. **ERROR**: Use for failures requiring immediate attention 2. **WARN**: Use for unusual but non-critical situations 3. **INFO**: Use for significant but normal operations 4. **DEBUG**: Use for detailed diagnostic information that won't impact performance Consider the frequency and volume of log entries when selecting levels. High-frequency operations should use lower log levels (DEBUG) to avoid log pollution and performance degradation. Format log messages for runtime readability with variables logically positioned: ```rust // Bad - Poor readability in runtime logs warn!( "could not create image layers due to {}; this is not critical because the requested image LSN is below the GC curoff", err ); // Good - Clear and readable at runtime warn!( "failed to create image layers but lsn is <= gc_cutoff, therefore this can happen: lsn={lsn} gc_cutoff={gc_cutoff}: {err}", ); ``` Ensure log messages accurately describe the operation being performed and provide meaningful context. Avoid logging routine operations in tight loops, especially at higher log levels. --- ## Defensive null handling Always handle null references and values defensively to prevent NullPointerExceptions and unexpected behavior. Follow these practices: 1. **Check nulls before dereferencing**: Always verify a reference is not null before accessing its methods or properties: ```java // Instead of directly accessing possibly null objects Engine engine = shard.getEngineOrNull(); assertThat(engine != null && engine.isThrottled(), equalTo(true)); // Or for parsing operations if (token == null) { token = parser.nextToken(); } ``` 2. **Use APIs that handle nulls gracefully**: Prefer methods that allow specifying default values when parsing or converting: ```java // Instead of Boolean.parseBoolean which doesn't handle nulls well assertThat(Booleans.parseBoolean((String) indexSettings.get(setting.getKey()), false), matcher); ``` 3. **Defensive coding for asynchronous operations**: When objects might be deleted or changed by other threads: ```java var project = clusterService.state().metadata().projects().get(projectId); PersistentTasksCustomMetadata.PersistentTask persistentTask = project == null ? null : PersistentTasksCustomMetadata.getTaskWithId(project, getPersistentTask()); ``` These patterns help create more robust code that can handle edge cases and prevent crashes in production. --- ## Parallel branch traceability When implementing algorithms with parallel processing branches, ensure proper traceability and data consistency across all branches to facilitate result analysis and debugging: 1. Add discriminator fields to identify the source branch for each result entry 2. Maintain consistent data types for identical field names across all branches 3. Establish a clear strategy for handling missing fields (e.g., null values) 4. Preserve the ordering of results within each branch while providing mechanisms to control inter-branch ordering Example implementation: ``` function processBranches(input) { // Create parallel branches const branches = [branchA(input), branchB(input), branchC(input)]; // Combine results with branch identifiers const results = []; branches.forEach((branchResult, index) => { branchResult.forEach(item => { // Add discriminator field item.branchId = `branch${index + 1}`; // Ensure all fields exist (with nulls if needed) ensureConsistentFields(item, getAllFieldsAcrossBranches(branches)); results.push(item); }); }); // Option to sort by branch for clearer output return sortByBranchIfNeeded(results); } ``` This approach ensures that outputs from complex multi-branch algorithms remain traceable, consistent, and analyzable, which is critical for debugging and maintaining algorithmic correctness. --- ## Stage intensive operations carefully When implementing operations that consume significant system resources (CPU, memory, I/O), introduce changes gradually while monitoring performance. This prevents cascading performance issues that can degrade the entire system. For example: 1. When modifying configuration parameters that trigger resource-intensive tasks (like decreasing `min_age` in Elasticsearch ILM policies), consider how many tasks might execute simultaneously 2. For threadpool size adjustments (like `thread_pool.force_merge.size`), increase gradually while monitoring system metrics 3. Consider where operations execute and potential trade-offs - operations on high-performance nodes may complete faster but could impact critical workflows like data ingestion This approach helps maintain system stability while optimizing resource-intensive operations. --- ## Prefer callbacks over blocking Always structure concurrent code to use asynchronous callbacks instead of blocking operations. Blocking calls like CountDownLatch, Thread.join(), or Future.get() can lead to thread starvation, reduced throughput, and complex deadlock scenarios. Replace blocking patterns with callback-based approaches: ```java // AVOID: Blocking with CountDownLatch CountDownLatch latch = new CountDownLatch(1); final AtomicBoolean result = new AtomicBoolean(false); service.getResource(id, new ActionListener<>() { @Override public void onResponse(Resource resource) { result.set(true); latch.countDown(); } @Override public void onFailure(Exception e) { logger.error("Failed: {}", e.getMessage()); latch.countDown(); } }); latch.await(5, TimeUnit.SECONDS); // BLOCKS thread! return result.get(); // PREFER: Fully asynchronous with listeners void getResourceAsync(String id, ActionListener listener) { service.getResource(id, ActionListener.wrap( resource -> listener.onResponse(Boolean.TRUE), exception -> { if (exception instanceof ResourceNotFoundException) { listener.onResponse(Boolean.FALSE); } else { listener.onFailure(exception); } } )); } ``` Similarly, for testing asynchronous code, avoid spawning threads and waiting with Thread.join(). Instead, use futures or test-friendly abstractions that allow controlling when callbacks are triggered. --- ## Optimize memory allocation patterns Minimize memory allocations and optimize allocation patterns to improve performance. Key practices: 1. Pre-allocate collections with known sizes 2. Avoid unnecessary intermediate allocations 3. Reuse allocated memory when possible 4. Use appropriate capacity reservations Example - Before: ```rust // Allocates new Vec for each row result.try_push(Some( x.chunks_exact(element_size) .map(|val| T::cast_le(val)) .collect::>(), )); ``` Example - After: ```rust // Pre-allocate with known capacity let mut builder = make_builder(arr.dtype()); builder.reserve(output_length); // Avoid intermediate allocations result.try_push( x.chunks_exact(element_size) .map(|val| T::cast_le(val)) ); ``` Benefits: - Reduced memory churn - Better performance through fewer allocations - More predictable memory usage patterns - Lower risk of allocation-related performance spikes --- ## Exceptions for critical errors Use exceptions rather than assertions for handling critical error conditions that need to be caught in production. Assertions should only be used for development-time invariant checking that should never occur in production code. Bad: ```java assert bestAssignment != -1 : "Failed to assign vector to centroid"; ``` Good: ```java if (bestAssignment == -1) { throw new IllegalStateException("Failed to assign vector to centroid"); } ``` Assertions are disabled by default in production environments, which means critical error conditions could be silently ignored. This can lead to data corruption or system instability. Instead: 1. Use exceptions for error conditions that: - Need to be handled by calling code - Represent invalid states that could occur in production - Should trigger error reporting/monitoring 2. Reserve assertions for: - Validating internal implementation invariants - Checking preconditions during development/testing - Documenting assumptions that should never be violated When choosing between exceptions and assertions, ask "Should this error be caught and handled in production?" If yes, use an exception. --- ## Scope and document configurations When designing and implementing configuration options, carefully consider two key aspects: 1. **Choose appropriate configuration scope**: Place configuration settings at the level that makes the most sense for their purpose and usage pattern. Avoid overly granular scopes when settings could logically apply at a broader level. ``` // Prefer index-level settings for behaviors that affect the entire index // Example from Discussion 1: "early_termination": true // At index level, not field level ``` 2. **Document configurations completely**: Provide clear documentation that covers all possible values, edge cases, and boundary conditions. Explicitly state limitations, especially for managed environments where configuration options may be restricted. ``` // Example of improved documentation clarity: // Before: "50% of total system memory when more than 1 GB, with a maximum of 31 GB." // After: "50% of total system memory when 1 GB or more, with a maximum of 31 GB." ``` For managed or restricted environments, clearly explain which configuration options are available to users and why certain configurations might be limited or handled automatically. --- ## Appropriate error handling Distinguish between implementation errors (invariant violations) and expected failure cases. For implementation errors that should never occur, use `unreachable!()` with explanatory comments rather than generic `panic!()` or `todo!()`. For expected failure modes, return proper error types with informative messages. Implementation invariant violations: ```rust // Good: Clear explanation with unreachable!() if parsed != incr { unreachable!("Parsing invariant violated: expected {} parsed digits, got {}", incr, parsed); } // Bad: Using todo() for cases that should never occur match inner_fn { IRBitwiseFunction::And => (new_bitwise_and_reduction(get_dt(input)?), input), IRBitwiseFunction::Or => (new_bitwise_or_reduction(get_dt(input)?), input), IRBitwiseFunction::Xor => (new_bitwise_xor_reduction(get_dt(input)?), input), _ => todo!(), // Should be unreachable!() if this case can't happen } ``` Expected failures: ```rust // Good: Informative error with context if ac.is_aggregated() { polars_bail!(InvalidOperation: "cannot slice() an aggregated value") } // Good: Using try_ methods with ? operator to propagate errors offset_fn(&Duration::try_parse(offset)?, timestamp, time_zone).map(Some) ``` Document error conditions with appropriate doc comments: ```rust /// # Panics /// Panics if the parsing invariant is violated. /// # Errors /// Returns an error if the input cannot be parsed as a valid timestamp. ``` In debug builds, use `debug_assert!()` for checks that shouldn't affect release performance, but always check critical invariants in both debug and release builds. --- ## Database provider compatibility Ensure database code properly handles provider-specific differences and capabilities. Different database providers have varying syntax requirements, feature support, and connection methods that must be accounted for in the implementation. Key areas to consider: - **Parameter binding**: Some databases use numbered parameters (`$1`, `$2`) while others use anonymous placeholders (`?`) - **Feature availability**: Certain features like transaction isolation levels may only be available for specific providers (e.g., "this is really only available for SQL servers") - **Connection methods**: Different providers support different connection approaches (TCP, Unix sockets, etc.) - **Data types**: Column type mapping varies between providers and may depend on additional flags - **Schema generation**: Default schemas must be valid for the target provider Example of proper provider-specific handling: ```typescript // Handle parameter binding differences const usesAnonymousParams = [Providers.MYSQL, Providers.SQLITE].includes(provider) // Skip provider-specific features appropriately if (datasource.provider !== 'sqlite' && parseEnvValue(datasource.url) === defaultURL(datasource.provider)) { // Show warning for non-SQLite providers only } // Handle provider-specific connection methods if (credentials.type === 'postgresql' || credentials.type === 'cockroachdb') { // Unix socket handling for PostgreSQL-compatible databases } ``` Always test provider-specific code paths and avoid making assumptions about universal database feature support. --- ## Optimize cargo dependencies Maintain clean and efficient dependency configurations in Cargo.toml files by following these practices: 1. **Use workspace inheritance** when available for common settings and dependencies: ```toml [package] name = "your_package" version = "0.1.0" edition.workspace = true license.workspace = true [dependencies] some_dependency.workspace = true ``` 2. **Remove unused dependencies** to keep builds efficient and prevent unnecessary bloat. 3. **Maintain alphabetical ordering** of dependencies to improve readability and make changes easier to track. 4. **Avoid duplicate version specifications** when a dependency is already defined in the workspace. Use the workspace version unless you specifically need a different version: ```toml # Prefer this tokio-util.workspace = true # Instead of this tokio-util = { version = "0.7", features = ["compat"] } ``` 5. **When upgrading a dependency version**, ensure compatibility with existing code and consider whether the upgrade should happen at the workspace level. Following these practices helps maintain a clean dependency tree, reduces build times, and makes configuration files easier to understand and maintain. --- ## Extract duplicated code Identify and extract duplicated code into reusable functions or move common fields to parent structures. This makes the codebase more maintainable and reduces the risk of inconsistencies when code needs to be updated. Consider these refactoring approaches: 1. **Move common fields to parent structures** when multiple states or variants contain the same fields: ```rust // Instead of this: struct ShiftNode { state: State, } enum State { StateA { buffer: Vec, seq: MorselSeq }, StateB { buffer: Vec, seq: MorselSeq }, } // Do this: struct ShiftNode { state: State, buffer: Vec, seq: MorselSeq, } enum State { StateA, StateB, } ``` 2. **Extract repeated patterns into functions** when similar code appears in multiple locations: ```rust // Instead of repeating this pattern: match self.dtype() { #[cfg(feature = "timezones")] Datetime(_, Some(tz)) => { /* complex logic */ }, _ => { /* similar complex logic */ } } // Create a helper function: fn process_with_timezone(&self, has_timezone: bool) -> Result { // Implementation that handles both cases } ``` 3. **Use named constants** instead of magic numbers: ```rust // Instead of: Scalar::new(Time, AnyValue::Time(86_399_999_999_999)) // Use a descriptive constant: Scalar::new(Time, AnyValue::Time(NS_IN_DAY - 1)) ``` 4. **Parameterize functions** to handle variation rather than duplicating similar functions: ```rust // Instead of two nearly identical functions: fn prepare_expression_for_context() { /* ... */ } fn prepare_expression_for_context_with_schema() { /* ... */ } // Create one function with an optional parameter: fn prepare_expression_for_context(schema: Option) { /* ... */ } ``` These refactorings improve readability, maintainability, and reduce the risk of bugs when one instance of duplicated code is updated but others are missed. --- ## Hide implementation details Design public APIs to hide implementation details and focus on the user's mental model of the system. Avoid exposing internal classes, implementation-specific parameters, or technical approaches that might change in the future. Key principles: - Use terminology that matches the user's conceptual model, not internal implementation - Avoid exposing implementation-specific parameters like `parallel` or internal classes - Prefer extensible parameter designs (e.g., string literals) over booleans for features that might be expanded - Use full descriptive names in public APIs instead of abbreviations Example - Before: ```python def set_gpu_engine(cls, active: bool | None = None) -> type[Config]: """Set the default engine to use the GPU.""" # ... def filter(self, predicate: Expr, *, parallel: bool = False) -> Series: """Filter elements in each list by a boolean expression.""" # ... ``` Example - After: ```python def set_default_engine(cls, engine: Literal["cpu", "gpu"]) -> type[Config]: """Set the default engine type.""" # ... def filter(self, predicate: Expr) -> Series: """Filter elements in each list by a boolean expression.""" # ... ``` This approach creates more maintainable APIs, gives implementation flexibility, and provides a clearer mental model for users. --- ## Avoid unnecessary allocations Minimize memory allocations by avoiding intermediate objects, sharing underlying buffers, and eliminating unnecessary array operations. This is particularly important in hot code paths and when processing large datasets. Key strategies: - Share underlying buffers instead of copying data: `Buffer.from(arg.buffer, arg.byteOffset, arg.byteLength)` instead of `Buffer.from(arg)` - Avoid creating intermediate arrays when chaining operations: inline map operations rather than creating separate arrays - Construct objects directly when possible: `new Uint8Array(value)` instead of `new Uint8Array(Buffer.from(value).buffer)` - Move expensive computations out of frequently called functions to constructors or initialization phases - Only store data structures when the intermediate results are actually needed Example of optimization: ```typescript // Before: Creates unnecessary intermediate array const results: Value[] = [] for (const arg of node.args) { results.push(await this.interpretNode(arg, queryable, scope, generators)) } return results[results.length - 1] // After: Only compute what's needed let lastResult: Value for (const arg of node.args) { lastResult = await this.interpretNode(arg, queryable, scope, generators) } return lastResult ``` --- ## Prevent deadlock conditions Carefully manage resource acquisition and release to prevent deadlocks in concurrent code. Deadlocks typically occur when multiple threads hold resources while waiting for resources held by each other. To avoid deadlocks: 1. **Release resources before blocking operations**: Always drop locks, tokens, or other exclusive resources before operations that might block or wait on results from other threads. 2. **Be careful with resource ordering**: When a component needs to interact with multiple synchronization primitives, establish and maintain a consistent acquisition order. 3. **Properly propagate blocking states**: When implementing custom synchronization mechanisms, ensure blocking states are correctly propagated between components to prevent hidden deadlocks. 4. **Ensure proper cancellation**: When using async tasks, wrap each task handle in cancellation mechanisms (like `AbortOnDropHandle`) immediately after creation to prevent resource leaks during cancellation. Example of fixing a deadlock-prone pattern: ```rust // Problematic code - potential deadlock fn process_data(mut morsel: Morsel, sender: &mut Sender, linearizer: &mut Linearizer) { // Holding consume_token while inserting into linearizer can create circular wait let consume_token = morsel.take_consume_token(); linearizer.insert(morsel); // If linearizer is waiting for another thread, and that thread needs consume_token // to be dropped to proceed, we have a deadlock } // Fixed version fn process_data(mut morsel: Morsel, sender: &mut Sender, linearizer: &mut Linearizer) { // Drop the token before potentially blocking operations let consume_token = morsel.take_consume_token(); drop(consume_token); // Explicitly drop before linearizer operation linearizer.insert(morsel); } ``` --- ## validate inputs early Validate function inputs, preconditions, and assumptions as early as possible in the execution flow, preferably during binding or initialization phases rather than deep within implementation logic. This prevents invalid states from propagating through the system and ensures errors are caught at the appropriate architectural level. Early validation should check for: - Input parameter validity and type correctness - Preconditions that must be met before processing - Resource availability and state consistency - Duplicate or conflicting arguments Example of good early validation: ```cpp // In bind phase - catch errors before execution starts static unique_ptr ListContainsBind(ClientContext &context, ScalarFunction &bound_function, vector> &arguments) { // Check for unsupported nested types early if (arguments[1]->return_type.id() == LogicalTypeId::LIST) { throw NotImplementedException("This function has not yet been implemented for nested types"); } // Validate map type assumption before using it if (map_logical_type.id() != LogicalTypeId::MAP) { throw InvalidInputException("Expected MAP type, got %s", map_logical_type.ToString()); } } ``` Example of problematic late validation: ```cpp // Bad - validating deep in execution logic static void SomeFunction(DataChunk &args, ExpressionState &state, Vector &result) { // ... lots of processing ... if (children.size() != 1) { // Should be caught much earlier throw InvalidInputException("Expected single child"); } } ``` This approach reduces debugging complexity, provides clearer error messages with better context, and prevents resource waste on operations that are destined to fail. --- ## Structure endpoints for REST Organize API endpoints hierarchically by resource type and use appropriate HTTP methods based on operation semantics. Group related operations under resource-based paths, use debug prefixes for administrative endpoints, and select HTTP methods that match the operation's behavior: - Use resource paths like `/v1/resource/sub-resource` - Place debug/admin endpoints under `/debug/v1/` - Use GET for retrieval - Use POST for async operations - Use PUT for synchronous updates - Return 202 Accepted for async operations Example: ```rust Router::new() // Resource endpoints .route("/v1/lfc/prewarm", get(get_prewarm_status)) .route("/v1/lfc/prewarm", post(start_prewarm)) .route("/v1/lfc/offload", get(get_offload_status)) .route("/v1/lfc/offload", post(start_offload)) // Debug endpoints .route("/debug/v1/feature_flag", put(override_flag)) ``` --- ## Prevent cryptic errors Always implement proper validation and type checking to prevent cryptic error messages. When errors do occur, provide clear, actionable guidance for resolution. Key practices: 1. Use exact instance checking rather than isinstance() when needed to avoid attribute errors: ```python # Prefer this if isinstance(path, io.BytesIO): target = path # Over potentially error-prone checks that could lead to attribute errors ``` 2. Place input validation at appropriate scope levels to catch errors early: ```python # Validate inputs early with helpful error messages if len(exprs) == 1 and isinstance(exprs[0], Mapping): msg = ( "Cannot pass a dictionary as a single positional argument.\n" "If you merely want the *keys*, use:\n" " • df.method(*your_dict.keys())\n" "If you need the key–value pairs, use one of:\n" " • unpack as keywords: df.method(**your_dict)\n" " • build expressions: df.method(expr.alias(k) for k, expr in your_dict.items())" ) raise TypeError(msg) ``` 3. Consider returning None instead of raising exceptions for non-critical failures to allow more consistent error handling by callers. 4. Leverage built-in language error mechanisms when appropriate rather than implementing custom error checking that duplicates existing functionality. --- ## Secure authentication handling Always implement proper authentication checks and protect sensitive credentials throughout your codebase. This includes: 1. **Validate all authentication inputs properly** to prevent panics and security vulnerabilities: ```rust // Bad: Will panic on short inputs if &authorization[0..7] != "Bearer " { // ... } // Good: Safe parsing without panic risk if let Some(token) = authorization.strip_prefix("Bearer ") { // Process token safely } else { return Err(tonic::Status::invalid_argument("invalid authorization format")); } ``` 2. **Implement permission checks for all API endpoints**, even internal or testing ones: ```rust // Always add permission checks to new endpoints async fn your_api_endpoint( request: Request, _cancel: CancellationToken, ) -> Result, ApiError> { // Add this check to prevent unauthorized access check_permissions(&request, None)?; // Rest of your endpoint logic // ... } ``` 3. **Redact sensitive information in logs** to prevent credential exposure: ```rust // When logging URLs or other structures that might contain credentials match url::Url::parse(url) { Ok(mut url) => { if url.password().is_some() { let _ = url.set_password(Some("_redacted_")); } // Now safe to log info!("Using URL: {}", url); } // ... } ``` These practices prevent unauthorized access to sensitive endpoints, reduce risks of credential leakage, and improve the overall security posture of the system. --- ## Proper metrics design When designing metrics for observability systems like Prometheus, follow established best practices to ensure your metrics are useful, queryable, and maintainable: 1. Split metrics appropriately: Create separate metrics when aggregate operations (sum, avg) across dimensions wouldn't be meaningful. Don't combine unrelated measurements under a single metric name. 2. Use correct metric types: Select 'counter' for values that only increase (like event counts) and 'gauge' for values that can go up or down (like current status or measurements). Example of proper metrics design: ```jsonnet // GOOD: Separate metrics with clear meaning { metric_name: 'pg_autovacuum_oldest_mxid', type: 'gauge', help: 'Oldest multi-transaction ID across all databases', key_labels: ['database_name'], }, { metric_name: 'pg_autovacuum_oldest_frozen_xid', type: 'gauge', help: 'Oldest frozen transaction ID across all databases', key_labels: ['database_name'], }, { metric_name: 'pg_autovacuum_run_count', type: 'counter', // Counter for non-decreasing values help: 'Number of autovacuum runs', key_labels: ['database_name', 'table_name'], } // BAD: Combined unrelated metrics with unclear aggregation meaning { metric_name: 'pg_autovacuum_stats', type: 'gauge', // Wrong type for counts help: 'Autovacuum statistics for all tables across databases', key_labels: ['database_name'], value_label: 'metric', values: ['oldest_mxid', 'oldest_frozen_xid', 'run_count'], } ``` Reference: https://prometheus.io/docs/practices/naming/ --- ## Sync environment variables When converting hardcoded values to environment variables in configuration files (like docker-compose.yml), always update corresponding example files (e.g., .env.example) with appropriate default values. This ensures that developers can use standard setup processes without additional manual configuration. Example: ```diff # In docker-compose.yml - SECRET_KEY_BASE=UpNVntn3cDxHJpq99YMc1T1AQgQpc8kfYTuRgBiYa15BLrx8etQoXz3gZv1/u2oq + SECRET_KEY_BASE=${POOLER_SECRET_KEY_BASE} # You must also update .env.example + POOLER_SECRET_KEY_BASE=UpNVntn3cDxHJpq99YMc1T1AQgQpc8kfYTuRgBiYa15BLrx8etQoXz3gZv1/u2oq ``` This maintains compatibility for new developers setting up the project and prevents broken workflows when someone runs `cp .env.example .env && docker-compose up`. --- ## Safe null handling Always implement robust null handling patterns to prevent unexpected behavior and crashes. Consider all edge cases where null values might affect calculations, type conversions, or data operations: 1. Don't rely on potentially null values for sizing or allocation decisions: ```rust // Bad: Zero allocation if first element is null let capacity = from.get(0) .map(|bytes| bytes.len() / element_size) .unwrap_or(0); // Good: Use a reasonable default or derive from non-null elements let capacity = if from.is_empty() { DEFAULT_CAPACITY } else { // Find first non-null element or use default from.iter() .filter_map(|opt| opt.map(|bytes| bytes.len() / element_size)) .next() .unwrap_or(DEFAULT_CAPACITY) }; ``` 2. Use `Option` when a function might not be able to determine a result with certainty: ```rust // Bad: Returns bool even when we're uncertain pub fn can_cast_to(&self, to: &DataType) -> bool { // might return false when we're not sure } // Good: Explicitly communicates uncertainty pub fn can_cast_to(&self, to: &DataType) -> Option { // None = not sure, Some(false) = definitely cannot cast } ``` 3. Avoid unsafe shortcuts when accessing potentially null values - prefer safe methods that properly handle nulls and bounds checking: ```rust // Bad: Unsafe access for simple value retrieval if let Some(value) = unsafe { array.get_unchecked(0) } { // ... } // Good: Safe access with proper null handling if let Some(value) = array.get(0) { // ... } ``` --- ## Optimize memory allocation Always allocate data structures with appropriate initial capacity and use memory-efficient data types to reduce memory pressure and improve application performance. Pre-allocate containers when the size is known or can be estimated, avoid unnecessary intermediate data structures, and choose compact data representations. For example: ```go // Good: Pre-allocate with known capacity finalRes := make(map[string]string, len(qr.Rows)) errors := make(map[string][]error, 1) // When expecting few entries // Good: Use memory-efficient data types // TabletAlias object (8 bytes) vs string representation (16 bytes) type queueItem struct { TabletAlias *topodatapb.TabletAlias // More memory-efficient // Instead of: Key string // Less efficient } // Good: Avoid unnecessary intermediate data structures // Direct processing without creating intermediate collections var offset int64 var fieldsIndex int for i, loc := range bindLocations { field = tp.Fields[fieldsIndex] length = row.Lengths[fieldsIndex] // Process directly... } ``` Memory allocation patterns have significant impact on both CPU and memory performance, especially in high-throughput services. Inefficient allocations create unnecessary garbage collection pressure, leading to more frequent GC cycles and higher CPU usage. By being thoughtful about memory allocation, you can achieve substantial performance improvements with minimal code changes. --- ## Configuration context consistency Ensure configuration names, values, and settings accurately reflect their intended context and usage. Configuration mismatches can lead to runtime errors, incorrect behavior, and maintenance issues. Key practices: - Match configuration names to their actual context (e.g., use 'mysql' folder names in MySQL introspection functions, not 'postgresql') - Use generic, extensible configuration structures when the setting applies to multiple contexts (e.g., `set: { configs: [...], role: ... }` instead of `rlsConfig` for PostgreSQL transaction settings) - Verify runtime configurations work across target environments and Node.js versions - Configure build tools with strict settings to catch configuration inconsistencies early Example of good configuration naming: ```ts // Instead of context-specific naming interface PgTransactionConfig { rlsConfig?: { ... }; } // Use generic, extensible naming interface PgTransactionConfig { set?: { configs?: { name: string; value: string; isLocal?: boolean; }[]; role?: AnyPgRole; }; } ``` This approach prevents configuration drift and makes code more maintainable when contexts change or expand. --- ## preserve error context When propagating errors through promise rejections, catch blocks, or error transformations, always preserve the original error information to maintain debugging context and error traceability. Common anti-patterns include: - Rejecting promises without the original error: `reject()` instead of `reject(error)` - Generic error messages that lose specifics: `"An error occurred"` instead of the actual error - Not handling unknown error types properly when extracting messages **Good practices:** ```typescript // ✅ Pass the original error when rejecting } catch (error) { this.#setTransactionEnded(true, error); reject(error); // Preserve the original error } // ✅ Handle unknown error types safely while preserving info function makeAppErrorResponse(m: Mutation, e: unknown): MutationResponse { return { result: { error: 'app', details: e instanceof Error ? e.message : String(e), // Convert unknown to string } }; } // ✅ Return rejected promises instead of sync throws in async contexts run(): Promise> { return Promise.reject(new Error('AuthQuery cannot be run')); } ``` This practice significantly improves debugging experience by maintaining the full error chain and context, making it easier to trace issues back to their root cause. --- ## Flexible documented configurations Create configuration interfaces that are flexible, well-documented, and future-proof. When designing configuration parameters: 1. **Prefer flexible parameterization** - Use functions/interfaces with optional parameters and sensible defaults rather than separate specialized functions for related configuration operations. ```sql -- Prefer this approach: CREATE OR REPLACE FUNCTION anon.enable_transparent_masking_superuser( dbname TEXT, toggle BOOL DEFAULT = true, ) RETURNS VOID AS $$ BEGIN EXECUTE format('ALTER DATABASE %I SET anon.transparent_dynamic_masking TO %L', dbname, toggle::text); END; $$ ``` 2. **Document the rationale** - Explain not only what a parameter does, but also why it exists and the reasoning behind its format or structure. ``` /* * Comma-separated list of safekeeper connection strings * This format allows specifying multiple connection parameters for each safekeeper * while maintaining backward compatibility with the host:port format */ ``` 3. **Plan for future changes** - When hardcoding values or using temporary solutions, document the limitations and potential future improvements. 4. **Consider parameter precedence** - When configurations can be specified in multiple places, establish and document clear rules about which settings take precedence. 5. **Use stable sources** - For external dependencies, prefer stable, well-maintained sources (e.g., GitHub mirrors over potentially unstable direct links). --- ## Optimize what matters Focus optimization efforts on performance-critical paths rather than applying micro-optimizations everywhere. Balance code clarity and maintainability against performance gains, and only optimize when there's a measurable benefit. When considering optimizations: 1. Determine if the code is on a critical path or frequently executed 2. Measure the actual performance impact before and after changes 3. Evaluate the tradeoff between readability and performance For example, avoid premature optimization in code like this: ```c // PREFER: Simple and clear approach for non-critical code for (int bucketno = 0; bucketno < NUM_QT_BUCKETS; bucketno++) { uint64 threshold = qt_bucket_thresholds[bucketno]; metrics[i].bucket_le = (threshold == UINT64_MAX) ? INFINITY : ((double) threshold) / 1000000.0; } // AVOID: Premature optimization making code less clear for (int bucketno = 0; bucketno < NUM_QT_BUCKETS - 1; bucketno++) { metrics[i].bucket_le = ((double) qt_bucket_thresholds[bucketno]) / 1000000.0; } // Special case for last bucket metrics[NUM_QT_BUCKETS-1].bucket_le = INFINITY; ``` However, when there's significant performance impact (like looping from 3 to INT_MAX), optimization becomes necessary. Always document performance-critical sections and the rationale behind optimization decisions. --- ## Design for operation flexibility When implementing algorithms that operate on data structures (particularly arrays, lists, or collections), design them to handle both constant and expression-based inputs efficiently. Consider the constraints each imposes on the algorithm: 1. For constant inputs, you can determine output shapes/types at compile-time 2. For expression-based inputs, you may need runtime evaluation Implement pattern flags (like `as_array`) that indicate when constant values are required for type determination. This enables both flexibility and performance optimization. For example: ```python def slice(self, offset: int | Expr, length: int | Expr, as_array: bool = True) -> Expr: """Slice the array. Parameters ---------- offset The offset to start the slice. length The length of the slice. as_array If True, offset and length must be constants. If False, returns a list instead of an array. """ if as_array and (not isinstance(offset, int) or not isinstance(length, int)): raise ValueError("When as_array=True, offset and length must be constants") # Implementation continues... ``` This pattern enables algorithms to work efficiently with both scenarios while providing clear feedback when constraints cannot be met. --- ## Use specialized sensitive types When handling sensitive data like encryption keys, choose data types based on the data's lifecycle and security requirements. Use specialized types (e.g., `EncryptionKey`) for actual sensitive data that persists in memory, as these provide memory locking and secure deletion. Use regular strings only for temporary user input that gets immediately processed and wiped. For example: ```cpp // For temporary user input (acceptable) string encryption_key; // User input, immediately wiped // For actual encryption keys (preferred) EncryptionKey actual_key; // Locked memory, secure deletion ``` The distinction matters because temporary user input can be any length and exists briefly, while actual keys should be fixed-size, memory-locked, and securely managed throughout their lifecycle. --- ## Document parameter choices Always add explanatory comments for parameters or configuration options that aren't self-explanatory from their names or context. These comments should explain: 1. What the parameter does 2. Why a specific value was chosen 3. What different values would mean (especially in tests with multiple modes) This helps future developers understand your reasoning and the implications of parameter choices. Examples: ```python # Good - explains why retry_on_failures is needed env.storage_controller.reconcile_until_idle(timeout_secs=60, retry_on_failures=True) # retry_on_failures needed because reconciliation may encounter transient errors during split operations # Good - explains test parameters @pytest.mark.parametrize("with_compute_ctl", [False, True], ids=["standard", "compute-ctl"]) # with_compute_ctl: Tests prewarm behavior both with and without compute controller integration def test_lfc_prewarm(neon_simple_env: NeonEnv, with_compute_ctl: bool): ... ``` Especially important for: - Boolean flags (the reason for enabling/disabling) - Magic numbers (why this specific value) - Test parameters (what aspect each parameter tests) - Configuration options with non-default values --- ## Validate environment configurations Ensure that all environment-specific configurations work properly across all target environments, particularly when managing multiple architectures or CI systems. Test changes thoroughly in the actual environments where code will run. Document the purpose of specialized configurations and don't remove configuration blocks without verifying they're no longer needed, even if they appear obsolete. For example, when handling MySQL installations across different architectures, build packages against the appropriate dependencies for each target architecture. In CI workflows, preserve necessary system configurations like AppArmor settings that might be required for specific environments: ```yaml # Example: Preserving necessary environment-specific configurations sudo service mysql stop sudo service etcd stop sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld ``` --- ## dependency classification standards Ensure proper classification and versioning of package dependencies in package.json files. Dependencies should be classified based on their usage pattern: use regular `dependencies` for hard requirements that the package cannot function without, and `peerDependencies` for packages that consumers are expected to provide. When specifying version ranges, use appropriate semantic versioning operators - prefer `>=` over `>` for minimum versions, and understand that v0.x packages behave differently than v1+ packages in semver. For example, database drivers should typically be regular dependencies rather than peer dependencies: ```json { "dependencies": { "@libsql/client": "0.8.0" } } ``` For TypeScript requirements, specify minimum versions clearly: ```json { "peerDependencies": { "typescript": ">=5.1.0" } } ``` Remove unused dependencies to keep the dependency tree clean and avoid unnecessary bloat. When working with v0.x packages, be explicit about version ranges since `^0.48` won't match `0.49.1`, unlike v1+ packages. --- ## Names reveal clear intent Choose names that clearly communicate intent and context, avoiding ambiguity or confusion. Variable and function names should be self-documenting and reflect their domain purpose. Key guidelines: - Use domain-specific terminology consistently - Prefer descriptive names over generic ones - Make implicit context explicit in the name - Avoid positional/numeric references in favor of semantic names Example - Before: ```rust fn make_categoricals_compatible( input: &[column], // Uses numeric index input[1] array_width: usize, ) ``` After: ```rust fn make_rhs_categoricals_compatible( CategoryMapping { source_column, target_column, }: CategoryMapping, array_width_in_bytes: usize, ) ``` The improved version: - Clarifies the function's effect on right-hand side - Uses a semantic structure instead of array indices - Adds context to the width parameter - Makes the relationship between parameters explicit --- ## Proper Option type usage Use Option only for truly optional values that can meaningfully be None. Avoid using Option when a value is always required or when defaulting to empty/invalid values. Key guidelines: 1. Use Option when a value may legitimately be absent 2. Avoid Option if a field is never None in practice 3. Don't use empty/default values as substitutes for Option Example: ```rust // Good: Truly optional value struct Config { refresh_interval: Option, // May not be configured error_message: Option // Only present on error } // Bad: Using Option when value is never None struct AuthInterceptor { tenant_id: AsciiMetadataValue, shard_id: Option // If always present, remove Option } // Bad: Using empty string instead of Option struct Settings { base_url: String // Empty string is meaningless, use Option } ``` --- ## Guard against race conditions When working with concurrent operations, always implement proper guards to prevent race conditions between processes. This includes: 1. Adding explicit checks before operations that might be affected by concurrent processes: ```c // Before performing an operation that assumes file existence if (mdnblocks(reln, forknum) >= blocknum) { // Operation is safe to proceed // Comment explaining the race condition being prevented } ``` 2. Using precise mechanisms to detect resource availability in multi-process environments. For shared memory access, prefer `UsedShmemSegAddr` over `IsUnderPostmaster` since some processes may have detached shared memory: ```c // Correct check for shared memory availability if (newval && *newval != '\0' && UsedShmemSegAddr && shared_resource_available) ``` 3. Implementing robust interrupt handling for operations that may be suspended. Always recalculate timing after potential interruptions: ```c // Loop to handle interrupted sleeps while (elapsed_time < required_delay) { pg_usleep(required_delay - elapsed_time); // Handle potential interrupts CHECK_FOR_INTERRUPTS(); // Recalculate elapsed time with fresh timestamp current_time = GetCurrentTimestamp(); elapsed_time = current_time - start_time; } ``` Always add clear comments explaining the race condition being prevented to help future developers understand the reasoning behind these guards. --- ## Test algorithmic performance scaling When implementing or modifying algorithms, especially data structures like hash tables, bloom filters, or pattern matching logic, ensure performance characteristics are validated across different input scales. Many algorithms exhibit degraded performance or correctness issues that only become apparent with large datasets. For algorithms that may degrade at scale, create targeted performance tests that cover: - Small inputs (typical use cases) - Medium inputs (stress testing) - Large inputs (boundary conditions) When performance testing is impractical for regular CI due to runtime constraints, consider: - Creating separate benchmark suites that can be run periodically - Testing at smaller scales that still demonstrate the algorithmic behavior - Using library methods and established algorithms instead of custom implementations for edge cases Example from bloom filter testing: ```java // Test false positive rates at different scales @Test public void testBloomFilterScaling() { // Test at 1M elements - baseline performance testFalsePositiveRate(1_000_000, 0.03, 0.05); // within 5% tolerance // Test at 100M elements - where degradation may start testFalsePositiveRate(100_000_000, 0.03, 0.10); // allow higher tolerance } ``` Avoid hardcoded checks for algorithmic edge cases. Instead of manual pattern matching like `pattern.equals(lowercaseRegexPrefix)`, use library methods that can properly determine functional equivalence or emptiness. --- ## Verify dependency integrity Always verify the integrity of external dependencies, especially those downloaded from non-official or personal repositories. This helps prevent supply chain attacks where compromised packages could introduce security vulnerabilities. When downloading external packages: 1. Prefer official sources over personal repositories 2. Always validate package integrity using cryptographic checksums (SHA-256 recommended) 3. Fail the build/installation if verification fails 4. Document the expected checksums in your codebase Example implementation: ```bash # Download package wget -c https://example.com/package.deb # Define expected checksum EXPECTED_SHA256="e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" # Verify checksum and abort on mismatch echo "$EXPECTED_SHA256 package.deb" | sha256sum -c - || { echo "Checksum verification failed!"; exit 1; } # Only proceed with installation if verification passed sudo dpkg -i package.deb ``` For long-term solutions, work towards hosting critical dependencies in organization-controlled repositories with proper access controls and provenance tracking. --- ## avoid cosmetic formatting changes Avoid including purely cosmetic formatting changes in pull requests that serve a functional purpose. Automatic formatter changes (like Prettier reformatting, bracket placement, or line breaks) should be separated from functional changes to keep PRs focused and easier to review and merge. When submitting PRs, ensure changes contain the "absolute minimum amount of changes compared to the version you are editing" for faster approval. If you need to apply formatting changes, consider doing so in a separate, dedicated formatting-only PR. Example of cosmetic changes to avoid mixing with functional changes: ```typescript // Before (functional change mixed with formatting) - generated: T extends { generated: infer G } ? G extends undefined ? unknown : G : unknown; -} -& TTypeConfig + generated: T extends { generated: infer G } ? (G extends undefined ? unknown : G) : unknown; +} & TTypeConfig // Better: Keep functional changes separate from formatting changes ``` This practice improves code review efficiency and reduces the likelihood of merge conflicts. --- ## minimize hot path allocations Reduce memory allocations and garbage collection pressure in frequently executed code paths. Object creation through functional methods like `Object.fromEntries()`, `map()`, and repeated `toString()` calls can cause significant performance degradation in hot paths. **Prefer manual object construction over functional approaches:** ```typescript // ❌ Avoid - creates multiple intermediate objects return Object.fromEntries( keyColumns.map(col => [col, row[col]]) ); // ✅ Better - single allocation const result = {}; for (const col of keyColumns) { result[col] = row[col]; } return result; ``` **Accumulate strings efficiently:** ```typescript // ❌ Avoid - repeated toString() calls l < r && (this.currVal += chunk.subarray(l, r).toString('utf8')); // ✅ Better - collect segments, join once segments.push(chunk.subarray(l, r)); // Later: segments.join('') ``` **Prevent unbounded memory growth:** Monitor data structures that grow indefinitely (like histograms per keystroke) and implement cleanup strategies or size limits. Consider the memory footprint of operations that scale with user input frequency. **Use lazy evaluation for expensive operations:** Only create error stacks, stringify large objects, or perform expensive computations when actually needed, especially in debug scenarios or error paths. --- ## Avoid flaky tests Design tests to be deterministic and reliable across different environments. Tests that occasionally fail due to timing, race conditions, or environmental differences waste development time and reduce confidence in the test suite. When testing values that may vary based on environment conditions (like scores influenced by shard counts), use relative comparisons, before/after validations, or approximation matchers rather than exact values: ```yaml # Instead of relying on exact scores which can vary: - match: { hits.hits.0._score: 1.8918664 } # Better - validate scores are different before/after modification: - do: search: index: test-index body: query: match: field: "query text" - set: { hits.hits.0._score: base_score } - do: search: index: test-index body: query: match: field: query: "query text" boost: 5.0 - gt: { hits.hits.0._score: $base_score } ``` Avoid "hacky" approaches that depend on timing. Instead, focus on testing that functionality exists without creating timing dependencies: ```yaml # Avoid unreliable wait mechanisms: - do: catch: request_timeout cluster.health: wait_for_nodes: 10 timeout: "2s" # Better - verify functionality exists without timing dependencies: - do: cat.shards: index: foo h: refresh.is_search_idle - match: $body: /^(false|true)\s*\n?$/ ``` --- ## Use current configuration patterns Always use the current recommended configuration patterns for your project, avoiding deprecated approaches. When configuring tests, features, or resource access: 1. For test requirements, prefer capability-based specifications over version-based skipping: ```yaml # Instead of: requires: cluster_features: ["gte_v9.1.0"] # Use: requires: test_runner_features: [capabilities] capabilities: - method: DELETE path: /{index}/_block/{block} ``` 2. For resource access configurations, be precise about paths and understand the available options: ```yaml # Be specific with paths: files: - relative_path_setting: "reindex.ssl.certificate" # Consider whether glob patterns are appropriate: files: - relative_path_setting: "reindex.ssl.*" ``` Using current configuration patterns improves maintainability, compatibility, and ensures the system behaves as expected across environments. --- ## Justify CI resource additions Before adding new resources (Dockerfiles, jobs, images) to CI/CD pipelines, provide clear justification for their necessity and document how they will be maintained. Consider whether the resource will actually run in your CI environment, if existing resources could be reused, and who will maintain version updates. For example, instead of creating a new Dockerfile for a specific test: ``` # Before: Creating a new Dockerfile that may not be necessary FROM golang:1.24.3 # ... rest of specialized Dockerfile that might be redundant # After: Use existing Docker images when possible, or clearly document the need # for specialized images in CI documentation and CHANGELOG.md ``` Document additions in appropriate CHANGELOG files and consider potential maintenance overhead before expanding CI infrastructure. --- ## explicit null state management Make null state checks explicit and comprehensive rather than using implicit return values or redundant fields. Use dedicated methods like `ContainsNull()` to clearly indicate null state, avoid duplicate null tracking when child objects already maintain null state, and ensure hierarchical null checks cover all levels (self and children). For efficiency, only perform null-setting operations when actually needed: ```cpp // Good: Explicit null check with efficient setting A_TYPE input = A_TYPE::ConstructType(state, i); if (input.ContainsNull()) { FlatVector::SetNull(result, i, true); continue; } // Good: Hierarchical null detection bool ContainsNull() { return is_null || a_val.is_null || b_val.is_null; } // Bad: Redundant null fields when children track null state struct StructTypeBinary { A_TYPE a_val; // Already has is_null field B_TYPE b_val; // Already has is_null field bool a_is_null; // Redundant! bool b_is_null; // Redundant! }; ``` This pattern prevents null-related bugs by making null state explicit, reduces redundancy, and ensures comprehensive null detection across complex nested structures. --- ## Explicit null handling Always be explicit and consistent about how null values are handled in operations and documentation. This clarity prevents confusion and ensures predictable behavior across the codebase. When implementing functions that process data which might contain nulls: 1. Use "null" terminology (not "None") in documentation and comments to maintain conceptual consistency, even though Python uses `None` internally: ```python # Incorrect def eq_missing(self, other: Any) -> Self: """Equality operator where `None` is treated as a distinct value.""" # Correct def eq_missing(self, other: Any) -> Self: """Equality operator where null is treated as a distinct value.""" ``` 2. Document null behavior with concrete examples: ```python def sum_horizontal(*columns, ignore_nulls: bool = True): """Sum values horizontally across columns. Parameters ---------- ignore_nulls - If True: nulls are replaced by 0 (e.g., 1 + null + 2 = 3) - If False: nulls are propagated (e.g., 1 + null + 2 = null) """ ``` 3. When documenting behavior, avoid double negations and use clear, specific language: ```python # Avoid: "If there are no non-null values, the output is False." # Better: "If all values are null or the collection is empty, the output is False." ``` 4. For operations where input values might not match expected size/format, explicitly state how nulls are used: ```python # Example: Document how binary data of incorrect length is handled """Note that rows of the binary array where the length does not match the width of the output array will become NULL.""" ``` Consistently following these practices reduces ambiguity and helps users understand how their data will be processed in the presence of missing values. --- ## preserve serialization compatibility When making changes to serialized data structures, always preserve backward and forward compatibility to prevent breaking existing databases and migration paths. This applies to enums, serialized plans, function serialization, and version handling. Key practices: - **Append new enum values at the end** rather than reordering existing ones, as changing enum order breaks serialization - **Avoid regenerating serialized plans** - if this becomes necessary, fix the underlying serialization issue instead - **Use default values for new serialization fields** to maintain forward compatibility - fields with default values won't be serialized, allowing older versions to read newer serialized data - **Use open-ended version ranges** (e.g., `v1.2.0+`) instead of closed ranges to accommodate ongoing compatibility Example of safe enum extension: ```cpp enum class CatalogType : uint8_t { INVALID = 0, TABLE_ENTRY = 1, SCHEMA_ENTRY = 2, // ... existing entries ... DATABASE_ENTRY = 9, NEW_ENTRY = 10, // Append new entries at the end }; ``` Breaking serialization compatibility can render existing databases unreadable and disrupt migration workflows, making this a critical consideration for any schema or data structure changes. --- ## Use parameterized queries Always use parameterized queries with bind variables instead of string concatenation or formatting when constructing SQL statements. This prevents SQL injection attacks and improves query performance through better statement caching. **Instead of this:** ```go query := `SELECT variable_value FROM performance_schema.global_status WHERE variable_name IN (` for _, status := range statuses { query += `"` + status + `"` } query += ");" qr, err := mysqld.FetchSuperQuery(ctx, query) ``` **Do this:** ```go statusBv, err := sqltypes.BuildBindVariable(statuses) if err != nil { return nil, err } query, err := sqlparser.ParseAndBind( "SELECT variable_name, variable_value FROM performance_schema.global_status WHERE variable_name IN %a", statusBv, ) if err != nil { return nil, err } qr, err := mysqld.FetchQuery(ctx, query) ``` When building dynamic SQL statements, especially those with user-provided input, always use the SQL parser and bind variables. This approach not only prevents SQL injection but also allows the database to cache and reuse execution plans, improving performance for frequently executed queries with different parameter values. --- ## generate test data dynamically Instead of adding static test data files to the repository, generate test data programmatically within test cases. This approach improves test modularity, reduces repository bloat, and ensures tests are self-contained. Use temporary directories for any files that need to be written during testing. When you need test data, create it using code rather than committing large CSV files or other data files. For example: ```sql statement ok CREATE TABLE my_tbl (a int, b int, c varchar); statement ok INSERT INTO my_tbl SELECT range, 5000 - range, 'cooler string' FROM range(5000); statement ok COPY tbl TO '__TEST_DIR__/output.csv' (HEADER, DELIMITER ','); ``` This keeps tests modular, avoids persistent files between test runs, and makes the test data generation explicit and maintainable within the test itself. --- ## Extract and reuse Create focused utility functions for repeated or complex operations instead of duplicating logic across the codebase. When implementing functionality, first check if similar code already exists that can be reused. Benefits: - Improves code readability by abstracting implementation details - Reduces duplication, making maintenance easier - Makes testing isolated functionality simpler Example: ```c // Before: Repeated parsing logic in multiple places if (strncmp(wp->config->safekeeper_connstrings, "g#", 2) == 0) { char *endptr; errno = 0; wp->safekeepers_generation = strtoul(wp->config->safekeeper_connstrings + 2, &endptr, 10); if (errno != 0) { wp_log(FATAL, "failed to parse generation number: %m"); } // ... } // After: Extracted to a utility function bool split_off_safekeepers_generation(const char* connstring, uint32* generation, const char** remaining) { if (strncmp(connstring, "g#", 2) == 0) { char *endptr; errno = 0; *generation = strtoul(connstring + 2, &endptr, 10); if (errno != 0 || *endptr != ':') return false; *remaining = endptr + 1; return true; } *generation = INVALID_GENERATION; *remaining = connstring; return true; } // Usage const char* remaining; if (!split_off_safekeepers_generation(wp->config->safekeeper_connstrings, &wp->safekeepers_generation, &remaining)) { wp_log(FATAL, "failed to parse generation number"); } ``` When implementing new functionality, scan the codebase for similar operations that might already be implemented as utility functions. For complex operations like parsing, prefer creating dedicated functions that can be reused across the project. --- ## Consistent case style Use consistent case styles in your codebase, adapting to the conventions of the ecosystem you're interacting with. When working with systems that use snake_case (like many Python-based backends, databases, or MCP servers), maintain that convention in your interfaces to those systems. Benefits: - Improves code readability and maintainability - Creates consistency across ecosystem boundaries - Can improve performance with ML models that are trained predominantly on certain syntactic patterns **Example:** ```typescript // Good: Using snake_case for fields that interface with a snake_case system const logFields = { log_type: 'postgres', event_message: message, api_role: role }; // Not recommended: Mixing conventions when interfacing with snake_case systems const logFields = { logType: 'postgres', // camelCase event_message: message, // snake_case apiRole: role // camelCase }; ``` For new components, consider the primary systems they'll interact with and adopt the prevailing convention. Document your case style decisions in your team's style guide to maintain consistency. --- ## Escape SQL parameters Always escape parameters in database connection strings to prevent SQL injection attacks. Direct string concatenation with user-provided or database-sourced values creates security vulnerabilities that can be exploited. When using PostgreSQL's dblink function, apply proper escaping functions like quote_ident() for identifiers (e.g., database names) and quote_literal() for string values: Instead of unsafe concatenation: ```sql 'dbname=' || d.datname || ' user=' || current_user || ' connect_timeout=5' ``` Use escaped parameters: ```sql 'dbname=' || quote_ident(d.datname) || ' user=' || quote_literal(current_user) || ' connect_timeout=5' ``` This practice should extend to all SQL query construction, ensuring that any dynamic values incorporated into queries are properly sanitized before execution. --- ## Design domain-specific error types Create and use domain-specific error types instead of generic errors or anyhow. This improves error handling clarity and ensures proper error propagation through the codebase. Key guidelines: 1. Define specific error types for your domain using enums 2. Include variants for different failure modes (e.g., Cancelled, ShuttingDown) 3. Provide clear, lowercase error messages with context 4. Propagate errors using proper type mapping Example: ```rust // Instead of using anyhow or generic errors: fn process() -> anyhow::Result<()> { if cancelled { bail!("operation cancelled"); // Generic error } Ok(()) } // Create domain-specific error types: #[derive(Error, Debug)] pub enum ProcessError { #[error("operation cancelled")] Cancelled, #[error("failed to write to {path}: {source}")] IoError { path: String, source: std::io::Error }, #[error("invalid configuration: {0}")] Config(String), } fn process() -> Result<(), ProcessError> { if cancelled { return Err(ProcessError::Cancelled); } Ok(()) } // Proper error mapping when crossing domain boundaries: fn handle_process() -> Result<(), ApiError> { process().map_err(|e| match e { ProcessError::Cancelled => ApiError::ServiceUnavailable, ProcessError::IoError { .. } => ApiError::InternalError, ProcessError::Config(_) => ApiError::BadRequest, })?; Ok(()) } ``` --- ## Comprehensive code documentation Properly document code with clear, accurate, and useful comments using the correct syntax based on context: 1. Use `///` to document items (structs, functions, etc.) that follow the comment: ```rust /// Struct to store runtime state of the compute monitor thread. /// Handles tracking Postgres availability and managing downtime metrics. pub struct ComputeMonitor { // ... } ``` 2. Use `//!` for module or crate-level documentation that applies to the enclosing item. 3. Document all public structures, traits, and functions with explanations of: - Their purpose - Usage considerations - Potential pitfalls or caveats 4. For complex data structures, include explanations of fields and their relationships: ```rust /// Range of LSNs for page requests /// - request_lsn: LSN specifically requested by compute /// - effective_lsn: LSN actually used for the request (may differ from request_lsn) /// Note: Primary computes typically use request_lsn == MAX pub struct LsnRange { pub request_lsn: Lsn, pub effective_lsn: Lsn, } ``` 5. Ensure comments are accurate and up-to-date. Never merge incorrect comments with the intention to fix them later. 6. When referencing other code elements, use proper linking syntax: `[`Type::method()`]` to enable IDE navigation. 7. Avoid redundant comments that merely restate what is already clear from the code or function names. Good documentation serves as both a reference for users of your code and as context for future developers (including yourself) who will need to maintain it. --- ## secure sensitive data handling When handling sensitive data like encryption keys, passwords, or authentication tokens, avoid using standard string types that can be easily copied and may persist in memory. Instead, use specialized secure data structures that prevent accidental copying and provide controlled memory management. Key practices: 1. **Use secure data structures**: Replace `std::string` with custom classes that are not copy-constructible and handle their own memory locking/unlocking 2. **Explicit memory clearing**: When you must use standard types temporarily, explicitly clear sensitive data using `fill()` and `clear()` operations 3. **Memory locking**: Use platform-specific functions like `mlock()`/`VirtualLock()` to prevent sensitive data from being paged to disk Example of problematic code: ```cpp // BAD: std::string can be easily copied accidentally std::string encryption_key = user_input; storage_options.encryption_key = encryption_key; // Creates copy ``` Example of secure handling: ```cpp // GOOD: Explicit clearing of temporary sensitive data auto user_key = entry.second.GetValue(); storage_options.encryption_key = user_key; // Clear the user key from memory fill(user_key.begin(), user_key.end(), '\0'); user_key.clear(); ``` This approach prevents sensitive data from accidentally persisting in memory through unintended copies and reduces the attack surface for memory-based exploits. --- ## Use descriptive names Choose names that clearly communicate intent and purpose, avoiding vague or misleading terms. Names should be self-documenting and accurately reflect what the variable, function, or type represents. **Key principles:** - Replace vague names with specific, descriptive alternatives - Ensure function names accurately describe their behavior and return values - Use descriptive parameter names that clarify their role - Avoid generic terms when more specific ones are available **Examples:** ```typescript // ❌ Vague and misleading function getView() { /* doesn't return anything useful */ } const enabled: boolean; // too generic const tableName = 'users'; // unclear if client or server name // ✅ Clear and descriptive function useView() { /* name reflects side-effect behavior */ } const isServerConnected: boolean; // specific about what's enabled const serverTableName = 'users'; // clarifies which context // ❌ Generic parameter names function configure(name: string, args: unknown[]) { } // ✅ Descriptive parameter names function configure(--target-client-row-count: string, queryArgs: ReadonlyJSONValue[]) { } ``` This practice makes code self-documenting, reduces cognitive load for reviewers, and prevents misunderstandings about functionality. When names accurately reflect their purpose, the code becomes easier to maintain and debug. --- ## Reliable concurrency synchronization When handling concurrent operations, prefer completion signals and proper thread management over arbitrary timeouts. This improves code reliability and prevents hangs or deadlocks. For waiting on asynchronous processes: - Instead of arbitrary timeouts that can cause flaky tests: ```python # Avoid this wait_until(lambda: assert_condition(), timeout=120) # Prefer this env.storage_controller.reconcile_until_idle() wait_until(lambda: assert_condition()) ``` For operations that might block indefinitely: - Use thread pools to prevent the main execution thread from hanging: ```python # Instead of potentially blocking code ps.restart() # Could hang waiting for active status # Use a ThreadPoolExecutor for potentially blocking operations with concurrent.futures.ThreadPoolExecutor() as executor: future = executor.submit(ps.restart) # Continue execution or implement proper timeout handling ``` These patterns ensure that your concurrent code is more predictable, testable, and less prone to timing-dependent issues. --- ## Choose appropriate abstractions When designing APIs, select data types and patterns that match how they will be consumed while facilitating long-term maintainability: 1. Use data types that naturally fit the intended usage pattern. For cloud paths, prefer `String`/`Box` over `OsStr` when APIs you interact with expect string types: ```rust // Instead of: path: Box, // requires conversions when used with string APIs // Prefer: path: Box, // directly compatible with string-based APIs ``` 2. Prefer serialization approaches that decouple interface from implementation. When handling complex structures that may change over time, consider JSON serialization rather than field-by-field matching: ```rust // Instead of field-by-field matching: IR::Sink { input, payload } => Sink { // Complex, brittle field extraction... } // Prefer: IR::Sink { input, payload } => Sink { input: input.0, payload: serde_json::to_string(payload)? } ``` 3. Design for capability-based extensibility rather than type discrimination. Use flags to indicate capabilities instead of checking specific types, allowing for plugins and extensions: ```rust // Instead of checking specific reader types if let FileReaderType::CSV = reader_type { // CSV-specific handling } // Prefer capability flags if reader.supports_projection() { // Handle projection for any reader supporting it } ``` 4. Structure APIs to avoid complex state management. Builder patterns with clear ownership transfer make code more predictable: ```rust // Instead of: let df = LazyCsvReader::new("reddit.csv") // Path stored as internal state // Prefer: let df = LazyCsvReader::new().load("reddit.csv") // No internal state tracking ``` These approaches lead to more maintainable APIs that are easier to extend and evolve over time. --- ## Optimize data transformations When implementing data processing operations, avoid unnecessary data transformations, copies, and conversions that can impact query performance. Consider these practices: 1. Prefer direct pattern matching over string conversions: ```rust // Avoid this: matches!(function, FunctionExpr::Range(f) if f.to_string() == "int_range") // Prefer this: matches!(function, FunctionExpr::Range(RangeFunction::IntRange { .. })) ``` 2. Avoid creating temporary data structures when existing ones can be reused: ```rust // Avoid duplicate data with unnecessary temp dataframes: let tmp_df = value_col.into_frame().hstack(pivot_df.get_columns()).unwrap(); // Instead, pass existing data directly: // Use pivot_df directly in subsequent operations ``` 3. Ensure schema is cleared after operations that modify DataFrame structure to prevent incorrect schema information from being cached: ```rust // After modifying DataFrame columns df.clear_schema(); ``` 4. Be precise about row handling in data processing operations, clearly distinguishing between rows scanned vs. rows read, especially important for operations like slicing, filtering, and maintaining correct row indices. --- ## Configuration context alignment Choose the appropriate configuration context based on how changes will be handled by the system. When defining custom configuration variables: 1. Use `PGC_POSTMASTER` for options that require a server restart or cannot be changed while the server is running 2. Use `PGC_SIGHUP` only when there is code to properly handle configuration changes at runtime 3. Implement appropriate handler functions for configurations that need special processing when changed Example of proper context selection: ```c DefineCustomStringVariable( "neon.safekeeper_extra_conninfo_options", "libpq keyword parameters and values to apply to safekeeper connections", NULL, &safekeeper_extra_conninfo_options, "", PGC_POSTMASTER, // Requires restart since we don't have reconnection code ... ); // For configurations that need special handling when changed: DefineCustomStringVariable( "neon.safekeepers", "List of safekeepers", NULL, &safekeepers_list, "", PGC_SIGHUP, // Can change during runtime ... NULL, assign_neon_safekeepers, NULL // Custom handler function ); ``` Adding clear comments when configurations have special behavior enhances code maintainability. --- ## Use descriptive names Choose names that clearly convey purpose and context rather than generic or vague terms. Names should be self-documenting and provide sufficient information for developers to understand their role without additional investigation. Avoid generic prefixes like "My" or overly broad terms without context. Instead, use specific, descriptive names that indicate the entity's purpose or functionality. Examples of improvements: - `MyByteArrayOutputStream` → `ExposedBufferByteArrayOutputStream` (describes the specific functionality) - `Inner` → `InnerJoinType` (provides domain context) - `mightContainEven` → `evenNumbersFoundCount` (clarifies what the variable represents) When naming decisions might seem unusual or counterintuitive, document the rationale to help future maintainers understand the choice. This is especially important when maintaining compatibility with existing APIs or external libraries. --- ## Database entity configuration Configure database entities with appropriate defaults and clear type distinctions. For triggers, prefer AFTER/ROW over BEFORE/STATEMENT as defaults for most use cases. When working with database views, clearly distinguish between regular views and materialized views in your code. When counting database resources like replicas, ensure accurate counting by excluding the primary instance. ```typescript // Good: Clear distinction between view types const isView = entity.type === ENTITY_TYPE.VIEW; const isMaterializedView = entity.type === ENTITY_TYPE.MATERIALIZED_VIEW; const isViewEntity = isView || isMaterializedView; // Good: Accurate counting of replicas const replicasCount = (replicasData?.length ?? 1) - 1; // Subtract primary instance // Good: Appropriate trigger defaults const defaultValues = { name: '', schema: '', table: '', activation: 'AFTER', // AFTER is typically more common orientation: 'ROW', // ROW level is more frequently needed } ``` --- ## Harden CI/CD runners All CI/CD workflow jobs must implement security controls for network traffic, particularly using the step-security/harden-runner action or equivalent technology. This prevents potential supply chain attacks or unauthorized data exfiltration during automated builds and tests. Example: ```yml - name: Harden the runner (Audit all outbound calls) uses: step-security/harden-runner@4d991eb9b905ef189e4c376166672c3f2f230481 # v2.11.0 with: egress-policy: audit ``` Ensure this step is added consistently to all jobs in CI/CD workflows to maintain a strong security posture across the entire build pipeline. --- ## Follow API conventions Design APIs following modern conventions and best practices to improve usability, maintainability, and consistency across your codebase. APIs should prioritize: 1. **Use intuitive method naming** that clearly expresses purpose and follows a consistent pattern: ```rust // Instead of: .query_sql("foo").with_query("SELECT * FROM bar") // Prefer: .query("foo").with_sql("SELECT * FROM bar") ``` 2. **Reuse existing functionality** rather than reimplementing similar features: ```rust // Instead of: let now = self.time_provider.now().timestamp_nanos(); Some(now - retention_period as i64) // Prefer: self.time_provider.now().checked_sub(Duration::nanoseconds(retention_period)) ``` 3. **Leverage library conveniences** for cleaner API construction: ```rust // Instead of: let db_query_param = format!("db={}", db.into()); let mut url = self.base_url.join(api_path)?; url.set_query(Some(&db_query_param)); let mut req = self.http_client.delete(url); // Prefer: let mut url = self.base_url.join(api_path)?; let mut req = self.http_client.delete(url).query(&[("db", db.as_ref())]); ``` 4. **Standardize types across API boundaries** to maintain consistency: ```rust // Instead of duplicating types in multiple crates: // influxdb3_client::Precision and influxdb3_server::Precision // Consider moving shared types to a common crate or re-exporting // from a single source to ensure they're always identical ``` 5. **Maintain proper encapsulation** - public APIs should hide implementation details and provide only the intended functionality to consumers. --- ## Descriptive semantic naming Create identifiers that clearly convey meaning through descriptive names and appropriate types. Two key practices improve code readability and prevent errors: 1. **Use semantic types instead of primitives** for domain concepts to provide type safety and clarity: ```rust // Instead of raw primitives: pub fn get_retention_period_cutoff_ts_nanos(&self, db_id: &DbId) -> Option { ... } // Use semantic types: pub struct UnixTimestampNanos(i64); pub fn get_retention_period_cutoff(&self, db_id: &DbId) -> Option { ... } ``` 2. **Choose clear, fully descriptive names** that convey complete context: ```rust // Prefer: let last_snapshotted_wal_sequence_number = get_sequence(); // Clear purpose // Over: let last_wal_sequence_number = get_sequence(); // Ambiguous ``` 3. **Include units in variable names** when working with primitive numeric types: ```rust let timestamp_ns: i64 = 1682939402000000000; // Unit 'ns' clarifies meaning ``` 4. **Follow established naming conventions** for your ecosystem. For metrics, follow standards like Prometheus conventions: ```rust // Correct: write_lines_total: Metric // Unit-less counters end with _total // Avoid: write_lines_count: Metric // Non-standard naming ``` These practices substantially improve code maintainability, preventing confusion and subtle bugs related to type misuse. --- ## Prefer explicit nullability Always make nullable states explicit in your code by leveraging Rust's type system. Use `Option` rather than sentinel values (like empty structs or default values) to represent potentially missing data. When handling options, prefer idiomatic patterns like `.map()` with `.unwrap_or()` for transforming with fallbacks. Consider specialized types like `NonZeroUsize` when values should never be null or zero. ```rust // Good: Return Option instead of empty PathBuf fn find_python_install() -> Option { // Return Some(path) when found, None otherwise } // Good: Idiomatic handling of nullable values snap.retention_period .map(Snapshot::from_snapshot) .unwrap_or(RetentionPeriod::Indefinite) // Good: Using specialized non-null type #[derive(Debug, Serialize, Eq, PartialEq, Clone, Copy)] pub struct LastCacheSize(pub(crate) NonZeroUsize); // Good: Explicit null representation in data structures struct LastCacheKey { column_name: String, value_map: HashMap, LastCacheState> } ``` When working with external libraries like Arrow, use appropriate null-checking methods (e.g., `is_valid()`) and handle null values consistently rather than treating them as error conditions. --- ## Use structured logging fields Always use structured logging with descriptive field names rather than string interpolation. Include relevant context variables such as identifiers, durations, and parameters to make logs more useful for debugging and monitoring. This practice makes logs more searchable, filterable, and easier to analyze. ```rust // Bad: String interpolation lacks structured fields info!("Created new instance id {:?}", instance_id); error!("Error splitting table buffer: {}", e); // Good: Structured logging with descriptive field names info!(instance_id = ?instance_id, "catalog not found, creating new instance id"); error!(error = %e, table = %table_name, db = %db_name, "Error splitting table buffer for persistence"); ``` When logging errors, use descriptive field names like `error` rather than single-letter variables to improve log readability. For logging important operations, include all relevant parameters to provide context for later analysis. --- ## Performance-conscious metrics implementation Implement metrics collection that is both comprehensive and minimally impactful on system performance. Design your metrics system to avoid creating bottlenecks in critical paths. Key principles: 1. **Avoid high-overhead operations in critical paths**: Don't spawn new tasks for each metric update in hot code paths. ```rust // Avoid this: tokio::spawn(async move { // Update metrics }); // Prefer this: store.add_write_metrics(num_lines, payload_size); ``` 2. **Use appropriate buffer sizing** for telemetry channels based on expected throughput. For high-volume services, consider larger buffers (e.g., 10k) or direct counter updates instead of channels. ```rust // Consider higher capacity for high-volume metrics let (sender, receiver) = mpsc::channel(10_000); ``` 3. **Track both success and failure metrics** for all critical operations to provide a complete picture of system behavior. ```rust // Example counters for both successful and failed operations const WRITE_LINES_TOTAL_NAME: &str = "influxdb_write_lines_total"; const WRITE_LINES_REJECTED_TOTAL_NAME: &str = "influxdb_write_lines_rejected_total"; ``` 4. **Ensure proper isolation of metrics** across components. When using multiple executors or processing paths, avoid sharing metric registries if they cause conflicts. Remember that metrics collection should provide valuable insights without becoming a performance bottleneck itself. --- ## Database replica promotion safeguards When implementing database replica promotion logic, avoid temporary workarounds that bypass validation checks. Instead, design a comprehensive approach that maintains data integrity throughout the promotion process. Consider these best practices: 1. Carefully evaluate whether streams need to be reestablished when a replica is promoted to primary 2. Explicitly handle timeline management during promotion events 3. Test promotion scenarios thoroughly to identify edge cases For example, instead of using flags to bypass validation checks: ``` // Avoid this approach if (SkipXLogPageHeader(wp, wp->propTermStartLsn) != wp->api.get_redo_start_lsn(wp) && !replica_promote) { // Validation logic here } ``` Consider implementing proper state transition handling: ``` // Better approach if (isReplicaPromotion) { // Handle LSN alignment explicitly for promotion case SetRedoStartLsn(wp, SkipXLogPageHeader(wp, wp->propTermStartLsn)); } else if (SkipXLogPageHeader(wp, wp->propTermStartLsn) != wp->api.get_redo_start_lsn(wp)) { // Normal validation logic for non-promotion case } ``` This ensures proper state management during replica promotion while maintaining the integrity of validation checks for normal operations. --- ## Mind transaction boundaries Be conscious of implicit transaction boundaries when working with databases. Programming constructs can create unexpected transaction scopes that affect behavior and performance. For example, in Python, the `with` block for database connections implicitly starts a transaction, causing all operations within the block to execute in a single transaction: ```python # CAUTION: All operations execute in one transaction with db.connect() as conn: conn.execute("INSERT INTO table1 VALUES (1)") conn.execute("INSERT INTO table2 VALUES (2)") # Both inserts committed or rolled back together ``` Similarly, be aware of which database commands create transaction boundaries. Commands like `CHECKPOINT` and `COMMIT` flush WAL logs, while others may have different transactional behavior: ```sql -- Both sufficient to flush current WAL: CHECKPOINT; COMMIT; -- Better than using pg_switch_wal() in many cases: SELECT pg_current_wal_insert_lsn(); CHECKPOINT; ``` Choosing appropriate transaction boundaries improves reliability, performance, and maintainability of database operations. --- ## maintain clean CI configuration Keep CI/CD configuration files clean and self-documenting by removing outdated comments, using descriptive parameter names, and avoiding redundant specifications. Remove TODO comments that no longer reflect the current state of the codebase, choose parameter names that clearly indicate their scope and purpose, and rely on existing configuration files (like package.json) rather than duplicating version specifications. Example of good practices: ```yaml # Remove outdated comments flavor: ['js_pg', 'js_libsql', 'js_d1', 'js_better_sqlite3'] # Use descriptive parameter names driverAdapterTestJobTimeout: # clearly indicates job-level timeout # Avoid redundant version specs when package.json handles it - name: Setup PNPM uses: pnpm/action-setup@v4 # No version specified - uses packageManager from package.json ``` This approach reduces maintenance overhead, prevents confusion from stale documentation, and makes CI configurations more reliable and easier to understand. --- ## Minimize critical path allocations Avoid unnecessary memory allocations in performance-critical code paths. These allocations not only consume memory but also trigger expensive CPU operations that can significantly impact system performance. Key optimization techniques: 1. **Use efficient data structures**: - Box large enum variants to reduce memory footprint ```rust // Before enum Schedule { Cron(OwnedScheduleIterator), // 288 bytes Every(Duration), // 16 bytes } // After enum Schedule { Cron(Box>), // 16 bytes Every(Duration), // 16 bytes } ``` - Prefer `HashMap` over `BTreeMap` for lookup-heavy operations - Consider `Arc` or `&str` over `String` when appropriate 2. **Optimize container operations**: - Use `contains()` instead of `iter().any()` for membership checks ```rust // Less efficient if paths_without_authz.iter().any(|disabled_authz_path| *disabled_authz_path == path) { // ... } // More efficient if paths_without_authz.contains(&path) { // ... } ``` - Consider how you handle capacity in vectors (clear vs. take) - Use iterators for lazy evaluation rather than materializing full collections 3. **Avoid string overhead**: - Use `to_string()` instead of `format!()` for simple conversions ```rust // Less efficient let partition_key = data_types::PartitionKey::from(format!("{}", parquet_file.chunk_time)); // More efficient let partition_key = data_types::PartitionKey::from(parquet_file.chunk_time.to_string()); ``` - Avoid rebuilding schemas or data structures repeatedly in hot paths Remember that even small allocations can have a significant impact when they occur frequently in critical code paths, particularly in high-throughput systems handling many requests or processing large datasets. --- ## Feature flag compatibility Design code to work correctly with any combination of feature flags. When implementing conditional compilation with feature flags: 1. Use conditional imports for dependencies that are only needed when certain features are enabled: ```rust // Good: Only import when feature is enabled use polars_core::POOL; #[cfg(feature = "new_streaming")] use polars_core::StringCacheHolder; // Bad: Requiring dependencies unconditionally use polars_core::{POOL, StringCacheHolder}; // Breaks when "new_streaming" is disabled ``` 2. Consider user scenarios with different feature combinations. Test both with and without features enabled. 3. Ensure error messages reference correct feature names: ```rust // Good: Reference the correct feature name polars_bail!( ComputeError: "consider compiling with polars-bigidx feature, or set 'streaming'" ) // Bad: Referencing incorrect feature name polars_bail!( ComputeError: "consider compiling with polars-u64-idx feature, or set 'streaming'" ) ``` 4. When adding code conditionally compiled with feature flags, check that the code still compiles and functions correctly when those features are disabled. 5. Consider organizing feature-dependent code to minimize duplication while maintaining compatibility with all feature combinations. --- ## Modern shell syntax Prefer double brackets (`[[ ]]`) over single brackets (`[ ]`) in shell scripts for improved functionality and consistency. While double brackets don't strictly require quotes around variables, maintaining consistent quoting practice is recommended even with double brackets: ```bash # Preferred style if [[ "$variable" == "value" ]]; then # code fi # Instead of if [ $variable == "value" ]; then # code fi ``` This practice enhances readability and offers protection if scripts are later modified to use single brackets or embedded in contexts with stricter requirements. --- ## Document intent clearly Add clear documentation that explains not just what code does, but why certain approaches were chosen. This applies to: 1. **Complex code sections**: Add explanatory comments to complex logic, particularly SQL queries ```sql -- last-used-api-keys: retrieves timestamp of most recent API key usage by role SELECT unix_millis(max(timestamp)) as timestamp, payload.role, payload.signature_prefix FROM edge_logs cross join unnest(metadata) as m -- additional joins... ``` 2. **Implementation decisions**: Document the reasoning behind specific coding patterns ```javascript // Create a shallow copy to prevent unintended mutations to the original object custom_properties: 'properties' in event ? event.properties : {}, ``` 3. **UI elements**: Include tooltips for interactive elements to clarify their purpose 4. **Technical limitations**: Explicitly state constraints in user-facing documentation (e.g., "disk size can only be expanded, not shrunk") Well-documented code improves maintainability, reduces onboarding time for new developers, and prevents misunderstandings about system behavior. --- ## prefer nullish coalescing operator Use the nullish coalescing operator (`??`) instead of the logical OR operator (`||`) when you specifically want to provide fallback values only for `null` or `undefined`, not for other falsy values. The logical OR operator (`||`) treats all falsy values (empty strings, 0, false, null, undefined) as conditions to use the fallback value. The nullish coalescing operator (`??`) only triggers the fallback for `null` and `undefined`, preserving other falsy values that might be valid data. **Use `??` when:** - Empty strings, 0, or false are valid values that shouldn't trigger fallbacks - You're specifically handling null/undefined cases - Working with optional properties or nullable database fields **Examples:** ```typescript // ❌ Problematic - empty string triggers fallback to 'public' const typeSchema = column.enum.schema || 'public'; // ✅ Correct - only null/undefined triggers fallback const typeSchema = column.enum.schema ?? 'public'; // ❌ Problematic - treats null as invalid when it might be valid data if (!!set[colName]) { /* ... */ } // ✅ Correct - explicitly check for undefined if (set[colName] !== undefined) { /* ... */ } // ❌ Problematic - empty object created for any falsy value Object.keys(obj || {}) // ✅ Correct - empty object only for null/undefined Object.keys(obj ?? {}) ``` This pattern is especially important when working with database schemas, optional configurations, and API responses where distinguishing between "no value provided" (null/undefined) and "falsy value provided" (empty string, 0, false) is crucial for correct behavior. --- ## Maintain code readability Ensure code remains readable and maintainable by following these practices: 1. **Combine case statements with identical outcomes** to reduce duplication. When multiple cases have identical code blocks, group them together: ```go // GOOD switch planType { case tsm1.PT_Standard, tsm1.PT_SmartOptimize: return common case tsm1.PT_NoOptimize: return testLevelResults{} } // AVOID switch planType { case tsm1.PT_Standard: return common case tsm1.PT_SmartOptimize: return common case tsm1.PT_NoOptimize: return testLevelResults{} } ``` 2. **Extract common values** to avoid repetition, especially in maps or switch statements: ```go // GOOD common := testLevelResults{ level4Groups: []tsm1.PlannedCompactionGroup{ { tsm1.CompactionGroup{"01-05.tsm", "02-05.tsm", "03-05.tsm", "04-04.tsm"}, tsdb.DefaultMaxPointsPerBlock, }, }, } switch planType { case tsm1.PT_Standard, tsm1.PT_SmartOptimize, tsm1.PT_NoOptimize: return common } ``` 3. **Use named constants instead of magic numbers**. Operations between constants should also be assigned to named constants: ```go // GOOD const AggressiveMaxPointsPerBlock = DefaultMaxPointsPerBlock * 100 // AVOID e.Compactor.Size = tsdb.DefaultMaxPointsPerBlock * 100 ``` 4. **Simplify nested conditionals** where possible: ```go // GOOD var loaded bool if f, loaded = m.fields.LoadOrStore(newField.Name, newField); f.Type != typ { return f, !loaded, ErrFieldTypeConflict } return f, !loaded, nil // AVOID if f, loaded := m.fields.LoadOrStore(newField.Name, newField); loaded { if f.Type != typ { return f, false, ErrFieldTypeConflict } return f, false, nil } else { return f, true, nil } ``` These practices make code easier to understand, maintain, and reduce bugs from copy-paste errors or inconsistencies. --- ## Document concurrency design decisions Always document key concurrency design decisions in code, including: 1. Locking protocols and ordering between multiple locks 2. Assumptions about data access patterns and thread safety 3. State transition guarantees and potential race conditions 4. Rationale for chosen concurrency primitives Example: ```rust /// The locking protocol for this struct is: /// - `index` must be acquired before `inner` /// - `inner` is append-only, so it is safe to: /// * read and release `index` before locking and reading from `inner` /// * write and release `inner` before locking and updating `index` /// /// This avoids holding `index` locks across IO operations and is crucial /// for avoiding read tail latency. pub struct ConcurrentStorage { index: RwLock>, inner: RwLock, } ``` Clear documentation of concurrency decisions helps prevent subtle bugs, makes code more maintainable, and enables safe evolution of concurrent systems. When concurrent code lacks proper documentation, reviewers and maintainers must reverse-engineer the intended synchronization patterns, which is error-prone and time-consuming. --- ## Follow hooks rules Adhere strictly to React hooks rules and best practices to ensure your components behave correctly and predictably. Common issues include: 1. **Manage dependencies properly**: Always include all dependencies in useEffect/useCallback dependency arrays or use useLatest() for values you intentionally exclude to prevent stale data issues. ```jsx // INCORRECT: Missing snap.enforceExactCount and error.code in dependencies useEffect(() => { if (isError && snap.enforceExactCount && error.code === 408) { // ... } }, [isError]); // CORRECT: Include all dependencies useEffect(() => { if (isError && snap.enforceExactCount && error.code === 408) { // ... } }, [isError, snap.enforceExactCount, error.code]); // ALTERNATIVE: Use useLatest for values you want to read but not trigger re-runs const latestSnap = useLatest(snap); const latestError = useLatest(error); useEffect(() => { if (isError && latestSnap.current.enforceExactCount && latestError.current.code === 408) { // ... } }, [isError]); ``` 2. **Never conditionally call hooks**: Instead of conditionally calling hooks, use the enabled option or similar patterns. ```jsx // INCORRECT: Conditionally calling a hook breaks the rules of hooks const { hasAcceptedConsent } = IS_PLATFORM ? useConsentToast() : { hasAcceptedConsent: true } // CORRECT: Use a parameter to conditionally enable the hook's behavior const { hasAcceptedConsent } = useConsentToast({ enabled: IS_PLATFORM }) ``` 3. **Use useCallback for function props**: When passing function props to memoized components, wrap them in useCallback to preserve memoization. ```jsx // INCORRECT: Inline function breaks memoization on each render snap.updateMessage({ id: message.id, ...props })} /> // CORRECT: Preserve memoization with useCallback const handleResults = useCallback((props) => { snap.updateMessage({ id: message.id, ...props }) }, [message.id, snap]) ``` 4. **Avoid hooks in loops**: Never use hooks inside loops, conditions, or nested functions. Extract them to custom hooks if needed. Remember that hook calls must be at the top level of your component to ensure they're called in the same order on each render. --- ## Metric design best practices Design metrics to be reliable and maintainable by following these key principles: 1. Initialize metrics with zero values to ensure consistent existence: ```go labelValues := []string{keyspace, shard, tabletType} metrics.vstreamsEndedWithErrors.Add(labelValues, 0) ``` 2. Choose appropriate metric granularity: - Consider per-component metrics when aggregation could mask issues - For streaming/processing, track per-shard metrics to identify bottlenecks - Avoid overly granular labels (e.g., hostnames) that can explode cardinality 3. Select stable label dimensions: - Use static identifiers (keyspace, shard, type) - Avoid ephemeral values like hostnames in container environments - Consider the metric lifespan and deployment environment 4. Handle metric lifecycle properly: - Avoid metric re-registration issues - Implement proper cleanup for deprecated metrics - Document metric deprecation with clear replacement paths These practices ensure metrics remain useful for debugging production issues while staying maintainable over time. --- ## Configurable cache parameters Cache configurations should be runtime-configurable rather than hardcoded, with support for dynamic resizing when configuration changes. This practice improves adaptability to varying workloads and environments without requiring service restarts. When implementing caches: 1. Define cache capacities as configurable parameters: ```rust // Instead of this: const REL_SIZE_CACHE_CAPACITY: usize = 1024; // Do this: #[derive(Debug, Clone, Deserialize)] pub struct TenantConfig { pub relsize_snapshot_cache_capacity: usize, // other config parameters... } ``` 2. Implement handlers to resize caches when configuration changes: ```rust fn tenant_conf_updated(&self, new_conf: TenantConfig) { // Update the cache capacity when config changes let mut cache = self.rel_size_cache.write().unwrap(); cache.resize_capacity(new_conf.relsize_snapshot_cache_capacity); } ``` 3. Handle existing entries properly during cache operations to prevent memory leaks: ```rust fn insert(&mut self, key: K, value: V) { if self.cache.contains_key(&key) { // Handle existing entry - consider evicting old entry, // updating statistics, or queueing for cleanup self.deletion_queue.send(key.clone()); } self.cache.insert(key, value); } ``` This approach ensures that caches can be tuned for optimal performance based on actual usage patterns without service disruptions, and properly manages the lifecycle of cached entries. --- ## use default serialization methods When adding new properties to serialization methods, use `WritePropertyWithDefault` and `ReadPropertyWithDefault` instead of `WriteProperty` and `ReadProperty` to maintain backwards compatibility. This ensures that older versions can still deserialize data written by newer versions by providing sensible defaults for missing properties. Without default methods, adding new serialized properties breaks backwards compatibility because older versions cannot handle the new data format. Using default methods allows graceful degradation where older versions use default values for properties they don't understand. Example: ```cpp // Bad - breaks backwards compatibility serializer.WriteProperty(212, "extra_info", extra_info); // Good - maintains backwards compatibility serializer.WritePropertyWithDefault(212, "extra_info", extra_info); ``` Additionally, be cautious with `ShouldSerialize()` checks that might cause information loss when serializing to older storage versions. If critical data would be lost, prefer throwing an error on deserialization rather than silently dropping information. --- ## avoid redundant computations Move loop-invariant conditions and computations outside of iteration blocks to improve performance and reduce redundant processing. When the same check or calculation doesn't depend on loop variables, extract it to run once before the loop rather than repeating it for each iteration. For example, instead of checking the same condition inside every loop iteration: ```python for regression in regression_list: if isinstance(time_old, float) and isinstance(time_new, float): # process regression ``` Extract the invariant check outside the loop: ```python if isinstance(time_old, float) and isinstance(time_new, float): geomean_time_slower = time_new <= time_old for regression in regression_list: if geomean_time_slower and individual_regression_diff_perc <= 10.0: # process regression ``` This optimization is particularly important in performance-critical code paths, benchmarking systems, and data processing pipelines where loops may process large datasets. Additionally, avoid duplicate processing by leveraging existing classifications or computations rather than recalculating the same values. --- ## Centralize workspace configurations Centralize configuration settings like dependency versions and feature flags at the workspace level rather than duplicating them across individual crate files. This reduces maintenance overhead, prevents inconsistencies, and simplifies dependency management. In a Rust workspace: - Define common dependency versions in the root `Cargo.toml` - Configure feature flags at the workspace level when they apply to multiple crates - Reference workspace configurations in individual crates using the `workspace = true` property Example: ```toml # Root Cargo.toml [workspace.dependencies] schema = { version = "1.0.0", features = ["v3"] } # Individual crate's Cargo.toml [dependencies] schema = { workspace = true } # Inherits version and features automatically ``` This approach prevents situations where developers need to update the same dependency in multiple places or encounter unexpected behavior due to mismatched feature flags. --- ## Consistent naming standards Maintain consistent and standardized naming throughout the codebase: 1. **Use snake_case for multi-word identifiers**: Separate words in variable names, function names, and parameters with underscores. 2. **Be consistent with identifier names**: Use the same name for a concept throughout function signatures, implementations, and documentation. 3. **Choose descriptive, semantic names**: Select names that clearly reflect the purpose and context of the identifier. More specific names improve code clarity and maintainability. Example: ```python # Incorrect def filter(self, predicate: Expr, *, use_abspath: bool = False) -> Series: """ Filter elements by expr """ # implementation using expr # Correct def filter(self, predicate: Expr, *, use_abs_path: bool = False) -> Series: """ Filter elements by predicate """ # implementation using predicate # More descriptive naming MaintainOrderJoin: TypeAlias = Literal["none", "left", "right", "left_right", "right_left"] ``` Consistent naming reduces cognitive load, improves code readability, and helps prevent bugs that could arise from naming confusion. --- ## Extract duplicate code When you notice code patterns being repeated across multiple locations, extract them into reusable functions or constants to improve maintainability and follow DRY principles. Look for these common duplication patterns: - **Identical logic blocks**: Extract into shared functions - **Repeated literal arrays/objects**: Extract into named constants - **Similar data transformations**: Create utility functions - **Complex type conversions**: Move to dedicated helper functions Example from the codebase: ```typescript // Before: Duplicated logic in multiple methods async storeCredentials(data: AuthFile): Promise { const authData: AuthFile = { tokens: this.loadedCredentials } await mkdir(path.dirname(this.authFilePath), { recursive: true }) await writeFile(this.authFilePath, JSON.stringify(authData, null, 2)) } async deleteCredentials(workspaceId: string): Promise { this.loadedCredentials = this.loadedCredentials?.filter((c) => c.workspaceId !== workspaceId) || [] const data: AuthFile = { tokens: this.loadedCredentials } await mkdir(path.dirname(this.authFilePath), { recursive: true }) await writeFile(this.authFilePath, JSON.stringify(data, null, 2)) } // After: Extract shared logic private async writeAuthFile(data: AuthFile): Promise { await mkdir(path.dirname(this.authFilePath), { recursive: true }) await writeFile(this.authFilePath, JSON.stringify(data, null, 2)) } ``` This practice reduces maintenance burden, eliminates inconsistencies, and makes code changes easier to implement across the codebase. --- ## Secure authentication flows Design APIs with secure authentication flows by following proper error handling and documentation practices. Avoid non-null assertions in authorization code, properly document URL format requirements, and provide clear configuration steps for auth providers. For authentication headers, handle potential null values safely: ```typescript // Good: Handle potential null values safely headers: { Authorization: req.headers.get('Authorization') } // Bad: Using non-null assertion can lead to runtime errors headers: { Authorization: req.headers.get('Authorization')! } ``` For redirect URLs, document format requirements explicitly: ``` // Important: Redirect URLs must end with a trailing slash https://example.com/auth/callback/ // Correct https://example.com/auth/callback // May cause authentication failures ``` Remember to document all necessary provider configuration steps: "Enable the provider you want to use under Auth Providers in the Supabase Dashboard and add the necessary credentials." --- ## Use null strategically When handling empty or missing values, be intentional about using `null`, `undefined`, or empty strings based on how downstream systems interpret these values. In particular, when working with database operations (like PostgreSQL), use `null` instead of empty strings for fields that need to be cleared or constraints that should be dropped. ```typescript // Bad: Using empty string or keeping values as-is const comment = ((field.comment?.length ?? '') === 0 ? '' : field.comment)?.trim() const check = field.check?.trim() // Good: Converting empty values to null for database operations const comment = field.comment?.trim() || null const check = field.check?.trim() || null ``` This pattern ensures proper behavior when working with systems that treat null specially. For PostgreSQL specifically, NULL is required to remove comments or constraints, as empty strings are treated differently. Use the nullish coalescing operator (`||`) with null to handle this pattern concisely. --- ## Balance flexibility with performance When designing APIs, carefully balance flexibility against performance constraints. More flexible APIs often come with implementation complexity and potential performance costs. Start by understanding how clients actually use your API, then optimize for those common patterns while providing just enough flexibility for legitimate edge cases. For example, in an RPC service for batch operations, consider whether clients typically need scattered or contiguous page requests: ```protobuf // More flexible but potentially less optimized approach message GetPageRequestBatch { repeated GetPageRequest requests = 1; // Allows arbitrary pages } // More constrained but potentially more optimized approach message GetContiguousPageRequest { uint64 request_id = 1; GetPageClass request_class = 2; ReadLsn read_lsn = 3; RelTag rel = 4; uint32 start_block_number = 5; uint32 block_count = 6; // Enforces contiguous ranges } ``` When evaluating API design decisions, ask: "Is this flexibility actually needed by clients?" As one reviewer noted: "Since the batching exists for performance, I think it's best to limit the freedom. It might allow skipping some overhead." At the same time, consider consistent patterns across similar endpoints to make your API more intuitive and maintainable. --- ## Secure token lifecycle Implement comprehensive lifecycle controls for authentication tokens to maintain security throughout token creation, usage, and deletion processes. Essential practices include: 1. Special handling for privileged tokens (like admin/operator tokens) with appropriate restrictions and user feedback 2. Unique identifiers for each token to prevent duplication or reuse of token IDs 3. Proper validation of token deletion permissions with clear error messages 4. Explicit regeneration paths for tokens that cannot be directly deleted Example implementation for special token handling: ```rust if let Err(e) = client.api_v3_configure_token_delete(&token_name).await { match e { Error::ApiError { code, ref message } => { if code == StatusCode::METHOD_NOT_ALLOWED && message == "cannot delete operator token" { println!( "Cannot delete operator token, to regenerate an operator token, use `influxdb3 create token --admin --regenerate --token $TOKEN`" ); } } _ => return Err(e.into()), } } ``` This approach prevents security vulnerabilities that could arise from improper token management while providing clear guidance to users when special procedures are required. --- ## Performance test pragmatism When designing performance tests, focus on efficiency and meaningful insights rather than exhaustive combinations. Consider these principles: 1. **Limit parameter combinations** to avoid excessive test duration. For example, instead of: ```python @pytest.mark.parametrize( "pipelining_config,ps_io_concurrency,l0_stack_height,queue_depth,name", [ (config, ps_io_concurrency, l0_stack_height, queue_depth, f"{dataclasses.asdict(config)}") for config in LATENCY_CONFIGS for ps_io_concurrency in PS_IO_CONCURRENCY for queue_depth in [1, 2, 3, 4, 16, 32] for l0_stack_height in [0, 3, 10] ], ) ``` Consider testing only critical parameter combinations such as production values and a few boundary cases. 2. **Use robust performance measurement techniques** when comparing optimizations. For significant features, measure performance both with and without the optimization and average multiple runs to account for system variability: ```python # Run each test multiple times and average the results run_times_with_feature = [] run_times_without_feature = [] for _ in range(SAMPLE_SIZE): run_times_with_feature.append(measure_execution(feature_enabled=True)) run_times_without_feature.append(measure_execution(feature_enabled=False)) avg_with_feature = sum(run_times_with_feature) / len(run_times_with_feature) avg_without_feature = sum(run_times_without_feature) / len(run_times_without_feature) ``` 3. **Balance thoroughness with execution time**. Consider randomized parameter selection for exploratory testing when exhaustive testing is impractical, but ensure values are logged for reproducibility. --- ## Pin dependency versions Always specify exact versions for dependencies in your configuration files and import statements to ensure consistent behavior across different environments. Unpinned dependencies can lead to unexpected breaking changes when packages are automatically updated to newer versions. For npm/JavaScript dependencies: ```json { "dependencies": { "elevenlabs": "1.0.5", "@supabase/supabase-js": "2.38.4" } } ``` For imports in Deno/Edge Functions: ```ts import { ElevenLabsClient } from 'npm:elevenlabs@1.0.5'; import { createClient } from 'jsr:@supabase/supabase-js@2.38.4'; ``` For Docker/container configurations: ```dockerfile FROM node:20.9.0-alpine ``` This practice ensures reproducible builds, predictable behavior across development, testing, and production environments, and makes debugging easier by eliminating version inconsistencies as a potential source of problems. --- ## Stage configuration changes gradually When introducing configuration changes that affect multiple system components, implement them in stages to ensure smooth transitions and backward compatibility. Follow these steps: 1. Add new configuration option while maintaining old one 2. Wait for new option to reach release 3. Switch to new option in dependent components 4. Remove old option after full migration Example: ```rust // Step 1: Add new option while keeping old impl Config { fn get_safekeeper_info(&self) -> String { // Support both old and new formats self.safekeeper_connstrings.clone() .or_else(|| self.safekeepers.map(|s| convert_to_new_format(s))) .unwrap_or_default() } } // Step 2: Wait for release with dual support // Step 3: Switch dependent components to new format conf.append("neon.safekeeper_connstrings", &safekeepers); // Instead of old: conf.append("neon.safekeepers", &safekeepers); // Step 4: Remove old option in future release ``` This approach: - Prevents breaking changes in production - Allows gradual testing in staging/pre-prod - Maintains backward compatibility during transition - Reduces deployment risks through phased rollout --- ## Safe database operations When modifying database structures or executing dynamic SQL queries, prioritize both performance and safety: 1. **Use non-blocking operations** for schema changes in production databases. For example, when creating indexes on potentially busy tables, use `CREATE INDEX CONCURRENTLY` instead of standard `CREATE INDEX`: ```typescript // Avoid this - can lock tables during index creation const sql = `CREATE INDEX ON "${schema}"."${entity}" USING ${type} (${columns})`; // Prefer this - allows concurrent writes while building the index const sql = `CREATE INDEX CONCURRENTLY ON "${schema}"."${entity}" USING ${type} (${columns})`; ``` 2. **Properly handle SQL string escaping** when building dynamic queries, especially with date values and other complex types. Be careful about escaping clashes between JavaScript and SQL: ```typescript // Problematic - potential escaping conflicts const sql = `valid until ${literal(validUntil)}`; // Better approach - use format specifiers for proper escaping const sql = `valid until %L`; const formattedSql = format(sql, validUntil); // Or use parameterized queries when possible const result = await client.query('UPDATE users SET valid_until = $1', [validUntil]); ``` These practices help prevent database locks that can impact application performance and avoid SQL injection vulnerabilities while ensuring query correctness. --- ## Connection resilience patterns Implement resilient networking connections with retry mechanisms for all client-service interactions. When establishing connections to external services or databases, include proper retry logic with backoff strategies to handle temporary network instabilities or service unavailability. Monitor connection counts and states across different connection types to detect potential issues. ```typescript // Example implementation of connection retry logic const getClientConnection = async (maxRetries = 3, backoffMs = 500): Promise => { let attempts = 0; while (attempts < maxRetries) { try { const connectionString = await getValidConnectionString(); return new Client(connectionString); } catch (error) { attempts++; if (attempts >= maxRetries) throw error; console.log(`Connection attempt ${attempts} failed, retrying in ${backoffMs}ms...`); await new Promise(resolve => setTimeout(resolve, backoffMs)); // Exponential backoff for next attempt backoffMs *= 2; } } } ``` --- ## Preserve API backward compatibility When modifying existing APIs, ensure that current usage patterns continue to work unchanged. This applies to command-line interfaces, library APIs, REST endpoints, and any client-facing interfaces. Always provide migration paths that maintain existing behavior while introducing improvements. For command-line tools, use positional arguments and optional flags that preserve the original calling convention: ```python # Good: Maintains backward compatibility parser.add_argument('revision', nargs='?', default='HEAD', help='Git revision to check (default: HEAD)') parser.add_argument('directories', nargs='*', help='Directories to format') # This allows both: # python format.py HEAD (original usage) # python format.py --revision HEAD dir1 dir2 (new usage) ``` Before making API changes, identify all existing usage patterns and ensure they remain functional. Document any new capabilities as additive features rather than replacements. This principle helps maintain user trust and reduces friction during upgrades. --- ## Proper synchronization patterns When implementing synchronization mechanisms, avoid common anti-patterns that can lead to performance issues or incorrect behavior. Use proper condition variables with timeouts instead of indefinite or busy waiting, and ensure condition checks occur after wait operations. Key guidelines: 1. **Use timeouts for blocking operations**: Replace indefinite waits with configurable timeouts to prevent threads from blocking forever 2. **Prefer condition variables over busy waiting**: Use `wait()/notify()` mechanisms instead of polling in loops with `Thread.sleep()` 3. **Check conditions after waiting**: Always re-evaluate the condition after a wait operation, as spurious wakeups can occur Example of proper synchronization pattern: ```scala private def awaitProcessThisPartition( id: StateStoreProviderId, timeoutMs: Long): Boolean = maintenanceThreadPoolLock synchronized { val endTime = System.currentTimeMillis() + timeoutMs var canProcessThisPartition = processThisPartition(id) while (!canProcessThisPartition && System.currentTimeMillis() < endTime) { maintenanceThreadPoolLock.wait(timeoutMs) canProcessThisPartition = processThisPartition(id) // Check condition AFTER wait } canProcessThisPartition } ``` This approach prevents resource waste from busy waiting, avoids indefinite blocking, and ensures correct behavior even with spurious wakeups. --- ## maintain formatting consistency Ensure consistent formatting patterns and styles throughout the codebase, both when writing new code and refactoring existing code. When making style improvements or refactoring, preserve the original behavior while applying consistent formatting standards. Key principles: - Use consistent formatting patterns for similar constructs within the same file and across the codebase - When refactoring code for style improvements, maintain the original algorithm and behavior as closely as possible - Establish and follow consistent standards for spacing, string formatting, and code structure Example of inconsistent formatting to avoid: ```python # Inconsistent f-string spacing and multi-line formatting print(f"{i}: ", failure_message["benchmark"]) # Note the space after colon # vs elsewhere in code: print(f"{i}. ", other_message) # Note the period and different spacing # Inconsistent multi-line string formatting print('''single line approach''') # vs print( ''' multi-line approach ''' ) ``` Example of consistent formatting: ```python # Consistent f-string spacing print(f"{i}: {failure_message['benchmark']}") print(f"{j}: {other_message}") # Consistent multi-line formatting approach print(""" ==================================================== ================ FAILURES SUMMARY ================ ==================================================== """) ``` When refactoring substantial portions of code, break changes into separate commits to clearly distinguish between behavioral changes and pure style improvements, making it easier to verify that the original functionality is preserved. --- ## Size fields appropriately When designing database schemas, choose field types and sizes that accommodate both current and anticipated future data volumes. Undersized fields can cause production issues and limit functionality when data grows beyond initial expectations. Additionally, ensure consistency between related fields that store similar data or participate in the same operations. Example: ```sql -- Avoid restrictive size limits for fields that may grow -- Bad: `pos` varbinary(10000) NOT NULL, `stop_pos` varbinary(10000) DEFAULT NULL, -- Better: `pos` mediumblob NOT NULL, -- Up to 16MB `stop_pos` mediumblob DEFAULT NULL, -- Matching type and capacity ``` For fields containing potentially large data (like position coordinates, serialized objects, or binary data), prefer types with sufficient headroom rather than arbitrary size limitations. This prevents operational issues when edge cases are encountered in production. --- ## consistent error object usage Always use proper Error objects when throwing exceptions, maintain consistent error handling contracts, and ensure type safety in error scenarios. Avoid throwing primitive strings, mixing throw/return patterns, or unsafe type assertions on error objects. Key principles: - Throw Error objects, not strings: Use `throw new Error('message')` instead of `throw 'message'` - Maintain consistent function contracts: If a function signature suggests it returns `undefined` on failure, don't throw instead - Use proper typing for error handling: Cast caught exceptions appropriately (`error instanceof Error`) before accessing properties - Handle error causes safely: When unwrapping error causes, verify the error is an Error instance first Example of problematic patterns: ```typescript // Bad: throwing string throw `External error with reported id was not registered` // Bad: inconsistent with function signature that returns undefined lookupError(error: number): ErrorRecord | undefined { if (!errorRecord) throw `External error with reported id was not registered` return errorRecord } // Bad: unsafe destructuring from any catch (e) { const { message } = e // e is any, not type-safe } ``` Example of improved patterns: ```typescript // Good: proper Error object throw new Error('External error with reported id was not registered') // Good: consistent with signature lookupError(error: number): ErrorRecord | undefined { const errorRecord = this.registeredErrors[error] return errorRecord // returns undefined if not found } // Good: type-safe error handling catch (error) { if (error instanceof Error) { return err(error.cause) } throw error } ``` --- ## Optimize data structures Choose and implement data structures with careful consideration of algorithmic complexity, memory usage, and Go's specific performance characteristics. Pay attention to: 1. **Placement of operations in code flow**: Consider when and where operations are performed, especially in loops or before early returns. ```go // Be careful with operations inside loops func (set Mysql56GTIDSet) AddGTIDInPlace(gtid GTID) GTIDSet { // ... for _, iv := range intervals { // ... processing logic ... // Incorrect: Updating data structure in every loop iteration set[gtid56.Server] = newIntervals } // Correct: Update once after loop completes (when appropriate) set[gtid56.Server] = newIntervals } ``` However, be mindful of early returns that might skip operations: ```go // Early returns may require updates within the loop for _, iv := range intervals { if condition { // Update needed here if we might return set[gtid56.Server] = newIntervals return set } } ``` 2. **Efficient implementation choices**: Choose implementations that avoid unnecessary overhead: - Prefer direct binary operations over reflection-based alternatives like `binary.Write` - Consider specialized data structures for specific use cases (e.g., Disjoint Set Union for transitive closures) - Order type assertions from most specific to most general to avoid unnecessary checks 3. **Accurate metrics and counts**: Ensure methods that report on data structure state (like length or size) accurately reflect the true state: - Avoid double-counting elements in queue implementations - Be precise about what your metrics represent (buffer capacity vs. element count) By thoughtfully designing and implementing your data structures, you can significantly improve both performance and maintainability of your code. --- ## Prefer opt-in security When implementing security features that modify data presentation or alter normal data access patterns (like anonymization, masking, or redaction), design them to be explicitly enabled by users rather than activated by default. Default-enabled data transformation creates security and usability concerns as users may not expect or desire their data to be modified. For example, instead of: ``` # Adding anonymization library to default preloaded libraries COPY --from=pg-anon-pg-build /usr/local/pgsql/ /usr/local/pgsql/ # Configuration that auto-enables the feature shared_preload_libraries = '...,anon' ``` Prefer: ``` # Make the library available but not preloaded by default COPY --from=pg-anon-pg-build /usr/local/pgsql/ /usr/local/pgsql/ # Provide documentation or helper functions to enable when needed CREATE OR REPLACE FUNCTION enable_data_masking() RETURNS VOID AS $$ BEGIN PERFORM set_config('session_preload_libraries', current_setting('session_preload_libraries') || ',anon', false); END; $$ LANGUAGE plpgsql SECURITY DEFINER; ``` --- ## Explicit nil handling Always handle nil values explicitly in your code to improve clarity and prevent subtle bugs. When a function needs to deal with potentially nil values: 1. Document the expected behavior when nil values are encountered 2. Use early returns to handle nil cases at the beginning of functions 3. Avoid silent fall-throughs in switch statements when handling nil 4. Know when nil checks are necessary versus when Go handles nil cases automatically For example, instead of using a switch statement with a fall-through for nil: ```go func AppendGTIDInPlace(rp Position, gtid GTID) Position { switch { case gtid == nil: case rp.GTIDSet == nil: rp.GTIDSet = gtid.GTIDSet() default: // Handle non-nil case } // ... } ``` Prefer explicit handling with early returns: ```go func AppendGTIDInPlace(rp Position, gtid GTID) Position { // If gtid is nil, treat it as a no-op and return the input Position. if gtid == nil { return rp } if rp.GTIDSet == nil { rp.GTIDSet = gtid.GTIDSet() } else { // Handle non-nil case } // ... } ``` Also, remember that Go has built-in nil handling for some operations. For instance, append works safely with nil slices: ```go // This is unnecessary if req.FilterRules != nil { bls.Filter.Rules = append(bls.Filter.Rules, req.FilterRules...) } // This is sufficient bls.Filter.Rules = append(bls.Filter.Rules, req.FilterRules...) ``` This approach makes the intention clear, improves code readability, and prevents unexpected behavior. --- ## Defer expensive operations Avoid triggering expensive computations prematurely in your code. Operations like `collect()`, intensive IO operations, or algorithms with quadratic complexity should be deferred until absolutely necessary to prevent wasted computational resources. For example, instead of: ```python # This could trigger heavy computation that gets discarded if strict: nrows = first.select(F.len()).collect()[0, 0] ``` Consider deferring the operation: ```python # Defer the expensive operation to when it's actually needed pl.scan_iceberg(iceberg_path, snapshot_id=1234567890).collect() ``` When optimizing performance-critical code: 1. Identify operations that could be computationally expensive (IO, collect, complex transformations) 2. Move these operations as late as possible in the execution flow 3. Use benchmarks to verify performance improvements and prevent regressions 4. Be especially vigilant with operations that might have quadratic complexity when scaling This approach minimizes resource utilization and significantly improves performance, especially when working with large datasets and complex query plans. --- ## CI workflow configuration best Configure GitHub Actions workflows to maximize reliability and maintainability. Follow these key practices: 1. **Always test with latest patches**: Use `check-latest: true` when setting up language environments to ensure you're testing with the most recent patch versions rather than potentially outdated cached versions. ```yaml - uses: actions/setup-python@v5 with: python-version: '3.12' check-latest: true ``` 2. **Define complete job dependencies**: Ensure publication jobs depend on all build and test jobs to prevent publishing artifacts if any part of the build process fails. ```yaml publish-to-pypi: needs: [create-sdist, build-wheels, build-wheel-pyodide] ``` 3. **Use readable matrix configurations**: Consider using `include` rather than `exclude` when it results in clearer, more maintainable matrix definitions for cross-platform testing. ```yaml matrix: package: [polars, polars-lts-cpu, polars-u64-idx] os: [ubuntu-latest, macos-13] architecture: [x86-64, aarch64] include: - os: windows-latest architecture: x86-64 - os: windows-arm64-16gb architecture: aarch64 ``` These practices help create reliable CI pipelines that catch issues early and prevent problematic releases. --- ## Structured logging best practices Use structured logging with appropriate field types and context to make logs more useful for troubleshooting. Choose specific field types over generic ones, include relevant metrics, and select appropriate log levels based on information importance. **Use appropriate field types:** ```go // Bad: May generate excessively large logs with unpredictable content logger.Info("TSM scheduled for compaction", zap.Any("file", group)) // Good: Clearly conveys the data with appropriate type logger.Info("TSM scheduled for compaction", zap.Strings("files", fileNames)) ``` **Include relevant context:** ```go // Bad: Generic error with minimal context logger.Error("error creating log writer", zap.String("path", path)) // Good: Includes specific error and context logger.Error("error creating log writer", zap.String("path", path), zap.Error(err)) ``` **Select appropriate log levels:** ```go // Summary information at Info level logger.Info("added files", zap.Int("count", len(fileNames))) // Detailed information at Debug level logger.Debug("purging", zap.Strings("files", fileNames)) // Warning for potential issues logger.Warn("waiting for operation", zap.Duration("elapsed", elapsed)) ``` For high-volume events (like dropped points), be selective about what you log to avoid overwhelming logs. Consider logging only the first N occurrences or aggregate information. --- ## Clear configuration parameters Configuration parameters should be descriptively named, well documented, and have sensible defaults that are visible to users. This makes your application more user-friendly and reduces confusion. **Descriptive naming:** - Use precise, unambiguous names that clearly indicate the parameter's purpose - Example: Use `query-file-limit` instead of `file-limit`, or `snapshotted-wal-files-to-keep` instead of `num-wal-files-to-keep` **Document defaults:** - Make default values visible in CLI documentation to match coded defaults - Example: ```rust /// The maximum number of distinct value combinations to hold in the cache #[clap(long = "max-cardinality", default_value = "100000")] max_cardinality: Option, ``` **Use user-friendly units:** - Choose appropriate units for your configurations - Consider using percentages for memory limits to work across different system configurations - For file sizes, use MB instead of bytes when appropriate **Provide helpful documentation:** - Include examples in help text for required parameters - Explain valid types and formats - Add contextual information to help users understand the implications of settings Following these guidelines will reduce support issues and improve the usability of your application's configuration interface. --- ## Prioritize searchable names {% raw %} Choose names that are easily searchable and immediately understandable, avoiding unclear abbreviations and symbols that hinder discoverability. Names should communicate their purpose clearly without requiring domain knowledge to decode. Avoid unclear acronyms that aren't established in your codebase - if "DA" isn't used elsewhere, spell out "Driver Adapter". Similarly, avoid symbols like emojis in contexts where searchability matters, as they make it difficult to find and reference specific components. Example from GitHub workflows: ```yaml # Avoid - unclear acronym name: DA Unit Tests [v${{ matrix.node }}] # Avoid - not searchable name: 🧪 # Better - clear and searchable name: Driver Adapter Unit Tests [v${{ matrix.node }}] name: Tests ``` When space constraints exist, prioritize clarity over brevity. A slightly longer name that clearly communicates purpose is preferable to a short name that requires explanation or context to understand. {% endraw %} --- ## proper async error testing When testing for expected errors in async code, use Jest's built-in async error testing patterns instead of try-catch blocks with expectations inside catch handlers. Expectations in catch blocks can be silently skipped if the code stops failing, leading to false positives. Use `await expect(promise).rejects.toThrow()` or similar matchers: ```typescript // ❌ Avoid - expectation can be skipped if error stops occurring it('should handle error', async () => { await ctx.cli('generate').catch((e) => { expect(e.stderr).toMatchInlineSnapshot(`Error: ...`) }) }) // ✅ Preferred - ensures the promise actually rejects it('should handle error', async () => { const result = ctx.cli('generate') await expect(result).rejects.toThrowErrorMatchingInlineSnapshot(`Error: ...`) }) ``` Additionally, ensure your tests actually verify what they claim to test. For example, when testing that an example file was used, assert that the expected content is present, not just that some generic content exists. For tests that are expected to fail temporarily, use `it.failing()` to make the intent explicit and easier to fix later when the underlying issue is resolved. --- ## Document configuration alternatives When documenting configuration setup, provide multiple formats and approaches to accommodate different tools, platforms, and environments. Include both the base command and tool-specific configuration examples when applicable. This ensures users can successfully configure the system regardless of their specific environment or tooling choices. For example, when documenting MCP server setup, provide both the CLI command and JSON configuration: ```bash npx prisma mcp ``` And the JSON configuration for AI tools: ```json { "mcpServers": { "Prisma": { "command": "npx", "args": ["-y", "prisma", "mcp"] } } } ``` Additionally, clearly document environment-specific behaviors and requirements, such as Windows-specific installation steps or path resolution differences, to help users understand and troubleshoot platform-specific issues. --- ## Avoid unnecessary work When optimizing performance-critical code paths, eliminate redundant operations and unnecessary processing: 1. **Exit loops early** when a decision has been made and no further iterations are needed: ```go // Good: Break out of the loop once we find what we need for _, f := range files { if tsmPointsPerBlock := fileStore.BlockCount(f, 1); tsmPointsPerBlock == aggressivePointsPerBlock { pointsPerBlock = aggressivePointsPerBlock break // No need to check remaining files } } ``` 2. **Cache calculated values** that are used multiple times, especially expensive calculations: ```go // Good: Calculate time bounds once and reuse futureTimeLimit := time.Now().Add(rp.FutureWriteLimit) pastTimeLimit := time.Now().Add(-rp.PastWriteLimit) // Check points against cached limits for _, p := range points { if p.Time().After(futureTimeLimit) || p.Time().Before(pastTimeLimit) { // Point outside time window } } ``` 3. **Skip reprocessing** data that already meets target criteria: ```go // Good: Use >= instead of == to avoid recompacting already optimized files if blockCount >= targetBlockCount && !hasTombstones { // Skip this file, it's already optimized } ``` 4. **Use direct operations** rather than unnecessary function calls for deterministic results: ```go // Good: Use copy directly instead of looping with a function that always returns true shards := make([]*Shard, 0, len(s.shards)) copy(shards, s.shards) ``` These optimizations reduce CPU cycles, memory allocations, and I/O operations, leading to more efficient code execution, particularly in performance-sensitive areas. --- ## Prefer configurable values Always use configurable values instead of hardcoded defaults when available. This ensures that user preferences are respected throughout the application lifecycle and prevents unexpected behavior. When defining related configuration constants, establish their relationship explicitly rather than using magic numbers: ```go // Bad const DefaultMaxPointsPerBlock = 1000 const AggressiveMaxPointsPerBlock = 100000 // Magic number with unclear relationship // Good const DefaultMaxPointsPerBlock = 1000 const AggressiveMaxPointsPerBlock = DefaultMaxPointsPerBlock * 100 // Relationship is clear and maintainable ``` When referencing configuration values in logs or error messages, clearly indicate whether you're using a default or a user-configured value: ```go // Bad e.logger.Info("TSM optimized compaction running", zap.Int("points-per-block", tsdb.DefaultAggressiveMaxPointsPerBlock)) // Good e.logger.Info("TSM optimized compaction running", zap.Int("points-per-block", e.CompactionPlan.GetAggressiveCompactionPointsPerBlock())) // Good - when referencing defaults in messages fmt.Sprintf("points per block count (default: %d points)", DefaultAggressiveMaxPointsPerBlock) ``` Always check against configured values rather than defaults when making decisions in your code. This respects user settings and prevents unexpected behavior when configurations change. --- ## Use descriptive names Names in code should be self-documenting, accurately reflect purpose, and follow consistent conventions. Apply these principles throughout your code: 1. **Function/method names must accurately reflect behavior** ```go // Poor: Name doesn't match behavior (returns true even for running compactions) func FullyCompacted() (bool, string) // Better: Name accurately reflects behavior func CompactionOptimizationAvailable() (bool, string) ``` 2. **Variable names should describe their content** ```go // Poor: Name suggests future action, but tracks completed fields var fieldsToCreate []*FieldCreate // Better: Name reflects actual content var createdFields []*FieldCreate ``` 3. **Use named return values for clarity** ```go // Poor: Purpose of return values unclear func CreateFieldIfNotExists(name string, typ influxql.DataType) (*Field, bool, error) // Better: Return values purpose is clear func CreateFieldIfNotExists(name string, typ influxql.DataType) (f *Field, created bool, err error) ``` 4. **Define constants for literal values** ```go // Poor: Magic strings scattered in code if t.HashedToken != "" { token = "REDACTED" } else { variantName = "N/A" } // Better: Named constants with clear semantics const ( TokenRedacted = "REDACTED" ValueNotAvailable = "N/A" ) ``` 5. **Function naming should align with behavior** ```go // Poor: Name implies it starts a goroutine but doesn't func startFileLogWatcher(w WatcherInterface, e *Executor, ctx context.Context) // Better: Name reflects actual function behavior func fileLogWatch(w WatcherInterface, e *Executor, ctx context.Context) ``` Clear, descriptive naming reduces cognitive load, improves maintainability, and helps prevent bugs by making code behavior more obvious. --- ## WebSocket lifecycle management Ensure proper WebSocket connection lifecycle management by using `once()` instead of `on()` for cleanup operations and separating initialization from construction to avoid race conditions. When setting up WebSocket connections, use `once()` for events that should only happen once, particularly for cleanup operations: ```typescript // Good: Use once() for cleanup events ws.once('open', () => startHeartBeats()); ws.once('close', () => clearInterval(heartbeatTimer)); ws.once('close', () => clearTimeout(missedPingTimer)); // Avoid: Using on() for one-time cleanup ws.on('close', () => clearInterval(heartbeatTimer)); // Could fire multiple times ``` Separate connection initialization from construction to prevent race conditions where close handlers might execute before the connection is properly registered: ```typescript // Good: Two-phase initialization const connection = new Connection(..., onCloseHandler); this.connections.set(clientID, connection); connection.init(); // Now safe to close if needed // Avoid: Initialization in constructor that might close immediately ``` This pattern prevents resource leaks, ensures cleanup operations execute exactly once, and avoids timing issues where connections might be closed before being properly tracked. --- ## Design runtime-specific API exports When designing APIs that need to work across different JavaScript runtimes (Node.js, edge environments, browsers), create explicit export configurations that account for runtime-specific implementations. Different runtimes may require different API implementations that aren't interchangeable, so provide separate entry points rather than trying to create a one-size-fits-all solution. Use package.json exports to define runtime-specific entry points: ```json { "exports": { ".": { "require": { "types": "./dist/index-node.d.ts", "default": "./dist/index-node.js" }, "import": { "types": "./dist/index-node.d.mts", "default": "./dist/index-node.mjs" } }, "./web": { "require": { "types": "./dist/index-web.d.ts", "default": "./dist/index-web.js" }, "import": { "types": "./dist/index-web.d.mts", "default": "./dist/index-web.mjs" } } } } ``` Make runtime-specific implementations opt-in rather than automatic to avoid confusion. Plan for ecosystem testing to verify that exports work correctly in target environments like Cloudflare Workers and Vercel Edge. When selecting dependencies, prefer libraries that provide ergonomic abstractions while maintaining broad runtime compatibility. --- ## Extract shared code patterns Identify and extract repeated code patterns into reusable functions to improve maintainability and reduce duplication. When similar code appears in multiple places, create a shared helper function that captures the common logic. Key guidelines: - Extract code that differs only in a few parameters - Create well-named helper functions that clearly describe their purpose - Pass varying elements as parameters - Consider creating dedicated types for related helpers Example - Before: ```go func (tc *TransitiveClosures) Add(exprA, exprB sqlparser.Expr, comp string) { // Duplicate logic for handling expressions if tc.setA.contains(exprA) { // Complex logic A } if tc.setB.contains(exprB) { // Complex logic A repeated } } ``` After: ```go func (tc *TransitiveClosures) handleExpr(expr sqlparser.Expr, set *exprSet) { // Shared logic extracted if set.contains(expr) { // Complex logic in one place } } func (tc *TransitiveClosures) Add(exprA, exprB sqlparser.Expr, comp string) { tc.handleExpr(exprA, tc.setA) tc.handleExpr(exprB, tc.setB) } ``` This pattern reduces maintenance burden, improves readability, and makes the code easier to modify since changes only need to be made in one place. --- ## comprehensive test coverage Ensure thorough test coverage by systematically testing edge cases, boundary conditions, failure scenarios, and different data types. When implementing new functionality, consider all possible input variations, error conditions, and integration points. Key areas to cover: - **Boundary values**: Test minimum, maximum, and overflow conditions for numeric types - **Edge cases**: Empty inputs, null values, special characters, and unusual data patterns - **Failure conditions**: Invalid inputs, type mismatches, and error scenarios - **Data type variations**: Test across different compatible types and type conversions - **Integration scenarios**: Test interactions with related features and nested operations Use systematic approaches like foreach loops to ensure comprehensive coverage: ```sql foreach type varchar query T SELECT list_contains([1,2,3], $1::$2) ---- # expected results endloop ``` Examples of comprehensive test cases: - For hex parsing: test empty hex (`'0x'::INT`), case variations (`'0XFF'::INT`), overflow conditions (`'0xFFFFFFFFFFFFFFFFF'::INT`), and exact boundary values - For string operations: test strings longer than 12 characters (handled differently in DuckDB) - For null handling: test null inputs, null in collections, and null propagation - For new parameters: test across related features (e.g., if implementing for Parquet, also test for CSV) This approach helps achieve near 100% code coverage and prevents regression issues by thoroughly exercising all code paths. --- ## avoid quadratic complexity When processing collections, be mindful of time complexity and avoid accidentally creating O(N²) algorithms, especially when simpler O(N) alternatives exist. A common anti-pattern is using `reduce()` with object spread operations or similar approaches that create new objects in each iteration. This can lead to quadratic time complexity where linear complexity would suffice. **Example of problematic O(N²) code:** ```javascript const keysPerRow = rows.map((item) => response.keys.reduce((acc, key) => ({ [key]: item[key], ...acc }), {}) ) ``` **Improved O(N) alternative:** ```javascript const keysPerRow = rows.map((item) => { const obj = {} for (const key of response.keys) { obj[key] = item[key] } return obj }) ``` Additionally, implement early returns for empty collections to avoid unnecessary processing: ```javascript if (queries.length === 0) { return [] } ``` When choosing between implementation approaches, consider the computational complexity implications. Even micro-optimizations like using enums instead of strings can have measurable performance benefits in hot code paths, though the primary focus should be on avoiding algorithmic inefficiencies that scale poorly with input size. --- ## Centralize configuration logic Avoid scattering configuration defaults, validation, and loading logic across multiple functions. Instead, centralize these concerns in dedicated configuration modules or at the entry points of your application. **Problems with scattered configuration:** - Default values redefined in multiple places lead to inconsistency - Configuration loading logic duplicated across different modules - Harder to maintain and debug configuration-related issues **Best practices:** 1. **Centralize defaults at entry level** - Define default values once at the top-level configuration rather than in each function that needs them 2. **Extract configuration logic into dedicated functions** - Move complex configuration loading and validation into separate, testable functions 3. **Consolidate environment variable handling** - Handle environment variable loading in a single place rather than scattered throughout the codebase **Example of the problem:** ```typescript // Scattered defaults - AVOID function buildClient({ target = 'nodejs' }) { /* ... */ } function generateClient({ target = 'nodejs' }) { /* ... */ } function processConfig({ target = 'nodejs' }) { /* ... */ } ``` **Better approach:** ```typescript // Centralized configuration - PREFER const DEFAULT_CONFIG = { target: 'nodejs', // other defaults... }; function buildClient({ target = DEFAULT_CONFIG.target }) { /* ... */ } function generateClient(options = DEFAULT_CONFIG) { /* ... */ } ``` This approach makes configuration changes easier to manage, reduces the risk of inconsistent defaults, and improves code maintainability by having a single source of truth for configuration values. --- ## prefer environment variables {% raw %} When configuring behavior that needs to work across different execution contexts (CI workflows, manual runs, different build systems), prefer environment variables over command-line flags or hardcoded options. Environment variables provide broader compatibility and can be easily overridden without modifying scripts or commands. Additionally, ensure all environment variables are properly defined and have clear dependencies. Avoid referencing undefined variables or creating confusing variable chains. Example: ```yaml # Instead of hardcoded command-line flags: run: build/reldebug/test/unittest --force-reload --force-storage --summarize-failures # Prefer environment variables that can be read by the application: env: SUMMARIZE_FAILURES: 1 run: build/reldebug/test/unittest --force-reload --force-storage # Ensure variables are properly defined: env: ENABLE_EXTENSION_AUTOLOADING: 1 # Clear, defined value # Not: ENABLE_EXTENSION_AUTOLOADING: ${{ inputs.undefined_variable }} ``` This approach makes configuration more flexible and allows the same settings to work with different execution methods like `make allunit` or direct test execution. {% endraw %} --- ## Validate sensitive operations Always implement safety checks before performing operations that could expose sensitive data or cause destructive changes. This includes validating targets before file deletion and sanitizing credentials in error messages. For destructive file operations, verify the target contains expected artifacts: ```typescript // Before deleting, ensure directory contains expected generated files if (!files.includes('client.d.ts')) { throw new Error( `${outputDir} exists and is not empty but doesn't look like a generated Prisma Client. ` + 'Please check your output path and remove the existing directory if you indeed want to generate the Prisma Client in that location.', ) } ``` For credential handling, redact sensitive information from logs and error messages: ```typescript function redactFailedCommand(message: string) { // remove the connection url that follows '--datasource' from the given `message`. // Note: if the command isn't properly formed, i.e., no connection url follows `--datasource`, we risk redacting an irrelevant part of the command } ``` This prevents accidental data loss and credential exposure, two common security vulnerabilities that can have severe consequences. --- ## optimize algorithmic performance Prioritize algorithmic efficiency and avoid unnecessary computational overhead, especially in type-level operations and validation logic. When multiple approaches achieve the same result, choose the one with better performance characteristics. Key optimization strategies: 1. **Avoid expensive validation when unnecessary** - Skip complex validation for cases that cannot fail 2. **Use direct property access over type inference** - Prefer `obj['_']['property']` over complex type inference patterns 3. **Choose simpler algorithms when equivalent** - Replace complex recursive or iterative approaches with direct lookups when possible Example of optimization: ```typescript // Inefficient: Complex validation that always passes const schema = type('Record') // Optimized: Direct type assertion since validation cannot fail const schema = type.object.as>() // Inefficient: Complex type inference type Result = TValue extends Interface ? TResult : never // Optimized: Direct property access type Result = TValue['_']['result'] ``` This is particularly critical in type-level operations where "performance is our bottleneck in general" and unnecessary complexity can significantly impact compilation times and developer experience. --- ## Explicit role security management Always be explicit about role privileges when configuring database security. Remember that both `authenticated` and `anon` roles typically have default privileges that need to be explicitly revoked. Use SQL migrations rather than GUI tools to manage role permissions for better version control and reproducibility. ```sql -- Example: Revoking update privileges from both authenticated and anon roles revoke update on table public.posts from authenticated; revoke update on table public.posts from anon; -- Then grant specific column-level privileges as needed grant update (title, content) on table public.posts to authenticated; ``` When implementing Row Level Security policies with JWT claims, ensure proper SQL syntax to extract JWT values. Use subqueries when referencing JWT values in comparisons: ```sql -- Correct way to reference JWT values in RLS policies create policy "Only organization admins can insert" on table_name to authenticated with check ( (auth.jwt()->>'org_role' = 'org:admin') and (organization_id = (select auth.jwt()->>'org_id')) ); ``` This approach ensures that your database access controls are precisely defined and consistently maintained through version-controlled code. --- ## intuitive API method design Design API methods with intuitive names and signatures that follow established conventions and provide good ergonomics. Avoid method names that conflict with well-known patterns from other domains, and prefer streamlined method signatures that reduce unnecessary verbosity. Key principles: - Avoid overloading method names that have established meanings (e.g., `.then()` is associated with Promises) - Design method signatures to accept parameters directly when possible, rather than requiring wrapper functions - Consider alternative naming that better reflects the method's purpose Example of problematic naming: ```typescript // Confusing - .then() suggests Promise-like behavior caseWhen(condition).then(value) // Better - clearer intent when(condition, value) ``` Example of improved method signature: ```typescript // Verbose - requires unnecessary type wrapper literalSchema.or(type('unknown.any[] | Record')) // Streamlined - accepts definition directly literalSchema.or('unknown.any[] | Record') ``` This approach makes APIs more discoverable, reduces cognitive load for developers, and prevents confusion with established patterns from other libraries or language features. --- ## Concise performance documentation When documenting performance metrics, benchmarks, or scalability information, prioritize clarity and conciseness. Avoid redundant wording, use consistent terminology when listing features, and structure information in a scannable format. Clear performance documentation helps developers understand system capabilities, set appropriate expectations, and make informed optimization decisions. Example: ```diff - The proposed workloads are designed to demonstrate Supabase Realtime's throughput and scalability capabilities. These benchmarks focus on core functionality and common usage patterns. + The proposed workloads are designed to demonstrate Supabase Realtime's throughput and scalability. These benchmarks focus on core functionality and common usage patterns. - This guide explores the scalability of Realtime's features: Broadcast, Presence performance, and Postgres Changes. + This guide explores the scalability of Realtime's features: Broadcast, Presence, and Postgres Changes. ``` Clear performance documentation is critical for performance optimization efforts as it establishes baseline expectations and provides the context needed for future optimizations. --- ## Organize tests efficiently Write maintainable, well-structured tests that are easy to understand and extend. Tests should remain simple and focused on their specific validation purpose. Three key practices to follow: 1. **Use pytest parametrization** to express multiple test scenarios concisely rather than duplicating similar test functions: ```python # Instead of multiple similar test functions: def test_unique_counts_on_bool_only_true() -> None: s = pl.Series("bool", [True, True, True]) expected = pl.Series("bool", [3], dtype=pl.UInt32) assert_series_equal(s.unique_counts(), expected) def test_unique_counts_on_bool_only_false() -> None: s = pl.Series("bool", [False, False, False]) expected = pl.Series("bool", [3], dtype=pl.UInt32) assert_series_equal(s.unique_counts(), expected) # Prefer a single parametrized test: @pytest.mark.parametrize( ("input", "expected"), [ ([True, True, True], [3]), ([False, False, False], [3]), ([True, False, False, True, True], [3, 2]), ] ) def test_unique_counts_bool(input: list[bool], expected: list[int]): assert_series_equal( pl.Series("bool", input).unique_counts(), pl.Series("bool", expected, dtype=pl.UInt32) ) ``` 2. **Avoid complex logic in test cases**. Each test should be straightforward and focused on specific validation. If a test contains intricate logic, consider refactoring into multiple simpler tests. 3. **Use appropriate test fixtures and environments**: - Use the `tmp_path` fixture for file operations instead of hardcoded paths - Prefer in-memory testing with `io.BytesIO()` over disk operations when possible for better performance - Ensure tests are isolated and don't interfere with each other --- ## Cache performance preservation When implementing database failover or restart mechanisms, ensure performance consistency by preserving and prewarming caches. Database performance can significantly degrade after restarts due to cold caches (buffer pools, file system cache, query plan cache), especially for large workloads. Implement a cache prewarming strategy: 1. Periodically persist cache state to external storage (e.g., S3) 2. During failover, load cache state into the new instance before accepting traffic 3. Ensure clean handover between primaries during failover with proper sequencing: - Avoid concurrent primaries on the same timeline - Complete promotion before routing traffic - Validate replay of all committed transactions Example code for cache state persistence: ```rust struct ComputeSpec { // ...existing fields /// Whether to do auto-prewarm at start pub lfc_auto_prewarm: bool, /// Interval in seconds for periodic cache dumps pub lfc_dump_interval_sec: Option } // In the cache manager fn periodic_cache_dump() { if let Some(interval) = config.lfc_dump_interval_sec { schedule_task(Duration::from_secs(interval as u64), || { // Dump cache state to persistent storage store_cache_state_to_s3(); }); } } ``` For read-heavy workloads, consider implementing hot secondaries that can handle read traffic while maintaining cache warmth, providing both scaling benefits and faster failover. --- ## Document connection transitions When implementing systems that involve network connection state changes (such as during failovers, restarts, or component promotions), explicitly document and implement connection handling strategy for all affected components. Key considerations: 1. Identify which component is responsible for terminating existing connections during transitions 2. Document the sequence of connection-related operations during state changes 3. Implement safeguards against stale reads from outdated connections 4. Consider connection invalidation mechanisms for proxies and clients For example, when designing a compute promotion flow: ``` 3.1. Terminate the primary compute. Starting from here, this is a critical section. 3.2. Send cache invalidation message to all proxies, notifying them that all new connections should request and wait for the new connection details. 3.3. Ensure proxies drop any existing connections to the old primary to prevent stale reads. ``` This approach prevents inconsistent state during transitions and makes connection handling behavior explicit rather than implicit, reducing the risk of connection-related bugs that can be difficult to diagnose in distributed systems. --- ## Scope JWT authentication tokens Always include tenant, timeline, and endpoint identifiers in JWT tokens used for service authentication. This ensures proper isolation between tenants and prevents unauthorized access across endpoints or timelines, even within the same tenant hierarchy. Without proper token scoping, services cannot validate that requests come from authorized sources with the correct endpoint_id, creating potential security vulnerabilities where one endpoint could access another endpoint's data. Implementation guidelines: 1. Control plane should generate fully-scoped JWT tokens 2. Include endpoint_id, tenant_id, and timeline_id in token claims 3. Use structured paths in storage that correspond to these identifiers 4. Services should validate all relevant identifiers before granting access Example token generation: ```rust // INSECURE: Token lacks proper scope let token = generate_jwt_token(claims! { "tenant_id": tenant_id, // Missing endpoint and timeline identifiers! }); // SECURE: Token includes all necessary identifiers let token = generate_jwt_token(claims! { "tenant_id": tenant_id, "timeline_id": timeline_id, "endpoint_id": endpoint_id, "exp": expiration_time }); ``` Example path structure for storage access: ``` s3:///tenant-/tl-// ``` This approach protects against cross-tenant and cross-timeline attacks, as services can validate that requesters can only access resources matching their token scope. --- ## optimize hot path performance Avoid expensive operations in frequently executed code paths by implementing performance optimizations such as lookup tables, result caching, conditional initialization, and object reuse. Key optimization strategies: 1. **Replace expensive algorithms with faster alternatives**: Use lookup tables instead of linear searches like `std::find_first_of` when checking for special characters in strings. 2. **Cache redundant computations**: Avoid performing the same expensive checks multiple times by caching results in appropriate data structures like `unsafe_unique_array`. 3. **Minimize initialization overhead**: Only perform expensive initialization when necessary (e.g., after state changes) rather than on every iteration or function call. 4. **Reuse expensive objects**: Make costly-to-create objects like compiled regex patterns static to avoid recreation overhead. Example of lookup table optimization: ```cpp // Instead of: std::find_first_of(string_data, string_end, special_chars, special_chars + special_chars_length) // Use a lookup table for O(1) character checking: static bool special_char_lookup[256] = {false}; // Initialize once // Check: if (special_char_lookup[static_cast(ch)]) ``` Example of conditional initialization: ```cpp // Instead of initializing on every chunk: static void MultiFileScan(...) { InitializeFileScanState(context, data, gstate.projection_ids); // Every call } // Initialize only when needed: static void MultiFileScan(...) { if (file_changed || !initialized) { InitializeFileScanState(context, data, gstate.projection_ids); } } ``` These optimizations can provide significant performance improvements, with reported speedups of 2x or more in critical code paths. --- ## Optimize large field queries When working with database queries that process large text fields or arrays, choose operations that minimize data conversion and processing overhead. This can lead to dramatic performance improvements. For checking text length in PostgreSQL queries, use `octet_length()` instead of `length()` when dealing with large fields: ```sql -- Less efficient (could be 100x slower for large fields) when length(field_name::text) > 10240 then ... -- More efficient when octet_length(field_name::text) > 10240 then ... ``` In benchmark testing with ~50MB text fields, this change reduced query execution time from ~11 seconds to under 100ms by avoiding full text conversion. For array columns, implement size limits to prevent excessive data processing: ```sql case when octet_length(array_column::text) > ${maxSize} then (select array_cat(array_column[1:${maxArraySize}]::text[], array['...']))::text[] else array_column::text[] end ``` These techniques are particularly important in data-intensive applications where rendering large text fields or arrays could create bottlenecks. --- ## Avoid skipping e2e tests Do not use the `skip_e2e` flag to bypass end-to-end tests that fail. Instead, fix the test implementation by providing appropriate test data or modifying the test to make it pass correctly. This maintains the integrity of the test suite and ensures that code changes are properly validated. When adding new test cases, especially those involving complex operations like joins that require specific verification conditions, include the necessary sample data: ```go // Instead of this: { "comment": "Between clause on a binary vindex field with values from a different table", "query": "select s.oid,s.col1, se.colb from sales s join sales_extra se on s.col1 = se.cola where s.oid between se.start and se.end", "plan": { // Plan details... }, "skip_e2e": true // DON'T DO THIS } // Do this: // 1. Add necessary sample data // Example: Add data loading for test tables // 2. Remove the skip_e2e flag { "comment": "Between clause on a binary vindex field with values from a different table", "query": "select s.oid,s.col1, se.colb from sales s join sales_extra se on s.col1 = se.cola where s.oid between se.start and se.end", "plan": { // Plan details... } // No skip_e2e flag - test will run } ``` This approach ensures comprehensive test coverage and helps catch issues that might otherwise go undetected in production code. Maintaining a policy of not adding `skip_e2e` flags to new tests encourages better test design and more robust code. --- ## verify authorization before operations Always verify that users have proper authorization to access and modify resources before performing any data operations. This prevents privilege escalation attacks and unauthorized data access. Key principles: 1. **Check visibility first**: Ensure users can see/access the resource before allowing any operations on it 2. **Validate operation permissions**: Verify the user has specific permissions for the intended operation (read, write, delete) 3. **Prevent impersonation**: Never allow users to change ownership or creator fields to other users, as this constitutes impersonation Example implementation: ```typescript async update(tx, change: UpdateValue) { // First verify user can access the resource await assertIsAdminOrCreator(tx, tx.query.issue, change.id); // Then perform the operation await tx.mutate.issue.update(change); } // In permissions schema select: [ (authData, {exists}) => exists('issue', q => q.where(eb => canSeeIssue(authData, eb))), ] ``` This pattern prevents scenarios where users could perform operations on resources they cannot see due to permission changes, and ensures consistent security enforcement across all data access patterns. --- ## avoid redundant cache lookups When implementing cache functionality, avoid performing the same cache lookup multiple times. Instead, store and reuse the result from the first lookup to improve performance. This is particularly important in scenarios where you need to check if content is cached (e.g., in `isContentCached`) and then immediately perform the actual cache operation (e.g., in `nextImpl`). Rather than doing separate `contains()` and `getOrSet()` calls, perform `getOrSet()` once and store the result for subsequent use. Example approach: ```cpp // Instead of checking cache twice: bool isContentCached(size_t offset, size_t size) { return cache->contains(key); // First lookup } bool nextImpl() { auto result = cache->getOrSet(key, ...); // Second lookup } // Store the result from first lookup: bool isContentCached(size_t offset, size_t size) { cached_result = cache->getOrSet(key, ...); // Single lookup, store result return cached_result != nullptr; } bool nextImpl() { // Reuse cached_result instead of looking up again } ``` This pattern reduces cache overhead and improves overall system performance, especially in high-throughput scenarios where cache operations are frequent. --- ## Document versioning strategies Establish and clearly document versioning strategies in configuration files, both for your application and its dependencies. For application versioning, include detailed explanations of each component in the version string: ``` v3.M.m-B[.Q.q.b] # eg, v3.0.0-0.beta.1.0 or v3.0.0-1 | | | | | | | | | | | | | ----------> package build number for 'quality' | | | | | ------------> 'quality' release number | | | | --------------> release quality (optional: alpha, beta, rc) | | | -----------------> package build number | | -------------------> micro version (non-breaking changes) | ---------------------> minor version (breaking changes) -----------------------> major version (hugely breaking changes) ``` For dependencies in CI/CD configurations, make deliberate decisions about whether to pin specific versions (`tool@v1.2.3`) or use floating references (`tool@latest`). Document the reasoning behind your choice, and regularly review dependencies with floating versions to ensure they remain stable. When using `@latest`, verify that upstream maintainers handle versioning responsibly to prevent unexpected breaking changes in your build pipeline. --- ## Pin environment versions Always use explicit version tags for CI/CD environments (runners, containers, images, tool versions) instead of floating references like 'latest'. This practice prevents unexpected breakages when upstream environments are updated and gives the team control over when to upgrade. When upgrading is necessary, do it in a dedicated PR to properly test the changes without blocking other development work. Example: ```yaml jobs: build: # Good: Explicitly pinned version runs-on: ubuntu-22.04 # Avoid: Floating version tag # runs-on: ubuntu-latest test: container: # Good: Pinned image version image: python:3.11.4 # Avoid: Floating tag # image: python:latest ``` --- ## Manage complete cache lifecycle Implement comprehensive cache lifecycle management focusing on three key aspects: 1. Idempotent Creation: Make cache creation idempotent by returning success for identical configurations and error only for conflicting ones. This enables reliable automation and prevents redundant cache instances. Example: ```rust impl Cache { fn create(&self, config: CacheConfig) -> Result<(), Error> { match self.get_existing_config(&config.name) { Some(existing) if existing == config => Ok(()), // Identical config Some(_) => Err(Error::ConflictingConfig), // Different config None => { self.insert_new_cache(config); Ok(()) } } } } ``` 2. Efficient Maintenance: Schedule cache maintenance operations (pruning, eviction) at appropriate intervals: - Perform heavy operations like eviction during low-activity periods (e.g., during snapshots) - Use reasonable intervals for routine checks (e.g., once per second vs every 10ms) - Hold locks for minimum duration needed 3. Complete Cleanup: Implement thorough cleanup that removes all artifacts when entries expire: - Remove expired values from cache stores - Clean up empty key entries in hierarchical caches - Walk up cache trees to remove empty branches - Consider using weak references for temporary entries This ensures optimal memory usage and prevents accumulation of stale metadata. --- ## API abstraction levels Functions and utilities should operate at appropriate abstraction levels without being aware of higher-level concepts or implementation details. This prevents tight coupling and improves reusability and testability. When designing APIs, ensure that: - Utilities take abstract parameters rather than concrete types from higher layers - Functions accept processed inputs instead of calling internal functions themselves - Conditional logic is abstracted into proper interfaces rather than scattered throughout classes **Example:** Instead of making a utility aware of a specific `Client` type: ```typescript // ❌ Bad - utility knows about high-level Client concept export function createCompositeProxy(client: Client, target: T, layers: CompositeProxyLayer[]): T { // ... getPrototypeOf: () => Object.getPrototypeOf(client._originalClient), } ``` Use an abstract parameter that provides only what's needed: ```typescript // ✅ Good - utility works with abstract prototype provider export function createCompositeProxy(prototypeProvider: {}, target: T, layers: CompositeProxyLayer[]): T { // ... getPrototypeOf: () => Object.getPrototypeOf(prototypeProvider), } ``` Similarly, abstract scattered conditionals into proper interfaces: ```typescript // ❌ Bad - conditionals scattered throughout class if (TARGET_BUILD_TYPE === 'rn') { // React Native specific logic } // ✅ Good - abstracted into library interface const library = await libraryLoader.load() await library.startTransaction(options) ``` This approach makes code more modular, testable, and maintainable by ensuring each component operates at its appropriate level of abstraction. --- ## Favor clarity over brevity Always prioritize code readability and maintainability over concise but cryptic implementations. Extract repeated logic into well-named helper functions, use keyword-only arguments for clearer APIs, and structure complex string operations for better readability. For repeated code: ```python # Instead of this: if engine == "auto" and get_engine_affinity() != "auto": engine = get_engine_affinity() # Extract into a reusable function: def _select_engine(engine: EngineType) -> EngineType: return get_engine_affinity() if engine == "auto" else engine # Then use it: engine = _select_engine(engine) ``` For clearer APIs, use keyword-only arguments: ```python # Instead of this: def from_buffer(self, dtype: PolarsDataType, endianness: Endianness = "little"): ... # Prefer this: def from_buffer(self, *, dtype: PolarsDataType, endianness: Endianness = "little"): ... ``` For complex string operations, prefer readable structures: ```python # Instead of complex concatenation: categories = ( [",".join(f"{cat!r}" for cat in categories[:3])] + ["…"] + [",".join(f"{cat!r}" for cat in categories[-3:])] ) # Use more readable structure: categories = [ ",".join(repr(cat) for cat in categories[:3]), "…", ",".join(repr(cat) for cat in categories[-3:]), ] ``` --- ## Database API abstraction When designing database interaction layers, carefully consider when to create wrapper methods versus allowing direct use of underlying libraries. Add abstraction only when it provides significant value through error handling, feature consolidation, or version compatibility management. For instance, instead of implementing a simple wrapper method like: ```python def write_iceberg(self, table, mode: Literal["append", "overwrite"]): data = self.to_arrow() if mode == "append": table.append(data) else: table.overwrite(data) ``` Consider whether users could just as easily write: ```python # Direct usage without wrapper table.append(df.to_arrow()) # or table.overwrite(df.to_arrow()) ``` Create wrappers when they provide meaningful value, such as handling version compatibility: ```python if parse_version(cx.__version__) < (0, 4, 2): # Handle older version with specific parameters tbl = cx.read_sql( conn=connection_uri, query=query, return_type="arrow2", # other parameters ) else: # Handle newer version with additional features tbl = cx.read_sql( conn=connection_uri, query=query, return_type="arrow", pre_execution_query=pre_execution_query, # other parameters ) ``` This approach keeps your database layer clean and maintainable while still providing convenience where it truly adds value. When implementing database operations, aim for the right balance between abstraction and simplicity. --- ## avoid password conversions When handling sensitive data like passwords, avoid unnecessary type conversions that create additional copies in memory. Pass char[] arrays directly to methods that accept them instead of converting to String and back to char[]. This minimizes the number of sensitive data copies in memory and reduces the attack surface. Example of what to avoid: ```java // Problematic: creates unnecessary String copy byte[] passwordBytes = ScramFormatter.normalize(new String(password).toCharArray()); ``` Preferred approach: ```java // Better: pass char[] directly byte[] passwordBytes = ScramFormatter.normalize(password); ``` This practice is important because String objects are immutable and remain in memory until garbage collected, while char[] arrays can be explicitly cleared after use. --- ## Explicit configuration precedence Implement a clear configuration resolution chain that follows a consistent precedence pattern: explicit parameters first, then environment variables, then default values. This ensures predictable behavior and proper respect for user-provided settings at different levels. When a parameter can be configured through multiple means (function parameters, environment variables, config files), default it to `None` in function signatures and explicitly handle the resolution sequence in your code: ```python def process_data(engine=None): # Check explicit parameter first if engine is None: # Then check environment variable engine = get_engine_from_environment() if engine is None: # Finally use the default value engine = "cpu" ``` This approach offers several benefits: - Users can understand and predict which configuration source takes precedence - Function calls without explicit parameters will respect environment configurations - You can track the source of a configuration setting for better error messages - The code clearly documents the fallback behavior - It becomes easier to add new configuration sources without breaking existing code When documenting these parameters, clearly communicate the precedence chain so users understand how their configurations will be applied. --- ## Evaluate algorithmic complexity tradeoffs When implementing algorithms, carefully evaluate tradeoffs between performance optimizations and code maintainability. Consider: 1. Early vs Late Optimization: - Assess whether complexity can be handled at initialization vs runtime - Example: For search operations, evaluate if preprocessing (like sorting) justifies the overhead 2. Memory Allocation Patterns: - Minimize unnecessary allocations and copies - Look for opportunities to reuse existing buffers 3. Algorithm Selection: - Choose algorithms based on expected data characteristics - Consider special cases that could enable optimizations Example of evaluating tradeoffs in search implementation: ```rust // Consider tradeoffs between approaches: // Approach 1: Simple but has runtime overhead fn index_of_predicate(ca: &ChunkedArray, predicate: P) -> Option { // O(n) scan with predicate check per element for (idx, val) in ca.iter().enumerate() { if predicate(val) { return Some(idx); } } None } // Approach 2: More complex but constant overhead fn index_of(value: T) -> Option { // O(1) special case check upfront if value.is_special_case() { return handle_special_case(); } // O(n) scan for normal case linear_scan(value) } ``` The choice between approaches should consider: - Expected data distributions - Frequency of operation - Maintenance complexity - Memory usage patterns --- ## ensure comprehensive test coverage Tests should validate all relevant code paths and actually exercise the functionality they claim to test. When adding tests for new features, ensure they stress the new behavior rather than just existing functionality that was already covered. For functions with multiple execution paths (like optional parameters), write test cases that cover each path: ```python def test_try_to_timestamp(self, spark): df = spark.createDataFrame([('1997-02-28 10:30:00',), ('2024-01-01',), ('invalid')], ['t']) # Test default format (format=None) res = df.select(F.try_to_timestamp(df.t).alias('dt')).collect() # Test explicit format res_fmt = df.select(F.try_to_timestamp(df.t, 'yyyy-MM-dd HH:mm:ss').alias('dt')).collect() ``` When testing new functionality, use inputs that specifically exercise the new behavior rather than inputs that would work with the old implementation. Avoid test cases where "your string is just 'i' this was already covered by the old behavior" - instead, create test cases that would fail without the new functionality. --- ## Promote code clarity Write code that prioritizes clarity and maintainability over brevity. This involves several key practices: 1. **Extract repeated code blocks into helper functions** When code is used in multiple places, move it to a dedicated function with a descriptive name: ```rust // Instead of repeating this logic: if let Some(path) = output_file_path { let mut f = OpenOptions::new() .write(true) .create(true) .truncate(true) .open(path) .await?; f.write_all_buf(&mut bs).await?; } else { if output_format.is_parquet() { Err(Error::NoOutputFileForParquet)? } println!("{}", String::from_utf8(bs.as_ref().to_vec()).unwrap()); } // Create a helper function: async fn write_output(bs: &mut Bytes, output_format: &OutputFormat, output_path: &Option) -> Result<()> { // ... same logic here ... } ``` 2. **Use constants for hardcoded values** Replace magic strings and numbers with named constants: ```rust // Instead of: let python_exe = if cfg!(target_os = "windows") { /* ... */ } // Use: const PYTHON_EXECUTABLE: &str = "python3"; const PYTHON_EXECUTABLE_WINDOWS: &str = "python"; let python_exe = if cfg!(target_os = "windows") { PYTHON_EXECUTABLE_WINDOWS } else { PYTHON_EXECUTABLE }; ``` 3. **Design clear function interfaces** Use descriptive parameter names and separate compound parameters when clarity is improved: ```rust // Instead of: async fn cleanup_after_snapshot(&self, cleanup_after_snapshot: Option<(oneshot::Receiver, SnapshotInfo, OwnedSemaphorePermit)>) // Use: async fn cleanup_after_snapshot( &self, snapshot_finished_receiver: oneshot::Receiver, snapshot_info: SnapshotInfo, snapshot_permit: OwnedSemaphorePermit, ) ``` 4. **Respect encapsulation** Create proper accessor methods rather than exposing internals: ```rust // Instead of: let triggers = result.catalog().inner().read().triggers(); // Add a method to Catalog: impl Catalog { pub fn triggers(&self) -> Vec { self.inner().read().triggers() } } // Then use: let triggers = result.catalog().triggers(); ``` 5. **Use clear destructuring** Simplify complex destructuring to improve readability: ```rust // Instead of: let Self { data, meta } = self; let ObjectMeta { location, last_modified: _, size: _, e_tag, version, } = meta; // Use: let Self { data, meta: ObjectMeta { location, last_modified: _, size: _, e_tag, version, } } = self; ``` 6. **Organize code logically** Group related functionality into modules and keep single-purpose code in dedicated files. By consistently applying these practices, code becomes more readable, maintainable, and less prone to errors. --- ## Prevent concurrent access races When sharing data across goroutines, always use proper synchronization mechanisms to prevent race conditions. Race conditions are difficult to debug and can cause intermittent failures that only appear under load. **Key practices:** 1. **Always protect mutex unlocking with defer**: ```go func() { mu.Lock() defer mu.Unlock() // Code that might panic or return early }() ``` 2. **Clone data structures before sharing across goroutines**: ```go // When parallelism > 1, clone to prevent concurrent modifications if parallelism > 1 { resp = resp.CloneVT() } ``` 3. **Use atomic operations for simple flags**: ```go var rowAdded atomic.Bool // Later in code executed by multiple goroutines if len(rresult.Rows) > 0 { rowAdded.Store(true) } // Check safely if rowAdded.Load() { // ... } ``` 4. **Return copies of slices to prevent external mutation**: ```go func (ml *MemoryLogger) LogEvents() []*logutilpb.Event { ml.mu.Lock() defer ml.mu.Unlock() return slices.Clone(ml.Events) } ``` 5. **Always clean up resources with defer**: ```go ctx, cancel := context.WithCancel(ctx) defer cancel() // Prevents context leaks ``` 6. **Consider message queues for channels with unbounded consumers**: ```go // For high-throughput messaging where backpressure is a concern mq := concurrency.NewMessageQueue[*TabletHealth]() go func() { for { th, validRes := mq.Receive() if validRes { c <- th } } }() ``` 7. **Use `chan struct{}` over `chan bool` for signaling**: ```go // More efficient and idiomatic for simple signaling semAcquiredChan := make(chan struct{}) // When signaling semAcquiredChan <- struct{}{} ``` Run tests with the `-race` flag to catch race conditions during development. --- ## Clear metric documentation When adding, modifying, or deprecating metrics, ensure comprehensive and clear documentation. Include: 1. Descriptive names for new metrics that reflect their purpose 2. Clear explanations of what each metric measures 3. The dimensions/labels available for the metric 4. Explicit mention of any deprecated metrics and their replacements For example, when documenting a new metric: ``` ### VTGate Metrics Added two new metrics with query type, plan type and tablet type as dimensions: 1. QueryProcessed - This counts the number of queries processed 2. QueryRouted - This counts the number of shards the query was executed on Deprecated: 1. QueriesProcessed - Replaced by QueryProcessed 2. TableMetrics - Replaced by QueryTables ``` This practice ensures users can effectively monitor their systems, understand metric changes, and maintain accurate dashboards and alerts as your observability tools evolve. --- ## Prevent nil dereferences Always verify that pointers, slices, or arrays are non-nil and have sufficient elements before attempting to access their members. Use appropriate checks in the correct order to prevent runtime panics. For pointers: ```go // Do this if a != nil { a.Token = "value" // Safe access } // Not this a.Token = "value" // Potential panic if a is nil ``` For arrays and slices: ```go // Do this - check length before indexing if len(gens) > 0 { size := gens[0].size() // Safe access } // Not this size := gens[0].size() // Potential panic if gens is empty ``` When checking for nil slices, remember that `len()` already handles nil slices correctly (returning 0), so separate nil checks are often redundant: ```go // This is sufficient if len(fns) == 0 { // handle empty case } // Not this - redundant check if fns == nil || len(fns) == 0 { // handle empty case } ``` However, if you do need both checks for clarity or other reasons, always check for nil first: ```go // Do this - nil check first to avoid potential panic if fns == nil || someOtherCondition(fns) { // handle nil case } ``` This practice helps prevent common runtime errors like nil pointer dereferences and index out of range panics. --- ## Use testify assertion libraries Replace manual if-error checks with `testify`'s `assert` and `require` packages to make tests more readable, maintainable, and with better error messages. **When to use each:** - Use `assert` for non-critical checks that should report failures but continue testing - Use `require` for checks that should halt the test if they fail **Key improvements:** 1. **Replace manual comparisons with appropriate assertions** ```go // Before if !ok || v != data { t.Errorf("Cache has incorrect value: %v != %v", data, v) } // After assert.True(t, ok, "Value should be found in cache") assert.Equal(t, data, v, "Cache should return correct value") ``` 2. **Split compound conditions into separate assertions** ```go // Before if !triggered1 || !triggered2 { t.Errorf("not all matching listeners triggered") } // After assert.True(t, triggered1, "First listener should be triggered") assert.True(t, triggered2, "Second listener should be triggered") ``` 3. **Use type-compatible assertions to avoid unnecessary casting** ```go // Before if got := int(h.Records()[0].(mod10)); got != 1 { t.Errorf("h.Records()[0] = %v, want %v", got, 1) } // After assert.EqualValues(t, 1, h.Records()[0].(mod10)) ``` 4. **Use specialized assertions for common operations** ```go // Before if len(values) != 1 { t.Errorf("Expected exactly one value, got %d", len(values)) } // After require.Len(t, values, 1, "Should have exactly one value") ``` 5. **Use appropriate error validation** ```go // Before if err == nil { t.Fatalf("Expected error but got none") } // After require.Error(t, err, "Operation should return an error") // Or to check specific error text require.ErrorContains(t, err, "expected error message") ``` Using these assertion functions consistently improves readability, provides better error messages, and makes tests more maintainable. --- ## Robust network handling Always implement proper network timeout handling and address formatting to ensure robust connectivity across different network conditions and IP protocols. **Timeouts:** - Use explicit timeouts for all network operations rather than relying on indefinite waits - Leverage existing timeout constants (like `RemoteOperationTimeout`) instead of hardcoding arbitrary values - Pass timeout parameters explicitly to connection operations **Address formatting:** - Use `net.JoinHostPort()` when combining host and port values to ensure compatibility with both IPv4 and IPv6 - Be explicit about network binding addresses and understand the implications (127.0.0.1 for localhost-only vs 0.0.0.0 for all interfaces) Example of proper implementation: ```go // GOOD: Using proper timeout and address formatting ctx, cancel := context.WithTimeout(context.Background(), config.RemoteOperationTimeout) defer cancel() // Using net.JoinHostPort for proper IPv4/IPv6 handling address := net.JoinHostPort(host, port) conn, err := grpc.DialContext(ctx, address, grpc.WithBlock()) // BAD: Hardcoded timeout and manual address formatting ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel() // Manual concatenation doesn't handle IPv6 properly address := host + ":" + port conn, err := grpc.DialContext(ctx, address, grpc.WithBlock()) ``` This approach prevents connection hangs during network issues while ensuring compatibility across IP protocol versions. --- ## Explicit security parameters Security-critical features should be implemented as required parameters rather than optional parameters or option functions. This forces developers to make explicit decisions about security settings, preventing accidental omissions that could introduce vulnerabilities. For example, when implementing authentication features like token hashing: ```go // Good: Security parameter is required, must be explicitly set func NewStore(ctx context.Context, kvStore kv.Store, useHashedTokens bool) (*Store, error) { // Implementation ensures token hashing is explicitly specified } // Risky: Security parameter could be accidentally omitted func NewStore(ctx context.Context, kvStore kv.Store, opts ...StoreOption) (*Store, error) { // If WithHashedTokens option is forgotten, could create security vulnerability } ``` This pattern ensures developers cannot accidentally omit critical security choices and makes security-relevant decisions more visible during code review. By making security parameters explicit, you enforce conscious decisions about security features rather than relying on defaults or optional configurations that might be overlooked. --- ## verify authorization permissions Ensure that authorization checks use the appropriate permission level for the specific operation being performed. Operations that only read data should use READ permissions, while operations that modify data should use WRITE permissions. Using overly permissive authorization can lead to privilege escalation vulnerabilities. For example, when implementing authorization for share group operations: ```scala // Incorrect - using WRITE for a read operation if (!authHelper.authorize(request.context, WRITE, GROUP, groupId)) { // Correct - using READ for a read operation if (!authHelper.authorize(request.context, READ, GROUP, groupId)) { ``` Always verify that the permission level matches the actual data access pattern of the operation to maintain proper access control boundaries. --- ## Log levels and clarity Choose appropriate log levels and write clear, meaningful log messages that provide necessary context without creating noise. Follow these guidelines: 1. Use appropriate log levels: - ERROR: For issues requiring immediate attention - WARNING: For potentially problematic situations - INFO: For significant state changes or important operations - Avoid logging routine operations at INFO level 2. Include relevant context in messages: - Add identifiers (e.g., workflow names, IDs) - Explain the action being attempted, not just the result - Use structured formatting for complex objects (%+v) Example of good logging: ```go // Bad log.Info("Operation failed") log.Info("Starting operation") // Good log.Errorf("Failed to revert denied tables for workflow %v: %v", workflowName, err) log.Infof("Starting VReplication player id: %v, settings: %+v", id, settings) ``` 3. Avoid duplicate logging: - Don't log the same information at multiple levels - Choose the most appropriate single level for each message 4. Consider log readers: - Write messages that make sense without deep system knowledge - Include enough context to understand the operation's purpose --- ## Lock with defer unlock Always follow the lock-defer-unlock pattern when protecting shared resources with mutexes. Acquire the lock, immediately use defer to ensure unlock, then perform operations on the protected resource. This prevents deadlocks and ensures locks are released even when functions exit unexpectedly. When returning data from mutex-protected resources, always create a deep copy before returning to prevent race conditions after the lock is released. ```go // Bad func (s *Store) ClearBadShardList() map[uint64]error { s.badShards.mu.Lock() // Missing defer - risk of not unlocking if code between lock and unlock panics if s.badShards.shardErrors == nil { s.badShards.shardErrors = make(map[uint64]error) } badShards := maps.Clone(s.badShards.shardErrors) clear(s.badShards.shardErrors) s.badShards.mu.Unlock() return badShards } // Good func (s *Store) ClearBadShardList() map[uint64]error { s.badShards.mu.Lock() defer s.badShards.mu.Unlock() if s.badShards.shardErrors == nil { s.badShards.shardErrors = make(map[uint64]error) } // Clone before returning to prevent race conditions badShards := maps.Clone(s.badShards.shardErrors) clear(s.badShards.shardErrors) return badShards } ``` For read-only operations, consider using RWMutex with RLock() to allow concurrent reads. This approach increases concurrency while maintaining thread safety. --- ## Vet security-critical dependencies When introducing new dependencies, especially those handling sensitive operations like language interpreters, perform comprehensive security due diligence before adoption. Evaluate each dependency against these security criteria: 1. **Maintenance health**: Check for active development, large user base, and responsive contributors. 2. **Security track record**: Review CVE history to confirm security issues are promptly addressed. 3. **Security integration**: Verify integration with vulnerability tracking systems like GitHub advisories. 4. **Contingency planning**: Document alternatives if the dependency becomes unmaintained. For critical dependencies, document your findings in a security assessment that includes: - Vulnerability history analysis - Update frequency and responsiveness - Embedded component maintenance strategy (if applicable) - Security monitoring approach Example from assessment: ```rust // Security-vetted dependency [dependencies.pyo3] version = "0.23.3" // Includes fix for CVE-2024-9979 ``` Always maintain awareness of dependency status. When a critical dependency becomes unmaintained (as happened with PyOxidizer), promptly implement your contingency plan by switching to maintained alternatives (like using pyo3 with python-build-standalone). --- ## Document complete data flows When documenting database systems, ensure all documentation includes complete end-to-end data flows. Both diagrams and textual descriptions should clearly indicate: 1. Where and how user requests enter the system 2. The complete path data takes through system components 3. How responses/confirmations are returned to clients 4. Precise timing and conditions for internal processes For diagrams: - Include entry points for user requests with clear directional arrows - Link diagram components to numbered steps in textual descriptions - Show all components involved in request processing For technical descriptions: - Include client notification mechanisms (like the oneshot channel that gets called back after flush operations) - Provide accurate details about internal timing conditions (like exactly when WAL periods trigger snapshots) - Clarify edge cases and exceptions to normal flows Example improvement: ``` // Before: Incomplete diagram missing user entry point ┌────────────┐ ┌────────────┐ │flush buffer│──────────►│ wal buffer │ └────────────┘ └────────────┘ // After: Complete diagram showing user entry point and process flow User Write │ ▼ ┌────────────┐ ┌────────────┐ │flush buffer│──────────►│ wal buffer │ (Step 1: Buffer incoming writes) └────────────┘ └────────────┘ │ ▼ (Step 2: Flush to disk) ``` Complete documentation helps new team members understand the system and makes debugging easier during incidents. --- ## Environment-portable configuration management Ensure all configurations are environment-portable and follow current best practices for the target platforms. This includes: 1. Using environment variables instead of hardcoded paths or user-specific values 2. Keeping base images and dependencies updated to supported versions 3. Adapting installation methods to platform-specific best practices 4. Verifying package availability across target environments **Examples:** Instead of: ```bash rm -rf /Users/username/vtdataroot/* ``` Use: ```bash rm -rf "${VTDATAROOT:-./vtdataroot}"/* ``` For Docker configurations, regularly review base images: ```dockerfile # Review regularly to keep updated FROM --platform=linux/amd64 golang:1.23.4-bookworm # Research platform-specific installation methods RUN echo "deb http://repo.mysql.com/apt/debian/ bookworm mysql-8.4-lts" > /etc/apt/sources.list.d/mysql.list ``` When packages change across OS versions, document and adapt: ``` # For bookworm, use these packages instead of the deprecated 'etcd' apt-get install -y etcd-client etcd-server ``` --- ## Standardize error wrapping patterns Use consistent error wrapping patterns to preserve error context and ensure proper error code propagation. Always: 1. Use vterrors.New/Errorf when creating new errors to include error codes 2. Use vterrors.Wrapf when wrapping existing errors to maintain the error chain 3. Use errors.As for type checking to handle error wrapping chains 4. Apply specific error codes (e.g. VT13001) for known error categories Example: ```go // Don't do this: if err != nil { return fmt.Errorf("failed to delete workflow %s: %v", name, err) } // Do this instead: if err != nil { return vterrors.Wrapf(err, "failed to delete workflow %s", name) } // Don't do this: if _, ok := err.(ConfigFileNotFoundError); ok { return true } // Do this instead: var configErr ConfigFileNotFoundError if errors.As(err, &configErr) { return true } // Don't do this: return nil, fmt.Errorf("opcode: %v not supported", opcode) // Do this instead: return nil, vterrors.VT13001(fmt.Sprintf("opcode: %v not supported", opcode)) ``` --- ## Limit concurrent access slots Design concurrency mechanisms based on actual usage patterns rather than theoretical maximum connections. When implementing locking or state tracking for concurrent operations, limit the number of concurrent slots to a reasonable value (e.g., 128) that reflects the expected concurrency level, rather than scaling to the maximum number of possible backends (max_connections). This approach reduces memory overhead and lock contention. For example, instead of: ```c // Creating state tracking for every possible backend prewarm_worker_state[MaxConnections]; ``` Use a more efficient approach: ```c // Limit concurrent slots to what's actually needed #define MAX_CONCURRENT_PREWARMING 128 prewarm_worker_state[MAX_CONCURRENT_PREWARMING]; ``` This is especially effective when the operation is quick (as with prewarming or prefetching), making it unlikely that all backends would need concurrent access. The same principle applies to other synchronization mechanisms like locks, semaphores, or condition variables. --- ## Handle errors by criticality Choose error handling strategies based on operation criticality: 1. For critical operations that could corrupt data or state: - Use assertions/panics to fail fast - Example: Validating snapshot details before WAL deletion ```rust // Critical operation - use assert assert_eq!(snapshot_details, details, "Snapshot mismatch before WAL deletion"); ``` 2. For recoverable operations: - Return Result/Error types - Implement retry with backpressure - Log appropriately (warn for temporary, error for persistent issues) ```rust // Recoverable operation - return Result async fn persist_data(&self) -> Result<(), Error> { loop { match self.try_persist().await { Ok(_) => return Ok(()), Err(e) => { warn!("Temporary persist error, retrying: {}", e); tokio::time::sleep(Duration::from_secs(1)).await; } } } } ``` 3. For race conditions and invariant violations: - Use panic/assert when they indicate programming errors - Return errors when they represent expected edge cases This approach ensures critical errors fail fast while allowing recovery from transient failures. --- ## Avoid flaky test patterns Write reliable tests by avoiding common patterns that can lead to flaky behavior. Specifically: 1. Avoid arbitrary timeouts and sleeps in tests. Instead, use proper test infrastructure or wait for specific conditions. 2. Don't rely on snapshot testing for non-deterministic data like serialized maps. 3. Use proper test abstractions rather than low-level timing controls. Example of problematic code: ```rust #[test] async fn test_telemetry() { let store = TelemetryStore::new().await; tokio::time::sleep(Duration::from_secs(1)).await; // Flaky! assert!(store.duration > 0); } ``` Better approach: ```rust #[test] async fn test_telemetry() { let store = TelemetryStore::new().await; // Use TestServer or similar infrastructure let server = TestServer::spawn().await; // Waits for actual readiness assert!(store.is_ready()); // Or implement explicit condition checking wait_for_condition(|| store.duration > 0, "store duration").await; } ``` For serialization testing, prefer explicit equality checks over snapshot comparisons when dealing with non-deterministic data structures. --- ## Use testify assertions Always use the testify package (require/assert) in tests instead of standard Go testing assertions. The testify library provides more descriptive failure messages that make debugging test failures easier and faster. Replace standard testing patterns with testify equivalents: ```go // Instead of this: if err := s.CreateShard("db0", "rp0", 1, true); err != nil { t.Fatal(err) } if len(s.Store.GetBadShardList()) != 0 { t.Fatalf("expected empty list, got %v", s.Store.GetBadShardList()) } if !reflect.DeepEqual(expected, actual) { t.Fatalf("values don't match") } // Use these testify equivalents: require.NoError(t, s.CreateShard("db0", "rp0", 1, true), "creating shard") require.Empty(t, s.Store.GetBadShardList(), "bad shard list should be empty") require.Len(t, s.Store.GetBadShardList(), 1, "should have exactly one bad shard") require.Equal(t, expected, actual, "values should match") ``` Also use modern testing utilities for resource management: - Use `t.TempDir()` instead of manual directory creation and cleanup - Use `t.Cleanup()` for all resource cleanup instead of defer statements These patterns make tests more maintainable, provide better error messages when tests fail, and ensure proper cleanup even when tests panic or fail early. --- ## Manage workflow state transitions When working with temporal workflows, always implement explicit state transitions rather than abrupt deletions. Workflows should proceed through well-defined states (like running, stopped, or frozen) to ensure proper lifecycle management and durable execution. Instead of directly deleting workflows: ```go // Not recommended if _, derr := s.WorkflowDelete(ctx, &vtctldatapb.WorkflowDeleteRequest{ Keyspace: req.TableKeyspace, Workflow: req.Name, KeepData: true, KeepRoutingRules: true, }); derr != nil { return nil, vterrors.Errorf(vtrpcpb.Code_FAILED_PRECONDITION, "failed to delete workflow %s: %v", req.Name, derr) } ``` Implement proper state transitions: ```go // Recommended // First stop the workflow _, err = s.tmc.UpdateVReplicationWorkflow(ctx, tabletInfo.Tablet, &tabletmanagerdatapb.UpdateVReplicationWorkflowRequest{ Workflow: req.Name, State: ptr.Of(binlogdatapb.VReplicationWorkflowState_Stopped), }) if err != nil { return vterrors.Wrapf(err, "failed to stop workflow %s on shard %s/%s", req.Name, tabletInfo.Keyspace, tabletInfo.Shard) } // Then mark workflow as frozen query := fmt.Sprintf(SqlFreezeWorkflow, Frozen, encodeString(tabletInfo.DbName()), encodeString(req.Name)) _, err = s.tmc.VReplicationExec(ctx, tabletInfo.Tablet, query) ``` Commands that operate on workflows should check and enforce preconditions related to workflow state. As seen in the discussion about `complete` and `cancel` commands, make sure each command has clear semantics regarding workflow state requirements. --- ## Document configuration precedence When implementing multiple configuration methods (e.g., config files, command-line flags, environment variables), clearly document the precedence order and warn users about potential conflicts. Unclear precedence rules can lead to unexpected system behavior and difficult troubleshooting. For example, if command-line flags override config file settings (or vice versa), this relationship should be explicitly stated in documentation. Consider adding runtime warnings when potentially conflicting configuration methods are detected: ```go // Example of warning about conflicting configuration methods if configFileProvided && legacyConfigFlagProvided { log.Warning("Both --config-file and --legacy-config flags detected. " + "Values from --legacy-config will take precedence over config file values. " + "Please migrate to using only --config-file as --legacy-config is deprecated.") } ``` When introducing new configuration approaches, mark older methods as deprecated with clear migration instructions to help users transition smoothly. This helps maintain backward compatibility while encouraging adoption of improved configuration patterns. --- ## Redact sensitive credentials When implementing authentication or handling credentials, always redact sensitive information (keys, tokens, passwords) in logs and debug output. Using placeholder text like "" or obfuscation techniques ensures credentials aren't accidentally exposed through logs, which could lead to security breaches. The example demonstrates using a mapping function to replace the actual credential content with a redacted placeholder: ```rust if verbose { eprintln!( "[CloudOptions::build_azure]: azure_cli_account_key: {:?}", azure_cli_account_key.as_ref().map(|_| "") ); } ``` This pattern ensures that even when debugging is enabled, sensitive credentials remain protected. Implementing this practice consistently across the codebase helps prevent accidental exposure of authentication details in log files, console output, or error reports that might be shared during troubleshooting. --- ## Wrap errors with context Always wrap errors with meaningful context using fmt.Errorf and %w verb. Include relevant identifiers (filenames, IDs, paths) in error messages to aid debugging. This helps pinpoint error sources and simplifies troubleshooting. Example: ```go // Bad: if err := os.Open(path); err != nil { return err // Original context is lost } // Good: if err := os.Open(path); err != nil { return fmt.Errorf("failed to open config file %q: %w", path, err) } // Bad: func (s *Store) SetShardNewReadersBlocked(shardID uint64, blocked bool) error { if sh == nil { return ErrShardNotFound // Missing context } } // Good: func (s *Store) SetShardNewReadersBlocked(shardID uint64, blocked bool) error { if sh == nil { return fmt.Errorf("shard not found: id=%d blocked=%v: %w", shardID, blocked, ErrShardNotFound) } } ``` Key points: - Use fmt.Errorf with %w to wrap errors while preserving the error chain - Include relevant identifiers (paths, IDs) in error messages - Add operation context to clarify what failed - Preserve the original error for error type checking with errors.Is() --- ## Consistent database APIs Design database APIs with consistent patterns for response structures and error handling. Follow established conventions in your codebase, even when it means returning empty response objects for error-only operations. For RPC or service endpoints: 1. Return structured response objects consistently, even when empty 2. Avoid duplicating fields across response structures 3. Maintain a predictable error handling pattern **Example:** ```protobuf // Good: Follows convention of always having a response message message ValidatePermissionsKeyspaceRequest { string keyspace = 1; repeated string shards = 2; } message ValidatePermissionsKeyspaceResponse { // Even if initially empty, this maintains API consistency // and allows for future extension without breaking changes } // Avoid: Duplicating fields across structures message StopReplicationAndGetStatusResponse { replicationdata.StopReplicationStatus status = 2; // Don't add backup_running here if it already exists in status } ``` This pattern creates more maintainable database interfaces that can evolve without breaking changes, while keeping a consistent experience for developers consuming your API. --- ## Include explanatory examples Always enhance documentation with concrete, illustrative examples that demonstrate expected inputs, formats, or outputs. Examples significantly improve comprehension and reduce ambiguity, especially for complex parameters, commands, or return values. When documenting: - CLI commands: Show sample invocations with typical arguments - Parameters: Provide examples of valid formats and constraints - Functions: Include examples of return values or expected output - Data structures: Demonstrate typical usage patterns For instance, instead of simply stating: ```rust /// Which columns in the table to use as keys in the cache #[clap(long = "key-columns")] ``` Enhance it with format information and an example: ```rust /// Which columns in the table to use as keys in the cache (comma-separated list) /// Example: --key-columns="customer_id,region,timestamp" #[clap(long = "key-columns")] ``` Similarly, when documenting format expectations or naming rules: ```rust /// The database name to create (alphanumeric with underscore and dash, /// must start with a letter or number) /// Example: "production_metrics" or "test-db-2" #[clap(required = true)] ``` For code variables that might be unclear, add a comment with sample output: ```rust // Format: "{package_name}-{version}", e.g. "influxdb3-3.0.1" let influx_version = format!("{}-{}", influxdb_pkg_name, influxdb_pkg_version); ``` Examples bridge the gap between abstract documentation and practical application, making your code more accessible and easier to use correctly. --- ## consistent naming patterns Maintain consistent naming conventions across similar constructs in your codebase. This includes using consistent prefixes for related methods and following established patterns for file imports and extensions. When you have similar methods or functions that perform related operations, use consistent naming patterns. For example, if you have a `createJoin` method, related methods should follow the same pattern like `createSetOperator` rather than just `setOperator`. Similarly, maintain consistency in file import conventions. If your project uses explicit file extensions in imports, apply this pattern consistently across all imports. Example of consistent method naming: ```typescript // Good - consistent "create" prefix private createJoin(...) private createSetOperator(...) // Inconsistent - mixed naming patterns private createJoin(...) private setOperator(...) ``` Example of consistent import conventions: ```typescript // Good - consistent .ts extensions import type { ColumnBaseConfig } from '~/column.ts'; import { entityKind } from '~/entity.ts'; // Inconsistent - missing extension import type { ColumnBaseConfig } from '~/column'; ``` Consistent naming patterns improve code readability, reduce cognitive load for developers, and make the codebase easier to navigate and maintain. --- ## Cross-platform feature flags When documenting package installation commands with feature flags, ensure compatibility across different operating systems. Windows handles quoted package specifications differently than Unix-based systems, which can lead to installation failures. For maximum compatibility in documentation, use double quotes or provide OS-specific instructions: ```bash # Most compatible approach across platforms pip install "polars[feature]" # Or provide platform-specific examples when needed # For Unix/MacOS: pip install 'polars[feature]' # For Windows: pip install polars[feature] ``` This approach ensures users can successfully install packages with feature flags regardless of their operating system, reducing confusion and installation errors. When providing environment-specific configuration instructions (like for CUDA 11 systems), clearly indicate which environments require additional steps and separate them from general instructions. --- ## Database type consistency Ensure database-specific types, imports, and serialization are used consistently throughout the codebase. This prevents runtime errors, improves type safety, and maintains proper data handling across different database systems. Key areas to verify: 1. **Use correct database-specific imports**: Import types and utilities from the appropriate database core module (e.g., `mysql-core`, `pg-core`, `sqlite-core`) rather than mixing imports from different databases. 2. **Handle data type serialization properly**: Convert problematic types like BigInt to appropriate formats before serialization to avoid "Do not know how to serialize a BigInt" errors. 3. **Manage timezone and date handling consistently**: Override default database driver parsers when needed to ensure consistent date/timestamp behavior across environments. Example of incorrect usage: ```typescript // Wrong - using PostgreSQL types for MySQL import { AnyPgColumn, AnyPgTable } from 'drizzle-orm/mysql-core'; // Wrong - BigInt serialization issue declare global { interface BigInt { toJSON(): Number; } } ``` Example of correct usage: ```typescript // Correct - use appropriate database types import { AnyMySqlColumn, AnyMySqlTable } from 'drizzle-orm/mysql-core'; // Correct - handle BigInt conversion in serializer if (typeof defaultValue === 'bigint') { defaultValue = defaultValue.toString(); } // Correct - override timestamp parsing for consistency const transparentParser = (val: any) => val; for (const type of ['1184', '1082', '1083', '1114']) { client.types.setTypeParser(type, transparentParser); } ``` This ensures type safety, prevents serialization errors, and maintains consistent behavior across different database systems and environments. --- ## Document function signatures Always document function parameters and return values in the function header comment or interface definition. This is particularly important when a function returns multiple values or when the meaning of return values might not be immediately obvious. Example: ```go // PlanOptimize performs optimization planning and returns three values: // - []CompactionGroup: groups of files to compact // - int64: estimated bytes that will be written // - int64: the generation length func PlanOptimize() ([]CompactionGroup, int64, int64) ``` By clearly documenting what each return value represents, you make the code more maintainable and reduce the cognitive load for other developers who need to use your functions. Documentation should be placed in the function header comment rather than as a separate comment within the function body. --- ## avoid redundant lookups When working with associative containers (sets, maps, unordered_set, unordered_map), avoid performing redundant lookups by using single operations that combine lookup and insertion/modification. The common anti-pattern is to first check if an element exists with `find()`, then perform a separate `insert()` operation, which results in two hash table lookups. **Anti-pattern:** ```cpp if (result.find(index) != result.end()) { throw InternalException("Duplicate table index found"); } result.insert(index); ``` **Preferred approach:** ```cpp const bool is_new = result.emplace(index).second; if (!is_new) { throw InternalException("Duplicate table index found"); } ``` For sets, use `emplace()` or `insert()` with `.second` to check if insertion occurred. For maps, use `emplace()`, `try_emplace()`, or `insert_or_assign()` depending on the desired semantics. This optimization reduces algorithmic complexity from O(2) to O(1) hash table operations per element, which can significantly improve performance in tight loops or when processing large datasets. The same principle applies to other associative containers and operations - always prefer single operations that provide the information you need rather than separate lookup and modification steps. --- ## Database configuration best practices When implementing or modifying database-related configurations, follow these principles: 1. Use semantically appropriate types for configuration parameters (e.g., `Duration` instead of `float` for time values) 2. Streamline configuration by consolidating related settings (e.g., control transaction behavior through modes rather than multiple flags) 3. Include comprehensive documentation with links to related PRs or issues when documenting database feature changes **Example:** ```go // Preferred approach config.RegisterFlag(Duration("twopc_abandonage", time.Hour, "time to wait before abandoning transactions")) // Instead of config.RegisterFlag(Float("twopc_abandonage", 3600.0, "seconds to wait before abandoning transactions")) config.RegisterFlag(Bool("twopc_enable", false, "enable two-phase commit")) ``` For documentation: ```markdown ### Added support for LAST_INSERT_ID(expr) Added support for `LAST_INSERT_ID(expr)` to align with MySQL behavior, enabling session-level assignment of the last insert ID via query expressions. For more information about this change see [#17295](https://github.com/vitessio/vitess/pull/17295). ``` This approach ensures settings are intuitive to configure, prevents redundancy, and provides clear paths to additional context about database functionality. --- ## Create demonstrative examples Include clear, concise examples in documentation that effectively demonstrate functionality. Follow these principles for better documentation examples: 1. Include specific examples for each new parameter or feature 2. Show both input and output when demonstrating transformations 3. Use examples that show meaningfully different results to illustrate behavior 4. Keep examples simple but comprehensive enough to demonstrate functionality Example of an effective documentation example: ```python # Good example - shows different results with before/after from datetime import datetime s = pl.Series("datetime", [datetime(2024, 1, 1), datetime(2024, 1, 2)]) # Original series print(s) # shape: (2,) # Series: 'datetime' [datetime[μs]] # [ # 2024-01-01 00:00:00 # 2024-01-02 00:00:00 # ] # After replacement - shows varied results result = s.dt.replace(year=2022) print(result) # shape: (2,) # Series: 'datetime' [datetime[μs]] # [ # 2022-01-01 00:00:00 # 2022-01-02 00:00:00 # ] ``` Users learn more effectively from examples than from descriptions alone. Well-constructed examples that demonstrate real usage patterns significantly improve API usability. --- ## Use consistent temporal types When implementing or modifying temporal operations, maintain consistent data types that align with existing temporal functions. Specifically, use Int8 for datetime components with small ranges (months, days, hours, minutes, seconds) and Int32/Int64 for larger values (years, microseconds). This approach reduces unnecessary type casting, improves code readability, and ensures consistency with temporal functions like `.month()` that return specific types. Example: ```rust // Recommended approach - consistent types let mut month = month.cast(&DataType::Int8)?; let month = month.i8()?; // Instead of // let mut month = month.cast(&DataType::UInt32)?; // let month = month.u32()?; ``` --- ## Edge case algorithm handling When implementing algorithms, pay special attention to edge cases, particularly empty collections. Define and document how your algorithm behaves in these boundary conditions following consistent mathematical or programming conventions. Different paradigms handle edge cases differently. For example, when aggregating empty collections: - Set theory and Python conventions might return a default value (e.g., sum of empty collection = 0) - SQL conventions might return NULL or None Example from Polars data processing library: ```python # Polars follows Python conventions for empty collections pl.Series([], dtype=pl.Int32).sum() # Returns 0 # This differs from SQL which would return NULL # Document these differences when your algorithm implementation # might surprise users familiar with other systems ``` Document your chosen approach clearly, especially when it differs from related systems. This improves predictability and prevents unexpected behavior when algorithms operate at boundary conditions. Consider adding explicit tests for edge cases to verify consistent handling across your system. --- ## track migration state immediately Ensure migration state is recorded in the database immediately after each migration file is successfully applied, rather than batching all state updates at the end. This prevents inconsistent states where migrations are partially applied but not tracked, leaving users unable to determine which migrations have been executed. When a migration process fails partway through, users should be able to resume from the last successfully applied migration. If state tracking is deferred until all migrations complete, a failure will leave the database in an unknown state. Example of problematic pattern: ```typescript // BAD: State tracking deferred until end for await (const migration of migrations) { if (!lastDbMigration || Number(lastDbMigration.created_at) < migration.folderMillis) { for (const stmt of migration.sql) { await db.session.execute(sql.raw(stmt)); } rowsToInsert.push(sql`insert into ${migrationsTable}...`); // Deferred } } // Execute all inserts at end - if this fails, no state is tracked ``` Better approach: ```typescript // GOOD: State tracked immediately after each migration for await (const migration of migrations) { if (!lastDbMigration || Number(lastDbMigration.created_at) < migration.folderMillis) { for (const stmt of migration.sql) { await db.session.execute(sql.raw(stmt)); } // Track state immediately after successful application await db.session.execute(sql`insert into ${migrationsTable} ("hash", "created_at") values(${migration.hash}, ${migration.folderMillis})`); } } ``` This principle also applies when making changes to migration tooling - always consider the impact on existing migration state and provide clear guidance for users to handle state inconsistencies. --- ## Choose optimal data structures Select data structures based on specific access patterns and performance requirements. When both fast lookup and predictable iteration order are needed, specialized structures like IndexMap offer benefits over standard HashMap: ```rust // When order matters and lookups are frequent: use indexmap::IndexMap; // Fast lookups + preserved insertion order let cache: IndexMap = IndexMap::new(); // Regular HashMap when only lookup speed matters let simple_lookup: HashMap = HashMap::new(); ``` For serialization, separate concerns from runtime efficiency - use simple Vector structures for serialization, then transform into Maps for runtime lookups. This avoids serializing complex key structures while maintaining fast access patterns: ```rust // For serialization - simple Vec let serializable_data: Vec = items.into_iter().collect(); // For runtime - efficient HashMap let lookup_map: HashMap = serializable_data.into_iter() .map(|item| (item.id, item)) .collect(); ``` When implementing caches, consider using reference-based APIs like `entry_ref` from hashbrown to avoid unnecessary cloning: ```rust use hashbrown::HashMap; // Avoids cloning the key for lookup map.entry_ref(key_reference).or_insert_with(|| create_value()); ``` Document your data structure choices to explain performance tradeoffs, especially when choosing specialized structures over standard library alternatives. --- ## Stable schema identifiers Use persistent identifiers for schema elements rather than relying on positional information or enumeration, which can break when schema changes occur. Assign unique, immutable IDs to database objects like tables and columns, and ensure these IDs remain stable throughout the object's lifecycle. For example, instead of: ```rust // Problematic - using enumeration for column IDs .enumerate() .map(|(idx, (col_type, f))| { // Using idx as column ID is dangerous ColumnDefinition::new(idx, f.name(), col_type, f.is_nullable()) }) ``` Use a system that generates and persists stable IDs: ```rust // Better - using dedicated ID generation let column_id = table_def.next_column_id(); ColumnDefinition::new(column_id, f.name(), col_type, f.is_nullable()) ``` When designing schema access patterns, minimize catalog locks by including ID-to-name mappings in cached schema objects. Use IDs for internal lookups and operations for performance, while preserving names for user-facing interfaces. When serializing schema objects, ensure both IDs and names are properly preserved to maintain the mapping integrity. During schema changes like adding columns, preserve existing IDs and assign new IDs to new elements, rather than reassigning all IDs. This guarantees that references to existing columns remain valid even as the schema evolves. --- ## Choose appropriate lock primitives Select lock types based on access patterns - prefer RWLock over Mutex for read-heavy operations to enable concurrent reads while allowing exclusive write access. Minimize lock scope to essential operations and avoid holding locks across await points in async code. ```rust // AVOID: Using Mutex for read-heavy operations type MetaData = Mutex>>; // BETTER: Using RWLock for read-heavy operations type MetaData = RwLock>>; // AVOID: Taking lock before doing computation async fn persist_parquet_file(&self) -> Result<(), Error> { // Lock held during entire function including computation let mut meta_data_lock = self.meta_data.write(); // Long computation while holding lock let result = compute_expensive_result().await?; // Update metadata } // BETTER: Taking lock only when needed async fn persist_parquet_file(&self) -> Result<(), Error> { // Do computation without holding lock let result = compute_expensive_result().await?; // Only take lock when updating shared state let mut meta_data_lock = self.meta_data.write(); // Update metadata with minimal lock time } ``` For high-contention scenarios, consider specialized concurrent data structures like DashMap that distribute lock contention. Keep related data under the same lock to ensure atomic updates across multiple data structures. --- ## Type over primitives Use domain-specific types instead of primitives (like strings, []byte, or generic maps) to represent domain concepts in algorithms. This makes assumptions explicit, prevents errors, and enables stronger type checking. For example, rather than passing raw byte slices for different kinds of series keys: ```go // Problematic approach - using raw []byte func parseSeriesKey(key []byte) (measurement, tags []byte) { // Code assumes specific format without enforcing it // Easy to pass wrong key format! } // Better approach - using typed wrapper type SeriesKeyV1 struct { B []byte } type SeriesKeyV2 struct { B []byte } func parseSeriesKeyV1(key SeriesKeyV1) (measurement, tags []byte) { // Implementation specific to V1 format } func parseSeriesKeyV2(key SeriesKeyV2) (measurement, tags []byte) { // Implementation specific to V2 format } ``` This pattern applies to many algorithms like filtering functions, composite predicates, and data transformations. By creating specific types, you prevent algorithmic errors, make code more readable, and ensure operations only apply to valid inputs. Type-safe algorithms are easier to reason about and less prone to subtle bugs. --- ## Complete schema management When working with database systems that have flexible schemas (like InfluxDB), ensure complete schema discovery and proper merging from all data sources before processing operations. Incomplete schema handling leads to missing columns, data corruption, or export failures. To properly handle schemas: 1. Gather the complete schema by examining all relevant series or records 2. Merge schemas (tags and fields) from all sources to create a comprehensive union schema 3. Use efficient APIs for schema discovery rather than inferring from data scans 4. Remember that schemas can change within files or between series For example, in InfluxDB, instead of assuming schema consistency: ```go // Efficiently gather schema using existing indices tagKeys, err := e.tsdbStore.TagKeys(context.Background(), query.OpenAuthorizer, []uint64{shard.ID()}, cond) fields := shard.MeasurementFields([]byte("")) // When processing mixed tag sets, merge schemas properly: allTagKeys := make(map[string]struct{}) allFields := make(map[string]influxql.DataType) // Iterate through all series to build complete schema for seriesKey, fieldValues := range data { tags := models.ParseTags([]byte(seriesKey)) // Add all tags to the complete schema for _, tag := range tags { allTagKeys[string(tag.Key)] = struct{}{} } // Add all fields to the complete schema for fieldName, values := range fieldValues { allFields[fieldName] = values[0].Type() } } ``` This approach ensures data integrity when schema varies across records, producing complete and accurate results for queries, exports, and other database operations. --- ## Secure GPG verification When downloading and verifying packages using GPG signatures, follow secure practices to ensure authenticity and prevent security vulnerabilities. Use hardcoded GPG keys rather than dynamic ones, employ reliable keyservers, set proper GNUPGHOME environment, and clean up GPG processes after verification. Key practices include: - Hardcode GPG_KEY values as environment variables for transparency - Use only reliable, maintained keyservers (e.g., hkp://keys.openpgp.org, keyserver.ubuntu.com) - Set GNUPGHOME to isolate GPG operations - Clean up GPG processes with `gpgconf --kill all` after verification - Download from HTTPS sources when possible Example implementation: ```dockerfile ENV GPG_KEY CF9500821E9557AEB04E026C05EEA67F87749E61 RUN set -eux ; \ for server in hkp://keys.openpgp.org keyserver.ubuntu.com ; do \ gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; \ done && \ wget -nv -O package.tgz "$package_url"; \ wget -nv -O package.tgz.asc "$package_url.asc"; \ gpg --batch --verify package.tgz.asc package.tgz; \ gpgconf --kill all ``` This approach follows Docker official images guidelines and established practices from projects like Apache Flink, ensuring package integrity while maintaining security best practices. --- ## Document network configuration When configuring network settings in containerized services, always document the reasoning behind specific choices, especially for port mappings and security-related flags. This helps other developers understand the configuration and prevents accidental misconfigurations. For port mappings, use reasonable external port numbers and explain any non-standard choices: ```yaml ports: - 9432:5432 # Postgres port - using 9432 to avoid conflicts ``` For network security flags, document their purpose and security implications: ```yaml healthcheck: test: ['CMD', '/opt/mssql-tools18/bin/sqlcmd', '-C', '-Usa', '-PPr1sm4_Pr1sm4', '-Q', 'select 1'] # -C flag trusts server certificate without validation ``` This practice prevents confusion during deployment and helps maintain consistent network configuration across environments. --- ## Precise algorithm terminology When implementing and documenting algorithms, use precise terminology and be explicit about metrics, operations, and data structures to avoid ambiguity and ensure correctness. **Key practices:** 1. **Be explicit about threshold metrics:** Clearly distinguish between counting operations (`count`) versus measuring their size or impact (`sum`). For example: ```rust // Be specific about threshold conditions if count(deltas) < threshold { // Clear that we're counting operations // retain deltas } else if sum(delta_sizes) < size_threshold { // Clear that we're measuring size // different logic based on total size } ``` 2. **Use accurate descriptors for data structures:** Choose precise terminology like "sparse" instead of "partial" when describing specialized data structures. This communicates the exact properties and access patterns. 3. **Document algorithmic complexity accurately:** When describing performance characteristics, ensure grammatical accuracy and precision (e.g., "does not have such a constant factor" rather than "does not has"). Proper terminology choices make algorithms easier to understand, maintain, and optimize. They also help prevent subtle bugs that can arise from misinterpreting how an algorithm should behave based on ambiguous descriptions. --- ## Document performance tradeoffs Always explicitly document the performance implications of API parameters, limit changes, and features that could significantly impact resource usage. When introducing functionality that offers a tradeoff between capabilities and performance, provide clear warnings and specific details about: 1. Memory consumption impacts 2. Response time implications 3. Scaling considerations for different cluster sizes For example: ```asciidoc `include_empty_fields`:: (Optional, Boolean) If `false`, fields that never had a value in any shards are not included in the response. *WARNING*: Setting this parameter to `false` may significantly increase response times on large datasets due to additional field existence checks across all documents. Consider using the default value in performance-sensitive applications. ``` or ```asciidoc NOTE: Synonyms sets consume heap memory proportional to their size. While the system supports up to 100,000 synonym rules per set, large synonym sets may impact query performance and memory usage, especially on smaller clusters. Monitor heap usage when working with large synonym sets. ``` This practice helps users make informed decisions about performance-sensitive configurations and prevents unexpected resource usage issues in production environments. --- ## Research configuration format support When migrating configuration files to newer formats, research current ecosystem support before defaulting to compatibility layers. Many tools and plugins may already support modern formats even if documentation suggests otherwise. Before using compatibility shims like `@eslint/eslintrc`'s `FlatCompat`, check if your required plugins and parsers have native support for the new format. This prevents unnecessary dependencies and technical debt while ensuring you benefit from modern configuration features. Example approach: ```javascript // Instead of immediately using compatibility layer: const { FlatCompat } = require('@eslint/eslintrc') const compat = new FlatCompat({...}) // First research if plugins support new format: // Check plugin documentation, GitHub issues, or compatibility matrices // Many plugins like @typescript-eslint and eslint-plugin-prettier // already support ESLint v9 flat config natively // Then use native configuration when possible: module.exports = [ { files: ['**/*.ts'], // Native flat config approach } ] ``` This approach reduces maintenance burden and ensures you're using the most current, supported configuration patterns rather than legacy compatibility workarounds. --- ## Complete API parameter documentation API endpoints must include comprehensive documentation for all parameters. For each parameter, clearly specify: 1. Whether it's required or optional 2. The data type and format 3. Default values when omitted 4. Validation rules and constraints (e.g., maximum values, character limits) 5. Accepted formats and encoding requirements 6. Behavior details (exact matching vs pattern matching) This ensures consistency between API implementation and documentation, prevents integration issues, and improves developer experience. Example: ``` `connector_name`:: (Optional, string) A comma-separated list of connector names, used to filter search results. Default: Returns all connectors if omitted. Maximum: 100 connector names. Format: Exact matches only, URL-encoded values required for special characters. Validation: Names must follow the same validation rules as defined in the creation API. ``` When updating APIs, ensure that parameter behavior aligns with UI expectations and vice versa to prevent disconnects between API functionality and user experience. --- ## Official product naming When referencing external products, libraries, or services in documentation and code, always use their official names exactly as specified by the vendor, and treat them as proper nouns without articles. Key principles: - Use official product names without modification (e.g., "Cloudflare D1" not "Cloudflare's Serverless Driver D1") - Treat product names as proper nouns - don't prefix with articles like "the" (e.g., "Prisma Client" not "the Prisma Client") - Maintain consistency across all documentation and comments Example from documentation: ```markdown // Incorrect Install the Prisma Client, the Prisma adapter for Cloudflare's Serverless Driver D1 // Correct Install Prisma Client, Prisma adapter for Cloudflare D1 ``` This ensures accuracy, professionalism, and consistency when referencing third-party technologies, while avoiding confusion about official product capabilities or features. --- ## Document configuration decisions {% raw %} Add explanatory comments to configuration files that clarify the reasoning behind non-obvious choices, feature exclusions, conditional logic, and environment-specific settings. This helps future maintainers understand why certain configurations exist and prevents accidental modifications. Key areas requiring documentation: - Feature flags and their exclusions/inclusions - Conditional logic in workflows or config files - Environment-specific behavior - Default values that may not be immediately obvious Example: ```yaml env: # Driver adapters are a special feature, # so in our "normal" tests, we don't enable it yet to keep things separated, # also because they are tested in a separate job. EXCLUDED_PREVIEW_FEATURES: 'driverAdapters' # Exclude relationJoins tests with WASM because the WASM engine does not include the necessary changes yet. EXCLUDED_PREVIEW_FEATURES: ${{ matrix.engine-type == 'wasm' && 'relationJoins' || '' }} ``` This practice prevents confusion about configuration intent and reduces the likelihood of breaking changes when configurations are modified. {% endraw %} --- ## Deterministic Control-Plane Protocols When multiple agents participate in cluster control-plane updates (e.g., TD, FC, nodes), treat the protocol as an algorithm and make its conflict-resolution and time/progression semantics explicit in the spec/code review contract. **Standards** 1) **Define a strict authority hierarchy (deterministic winner).** If agents can reject or delay configs (e.g., FC rejects “invalid” topology), the spec must state what happens in disagreement and who has the final vote (e.g., TD decides topology metadata; FC decides primary/replica status). Avoid designs where an agent bug can permanently wedge the system. 2) **Define recovery contracts, not just “fix bugs.”** Specify what the system does if validation fails or messages arrive out of order, including anti-deadlock guarantees and how to un-wedge (e.g., force flags, admin-driven reconciliation). 3) **Use monotonic progression signals for all time-based decisions.** Do not rely on multiple unsynchronized clocks across components. Have the control-plane coordinator maintain a monotonically increasing counter (tick) and include it in responses so nodes can compute “time since last primary update” from tick deltas. 4) **Separate validation from decision responsibilities.** Keep validation scope tight and named: for example, FC can validate epoch correctness and ownership/ownership changes, while nodes perform failover triggering based on the FC-provided state and tick-based timeouts. 5) **Enforce critical invariants to prevent state-space races.** Example invariant: no concurrent updates for the same shard until prior epochs are acknowledged by all relevant nodes/FCs. **How to apply (pseudo-code)** ```pseudo // Coordinator-provided monotonic tick included in every response on FC.heartbeat(node_id, node_epoch, shard_epoch): if node_epoch < current_epoch: return NACK if node_shard_coordinator(node_id) != this.FC_id: return MOVED(target_FC_id) record(node_id, role, shard_epoch, replication_offset, tick=current_tick) return ACK(current_topology_if_needed, shard_results, tick_per_node) // Node uses FC tick deltas to decide failover (no local multi-clock dependence) on node.tick_based_timeout_primary_missing(primary_id): last_primary_tick = tick_from_FC(primary_id) if current_tick_from_FC - last_primary_tick > N: if node_is_most_up_to_date_replica(): propose_promote(role=PRIMARY, bump(shard_epoch)) ``` **Review checklist** - Can we point to the exact “winner rule” in every disagreement case? - Is there an explicit anti-deadlock/un-wedging plan for invalid or rejected updates? - Are all timeout/failover decisions derived from coordinator-provided monotonic counters? - Does each component validate only what it should, and does every decision-maker know its inputs? - Are concurrency/race invariants (like “no concurrent updates per shard until acknowledged”) stated and enforced? --- ## Structured configuration management Avoid using global configuration singletons that rely on string-based lookups, as they lead to brittle code that's hard to maintain. Instead, implement a structured approach where configuration values are explicitly declared with their types, sources (flags, environment variables, config files), and default values. This improves compile-time safety, documentation, and maintainability. For dynamic configurations that can be reloaded at runtime, ensure thread-safety and consider performance implications. When implementing config reloading, handle invalid configurations gracefully by logging errors and maintaining the last valid configuration rather than crashing. Example: ```go // BAD: Global, string-based configuration func doSomething() { name := viper.GetString("name") // Brittle, no type safety fmt.Println(name) } // GOOD: Structured, type-safe configuration var ( configKey = viperutil.KeyPrefixFunc("myapp") // Explicitly declare configuration with sources accountName = viperutil.Configure( configKey("account.name"), viperutil.Options[string]{ EnvVars: []string{"MY_APP_ACCOUNT_NAME"}, FlagName: "account_name", Default: "", Dynamic: false, // Static config loaded only at startup }, ) ) // Register flags and bind them to config values func registerFlags(fs *pflag.FlagSet) { fs.String("account_name", accountName.Default(), "Account name for the application") viperutil.BindFlags(fs, accountName) } // Use the configuration value func doSomething() { name := accountName.Get() fmt.Println(name) } ``` --- ## Document security requirements explicitly Always document security-related configurations, permissions, and behaviors explicitly and comprehensively. When documenting security features: 1. Clearly specify required permissions and roles 2. Note version-specific security behavior changes 3. Explicitly state configuration inheritance rules and exceptions 4. Highlight security-critical settings with warnings or notes For example, when documenting an API that interacts with protected resources: ``` IMPORTANT: This action requires specific permissions. In {es} 8.1 and later, the superuser role doesn't have write access to system indices. If you execute this request as a user with the superuser role, you must have an additional role with the `allow_restricted_indices` privilege set to `true` to delete system indices. ``` For configuration documentation: ``` NOTE: Transport profiles do not inherit TLS/SSL settings from the default transport. The `xpack.security.transport.ssl.enabled` setting is an exception that controls SSL for both default transport and any transport profiles. ``` Clear and complete security documentation prevents misconfigurations that could lead to vulnerabilities or access issues. --- ## vet third-party actions Before using third-party GitHub Actions or similar external dependencies, thoroughly review their security implications. Specifically: 1) Understand what tokens or credentials they require and how they use them, 2) Verify that permission checks and security features work as expected, 3) Avoid providing unnecessary credentials (like PATs when not required), and 4) Add tests to validate the security functionality works correctly. Example from the discussion: ```yaml # Before using actions-cool/check-user-permission@v2 # Review: what does it do with tokens? Does it properly check permissions? - name: Check write permission id: check uses: actions-cool/check-user-permission@v2 # Remove PAT if not required, add tests for permission validation ``` This practice prevents security vulnerabilities from poorly understood or misconfigured third-party components and ensures your security controls work as intended. --- ## Control-plane connectivity rules Define a single, deterministic networking contract between data-plane nodes and the control plane (TD/FC), and make it resilient to CP reachability and multi-FC state consistency. Apply these rules: 1) **Minimize DP discovery/knowledge**: Data-plane code should not need to discover TD/FC addresses/locations. Prefer a design where the control plane learns ownership and connects/publishes to the assigned nodes, so the data plane only speaks a stable “FC-facing” protocol endpoint. 2) **Choose connection ownership intentionally**: If DP→CP, the control plane must rate-limit/jitter/retry to prevent thundering herd. If CP→DP, ensure DP listens on a well-defined port and accepts long-lived connections. 3) **Implement explicit “CP unreachable” behavior**: When a node can’t reach its owning FC for *X* seconds, it must stop serving all traffic or at least stop writes, based on configuration (e.g., `cluster-allow-reads-during-cluster-down`). 4) **Propagate routing-relevant state across FCs**: In a multi-FC deployment, ensure that failover/topology changes that affect how clients get slot routing (e.g., `CLUSTER SLOTS` / MOVED behavior) are visible to the FCs that serve those clients; otherwise different FCs will return conflicting views. Example (CP-unreachable gating): ```pseudo onHeartbeatTimeout(): if timeSinceLastContact(owningFC) >= FC_UNREACHABLE_TIMEOUT: if clusterAllowReadsDuringClusterDown: blockWrites() else: stopServingAllTraffic() ``` These constraints keep the data plane’s networking simple, reduce overload risk on the control plane, and prevent stale routing answers across FC boundaries. --- ## Required API Partitioning When designing cluster APIs/specs, explicitly partition endpoints into categories by *actor* and *compatibility obligation*, and document which categories alternative implementations must implement. **Standard** 1. **Define the compatibility surface** (what non-admin clients and core node behavior need). Mark these endpoints as **Required**. 2. **Separate reference/admin orchestration** (what operators use to create/configure/manage entities). Mark these as **Admin/Optional**. 3. **Keep internal control-plane comms** (e.g., TD↔FC status propagation) out of the external compatibility surface. Mark these as **Internal/Not Required**. 4. For any advanced controls (e.g., externally imposed shard identity), ensure the **default flow remains ergonomic**, and treat identity/placement parameters as **opt-in options** on the required or admin interfaces. **Practical checklist** - For each command/endpoint, label: `{Actor: node|client|admin|internal, Obligation: Required|Optional|Not Required}`. - Add a short compatibility statement: “Any alternative implementation that claims drop-in compatibility MUST implement all *Required* endpoints (optionally subsets), but may ignore *Internal/Not Required* endpoints.” - Ensure optional power-user fields (e.g., `shard_id`) are not required for the default path. **Example (shape of the spec, pseudo-code)** ```text Endpoint: HEARTBEAT Actor: node Obligation: Required Purpose: node↔FC liveness, role decisions Endpoint: TD/FC internal messages (FAILOVER_STATUS, TOPology_RESPONSE) Actor: internal Obligation: Not Required Purpose: FC↔TD propagation; no compatibility guarantee Endpoint: ADMIN.CREATE_SHARD / SHARD_ID on NODE_JOIN Actor: admin Obligation: Optional Default: shard IDs auto-managed Power-user: allow specifying shard_id to match external orchestration needs ``` This prevents “spec creep” where admin/internal endpoints accidentally become de-facto requirements, and it allows alternative implementations to remain compatible without re-creating internal details. --- ## Define Non-Ack Behavior For any control-plane action that depends on acknowledgement (epochs/status updates, joins/removes, metadata propagation), the codebase must explicitly define end-to-end error handling: timeout, retry/escalation, cascade effects, and the operator override path. Apply this standard by ensuring each such operation documents/implements: 1. **Non-ack contract**: what the system does when a target node (or FC/TD hop) never acknowledges (stop progress, keep serving with defined mode, or mark entities unavailable). 2. **Timeout and retry strategy**: exact thresholds and backoff; whether retries continue indefinitely or transition to an escalated state. 3. **Cascade/feedback clarity**: whether non-ack at the node level affects TD/other components, and what errors are propagated upward. 4. **Operator override**: if an escape hatch exists, define a *safe default* (refuse or limit impact) and a *forced* path with prominent warnings and justification. 5. **Troubleshooting path**: a dedicated “everything is broken”/unwedging procedure, including what signals/metrics/errors to look for. 6. **Recovery-mode correctness**: distinguish **automatic recovery** from **rejoin required** when persistence/durability expectations aren’t met; avoid wording that implies guarantees the system doesn’t actually provide. 7. **Admin guardrails for unsafe actions**: when removal/failover can be destructive (e.g., removing a primary), require safer sequencing (failover first, then remove) or ensure force flags are clearly constrained; optionally support client-safe mitigation like connection draining/rotation. Example (pseudocode pattern for non-ack handling): ```pseudo function applyTopologyChange(change, targetNode, basedEpoch): reqId = newReqId() sendRequest(targetNode, change, basedEpoch, reqId) deadline = now() + ACK_TIMEOUT attempts = 0 while now() < deadline and attempts < MAX_RETRIES: status = waitForAck(reqId, minRemainingTime()) if status == ACKED and status.epoch == basedEpoch: return OK attempts++ // Escalation (defined behavior) markEntity(DEGRADED, reason="no ack", epoch=basedEpoch) propagateUpstreamIfNeeded(reason="no ack") logError(warn="Operation not acknowledged; escalation to operator required", reqId) if operatorOverrideEnabled and operatorConfirmed: applyForced(change) else return ERROR("non-ack timeout; operation refused by contract") ``` This turns ambiguous failure modes into deterministic, testable behavior and ensures operators have a clear, warned path to recover when the normal acknowledgement-based workflow cannot complete. --- ## Evidence-Based Failover Targets When proposing performance/SLA goals (e.g., failover completion time), require an evidence-based rationale and a repeatable way to validate them—especially for worst-case behavior. Apply this by documenting: - Target definition: what exactly “10s failover” includes/excludes (detection, consensus/propagation, promotion, client-observable recovery). - Worst-case assumptions: network latency bounds, timeouts, quorum/heartbeat intervals, failure model, and any upper bounds. - Measurement basis: links to benchmarks/profiles or at least a structured summary of real-world observations (cloud regions/AZs, instance/network reliability stats). - Acceptance tests: automated tests (or staging runs) that exercise the failure mode(s) producing the worst case and assert the target. Example standard text developers can follow: ```text Failover target: <=10s (worst-case), <=5s (some configs). Definition: time from heartbeat timeout detection to topology+role update propagated to all impacted nodes, including client observable readiness. Assumptions: cross-AZ RTT p99=..., heartbeat interval=..., coordinator tick=..., quorum=... Evidence: benchmark/staging results in regions X/Y with injected failures; include traces/profiles. Regression guard: CI/staging test fails if p99/worst-case exceeds target. ``` This ensures performance claims aren’t just experience-based anecdotes, and it prevents future changes from silently breaking latency guarantees. --- ## Accurate method descriptions API documentation must precisely describe method behavior, especially return values and cardinality. Inaccurate descriptions mislead developers about what methods actually return, potentially causing runtime errors or incorrect assumptions. Key areas to verify: - **Return value nullability**: Clearly state when methods can return null/undefined - **Result cardinality**: Use precise language like "zero or one" vs "one", or "zero or more" vs "one or more" - **Actual behavior alignment**: Ensure descriptions match the method's real implementation Example of incorrect vs correct documentation: ```javascript // ❌ Incorrect - misleading about nullability /** * Find one User that matches the filter. */ findUnique(args) { ... } // ✅ Correct - accurate about possible null return /** * Returns User that matches the filter or `null` if nothing is found. */ findUnique(args) { ... } // ❌ Incorrect - wrong cardinality /** * Find one or more Users that matches the filter. */ findMany(args) { ... } // ✅ Correct - accurate cardinality /** * Find zero or more Users that matches the filter. */ findMany(args) { ... } ``` This prevents developers from making incorrect assumptions about method behavior and reduces debugging time caused by unexpected null values or empty results.