Awesome Reviewers

Verification catalog

10 tactics, 70 techniques, 179 sub-techniques. Usage counts are from 256 scanned source repositories. Leaderboard: https://awesomereviewers.com/raw/leaderboard.json

VA01 Formatting and hygiene

Normalises layout, spelling and file hygiene so the diff only shows what changed.

When: On every change, before anything else. Almost always autofixable: run the fix, then commit.

V1001 Formatting

A formatter rewrites code to one layout so diffs show only real changes; in CI it runs in check mode and fails on drift. Run by 112 of 256 scanned repositories.

Sub-techniques:

• V1001.001 Code formatter: One opinionated formatter per language, run on save and in CI.
• V1001.002 Import ordering: Sort and group imports so they never cause conflicts or noise.
• V1001.003 Check mode gate: Run the formatter with –check or –diff in CI so unformatted code fails instead of being rewritten.

Craft it:

1. Pick one formatter per language and commit its config at the repository root.
2. Run it once over the whole tree in a single commit and add that commit to .git-blame-ignore-revs.
3. Add a pre-commit hook that rewrites, and a CI step in check mode that fails.

Reverse engineer it (look for):

• Formatter config in pyproject.toml, .prettierrc, rustfmt.toml or biome.json.
• A CI step running the formatter with –check, –diff or format –check.
• A .git-blame-ignore-revs file pointing at a mass-format commit.
• .github/workflows/*.yml (configures it in 74 scanned repositories)
• package.json (configures it in 47 scanned repositories)
• .pre-commit-config.yaml (configures it in 33 scanned repositories)
• Makefile (configures it in 22 scanned repositories)
• .prettierrc (configures it in 17 scanned repositories)
• rustfmt.toml (configures it in 15 scanned repositories)

Open-source packages: Ruff, Black, isort, Prettier, Biome, rustfmt Tools seen in scans: prettier (98), rustfmt (46), ruff-format (43), black (14), isort (9), gofmt (7)

V1002 File hygiene

Cheap checks on files as files: valid syntax for config formats, no merge markers, no trailing whitespace, no oversized or binary blobs. Run by 33 of 256 scanned repositories.

Sub-techniques:

• V1002.001 Config syntax validation: Parse every JSON, YAML and TOML file, and validate the important ones against a schema.
• V1002.002 Merge marker and whitespace checks: Reject conflict markers, trailing whitespace and missing final newlines.
• V1002.003 Large file guard: Refuse files above a size limit so binaries go to LFS or object storage.

Craft it:

1. Add pre-commit-hooks with check-yaml, check-json, check-toml, check-merge-conflict and check-added-large-files.
2. Validate CI and app config against published JSON schemas with check-jsonschema.
3. Run pre-commit run –all-files in CI so the hooks are not optional.

Reverse engineer it (look for):

• A .pre-commit-config.yaml listing pre-commit-hooks ids.
• A CI job that runs pre-commit run –all-files.
• A .yamllint or schema files referenced from hooks.
• .pre-commit-config.yaml (configures it in 33 scanned repositories)
• .github/workflows/*.yml (configures it in 12 scanned repositories)
• Makefile (configures it in 1 scanned repositories)
• pyproject.toml (configures it in 1 scanned repositories)

Open-source packages: pre-commit-hooks, check-jsonschema, yamllint Tools seen in scans: pre-commit-hooks (33)

V1003 Spelling

Finds misspellings in identifiers, comments and documentation, where a typo becomes a public API or a search miss. Run by 19 of 256 scanned repositories.

Sub-techniques:

• V1003.001 Identifier typo detection: Split camelCase and snake_case identifiers and flag known misspellings.
• V1003.002 Prose spell check: Dictionary check over docs and comments with a project word list.

Craft it:

1. Start with a low false-positive tool (typos or codespell) on the whole tree.
2. Add an allow-list file for domain words instead of disabling the check.
3. Run it in pre-commit and in CI with autofix locally.

Reverse engineer it (look for):

• _typos.toml, .codespellrc or cspell.json in the repository.
• A codespell or typos hook in .pre-commit-config.yaml.
• .github/workflows/*.yml (configures it in 15 scanned repositories)
• .pre-commit-config.yaml (configures it in 11 scanned repositories)
• pyproject.toml (configures it in 10 scanned repositories)
• .codespellrc (configures it in 5 scanned repositories)
• Makefile (configures it in 3 scanned repositories)
• tox.ini (configures it in 1 scanned repositories)

Open-source packages: typos, codespell, CSpell Tools seen in scans: codespell (22)

V1004 Commit and pull request hygiene

Rules on the change itself rather than the code: commit message format, PR description, size limits, required labels or linked tickets.

Sub-techniques:

• V1004.001 Commit message lint: Enforce a message convention so changelogs and release tooling can parse history.
• V1004.002 Pull request rules: Fail or warn when a PR is too large, lacks a description, or touches files without matching tests or docs.

Craft it:

1. Write the rules you already enforce by hand in review as a Dangerfile or commitlint config.
2. Start as warnings, then turn the rules people respect into failures.
3. Keep size thresholds per directory, not global.

Reverse engineer it (look for):

• commitlint.config.js, a Dangerfile, or a PR title check in CI.
• Release tooling that parses conventional commits (release-please, semantic-release).

Open-source packages: commitlint, Danger JS

VA02 Static analysis

Reads the code without running it: linters, type checkers, security and workflow scanners.

When: On every change. Cheapest signal, runs in seconds, and most findings point at an exact line.

V1005 Linting

Rule-based detection of bug-prone patterns, unused code and conventions the formatter does not cover. Run by 144 of 256 scanned repositories.

Sub-techniques:

• V1005.001 Correctness rules: Rules for likely bugs: unused results, shadowing, unreachable code, wrong comparisons.
• V1005.002 Convention rules: Project conventions encoded as lint rules instead of review comments.
• V1005.003 Custom lint rules: Repository-specific rules written as plugins or pattern rules for mistakes that recur in review.

Craft it:

1. Enable the linter’s recommended set and fix or baseline existing findings.
2. Turn recurring review comments into custom rules (an ESLint plugin, a Semgrep rule, a Clippy lint config).
3. Fail CI on new findings only, so the baseline shrinks without blocking work.

Reverse engineer it (look for):

• Lint config in pyproject.toml ([tool.ruff]), eslint.config.js, .golangci.yml or clippy.toml.
• A CI job named lint, or a lint script in package.json, Makefile or justfile.
• Inline suppressions (noqa, eslint-disable) show which rules are enforced.
• .github/workflows/*.yml (configures it in 101 scanned repositories)
• package.json (configures it in 64 scanned repositories)
• pyproject.toml (configures it in 40 scanned repositories)
• .pre-commit-config.yaml (configures it in 37 scanned repositories)
• Makefile (configures it in 27 scanned repositories)
• eslint.config.mjs (configures it in 23 scanned repositories)

Open-source packages: Ruff, ESLint, golangci-lint, Clippy, ShellCheck, Error Prone Tools seen in scans: eslint (107), ruff-check (64), clippy (45), golangci-lint (26), biome (24), flake8 (17), pylint (17), shellcheck (10), go-vet (9), staticcheck (1)

V1006 Type checking

Proves call sites and data shapes agree across modules: wrong types, unhandled null or None, signatures broken by a change. Run by 106 of 256 scanned repositories.

Sub-techniques:

• V1006.001 Gradual typing: Type-check annotated code first and tighten strictness per module.
• V1006.002 Strict mode: Disallow implicit any, untyped definitions and unchecked optional access.
• V1006.003 Compiler as checker: In compiled languages the build is the type check; a failing build is this check failing.

Craft it:

1. Run the checker in non-strict mode on the whole repository to get a baseline.
2. Turn on strict flags for new and frequently changed modules first.
3. Run it in CI as its own job so type errors are not hidden in test failures.

Reverse engineer it (look for):

• [tool.mypy] or pyrightconfig.json, tsconfig.json with strict flags.
• A tsc –noEmit, mypy or pyright step in CI or a typecheck script.
• package.json (configures it in 80 scanned repositories)
• .github/workflows/*.yml (configures it in 74 scanned repositories)
• pyproject.toml (configures it in 17 scanned repositories)
• Makefile (configures it in 12 scanned repositories)
• .pre-commit-config.yaml (configures it in 9 scanned repositories)
• mypy.ini (configures it in 4 scanned repositories)

Open-source packages: mypy, Pyright, TypeScript (tsc) Tools seen in scans: tsc (117), mypy (35), cargo-check (21), pyright (16)

V1007 Static application security testing

Finds injection, unsafe APIs, weak crypto and tainted data reaching a sink without running the code. Run by 71 of 256 scanned repositories.

Sub-techniques:

• V1007.001 Pattern rules: Syntax-aware patterns for dangerous calls and configurations (shell=True, yaml.load, disabled TLS verification).
• V1007.002 Taint tracking: Follow untrusted input from sources to sinks across functions and files.
• V1007.003 Language security linters: Per-language security rule sets maintained by each ecosystem.
• V1007.004 Custom rule packs: Rules for the framework and helpers your code uses, with tests for each rule.

Craft it:

1. Run a pattern scanner with a community rule pack and record the baseline.
2. Write custom rules for your own wrappers (your query builder, your HTTP client), each with a positive and negative test file.
3. Scan only changed files on PRs (diff-aware mode) and the whole repository on a schedule.

Reverse engineer it (look for):

• A codeql-analysis workflow, .semgrep.yml or a rules/ directory with YAML rules.
• bandit, gosec or security-code-scan in CI or pre-commit.
• SARIF uploads (github/codeql-action/upload-sarif) in workflows.
• .github/workflows/*.yml (configures it in 70 scanned repositories)
• .pre-commit-config.yaml (configures it in 4 scanned repositories)
• pyproject.toml (configures it in 1 scanned repositories)
• tox.ini (configures it in 1 scanned repositories)

Open-source packages: Opengrep, Semgrep CE, Bandit, gosec, Security Code Scan, Find Security Bugs, eslint-plugin-security Tools seen in scans: codeql (63), semgrep (15), bandit (6)

V1008 Secret scanning

Finds credentials and tokens in the working tree and in history before they are pushed or published. Run by 5 of 256 scanned repositories.

Sub-techniques:

• V1008.001 Pre-commit secret scan: Block the commit locally, before the secret reaches a remote.
• V1008.002 History scan: Scan full git history, since deleting a secret in a later commit does not remove it.
• V1008.003 Live verification: Test whether a found credential is still valid to prioritise rotation.

Craft it:

1. Add a pre-commit hook and a CI job running the same scanner and config.
2. Keep a baseline or allow-list for test fixtures, reviewed like code.
3. Scan full history once and rotate anything found.

Reverse engineer it (look for):

• .gitleaks.toml, .secrets.baseline or a trufflehog step in CI.
• A gitleaks or detect-secrets hook in .pre-commit-config.yaml.
• .gitleaks.toml (configures it in 3 scanned repositories)
• .pre-commit-config.yaml (configures it in 3 scanned repositories)
• .github/workflows/*.yml (configures it in 2 scanned repositories)
• .husky/pre-commit (configures it in 1 scanned repositories)

Open-source packages: Gitleaks, TruffleHog, detect-secrets Tools seen in scans: gitleaks (7)

V1009 CI workflow linting

Checks CI definitions themselves: syntax, script injection through untrusted inputs, over-broad token permissions, unpinned actions. Run by 38 of 256 scanned repositories.

Sub-techniques:

• V1009.001 Workflow syntax: Validate workflow files and embedded shell before they run.
• V1009.002 Workflow security audit: Flag template injection, pull_request_target misuse, unpinned third-party actions and write-all permissions.

Craft it:

1. Run actionlint on every change to .github/workflows.
2. Add zizmor and fix its high findings first: injection and permissions.
3. Pin third-party actions to commit SHAs.

Reverse engineer it (look for):

• actionlint or zizmor in CI or pre-commit.
• permissions: blocks at workflow level and SHA-pinned uses: lines.
• .github/workflows/*.yml (configures it in 29 scanned repositories)
• .pre-commit-config.yaml (configures it in 13 scanned repositories)
• Makefile (configures it in 1 scanned repositories)
• lefthook.yml (configures it in 1 scanned repositories)
• tox.ini (configures it in 1 scanned repositories)

Open-source packages: actionlint, zizmor, ShellCheck Tools seen in scans: zizmor (33), actionlint (12)

V1010 Documentation build

Builds documentation as a check so broken cross-references, links and markup fail like code. Run by 13 of 256 scanned repositories.

Sub-techniques:

• V1010.001 Strict docs build: Treat warnings as errors (sphinx -W, mkdocs –strict).
• V1010.002 Link checking: Check internal and external links on a schedule.
• V1010.003 Doctests: Run code examples in documentation as tests.

Craft it:

1. Build docs in CI with warnings as errors.
2. Run a link checker on PRs for internal links and weekly for external ones.
3. Execute examples so docs and code cannot drift.

Reverse engineer it (look for):

• docs/conf.py or mkdocs.yml plus a CI docs job.
• A lychee or linkcheck step, or doctest options in pytest config.
• .github/workflows/*.yml (configures it in 10 scanned repositories)
• pyproject.toml (configures it in 7 scanned repositories)
• Makefile (configures it in 2 scanned repositories)
• tox.ini (configures it in 2 scanned repositories)
• justfile (configures it in 1 scanned repositories)

Open-source packages: Sphinx, MkDocs, lychee Tools seen in scans: mkdocs (11), sphinx (9)

V1011 Duplicate code detection

Finds copied blocks so a fix in one copy is not missed in the others, and flags new code that re-implements an existing helper.

Sub-techniques:

• V1011.001 Token clone detection: Report token-identical or near-identical blocks above a size threshold.
• V1011.002 Semantic reuse check: Search the codebase for an existing function that does what the new code does, by embedding or by name.

Craft it:

1. Run a clone detector with a minimum token count that ignores tests and generated code.
2. Report only clones introduced by the change.
3. For reuse, index function signatures and docstrings and query with each new function.

Reverse engineer it (look for):

• .jscpd.json or a cpd task in the build.
• Review comments asking to reuse an existing helper point at where this is missing.

Open-source packages: jscpd, PMD (CPD), Sentence Transformers

V1012 Complexity limits

Measures cyclomatic complexity, function length and nesting and fails when a change pushes a function past a limit.

Sub-techniques:

• V1012.001 Complexity thresholds: Per-function limits on cyclomatic or cognitive complexity.
• V1012.002 Size budgets: Limits on function, file and parameter counts.

Craft it:

1. Measure the current distribution and set the limit at the existing worst case.
2. Lower it over time; never block on files the change did not touch.
3. Prefer a linter rule (ruff C901, ESLint complexity) over a separate tool.

Reverse engineer it (look for):

• max-complexity or complexity rules in linter config.
• radon or lizard reports in CI.

Open-source packages: Radon, Lizard, Ruff

V1013 API compatibility checking

Diffs a public interface between the base and the change and reports breaking changes: removed endpoints, changed types, narrowed inputs.

Sub-techniques:

• V1013.001 HTTP API diff: Compare OpenAPI documents for breaking changes, and lint them for style.
• V1013.002 Schema and protobuf compatibility: Wire-compatibility rules for protobuf, Avro or JSON Schema.
• V1013.003 Library API diff: Compare the exported symbols and signatures of a package against its last release.

Craft it:

1. Generate the interface description (OpenAPI, .proto, public symbols) in CI on both base and head.
2. Fail on breaking differences unless the version is bumped or a label allows it.
3. Publish the diff as a PR comment so reviewers see the contract change.

Reverse engineer it (look for):

• buf.yaml with a breaking section, oasdiff in CI, api-extractor.json.
• Semver checks in release workflows.

Open-source packages: oasdiff, Spectral, Buf (buf breaking), Griffe (griffe check), cargo-semver-checks, japicmp, API Extractor (Rush Stack)

V1014 Data and control flow analysis

Builds a code property graph or interprocedural model to answer reachability questions: can this null reach that dereference, which callers reach this function.

Sub-techniques:

• V1014.001 Interprocedural null and resource analysis: Report null dereferences and leaks along concrete paths.
• V1014.002 Code property graph queries: Query syntax, control flow and data flow together to find custom bug patterns.
• V1014.003 Call graph impact: List every caller and downstream consumer of a changed symbol.

Craft it:

1. Run the analyzer on the default branch to establish a baseline.
2. Run it on the change and report only new paths.
3. Save the queries that found real bugs as regression rules.

Reverse engineer it (look for):

• infer or spotbugs tasks in the build, or Joern scripts in the repository.
• SCIP or LSIF index uploads in CI for code navigation.

Open-source packages: Infer, Joern, SpotBugs, SCIP

VA03 Supply chain

Checks dependencies: known vulnerabilities, disallowed licenses, lockfiles that disagree with manifests.

When: When a manifest or lockfile changes, and on a schedule, since advisories arrive after the code ships.

V1015 Dependency audit

Matches dependencies against advisory databases and license policy, and fails on known-vulnerable or disallowed packages. Run by 37 of 256 scanned repositories.

Sub-techniques:

• V1015.001 Advisory matching: Compare resolved versions with OSV, GitHub and ecosystem advisories.
• V1015.002 Reachability-aware audit: Report a vulnerable dependency only when the vulnerable function is called.
• V1015.003 License policy: Allow or deny dependencies by license.

Craft it:

1. Run an audit on every manifest or lockfile change and on a daily schedule.
2. Fail on high severity with a fix available; warn on the rest.
3. Use a reachability-aware tool where the ecosystem has one.

Reverse engineer it (look for):

• deny.toml, osv-scanner.toml, a dependency-review workflow or an audit script.
• Dependabot or Renovate config with security updates enabled.
• .github/workflows/*.yml (configures it in 35 scanned repositories)
• deny.toml (configures it in 14 scanned repositories)
• package.json (configures it in 5 scanned repositories)
• .pre-commit-config.yaml (configures it in 1 scanned repositories)
• Makefile (configures it in 1 scanned repositories)
• pyproject.toml (configures it in 1 scanned repositories)

Open-source packages: OSV-Scanner, pip-audit, govulncheck, cargo-deny, Grype Tools seen in scans: cargo-deny (19), dependency-review (12), npm-audit (9), govulncheck (4), cargo-audit (2), pip-audit (2)

V1016 Lockfile consistency

Fails when the lockfile no longer matches the manifest, so CI installs exactly what was reviewed. Run by 16 of 256 scanned repositories.

Sub-techniques:

• V1016.001 Frozen install: Install with the frozen or locked flag so a stale lockfile fails.
• V1016.002 Lock check: Regenerate the lock and fail on any diff.

Craft it:

1. Commit lockfiles for applications.
2. Use frozen installs in CI (npm ci, uv sync –locked, cargo build –locked).
3. Add an explicit lock check step for clearer failures.

Reverse engineer it (look for):

• –frozen-lockfile, –locked or npm ci in CI.
• uv lock –check or a lockfile diff step.
• .github/workflows/*.yml (configures it in 10 scanned repositories)
• .pre-commit-config.yaml (configures it in 9 scanned repositories)
• justfile (configures it in 1 scanned repositories)
• pyproject.toml (configures it in 1 scanned repositories)

Open-source packages: uv (uv lock –check) Tools seen in scans: uv-lock (16)

V1017 SBOM and provenance

Records what went into a build and signs it, so consumers can verify where an artifact came from and what it contains.

Sub-techniques:

• V1017.001 SBOM generation: Emit an SPDX or CycloneDX bill of materials per release.
• V1017.002 Signing and attestation: Sign artifacts and attach build provenance (SLSA).

Craft it:

1. Generate an SBOM in the release workflow and publish it with the artifact.
2. Sign with keyless signing from CI identity.
3. Verify signatures in deployment.

Reverse engineer it (look for):

• Release workflows calling syft, cosign or actions/attest-build-provenance.
• *.sbom.json or *.intoto.jsonl assets on releases.

Open-source packages: Syft, Sigstore cosign, SLSA GitHub generator

V1018 Dependency risk assessment

Judges a new dependency beyond known CVEs: maintenance, project security practices and signs of a malicious package.

Sub-techniques:

• V1018.001 Project health scoring: Score a dependency’s repository practices (branch protection, signed releases, CI).
• V1018.002 Malicious package heuristics: Scan package contents for install-time scripts, obfuscation and exfiltration.

Craft it:

1. Run the scan only when a dependency is added or its source changes.
2. Require a reviewer sign-off below a score threshold rather than failing outright.

Reverse engineer it (look for):

• An ossf/scorecard-action workflow, or guarddog in CI.

Open-source packages: OpenSSF Scorecard, GuardDog

VA04 Testing

Runs the code: unit and integration tests, browser tests, property-based tests, fuzzing, mutation testing.

When: On every change that alters behaviour. Add generated inputs (property tests, fuzzing) when the code parses, transforms or computes, and mutation testing when the tests themselves change.

V1019 Unit and integration tests

Example-based tests that run the code and assert on behaviour, from single functions up to services with real dependencies. Run by 166 of 256 scanned repositories.

Sub-techniques:

• V1019.001 Unit tests: Fast, isolated tests per function or class.
• V1019.002 Integration tests with real services: Run against real databases and queues in containers instead of mocks.
• V1019.003 Regression test per bug: Every fixed bug gets a test that failed before the fix.
• V1019.004 Coverage gate: Fail when coverage of changed lines drops below a threshold.

Craft it:

1. Write the failing test before the fix for every bug.
2. Run integration tests against containers, not mocks, for storage and messaging code.
3. Gate on coverage of changed lines, not overall coverage.

Reverse engineer it (look for):

• tests/ or _test. files, plus test config in pyproject.toml, jest.config or vitest.config.
• A CI test job and a coverage upload step.
• .github/workflows/*.yml (configures it in 138 scanned repositories)
• package.json (configures it in 76 scanned repositories)
• vitest.config.ts (configures it in 32 scanned repositories)
• Makefile (configures it in 31 scanned repositories)
• pyproject.toml (configures it in 26 scanned repositories)
• vitest.config.mts (configures it in 11 scanned repositories)

Open-source packages: pytest, Jest, Vitest, Testcontainers (Python) Tools seen in scans: vitest (91), pytest (77), cargo-test (44), jest (38), node-test (35), go-test (29), unittest (20), mocha (10)

V1020 End-to-end browser tests

Drives a real browser through user flows to catch breakage across components, routing and the network. Run by 25 of 256 scanned repositories.

Sub-techniques:

• V1020.001 Scripted flows: Deterministic scripts for critical paths: sign-in, checkout, settings.
• V1020.002 Agent-driven exploration: An LLM agent drives the browser from a natural-language description of the flow.

Craft it:

1. Cover the few flows that would page someone if broken.
2. Seed test data per run and use stable selectors (roles, test ids).
3. Record traces and screenshots on failure as CI artifacts.

Reverse engineer it (look for):

• playwright.config or cypress.config and an e2e/ directory.
• A CI job that starts the app and runs the browser suite.
• playwright.config.ts (configures it in 21 scanned repositories)
• package.json (configures it in 19 scanned repositories)
• .github/workflows/*.yml (configures it in 17 scanned repositories)
• Makefile (configures it in 1 scanned repositories)

Open-source packages: Playwright, Cypress, Browser Use Tools seen in scans: playwright (55)

V1021 Property-based tests

Generates inputs and checks invariants that must hold for all of them, then shrinks failures to a minimal case. Run by 23 of 256 scanned repositories.

Sub-techniques:

• V1021.001 Round-trip properties: decode(encode(x)) == x for serializers and parsers.
• V1021.002 Model-based properties: Compare the implementation against a simple reference model.
• V1021.003 Stateful testing: Generate sequences of operations and check invariants after each.

Craft it:

1. Start with round-trip and idempotence properties; they need no oracle.
2. Persist the failure database so found cases are replayed.
3. Promote shrunk failures to example tests.

Reverse engineer it (look for):

• @given decorators, fc.assert or proptest! macros in tests.
• .hypothesis/ in .gitignore.
• Cargo.toml (configures it in 11 scanned repositories)
• pyproject.toml (configures it in 10 scanned repositories)
• package.json (configures it in 3 scanned repositories)

Open-source packages: Hypothesis, fast-check, proptest Tools seen in scans: proptest (11), hypothesis (10), fast-check (3)

V1022 Fuzzing

Feeds coverage-guided random input to parsers and decoders to find crashes, panics and hangs. Run by 2 of 256 scanned repositories.

Sub-techniques:

• V1022.001 Coverage-guided fuzzing: libFuzzer or AFL style harnesses per entry point.
• V1022.002 Continuous fuzzing: Run fuzzers continuously (for example OSS-Fuzz) and file crashes as issues.
• V1022.003 Corpus regression: Replay the saved corpus and crash cases in normal CI.

Craft it:

1. Write one harness per parser or decoder entry point.
2. Seed the corpus with real inputs and commit it.
3. Run short fuzz jobs on PRs and long ones on a schedule.

Reverse engineer it (look for):

• fuzz/ directories, fuzz_targets, or project.yaml for OSS-Fuzz.
• CIFuzz or cargo fuzz steps in workflows.
• .github/workflows/*.yml (configures it in 2 scanned repositories)
• Cargo.toml (configures it in 1 scanned repositories)
• fuzz_stream_map.rs (configures it in 1 scanned repositories)
• ruff_fix_validity.rs (configures it in 1 scanned repositories)

Open-source packages: cargo-fuzz, Atheris, Jazzer, AFL++ Tools seen in scans: cargo-fuzz (6)

V1023 Mutation testing

Changes the code on purpose and checks that some test fails; surviving mutants show tests that do not constrain behaviour. Run by 2 of 256 scanned repositories.

Sub-techniques:

• V1023.001 Diff-scoped mutation: Mutate only lines changed in the PR to keep runtime bounded.
• V1023.002 Mutation score gate: Fail when the score on changed code drops below a threshold.

Craft it:

1. Run on one critical module first to see the runtime.
2. Scope to changed lines on PRs.
3. Treat each surviving mutant as a missing assertion, not a number to optimise.

Reverse engineer it (look for):

• stryker.conf, mutmut config or cargo-mutants in CI.
• Mutation reports uploaded as artifacts.
• .github/workflows/*.yml (configures it in 2 scanned repositories)
• package.json (configures it in 1 scanned repositories)
• pyproject.toml (configures it in 1 scanned repositories)
• stryker.conf.json (configures it in 1 scanned repositories)

Open-source packages: mutmut, StrykerJS, cargo-mutants, PIT Tools seen in scans: stryker (3), mutmut (1)

V1024 Version and platform matrix

Runs the suite across runtime versions, operating systems and feature combinations you do not run locally. Run by 116 of 256 scanned repositories.

Sub-techniques:

• V1024.001 Runtime matrix: Every supported language runtime version.
• V1024.002 Feature powerset: Build and test each combination of optional features or extras.
• V1024.003 Minimum versions: Test with the lowest declared dependency versions.

Craft it:

1. Declare supported versions once and generate the matrix from that list.
2. Test the oldest and newest on PRs, the full matrix on main.
3. Add a minimum-versions job so lower bounds are true.

Reverse engineer it (look for):

• strategy.matrix in workflows, tox.ini or noxfile.py.
• cargo hack –feature-powerset or -Z minimal-versions steps.
• .github/workflows/*.yml (configures it in 115 scanned repositories)
• Makefile (configures it in 2 scanned repositories)

Open-source packages: tox, nox, cargo-hack Tools seen in scans: ci-matrix (115), cargo-hack (4)

V1025 Visual regression testing

Compares screenshots of rendered pages or components with approved baselines and reports pixel differences.

Sub-techniques:

• V1025.001 Component snapshots: Screenshot each component state in isolation.
• V1025.002 Page snapshots: Screenshot full pages at fixed viewports.

Craft it:

1. Render in a pinned browser and font environment to avoid noise.
2. Store baselines per branch and approve changes in the PR.
3. Set per-component diff thresholds rather than one global value.

Reverse engineer it (look for):

• screenshots or backstop_data directories.
• toHaveScreenshot assertions or a visual testing step in CI.

Open-source packages: pixelmatch, BackstopJS, Lost Pixel, reg-suit, Playwright

V1026 Sandboxed execution

Builds and runs the change in an isolated environment to confirm it installs, starts and passes its checks before anyone relies on it.

Sub-techniques:

• V1026.001 Ephemeral environment: A fresh container or microVM per change with the repository’s toolchain.
• V1026.002 Reproduce-then-verify: Reproduce a reported bug in the sandbox, apply the fix, and show the same command now passes.

Craft it:

1. Describe the environment as code (a Dagger pipeline or container image).
2. Run the repository’s own install, lint and test commands inside it.
3. Keep the logs as evidence attached to the change.

Reverse engineer it (look for):

• dagger.json, devcontainer.json or a setup script used by CI.
• Bots that post reproduction logs on issues or PRs.

Open-source packages: E2B, Dagger, Testcontainers (Python)

V1027 Dynamic security testing

Attacks a running instance: crawls endpoints and sends payloads to confirm injection, misconfiguration and exposure.

Sub-techniques:

• V1027.001 Baseline scan: Passive scan of a deployed preview for headers, cookies and known misconfigurations.
• V1027.002 Template-based checks: Targeted requests that confirm a specific vulnerability class.

Craft it:

1. Run a baseline scan against preview deployments.
2. Write templates for findings from static analysis so they can be confirmed at runtime.
3. Run active scans only against disposable environments.

Reverse engineer it (look for):

• zap-baseline or nuclei steps in deployment workflows.
• Custom nuclei templates in the repository.

Open-source packages: ZAP, Nuclei

VA05 Formal verification and model checking

Proves or exhaustively explores properties: model checking, proofs, undefined-behaviour interpreters.

When: When a bug would be a memory-safety violation or an interleaving that tests are unlikely to hit: unsafe code, lock-free structures, consensus and protocol logic.

V1028 Undefined-behaviour detection

Executes code under an interpreter or instrumentation that detects memory-safety and aliasing violations. Run by 4 of 256 scanned repositories.

Sub-techniques:

• V1028.001 Interpreter-based checking: Run tests under an interpreter that tracks every pointer (Miri).
• V1028.002 Sanitizers: Compile with address, thread or undefined-behaviour sanitizers and run the tests.

Craft it:

1. Run the unsafe-heavy test subset under the checker in CI.
2. Add sanitizer builds as separate matrix entries.
3. Treat every report as a bug; there are no false positives to baseline.

Reverse engineer it (look for):

• cargo miri test or -Zsanitizer flags in workflows.
• ASAN_OPTIONS or -fsanitize in build scripts.
• .github/workflows/*.yml (configures it in 4 scanned repositories)
• Makefile (configures it in 1 scanned repositories)

Open-source packages: Miri, Sanitizers (ASan, TSan, UBSan) Tools seen in scans: miri (4)

V1029 Concurrency model checking

Explores thread interleavings systematically to find races and orderings that fail only rarely. Run by 1 of 256 scanned repositories.

Sub-techniques:

• V1029.001 Exhaustive interleaving: Enumerate all interleavings of a small test (loom).
• V1029.002 Randomized scheduling: Sample schedules with a controlled scheduler (shuttle).

Craft it:

1. Isolate the concurrent primitive behind an interface the checker can swap.
2. Write small tests with two or three threads.
3. Run them in a dedicated CI job with the checker’s cfg flag.

Reverse engineer it (look for):

• –cfg loom in RUSTFLAGS or loom::model in tests.
• ThreadSanitizer builds in CI.
• .github/workflows/*.yml (configures it in 1 scanned repositories)

Open-source packages: loom, Shuttle, Sanitizers (ASan, TSan, UBSan) Tools seen in scans: loom (1)

V1030 Proofs and specifications

Specifies a protocol or algorithm and checks it exhaustively or proves it, catching design bugs before code exists. Run by 2 of 256 scanned repositories.

Sub-techniques:

• V1030.001 Model checking a design: Write a TLA+ spec and check invariants over all reachable states.
• V1030.002 Verified code: Write code with pre- and post-conditions that a verifier proves (Dafny, Lean, Rocq).
• V1030.003 Proof harnesses on real code: Bounded proofs over the production code itself (Kani).

Craft it:

1. Specify the part where a bug is catastrophic and tests are weak: consensus, replication, locking.
2. Check the spec in CI on every change to it.
3. Link spec actions to code with comments or trace checking.

Reverse engineer it (look for):

• *.tla, *.dfy, *.lean or *.v files and a CI job that runs the checker.
• #[kani::proof] harnesses.
• show_manual_release_cmd_test.v (configures it in 1 scanned repositories)
• v.v (configures it in 1 scanned repositories)

Open-source packages: TLA+ tools, Dafny, Lean 4, Rocq (Coq), Kani Tools seen in scans: coq (2), lean (1), tla-plus (1)

V1031 Symbolic execution

Explores program paths with symbolic inputs and a solver to find inputs that violate an assertion.

Sub-techniques:

• V1031.001 Bounded model checking: Unroll loops to a bound and check all paths (CBMC).
• V1031.002 Contract checking: Search for inputs that break a function’s pre- and post-conditions (CrossHair).

Craft it:

1. Pick small pure functions with clear contracts.
2. Write contracts as assertions or type annotations.
3. Run with a time budget per function in CI.

Reverse engineer it (look for):

• CBMC harnesses, crosshair config or contract decorators in code.

Open-source packages: CBMC, KLEE, CrossHair

VA06 Code review

Reads the change the way a reviewer does: what it is for, what it touches, and where its logic, conventions or security fall short.

When: On every pull request, after the automated checks pass, so reviewers spend attention on what the checks cannot see.

V1032 Change intent extraction

States what the change is meant to do, from title, description, commits and linked ticket, so every later check can ask whether a finding is in scope.

Sub-techniques:

• V1032.001 Change summary: Generate a description and per-file walkthrough of the diff.
• V1032.002 Topic grouping: Group changed files by topic and rate each topic’s importance.

Craft it:

1. Feed title, body, commit messages and linked ticket to a model with a structured output schema.
2. Use a structural diff so moved code is not reported as new.
3. Store the intent with the review so findings can be checked against it.

Reverse engineer it (look for):

• A PR description bot or a generated summary section in PR bodies.

Open-source packages: PR-Agent, Instructor, Difftastic, GumTree

V1033 Review context retrieval

Collects what a reviewer needs beyond the diff: callers, definitions, related tests, docs and repository rules.

Sub-techniques:

• V1033.001 Repository map: A ranked map of symbols and files built from syntax trees.
• V1033.002 Precise code navigation: Definitions and references from a compiler-grade index.
• V1033.003 Cross-repository context: Look up how other services call or are called by the changed code.

Craft it:

1. Index the repository with tree-sitter or SCIP on the default branch.
2. For each changed symbol, retrieve definitions, callers and tests.
3. Cap context by relevance, not file order.

Reverse engineer it (look for):

• SCIP index uploads, tree-sitter grammars or a repo-map cache in tooling.

Open-source packages: tree-sitter, Aider, SCIP, LlamaIndex

V1034 Agentic exploration

An agent with read, search and run tools investigates a change: follows a suspicion across files, runs code, and returns findings with evidence.

Sub-techniques:

• V1034.001 Breadth survey: A first pass that lists concerns across the whole diff.
• V1034.002 Deep exploration: Focused sub-agents that investigate one concern each.
• V1034.003 Tiered depth: Route small or low-risk changes to a shallow pass and risky ones to deep exploration.

Craft it:

1. Give the agent read-only tools first: read file, grep, list directory.
2. Split into a survey step and per-concern investigation steps.
3. Require each finding to cite file, line and the evidence that supports it.

Reverse engineer it (look for):

• Agent configuration files (AGENTS.md, CLAUDE.md) and review bot comments that cite multiple files.

Open-source packages: OpenHands, SWE-agent, LangGraph, RouteLLM

V1035 Logic and correctness review

Reads the change for wrong behaviour: off-by-one, inverted conditions, missed error paths, broken invariants, race conditions.

Sub-techniques:

• V1035.001 Logic bug review: Look for conditions, loops and state transitions that do not match intent.
• V1035.002 Error path review: Check failures, retries and partial writes.
• V1035.003 Type and contract consistency: Check that values crossing module boundaries keep their meaning.

Craft it:

1. Give the reviewer the intent, the diff and retrieved context.
2. Ask for findings only with a concrete triggering input or state.
3. Validate each finding before posting (see Finding validation).

Reverse engineer it (look for):

• Review bot workflows (claude-code-action, pr-agent) and their prompt files.

Open-source packages: PR-Agent, Claude Code Action, OpenHands

V1036 Convention and guideline review

Checks a change against the repository’s written rules: style guides, AGENTS.md, architecture decisions, naming conventions.

Sub-techniques:

• V1036.001 Rule file review: Apply the rules in AGENTS.md, CLAUDE.md or CONTRIBUTING.md to the diff.
• V1036.002 Naming and conciseness: Flag unclear names, typos and needless code.
• V1036.003 Rule-to-lint promotion: Turn rules that can be matched syntactically into lint or Semgrep rules.

Craft it:

1. Keep rules in the repository in one file per scope.
2. Promote any rule that a pattern can match into a lint rule.
3. Leave the rest to a reviewer prompt that quotes the rule it applies.

Reverse engineer it (look for):

• AGENTS.md, CLAUDE.md, .cursor/rules or CONTRIBUTING.md with coding rules.

Open-source packages: PR-Agent, Semgrep CE, typos

V1037 Reviewers learned from past reviews

Mines a repository’s review discussions for recurring feedback and turns each cluster into a reusable review instruction.

Sub-techniques:

• V1037.001 Discussion mining: Collect resolved review threads with the code they were on.
• V1037.002 Topic clustering: Cluster comments into themes with a minimum number of supporting discussions.
• V1037.003 Instruction distillation: Write one instruction per theme with examples from the threads.
• V1037.004 Memory of facts and rules: Extract facts from new discussions and match them to existing rules to update or add them.

Craft it:

1. Export review threads with their file paths and diffs.
2. Embed and cluster comments; keep clusters with enough independent threads.
3. Distill each cluster into an instruction and evaluate it on held-out PRs.

Reverse engineer it (look for):

• The instructions on this site are this technique applied to each source repository.

Open-source packages: Awesome Reviewers, BERTopic, HDBSCAN, Sentence Transformers, Mem0

V1038 Security-focused review

A reviewer that reads for exploitable behaviour across files: authorization gaps, injection through helpers, unsafe trust between services.

Sub-techniques:

• V1038.001 Should-review gate: Decide from the diff whether a deeper security pass is warranted.
• V1038.002 Call-chain investigation: Follow user input from entry point to sink with a model, file by file.
• V1038.003 OWASP classification: Tag each finding with an OWASP Top 10 category for routing and reporting.

Craft it:

1. Gate on diff signals: new routes, auth code, deserialization, subprocess, SQL.
2. Investigate from the entry point to the sink, not file by file.
3. Require a concrete input that reaches the sink.

Reverse engineer it (look for):

• SECURITY.md, threat model docs, or a security review workflow on paths like auth/ or api/.

Open-source packages: Vulnhuntr, Semgrep CE, Joern, SetFit

V1039 LLM application security review

Reviews code that calls models for the OWASP Top 10 for LLM applications: prompt injection, output handling, excessive agency, unbounded consumption.

Sub-techniques:

• V1039.001 Static review of LLM call sites: Check how untrusted text reaches prompts and how outputs reach tools or HTML.
• V1039.002 Red-team probing: Send adversarial prompts to a running application and score the responses.
• V1039.003 Runtime input and output scanning: Scan prompts and responses in production for injection and leakage.

Craft it:

1. List every place untrusted text enters a prompt and every place model output is executed or rendered.
2. Run a red-team suite against those paths in CI.
3. Cap tokens, tool calls and spend per request.

Reverse engineer it (look for):

• promptfoo or garak config, prompt templates, and tool definitions in the repository.

Open-source packages: garak, promptfoo, LLM Guard

V1040 Review question checklists

A fixed bank of questions a reviewer must answer for a kind of change, ranked so the most relevant ones are asked first.

Sub-techniques:

• V1040.001 Universal questions: Questions that apply to any change: error handling, tests, naming.
• V1040.002 Behavioural questions: Questions about what changes for users and callers.
• V1040.003 Security requirement checklists: Verification requirements per area from a standard such as OWASP ASVS.

Craft it:

1. Start from a published checklist and keep only questions your reviews actually raise.
2. Tag each question with the kinds of change it applies to.
3. Rank by relevance to the diff and ask the top few.

Reverse engineer it (look for):

• PR templates with checkbox lists, or review guidelines in CONTRIBUTING.md.

Open-source packages: OWASP ASVS, Google engineering practices, rerankers

V1041 Reviewer routing

Chooses which specialist reviewers, human or automated, apply to a change based on its files, topics and history.

Sub-techniques:

• V1041.001 Path ownership: Route by file path ownership rules.
• V1041.002 Semantic matching: Match the change to reviewer descriptions by embedding similarity and reranking.
• V1041.003 Reviewer merging: Combine overlapping reviewers so one concern is not reviewed twice.

Craft it:

1. Start with CODEOWNERS for humans.
2. Describe each automated reviewer in one paragraph and match changes to it.
3. Log which reviewer produced accepted findings and prune the rest.

Reverse engineer it (look for):

• CODEOWNERS, review assignment bots, or reviewer config files.

Open-source packages: Sentence Transformers, rerankers, SetFit

VA07 Finding validation

Checks the findings themselves before anyone acts on them: is there proof, can it happen, is it new, how bad is it, is it in scope.

When: Whenever a reviewer, human or model, produces more findings than people will read, or findings that are often wrong.

V1042 Evidence-backed verification

Re-checks each finding against the code before it is shown: look for counterevidence, require proof, and refute or suppress what does not hold.

Sub-techniques:

• V1042.001 Counterevidence search: Look for guards, callers or tests that show the claimed failure cannot happen.
• V1042.002 Proof requirement: Drop findings that cite no line, input or behaviour.
• V1042.003 Run to confirm: Write and run a test or script that demonstrates the failure.
• V1042.004 Atomic findings: Split findings that make more than one claim so each is verified separately.

Craft it:

1. Run a second, independent pass per finding with the code and the claim only.
2. Record a verdict per finding: confirmed, refuted, suppressed or incomplete, with the reason.
3. Only post confirmed findings; measure refutation rate per reviewer.

Reverse engineer it (look for):

• Bot comments that cite evidence, or review tooling that records dismissed findings.

Open-source packages: DSPy, DeepEval, Ragas, E2B

V1043 Path reachability check

Decides whether the triggering state can reach the flagged line on this code path, given guards, early returns and types.

Sub-techniques:

• V1043.001 Guard analysis: Find early returns, enclosing conditions and defaults that exclude the trigger.
• V1043.002 Caller analysis: Find a real caller, config or workflow that produces the triggering state.

Craft it:

1. For each finding, list the callers of the flagged function.
2. Check each caller for guards that exclude the trigger.
3. Mark findings with no established producer as unestablished, not confirmed.

Reverse engineer it (look for):

• Static analyzer configs with path-sensitive checks enabled.

Open-source packages: Infer, Joern, SCIP

V1044 External contract verification

Checks claims about third-party behaviour (an API can return null, a library throws) against authoritative documentation, and grades how strong that proof is.

Sub-techniques:

• V1044.001 Documentation lookup: Fetch the exact library version’s documentation or source.
• V1044.002 Proof grading: Grade as definitive (spec or exhaustive enum), illustrative (an example) or unresolved.

Craft it:

1. Resolve the dependency version from the lockfile.
2. Fetch that version’s docs or source, not the latest.
3. Keep findings that rest on illustrative proof at lower severity.

Reverse engineer it (look for):

• Documentation MCP servers or doc lookup tools configured for the review agent.

Open-source packages: Context7, SearXNG

V1045 Finding deduplication

Merges findings that describe the same problem from different reviewers, files or runs into one.

Sub-techniques:

• V1045.001 Semantic dedup: Cluster findings by embedding similarity and location.
• V1045.002 Consolidation: Merge a cluster into one finding that keeps all evidence.
• V1045.003 Fingerprinting across runs: Stable fingerprints so a finding is recognised on the next commit.

Craft it:

1. Emit findings as SARIF or a common schema with file, line and rule.
2. Fingerprint by rule, file and normalised code, not line number.
3. Cluster the remainder by embedding and merge.

Reverse engineer it (look for):

• SARIF partialFingerprints, or a findings table keyed by fingerprint.

Open-source packages: Sentence Transformers, FAISS, sarif-tools

V1046 Severity scoring

Scores a finding from separate evidence axes (impact, persistence, detectability, reachability, scope, certainty) instead of a single guess.

Sub-techniques:

• V1046.001 Impact category: Data loss, security, crash, wrong behaviour, performance, leak or maintainability.
• V1046.002 Persistence and detectability: Transient, until restart or permanent; loud, silent or delayed.
• V1046.003 Blast radius: One operation, one user, a subset, broad or systemic.
• V1046.004 Certainty basis: Demonstrated by a run, traced in code, inferred from a name, or assumed.
• V1046.005 Standard vulnerability scoring: CVSS vectors, SSVC decisions and EPSS likelihood for security findings.

Craft it:

1. Have the reviewer fill each axis separately with evidence.
2. Derive severity from the axes with a fixed table, never from free text.
3. Cap severity when certainty is inferred or assumed.

Reverse engineer it (look for):

• Severity rubrics in review guidelines, or CVSS vectors in security advisories.

Open-source packages: cvss (Python), CERT/CC SSVC

V1047 Scope and relevance filtering

Keeps only findings that belong to this change: on changed lines, within the PR’s intent, and fixable without reworking unchanged code.

Sub-techniques:

• V1047.001 Diff scoping: Report only on changed lines or their immediate context.
• V1047.002 Intent scoping: Move findings outside the PR’s intent to follow-up work.
• V1047.003 Remediation scope: Check whether the fix stays inside the diff or reaches unchanged code.

Craft it:

1. Filter by diff hunks first; it is cheap and removes most noise.
2. Check each remaining finding against the stated intent.
3. File out-of-scope findings as issues instead of review comments.

Reverse engineer it (look for):

• reviewdog -filter-mode settings, or bots that only comment on changed lines.

Open-source packages: reviewdog, Danger JS

V1048 Noise budget

A fixed cap on how many findings are posted per change, filled in severity order, so the important ones are read.

Sub-techniques:

• V1048.001 Volume cap: At most N comments per review, highest severity first.
• V1048.002 Carried findings: Findings that did not fit are kept and offered on the next run instead of dropped.

Craft it:

1. Set the cap from how many comments reviewers actually address.
2. Sort by severity, then certainty.
3. Store the overflow and resurface it when the budget allows.

Reverse engineer it (look for):

• Review bot config with a maximum comment count.

Open-source packages: reviewdog, sarif-tools

V1049 Confidence calibration

Maps a reviewer’s confidence to the observed rate of accepted findings and sets the posting threshold from that curve.

Sub-techniques:

• V1049.001 Calibration curve: Fit confidence against accepted or rejected outcomes.
• V1049.002 Threshold per reviewer: Each reviewer gets its own threshold.

Craft it:

1. Log confidence and the human outcome for every posted finding.
2. Fit a calibration model monthly.
3. Pick the threshold that meets a target precision.

Reverse engineer it (look for):

• Thumbs up or down reactions collected on bot comments.

Open-source packages: scikit-learn

V1050 Reflection pass

A self-critique step where the reviewer re-reads its own findings with fresh context and keeps, revises or withdraws each.

Sub-techniques:

• V1050.001 Finding reflection: Re-examine each finding against the code and the base-to-head diff.
• V1050.002 Discussion-aware reflection: Include existing review threads so the reviewer does not repeat or contradict them.

Craft it:

1. Run reflection with a different prompt and, ideally, a different model than the reviewer.
2. Pass the code, not the reviewer’s reasoning.
3. Record why each withdrawn finding was withdrawn.

Reverse engineer it (look for):

• Multi-step review agent graphs with a critique node.

Open-source packages: LangGraph, DSPy

V1051 Past discussion check

Compares a finding with earlier review discussions: a repeat of something already decided is suppressed, a regression of a fixed issue is raised.

Sub-techniques:

• V1051.001 Repeat suppression: Do not re-raise what a human already dismissed.
• V1051.002 Regression detection: Raise severity when a change reintroduces a previously fixed problem.

Craft it:

1. Store resolved and dismissed threads with their code location and outcome.
2. Retrieve similar threads for each new finding.
3. Suppress repeats; escalate regressions.

Reverse engineer it (look for):

• Resolved review threads that mention an earlier PR or issue.

Open-source packages: Mem0, Graphiti, Sentence Transformers

VA08 Intent and spec conformance

Compares the change to what was asked for: the ticket, the plan or spec, the design, the review comments it was meant to address.

When: When a change implements a ticket, a plan or a design, and before merging, to confirm the thing built is the thing requested.

V1052 Ticket intent alignment

Checks a finding or a change against the linked ticket: behaviour the ticket declares intentional is not a bug, and requirements it lists must be present.

Sub-techniques:

• V1052.001 Intended per ticket: Suppress findings for behaviour the ticket explicitly asks for or defers.
• V1052.002 Ticket coverage: List ticket requirements and check each against the diff.

Craft it:

1. Link tickets in PR titles or branch names so they can be fetched.
2. Extract requirements as a structured list.
3. Match only on explicit statements; treat silence as unknown.

Reverse engineer it (look for):

• Ticket keys in branch names or PR titles, and issue-tracker integrations.

Open-source packages: jira (Python), Instructor

V1053 Plan and spec traceability

Checks each requirement in a plan or specification against the implementation: done in this change, done elsewhere, or not started, with gaps graded by severity.

Sub-techniques:

• V1053.001 Requirement tracing: Link requirement ids to code and tests.
• V1053.002 Gap analysis: Report where the implementation departs from the plan’s intent.

Craft it:

1. Write specs with stable requirement ids.
2. Reference the ids in code comments or tests.
3. Report untraced requirements and code with no requirement.

Reverse engineer it (look for):

• specs/ or docs/adr directories, requirement ids in tests.

Open-source packages: Spec Kit, OpenFastTrace, Instructor

V1054 Functional spec verification

Runs the application and checks each functional requirement in a browser, returning met, partially met, not met or unverifiable with screenshot evidence.

Sub-techniques:

• V1054.001 Browser verification: An agent or script performs the flow and checks the outcome.
• V1054.002 Evidence capture: Screenshot and annotate each verdict.

Craft it:

1. Deploy a preview per PR.
2. Turn each acceptance criterion into a step and an observable outcome.
3. Attach screenshots to every verdict, including failures.

Reverse engineer it (look for):

• Preview deployment workflows and acceptance criteria in tickets.

Open-source packages: Browser Use, Playwright, DeepEval

V1055 Design conformance

Compares the rendered UI with the design it implements and reports layout, spacing, colour and copy differences.

Sub-techniques:

• V1055.001 Design-to-render comparison: Diff a screenshot against the exported design frame.
• V1055.002 Annotated screenshots: Mark the differing regions so the fix is obvious.

Craft it:

1. Export frames at the same viewport the test renders.
2. Compare with a tolerance and mask dynamic content.
3. Annotate differing regions on the screenshot.

Reverse engineer it (look for):

• Design links in tickets or PRs, and exported frames in the repository.

Open-source packages: pixelmatch, OpenCV, Pillow, Playwright

V1056 Review comment resolution

Checks whether a new commit actually addresses each open review comment before the thread is resolved.

Sub-techniques:

• V1056.001 Addressed check: Compare the comment’s request with the code that changed after it.
• V1056.002 Thread classification: Classify threads as addressed, disputed, deferred or open.

Craft it:

1. Fetch open threads and the commits after each.
2. Ask whether the requested change is present, with the diff hunk as evidence.
3. Resolve only threads with evidence; reply on the rest.

Reverse engineer it (look for):

• Bots that resolve threads or post follow-ups after pushes.

Open-source packages: PyGithub, Instructor

V1057 Merge readiness

Combines checks, reviews, open threads and findings into one verdict on whether a change can merge.

Sub-techniques:

• V1057.001 Merge rules: Required checks, approvals and labels as code.
• V1057.002 Merge verdict summary: A summary of what is still blocking, with links.

Craft it:

1. Encode required checks and approvals in branch protection.
2. Add a bot that lists what is still blocking.
3. Merge automatically only when the verdict is clean.

Reverse engineer it (look for):

• Branch protection rules, .kodiak.toml or merge queue config.

Open-source packages: Kodiak, Danger JS

V1058 Output guardrails

Validates what an automated reviewer or fixer produces before it is posted or committed: schema, policy and safety checks.

Sub-techniques:

• V1058.001 Schema validation: Reject malformed structured output and retry.
• V1058.002 Policy rails: Block outputs that leak secrets, change protected files or exceed scope.

Craft it:

1. Define the output schema and validate every response.
2. List protected paths and actions the bot may not take.
3. Log every blocked output for review.

Reverse engineer it (look for):

• Guardrail config files or validators around model calls.

Open-source packages: Guardrails AI, NeMo Guardrails, Instructor

VA09 Threat modeling

Maps where a system can be attacked, finds the weaknesses behind each entry point and checks which are mitigated and exposed.

When: On a whole repository periodically, and on changes that add an entry point, a trust boundary or a new dependency on external input.

V1059 Attack surface inventory

Lists every entry point and trust boundary: API endpoints, webhooks, queues, schedulers, parsers, CLIs, third-party scripts, model calls.

Sub-techniques:

• V1059.001 Network entry points: HTTP, gRPC, WebSocket, raw socket and webhook receivers.
• V1059.002 Asynchronous entry points: Queue consumers and scheduled jobs that act on external data.
• V1059.003 Data parsers and deserialization: File, archive and object parsers that take untrusted input.
• V1059.004 Infrastructure and configuration: Infrastructure as code, secrets management, caching and edge routing.
• V1059.005 AI and model interactions: Prompts, tools and retrieved content that cross a trust boundary.

Craft it:

1. Enumerate entry points from route tables, consumers and schedulers in code.
2. Describe the system as a model (pytm, Threagile) with data flows and trust boundaries.
3. Regenerate the model when entry points change.

Reverse engineer it (look for):

• threagile.yaml, pytm models or threat model documents in docs/.

Open-source packages: OWASP pytm, Threagile, OWASP Threat Dragon

V1060 Weakness discovery

For each entry point, searches for the weakness classes that apply to it: injection, broken access control, SSRF, deserialization, path traversal.

Sub-techniques:

• V1060.001 Per-surface weakness search: Apply the weakness classes relevant to each surface category.
• V1060.002 Multi-tenancy isolation: Check every query and cache key for tenant scoping.

Craft it:

1. Map each surface category to the weakness classes that apply.
2. Search with taint queries and an investigating agent.
3. Record each weakness with the entry point it is reachable from.

Reverse engineer it (look for):

• Security audit reports or issues labelled security.

Open-source packages: Joern, Vulnhuntr, Semgrep CE

V1061 Mitigation mapping

Checks whether each weakness is already mitigated by a control (validation, authorization middleware, rate limits) before reporting it.

Sub-techniques:

• V1061.001 Control inventory: List the security controls and where they apply.
• V1061.002 Coverage check: Report entry points that bypass the controls.

Craft it:

1. List controls as code locations (middleware, decorators, gateways).
2. Check each entry point passes through them.
3. Report only unmitigated weaknesses.

Reverse engineer it (look for):

• Auth middleware, rate limiter and validation layers in the code.

Open-source packages: Threagile, OWASP pytm

V1062 Exposure verification

Confirms which weaknesses are reachable from outside: external asset discovery and targeted checks against deployed endpoints.

Sub-techniques:

• V1062.001 External asset discovery: Enumerate domains, subdomains and exposed services.
• V1062.002 Targeted confirmation: Send a request that demonstrates the weakness on a test deployment.

Craft it:

1. Discover assets on a schedule.
2. Map findings to exposed assets.
3. Confirm with targeted templates against non-production environments.

Reverse engineer it (look for):

• Scheduled scan workflows and nuclei templates.

Open-source packages: OWASP Amass, Nuclei, ZAP

V1063 Risk prioritization

Orders confirmed weaknesses by exposure, exploit likelihood and impact so fixes start where risk is highest.

Sub-techniques:

• V1063.001 Decision trees: SSVC decisions from exploitation, exposure and impact.
• V1063.002 Likelihood scores: EPSS for known vulnerabilities.

Craft it:

1. Score exposure from the attack surface inventory.
2. Combine with likelihood and impact through a fixed decision tree.
3. Recompute when exposure changes.

Reverse engineer it (look for):

• Vulnerability management dashboards or SECURITY.md response targets.

Open-source packages: CERT/CC SSVC, cvss (Python)

VA10 Feedback and evaluation

Closes the loop: production errors and missed bugs become new checks, and the reviewers themselves are measured and tuned.

When: Continuously, once checks and reviewers are in place. Every escaped bug is a missing technique somewhere to the left.

V1064 Production error investigation

Investigates production errors from observability data, finds the responsible code and proposes a fix with a regression test.

Sub-techniques:

• V1064.001 Error-to-code mapping: Map a stack trace or log pattern to the commit and lines responsible.
• V1064.002 Automated fix proposal: Open a change with the fix and a test that reproduces the error.

Craft it:

1. Connect the agent to logs, traces and the repository read-only.
2. Require a reproduction before any fix.
3. Route fixes through the same review as human changes.

Reverse engineer it (look for):

• Observability integrations and incident runbooks in the repository.

Open-source packages: HolmesGPT, K8sGPT, OpenHands

V1065 Alert deduplication and triage

Groups related production alerts and errors and ranks them so investigation starts with the most impactful.

Sub-techniques:

• V1065.001 Alert correlation: Group alerts by fingerprint, service and time window.
• V1065.002 Impact ranking: Rank by users affected, frequency and recency.

Craft it:

1. Fingerprint errors by normalised stack trace.
2. Correlate within a time window per service.
3. Rank before routing to people or agents.

Reverse engineer it (look for):

• Alert routing and grouping rules in monitoring config.

Open-source packages: Keep, HDBSCAN

V1066 Missed bug analysis

For each bug that escaped review, finds which technique should have caught it and why it did not, then clusters the causes.

Sub-techniques:

• V1066.001 Root cause trace: Replay the review trace for the PR that introduced the bug.
• V1066.002 Cause clustering: Cluster root causes to find the systematic gaps.
• V1066.003 Eval set growth: Add each missed bug to an evaluation set for the reviewers.

Craft it:

1. Link every bug fix to the PR that introduced the bug.
2. Trace what each check and reviewer saw on that PR.
3. Add the case to the eval set and re-run reviewers on it.

Reverse engineer it (look for):

• Postmortems that name the introducing PR, and tracing for review agents.

Open-source packages: Langfuse, Arize Phoenix, BERTopic

V1067 Reviewer evaluation

Benchmarks reviewers on labelled changes: did they find the known bugs, and how many findings were wrong.

Sub-techniques:

• V1067.001 Labelled benchmark: PRs with known bugs and known clean changes.
• V1067.002 Model as judge: A judge model matches reviewer findings to the labelled bugs.
• V1067.003 Side-by-side comparison: Run two reviewer versions on the same set and compare.

Craft it:

1. Build the set from real escaped bugs and from changes that needed no comments.
2. Score recall on bugs and precision on findings.
3. Gate reviewer changes on the benchmark.

Reverse engineer it (look for):

• eval/ or benchmarks/ directories with labelled cases.

Open-source packages: Inspect, promptfoo, DeepEval

V1068 Reviewer optimization

Tunes reviewer prompts and instructions against the evaluation set instead of by hand.

Sub-techniques:

• V1068.001 Prompt optimization: Search prompt variants against a metric.
• V1068.002 Instruction updates from feedback: Update reviewer instructions from accepted and rejected findings.

Craft it:

1. Define the metric first (from reviewer evaluation).
2. Optimize on a train split; report on a held-out split.
3. Review every prompt change like code.

Reverse engineer it (look for):

• Optimizer scripts and versioned prompt files in the repository.

Open-source packages: DSPy, TextGrad

V1069 Developer feedback capture

Records how developers respond to findings (fixed, dismissed, argued) and the sentiment of replies, as the ground truth for every other loop.

Sub-techniques:

• V1069.001 Outcome tracking: Was the finding fixed in a later commit, dismissed or ignored.
• V1069.002 Reply sentiment: Classify replies as agreeing, disputing or neutral.

Craft it:

1. Store every finding with its later outcome.
2. Sample disputed findings for human labelling.
3. Feed labels to calibration and evaluation.

Reverse engineer it (look for):

• Reactions and replies on bot comments, collected by a webhook.

Open-source packages: Argilla, Label Studio, VADER

V1070 CI failure analysis

Reads failing CI runs, separates real failures from flaky ones, and maps each to the job, step and change that caused it.

Sub-techniques:

• V1070.001 Flake detection: Re-run and compare to tell flaky tests from real failures.
• V1070.002 Workflow structure analysis: Map jobs and steps to the checks they run (the scanner behind this site does this).

Craft it:

1. Record test outcomes per run, not just the job status.
2. Quarantine tests that fail and pass on the same commit, and file an issue.
3. Never skip a test to get a green run.

Reverse engineer it (look for):

• Retry settings, flaky markers and test result uploads in CI.

Open-source packages: actionlint, pytest

This site is open source. Improve this page.