Provide a clear, stable, and discoverable configuration schema. Apply the following rules to all TOML/environment configs:

1) Use self-documenting values

• Avoid magic integers/booleans for multi-way choices. Prefer explicit enum/string values. Example: replace direction numeric codes with names:

scorers = [ { plugin = “heretic.scorers.refusal_rate.RefusalRate”, direction = “MINIMIZE”, scale = 1.0 } ]

• Use named methods for optional features (future-proofing):

quantization_method = “bnb_4bit” # instead of load_in_4bit = true

2) Encode mutually exclusive or dependent options as a single enum

• When multiple booleans imply combinations (e.g., compute_normalized & preserve_magnitudes), replace them with an enum so the intent is explicit and invalid combos are impossible:

3) Standardize plugin-scoped tables and plugin import paths

• Use a documented convention for plugin configuration keys so settings are discoverable and validated. Example conventions:

For scorer-wide settings

[scorer.RefusalRate]

For instance-specific settings (suffix with instance name)

[scorer.RefusalRate_instanceA]

• Plugin implementations should be specified as importable module paths (importlib-style). Example:

tagger = “heretic.taggers.keyword.KeywordRefusalDetector”

• If filesystem-path loading is required, document and implement it explicitly; otherwise require importable module paths and document how to extend PYTHONPATH.

4) Validate and warn on conflicts or removed/renamed keys

• Emit a config validation step that:
  • Ensures file ordering/keys are present and documents preferred order (e.g., mirror config.py for readability).
  • Warns when plugin-scoped keys collide with instance-name conventions.
  • Flags removed or interdependent settings (e.g., deprecated kl_divergence_scale/kl_divergence_target) and provide migration guidance.

5) Maintain backward-compatibility with deprecation policy

• When changing a config shape, provide a clear migration path and keep the old key working with warnings for at least one release, or provide a converter tool to update user configs.

How to apply in practice

• Implement a config schema validator that enforces enums, recognized plugin table names (scorer. and scorer._), and emits warnings for collisions and deprecated keys.
• Document examples in config.default.toml and note the canonical order (e.g., follow config.py order) so users can scan settings easily.

Example (combined TOML):

Top-level list of scorers with explicit directions

scorers = [ { plugin = “heretic.scorers.refusal_rate.RefusalRate”, direction = “MINIMIZE”, scale = 1.0 } ]

Plugin path (importable module path)

tagger = “heretic.taggers.keyword.KeywordRefusalDetector”

Plugin-specific settings

[scorer.RefusalRate] good_evaluation_prompts = { dataset = “mlabonne/harmless_alpaca”, split = “test[:100]”, column = “text” }

Enum-valued options

quantization_method = “bnb_4bit” row_normalization = “row_normalize_and_restore_magnitudes”

References: discussions 0–7.