When implementing parsing/classification logic, treat the input grammar as a contract: (1) constrain pattern matches to the intended envelope/boundaries/character sets (avoid prefix/superset regexes and overly broad Unicode ranges), (2) ensure every computed signal/metric is actually used in the final decision, and (3) add targeted regression tests for each boundary/grammar edge.
Apply:
\b, attribute-aware tag opening).Example (grammar-constrained marker stripping):
import re
def strip_asr_envelope(text: str) -> str:
# Only remove the Qwen-style start-anchored envelope, not any occurrence.
m = re.match(r"^language\s+.*?<asr_text>", text, flags=re.IGNORECASE | re.DOTALL)
if not m:
return text
# Remove the first envelope marker then keep the rest.
return text.replace(m.group(0), "", 1).strip()
Checklist for PRs in this area:
Enter the URL of a public GitHub repository