Metric design best practices

Design metrics to be reliable and maintainable by following these key principles:

1. Initialize metrics with zero values to ensure consistent existence:

   labelValues := []string{keyspace, shard, tabletType}
   metrics.vstreamsEndedWithErrors.Add(labelValues, 0)
   

2. Choose appropriate metric granularity:
   • Consider per-component metrics when aggregation could mask issues
   • For streaming/processing, track per-shard metrics to identify bottlenecks
   • Avoid overly granular labels (e.g., hostnames) that can explode cardinality
3. Select stable label dimensions:
   • Use static identifiers (keyspace, shard, type)
   • Avoid ephemeral values like hostnames in container environments
   • Consider the metric lifespan and deployment environment
4. Handle metric lifecycle properly:
   • Avoid metric re-registration issues
   • Implement proper cleanup for deprecated metrics
   • Document metric deprecation with clear replacement paths

These practices ensure metrics remain useful for debugging production issues while staying maintainable over time.