Performance-conscious metrics implementation

Implement metrics collection that is both comprehensive and minimally impactful on system performance. Design your metrics system to avoid creating bottlenecks in critical paths.

Key principles:

1. Avoid high-overhead operations in critical paths: Don’t spawn new tasks for each metric update in hot code paths.

// Avoid this:
tokio::spawn(async move {
    // Update metrics
});

// Prefer this:
store.add_write_metrics(num_lines, payload_size);

1. Use appropriate buffer sizing for telemetry channels based on expected throughput. For high-volume services, consider larger buffers (e.g., 10k) or direct counter updates instead of channels.

// Consider higher capacity for high-volume metrics
let (sender, receiver) = mpsc::channel(10_000);

1. Track both success and failure metrics for all critical operations to provide a complete picture of system behavior.

// Example counters for both successful and failed operations
const WRITE_LINES_TOTAL_NAME: &str = "influxdb_write_lines_total";
const WRITE_LINES_REJECTED_TOTAL_NAME: &str = "influxdb_write_lines_rejected_total";

1. Ensure proper isolation of metrics across components. When using multiple executors or processing paths, avoid sharing metric registries if they cause conflicts.

Remember that metrics collection should provide valuable insights without becoming a performance bottleneck itself.