When shared state is mutated by multiple threads, background readers/waiters, or async callbacks, treat every check-then-act sequence as a potential race and design for it explicitly:
Re-validate state after acquiring a lock, not before. A “shutting down” or “already running” flag checked outside a lock can go stale by the time the lock is taken; recheck it inside the critical section before spawning, replacing, or mutating.
let generation = NEXT_GENERATION.fetch_add(1, Ordering::Relaxed);
// ... later, in the waiter thread:
if map.get(&id).is_some_and(|s| s.generation == generation) {
map.remove(&id); // only remove if we still own this slot
}
When two locks must ever be held together, pick and document one fixed acquisition order (e.g., “process before ws_endpoint, never the reverse”) and ensure every code path, including teardown/Drop, follows it to avoid deadlocks.
Prefer holding a single lock across an entire check-and-act operation (e.g., “is a process already running? if not, spawn one”) over separate lock/unlock steps, so concurrent callers serialize instead of racing to spawn duplicates or overwrite each other’s work. For TOCTOU-sensitive operations like replacing an installed binary, re-verify the precondition (e.g., version/epoch) under the same lock immediately before the destructive action.
Applying these practices prevents duplicate spawns, resource leaks, deadlocks, and stale background work from corrupting state owned by its replacement.