How do you debug a bug that only happens sometimes?

Turn “sometimes” into “under these conditions”. Intermittent bugs almost always have a pattern; the job is to collect enough evidence to see it.

Gather data

• Add logging around the suspect area: inputs, timing, state
• Record every occurrence with time, user, environment and what was running
• Look for patterns: time of day, load, specific data, a certain device or browser

Common causes

• Race conditions between threads, requests or background jobs
• Caches serving stale data
• Timeouts under slow networks or heavy load
• Order-dependent tests or shared test data
• Differences between environments

Force the conditions

Once you suspect a cause, try to make it happen more often: add artificial delays, run the scenario in a loop, throttle the network or run tests in random order. A bug that fails one time in fifty becomes manageable when you can make it fail on demand.

Fix with care

Keep the extra logging until the fix has run in production for a while. Then trim it.

This fits step 3 of the debugging workflow.

All questions