Engineering Workflows
Investigating Customer Bugs You Cannot Reproduce
A practical investigation method for intermittent customer bugs: document failed attempts, test meaningful differences, collect targeted evidence, and pause with a clear restart condition.
An investigation note that says “could not reproduce” leaves the next engineer with little to use. In a hypothetical invitation failure, an engineer sends five fresh invitations from a test administrator account and accepts each one. The customer pressed Accept on an invitation sent the previous day, after an administrator changed the invitee's intended role, and saw an error instead of confirmation. The engineer and customer tested different starting states.
Support and engineering should investigate an intermittent customer bug by preserving one identifiable occurrence, documenting the conditions of each reproduction attempt, and choosing experiments that distinguish plausible causes. After a bounded investigation, either pursue a supported hypothesis, add targeted diagnostics for the next occurrence, or pause with a named reopening condition. Keep customer impact separate from reproduction confidence so an uncertain cause does not obscure an urgent problem.
Record the limits of a failed attempt
An intermittent bug is a reported failure that occurs on some occasions but does not occur on demand under the team's current reproduction procedure. That definition describes the investigation state. Engineers may later find a repeatable trigger, such as a particular sequence of role changes.
Mozilla asks reporters to state whether they can reproduce a problem at will, occasionally, or not at all. For a failure they cannot reproduce, its guidance asks for unique information about the occurrence. Mozilla's Bug Writing Guidelines also distinguish observations from speculation.
Use that distinction in the ticket. Preserve the customer's observed outcome, the intended behavior, a time with timezone, the affected invitation or other object, and any available request identifier. Mark approximate times and remembered details as uncertain. The timestamp or request identifier gives an engineer a search path even when the customer cannot repeat the sequence.
A “cannot reproduce” result is an observation about a named test setup: the engineer did not observe the reported failure under the recorded conditions and attempts. A useful reproduction record states the starting state, product build, relevant configuration, actions, attempt count, observed result, and remaining differences from the customer occurrence. This proposed record applies Mozilla's distinction between observations and speculation to engineering attempts.
For the invitation example, replace the empty note with: engineer tested build 6.4 in staging, created five fresh invitations for the member role, accepted them in new sessions, and observed five successful memberships. The engineer has not yet tested old invitations, changed roles, or an existing signed-in session.
Protect the customer's work before testing theories
Support should establish whether the customer can continue and whether the failed action left uncertain state. In the invitation example, inspect membership before recommending another invitation. A visible error can accompany a completed operation; the server might have added the member before the interface failed to show confirmation.
Use a verified workaround where one exists, record its limits, and assign a customer reply owner. Ask an authorized administrator to use a verified membership-management path if that avoids the suspected invitation flow.
For data loss, unauthorized access, or a widespread interruption, invoke the team's urgent response process. Google SRE's Effective Troubleshooting recommends matching response to impact and prioritizing mitigation during a major outage. Engineers should preserve available evidence while protecting users from further harm.
Keep customer consequence and diagnostic confidence in separate fields. An engineer's clean staging run should not lower the consequence of a production access failure.
Choose an experiment that can change your mind
Begin with a small set of hypotheses tied to the occurrence. A hypothesis is a proposed explanation that an engineer can test. For the invitation failure, candidates might include outdated role information in the invitation, interaction with an existing session, or a delayed membership write. These are illustrative possibilities, not findings about a real product.
Google SRE's troubleshooting method starts with forming hypotheses from observations and testing them against system evidence or controlled changes. It also cautions that tests can have side effects and produce suggestive rather than definitive results. Apply that method by stating the predicted difference before running the test.
Use an attempt table in the existing issue:
| Proposed explanation | Controlled change | Evidence to inspect | Meaning of a clean run |
|---|---|---|---|
| Role information differs after an administrator edits an invitation | Compare unchanged and role-edited invitations with the same age and session setup | Recorded invitation role and resulting membership role | This tested edit sequence did not trigger the symptom |
| Existing session state contributes | Accept equivalent invitations in signed-out and signed-in sessions | Session identity, response, resulting membership | These session conditions did not trigger the symptom |
| Membership completion is delayed | Introduce a controlled delay in an isolated test environment | Order of completion and interface updates | The selected delay did not expose the suspected timing failure |
Change one factor first so the owner can interpret the result. Then combine factors if the customer occurrence requires an interaction. Testing only isolated factors would miss a failure that needs both an older invitation and a changed role.
Reset the starting state between attempts. Record which invitation, session, configuration, and build the engineer used. Use synthetic fixtures in an isolated environment.
Control state and timing before counting clean runs
Repeated runs can help expose a failure, but an engineer should know what each run exercises. Ten fresh sessions explore a different condition from ten attempts in one accumulated session. Reusing an invitation after acceptance may test an invalid-token path instead of the original bug.
Martin Fowler's 2011 article, Eradicating Non-Determinism in Tests, identifies test isolation, asynchronous behavior, remote services, time, and resource leaks as common sources of unstable test outcomes. The article concerns automated tests; those categories are useful prompts for designing an intermittent-bug experiment, not proof of the customer's cause.
For invitation age, control the test clock instead of waiting overnight. For delayed completion, wait for a named state with a timeout rather than adding an arbitrary sleep. For shared state, rebuild a known fixture before each run. A test double, a controllable substitute for an external dependency, can expose a specific failure response, although the engineer still needs checks against the real integration.
An engineer who gets a failure once should save the exact setup before simplifying it. Remove steps or vary state until the team understands which conditions matter. Keep the original failing case so the simplification does not erase the evidence.
A clean run after a code change supports a fix only to the extent that the check would have detected the failure before the change. If the team never captured the trigger, label the change as a mitigation or suspected fix and define how the team will evaluate it after release.
Make the next occurrence cheaper to investigate
After repeated clean runs with no new distinction, choose what engineering needs to observe next. Targeted diagnostics are measurements added to answer a specific unresolved question.
For the invitation example, an engineer might need the invitation state at acceptance, the member-creation result, and the order of completion. Attach a correlation identifier, an identifier that connects related events, so an authorized investigator can follow one attempt across components.
Write the diagnostic request as a decision: determine whether membership creation completed before the interface reported failure. Name the event fields, access scope, collection window, and person who will inspect the result. Prefer state labels and identifiers over invitation contents, email addresses, or tokens. Review whether an identifier itself links to personal data.
Ask the customer for one lightweight action if the problem happens during normal work: send the ticket reference and approximate time with timezone. Explain which facts the team can retrieve from its own systems. Offer an approved evidence route if the visible result matters. Do not ask the customer to keep retrying an action that might duplicate work or change access.
Before adding verbose logging, consider overhead and timing effects. Google SRE's guidance on experimental side effects warns that added logging can worsen latency and confuse experimental results. Give temporary diagnostics an expiry and removal owner.
Pause by evidence value, with a clear restart condition
Use a proposed investigation budget that fits the impact: for example, one engineer spends an hour comparing the known occurrence with staging and running the two most discriminating safe tests. Adapt the budget to impact; a serious integrity failure warrants a different response.
At the review point, record what the engineer learned, which conditions remain untested, and whether another attempt would separate the remaining explanations.
An intermittent-bug investigation should continue when the next safe step can distinguish plausible causes or protect affected work. Pause repeated reproduction attempts when they provide no new distinction; name the missing evidence, its collection owner, the next review date, and the event that will restart investigation. Escalate according to customer impact even while the cause remains uncertain. This is a proposed operating rule, not an automatic closure policy.
Define an observable restart condition: another acceptance failure with a correlation identifier, a second affected account with matching state, or a failure under the role-edit experiment. Avoid leaving “more reports” undefined. Support should tell the customer what remains unresolved, what the team has tried, and when the team will review the issue again.
Keep agent investigations attached to the attempt history
Ask a coding agent to inspect a relevant code path, compare a recent change with a hypothesis, or prepare a controlled test fixture. Give the agent the existing attempt history and request one discriminating investigation. Ask it to return conditions, findings, and uncertainty before proposing a patch. Human reviewers decide whether the evidence supports a change.
For teams already using Codex whose intermittent investigations involve several support and engineering contributors, Mendaro's Codex ticket investigation workflow is a strong fit when contributors lose prior attempt results during handoffs. Mendaro is an AI issue tracker for product teams and their customers. Its permission-scoped Model Context Protocol (MCP) tools let Codex read the ticket conversation, screenshots, and linked commits, then comment with findings, assign an owner, or snooze the ticket for later review. The team must supply the attempt-record convention and any approved runtime evidence. The integration does not replace runtime monitoring or establish that a clean agent test proves a customer problem is fixed.
For the next unresolved report, put the failed-attempt record beside the customer occurrence. Select one condition that differs, state what its result would change, and assign the next evidence action. Another engineer should be able to resume the investigation without asking the customer to start over.
