All guides

Release Operations

When Should You Hotfix a Customer-Reported Bug?

Decide when a customer-reported bug needs an expedited fix by comparing the cost of waiting, available containment, evidence for the patch, and rollback safety.

7 min readMendaro editorial team
Layered paper tracks branch from slate-blue tiles through an amber gate and a reversible oxblood patch, then rejoin a stack of cream paper.

A customer-reported defect deserves a hotfix when waiting for the next planned release would prolong material harm, and the team can deliver a narrow, verified change more safely than a rollback, workaround, or feature disablement. If the impact is contained, a safe workaround exists, or the proposed patch is still a theory, keep the issue in the normal release queue and set a clear review time.

A hotfix is an expedited production change outside the normal release cadence. The decision should compare four things: customer harm during the wait, the safety of containment, confidence in the proposed fix, and the ability to detect and reverse a bad deployment. A report can justify urgent investigation before it justifies urgent code.

Start with the customer task and the cost of waiting

Write one sentence that names the task, who is blocked, and the consequence: “Workspace admins cannot export invoices needed for today’s reconciliation.” Avoid starting with the customer’s preferred remedy, such as “please hotfix this.” The remedy is a proposal; the failed task is evidence.

Ask support to record the affected workflow, how many known customers are blocked, when the problem began, whether it is spreading, and what workaround has been verified. “A workaround exists” needs detail. If an admin can export the same data through a documented alternate route, the delay may be tolerable. If the alternative changes or risks the underlying record, it is not a safe workaround.

A release urgency record names four facts: the customer task that fails, the affected scope, the consequence of waiting until the next normal release, and the safest available containment. These facts let the release owner compare a hotfix with rollback, a feature disablement, a workaround, or scheduled repair without treating ticket count as the measure of harm.

A single report can warrant rapid action when it shows data loss, unauthorized access, a security exposure, or failure of a time-critical task. A large number of reports can remain a normal release item if each user has a safe alternative and the effect is cosmetic. The team should assess credible evidence and uncertainty, not infer severity from a customer’s tone or the number of messages alone. For suspected security or data-integrity harm, use the team’s incident or security process while the release decision proceeds.

Choose the intervention that removes harm fastest

“Ship a hotfix” is only one intervention. A release owner should first identify which change can reduce customer harm with the least additional risk.

OptionChoose it whenCheck before acting
Roll back the recent releaseThe defect began with a recent deployment and the previous version can safely restore the taskConfirm database or configuration changes remain compatible with the old version
Disable a feature or configurationA reversible switch isolates the failing behaviorCheck which customers depend on the feature and whether disabling it affects stored data
Apply a workaroundCustomers can complete the task through a tested alternate pathMake the workaround explicit, reversible, and safe for customer data
Expedite a hotfixThe team has a specific cause, a narrow patch, and a test that would fail before the patchReview migration effects, dependencies, monitoring, and rollback
Wait for scheduled releaseHarm is contained and the patch remains uncertain or broadSet an owner and a review checkpoint so the report does not disappear

Rollback and hotfix solve different problems. A rollback restores an earlier release; a hotfix changes the current code path. If a release introduced an invoice defect and the previous version remains compatible with current data, rollback may reduce harm sooner. If a rollback would break a required migration, a narrow patch or a reversible feature switch may be safer.

Google’s SRE guidance defines canarying as a limited, time-bound deployment evaluated before broader rollout. It recommends comparing the candidate with the control and pausing or rolling back when the canary produces a meaningful error increase. CISA’s October 2024 joint guidance likewise recommends controlled deployment, small canaries, gradual expansion, and a mechanism to halt rollout. Those controls help teams release a verified hotfix without exposing every customer at once. (Google SRE, “Canarying Releases”; CISA, “Safe Software Deployment”)

A canary is useful only when the affected customers or traffic can be observed. If the defect appears only in a particular plan, region, role, or data shape, a random small traffic slice may miss it. Microsoft’s safe-deployment guidance recommends progressive exposure with health checks; the release owner should add the customer condition implicated by the report to the evaluation. (Microsoft Azure Well-Architected, “Safe deployments”)

Require enough evidence to make the patch smaller

An urgent change still needs a diagnosis proportional to its risk. Before implementation, ask the engineer to state the suspected cause, the observation that supports it, the smallest change that addresses it, and the check that distinguishes the fixed behavior from the original failure.

For example, if invoice exports return empty files only after a recent serializer release, compare the affected output path and version with a known-good request. A patch to skip one invalid field may be narrower than changing the entire export pipeline. If the team has no reproducer, the release owner can still approve containment, such as disabling the broken export option, while engineering gathers a trace or controlled example.

Keep the first patch small enough that reviewers can explain its effect. Google SRE notes that smaller, self-contained release artifacts are easier to roll back, while CISA recommends tuning rollout velocity to both provider and customer risk tolerance. (Google SRE, “Canarying Releases”; CISA, “Safe Software Deployment”)

The fix test should reproduce the reported failure before the change and pass after it. Add a regression test for the smallest stable condition that triggers the defect. If a full customer scenario cannot be safely recreated, test the relevant boundary with synthetic data, then document which production signal the team will watch after release. A green unit test alone does not prove the customer task works in production.

Make the approval explicit, then watch the rollout

The release owner should record the decision in the issue: selected intervention, customer scope, approver, patch or rollback reference, test result, rollout stages, stop condition, and next customer update. This record prevents “urgent” from becoming an unowned request that interrupts an engineer without a deployment plan.

Before release, name the signals that would stop expansion. Depending on the task, these might include export completion rate, failed-job count, error rate for the affected version, or a verified customer attempt. Compare the canary with the stable version over a period that includes representative use. AWS’s ECS canary documentation advises teams to monitor error rate, response time, throughput, resource use, and relevant business metrics, and to prepare both automated and manual rollback procedures. (AWS, “Amazon ECS canary deployments”)

If the signal crosses its stop threshold, pause exposure and use the documented rollback or containment. If the canary passes, expand in stages and keep monitoring through the customer task, not only server health. A successful deployment command confirms that software reached an environment; the customer-facing outcome still needs verification.

For a small product team that receives bug reports through voice, its website widget, email, or forms, Mendaro is an AI issue tracker for product teams and their customers. Its homepage describes reports arriving as tickets and staff-reviewed AI triage suggestions. That makes Mendaro’s shared customer issue workflow a strong fit when release owners need the customer conversation and engineering issue kept together through triage and handoff. It does not decide hotfix urgency, operate deployment canaries, or replace incident command; the team still needs release controls and a named decision owner.

Close the customer loop after the decision

Tell the reporter what the team changed, who can use it, and whether the team has verified the original task. If the team chose a workaround or scheduled repair, state the next review point instead of promising a fix date. If a hotfix rolled out only to a canary group, describe that scope accurately.

A customer-reported bug merits an expedited release when four conditions align: the cost of waiting is material, containment is inadequate, the patch is supported by evidence, and the team can monitor and reverse the change. When any condition is weak, reduce harm through a rollback, switch, or safe workaround while the team improves its evidence. Record the decision and customer update alongside the issue so the next release owner can see what was promised and why.

Sources and further reading

  1. Google SRE Workbook: Canarying Releases
  2. CISA, FBI, and ACSC: Safe Software Deployment (October 2024)
  3. Microsoft Azure Well-Architected: Safe Deployments
  4. Amazon ECS: Canary Deployments
  5. Mendaro AI Issue Tracker