Practice Production Incident Response

What Is a Production Incident?

A production incident is an event that degrades or breaks a production service. It might be a complete outage — the service is down, users cannot reach it. It might be a partial degradation — the service is up but slow, or returning errors for some requests, or missing data.

Incidents are not always caused by a recent deployment. They can be caused by a configuration change, a dependency failure, a resource running out, a network problem, or a bug that was latent and only surfaces under specific conditions. The cause is not always obvious from the symptom.

Severity levels

• SEV-1: A complete outage affecting all users. The service is down or returning errors for everyone. This is the highest priority — restore service immediately.
• SEV-2: A significant degradation affecting many users or a critical feature. The service is partially working but something important is broken.
• SEV-3: A minor issue affecting few users or a non-critical feature. The service is working, but something is wrong and needs attention.

The severity determines the response — how quickly to act, who to involve, whether to wake someone up. A SEV-1 at 3am gets a page. A SEV-3 might wait until morning.

The Incident Lifecycle

An incident goes through stages. Understanding the stages helps you know what to do at each point — and what not to do.

1. Detection

An incident is detected when someone notices — an alert fires, a user reports a problem, a monitor shows a spike. Detection is the first step, and it is not always fast. A monitoring system that does not alert on the right things will let incidents go undetected until users complain.

Good detection means monitoring the right signals — health checks, error rates, latency, resource usage — and alerting on them before users notice. A health check that fails should page someone. An error rate that spikes should page someone. The goal is to detect incidents before users report them.

2. Triage

Triage is the first response — confirming the incident is real, assessing its severity, and deciding what to do. The first questions are: is this actually a problem? How bad is it? What is affected? Is it getting worse?

Triage is not investigation. It is the quick assessment that determines the response. If the site is down, you do not spend ten minutes reading logs before deciding to escalate — you escalate, then investigate. Triage is about deciding what is happening and how urgently to act.

3. Investigation

Once the incident is confirmed and the response is organized, investigation starts. The goal is to find the root cause — what changed, what broke, why. Investigation uses the same tools as deployment troubleshooting: logs, metrics, environment variables, configuration, the running state of the service.

Investigation is methodical. You start from the symptom — the error, the latency, the crash — and work backward. What changed recently? What does the log say? What does the configuration say? What is the state of the dependencies? Each piece of evidence narrows the possibilities.

4. Mitigation

Mitigation is the step that reduces the impact of the incident — ideally fast, even if it does not fix the root cause. A rollback is a mitigation: it restores service by going back to the last known-good state, even if you have not figured out what caused the failure. Turning off a feature is a mitigation. Scaling up a resource is a mitigation.

The goal of mitigation is to stop the bleeding. Restore service. Reduce the error rate. Bring latency back down. The root cause fix comes after — but mitigation is what protects users in the meantime.

5. Resolution

Resolution is fixing the root cause — the actual problem, not just the symptom. If the incident was caused by a bad deployment, resolution is rolling back or fixing the deployment. If it was caused by a configuration error, resolution is correcting the configuration. If it was caused by a resource running out, resolution is adding capacity or fixing the leak.

Resolution is confirmed when the service is healthy and the incident is over — the error rate is back to normal, the latency is back to normal, the health check is passing. The resolution is verified, not assumed.

6. Postmortem

A postmortem is the review after the incident — what happened, why, what was done, what can be done differently next time. A good postmortem is blameless: it focuses on the system and the process, not on who made a mistake. The goal is to learn from the incident so it does not happen again — or if it does, the response is faster and better.

A postmortem typically covers: the timeline of the incident, the impact, the root cause, what was done to mitigate and resolve it, what went well in the response, what went poorly, and what actions will prevent recurrence. The actions are the most important part — they turn the incident into improvement.

Incident Tools

Incident response uses a specific set of tools. The simulator includes these tools so you can practice using them in a realistic environment.

Logs

Logs are the ground truth. They record what the service did — requests, errors, startup, shutdown, configuration. When an incident happens, the logs are the first place to look. They tell you what changed, what failed, and when.

Metrics

Metrics show the state of the service over time — error rate, latency, CPU, memory, request volume. Metrics tell you when an incident started, how bad it is, and whether your mitigation is working. A spike in the error rate graph is the clearest signal that something is wrong.

Terminal

A terminal gives you direct access to the running environment — check processes, inspect network state, read files, run commands. The terminal is where you verify what the logs and metrics suggest — is the process actually running? What port is it listening on? What does the environment look like right now?

CI/CD and Deployments

The deployment history tells you what changed recently. If an incident started right after a deployment, the deployment is the prime suspect. The CI/CD tools help you see what was deployed, when, and whether you can roll back.

Database and Redis

Many incidents involve a dependency — the database is slow, Redis is unavailable, the connection pool is exhausted. Database and cache tools let you check the state of these dependencies directly — query the database, check Redis keys, verify the connection.

Status Page

A status page is what users see during an incident — a public page that communicates whether the service is healthy and what is affected. Keeping the status page current during an incident is part of the response: it tells users the problem is known and being worked on.

Runbook

A runbook is a structured guide for handling specific incidents — steps to follow, checks to make, commands to run. A runbook turns an incident response into a repeatable process. During an incident, a runbook keeps the response organized and ensures nothing is missed.

Team Chat and On-Call

Incident response is often a team effort. On-call tools simulate the paging flow — acknowledgment, escalation, incident commander assignment. Team chat simulates the coordination that happens during a real incident — sharing findings, assigning tasks, communicating status.

Chaos Injection

Chaos injection is the deliberate introduction of failure into a healthy service — to practice troubleshooting before a real incident happens. Inject a latency spike, drop requests, kill a process, and then practice diagnosing the problem. Chaos injection turns incident response into a trainable skill.

Incident Response Skills

Incident response is a skill that improves with practice. The simulator provides scenarios that exercise the core skills:

• Reading logs under time pressure. The logs are there, but they are long and the important line is buried in them. Finding it quickly is a skill.
• Correlating symptoms with causes. A spike in error rate is a symptom. The cause might be a bad deployment, a database problem, a configuration change. Connecting the two is the investigation.
• Deciding when to mitigate versus fix. If the site is down, roll back first and ask questions later. If the site is degraded but functional, you might have time to investigate before acting. Knowing the difference is a judgment call.
• Communicating during an incident. Status updates, team coordination, postmortem writing — incident response is not just technical, it is also communication.
• Following a runbook. A runbook is a checklist for a reason — it ensures the response is complete and consistent. Following it under pressure is a practiceable skill.

Practice Incidents in the Simulator

The LearnToDeploy simulator includes production incident scenarios across multiple environments — each with its own services, dependencies, and failure modes. You page in, assess the situation, investigate using the tools, mitigate the impact, and resolve the incident.

Each scenario has a hidden root cause that you have to find — not a scripted sequence, but a real configuration or code problem that you diagnose the way you would in a real incident: read the logs, check the state, form a hypothesis, test it, fix it.

The simulator tracks your response — how long the incident lasted, what tools you used, whether you found the root cause. The goal is not to be fast at the expense of accuracy — it is to be methodical and correct, because that is what good incident response is.

For the underlying concepts behind each incident type, see the deployment concepts library, including entries on incident response, logs and debugging, health checks, and rollbacks.