Deployment Troubleshooting

The Troubleshooting Process

A deployment failure is a symptom. The service is not responding, the health check is failing, the container is crashing. The symptom tells you something is wrong; it does not tell you why. Troubleshooting is the process of finding the cause.

A systematic approach

• Read the logs. Logs are the ground truth of what happened. The application logs, the platform logs, the build logs — they tell you what the system actually did, in the order it did it.
• Check the current state. Is the container running? What port is it listening on? What environment variables are set? What does the health endpoint return?
• Compare against the expected state. The application should be listening on a specific port. The environment should have specific variables. The health check should return a specific response. Compare what is to what should be.
• Form a hypothesis. Based on what you see, what is the most likely cause? The port is wrong. The environment variable is missing. The health check is misconfigured. State it clearly.
• Test the hypothesis. Change one thing and check the result. If you change the port and the health check passes, you found the cause. If it does not, revise the hypothesis.
• Fix the root cause. Not the symptom — the cause. If the health check fails because the port is wrong, fix the port, not the health check.

502 Bad Gateway

A 502 Bad Gateway means the platform — the proxy, the load balancer, the ingress — cannot reach the application. The platform is up, but it cannot get a valid response from the service it is trying to route traffic to.

A 502 is a platform-level message, not an application-level one. The application might be running. It might be listening. It might be healthy by its own definition. But the platform cannot reach it, so it returns 502.

Common causes of 502

• The application is not listening on the expected port. The platform routes traffic to port 8000. The application listens on 8080. The container is running, but the platform cannot reach it.
• The container crashed after starting. The application started, then crashed — out of memory, unhandled exception, missing dependency. The container is no longer running, so the platform gets nothing.
• The application is listening on the wrong interface. The application binds to 127.0.0.1 inside the container, which is not reachable from outside the container. It should bind to 0.0.0.0.
• The health check is failing. The platform uses the health check to decide whether to route traffic. If the health check fails, the platform stops routing — even if the application is otherwise running.
• The container did not start at all. The startup command is wrong, the image is broken, the entrypoint fails. The container never becomes ready.

# Platform-side view: 502 from the proxy
curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/health
# 502

# Check if the container is running
docker ps | grep myapp
# Check what port the app is actually listening on
docker exec myapp ss -tlnp

Connection Refused

Connection refused means the platform reached the host, but nothing was listening on the port. This is different from a timeout — the host responded, but said there is nothing at that port.

Common causes of connection refused

• The application is not running. The container exited, the process crashed, the service was never started.
• The application is listening on a different port. The platform checks port 8000. The application listens on 8080. Nothing is on 8000, so connection refused.
• The application is listening on the wrong interface. Binding to 127.0.0.1 means only local connections reach it — external connections are refused.
• A firewall or security group is blocking the port. The application is listening, but the network path is blocked.

# Check what is actually listening
ss -tlnp | grep 8000
# If nothing is listening on 8000, the application is not bound to that port

# Check the application's configured port
grep -r "PORT" .env
grep -r "port" config.yaml
# Compare against what the platform expects

Health Check Failures

A health check is a request the platform makes to verify the service is alive. If the health check fails, the platform considers the service unhealthy and stops routing traffic to it — even if the application is running and handling some requests.

A health check failure is not always a problem with the health check itself. It is often a symptom of a deeper problem: the database is not connected, the cache is unavailable, the application is too slow to respond. The health check is reporting that the service is not healthy, and it is usually right.

What a health check verifies

• The application responds. The endpoint returns HTTP 200. If it returns anything else, the health check fails.
• Critical dependencies are reachable. A good health check verifies the database, the cache, the message queue — whatever the application needs to function.
• The response is timely. A health check with a timeout will fail if the application takes too long to respond — which might indicate a problem even if it eventually responds.

# A health check that fails when the database is not connected
@app.route("/health")
def health():
    try:
        db.execute("SELECT 1")
        return {"status": "healthy"}, 200
    except Exception:
        return {"status": "unhealthy", "reason": "database"}, 503

If the health check returns 503, do not just make the health check more lenient. Find out why the database is not connected — missing environment variable, wrong host, wrong credentials — and fix that.

Missing or Wrong Environment Variables

Environment variables configure the application at runtime — database connections, ports, secret keys, API endpoints. If a required variable is missing, the application might fail to start, fail to connect to a dependency, or silently fall back to a wrong default.

Signs of a missing environment variable

• The application logs show a connection error to a database or external service — but the code looks correct.
• The application starts but does not respond — it failed during initialization and exited, or it is stuck waiting for something.
• The application is using a default value that does not match the platform — the port is 8080 instead of 8000, the database host is localhost instead of the platform's database URL.

# Compare the live environment against the documented example
diff .env.example .env
# Or check specific variables
echo $DATABASE_HOST
echo $PORT
echo $DATABASE_URL

Port Mismatches

A port mismatch is one of the most common deployment failures and one of the easiest to diagnose once you know to check for it. The application listens on one port, and the platform expects traffic on another. The application is running — it just is not reachable on the port the platform is checking.

# The application is configured to listen on 8080
# The platform expects 8000
# The container starts, the application runs, the health check fails

# Check the application's port configuration
grep -r "PORT" .env .env.example config.yaml server.py
# Check what the platform expects
grep -r "port" platform-config.yaml deployment.yaml

# The fix: make them match
PORT=8000

A port mismatch is often caused by a default that differs from the platform's expectation. The application default is 8080. The platform expects 8000. If the PORT environment variable is not set, the application uses its default — and the deployment fails.

Build Failures

A build failure means the artifact was never produced. The deployment cannot proceed because there is nothing to deploy. Build failures happen in the CI stage, before the deployment stage, and they should fail the pipeline — that is what CI is for.

Common build failures

• Syntax errors in the source code. The compiler or interpreter catches them during the build.
• Missing dependencies. The dependency file lists a package that does not exist, or a version that is not available.
• A build command that references a missing file. The Dockerfile copies a file that is not in the build context. The build script references a path that does not exist.
• Tests that fail. The test suite is part of the build, and a failing test fails the build.
• A dependency conflict. Two packages require incompatible versions of the same dependency.

# Read the build log — the failure is usually at the end
docker build -t myapp:latest . 2>&1 | tail -30
# Or check the CI pipeline log
# The build stage that failed is the first place to look

Practice Troubleshooting in the Simulator

The LearnToDeploy deployment simulator includes deployment failures that you diagnose and fix using the same tools you would use in a real environment: reading logs, checking environment variables, inspecting the running service, editing configuration, and verifying the fix with a health check.

Each failure in the simulator is a realistic problem — a wrong port, a missing variable, a broken health check, a container that exits immediately. The simulator gives you the logs and the tools to figure out what is wrong, and it tells you when you have fixed it — when the health check passes, the deployment is healthy.

For the concepts behind the failures, see the deployment concepts library, including entries on health checks, ports and networking, environment variables, and logs and debugging.