Hardening for Production

AI excels at producing something that works on your machine for one user. Production means thousands of users, hostile inputs, partial failures, and 3 a.m. incidents.

Hardening is the deliberate gap-closing between "it runs" and "it holds." The model will not cross that gap unless you direct it.

This lesson is the checklist for making AI-built software ready to actually ship.

Production code must read every secret and tunable from the environment, never from constants. Different environments need different values without code changes.

Ask the model to externalize configuration and fail fast on startup if a required variable is missing, rather than crashing mysteriously later.

A clear startup error beats a silent misconfiguration in production.

Scattered print statements do not survive contact with production. You need structured logs with levels, timestamps, request ids, and no secrets.

When something breaks at scale, the log is your only witness. Make it queryable and correlated across a request's lifecycle.

Ask explicitly for structured output and a correlation id threaded through each request.

Demo code lets exceptions bubble up and crash the process. Production code catches at boundaries, returns sane responses, and keeps serving other requests.

Map errors to correct status codes, distinguish user mistakes from server faults, and never expose internals.

A single bad request must not take down the whole service.

Every network call can hang. Without timeouts, one slow dependency exhausts your connection pool and the whole app stalls.

Add timeouts to outbound calls, bounded retries with backoff for transient failures, and circuit breakers for repeatedly failing dependencies.

Models almost never add these unprompted; they assume the network is reliable.

Production must survive abuse: oversized payloads, huge result sets, and traffic spikes. Cap request body size, paginate every list, and bound queue depth.

Without limits, a single large request or a burst of traffic can exhaust memory and take the service down.

Define the ceilings explicitly so the system degrades instead of collapsing.

The database is the usual production bottleneck. Connection pools need sizing, slow queries need indexes, and migrations need to be reversible.

AI-generated queries often miss indexes and trigger full table scans that only hurt once data grows. Wrap multi-step writes in transactions to avoid partial state.

Review the data layer for pooling, indexing, and atomicity before launch.

Orchestrators need to know if your service is alive and ready. A liveness probe answers "is the process running" and a readiness probe answers "can it serve traffic."

Without them, a deploy can route requests to an instance still warming up, or keep a wedged instance in rotation.

Add lightweight endpoints that reflect real dependency health.

You cannot fix what you cannot see. Production needs metrics on latency, error rate, and throughput, plus alerts when they cross thresholds.

The goal is to learn about problems from a dashboard, not from angry users. Instrument the paths that matter and set alerts on symptoms users feel.

Ask the model to add metrics and define what should page someone.

The riskiest moment is the deploy itself. Production needs a way to ship gradually and roll back instantly when something goes wrong.

Use migrations that are backward compatible, feature flags to decouple deploy from release, and a tested rollback path you have actually exercised.

A deploy you cannot undo is a gamble, not a release.

Before launch, run one final pass: secrets externalized, logging structured, errors handled, timeouts set, limits enforced, database indexed, health checks live, metrics and alerts wired, and rollback rehearsed.

Have the assistant audit the whole app against this list and report what is missing, then verify each claim yourself.

Hardening is finished only when an outage would be boring, not catastrophic.

Test your production hardening judgment.

Production hardening closes the gap between "it runs" and "it holds": externalize config, log with structure, handle errors at boundaries, add timeouts, retries, and limits, harden the database, expose health checks, wire metrics and alerts, and make deploys reversible.

Run the pre-launch checklist with the assistant and verify every item yourself. You now have a full workflow to test and harden AI-built software for the real world.