Inferencia online en tiempo real

Online inference answers one request at a time, the instant it arrives, so a user or service gets a fresh prediction right away. ⚡

Unlike batch, here someone waits on the other end. The prediction must come back in milliseconds, not minutes, to feel responsive.

An online model sits behind an HTTP endpoint. A client sends features in a request and gets the prediction back in the response.

Each call carries the input for a single case. The service runs predict on it and returns the score, then handles the next caller.

Because features arrive with the request, the score reflects the latest state, perfect when inputs change second to second.

The number you watch is latency: how long from request to response. People often track the slow 95th and 99th percentile, not just the average.

Loading the model on every request would be slow. Instead you load it once at startup and reuse it across all incoming calls.

Many users hit the service at the same time. It must serve requests concurrently, often with multiple workers, to keep latency low under load.

Fraud checks at checkout, search ranking, and recommendation widgets all need instant answers, so they run as online inference.

An online service must stay running and ready around the clock. That always-on footprint costs more than a job that runs and stops.

Choose online when inputs are unknown ahead of time and users need a fresh answer now, accepting more cost and operational care.

What metric matters most for online inference?

Online inference serves one fresh prediction per request behind an API, prizes low latency, and costs more because it must stay always on.