Loading and Serving ML Models
Loading joblib/keras model at startup, thread-safe inference, batch prediction endpoints.
Loading and Serving ML Models is a free Learn AI with Python lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Loading Problem
Loading a model from disk is slow. If you reload it on every request, latency skyrockets. The fix: load the model once at startup and reuse it for all requests.
The startup Event
The @app.on_event("startup") hook runs once when the server boots, the perfect place to load the model before any request arrives.
from fastapi import FastAPI
import joblib
app = FastAPI()
@app.on_event("startup")
def load_model():
app.state.model = joblib.load("model.joblib")app.state for Thread-Safe Storage
app.state is the recommended place to stash shared objects like the model. Stored once at startup and only read during requests, it avoids the pitfalls of mutable module-level globals.
from fastapi import Request
@app.post("/predict")
def predict(req: PredictRequest, request: Request):
model = request.app.state.model
return {"prediction": model.predict([req.features])[0]}Why Not a Global Variable?
A bare module global works but couples loading to import time and is harder to test or swap. app.state ties the object lifecycle to the app, which is cleaner and the FastAPI-recommended pattern.
The Modern lifespan Approach
Newer FastAPI replaces on_event with a lifespan context manager, which handles both startup and shutdown in one place.
from contextlib import asynccontextmanager
@asynccontextmanager
async def lifespan(app):
app.state.model = joblib.load("model.joblib") # startup
yield
app.state.model = None # shutdown
app = FastAPI(lifespan=lifespan)Model Warm-Up
The first prediction is often slow because frameworks lazily initialize on first call. Warm up the model at startup with a dummy prediction so the first real user request is already fast.
@app.on_event("startup")
def load_and_warm():
app.state.model = joblib.load("model.joblib")
app.state.model.predict([[0.0, 0.0, 0.0, 0.0]]) # warm-upWhy Warm-Up Matters
Without warm-up, the first request after deploy can be many times slower, hurting tail latency and possibly failing health checks. A single dummy inference at startup pays this cost off the critical path.
A Batch Prediction Endpoint
Predicting many rows in one call is far more efficient than many single calls, since models vectorize over a batch. Accept a list of feature vectors and return a list of predictions.
class BatchRequest(BaseModel):
items: list[list[float]]
@app.post("/predict/batch")
def predict_batch(req: BatchRequest, request: Request):
model = request.app.state.model
preds = model.predict(req.items)
return {"predictions": preds.tolist()}Single vs Batch
- Single: one input, lowest latency per call, simple clients.
- Batch: many inputs at once, far higher throughput, ideal for offline scoring.
Offering both covers real-time and bulk use cases.
Reading From State in Endpoints
Every endpoint reads the shared model via request.app.state.model, never reloading it, so all requests share one in-memory instance.
Putting It Together
A production-ready setup: load and warm the model in lifespan/startup, store it in app.state, and expose both single and batch prediction endpoints reading from state. This minimizes latency and maximizes throughput.
Quick Check
Test your model-serving knowledge.
Recap
You loaded the model once via startup/lifespan, stored it in app.state for thread-safe reuse, added a warm-up dummy prediction, and exposed a batch endpoint for throughput. Next: dockerizing the API.
Frequently asked questions
Is the “Loading and Serving ML Models” lesson free?
Yes — the full text of “Loading and Serving ML Models” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Loading and Serving ML Models”?
Loading joblib/keras model at startup, thread-safe inference, batch prediction endpoints. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Loading and Serving ML Models” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- FastAPI Basics for ML Engineers
- Pydantic Schemas for Request and Response
- Loading and Serving ML Models
- Dockerizing the Model API