Aggregation Pipeline Optimization Tips

Aggregation pipelines can be expensive — they can scan millions of documents, build large in-memory structures, and block for seconds. The key to fast pipelines is applying data-reduction stages as early as possible so later stages work on the smallest possible dataset. MongoDB also has an internal optimizer that automatically reorders certain stages, but understanding manual optimizations gives you the most control.

$match is MongoDB's filter stage. Placing it as early as possible in the pipeline reduces the number of documents that flow into every subsequent stage. If $match can use an index, it becomes an extremely fast first step. Even a $match without an index early in the pipeline is better than a late one — it avoids processing documents that will be discarded anyway.

Use $project early to drop fields you will not need in later stages. Fewer fields per document means smaller in-memory representations flowing through the pipeline, reducing memory pressure and CPU time. Only project away fields you are certain are not needed — an over-aggressive early $project that drops a field used in a later stage will fail.

MongoDB's aggregation optimizer automatically applies several rewrites before executing your pipeline. Key auto-rewrites: 1) Merges consecutive $match stages into one. 2) Moves $match before $skip and $limit when possible. 3) Merges consecutive $limit stages. 4) Pushes $match before $lookup to filter before the join. Use explain() to see the optimized pipeline.

$unwind explodes an array into multiple documents — one per array element. If you place $match after $unwind, you pay the full cost of expansion before filtering. When possible, filter before the $unwind to reduce the number of array elements that need expansion. This can reduce document count by an order of magnitude on large arrays.

The first $match stage of a pipeline can use a collection index just like a find() query. The first $sort stage can also use an index to avoid an in-memory sort — but only if it appears before any stage that modifies the document shape (like $project or $group). Structure your pipelines so early $match and $sort stages benefit from indexes.

By default, each aggregation stage is limited to 100 MB of RAM. If a $sort, $group, or $bucket stage exceeds this limit, the pipeline fails with an error. Setting allowDiskUse: true lets the pipeline spill to disk, enabling arbitrarily large sorts at the cost of slower I/O. The right fix is usually to add an index or filter more aggressively before the sort.

$lookup is the aggregation equivalent of a SQL JOIN and is one of the most expensive stages. To minimise cost: 1) Place $match before $lookup so fewer documents need joining. 2) Ensure the joined collection has an index on the foreignField. 3) Use the pipeline form of $lookup with a $match inside it to filter the joined results immediately.

When building paginated list endpoints, push $limit as early as possible after the filter and sort. A pipeline that processes 100,000 documents through $lookup and $addFields before limiting to 20 is 5,000x more expensive than one that limits first. Keyset pagination with a range filter often eliminates the need for $skip entirely.

For expensive aggregations that power dashboards or reports, consider materialised views using $merge or $out. Run the heavy aggregation on a schedule (e.g., every hour) and write results to a dedicated collection. Queries against the materialised view are cheap point-lookups instead of expensive pipeline scans. Atlas also supports On-Demand Materialised Views via triggers.

Use .explain('executionStats') on an aggregate to see how many documents each stage emitted. Look for stages where nReturned drops sharply — that is where the real work happens. If a stage is still processing millions of documents, add a $match or improve the index before it. For Atlas, the Query Profiler also shows aggregation pipeline performance over time.

Test your understanding of MongoDB & NoSQL Databases concepts from this lesson.

In this lesson you learned: push $match and $project as early as possible to minimize document volume in later stages, ensure the first $match and $sort use indexes to avoid collection scans and in-memory sorts, and use allowDiskUse for unavoidably large sorts, but prefer better indexes or earlier filters as the permanent fix. Next up we explore Atlas Data Federation.