O que é o Atlas Data Federation

Modern applications generate data across multiple systems: live operational data in MongoDB Atlas, historical archives in Amazon S3, and analytics exports in data lakes. Querying across these silos traditionally requires data pipelines, ETL jobs, and separate query engines. Atlas Data Federation solves this by letting you query all these sources with a single MongoDB connection and the familiar aggregation pipeline.

Atlas Data Federation is a fully managed query engine built into MongoDB Atlas. It creates a federated database instance — a virtual MongoDB deployment that maps data from multiple sources (Atlas clusters, S3 buckets, Atlas Data Lake, HTTP endpoints) to virtual collections. You connect with a standard MongoDB connection string and use the same aggregation pipeline you already know.

Atlas Data Federation can query: Atlas clusters — live MongoDB collections. Amazon S3 — JSON, BSON, CSV, TSV, Avro, ORC, and Parquet files stored in S3 buckets. Atlas Data Lake — processed and enriched datasets. HTTP/HTTPS endpoints — external REST APIs that return JSON. All sources are mapped to virtual namespaces within the federated instance.

A federated database has three layers: Storage configuration — defines which sources map to which virtual databases and collections. Query engine — the distributed SQL/MQL processor that reads from multiple sources, applies pipeline stages, and merges results. Connection layer — a mongos-compatible endpoint you connect to with any MongoDB driver or mongosh.

You create a federated database instance through the Atlas UI, Atlas Admin API, or Atlas CLI. During setup you: 1) Name the instance. 2) Add stores (S3 buckets with IAM credentials, Atlas clusters, etc.). 3) Define virtual databases and collections that point to those stores. 4) Copy the connection string and connect with your MongoDB driver.

Virtual collections in a federated database do not store data — they are logical views over the underlying source files or collections. You can query a virtual collection named analytics.orders that actually reads S3 Parquet files at s3://my-bucket/orders/2024/. To MongoDB drivers and tools, the virtual collection looks and behaves like a regular MongoDB collection.

One of the most powerful features is joining a live Atlas collection with archived S3 data in a single pipeline. For example: look up active customer details from a live Atlas cluster and join them with their 3-year purchase history stored in S3 Parquet files — all in one aggregation with no ETL job required.

Data Federation reads S3 files in many formats: JSON (one document per line or array), BSON (MongoDB native binary), CSV/TSV (with header row), Avro, ORC, and Parquet (columnar formats widely used in data lakes). For columnar formats, Data Federation can push projection and filter predicates into the file reader for even faster scans.

Atlas Data Federation charges based on data processed (bytes scanned), not uptime. This makes it cost-effective for infrequent analytical queries over large S3 archives — you pay nothing when no queries run. However, scanning entire unpartitioned S3 datasets can become expensive. Partitioning your S3 data and using projection to reduce scanned bytes are critical for cost control.

Federated database instances use the same Atlas database users and roles as regular Atlas clusters. You can apply Atlas network peering, private endpoints (AWS PrivateLink), and IP Access Lists to restrict who can connect. The connection to S3 uses IAM roles rather than storing AWS keys directly, following AWS security best practices.

Data Federation is a good fit when: 1) You need to run ad-hoc queries across historical S3 archives without loading data into a live cluster. 2) You want to join live transactional data with archived data in a single query. 3) You need a unified analytics interface across multiple Atlas clusters. 4) You want to avoid building and maintaining a separate ETL pipeline for each analytical use case.

Test your understanding of MongoDB & NoSQL Databases concepts from this lesson.

In this lesson you learned: Atlas Data Federation creates a virtual MongoDB namespace over S3, Atlas clusters, and other sources, you use the standard aggregation pipeline to query and join data across all sources in one operation, and costs are based on bytes scanned, so partitioning and projection are essential for cost control. Next up we map S3 and Atlas sources to virtual namespaces.