Mapping S3 and Atlas Sources to a Virtual Namespace

In Atlas Data Federation, the storage configuration is a JSON document that defines two things: stores (where the raw data lives — S3 buckets, Atlas clusters) and databases/collections (the virtual namespace that applications query). The mapping between them tells the query engine which store to read when you query a virtual collection.

An S3 store definition names the store, specifies the AWS region and bucket name, and associates it with IAM credentials (via an Atlas cloud provider access role). You can optionally set a delimiter and prefix to scope the store to a particular S3 prefix. A single federated instance can have multiple stores pointing to different buckets or regions.

An Atlas cluster store connects a federated instance to a live Atlas replica set. You reference it by the cluster name within the same Atlas project. This lets you write pipelines that read from live operational collections in the cluster alongside archived S3 data, enabling hybrid queries without any data duplication.

A virtual collection is mapped to a store and a path pattern. The path is a glob pattern that tells Data Federation which S3 objects belong to this collection. The pattern can include literal path segments or wildcards. The query engine will scan all matching objects when the collection is queried.

Partition attributes encode metadata directly in the S3 path using curly-brace syntax: {year int}/{month int}/{day int}/. When a query filters on year, month, or day, Data Federation prunes — skips — all S3 objects whose path does not match the filter values. This is analogous to Hive partitioning and dramatically reduces bytes scanned.

A single virtual collection can be mapped to multiple data sources — for example, an S3 archive plus a live Atlas collection. Queries merge results from all sources transparently. This is useful for a 'full history' collection where recent data is in Atlas and older data is archived in S3, yet applications query both through a single namespace.

You can define a wildcard collection (*) that maps all files in an S3 prefix to dynamically named virtual collections. When you query a collection name that matches the wildcard pattern, Data Federation derives the path from the collection name. This is useful for partitioned data lakes where you have thousands of prefix-based 'tables' and cannot enumerate them all in the config.

You can update the storage configuration at any time through the Atlas UI, the Atlas Admin API, or mongosh. Changes take effect immediately — you do not need to restart the federated instance. This lets you add new S3 paths, remap collections to different stores, or add new Atlas cluster sources without any downtime.

After defining the storage configuration, test it by connecting to the federated instance with mongosh and listing databases and collections. Use show dbs, show collections, and a simple find() to verify the mapping is correct. Check that the correct files are being read by inspecting the first few documents returned.

Atlas Data Federation accesses S3 via AWS IAM. The recommended approach is to use an IAM role delegated to Atlas's AWS account (Atlas Cloud Provider Access) rather than storing access key/secret pairs. The role is granted read access to the S3 bucket via an IAM policy, and Atlas assumes the role when executing queries. This is more secure than long-lived access keys.

You can create multiple virtual collections that point to the same underlying S3 prefix with different path patterns — effectively creating multiple views of the same data. For example, one collection exposes all data, another exposes only the current year's partition, and a third maps only Parquet files while excluding JSON. This lets you control what each application tier sees.

Test your understanding of MongoDB & NoSQL Databases concepts from this lesson.

In this lesson you learned: stores define where raw data lives (S3 buckets or Atlas clusters) and virtual databases/collections define the namespace applications query, partition attributes in S3 paths enable pruning that dramatically reduces bytes scanned, and a single virtual collection can merge data from multiple stores simultaneously. Next up we write cross-source aggregation pipelines.