Mapping S3 and Atlas Sources to a Virtual Namespace
Learners will configure a federated database instance that maps S3 prefixes and Atlas collections to virtual databases and collections.
Storage Configuration: The Core Concept
In Atlas Data Federation, the storage configuration is a JSON document that defines two things: stores (where the raw data lives — S3 buckets, Atlas clusters) and databases/collections (the virtual namespace that applications query). The mapping between them tells the query engine which store to read when you query a virtual collection.
Defining a Store: S3 Bucket
An S3 store definition names the store, specifies the AWS region and bucket name, and associates it with IAM credentials (via an Atlas cloud provider access role). You can optionally set a delimiter and prefix to scope the store to a particular S3 prefix. A single federated instance can have multiple stores pointing to different buckets or regions.
// S3 store definition in storage config
{
'stores': [{
'name': 's3ArchiveStore',
'provider': 'S3',
'region': 'us-east-1',
'bucket': 'mycompany-analytics-archive',
'delimiter': '/',
'additionalStorageClasses': ['STANDARD_IA', 'GLACIER']
}]
}All lessons in this course
- What Is Atlas Data Federation?
- Mapping S3 and Atlas Sources to a Virtual Namespace
- Running Cross-Source Aggregation Pipelines
- Partitioning S3 Data for Query Performance