MongoDB Academy · 课时

将 S3 和 Atlas 数据源映射到虚拟命名空间

您将配置联邦数据库实例，把 S3 前缀和 Atlas 集合映射到虚拟数据库和集合。

将 S3 和 Atlas 数据源映射到虚拟命名空间 是 CoddyKit 上的免费 MongoDB Academy 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 MongoDB Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 MongoDB Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Storage Configuration: The Core Concept

In Atlas Data Federation, the storage configuration is a JSON document that defines two things: stores (where the raw data lives — S3 buckets, Atlas clusters) and databases/collections (the virtual namespace that applications query). The mapping between them tells the query engine which store to read when you query a virtual collection.

Defining a Store: S3 Bucket

An S3 store definition names the store, specifies the AWS region and bucket name, and associates it with IAM credentials (via an Atlas cloud provider access role). You can optionally set a delimiter and prefix to scope the store to a particular S3 prefix. A single federated instance can have multiple stores pointing to different buckets or regions.

// S3 store definition in storage config
{
  'stores': [{
    'name': 's3ArchiveStore',
    'provider': 'S3',
    'region': 'us-east-1',
    'bucket': 'mycompany-analytics-archive',
    'delimiter': '/',
    'additionalStorageClasses': ['STANDARD_IA', 'GLACIER']
  }]
}

Defining a Store: Atlas Cluster

An Atlas cluster store connects a federated instance to a live Atlas replica set. You reference it by the cluster name within the same Atlas project. This lets you write pipelines that read from live operational collections in the cluster alongside archived S3 data, enabling hybrid queries without any data duplication.

// Atlas cluster store definition
{
  'stores': [{
    'name': 'liveClusterStore',
    'provider': 'atlas',
    'clusterName': 'MyProdCluster',
    'projectId': 'proj123abc'
  }]
}

Mapping Collections to S3 Paths

A virtual collection is mapped to a store and a path pattern. The path is a glob pattern that tells Data Federation which S3 objects belong to this collection. The pattern can include literal path segments or wildcards. The query engine will scan all matching objects when the collection is queried.

// Map virtual collection to S3 path pattern
{
  'databases': [{
    'name': 'analytics',
    'collections': [{
      'name': 'events_2024',
      'dataSources': [{
        'storeName': 's3ArchiveStore',
        'path': '/data/events/2024/*'
      }]
    }]
  }]
}

Partition Attributes in Paths

Partition attributes encode metadata directly in the S3 path using curly-brace syntax: {year int}/{month int}/{day int}/. When a query filters on year, month, or day, Data Federation prunes — skips — all S3 objects whose path does not match the filter values. This is analogous to Hive partitioning and dramatically reduces bytes scanned.

// Path with partition attributes (Data Federation parses the directory structure)
{
  'dataSources': [{
    'storeName': 's3ArchiveStore',
    'path': '/events/{year int}/{month int}/{day int}/*.json'
  }]
}

// Query that uses partition pruning:
db.events.find({ year: 2025, month: 3 })
// Data Federation only reads /events/2025/3/ prefix

Mapping Multiple Sources to One Collection

A single virtual collection can be mapped to multiple data sources — for example, an S3 archive plus a live Atlas collection. Queries merge results from all sources transparently. This is useful for a 'full history' collection where recent data is in Atlas and older data is archived in S3, yet applications query both through a single namespace.

{
  'collections': [{
    'name': 'orders',
    'dataSources': [
      {
        'storeName': 'liveClusterStore',
        'database': 'mydb',
        'collection': 'orders'           // live Atlas data
      },
      {
        'storeName': 's3ArchiveStore',
        'path': '/orders/archive/*.parquet'  // S3 archive
      }
    ]
  }]
}

Wildcard Collections: Schema-on-Read

You can define a wildcard collection (*) that maps all files in an S3 prefix to dynamically named virtual collections. When you query a collection name that matches the wildcard pattern, Data Federation derives the path from the collection name. This is useful for partitioned data lakes where you have thousands of prefix-based 'tables' and cannot enumerate them all in the config.

// Wildcard: each year-month subdirectory becomes a virtual collection
{
  'collections': [{
    'name': '*',
    'dataSources': [{
      'storeName': 's3ArchiveStore',
      'path': '/data/{collectionName string}/'
    }]
  }]
}

// Now query any 'table' by name:
db['orders-2024-01'].find({})
db['events-2025-03'].aggregate([...])

Updating the Storage Configuration

You can update the storage configuration at any time through the Atlas UI, the Atlas Admin API, or mongosh. Changes take effect immediately — you do not need to restart the federated instance. This lets you add new S3 paths, remap collections to different stores, or add new Atlas cluster sources without any downtime.

// Update storage config via Admin API
// PATCH /api/atlas/v1.0/groups/{groupId}/dataFederation/{name}
// Body: updated storage config JSON

// Or via mongosh using the Atlas admin command
db.adminCommand({
  setQueryableEncryptionBackend: 1,
  dataFederationConfig: { /* new config */ }
})

Testing Your Mapping

After defining the storage configuration, test it by connecting to the federated instance with mongosh and listing databases and collections. Use show dbs, show collections, and a simple find() to verify the mapping is correct. Check that the correct files are being read by inspecting the first few documents returned.

// Connect to federated instance and test
// mongosh 'mongodb+srv://federated.mongodb.net/'
show dbs                // lists virtual databases
use analytics
show collections        // lists virtual collections
db.events_2024.findOne()
// Verify the document shape matches your S3 files

IAM Role vs Access Key Auth for S3

Atlas Data Federation accesses S3 via AWS IAM. The recommended approach is to use an IAM role delegated to Atlas's AWS account (Atlas Cloud Provider Access) rather than storing access key/secret pairs. The role is granted read access to the S3 bucket via an IAM policy, and Atlas assumes the role when executing queries. This is more secure than long-lived access keys.

// Minimal S3 IAM policy for Data Federation read access
// {
//   'Version': '2012-10-17',
//   'Statement': [{
//     'Effect': 'Allow',
//     'Action': ['s3:GetObject', 's3:ListBucket'],
//     'Resource': [
//       'arn:aws:s3:::mycompany-analytics-archive',
//       'arn:aws:s3:::mycompany-analytics-archive/*'
//     ]
//   }]
// }

Namespace Aliasing and Multiple Views

You can create multiple virtual collections that point to the same underlying S3 prefix with different path patterns — effectively creating multiple views of the same data. For example, one collection exposes all data, another exposes only the current year's partition, and a third maps only Parquet files while excluding JSON. This lets you control what each application tier sees.

Quick Check

Test your understanding of MongoDB & NoSQL Databases concepts from this lesson.

Lesson Recap

In this lesson you learned: stores define where raw data lives (S3 buckets or Atlas clusters) and virtual databases/collections define the namespace applications query, partition attributes in S3 paths enable pruning that dramatically reduces bytes scanned, and a single virtual collection can merge data from multiple stores simultaneously. Next up we write cross-source aggregation pipelines.

常见问题解答

「将 S3 和 Atlas 数据源映射到虚拟命名空间」课时是免费的吗？

是的 — 「将 S3 和 Atlas 数据源映射到虚拟命名空间」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 MongoDB Academy 课程的其余内容，请升级到 CoddyKit PRO。 MongoDB Academy 课程共包含 4 节课。

「将 S3 和 Atlas 数据源映射到虚拟命名空间」这节课中我会学到什么？

您将配置联邦数据库实例，把 S3 前缀和 Atlas 集合映射到虚拟数据库和集合。你通过在浏览器中直接运行的动手代码来练习 MongoDB Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 MongoDB Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 MongoDB Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「将 S3 和 Atlas 数据源映射到虚拟命名空间」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 MongoDB Academy 课中编写并运行代码吗？

能。每节 MongoDB Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 MongoDB Academy