Scenarios
When a single Replica Set cannot meet the storage capacity or concurrent throughput requirements of a business, a Sharded Cluster provides horizontal scaling capabilities. In actual production environments, improper shard design often leads to serious availability and performance issues. Common pain points include:
Data hotspots cause single-node overload: If the shard Key is improperly selected (for example, using monotonically increasing fields such as auto-incrementing IDs or timestamps), all concurrent writes will concentrate on a single shard. This saturates the CPU and disk I/O of that node, while other shards remain idle, failing to leverage the advantages of a distributed architecture.
Broadcast queries slow overall performance: When a query condition does not include the shard key, the routing node (Mongos) must broadcast the request to all shards and then merge the results. This causes latency to increase linearly with the shard quantity.
Jumbo Chunks cause storage skew: If the shard key has low Cardinality, a large amount of data will be concentrated in a few chunks. When a Chunk exceeds the size limit and cannot be split, the Balancer cannot migrate it. This eventually leads to severe storage skew across shards.
High Ops risk for shard key changes: Versions earlier than MongoDB 5.0 do not support modifying the shard key. If the design is flawed, the cluster must be migrated with service suspension. Although MongoDB V5.0 and later introduced the online Resharding feature, this operation consumes a significant amount of additional disk space and I/O resources. Therefore, flawed architecture design in the early stage still poses a substantial Ops risk.
This guide aims to provide standard design and Ops specifications for sharded clusters. It guides you in making reasonable shard key selections, configuring secure access, and efficiently controlling data balancing, thereby ensuring high availability and high scalability of the cluster.
Shard Key Design
Specification 1: Shard Key with High Cardinality
Core action: The shard key's value range should be sufficiently large (at least tens of thousands of distinct values) to ensure data is evenly distributed across all shards.
Cardinality refers to the number of distinct values in a field. A lower Cardinality makes data more likely to concentrate in a few chunks, creating storage and performance hotspots. When the data volume in a Chunk exceeds the threshold but cannot be split further due to identical shard key values, a Jumbo Chunk is formed. This causes storage pressure on some shards while leaving resources on other shards idle.
|
Extremely Low Cardinality | < 100 | Not recommended (hotspot inevitable) |
Low Cardinality | 100 ~ 10,000 | Use with caution (distribution evaluation required) |
High Cardinality | 10,000 ~ 1,000,000 | Recommended |
Extremely High Cardinality | > 1,000,000 | Ideal choice |
Business application: A logistics system uses the city field as the shard key. The entire country has only 50 cities.
Problem manifestation: As business grows, data from the four cities of Beijing, Shanghai, Guangzhou, and Shenzhen accounts for 60% of the total volume. The corresponding shards face storage pressure, while shards for other remote cities remain almost completely idle, resulting in severe data skew. Since the system runs on a MongoDB version earlier than 5.0, which natively does not support modifying the shard key, the cluster ultimately had to be migrated to a new cluster for redesign. This caused an 8-hour service suspension.
Optimization operation: Select a high-cardinality field (such as orderId or customerId) as the shard key. Such fields have a value range that can reach millions or even higher. This ensures data and read/write workloads are distributed extremely evenly across all shards in the cluster, fully unleashing the horizontal scaling capability of the distributed architecture.
Cardinality evaluation example: The following example demonstrates how to evaluate whether a field's Cardinality meets the shard key requirements through an aggregation query.
db.t_orders.aggregate([
{ $group: { _id: "$customer_id" } },
{ $count: "cardinality" }
], { allowDiskUse: true })
db.t_orders.aggregate([
{ $group: { _id: "$status" } },
{ $count: "cardinality" }
])
Specification 2: Shard Key Ensuring Even Write Distribution
Core action: Select a field with evenly distributed data during writes, and avoid monotonically increasing fields (such as ObjectId, auto-incrementing IDs, and timestamps) as range shard keys.
When a monotonically increasing field is used as a range shard key, all new data is written to the Chunk with the maximum value, creating a "write hotspot" where a single shard bears 100% of the write load while other shards remain completely idle. Hashed sharding distributes writes evenly across all shards by hashing the shard key values before assignment. When data possesses inherent partitioning attributes (such as region, business line, or data temperature), Zone Sharding can be used to direct data to designated shards.
Illustration of the Hotspot Issue with Monotonically Increasing Fields:
Timeline ─────────────────────────────────────────────────────►
Chunk 1 Chunk 2 Chunk 3 Chunk 4
[2024-01-01] [2024-02-01] [2024-03-01] [2024-04-01]
↑ ↑ ↑ ↑
Cold Data Cold Data Cold Data ←── All writes are concentrated here (hotspot)
|
Write distribution | Potential hotspot (monotonic field) | Even distribution | Distributed according to Zone rules |
Range query | Efficient (targeted query) | Inefficient (full shard scan) | Efficient within a Zone |
Equality query | Efficiency | Efficiency | Efficient within a Zone |
Data location | By value range | By hash value | Based on Zone tags |
Scenario | Time-range queries, interval statistics | High-concurrency writes and equality queries | Based on Zone tags |
Business Application: A logging system uses createTime (timestamp) as a range shard key.
Problem Manifestation: All new logs are written to the Chunk with the maximum timestamp. A single shard bears 100% of the write load, its disk I/O becomes saturated, while other shards remain completely idle.
Optimization Operation: Switch to Hashed sharding on _id. Writes are evenly distributed across all shards, reducing the load on a single shard by 80%.
Range Sharding and Hashed Sharding Example: The following example compares the configuration methods of range sharding and Hashed sharding, demonstrating how to avoid write hotspot issues caused by incrementing keys.
sh.shardCollection("mydb.t_orders", { _id: 1 })
sh.shardCollection("mydb.t_orders", { createTime: 1 })
sh.shardCollection("mydb.t_orders", { _id: "hashed" })
sh.shardCollection("mydb.t_orders", { orderNo: "hashed" })
Specification 3: Shard Key Covering Major Query Conditions
Core action: The shard key should include the filter conditions from the most frequent queries to enable "Targeted Query" and avoid broadcast queries.
When a query condition does not include the shard key, mongos cannot determine which shard contains the target data. It must broadcast the query to all shards and then merge the results (Scatter-Gather). The greater the shard quantity, the higher the latency of broadcast queries. Targeted queries are routed directly to the target shard, so their latency is independent of the shard quantity.
|
Targeted query (Targeted) | The query condition contains a shard key and is directly routed to the target shard. | Low latency |
Scatter-gather query (Scatter-Gather) | The query condition does not contain a shard key and is broadcast to all shards. | High latency |
Business Application: A social media platform uses Hashed sharding on _id (for even write distribution), but 99% of its queries are filtered by user_id.
Problem Manifestation: Each query cannot locate the target shard. The system must send requests to all 16 shards and then merge the results, causing latency to surge from 10ms to 200ms.
Optimization Operation: Switch to {user_id: 1} as the shard key. Queries are then routed directly to the target shard, and latency recovers to 15ms.
Query Routing Example: The following example demonstrates the difference between targeted queries and broadcast queries.
db.t_messages.find({ user_id: "U001", create_time: { $gt: ISODate("2024-01-01") } })
db.t_messages.find({ create_time: { $gt: ISODate("2024-01-01") } })
Specification 4: Compound Shard Key Balancing Read and Write Requirements
Core action: Use a compound shard key when a single field cannot simultaneously meet the requirements for high cardinality, even write distribution, and query coverage.
The prefix field of a compound shard key is used for targeted query routing, while the suffix field is used to further scatter data within the same prefix. This design supports targeted queries based on the prefix field and avoids hotspots caused by excessive data volume under a single prefix value.
Business Application: A multi-tenant SaaS platform where 90% of queries are filtered by tenant_id, but the data volume of some large tenants is 1000 times that of smaller tenants.
Problem Manifestation: Using tenant_id alone as the shard key causes data from large tenants to concentrate on a few shards, resulting in severe imbalance in both storage and performance.
Optimization Operation: Adopt a compound shard key {tenant_id: 1, _id: "hashed"}. Data from the same tenant is further scattered by hashing the _id. This approach supports tenant-specific targeted queries while avoiding hotspots caused by large tenants.
Compound Shard Key Design Example: The following example demonstrates two common design patterns for compound shard keys.
sh.shardCollection("db.t_orders", { customer_id: 1, order_id: "hashed" })
sh.shardCollection("db.t_logs", { month: 1, order_id: "hashed" })
Sharded Cluster Access Specifications
Specification 5: Prohibiting Direct Connection to Shard Nodes
Core Action: Applications must access the sharded cluster through mongos routing nodes. Direct connections to the underlying mongod shards are prohibited.
mongos is responsible for shard routing, query result merging, and metadata management. Directly connecting to a Shard node bypasses the routing logic. Consequently, data being written cannot be correctly routed, leading to chaotic data distribution. More critically, direct connection operations do not update the metadata on the Config Server. This causes the Balancer to misjudge data distribution and triggers a large number of unnecessary data migrations.
Architecture Relationship Diagram:
┌─────────────┐
│ Application │
└──────┬──────┘
│
┌──────▼──────┐
│ mongos │ ◄── Access must go through mongos.
└──────┬──────┘
┌───────────────┼───────────────┐
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ Shard 1 │ │ Shard 2 │ │ Shard 3 │
│ (Replica Set) │ │ (Replica Set) │ │ (Replica Set) │
└─────────────┘ └─────────────┘ └─────────────┘
↑ ↑ ↑
└───────────────┴───────────────┘
❌ Do not connect directly.
Business Application: An Ops person, in an attempt to "improve performance", allowed some applications to connect directly to Shard nodes.
Problem Manifestation: Direct connection operations bypass the shard routing logic of mongos, resulting in chaotic data distribution. The Balancer misjudges the data distribution, triggering a large number of unnecessary data migrations and causing a sharp decline in cluster performance.
Optimization Action: All applications must access the cluster uniformly through mongos. Configure the mongos node address in the connection string.
Connection String Configuration Example: The following example demonstrates the correct connection method for a sharded cluster.
mongodb://mongouser:password@10.0.0.1:27017,10.0.0.2:27017/admin
mongodb://mongouser:password@shard1-node1:27018/admin
Specification 6: Configuring Multiple mongos for High Availability
Core Action: Configure all mongos node addresses in the connection string. The driver automatically performs CLB and failover.
If a single mongos node fails and its address is the only one configured in the connection string, the entire application will lose database access. After multiple mongos nodes are configured, the driver automatically switches between healthy nodes, ensuring no service disruption during a single point of failure.
Business Application: A system is configured with only one mongos address.
Problem Manifestation: After this mongos node fails, the entire application loses database access, resulting in service interruption.
Optimization Action: Configure all mongos node addresses in the connection string. If a single point of failure occurs, the driver automatically switches to another mongos, ensuring no service disruption.
Multiple mongos Connection String Example: The following example demonstrates how to configure multiple mongos nodes in a connection string.
uri = "mongodb://user:pass@mongos1:27017,mongos2:27017,mongos3:27017/admin"
uri = "mongodb://user:pass@mongos1:27017/admin"
Balancer Configuration
Specification 7: Running the Balancer During Off-Peak Service Period
Core Action: Configure the Balancer time window to avoid performing Chunk migration during peak business hours.
The Balancer is responsible for migrating chunks between shards to maintain data balance. Chunk migration is an I/O-intensive operation that consumes significant disk and network bandwidth. If performed during peak business hours, it can lead to increased query latency and a higher timeout rate.
Business Application: During a major e-commerce promotion, the Balancer migrates chunks during peak hours.
Problem Manifestation: Data migration consumes significant disk I/O and network bandwidth, causing the user query timeout rate to surge to 5%.
Optimization Action: Configure the Balancer to run only between 2:00 AM and 6:00 AM, ensuring the system runs stably during major promotions.
Balancer Time Window Configuration Example: The following example demonstrates how to configure the Balancer's running time window and temporarily stop the Balancer during major promotions.
use config
db.settings.updateOne(
{ _id: "balancer" },
{
$set: {
activeWindow: {
start: "02:00",
stop: "06:00"
}
}
},
{ upsert: true }
)
sh.stopBalancer()
sh.isBalancerRunning()
sh.startBalancer()
Collection-Level Balancer Control: When you need to pause or enable Chunk migration only for specific collections (for example, when a collection is performing a batch import while other collections still require the Balancer to maintain balance), you can use collection-level Balancer control commands. This avoids affecting the data balance of other collections due to globally disabling the Balancer.
Note:
After the collection-level Balancer is disabled, chunks in that collection are not automatically migrated, but the Balancer for other collections continues to run normally.
After the batch import is complete, promptly re-enable the Balancer for that collection to avoid long-term data skew.
sh.disableBalancing("mydb.t_orders")
sh.enableBalancing("mydb.t_orders")
db.getSiblingDB("config").collections.findOne({_id: "mydb.t_orders"}).noBalance
Specification 8: Monitoring and Handling Jumbo Chunks
Core Action: Periodically check for Jumbo Chunks (chunks that exceed maxChunkSize but cannot be split) and handle them promptly.
A Jumbo Chunk is a chunk whose data volume exceeds the threshold but cannot be further split because the shard key values are identical. Jumbo Chunks cannot be migrated by the Balancer, causing some shards to hold significantly more data than others. This eventually triggers storage alarms and performance skew.
Business Application: The shard key cardinality of a system is insufficient, causing a large amount of data to be concentrated in a few chunks.
Problem Manifestation: Jumbo Chunks are formed, cannot be migrated or split, the storage usage of some shards is five times that of other shards, and disk alarms are triggered.
Optimization Action: Investigate the distribution of Jumbo Chunks and assess whether the shard key cardinality meets the requirements. If the cardinality is insufficient, redesign the shard key and migrate the data.
Jumbo Chunk Check Example: The following example demonstrates how to investigate Jumbo Chunks and the chunk distribution across shards.
use config
db.chunks.find({ jumbo: true }).forEach(function(chunk) {
print("Jumbo Chunk: " + chunk.ns + " [" +
tojson(chunk.min) + " -> " + tojson(chunk.max) + "]")
})
db.chunks.aggregate([
{ $match: { ns: "mydb.mycollection" } },
{ $group: { _id: "$shard", count: { $sum: 1 } } },
{ $sort: { count: -1 } }
])
Sharded Collection Management
Specification 9: Unique Indexes for Sharded Collections Must Include the Shard Key
Core action: When you create a unique index on a sharded collection, the index field must be prefixed with the shard key or contain the complete shard key.
The data of a sharded collection is distributed across multiple shards. Each shard can only guarantee uniqueness within itself and cannot guarantee global uniqueness across shards. Therefore, MongoDB requires that a unique index on a sharded collection must include the shard key. This uses the shard key to confine the uniqueness constraint within a single shard.
Business Application: A system attempts to create a unique index on the email field of a sharded collection, but the shard key is user_id.
Problem Manifestation: The creation fails with the error can't shard collection with unique index on { email: 1 }. Each shard can only guarantee the uniqueness of email within itself and cannot guarantee global uniqueness across shards.
Optimization Action: Change the unique index to a compound index {user_id: 1, email: 1}, or implement deduplication logic at the application layer to guarantee global uniqueness.
Sharded Collection Unique Index Example: The following example demonstrates the correct way to create a unique index on a sharded collection.
db.t_users.createIndex({ email: 1 }, { unique: true })
db.t_users.createIndex({ user_id: 1, email: 1 }, { unique: true })
Specification 10: Pre-Sharding to Avoid Initial Hotspots
Core action: For a newly created sharded collection, pre-split chunks and distribute them across shards to prevent all data from being initially written to a single shard.
A newly created sharded collection initially has only one Chunk located on a single shard. If a large batch of data is imported immediately, all data is first written to that shard. The Balancer cannot migrate the data in time, causing excessive pressure on that shard. Pre-splitting distributes chunks evenly across shards in advance, so that data is written directly to the target shards during import.
Business Application: A system immediately begins importing a large batch of data after creating a new sharded collection.
Problem Manifestation: Initially, only one Chunk is located on a single shard. All data is first written to that shard, and the Balancer cannot migrate it in time, resulting in excessive pressure on that shard.
Optimization Operation: When creating a sharded collection, specify the pre-split shard quantity. Chunks are pre-distributed evenly across shards, and data is written directly to the target shards during import to avoid initial hotspots.
Pre-split Configuration Example: The following example demonstrates the pre-split method for Hashed sharding and range sharding.
sh.shardCollection(
"mydb.t_orders",
{ order_id: "hashed" },
false,
{ numInitialChunks: 64 }
)
sh.shardCollection("mydb.t_logs", { month: 1, _id: 1 })
sh.splitAt("mydb.t_logs", { month: "2024-01", _id: MinKey })
sh.splitAt("mydb.t_logs", { month: "2024-02", _id: MinKey })
sh.splitAt("mydb.t_logs", { month: "2024-03", _id: MinKey })
sh.moveChunk("mydb.t_logs", { month: "2024-01", _id: MinKey }, "shard1")
sh.moveChunk("mydb.t_logs", { month: "2024-02", _id: MinKey }, "shard2")
Shard Key Design Checklist
Pre-Launch Checklist
|
Whether Cardinality Is Sufficiently High | `db.collection.distinct("field").length` Note: Executing queries on large tables triggers a full table scan. Perform this operation during off-peak hours or on a secondary node. | > 10,000 |
Whether write distribution is uniform | Observe the write QPS of each shard. | Deviation < 20% |
Whether major queries are covered | Analyze the TOP 10 query statements. | > 80% targeted queries |
Whether monotonic increase | Analyze field characteristics. | Not a timestamp/auto-increment ID |
Whether data skew is considered | Analyze field value distribution. | No obvious hot spots |
Regular Inspection
|
Number of Chunks per Shard | `sh.status()` | Investigate if deviation exceeds 20%. |
Number of Jumbo Chunks | `db.chunks.find({ jumbo: true })` | Handle if Jumbo Chunks exist. |
Shard Storage Balance | Disk usage per shard | Investigate if deviation exceeds 30%. |
Sharding Architecture Selection Recommendations
|
Data volume < 500 GB, QPS < 10,000 | Replica Set | No sharding required |
High-concurrency writes with equality queries as the primary operation | Sharded cluster + hashed sharding | Hash the business primary key |
Time-series data with range queries as the primary operation | Sharded cluster + compound sharding | `{time_bucket: 1, _id: "hashed"}` |
Multi-tenant SaaS | Sharded cluster + compound sharding | `{tenant_id: 1, _id: "hashed"}` |
Geographically distributed data | Zone Sharding | `{region: 1, _id: 1}` |
FAQs
Q1: How to Determine Whether the System Truly Needs to Be Upgraded to a Sharded Cluster?
Answer: Consider using a sharded cluster when one of the following conditions is met:
1. Storage physical bottleneck: The data volume of a single replica set exceeds the maximum allowed storage capacity, and storage is approaching its limit.
2. Concurrent write bottleneck: The write concurrency for core business is extremely high, with write QPS consistently exceeding 10,000. This causes the CPU and disk I/O of the Primary node in a single replica set to remain in a state of high saturation for extended periods, and this condition cannot be alleviated by optimizing indexes or adding read-only replicas.
Q2: What Are the Remediation Options If the Wrong Shard Key Is Selected in a Production Environment?
1. Minor issue: Only some queries are inefficient. This can be mitigated by creating compound indexes and optimizing frontend query conditions (by mandating the inclusion of the shard key), thereby alleviating the broadcast query pressure on the router.
2. Critical issue: This occurs if severe data skew is caused or a Jumbo Chunk cannot be migrated.
For MongoDB versions earlier than 5.0: You must create a new collection within the cluster and specify the optimized new shard key. Then, use a data migration tool (such as mongodump/mongorestore) to synchronize historical data to the new collection. Finally, switch the table name at the application layer.
For MongoDB V5.0 and later: You can utilize the natively supported online Resharding feature. Change the shard key online by executing the sh.reshardCollection() command. (Note: This operation consumes a significant amount of temporary disk space and cluster I/O, and must be performed during an off-peak service period).
3. Extreme case: Migrating to a new cluster and redesigning it may require service suspension.
4. Additional note: In MongoDB versions earlier than 4.2, the value of a document's shard key field is immutable. Starting from MongoDB version 4.2, you can update a document's shard key value unless the shard key field is the immutable _id field. To perform an update, use the following method to update a document's shard key value:
|
| |
| |
- | Note: If modifying a shard key causes a document to be moved to another shard, you cannot specify multiple shard key modifications in a batch operation. This means the batch size must be 1. If modifying a shard key does not cause a document to be moved to another shard, you can specify multiple shard key modifications in a batch operation. |
Note:
Operations must be run within a transaction or using a retryable write method on mongos. They cannot be executed directly on a shard.
The query filter must include equality conditions on the full shard key. For example, if a sharded collection uses {country: 1, userid: 1} as the shard key, to update a document's shard key, the query filter must include country: and userid:. You can also include other fields in the query as needed.
Q3: How to Safely Migrate Data in a Sharded Cluster?
1. Create the target collection: Based on the optimized plan, pre-create the indexes in the new cluster (or new collection) and execute sh.shardCollection() to initialize the new shard key.
2. Stop the Balancer: Before starting the migration, you must run the following command to disable the balancer on the source cluster. This prevents data inconsistency or migration tool interruption caused by frequent Chunk splitting and migration during the migration process:
sh.stopBalancer()
sh.getBalancerState()
3. Migrate data in batches: Use DTS to synchronize the historical full data to the target in batches, and enable incremental real-time synchronization.
4. Verify data consistency: Compare the document counts and sampled data between the source and target collections. For specific operations, see Create a Data Consistency Check. Statically compare the total document counts between the source and target collections (db.collection.countDocuments()).
Perform a sample comparison on core business fields, verify the MD5 checksum or key data structures, and ensure zero data loss or corruption.
5. Switch Traffic: During an off-peak service period, modify the database Connection String at the application layer to point to the new cluster/new collection. First, switch the write traffic. After the incremental data is fully caught up, switch the read traffic. Monitor business metrics (such as latency, slow log queries, and error rates) in real time.
6. Start the Balancer: After the traffic cutover is complete and the new cluster is running stably, remember to restore the balancer status on the source cluster (or confirm it on the new cluster), allowing the system to automatically perform subsequent data balancing: