explain("executionStats") to confirm index hits and reasonable scan efficiency.explain() is the only reliable method for validating execution plans, as it can reveal key information including scan efficiency, sorting methods, and index hits. By comparing and analyzing these key metrics, you can quickly locate query performance bottlenecks:Metric | Ideal Value | Requires Optimization |
`winningPlan.stage` | `IXSCAN` | `COLLSCAN` (full table scan, missing index) |
`totalKeysExamined / nReturned` | ≈ 1 | >> 10 (low index efficiency) |
Whether `SORT` Stage Exists | None. | Yes (in-memory sorting; create an index for the sorting field) |
Stage | Description | Ideal or Not |
`COLLSCAN` | Full Table Scan | Optimization recommended |
`IXSCAN` | Index scanning | Ideal |
`FETCH` | Fetching documents from the table | Optimization recommended |
`SORT` | In-memory sorting | Create an index for the sorting field. |
userId_createTime composite index actually triggered a full table scan.explain("executionStats") to validate the execution plan. Ensure that winningPlan.stage is IXSCAN and the scan efficiency ratio is close to 1 before proceeding with the release.// Obtain execution statistics (Recommended)db.t_orders.find({status: "paid",createTime: { $gte: ISODate("2024-01-01") }}).explain("executionStats")
// Error: Returning the entire document (including unnecessary large fields)db.t_products.find({ productId: "P001" })// Correct: Returning only the required fieldsdb.t_products.find({ productId: "P001" },{ _id: 1, name: 1, price: 1, stock: 1 })// Correct: Excluding large fieldsdb.t_products.find({ productId: "P001" },{ description: 0, htmlContent: 0 })
skip + limit for deep pagination, and instead use cursor-based pagination on sorted fields.skip(N) works by first scanning and discarding the first N records, and then returning the subsequent results. When N reaches tens of thousands or even hundreds of thousands, the query time increases linearly with the page number, and the response time for deep pagination can reach several seconds or even time out. Cursor-based pagination records the sort field value of the last record on the previous page and uses an index to directly locate the starting point of the next page, making its performance independent of the page number.skip(100000).limit(20) to navigate to page 5000._id and sort fields. Regardless of which page is navigated to, the query time remains stable within 10ms.// Error: skip pagination, where performance degrades sharply during deep pagination.db.t_orders.find().sort({ createTime: -1 }).skip(100000).limit(20)// Correct: Cursor pagination, stable performance// Page 1db.t_orders.find().sort({ createTime: -1, _id: -1 }).limit(20)// Subsequent pages (using the createTime and _id of the last record from the previous page)db.t_orders.find({$or: [{ createTime: { $lt: lastTime } },{ createTime: lastTime, _id: { $lt: lastId } }]}).sort({ createTime: -1, _id: -1 }).limit(20)
$in and $or within 50 items. If it exceeds this limit, perform queries in batches.$in query, it performs a multi-interval index scan. In the Plan Stage, $or is split into multiple sub-plans, where each branch may use a different index, requiring deduplication and merging. When the list is too large, the number of index lookups and the overhead of result merging increase linearly. This can cause a single query delay of up to several hundred milliseconds and impose an instantaneous load on the database.{productId: {$in: [100 IDs]}}.$in query into multiple batches for execution.// Exercise caution in high-concurrency scenarios: large list $indb.t_orders.find({ productId: { $in: [/* 100 IDs */] } })// Correct: Batch Queryingasync function batchQuery(productIds, batchSize = 50) {const results = [];for (let i = 0; i < productIds.length; i += batchSize) {const batch = productIds.slice(i, i + batchSize);const docs = await db.t_orders.find({productId: { $in: batch }}).toArray();results.push(...docs);}return results;}
replaceOne) overwrites the entire document, causing unspecified fields to be lost. Operators such as $set modify only the specified fields, resulting in smaller data writes and no risk of concurrent overwrites.$set. This completely resolves concurrency issues and reduces the amount of data written from 10KB to 100B.$set.// Incorrect: Full document replacement (unspecified fields are lost, and concurrency issues may arise).const user = await db.t_users.findOne({ _id: userId });user.nickName = "new nickname";await db.t_users.replaceOne({ _id: userId }, user); // Dangerous!// Correct: Partial update with $setawait db.t_users.updateOne({ _id: userId },{ $set: { nickName: "new nickname" } });// Correct: Combine multiple update operatorsawait db.t_users.updateOne({ _id: userId },{$set: { nickName: "new nickname" },$inc: { loginCount: 1 },$currentDate: { lastLoginTime: true }});
0, triggering a major capital loss incident.// Danger: An empty condition matches all documents, making it highly prone to erroneously updating non-target data.db.t_users.updateOne({}, { $set: { status: "inactive" } })// Correct: With explicit query conditions, precisely targeting the intended documents.db.t_users.updateOne({ userId: "U001" },{ $set: { status: "inactive" } })// Correct: Batch updates must include explicit conditions, and the impact scope should be assessed before updating.db.t_users.updateMany({ status: "pending", createTime: { $lt: ISODate("2024-01-01") } },{ $set: { status: "expired" } })
# Python Example: Preventing Empty Condition Updatesdef safe_update(collection, filter_query, update_doc, **kwargs):if not filter_query or filter_query == {}:raise ValueError("Update filter cannot be empty!")return collection.update_one(filter_query, update_doc, **kwargs)
$set + a positional operator to precisely update a single element, reducing the write volume by 99%.// Incorrect: Full array replacement.const order = await db.t_orders.findOne({ orderId: "ORD001" });order.items[0].quantity = 5;await db.t_orders.updateOne({ orderId: "ORD001" },{ $set: { items: order.items } } // Full write);// Correct: Update matched elements using the $ positional operator.await db.t_orders.updateOne({ orderId: "ORD001", "items.productId": "P001" },{ $set: { "items.$.quantity": 5 } } // Precise update);// Correct: Use arrayFilters to update multiple elements (MongoDB 3.6+)await db.t_orders.updateOne({ orderId: "ORD001" },{ $set: { "items.$[elem].quantity": 5 } },{ arrayFilters: [{ "elem.productId": { $in: ["P001", "P002"] } }] });// Correct: Append elements to an array.await db.t_orders.updateOne({ orderId: "ORD001" },{ $push: { items: { productId: "P003", quantity: 1 } } });// Correct: Remove elements from an array.await db.t_orders.updateOne({ orderId: "ORD001" },{ $pull: { items: { productId: "P003" } } });
# Python Example: Batch Insertionimport timedef batch_insert(collection, documents, batch_size=1000, interval=0.1):for i in range(0, len(documents), batch_size):batch = documents[i:i + batch_size]collection.insert_many(batch, ordered=False)# Pause between batches to reduce database pressure.if i + batch_size < len(documents):time.sleep(interval)
remove operations on large volumes of data. Use TTL indexes, batch deletion, or drop the collection instead.remove operation on a large volume of data generates a substantial amount of delete oplog logs in a short time, saturating CPU and disk I/O resources and causing a sharp spike in primary-secondary replication delay. For time-based expiration cleanup scenarios, TTL indexes are smoothly executed by MongoDB background tasks, resulting in minimal impact on business operations. Options for deletion strategies:Solution | Use Cases | Performance Impact |
TTL index | Automatically expires based on time | Runs smoothly in the background with no perceptible impact. |
drop collection | Emptying the entire collection | Completes instantly. |
Batch deletion | Deleting partial data | Controllable, requires rate control. |
Rename + create new | Retaining table structure and clearing data | Fast. |
db.logs.remove({createTime: {$lt: a_timestamp}}).// Batch deletion, 1000 records per batch, with a 100ms interval between batches.async function batchDelete(collection, filter, batchSize = 1000) {let totalDeleted = 0;while (true) {// First, query the document IDs to be deleted.const docs = await collection.find(filter, { _id: 1 }).limit(batchSize).toArray();if (docs.length === 0) break;const ids = docs.map(d => d._id);const result = await collection.deleteMany({ _id: { $in: ids } });totalDeleted += result.deletedCount;console.log(`${totalDeleted} records deleted...`);// Pause between intervals.await new Promise(resolve => setTimeout(resolve, 100));}return totalDeleted;}
Operating System: Ubuntu 24.04.3 LTS / x86_64
Runtime Version: Node.js v22.13.1
// Route writes to the corresponding collection based on month.const collName = "t_events_" + new Date().toISOString().slice(0, 7).replace("-", "")db.getCollection(collName).insertOne({userId: "U001",action: "click_buy",createTime: new Date()})// Clean up expired data: Directly drop collections older than two months. This operation completes instantly.db.t_events_202601.drop()
writeConcern: { w: "majority", j: true } to ensure that success is returned only after data is written to a majority of nodes.w:1 acknowledges a write as successful only after the data is written to the Primary node. If the Primary node fails before the data is synchronized to a Secondary node, writes that have been acknowledged as successful will be lost after a primary/secondary switchover. w:"majority" requires that data be written to a majority of nodes before acknowledgment, ensuring that data is not lost even if the Primary node fails.Configuration | Data Security | Write Latency | Use Cases |
`{ w: 1 }` | Medium | Lowest | Scenarios with low data security requirements, such as logging |
`{ w: "majority" }` | High | Relatively high | Core business, transactions, and orders |
`{ w: 1, j: false }` | Extremely low | Lowest | Do not use. Data loss may occur. |
`{ w: "majority", j: true }` | Extremely high | Relatively high | Finance, payments, and other capital-related operations |
w:1 configuration, and success is returned immediately after data is written to the Primary node.w:"majority". Success is returned only after data is synchronized to a majority of nodes. Write latency increases slightly, but data loss is zero during a primary/secondary switchover.// Prohibited: j:false may cause data loss.db.t_orders.insertOne({ orderId: "ORD001" },{ writeConcern: { w: 1, j: false } } // Dangerous!)// Correct: Use majority for core services.db.t_orders.insertOne({ orderId: "ORD001", amount: 199.00 },{ writeConcern: { w: "majority", j: true, wtimeout: 5000 } })// Connection String Configuration Method (Globally Effective)// mongodb://user:pwd@host1,host2,host3/db?w=majority&j=true&wtimeout=5000
$match at the beginning and follow it immediately with $project to reduce fields. Avoid overusing $unwind.$unwind to expand arrays and then $match to filter, the intermediate data volume can be expanded by tens of times. Conversely, if you first execute $match to filter and then $unwind, the data volume can be significantly reduced at the source.$unwind to expand arrays (expanding 1 million documents to 100 million entries), then uses $match to filter (leaving 100,000 entries), and finally uses $group for aggregation.$match (reducing from 1 million to 10,000), then expand with $unwind (expanding to 1 million entries). This reduces memory usage to a manageable range and cuts execution time from a timeout to 5 seconds.// Error: Expanding before filtering causes intermediate data volume to expand.db.t_orders.aggregate([{ $unwind: "$items" }, // Unwinding first causes data volume to expand.{ $match: { status: "paid" } }, // Then filter{ $group: { _id: "$items.category", total: { $sum: "$items.amount" } } }])// Correct: Filtering before expanding reduces data volume at the source.db.t_orders.aggregate([{ $match: { status: "paid" } }, // Filter first to reduce data volume.{ $project: { items: 1 } }, // Keep only the required fields.{ $unwind: "$items" }, // Then unwind.{ $group: { _id: "$items.category", total: { $sum: "$items.amount" } } }])
$lookup joins or large-scale $group aggregations, a single aggregation can take tens of seconds. During peak hours, even a small number of such requests can saturate the database CPU, impacting core business operations.Scenario | Not Recommended | Recommended |
Complex reports | Real-time Aggregation | Offline computing + caching |
Multi-table Join | Multiple `$lookup` operations | Application-layer stitching |
Real-time statistics | MapReduce | Pre-aggregation + caching |
$lookup, $unwind, and $group operations. A single aggregation takes more than 10 seconds.Check Item | Verification Method | Passing Criteria |
All queries hit indexes. | `explain("executionStats")` | The `stage` is `IXSCAN`, with no `COLLSCAN`. |
No in-memory sorting | `explain("executionStats")` | No `SORT` Stage |
Reasonable scan efficiency | `totalDocsExamined / nReturned` | A ratio close to 1 |
Using projection | Code review | Returning only the necessary fields |
Conditional Update | Code review | No empty conditions |
Using $set for updates | Code review | Full document replacement is not recommended for scenarios involving updates to only a few fields. |
Controlled batch operations | Code review | Limit each batch to ≤ 1000 items, and appropriately control the concurrency. |
Core business w:majority | Check Write Concern. | Using majority for payment/order transactions. |
Check Item | Verification Method | Recommended Action |
Slow query analysis | Slow query log | Regularly analyze queries that take longer than 100 ms. |
Write latency monitoring | Monitoring dashboard | Investigate the cause when the P99 latency is abnormal. |
Oplog utilization | `rs.printReplicationInfo()` | Oplog window period > 24 hours |
// 1. Use explain to view the execution plan.var result = db.t_orders.find({ status: "paid" }).explain("executionStats")// 2. Check whether indexes are used.// A winningPlan.stage value of IXSCAN indicates that an index is hit.// A winningPlan.stage value of COLLSCAN indicates a full collection scan, which requires immediate optimization.// 3. Check scan efficiency.var stats = result.executionStatsif (stats.nReturned === 0) {print("The returned result is empty. Please verify that the query conditions are correct.")} else if (stats.totalKeysExamined === 0) {print("No index is used (full collection scan). Please refer to Step 2 to determine whether an index needs to be created.")} else {print("Index Scan Ratio:", (stats.totalKeysExamined / stats.nReturned).toFixed(2))print("Document Scan Ratio:", (stats.totalDocsExamined / stats.nReturned).toFixed(2))// A value close to 1 for both ratios is ideal.// An index scan ratio significantly greater than 1 indicates insufficient index selectivity or redundant scanning.// A document scan ratio significantly greater than 1 indicates that a large number of documents that do not meet the criteria still need to be filtered by returning to the table after index filtering.}// 4. Check whether in-memory sorting exists.// The presence of a SORT stage in executionStages indicates that sorting is not performed using an index and has degraded to in-memory sorting.// Common causes: The sorting field is not included in the index, or the sort order of the field in the index does not match the query.
count() to confirm the number of matching documents._id of the last record in each batch to support resumable execution.Esta página foi útil?
Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.
comentários