In some cases, the bulk rejection rate of a cluster increases. Specifically, an error message such as the following will appear when bulk writes are performed:
[2019-03-01 10:09:58][ERROR]rspItemError: {"reason":"rejected execution of org.elasticsearch.transport.TransportService$7@5436e129 on EsThreadPoolExecutor[bulk, queue capacity = 1024, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6bd77359[Running, pool size = 12, active threads = 12, queued tasks = 2390, completed tasks = 20018208656]]","type":"es_rejected_execution_exception"}
GET _cat/thread_pool/bulk?s=queue:desc&v
Generally, the default value for a queue is 1024. If there is 1024 under a queue, rejections have occurred on the node.Bulk rejections are typically caused by a too large shard capacity or uneven allocation of shards. The specific cause can be identified and analyzed by following the steps below.
GET _cat/shards?index=index_name&v
curl "$p:$port/_cat/shards?index={index_name}&s=node,store:desc" | awk '{print $8}' | sort | uniq -c | sort
The results are as follows (the first column shows the number of shards, and the second shows the node ID), where some nodes are allocated with one shard, while some eight.number_of_shards
parameter in the index template (after the template is created, it will take effect when you create new indexed, and previous indices will not be adjusted).routing.allocation.total_shards_per_node
parameter. For more information, please see Total Shards Per Node.Reference command:A certain buffer should be reserved for
total_shards_per_node
so as to prevent any machine failure from rendering allocation of shards impossible (for example, if there are 10 machines and an index has 20 shards,total_shards_per_node
should be set to above 2, such as 3).
PUT {index_name}/_settings
{
"settings": {
"index": {
"routing": {
"allocation": {
"total_shards_per_node": "3"
}
}
}
}
}
PUT _template/{template_name}
{
"order": 0,
"template": "{index_prefix@}*", // Prefix of the index to be adjusted
"settings": {
"index": {
"number_of_shards": "30", // Specify the number of shards allocated to the index based on a shard size of about 30 GB
"routing.allocation.total_shards_per_node":3 // Specify the maximum number of shards that a node can accommodate
}
},
"aliases": {}
}
Was this page helpful?