Your search cluster stops accepting writes. Reads keep working, so nothing pages. Then someone notices that a dashboard has been showing an empty result set for a week, and it is not empty because there is no data. It is empty because the index holding that data was never created.
This is the shard ceiling, and it is one of the few capacity limits that degrades into wrong answers rather than errors. This post is what the limit is, why it is usually reached by accident rather than by growth, and how to tell whether you are near it. For whoever operates an Elasticsearch or OpenSearch cluster that somebody else's application writes to.
The limit
A node will refuse to hold more than a fixed number of shards. The default in recent versions is 1,000 per node, and cluster.max_shards_per_node is the setting.
Total capacity is that number multiplied by your data node count. On a single-node cluster, which is more common in production than people admit, the ceiling is exactly 1,000 unless somebody raised it.
When you hit it, index creation fails:
validation_exception: Validation Failed: 1: this action would add [2] shards,
but this cluster currently has [3000]/[3000] maximum normal shards open
That is a clear error. The problem is where it surfaces.
Why it turns into empty reads
The failure lands on index creation, which for most applications happens inside a write path that creates a time-based index: logs-2026-09-17, events-2026-09, one per tenant, one per scan.
So the sequence is: the write fails, the application logs it and moves on, and the index is never created. Later a read queries that index pattern. The pattern matches nothing. Elasticsearch returns an empty result set, with a 200, because asking for documents in a pattern that matches no index is not an error. It is a question with the answer "none".
Nothing in the read path can tell "no documents matched" apart from "the index was never created". Both are an empty list and a success status. Your dashboard renders zero, your alert on that metric never fires because zero is below the threshold, and the health check is green because the cluster is up.
That is the whole failure. A capacity limit on the write side becomes a silent data-absence bug on the read side, separated by hours or weeks.
Why it is reached by accident
Almost never because you have too much data. Nearly always because you have too many indices holding very little.
The usual causes:
One index per small thing. Per tenant, per day, per job. A thousand tenants on a daily index is a thousand shards a day. This is the dominant cause and it is a design decision that looked fine at ten tenants.
Default shard counts nobody changed. Older versions defaulted to five primaries plus one replica each, so every index cost ten shards regardless of holding four documents. The modern default of one primary is much better, and plenty of long-lived clusters still carry index templates written when it was five.
No lifecycle policy. Indices are created and never deleted. The cluster fills with empty ones. On a cluster we looked at, more than 1,500 indices held zero documents and together consumed a third of the ceiling.
The tell for all three is a cluster with thousands of shards and very little data. Shard count and data volume are independent, and it is shard count that runs out.
Finding out where you stand
# total shards against the ceiling
curl -s 'localhost:9200/_cluster/health?pretty' | grep active_shards
curl -s 'localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true' \
| grep max_shards_per_node
# indices sorted by document count: the empty ones are at the bottom
curl -s 'localhost:9200/_cat/indices?v&h=index,pri,rep,docs.count,store.size&s=docs.count:asc' \
| head -40
# how many hold nothing at all
curl -s 'localhost:9200/_cat/indices?h=index,docs.count' | awk '$2==0' | wc -l
If active shards are above roughly 80 percent of your ceiling, act now. The remaining margin disappears in one busy day, and the failure will not be the one you are watching for.
The order to fix it in
Raise the ceiling first, as a stopgap. It is a dynamic setting and it buys you time to do the real work.
curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '
{"persistent": {"cluster.max_shards_per_node": 4000}}'
Understand what this is: the limit exists because each shard costs heap on the master and file handles on the node, and a cluster with many thousands of small shards has slow cluster state updates and poor query performance. Raising it converts a hard failure into gradual degradation. Do it to stop the bleeding, not as the fix.
Delete the empty indices. Immediate, large, and safe once you have confirmed they hold nothing.
Fix the shard count in the index template, so new indices do not inherit five primaries for a hundred documents. One primary and one replica is right for almost every small index.
Consolidate the index-per-thing pattern. One index per time period with a tenant field, rather than one per tenant. Filter on the field. This is the change that actually fixes it, and it is the largest.
Add lifecycle management, so indices roll over on size and are deleted on age without anyone remembering.
The monitoring that would have caught it
Two alerts, and the second is the important one.
Alert on shard count as a percentage of the ceiling, at 80 percent. Ordinary capacity alerting.
Then alert on index creation failures specifically, at the application, as an error rather than a logged warning. This is the signal that distinguishes "no data yet" from "we failed to make somewhere to put the data", and it is the only place in the whole sequence where the difference is visible. Everywhere downstream, both look like zero.
The concession
A single-node cluster with a thousand shards is a configuration that many teams run deliberately, and correctly, for a workload that fits. The argument here is not that small clusters are wrong.
It is that the ceiling is one of the few limits whose failure mode is a wrong answer rather than an outage, so it needs an alert that a generic infrastructure dashboard will not give you. A cluster at 99 percent of its shard ceiling is green on every standard health check, right up to the moment it starts quietly not storing things.
The implication
The general shape is worth carrying beyond search clusters: when a write failure produces an absence, and the read path cannot distinguish absence from emptiness, you have built a system that lies to you rather than one that breaks.
Wherever that pattern exists, the alert belongs on the write, because by the time the read is wrong there is nothing left to detect.
