Rob Pankow 
The ClickHouse storage guide covers the columnar analytics side of the observability stack, and the Elasticsearch storage guide covers the original search-and-log-indexing engine this fork descends from. OpenSearch, the Apache 2.0 fork maintained under the OpenSearch Software Foundation, runs the same Lucene-based indexing core but is increasingly the default choice for teams that want an open-source search and observability backend without a source-available license. Its storage behavior on Kubernetes carries the same structural demands as any other Lucene-based system, and a generic StorageClass does not meet them by default.
Every OpenSearch index is really a set of Lucene shards, and every shard is its own append-only segment store. Documents are written into new segments, and a background merge process continuously combines smaller segments into larger ones to keep query performance from degrading as segment count grows. That merge activity runs on the same data node, competing for the same disk I/O as the queries the cluster is supposed to be answering.
Why OpenSearch’s Segment-Merge I/O Breaks Generic Kubernetes Storage
An OpenSearch data node’s I/O has two workloads sharing one volume in most Kubernetes deployments:
Segment writes and refreshes happen continuously during indexing. Every index refresh interval (one second by default) flushes buffered documents into a new, immutable Lucene segment. A high-ingest logging or metrics pipeline is writing new segments on a steady cadence, not in occasional bursts.
Background segment merges read several existing segments, combine them into one larger segment, and delete the originals. This is sustained sequential I/O that runs whenever segment count crosses OpenSearch’s internal merge policy thresholds, and it runs on the same node-local disk that is simultaneously serving indexing writes and search-query reads. On a persistent volume sized for steady-state capacity rather than merge throughput, a merge cycle on one shard can add tail latency to searches hitting a completely different index on the same node.
This is the same structural cost documented for other log-structured merge tree and segment-based engines: compaction-style rewrites are how the design keeps reads fast, and the storage layer underneath has to absorb that rewrite traffic without stealing I/O from the query path, the same argument covered in the ClickHouse MergeTree storage guide for its own background-merge cycle.
OpenSearch Storage Tiers: Matching I/O Patterns to Backends
The table below maps OpenSearch’s storage tiers to the backend each one needs, using a representative high-ingest logging or observability cluster as the reference point.
| OpenSearch storage tier | I/O pattern | Latency sensitivity | Best-fit backend |
|---|---|---|---|
| Active segment writes / refresh | Small, continuous sequential writes | High, gates index refresh latency | Disaggregated NVMe/TCP or NVMe/RoCE block storage |
| Background segment merges | Large sequential read + write, bursty | Medium, but contends with the query path if slow | Same NVMe/TCP tier, sized for merge burst throughput |
| Hot query working set | Random reads across recent shards | High, gates search response time | Same tier, ideally with enough page cache to avoid most disk reads |
| Cold / warm tier (ISM-managed indices) | Large sequential reads, infrequent | Low | S3-compatible object storage, via OpenSearch’s Index State Management policies |
Table 1: OpenSearch storage tier characteristics and the backend each tier fits best.
OpenSearch’s own Index State Management (ISM) plugin already assumes this split: it ages indices from hot to warm to cold storage on a policy, moving older, less-queried indices off expensive fast storage. That policy handles the cold tier correctly. What it does not solve is the hot tier: the actively-indexing and actively-queried shards still need to sit on storage provisioned for continuous write and burst-merge throughput, not sized like ordinary capacity.
A slow merge tier turns into slow indexing and slow search at the same time. Talk to a storage architect about sizing an NVMe/TCP tier for OpenSearch’s active shards and merge bursts, separate from your ISM warm/cold tiers. Talk to a storage architect
How Disaggregated NVMe/TCP Absorbs OpenSearch’s Merge and Rebalance I/O
Running OpenSearch data nodes with persistent volumes backed by disaggregated NVMe/TCP (or NVMe/RoCE on RDMA-capable fabrics) closes the two gaps a single generic volume leaves open:
Merge bursts do not steal I/O from the search path. Because the hot-tier volume sits on a shared NVMe fabric rather than one node’s local disk, the throughput available to a shard’s background merge is not capped by whatever spare bandwidth is left on that node alongside every other pod scheduled there. A merge cycle on one shard does not degrade search latency for a neighboring index sharing the same physical node.
Shard relocation becomes a volume reattachment, not a full segment copy. OpenSearch rebalances shards across data nodes for load distribution and recovers from node loss by copying a full replica shard’s segment files from a healthy copy, an operation whose cost scales with index size. When the underlying volume lives on the storage fabric instead of node-local disk, a rescheduled pod reattaches the same PVC and resumes serving from where it left off, without the full-shard data transfer.
OpenSearch deployments on Kubernetes also inherit the same snapshot pattern already covered for CSI snapshot architecture: a VolumeSnapshot of a data node’s active-shard volume gives a point-in-time, storage-side capture of the current segment set, restorable in seconds, as a complement to OpenSearch’s own repository-based snapshot API, which streams shard data through a registered repository (typically S3-compatible object storage) and takes longer as index size grows. A VolumeSnapshotClass per index tier, frequent snapshots for actively-indexing hot indices and longer-retention snapshots for stable warm indices, mirrors the pattern already covered for the ClickHouse and CockroachDB deployments elsewhere in this series.
The same architecture supports OpenSearch running hyperconverged, with storage and compute co-located on data nodes, or disaggregated, with a dedicated storage fabric behind the cluster, on vSphere, bare-metal Kubernetes, or OpenShift. Simplyblock licenses per usable storage TB per year, with no per-core fee, so scaling a data-node fleet horizontally to absorb higher ingest volume is a compute decision, not an automatic storage cost increase.
Questions and Answers
Why does OpenSearch need a dedicated hot storage tier instead of one generic volume for everything? OpenSearch’s Lucene-based shards write new segments on every index refresh and continuously merge them in the background, generating sustained sequential write and read traffic on top of whatever read traffic search queries add. A single generic Kubernetes persistent volume sized for capacity rather than throughput lets merge activity compete with search reads for the same disk, which shows up as query latency spikes unrelated to query complexity.
What happens when an OpenSearch shard is relocated or a node is lost on generic Kubernetes storage? OpenSearch normally recovers a lost or relocated shard by copying its full segment set from a healthy replica, a transfer that scales with index size and can take a long time for large indices. A persistent volume claim backed by simplyblock’s NVMe/TCP storage reattaches to a rescheduled pod in seconds instead, avoiding the full-shard data transfer in most node-failure or rebalance scenarios.
Can OpenSearch’s Index State Management (ISM) tiering replace a dedicated NVMe/TCP tier? ISM correctly ages indices from hot to warm to cold, moving less-queried data to cheaper, slower storage on a policy, and that part of the architecture does not need to change. The gap is the hot tier: actively-indexing shards, background merges, and recently-queried indices still need low-latency block storage, which ISM assumes exists but does not provision on its own.
How does simplyblock fit into an OpenSearch deployment on Kubernetes? Simplyblock provides NVMe/TCP (or NVMe/RoCE) block storage as persistent volumes for OpenSearch data nodes, sized for the continuous segment-write and merge-burst I/O that generic Kubernetes storage classes are not built to absorb. Simplyblock’s CSI driver adds fast, storage-side snapshots for index-level backup and rollback, and the same architecture supports both hyperconverged and disaggregated deployment models.
How should I size storage for OpenSearch on Kubernetes? Size the NVMe/TCP hot tier for the working set of actively-indexing shards plus enough burst throughput headroom for background segment merges at peak ingest rate, and let OpenSearch’s own ISM policy move aging indices to warm or cold, S3-compatible object storage. Simplyblock’s pricing is per usable TB per year with no per-core fee, so scaling the data-node fleet horizontally for higher ingest volume is a compute decision, not an automatic storage cost increase.
