Elasticsearch on Kubernetes: Persistent Storage Architecture for Production

Elasticsearch is the dominant platform for full-text search and log analytics on Kubernetes. Clusters serving production observability pipelines or enterprise search routinely index gigabytes per minute, and the storage layer underneath determines how much of that write budget is available to your application before a node falls behind.

Unlike stateless web services, Elasticsearch nodes carry state: Lucene index segments accumulate on persistent storage as data is ingested, and the merge operations that compact those segments into larger files compete for the same disk I/O as incoming writes and outgoing queries. On Kubernetes, this creates a persistent-volume design problem. You need enough IOPS to sustain indexing throughput, enough capacity for retention, and enough isolation to ensure one noisy shard does not degrade another tenant’s queries.

This post covers how to size and configure persistent storage for Elasticsearch on Kubernetes, with specific attention to the hot-tier IOPS requirement and how disaggregated NVMe block storage changes the tradeoffs for production clusters.

Why Elasticsearch Storage Is Different on Kubernetes

Elasticsearch uses Apache Lucene as its indexing engine. Documents written to an index are buffered in memory (the JVM heap’s indexing buffer), then flushed to disk as immutable segment files every few seconds or when the buffer fills. The merge scheduler runs concurrently in the background, combining small segments into fewer, larger ones to keep query performance stable and disk usage bounded.

The practical I/O profile on a busy hot-tier node is a mix of three access patterns:

• Sequential writes (segment flush, merge write): 50 to 500 MB/s depending on ingest rate
• Sequential reads (merge source reads): comparable to merge write bandwidth
• Random reads (query shard fetches): workload-dependent, but tail latency matters

For Kubernetes deployments, this mix makes local NVMe the obvious choice from a raw performance standpoint. But local storage creates a scheduling problem: if the node fails or needs to drain, the Elasticsearch pod cannot reschedule because its data is tied to a specific host. Teams compensate by over-provisioning Elasticsearch replicas, which doubles storage cost and spreads the I/O load across more nodes rather than solving the root constraint.

Network-attached block storage that delivers NVMe-class performance changes this tradeoff. The pod can reschedule to any node in the cluster without data loss, and you size replicas for query capacity rather than host availability.

NVMe/TCP Block Storage for Elasticsearch Hot-Tier Nodes

Hot-tier nodes carry recently ingested data that is still being actively queried and merged. They require the highest IOPS and throughput of any data tier. Cloud block storage (AWS gp3, Azure Premium SSD) delivers adequate throughput at small shard counts but becomes expensive at the volume and IOPS levels needed by a production ingest pipeline processing tens of billions of events per day.

NVMe over TCP (NVMe/TCP) is a block storage protocol that carries native NVMe commands over standard Ethernet without specialized hardware. The Linux kernel NVMe/TCP driver has been upstream since kernel 5.0. Because NVMe/TCP uses the same command queue model as local NVMe, up to 65,535 queues with up to 65,535 outstanding commands each, it delivers latency in the 100 to 400 microsecond range on a well-tuned 25 GbE link. That range is adequate for Elasticsearch segment flush and merge workloads, which are throughput-bound rather than latency-bound.

simplyblock provides a disaggregated NVMe/TCP block storage pool for Kubernetes, integrated via a standard CSI driver. Each Elasticsearch StatefulSet pod gets a persistent volume claim backed by a simplyblock volume. Because the storage is network-attached:

• Pod rescheduling attaches the same volume to the new node without copying data
• simplyblock applies multi-replica protection at the storage layer, so you do not need to run replica shards solely for data durability
• The storage pool scales independently of the number of Elasticsearch nodes, so you add disk capacity without adding compute

Running Elasticsearch at scale on Kubernetes? NVMe/TCP storage removes the local-disk scheduling constraint without sacrificing hot-tier indexing throughput. simplyblock’s CSI-integrated disaggregated NVMe/TCP pool lets Elasticsearch pods reschedule freely while keeping NVMe-class performance for segment writes and merges. Talk to a storage architect

Sizing Persistent Volumes for Elasticsearch Data Tiers

Elasticsearch organizes data across four tiers based on access frequency and age. Each tier has different storage performance requirements and different sizing rules.

Table 1: Storage requirements per Elasticsearch data tier on Kubernetes.

The 3× raw data sizing factor for the hot tier accounts for three overlapping components: primary shard data (1×), one replica shard (1×), and merge space plus temporary segment files (0.5 to 1×).

A cluster indexing 500 GB of raw data per day with a 7-day retention policy needs approximately 500 GB × 7 days × 3 = 10.5 TB of hot-tier PV capacity. If you run two replica shards (the Elasticsearch default for production), factor in an additional 1× for the second replica, bringing the total closer to 14 TB.

For warm-tier nodes that hold data for 7 to 30 days after it ages out of hot, size at 2× raw data and use a cheaper SSD StorageClass. Cold and frozen tiers are best served by separate StorageClasses or object storage backends.

Configuring Kubernetes Storage for Elasticsearch

Elasticsearch on Kubernetes should always run as a StatefulSet, never a Deployment. StatefulSets provide stable pod identity (which Elasticsearch uses for node names and cluster membership) and stable PVC bindings (which ensure the same pod always reattaches to the same volume after rescheduling).

A minimal StorageClass for hot-tier Elasticsearch using simplyblock:

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: simplyblock-nvme-hot
5
provisioner: csi.simplyblock.io
6
parameters:
7
  pool: hot-tier
8
  replication: "2"
9
volumeBindingMode: WaitForFirstConsumer
10
allowVolumeExpansion: true
11
reclaimPolicy: Retain

Setting volumeBindingMode: WaitForFirstConsumer ensures the volume is provisioned in the same availability zone as the pod, preventing cross-zone I/O charges. Setting reclaimPolicy: Retain protects against accidental data loss if a PVC is deleted.

The StatefulSet volumeClaimTemplates block for hot-tier Elasticsearch nodes:

1
volumeClaimTemplates:
2
  - metadata:
3
      name: elasticsearch-data
4
    spec:
5
      accessModes: ["ReadWriteOnce"]
6
      storageClassName: simplyblock-nvme-hot
7
      resources:
8
        requests:
9
          storage: 500Gi

For host-level preparation, Kubernetes nodes running Elasticsearch pods must have vm.max_map_count set to at least 262144. Apply this with a DaemonSet init container:

1
initContainers:
2
  - name: set-vm-max-map-count
3
    image: busybox
4
    command: ["sysctl", "-w", "vm.max_map_count=262144"]
5
    securityContext:
6
      privileged: true

For multi-tier clusters, use node affinity and separate StorageClasses to route hot, warm, and cold tier pods to the appropriate node pools. Label hot-tier Kubernetes nodes with elasticsearch-tier: hot and add a nodeSelector or nodeAffinity rule to the hot-tier StatefulSet spec to enforce placement and avoid cross-tier I/O contention.

Questions and Answers

What are the persistent storage requirements for Elasticsearch on Kubernetes?

Elasticsearch hot-tier nodes require high-IOPS block storage, typically 10,000 IOPS or more per node for production ingest rates, with 500 MB/s or higher sustained write throughput. Warm-tier nodes can use standard SSD storage with 2,000 to 5,000 IOPS. Cold-tier nodes prioritize capacity over performance. All tiers should use ReadWriteOnce PVCs attached to StatefulSets, with volumeBindingMode: WaitForFirstConsumer to enforce zone-local placement.

Should Elasticsearch on Kubernetes use local storage or network-attached storage?

Local NVMe delivers the lowest latency, but it creates pod scheduling constraints: if the host fails, the pod cannot reschedule to a node that does not have the data. Network-attached NVMe storage (NVMe/TCP) resolves this by decoupling the pod lifecycle from the physical disk location. simplyblock delivers NVMe-class IOPS over the network via a CSI driver, so Elasticsearch pods can reschedule freely while retaining full storage performance and durability guarantees.

How much Kubernetes persistent volume storage does each Elasticsearch node need?

A practical sizing rule is to provision 3× the raw data volume for hot-tier nodes: 1× for primary shards, 1× for replica shards, and 1× for merge space and temporary segment files. For warm-tier, use 2× raw data. For cold-tier, use 1.5× raw data. A hot-tier cluster retaining 7 days of data at 500 GB/day of raw ingest needs approximately 10.5 TB of PV capacity.

What Kubernetes StorageClass settings work best for Elasticsearch?

Use volumeBindingMode: WaitForFirstConsumer to ensure volumes are provisioned in the same availability zone as the pod. Use reclaimPolicy: Retain to prevent accidental data loss on PVC deletion. For hot-tier nodes, choose a StorageClass backed by NVMe block storage with at least 10,000 IOPS per volume and allowVolumeExpansion: true so you can grow volumes as data retention requirements increase.

How does simplyblock compare to cloud block storage for Elasticsearch hot-tier nodes?

Cloud block storage (AWS gp3, Azure Premium SSD) typically caps at 16,000 IOPS per volume and charges extra for provisioned throughput beyond the base tier. simplyblock’s disaggregated NVMe/TCP pool provides NVMe-class IOPS without per-volume throughput caps, and the storage pool scales independently of compute, so you add capacity without adding Elasticsearch nodes. For clusters with multi-terabyte hot tiers and sustained high ingest rates, the total cost of ownership difference between provisioned cloud block storage and simplyblock is significant at scale.