Rob Pankow 
The Milvus storage guide covers the retrieval-time storage pattern for vector search, and the etcd storage guide covers the control-plane database underneath Kubernetes itself. ClickHouse sits in a third, increasingly common spot in the same stack: the columnar analytics and observability backend that ingests logs, metrics, and event data at high volume and answers aggregate queries over it in milliseconds. That ingest-and-merge pattern has its own storage requirements, and a generic Kubernetes StorageClass does not meet them by default.
ClickHouse’s MergeTree family of table engines never updates a row in place. Every insert lands as a new, immutable data part on disk. A background merge process continuously combines smaller parts into larger ones, rewriting data that was already written minutes or hours earlier so that read queries touch fewer, larger files. That background merge activity is not an edge case: it is the mechanism that keeps ClickHouse fast, and it runs constantly on any table taking a steady stream of inserts.
Why ClickHouse’s MergeTree Engine Breaks Generic Kubernetes Storage
A MergeTree table’s I/O has two distinct phases, and they compete for the same volume on most Kubernetes deployments:
Part writes happen on every insert batch. ClickHouse’s own documentation recommends batching inserts (thousands of rows per batch, not one row at a time) precisely because each insert creates a new part, and too many small parts overwhelm the merge scheduler. Even batched, a busy ingestion pipeline is writing new parts continuously.
Background merges read multiple existing parts, combine them, and write a new, larger part, then delete the originals. This is large sequential I/O, but it happens on the same disk that is simultaneously serving part writes and read queries. On a generic Kubernetes persistent volume sized for steady-state capacity rather than burst throughput, a merge storm competing with pod-local write traffic on the same node is a common cause of query latency spikes that have nothing to do with query complexity.
This is the same class of problem covered in the log-structured merge tree pattern used by RocksDB and other LSM-based engines: write amplification from compaction is a structural cost of the design, not a misconfiguration, and the storage layer underneath has to absorb it without stealing I/O from the read path.
ClickHouse Storage Tiers: Matching I/O Patterns to Backends
The table below maps ClickHouse’s storage tiers to the backend each one actually needs, using a representative high-ingest observability workload as the reference point.
| ClickHouse storage tier | I/O pattern | Latency sensitivity | Best-fit backend |
|---|---|---|---|
| Active part writes | Small-to-medium sequential writes, continuous | High, gates insert acknowledgment | Disaggregated NVMe/TCP or NVMe/RoCE block storage |
| Background merges | Large sequential read + write, bursty | Medium, but contends with writes and reads if slow | Same NVMe/TCP tier, sized for burst throughput |
| Hot query working set | Random reads across recent parts | High, gates query response time | Same tier, ideally with a page cache large enough to avoid most disk reads |
| Cold historical parts (tiered storage) | Large sequential reads, infrequent | Low | S3-compatible object storage, via ClickHouse’s native tiered-storage policies |
Table 1: ClickHouse storage tier characteristics and the backend each tier fits best.
ClickHouse’s own tiered-storage feature already assumes this split: it moves cold parts to object storage on a policy while keeping hot parts on local or block storage. The gap most teams hit on Kubernetes is not the cold tier, which ClickHouse handles natively. It is provisioning the hot tier, the one absorbing continuous part writes and merge bursts, on storage that is sized and provisioned like ordinary capacity rather than like a throughput-bound workload.
A slow merge tier turns into slow inserts and slow queries at the same time. Talk to a storage architect about sizing an NVMe/TCP tier for ClickHouse’s active parts and merge bursts, separate from your cold object-storage tier. Talk to a storage architect
How Disaggregated NVMe/TCP Absorbs Merge-Driven I/O Bursts
Running ClickHouse’s data nodes with persistent volumes backed by disaggregated NVMe/TCP (or NVMe/RoCE on RDMA-capable fabrics) addresses the two problems a single generic volume leaves open:
Merge bursts do not steal I/O from the query path. Because the hot-tier volume sits on a shared NVMe fabric rather than a single node’s local disk, the sustained throughput available to background merges is not capped by whatever spare bandwidth exists on one node alongside every other pod scheduled there. A merge storm on one ClickHouse shard does not degrade query latency on a neighboring shard sharing the same physical node.
Node failures become volume reattachments, not full replica rebuilds. ClickHouse typically relies on ReplicatedMergeTree and a coordination layer (ClickHouse Keeper or ZooKeeper) to keep replicas consistent; a lost replica normally means streaming a full copy of that shard’s data from a healthy replica. When the underlying volume lives on the storage fabric instead of node-local disk, a rescheduled pod reattaches the same PVC and resumes from where it left off, without the full-replica data transfer.
ClickHouse deployments also inherit the same snapshot-based backup pattern already covered for CSI snapshot architecture: a VolumeSnapshot of the active-parts volume gives a point-in-time, storage-side capture of a table’s current data parts, restorable in seconds, as a complement to ClickHouse’s own BACKUP command, which typically streams data through an object-storage destination and takes longer as table size grows. For teams running ClickHouse across multiple environments, a VolumeSnapshotClass per cluster tier (frequent snapshots for actively-ingesting production tables, longer-retention snapshots for stable analytical tables) mirrors the pattern already in production for the Milvus and CockroachDB deployments covered elsewhere in this series.
Simplyblock’s own ClickBench benchmark results already show the throughput gain this hot tier delivers under ClickHouse’s own industry-standard benchmark suite. Simplyblock licenses per usable storage TB per year, with no per-core fee, so a cluster that scales data nodes horizontally to absorb higher ingest volume does not pay more for the storage tier unless the actual volume of hot-tier data grows. The architecture works the same way whether ClickHouse runs hyperconverged, with storage and compute on the same nodes, or disaggregated, with a dedicated storage fabric behind the data nodes, on vSphere, bare-metal Kubernetes, or OpenShift.
Questions and Answers
Why does ClickHouse need a dedicated hot storage tier instead of one generic volume for everything? ClickHouse’s MergeTree engine writes new immutable parts on every insert and continuously merges them in the background, generating sustained sequential write and read traffic on top of whatever read traffic query workloads add. A single generic Kubernetes persistent volume sized for capacity rather than throughput lets merge activity compete with query reads for the same disk, which shows up as query latency spikes unrelated to query complexity.
What happens when a ClickHouse replica is lost on generic Kubernetes storage? ClickHouse’s ReplicatedMergeTree engine normally recovers a lost replica by streaming a full copy of that shard’s data from a healthy replica, which can take a long time for large tables. A persistent volume claim backed by simplyblock’s NVMe/TCP storage reattaches to a rescheduled pod in seconds instead, avoiding the full-replica data transfer in most node-failure scenarios.
Can ClickHouse’s built-in tiered storage replace a dedicated NVMe/TCP tier? ClickHouse’s native tiered-storage policies correctly move cold, infrequently-queried parts to S3-compatible object storage, and that part of the architecture does not need to change. The gap is the hot tier: active part writes, merge bursts, and recently-queried parts still need low-latency block storage, which tiered storage assumes exists but does not provide on its own.
How does simplyblock fit into a ClickHouse deployment on Kubernetes? Simplyblock provides NVMe/TCP (or NVMe/RoCE) block storage as persistent volumes for ClickHouse’s data nodes, sized for the continuous part-write and merge-burst I/O that generic Kubernetes storage classes are not built to absorb. Simplyblock’s CSI driver adds fast, storage-side snapshots for table-level backup and rollback, and the same architecture supports both hyperconverged and disaggregated deployment models.
How should I size storage for ClickHouse on Kubernetes? Size the NVMe/TCP hot tier for the working set of actively-written parts plus enough burst throughput headroom for background merges at peak ingest rate, and let ClickHouse’s own tiered-storage policy move cold, historical parts to S3-compatible object storage. Simplyblock’s pricing is per usable TB per year with no per-core fee, so scaling the data-node fleet horizontally for higher ingest volume is a compute decision, not an automatic storage cost increase.
