Ceph

Ceph is an open-source, distributed storage system that provides block storage (RBD), file storage (CephFS), and object storage (RGW/S3-compatible) from a single unified backend called RADOS — the Reliable Autonomic Distributed Object Store. Originally developed at the University of California and now maintained by Red Hat and the community, Ceph runs on commodity hardware and scales to exabytes. Its self-healing and self-managing architecture eliminates single points of failure by distributing data across all nodes using an algorithm called CRUSH.

Ceph’s strength is breadth: one cluster can serve block volumes to virtual machines, filesystem mounts to applications, and S3-compatible object storage to data pipelines simultaneously. Its challenge is operational complexity — deploying, tuning, and operating Ceph requires significant expertise, and the architecture introduces overhead that can affect latency-sensitive workloads.

How Ceph Works

Every write to a Ceph cluster is broken into objects and distributed across the cluster using the CRUSH algorithm. CRUSH computes placement deterministically — without a central lookup table — which means placement decisions scale with the cluster rather than creating a metadata bottleneck.

The core daemons:

• OSD (Object Storage Daemon): Runs on each storage node, handles actual data reads/writes, and manages replication. Each OSD typically owns one disk.
• MON (Monitor): Maintains the cluster map (which OSDs exist, what their state is, how data is placed). A quorum of at least three MONs is required for cluster operation.
• MDS (Metadata Server): Manages filesystem metadata for CephFS. Not needed for block or object workloads.
• RGW (RADOS Gateway): Provides an S3 and Swift-compatible object storage API on top of RADOS.
• MGR (Manager): Collects cluster metrics, provides a dashboard, and runs orchestration modules.

When a client writes to an RBD block volume or CephFS mount, the Ceph client library (librados) computes the target OSD set using the CRUSH map and writes directly to those OSDs. Replication (typically 3×) or erasure coding is applied at the RADOS layer.

🚀 Evaluating Ceph for Kubernetes? simplyblock delivers NVMe-first block storage with a fraction of the operational overhead — CSI-native, HCI or disaggregated, with per-volume QoS and sub-millisecond latency. 👉 See Why Teams Choose simplyblock over Ceph →

Key Features of Ceph

• Unified storage: Single cluster serves block (RBD), file (CephFS), and object (RGW) workloads.
• Scalability: Scales horizontally to thousands of nodes and exabytes of data.
• Self-healing: Detects OSD failures and automatically rebalances and re-replicates affected data.
• CRUSH placement: Deterministic, topology-aware data placement without central metadata lookup.
• Erasure coding: Configurable EC profiles reduce storage overhead compared to 3× replication, at the cost of write latency.
• Snapshot support: RBD snapshots and CephFS snapshots for point-in-time recovery.
• Thin provisioning: RBD volumes allocate space on write, not at creation.

Ceph in Kubernetes: Rook

Rook is the Kubernetes operator for Ceph. It automates deployment and management of a Ceph cluster inside Kubernetes, and exposes CSI drivers for RBD (block) and CephFS (filesystem) volumes. This allows PVCs to be backed by Ceph storage with no separate Ceph administration plane.

In practice, Rook-Ceph in Kubernetes means the cluster is responsible for its own storage infrastructure — OSDs run as pods, the MON quorum runs as pods, and storage capacity is tied to the nodes where Ceph OSDs are scheduled. This is a hyper-converged architecture: compute and storage share the same nodes.

The operational implications:

• Storage I/O competes with application workloads on the same nodes.
• Adding storage capacity means adding or configuring nodes in the Kubernetes cluster.
• The Ceph control plane (MONs, MGR) consumes cluster resources continuously.
• Tuning Ceph for performance (BlueStore, RocksDB, OSD memory targets) requires specialized expertise.

Ceph vs. Other Distributed Storage Solutions

Challenges of Using Ceph

Ceph’s capability comes with well-documented operational challenges:

• Deployment complexity: A production Ceph cluster requires careful planning — OSD placement, MON quorum, network separation for public and cluster traffic, BlueStore tuning.
• Resource consumption: MONs, MGR, and OSDs consume meaningful CPU and RAM continuously. On small clusters this overhead is proportionally high.
• Latency floor: RADOS’s object distribution and journaling add latency that is difficult to eliminate. For workloads requiring consistent sub-millisecond response times — databases, real-time analytics — Ceph’s architecture is a constraint.
• Slow recovery: When an OSD fails, Ceph rebalances affected placement groups across the cluster. On large or heavily loaded clusters, recovery can take hours and degrades performance during the process.
• Expertise requirement: Diagnosing performance issues, managing cluster topology changes, and tuning for specific workload mixes requires deep Ceph-specific knowledge.

simplyblock as a Ceph Alternative

simplyblock addresses the same Kubernetes block storage problem as Ceph/Rook-RBD, with a narrower scope and lower operational overhead. It focuses exclusively on block storage, using NVMe/TCP as the transport rather than RADOS’s object distribution layer.

simplyblock can be deployed in either hyper-converged mode (storage running alongside application workloads on the same nodes, like Rook-Ceph) or disaggregated mode (dedicated storage nodes, storage and compute scale independently). This flexibility means it fits the same initial deployment topology as Ceph while leaving the option to separate the layers as the cluster grows.

Specific differences at production scale:

• Lower latency: NVMe/TCP kernel-path transport avoids the RADOS object layer overhead. For database and analytics workloads, this translates to more consistent tail latency.
• Multi-tenant QoS: Per-volume IOPS and bandwidth limits enforced at the storage controller. Ceph has no per-volume QoS equivalent at this granularity.
• Simpler day-2 operations: No MON quorum to manage, no OSD tuning, no BlueStore configuration. The operational surface is substantially smaller.
• Erasure coding: Configurable EC profiles provide fault tolerance with less storage overhead than 3× replication.

For teams needing CephFS or S3 object storage in addition to block volumes, Ceph remains the right tool. For teams whose primary need is Kubernetes block storage with strong performance and manageable operations, simplyblock is worth evaluating.

Related Terms

Longhorn ZFS Thin Provisioning NVMe over TCP

Questions and Answers

Why are companies replacing Ceph for Kubernetes block storage?

Ceph is reliable but operationally demanding. For Kubernetes block storage specifically, teams often find that the Rook-Ceph stack — MON quorum, OSD management, network separation, BlueStore tuning — is disproportionate to the problem they need to solve. Purpose-built Kubernetes block storage platforms deliver comparable resilience with a significantly smaller operational footprint.

How does Ceph compare to simplyblock for database workloads?

Ceph introduces RADOS-layer overhead that creates a latency floor unsuitable for latency-sensitive databases. simplyblock uses NVMe/TCP kernel-path transport, which eliminates much of that overhead and delivers more consistent sub-millisecond latency. For databases running in Kubernetes, the difference is measurable in query tail latency.

What are the downsides of Rook-Ceph in Kubernetes?

Running Ceph via Rook consumes significant cluster resources for MON, MGR, and OSD pods. Storage I/O competes with application workloads unless nodes are dedicated to storage. Cluster topology changes (adding nodes, replacing disks) require careful coordination through the Rook operator. Recovery from OSD failures can affect cluster performance for extended periods.

Is Ceph still useful in modern storage architectures?

Yes — particularly for workloads that need all three storage types (block, file, object) from a single platform, or for very large-scale deployments where Ceph’s CRUSH-based scaling is a genuine advantage. For Kubernetes environments focused on block storage performance, dedicated block-storage platforms are typically a better fit.

Does Ceph support NVMe performance levels?

Ceph can use NVMe devices as OSD backing, and BlueStore’s direct device access eliminates some filesystem overhead. However, RADOS’s network and journaling path still adds latency that prevents Ceph from fully exploiting NVMe’s capabilities. NVMe/TCP-based platforms like simplyblock are designed from the ground up around NVMe’s performance characteristics.

Can simplyblock and Ceph coexist in the same Kubernetes cluster?

Yes — they can serve different workloads from separate StorageClasses. A common pattern during migration is running both storage platforms simultaneously, with new PVCs provisioned on simplyblock and existing Ceph-backed volumes retained until workloads are migrated. CSI makes this straightforward since both platforms expose standard Kubernetes PVC interfaces.

What is the CRUSH algorithm in Ceph?

CRUSH (Controlled Replication Under Scalable Hashing) is Ceph’s placement algorithm. Given an object name and a cluster map, CRUSH deterministically computes which OSDs should store that object — without a central lookup table. This enables placement decisions to scale with cluster size and supports topology-aware replication (e.g., place replicas in different racks or availability zones).