Database / ScyllaDB Interview questions

ScyllaDB is a distributed, wide-column NoSQL database built in C++ as a drop-in replacement for Apache Cassandra, designed to squeeze far more throughput out of the same hardware. It re-implements Cassandra's data model and wire protocol from scratch on top of the Seastar framework, avoiding the JVM entirely.

Because it's written in C++ with a shard-per-core architecture, ScyllaDB sidesteps garbage collection pauses and can drive a single machine's CPU cores far closer to their limit than a JVM-based database can. It supports the CQL query language, is wire-compatible with existing Cassandra drivers, and also ships Alternator, a DynamoDB-compatible API, so teams can migrate from either Cassandra or DynamoDB without rewriting application code. It's commonly used for workloads needing high write throughput and low tail latency at scale, such as IoT ingestion, ad tech, and real-time analytics.

ScyllaDB's design goals center on extracting maximum throughput per node while keeping the operational model familiar to Cassandra users.

• Shard-per-core architecture - each CPU core owns its own data and connections, avoiding lock contention.
• No JVM / no GC pauses - written in C++ on the Seastar async framework.
• CQL and Cassandra wire compatibility - existing Cassandra drivers and tools work with minimal changes.
• Alternator - a DynamoDB-compatible API for easy migration off DynamoDB.
• Tunable consistency - per-query consistency levels like ONE, QUORUM, ALL.
• Built-in Change Data Capture (CDC) and workload prioritization for mixed OLTP/analytics workloads.

Together these let ScyllaDB clusters handle the same workload as a larger Cassandra cluster with meaningfully fewer nodes.

Shard-per-core means each CPU core on a node runs an independent, single-threaded execution engine that owns a dedicated slice of RAM, a dedicated set of data (by token range), and its own network connections. Instead of one process sharing memory across threads with locks, ScyllaDB partitions everything up front so cores rarely need to coordinate.

Node with 8 cores
 -> 8 independent shards
 -> each shard owns: its own memtable, its own CPU, its own connections
 -> cross-shard work happens via explicit message passing, not shared memory locks

This avoids the lock contention and cache-line bouncing that plague traditional multi-threaded databases as core counts grow. A shard-aware driver takes this further by routing each request directly to the specific shard that owns the relevant data, skipping an extra network hop inside the node entirely.

A partition key is the portion of a table's primary key that determines which node(s) store a given row, by hashing the key value into a token that maps to a range owned by a specific replica set. Every row with the same partition key lands in the same partition, physically together on disk.

CREATE TABLE sensor_readings (
  sensor_id text,
  reading_time timestamp,
  value double,
  PRIMARY KEY (sensor_id, reading_time)
);

Here sensor_id is the partition key: all readings for one sensor are stored together and can be read efficiently as a range. Choosing a partition key with high cardinality and even access patterns is critical, since a key with too few distinct values, or one that concentrates traffic, creates an oversized or overloaded partition that a single set of replicas must absorb.

A clustering key is the part of the primary key that comes after the partition key and determines how rows within the same partition are sorted on disk. Where the partition key decides which node holds the data, the clustering key decides the physical order of rows inside that partition.

PRIMARY KEY (sensor_id, reading_time)
-- sensor_id = partition key
-- reading_time = clustering key (rows sorted by time within each sensor's partition)

Because rows are pre-sorted by clustering key, range queries like "give me readings for sensor X between two timestamps" are efficient sequential disk reads rather than random lookups. Multiple clustering columns can be declared to create a compound sort order, and the sort direction of each can be set independently with CLUSTERING ORDER BY.

ScyllaDB, following CQL, supports a broad set of scalar and collection types for schema design:

• text / varchar, ascii - string data.
• int, bigint, smallint, tinyint, varint - integer types of varying width.
• float, double, decimal - numeric types, with decimal for exact precision.
• boolean - true/false.
• uuid, timeuuid - unique identifiers, with timeuuid encoding a sortable timestamp.
• timestamp, date, time - temporal types.
• blob - raw binary data.
• list, set, map - collection types.
• counter - a specialized type for distributed increment/decrement.
• User Defined Types (UDTs) - custom structured types composed of other types.

Counters and collections have special replication behavior, so they're used deliberately rather than as general-purpose containers for arbitrary data.

Seastar is the open-source C++ asynchronous programming framework, originally built for ScyllaDB, that underlies its shard-per-core, high-performance design. Seastar provides a future-promise concurrency model, its own userspace network stack and memory allocator, and asynchronous I/O so applications built on it can drive hardware close to its physical limits.

Rather than relying on OS threads and blocking system calls, Seastar programs run one thread per core, each managing its own event loop and non-blocking I/O queue. This eliminates cross-core synchronization overhead and lets ScyllaDB scale near-linearly with core count on a single machine, which is the foundation the shard-per-core architecture is built on top of. Seastar is also used by other performance-sensitive projects beyond ScyllaDB itself.

ScyllaDB, like Cassandra, offers tunable consistency, letting each read or write specify how many replicas must acknowledge before the operation succeeds.

Choosing a consistency level is a trade-off between latency/availability and strictness: ONE is fastest but can return stale data, while ALL guarantees freshness but fails if even one replica is unreachable. LOCAL_QUORUM is a common default for multi-datacenter clusters since it gives strong consistency within a region without paying cross-datacenter round-trip latency.

Compaction merges SSTables together to remove obsolete data and reduce read amplification; ScyllaDB supports several strategies suited to different workloads.

• Size-Tiered Compaction Strategy (STCS) - merges similarly-sized SSTables together; good default for write-heavy workloads.
• Leveled Compaction Strategy (LCS) - organizes SSTables into levels with bounded size; reduces read amplification and space overhead at the cost of more I/O during compaction, good for read-heavy workloads.
• Time-Window Compaction Strategy (TWCS) - groups SSTables by time window; ideal for time-series data with TTL-based expiration.
• Incremental Compaction Strategy (ICS) - a ScyllaDB-specific strategy that avoids the temporary doubling of disk space STCS needs during large compactions.

Picking the wrong strategy for the access pattern is a common source of either excessive read latency or excessive disk churn, so the strategy is usually revisited if compaction backlog or read amplification metrics look unhealthy.

Tables are created with standard CQL CREATE TABLE statements through cqlsh, a driver, or ScyllaDB's REST/CQL tooling, specifying the partition key and any clustering columns as part of the primary key.

CREATE TABLE ecommerce.orders (
  customer_id uuid,
  order_id timeuuid,
  status text,
  total decimal,
  PRIMARY KEY (customer_id, order_id)
) WITH CLUSTERING ORDER BY (order_id DESC);

Table options can also set the compaction strategy, default TTL, and compression algorithm at creation time. Because the primary key layout directly determines physical data distribution and sort order, table design in ScyllaDB is driven by the queries the application needs to run, often called query-first or query-driven schema design, rather than starting from a normalized relational model.

A materialized view is a server-managed table that automatically re-derives its rows from a base table using a different primary key layout, letting the same data be queried efficiently by a column that isn't part of the base table's partition key.

CREATE MATERIALIZED VIEW orders_by_status AS
  SELECT * FROM orders
  WHERE status IS NOT NULL AND customer_id IS NOT NULL AND order_id IS NOT NULL
  PRIMARY KEY (status, customer_id, order_id);

ScyllaDB keeps the view in sync with the base table automatically whenever the base table changes, so the application doesn't have to maintain a second table manually. The trade-off is write amplification, since every write to the base table triggers a corresponding write to each materialized view, and views are eventually consistent with the base table rather than updated in the same atomic write, which matters for applications that need strict read-after-write guarantees on the view.

The commit log is an append-only, on-disk log that ScyllaDB writes to before acknowledging any write, purely for crash recovery. Every mutation is appended sequentially to the commit log at the same time it's applied to the in-memory memtable, so a sequential disk write, which is fast, stands in for a full random-access flush on every write.

If a node crashes before its memtables are flushed to SSTables, the commit log is replayed on restart to reconstruct any data that hadn't yet been persisted as an SSTable. Once a memtable is flushed to disk as an immutable SSTable, the corresponding commit log segments are no longer needed for recovery and get recycled. This design is what lets ScyllaDB acknowledge writes quickly while still guaranteeing durability against a node crash.

Alternator is ScyllaDB's implementation of the DynamoDB HTTP API, letting applications written against AWS DynamoDB's SDKs point at a ScyllaDB cluster instead, with no application code changes beyond the endpoint URL.

aws dynamodb list-tables --endpoint-url http://scylla-node:8000

It supports DynamoDB's core operations, including GetItem, PutItem, Query, Scan, and Global/Local Secondary Indexes, translating them internally onto ScyllaDB's storage engine. Alternator exists mainly to give teams an exit path from DynamoDB's proprietary pricing and vendor lock-in, letting them run the same workload on self-managed or ScyllaDB Cloud infrastructure, on-premises or across multiple clouds, without a rewrite.

Time-to-live (TTL) can be set per-write in seconds, either on an individual INSERT/UPDATE statement or as a table-wide default, after which the data is automatically marked for deletion.

INSERT INTO sessions (session_id, user_id, data)
VALUES (uuid(), 'user123', 'payload')
USING TTL 3600;  -- expires in 1 hour

CREATE TABLE cache (k text PRIMARY KEY, v text) WITH default_time_to_live = 86400;

Once a row or column's TTL elapses, it isn't deleted immediately; ScyllaDB writes a tombstone that marks it as expired, and the underlying data is physically removed later during compaction. TTL is a common building block for session data, caches, and rate-limiting counters, but heavy TTL usage benefits from the Time-Window Compaction Strategy, since TWCS is specifically designed to purge fully-expired SSTables efficiently.

ScyllaDB Manager is a centralized operations tool for automating and scheduling cluster-wide maintenance tasks that would otherwise need to be run manually node by node, such as repairs, backups, and rolling restarts.

• Automated repair scheduling - runs anti-entropy repairs on a rolling schedule across the cluster.
• Backup management - schedules and tracks backups to object storage (S3-compatible).
• Health checks and monitoring integration - feeds status into the ScyllaDB Monitoring Stack dashboards.
• Cluster-wide task orchestration - coordinates operations across many nodes without manual per-node scripting.

It's a separate component from the database itself, typically deployed once per organization to manage multiple ScyllaDB clusters, and is considered a standard part of running ScyllaDB in production rather than an optional add-on.

Cassandra runs on the JVM, which means its performance is subject to garbage collection pauses: periodically the JVM has to stop application threads to reclaim memory, and under heavy load or large heaps these "stop-the-world" pauses can spike into hundreds of milliseconds, directly hurting tail latency.

ScyllaDB was built in C++ specifically to eliminate this class of problem. Without a JVM, there's no garbage collector to pause application threads, and the Seastar framework it's built on manages memory explicitly per shard instead of relying on automatic collection. The trade-off is that C++ development is more demanding than JVM-based development, requiring careful manual memory management, but the payoff is materially more predictable p99/p999 latencies and higher achievable throughput per core, which is the core value proposition that differentiates ScyllaDB from Cassandra.

Per-node scalability in ScyllaDB comes from combining the shard-per-core architecture with a design that avoids shared, contended state as core counts grow. Each core runs its own shard with its own memtable, cache, and connections, so adding cores adds independent capacity rather than more contention on a single shared structure.

Cross-shard communication happens through explicit, asynchronous message passing rather than shared-memory locks, which keeps the cost of coordination bounded and predictable even as core count rises. This is why doubling a node's CPU cores in ScyllaDB tends to roughly double its achievable throughput, a property that's much harder to guarantee in a JVM-based database where garbage collection and lock contention scale poorly with thread count. At the cluster level, this per-node scalability compounds with horizontal scaling across nodes via consistent hashing, giving both dimensions of scale.

Because ScyllaDB re-implements Cassandra's data model and wire protocol, most existing CQL schemas, queries, and drivers work against either system with little to no change, which is why migrations are typically described as a drop-in replacement rather than a rewrite. The practical difference teams notice is throughput per node and tail latency consistency, particularly under high load or with large heaps, rather than a difference in the CQL data model itself.

A local secondary index is stored on the same node as the base data it indexes, so a query using it can only be efficiently satisfied by first knowing (or scanning across) the partition, making it most useful when the query already includes the partition key alongside the indexed column. A global secondary index (as used in Alternator/DynamoDB-style tables) is maintained as its own independently partitioned table, letting you query by the indexed column without knowing the base table's partition key at all, at the cost of extra write overhead to keep it in sync.

Use a local index when filtering within a partition you can already identify, since it avoids the overhead of a separate global structure. Reach for a global index when the access pattern genuinely needs to search across all partitions by an attribute unrelated to the partition key, accepting the additional write cost and (for materialized-view-style implementations) eventual consistency between the base table and the index.

A wide partition occurs when a single partition key accumulates an unusually large number of rows or a very large amount of data, often from a low-cardinality key or unbounded time-series growth without bucketing. Because a partition is the unit that a single set of replicas must serve, the node(s) owning that partition end up doing disproportionate work.

Symptoms include elevated read and compaction latency for that specific partition, larger memtable and SSTable footprints concentrated on fewer nodes, and in extreme cases, timeouts when scanning the partition or during repair, since a wide partition takes longer to stream and validate. Monitoring tools flag large partition warnings so they can be caught before becoming critical. The fix is schema-level: bucket the partition key by a natural time window or hash suffix so what was one unbounded partition becomes many bounded ones spread across the cluster.

ScyllaDB uses consistent hashing to map each partition key to a token, and that token determines a position on a logical ring of token ranges spread across the cluster's nodes. Each token range is owned by a set of replicas equal to the table's replication factor, with the specific placement strategy (e.g. NetworkTopologyStrategy) controlling how replicas are spread across datacenters and racks.

CREATE KEYSPACE ecommerce WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'dc1': 3,
  'dc2': 3
};

A write is sent to all replicas responsible for its token range in parallel, and the client's requested consistency level determines how many acknowledgments must return before the write is considered successful. This peer-to-peer replication model, with no single master, means any replica can accept a write or serve a read, which is central to ScyllaDB's high availability - there's no single point of failure the way a primary/replica relational system has.

CQL's BATCH statement groups multiple writes together, but unless every statement in the batch shares the same partition key, ScyllaDB has to coordinate the batch as a distributed operation across multiple partitions, which adds a lot of overhead compared to sending the same writes individually.

A large multi-partition batch forces a single coordinator node to buffer and manage writes destined for many different replica sets at once, increasing memory pressure and the risk of timeouts, and it doesn't actually make those writes atomic in the way a relational transaction would - it just batches network round trips. If all statements in a batch do share one partition key, the batch is efficient and reasonable, since it's genuinely a single-partition operation. The general guidance is to keep batches small, single-partition, and use them for genuine logical grouping rather than as a shortcut to bulk-load unrelated writes.

DynamoDB trades operational control for zero-ops convenience within AWS, which suits teams that want to avoid managing infrastructure entirely. ScyllaDB (via Alternator) suits teams that want DynamoDB-style access patterns but need multi-cloud or on-premises deployment, more predictable and often lower cost at scale, or deeper visibility into performance internals for tuning - at the cost of taking on more operational responsibility than a fully managed service.

When a write's coordinator can't reach one of the replicas responsible for storing it, perhaps because that node is temporarily down or unreachable, ScyllaDB can store a hint: a record of the missed write, kept on a live node (often the coordinator) until the target replica comes back.

Once the failed replica rejoins the cluster, hints stored for it are replayed, bringing it back in sync without needing a full repair for that data. Hints have a configurable time window (by default a few hours); if a node is down longer than that window, the hint is discarded and the node instead relies on read repair or a full anti-entropy repair to catch up. Hinted handoff lets ScyllaDB tolerate short, transient node failures without sacrificing write availability, since the write can still succeed at the consistency level requested as long as enough other replicas are reachable.

QUORUM requires a majority of replicas across all datacenters to respond, which means every read or write pays a cross-datacenter network round trip, even when the application only cares about consistency within its own region. LOCAL_QUORUM requires a majority only among replicas in the coordinator's local datacenter, avoiding that cross-region latency entirely.

Choose LOCAL_QUORUM for the common case of a multi-datacenter deployment where each region primarily serves its own local traffic and cross-datacenter replication exists mainly for disaster recovery or read locality, not for enforcing strict global ordering on every request. Reach for QUORUM only when an operation genuinely needs to reflect state across all datacenters immediately, which is rare and usually confined to specific coordination paths, since the latency cost of a full cross-datacenter quorum on every request is substantial at scale.

• Design partition keys to avoid hotspots - uneven key distribution caps throughput on a few nodes regardless of cluster size.
• Use a shard-aware driver so writes go directly to the owning shard, avoiding an internal network hop.
• Prefer single-partition batches over multi-partition ones, or skip batching for unrelated writes entirely.
• Pick an appropriate consistency level - LOCAL_QUORUM instead of QUORUM avoids unnecessary cross-datacenter latency for most workloads.
• Match compaction strategy to workload - STCS or ICS for write-heavy tables reduces compaction overhead competing with foreground writes.
• Use asynchronous/pipelined writes from the client instead of waiting for each write to complete before issuing the next.

As with reads, most sustained write bottlenecks trace back to schema design (a hot partition key) rather than something tunable purely at the client or cluster-configuration level, so it's worth checking per-partition and per-shard metrics before assuming the fix is purely operational.

Every write lands first in the memtable (alongside the commit log for durability), so recent data is served straight from memory, which is fast. When a memtable fills up, it's flushed to disk as a new, immutable SSTable, and a read for a given row may need to check the memtable plus one or more SSTables, merging results together, since a row's data can be spread across several SSTables written at different times. Compaction periodically merges multiple SSTables into fewer, larger ones to keep this per-read merge work bounded.

Because SSTables are immutable once written, ScyllaDB can't simply erase a row or column in place the way an update-in-place database would. Instead, a delete (explicit, or implicit via TTL expiration) is recorded as a tombstone, a special marker written like any other mutation, that tells later reads "ignore any older value you find for this row/column."

Tombstones let deletes be fully consistent with ScyllaDB's replication and compaction model: they replicate to other nodes exactly like a write would, and they get merged and eventually purged during compaction once the deleted data is safely older than gc_grace_seconds across all replicas. The trade-off is that a workload with heavy deletes (or short TTLs) accumulates many tombstones, and reads that must scan past a large number of tombstones to find live data suffer degraded latency, which is why tombstone-heavy access patterns get specific attention in schema and compaction strategy design.

A shard-aware driver understands not just which node owns a given partition key, via the same token-ring logic every driver uses, but which specific CPU shard on that node owns it, since ScyllaDB partitions data by core within a node as well as across nodes.

Normal (non-shard-aware) drivers connect to a node and let ScyllaDB internally forward the request to the correct shard, adding a small extra hop inside the node. A shard-aware driver instead opens multiple connections per node, one (or more) per shard, and computes the target shard client-side from the partition key's token before sending the request, so it connects directly to the connection already bound to the owning shard. This removes the internal cross-shard forwarding step entirely, which measurably reduces latency and CPU overhead at high request rates, making shard-aware drivers the recommended choice for latency-sensitive ScyllaDB workloads.

STCS is the simpler default and works well when write throughput matters most and occasional read amplification is tolerable. LCS trades additional background I/O (more frequent, smaller compactions) for a guarantee that a read only has to check a small, bounded number of SSTables, which keeps read latency more consistent as data grows - the reason it's often recommended for read-heavy tables where p99 latency matters more than raw compaction overhead.

Lightweight transactions (LWT), expressed with IF clauses like INSERT ... IF NOT EXISTS or UPDATE ... IF column = value, provide linearizable compare-and-swap semantics using a Paxos-based consensus protocol among replicas, rather than the normal fire-and-acknowledge write path.

UPDATE inventory SET quantity = quantity - 1
WHERE item_id = 'sku-42' IF quantity > 0;

Use LWT specifically when correctness genuinely depends on a conditional check being atomic across replicas, such as claiming a unique username, decrementing limited inventory without overselling, or implementing a distributed lock. Because Paxos-based consensus requires multiple round trips between replicas, LWT is meaningfully slower and more resource-intensive than a normal write, so it should be reserved for the specific operations that need this guarantee rather than used as a default for all writes.

Repair is ScyllaDB's anti-entropy process for reconciling data drift between replicas that can build up from missed writes, expired hints, or clock/network issues. It compares data across replicas, typically using Merkle trees, hash-tree structures that let two replicas efficiently find which ranges of data actually differ without transferring and comparing every row.

Each replica builds a Merkle tree over its data for a given token range; comparing trees quickly narrows down to the specific sub-ranges that are out of sync, and only those are streamed and reconciled, rather than re-transferring the entire dataset. Repair is typically scheduled and orchestrated cluster-wide through ScyllaDB Manager rather than triggered manually per node, since running it on a rolling, regular basis (well within gc_grace_seconds) is what guarantees that tombstones are consistent across all replicas before they're purged by compaction, preventing deleted data from silently reappearing.

ScyllaDB is architected so that any single query can be efficiently routed and answered by the specific node(s) owning the relevant partition, keeping latency predictable at scale. An arbitrary join across two large tables would require correlating data that could live on completely different, unrelated nodes, which breaks that guarantee and can force a scatter-gather operation across the entire cluster for a single query.

Instead, ScyllaDB pushes the "join" work to schema design time: applications denormalize data, duplicating or pre-joining information into a single table (or a materialized view) shaped around the exact query pattern needed, so that what would be a join in a relational database becomes a single-partition read here. This trades some storage and write-time complexity (keeping denormalized copies in sync) for read-time predictability, which is the core design philosophy behind query-first schema design in wide-column databases.

Vnodes were ScyllaDB's original approach, inherited from Cassandra's design, where each physical node owns many virtual token ranges to smooth out load distribution, but the granularity is still fixed once nodes join. Tablets are a newer, more dynamic model where each table's data is divided into independently-sized tablets that ScyllaDB can split, merge, and move much more responsively as load or cluster size changes, similar in spirit to how some distributed SQL systems handle splits, and they significantly speed up operations like adding a node or scaling a cluster.

1. Check for wide partitions - a large partition takes longer to scan and can dominate latency for that key.
2. Check tombstone counts in query tracing - excessive tombstones force reads to skip past dead data.
3. Review the consistency level - QUORUM or ALL reads pay more coordination latency than ONE or LOCAL_QUORUM.
4. Look at SSTable count per read - a high count (common with STCS under heavy churn) suggests switching to LCS or tuning compaction.
5. Check cache hit rates - frequent misses on the row/key cache push more reads to disk.
6. Confirm the driver is shard-aware - a non-shard-aware driver adds an internal hop on every request.

ScyllaDB's built-in tracing (TRACING ON in cqlsh, or the Monitoring Stack's per-query breakdown) is usually the fastest way to see exactly where time is going for a specific slow query, rather than guessing from cluster-wide metrics alone.

Seastar's concurrency model is built around futures (a placeholder for a value that will eventually be ready) and promises (the producer side that eventually fulfills that value), composed together instead of using blocking calls or OS-level thread synchronization.

Each core runs a single-threaded reactor loop that polls for completed I/O events and dispatches ready continuations, so nothing on that core ever blocks waiting for disk or network; instead, code chains .then() continuations onto a future, and the reactor invokes them once the underlying operation completes. Because everything on a given core runs cooperatively on one thread, there's no need for locks to protect that core's own data structures - the only synchronization required is for the deliberately explicit, asynchronous messages sent between cores. This is what allows ScyllaDB to scale near-linearly with core count without traditional multi-threaded contention.

A write in ScyllaDB moves from client to coordinator to replicas, with durability and consistency handled at distinct points along the way.

The client sends the write to any node, which acts as coordinator for that request and computes which replicas own the relevant token range. It forwards the write to all of them in parallel; each replica durably appends it to its commit log and applies it to its in-memory memtable before acknowledging. The coordinator waits only for the number of acknowledgments required by the requested consistency level (e.g. a quorum) before telling the client the write succeeded - it doesn't need every replica to respond immediately, and any replica it couldn't reach gets a hint stored for later replay. This is why writes in ScyllaDB are fast even at high consistency levels: the expensive part (durable commit log append) is spread in parallel across replicas rather than serialized through a single leader.

An SSTable's life begins in memory and ends when its data is either merged into a newer SSTable or fully expired and removed.

Once a memtable fills past its threshold, ScyllaDB flushes it to disk as a new, immutable SSTable, complete with its own bloom filter (to quickly rule out SSTables that definitely don't contain a key) and index for locating rows within the file. As more SSTables accumulate from ongoing writes, the configured compaction strategy periodically selects a set of them to merge: overlapping row versions are reconciled (keeping the latest write, respecting tombstones), and a new, consolidated SSTable is written while the old input SSTables are deleted once the merge completes successfully. Fully-expired SSTables, where every row has passed its TTL and gc_grace_seconds, can sometimes be dropped entirely without a full merge, which is one reason time-bucketed compaction strategies like TWCS are efficient for TTL-heavy data.

Older versions of ScyllaDB (and Cassandra) propagated schema changes via gossip, an eventually-consistent protocol, which could momentarily leave different nodes with slightly different views of the schema during a rollout, an acceptable risk for schema but not ideal. ScyllaDB has since adopted Raft, a leader-based consensus protocol, specifically for schema and cluster topology changes.

Under Raft, schema changes are proposed to a Raft group spanning the cluster's nodes; a leader replicates the change as a log entry, and it's only considered committed once a majority of the group has durably persisted it. Every node applies committed log entries in the same order, which guarantees that at any point every node either has the old schema or the new one - never an inconsistent in-between state - unlike gossip's best-effort convergence. The same Raft-based mechanism is used for topology changes (adding/removing nodes, tablet migrations), which is what makes those operations safer and faster to reason about than the older gossip-driven approach.

Compaction reads several existing SSTables, merges their contents row by row, and writes the result as new, consolidated SSTables, all while the affected data remains readable and writable through the process.

For each partition present in the input SSTables, compaction merges all the row fragments it finds, keeping the most recent value for each column based on write timestamps, and applying any tombstones so deleted data doesn't reappear. Data that's past both its TTL and gc_grace_seconds is dropped entirely rather than rewritten. Once the new SSTable (or SSTables, for LCS which produces multiple leveled outputs) is fully written and fsynced, the old input SSTables are deleted. Because compaction is CPU and I/O intensive, ScyllaDB throttles it via configurable bandwidth limits and prioritizes foreground read/write traffic, so compaction runs continuously in the background without starving the workload it's supporting.

• Use LOCAL_QUORUM (or ONE/LOCAL_ONE) for regular application traffic so requests don't pay cross-datacenter round trips.
• Set NetworkTopologyStrategy replication per-datacenter so each region has enough local replicas to satisfy local consistency without depending on a remote datacenter.
• Route application traffic to the nearest datacenter at the load balancer or driver level, rather than letting requests land anywhere and potentially hit a remote coordinator.
• Reserve cross-datacenter QUORUM reads/writes for the few operations that truly need global strong consistency, not as a default.
• Monitor inter-DC replication lag - if remote datacenters fall behind, LOCAL_QUORUM reads stay fast but the data available there becomes more stale.

The general principle mirrors other distributed databases: cross-region network latency is a physical constraint no amount of tuning removes, so the optimization goal is minimizing how often a request actually needs to cross that boundary rather than trying to make the crossing itself faster.

Neither is universally better; the choice trades off consistency strength against latency and availability, and the right answer depends on what the read or write is for.

ONE is the right choice for workloads where absolute lowest latency matters more than freshness, such as non-critical telemetry ingestion or best-effort caching, and where the application can tolerate occasionally reading slightly stale data. LOCAL_QUORUM is the better default for most application data, like user profiles, orders, or inventory, where the application logic depends on reads reflecting recent writes within the region, since combining LOCAL_QUORUM reads and writes guarantees that overlap. The practical guidance is to default to LOCAL_QUORUM for correctness-sensitive paths and reserve ONE for specific, well-understood cases where staleness is genuinely acceptable.

Without token awareness, a driver sends a request to an arbitrary node, which then has to act as a coordinator and forward the request to whichever node(s) actually own the relevant data, adding an extra network hop before the "real" work even starts.

// Token-aware driver (conceptual)
token = murmur3(partitionKey)
replicas = tokenRing.getReplicasFor(token)
connection = pickClosest(replicas)  // often combined with a local DC preference
sendRequest(connection, query)

A token-aware driver computes the same token-ring hash the cluster itself uses, based on the partition key in the query, and sends the request directly to one of the nodes that actually owns that data - skipping the coordinator forwarding hop entirely for the common case. Combined with a shard-aware driver (which goes one step further and picks the exact CPU shard), this removes essentially all unnecessary internal routing, cutting both latency and the coordinator load that would otherwise accumulate on whichever node happened to receive the request first.

In a large peer-to-peer cluster with no central coordinator for membership, every node needs a way to learn which other nodes exist, whether they're alive, and their basic state (load, schema version, token ownership) without a single point of failure or a central registry becoming a bottleneck. Gossip solves this by having each node periodically exchange state with a few random peers, and that state naturally propagates across the whole cluster within a few rounds, exponentially fast, similar to how a rumor spreads through a social network.

Because gossip is decentralized and doesn't rely on any single node staying up, cluster membership information keeps propagating correctly even while individual nodes fail or restart, which is essential for a system with no master node coordinating everything centrally. ScyllaDB still uses gossip for liveness/failure detection and general cluster state today, even though schema and topology changes themselves have moved to Raft for strong consistency - the two mechanisms serve different needs: gossip for scalable, resilient state dissemination, Raft for operations that must never be ambiguous.

1. Check pending compaction metrics in the Monitoring Stack - a steadily growing backlog means compaction can't keep pace with incoming writes.
2. Review the compaction strategy fit - STCS under heavy, uneven write patterns can lag; LCS or ICS may handle the shape of the workload better.
3. Check disk I/O saturation - compaction is I/O-heavy, and a disk that's already near its throughput limit from foreground traffic will starve compaction.
4. Look for oversized partitions - a few very large partitions can dominate compaction time disproportionately.
5. Review compaction throughput throttling settings - an overly conservative bandwidth cap can be safe for foreground latency but let backlog build up over time; it may need raising if the cluster has I/O headroom.
6. Check for repair-driven compaction spikes - large repairs stream a lot of new data that then needs compacting, so backlog can correlate with a recent repair window.

A persistent backlog usually means the table is fundamentally under-provisioned for its write rate rather than a one-time blip, so alongside these checks it's worth evaluating whether the table needs more nodes or a schema change (like reducing tombstone-heavy delete patterns) rather than only tuning compaction settings.

A read has to reconstruct the current value of a row from potentially several places at once, since data for one partition can be spread across the memtable and multiple SSTables.

Each replica first checks its memtable (freshest data) and row cache (hot data already assembled), then uses per-SSTable bloom filters to cheaply skip SSTables that provably don't contain the requested key, and consults the index of the remaining candidates to locate the row's position on disk. All the fragments found are merged, respecting write timestamps and tombstones, into a single up-to-date view of the row. If the read's consistency level requires multiple replicas to respond, the coordinator compares their answers and, if they disagree, can trigger a foreground read repair to reconcile the difference before returning the final result to the client - which is one way ScyllaDB heals minor replica drift opportunistically, on top of scheduled anti-entropy repair.

Because data is replicated across multiple nodes per the table's replication factor, a single node going down doesn't make its data unavailable - the remaining replicas for its token ranges continue serving reads and writes.

Other nodes' failure detectors (built on gossip-based heartbeat exchange) mark the node as down after missing expected heartbeats, and the cluster's topology view updates accordingly. Writes destined for the down node are handled via hinted handoff, stored on a live coordinator or replica and replayed once the node returns. Reads at consistency levels like QUORUM or LOCAL_QUORUM continue succeeding as long as enough of the remaining replicas can form the required majority; only if too many replicas for a given range are simultaneously down does an operation at that consistency level start failing. When the node comes back, it replays any hints waiting for it and, on a regular schedule, repair reconciles any data it missed while down that exceeded the hint window, bringing it fully back in sync with the rest of its replica set.

When CDC is enabled on a table, ScyllaDB automatically creates a companion log table alongside it, and every insert, update, or delete on the base table also writes a corresponding row into that log table describing the change, as part of the same write path.

The CDC log table is a regular ScyllaDB table under the hood, partitioned by time-based "streams" so that reading recent changes is an efficient, sequential operation rather than a full scan, and it carries its own TTL so change history doesn't grow unbounded. Consumers, whether custom CQL clients or the official Kafka Source Connector, read from this log table incrementally, tracking their own progress per stream so they can resume where they left off after a restart. Because the log write happens as part of the same mutation as the base table write, CDC records are captured with the same durability and ordering guarantees as the underlying data, avoiding the missed-event problems that separate, out-of-band change-tracking mechanisms can suffer from.

A lightweight transaction (LWT) needs every replica involved to agree on both the current value and the outcome of the conditional check before anything is applied, so it runs a Paxos round instead of the normal single-phase write path.

The coordinator first runs a Paxos "prepare" phase, proposing a ballot number to the replicas and asking them to promise not to accept any older proposal, while also learning the current value each replica holds. It then evaluates the IF condition against that value; if it holds, it runs the "propose" phase, asking replicas to accept the new value, and once a majority accepts, a final commit message applies it everywhere. This extra round-trip machinery, compared to a normal write's single pass, is exactly what guarantees the compare-and-swap is linearizable even if multiple clients race to update the same row simultaneously - only one will see its condition succeed.

• Bucket time-series data by a natural window (hour/day) appended to the partition key, so one logical entity's data spreads across many bounded partitions instead of one unbounded one.
• Add a synthetic bucket suffix (e.g. a hash or modulo of an ID) when a naturally low-cardinality key would otherwise concentrate rows in a single partition.
• Set a maximum expected partition size upfront during design and validate against realistic data volume projections, not just current test data size.
• Monitor large-partition warnings in the logs and Monitoring Stack proactively, since partitions often start small and only become a problem months into production.
• Reconsider clustering key design so queries can still efficiently retrieve data across buckets when needed, for example querying several time-buckets in parallel rather than one giant partition sequentially.

The underlying principle is the same one that drives most ScyllaDB schema decisions: because a partition is the unit of physical placement and per-request work, any single partition that grows unbounded relative to others will eventually dominate the resource usage of whichever node(s) hold it, regardless of how well-provisioned the rest of the cluster is.