Performance tuning in Cassandra

Cassandra Performance Tunning:

Read minimization:

+++++++++++++++++++

Of course, we can not afford full SStable scan on each reads. Several mechanisms are implemented by default in Cassandra to ease read.

+ The first one is about Bloom filters.

Bloom filters is a probabilistic data structure to check for the existence of a key. It allows Cassandra to almost all of the time avoid to read on the disk for keys that do not exist.

+ Key Cache storing on disk location for a specific key.

Thus we can save much time when reading it.

+ Cassandra has a Row Cache,

storing in memory specific values for rows. It manages to keep ‘hot’, frequently access values in it. It allows finer grained caching than file system cache. Row cache can avoid disk reads.

+ compaction! SSTables are immutable.

However update and delete heavy workload will generate large SStables. Sometime we will need to remove entries and associated tombstone. Sometimes we want to summarize an entry and it’s updates into one entry. And we also want to have clustering partitions located on the same SSTable. So Cassandra will on a regular basis read and re-write the SSTables to make our reads easier, and save disk space.

+ Metadata Analysis:

describe table;

You can there check:

— compression

— compaction

— Bloom filter settings

— Cache settings

To see Cassandra level metrics (global caches results, load, etc…):

+ nodetool info

To see detailed statistics about your tables, including:

— Read/Write request counts

— Read/Write latencies

— Space on disk

— SSTable count

— Bloom filter statistics

+ nodetool cfstats

You can also monitor detailed percentile latencies, per table, using:

+ nodetool cfhistograms apache_james messageidtable

Adding row cache

You can enable row caching. It can avoid significant READ load on your disks.

War story: You need to reboot the node when enabling row-cache though row_cache_size_in_mb cassandra.yaml configuration file. nodetool will not be enough.

Once this (annoying) configuration parameter is enabled, you can use CQL per table to enable row cache:

use apache_james ;

ALTER TABLE modseq

WITH caching = {‘keys’: ‘ALL’, ‘rows_per_partition’: ‘10’} ;

Updating the compaction strategy of your tables can be done without downtime, at the cost of running compactions. Warning: this might consume IO and memories, and thus decrease performances when the compaction is running.

You need to modify the CQL table declaration to change the compaction strategy:

use apache_james ;

ALTER TABLE modseq

WITH compaction = { ‘class’ : ‘LeveledCompactionStrategy’ };

For the changes to take effect, you need to compact the SSTables. To force this, you need to use nodetool:

+ nodetool compact keyspace table

For the following compaction on large tables, you can use:

+ nodetool compactionstats

The rule of thumb for compaction time estimate is, with our hardware (16GB, HDD), approximatively one hour per GB stored on the table.

JVM tunning for cassandra:

https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html

Methodology for Perf tuning:

• Methodology
• Observability
• Practice

OODA Loop

Try to identify if there is a saturation at any level.Especially at the device IO level.What  it means the device is having more request probably queued , then it can served.

1. IOSTAT 
2. Error : tcp retransmit

Anti-Patterns:

1. Average latency watch 

Some useful commands to find internal working of cassandra:

• chunk_length_in_kb (default: 16KiB): specifies the number of kilobytes of data per compression chunk. The main tradeoff here is that larger chunk sizes give compression algorithms more context and improve their ratio, but require reads to deserialize and read more off disk. This means there is a configurable tradeoff between read and disk space here.

• Advanced Use

• Advanced users can provide their own compression class by implementing the interface at org.apache.cassandra.io.compress.ICompressor.