G1GC tuning

Reduced heap (8GB) with hard-coded (8M) region size, and an RSet-updating-time of 5 percent.

Another test of 8MB regions, this time with -XX:G1RSetUpdatingPauseTimePercent=5.


Baseline

-Xms16g \
-Xmx16g \
-Xss256k \
-XX:+UseG1GC \
-XX:G1RSetUpdatingPauseTimePercent=5 \
-XX:MaxGCPauseMillis=300 \
-XX:+PrintGCDetails \
-XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC \
-XX:+PrintTenuringDistribution \
-XX:+PrintGCApplicationStoppedTime \
-XX:+PrintPromotionFailure \
-XX:PrintFLSStatistics=1 \
-Xloggc:/var/log/cassandra/gc.log \
-XX:+UseGCLogFileRotation \
-XX:NumberOfGCLogFiles=10 \
-XX:GCLogFileSize=10M \
-XX:+PrintAdaptiveSizePolicy

8GB heap, G1HeapRegionSize of 8MB, RSetUpdatingPauseTimePecent of 5

-Xms8g \
-Xmx8g \
-Xss256k \
-XX:+UseG1GC \
-XX:G1RSetUpdatingPauseTimePercent=5 \
-XX:G1HeapRegionSize=8M \
-XX:MaxGCPauseMillis=300 \
-XX:+PrintGCDetails \
-XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC \
-XX:+PrintTenuringDistribution \
-XX:+PrintGCApplicationStoppedTime \
-XX:+PrintPromotionFailure \
-XX:PrintFLSStatistics=1 \
-Xloggc:/var/log/cassandra/gc.log \
-XX:+UseGCLogFileRotation \
-XX:NumberOfGCLogFiles=10 \
-XX:GCLogFileSize=10M \
-XX:+PrintAdaptiveSizePolicy
Cassandra read rate
Cassandra write rate
Cassandra read latency (75p)
Cassandra read latency (99p)
Cassandra write latency (75p)
Cassandra write latency (99p)
Cassandra connection timeouts/sec
Collection time (accumulated, 5 minute moving average)
Collections

Observations

Only one allocation failure here, resulting in a young collection being extended to ~1.9s (selected out of scatter plot above).

17 humongous allocations resulted in premature starts to concurrent cycles (a significant improvement over observations at 4MB region size).