G1GC tuning

Small heap, hard-coded region, and reduced initiating occupancy.

Another small heap test of 8MB regions, with -XX:G1RSetUpdatingPauseTimePercent=5, and -XX:InitiatingHeapOccupancyPercent=35.


Baseline


-Xms16g \
-Xmx16g \
-Xss256k \
-XX:+UseG1GC \
-XX:G1RSetUpdatingPauseTimePercent=5 \
-XX:MaxGCPauseMillis=300 \
-XX:+PrintGCDetails \
-XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC \
-XX:+PrintTenuringDistribution \
-XX:+PrintGCApplicationStoppedTime \
-XX:+PrintPromotionFailure \
-XX:PrintFLSStatistics=1 \
-Xloggc:/var/log/cassandra/gc.log \
-XX:+UseGCLogFileRotation \
-XX:NumberOfGCLogFiles=10 \
-XX:GCLogFileSize=10M \
-XX:+PrintAdaptiveSizePolicy

8GB heap, G1HeapRegionSize of 8MB, RSetUpdatingPauseTimePecent of 5, and InitiatingHeapOccupancyPercent of 35. 

-Xms8g \
-Xmx8g \
-Xss256k \
-XX:+UseG1GC \
-XX:G1RSetUpdatingPauseTimePercent=5 \
-XX:G1HeapRegionSize=8M \
-XX:InitiatingHeapOccupancyPercent=35 \
-XX:MaxGCPauseMillis=300 \
-XX:+PrintGCDetails \
-XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC \
-XX:+PrintTenuringDistribution \
-XX:+PrintGCApplicationStoppedTime \
-XX:+PrintPromotionFailure \
-XX:PrintFLSStatistics=1 \
-Xloggc:/var/log/cassandra/gc.log \
-XX:+UseGCLogFileRotation \
-XX:NumberOfGCLogFiles=10 \
-XX:GCLogFileSize=10M \
-XX:+PrintAdaptiveSizePolicy
Cassandra read rate
Cassandra write rate
Cassandra read latency (75p)
Cassandra read latency (99p)
Cassandra write latency (75p)
Cassandra write latency (99p)
Cassandra connection timeouts/sec
Collection time (accumulated, 5 minute moving average)
Collections

Observations

There were two allocation failures that triggered early concurrent cycles and resulted in long (young) evacuations of 800ms and 3.36s (selected out of the scatter plot above) respectively.

Decreasing the occupancy threshold for initiaing a concurrent cycle seems to increase collection (compare these results to the same configuration with a threshold of 45%, and did not eliminate the to-space exhaustion.

A view of average collection time relative to the other nodes.

A view of heap usage relative to the other nodes