AnsweredAssumed Answered

Cassandra Service Rolling

Question asked by Hedde van der Hoeven on Nov 6, 2017
Latest reply on Nov 6, 2017 by Hedde van der Hoeven

Dear community,

 

I found our HCI cluster in a bit of a state this morning and after restarting HCI on all 4 instances, I found that the Cassandra (database) service is rolling on a single instance.

 

I have raised a support case for this (HDS04614166) and uploaded the logs to this case.

 

I did some troubleshooting and found this in the Cassandra logs on the watchdog service

 

 

ta/data/commitlog/CommitLog-6-1509980246831.log, /opt/service/data/data/commitlog/CommitLog-6-1509980266821.log, /opt/service/data/data/commitlog/CommitLog-6-1509980286936.log, /opt/service/data/data/commitlog/CommitLog-6-1509980306938.log, /opt/service/data/data/commitlog/CommitLog-6-1509980326989.log, /opt/service/data/data/commitlog/CommitLog-6-1509980346906.log, /opt/service/data/data/commitlog/CommitLog-6-1509980366063.log, /opt/service/data/data/commitlog/CommitLog-6-1509980385082.log, /opt/service/data/data/commitlog/CommitLog-6-1509980404374.log, /opt/service/data/data/commitlog/CommitLog-6-1509980423964.log, /opt/service/data/data/commitlog/CommitLog-6-1509980445377.log, /opt/service/data/data/commitlog/CommitLog-6-1509980465282.log, /opt/service/data/data/commitlog/CommitLog-6-1509980484494.log, /opt/service/data/data/commitlog/CommitLog-6-1509980503814.log, /opt/service/data/data/commitlog/CommitLog-6-1509980528362.log, /opt/service/data/data/commitlog/CommitLog-6-1509980547628.log, /opt/service/data/data/commitlog/CommitLog-6-1509980566666.log, /opt/service/data/data/commitlog/CommitLog-6-1509980586131.log, /opt/service/data/data/commitlog/CommitLog-6-1509980602994.log, /opt/service/data/data/commitlog/CommitLog-6-1509980623198.log

ERROR [main] 2017-11-06 15:04:04,356 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization.

org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 28040501 in CommitLog-6-1509393715114.log

        at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:681) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:545) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:445) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:145) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:293) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) [apache-cassandra-3.0.9.jar:3.0.9]

        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) [apache-cassandra-3.0.9.jar:3.0.9]

 

Corrupted replay log maybe?

 

-Hedde-

Outcomes