text:
WIP: Raft WAL deadlock test

WAL deadlock recreation
Type of change

Bug fix
Test update

Description
Please look at #4199, and PR #4201 .

Log of the test when everything works:
GOROOT=/usr/local/go #gosetup
WORKING_DIRECTORY=/Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft #gosetup
/Users/emil/go/bin/ginkgo -v "--focus=Chain 3-node Raft cluster (when )?reconfiguring raft cluster reconfiguration wtf 2"
2023-06-05 11:07:58.656 IDT 0001 INFO [test] confirmSuspicion -> Suspecting our own eviction from the channel for 10m0s
2023-06-05 11:07:58.656 IDT 0002 PANI [test] confirmSuspicion -> Failed creating a block puller: oops
2023-06-05 11:07:58.657 IDT 0003 INFO [test] confirmSuspicion -> Suspecting our own eviction from the channel for 10m0s
2023-06-05 11:07:58.658 IDT 0004 INFO [test] confirmSuspicion -> Last config block was found to be block [9]
2023-06-05 11:07:58.658 IDT 0005 INFO [test] confirmSuspicion -> Cannot confirm our own eviction from the channel: bad block
2023-06-05 11:07:58.658 IDT 0006 INFO [test] confirmSuspicion -> Suspecting our own eviction from the channel for 10m0s
2023-06-05 11:07:58.659 IDT 0007 INFO [test] confirmSuspicion -> Last config block was found to be block [9]
2023-06-05 11:07:58.659 IDT 0008 INFO [test] confirmSuspicion -> Cannot confirm our own eviction from the channel, our certificate was found in config block with sequence 9
2023-06-05 11:07:58.659 IDT 0009 INFO [test] confirmSuspicion -> Suspecting our own eviction from the channel for 10m0s
2023-06-05 11:07:58.659 IDT 000a INFO [test] confirmSuspicion -> Last config block was found to be block [9]
2023-06-05 11:07:58.659 IDT 000b WARN [test] confirmSuspicion -> Detected our own eviction from the channel in block [9]
2023-06-05 11:07:58.659 IDT 000c INFO [test] confirmSuspicion -> Waiting for chain to halt
2023-06-05 11:07:58.659 IDT 000d INFO [test] confirmSuspicion -> Our height is higher or equal than the height of the orderer we pulled the last block from, aborting.
2023-06-05 11:07:58.659 IDT 000e INFO [test] confirmSuspicion -> Suspecting our own eviction from the channel for 10m0s
2023-06-05 11:07:58.660 IDT 000f INFO [test] confirmSuspicion -> Last config block was found to be block [9]
2023-06-05 11:07:58.660 IDT 0010 WARN [test] confirmSuspicion -> Detected our own eviction from the channel in block [9]
2023-06-05 11:07:58.660 IDT 0011 INFO [test] confirmSuspicion -> Waiting for chain to halt
2023-06-05 11:07:58.660 IDT 0012 INFO [test] confirmSuspicion -> Chain has been halted, pulling remaining blocks up to (and including) eviction block.
2023-06-05 11:07:58.660 IDT 0013 INFO [test] confirmSuspicion -> Will now pull blocks 8 to 9
2023-06-05 11:07:58.660 IDT 0014 INFO [test] confirmSuspicion -> Pulling block [8]
2023-06-05 11:07:58.660 IDT 0015 INFO [test] confirmSuspicion -> Pulling block [9]
2023-06-05 11:07:58.660 IDT 0016 INFO [test] confirmSuspicion -> Pulled all blocks up to eviction block.
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestOpenWALLast_WAL_file_is_broken155635490/001/snapshot"}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestOpenWALLast_WAL_file_is_broken155635490/001/wal'"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 0 entries to memory storage"}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestOpenWALLast_WAL_file_is_broken155635490/001/snapshot"}
{"level":"info","msg":"Found WAL data at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestOpenWALLast_WAL_file_is_broken155635490/001/wal', replaying it"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0"}
{"level":"warn","msg":"Failed to read WAL: unexpected EOF"}
{"level":"info","msg":"repairing","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestOpenWALLast_WAL_file_is_broken155635490/001/wal/0000000000000000-0000000000000000.wal"}
{"level":"info","msg":"repaired","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestOpenWALLast_WAL_file_is_broken155635490/001/wal/0000000000000000-0000000000000000.wal","error":"unexpected EOF"}
{"level":"warn","msg":"ignored file in WAL directory","path":"0000000000000000-0000000000000000.wal.broken"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 2 entries to memory storage"}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/snapshot"}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal'"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 0 entries to memory storage"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000001-0000000000000001.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000002-0000000000000002.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000003-0000000000000003.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000004-0000000000000004.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000005-0000000000000005.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000006-0000000000000006.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000007-0000000000000007.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000008-0000000000000008.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000009-0000000000000009.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/000000000000000a-000000000000000a.wal"}
{"level":"debug","msg":"Creating snapshot at index 3 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 3) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 3"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 3"}
{"level":"info","msg":"Snapshot is taken at index 3"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000000-0000000000000000.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000001-0000000000000001.wal"}
{"level":"debug","msg":"Creating snapshot at index 5 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 5) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 5"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 5"}
{"level":"info","msg":"Snapshot is taken at index 5"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000002-0000000000000002.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000003-0000000000000003.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/snapshot/0000000000000000-0000000000000003.snap"}
{"level":"debug","msg":"Loaded snapshot at Term 0 and Index 5, Nodes: [1]"}
{"level":"info","msg":"Found WAL data at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal', replaying it"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 5"}
{"level":"debug","msg":"Applying snapshot to raft MemoryStorage"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 4 entries to memory storage"}
{"level":"debug","msg":"Creating snapshot at index 7 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 7) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 7"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 7"}
{"level":"info","msg":"Snapshot is taken at index 7"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000004-0000000000000004.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000005-0000000000000005.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/snapshot/0000000000000000-0000000000000005.snap"}
{"level":"debug","msg":"Creating snapshot at index 9 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 9) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 9"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 9"}
{"level":"info","msg":"Snapshot is taken at index 9"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000006-0000000000000006.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/wal/0000000000000007-0000000000000007.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==12413103824/001/snapshot/0000000000000000-0000000000000007.snap"}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/snapshot"}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal'"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 0 entries to memory storage"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000001-0000000000000001.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000002-0000000000000002.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000003-0000000000000003.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000004-0000000000000004.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000005-0000000000000005.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000006-0000000000000006.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000007-0000000000000007.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000008-0000000000000008.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000009-0000000000000009.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/000000000000000a-000000000000000a.wal"}
{"level":"debug","msg":"Creating snapshot at index 3 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 3) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 3"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 3"}
{"level":"info","msg":"Snapshot is taken at index 3"}
{"level":"debug","msg":"Snapshots on disk (1) < limit (2), no need to purge wal/snapshot"}
{"level":"debug","msg":"Creating snapshot at index 5 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 5) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 5"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 5"}
{"level":"info","msg":"Snapshot is taken at index 5"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000000-0000000000000000.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000001-0000000000000001.wal"}
{"level":"debug","msg":"Loaded snapshot at Term 0 and Index 5, Nodes: [1]"}
{"level":"info","msg":"Found WAL data at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal', replaying it"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 5"}
{"level":"debug","msg":"Applying snapshot to raft MemoryStorage"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 4 entries to memory storage"}
{"level":"debug","msg":"Creating snapshot at index 7 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 7) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 7"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 7"}
{"level":"info","msg":"Snapshot is taken at index 7"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000002-0000000000000002.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000003-0000000000000003.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/snapshot/0000000000000000-0000000000000003.snap"}
{"level":"debug","msg":"Creating snapshot at index 9 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 9) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 9"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 9"}
{"level":"info","msg":"Snapshot is taken at index 9"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000004-0000000000000004.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/wal/0000000000000005-0000000000000005.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotGoodMaxSnapshotFiles==22423456389/001/snapshot/0000000000000000-0000000000000005.snap"}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot"}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal'"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 0 entries to memory storage"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000001-0000000000000001.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000002-0000000000000002.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000003-0000000000000003.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000004-0000000000000004.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000005-0000000000000005.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000006-0000000000000006.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000007-0000000000000007.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000008-0000000000000008.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000009-0000000000000009.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/000000000000000a-000000000000000a.wal"}
{"level":"debug","msg":"Creating snapshot at index 3 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 3) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 3"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 3"}
{"level":"info","msg":"Snapshot is taken at index 3"}
{"level":"debug","msg":"Snapshots on disk (1) < limit (2), no need to purge wal/snapshot"}
{"level":"debug","msg":"Creating snapshot at index 5 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 5) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 5"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 5"}
{"level":"info","msg":"Snapshot is taken at index 5"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000000-0000000000000000.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000001-0000000000000001.wal"}
{"level":"warn","msg":"failed to unmarshal snappb.Snapshot","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap","error":"proto: illegal wireType 7"}
{"level":"error","msg":"Snapshot file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap is corrupted: proto: illegal wireType 7"}
{"level":"debug","msg":"Renaming corrupted snapshot file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap to /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap.broken"}
{"level":"warn","msg":"failed to unmarshal snappb.Snapshot","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap","error":"proto: illegal wireType 7"}
{"level":"warn","msg":"failed to read a snap file","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap","error":"proto: illegal wireType 7"}
{"level":"warn","msg":"renamed to a broken snap file","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap","broken-path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap.broken"}
{"level":"debug","msg":"Loaded snapshot at Term 0 and Index 3, Nodes: [1]"}
{"level":"info","msg":"Found WAL data at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal', replaying it"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 3"}
{"level":"debug","msg":"Applying snapshot to raft MemoryStorage"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 6 entries to memory storage"}
{"level":"debug","msg":"Creating snapshot at index 7 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 7) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 7"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 7"}
{"level":"info","msg":"Snapshot is taken at index 7"}
{"level":"warn","msg":"Found broken snapshot file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap.broken, it can be removed manually"}
{"level":"debug","msg":"Creating snapshot at index 9 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 9) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 9"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 9"}
{"level":"info","msg":"Snapshot is taken at index 9"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000002-0000000000000002.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000003-0000000000000003.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000004-0000000000000004.wal"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/wal/0000000000000005-0000000000000005.wal"}
{"level":"warn","msg":"Found broken snapshot file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000005.snap.broken, it can be removed manually"}
{"level":"debug","msg":"Purged file /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestTakeSnapshotBadMaxSnapshotFiles==21524095514/001/snapshot/0000000000000000-0000000000000003.snap"}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/snapshot"}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal'"}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0"}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}"}
{"level":"debug","msg":"Appending 0 entries to memory storage"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000001-0000000000000001.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000002-0000000000000002.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000003-0000000000000003.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000004-0000000000000004.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000005-0000000000000005.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000006-0000000000000006.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000007-0000000000000007.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000008-0000000000000008.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/0000000000000009-0000000000000009.wal"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/000000000000000a-000000000000000a.wal"}
{"level":"debug","msg":"Creating snapshot at index 3 from MemoryStorage"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 3) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 3"}
{"level":"debug","msg":"Purging in-memory raft entries prior to 3"}
{"level":"info","msg":"Snapshot is taken at index 3"}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot"}
{"level":"warn","msg":"Attempted to apply out-of-date snapshot at Term 0 and Index 3"}
{"level":"info","msg":"created a new WAL segment","path":"/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/TestApplyOutOfDateSnapshotApply_out_of_date_snapshot2379159986/001/wal/000000000000000b-000000000000000b.wal"}
{"level":"info","msg":"Persisting snapshot (term: 0, index: 3) to WAL and disk"}
{"level":"debug","msg":"Releasing lock to wal files prior to 3"}
{"level":"warn","msg":"Attempted to apply out-of-date snapshot at Term 0 and Index 3"}
2023-06-05 11:08:04.588 IDT 0017 WARN [test] OnConsensus -> An attempt to send a consensus request to a non existing channel (notmychannel) was made by 1
2023-06-05 11:08:04.588 IDT 0018 WARN [test] OnSubmit -> An attempt to submit a transaction to a non existing channel (notmychannel) was made by 1
Running Suite: Etcdraft Suite - /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft
===========================================================================================================
Random Seed: 1685952477

Will run 1 of 121 specs

------------------------------
Chain 3-node Raft cluster reconfiguring raft cluster reconfiguration
  wtf 2
  /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:2123
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/node-1-1868384565/snapshot","channel":"multi-node-channel","node":1}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/node-1-1868384565/wal'","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/node-2-4000201978/snapshot","channel":"multi-node-channel","node":2}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/node-2-4000201978/wal'","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/node-3-2608325559/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/node-3-2608325559/wal'","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 0","channel":"multi-node-channel","node":3}
{"level":"info","msg":"newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became follower at term 0","channel":"multi-node-channel","node":1}
{"level":"info","msg":"newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became follower at term 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Suspecting our own eviction from the channel for 100.665194ms","channel":"multi-node-channel","node":3}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 0","channel":"multi-node-channel","node":2}
{"level":"info","msg":"newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"This node is picked to start campaign","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Suspecting our own eviction from the channel for 101.05854ms","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Suspecting our own eviction from the channel for 201.147869ms","channel":"multi-node-channel","node":3}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
Send artificial MsgTimeoutNow to elect node 1
{"level":"info","msg":"1 [term 1] received MsgTimeoutNow from 0 and starts an election to get leadership.","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 is starting a new election at term 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became candidate at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 received MsgVoteResp from 1 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 [logterm: 1, index: 3] sent MsgVote request to 2 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 [logterm: 1, index: 3] sent MsgVote request to 3 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Suspecting our own eviction from the channel for 100.755693ms","channel":"multi-node-channel","node":2}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Suspecting our own eviction from the channel for 201.340033ms","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Suspecting our own eviction from the channel for 301.265395ms","channel":"multi-node-channel","node":3}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"3 [term: 1] received a MsgVote message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 [logterm: 1, index: 3, vote: 0] cast MsgVote for 1 [logterm: 1, index: 3] at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"2 [term: 1] received a MsgVote message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"2 became follower at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 [logterm: 1, index: 3, vote: 0] cast MsgVote for 1 [logterm: 1, index: 3] at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"1 received MsgVoteResp from 3 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 has received 2 MsgVoteResp votes and 0 vote rejections","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became leader at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 1 elected leader 1 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 3 elected leader 1 at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"There are in flight blocks, new leader should not serve requests","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 2 elected leader 1 at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Suspecting our own eviction from the channel for 201.311031ms","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Last config block was found to be block [0]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Cannot confirm our own eviction from the channel: block data does not carry an envelope at index 0: error unmarshalling Envelope: proto: cannot parse invalid wire-format data","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Received artificial snapshot to trigger catchup","channel":"multi-node-channel","node":2}
{"level":"warn","msg":"Snapshot is at block [0], local block number is 0, no sync needed","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Leader 1 is present, quit campaign","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Start accepting requests as Raft leader at block [0]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
STEP: Submitting first tx to cut the block 06/05/23 11:08:05.325
{"level":"info","msg":"Created block [1], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [1] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":2}
STEP: Disconnecting a follower node 06/05/23 11:08:05.398
STEP: Configuring an additional node 06/05/23 11:08:05.398
STEP: Sending config transaction 06/05/23 11:08:05.399
{"level":"debug","msg":"Node 1 is still part of the consenters set","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [2], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Received config transaction, pause accepting transaction till it is committed","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [2] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Accumulated 6374 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [2] (index: 6), last snapshotted block number is 0, current nodes: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block or ConfChange in flight, pause accepting transaction","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Creating snapshot at index 6 from MemoryStorage","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 not forwarding to leader 1 at term 2; dropping proposal","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Accumulated 6374 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [2] (index: 6), last snapshotted block number is 0, current nodes: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Config block or ConfChange in flight, pause accepting transaction","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Creating snapshot at index 6 from MemoryStorage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Purging in-memory raft entries prior to 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Snapshot is taken at index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Purging in-memory raft entries prior to 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Snapshot is taken at index 6","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":3}
STEP: Starting the new node 06/05/23 11:08:05.567
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 4 because it is unreachable [StateProbe match=0 next=7 paused]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 4, because: connection refused","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/snapshot","channel":"multi-node-channel","node":4}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-4191076620/wal'","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Starting raft node: #peers: 4","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Starting raft node to join an existing channel","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 switched to configuration voters=()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 became follower at term 0","channel":"multi-node-channel","node":4}
{"level":"info","msg":"newRaft 4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Suspecting our own eviction from the channel for 154ns","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Last config block was found to be block [2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Cannot confirm our own eviction from the channel: channelconfig Config cannot be nil","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Received artificial snapshot to trigger catchup","channel":"multi-node-channel","node":4}
{"level":"warn","msg":"Snapshot is at block [2], local block number is 2, no sync needed","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Successfully sent StepRequest to 4 after failed attempt(s)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 [term: 0] received a MsgHeartbeat message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 became follower at term 2","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"raft.node: 4 elected leader 1 at term 2","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Current active nodes in cluster are: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 decreased progress of 4 to [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 [firstindex: 3, commit: 7] sent snapshot[index: 6, term: 2] to 4 [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 paused sending replication messages to 4 [StateSnapshot match=0 next=1 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 snapshot succeeded, resumed sending replication messages to 4 [StateProbe match=0 next=7]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 [commit: 6, lastindex: 6, lastterm: 2] restored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 [commit: 6] restored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 decreased progress of 4 to [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 [firstindex: 3, commit: 7] sent snapshot[index: 6, term: 2] to 4 [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 paused sending replication messages to 4 [StateSnapshot match=0 next=1 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 [commit: 6] ignored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 snapshot succeeded, resumed sending replication messages to 4 [StateProbe match=0 next=7]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
STEP: Sending data blocks to leader 06/05/23 11:08:05.755
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [3], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [4], there are 1 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [5], there are 2 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [3] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Proposed block [4] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [6], there are 3 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [5] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [6] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [7], there are 4 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [7] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [8], there are 5 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [8] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [9], there are 6 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [10], there are 7 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [9] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [11], there are 8 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [10] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [11] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [12], there are 9 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [12] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [13], there are 10 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [14], there are 11 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [13] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [15], there are 12 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [14] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [16], there are 13 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [15] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [17], there are 14 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [16] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [17] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [18], there are 15 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [19], there are 16 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [18] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [19] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [20], there are 17 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [21], there are 18 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [20] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [22], there are 19 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [21] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [23], there are 20 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [22] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [23] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [24], there are 21 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [24] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [25], there are 22 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [26], there are 23 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [25] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [26] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [27], there are 24 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [28], there are 25 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [29], there are 26 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [27] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [30], there are 27 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [28] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [29] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [31], there are 28 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [30] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [31] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [32], there are 29 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [32] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [33], there are 30 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [34], there are 31 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [33] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [35], there are 32 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [34] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [36], there are 33 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [35] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [37], there are 34 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [36] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [37] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [38], there are 35 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [39], there are 36 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [38] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [39] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [40], there are 37 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [40] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [41], there are 38 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [42], there are 39 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [41] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [43], there are 40 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [42] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [43] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [44], there are 41 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [45], there are 42 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [46], there are 43 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [44] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [45] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [47], there are 44 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [46] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [47] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [48], there are 45 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [49], there are 46 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [48] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [49] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [50], there are 47 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [51], there are 48 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [52], there are 49 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [50] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [53], there are 50 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [51] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [52] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [54], there are 51 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [53] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [55], there are 52 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [54] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [55] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [56], there are 53 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [57], there are 54 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [56] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [58], there are 55 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [59], there are 56 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [57] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [60], there are 57 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [58] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [59] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [61], there are 58 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [60] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [61] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [62], there are 59 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [63], there are 60 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [62] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [64], there are 61 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [63] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [64] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [65], there are 62 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [65] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [66], there are 63 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [66] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [67], there are 64 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [68], there are 65 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [69], there are 66 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [70], there are 67 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [71], there are 68 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [72], there are 69 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [67] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [73], there are 70 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [68] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [74], there are 71 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [69] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [70] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [75], there are 72 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [71] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [76], there are 73 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [72] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [77], there are 74 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [73] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [74] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [78], there are 75 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [75] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [79], there are 76 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [76] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [77] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [80], there are 77 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [78] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [79] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [81], there are 78 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [80] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [82], there are 79 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [81] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [82] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [83], there are 80 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [83] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [84], there are 81 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [85], there are 82 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [84] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [86], there are 83 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [85] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [86] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [87], there are 84 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [88], there are 85 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [87] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [88] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [89], there are 86 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [90], there are 87 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [89] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [91], there are 88 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [90] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [92], there are 89 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [91] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [92] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [93], there are 90 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [94], there are 91 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [95], there are 92 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [93] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [96], there are 93 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [94] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [95] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [96] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [97], there are 94 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [98], there are 95 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [97] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [99], there are 96 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [98] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [99] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [100], there are 97 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [101], there are 98 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [102], there are 99 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [100] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [101] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [102] to raft consensus","channel":"multi-node-channel","node":1}
STEP: Reconnecting the follower node 06/05/23 11:08:05.758
STEP: Checking correct synchronization 06/05/23 11:08:05.758
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":4}
{"level":"warn","msg":"Snapshot is at block [2], local block number is 2, no sync needed","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Successfully sent StepRequest to 2 after failed attempt(s)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Accumulated 366 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [5] (index: 10), last snapshotted block number is 2, current nodes: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Creating snapshot at index 10 from MemoryStorage","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 10) to WAL and disk","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"2 not forwarding to leader 1 at term 2; dropping proposal","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Accumulated 6374 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [2] (index: 7), last snapshotted block number is 0, current nodes: [1 2 3 4]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Creating snapshot at index 7 from MemoryStorage","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 7) to WAL and disk","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":1}
{"level":"warn","msg":"Snapshotting is in progress, it is very likely that SnapshotIntervalSize is too small","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Accumulated 12200 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [102] (index: 107), last snapshotted block number is 2, current nodes: [1 2 3 4]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Creating snapshot at index 107 from MemoryStorage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 107) to WAL and disk","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":2}
{"level":"warn","msg":"Snapshotting is in progress, it is very likely that SnapshotIntervalSize is too small","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Releasing lock to wal files prior to 10","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Purging in-memory raft entries prior to 6","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Snapshot is taken at index 10","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Snapshots on disk (2) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Releasing lock to wal files prior to 7","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Releasing lock to wal files prior to 107","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Purging in-memory raft entries prior to 103","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Snapshot is taken at index 107","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Snapshots on disk (2) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft node stopped","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Stop serving requests","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Purging in-memory raft entries prior to 3","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Snapshot is taken at index 7","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Stop garbage collecting","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Raft node stopped","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Stop garbage collecting","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Stop serving requests","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft node stopped","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Stop garbage collecting","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Stop serving requests","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Raft node stopped","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Stop serving requests","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Quit proposing blocks, discarded 0 blocks in the queue","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Stop garbage collecting","channel":"multi-node-channel","node":1}
• [1.636 seconds]
Chain
/Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:102
  3-node Raft cluster
  /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:1580
    reconfiguring raft cluster
    /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:1730
      reconfiguration
      /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:1806
        wtf 2
        /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:2123

  Begin Captured GinkgoWriter Output >>
    Send artificial MsgTimeoutNow to elect node 1
    STEP: Submitting first tx to cut the block 06/05/23 11:08:05.325
    STEP: Disconnecting a follower node 06/05/23 11:08:05.398
    STEP: Configuring an additional node 06/05/23 11:08:05.398
    STEP: Sending config transaction 06/05/23 11:08:05.399
    STEP: Starting the new node 06/05/23 11:08:05.567
    STEP: Sending data blocks to leader 06/05/23 11:08:05.755
    STEP: Reconnecting the follower node 06/05/23 11:08:05.758
    STEP: Checking correct synchronization 06/05/23 11:08:05.758
  << End Captured GinkgoWriter Output
------------------------------

Ran 1 of 121 Specs in 1.637 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 120 Skipped
2023-06-05 11:08:09.522 IDT 0019 INFO [test] Stop -> Periodic check is stopping.
PASS

Ginkgo ran 1 suite in 12.235502476s
Test Suite Passed

Process finished with exit code 0

Log of the test on 2 cases of deadlock (same error different order of execution)
Chain 3-node Raft cluster reconfiguring raft cluster reconfiguration
  wtf 2
  /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:2123
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/node-3-400212855/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/node-3-400212855/wal'","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/node-1-741347575/snapshot","channel":"multi-node-channel","node":1}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/node-1-741347575/wal'","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/node-2-4022112389/snapshot","channel":"multi-node-channel","node":2}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/node-2-4022112389/wal'","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became follower at term 0","channel":"multi-node-channel","node":1}
{"level":"info","msg":"newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became follower at term 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 0","channel":"multi-node-channel","node":2}
{"level":"info","msg":"newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"This node is picked to start campaign","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Suspecting our own eviction from the channel for 101.086032ms","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 0","channel":"multi-node-channel","node":3}
{"level":"info","msg":"newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Suspecting our own eviction from the channel for 100.865014ms","channel":"multi-node-channel","node":2}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Suspecting our own eviction from the channel for 201.477892ms","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
Send artificial MsgTimeoutNow to elect node 1
{"level":"info","msg":"1 [term 1] received MsgTimeoutNow from 0 and starts an election to get leadership.","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 is starting a new election at term 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became candidate at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 received MsgVoteResp from 1 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 [logterm: 1, index: 3] sent MsgVote request to 2 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 [logterm: 1, index: 3] sent MsgVote request to 3 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Suspecting our own eviction from the channel for 100.106602ms","channel":"multi-node-channel","node":3}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Suspecting our own eviction from the channel for 201.355157ms","channel":"multi-node-channel","node":2}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Suspecting our own eviction from the channel for 301.767619ms","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"3 [term: 1] received a MsgVote message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 [logterm: 1, index: 3, vote: 0] cast MsgVote for 1 [logterm: 1, index: 3] at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"2 [term: 1] received a MsgVote message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 [logterm: 1, index: 3, vote: 0] cast MsgVote for 1 [logterm: 1, index: 3] at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"1 received MsgVoteResp from 2 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 has received 2 MsgVoteResp votes and 0 vote rejections","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became leader at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 1 elected leader 1 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 3 elected leader 1 at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"raft.node: 2 elected leader 1 at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"There are in flight blocks, new leader should not serve requests","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Leader 1 is present, quit campaign","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Start accepting requests as Raft leader at block [0]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Suspecting our own eviction from the channel for 200.532068ms","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Last config block was found to be block [0]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Cannot confirm our own eviction from the channel: block data does not carry an envelope at index 0: error unmarshalling Envelope: proto: cannot parse invalid wire-format data","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Received artificial snapshot to trigger catchup","channel":"multi-node-channel","node":3}
{"level":"warn","msg":"Snapshot is at block [0], local block number is 0, no sync needed","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
STEP: Submitting first tx to cut the block 06/05/23 11:02:46.09
{"level":"info","msg":"Created block [1], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Proposed block [1] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":2}
STEP: Disconnecting a follower node 06/05/23 11:02:46.144
STEP: Configuring an additional node 06/05/23 11:02:46.144
STEP: Sending config transaction 06/05/23 11:02:46.145
{"level":"debug","msg":"Node 1 is still part of the consenters set","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [2], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Received config transaction, pause accepting transaction till it is committed","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [2] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Accumulated 6358 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [2] (index: 6), last snapshotted block number is 0, current nodes: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block or ConfChange in flight, pause accepting transaction","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Creating snapshot at index 6 from MemoryStorage","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Accumulated 6358 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [2] (index: 6), last snapshotted block number is 0, current nodes: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Config block or ConfChange in flight, pause accepting transaction","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Creating snapshot at index 6 from MemoryStorage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 not forwarding to leader 1 at term 2; dropping proposal","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Purging in-memory raft entries prior to 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Snapshot is taken at index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Purging in-memory raft entries prior to 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Snapshot is taken at index 6","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":3}
STEP: Starting the new node 06/05/23 11:02:46.3
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/snapshot","channel":"multi-node-channel","node":4}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-1839957041/wal'","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 4 because it is unreachable [StateProbe match=0 next=7 paused]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 4, because: connection refused","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Starting raft node: #peers: 4","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Starting raft node to join an existing channel","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 switched to configuration voters=()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 became follower at term 0","channel":"multi-node-channel","node":4}
{"level":"info","msg":"newRaft 4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Suspecting our own eviction from the channel for 96ns","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Last config block was found to be block [2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Cannot confirm our own eviction from the channel: channelconfig Config cannot be nil","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Received artificial snapshot to trigger catchup","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"warn","msg":"Snapshot is at block [2], local block number is 2, no sync needed","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Successfully sent StepRequest to 4 after failed attempt(s)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 [term: 0] received a MsgHeartbeat message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 became follower at term 2","channel":"multi-node-channel","node":4}
{"level":"info","msg":"raft.node: 4 elected leader 1 at term 2","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Current active nodes in cluster are: [1 3 4 2]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 decreased progress of 4 to [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 [firstindex: 3, commit: 7] sent snapshot[index: 6, term: 2] to 4 [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 paused sending replication messages to 4 [StateSnapshot match=0 next=1 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 snapshot succeeded, resumed sending replication messages to 4 [StateProbe match=0 next=7]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 decreased progress of 4 to [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 [firstindex: 3, commit: 7] sent snapshot[index: 6, term: 2] to 4 [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 paused sending replication messages to 4 [StateSnapshot match=0 next=1 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 snapshot succeeded, resumed sending replication messages to 4 [StateProbe match=0 next=7]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 [commit: 6, lastindex: 6, lastterm: 2] restored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 [commit: 6] restored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 [commit: 6] ignored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 decreased progress of 4 to [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 [firstindex: 3, commit: 7] sent snapshot[index: 6, term: 2] to 4 [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 paused sending replication messages to 4 [StateSnapshot match=0 next=1 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 [commit: 6] ignored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 snapshot succeeded, resumed sending replication messages to 4 [StateProbe match=0 next=7]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
STEP: Sending data blocks to leader 06/05/23 11:02:46.499
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [3], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [3] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [4], there are 1 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [5], there are 2 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [4] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [6], there are 3 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [5] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [6] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [7], there are 4 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [8], there are 5 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [9], there are 6 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [7] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [8] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [9] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [10], there are 7 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [10] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [11], there are 8 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [12], there are 9 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [11] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [12] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [13], there are 10 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [14], there are 11 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [13] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [15], there are 12 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [16], there are 13 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [17], there are 14 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [14] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [15] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [16] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [18], there are 15 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [17] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [18] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [19], there are 16 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [19] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [20], there are 17 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [20] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [21], there are 18 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [22], there are 19 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [23], there are 20 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [21] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [22] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [24], there are 21 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [23] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [24] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [25], there are 22 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [26], there are 23 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [25] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [27], there are 24 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [28], there are 25 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [26] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [29], there are 26 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [27] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [28] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [30], there are 27 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [29] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [30] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [31], there are 28 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [32], there are 29 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [33], there are 30 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [31] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [34], there are 31 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [32] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [33] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [34] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [35], there are 32 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [36], there are 33 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [35] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [37], there are 34 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [36] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [38], there are 35 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [37] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [38] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [39], there are 36 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [40], there are 37 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [39] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [41], there are 38 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [40] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [42], there are 39 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [41] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [43], there are 40 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [42] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [44], there are 41 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [43] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [44] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [45], there are 42 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [46], there are 43 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [45] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [46] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [47], there are 44 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [47] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [48], there are 45 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [49], there are 46 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [48] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [49] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [50], there are 47 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [51], there are 48 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [50] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [52], there are 49 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [51] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [53], there are 50 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [52] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [53] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [54], there are 51 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [55], there are 52 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [54] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [56], there are 53 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [55] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [56] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [57], there are 54 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [57] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [58], there are 55 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [59], there are 56 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [58] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [59] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [60], there are 57 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [61], there are 58 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [60] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [62], there are 59 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [61] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [62] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [63], there are 60 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [63] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [64], there are 61 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [65], there are 62 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [64] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [66], there are 63 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [65] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [66] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [67], there are 64 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [67] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [68], there are 65 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [69], there are 66 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [68] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [70], there are 67 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [69] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [70] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [71], there are 68 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [72], there are 69 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [71] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [73], there are 70 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [72] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [73] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [74], there are 71 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [75], there are 72 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [74] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [76], there are 73 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [75] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [77], there are 74 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [76] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [77] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [78], there are 75 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [79], there are 76 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [78] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [80], there are 77 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [79] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [80] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [81], there are 78 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [82], there are 79 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [81] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [82] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [83], there are 80 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [84], there are 81 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [83] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [85], there are 82 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [84] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [85] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [86], there are 83 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [87], there are 84 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [86] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [88], there are 85 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [87] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [89], there are 86 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [88] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [89] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [90], there are 87 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [91], there are 88 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [92], there are 89 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [90] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [91] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [92] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [93], there are 90 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [93] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [94], there are 91 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [95], there are 92 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [94] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [95] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [96], there are 93 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [97], there are 94 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [96] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [98], there are 95 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [97] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [98] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [99], there are 96 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [100], there are 97 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [99] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [101], there are 98 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [100] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [101] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [102], there are 99 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [102] to raft consensus","channel":"multi-node-channel","node":1}
STEP: Reconnecting the follower node 06/05/23 11:02:46.501
STEP: Checking correct synchronization 06/05/23 11:02:46.501
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 decreased progress of 4 to [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 [firstindex: 3, commit: 7] sent snapshot[index: 6, term: 2] to 4 [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 paused sending replication messages to 4 [StateSnapshot match=0 next=1 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"warn","msg":"Snapshot is at block [2], local block number is 2, no sync needed","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 recovered from needing snapshot, resumed sending replication messages to 4 [StateSnapshot match=6 next=7 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Successfully sent StepRequest to 2 after failed attempt(s)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"4 [commit: 6] ignored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Accumulated 244 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [4] (index: 9), last snapshotted block number is 2, current nodes: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Creating snapshot at index 9 from MemoryStorage","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 9) to WAL and disk","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"2 not forwarding to leader 1 at term 2; dropping proposal","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":1}
{"level":"warn","msg":"Snapshotting is in progress, it is very likely that SnapshotIntervalSize is too small","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Creating snapshot at index 107 from MemoryStorage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 107) to WAL and disk","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Accumulated 12200 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [102] (index: 107), last snapshotted block number is 2, current nodes: [1 2 3 4]","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Releasing lock to wal files prior to 9","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Purging in-memory raft entries prior to 5","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Snapshot is taken at index 9","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Snapshots on disk (2) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Releasing lock to wal files prior to 107","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Purging in-memory raft entries prior to 103","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Snapshot is taken at index 107","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Snapshots on disk (2) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft node stopped","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Stop serving requests","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Stop garbage collecting","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft node stopped","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Stop garbage collecting","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Stop serving requests","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Raft node stopped","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Stop serving requests","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Quit proposing blocks, discarded 0 blocks in the queue","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Stop garbage collecting","channel":"multi-node-channel","node":1}

Chain 3-node Raft cluster reconfiguring raft cluster reconfiguration
  wtf 2
  /Users/emil/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:2123
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/node-1-2525558497/snapshot","channel":"multi-node-channel","node":1}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/node-1-2525558497/wal'","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/node-2-1165583541/snapshot","channel":"multi-node-channel","node":2}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/node-2-1165583541/wal'","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/node-3-1570635261/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/node-3-1570635261/wal'","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became follower at term 0","channel":"multi-node-channel","node":1}
{"level":"info","msg":"newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became follower at term 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":2}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 0","channel":"multi-node-channel","node":2}
{"level":"info","msg":"newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"This node is picked to start campaign","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Suspecting our own eviction from the channel for 100.870152ms","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Starting raft node: #peers: 3","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Starting raft node as part of a new channel","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 0","channel":"multi-node-channel","node":3}
{"level":"info","msg":"newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Suspecting our own eviction from the channel for 100.361587ms","channel":"multi-node-channel","node":2}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 1, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 2, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 3, current nodes in channel: [1 2 3]","channel":"multi-node-channel","node":3}
Send artificial MsgTimeoutNow to elect node 1
{"level":"info","msg":"1 [term 1] received MsgTimeoutNow from 0 and starts an election to get leadership.","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 is starting a new election at term 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became candidate at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 received MsgVoteResp from 1 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 [logterm: 1, index: 3] sent MsgVote request to 2 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 [logterm: 1, index: 3] sent MsgVote request to 3 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"3 [term: 1] received a MsgVote message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 became follower at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"2 [term: 1] received a MsgVote message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 became follower at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"2 [logterm: 1, index: 3, vote: 0] cast MsgVote for 1 [logterm: 1, index: 3] at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"3 [logterm: 1, index: 3, vote: 0] cast MsgVote for 1 [logterm: 1, index: 3] at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Suspecting our own eviction from the channel for 201.076236ms","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 received MsgVoteResp from 3 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 has received 2 MsgVoteResp votes and 0 vote rejections","channel":"multi-node-channel","node":1}
{"level":"info","msg":"1 became leader at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 1 elected leader 1 at term 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Suspecting our own eviction from the channel for 100.503697ms","channel":"multi-node-channel","node":3}
{"level":"error","msg":"Failed pulling the last config block: ledger not available","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 3 elected leader 1 at term 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"There are in flight blocks, new leader should not serve requests","channel":"multi-node-channel","node":1}
{"level":"info","msg":"raft.node: 2 elected leader 1 at term 2","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Start accepting requests as Raft leader at block [0]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Suspecting our own eviction from the channel for 200.994339ms","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Last config block was found to be block [0]","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Cannot confirm our own eviction from the channel: block data does not carry an envelope at index 0: error unmarshalling Envelope: proto: cannot parse invalid wire-format data","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Received artificial snapshot to trigger catchup","channel":"multi-node-channel","node":2}
{"level":"warn","msg":"Snapshot is at block [0], local block number is 0, no sync needed","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Leader 1 is present, quit campaign","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
STEP: Submitting first tx to cut the block 06/05/23 11:26:25.3
{"level":"info","msg":"Created block [1], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [1] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [1] (Raft index: 5) to ledger","channel":"multi-node-channel","node":2}
STEP: Disconnecting a follower node 06/05/23 11:26:25.351
STEP: Configuring an additional node 06/05/23 11:26:25.352
STEP: Sending config transaction 06/05/23 11:26:25.352
{"level":"debug","msg":"Node 1 is still part of the consenters set","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [2], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Received config transaction, pause accepting transaction till it is committed","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [2] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Accumulated 6362 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [2] (index: 6), last snapshotted block number is 0, current nodes: [1 2 3]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block or ConfChange in flight, pause accepting transaction","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Creating snapshot at index 6 from MemoryStorage","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Accumulated 6362 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [2] (index: 6), last snapshotted block number is 0, current nodes: [1 2 3]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Config block or ConfChange in flight, pause accepting transaction","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Creating snapshot at index 6 from MemoryStorage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":3}
{"level":"info","msg":"3 not forwarding to leader 1 at term 2; dropping proposal","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Purging in-memory raft entries prior to 2","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Snapshot is taken at index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Purging in-memory raft entries prior to 2","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Snapshot is taken at index 6","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Snapshots on disk (1) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"1 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"3 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":3}
STEP: Starting the new node 06/05/23 11:26:25.51
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 4, because: connection refused","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 4 because it is unreachable [StateProbe match=0 next=7 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"No snapshot found at /var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/snapshot","channel":"multi-node-channel","node":4}
{"level":"info","msg":"No WAL data found, creating new WAL at path '/var/folders/sh/cx2kqdf52bbgygbr8vchj1qc0000gn/T/raft-test-808286032/wal'","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Loading WAL at Term 0 and Index 0","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Setting HardState to {Term: 0, Commit: 0}","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Appending 0 entries to memory storage","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Starting Raft node","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Starting raft node: #peers: 4","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Starting raft node to join an existing channel","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 switched to configuration voters=()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 became follower at term 0","channel":"multi-node-channel","node":4}
{"level":"info","msg":"newRaft 4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Suspecting our own eviction from the channel for 194ns","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Last config block was found to be block [2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Cannot confirm our own eviction from the channel: channelconfig Config cannot be nil","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Received artificial snapshot to trigger catchup","channel":"multi-node-channel","node":4}
{"level":"warn","msg":"Snapshot is at block [2], local block number is 2, no sync needed","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"error","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Successfully sent StepRequest to 4 after failed attempt(s)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 [term: 0] received a MsgHeartbeat message with higher term from 1 [term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 became follower at term 2","channel":"multi-node-channel","node":4}
{"level":"info","msg":"raft.node: 4 elected leader 1 at term 2","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Current active nodes in cluster are: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Raft leader changed: 0 -> 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 decreased progress of 4 to [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 [firstindex: 3, commit: 7] sent snapshot[index: 6, term: 2] to 4 [StateProbe match=0 next=1]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 paused sending replication messages to 4 [StateSnapshot match=0 next=1 paused pendingSnap=6]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"4 [logterm: 0, index: 6] rejected MsgApp [logterm: 2, index: 6] from 1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 4 for index 6","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"1 snapshot succeeded, resumed sending replication messages to 4 [StateProbe match=0 next=7]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 [commit: 6, lastindex: 6, lastterm: 2] restored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"4 [commit: 6] restored snapshot [index: 6, term: 2]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 6) to WAL and disk","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
STEP: Sending data blocks to leader 06/05/23 11:26:25.69
{"level":"debug","msg":"1 failed to send message to 2 because it is unreachable [StateProbe match=5 next=6 paused]","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Failed to send StepRequest to 2, because: connection lost","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [3], there are 0 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Created block [4], there are 1 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [3] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [4] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [5], there are 2 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [6], there are 3 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [5] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [6] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [7], there are 4 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [8], there are 5 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [9], there are 6 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [7] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [8] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [9] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [10], there are 7 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [10] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [11], there are 8 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [12], there are 9 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [11] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [13], there are 10 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [12] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [13] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [14], there are 11 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [15], there are 12 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [14] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [15] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [16], there are 13 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [17], there are 14 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [16] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [18], there are 15 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [17] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [18] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [19], there are 16 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [20], there are 17 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [19] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [20] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [21], there are 18 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [22], there are 19 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [23], there are 20 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [21] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [24], there are 21 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [22] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [25], there are 22 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [23] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [24] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [25] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [26], there are 23 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [27], there are 24 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [28], there are 25 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [29], there are 26 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [30], there are 27 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [26] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [27] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [28] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [31], there are 28 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [29] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [30] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [32], there are 29 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [31] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [32] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [33], there are 30 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [34], there are 31 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [33] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [34] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [35], there are 32 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [36], there are 33 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [35] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [37], there are 34 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [36] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [37] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [38], there are 35 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [39], there are 36 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [38] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [40], there are 37 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [39] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [40] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [41], there are 38 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [42], there are 39 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [41] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [43], there are 40 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [42] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [43] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [44], there are 41 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [45], there are 42 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [44] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [46], there are 43 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [47], there are 44 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [45] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [46] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [47] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [48], there are 45 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [49], there are 46 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [48] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [50], there are 47 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [49] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [51], there are 48 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [50] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [51] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [52], there are 49 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [53], there are 50 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [52] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [53] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [54], there are 51 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [54] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [55], there are 52 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [56], there are 53 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [55] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [56] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [57], there are 54 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [58], there are 55 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [59], there are 56 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [57] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [60], there are 57 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [58] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [61], there are 58 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [59] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [60] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [62], there are 59 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [61] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [62] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [63], there are 60 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [64], there are 61 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [63] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [65], there are 62 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [64] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [65] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [66], there are 63 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [67], there are 64 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [66] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [68], there are 65 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [67] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [69], there are 66 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [68] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [69] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [70], there are 67 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [70] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [71], there are 68 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [72], there are 69 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [71] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [73], there are 70 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [72] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [74], there are 71 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [73] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [74] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [75], there are 72 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [76], there are 73 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [75] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [77], there are 74 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [76] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [78], there are 75 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [77] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [78] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [79], there are 76 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [80], there are 77 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [79] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [81], there are 78 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [80] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [81] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [82], there are 79 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [82] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [83], there are 80 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [84], there are 81 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [83] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [85], there are 82 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [84] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [85] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [86], there are 83 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [86] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [87], there are 84 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [88], there are 85 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [87] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [89], there are 86 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [88] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [90], there are 87 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [89] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [90] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [91], there are 88 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [91] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [92], there are 89 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [92] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [93], there are 90 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [94], there are 91 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [93] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [95], there are 92 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [94] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [96], there are 93 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [95] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [96] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [97], there are 94 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [98], there are 95 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [97] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [99], there are 96 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [98] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [100], there are 97 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [99] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [100] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [101], there are 98 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Created block [102], there are 99 blocks in flight","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [101] to raft consensus","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Proposed block [102] to raft consensus","channel":"multi-node-channel","node":1}
STEP: Reconnecting the follower node 06/05/23 11:26:25.693
STEP: Checking correct synchronization 06/05/23 11:26:25.693
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Releasing lock to wal files prior to 6","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: n.chain.snapC <- &rd.Snapshot","channel":"multi-node-channel","node":4}
{"level":"warn","msg":"Snapshot is at block [2], local block number is 2, no sync needed","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Successfully sent StepRequest to 2 after failed attempt(s)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: app := <-c.applyC:","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 4:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 2:: c.apply(app.entries)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: <<<<<<<<<<<< case <-n.storage.WALSyncC","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: @@@@@@@@@@@@@@+1","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [2] (Raft index: 6) to ledger","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 1:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":1}
{"level":"info","msg":"4 switched to configuration voters=(1 2 3 4)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Applied config change to add node 4, current nodes in channel: [1 2 3 4]","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Accumulated 244 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [4] (index: 9), last snapshotted block number is 2, current nodes: [1 2 3 4]","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: app := <-c.applyC:","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 1:: c.apply(app.entries)","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Creating snapshot at index 9 from MemoryStorage","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 9) to WAL and disk","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Update snapshot interval size to 200 bytes (was 16777216)","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"2 not forwarding to leader 1 at term 2; dropping proposal","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":1}
{"level":"warn","msg":"Snapshotting is in progress, it is very likely that SnapshotIntervalSize is too small","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Node 4:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: app := <-c.applyC:","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 4:: c.apply(app.entries)","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: ?????????? rd := <-n.Ready()","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Node 3:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: app := <-c.applyC:","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Node 3:: c.apply(app.entries)","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [3] (Raft index: 8) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [4] (Raft index: 9) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [5] (Raft index: 10) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [6] (Raft index: 11) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [7] (Raft index: 12) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [10] (Raft index: 15) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [11] (Raft index: 16) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [12] (Raft index: 17) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [13] (Raft index: 18) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [14] (Raft index: 19) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [15] (Raft index: 20) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [16] (Raft index: 21) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [17] (Raft index: 22) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [18] (Raft index: 23) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [19] (Raft index: 24) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [20] (Raft index: 25) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [21] (Raft index: 26) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [22] (Raft index: 27) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [23] (Raft index: 28) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [24] (Raft index: 29) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [25] (Raft index: 30) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [26] (Raft index: 31) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [27] (Raft index: 32) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [28] (Raft index: 33) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [29] (Raft index: 34) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [30] (Raft index: 35) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [31] (Raft index: 36) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [32] (Raft index: 37) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [33] (Raft index: 38) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [34] (Raft index: 39) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [35] (Raft index: 40) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [36] (Raft index: 41) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [37] (Raft index: 42) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [38] (Raft index: 43) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [39] (Raft index: 44) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [40] (Raft index: 45) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [41] (Raft index: 46) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [42] (Raft index: 47) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [43] (Raft index: 48) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [44] (Raft index: 49) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [45] (Raft index: 50) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [46] (Raft index: 51) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [47] (Raft index: 52) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [48] (Raft index: 53) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [49] (Raft index: 54) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [50] (Raft index: 55) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [51] (Raft index: 56) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [52] (Raft index: 57) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [53] (Raft index: 58) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [54] (Raft index: 59) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [55] (Raft index: 60) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [56] (Raft index: 61) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [57] (Raft index: 62) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [58] (Raft index: 63) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [59] (Raft index: 64) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [60] (Raft index: 65) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [61] (Raft index: 66) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [62] (Raft index: 67) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [63] (Raft index: 68) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [64] (Raft index: 69) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [65] (Raft index: 70) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [66] (Raft index: 71) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [67] (Raft index: 72) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [68] (Raft index: 73) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [69] (Raft index: 74) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [70] (Raft index: 75) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [71] (Raft index: 76) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [72] (Raft index: 77) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [73] (Raft index: 78) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [74] (Raft index: 79) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [75] (Raft index: 80) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [76] (Raft index: 81) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [77] (Raft index: 82) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [78] (Raft index: 83) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [79] (Raft index: 84) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [80] (Raft index: 85) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [81] (Raft index: 86) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [82] (Raft index: 87) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [83] (Raft index: 88) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [84] (Raft index: 89) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [85] (Raft index: 90) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [86] (Raft index: 91) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [87] (Raft index: 92) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [88] (Raft index: 93) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [89] (Raft index: 94) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [90] (Raft index: 95) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [91] (Raft index: 96) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [92] (Raft index: 97) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [93] (Raft index: 98) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [94] (Raft index: 99) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [95] (Raft index: 100) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [96] (Raft index: 101) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [97] (Raft index: 102) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [98] (Raft index: 103) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Accumulated 12200 bytes since last snapshot, exceeding size limit (200 bytes), taking snapshot at block [102] (index: 107), last snapshotted block number is 2, current nodes: [1 2 3 4]","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [99] (Raft index: 104) to ledger","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Creating snapshot at index 107 from MemoryStorage","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Persisting snapshot (term: 2, index: 107) to WAL and disk","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Writing block [100] (Raft index: 105) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [101] (Raft index: 106) to ledger","channel":"multi-node-channel","node":4}
{"level":"info","msg":"Writing block [102] (Raft index: 107) to ledger","channel":"multi-node-channel","node":4}
{"level":"debug","msg":"Releasing lock to wal files prior to 9","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Purging in-memory raft entries prior to 5","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Snapshot is taken at index 9","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Snapshots on disk (2) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":1}
{"level":"debug","msg":"Releasing lock to wal files prior to 107","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Purging in-memory raft entries prior to 103","channel":"multi-node-channel","node":3}
{"level":"info","msg":"Snapshot is taken at index 107","channel":"multi-node-channel","node":3}
{"level":"debug","msg":"Snapshots on disk (2) < limit (4), no need to purge wal/snapshot","channel":"multi-node-channel","node":3}

Thanks @semil .
@yacovm @denyeart The log above shows the deadlock. In node2, the two goroutines are waiting for each other on two different channels:

one on n.chain.applyC <- apply
the other on c.Node.storage.WALSyncC <- struct{}{}
node2 gets stuck there.

{"level":"info","msg":"Node 2:: !!!!!!!!!!!!!!!!!! n.chain.applyC <- apply{rd.CommittedEntries, rd.SoftState}","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [8] (Raft index: 13) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"2 not forwarding to leader 1 at term 2; dropping proposal","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Writing block [9] (Raft index: 14) to ledger","channel":"multi-node-channel","node":1}
{"level":"info","msg":"Config block just committed adds node 4, pause accepting transactions till config change is applied","channel":"multi-node-channel","node":2}
{"level":"info","msg":"Node 2:: @@@@@@@@@@@@@@ c.Node.storage.WALSyncC <- struct{}{}","channel":"multi-node-channel","node":2}

authors: semil tock-ibm