A new Write-Ahead Log for Raft

[sakno]

.NEXT 5.22.0 introduces a new WAL that is a replacement of existing PersistentState class. it is marked as experimental for now. However, it will be the only implementation in the next major version of .NEXT, which will be released in the end of 2025 year.

The new WAL has the performance comparable to FasterLog, the append method is not blocked by disk I/O, it has simpler configuration, simpler implementation and better state machine API.

[guillaume-chervet]

Hi @sakno, thank you again so much for this wonderful library !

More performance will be awesome :)
Do you have a kind of migration guide or something that can help me to migrate to latest version ?

I was using PackageReference Include="DotNext.AspNetCore.Cluster" Version="5.19.1" but latest version seem to have many breaking change.

Here my actual and simple code :
https://github.com/SlimPlanet/SlimFaas/blob/main/src/SlimData/SlimData.csproj

https://github.com/SlimPlanet/SlimFaas/blob/main/src/SlimData/SlimDataInterpreter.cs

https://github.com/SlimPlanet/SlimFaas/blob/main/src/SlimData/SlimPersistentState.cs

[sakno]

For the command handler, take a look at CommandInterpreterTests test. The interpreter framework is simplified a lot. The documentation is also updated: https://dotnet.github.io/dotNext/features/cluster/wal.html#command-interpreter

Do you wanna migrate to a new Write-Ahead Log? It's also relatively simple: https://dotnet.github.io/dotNext/features/cluster/wal.html#simple-state-machine. You need to implement IStateMachine interface or you can use SimpleStateMachine abstract implementation with out-of-the-box snapshotting. The semantics is very close to the old WAL.

[sakno]

I'll try to do a PR for you.

[sakno]

@guillaume-chervet , there is PR: SlimPlanet/SlimFaas#145

[guillaume-chervet]

Thank you so much @sakno !

[guillaume-chervet]

Hi @sakno ,

I have migrated to the new SimpleStateMachine (Code migration was easy, thank you for your very good your documentation), but I encounter some problems.
If I run SlimFaas with 3 nodes (the same with only 1 node), I have after some random time this error (it was working with the previous system):

https://github.com/SlimPlanet/SlimFaas/blob/605b823cb75b185df8cf876d94aae01edaa0a280/src/SlimData/SlimPersistentState.cs#L11

fail: SlimFaas.Database.SlimDataService[0]
      SlimData Service DoAsync
      System.InsufficientMemoryException: Insufficient memory to continue the execution of the program.
         at DotNext.Net.Cluster.Consensus.Raft.StateMachine.WriteAheadLog.PagedBufferWriter.System.Buffers.IBufferWriter<System.Byte>.GetSpan(Int32) + 0x82
         at System.Text.EncodingExtensions.Convert(Encoder, ReadOnlySpan`1, IBufferWriter`1, Boolean, Int64&, Boolean&) + 0x103
         at DotNext.Buffers.BufferWriter.Encode(IBufferWriter`1, ReadOnlySpan`1, EncodingContext&, Nullable`1) + 0xd0
         at DotNext.IO.AsyncBufferWriter.DotNext.IO.IAsyncBinaryWriter.EncodeAsync(ReadOnlyMemory`1, EncodingContext, Nullable`1, CancellationToken) + 0x101
      --- End of stack trace from previous location ---
         at SlimData.Commands.ListCallbackBatchCommand.<WriteToAsync>d__10`1.MoveNext() + 0x5c3
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.StateMachine.WriteAheadLog.<AppendUnbufferedAsync>d__35`1.MoveNext() + 0x237
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<ReplicateAsync>d__120`1.MoveNext() + 0x441
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<ReplicateAsync>d__120`1.MoveNext() + 0x71c
      --- End of stack trace from previous location ---
      at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x1c
         at SlimData.Endpoints.<SafeReplicateAsync>d__2`1.MoveNext() + 0x339
      --- End of stack trace from previous location ---
         at SlimData.Endpoints.<ListCallbackBatchCommand>d__18.MoveNext() + 0x9c
      --- End of stack trace from previous location ---
         at SlimFaas.Database.SlimDataService.<BatchListCallbackHandlerAsync>d__22.MoveNext() + 0x113
      --- End of stack trace from previous location ---
         at SlimData.MultiRateAdaptiveBatcher.<>c__DisplayClass14_0`2.<<RegisterKind>b__3>d.MoveNext() + 0x60
      --- End of stack trace from previous location ---
         at SlimData.MultiRateAdaptiveBatcher.<LoopAsync>d__18.MoveNext() + 0x759
      --- End of stack trace from previous location ---
         at SlimData.MultiRateAdaptiveBatcher.<EnqueueAsync>d__15`2.MoveNext() + 0x26b
      --- End of stack trace from previous location ---
         at SlimFaas.Database.SlimDataService.<>c__DisplayClass21_0.<<ListCallbackAsync>b__0>d.MoveNext() + 0x5d
      --- End of stack trace from previous location ---
         at SlimFaas.Retry.<DoAsync>d__1.MoveNext() + 0x15c

————

If I run my test https://github.com/SlimPlanet/SlimFaas/blob/605b823cb75b185df8cf876d94aae01edaa0a280/tests/SlimData.Tests/RaftClusterTests.cs#L311
I have at the end 512 elements instead of 1000 (2 commands run on the background with 2 await cluster.ReplicateAsync(cmd, ct);)

Do you have an idea why ?

Thank you so much for this library.

[sakno]

This is experimental API, so the bugs are possible. Please open a separate task.

[sakno]

After closer look, I think that ListCallbackBatchCommand.Length implementation returns a length in bytes that is not equal to the actual number of bytes written by WriteToAsync method.

[guillaume-chervet]

Hi @sakno sorry for the delay.
I have fixed Length issue.

I send you some feedbacks as soon as I can test it again. Again, thank you for your work and help.

[sakno]

.NEXT 6.0 has been released with a new Write Ahead Log. I'm closing this discussion.

[guillaume-chervet]

So cool thank you @sakno ! I will be happy to play with this new version :)