Service Protocol V4 and Immutable journal

[slinkydeveloper]

This PR implements the ServiceProtocol V4 and the immutable journal (often referred in the codebase as Journal V2 or Journal Table V2).

The Immutable Journal is a rework of the journal model, splitting commands and completions in distinct entries. The commit c484925 defines the new data model, which roughly works as follows:

• The journal is composed of entries, which can be:
  • Commands (e.g. call, set state, etc)
  • Notifications
  • Events
• Notifications are sub-categorized into:
  • Completions (what we used to write back in the entries)
  • Signals, which can be addressed with either a numeric id or a string

In order to implement the new protocol and the immutable journal, there are two places where most of the interesting stuff happens:

• In the invoker, commit 4edf7d4 (@tillrohrmann PTAL).
• In the state machine, commits c1616ca (@tillrohrmann PTAL) and 0e13883 and 5f1ae6f.
  • Notably, a lot of code was copy-pasted from the old state machine to the new state machine. This was done to reduce the risk of this changes and to make as obvious as possible the branching points between old and new state machine. All these branching points for the state machine are in the old state machine mod.rs file.
  • There are few new commands for the wal protocol that we don't write yet, but will be used in future releases once we remove the old service protocol.
  • Because the invoker is the component that ultimately decides the service protocol version, we initialize the journal table always using the old table, and then when the invoker sends the PinnedDeployment effect we perform, if needed, a migration from the old table to the new table. This enforces the following invariants:
    • if pinned service protocol is none or <= 3 then use old journal table, else use new journal table
    • For a given invocation id, either old journal table contains something, or new journal table contains something, but not both at the same time
  • The aforementioned invariant has two corner cases:
    • If we get an awakeable completion before the PinnedDeployment effect. To solve this case, I introduced a new id scheme called ExternalSignalIdentifier, which will be produced only by SDKs using service protocol v4.
    • If we get cancelled before the PinnedDeployment effect. In this case, the journal contains only the input entry, thus we can ignore the cancel for now.

The storage changes are minimal (commit 2ba04cc), they add a JournalTableV2 which is very similar to the old JournalTable, but uses the new data model and adds a reverse index from notification id to notification and from completion id to command.

There are few other code organization changes that have been made in this PR and will need subsequent work to finalize:

• Sunsetting the https://github.com/restatedev/service-protocol and just move all that stuff here. We don't need to publish the proto files to BSR anymore, and there's no need to have a repo only for that, so it reduces maintenance burden to consolidate those protobuf files just here under /service-protocol.
• Protobuf messages were moved to restate_types (presumably for the service protocol version data structure) but I would like to reverse this and simply generate the service protocol version data structure (or even just write it in code, no need to have this in protobuf), and will do it once we remove the old service-protocol.

This PR misses the following, which will be tackled in followups:

• Wiring the new journal table to datafusion
• Service protocol spec, which needs to be rewritten
• Unit test coverage of the new state machine

[slinkydeveloper]

Good news! All the feature tests for protocol V4 on our WIP python SDK branch are passing!

[igalshilman]

Tested this locally with the verification test. Mixed protocols v4 and v3.