WIP: Progress on py-libp2p and rust-libp2p interop for Ping #532

paschal533 · 2025-03-06T12:38:14Z

Description

This is a work-in-progress (WIP) pull request to improve interoperability between py-libp2p and rust-libp2p, focusing on the Noise XX handshake and Yamux multiplexing. The goal is to enable a successful ping exchange between a Python client and a Rust server.

Changes

examples/ping/ping.py:
- Modified to use a KeyPair derived from Rust’s private and public keys for testing (e.g., private: 6866d018..., public: 0d3fb3dc...). These are placeholders and change per Rust run; future work should dynamically fetch them.
- Updated Transport instantiation to pass noise_privkey=key_pair.private_key instead of noise_keypair (fixed call-arg error).
- Added type annotations and cast(TProtocol, ...) for muxer_transports_by_protocol to resolve mypy errors.
libp2p/stream_muxer/yamux/yamux.py:
- Updated header size to 12 bytes (!BBHI) per the Yamux spec, fixing the previous 8-byte mismatch.
- Improved stream handling for FLAG_SYN and FLAG_FIN.
libp2p/security/noise/transport.py:
- Adjusted secure_inbound signature to match ISecureTransport (removed peer_id parameter).
libp2p/security/noise/patterns.py:
- Made remote_peer_id optional in PatternXX.handshake_inbound to match IPattern interface, fixing override error.

Current Status

Noise Handshake:
- Completes on Python side but fails on Rust side with BadSignature during Msg define interfaces for milestone 1 #3 verification. Logs show Python’s Noise public key (ecc32c64...) doesn’t match the expected Rust key (0d3fb3dc...), despite setting the KeyPair.
- The hardcoded Rust keys in ping.py are temporary and change per Rust run, indicating a need for dynamic key exchange or configuration.
Muxing:
- Fails with IncompleteReadError after the handshake, likely due to Rust closing the connection post-BadSignature.
- Yamux header updated to 12 bytes, but compatibility untested until handshake succeeds.
Next Steps:
- Debug why Transport uses a different Noise key despite setting noise_privkey.
- Implement dynamic key retrieval from Rust instead of hardcoding.
- Verify Yamux interoperability once the handshake is fixed.
- Run full interop test with updated Rust logs.

Logs

Python:

DEBUG:root:Local Peer ID: 12D3KooWRhnLFgm53oxqcC3teSWQKFaCt1oa2NfmCRps7yrC5SQF
DEBUG:root:Public Key from Private: ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8
DEBUG:root:Expected Public Key: aa7f2550cd849dc2a78e6a4d76c103f492c55bb75cc41adc6ae09ff2312d0a9e
DEBUG:async_service.Manager:<Manager[Swarm] flags=SRcfe>: _handle_cancelled waiting for cancellation
DEBUG:libp2p.transport.tcp:serve_tcp 0.0.0.0 8001
DEBUG:libp2p.network.swarm:attempting to dial peer 12D3KooWGjG2AC1hoU3DURkkvLSEJnvg1cTyLRajW9YMe8ejS5ac
DEBUG:libp2p.network.swarm:dialed peer 12D3KooWGjG2AC1hoU3DURkkvLSEJnvg1cTyLRajW9YMe8ejS5ac over base transport
DEBUG:root:Using libp2p key: <libp2p.crypto.ed25519.Ed25519PrivateKey object at 0x000001E4C4485050>, noise key: None
DEBUG:root:libp2p_privkey pubkey: ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8
DEBUG:root:noise_static_key pubkey: ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8
DEBUG:root:Starting outbound handshake with peer 12D3KooWGjG2AC1hoU3DURkkvLSEJnvg1cTyLRajW9YMe8ejS5ac
DEBUG:root:Wrote encrypted message: 6c743a7731bc7c34ea196f821404b05f8b3a3803b2b19efdcf1220c395d3040c
DEBUG:root:Sent Msg #1: 6c743a7731bc7c34ea196f821404b05f8b3a3803b2b19efdcf1220c395d3040c
DEBUG:root:Received Msg #2: 0a240801122066b2f3edfd6f987a0d9e9e070a52154c2592e81dd2b7ee84a472ab50d8fd41151240cae7954cd98fdd8d81349a58af954589fd079a118c7471c54165d7767c66a1ed1658548fd92734580f3d30fd970754d133b56f73a5210fd983c423e909cda604
DEBUG:root:Public key bytes: ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8
DEBUG:root:Transcript for signature: 6e6f6973652d6c69627032702d7374617469632d6b65793aec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8
DEBUG:root:Signing with privkey, pubkey: ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8
DEBUG:root:Signature: 1d9e7046271c6bc4a797686f446f77e78007bcf3d9edcfcb309bffee4ce39a2ca1ceca898edcea541c211fd46d275b2409cd24f39b5ba73ace8f229f55dd0c08
DEBUG:root:Signature verified locally
DEBUG:root:Raw public key bytes: ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8, length: 32
DEBUG:root:Serialized pubkey: 08011220ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b8
DEBUG:root:Serialized payload: 0a2408011220ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b812401d9e7046271c6bc4a797686f446f77e78007bcf3d9edcfcb309bffee4ce39a2ca1ceca898edcea541c211fd46d275b2409cd24f39b5ba73ace8f229f55dd0c08
DEBUG:root:Prepared Msg #3 raw: 0a2408011220ec0b3f57bd7cd213ea0f913947b374c27a3035654b068bd334722b85aaf176b812401d9e7046271c6bc4a797686f446f77e78007bcf3d9edcfcb309bffee4ce39a2ca1ceca898edcea541c211fd46d275b2409cd24f39b5ba73ace8f229f55dd0c08, length: 104     
DEBUG:root:Wrote encrypted message: 6bc32048aef6fd1013ac1f8f1eb7f68a7c6fcaa6832589426c556bac761b27d653cf0a13ca6653a647c60126f2d2afa5c1f2262a80e4b373a9c6f9fc6d96b505adcc578c0a358ca237fb01ccecc6a9b7a9cf5fd082b6e393f80eb8996a713f0837244db4a916e474437e73b1cb0ed297a3b77ba312f859a49f1c8bcd8b47d577ac0b2641de96bca24d82be33bc4615ea83dbadc19944662349cb1f72017a99ca83d48d986c7f3c37
DEBUG:root:Sent Msg #3 (s, se): 6bc32048aef6fd1013ac1f8f1eb7f68a7c6fcaa6832589426c556bac761b27d653cf0a13ca6653a647c60126f2d2afa5c1f2262a80e4b373a9c6f9fc6d96b505adcc578c0a358ca237fb01ccecc6a9b7a9cf5fd082b6e393f80eb8996a713f0837244db4a916e474437e73b1cb0ed297a3b77ba312f859a49f1c8bcd8b47d577ac0b2641de96bca24d82be33bc4615ea83dbadc19944662349cb1f72017a99ca83d48d986c7f3c37
DEBUG:root:Handshake completed
DEBUG:libp2p.network.swarm:upgraded security for peer 12D3KooWGjG2AC1hoU3DURkkvLSEJnvg1cTyLRajW9YMe8ejS5ac
DEBUG:libp2p.network.swarm:failed to upgrade mux for peer 12D3KooWGjG2AC1hoU3DURkkvLSEJnvg1cTyLRajW9YMe8ejS5ac
DEBUG:libp2p.network.swarm:encountered swarm exception when trying to connect to /ip4/127.0.0.1/tcp/35557, trying next address...Traceback (most recent call last):

Rust:

2025-03-06T10:12:44.209617Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: multistream_select::protocol: Received message Header(V1)
2025-03-06T10:12:44.211197Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: multistream_select::protocol: Received message Protocol(Protocol("/noise"))
2025-03-06T10:12:44.211305Z DEBUG new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: multistream_select::listener_select: Listener: confirming protocol protocol=/noise
2025-03-06T10:12:44.213647Z DEBUG new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: multistream_select::listener_select: Listener: sent confirmed protocol protocol=/noise
2025-03-06T10:12:44.289501Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: libp2p_noise::io::framed: Incoming ciphertext has 32 bytes
2025-03-06T10:12:44.289570Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: libp2p_noise::io::framed: Decrypted cleartext has 0 bytes
2025-03-06T10:12:44.289588Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: libp2p_noise::io::framed: Encrypting 104 bytes
2025-03-06T10:12:44.290024Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: libp2p_noise::io::framed: Outgoing ciphertext has 200 bytes
2025-03-06T10:12:44.314637Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: libp2p_noise::io::framed: Incoming ciphertext has 168 bytes
2025-03-06T10:12:44.315002Z TRACE new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: libp2p_noise::io::framed: Decrypted cleartext has 104 bytes
2025-03-06T10:12:44.320755Z DEBUG new_incoming_connection{remote_addr=/ip4/127.0.0.1/tcp/49616 id=1}: libp2p_core::upgrade::apply: Failed to upgrade inbound stream upgrade=/noise
2025-03-06T10:12:44.321646Z DEBUG Swarm::poll: libp2p_swarm: Incoming connection failed: Transport(Other(Custom 
{ kind: Other, error: Other(Left(Right(Apply(BadSignature)))) }))

Request for Feedback

Is noise_privkey in Transport being applied correctly to the Noise handshake? Logs suggest a mismatch.
Why does the Noise handshake generate a different key pair (ecc32c64...) instead of using the provided 0d3fb3dc...?
Suggestions for dynamically fetching Rust’s key pair instead of hardcoding?
Any tips to ensure Yamux compatibility with rust-libp2p once the handshake succeeds?

Notes

The Rust key pair (6866d018... and 0d3fb3dc...) changes per run, so hardcoding is a temporary test measure. A production solution should query or negotiate keys.
This PR is marked WIP and will be updated as progress continues.

seetadev · 2025-03-10T22:32:07Z

@paschal533 : Thank you for submitting the PR. Appreciate it.

Please have a look at the 12 checks that are failing in this PR.

Looking forward to seeing a revised PR soon.

seetadev · 2025-03-27T22:14:29Z

@paschal533 : Thank you for submitting the PR. Appreciate it.

Please have a look at the 12 checks that are failing in this PR.

Looking forward to seeing a revised PR soon.

@paschal533 : This PR needs some work as a number of tests are failing. Please connect with @panditdhamdhere , who was there in the maintainer's call today. He can work with you on the bridging the interoperability with rust-libp2p and has expressed interest to work on rust-libp2p related projects.

On the same note, I will open up an issue on the interoperability with go-libp2p. @pranavkonde can help us here as shared by @panditdhamdhere.

Thank you once again. Looking forward to seeing the interoperability with rust-libp2p.

paschal533 · 2025-03-31T21:42:29Z

Hi, @panditdhamdhere, are you free this week? let's work on this interop testing.

panditdhamdhere · 2025-04-01T01:42:53Z

Yeah @paschal533 offcourse let's do it.

lla-dane · 2025-05-07T14:52:49Z

Hello, @paschal533 and @panditdhamdhere, I am a new contributor in PLDG cohort-3. @seetadev recommended me work on this as a starting issue. I will go through libp2p/rust-libp2p#5717 and https://github.com/quinn-rs/quinn as @seetadev recommended for a few time, and then start to research about this PR.
Thanks!

seetadev · 2025-05-07T15:06:37Z

Hello, @paschal533 and @panditdhamdhere, I am a new contributor in PLDG cohort-3. @seetadev recommended me work on this as a starting issue. I will go through libp2p/rust-libp2p#5717 and https://github.com/quinn-rs/quinn as @seetadev recommended for a few time, and then start to research about this PR. Thanks!

@lla-dane : Hi Abhinav. Great to learn about your interest. You can go through libp2p/rust-libp2p#5717 and the ping example at https://py-libp2p.readthedocs.io/en/stable/examples.html . In reference to py-libp2p, please study https://github.com/libp2p/py-libp2p ; https://py-libp2p.readthedocs.io/en/stable/getting_started.html .

Please also study the test plans repo: https://github.com/libp2p/test-plans. @paschal533 and @panditdhamdhere : this would be very helpful from understanding interoperability. Interoperability in reference to our project means that a node using py-libp2p can seamlessly communicate with a rust-libp2p node using shared protocol modules (e.g., /ipfs/ping/1.0.0).

paschal533 · 2025-05-08T11:07:40Z

Hello, @paschal533 and @panditdhamdhere, I am a new contributor in PLDG cohort-3. @seetadev recommended me work on this as a starting issue. I will go through libp2p/rust-libp2p#5717 and https://github.com/quinn-rs/quinn as @seetadev recommended for a few time, and then start to research about this PR. Thanks!

Hi @lla-dane, welcome to Py-libp2p 🎉. Let's get this done once Yamux has been implemented. Looking forward to working with you on this. :)

WORKING: Connection, handshake, Yamux setup, initial ping/pong. ISSUE: Frame parser corruption after 2-3 frames (boundary sync)

paschal533 · 2025-05-31T12:02:59Z

Ping Interoperability Progress Report

✅ SUCCESS: Basic Ping Interop Working

Hi @lla-dane, @acul71, @seetadev, I have successfully established basic ping interoperability between py-libp2p and rust-libp2p The connection, handshake, and initial ping/pong exchange are working correctly.

What's Working:

✅ Connection establishment between Python and Rust nodes
✅ Noise security protocol negotiation
✅ Yamux stream multiplexing setup
✅ Initial Yamux ping/pong frames (connection-level)
✅ Stream creation and protocol negotiation for /ipfs/ping/1.0.0

⚠️ CURRENT ISSUE: Frame Parsing Corruption After Initial Success

After the successful ping exchange, the Yamux frame parser encounters corrupted frames that cause connection termination.

Error Details

Issue: After processing 2-3 valid Yamux frames, the parser receives malformed frames with:

Invalid frame types (47, 115 instead of 0-3)
Massive frame lengths (>1GB instead of reasonable sizes)
Invalid flag values

This suggests a frame boundary synchronization issue where the parser gets out of sync with the byte stream.

Logs - Python Side (py-libp2p)

2025-05-31 00:15:13,264 - DEBUG - Received header for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV:type=2, flags=1, stream_id=0,length=858059319 2025-05-31 00:15:13,264 - DEBUG - Received ping request with value858059319 for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV 2025-05-31 00:15:13,267 - DEBUG - Received header for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV:type=0, flags=1, stream_id=1,length=38 2025-05-31 00:15:13,268 - DEBUG - Sending stream 1to channel for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV ❌ CORRUPTION STARTS HERE

2025-05-31 00:15:13,270 - DEBUG - Received header for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV:type=47, flags=28021, stream_id=1819568499,length=1953654113 2025-05-31 00:15:13,270 - DEBUG - Received header for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV:type=47, flags=12590, stream_id=808333322,length=288319856 2025-05-31 00:15:13,271 - DEBUG - Received header for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV:type=115, flags=12144, stream_id=1768843055,length=825110574 2025-05-31 00:15:13,272 - DEBUG - Connection closed orincomplete header for peer 12D3KooWJS75TdNJJd24hWpdSGGQC387FjQeTnQ3c2uG1XoUreqV

Logs - Rust Side (rust-libp2p)

2025-05-30T23:15:13.256670Z DEBUG yamux::connection::rtt: sending ping 858059319
2025-05-30T23:15:13.259217Z DEBUG yamux::connection: fcc994e4: new outbound (Stream fcc994e4/1) of (Connection fcc994e4 Client (streams 0))
2025-05-30T23:15:13.260804Z DEBUG multistream_select::dialer_select: Dialer: Proposed protocol protocol=/ipfs/ping/1.0.0
2025-05-30T23:15:13.264582Z DEBUG multistream_select::negotiated: Negotiated: Received confirmation for protocol protocol=/yamux/1.0.0
2025-05-30T23:15:13.266716Z DEBUG yamux::connection::rtt: received pong 858059319, estimated round-trip-time 10.0389ms
2025-05-30T23:15:13.273402Z DEBUG Connection closed with error IO(Custom { kind: Other, error: Error(Right(Closed)) })

📊 Test Results Summary

Component	Status	Notes
Connection	✅ Pass	TCP connection established
Noise Protocol	✅ Pass	Security handshake successful
Yamux Setup	✅ Pass	Stream muxer negotiated
Initial Ping	✅ Pass	Connection-level ping works
Stream Creation	✅ Pass	New stream opened successfully
Frame Parsing	❌ Fail	Corruption after 2-3 frames
Protocol Negotiation	✅ Pass	/ipfs/ping/1.0.0 confirmed
Application Ping	⏸️ Blocked	Cannot test due to frame issue

Overall: Significant progress made, core interop proven feasible. One critical parsing bug remains.

lla-dane · 2025-05-31T12:21:33Z

Great, thanks @paschal533. I will read through the chnages in ping.py here and do the same thing for ping in interop tests in #620

paschal533 · 2025-05-31T12:23:40Z

Great, thanks @paschal533. I will read through the chnages in ping.py here and do the same thing for ping in interop tests in #620

Alright
Great work at your end too 👏
Keep me posted with your progress

seetadev · 2025-05-31T14:22:34Z

@lla-dane and @paschal533 : Great progress :) Appreciate the initiative.

Looking forward to seeing the interop test plans with rust-libp2p soon at https://github.com/libp2p/test-plans

seetadev · 2025-06-09T04:36:29Z

@lla-dane , @varun-r-mallya and @paschal533 : Wish to share that there are some merge conflicts in this branch that must be resolved. Please check examples/ping/ping.py

interop

…l stream handling

…ecting peer detection

paschal533 changed the title ~~WIP: Progress on py-libp2p and rust-libp2p interop for Noise and Yamux~~ WIP: Progress on py-libp2p and rust-libp2p interop for Ping Mar 6, 2025

paschal533 mentioned this pull request Mar 7, 2025

Replace Deprecated mplex with yamux for Improved Interoperability #534

Open

seetadev marked this pull request as ready for review March 10, 2025 22:30

pacrob mentioned this pull request May 8, 2025

delete old interop, turn on with placeholders #588

Merged

3 tasks

paschal533 closed this May 23, 2025

paschal533 force-pushed the feat/interop-test-ping branch from 2099b1a to 386f0a2 Compare May 23, 2025 09:42

paschal533 added 2 commits May 31, 2025 12:25

feat: achieve ping interop py-libp2p - rust-libp2p

1d9849c

WORKING: Connection, handshake, Yamux setup, initial ping/pong. ISSUE: Frame parser corruption after 2-3 frames (boundary sync)

fix: pre-commit issues

4b331d9

paschal533 reopened this May 31, 2025

lla-dane mentioned this pull request Jun 6, 2025

WIP: Ping interop in py-libp2p and rust-libp2p #620

Open

paschal533 closed this Jun 8, 2025

seetadev reopened this Jun 9, 2025

paschal533 and others added 6 commits June 9, 2025 12:38

feat: update ping.py with recent changes

8a9465c

fix linting issues

ec225ea

fix

376492b

basic host import

b3970e8

Merge pull request #6 from Shonen-Labs/fix

edba9c8

interop

Fix py-libp2p to py-libp2p ping interoperability by adding conditiona…

921be23

…l stream handling

Fix py-libp2p client interoperability with rust-libp2p server by corr…

6d0f0f3

…ecting peer detection

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Progress on py-libp2p and rust-libp2p interop for Ping #532

WIP: Progress on py-libp2p and rust-libp2p interop for Ping #532

Uh oh!

paschal533 commented Mar 6, 2025

Uh oh!

seetadev commented Mar 10, 2025 •

edited

Loading

Uh oh!

seetadev commented Mar 27, 2025

Uh oh!

paschal533 commented Mar 31, 2025

Uh oh!

panditdhamdhere commented Apr 1, 2025

Uh oh!

lla-dane commented May 7, 2025

Uh oh!

seetadev commented May 7, 2025

Uh oh!

paschal533 commented May 8, 2025 •

edited

Loading

Uh oh!

paschal533 commented May 31, 2025

❌ CORRUPTION STARTS HERE

Uh oh!

lla-dane commented May 31, 2025 •

edited

Loading

Uh oh!

paschal533 commented May 31, 2025

Uh oh!

seetadev commented May 31, 2025

Uh oh!

seetadev commented Jun 9, 2025

Uh oh!

Uh oh!

WIP: Progress on py-libp2p and rust-libp2p interop for Ping #532

Are you sure you want to change the base?

WIP: Progress on py-libp2p and rust-libp2p interop for Ping #532

Uh oh!

Conversation

paschal533 commented Mar 6, 2025

Description

Changes

Current Status

Logs

Request for Feedback

Notes

Uh oh!

seetadev commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seetadev commented Mar 27, 2025

Uh oh!

paschal533 commented Mar 31, 2025

Uh oh!

panditdhamdhere commented Apr 1, 2025

Uh oh!

lla-dane commented May 7, 2025

Uh oh!

seetadev commented May 7, 2025

Uh oh!

paschal533 commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paschal533 commented May 31, 2025

Ping Interoperability Progress Report

✅ SUCCESS: Basic Ping Interop Working

What's Working:

⚠️ CURRENT ISSUE: Frame Parsing Corruption After Initial Success

Error Details

Logs - Python Side (py-libp2p)

❌ CORRUPTION STARTS HERE

Logs - Rust Side (rust-libp2p)

📊 Test Results Summary

Uh oh!

lla-dane commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paschal533 commented May 31, 2025

Uh oh!

seetadev commented May 31, 2025

Uh oh!

seetadev commented Jun 9, 2025

Uh oh!

Uh oh!

seetadev commented Mar 10, 2025 •

edited

Loading

paschal533 commented May 8, 2025 •

edited

Loading

lla-dane commented May 31, 2025 •

edited

Loading