BNB Chain nodes: Some nodes can send transactions while others fail with "indexing in progress" error

[h6dworker]

System information

• Geth version: v1.5.12
• OS & Version: Linux (Ubuntu 22.04)
• Commit hash: d60167c

Expected behaviour

All nodes with similar configuration should have the same transaction capability - either all should be able to send transactions or none should.

Actual behaviour

Some nodes can successfully send transactions while others fail with "transaction indexing is in progress" error, despite all nodes appearing to sync properly with the same configuration.

Details

/opt/bnb/geth --config /opt/bnb/config.toml --cache 16384 --rpc.allow-unprotected-txs --history.transactions=0 --syncmode=full --tries-verify-mode=none --pruneancient --rpc.enabledeprecatedpersonal --db.engine=pebble --datadir /opt/bnb/node

Transaction index progress is only ~2.6% on all nodes (1,294,933 indexed blocks, 48,634,071 remaining).

Steps to reproduce the behaviour

1. Set up multiple BNB Chain nodes with the configuration options listed above
2. Verify all nodes are syncing (same eth.syncing output)
3. Attempt to send transactions from each node
4. Some nodes succeed while others fail with the error below

Backtrace

> eth.syncing
{
  currentBlock: 49929003,
  healedBytecodeBytes: 0,
  healedBytecodes: 0,
  healedTrienodeBytes: 0,
  healedTrienodes: 0,
  healingBytecode: 0,
  healingTrienodes: 0,
  highestBlock: 49816655,
  startingBlock: 49816653,
  syncedAccountBytes: 0,
  syncedAccounts: 0,
  syncedBytecodeBytes: 0,
  syncedBytecodes: 0,
  syncedStorage: 0,
  syncedStorageBytes: 0,
  txIndexFinishedBlocks: 1294933,
  txIndexRemainingBlocks: 48634071
}

// Log messages on all nodes (both working and failing):
lvl=warn msg="Failed to decode block body" block=48634070 error=EOF
lvl=warn msg="Failed to decode block body" block=48634069 error=EOF
lvl=warn msg="Failed to decode block body" block=48634068 error=EOF

// Error on failing nodes:
Error occurred:
Failed to check for transaction receipt:
{
  "data": "transaction indexing is in progress"
}

[h6dworker]

BNB Chain Node Transaction Capability Detection Issue
Problem: Unable to determine transaction sending capability for BNB Chain nodes using standard monitoring methods.
Current Status:

eth.syncing never returns false even when transactions work
eth_getTransactionReceipt test is unreliable (works on some non-functional nodes)
Hardware performance shows no correlation with transaction capability

Expected: Clear API method to detect when a BNB Chain node can process new transactions
Actual: No reliable indicator exists - only actual transaction testing confirms capability

Latest(v1.5.13) transactions can be sent successfully

Impact: Production monitoring is impossible - no way to programmatically verify node transaction capability without actual transaction attempts

[emailtovamos]

@h6dworker
Does it happen with the latest bsc?
There were upstream bug fixes picked up in v1.5.13.

[h6dworker]

With v1.5.13, transactions work now but eth.syncing still never returns false.
Is this expected behavior for BNB Chain?

[emailtovamos]

Since the block times are lower, the current block is likely going to be a bit behind the highest block.
In the fully synced state the difference might be non-zero but very close to zero. So if it is very close to zero AND txIndexRemainingBlocks is 0 then likely scenario is that the syncing is basically done and transaction can be processed.

[h6dworker]

Thanks for the clarification!

However, my current state shows currentBlock: 51303548 > highestBlock: 51272400 (diff: 31,148) and txIndexRemainingBlocks: 50,151,285, which doesn't match the "very close to zero" criteria you mentioned. Yet transactions work fine.
Is there perhaps a different threshold or condition for BNB Chain's transaction capability that doesn't require complete indexing?

[emailtovamos]

The header downloader and body-importer run independently. So if there is a difference in the rate at which these happen then you can see currentBlock > highestBlock

Is there perhaps a different threshold or condition for BNB Chain's transaction capability that doesn't require complete indexing?

Currently when you send a transaction there is no waiting for indexing. It is directly sent to txpool (after basic checks) where it is then addressed. Acceptance into the local pool triggers gossip, and the transaction propagates outward until it lands in a validator’s pool and gets mined.
The miner/validator is supposed to include a transaction in a block so for him the currentBlock being up to date matters more.

[h6dworker]

Thanks for the detailed explanation about the txpool mechanism! That really helps me understand how BNB Chain handles transactions.

Based on your explanation and my testing, it seems like monitoring currentBlock progression is more important than waiting for complete indexing.

Is there a recommended way to monitor node health for transaction capability? Currently I'm checking the gap between eth.blockNumber and currentBlock from eth.syncing.

[krakovia-evm]

Getting same error on the ws mempool subscription