Refactor Staged Stream Sync Long Range #4824

GheisMohammadi · 2024-12-30T11:55:31Z

Issue

This PR refactors Staged Stream Sync Long Range stage. It adds BlockHashes stage to Staged Stream Sync. By calculating the correct hash first, this ensures data integrity and reliability in the block download process.

sophoah · 2025-01-09T09:47:33Z

is it possible to write test in this PR as well ?

GheisMohammadi · 2025-01-10T06:58:12Z

is it possible to write test in this PR as well ?

done

sophoah · 2025-01-10T09:16:36Z

@GheisMohammadi for long range how many block behind are we talking about

GheisMohammadi · 2025-01-10T09:23:59Z

@GheisMohammadi for long range how many block behind are we talking about

The node starts in long-range sync mode by default. Once it catches up, it switches to short-range sync.

GheisMohammadi · 2025-01-10T09:26:14Z

@GheisMohammadi for long range how many block behind are we talking about

The node starts in long-range sync mode by default. Once it catches up, it switches to short-range sync.

Also, I am currently working on a PR that dynamically switches between short-range and long-range sync based on the block gap.

sophoah · 2025-01-23T04:59:55Z

@GheisMohammadi did you test the issue #4826 (comment) ?

GheisMohammadi · 2025-01-23T05:48:51Z

@GheisMohammadi did you test the issue #4826 (comment) ?

in progress

…functionalities

…rted block results

…ed stream sync

sophoah · 2025-01-24T07:59:48Z

@GheisMohammadi i am still facing panic when i try to restart the harmony process that I killed just previously

My Genesis Account: one1pf75h0t4am90z8uv3y0dgunfqp4lj8wr3t5rsp/e751ec995defe4931273aaebcb2cd14bf37e629c554a57d3f334c37881a34a6188a93e76113c55ef3481da23b7d7ab09:0
Legacy mode; node key e751ec995defe4931273aaebcb2cd14bf37e629c554a57d3f334c37881a34a6188a93e76113c55ef3481da23b7d7ab09; -> shard 0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x112887e]

goroutine 1 [running]:
github.com/harmony-one/harmony/core.(*StateProcessor).Process(_, _, _, {0x0, {0x0, 0x0}, 0x0, 0x0, {{0x0, 0x0, ...}, ...}, ...}, ...)
        /home/soph/harmony/harmony2/core/state_processor.go:110 +0x9e
github.com/harmony-one/harmony/core.(*BlockChainImpl).insertChain(0xc000f26008, {0xc00089af50, 0x1, 0x1}, 0x1)
        /home/soph/harmony/harmony2/core/blockchain_impl.go:1860 +0xc56
github.com/harmony-one/harmony/core.(*BlockChainImpl).repairRecreateStateTries(0xc000f26008, 0xc00089b2d8)
        /home/soph/harmony/harmony2/core/blockchain_impl.go:800 +0x72
github.com/harmony-one/harmony/core.(*BlockChainImpl).repair(0xc000f26008, 0xc00089b2d8)
        /home/soph/harmony/harmony2/core/blockchain_impl.go:820 +0xa5
github.com/harmony-one/harmony/core.(*BlockChainImpl).loadLastState(0xc000f26008)
        /home/soph/harmony/harmony2/core/blockchain_impl.go:609 +0x116b
github.com/harmony-one/harmony/core.newBlockChainWithOptions({_, _}, {_, _}, {_, _}, _, _, {0x2d26918, 0xc00064f920}, ...)
        /home/soph/harmony/harmony2/core/blockchain_impl.go:331 +0x9b3
github.com/harmony-one/harmony/core.NewBlockChainWithOptions(...)
        /home/soph/harmony/harmony2/core/blockchain_impl.go:237
github.com/harmony-one/harmony/internal/shardchain.(*CollectionImpl).ShardChain(0xc0007da240, 0x0, {0x0, 0x0, 0x2d1a2c8?})
        /home/soph/harmony/harmony2/internal/shardchain/shardchains.go:147 +0x765
main.setupChain({{0x25e780e, 0x5}, {{0x25efa8f, 0x9}, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
        /home/soph/harmony/harmony2/cmd/harmony/main.go:695 +0x7c7
main.setupConsensusAndNode({{0x25e780e, 0x5}, {{0x25efa8f, 0x9}, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
        /home/soph/harmony/harmony2/cmd/harmony/main.go:735 +0x347
main.setupNodeAndRun({{0x25e780e, 0x5}, {{0x25efa8f, 0x9}, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...})
        /home/soph/harmony/harmony2/cmd/harmony/main.go:247 +0x389
main.runHarmonyNode(0x3fa57a0, {0x25e6aef?, 0x4?, 0x25e68d7?})
        /home/soph/harmony/harmony2/cmd/harmony/main.go:127 +0x205
github.com/spf13/cobra.(*Command).execute(0x3fa57a0, {0xc000052258, 0x22, 0x22})
        /home/soph/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x69b
github.com/spf13/cobra.(*Command).ExecuteC(0x3fa57a0)
        /home/soph/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x38d
github.com/spf13/cobra.(*Command).Execute(...)
        /home/soph/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /home/soph/harmony/harmony2/cmd/harmony/main.go:107 +0x1a

[1]+  Exit 2

GheisMohammadi · 2025-01-24T08:04:06Z

@GheisMohammadi i am still facing panic when i try to restart the harmony process that I killed just previously


My Genesis Account: one1pf75h0t4am90z8uv3y0dgunfqp4lj8wr3t5rsp/e751ec995defe4931273aaebcb2cd14bf37e629c554a57d3f334c37881a34a6188a93e76113c55ef3481da23b7d7ab09:0

Legacy mode; node key e751ec995defe4931273aaebcb2cd14bf37e629c554a57d3f334c37881a34a6188a93e76113c55ef3481da23b7d7ab09; -> shard 0

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x112887e]



goroutine 1 [running]:

github.com/harmony-one/harmony/core.(*StateProcessor).Process(_, _, _, {0x0, {0x0, 0x0}, 0x0, 0x0, {{0x0, 0x0, ...}, ...}, ...}, ...)

        /home/soph/harmony/harmony2/core/state_processor.go:110 +0x9e

github.com/harmony-one/harmony/core.(*BlockChainImpl).insertChain(0xc000f26008, {0xc00089af50, 0x1, 0x1}, 0x1)

        /home/soph/harmony/harmony2/core/blockchain_impl.go:1860 +0xc56

github.com/harmony-one/harmony/core.(*BlockChainImpl).repairRecreateStateTries(0xc000f26008, 0xc00089b2d8)

        /home/soph/harmony/harmony2/core/blockchain_impl.go:800 +0x72

github.com/harmony-one/harmony/core.(*BlockChainImpl).repair(0xc000f26008, 0xc00089b2d8)

        /home/soph/harmony/harmony2/core/blockchain_impl.go:820 +0xa5

github.com/harmony-one/harmony/core.(*BlockChainImpl).loadLastState(0xc000f26008)

        /home/soph/harmony/harmony2/core/blockchain_impl.go:609 +0x116b

github.com/harmony-one/harmony/core.newBlockChainWithOptions({_, _}, {_, _}, {_, _}, _, _, {0x2d26918, 0xc00064f920}, ...)

        /home/soph/harmony/harmony2/core/blockchain_impl.go:331 +0x9b3

github.com/harmony-one/harmony/core.NewBlockChainWithOptions(...)

        /home/soph/harmony/harmony2/core/blockchain_impl.go:237

github.com/harmony-one/harmony/internal/shardchain.(*CollectionImpl).ShardChain(0xc0007da240, 0x0, {0x0, 0x0, 0x2d1a2c8?})

        /home/soph/harmony/harmony2/internal/shardchain/shardchains.go:147 +0x765

main.setupChain({{0x25e780e, 0x5}, {{0x25efa8f, 0x9}, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)

        /home/soph/harmony/harmony2/cmd/harmony/main.go:695 +0x7c7

main.setupConsensusAndNode({{0x25e780e, 0x5}, {{0x25efa8f, 0x9}, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)

        /home/soph/harmony/harmony2/cmd/harmony/main.go:735 +0x347

main.setupNodeAndRun({{0x25e780e, 0x5}, {{0x25efa8f, 0x9}, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...}, ...})

        /home/soph/harmony/harmony2/cmd/harmony/main.go:247 +0x389

main.runHarmonyNode(0x3fa57a0, {0x25e6aef?, 0x4?, 0x25e68d7?})

        /home/soph/harmony/harmony2/cmd/harmony/main.go:127 +0x205

github.com/spf13/cobra.(*Command).execute(0x3fa57a0, {0xc000052258, 0x22, 0x22})

        /home/soph/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x69b

github.com/spf13/cobra.(*Command).ExecuteC(0x3fa57a0)

        /home/soph/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x38d

github.com/spf13/cobra.(*Command).Execute(...)

        /home/soph/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918

main.main()

        /home/soph/harmony/harmony2/cmd/harmony/main.go:107 +0x1a



[1]+  Exit 2

It seems it is not related to stream sync. Based on the logs, it happened even before sync module gets started on the setup chain.

sophoah · 2025-01-24T08:11:22Z

ok i just tested the same from the dev branch and seems we are also facing the same issue

GheisMohammadi self-assigned this Dec 30, 2024

GheisMohammadi marked this pull request as draft December 30, 2024 11:55

GheisMohammadi requested review from Frozen, mur-me and sophoah January 6, 2025 11:44

GheisMohammadi marked this pull request as ready for review January 6, 2025 11:44

Frozen approved these changes Jan 6, 2025

View reviewed changes

GheisMohammadi mentioned this pull request Jan 10, 2025

Enhance P2P Streams with Reserved Stream Replacement #4826

Merged

GheisMohammadi added 15 commits January 24, 2025 11:15

refactor download manager in staged stream sync

8e05122

rename block download manager to download manager and generalize its …

be3649e

…functionalities

add block hashes to list of stream sync stages

f03d97a

add block hashes stage to staged stream sync

ba6f577

optimize final block hash calculations, remove invalid streams

6f2865f

add new function to stream client for requesting blocks by hashes

5b41be5

refactor stage bodies to use hashes rather than block numbers

d1d7191

fix download manager issue of skipping blocks

047d69e

add comment and TODO to download manager and stage bodies to check so…

3734a1d

…rted block results

add contains function to prioritizedNumbers

8d93ecc

add tests for stream sync download manager

aea57f4

fix using internal transaction in stage bodies

c74880a

refactor save progress in stage block hashes

997e8fd

update logs identifiers to differentiate between staged sync and stag…

1ea865e

…ed stream sync

add block number to logs for re-download bad block function

70175de

save progress for last two stages; block bodies and states

ce62cf5

GheisMohammadi force-pushed the refactor/staged_stream_sync_long_range branch from 6c57170 to ce62cf5 Compare January 24, 2025 03:21

Frozen approved these changes Jan 24, 2025

View reviewed changes

sophoah approved these changes Jan 24, 2025

View reviewed changes

sophoah merged commit 1b4d625 into dev Jan 24, 2025
4 checks passed

sophoah deleted the refactor/staged_stream_sync_long_range branch January 24, 2025 08:31

mur-me mentioned this pull request May 29, 2025

Release Candidate 2025.1.1 (dev -> main) #4889

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor Staged Stream Sync Long Range #4824

Refactor Staged Stream Sync Long Range #4824

Uh oh!

GheisMohammadi commented Dec 30, 2024 •

edited

Loading

Uh oh!

sophoah commented Jan 9, 2025

Uh oh!

GheisMohammadi commented Jan 10, 2025

Uh oh!

sophoah commented Jan 10, 2025 •

edited by GheisMohammadi

Loading

Uh oh!

GheisMohammadi commented Jan 10, 2025 •

edited

Loading

Uh oh!

GheisMohammadi commented Jan 10, 2025

Uh oh!

sophoah commented Jan 23, 2025

Uh oh!

GheisMohammadi commented Jan 23, 2025

Uh oh!

sophoah commented Jan 24, 2025

Uh oh!

GheisMohammadi commented Jan 24, 2025

Uh oh!

sophoah commented Jan 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Refactor Staged Stream Sync Long Range #4824

Refactor Staged Stream Sync Long Range #4824

Uh oh!

Conversation

GheisMohammadi commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Uh oh!

sophoah commented Jan 9, 2025

Uh oh!

GheisMohammadi commented Jan 10, 2025

Uh oh!

sophoah commented Jan 10, 2025 • edited by GheisMohammadi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GheisMohammadi commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GheisMohammadi commented Jan 10, 2025

Uh oh!

sophoah commented Jan 23, 2025

Uh oh!

GheisMohammadi commented Jan 23, 2025

Uh oh!

sophoah commented Jan 24, 2025

Uh oh!

GheisMohammadi commented Jan 24, 2025

Uh oh!

sophoah commented Jan 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

GheisMohammadi commented Dec 30, 2024 •

edited

Loading

sophoah commented Jan 10, 2025 •

edited by GheisMohammadi

Loading

GheisMohammadi commented Jan 10, 2025 •

edited

Loading