Add hardlink support #1954
Add hardlink support #1954
Conversation
ChengyuZhu6
force-pushed
the
hardlink
branch
3 times, most recently
from
ff5ecf3 to
3e921b6
Compare
cache/cache.go
Outdated
| if linkPath, exists := dc.hlManager.GetLink(key); exists { | ||
| if r, err := os.Open(linkPath); err == nil { |
There was a problem hiding this comment.
What is the performance benefit comparing to just doing os.Open(dc.cachePath(key))?
There was a problem hiding this comment.
That's not related to the os.Open function. However, I think it's a good point, and I can reuse the dc.cachePath(key) function.
There was a problem hiding this comment.
Got your point. I'll refactor my code.
ChengyuZhu6
force-pushed
the
hardlink
branch
2 times, most recently
from
b879c95 to
457b45b
Compare
ChengyuZhu6
force-pushed
the
hardlink
branch
2 times, most recently
from
8829e66 to
92702af
Compare
|
Needs rebase |
ChengyuZhu6
force-pushed
the
hardlink
branch
from
92702af to
1d5fd08
Compare
Done. |
ChengyuZhu6
force-pushed
the
hardlink
branch
from
1d5fd08 to
8694c9a
Compare
|
Just retrigger ci, no code change. |
|
@ktock I conducted experiments with several basic images, converting them to the estargz format and running them in containers with a simple 'echo "hello"' command. These tests used only background threads of stargz to pull images to the local machine. By measuring the overall memory and disk usage, I observed that implementing hardlinks resulted in a 20-30% reduction in both memory consumption and disk space requirements. 
|
|
When working with different versions of an image, implementing hardlinks can achieve a memory and disk deduplication effect of nearly 50%. I verified this by conducting tests with various development versions of Golang images. 
|
|
kindly ping @ktock |
ChengyuZhu6
force-pushed
the
hardlink
branch
3 times, most recently
from
efbcc75 to
01b70ae
Compare
This commit adds a hardlink system for the Stargz Snapshotter cache to optimize storage and improve performance. The system intelligently creates hardlinks between identical content chunks, significantly reducing disk space usage in environments with many containers using the same base layers. Key changes: - Add new HardlinkManager that tracks files by chunk digest - Enable hardlinking between chunk files with same content - Add configuration option `EnableHardlink` to control the feature - Preserve file digest mapping across snapshotter restarts - Add documentation on hardlink usage and configuration The implementation includes: - Chunk-level digest tracking for optimizing cache lookups - Background persistence of hardlink mappings to survive restarts - Automatic cleanup of unused digest mappings - Test suite for hardlink functionality Signed-off-by: ChengyuZhu6 <[email protected]>
Propose the implementation of a hardlink feature in the caching mechanism to optimize memory usage, improve performance and save disk space.
Fixes: #1953