Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: add pure Go build mode #94

Closed
valyala opened this issue Jul 11, 2019 · 4 comments
Closed

Feature: add pure Go build mode #94

valyala opened this issue Jul 11, 2019 · 4 comments
Assignees
@hagen1778
Labels
enhancement New feature or request

Comments

@valyala
Copy link
Collaborator

valyala commented Jul 11, 2019

The issue

VictoriaMetrics build requires cgo, since it uses valyala/gozstd wrapper for the upstream zstd library written in C. This complicates build process, especially for cross-compliation - it requires C compiler toolchain for the target architecture additionally to Go.

The solution

To add ability to use pure Go implementation for zstd - github.com/klauspost/compress/tree/master/zstd. This will simplify cross-compilation builds for various GOOS/GOARCH combinations.

See this thread on golang-nuts for more details.
@valyala valyala added the enhancement New feature or request label Jul 11, 2019
@hagen1778 hagen1778 self-assigned this Jul 13, 2019
@hagen1778 hagen1778 mentioned this issue Jul 20, 2019
Merged
valyala pushed a commit that referenced this issue Jul 23, 2019
@hagen1778 @valyala
all: add Pure Go build (pull request #114)
5bf4e5f 
Updates #94
valyala pushed a commit that referenced this issue Jul 24, 2019
@hagen1778 @valyala
all: add Pure Go build (pull request #114)
fcf09aa 
Updates #94
@valyala
Copy link
Collaborator Author

valyala commented Jul 24, 2019

pure Go build mode with https://github.com/klauspost/compress/tree/master/zstd has been implemented. It provides slightly lower compression ratio and slightly lower decompression performance comparing to cgo. See instructions on how to create GCO_ENABED=0 builds for VictoriaMetrics.

The pure Go mode may become default in the future when it will gain better compression ratio comparable to the gozstd.

@valyala valyala closed this as completed Jul 24, 2019
@valyala
Copy link
Collaborator Author

valyala commented Jul 24, 2019

@klauspost, is it possible to remove the allocation here? It looks like this allocation reduces the decompression speed for small blocks.

@klauspost
Copy link

@valyala 40 bytes per block? ok - I will take a look.

@klauspost
Copy link

@valyala Added here: klauspost/compress#142

Seems quite harmless so I will probably merge it later today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hagen1778 hagen1778

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@valyala @hagen1778 @klauspost