Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a way to enable checksums #43

Open
bobrik opened this issue Oct 1, 2018 · 1 comment
Open

Add a way to enable checksums #43

bobrik opened this issue Oct 1, 2018 · 1 comment
Labels
enhancement help wanted

Comments

@bobrik
Copy link

bobrik commented Oct 1, 2018

What did you do?

I compressed an empty file with zstd:

$ touch empty

$ zstd empty
empty                :1300.00%   (     0 =>     13 bytes, empty.zst)

Then I compressed an empty slice in Go:

package main

import (
	"log"

	"github.com/DataDog/zstd"
)

func main() {
	input := []byte{}

	output, err := zstd.CompressLevel(nil, input, 0)
	if err != nil {
		log.Fatalf("Error compressing: %s", err)
	}

	log.Printf("input      [len = %d]: %#v", len(input), input)
	log.Printf("compressed [len = %d]: %#v", len(output), output)
}

What did you expect to see?

Both outputs are the same.

What did you see instead?

13 bytes in zstd cli output:

$ ls -la empty*
-rw-r--r--  1 bobrik  wheel   0 Oct  1 11:07 empty
-rw-r--r--  1 bobrik  wheel  13 Oct  1 11:07 empty.zst

$ cat empty.zst | hexdump -C
00000000  28 b5 2f fd 24 00 01 00  00 99 e9 d8 51           |(./.$.......Q|
0000000d

9 bytes in Go output:

$ go run zstd.go
2018/10/01 11:09:09 input      [len = 0]: []byte{}
2018/10/01 11:09:09 compressed [len = 9]: []byte{0x28, 0xb5, 0x2f, 0xfd, 0x20, 0x0, 0x1, 0x0, 0x0}

Last 4 bytes are missing, which means no optional checksum:

Checksums are great and I think there should be a way to enable them.
@bobrik bobrik mentioned this issue Oct 1, 2018
Merged
@Viq111
Copy link
Collaborator

Viq111 commented Oct 1, 2018

Yes, if you look at the zstd cli, it enables the checksum via the zstd context

Currently this library uses the standard ZSTD_compress methods and not the ZSTD_compressCCtx ones.
In zstd source code:

Marking this an improvement as we would need to migrate the cgo calls to those methods instead
If you want to do a PR for this, it would be of great help!

@Viq111 Viq111 mentioned this issue Apr 29, 2019
Closed
@Viq111 Viq111 mentioned this issue Aug 12, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bobrik @Viq111