Skip to content

[redis]: Use Readline to read large value#471

Merged
yesnault merged 2 commits into
ovh:masterfrom
marcaudefroy:fix_redis_executor_when_large_value
Dec 21, 2021
Merged

[redis]: Use Readline to read large value#471
yesnault merged 2 commits into
ovh:masterfrom
marcaudefroy:fix_redis_executor_when_large_value

Conversation

@marcaudefroy
Copy link
Copy Markdown
Collaborator

@marcaudefroy marcaudefroy commented Dec 20, 2021

By default, Scanner uses a buffer with 64K capacity. So when we try to set a huge redis value (>64K), executor failed to read file. The fix consists to use Readline method. (source https://devmarkpro.com/working-big-files-golang).

Before :

 ./venom run ./../../tests/redis.yml
 • Redis testsuite (./../../tests/redis.yml)
 	• Commands_Test_Case SUCCESS
 	• File_Test_Case SUCCESS
 	• Long_Value_File_Test_Case FAILURE
Testcase "Long_Value_File_Test_Case", step #1: Failed to load file: bufio.Scanner: token too long (./../../tests/redis.yml:50)
 	• Commands_Nested_Array_Response_Test_Case SUCCESS
	  [info] streamEntry1ID: OK (./../../tests/redis.yml:70) (./../../tests/redis.yml:70)
	  [info] streamEntry2ID: 1639752707532-1 (./../../tests/redis.yml:72) (./../../tests/redis.yml:72)

After :

   ./venom  run -v ./../../tests/redis.yml
	  [trac] writing venom.log
 • Redis testsuite (./../../tests/redis.yml)
 	• Commands_Test_Case SUCCESS
 	• File_Test_Case SUCCESS
 	• Long_Value_File_Test_Case SUCCESS
 	• Commands_Nested_Array_Response_Test_Case SUCCESS
	  [info] streamEntry1ID: OK (./../../tests/redis.yml:70) (./../../tests/redis.yml:70)
	  [info] streamEntry2ID: 1639753471610-1 (./../../tests/redis.yml:72) (./../../tests/redis.yml:72)

@marcaudefroy
[redis]: Use File Stat to determine the size of scanner buffer
2e0eb2f 
Signed-off-by: marc audefroy <marc.audefroy@corp.ovh.com>
@marcaudefroy marcaudefroy changed the title [redis]: Use File Stat to determine the size of scanner buffer [redis]: Use Readline to read large value Dec 20, 2021
@marcaudefroy
[redis] Fix global value dialURL in README
9515e29 
Signed-off-by: marc audefroy <marc.audefroy@corp.ovh.com>
@marcaudefroy marcaudefroy force-pushed the fix_redis_executor_when_large_value branch from cc4cdef to 9515e29 Compare December 20, 2021 09:36
@yesnault yesnault merged commit eeb7e4c into ovh:master Dec 21, 2021
storagebits pushed a commit to storagebits/venom that referenced this pull request Apr 17, 2026
@marcaudefroy
[redis]: Use Readline to read large value (ovh#471)
73c31ba 
* [redis]: Use File Stat to determine the size of scanner buffer

Signed-off-by: marc audefroy <marc.audefroy@corp.ovh.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yesnault yesnault yesnault approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@marcaudefroy @yesnault