Skip to content

Conversation

@carolynhu
Copy link
Contributor

@carolynhu carolynhu commented Aug 12, 2020

Here is the error log when we run the istio perf benchmark test:

2dac87db_qps_3000_c_16_1024_v2-stats-wasmbothsidecars_perf.data
kubectl --namespace twopods-istio exec fortioclient-69f64854bb-t7gfm  -- fortio load  -jitter=True -c 16 -qps 3000 -t 240s -a -r 0.00005   -httpbufferkb=128 -labels 2dac87db_qps_3000_c_16_1024_v2-stats-wasm_both http://fortioserver:8080/echo?size=1024
Defaulting container name to captured.
Use 'kubectl describe pod/fortioclient-69f64854bb-t7gfm -n twopods-istio' to see all of the containers in this pod.
Fortio 1.3.2-pre running at 3000 queries per second, 32->32 procs, for 4m0s: http://fortioserver:8080/echo?size=1024
13:22:30 I httprunner.go:82> Starting http test for http://fortioserver:8080/echo?size=1024 with 16 threads at 3000.0 qps
Starting at 3000 qps with 16 thread(s) [gomax 32] for 4m0s : 45000 calls each (total 720000)
panic: invalid argument to Int63n

goroutine 133 [running]:
math/rand.(*Rand).Int63n(0xc0000c6150, 0x0, 0x758cda975)
	/go1.11/go/src/math/rand/rand.go:111 +0x10d
math/rand.Int63n(0x0, 0xd73840)
	/go1.11/go/src/math/rand/rand.go:319 +0x37
fortio.org/fortio/periodic.getJitter(0x5, 0x2a8b4e6)
	/go/src/fortio.org/fortio/periodic/periodic.go:565 +0x66
fortio.org/fortio/periodic.runOne(0x2, 0xc00025a060, 0xc000394be0, 0xc000394c30, 0xafc8, 0xbfc47065a37708f9, 0x2a8b4e6, 0xd73840, 0xc0001d4500)
	/go/src/fortio.org/fortio/periodic/periodic.go:519 +0x96b
fortio.org/fortio/periodic.(*periodicRunner).Run.func1(0xc00025a060, 0xafc8, 0xbfc47065a37708f9, 0x2a8b4e6, 0xd73840, 0xc0001d4500, 0xc0002ec334, 0x2, 0xc000394be0, 0xc000394c30)
	/go/src/fortio.org/fortio/periodic/periodic.go:415 +0x8b
created by fortio.org/fortio/periodic.(*periodicRunner).Run
	/go/src/fortio.org/fortio/periodic/periodic.go:414 +0x4f1
command terminated with exit code 2

The root cause for this is that according to Golang math/rand source code: https://golang.org/src/math/rand/rand.go (line111) if the seed number is <= 0 it will panic.

Therefore, I set the minimum i value here to be int64(1) [duration type] when i <=0.

cc @ldemailly @mandarjog

@qlty-cloud-legacy
Copy link

qlty-cloud-legacy bot commented Aug 12, 2020

Code Climate has analyzed commit 58a0d65 and detected 0 issues on this pull request.

View more on Code Climate.

if t <= 0 {
return time.Duration(0)
}
i := int64(t / 10)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this probably should be something like

func getJitter(t time.Duration) time.Duration {)
   i:= int64(float64(t)/10.+ 0.5) // round to nearest instead of truncate
   if i <= 0 {
 	return time.Duration(0)
   }
   j := rand.Int63n(2*i) - i
   return time.Duration(j)
}

ie not test for 0 twice
(could also use math.Ceil and maybe use /5 and 2*i - i/2 instead of i/2 and -i) or use rand.Float64()...)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or could check for <= 5 (or 10) at the beginning

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe something like:

func getJitter(t time.Duration) time.Duration {
   if t < 10 {
      return time.Duration(0)
   }
   i:= int64(t/10)    # if t < 10 check to make sure i > 0
   j := rand.Int63n(2*i) - i
   return time.Duration(j)
}

do not add +0.5, since when float64(t)/10+ 0.5 is a constant, golang will throw constant xx.xx truncated to integercompile error, as the constant does not fit into the specified type (int64).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

odd, /10. should keep everything float until after the int64() but not a huge deal

I wonder if you can you add a test that recreates the panic before the fix and shows the fix but it may be tricky to get sleep to be < 10 yet > 0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird, i cannot reproduce the previous error i got. Yes, that looks fine. Will update!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I pass:

t := 6
i := int64(float64(t)/10 + 0.5)

or

t := float64(6)
i := int64(t/10 + 0.5)

it works

But when I pass:
i := int64(float64(6)/10 + 0.5)

got constant 1.1 truncated to integer

@ldemailly
Copy link
Member

also make the linter happy

go get mvdan.cc/gofumpt 
gofumpt -s -w  periodic/periodic.go

Copy link
Member

@ldemailly ldemailly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^

@ldemailly
Copy link
Member

ldemailly commented Aug 14, 2020

for lint, I should have specified, don't go get gofumt in the fortio directory as that is adding a dependency, please revert go.sum and go.mod changes, and run the formatter

also in general avoid force pushing a pr after review has started (unless you accidentally commit unrelated stuff) because it makes it hard for reviewing what changed

@ldemailly
Copy link
Member

unit test?

@ldemailly
Copy link
Member

ldemailly commented Aug 14, 2020

for the build failure:

periodic/periodic.go:563:30: mnd: Magic number: 0.5, in <argument> detected (gomnd)
	i := int64(float64(t)/10. + 0.5) // round to nearest instead of truncate
	                            ^

change the comment to be // nolint: gomnd // rounding to nearest instead of truncate

also you can run golangci-lint locally (brew install golangci-lint if on mac) and/or in your ide (like vscode)

@carolynhu
Copy link
Contributor Author

carolynhu commented Aug 14, 2020

for lint, I should have specified, don't go get gofumt in the fortio directory as that is adding a dependency, please revert go.sum and go.mod changes, and run the formatter

also in general avoid force pushing a pr after review has started (unless you accidentally commit unrelated stuff) because it makes it hard for reviewing what changed

-reverted and will rerun the formatter, seems that nothing changes after I run linter, the issue is:

golangci-lint --concurrency=2 run ./...
periodic/periodic.go:563:30: mnd: Magic number: 0.5, in <argument> detected (gomnd)
	i := int64(float64(t)/10. + 0.5) // round to nearest instead of truncate
	                            ^
make: *** [Makefile:55: local-lint] Error 1
  • oops, i thought that was the last review, so i just want to squash it into one commit by forcing. sure, will keep it there.

unit test?

test against two branches: master (with fix) and the other (without fix)

Copy link
Member

@ldemailly ldemailly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@ldemailly
Copy link
Member

linters seems to have forgotten to show only new issues so merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants