[docker_registry] Turn off Nginx buffering to improve docker push speed; RFC - #1565

This sounds like a great idea, but I'm also a bit wary about changing these (supposedly carefully-selected) defaults. Maybe you can elaborate this a bit:

1. Why does nginx use buffering on proxy requests and responses, and could disabling hurt performance/security/etc?
2. If storage speed is the only issue here, why not just limit or disable buffering to disk with proxy_max_temp_file_size?

I will test this later. I posted this on the ML so see if people on their can comment on this.

I revisited this PR today and read the ngx_http_proxy_module documentation in an attempt to understand how buffering works in nginx, and if these changes would indeed solve some issues without causing others. I've summarised things below; feel free to comment and point out any mistakes I might've made.

Let's say that we have a typical reverse proxy setup with two connections:

1. Between client and proxy
2. Between proxy and back-end server

Data can of course flow both ways, i.e. client -> proxy -> server and server -> proxy -> client.

Now, if both connections are equally fast, then we can turn off buffering in order to speed things up a little. The proxy just passes HTTP packets on as soon as it receives them, which is obviously faster than holding them in a buffer for some time. In reality though, the connection between the client and the proxy will be slower than the connection between the proxy and the server.

Let's say that we don't do any buffering at all. If the proxy streams a large blob towards the client, it will have to keep the connection to the server open until all HTTP packets have travelled from the server, through the proxy, to the client. The same is true for clients uploading large blobs. In other words, buffering is useful for freeing up back-end resources.

If we keep proxy_buffering on, then nginx will buffer data that's flowing from server to client. If we keep proxy_request_buffering on, then nginx will buffer data that's flowing from client to server. Ideally we keep both options on, and prevent the proxy from writing buffer data to disk. As @ypid points out, writing buffer data to disk happens when the request/response bodies don't fit in the regular RAM buffer, whose size is governed by the proxy_buffers option, which defaults to 8 x 4 KiB. It's probably safe to increase this a bit, but I find it hard to set a good default. I imagine that tweaking this the wrong way can cause a DoS risk, for example.

An easy way to prevent buffer data from being written to disk is by setting proxy_max_temp_file_size to zero. My guess is that nginx will then only buffer 8 x 4 KiB in RAM and pass on the rest directly. This would effectively buffer small HTTP bodies, while big blobs are passed on without buffering. This might be a nice compromise, however, we need to test this.

The nginx documentation is a bit vague regarding whether options like proxy_buffers and proxy_max_temp_file_size are only related to response buffering or to request buffering as well. My guess is that they also apply to request buffering if proxy_request_buffering is on. We should test this.

I'll try to find some time for testing this in the next two weeks. @ypid, did you already do some testing yourself?

That is very detailed and a better read than the Nginx docs when I last checked them out. Just a small pointer, this PR is only about changing buffering for docker-registry. I am just mentioning that go avoid misunderstanding as you don’t mention docker-registry in your text.

I tested a bit using iotop:

proxy_max_temp_file_size 0;

Does what the docs say: It only affects responses.

So I am not sure how to do it. The small cache in RAM can stay if we can also disable caching to file for requests (docker pushes).

On a more generic note, docker-registry should be perfectly capable of being run without reverse proxy, so I am not sure we really need the caching. I am using:

proxy_request_buffering off;
proxy_max_temp_file_size 0;

now.