HTTPS support for secondary storage? #894

jrosdahl · 2021-07-12T19:24:14Z

jrosdahl
Jul 12, 2021
Maintainer

I kind of hoped not to have to start a discussion about HTTPS at this time by saying "Not in scope: HTTPS" in #858, but now when PR #890 already exists I guess I have to anyway. 🙂 The thing is that I'm not yet sure whether we actually should support HTTPS at all, so I would like to discuss our options first.

I have two concerns with adding support like in #890, i.e. just linking with OpenSSL and letting each ccache invocation start a new HTTPS connection:

Startup cost when not using HTTPS storage. If a user (or distributor) builds ccache with HTTPS support then all users will pay the cost of a slightly slower ccache startup. This is similar to my concerns about using libcurl. The cost is smaller for just OpenSSL, but clearly measurable. On my system, ccache -V goes from ≈1.5 ms to ≈2.9 ms with Add secondary HTTPS storage #890 (dynamic linking). With static linking of OpenSSL it goes down to ≈2.6 ms, which surprised me since I expected most of the slowdown to go away. That seems to be because libssl/libcrypto perform some initialization steps before main is reached. A couple of ms slower is probably acceptable for many people, but many small slowdowns can turn into a larger slowdown, so we have to compare this with the value the feature adds.
Connection cost when using HTTPS storage. On my system, an HTTPS connection to localhost has an overhead of ≈7 ms compared to HTTP. That overhead is mostly due the TLS handshake, and the time for round trips will be amplified when connecting to a server farther away than localhost. I guess this is sort of OK since it's opt-in to use HTTPS storage, but a more reasonable architecture would be to start a long-lived process that keeps a connection alive – and thus offsets the session setup cost – and let ccache proxy its calls via that local process.

Also: is HTTPS storage a valid use case? Do we know of anybody who wants to use HTTPS for ccache, especially in this one-connection-per-invocation way?

gjasny · 2021-07-13T05:45:52Z

gjasny
Jul 13, 2021
Collaborator

I mostly implemented HTTPS support because it could be used to protect the secrets in a HTTP Basic Authentication scenario. There might be industries which mandate encryption and authentication for storing build results.

We might be able to reduce the roundtrips needed for a handshake with TLS Session Resumption reducing ccache startup time would involve dynamic loading of either OpenSSL or a hypothetical ccache HTTPS plugin.

Maybe we could disable OpenSSL / HTTPS support by default and add a big warning sign above the CMake option to convince distributions to not enable it for the standard packages?

5 replies

jrosdahl Jul 13, 2021
Maintainer Author

We might be able to reduce the roundtrips needed for a handshake with TLS Session Resumption

Do you mean with the one-connection-per-invocation model? If so, where should ccache store the session parameters?

I have no measurements to back this up, but my gut feeling is that if a user needs encryption we need to have a long-lived process that keeps the connection alive to get any reasonable speed. (That would certainly help for other session-based network protocols as well.) That is, let ccache start a helper process (if not already started) and communicate with it over a Unix socket. What do you think about that model?

reducing ccache startup time would involve dynamic loading of either OpenSSL or a hypothetical ccache HTTPS plugin.

Right, dynamic plugins are what I envisioned in #414, partly to not have all users pay the cost of all different backends when not using them.

Maybe we could disable OpenSSL / HTTPS support by default and add a big warning sign above the CMake option to convince distributions to not enable it for the standard packages?

I would prefer if all bundled backends are expected to be enabled by default so that the configuration doesn't have to say "if enabled", but yeah, this is the easiest option. The idea of adding a big warning sign for enabling opt-in backends didn't occur to me before... Hmm. Yes, that's maybe good enough. Distributors can then provide a default ccache package and an alternative ccache-tls package with HTTPS and Redis TLS support, etc.

I would however prefer the helper process model, but it's of course a lot more work to get there.

afbjorklund Jul 16, 2021

Couldn't you just use a local web proxy (going from http to https), instead of inventing a new backend or package ?

jrosdahl Jul 17, 2021
Maintainer Author

Couldn't you just use a local web proxy (going from http to https)

I guess you mean letting the user start a local web proxy themselves and point ccache to it? I've tried to find a simple enough HTTP(S) forward proxy to recommend but didn't find anything. Do you have any suggestion? In any case, this model seems quite complicated for the user.

Or do you mean ccache starting a local off the shelf web proxy on demand?

instead of inventing a new backend or package ?

To clarify, I don't see this as inventing a new backend but to change the way ccache handles all its network backends.

afbjorklund Jul 18, 2021

I did mean an external process, like the ones that exist for memcached and redis...

I have only used the web proxies "the other way", to terminate https before http backend.
Mostly just apache or nginx, but I guess it is a hassle - especially this way, adding certs.

https://en.wikipedia.org/wiki/TLS_termination_proxy

Adding the SSL backends using dlopen sounded good otherwise, even if complicated.

That is, having https and rediss backends available by loading extra code not in "main"
Maybe "easiest" is the different ccache builds, like having a separate ccache-tls package.

afbjorklund Jul 18, 2021

I found some simple proxy, but I forgot the name of it... Will update with link when I find it, but haven't tried it.
There were lots of proxies that handled GET request, but not so many that would also handle PUT requests.

Maybe this: https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/

We will probably need some documentation on how to set up the web server anyway, with DAV support etc ?
I found the Bazel documentation on NGINX was OK, it discussed the packages needed and example config:

https://docs.bazel.build/versions/main/remote-caching.html#setting-up-a-server-as-the-caches-backend

afbjorklund · 2021-07-18T08:26:07Z

afbjorklund
Jul 18, 2021

Depending on the outcome of the discussion (about HTTPS), one can use hiredis_ssl to add a similar option for Redis.

The trade-offs are quite the same, and TLS support is quite new (it requires Redis 6): https://redis.io/topics/encryption

NOTE: requires hiredis version 1.0.0 (with ENABLE_SSL=ON, not the default OFF)

int redisInitiateSSLWithContext(redisContext *c, redisSSLContext *redis_ssl_ctx);

Some reports talk about a 36% performance drop (adding TLS), the single-threaded model makes it extra visible.,

There is also an increased CPU load on the server, normally it is very light (just sending from memory to network)

1 reply

afbjorklund Jul 18, 2021

For the memcached backend, TLS was ignored (~~and handled by Couchbase "moxi"~~).

It seems like the twemproxy "nutcracker" does not have ~~the same~~ TLS/SSL support.

For hostile networks, such as cloud, tunneling was used: (and UDP was disabled, only TCP)

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/security_guide/index#sect-Security_Guide-Available_Network_Services-Insecure_Services

Encrypt communication between memcached clients and servers with stunnel. Since memcached does not support TLS, a workaround is to use a proxy, such as stunnel, which provides TLS on top of the memcached protocol.

See https://redislabs.com/blog/stunnel-secure-redis-ssl/

Compare: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html

EDIT: I think I misremembered, it was only username/password that was added :-)
I don't think anyone considered encrypting internal traffic back then, "too expensive" ?

TLS should be available for memcached now (2019) though, again with similar tradeoffs:
https://github.com/memcached/memcached/blob/master/doc/tls.txt (also using OpenSSL)

afbjorklund · 2021-07-18T19:14:21Z

afbjorklund
Jul 18, 2021

Added a Redis SSL (hiredis_ssl) implementation in PR #902, matching the HTTPS implementation

Adding a test case requires "redis-server" and "libhiredis-dev" newer than what is available in Ubuntu...

2 replies

afbjorklund Jul 19, 2021

Even "redis-tools" (redis-cli) needs to be newer (1.0.0) and compiled with SSL support (BUILD_TLS=yes).

afbjorklund Jul 19, 2021

Maybe it would be possible to use brew also on Linux, then it can be tested using the same packaging as on the Mac

https://docs.brew.sh/Homebrew-on-Linux

$ brew install ninja pkg-config hiredis redis

afbjorklund · 2021-08-15T11:34:35Z

afbjorklund
Aug 15, 2021

Maybe this feature (SSL) can be added in 4.5 instead ?

2 replies

jrosdahl Aug 15, 2021
Maintainer Author

Maybe this feature (SSL) can be added in 4.5 instead ?

It's definitely out of scope for 4.4 and I don't have any immediate plans to work on it for 4.5 either.

afbjorklund Aug 15, 2021

I think I meant to say "after 4.4", as in "maybe later".

afbjorklund · 2022-11-02T20:26:41Z

afbjorklund
Nov 2, 2022

There are some suggestions to add new cloud storage backends:

The SDKs that they use are quite big, so probably OFF by default ?

0 replies

afbjorklund · 2022-11-02T22:06:07Z

afbjorklund
Nov 2, 2022

Here is an example of a project using plugins:

https://github.com/VgTajdd/plugin_manager

Some could be compiled in, some loaded ?

It would still need RPC, when actually used...

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPS support for secondary storage? #894

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

HTTPS support for secondary storage? #894

jrosdahl Jul 12, 2021 Maintainer

Replies: 6 comments · 10 replies

gjasny Jul 13, 2021 Collaborator

jrosdahl Jul 13, 2021 Maintainer Author

jrosdahl Jul 17, 2021 Maintainer Author

jrosdahl Aug 15, 2021 Maintainer Author

jrosdahl
Jul 12, 2021
Maintainer

Replies: 6 comments 10 replies

gjasny
Jul 13, 2021
Collaborator

jrosdahl Jul 13, 2021
Maintainer Author

jrosdahl Jul 17, 2021
Maintainer Author

jrosdahl Aug 15, 2021
Maintainer Author