Skip to content

Add server GUnzip middleware for request decompression - #7631

Open
bio-aeon wants to merge 12 commits into
http4s:series/0.23from
bio-aeon:gunzip-middleware
Open

bio-aeon wants to merge 12 commits into
http4s:series/0.23from
bio-aeon:gunzip-middleware

Conversation

@bio-aeon

Copy link
Copy Markdown
Contributor

This PR is a continuation of #7119, since I didn't get any response to my comment there during 1.5 months, but at the same time I'm also interested in having such decompression functionality.

@matkob feel free to leave a comment if you disagree with my changes for any reason.

What has been changed compared to #7119:

  1. Asking for a LoggerFactory doesn't seem feasible without breaking binary compatibility in GZip, so I extracted request decompression to a separate GUnzip middleware.
  2. In GUnzip middleware logger is being created via LoggerFactory and we don't need to call .unsafeRunSync().
  3. MalformedMessageBodyFailure is raised when request body isn't in valid gzip format (and the test is added for this case).

@mergify mergify Bot added series/0.23 PRs targeting 0.23.x module:server docs Relates to our website or tutorials labels Feb 11, 2025
@armanbilge
armanbilge requested a review from a team February 18, 2025 12:47
}
val response = req
.removeHeader[`Content-Length`]
.removeHeader[`Content-Encoding`]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to add TransferEncoding chunked here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you are right, since we are removing Content-Length. Updated.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hamnis Is there anything else I should adjust or does it look fine to you?

.removeHeader[`Content-Encoding`]
.putHeaders(`Transfer-Encoding`(TransferCoding.chunked))
.pipeBodyThrough(decompressPipe)
logger.trace("GUnzip middleware decoding content").as(unzippedRequest)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this log worth the hassle of propagating the log4cats.LoggerFactory -> log4cats.Logger into API? We won't lose many if we remove it. Alternatively, if we badly want it, let's use org.http4s.Platform#loggerFactory which is spread across the codebase already.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using org.http4s.Platform#loggerFactory, as I understood from the discussion in the original PR, this is considered an anti-pattterns and we don't want to use it in new code in favor of org.typelevel.log4cats.LoggerFactory.
Yes, of course we can just remove logging from this middleware at all, but I see a couple of minor issues with such a way:

  • We lose an opportunity of investigation by looking at order of middlewares application in logs.
  • In the future we won't be able to add logging even if necessary without breaking binary compatibility.


def apply[F[_]: Monad: log4cats.LoggerFactory, G[_]: ApplicativeThrow: Compression](
http: Http[F, G],
bufferSize: Int = 32 * 1024,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move this const into a final val with proper naming within GUnzip.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it'd be nice to tweak the org.http4s.server.middleware.GZip#apply to use a single default val.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done both for GUnzip and GZip.

Comment thread server/shared/src/main/scala/org/http4s/server/middleware/GUnzip.scala Outdated

import java.util.Arrays

class GUnzipSuite extends Http4sSuite {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a test for a "gzip bomb" request, such a request with a body of "a" repeated for a few gigabytes but compressed into something tiny?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please take a look if this is what you had in mind?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite, I would like to see that it returns "request too large" or something similar if the request sent is small (and has a similarly small content-length header), but the decompressed body is much, much larger, and exceeds withMaxRequestLength - I just took that from Armeria, but there should be something similar for other middlewares or requests, I guess? I haven't used http4s in a while, so I don't remember where to look.

Your test is still useful to verify that this middleware lazily decodes potentially infinite request bodies in a streaming manner, but the story will be different with Entity.Strict in http4s 1.x: https://http4s.org/v1/docs/entity.html#why-entity

It is less of an issue when all request bodies are streaming, but could still open up for smuggling in requests that end up being much larger than they should be allowed to be.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Http4s has the separate EntityLimiter middleware for entity size control. I've added one more test for GUnzip + EntityLimiter combination.
What about Entity.Strict in Http4s 1.x, I'm not sure that this will be a case, since GUnzip adds the Transfer-Encoding: chunked header, so I think such a request should be treated as a streaming one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, I'm a bit stuck figuring out why tests are failing for rootJS and rootNative, while they succeed for JVM. I'll write an update when I unlock this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reworked the gzip bomb test. The test compresses 2 MiB of zeros (which gzip shrinks to ~2 KB) and sets an EntityLimiter of 1 MiB. I believe this is enough to demonstrate the required scenario: the compressed payload is tiny, but decompression produces a body that exceeds the configured limit, resulting in EntityTooLarge. I dropped the approach with gigabytes-scale payload because it was causing CI timeouts on JS/Native.

.withBodyStream(
Stream
.emits(request.getBytes())
.repeatN(1024L * 1024L * 1024L)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really large, JS/Native are a bit slower, that's why they are timing out and failing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, thanks! Replaced with a 2 MiB compressed-zeros payload that exercises the same path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Relates to our website or tutorials module:server series/0.23 PRs targeting 0.23.x

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants