1. We have lots of rewrite rules within a server block outside of location blocks.
2. The bypass routines in cases of known errors and/or rate limiting used relative paths in the config (internal redirects of sorts)

Our config had a bit of this shape:

http {
  server {
    server_name x.example.com localhost;
    listen 80;

    error_page  404 /404.html;
    error_page  403 /403.html;

    # hundreds of rewrite rules go here, intended to "normalize" matching URLs
    # rewrite {regex} ...
    # rewrite {regex} ...
    # rewrite {regex} ...
    # ... and so on

    location /404.html {
      root /srv/www/html;
    }

    location /403.html {
      root /srv/www/html;
    }

    location / {
      proxy_pass http://upstream;
    }
    # ... and so on
  }
}

In our case the configuration is spread over hundreds of files, and the rewrite rules shown here are all in a separate file which gets included right after the main server block configurations end and any location block definitions start.

A long time ago on the project a decision was made to add a file that contained a few URL matching rules that had to run in-order for every URL, before any location matching runs, sort of a pre-processor to “normalize” URLs.

This is okay as long as it doesn’t get abused. But slowly over time there were additions that should’ve simply been location blocks instead. For the uninitiated, location matching tends to be more efficient in matching URLs as nginx builds out a tree of these at startup rather than going at it serially one by one, which is what our problem technically became. This set of rules started growing as it became a kitchen-sink of sorts for every “wrong” URL—at the time of debugging the number of such rules were in the hundreds. These rules get executed multiple times if there are internal redirects; rewrite rules themselves can be internal redirects if they don’t use one of the bypassing flags like break, redirect, permanent, which further exacerbates the probem. This was true in our case since the intent was to run these in-order, which means last and break can’t be used by definition.

Secondly, we used relative paths for all the error_page configurations mostly as a carry-over from nginx configurations that are documented pretty much everywhere¹ . So when an error status is triggered nginx will redo the matching from the beginning. In isolation this is not a problem, and I can understand why the default documentation snippets use this pattern. In our case these two problems in combination create a cascading effect: when testing out our rate limiting and error handling checks, which should’ve bypassed the relatively-costly location matching, every rewrite rule got run twice, which made the performance pathologically worse!

So here’s a PSA of sorts:

1. Try not to have rewrite rules outside of a location block
2. Prefer named routes for “jump”s or bypass internal redirects instead of normal URLs/paths. This would’ve avoided the second execution of the rewrite rules. Something like the below snippet:

error_page 404 @notfound;

location @notfound {
  try_files 404.html =500;
}

Example

Just to demonstrate this with an example, I’m going to use this configuration in nginx, which is deliberately close to what we had structurally:

events {  }

error_log   /dev/stderr notice;

http {
    include         /etc/nginx/mime.types;
    default_type    application/octet-stream;
    access_log      /dev/stdout combined;
    rewrite_log     on;

    # The 404 page definition is copied verbatim from the example config that
    # every debian Nginx package ships with.
    #
    # Culprit 1:
    error_page 404 /404.html;
    error_page 403 /403.html;

    server {
        listen      80;
        server_name localhost;
        root /usr/share/nginx/html;

        # Culprit 2: Naked rewrite rules that try to match for every route, even
        # internal redirects
        rewrite /unknown1/(.*)  /unknown/$1;
        rewrite /unknown2/(.*)  /unknown/$1;
        rewrite /unknown3/(.*)  /unknown/$1;
        rewrite /unknown4/(.*)  /unknown/$1;

        # This will try to send out the file /usr/share/nginx/html/main.html or
        # else respond with a 404 error page.
        location = /main {
            try_files index.html /index.html =404;
        }

        location /notauthorized {
            return 403;
        }

        location /nonexistent {
            return 404;
        }

        # Only the 50x.html template is present in the latest nginx container,
        # so using that as a generic error page to keep things simple.
        location /404.html { try_files 50x.html /50x.html =500; }
        location /403.html { try_files 50x.html /50x.html =500; }

        # Always go to /main by issuing an internal rewrite
        location / {
            rewrite ^/.*$ /main;
        }
    }
}

This sets up three main user-facing routes: /, /main, /notauthorized, /nonauthorized. / redirects to /main internally, although the user won’t see any 3xx, while /notauthorized returns a 403 response. The latter too (return, as used in this case) is implemented as an internal redirect within nginx, so the routing and rule execution behaviour is going to be similar between the 404 case and the 403 case. For the uninitiated, try_files (as used here) in nginx checks the paths given to it within the root path, or else return the status code mentioned at the end. error_page allow for configuring extra routes when nginx has to respond to a particular status code. Effectively, this too is an internal redirect before and after: when a location block has the redirect rule, and when the redirect rule itself has a path as the target location.

I’ll try the following four routes:

curl localhost/
curl localhost/main
curl localhost/notauthorized
curl localhost/nonexistent

The / and /main runs are just to demonstrate the extra rewrite between them. rewrite_log on; does what it says on the tin, and here’s a filtered snippet from the logs:

GET /
"/unknown1/(.*)" does not match "/"
"/unknown2/(.*)" does not match "/"
"/unknown3/(.*)" does not match "/"
"/unknown4/(.*)" does not match "/"
"^/.*$" matches "/"
rewritten data: "/main", args: ""

GET /main
"/unknown1/(.*)" does not match "/main"
"/unknown2/(.*)" does not match "/main"
"/unknown3/(.*)" does not match "/main"
"/unknown4/(.*)" does not match "/main"

GET /notauthorized
"/unknown1/(.*)" does not match "/notauthorized"
"/unknown2/(.*)" does not match "/notauthorized"
"/unknown3/(.*)" does not match "/notauthorized"
"/unknown4/(.*)" does not match "/notauthorized"
"/unknown1/(.*)" does not match "/403.html"
"/unknown2/(.*)" does not match "/403.html"
"/unknown3/(.*)" does not match "/403.html"
"/unknown4/(.*)" does not match "/403.html"

GET /nonexistent
"/unknown1/(.*)" does not match "/nonexistent"
"/unknown2/(.*)" does not match "/nonexistent"
"/unknown3/(.*)" does not match "/nonexistent"
"/unknown4/(.*)" does not match "/nonexistent"
"/unknown1/(.*)" does not match "/404.html"
"/unknown2/(.*)" does not match "/404.html"
"/unknown3/(.*)" does not match "/404.html"
"/unknown4/(.*)" does not match "/404.html"

Both the / route and /main work as expected: the naked rewrite rules run once, but in the cases of the other two these get executed twice. With the current config it’s a bit hard to demonstrate, but the pathological case happens even when those 403, 404 cases happen naturally: an undefined location etc. Using named routes this is the rewritten (no pun intended) config:

events {  }

error_log   /dev/stderr notice;

http {
    include         /etc/nginx/mime.types;
    default_type    application/octet-stream;
    access_log      /dev/stdout combined;
    rewrite_log     on;

    error_page 404 @404.html;
    error_page 403 @403.html;

    server {
        listen      80;
        server_name localhost;
        root /usr/share/nginx/html;

        # Culprit 2: Naked rewrite rules that try to match for every route, even
        # internal redirects
        rewrite /unknown1/(.*)  /unknown/$1;
        rewrite /unknown2/(.*)  /unknown/$1;
        rewrite /unknown3/(.*)  /unknown/$1;
        rewrite /unknown4/(.*)  /unknown/$1;

        # This will try to send out the file /usr/share/nginx/html/main.html or
        # else respond with a 404 error page.
        location = /main {
            try_files index.html /index.html =404;
        }

        location /notauthorized {
            return 403;
        }

        location /nonexistent {
            return 404;
        }

        # Only the 50x.html template is present in the latest nginx container,
        # so using that as a generic error page to keep things simple.
        location @404.html { try_files 50x.html /50x.html =500; }
        location @403.html { try_files 50x.html /50x.html =500; }

        # Always go to /main by issuing an internal rewrite
        location / {
            rewrite ^/.*$ /main;
        }
    }
}

GET /
"/unknown1/(.*)" does not match "/"
"/unknown2/(.*)" does not match "/"
"/unknown3/(.*)" does not match "/"
"/unknown4/(.*)" does not match "/"
"^/.*$" matches "/"
rewritten data: "/main", args: ""

GET /main
"/unknown1/(.*)" does not match "/main"
"/unknown2/(.*)" does not match "/main"
"/unknown3/(.*)" does not match "/main"
"/unknown4/(.*)" does not match "/main"

GET /notauthorized
"/unknown1/(.*)" does not match "/notauthorized"
"/unknown2/(.*)" does not match "/notauthorized"
"/unknown3/(.*)" does not match "/notauthorized"
"/unknown4/(.*)" does not match "/notauthorized"

GET /nonexistent
"/unknown1/(.*)" does not match "/nonexistent"
"/unknown2/(.*)" does not match "/nonexistent"
"/unknown3/(.*)" does not match "/nonexistent"
"/unknown4/(.*)" does not match "/nonexistent"

As expected, only one set of rewrite rule runs. That said, the actual fix would be to refactor the rewrites into location blocks to improve the matching performance a little further.

Footnotes

There’s a 1GiB limit for a single Read call for an os.File entity (object? struct?) in Go, even though native read syscall can fill a 2GiB buffer (as tested in my arm macos and Intel Linux machine). I ran into this when looking at a pprof profile of a sample word count program I was writing, which showed the program was spending way too much time in the syscall module. That in this context can only mean one thing: way too many read syscalls were getting called. Something like this would show this behaviour:

f, err := os.Open("superlargefile.txt")
if err != nil {
    log.Fatal("error opening input file: ", err)
}
defer f.Close()

buf := make([]byte, 1024*1024*1024*2) // 2GiB buffer
fmt.Println("buffer size", len(buf))

for iter := 1; ; iter += 1 {
    n, err := f.Read(buf)

    if err != nil {
        if err == io.EOF {
            fmt.Println("done")
            break
        }

        log.Fatal("error reading input file: ", err)
    }

    fmt.Println("bytes read: ", n)
    fmt.Println("iter: ", iter)
}

That, on a 2.5G file would output something like:

buffer size 2147483648
bytes read:  1073741824
iter:  1
bytes read:  1073741824
iter:  2
bytes read:  490442752
iter:  3
done

Even though the initialised buffer size is 2GiB, only 1GiB is read into the buffer per iteration. Upon digging into the source code, it looks like this is a deliberate choice. The main change logs from the history point to the following:

1. https://codereview.appspot.com/89900044 as a fix for golang/go#7812. This had a fix for failing reads on file sizes greater than or equal to 2GiB on macos and freebsd by capping each read syscall to only read a 2GiB-1 bytes. For the rest of operating systems, at this point, there was no cap.
2. https://codereview.appspot.com/94070044 as a followup of 1, where the limit was decreased without any OS checks to 1GiB, with an explanation that at least it would allow for aligned reads from disk, as opposed to an odd number that might miss page caches (my understanding).

Note that a lot has changed since that changeset, and the current file reference for that _unix.go file in the changeset is src/internal/poll/fd_unix.go.

Aside: System limits

As per the linux read syscall documentation, the maximum bytes that can be transferred is 2GiB. And I tested this out with rudimentary scripts in Rust and C. The Rust program is taken verbatim from the example for read_to_end(). Running that under strace has the following output (truncated here):

read(3, ..., 6594816000) = 2147479552
read(3, ..., 4447336448) = 2147479552
read(3, ..., 2299856896) = 2147479552
read(3, ..., 152377344) = 152377344
read(3, "", 32)         = 0

And a similar, simple C program results in similar output, when using the read syscall in a loop until the file is read:

SSIZE_MAX: 9223372036854775807 # outputting the limits.h constant
bytes read: 2147479552
bytes read: 2147479552
bytes read: 2147479552
bytes read: 152377344

Although that’s neither here nor there, it’s still interesting that Go’s choice has been to pick 2GiB-1 and then 1GiB justifying the odd buffer size in the former.

The classnames library is a very handy tool to apply CSS classes conditionally in JavaScript components. Since the output of the function is just a string, this can be composed very well on multiple conditionals layered on on various parts of the code.

For example, consider the following:

import cx from 'classnames';

switch (type) {
  case: 'textarea':
    const textareaClassNames = cx('text-area', 'text-input', 'invalid': !this.state.valid);
    return <textarea className={textareaClassNames} />
  default:
    const inputClassNames = cx('text-input', 'invalid': !this.state.valid);
    return <input type={type} className={inputClassNames} />
}

The class-names are the same except for one extra item in the case of textarea type input field. Until today, I would’ve done something like the above example, since I never bothered to look at the actual output of the call. A quick glance at the source code of the library made it evident that the library would enable composition with output of another classname-generated value (which is a String). So that code can be simplified to:

import cx from 'classnames';
const className = cx('text-input', 'invalid': !this.state.valid);

switch (type) {
  case: 'textarea':
    return <textarea className={cx(className, 'text-area')} />
  default:
    return <input type={type} className={className} />
}

Much better.

node enum-to-const-object.mjs source.ts [...]
node enum-to-const-object.mjs -w source.ts [...]
node enum-to-const-object.mjs --write source.ts [...]

The first invocation would print out the result to standard out, whereas the latter two would update the source file in-place, exactly how prettier works. The standard library API is pretty neat for such a simple interface:

// file: enum-to-const-object.mjs
// Note the mjs extension, which is why I'm able to use import. Otherwise,
// you'll have to use require in place of the following line
import { parseArgs } from 'node:util';

const options = {
  write: {
    type: 'boolean',
    short: 'w',
    default: false
  }
};

const { values, positionals } = parseArgs({ options, allowPositionals: true });
// values is of the shape { write: <flag value> }
// positionals: [ source.ts, ... ]

The options object is the one used by the parser as the configuration of the flags. The keys of this object are the expected flags in long-hand. The short property for each of these long-hand flags helps with adding aliases.

In addition to the flag format and strings, there is one more additional option that I had to configure: allowPositionals. This returns rest of the arguments that are not flags, which in my case are the files I wanted to transform. Once parseArgs is called using the configuration, and (by default) on process.argv, the flag values as an key-value pair, and the rest of the arguments are returned―values which contains the flag values, and positionals which contains the file list.

Docs Link

Content-Security-Policy: default-src 'self' images.kgrz.io; report-uri: /report-block
Content-Security-Policy-Report-Only: default-src 'self'; report-uri: /report-only

This CSP setting ensures resources only from images.kgrz.io are successfully loaded onto the page. I had always had the implicit understanding that the image load will be blocked, and a report sent out to /report-block path. But this is not the case: there’ll be two reports sent-out: one to /report-block, and one to /report-only.

The sample application that demonstrates this example is hosted at /apps/csp. You may have to have a modern-ish website to use this since I’m using no build pipeline for the JS that’s used on the page. (Anything that [supports <script type="module">][module_can_i_use])

"/p

pastes the search pattern in the current buffer. This same register value is used for repeat searches (n in normal mode). I’m not sure when I’m ever going to use this, but at least I’ll start to understand some shortcut on vim.fandom.com or SO that has an odd / in the command.

So, here’s a small reminder for the future-me that if the container has to be in running state, the state should be set to started. present only ensures the container is there in the process list, and not in running state. i.e., ignoring all the other interactions that this parameter has with other options, state: present is effectively docker create, whereas state: started is analogous docker run.

Aside: Why use ansible to run docker?

This whole setup might seem convoluted, but there’s a reason why I chose this. Before I had an M1-based Macbook, I was using Vagrant for local testing of the ansible pipeline. Everytime I make code changes for the server configuration, I’ll run the entire Ansible playbook end to end to setup the local VM provisioned by Vagrant, followed by basic curl-based tests that check for status codes and cache headers against the server. Vagrant uses VirtualBox for running the virtual machines, which is not supported on M1 chips. The process of converting this exact setup to instead use Docker is not that straight forward in my experience so far. Running docker build and docker run directly won’t work in all cases since I use Ansible template that interpolates variables into the nginx configuration files, which have to be compiled before I actually build the docker image. So the short-term alternative I was trying out was to use another playbook that runs every Ansible task in it locally, which includes letting Ansible take care of building, running the server container and the tests.

The ubiquitous IEEE floating-point standard defines two numbers to represent zero, the positive and the negative zeros. You also have the positive and negative infinity. If you compute the inverse of the positive zero, you get the positive infinity. If you compute the inverse of the negative zero, you get the negative infinity.

I wanted to check this out for the most frequent languages I tend to use—Ruby and JavaScript.

First up, Ruby:

minus_zero = -0.0
plus_zero = 0.0

converted = "-0.0".to_f

puts 1.0 / minus_zero
puts 1.0 / plus_zero
puts 1.0 / converted

Output: -Infinity, Infinity, -Infinity

Next, JavaScript:

const minus_zero = -0.0
const plus_zero = 0.0

const converted = parseFloat("-0.0", 10)

console.log(1.0 / minus_zero)
console.log(1.0 / plus_zero)
console.log(1.0 / converted)

Output: -Infinity, Infinity, -Infinity

Both languages (Ruby v. 2.7.1, JavaScript(NodeJS) v. 12.14.x & Chrome 91.x) return the values as expected in the post.

A very simple way to circumvent this restriction is by using a webfont. The simplest option is to use an @import statement and load a webfont from, say, Google fonts or some other service (your own web server too!).

Example stylesheet:

@import url('https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:ital@0;1&display=swap');

pre, code {
	font-family: 'IBM Plex Mono', monospace;
}

And use the Preferences > Advanced > Style sheet, and load this file from disk. This trick, and a little bit of default zoom makes the IETF spec docs much much better to read.

Sample screenshot of one of the IETF spec docs I’m reading:

This is not perfect, but goes a long way without needing to install extra plugins ala Stylebot or Cascadea for the basic usecase.

I’ll use an example from GitHub’s /commits REST API, using PR: ruby/ruby#3365. I’ve saved the response in the repo where I’ve added full implementation of the example used in this post. The commits response from GitHub REST API is very verbose, depending on the PR size, and having depth greater than 1. In the hypothetical application that I’m writing, I need a list of “objects” that have the following information:

type MetaData struct {
	Author string
	Committer string
	SHA string
	Message string
}

That is, I want parse this response into a []MetaData slice. I do not want to traverse the structs in the format of the responses in my main “business logic”, as that makes it hard to follow the important bits. I don’t want to use interface{} as a placeholder. A better trade-off, in my opinion and use case, is to do as much as possible during the parse phase to massage the data into the structure you want¹. I’m positive that this is a common use case. I ended up learning one way to do this cleanly almost by accident. First, the components involved:

Use anonymous structs

Anonymous structs can be used to avoid defining a concrete type and skip giving it a name for one-off use-cases. It’s heavily used in parsing and marshalling code paths, and testing. In our case, this technique can be used to define a “dirty” struct inside the UnmarshalJSON function on the fly, and use that for parsing the JSON.

Implementing Unmarshaler interface

Any type that has a UnmarshalJSON function on it implements the Unmarshaler interface. This type then can be used as the target for parsing a JSON sub tree or the entire JSON itself!

Implementation

First step is to mock out the main function:

func main() {
	// This variable contains the raw json bytes that resulted from the
	// API call. I'm not adding the code for the actual network fetch
	// for now, but in the example repository, I read the commits
	// response from a file
	var jsonb []byte
	jsonb = JSONFromSomewhere()

	var metadatas []MetaData
	if err := json.Unmarshal(jsonb, &metadatas); err != nil {
		log.Fatalln("error parsing JSON", err)
	}

	fmt.Println(metadatas)
}

The JSON response of /commits endpoint is a list of commit objects, and I’m using a list of MetaData types to match that interface. For each commit item from the JSON array, the raw bytes get passed as the argument to the UnmarshalJSON function on MetaData.

Next step is to implement the UnmarshalJSON function using an anonymous struct to parse out the raw commit object JSON string into it:

func (m *MetaData) UnmarshalJSON(buf []byte) error {
	var commit struct {
		SHA    string `json:"sha"`
		Commit struct {
			Author struct {
				Name string `json:"name"`
			} `json:"author"`
			Committer struct {
				Name string `json:"name"`
			} `json:"committer"`
			Message string `json:"message"`
		} `json:"commit"`
	}

	if err := json.Unmarshal(buf, &commit); err != nil {
		return errors.Wrap(err, "parsing into MetaData failed")
	}

	// continued
}

Final step is to process the commit struct, and set the appropriate fields on MetaData struct:

func (m *MetaData) UnmarshalJSON(buf []byte) error {
	// same as above

	m.AuthorName = commit.Commit.Author.Name
	m.CommitterName = commit.Commit.Committer.Name
	m.SHA = commit.SHA
	m.Message = commit.Commit.Message

	return nil
}

That’s it! An additional advantage to this type of narrow types is it’s easier to test.

Bonus: Filtering the slice further

For bonus points, I want to skip certain []MetaData elements based on a condition. A way to do this, keeping the same principles as above in mind, is to define a type that covers []MetaData, which implements the Unmarshaler interface:

type MetaDatas []MetaData

func (ms *MetaDatas) UnmarshalJSON(buf []byte) error {
	// []MetaData is not the same as MetaDatas, and this difference is
	// important!
	var metadatas []MetaData

	if err := json.Unmarshal(buf, &metadatas); err != nil {
		log.Fatalln("error parsing JSON", err)
	}

	// filtering without allocations
	// https://github.com/golang/go/wiki/SliceTricks#filtering-without-allocating
	cleanedms := metadatas[:0]
	for _, metadata := range metadatas {
		if !strings.HasPrefix(metadata.Message, "WIP") {
			cleanedms = append(cleanedms, metadata)
		}
	}
	*ms = cleanedms

	return nil
}

Like before, I’m using a temporary type of the kind that matches our main type, and using that to parse into. Then I’m clean out slice based on a condition—I want to skip all the commits that start with WIP. Note that the metadatas variable defined inside the UnmarshalJSON function is defined as []MetaData and not as MetaDatas, since doing that would result in a parse-loop. By design, var metadatas Metadatas and var metadatas []MetaData are not the same type.

Finally, the filtered slice gets assigned to the underlying object that the JSON is getting parsed into.

A note about performance

In these examples, the parse flow will create the entire []MetaData slice, even though we filter out many of the elements. To my knowledge, this seems like a necessary hit to take. I’m not aware if there’s a way to avoid allocations by pre-pre-processing the incoming bytes to avoid the allocation in the first place. My thought process here is that if we didn’t filter, or cleanup the JSON data, it will anyway allocate all the objects, so this may not be a huge difference in allocations per se, but that’s just my opinion at this point.