Skip to content
 
 

Repository files navigation

net

github.com/josexy/net is a fork of golang.org/x/net. It retains the upstream networking packages and extends http2 with header ordering, exact header-block forwarding, and HTTP/2 frame fingerprint capture and replay.

Key changes

HTTP/2 header ordering

  • Servers can inspect the original field order of request headers and trailers.
  • Clients can inspect the original field order of all 1xx responses, final response headers, and trailers.
  • Clients can control the field order of request headers and trailers.
  • Servers can control the field order of 1xx responses, final response headers, and trailers.
  • Pseudo-headers such as :method and :status, duplicate fields, field values, and HPACK never-indexed (Sensitive) metadata are preserved.
  • Proxies can stream complete ordered fields without buffering the body while waiting for trailers.

Here, "original order" means the field order after an HTTP/2 header block is decoded by HPACK. It does not mean the original HPACK bytes, and it does not preserve HEADERS/CONTINUATION frame boundaries.

HTTP/2 fingerprints

A server can capture the following four connection/request characteristics, and a client transport can replay them on an outbound connection:

  1. Initial SETTINGS.
  2. The first connection-level WINDOW_UPDATE.
  3. PRIORITY frames received before the first request.
  4. The current request's pseudo-header order.

When http2.Transport uses its default connection pool, connections are isolated by target and connection-level fingerprint. Requests that differ only in pseudo-header order can still reuse the same connection.

Installation

go get github.com/josexy/net

The module currently requires Go 1.25 or later:

import (
	"github.com/josexy/net/http2"
	"github.com/josexy/net/http2/h2c"
)

Clients must use this project's http2.Transport. Servers must be connected through this project's http2.Server, h2c.NewHandler, or the corresponding configuration helpers. HTTP/2 connections created solely by the standard library do not carry the ordering or fingerprint metadata recorded by this fork.

API overview

The primary data types are:

type HeaderOrder struct {
	Headers  []string // Initial request, 1xx, or final response headers.
	Trailers []string // Trailers.
}

type HeaderBlock struct {
	Kind      HeaderBlockKind
	Fields    []HeaderField
	Truncated bool
}

type HeaderField struct {
	Name      string
	Value     string
	Sensitive bool
}

type Fingerprint struct {
	Settings          []Setting
	WindowUpdate      uint32
	Priorities        []FingerprintPriority
	PseudoHeaderOrder []string
}

The main header-order and header-block APIs are:

  • RequestHeaderBlocks(*http.Request): inspect request header blocks on a server.
  • ResponseHeaderBlocks(*http.Response): inspect response header blocks on a client.
  • WithRequestHeaderOrder(*http.Request, HeaderOrder): set client request ordering.
  • SetResponseHeaderOrder(http.ResponseWriter, HeaderOrder): set server response ordering.
  • WithRequestHeaderBlocks: send an exact initial request block and obtain the trailer block dynamically after body EOF.
  • WithInformationalResponseHandler: receive each 1xx block immediately.
  • WriteResponseHeaderBlock: send one exact 1xx or final response block.
  • SetResponseTrailerBlock: set an exact trailer block after the response body has started.

The main HTTP/2 fingerprint APIs are:

  • RequestFingerprint: inspect the current request's four-part fingerprint on a server.
  • WithRequestFingerprint: bind a fingerprint to a client request for replay.
  • ParseFingerprint: parse and validate a fingerprint string.
  • Fingerprint.String: return the canonical representation.
  • Fingerprint.Hash: return the lowercase hexadecimal MD5 of the canonical representation.

HeaderBlock.Kind has the following values:

  • HeaderBlockInitial: initial request headers or final, non-1xx response headers.
  • HeaderBlockInformational: one 1xx response header block; every 1xx response has its own block.
  • HeaderBlockTrailer: trailers.

If Truncated is true, the configured header-list size limit was reached and Fields is incomplete. The inspection functions return snapshots, so changing a returned value does not affect later queries.

Client usage

Use this fork's http2.Transport. Apply WithRequestHeaderOrder to control ordinary request headers and trailers, and apply WithRequestFingerprint to control initial frames and pseudo-header order:

transport := &http2.Transport{}
defer transport.CloseIdleConnections()

client := &http.Client{Transport: transport}
req, err := http.NewRequest(
	http.MethodPost,
	"https://example.com/upload",
	strings.NewReader("hello"),
)
if err != nil {
	log.Fatal(err)
}

req.Header.Set("Content-Type", "text/plain")
req.Header.Set("X-Request-A", "a")
req.Header.Set("X-Request-B", "b")
req.Trailer = http.Header{
	"X-Trailer-A": {"trailer-a"},
	"X-Trailer-B": {"trailer-b"},
}

req, err = http2.WithRequestHeaderOrder(req, http2.HeaderOrder{
	Headers: []string{
		":method",
		":authority",
		":scheme",
		":path",
		"x-request-b",
		"content-length",
		"content-type",
		"x-request-a",
	},
	Trailers: []string{
		"x-trailer-b",
		"x-trailer-a",
	},
})
if err != nil {
	log.Fatal(err)
}

fingerprint, err := http2.ParseFingerprint(
	"1:65536;3:1000;4:6291456;6:262144|15663105|0|m,a,s,p",
)
if err != nil {
	log.Fatal(err)
}
req, err = http2.WithRequestFingerprint(req, fingerprint)
if err != nil {
	log.Fatal(err)
}

resp, err := client.Do(req)
if err != nil {
	log.Fatal(err)
}
defer resp.Body.Close()

// All 1xx blocks and final response headers are available after RoundTrip.
log.Printf("response headers: %#v", http2.ResponseHeaderBlocks(resp))

// Trailers usually arrive while the body is read. Query again after EOF.
if _, err := io.Copy(io.Discard, resp.Body); err != nil {
	log.Fatal(err)
}
log.Printf("response headers and trailers: %#v", http2.ResponseHeaderBlocks(resp))

For h2c prior knowledge (cleartext HTTP/2), also configure AllowHTTP and DialTLSContext. The first HTTP/1.1 h2c Upgrade request has no HTTP/2 pseudo-headers, so RequestFingerprint returns false for that request. Subsequent HTTP/2 requests can be captured normally.

Server usage

Initial request headers and the fingerprint are available when the handler starts. Request trailers usually arrive while the body is read, so call RequestHeaderBlocks again after EOF.

Call SetResponseHeaderOrder before the first Write, WriteHeader, or Flush:

func handler(w http.ResponseWriter, r *http.Request) {
	log.Printf("request headers: %#v", http2.RequestHeaderBlocks(r))
	if fingerprint, ok := http2.RequestFingerprint(r); ok {
		log.Printf("fingerprint=%s hash=%s", fingerprint.String(), fingerprint.Hash())
	}

	if _, err := io.Copy(io.Discard, r.Body); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}
	log.Printf("request headers and trailers: %#v", http2.RequestHeaderBlocks(r))

	if err := http2.SetResponseHeaderOrder(w, http2.HeaderOrder{
		Headers: []string{
			":status",
			"x-response-b",
			"content-type",
			"x-response-a",
		},
		Trailers: []string{
			"x-server-trailer-b",
			"x-server-trailer-a",
		},
	}); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	w.Header().Set("X-Response-A", "a")
	w.Header().Set("X-Response-B", "b")
	w.WriteHeader(http.StatusEarlyHints)

	w.Header().Set("Content-Type", "text/plain")
	w.Header().Set("Trailer", "X-Server-Trailer-A, X-Server-Trailer-B")
	w.WriteHeader(http.StatusOK)
	_, _ = io.WriteString(w, "ok\n")

	w.Header().Set("X-Server-Trailer-A", "trailer-a")
	w.Header().Set("X-Server-Trailer-B", "trailer-b")
}

mux := http.NewServeMux()
mux.HandleFunc("POST /header-order", handler)

server := &http.Server{
	Addr:    "127.0.0.1:8080",
	Handler: h2c.NewHandler(mux, &http2.Server{}),
}
log.Fatal(server.ListenAndServe())

TLS HTTP/2 servers can use http2.ConfigureServer. If middleware wraps an http.ResponseWriter, the wrapper must implement Unwrap() http.ResponseWriter so SetResponseHeaderOrder can reach the underlying HTTP/2 writer.

Header ordering rules

  • HeaderOrder contains field names, not values. Names are normalized to lowercase.
  • Missing fields are ignored. Once ordering is enabled for a block, unlisted ordinary fields follow the listed fields in stable lowercase-name order.
  • Duplicate fields with the same name are emitted as a group while preserving their value order.
  • Pseudo-headers always precede ordinary fields. Requests support :method, :authority, :scheme, :path, and :protocol; responses support only :status; trailers cannot contain pseudo-headers.
  • Pseudo-headers in an order must precede ordinary fields, and a field name cannot appear more than once.
  • Implementation-generated fields such as content-length, user-agent, and date may also be included in an order.

On the receiving side, HeaderField.Name is the lowercase wire name. Duplicate fields are not merged or reordered as they may be in http.Header. Sensitive means the field used HPACK's never-indexed representation; it does not mean that Value has been hidden.

Exact header-block forwarding

HeaderOrder describes field-name order and groups fields with the same name. A transparent proxy that must preserve values, duplicate positions, interleaved fields, and Sensitive metadata should use the exact block APIs:

  1. Obtain the inbound HeaderBlockInitial with RequestHeaderBlocks.
  2. Bind the initial block and a dynamic trailer provider to the outbound request with WithRequestHeaderBlocks.
  3. If RequestFingerprint returns a fingerprint, bind it explicitly with WithRequestFingerprint.
  4. Forward every 1xx block immediately with WithInformationalResponseHandler.
  5. Forward final response headers with ResponseHeaderBlocks and WriteResponseHeaderBlock.
  6. Stream the body with io.Copy.
  7. After body EOF, query ResponseHeaderBlocks again and forward trailers with SetResponseTrailerBlock.

The exact block APIs have the following semantics:

  • Field names, values, duplicate positions, and Sensitive flags are encoded in the supplied sequence.
  • The initial block passed to WithRequestHeaderBlocks replaces serialization of req.Header, but req.URL still selects the actual target and the body still comes from req.Body.
  • A final block passed to WriteResponseHeaderBlock replaces the initial response fields in w.Header().
  • A trailer provider returns an empty HeaderBlockTrailer to indicate that there are no trailers.
  • Truncated blocks, invalid pseudo-headers, HTTP/2 connection-specific fields, and invalid field values are rejected.
  • To modify fields such as :authority, forwarded, or via, copy and edit Fields before calling an exact-send API.

Exact replay means replaying the field sequence after HPACK decoding. Different connections have different HPACK dynamic tables, so the original compressed HPACK bytes are neither copied nor expected to match.

HTTP/2 fingerprint format and limitations

A fingerprint string has four sections:

SETTINGS|WINDOW_UPDATE|PRIORITY|PSEUDO_HEADER_ORDER

For example:

1:65536;3:1000;4:6291456;6:262144|15663105|0|m,a,s,p

The second section is 00 when no initial connection-level WINDOW_UPDATE was observed, and the third section is 0 when no PRIORITY frame was observed. The third section contains only standalone PRIORITY frames, matching the canonical four-part fingerprint format. Priority information carried inside a request's initial HEADERS frame is available separately as Fingerprint.HeaderPriority. That value contains StreamDep, Exclusive, and the semantic Weight in the range 1..256; it has no StreamID because the outgoing connection allocates the request stream.

Fingerprint.String and Fingerprint.Hash deliberately exclude HeaderPriority, and ParseFingerprint therefore returns a fingerprint with a nil HeaderPriority. Code that needs complete wire replay must pass or clone the structured Fingerprint instead of round-tripping it through String.

The server freezes the connection-level portion when the first valid request HEADERS arrives. Each request uses its actual pseudo-header order, and returned fingerprints are independent snapshots.

A proxy must bind the inbound fingerprint explicitly. Cloning the inbound context alone does not enable replay:

fingerprint, ok := http2.RequestFingerprint(incoming)
if ok {
	outgoing, err = http2.WithRequestFingerprint(outgoing, fingerprint)
	if err != nil {
		return err
	}
}

Dynamic fingerprints require a directly used http2.Transport with its default connection pool. Custom ConnPool implementations, ConfigureTransport/ConfigureTransports, and direct ClientConn.RoundTrip calls are not supported; those paths return an error wrapping errors.ErrUnsupported.

The fingerprint does not include TLS JA3/JA4, original HPACK bytes, or arbitrary HEADERS/CONTINUATION frame boundaries. HeaderPriority records only the priority fields and presence on the initial request HEADERS frame.

Go 1.27 and later

Starting with Go 1.27, x/net/http2 uses the net/http-based wrapper implementation by default. This project's header-order and fingerprint extensions are implemented in the legacy path, so applications must use the http2legacy build tag:

go run -tags http2legacy ./cmd/your-app
go test -tags http2legacy ./...
go build -tags http2legacy ./...

Without this tag:

  • Header send control, exact block APIs, and ordered 1xx callbacks return an error detectable with errors.Is(err, errors.ErrUnsupported).
  • RequestHeaderBlocks and ResponseHeaderBlocks return nil.
  • Fingerprint parsing, formatting, and hashing remain available.
  • RequestFingerprint returns false, and WithRequestFingerprint returns errors.ErrUnsupported.

Testing

go test ./...

On Go 1.27 or later, test the legacy implementation with:

go test -tags http2legacy ./...

About

[mirror] Go supplementary network libraries

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages