of-watchdog

Reverse proxy/middleware for functions using STDIO/HTTP

The of-watchdog implements an HTTP server listening on port 8080, and acts as a reverse proxy for running functions and microservices. It can be used independently, or as the entrypoint for a container with OpenFaaS.

This version of the OpenFaaS watchdog adds support for HTTP proxying as well as STDIO, which enables reuse of memory and very fast serving of requests. It does not aim to replace the Classic Watchdog, but offers another option for those who need these features.

A download is made via GitHub releases, but the watchdog is meant to be copied from the container image published to ghcr.io in a multi-stage build:

FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/of-watchdog:0.9.11 as watchdog
FROM --platform=${TARGETPLATFORM:-linux/amd64} node:18-alpine as ship

COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog

See example templates

• Keep function process warm for lower latency / caching / persistent connections through using HTTP
• Enable streaming of large responses from functions, beyond the RAM or disk capacity of the container
• Cleaner abstractions for each "mode"

There are several modes available for the of-watchdog which changes how it interacts with your microservice or function code.

A comparison of three watchdog modes. Top left - Classic Watchdog, top right: afterburn (deprecated), bottom left HTTP mode from of-watchdog.

1. HTTP mode - the default and most efficient option all template authors should consider this option if the target language has a HTTP server implementation.
2. Serializing mode - for when a HTTP server implementation doesn't exist, STDIO is read into memory then sent into a forked process.
3. Streaming mode - as per serializing mode, however the request and response are both streamed instead of being buffered completely into memory before the function starts running.

Private endpoints, served by watchdog:

• /_/health - returns true when the process is started, or if a lock file is in use, when that file exists.
• /_/ready - as per /_/health, but if max_inflight is configured to a non-zero value, and the maximum number of connections is met, it will return a 429 status

Any other HTTP requests:

• /* any other Path and HTTP verbs are sent to the function

HTTP mode is recommend for all templates where the target language has a HTTP server implementation available.

See a few different examples of templates, more are available via faas-cli template store list, such as golang-middleware, python3-http and node*.

To get the repository for a specific template use faas-cli template store describe NAME.

A process is forked when the watchdog starts, we then forward any request incoming to the watchdog to a HTTP port within the container.

Pros:

• Fastest option for high concurrency and throughput
• More efficient concurrency and RAM usage vs. forking model
• Database connections can be persisted for the lifetime of the container
• Files or models can be fetched and stored in /tmp/ as a one-off initialization task and used for all requests after that
• Does not require new/custom client libraries like afterburn but makes use of a long-running daemon such as Express.js for Node or Flask for Python

Example usage for testing:

• Forward to an Nginx container:

$ go build && mode=http port=8081 fprocess="docker run -p 80:80 --name nginx -t nginx" upstream_url=http://127.0.0.1:80 ./of-watchdog

• Forward to a Node.js / Express.js hello-world app:

$ go build && mode=http port=8081 fprocess="node expressjs-hello-world.js" upstream_url=http://127.0.0.1:3000 ./of-watchdog

Cons:

• One more HTTP hop in the chain between the client and the function
• Daemons such as express/flask/sinatra can be unpredictable when used in this way so many need additional configuration
• Additional memory may be occupied between invocations vs. forking model

It is not currently possible to have the watchdog's own messages outputted in JSON:

2024/04/25 17:29:06 Listening on port: 8080
2024/04/25 17:29:06 Writing lock-file to: /tmp/.lock
2024/04/25 17:29:06 Metrics listening on port: 8081
2024/04/25 17:29:08 GET / - 301 Moved Permanently - ContentLength: 39B (0.0049s) [test]

However, you can write your own log lines in JSON. Just set the prefix_logs environment variable to false, to remove the default prefix that the watchdog emits otherwise.

With prefix_logs on:

2024-04-24T21:00:04Z {"msg": "unable to connect to database"}

With prefix_logs off:

{"msg": "unable to connect to database"}

The gateway sends an X-Call-Id header which should be used in your own logger to correlate and trace requests.

In HTTP mode, the watchdog will append the X-Call-Id to its own HTTP log messages in square brackets if you set the log_callid environment variable to true:

2024/04/25 17:29:58 GET / - 301 Moved Permanently - ContentLength: 39B (0.0037s) [079d9ff9-d7b7-4e37-b195-5ad520e6f797]

If a function has a timeout set via exec_timeout of a large value like 1h, but you need an individual request to timeout earlier, i.e. 1m, then you can pass in a HTTP header of X-Timeout with a Go duration to override the behaviour.

The value for X-Timeout must be equal to or shorter than the exec_timeout environment variable.

X-Timeout cannot be set when the exec_timeout is set to 0 or hasn't been specified.

This mode is designed to replicate the behaviour of the original watchdog for backwards compatibility.

Forks one process per request. Multi-threaded. Ideal for retro-fitting a CGI application handler i.e. for Flask.

Limited to processing files sized as per available memory.

Reads entire request into memory from the HTTP request. At this point we serialize or modify if required. That is then written into the stdin pipe.

• Stdout pipe is read into memory and then serialized or modified if necessary before being written back to the HTTP response.
• A static Content-type can be set ahead of time.
• HTTP headers can be set even after executing the function (not implemented).
• Exec timeout: supported.

Forks a process per request and can deal with a request body larger than memory capacity - i.e. 512mb VM can process multiple GB of video.

HTTP headers cannot be sent after function starts executing due to input/output being hooked-up directly to response for streaming efficiencies. Response code is always 200 unless there is an issue forking the process. An error mid-flight will have to be picked up on the client. Multi-threaded.

• Input is sent back to client as soon as it's printed to stdout by the executing process.
• A static Content-type can be set ahead of time.
• Exec timeout: supported.

This mode starts an HTTP file server for serving static content found at the directory specified by static_path.

See an example in the Hugo blog post.

Environmental variables:

Note: timeouts should be specified as Golang durations i.e. 1m or 20s.

Unsupported options from the Classic Watchdog:

The watchdog can add browser login to a function using an OAuth 2.0 or OpenID Connect (OIDC) provider. It runs the Authorization Code flow with PKCE on your function's behalf, keeps the resulting session in a signed cookie, and validates that cookie before forwarding each request. Your function never has to implement authentication itself: it reads the verified session cookie to identify the user.

On every request the watchdog checks for a valid session cookie:

• No cookie - the request is forwarded to the function unchanged, so public pages work without signing in.
• Valid cookie - the request is forwarded with the cookie intact, so the function can decode it to read the user's ID and access tokens.
• Invalid or expired cookie - the request is rejected with HTTP 401.

The function's frontend starts the flow by sending the visitor to the watchdog's GET /auth/login endpoint, for example from a "Sign in" link. The watchdog then runs the authorization-code exchange with the provider itself, and on success sets the signed session cookie and sends the visitor back to the function.

The watchdog serves the following routes under the function's public URL:

• GET /auth/login - start the login flow and redirect the browser to the provider
• GET /auth/callback - handle the provider's redirect and set the session cookie
• POST /auth/logout - clear the session cookie

Register {oauth_base_url}/auth/callback as the callback (redirect) URI on your provider and client.

Session cookies are HttpOnly and signed. Logout only clears the function's cookies; it does not end the visitor's session at the provider.

Set oauth_enabled=true and provide the function's public URL, the client ID registered with your provider, and a signing key. Secrets such as the signing key and any client secret are read from files mounted under /var/openfaas/secrets.

You then point the watchdog at your provider in one of two ways:

• OIDC (recommended) - set oauth_issuer_url to the provider's issuer URL. The watchdog discovers the authorization, token and keys endpoints and validates ID tokens automatically. Use this with Keycloak, Okta, Google, Microsoft Entra ID and other OIDC providers.
• Plain OAuth - set oauth_authorization_endpoint and oauth_token_endpoint to the provider's authorization and token exchange URLs. Use this with providers that do not support OIDC discovery, such as a GitHub OAuth App.

Session lifetimes use positive Golang durations in whole seconds, e.g. 30m or 8h.