Expose opt-in Mako Prometheus metrics - #14040

mako is distinct but was wondering why not use existing fdb otel or statsd? Why add 3rd model, and one that is pull based rather than push (is pull good for mako stats when runs are short).

We also include a minimal http server here. You might make the argument for it (I think its easy -- smile) otherwise folks will wonder why codebase has a second one apart from fdbrpc/HTTPServer.cpp.

Thanks.

Hi @saintstack ,
Thank you very much for reviewing this PR quickly!

The problem we want to solve is to run a distributed Mako workload against an FDB cluster to do the load testing and release performance testing. A single box Mako workload is not enough to drive enough workload against a FDB cluster, so we need multiple mako instances (on k8s) driving load against a FDB cluster.

We want to measure the client-side performance, including P50 and P99 latency of GET and update requests and the QPS, IOPS throughput metrics. That's why we want to expose the metrics from each micro instance and leverage Grafana to create the stats and client-side performance dashboard.

regarding FDB StatsD and OTEL, reasons are:

1. I think these are for fdbservers. mako is an independent workload/binary that do not rely on FDB's flow runtime.
2. Grafana default model is pull instead of push model.
3. As you mentioned, it's simplier, and more importantly cleaner (keeping mako and fdbserver separate), to create a server.

I'm open to suggestions.

LGTM

Will review when you move it out of draft mode.

Thanks.

Thank you! @saintstack i marked it ready.

Was looking at the http server again. No keep-alive, no chunking, no pipelining but I think its fine. One suggestion given you have it is to use more of boost beast: you can delete request.rfind("GET /metrics HTTP/1.", 0) == 0 and you get stuff like absolute-form URI, and leading-CRLF tolerance for the same LoC. beast::tcp_stream::expires_after gives you the read timeout in one line.

Suggestions from my 'friend' given you are doing the raw asio are:

• no read/idle deadline — a connection that opens and sends nothing pins a Session forever; no cap on concurrent connections either. That's the one real bug.
• IPv4-only (Tcp::v4(), wildcard bind). On an IPv6 k8s cluster the scrape just fails. Add --prometheus_addr.
• Session::render is a std::function const& into the parent; it survives only because of member declaration order. Copy it or hold a shared_ptr.
• constructor calls accept() and starts the thread before construction completes; if std::thread throws you destroy an io_context with a pending handler and never run the dtor.

I was asking my friend about the standalone, hand-written http server and it was more concerned about prometheus.hpp + native_latency.hpp with hand-rolled format and a 250-bucket cumulative histogram with hand-strided index math (for (int index = 426; index <= 1278; index += 4)).

I looked at having mako just dump a file and let something like a telegraf host it but more work.

Sorry for all the comments. Am interested in this project. Would be good to have something like it back in core fdb on each process: e.g. different context completely but just FYI #12680.

From local LLM. See what you think:

• OP_TRANSACTION latency is not exported. It is already computed and passed to addLatency (mako.cpp:363, async.cpp:206), and end-to-end transaction latency is the single most useful number for a workload scraper. native_latency.hpp:384 operationCount = 2 excludes it for no stated reason. Bumping to 3 costs ~32 KB per worker.
• Session::render is a dangling-reference hazard. prometheus_server.hpp:890 holds std::functionstd::string() const& to the server's member. Member destruction order is thread, render, acceptor, io — so io's destructor destroys abandoned handlers (and the last Session) after render is already gone. Benign today only because ~Session never calls render. Store it by value or as shared_ptr; one std::function copy per scrape is free.
• No read/write timeout or connection cap on the exporter. A client that connects and never sends a full header holds a Session until process exit. Bound on 0.0.0.0 with no auth. Opt-in mitigates this, but a steady_timer on the read is ~5 lines.