probes

Running a Minimal Prometheus

2024-09-07T00:00:00+00:00

Prometheus does not need to be hugely complicated, nor a massive resource hog, provided you follow some principles.

Background</h2>
My last #prometheus</a> posts have been exclusively about large scale production setups, and the difficulties this pulls in.
I would like to argue that these difficulties are often self-imposed, and a combination result of inadequate cardinality control</a> and induced demand</a>.

👺: You should be able to run a prometheus on your handful-of-machine-sized homelab with <10k time series active, using less than 512Mi memory, and 10m CPU. </blockquote>
Disclaimer: Who am I?</summary> I</a> am a platform engineer working maintenance of observability infrastructure, and a maintainer of kube-rs</a> working on rust integration of observability with Kubernetes platforms. My knowledge of prometheus is superficial and mostly based on practical observations around operating it for years. Suggestions or corrections are welcome (links at bottom). </details>
Signals, Symptoms & Causes</h2>
To illustrate this, let's try to answer the (perhaps obvious question): why do you install a metrics system at all?
Primarily; we want to be able to track and get notified on changes to key signals. At the very basic level you want to tell if your "service is down", because you want to be able to use it. That's the main, user-facing signal. Setup something that test if the service responds with a `2XX</code>, and alert on deviations. You` `don't even need</a> prometheus for this.`
`However, while you can do basic sanity outside the cluster, you need utilisation and saturation</a>, to tell you about less obvious / upcoming failures:`
message queues full :: rejected work</li> high cpu utilisation :: degraded latency</li> memory utilisation high :: oom kill imminent</li> disk space nearing full :: failures imminent</li> </ul> You can argue idealistically about whether you should only be "aware of something going wrong", or be "aware that something is in a risky state" (e.g. symptoms not causes</a>), but utilisation/saturation/errors are ultimately very good indications/predictors for degraded performance, so we will focus on these herein. How Many Signals</h2> MAIN POINT: You should be able to enumerate the basic signals that you want to have visibility of. </blockquote> Let's do some simplified enumeration maths on how many signals you actually want to properly identify failures quickly. Compute Utilisation/Saturation</h3> Consider an example cluster with 200 pods, and 5 nodes. Want {utilization,saturation} of {cpu,memory} at cluster level :: 2 * 2 = 4 signals</li> Want to break these down per Pod</code> :: 200 * 4 = 800 signals</li> Want to break these down per Node</code> :: 5 * 4 = 20 signals</li> </ul> So, in theory, we should be able to visualise cluster, node, and pod level utilisation for cpu and memory with only 820 metrics (but likely more if you want to break down node metrics). NB: Pods are only found on one node, so Pod</code> cardinality does not multiply with Node</code> cardinality. </blockquote> These will come from a combination of cadvisor</a> and kube-state-metrics</a>; huge beasts with lots of functionality and way more signals than we need. Depending on how much of the kubernetes mixin</a> you want, you may want more signals. Node State Breakdown</h3> If you want to break down things within a node on a more physical level, then you you can also grab node-exporter</a>. Assuming, for simplicity, 10 cores per node, 10 disk devices per node, and 10 network interfaces: Want {utilization,saturation,errors} of cpu :: 310 5 = 60 signals</li> Want {utilization,saturation,errors} for memory :: 35 = 15 signals</li> Want {utilisation,saturation,errors} of disks :: 310 * 5 = 100 signals</li> Want {utilisation,saturation,errors} of network interfaces :: 103 5 = 150 signals</li> </ul> In theory, you should be able to get decent node monitoring with less than 400 metrics. In Practice</h2> Doing this type of enumeration is helpful as a way to tell how close you are to your theoretical 100% optimised system, but how does the number hold up in practice? Metric Inefficiencies</h3> The problems with expecting this type of perfection in practice is that many metric producers are very inefficient / overly lenient with their output. You can click on the addendum below for some examples, but without extreme tuning you can minimally expect a 5x inefficiency factor to apply to the above in practice. Addendum: Inefficiency Examples</summary> Take for instance, job="node-exporter"</code> putting tons of labels on metrics such as node_cpu_seconds_total</code>: {cpu="15", mode="idle"} 1 {cpu="15", mode="iowait"} 1 {cpu="15", mode="irq"} 1 {cpu="15", mode="nice"} 1 {cpu="15", mode="softirq"} 1 {cpu="15", mode="steal"} 1 {cpu="15", mode="system"} 1 {cpu="15", mode="user"} </code></pre> Maybe you dont care about either of these splits, you just want to have a cpu</code> total and data for one mode</code>. Well, firstly, it's hard to get rid of mode</code> because the standard recording rules depend on mode</a>. Secondly, while you could try to rid of cpu, there's no great way of doing this without something like stream aggregation</a> as labeldrop: cpu</code> leads to collisions. Similarly, if you want to support a bunch of the mixin dashboards with container level breakdown, you are also forced to grab a bunch of container level info for e.g. k8s.rules.container_resource</a> unless if you want to rewrite the world. And on top of this, many of these exporters also export things in very suboptimal ways. E.g. job="kube-state-metrics"</code> denormalising disjoint phases (never letting old/zero phases go out of scope): kube_pod_status_phase{phase="Pending", pod="forgejo-975b98575-fbjz8"} 0 kube_pod_status_phase{phase="Succeeded", pod="forgejo-975b98575-fbjz8"} 0 kube_pod_status_phase{phase="Failed", pod="forgejo-975b98575-fbjz8"} 0 kube_pod_status_phase{phase="Unknown", pod="forgejo-975b98575-fbjz8"} 0 kube_pod_status_phase{phase="Running", pod="forgejo-975b98575-fbjz8"} 1 </code></pre> This particular one may get a fix</a> though. </details> Visualising Cardinality</h3> I have a dashboard</a> for this with this panel: </a> Shows metrics produced, how many we decided to keep, and a breakdown link leading to topk(50, count({job="JOB-IN-ROW"}) by (name))</code>. It's probably the panel that has helped the most in slimming down prometheus. Outing Out</h3> The most unfortunate reality is that this stuff is all opt-out. The defaults are crazy inclusive</a>, and new versions of exporters introduce new metrics forcing you to watch your usage graph upgrades. Thankfully, in a gitops repo</a> this is easy to do (if you actually generate your helm templates</a> into a deploy</code> folder) because all your dashboards and recording rules live there. Procedure: Unsure if you need this container_sockets</code> metric? rg container_sockets deploy/</code> and see if anything comes up: No hits? action: drop</code>.</li> Only used in a dashboard? Do you care about this dashboard? No? action: drop</code>.</li> Used in a recording rule? Does the recording rule go towards an alert/dashboard you care about? No? action: drop</code>.</li> </ol> Grafana cloud has its own opt-out ML based adaptive metrics thing</a> to do this, and it's probably helpful if you are locked into their cloud. The solution definitely has a big engineering solution feel</a> to it, but can't really blame it as it's not like you can easily remove polluting labels from needed metrics these days without causing collisions in OSS prom. Things You Don't Need</h2> Sub Minute Data Fidelity</h3> You are probably checking your homelab once every few days at most, so why do you expect you would need 15s/30s data fidelity? You can set 60s</code> scrape/eval intervals and be fine: scrapeInterval: 60s evaluationInterval: 60s </code></pre> You could go even higher, but above 1m</code> grafana does starts to look less clean. To compensate, you can hack your mixin dashboards</a> to increase the time window, and set a less aggressive refresh: # change default refreshes on grafana dashboards to be 1m rather than 10s fd -g '.yaml' deploy/promstack/promstack/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/ \ -x sd '"refresh":"10s"' '"refresh":"1m"' {} # change default time range to be now-12h rather than now-1h on boards that cannot handle 2 days... fd -g '.yaml' deploy/promstack/promstack/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/ \ -x sd '"from":"now-1h"' '"from":"now-12h"' {} # change default time range to be now-2d rather than now-1h on solid dashboards... fd -g 'k8s.yaml' deploy/promstack/promstack/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/ \ -x sd '"from":"now-12h"' '"from":"now-2d"' {} </code></pre> ..which is actually practical</a> if you use helm template</code> rather than helm upgrade</code>. 👺: Somewhere out there, jsonnet</code> users are gouging their eyes out reading this. </blockquote> Most Kubelet Metrics</h3> 97% of kubelet metrics are junk. In a small cluster it's the biggest waste producer in a small cluster, often producing 10x more metrics than anything else. Look at the top 10 metrics they produce with just 1 node and 30 pods: </a> None of these are useful / contribute towards my above goal. Number 4 and 5 on that list together produce more signals than I consume in TOTAL in my actual setup. The kube-prometheus-stack</code> chart does attempt to reduce this somewhat</a>, but it by far too little. Histograms</h3> Histograms generally account for a huge percentage of your metric utilisation. On a default kube-prometheus-stack</code> install with one node, the distribution is >70% histograms: </a> Of course, this is largely due to kubelet</code> metric inefficiencies, but it's a trend you will see repeated elsewhere (thankfully usually with less eye-watering percentages). Unless you have a need to see/debug detailed breakdown of latency distributions, do not ingest or produce these metrics (see addendum for more info). It's the easiest win you'll get in prometheus land; wildcard drop them from your ServiceMonitor</code> (et al) objects: metricRelabelings: - sourceLabels: [name] action: drop regex: '^.bucket.' </code></pre> Addendum: Why does histograms suck?</summary> Multiplicative Cardinality: histogram buckets multiply metric cardinality by 5-30x (the number of buckets).</li> </ol> This number is multiplicative with regular cardinality: pod</code> / node</code> labels added by prometheus operator :: if 2000 metric per pod, and scaling to 20 pods, now you have 40000 metrics</li> request information added by users (endpoint, route, status, error) :: 5 eps 10 routes * 8 statuses * 4 errors = 1600 metrics, but with 30 buckets = 48000</li> See the Containing Your Cardinality</a> talk for more maths details</li> </ul> Inefficiency 30x multiplied information is a bad way to store a few additional signals</li> </ol> If you want P50s or P99s you can compute these in the app with things like rolling averages</a> or rolling quantiles. Some of these are more annoying than others, but there's a lot you can do by just tracking a mean. Strive for one signal per question where possible. </blockquote> That said, if you do actually need them, try to decouple them from your other labels (drop pod labels, drop peripheral information) so that you can get the answer you need cheaply. A histogram should answer one question, if your histogram has extra parameters, you can break them down to smaller histograms (addition beats multiplication for cardinality) 👺: Histograms will get better with the native-histograms feature</a> (see e.g. this talk</a>). However, this requires the ecosystem to move on to protobufs and it being propagated into client libraries (and is at the moment still experimental at the time of writing</a>), it's only been 2 years</a> though. </blockquote> </details> A Solution</h2> Because I keep needing an efficient, low-cost homelab setup for prometheus (that still has the signals I care about), so now here is a chart. It's mostly a wrapper chart over kube-prometheus-stack</a> with aggressive tunings / dropping (of what can't be tuned), and it provides the following default values.yaml</a>. You could use it directly with: helm repo add clux https://clux.github.io/homelab helm install [RELEASE_NAME] clux/promstack </code></pre> but more sensibly, you can take/dissect the values.yaml</a> file and run with it in your own similar subchart. 👺: You shouldn't trust me for maintenance of this, and you don't want to be any more steps abstracted away from kube-prometheus-stack</code>. </blockquote> Architecture Diagram</h2> The chart is effectively a slimmed down 2022 thanos setup</a>; no HA, no thanos</code>, no prometheus-adapter</code>, but a including a lightweight tempo</code> for exemplars: </a> Features</h2> A low-compromises prometheus, all the signals you need at near minimum cost. Cardinality Control</h3> Drops 97% of kubelet metrics, configures minimal kube-state-metrics</code>, node-exporter</code> (with some cli arg configurations and some relabelling based drops) and a few other apps. In the end we have a cluster running with 7000 metrics. </a> It's higher than my idealistic estimate - particularly considering this is a one node cluster atm. It can definitely be tuned lower with extra effort, but this is a good checkpoint. This is low enough that grafana cloud considers it free</a>. So is it? Low Utilization</h3> The end result is a prometheus averaging <0.01</code> cores and with <512Mi</code> memory use over a week with 30d</code> retention: Auxilary Features</h3> Tempo for exemplars</a>, so we can cross-link from metric panels to grafana's trace viewer.</li> Metrics Server for basic HPA scaling and kubectl top</code>.</li> </ol> 👺: You technically don't need metrics-server</code> if you are in an unscalable homelab, but having access to kubectl top</code> is nice. Another avenue for homelab scaling is keda</a> with its scale to zero</a> ability. </blockquote> CX Dashboards</h3> The screenshots here are from my own homebrew cx-dashboards</a> released separately. See the future post for these. Links / Comments</h2> Posted on mastodon</a>. Feel free to comment / raise issues</a> / suggest an edit</a>.

You Will (Not) Scale Prometheus 2024-08-15T00:00:00+00:00 To aid a memory obese prometheus</code>, I recently helped in attempting to slowly shift a cluster over to prometheus agent mode</a> sending data to thanos receive</a> over the last couple of months. I have now personally given up on this goal due to a variety of reasons, and this post explores why. Background Setup</h2> The original setup we started out with is basically this (via 2022 ecosystem post</a>): </a> and we were planning to move to this: </a> The key changes here: enable prometheus agent feature</a> limiting it to scraping and remote writes to receive</li> deploy thanos receive</a> for short term metric storage + S3 uploader (no more sidecar)</li> deploy thanos rule</a> as new evaluator, posting to alertmanager</li> </ol> With the 3 components replacing prometheus (agent, receive, ruler) in-theory having better scaling characteristics by themselves, with a cleaner, and more delineated area of responsibility. Why chase better scaling characteristics? A single prometheus grows in size/requests with amount of time series it scrapes, and it can only grow as long as you have enough RAM available. Eventually you run out of super-sized cloud nodes to run them. Have personally had to provision a 300GB memory node during a cardinality explosion, and would like to not deal with this ticking time bomb in the future. Complexity & Pickup</h2> While the original setup can hardly be considered trivial, splitting one component into 3 sounds like a simple addition in theory. However, while splitting a monolith might sound like a nice idea, actually operating such a distributed monolith is a different proposition. The existing complexity is splattered across 3 components in statefulsets and helm template gunk, and the operational complexity is compounded by these components not being as battle tested as the more traditional prometheus/thanos setup. 3 Single Points of Failure</h3> You will need improved alert coverage with the previous single point of failure getting split into 3 parts: ruler query failures</a> means no alerts get evaluated even though metrics exists in the system.</li> agent mode downtime means evaluation works, but metrics likely absent</li> receive write failures / downtime means rule evaluation will fail</li> </ul> The setup suffers from the "who alerts about alerting failures" problem. A single deadman's switch is necessary, but not sufficient; as ruler</code> can successfully send the WatchDog alert</a> to your external ping service, despite the agent being down. The mixins</a> provide a good starting point for the new thanos components that can be adapted, but it takes a little time to grok it all. Configuration Splits</h3> Since thanos-ruler</code> is the new query evaluator, and we use PrometheusRule</code> crds, these crds must be provisioned into thanos-ruler</code> via prometheus-operator</code>. On the helm side, this only works with kube-prometheus-stack</code> creating the ThanosRuler</code> crd (which prometheus-operator</code> will use to generate thanos-ruler</code>), in the same way this chart normally creates the Prometheus</code> / PrometheusAgent</code> crd (to generate the prometheus</code> or prometheus-agent</code> pair). Specifically, we have to NOT enable ruler</code> from the bitnami/thanos</a> chart, and have a thanos component live inside kube-prometheus-stack</code> instead. Not a major stumbling block, but goes to show some of the many sources of confusion There's a bigger stumbling block for thanos-receive</code>, but more on that in the speculation section. </blockquote> New Features, Slow Iteration</h3> Agent mode, with a writing ruler also feels fairly new (in prometheus time), and support for all the features generally takes a long time to fully propagate from prometheus, to thanos, to the operator. As an example see; keep_firing_for</code>: Nov 2022 :: Raised in prometheus</a> (me, lazy)</li> Feb 2023 :: Implemented in prometheus</a></li> June 2023 :: Support in prometheus-operator for PrometheusRule</a></li> Jan 2024 :: Support in thanos</a></li> March 2024 :: Support in prometheus-operator for ThanosRuler</a></li> </ul> So as you can see, it's a long chain where ruler</code> sits at the very end, and to me this it is indicative of the amount of use ruler</code> realistically gets. To drive that home, I've also had to upstream remote write ruler functionality in the chart</a>, and my minor issues in thanos</a> sit untouched. Anyway, not really trying to shame these projects, things take time, and volunteer work is volunteer work. But the clear outcome here is that many features are not necessarily very battle tested. Performance Problems</h2> Unfortunately, the performance from this setup (after weeks of tuning) was still 2x-3x worse than the original HA prometheus pair setup (again from the 2022 post</a> / 1st diagram above). These new subcomponents individually perform worse than the original prometheus, and have worse scaling characteristics. Rule evaluation performance also seriously deteriorated. Benchmark</h3> Comparison is made using prometheus v2.53.1</a> and thanos 0.36.0</a> and consider mean</code> utilisation measurements from cadvisor metrics on a cluster with a ~2.5M</code> time series per prometheus replica. We only consider the biggest statefulsets (receive, prometheus/agent, ruler, storegw). In either setup receive</code> or prometheus</code> were running on a 3d</code> local retention. Results</h3> Over full workdays we saw ~13 cores</code> constantly churning, and ~80 GB</code> of memory used by the 3 statefulsets (10 cores and 50GB alone from receive</code>): </a> Compare to the same setup using a normal HA prometheus (no ruler, no receive, local eval) and we have ~4 cores</code> and <30GB</code> memory: </a> So cluster wise, we end up with a between 2x-3x drop by switching back to non-agented, monolithic prometheus. Receive Performance</h3> From the same graphs we see that the portion of prometheus that got factored out into thanos receive, is using roughly 2x the CPU and memory of a standalone prometheus, despite not doing any evaluation / scraping. Addendum: Configuration Attempts</summary> Tried various flags here over many iterations to see if anything had any practical effects. new ketama hashing algorithm</a></li> --enable-auto-gomemlimit</code> - barely helps</li> --tsdb.wal-compression</code> - disk benefit only afaikt</li> --receive.forward.async-workers=1000</code> - irrelevant, receive does not forward requests in our setup</li> </ul> The receiver was run as minimally with 3d</code> retention, and 1 replication factor, 2 replicas. More about this later. </details> Agent Performance</h3> The agents (which now should only do scraping and remote write into receivers) are surprisingly not free either. From graph above, the memory utilisation is close to a full prometheus! There is at least one open related bug for this</a>. Ruler Performance</h3> Ruler evaluation performance when having to go through queriers is also impacted, and it surprisingly scales non-linearly with number of ruler replicas. This panel evaluates sum(avg_over_time(prometheus_rule_group_last_duration_seconds[1h])) by (pod)</code> per pod over three modes: up until 08/06 12ish :: 2 replicas of thanos ruler</li> middle on 08/07 :: 1 replica of thanos ruler</li> end on 08/08 :: 2 replicas of prometheus (non-agent mode)</li> </ol> As you can see the query time increases within each pod when increasing replicas. Possibly this is load accumulating in the new distributed monolith, but a near 50% spike per ruler? In either case, the actual comparison of 3s avg</code> vs 50s avg</code> is kind of outrageous. Maybe this is misreading it, but the system definitely felt more sluggish in general. No rules generally missed evaluations, but it got close to it, and that was not a good sign given it was in our low-load testing cluster. Beyond this, this component is nice; seemingly not bad in terms of utilisation, and easy to debug for the basics. Ruler metrics docs</a> and sum(increase(prometheus_rule_evaluation_failures_total{}[1h])) by (pod)</code> in particular were very helpful. Speculation</h2> ..on why it performs like this, and on whether we are expecting too much from this setup. Removing Colocation</h3> This is tiny brain post-rationalisation, but maybe having a big block of memory directly available for 3 purposes (scrape / storage / eval) without having to go through 3 hops and buffer points (ruler → query → receive) is a big deal. Split Receivers</h3> There is a lot more complexity under the surface of for actually running receive</code> well. I ran the basic setup, and probably paid for it. For people that need to go deeper; there's a split receiver setup</a>, and a third-party controller to manage its hashring</a> that people recommend to avoid write downtime</a> (not a problem I even noticed). By using it right, supposedly we get to double the utilisation again</a>. My lazy take here is that if the system performs badly with replication factor 1, the prospect of more complexity and a futher utilisation increase is not particularly inviting. Even if such a system scales, paying your way out of it with this much pointless compute resources feels wrong. Bigger Evaluation Window</h3> There is a chance that a good portion of the ruler</code> time has come from going through thanos-query</code>. Routing like this was a deliberate choice so that people could write alerts referencing more than 3d</code> (local retention) worth of data to do more advanced rules for anomaly detection. This should not have impacted most of our rules since most do not do this type of long range computations.. I tried moving the ruler</code> query endpoint directly to receive</code> to try to falsify this assumption, but this did not work from ruler</code> using the same syntax as query/storegw. Wrong Tool For the Job</h3> Agent mode on the prometheus side seems perhaps more geared to network gapped / edge / multi-cluster setups than what we were looking for judging by the grafana original announce</a> + thanos receive docs</a>). It’s also possible that other solutions perform better / are better suited, e.g. grafana mimir. This is all speculation. Confusing Agent Promise</h2> Perhaps the most confusing thing to me is that agent mode does not act like an agent. You cannot run it as a DaemonSet</code>, you merely split the monolith out into a distributed monolith. This is a present-day limitation. Despite people willing to help out, the issue</a> remains unmoving. I had hoped google would upstream its actual statefulset agent design (mentioned in the issue), but so far that has not materialised. Who knows if agent mode will even become stable</a>. On the grafana cloud side, the grafana agent</a> did support running as a daemonset</a>, but it is now EOL</a>. It’s been only 3 years since agent mode was announced</a>. Now, 2 years later the whole remote write protocol is being updated</a> and just landed in prometheus</a>. So, what I am trying to say; who knows what the future really brings here. It might be another couple of years before new remote write gets propagated through the thanos ecosysystem. Scaling Alternatives</h2> Sharding</h3> Maybe the better way forward for scaling is not to twist prometheus into something it's not - and create a staggeringly complex system - but by making more prometheuses. For instance; prometheus operator's sharding guide</a> can help partition a classic prometheus, but you do need partition and label management, uneven shard request (cpu/mem) management (due to some shards scraping more metrics and thus having more load), so it's definitely on the more manual side. brain: ..you would also need to split kubelet metrics across namespaces (or whatever you use as your shard) via some templated servicemonitor, and you'd need a bunch of templated datasources in your master grafana that your dashboards would need to be parametrised for. Maybe you also need one main-cluster prometheus that can scrape all the kubelet metrics for cluster-wide views. </blockquote> Not impossible, but clearly also an amount of faff. This is standard configuration faff though; not distributed systems faff. Lowering Local Retention</h3> If the problem is delaying scaling up to something complex, we could also lean on the classic thanos split and keep reducing local prometheus retention</code> time down to a single day or lower (as long as you are quick on detecting sidecar failures so you don't lose data). This is a temporary solution though. On my homelab I can run 30d</code> retention, but with 5M time series - in a company setting - I need 3d</code> to maintain a sensible utilisation. Cardinality Enforcement</h3> This is the "unscaling" approach I run in my homelab. Granted it is easier to justify there, but there are real concrete steps you can do to really reduce the prometheus utilisation: drop big histograms (easy) / move to native histograms (..some day)</li> dropping pod enrichment (big replica counts X histograms = lots of cardinality, impossible without stream aggregation tho)</li> Monitor your ingestion: by (job)</code>, before and after relabellings, put alerts on fixed ingestion numbers</li> Make sure everyone uses {Service,Pod}Monitor</code>s so above step is feasible</li> Drop most of kubelet metrics (most metrics are unused by dashboards or mixin alerts)</li> </ul> I'll probably explore this approach in more detail later on, because I think it's the most sensible one; dilligence on the home court avoids all the complexity. In the mean time, post is on mastodon</a>, source is in the probes repo</a>. Future</h2> No matter how you slice it, agent mode with thanos is certainly a complex beast whose configuration entangles a huge number of services; agent, operator, receive, query, store, compactor, ruler, adapters, alertmanager, grafana. You have a choice in how difficult you make this. The performance characteristics measured above, while not initially impressive to me, is one point, but the complexity of the setup is what pushes it over the edge for me. If the stack becomes so complex that the entire thing cannot be understood if one key person leaves, then I would consider that a failure. This was hard enough to explain before receive</code> and agent mode. If I have to wrangle with cardinality limits, relabellings, label enrichment, or create charts to multiply prometheuses, then this all seems more maintainable than receive</code>. kubectl hackery 2024-08-11T00:00:00+00:00 Unusual helpers for cluster level resource debugging. Gratitude for 2023 2023-12-31T00:00:00+00:00 The year is coming to a close, and as a small amount of self-reflection of my relatively fortunate and stable situation, here's a list of things in life that made me happy. Video</h2> In general, I like well presented video exposés, impressive video game footage, and watch competitions from everything from piano competitions to SC2 esports tournaments. Some frequented returns: yt/BobbyBroccoli</a> :: for great long form science controvery deep dives</li> yt/Tantacrul</a> :: for a yearly hinge expose on music</li> yt/hbomberguy</a> :: for his yearly expose on who knows what</li> yt/thejuicemedia</a> :: for their snappy government parody ads</li> yt/MonarchsFactory</a> :: TTRPG concepts and mythology</li> yt/mcolville</a> :: TTRPG game mechanics and game design</li> yt/chopininstitute</a> :: classical piano recitals and the chopin competition</a> (last winner</a>)</li> yt/HarstemCasts</a> :: intelligent starcraft 2 casts from top level player when you need a break</li> yt/ESLArchives</a> :: SC2/Dota2 tournament vods from ESL</a></li> yt/AfreecaTV</a> :: more starcraft: GSL</a> + ASL</a></li> yt/DoshDoshington</a> :: for well-narrated Factorio Mod runthroughs</li> yt/KimuraTails</a> :: insane Trackmania (united) TASes</li> yt/cncf</a> :: boring conference talks on cloud tech, but they allowed me to avoid some flying</a></li> yt/HealthyGamerGG</a> :: Dr K. with a pragmatic stoicism + meditation focused philosophy takes</li> </ul> Would I still watch as much youtube without 2X</code> on all speech or with ads? Probably not. Long form media is a lesser toll on my continually stretched focus though. </blockquote> As for actual full-runtime content, most my streaming services were cancelled last year, and the only one left is crunchyroll, which I guess makes me a full time weeb now. I would fight that, but Love is War</a> and Spy x Family</a> were also the funniest shit this year to me. Hardware</h2> 2023 was an upgrade year, a first since 2017, so have once again delved into the increasingly confusing realm of hardware for my #linux</a> desktop workstation. Wendell from Level1Tech</a> + GamersNexus</a> were the most helpful for guiding me in good linux-compatible setups. Ended up going full AMD</code> to make my time with Wayland</code> less painful</a>, and with less funny flags</a>. The new rig is great. It has ~4x faster compile-iteration cycles, and it uses less power than my old one! Games</h2> On the steam front, my timesinks were heavier games for a change. One even from this year: Baldur's Gate 3</code> obviously. I have spent days just theorycrafting</a> its predecessors, and it's GOTY for a reason. Great storytelling, lots of great game mechanic ideas for making 5e</code> better. RIP September.</li> Borderlands 3</code>. BL2 still holds a special place in my heart for its chill coop, item hunting / diablo vibes, so this year, when r/bl3</code> started being positive again - after years of annoying users by breaking the game, nerfing builds, changing the endgame - and my PC got upgraded, it was time. The weapon farming system is fun, the builds are cool, and the Cthulhu themed expansion really stands out as exceptionally beautiful. RIP July.</li> Manifold Garden</a> was the surprise hit. Its infinitely repeating take on first-person puzzle solving, ventures into a deeply psychadelic fractal space, and one that is captivatingly beautiful.</li> </ul> The amount of theorycrafting videos consumed during the RPG periods also shows that the desire to min-max is still very much alive. Coffee</h2> To stay woke, I rabbit-holed into home-espresso making (like many of my tech friends), and am now able make a great cup with pretty cheap components (basic delonghi dedica + non-pressurized portafilter + a grinder), given good beans as a result. While the famous hoffmann</a> (or joffman</a>) was helpful for his insights into coffee science, I did not need to min-max this. That said, fun side-project. I occasionally try to replicate the perfect asian dirty coffee, or a thai style es-yen, but most of the time I just pull a shot, ice it, and pour over some oatly for a great lazy iced latte. Open Source</h2> The kube-rs</a> ecosystem continues with 13 new kube releases</a> this year. Lots of features landed; socks5 proxying</a>, rustls default</a>, new configs for watcher</a> + Controller</a>, streaming lists</a>, controller streams interface</a>, oidc refresh</a>, metadata api</a> + metadata_watcher</a>, store readiness</a>. Thankfully, I can largely PM the ship, and leave most larger features to others. Stream sharing</a> is close, k8s-pb</a> integration can be started, client-v2</a> is restarted, and am enjoying expanding docs on kube.rs</a> (particularly the controller guide</a>) to simplify future Q/A + upskill adopters. Stepping outside kube, python-yq</a> has been rewritten in rust</a>. This is both a fun and necessary</a> project for me (not solved by the go rewrite</a>). This is a small project, because we are just deferring</a> to jq</a>, but it is very satisfying how simple</a> things can be when you are not trying to re-invent the wheels</a>. Helix</h2> The helix editor</a> (replacing VS Code) is the first modal editor I've managed to stick to, and a big reason for this is how compassionate the UX is</a>; menus give me forgotten shortcuts, commands are searchable, and the speculative LSP integrations are amazing. Am actually using helix as my new #pkm</a> rather than foam</code> because the markman lsp</a> actually lets me do most of what I expect from a second-brain anyway! Symbol search for H1s or H2s, and Goto definition</code> for wikilinks are both unexpected LSP features. Positivity</h2> In a world full of systemic problems, mute + block has never felt more valuable. I have aggressively hit Dont recommend channel</code> on captive news channels, and Indefinite mute</code> on excessively drama-peddling / self-rightous profiles mastodon even when I largely agree with their general points. With global problems, the cost of exposure is a draining negative energy; powerlessness, constant exposure to virtue signalling / schadenfreude, and increasingly polarised and unhealthily dichotomous thinking, E.g. climate change discourse, frequently followed by either: denial - thanks to the existence of unrealistic and equally problematic solutions</li> anger - e.g. via activists that advocate for engaging in large scale industrial sabotage</li> </ol> Yes, i believe we are pretty fucked, but the fucking will continue whether I worry about it or not. </blockquote> Twitter was terrible in always exposing you to the angriest threads about this. Thankfully, in its demise, Mastodon is an improvement for what I was using twitter for; cross-link to/from blog-posts, follow posts from experts, and moderately engage while bored on a train. Here the Mute</code> option actually works, negative echo chambers quickly get defederated, and you can*</a> move instances. You could make several good arguments for leaving twitter</a>, but that was not the job of this post. I have found a new place</a>, and it's a happier one. If you want more positivity, you must opt-out of some negativity. Pick your battles. </blockquote> Mountains</h2> This year changed my approach to exercise, changing my goals from setting PBs in running events like 2022</a>, to focus on fun, and exploring nearby mountains when they are available. The biggest treks were the 5h</code> trail runs up Doi Pui</a> + Croix du Nivolet</a>, and the 3d</code> long Kumano Kodo</a> pilgrimage trail with my partner. I also found incredible beauty and focus on hills requiring much less effort such as Doi Suthep</a>, Fløya</a>, & Tindstinden</a>. There is a concrete beauty to trail running and hiking that I did not get while optimizing for performance. You are always present in the mountains. No podcasts, no "get-it-done" mentality; the journey is its own reward. The experience lasts, unlike a VO2max</code> bump. Bigger hikes are less realistic outside of travel for me, so it's still mostly flats. Air-travel is becoming a harder-to-defend privilege, but I have to recognise that it's not really fair for a society to offload climate guilt</a> on consumers despite a complete lack of systemic regulation. Like, I am car free and child free, but it's fucked that I feel the need to use that as an excuse. </blockquote> Routine</h2> Beyond the above, my life is still largely routine: Walk to the park. Grind the coffee beans. Blitz some fruit. Triage issues. Play music. Appreciate beauty. Try to embrace being content, and try to be compassionate. Enjoy time with my partner. </blockquote> Hoping that can continue in 2024. Here's to another year! Talk log from KubeCon Chicago 2023-12-22T00:00:00+00:00 As time goes by, I find myself increasingly disinterested in actually travelling out to a convention, when my preferred method of consuming conference talks is overwhelmingly VOD form with 2x/FF potential. This is doubly so when the convention is increasingly corporate (like KubeCon), and the CNCF youtube channel</a> is in general high quality. Get it before your ad-blockers break. This post contains a quick export from my personal notes on some significant talks at KubeConNA'23 (make of them what you will). My interest areas this kubecon fall broadly into these categories (and will group talks by these): observability related :: maintain a lot of metrics related tooling</li> continuous deployment :: maintain a lot of ad-hoc cd tooling</li> security :: maintain a bunch of controllers and ad-hoc validations</li> networking :: traffic apis, mesh and other network sledgehammers (using occasionally)</li> kubernetes :: everything else.. cool to see where the platform i build on top of end up going</li> maintainership :: am trying to scale kube-rs</a> beyond myself</li> </ul> Observability</h2> How Prometheus Halved It's Memory Storage</a></h3> nice journey into prometheus space optimization and how the layout of their labels matter a lot. great work that everyone who upgraded prometheus this year benefitted from. Evaluating Observability Agent Performance</a></h3> good overview of where the utilization overhead comes from and how to think in terms of backpressure. OpenTelemetry: What's Next</a></h3> they're graduated now and deal with metrics (but still experimental w.r.t. rust). imo it feels really strange to be shoehorning everything here into one agent, but we'll see what benefits it can bring later (they keep talking about consistent metadata between them). Exploring the Power of Metrics Collection with OpenTelemetry</a></h3> after the 45m+ workshop, they get into the horrific setup they've made inlining scrapeConfigs inside collector crds to support metrics. kind of look at this, and all i can think is "why tho?". KEDA Graduation Announcements</a></h3> looks increasingly nice. (i need to kill the prometheus-adapter garbage template format i currently deal with.) keda_ metrics, scaling modifiers, pausing, job scaling, prom caching, all seem like very nice features. All You Need to Know About Prometheus in 2023</a></h3> very interesting talk if you are invested in this ecosystem. they mention keep_firing_for</code> landing which feels great (because it was my suggestion</a>). q/a session reveal they think the otel collector is an anti-pattern for metrics (ruins active monitoring) and collector's unconventional label use fucks with perf/predictability. worth keeping in mind if you consider using otel agent for metrics. How and Why You Should Adopt and Expose OSS Interfaces Like Otel and Prometheus</a></h3> This one was funny (to me, not actually funny). Google Monarch (their internal monitoring tool) exposing a promql interface so they can translate it to monarch. lots of work for something people complain about so often (promql). goes to show that the thing people complain about is the thing people actually use. Continuous Delivery</h2> Flux 2.0 and Beyond; OCI + Cosign</a></h3> OCI feels like a better distribution method for Kubernetes yaml than with helm, and flux's source-controller</code> pulling oci packaged tarballs has a nice flow to it. OCI + Kustomization sounds workeable. their selling points: colocation of artifacts + images + signatures</li> passwordless auth + keyless integrity verification</li> increased cd/flux controller efficiency</li> </ul> personally, i just want to distance myself from helm upgrade/releases as much possible and this is a nice + efficient way of doing that. Wolfi: Intro to the Linux Undistro</a></h3> wolfi idea remains the same, and they got a lot of momentum behind it and chainguard. decent build system, but lots of yaml. Keeping Helm Reliable, Stable, and Usable</a></h3> kind of a boring talk, but kept it in here to highlight one point; helm is slowing down because it's the defacto standard. so do not expect any new major features, WYSIWYG (including gotpl, toYaml | nindent, manual schemas)... Security</h2> Security Showdown: The Overconfident Operator Vs the Nefarious...</a></h3> entertaining and great talk about problems with wide access on laptops. Arbitrary Code & File Execution in R/O FS – Am I Write?</a></h3> nice exploit demo of readOnlyRootFileSystem</code> and ways to bypass the ways it can be enabled. some truly horrendous and ugly reverse shell setups and /dev/termination-log</code> abuse.. The Cluster Killer Bug: Learning API Priority and Fairness the Hard Way</a></h3> nice intro to flowschemas, apipriority and fairness through a motivating bug example. An accompanying talk would be Kubernetes DoS Protection at Google Scale</a>. RBACdoors: How Cryptominers Are Exploiting RBAC Misconfigs</a></h3> hiding techniques for cryptominers if you ever had cluster admin access, and removed it later. decent talk. Networking</h2> Gateway API: The Most Collaborative API in Kubernetes History Is GA</a></h3> this is a big deal. it's needed to get canaries everywhere with HTTPRoute</code>, and it's cool to hear them talk about a mature rollout strategy for experimental crd fields. tbh, it was more fun to hear this framed as a way to improve Service</code>"the worst api in kubernetes"</a> UX Matters: Switching to GAMMA Without Ruining Your Reputation</a></h4> linkerd on working with the gateway api and issues and lessons with policy controller</li> presents rust in a way that's "because it's cool" but a bit manual atm. (..i'll take it)</li> mentioned their leader election impl and kubert. candid and moderately intersting.</li> </ul> Istio Past Present and Future</a></h3> fun dig at the rust evangelists: "rewrite it in rust? [..] no we want to be a lot better" </blockquote> then talk about their single ambient mesh thing that still ends up with 39% cpu overhead 22% memory overhead. maybe they should rewrite it in rust. jkjk. this does seem like a nice improvement for them. ran istio a while back and found the overhead insane. When Is a Secure Connection Not Encrypted? and Other Stories</a></h3> talk about the main working principles behind cilium</code> and its mutual encryption. very interesting wireguard + spiffee setup. they had a lot of momentum behind them. let's see if that continues after Cisco buys them (i know how that works). Demystifying Cilium: Learn How to Build an eBPF CNI Plugin from Scratch</a></h3> while we are on the cilium train. great workshop about how a CNI can be built. Kubernetes</h2> Building Better Controllers</a></h3> with my kube-rs hat on, there are some cool ideas coming out of istio here (would be nice if they published it). the ideas here could be partially implement yourself in your code, but it's interesting to see them commit to an interface like this at the controller level. Node Size Matters - Running K8s as Cheaply as Possible</a></h3> opencost and their metrics. does a nice investigation into their metrics and proving that small cloud provider instances are the most expensive instances you can use if you can fill your nodes. Cutting Climate Costs with Kubernetes and CAPI</a></h3> climate aware scheduler idea using watttime data, priorityclasses and KubeSchedulerConfiguration</code> to allow only running workloads during "low emission times". a later (much fluffier talk) shows how to integrate this with KEDA</a> What's up with Kubernetes Long Term Support?</a></h3> mentions the numerous recent KEPs to improve stability: KEP-1333 - 1.19+ ensures all APIs required to run clusters are GA</li> KEP-1693 - New APIs are not allowed to be reqired until they graduate to GA</li> 1.19+ has metrics for deprecated resource use</li> KEP-1194 - KEPs require better reviews w.r.t. production readiness</li> Deprecation policy</a> updated to make stable api versions permanent</li> KEP-3136 - New unstable APIs are OFF by default</li> KEP-3744 - Kubernetes 1.23+ use supported go versions (easier to bump security fixes going forward)</li> KEP-3935 - Kubernetes 1.28+ control plane nodes support n-3 version skew (annual upgrade now ok) otherwise talks about difficulties of widening support window too much (upgrade complexity, ecosystem dependencies, bugfix porting)</li> </ul> they want a longer cycle (WG-LTS</code>) - because currently >2/3rds of people have clusters out-of-support this feels like a companion talk to Swimming with the current make it easy to stay up to date</a> which also highlights how much regressions are a problem in Kubernetes (and mostly on patch releases..). What's New with Kubectl and Kustomize</a></h4> kubectl diff --prune</code> had a selector bug still...</li> kubectl auth whoami</code> new</li> </ul> can't help lol when they say you shouldn't use most of the imperative subcommands; {create, run, expose, autoscale, replace, rollout undo, edit, set, patch, scale, ..} Nix Kubernetes and the Pursuit of Reproducibility</a></h3> building a nix hypervisor around libvirt</code> with qcow2. pretty cool idea. nice sweetspot for nix in hypervisor space, because talos/flatcar/bottlerocket is probably better for VMs, and wolfi is better for docker images. Pods and Circumstance: CRI-O Graduation Celebration</a></h3> CRI-O metric setups with kubelet/cAdvisor and how they plan to optimize it. they use conmon-rs</a> - a rust lib for container runtime monitoring! Grifts Ahoy! Bracing for the AI Tide</a></h3> advocates for soft-AI as a tool to help generate useful context for small niche areas. mentions we are likely on the top of the S curve (before the trough of disillusionment) and there's tons of hype and not much focus on risks, tons of exaggerations, whether it's better than non-AI, or whether it even uses AI at all - and it needs some laws. Mentions a bunch of boring AI risks to consider (not the more crazy AI singularity transhumanist hype): hard to control risks when you don't know what the model is actually doing</li> research has shown it's very cheap to poison a model (60$ for data control of 0.1% - buying expired domains)</li> can be used against us; malware creations (easy to bypass protections by lieing to it)</li> deepfakes (passed a liveness test and scammed shanghai tax system)</li> ai often hallucinates package names (can typosquat those)</li> </ul> great talk. Declarative Everything</a></h3> My favourite talk about my favourite new feature in Kubernetes. Admission Validation and admission policies. talked more about it on mastodon</a> and ended up writing kube.rs/admission</a> as a result Safeguarding Clusters: Exploring the Benefits and Navigating the dangers of admission controllers</a></h3> goes into details about footguns (failurePolicy</code>, latency buildup, default timeout, scope), and some more exotic crazy failures if you accidentally block leases or flowschemas. notes how it was hard to target negative selections until CEL; matchConditions can be CEL inside webhook configuration resource now! 15,000 Minecraft Players Vs One K8s Cluster. Who Wins?</a></h3> very good talk about how a cloud minecraft provider moved from GCP to bare metal with "65% cost reduction". nice to see the stuff they take advantage off (still off-load some stuff to cloud providers), and heavy use of the cluster api. MinIO and TopoLVM for storage is also very cool. if nothing else, this talk is worth it for how they deal with lifecycle management of long lived games (how do you terminate the pods?). short answer; some automatic upgrades with cluster api, some planned maintenances with warnings and then very long terminationGracePeriodSeconds</code> on SIGTERM</code> with some advanced termination handlers. Maintainer Track</h3> Tools for Resolving Difficult Conflicts in Open Source Communities and Projects</a></h3> imo the most useful thing herein is the highlighting of non-violent communication</a> as a methodology for compassionate communication, proven to have good results (but best if practiced consistently). The Eight Fallacies of Distributed Cloud Native Communities</a></h3> similar maitainer points: maintainer bw is (not) infinite :: lack of control + lack of empathy towards you = recipe for burnout</li> compromise is (not) a rarity and (not) the norm :: everyone has their own agenda</li> cost of contributor onboarding is (not) zero :: maintainer bw is not infinite, ownership also hindered by undocumented context => episodic maintainers leave. need to put concious effort into uplifting and growing existing contributors to avoid gridlock.</li> staffing across areas is (not) homogeneous :: some hard areas have very few people who knows what is going on</li> </ul> Kubeburned Out? How to get things done efficiently</a></h3> good tips for contributing by making routines for contributions if employed (e.g. 1h before work, maybe every tuesday + thursday, 1-2h during weekend). make your work public: always communicate pr status</li> ask for help if stuck</li> unasign if you cannot work on it, maybe add snippets that may help others</li> </ul> recognise people for their work; blog posts are good</li> celebrate small achievements</li> </ul> say no - keep yourself healthy; take breaks, short or long</li> </ul> good advice, but then it's more about how to take on more responsibility within kubernetes, writing KEPs, and TAG work. timing improves success; propose things at the right point in time</li> raise questions at the right point in time (so you're likely to get the right answer)</li> establish an async schedule more likely to get people working together with you</li> </ul> find habits that feel right for you. if it's not high priority for you or who pays you, don't work on it. Drop Down Terminal 2023-10-01T00:00:00+00:00 As a former Quake player, I have been a long-term user of guake</a> for a quick access terminal on Linux. However, as guake</code> is gnome specific, continued use became problematic once a mac got thrust upon me for work. So herein is a workaround for making your own drop-down terminal for Linux and Mac by utilizing alacritty</a> + zellij</a>. Looks</h2> Behold; a semi-transparent, tabbed terminal hovering above two browser windows (showing the editing process of this very post): </a> This is using a transparent theme in helix</a>, with 80% opacity in alacritty, and disabled blur in hyprland</code> to selectively see through below (like for say live updates to the markdown rendering). On mac, the setup and looks are very similar to this. For times when transparency is too annoying you can bind a key to change the helix</code> theme</a> to something fully opaque (most themes are opaque). A quick F1</code> press ultimately slides the terminal away from view anyway. </blockquote> Core Concept</h2> The idea is to have a persistent terminal that is hidden away in a "special place" and bind a key to toggle the visibility of this terminal. Then we rely on zellij</code> to provide tabs. The choice of terminal emulator and multiplexer is not super important. You could probably use kitty</code> + tmux</code>, or straight wezterm</code> (which has tabs mgmt built-in) if you prefer, and achieve similar results. </blockquote> On wayland that place is a special workspace, whereas on mac we use sticky</code> windows. Alacritty Setup</h2> My preferred setup is to always launch zellij</code> at alacritty</code> start by configuring the default shell in the alacritty.yml</code> config: shell: program: zsh # start zellij with a small sleep to ensure it gets right dimensions args: - -ic - sleep 0.2 && zellij -l compact </code></pre> and, since this is a drop-down terminal we are selling, you probably also want transparency (because otherwise why do you need the window to go in front? just switch applications or workspaces): window: opacity: 0.8 </code></pre> on linux we can also make it as clean as possible and remove most window decorations: window: decorations: none startup_mode: Maximized </code></pre> whereas on mac a similar decoration setup is buttonless</code>: window: decorations: buttonless startup_mode: Maximized </code></pre> (and you can also experiment with binding a key to ToggleSimpleFullscreen</code> there). Zellij Setup</h2> Honestly, this is pretty nice out of the box. Mostly styling is nice here to make it cleaner: pane_frames false ui { pane_frames { hide_session_name true } } </code></pre> grab a theme</a>. The -l compact</code> startup option set in the alacritty config assumes familiarity in zellij</code>, you're likely to find it easier to onboard if you don't use compact mode until you are familiar. </blockquote> Tab Names</h3> If you don't want to faff around with naming every single zellij tab, you can also make zsh</code> do it automatically for you using chpwd_functions</code></a> using something like this in your .zshrc</code> etc: zz() { if [ -n "${ZELLIJ_SESSION_NAME}" ]; then zellij action rename-tab "${PWD##*/}" fi } # ensure directory traversal updates tab names (if terminal mux exists) chpwd_functions+=zz zz # initialize name for new tabs/panes </code></pre> This is nice because it catches all sources of traversal, be it through cd</code> or through jumpers like zoxide</a>. Drop Down Variants</h2> The most ergonomic one here is the hypland variant, but the mac setup is also decent. Hyprland</h3> On Linux with hyprland</a> (my current wayland compositor), plugging this in is very easy to do because you have access to special workspaces</a>, so the terminal is never technically hidden, it's just active in a different workspace. This might not seem like a big deal, but if you've tried sharing a terminal that hides itself (such as guake's or through the mac solution below) on a screen sharing video call, you will find various failure modes (from app crashes to stream closes) if you hide the window you are sharing. </blockquote> In your hyprland.conf</code>, add: exec-once = [workspace special] alacritty bind = , F1, togglespecialworkspace </code></pre> so it auto-starts in special, and you can toggle the workspace as you wish. Give it an animation so it pops in vertically, and avoid undercutting your opacity: animations { animation = specialWorkspace, 1, 1, default, slidevert } decoration { # don't dim modal terminal dim_special = 0 blur { enabled = false # make it easy to see-through from special workspace } } </code></pre> And the final hyprland quirk; special workspaces have a window scaling factor (default 0.8</code>) that controls how much large the window is allowed to be. If you are putting a terminal editor in there or something you probably want to max that out: dwindle { special_scale_factor = 1 # maximize special workspace } </code></pre> The end result is a terminal that pops in from the bottom in an inverted quake terminal feel - and honestly this makes more sense than top-to-bottom since you have to give some space to the (usually top) waybar</code>. Hammerspoon</h3> On Mac, the easiest setup is using hammerspoon</a>. Bind a toggle or start alacritty</code> in your hammerspoon's init.lua</code>: local function toggleApp(name) local app = hs.application.find(name) if not app or app:isHidden() then hs.application.launchOrFocus(name) elseif hs.application.frontmostApplication() ~= app then app:activate() else app:hide() end end -- Global terminal toggle hs.hotkey.bind({}, "F1", function() toggleApp("alacritty") end) </code></pre> It is a little flimsy on a multi-monitor setup, with sometimes having to manually move it across to a workspace on a different monitor on boot (window/workspace/monitor hierarchy seems less clean on mac in general), but it helps to not auto-start the app and let the bind start it on the monitor you have your cursor by only pressing the bind there. The alacritty opacity setting is respected out of the box. If you are using it with yabai</a> (for auto-tiling of windows, window moving shortcuts), then you also want to add a rule for yabai</code> to mark it as sticky to avoid it getting bunched up as a normal window tile: yabai -m rule --add app="^(Alacritty)$" sticky=on </code></pre> Because it is using sticky windows, screen sharing of this window will not be viable by itself (as the source disappears when it's hidden), so you'll have to share the entire workspace instead. This is usually not a big deal if you are already using workspaces. Xorg</h3> All setups I found for this were comically bad, but including one for completeness. Don't use this. Stay on guake</code> if you are on X</code>. The hack below majorly struggles with multi-monitor. Aside: you should consider moving onto wayland</a> if you haven't. </blockquote> You need a script that auto-runs in your shell, and a runnable function: # autorun via .zshrc if [ -n "${ZELLIJ_SESSION_NAME}" ] && [ ! -f /tmp/wraise ]; then # If running in zellij on linux, save the window for refocus keybinds xdotool getactivewindow > /tmp/wraise fi # function somewhere else (e.g. ~/.functions) # setup a bind run: /bin/zsh -c 'source .functions && terminal_toggle' terminal_toggle() { local -r terminal_id="$(cat /tmp/wraise)" # Check if it is active (stored in hex on a root prop) local -r active_id="$((16#$(xprop -root _NET_ACTIVE_WINDOW | choose 4 | cut -d'x' -f2)))" if [ $active_id -eq $terminal_id ]; then xdotool windowminimize "${terminal_id}" else wmctrl -ia "${terminal_id}" fi } </code></pre> This expects alacritty</code> is on autorun and you can configure binds somewhere. I put an F2</code> bind in cinnamon</code> (yeah, long time between changing WMs for me) to run this. Again; I don't use this. I wrote it as a short stop-gap and I hated it. </blockquote> Having to support Mac + X in my dotfiles</a> became a perfect storm of frustration, and wouldn't you know it, wayland made everything better. ..as with many things, the answer is to migrate away from X</a>. Running 2022 2022-12-07T00:00:00+00:00 In an attempt to torture my chair shaped husk of a body, I somehow ran almost 2500km this year. Mostly to see what would happen, but also to take part in the Royal Parks Half Marathon in London</a>. Milestones</h2> In the process of going up to 70km weeks - something which became routine remarkably quickly - I ended up beating pretty much all of my endurance records: 41:51 10k</a></del></li> 39:29 10k</a></li> 11:15 3k</a></li> 1:29:34 HM</a></li> </ul> The Half Marathon was my first longer event and feel I ran it cleanly: The 3k PB is also a 3200m</code> Cooper test</a> which, heuristically</a>, puts my VO2max at 60. All in all, very pleased with the progress. Particularly getting 10k sub 40 and HM sub 90. Mileage</h2> Garmin report</a> for the year: Month</th> Mileage</th> VO2max</th></tr></thead> Jan</td> 117k</td> 51</td></tr> Feb</td> 65k</td> 52</td></tr> Mar</td> 142k</td> 53</td></tr> Apr</td> 144k</td> 54</td></tr> May</td> 190k</td> 55</td></tr> Jun</td> 157k</td> 54</td></tr> Jul</td> 266k</td> 56</td></tr> Aug</td> 322k</td> 57</td></tr> Sep</td> 333k</td> 58</td></tr> Oct</td> 249k</td> 58</td></tr> Nov</td> 349k</td> 58</td></tr> Dec</td> 10k</td> covid</td></tr> </tbody></table> Q1/Q2: Volume Building</h3> The year started slow, but increased a little too fast initially without muscle training and ended up spending a lot more time on an elliptical in February. Back in London, when my foot muscles started catching up, I was able to raise the mileage quite a bit come summer (with an average of 60k weekly in July). This gave me confidence to start a pretty ambitious training plan from the end of July. Q3: Royal Parks</h3> Combined ideas from two generally available training plans: AgeUk's Sub 1:30 12w</a> - for general setup, but taking out some of the races</li> NSPCC Sub 1:30 14w</a> - for negative split long runs and more thresholds</li> </ul> also incorporated ~two sessions of muscle strength training a week (single leg dead lifts, weighted squats, hungarian squats, heel/soleus raises, plus misc core/oblique strength exercises). Ended up averaging 71k</code> weekly mileage during this block. Week</th> Mileage</th> VO2max</th> Fast %</th></tr></thead> 1</td> 60k</td> 56</td> 6k (10%)</td></tr> 2</td> 61k</td> 56</td> 7k (11%)</td></tr> 3</td> 66k</td> 56</td> 11k (17%)</td></tr> 4</td> 67k</td> 57</td> 14k (20%)</td></tr> 5</td> 74k</td> 57</td> 13k (17%)</td></tr> 6</td> 79k</td> 56</td> 14k (18%)</td></tr> 7</td> 69k</td> 57</td> 20k (29%)</td></tr> 8</td> 79k</td> 57</td> 10k (13%)</td></tr> 9</td> 77k</td> 57</td> 17k (22%)</td></tr> 10</td> 89k</td> 58</td> 24k (27%)</td></tr> 11</td> 79k</td> 58</td> 17k (21%)</td></tr> 12</td> 57k</td> 58</td> 24k (42%)</td></tr> </tbody></table> and calculated the following zonal paces: 3:32/km -> 17km/h ~= Z6 for mile?</li> 3:45/km -> 16km/h ~= Z5 for 5k (18:45)</li> 4:00/km -> 15km/h ~= Z4 for 10k (40:00)</li> 4:16/km -> 14.1km/h ~= Z3 for HM (1:29:47)</li> 5:00/km -> 12km/h ~= Z2</li> </ul> My journal with key-workouts for these weeks: 1; 6x1k</code> at ~4:10/km moltemyr</a>, 24k long tromøy</a>, 60km</li> 2; 6x400m</code> at 3:32/km (aimed for 1:24)</a>, 20min fast lysløypa</code></a>, 21k long</a>, 61km</li> 3; 4x1M</code> at 4:05/km greenwhich</a>, 1x5k</code> at 4:15 track</a>, 15k long</a>, 66km</li> 4; 4x1M</code> at 4:04/km</a>, fartlek ~5k</code></a>, (swapped out 10k race), 20k long with 3k</code>@4:15</a>, 67k</li> 5: 5x1k</code> at 3:45/km</a>, 5k</code> at 4:08</a>, minor hills</code></a>, 20k long</a>, 74k</li> 6: 8x500m</code> at 3:42/km</a>, 5k</code> at 4:35</a>, 21k long with 5k</code> at 4:08</a>, 77k</li> 7: 8x1k</code> at 4:09/km</a> (stitch), 6x1min</code> at 3:29</a>, 10k race at 3:57</a>, 69k</li> 8: 6x1k</code> at 4:09/km</a>, 10x1min at 3:40/km</a>, 18k long with 1k@4:18</a>, 79k</li> 9: 2x3k</code>+ 2x1500</code> at 4:04</a>, 21k long with 8k at 4:15</a>, 77k</li> 10: 2x3k</code>+2x2k</code> at 4:07</a>, 2x2x400m at 3:30</a>, 21k long with 10k at 4:15</a>, 89k</li> 11: 5k race attempt</a>, 4x1M at 4:01</a>, 18k long with 6k at 4:15</a> (big headwind), 79k</li> 12 taper week (intensive, but short mileage 60%, muscle monday only, 2k speed</a>, 1k hills</a>), then HM</a>, 57k</li> </ul> By many metrics (Garmin VO2max estimate, Strava fitness), I peaked nicely for the race: and ended up feeling OK after the race. Got one blood-shot big toenail to show off for a year, but will leave you with the mental image. Recovery</h4> 2 weeks of pretty low effort (40k, 50k), no speed, run every other day. Q4: Chiang Mai</h3> Second block. 8w. A bit of an impromptu registration for the Chiang Mai Marathon (Half), and a bit of fun that to test how my training translate across continents to a new humid environment. Unfortunately, this block got cancelled near the end. 1: 6x1k</code> at 4:00/km</a>, 19k long</a>, 67k</li> 2: 7x1k</code> at 3:57/km</a>, 21k long</a>, 80k (last in London)</li> 3: 3.5x1M</code> at 4:05/km</a>, 700m climb long</a>, 72k (Chiang Mai)</li> 4: 4x1M</code> at 4:05/km</a>, 20k long</a>, 85k</li> 5: 4x1M</code> at 4:03/km</a>, 22k long</a>, 92k</li> 6: 8K recovery week, 8k</code> MAF test</a>, and covid 🪦, 38k</li> </ul> Got covid two weeks ahead of the race. First time, but there are ample</a>, warnings</a> about running hard post-covid. On the positive side, with an ungodly 4AM start</code> for this race and a 90%</code> average humidity (which tanked my running efficiency); having to cancel a difficult masochistic exercise you set purely for yourself is not universally a negative. Q4: Covid</h3> Unfortunately, it had been one full year since my last (3rd) vaccination shot - and along with probably being hit with one of the Q</code> variants in southeast asia - this disease was not fun. Just as one symptom disappeared another came in to replace it. While my stress graph on the worst day looked pretty benign (mostly medium stress), my partner's stress graph on Garmin was the worst I have ever seen (with >10 hours of high stress in one day). Still, the fever was pretty bad for me, and am now getting light headed doing easy things 6 days on, so it's definitely not "just a cold". BMJ has a Graduated Return To Play Guidance Protocol</a> and based on only having some "below-the-neck" systemic symptoms (fever), I am only planning on doing light jogging during the second half of December if/once 🤞 it's possible to: complete all activities of daily living without excessive fatigue and/or breathlessness and be able to walk on the flat without getting breathless </blockquote> So, yeah. Use your masks while indoors people. Each time you get reinfected, the risks increase</a>. Hopefully, I can find a way back. Random Thoughts</h2> Next year</h3> Provided recovery allows; I hope to keep the volume. It is a bit much - and just doing a fraction of it will for sure be enough to just stay healthy - but I kind of want to see how far I can stretch it for a bit longer. It's also made me feel sharper. Want to train specifically for shorter distances like 1M / 3000 / 5k, even though beating PBs at this points does come with 8+ hours of mandatory training a week and injury risks so how much this is carried on next year is heavily conditional on nothing bad happening - a statement generally not served with good odds in the 2020s. Not planning on higher distance events like marathons just because of how detrimental it seems to be to your overall immune system</a> and how much recovery you need. My current line of thinking is; why train for 4 months for an event that takes you out of the game for 2-4 weeks? Better find a smooth average that you can sustain. brain: he says this while planning on beating PBs; requiring the opposite of smooth averages. </blockquote> Anyway, running has been a fun side-project that's not been software related. It's been interested reading about how professionals train</a>, and seeing effects first hand. Shoes</h3> The silliest thing about this accessible sport though; is the amount of shoes you burn through. If you run 100k</code> a week, and follow the 500-800k</a> recommendation for max shoe mileage, you'll blast through 5-10 pairs of running shoes a year, costing you somewhere from £500-2000</code> per year for high end shoes. The £60</code> I paid to run in Royal Parks was actually worth it just for the 15% off</code> coupon at a running shoe store. On the plus side, this is temporary. Wish you all good health in 2023. Factober 5400 2022-10-30T00:00:00+00:00 Round two of escapism this year. This is a post for myself with various information as a conclusion to my recent fixation into designing an efficient and compact factorio megabase</a>. Plan</h2> I keep coming back to this game about once a year. There is something very soothing about the type of play this game encourages. Playing factorio basically feels like being part of some parallel software project; there's builtin dashboards, circuit logic, sharable blueprints, benchmarks, optimizations, and all without the usual externalities of a large software org/project. And at this point for me, it's all about the optimization and theorycrafting. I set out with the following goal: produce 2 full blue lanes of every science or 5400 SPM (Science Per Minute) </blockquote> Certain cheats were used to avoid the less challenging parts of the game: spawning ore / resources (to minimize ore replacement and transport work)</li> speeding up the game (to simulate idling / expansion and simplify moving around)</li> disabling pollution (as the biters were contained and later defeated anyway)</li> </ul> this lets us focus on the parts of the game that's more interesting to me; optimizing the factory without having to deal with stuff running out or setting up new chaotic trains all the time. It also enables us to optimize the medium sized factory pieces in complete isolation without suddenly being limited by ore or UPS (that's the equivalent part of managing a software project I am trying to avoid). Copy-pasting</a> small cell bases</a> is currently the leading way</a> to get the most productive bases without choking the game (i.e. maintaining 60 UPS). </blockquote> We set out to make each science build be entirely self-contained with just a single train station as an output, and we built upon an old starter base that had already achieved almost 200 mining productivity. Sideprojects</h3> But before we get into it, here are several other things we built along the way that (in true factorio style) did not in any way contribute to the main goal. Wildlife Preserve</h4> ..to justify all those military upgrades: which we eventually eradicated in a comically inelegant way: </iframe> </div> Fishing Outpost</h4> A sustainable ecosystem catching 600 fish over its lifetime on its own power/robo network. Built right above the launchers: Wood Burner</h4> An automated train service to get rid of all that wood: </iframe> </div> (normally triggers automatically, but there were no wood in the requester chests). Factory Designs</h2> Rails</h3> When starting out with production/purple science I originally had 16 full blue belts of rails that fed into the purple science factories. Now we make rails inline in a single factory that feeds 4 purple science assemblers: I've not seen anyone else do this, but it works pretty well. It sushies sticks around two columns, and does a bit of crate magic to fit 6 input inserters, 3 input lanes, and 4 output inserters inside a 12 beaconed bounding box, but it actually works in the end. Green Circuit / Plastic Row</h3> One of the more optimized factory pieces. A single lane that produces a mostly full, mixed plastic/green blue belt. It requires very precise spawning of coal, iron, copper so that we can have 2 chemical plants for plastic, and 4 smelters of iron + copper for a single 12 beaconed green factory. Some of the time all the furnaces are running: The green factory is what requires special attention here: One green factory alone can actually ALMOST supply a half belt (22.4 out of 22.5)</a> if you can fully beacon it. It's just that doing this is very difficult because it takes more than a full blue belt of copper cables that it is very hard to supply it while making room for all the the iron and the outputs. This is why there's a car in between the beacons that serves as a chest. A car can be picked up from and inserted to at very strange angles at more than one length so it's great for fitting irregular gaps that doesn't work with chests (or would require more inserters). The game doesn't want you to do this though. It cannot be copy-pasted so you have to plant it manually. Doshington showcases the pain this type of design leads to</a> . You actually have to remove one middle beacon when the inserters starts running to trick the inserters to lock onto the car before planting the beacon down again (otherwise they want to grab the speed modules). The end result is that you can transfer 24 cables to the assembler through a one space gap on every swing cycle. Red Circuit Row</h3> Continuing onwards from the green / plastic row, is another inline copper smelter section that joins into a nearly full copper cable assembler, that sufficiently feeds 8 factories of red chips. The red chips are also inserter clocked so they all swing together which looks very satisfying, but I don't think it actually helps with UPS. This joins onto the green setup so we can end up with these huge stacks for large red factories like this one for RCUs: Low Density Row</h3> A full row of low density structures can consist of 10 fully beaconed factories supplied by two lanes of copper</a>. It needs a lot of belts, and manual input setup, plus a bunch of inserters to load onto the belt and back in from the belt, so we searched for a more direct setup (without sacrificing too much at the number of machines). The end setup has 20 smelters to 10 LDS assemblers, which is an extra 4 smelters than is needed, so we beacon them slightly less, but end up with 100% direct insertion on copper. The downside? Lots and lots of cars. This 70/s</code> LDS factory for space science consists of 14 rows of 10 cars each, which is definitely a hassle to setup and filter. Ideally the equally sized steel would be lined up to face LDS on the input side (like they are in yellow science below), because here we mix belts first then use splitters later just to avoid a 14 lane wide thing through the base. Regardless, it looks different to everything else and is kind of beautiful at night: Enjoying the efficiency of the ratios</a>. All the smelters are running (94% avg utilization). Labs</h3> Extracted this more conventional lab design to test it out UPS Wars 6: Labs</a> for this with this entry</a>. Nothing remarkable, except perhaps for the long winding belts in different direction. It's slightly overscaled to ensure it will fully drain half belts. Trains</h2> All of the science cities / space component cities unload onto one of two major unload depots. The main depots have robo networks to distribute nuclear fuel from a single supply train and they only stop at the hub (which is enough). Lab Depot</h3> Lab depot looks like a close-up of a complicated circuit: Very happy with this unload station. We pull exactly 4 half-belts and that can be perfectly filled via two inserters (with an 8 size override) at the cost of less buffer chests (that we don't need those). No splitters anywhere! Rocket Depot</h3> Probably not the most efficient train unloading, but gotta get 4 wagons of stuff into 6 launchers via 2 inserters per launcher so it's tricky. Lot's of 2x3 splitters and an eventually emptying unloader with a priority line down to satellites... It's ultimately a little overscaled compared to the other sciences so a few seconds of semi-filled belts, while not particularly efficient, doesn't hurt production. Nuclear Fuel</h3> A minimal fuel factory that supplies all trains (piggy-backing on any oil refinery because it uses basically nothing). It's almost never active because it only has destination stops available when the hubs have less than 200 items on the receiving chests (circuit network on the receiving side). Science Cities</h2> Quickly going through these with factorio calc links. Red</a></h3> Max beaconing reds would give an awkward ratio whereas doing pairs here gives us just enough with 12F each column. It would be possible to do 11F we did some at higher beaconing though. Green</a></h3> Max beaconed smelters and an ad-hoc <4 half belt output of green circuits, and only max beaconing the last pack factory to fill the lane. Again, max beaconing is just not enough with 12F per column pair, so doing pairs with one last max beaconed for 14Fx4. Zero splitters. More gear factories than necessary, but more direct insertion. both red + green on the map: Military</a></h3> Annoyingly large for something that's off most of the time. Lots of clocked smelters, and the actual main ingredient factories are tiny.. Did a cool inline wall thing for each row for the actual science pack, and a big grenade factory, but lots of room for improvement. One fun thing here though; this factory shuts off its own power when the train station buffer chests is full and</code> no trains are at the station (circuit network connected to the chests and the train station via a combinator). This saves on power, but not on UPS, sadly. Chemical</a></h3> All ore/oil/reds + inputs for reds on the left through the green car lane setup, and then on the right inline sulfur (turns out one chemical plant can supply 45/s chemical science). Uses basic oil here because it's not the most demanding, but other peoples' comments suggest it's probably a wash performance wise. Production</a></h3> a huge chunk of the factory is literally just dealing with steel for rails and furnaces: which is a slightly overscaled (as can be seen from the few inactive furnaces), but it's a bit unevenly drawn from atm. Utility</a></h3> Slightly overscaled (60F) and undersupplied (can supply 59F). 57F ended up in an awkward 3 column build that needed to be split a lot. Best to stick to 4 columns if you are going to load onto 4 wagons. Oil is split into two advanced oil parts; inner one for lube/sulfuric and the outer just for petroleum for LDS. Space Science</a></h3> Can just be done with six rocket launchers running near constantly provided we have the ingredients. RCU</a></h4> Slightly overscaled because the speed module row looked awkward when optimal. Probably one of the most streamlined parts, but also most heavily abuses ore spawning (mixed lane inputs for both speed modules and blue processors). Blue build is from Stevetrov's cell base with a slight beaconing modification. Tried tons of blue builds for this and they're either all massive (like this), or they need double the amount of inserters for slightly more production, or they are smaller, but need so many belt inputs of plates that it's super awkward to plot down and less efficient. Didn't benchmark this for UPS, just went with gut feel based on smelter uptime, inserter numbers and inserter activity. LDS</a></h4> Supplies both rockets and satellites. RF CF</a> + RF2</a></h4> Partially from coal liquefaction just because i wanted to learn how to do it well. It's probably not the most optimal choice since it needs lots of mixed coal belts to get the coal in that deep (yes, yes). CF build is undersupplied with 12 refineries, but doing it that way allows us not to copy paste the advanced oil one another time. It ends up being almost perfectly scaled for the rockets + satellites. Belts just start to fill before the next train arrives. All the solid fuel factories piggy-back on the ore clock for less swings. Pretty happy about the beaconing line-up with the refineries and chemical plants: Satellites</a></h4> The final component. It's better to look at calculator links for Solar</a> + Accumulators</a> + Processors</a> (the real output part of this chunk) because we leech LDS + RF from the trains. It's a pretty ugly mess that was designed early, but it's clocked and beaconed everywhere. Oh, and there's only 5 inserters per sulfuric acid factory because cars. Optimization</h2> We tried several major optimization strategies: inserter clocking</a> on all ore smelters/batteries/red circuits/steel</li> enabling large pages on linux</a></li> inlining green chip production (i.e. less belts in favour of direct insertion)</li> max beaconing almost everywhere</li> reducing robo networks to only the essentials</li> </ul> and in general downscaling and optimizing flow: minimizing all builds (as close to zero inactive machines as possible)</li> reducing number of active inserters, belts, hops per item</li> </ul> Beyond the linked posts, the following technicalfactorio subreddit</a> posts, and factorio benchmarks</a> were also helpful: bot/belt advice</a></li> splitters + belt advice</a></li> cars as chests benchmark</a></li> inserter minimal train unloaders</a></li> Trains waiting on TIME PASSED && Full/Empty</code> condition is more efficient than just Full/Empty</code></a></li> Minimizing number of bots so their usage is close to 100% is the best</a></li> </ul> We did not go to the full-out direct insertion lengths that the most efficient bases go to where everything is inlined (because imo it ends up looking chaotic and huge, with lots of factories ending up idle a lot of the time), but it was truly fascinating to see the amount of rabbit holes you can go down here to optimize. From the limited amount of testing I personally did. These helped: solid improvement by eliminating pollution (pre-measurements)</li> ~10%</code> improvement by using large pages</li> ~10%</code> improvement by clocking all smelters/batteries/solid fuel</li> ~10%</code> improvement by using clocked direct insertion steel and killing robo networks</li> ~1%</code> improvement by disabling the 300+ cars</li> ~0.5%</code> improvement by removing the standalone fishing base</li> ~1%</code> improvement by cutting 6x7x4 inserters at science unloading stations</li> </ul> while these had no noticable improvements: refactoring circuit clocks to run of the same central clock</li> eliminating biters (mine are generally passive except for artillery revenge attacks)</li> eliminating grass/trees/fish/cliffs</li> reducing visual quality of the game</li> prepending time passed conditions to trains</li> </ul> Initial Measurements</h2> When doing this set of measurements the base looked like this: After all the optimizations above; we measured</a> the following rough percentages of time of where the time was being spent in the game's update loop: Source</th> Percent</th> ms</th></tr></thead> Entity update</td> 67%</td> 4.45ms</td></tr> Elecric network</td> 13%</td> 0.85ms</td></tr> Heat manager</td> 9%</td> 0.62ms</td></tr> Transport lines</td> 8%</td> 0.57ms</td></tr> Circuit network</td> 2%</td> 0.18ms</td></tr> Trains</td> 1%</td> 0.06ms</td></tr> </tbody></table> where the entities were broken down further</a> using show-entity-time-usage</code></a>: Entity Class</th> Percent</th> ms</th></tr></thead> Inserter</a></td> 46%</td> 1.80ms</td></tr> Generator</a></td> 15%</td> 0.60ms</td></tr> Boiler</a></td> 11%</td> 0.44ms</td></tr> Furnace</a></td> 8%</td> 0.31ms</td></tr> Assembler</a></td> 7%</td> 0.29ms</td></tr> Miners</a></td> 5%</td> 0.19ms</td></tr> Labs</a></td> 1%</td> 0.04ms</td></tr> </tbody></table> which illustrates why inserter-clocking can be so effective - as it's always such a big part of a factory - but it also highlights the next offender; nuclear. Heat manager (heat exchangers) uses 9% of the update time</li> Generator + Boiler uses 26% of entity time (0.67=17% of the update time)</li> </ul> This is actually a over a quarter of the time. Not what I had expected from a not-that-badly-designed nuclear setup (no steam storage, minimal exchanger network, optimal reactor efficiency bonuses). It's a lot more than people publically estimate</a>, but to give them some credit, the percentage seen on my end would be amplified by already having optimized the hell out of all the inserters. We ran the base on two big nuclear reactors sites producing 33GW in total. </blockquote> This reactor block gave us ~60% of the power output with 120 of the 204 total reactors. Opening up the save in creative mode and ripping out nuclear in favour of an infinite energy interface bumped UPS from 150 to over 200.. how would that translate if we used nearly a million solar panels instead? how much space would that take? how much time would that take... Solar Project</h3> Well, I had already sunk this much time into it, so thought I'd spend a weekend on it. How fun could it be? Well, the answer is somewhat.. Provided you shut off the rest of the factory (so you can run 4-8x speed), and then run boost around with a bunch of rc'd spidertrons: </iframe> </div> with some strategic landfill factories taking advantage of huge mining prod: Still, definitely a repetivite background task (that took two days..) ..but how did it measure up? On gamespeed 5</code> we saw a near 40% improvement to UPS from 150</code> to 208</code>. </blockquote> Breaking the new measurements (1</a>, 2</a>) down, we see: Source</th> Percent</th> ms</th> Change</th></tr></thead> Entity update</td> 71%</td> 3.12ms</td> -1.33ms</td></tr> Transport lines</td> 13%</td> 0.57ms</td> 0</td></tr> Elecric network</td> 11%</td> 0.48ms</td> -0.38ms</td></tr> Circuit network</td> 4%</td> 0.17ms</td> 0</td></tr> Trains</td> 1%</td> 0.06ms</td> 0</td></tr> Heat manager</td> 0%</td> 0.00ms</td> -0.62ms</td></tr> </tbody></table> and looking into the entities: Entity Class</th> Percent</th> ms</th> Change</th></tr></thead> Inserter</a></td> 72%</td> 2.29ms</td> -0.49ms</td></tr> Furnace</a></td> 11%</td> 0.35ms</td> ~0</td></tr> Assembler</a></td> 9%</td> 0.29ms</td> ~0</td></tr> Miners</a></td> 6%</td> 0.19ms</td> 0</td></tr> Labs</a></td> 1%</td> 0.04ms</td> 0</td></tr> Boiler</a></td> 0%</td> 0.01ms</td> -0.43ms</td></tr> Generator</a></td> 0%</td> 0.00ms</td> -0.60ms</td></tr> </tbody></table> (There's likely only precision to one decimal place here as we saw saw 0.04ms changes in furnace and assemblers despite having no actual changes in game. The numbers do have some variance on their own in the big base and we are just operating on screenshots.) The update loop itself saw a nearly equivalent improvement with 6.24ms to 4.53ms. End Performance</h3> The UPS went from around 110</code> to 208</code> on average from redesigns, clocking and the mentioned tweaks, where a good chunk of this came from the base design itself. The most braindead change with the biggest improvement was switching to solar (with nearly 40% improvement at the time - your mileage will vary). The changes with the most performance gains by time investment was turning on large pages (5min job), and turning off + clearing pollution (1min job). In the end, the island now looks like this: i.e. largely the same, but without nuclear, and solar everywhere. how much solar? oh i'm sorry, that screenshot was cropped.. yeah. that's a lot more than expected. When designing solar, ensure you overscale by 42%</a> to cover the night. Blueprints</a></h2> Exported these pieces: </a> Commands</h3> Expand Commands</summary> One time commands: remove pollution</a></li> Set /c game.speed=5</code> to try to run as fast as possible</li> </ul> Spawn resources: /c game.player.surface.create_entity({name="crude-oil", amount=10000000, position={game.player.position.x, game.player.position.y}}) </code></pre> Delete resources (of any kind) in a 10x10 square around you: /c local surface=game.player.surface local size=10 local pos=game.player.position for _, e in pairs(surface.find_entities_filtered{area={{pos.x-size, pos.y-size},{pos.x+size, pos.y+size}}, type="resource"}) do e.destroy() end </code></pre> Spawn a 2x4 water pond</a> below you: /c local waterTiles = {} for y=2,4 do for x=-2,2 do table.insert(waterTiles, {name="water", position={game.players[1].position.x+x, game.players[1].position.y+y}}) end game.players[1].surface.set_tiles(waterTiles) end </code></pre> Disable cars (see the cars as chests benchmark</a>): /c for _, ent in pairs(game.player.surface.find_entities_filtered{name = "car"}) do ent.active = false end </code></pre> Large pages</h4> Install mimalloc</a> and set start command on factorio via steam: LD_PRELOAD=/usr/lib/libmimalloc.so MIMALLOC_PAGE_RESET=0 MIMALLOC_LARGE_OS_PAGES=1 %command% </code></pre> Settings are documented via mi-alloc</a>. Large pages makes the most difference, but huge pages also works well. </details> Results</h2> In the end, all the optimizations meant we could run the 5400 SPM base at 210 UPS on my machine (i.e. the game managed to run the game at 3.5x real speed on average). With 20% productivity modules in labs, and that game speed scale, we effectively research ~23k science per minute, or 1.4 million science per hour. If you want to view it interactively or play it, you can download or view full base/save on factoriobox</a>. Baldur's Gate: Multinomial Edition 2022-04-12T00:00:00+00:00 In a brief</del> bout of escapism from the world and responsibilities, I booted up Baldur's Gate 2</a> with my brother. It's an amazing game, once you have figured out how to roll</a> your character. For today's installment; rather than telling you about the game, let's talk about the maths behind rolling a 2e</code> character for BG2</code>, and then running simulations with weird X</code>-based linux tools.

In Practice</h2>
Doing this type of enumeration is helpful as a way to tell how close you are to your theoretical 100% optimised system, but how does the number hold up in practice?</p>

`CX Dashboards</h3> The screenshots here are from my own homebrew cx-dashboards</a> released separately. See the future post for these.</p> Links / Comments</h2> Posted on mastodon</a>. Feel free to comment / raise issues</a> / suggest an edit</a>.</p>`

Links / Comments</h2>
Posted on mastodon</a>. Feel free to comment / raise issues</a> / suggest an edit</a>.</p>

Removing Colocation</h3>
This is tiny brain post-rationalisation, but maybe having a big block of memory directly available for 3 purposes (scrape / storage / eval) without having to go through 3 hops and buffer points (ruler → query → receive) is a big deal.</p>

Observability</h2>

Continuous Delivery</h2>

Security</h2>

Networking</h2>

Kubernetes</h2>

Declarative Everything</a></h3>
My favourite talk about my favourite new feature in Kubernetes. Admission Validation and admission policies.</p>
talked more about it on mastodon</a> and ended up writing kube.rs/admission</a> as a result</p>

Recovery</h4>
2 weeks of pretty low effort (40k, 50k), no speed, run every other day.</p>

Factober 5400

Fishing Outpost</h4>
A sustainable ecosystem catching 600 fish over its lifetime on its own power/robo network. Built right above the launchers:</p>
</p>

Factory Designs</h2>

Science Cities</h2>
Quickly going through these with factorio calc links.</p>

Baldur's Gate: Multinomial Edition

probes

Running a Minimal Prometheus

You Will (Not) Scale Prometheus

kubectl hackery

Gratitude for 2023

Talk log from KubeCon Chicago

Drop Down Terminal

Running 2022

probes

Running a Minimal Prometheus

How Many Signals</h2> MAIN POINT: You should be able to enumerate the basic</strong> signals that you want to have visibility of.</p> </blockquote> Let's do some simplified enumeration maths</strong> on how many signals</strong> you actually want to properly identify failures quickly.</p>

In Practice</h2> Doing this type of enumeration is helpful as a way to tell how close you are to your theoretical 100% optimised system, but how does the number hold up in practice?</p>

Things You Don't Need</h2> Sub Minute Data Fidelity</h3> You are probably checking your homelab once every few days at most, so why do you expect you would need 15s/30s data fidelity? You can set 60s</code> scrape/eval intervals and be fine:</p>

Sub Minute Data Fidelity</h3> You are probably checking your homelab once every few days at most, so why do you expect you would need 15s/30s data fidelity? You can set 60s</code> scrape/eval intervals and be fine:</p>

Cardinality Control</h3> Drops 97% of kubelet metrics</strong>, configures minimal kube-state-metrics</code>, node-exporter</code> (with some cli arg configurations and some relabelling based drops) and a few other apps. In the end we have a cluster running with 7000 metrics</strong>.</p>

You Will (Not) Scale Prometheus

3 Single Points of Failure</h3> You will need improved alert coverage with the previous single point of failure getting split into 3 parts:</p>

Removing Colocation</h3> This is tiny brain post-rationalisation, but maybe having a big block of memory directly available for 3 purposes (scrape / storage / eval) without having to go through 3 hops and buffer points (ruler → query → receive) is a big deal.</p>

Split Receivers</h3> There is a lot more complexity under the surface of for actually running receive</code> well. I ran the basic setup, and probably paid for it.</p>

Scaling Alternatives</h2> Sharding</h3> Maybe the better way forward for scaling is not to twist prometheus into something it's not - and create a staggeringly complex system - but by making more prometheuses.</p>

Sharding</h3> Maybe the better way forward for scaling is not to twist prometheus into something it's not - and create a staggeringly complex system - but by making more prometheuses.</p>

kubectl hackery

Gratitude for 2023

Video</h2> In general, I like well presented video exposés, impressive video game footage, and watch competitions from everything from piano competitions to SC2 esports tournaments. Some frequented returns:</p>

Talk log from KubeCon Chicago

Observability</h2>

Continuous Delivery</h2>

Security</h2>

Networking</h2>

UX Matters: Switching to GAMMA Without Ruining Your Reputation</a></h4> linkerd on working with the gateway api</strong> and issues and lessons with policy controller</li> presents rust in a way that's "because it's cool" but a bit manual atm. (..i'll take it)</li>

Kubernetes</h2>

What's New with Kubectl and Kustomize</a></h4> kubectl diff --prune</code> had a selector bug still...</li>

Drop Down Terminal

Xorg</h3> All setups I found for this were comically bad</strong>, but including one for completeness. Don't use this</strong>. Stay on guake</code> if you are on X</code>. The hack below majorly struggles with multi-monitor.</p>

Running 2022

Recovery</h4> 2 weeks of pretty low effort (40k, 50k), no speed, run every other day.</p>

Factober 5400

Fishing Outpost</h4> A sustainable ecosystem catching 600 fish over its lifetime on its own power/robo network. Built right above the launchers:</p> </p>

Factory Designs</h2>

Science Cities</h2> Quickly going through these with factorio calc links.</p>

In Practice</h2>
Doing this type of enumeration is helpful as a way to tell how close you are to your theoretical 100% optimised system, but how does the number hold up in practice?</p>

Removing Colocation</h3>
This is tiny brain post-rationalisation, but maybe having a big block of memory directly available for 3 purposes (scrape / storage / eval) without having to go through 3 hops and buffer points (ruler → query → receive) is a big deal.</p>

Recovery</h4>
2 weeks of pretty low effort (40k, 50k), no speed, run every other day.</p>

Fishing Outpost</h4>
A sustainable ecosystem catching 600 fish over its lifetime on its own power/robo network. Built right above the launchers:</p>
</p>

Science Cities</h2>
Quickly going through these with factorio calc links.</p>