In Norway, where I am from, wine is centralized through Vinmonopolet. That comes with a long list of political, cultural, and practical tradeoffs, but from a consumer perspective it has one very convenient side effect: discovery is easy. If you want to know whether a wine is available, who imports it, what it costs, and where it can be picked up, there is only one obvious place to look.

Denmark, where i currently live, is different. There is no wine monopoly. The market is more open, more fragmented, and in many ways more exciting. Independent shops, small importers, niche webshops, auction platforms, and specialist stores all coexist. That is part of what makes the Danish wine scene interesting. But it also makes discovery harder.

If you are looking for a specific producer, vintage, grape, region, or bottle, you often end up searching shop by shop. You might check one webshop, then another, then a few importers you remember from Instagram, then a private marketplace, then maybe a PDF price list somewhere. If you are lucky, you find what you were looking for. If you are not, you never know whether the wine was unavailable or whether you simply searched in the wrong place.

Find That Wine was my attempt to solve that problem.

The idea was simple: gather publicly available wine listings from Danish wine shops and present them in one searchable interface. The site would not sell anything. It would not take a commission. It would not try to become a marketplace. It would simply help people discover where wines were available, then send them directly to the shop that had the bottle.

In theory, everyone wins. Consumers find wine faster. Shops get more organic traffic. Smaller importers become easier to discover. Public product data becomes more useful.

In practice, the project became more complicated than that.

This post is a retrospective on what I built, how it worked, what I learned from building most of it with Codex as an assistant, and why I eventually decided to shut it down.

The problem: Denmark has abundance, but not one index​

The Danish wine market has a lot of charm precisely because it is not centralized. There are serious specialist shops with deep Burgundy allocations, natural wine importers with tiny producer portfolios, auction listings, private collections, Italian specialists, Austrian specialists, Champagne shops, hybrid bottle shops, and plenty of small webshops with their own angle.

That diversity is good. It also means the market does not have one shared discovery layer.

If you know exactly which shop imports a wine, the problem is easy. You go there. If you know the importer behind a producer, the problem is still manageable. But if you only know the wine, or if you are exploring a category, the work shifts to the consumer.

I kept running into questions like:

• Who has this producer in Denmark?
• Is this bottle sold out everywhere, or just at the shop I checked?
• Which shops carry what im looking for?
• Is an auction listing the only place this bottle appears right now?

These are not exotic questions. They are normal consumer questions. The frustrating part was that the answers often existed in public, just scattered across many websites.

So the project started with a consumer-first assumption: if the data is already visible on public shop pages, then a search engine that points users back to those shops should be useful.

That assumption carried the project technically. It did not fully carry it socially or legally.

What the site was meant to do​

The product goal was deliberately modest.

Find That Wine was not meant to be a Danish Wine-Searcher clone with subscriptions, merchant dashboards, price history, paid placement, or lead generation. I wanted something smaller and more direct:

1. Load inventory from Danish wine shops.
2. Normalize the listings into one shared format.
3. Let users search across all stores at once.
4. Let users filter by store, producer, category, tags, stock status, and price.
5. Show enough information to evaluate a result.
6. Link directly to the seller for purchase.

There was no checkout. No basket. No payment flow. No margin. No affiliate tracking. No advertising model.

The site was free because the point was not to earn money. The point was to make life easier for consumers aka myself and, ideally, send better-qualified traffic to the wine shops.

That was the optimistic version of the project: a public, neutral discovery layer for Danish wine.

Normalize everything​

The core technical problem was not rendering a search page. It was making different sources look similar enough that one search page could work.

The project used a simple normalized item model. Each listing became a flat object with fields such as:

type NormalizedItem = {
  storeId: string;
  storeName: string;
  storeBaseUrl: string;
  productId?: string | number;
  productTitle: string;
  vendor?: string;
  productType?: string;
  tags?: string[];
  variantId?: string | number;
  variantTitle?: string;
  sku?: string;
  price?: string;
  available: boolean;
  inventoryQty?: number | null;
  productUrl?: string;
  imageUrl?: string;
  updatedAt?: string;
};

This was not a perfect wine data model. It did not properly separate producer, cuvee, vintage, region, appellation, grape varieties, bottle size, farming, importer, scores, or allocations. It was a pragmatic model for a hobby project: enough structure to search, filter, sort, and link out.

That flat model turned out to be one of the better choices in the project. It made ingestion easier. Shopify products, scraped HTML listings, and parsed PDF rows could all become the same shape. It also made the frontend simple because it could treat every product as one list item regardless of source.

I had a more ambitious data model in mind. It imagined separate source entities, product entities, wine-specific attributes, variants, traceability, categories, regions, grapes, certifications, and so on. That would be the better long-term model. But it would also require much better source data than most public webshop listings provide.

For the version I actually built, the flat model was the right compromise.

Open Shopify APIs was a game changer​

The project became realistic because many Danish wine shops use Shopify.

Shopify exposes product data through a public products.json endpoint on many stores. For a hobby project, this is almost dangerously convenient. You can request a store's products, paginate through them, normalize them, and cache them as JSON.

In the end i had configured more than thirty configured sources, including Shopify shops and custom scrape sources. A scheduled job fetched Shopify stores, requested /products.json?limit=250&page=N, normalized every variant, and wrote the output to JSON files.

I even tried to be nice about it:

• The fetcher used a custom user agent identifying the project and contact address.
• It supported retries.
• It handled HTTP 429 rate limits with backoff.
• It waited between pages.
• It capped pagination to avoid runaway scraping.
• Only one run per 24 hours
• Respect robots.txt

This was the part of the project that felt cleanest. A scheduled job could refresh data periodically, the frontend could load cached JSON, and users got fast search without each query hitting every shop.

For shops already on Shopify, the integration required no custom partnership and no invasive data access. The same public product data that powered their storefront could power discovery.

But that convenience also created one of the central tensions: just because something is publicly reachable does not automatically mean every shop wants it aggregated, cached and displayed somewhere else. Which is fair. Its their livelihood after all.

But not every source looked like Shopify​

The more interesting wine sources were not always Shopify stores.

The project had several custom ingestion paths for sources that did not expose a neat product feed. Some sources were ordinary web pages. Some were auction listings. Some were PDF price lists. Each of them needed a slightly different way of turning public information into structured search data.

For HTML pages, the scraper looked for repeated product or listing blocks and pulled out whatever useful fields were available: titles, prices, links, images, stock signals, seller information, colors, vintages, and auction status when relevant. The exact markup varied from site to site, so the scraper had to be written around the structure of each source rather than around one universal standard.

Some pages hid surprisingly useful data inside attributes or embedded JSON used by the webshop itself. In those cases, the scraper could extract cleaner product information than what was visible in the rendered text. Other pages required more basic parsing: reading category pages, following pagination, inferring product type from the section of the site, and deduplicating products that appeared in more than one category.

PDF lists were a different beast. Some importers and specialist shops still publish their assortment as a document rather than a webshop catalog. The parser tried to read those lists, identify producer names, regions, certifications, vintages, prices, grape varieties, bottle sizes, and product types, then turn the result into the same normalized JSON format as every other source.

This was fun engineering because every source was a small puzzle.

It was also fragile engineering. HTML changes. CSS classes change. Embedded tracking payloads disappear. PDF layouts shift. Product type labels vary from shop to shop. A scraper that works today can quietly produce poor data tomorrow.

A deliberately simple frontend​

The main user interface was a Next.js app using the App Router, React, and TypeScript. The architecture was intentionally simple: load all visible store data into the browser and filter client-side.

That sounds naive, but it worked for this scale. For a niche wine search tool, sending cached JSON to the browser and filtering client side was doable. It avoided building a search backend, database schema, hosted index, or API query layer before I knew whether the product should exist.

The search page did several useful things:

• Loaded the store manifest first.
• Loaded visible stores in parallel.
• Sorted initial store loading by file size so smaller stores appeared quickly.
• Built facets for vendors and tags client-side.
• Filtered by store, vendor, tags, product category, price range, stock status, and text query.
• Sorted by price, store, product, or vendor.
• Stored filter state in the URL.
• Expanded rows to show images, tags, product type, and outbound purchase links.

The row design was intentionally utility-focused. A result showed the store, product name, vendor, price, stock state, and category. Clicking expanded it. Available images were lazy-loaded from the upstream source only after expansion to avoid loading a large number of bottle photos unnecessarily.

The site treated different source types slightly differently in the stock badge. Normal webshop products showed in stock or sold out, auction-style listings were marked as auctions, and document-based entries were marked as coming from a parsed list.

It was not an advanced search engine, but it worked!

Categorization is hard work​

Wine data looks structured until you try to combine it.

One store might use Rødvin. Another might use red wine. Another might use Naturvin, which is not a color. Another might use a region like Bourgogne as the product type. Another might put important information only in tags. Some stores sell beer, spirits, cider, glassware, gift cards, subscriptions, tastings, olive oil, books, and accessories alongside wine.

The project had a product category configuration mapping raw product types into normalized categories:

• Red
• White
• Rose
• Sparkling
• Orange
• Sweet / Dessert
• Fortified Wine
• Sake - Umeshu
• Spirits
• Beer
• Cider
• Non-Alcoholic
• Imprecise Type

It also had an exclusion list for things that should not appear as wine search results such as gift cards and accessories.

It was a boring job to sort through endless amount of product type strings, but without doing it the search became polluted with vouchers, tasting events, and product bundles. The biggest problem was that every new store introduced new product type strings and every new string needed to be mapped or excluded.

The app even had product type monitoring. A nightly analysis could detect unmapped product types and expose it as metrics i could act upon.

That is a good example of how the project slowly became more serious than the original idea. I started with "search some wine shops" and ended up with category mapping, cache diagnostics, cron jobs, ingestion APIs, monitoring, and admin controls.

Data is messy​

Very early on it became clear that i needed more than json files to debug and administer the sources. The data ingestion needed supervision.

The portal was plain, but it supported:

• Checking whether each store had cached data
• Seeing item counts and last update times
• Forcing a refresh for a single store
• Refreshing all stores
• Adding, editing, and deleting store configs
• Toggling store visibility
• Inspecting sample products
• Viewing directory contents and diagnostics
• Running product type analysis

European infrastructure​

One infrastructure decision I cared about was keeping the project in Europe.

This was not because the application was technically complicated. It was a small website with cached product data, scheduled updates, and a few admin tools. It could have been hosted almost anywhere. But the project was about Danish wine shops, Danish consumers, and public data from the European market. For that reason, and that the US threatened to invade Greenland at the time, I wanted to keep things in Europe.

European infrastructure was also a way of keeping the project honest about its scope. This was not a global platform. It was not trying to optimize for the cheapest possible hosting or the most aggressive growth setup. It was a local tool for a local market. The infrastructure should reflect that.

On that note, i built it using an American AI, so that promise was quickly broken.

Building with AI​

Most of the coding was done with help from Codex, just ever so slightly guided by my own knowledge of SWE and by the concrete problems I wanted to solve.

I know the domain. I had opinions about what the product should do. I could read code, test behavior and ask for changes. Codex helped produce much of the implementation: Next.js pages, scraper modules, normalization logic and anything else i could imagine.

This made the project possible in a way it probably would not have been for me a few years earlier.

But AI assistance did not remove the need for judgment or manual intervention along the way. I am an engineer after all.

The uncomfortable questions​

The code was not the reason I shut the project down.

The project had bugs and rough edges, but nothing about the implementation was impossible. The bigger issue was that I could not get comfortable with three questions.

1. Am I breaking any laws?​

I am not a lawyer, and this post is not legal analysis.

But the question mattered. The site aggregated public product listings from shops, cached them, normalized them, displayed them in a different context, and linked back to the original sellers. That sounds harmless from a consumer point of view, especially when the data is already public and the site sends traffic back.

But "publicly visible" and "free to republish in aggregated form" are not the same thing.

There are possible questions around terms of service, database rights, copyright in product descriptions or images, robots.txt expectations, rate limits, commercial impact, and whether a shop has consented to being indexed this way. Even if the project is non-commercial, it still creates a copy of data and presents it as part of another service.

I tried to be as respectful as i could. The site cached data instead of hammering stores on every user search. It used an identifying user agent. It linked back to shops. It did not sell the products itself. It documented integration formats and invited shops to get listed or taken off the site.

Those choices and considerations help, but they do not answer the legal question.

2. Will this hurt my relationship with importers and stores?​

This question bothered me more than the purely technical ones.

The Danish wine scene is not a faceless market of commodity sellers. It is a network of small shops, importers, enthusiasts, and relationships. Many of the most interesting bottles come from people who care deeply about producers, allocations, customer trust, and how wines are presented.

From my perspective, Find That Wine was meant to help those shops. It could make their bottles easier to find. It could send them customers who were already looking for something specific. It did not take a commission or try to sit between them and the buyer.

But intent is not the same as perception.

A shop might see the site as helpful. Another might see it as scraping. Another might dislike price comparison. Another might worry about stale stock information. Another might not want certain products indexed because allocations are sensitive. Another might object to product images or metadata being displayed outside their own site. Another might simply prefer to be asked first.

All of those reactions would be understandable.

There is also a difference between "this is technically possible" and "this is polite." In a community that depends on trust, being polite matters.

3. Is the Danish market open to this idea?​

The third question was product-market fit, but not in the startup sense.

I did not need the site to become a business. I did not need revenue. But I did need the market to tolerate the idea.

A search platform for wine sounds obviously useful to consumers. It is less obviously welcome to sellers.

Price transparency can be uncomfortable. Aggregation can flatten the differences between shops. A wine that one importer presents with context, producer story, and allocation logic can become one row in a table sorted by price. A store's carefully designed website can become just another data source. A bottle with limited availability can be discovered by people outside the intended customer base.

In Norway, a centralized monopoly creates a centralized search experience almost by default. In Denmark, the absence of a monopoly means the search layer is not neutral. Whoever builds it makes choices: which shops to include, how to rank results, what fields matter, whether price is prominent, how stale data is handled, and whether sellers consent.

I was not sure the Danish market wanted a hobbyist-built version of that.

Why I shut it down​

I shut the project down because the unresolved social and legal questions outweighed the utility of keeping it live.

That may sound anticlimactic, especially after building all the machinery: scrapers, category mappings, PDF parsing, monitoring, data models, and a fairly usable frontend.

But it also felt like the responsible decision.

There is a seductive moment in many software projects where the technical system starts working and you want that to be the finish line. Data flows in. Filters work. Pages load. The admin dashboard shows green checks. It becomes tempting to say: the project is real now.

But it being real does not mean it is right.

Find That Wine solved a real consumer problem. I still believe that. I would personally use a trustworthy version of it. I think many Danish wine consumers would too.

But the trustworthy version probably needs clearer participation from shops and importers. It needs a stronger legal basis. It needs better data agreements, a better opt-out or opt-in process, and a more careful stance on product images, stale availability, and price presentation.

What I would do differently​

If I were starting over, I would change the order of operations.

Technically, I started with the data because that was the exciting part. Find shops, fetch products, normalize data, build search. That is a natural builder instinct.

But the better path would be:

1. Talk to shops and importers first.
2. Ask whether they want this kind of discovery tool.
3. Ask what concerns they have.
4. Offer an opt-in data feed format.
5. Start with fewer sources but explicit participation.
6. Treat scraping as a fallback, not the default.

There is a version of this project that could be sustainable. It would be slower, more relationship-driven, and probably less comprehensive at launch. It would ask shops to participate. It would give them control over how their inventory appears and it would clearly explain data usage.

It would avoid surprises.

The part I still like​

The product idea is great.

I like the idea of being able to search across the Danish wine scene without already knowing which shop to check. I like the idea of small importers becoming more discoverable. I like the idea of a consumer tool that sends traffic directly to shops instead of trying to own the transaction.

I also like the technical shape of the project. It was appropriately boring in many places:

• Next.js and React for the UI.
• JSON files for cached data.
• A flat normalized item model.
• Cron-based updates.
• Client-side filtering.
• Simple admin diagnostics.
• Scrapers only where structured sources were not available.

FIN​

Find That Wine ended as a working prototype.

It could aggregate Shopify stores. It had custom scrapers. It could parse PDF wine lists. It had a data model. It had a searchable frontend. It had filters, categories, price ranges, stock states, and outbound product links. It had admin tools and monitoring. It had documentation for store onboarding and data formats.

But it was not enough for me to feel comfortable keeping it online.

That is probably the main point of this retrospective: technical feasibility is only one part of a product.

As builders, especially with AI tools, we can now create working software faster than ever. That is genuinely exciting. But the speed can make it easy to skip the slower questions:

• Who is affected by this?
• Who has consented?
• Who might object?
• What assumptions am I making about public data?
• What relationships could this change?
• What responsibility do I take on by keeping it running?

For Find That Wine, those questions did not have answers I was satisfied with.

So I turned it off.

I am still glad I built it. It taught me a lot about scraping, handling unstructured data, search interfaces, operational tooling, and AI-assisted development. It also taught me that sometimes the responsible end to a project is not scaling it, monetizing it, or polishing it.

Sometimes the responsible end is writing down what you learned, shutting the server down, and leaving the idea in a better state for the next attempt.

The biggest differences between them have been ambition, mandate, and organizational maturity. There’s no one-size-fits-all, of course, but there are some real common denominators I believe we should be able to identify, structure, and turn into a playbook of sorts. This blog post is my understanding of platform engineering as of 2026, influenced by new and old experience, combined with the great resources already written and published by others. The goal: thriving in a modern world.

To better enable a common understanding of the subject, we all need to know what a platform engineering team's output normally is and how ambition, mandate, and organizational maturity influence the final product.

Platform Engineering Definition​

There are as many definitions of any IT buzzword as there are grains of sand in the universe. However, some common understandings do exist. One definition by Microsoft states the following:

The Importance of Developer Experience​

Traditionally, two models* of IT operations exist. One where everything is managed by a centralized infrastructure team, making them and their offerings the bottleneck. In the other, infrastructure is the developers’ responsibility. Both come with a lot of problems, but they provide value as well. One of the common challenges is the developer experience. A centralized solution can be experienced as slow and bureaucratic, while the other demands a lot of specialized knowledge of cloud administration, making it an uphill battle for new developers or teams that have no existing infrastructure in place. Both are bad developer experiences.

Platform engineers’ main job is to improve on either model. There should be no need for human interaction to get started creating business value as a developer. This requires a unified experience between operations, CI/CD, security, and governance teams.

asterisk: others do exist, but for simplicity’s sake I’ve highlighted the two edge cases I think make the point.

Different Platforms (Application, Data, Common Components)​

A common mistake in platform engineering is that we try to encompass everything that is IT-related. The scope has a tendency to creep before any real work has been done. We cannot treat compute, databases, identities, networks, analytics, etc., as the same type of delivery.

In the next chapter we will talk about organizational maturity, but before we do so, let’s describe a few potential deliveries:

Organizational Maturity​

One of the largest hurdles any platform team faces is getting people to change.

Change is needed when introducing a centralized platform, and for companies who don't have the option to build a greenfield platform from the beginning, getting people and the organization on board can be an uphill battle. The first step is to gauge the organization and figure out where in the landscape our current way of working is placed.

Microsoft has created a good framework for exactly this task. It divides maturity into 5 stages across 6 categories. These stages serve as a tool to map our maturity using the Platform Engineering Capability Model.

Using these categories with the stages above and the Platform Engineering Capability Model you can create a baseline with ambitions that can later be reviewed and used to measure yourself against.

• Investment - Allocate budget and demonstrate an ROI
• Adoption - Choosing services, tools and technologies
• Governance - Security and compliance policies
• Provisioning and management - Define provisioning processes and management practices
• Interfaces - Streamline provisioning and service requests
• Measurements and feedback - Establish feedback loops

Mandate, Ambition, and Business Value​

Now that you know where you place in the maturity landscape, it’s time to talk about the three pillars:

• Mandate - what are you allowed to do?
• Ambition - what do you want to do?
• Business value - how do you implement your ambitions without being a bottleneck?

As the platform engineering definition says, your main goal should be to improve your developers’ security, compliance, costs, and time-to-business value.

For you to be able to do this, you need two things: backing from management and ways to prove your value.

What are you allowed to do?​

Getting backing from management is easier when you can prove that your work contributes to business value. This means that one follows the other. Gathering feedback to figure out where your services would be an improvement is key.

You must be able to measure if what you create is providing value or if it is a failed experiment. There are many frameworks to do so, but you will need to choose one that focuses on continuous improvement, such as Plan-Do-Check-Act, Lean, or Kaizen.

Launching small experiments to gather feedback and measure value is a great way to get started. This will help prove your worth and get the mandate to do more. Don't reimplement everything at once; instead, focus on small parts that can be improved quickly. With mandate comes the ability to increase ambition.

What do you want to do?​

You are allowed to have ambitions, but these are also meant to change. Based on the maturity and capabilities discovered in the model, you can see which areas clearly need improvement and possibly some of the steps needed to get there. As with feedback, you should continuously evaluate your own ambitions and adjust.

To get an idea of where the ambitions tend to lie, here is an extract from the expanded description of what platform engineering entails:

What this means in practical terms and what kind of business value it brings takes us to the next point.

How do we implement our ambitions without being a bottleneck?​

Most companies have a long history in the cloud by now, and many still have an on-premises presence that needs to be considered. The consequences of early adoption of cloud environments are usually a lack of standardization and incomplete target architectures.

These issues can be helped by implementing the ideas from platform engineering, but not in isolation. You will depend heavily on proper implementation of a modern cloud architecture.

Since most are not building a greenfield platform, it is necessary to locate the improvements that can be done now, next, and later that are specific to the point-in-time architecture you already operate in.

I suggest following this four-phase strategy: Stabilize, Standardize, Enable, and Consolidate

• Stabilize - Document the current state, build alignment and trust
• Standardize - Introduce consistent patterns for infrastructure delivery
• Enable - Deliver self-service capabilities
• Consolidate - Gradually move workloads to shared services when the foundation is ready

In the matrix below, you can see a few deliveries and capabilities that might already exist or that can be created by a Platform Engineering team. These are by no means complete or intend to cover all aspects of what a Platform should be, but it gives an idea of the work ahead, choosing to go this route.

The Golden Carrot — Shared Compute Services​

On purpose, shared compute like Kubernetes is left out of the phases shown in the matrix above for two reasons:

1. Delivering shared compute is very attractive
2. It is also very all-consuming and can be difficult to deliver alongside other platform capabilities

Together as a team, you must decide if this is something you should plan to do or if it should be an ambition at a later stage. Most companies have quite a long way to go before it makes sense to provide centralized compute, both technically and in terms of team maturity.

Not to mention the expertise needed in the team to deliver such a complex infrastructure component to a broad set of consumers.

What Creates Business Value?​

A factor that needs consideration is how any of the described capabilities adds to the ongoing business value creation inside product teams. The platform must be able to prove its worth at every step of its implementation, and that’s no easy task.

This is also the reason why developer experience trumps enforced governance.

You need teams to choose your products because of their worth, not because a manager told them to do so.

Saying that is easy, but you need to be able to quantify what value is. This responsibility is not wholly on you, but the developers’ success is your success.

Tools like DORA could support you in this.

In the Plan-Do-Check-Act workflow we talked about earlier, Check becomes one of the most important phases. If we can embrace the practice of "failing fast" and create excellent ways of capturing feedback and measuring value in our workflow, the need for developer-specific performance metrics becomes a phase 3 or 4 delivery.

Next Steps​

Now that we have covered the basics of platform engineering from a modern perspective, it’s time to get started.

1. Make sure that all team members have a common understanding of what you want to achieve and that you speak the same language.
2. Agree on what platform engineering and DevOps are in your context.
3. Gather stakeholders, engineers, and dependents to map your current maturity using the Platform Engineering Capability Model. Claims should be challenged and new perspectives are strongly recommended. Facts over feelings.
4. Based on the capability model and its output, add to or define capabilities and deliveries that make sense for your context in each of the 4 phases.
5. Agree on a timeline and get to work!

Bye!​

Thanks for reading! If you have any questions or want to discuss platform engineering further, feel free to reach out to me on LinkedIn.

This playbook is meant to serve as inspiration and a starting point for your own platform engineering journey. Adapt it to your organization's unique needs and context.

I thought it was a fun talk and since we sadly did not record the talks, I figured that I should write a blog post instead! I already have a script (although written in Norwegian) to accompany the presentation, so why not translate it and publish most of the contents here?

Here goes!

Welcome to this written version of my talk!​

Pic by Håkon Broder Lund

My name is William (dahlen.dev - dev), and while typing this i work as a Lead Platform Engineer at Telenor Norway where we have the task of creating the world's best developer experience through our efforts with self-service, automation and building resilient platforms.

I have been working as a sysadmin since 2013, where the last few years have been spent on the cloud side of things. I have a passion for modern service development, especially infrastructure as code/data and making life simpler.

Chaos engineering as a subject is not something i have practiced much myself, at least not out of free will. It has however always fascinated me, hence this talk.

What is Chaos Engineering?​

I like using the wikipedia definition when describing new concepts, not because its always right, but it is an equalizer; everyone has access to it and you don't need to dig deep into books to find "the correct" one.

Chaos engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.

Its short and sweet. We are exposing a system to turbulent conditions to see how it reacts. This is done to build confidence in the system's capability to withstand turbulent conditions in production.

This is not revolutionary. We have been doing this in the tech industry since the eighties, but its not until Netflix introduced it to their platform during their big cloud migration project in 2011 that it got wind in its mainstream sails.

Personally I did not know about the concept until a few years ago, and even then I was not sure how I should approach it. I am sure most people are in the same boat, so i won't go to deep into chaos engineering, but we will look at one example of the tech enabling it. In addition I will present a few core concepts I believe are important in building a culture for embracing failure and modern system development. Finally i'll give my two cents on the need for multicloud redundancy or maybe the lack of it.

The tech​

In our community, a concept is not real until it has a tool to go with it. Usually more than one as well. Chaos engineering is no exception. As it has existed for a while, multiple tools have been developed and you are free to choose whatever suits your needs. Many of these tools come from internal developer platforms, which later has created a logo, a new name and added an open source license to it. There is nothing wrong with that, tools are made for a purpose and no shoe fits all.

Common for all the tools is that they in some way or another inject failure into your system in a controlled manner, such that we can observe and learn.

The environment​

To get some practical experience with Chaos Engineering and the tools that exist we need a realistic environment. Ideally a copy of a production environment, but as i don't have a production environment at home, we will have to settle for a small lab environment in my own tenant.

The infrastructure​

To build a functioning lab we need to scope out what we need. In my case i decided that running a two kubernetes clusters in Azure, alongside a PostgreSQL as a service would do the trick.

We also need a way to access services in both clusters as a unified service. I could use native azure services for this purpose, but my lab is already totally ingrained in CloudFlare, so it was easier to continue using that. Lastly we need a client that can connect to the services and observe the results.

The engine​

As i am an Azure-man, and most of my day to day tasks are related to this cloud provider, I felt it was natural to use what tools were available to me in this environment.

Funny enough, since i ended up not using the build-in tooling at all, but ill get back to this later.

The engine in question is Chaos Mesh, backed by Chaos Studio in Azure. Chaos Mesh is a software in the incubation phase of the CNCF, and is a tool for orchestrating chaos experiments on Kubernetes. With Chaos Mesh you can simulate common faults in your Kubernetes cluster, such as network partition, pod deletion and resource exhaustion. It works in the same way as many other kubernetes tools, using the controller pattern and CRDs to define the experiments. It also deploy daemonsets to inject faults into the container runtime of the pods.

You can use Chaos Mesh in different ways, the deployment comes included with a web interface, Chaos Dashboard. This is a web-based UI that allows you to create, manage and monitor chaos experiments. You can also use YAML manifests directly to define the experiments. Azure has integrated with Chaos Mesh through Chaos Studio, which is a managed service that provides a simplified experience for creating and managing chaos experiments. It is built on top of Chaos Mesh and provides a unified experience for creating and managing chaos experiments across multiple clusters.

The plan was to use Chaos Studio to define the experiments, but I must admit that I never got it to work in time. The error messages i got was not very helpful, so i decided that using the YAML manifests through a normal deployment pipeline was the way to go.

In the end I prefer this outcome as it keeps the experience totally platform agnostic. Kubernetes is kubernetes after all.

The experiments​

Now that we have a lab running we need a way to test the different experiments. To do this and document the results i wrote a little go application that offers a simple API to test the different scenarios. One endpoint called /ping returns information about the runtime environment where the instance of the application is running. Like the egress ip, pod name, namespace and cluster information. For every request, this information is returned in the response body and is written to the database.

The stat endpoint returns the number of requests each pod has received and how the distribution is between the two clusters and the running pods.

We need some way to show this information in a nice way, so i also wrote a small frontend that shows the information in a pie chart. This frontend is written in SvelteKit and is just hosted locally. It is not very pretty and i had some bugs i could not figure out in time, so you just have to trust me that it works and i'll explain how the data mapped to each test.

What do we want to achieve?​

Simple; how hard is it to get going? How does the injected faults work and will my infrastructure behave as expected?

The tests i have written are not very advanced, and they do not take into account the full scope of what a normal production environment would look like. They are however a good starting point and should give you a good idea of how to get started.

I decided to test these three scenarios:

• Injecting a fault in one of the two clusters
• Injecting faults in both clusters
• Simulate a cloud provider fallout

Injecting a fault in one of the two clusters​

In the first test I wanted to see what would happen if we used the included fault in Chaos Mesh called "Pod failure". I expect that requests will be redirected to the other cluster and that the service will continue to work.

This is how the experiment looks like in the Chaos Studio UI in Azure.

You describe the fault using a json structure, then you choose your target resource and run the experiment. As i mentioned i choose to use a manifest file applied directly in the cluster instead. Luckily there is little difference between the two.

In the manifest you describe basically the same as in Chaos Studio. This is the most minimal example, many more parameters are supported. Among them selectors and randomness for both the targets and the timing.

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
 name: pod-failure
 namespace: chaos-testing
spec:
 action: pod-failure
 mode: all
 duration: '600s'
 selector:
   namespaces:
 	- cluster-1

When the manifest is applied, the chaos mesh controller will start reconciling the state and inject the fault on the pods in the selected namespace. The fault itself consists of changing the image of the pods to google container pause image, which will make the pods unresponsive. The container looks available, but it will not respond to any requests. When the duration is up, the pods are restored to their original state.

And we can see that while our pods were unresponsive, the service continued to work as expected. The requests were redirected to the other cluster.

Injecting faults in both clusters​

Now that we understand how Pod Failure works, we can try to inject a different fault at the same time in the other cluster. This fault might be more realistic then a pod failure for some of us as it will mimic network issues in the cluster. Usually, from my own experience, network issues tend to occur outside the cluster, but this will give us a good idea of how the service will behave if the network is not reliable.

In this scenario we will introduce latency and jitter in one of the pods in the other cluster. All instances have two replicas, so we should be fine. However, I expect to only see one pod responding to requests while the others are unresponsive. Cloudflare should see that one of its backends are down and redirect the traffic to the other cluster. While kubernetes should detect that one of the pods is very slow to respond and should stop directing traffic to it.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
 name: delay
 namespace: chaos-testing
spec:
 action: delay
 mode: one
 selector:
   namespaces:
 	- cluster-2
 delay:
   latency: '15ms'
   correlation: '100'
   jitter: '5ms'

I have to admit that the frontend i wrote did not do a good job of showing the outcome of this test, i found a bug that i could not patch in time of recording, so we have to live with the results and you just have to trust me.

The network fault itself is introduced through the daemonset that runs on the same node as the pod. The daemonset can change the traffic control part of the linux namespace inside the container, adding shaping rules matching what we specify in the manifest. Pretty cool.

This network fault in combination with the previous pod failure resulted in the service being available, but with a higher latency and some timeouts. I was expecting something else, but in the rush to get the presentation ready I forgot to specify the correct liveness and readiness probes in the deployment. The frontend (and the underlying data) show that cloudflare did its job, but inside kubernetes the pod with the network fault was still receiving requests.

Simulate a cloud provider fallout​

In the last test I wanted to see what would happen if the cloud provider had a fallout. This is a scenario that is not very likely, and neither was this scenario to be honest. I started removing the lab environment while I had the services running, woops. If my application served static content, I could have utilized the Cloudflare CDN to serve the content for a while, but I did not.

What did we learn?​

Now that we have been through a few different scenarios with an expected outcome, which we mostly could confirm, what did we learn?

First, it shows that there is possible to simulate faults we experience in a production environment with little effort. Second, it also gives me confidence that services i build for my team and others will be able to withstand turbulent conditions in production. Lastly, it was inspiring. Maybe i can get my team and other developers to take part in testing like this in a more representative environment. Seeing the consequences of a fault in a controlled environment is a good way to learn how to build more resilient systems.

My application crashing has no consequences of course, it was only a test. But in a real world scenario, the consequences could be severe. Mitigation is usually some form of redundancy or high availability.

High availability vs high cost - Is it worth it?​

The question quickly boils down to risk and probability. Even if the application and its dependencies are critical, we know that criticality is expensive. The big cloud providers are painfully aware of this and without taking this seriously they would not have any customers. We have to remember that uptime of our chosen platform is only as good as the services that are built on top. If we look at the big cloud providers and their track record, we see that they are quite good at keeping the lights on.

Historical downtime​

A few nice redditors have looked at all regional outages in Azure since 2019 and its not much. 110 in total. We don't know how serious all of them were, but from my own experience in Azure since 2019, I cannot recall many occasions where the underlying infrastructure was outright responsible for any major downtime. It does occur , but it is a rare event. What is much more common is human error.

System failure or human error?​

We, the engineers responsible for the platform and the services, are the biggest risk to the uptime of the services we provide. We are usually the ones that make mistakes. As an industry we have really good tools to mitigate the risk. The use of infrastructure as code, CI/CD pipelines, testing and having multiple environments are all modern solutions to the problem. It is much harder to do wrong now, but the things that were hard to test still are.

Creating a culture for embracing failure​

It's these faults and errors that really have my interest captured and what inspired me to write this talk. I think there are a few ways to elevate the culture of embracing failure in a company. Here are some examples:

• Use the tools available With this I mean that we should do even more to prioritize insight through monitoring, logging, tracing and so on. Most of us already have a great stack to work with, but I am certain there are improvements to be made. Enablement, training and workshops are a good way to get started. Generalizing dashboards is a good example. When a developer deploys an application in our environment, we should be able to offer insights from our stack just using the context and metadata provided by the deployment. Giving every developer the same baseline of insight might enable even faster recovery times in an incident.

• Operations is development For many developers, this might be the most boring topic, especially when feature requests are being forced upon them, but actively participating, owning what you build, and perhaps even contributing to shared components used by others can be highly rewarding. I’m absolutely sure of that. The future is heading towards a more API-driven world regardless, so the skills gap is closing every day. Is it inconceivable that infrastructure could build itself within defined parameters based on needs and source code analysis? The AI revolution is likely here to stay, whether we like it or not.

• Fail more and fail together It’s great to have test, stage, and pre-prod environments, but these tend to be as stable—if not more stable—than our production environments. Simply because they rarely reflect real-world load or actual complexity. Should we instead focus on emulating failure scenarios we’ve encountered before, or ones we can foresee happening? Looking back at the past two years of post-mortems or incident reports, the services that were affected at the time are often already addressed, but if the issue had occurred slightly differently, with just a small variation, are we confident that the services that worked back then would still hold up now? Chaos Mesh, among others, allows you to integrate test workflows into CI/CD solutions. Perhaps this is a good way to introduce controlled failure? By building standardized workflows that the organization agrees applications running on the platform should be able to withstand we could set requirements. Do you want your application to be categorized as critical? Then you must first guarantee that it meets the uptime requirement.

There’s certainly more we could do, but I believe that these three points, if done right, will have a positive impact on both operations and culture. I’ll have to check back next year to report on the results!

Conclusion​

This is the end. Before I go we must talk about multicloud redundancy as I promised in the abstract. I don't think this is far fetched for most readers. It is more common now to be using multiple cloud providers, especially as they are forking off in both price and service availability. In smaller companies its less common, as maintaining the same governance level and core infrastructure is both expensive and time consuming. Not to mention the competency needed to do so.

The question is if it is worth it? Do you automatically get a more resilient platform by using multiple cloud providers? From my own personal experience we react much more strongly to incidents caused by the cloud provider than incidents caused by our own mistakes. It's kind of a cognitive dissonance where we accept 99.95% uptime from the cloud provider, which in theory is 4 hours of downtime a year. Then when the cloud provider has downtime, we don't accept any at all? It is a strange world we live in. I don't think another dot on the architecture diagram alone will make us more resilient, but I do think that the tools and culture we build around it will.

I believe there are more gains to be made in building a culture for embracing failure and at the same time building more redundancy at the system level. There is no one answer to the question of multicloud redundancy. Think twice about putting the dot on the diagram, it looks cool and it's fun to say "we are multicloud redundant", but it might not be the answer.

Should you just have one takeaway from this talk, let it be this: Embracing failure and training for it is important. Probably much more important and cheaper than spinning up extra copies of your infrastructure and applications in someone else's datacenter.

Thank you for reading this translated version of my talk. I hope I could inspire you to try out Chaos Engineering in your own environment. It is a fun and rewarding experience.

If you have any questions or comments, feel free to reach out to me on Bluesky or LinkedIn.