When engineers, especially developers working with modern cloud environments and technologies, mention “security,” it can mean both everything and nothing at the same time. Security can range from the office firewall to user tokens when reaching the API. It encompasses user permissions within a cloud environment and application permissions when running inside a container within a cluster.

So, where should the focus be? Which areas can be “outsourced”? And, should you even care?

Let’s take a step back:

What is DevOps Again?

Without reiterating the same old mantra, I’ll sum it up in one go: DevOps is not a job title or an engineering field. It’s a culture, a collection of technologies, and most importantly, a specific way of doing things with the goal of improving the development life cycle. While this explanation may seem generic, it’s a testament to the term’s overuse and varied interpretations over the years.

Who Needs to Worry About Security?

The answer is simple: all engineers. But worry how, you may ask. Well, HACK!

Web Developer? Learn how to scan the /dist directory and JS files. Understand the mechanisms behind authentication, keys, tokens, and cookies. Learn the different headers and types of requests that come in and leave the application. Backend Developer? Are you aware of IDORs? Do you know what they are? Is there a chance you’re vulnerable? There are too many vulnerabilities to cover in one go, but it’s essential to stay updated with the latest news and discoveries. Familiarize yourself with OWASP and their Top Ten list of hot vulnerabilities. Although OWASP focuses on web applications, this knowledge is relevant across all areas. Every app, deployed in any environment, is susceptible to some attack vector.

Best way to avoid vulnerabilities? That’s right, NO CODE 😉

Why it Matters

When it comes to security, it’s challenging to go overboard. While that’s an option, merely proposing the idea that all developers are involved in the process—from development through QA to Ops and Sec—broadens the pairs of eyes looking at the code, infrastructure, and bugs. No one is free of responsibility, and the more, the merrier.

Take SQL injection, for example. It’s a vulnerability every engineer should at least be able to explain during an interview. SQL injection can be found and blocked at many levels, from the code susceptible to query injections, through the QA process that can test for simple scenarios, to automated services that check for these vulnerabilities in CI. Additionally, infrastructure systems like a WAF can block suspicious queries with filters that identify malicious injections.

When is it Important?

Security is one of those things that’s easy to ignore, overlook, or get frustrated with, and for a good reason. Rotating your Google password every other Tuesday is annoying and unhelpful, but having your entire production authentication secret keys leaked in your JS files can be the end of an organization (don’t ask how I know).

“The primary reason security professionals advise against periodic password changes is that when human beings change passwords often, they tend to conform to a pattern. That is why ethical hackers at Packetlabs see passwords like Summer2021, Fall2021, Spring2021.” - Packetlabs

Things to Consider

Most engineers won’t naturally dive into the world of ethical hacking or bug bounty programs. It takes education and continuous growth. Much like code reviews or DevOps education, application security is part of that growth.

Most security teams and CISOs might ask you to rotate your password or use Chrome instead of your beloved Firefox. They might not check if the app is vulnerable to an RCE because the file uploading feature lacks basic scanning. This is unfortunate but often the reality.

Nobody knows everything. Understanding this is key. With that in mind, a team can start thinking about the right way to build a system to improve security. One great option is the system of guilds, where an engineer can be part of the “web app security” guild and meet once a week to discuss concepts, tools, or methodologies. There can be multiple guilds: one for QA, another for developers, and a third for ops engineers. As long as the team is aware and actively working to improve, changes will compound over time, hopefully preventing the next disaster from occurring.

Conclusion

Sharpen your axe and stay vigilant. Security is a continuous process that involves everyone in the development lifecycle. By fostering a culture of awareness and proactive measures, we can collectively enhance our systems’ security and resilience.

As more businesses shift toward microservices, Kubernetes is turning into a go-to tool for handling the nitty-gritty of today’s IT world, and its API plays a pivotal role. Think of the K8s API as the control hub, making the management of the Kubernetes cluster a breeze and letting users spell out how they want their apps and infrastructure to look and act.

Kubernetes controllers are the unsung heroes, constantly working to ensure the system’s actual state matches the user’s needs. Here, the operator pattern is a real game changer, showing how flexible and user-focused Kubernetes can be.

The operator pattern was created to meet the varied needs of businesses and devs and enables Kubernetes to do even more, including managing custom resources it wasn’t originally built for. This means Kubernetes can be customized to handle all sorts of systems, proving it’s ready to adapt to whatever the tech world throws at it. Kubernetes really stands out when you consider the cool features and extensions the operator pattern brings to the table, offering solid answers to today’s tech hurdles.

Kubernetes-Native Controllers

By operating on the principle of desired state management, Kubernetes allows users to dictate their system’s configuration through a centralized control plane, which serves as the decision-making and monitoring hub. At the heart of this control plane are the Kubernetes-native controllers, purpose-built to manage specific resources within the ecosystem. These controllers continuously monitor their respective resources and ensure the system’s current state aligns seamlessly with the user-defined desired state; controllers automatically make necessary adjustments to maintain this balance.

For instance, consider the deployment controller. When you deploy an application in Kubernetes using a Deployment, this controller jumps into action. It ensures that the specified number of replicas of your application is maintained. If a pod crashes or becomes unresponsive, the deployment controller will recognize the discrepancy and initiate the creation of a new pod to maintain the desired state.

Similarly, the ReplicaSet controller maintains the correct number of pod replicas. It’s closely related to the deployment controller but operates at a slightly lower level, focusing specifically on pod replicas without the additional features that deployments offer.

These are just two examples, but Kubernetes boasts a plethora of native controllers, each tailored for specific tasks, like managing services, volumes or network policies. Together, they contribute to Kubernetes’ reliability and resilience, ensuring that your applications and infrastructure run smoothly and consistently.

Custom Controller Use Case: Tracking New Volumes

In my journey through the dynamic world of Kubernetes, I’ve found myself in situations where the built-in controllers couldn’t meet specific needs that cropped up. That’s when I realized the true power of custom controllers.

I worked in a large-scale organization where we were constantly deploying and scaling storage volumes. It became evident that we needed an efficient system to keep track of these deployments. I imagined how great it would be if I could receive a Slack notification every time a new storage volume was deployed. In addition, the volumes would be annotated automatically for monitoring systems without human intervention. While Kubernetes doesn’t offer these features natively, I figured out that a custom controller could be the perfect solution to bridge this gap.

So, I mapped out a workflow for the controller to handle this scenario, which looked something like this:

• A new storage volume gets deployed in the Kubernetes cluster.
• The custom controller, which I designed to keep an eye on storage volumes, spots this new deployment.
• The controller reacts to this by triggering a predefined action—in this case, shooting off a notification to a Slack channel and annotating volumes for monitoring.
• My team received the Slack notification, giving me a heads-up about the new volume deployment.
• Armed with this info, I could quickly gauge whether the new storage was vital for a particular application or if we needed to make some tweaks. This hands-on experience highlighted the versatility and adaptability of custom controllers. I was able to tailor Kubernetes to my specific needs, ensuring seamless integration with the other tools and platforms I relied on while maintaining the system in its desired state. It turned out to be a practical solution and helped to streamline operations and keep everything running smoothly.

Kubernetes-Native Way: kubebuilder

Kubernetes boasts a rich ecosystem that not only allows for the creation of custom controllers but also offers tools to facilitate this process. A standout tool in this realm is kubebuilder, a scaffolding framework designed to construct Kubernetes APIs and controllers. This tool greatly simplifies the task of integrating custom resources and logic into Kubernetes.

The preference for kubebuilder over custom scripts stems from several of its advantages:

It provides a structured project layout, streamlining the development and maintenance of controllers and custom resources. It autogenerates much of the repetitive code essential for setting up controllers and APIs. It integrates seamlessly with Kustomize for configuration customization and is backed by thorough documentation to guide developers through its functionalities.

Walkthrough: Creating a Controller With kubebuilder

To get going on creating a controller with Kubebuiler, I highly recommend The Kubebuilder Book. It provides a comprehensive walkthrough of creating a container, covering all the steps and components. Even for the relative experts out there, this guide is worth looking into to further sharpen your skills.

Advantages of Controllers

Kubernetes controllers, both inherent and custom-made, serve as the foundational pillars of the Kubernetes ecosystem. Acting as silent custodians, they ensure the cluster’s current state consistently mirrors the user’s desired specifications. These controllers offer a multitude of benefits:

• High availability: Controllers are integral to Kubernetes’ promise of high availability. For instance, in the context of tracking newly created volumes, having a controller that notifies the team immediately ensures that any issues can be addressed promptly, maintaining the high availability of the volumes. This self-recovery feature ensures applications remain robust against failures.
• Versatility: Controllers in Kubernetes are designed to cater to diverse needs. Leveraging them in tracking volume creations showcases their versatility in adapting to different operational needs, including batch jobs, stateful services or daemon processes. This allows Kubernetes to manage varied workloads effectively.
• Appropriate permissions: By prioritizing security, controllers operate on a least-privilege principle. They possess only the essential permissions needed for their tasks, reducing potential security threats and limiting the impact of any compromised component.
• Resource optimization: Beyond state maintenance, controllers emphasize efficiency. In the scenario of tracking volumes, it aids in resource optimization by providing real-time updates, facilitating immediate actions to optimize resources based on the current state and guaranteeing cost efficiency.
• Extensibility: Kubernetes’ flexibility is evident in its support for custom controllers, allowing users to address unique needs beyond the capabilities of native controllers. This adaptability ensures Kubernetes stays relevant to changing business needs. For example, in the case of tracking volumes, it has extended its functionality to integrate seamlessly with tools like Slack, enhancing operational efficiency and responsiveness.

Conclusion

Controllers aren’t just a component of Kubernetes; they’re its lifeblood, ensuring that applications remain available, resilient and efficient. They are essentially the Ops engineers’ method of introducing automation to K8s in an elegant and resilient way and extending their capabilities. In most scenarios, controllers prove to be the optimal way to interact with clusters, outshining scripts and manual interventions. Controllers’ automated, continuous monitoring and action loops mean the system can stay in its desired state without constant human oversight.

It’s also worth delving deeper into how controllers aid in extending the system. The term “operator” defines a set of controllers and custom resource definitions (CRDs), which are—in essence—custom resources. I’ve touched upon this concept briefly in this article. Still, it’s fundamental to understand that operators allow for creating custom, application-specific controllers, thereby enhancing the extensibility of Kubernetes.

So, as you navigate your Kubernetes journey, remember the pivotal role of controllers and consider crafting your own. With the extensibility features of Kubernetes—especially through the use of operators—you stand to gain even more from the cloud-native infrastructure and ecosystem.

Platform engineering is a strategic concept in which engineers use a single pane of glass to streamline all processes, tools, and platforms. It optimizes security, DevOps, CloudOps, and more, so engineers can work more efficiently, without investing an excessive amount of time and effort.

This streamlining eliminates the overwhelming amount of tasks and responsibilities that come with working on complex microservices and cloud native environments. With less cognitive load needed to go into building applications, more products and services can go to market faster, supporting CI/CD practices, automation, security, and governance.

While platform engineering is seeing a lot of buzz these days, there is one aspect missing from the current paradigm, which focuses exclusively on balancing developer time vs. app stability and performance. This missing component is cost efficiency — which in today’s economic environment, should be front and center.

Platform engineering should enable fluid integration for cost-efficient cloud provisioning and automated scaling. This way, businesses get the full benefit of optimized developer time, performance, and cost efficiency.

Why Platform Engineering?

DevOps teams have a lot of technology, infrastructure, and dependencies to keep track of and maintain. They are often overwhelmed with an endless array of tasks and responsibilities, making them lose track of key technical dependencies.

Platform engineering seeks to solve this problem by creating a unified SaaS platform that better enables teams to build, deploy, and run software applications. Instead of opening tickets or waiting in line for requests, all tools and dependencies are unified and streamlined in one place so teams no longer need to worry about tools or infrastructure, and can instead focus on building services.

Bringing Cloud Costs into the Mix

Platform engineering is seeing widespread momentum in engineering circles, but integrating cost efficiency into this movement will help organizations even more, ensuring that profitability, or at least cost awareness, is embedded throughout developer processes.

There are dozens of ways that the concepts of platform engineering can be used to get a handle on costs, and many organizations will likely develop their own personalized systems for it, but there are some easy ways to start.

Automatic Notifications of Cost Thresholds

As engineers are building, coding, and delivering new apps, costs are frequently the furthest thing from their minds. But as an organization, someone (or something) needs to ensure that the cost of servicing a product is kept in check.

That’s where automation comes in. Imagine if you could embed the platform your engineers are already using with automatic notifications and alerts that anticipate costs before they’re generated. For example, if an engineer is about to spin up a cloud resource that goes above budget, they will receive a notification about estimated costs and prompts for approval.

This will provide an automatic barrier preventing engineers from going over budget. In addition, it will also cause the engineering team to have more awareness and ownership over the costs they incur, encouraging them to come up with more cost-efficient methods for releasing services.

A real-world example of something similar is InfraCost, which provides automatic cloud cost estimations for Terraform pull requests. When combined with automation, this tool can serve as a cost threshold with an approval chain that ensures costs are being managed and kept under control.

Ongoing Scaling of Cloud Resources

Another area where an automated cost optimization solution can be integrated into platform engineering is the continued scaling of cloud resources to fit business needs.

Today, the over-provisioning of cloud resources costs companies millions. But fixing this issue is complex. While limiting resources can cause your service to crash, over-provisioning is expensive, wasteful, and inefficient. The solution would be to auto-scale cloud resources so that application needs are consistently met at the most cost-efficient price point.

What does that look like through a platform engineering lens? Build this capability into the platform and provide it to the developer as an automated feature so that they no longer have to give any further thought to it.

One example where this can be applied is with containers, where engineers struggle to allocate the right amount of CPU and RAM they’ll need in complex and unpredictable environments. To ensure the stability and performance of their applications, they typically over-provision these resources, which drives up costs unnecessarily. According to Datadog, as much as 49% of containers are using less than 30% of the requested CPU, with similar levels for memory utilization.

Integrating this type of auto-scaling technology into developer processes can efficiently bring down costs, without any human effort. Such an automated resource allocation tool would detect the level of demand and then scale resources like CPU and RAM to a level that would enable applications to run as they should — at optimum stability and performance. Behind the scenes, the organization is dramatically cutting costs, and significantly reducing the manual work and the cognitive load for DevOps engineers.

Final Thoughts

As a new approach in the world of DevOps and software engineering, platform engineering is growing in popularity for its vision of streamlining and simplifying developer operations.

Taking it to the next level and imbuing it with an additional focus on cost-efficiency can help businesses ride out more difficult financial periods and ideally, put them on a flywheel for growth, for when the economic tide does eventually turn.

There is a lot of untapped potential within the idea of platform engineering and tech-savvy business leaders should consider how they could use it to develop new approaches to support application stability, performance, security, and cost efficiency. Economic downturns are never easy, but it also offers a great opportunity for leaders to cut inefficiencies and streamline processes that can enable your business to thrive for years to come.

Being in the tech industry, it’s quite common to come across jargon such as cloud computing, containers, serverless frameworks, etc. But what is cloud computing?

How does a container work? And how can a function be serverless?

This post will try to decode these technologies and explore how developers should consider containers or serverless functions within their tech stack. For example, if your application has a longer startup time, then a container would suit the need better. Highly efficient stateless functions that need to scale up and down massively would benefit from running serverless functions. How Does a Container Work? A container is a packaged application that contains code along with the necessary libraries and dependencies, which can be executed in any environment, irrespective of the operating system. It helps developers build, ship, deploy, and scale applications easily.

Previously, when containers were not that popular, developers used to deploy applications in separate virtual machines to achieve isolation. Each virtual machine had a guest operating system that required some definite CPU and memory from the physical hardware. This caused virtual machines to use up resources that were necessary for the applications. With the adoption of containers, a guest OS is no longer required since the container engine can share the physical operating system with one or more containers. This is a huge benefit over a virtual machine, as more resources can now be dedicated to the apps. At a fundamental level, a container image is created by the developer that contains instructions about how the container should run. This container image can then be used to spin up containers that run the actual application. It contains executable code that can run in wholly isolated environments. The libraries and dependencies for the app are also packaged in the image definition. Docker, Amazon ECS, Kubernetes, and GCP Autopilot are major containerization platforms.

What Are Serverless Functions?

As the name suggests, serverless computing is a paradigm in computer science where developers do not manage the servers themselves. Instead, a third party provides services to manage the server so that developers can focus more on application logic rather than on the maintenance of servers running the app. Serverless Functions, in general, are a further abstraction from containers and under the hood; both use the same underlying technology.

How Does Serverless Computing Work?

Serverless computing is not possible without using actual servers—it’s just that developers don’t need to interact with them because vendors, often a cloud provider like AWS, Azure, or GCP, take on server management. Frequently, there are containers underneath Serverless infrastructure. In AWS’s case, they open-sourced their Rust project that handles the instantiation of the functions (firecracker) and the lightweight operating system they use for the nodes hosting the containers (bottlerocket). In this paradigm, developers are provided with an environment where they can write and submit their code. The platform takes care of the execution, allocating physical memory, CPU, and output. The user is then charged for the duration the application runs and for the memory and CPU consumption. This is a huge benefit, as it greatly reduces costs compared to running or managing your own infrastructure. An important point to note is that serverless functions also leverage containers under the hood, which is abstract from developers. When you start a serverless function, the cloud provider spins up a container in which the application is executed. Thus, the underlying technology is similar. However, the way environments are deployed, scaled, and utilized are different.

Fundamental Differences Between Containers and Serverless

Below, you can see some key differences between containers and serverless functions.

Use Cases

There are several everyday use cases for containers and serverless computing. Some solutions can be achieved by either using containers or serverless functions. Still, the major difference is how both containers and serverless functions are priced, integration with other applications and services within the cloud, and their ability to scale based on the load.

For example, if your application is a short-lived process that looks up a geographic location based on IP addresses, it can be serverless. As the number of requests increases, the serverless function will scale up automatically without any manual intervention. If the application is a web application that requires it to stay up and running for long periods of time, then a container might suit best.

Also, processes that need faster access to storage can be accommodated using containers as they can be integrated with file systems. An example is integrating the containers running on Amazon ECS that can use highly scalable EFS or Azure Files for storage purposes.

The following decision tree might be helpful to decide between choosing a container or a serverless function.

Although there can be an overlap between the two, some of the common use cases of containers and serverless functions are discussed below.

Containers

Programing language support

When you package your application in a container, it becomes platform-independent. Serverless Functions support a few of the most common runtime environments like Java, Python, Go, etc. However, if the application programming language is not supported by the Serverless Function’s runtime, then containers would be an ideal solution. The container can be deployed using any container orchestration platform, such as Kubernetes.

Hosting long-lived applications

Containers make it easier to set up long-lived web applications that need to always run as a service. For example, a tracking application that tracks user behavior on a website can be deployed via containers. Based on the generated events, the container can be scaled up or down. Containers can also be orchestrated by leveraging popular technologies such as Kubernetes, AWS Fargate, etc.

Serverless

API endpoints Organizations can use a serverless application to deploy API endpoints for a web or mobile server. These APIs can be stateless and short-lived, which can be triggered based on an event.

IoT processing

With the increase in home and industrial automation, the use of IoT devices has increased significantly. These IoT devices can leverage the power of serverless computing by running functions when triggered.

Event streaming

In a real-time event-streaming scenario, where events need to be enriched or filtered based on certain conditions, developers can use short-lived serverless functions. You can then use these functions to check if an event is valid, enrich events by looking up demographic information, etc.

Final Verdict

Both containers and serverless applications are cloud-agnostic tools that benefit developers. As a rule of thumb, containers provide isolation and flexibility, whereas serverless aids development and helps you autoscale with minimum runtime costs.

Choosing to go with containers versus serverless applications depends on the use case. For example, a serverless framework would be better if the requirement is to build an API server that serves fast, short-lived responses. On the other hand, if the application needs always to be available and up and running, a containerized application is the way to go.

AWS services are at the top of the cloud services food chain, and a hot topic for nearly every business considering migrating to the cloud. Along any cloud migration journey, many hidden traps await the unprepared. The volumes you’ve already provisioned might seem like a perfect match—but are you sure they suit your workload and carry the best price for performance delivered? Today, we’ll make the difficult task of choosing a storage class for your workload a bit less complicated.

In the World of EBS AWS EBS currently provides three categories of considerable storage: Hard disk drives, general purpose SSD, provisioned IOPS, and one that you will most likely never need to use—magnetic. Although I’ll provide brief descriptions of each, I’ll focus on the most widely used volume types—general purpose.

GP Storage – the Common Choice This is the staple storage for pretty much all AWS users. It gives you the best bang for your buck and is commonly used for tasks that don’t require heavy lifting from a drive perspective. GP storage ranges from 1 gibibyte (GiB) to 16 tebibytes (TiB), providing from 128 to 1,000 mebibytes per second (MiB/s) of throughput, which will most probably fit your use case, unless the workload is significantly specific.

There are multiple types of GP storage, each with different pricing and characteristics that I’ll provide a few details on below.

GP2 – Old-Gen SSDs for the Masses GP2 is the typical solution for boot volumes of EC2 instances, development environments, and lighter/less important databases. This is one of the most popular options out there, although its popularity comes from old habits rather than actually being a perfect shot. GP2 volume performance scales with its size—the bigger they are, the faster they go. Still, these might seem a good option, but to be honest, they aren’t so perfect. And the biggest elephant in the room is the term you’ll have to remember – “burst credits.”

Burst Credits – Math Homework for Your Business Work Provisioning a GP2 volume, you get a one-time grant of 5.4 million burst credits, which you can use to make it handle input/output better during spikes in demand. More of those tokens means you’ll be able to sustain bigger performance demand for longer, boosting your storage up to 3,000 IOPS as long as you have enough of them in your pocket. While depleted when you go “turbo,” they accumulate when you’re not using your disks at full speed—saving up unused performance for when you need it most. Burn them all, and your IOPS performance drops to base value and throughput falls to the baseline IOPS multiplied by max I/O size.

The performance of GP2 storage volume can be calculated with this formula: Throughput in MiB/s = ((Volume size in GiB) × (3 IOPS per GiB) × (I/O size in KiB))

Until you reach the point where your volume is so big that its base performance exceeds burstable, you’d have to learn them by heart. Where is this point? Quite high, at about 1.1 TB (1024 GiB)—a lot of disk, even for a medium-sized business, especially since those volumes can’t be mounted to multiple targets.

Unless you’re deploying a disk-devouring monster such as an EC2 host instance for dozens or hundreds of Docker containers, or you plan to run an enormous Elasticsearch node for the next few years with no clean-up routines in place, you’re not likely to need this much space. Also, in either of the two cases mentioned, you’d probably run out of CPU and RAM first; there are better alternatives for these cases as well, not to mention that the lack of maintenance is definitely not a good practice.

In addition, for practically every use case—web serving, web applications, databases, casual EC2 instances for use with Docker, and more—pretty much anything is better than GP2.

Luckily, in 2014, the introduction of GP3 brought some great improvements.

GP3 – The “Newcomer” While not really a “new” player, GP3 volumes are a welcome improvement after experiencing the inner workings of GP2. They perform better and cost less in comparison, allowing you to easily scale your operations according to your needs much easier. Gone are the days of burst credit balances.

The golden rule of this storage class is “mix n’ match.” Baseline performance here is the maximum burst performance of a GP2 volume, 3,000 IOPS, and 125 MiB/s. If your business needs more, you can purchase additional amounts of either, paying extra only for what you’re interested in.

The limits are quite generous. You can purchase up to 500 IOPS per GiB (for at least 32 GiB volumes, capping at 16,000 IOPS), or up to 0.25 MiB/s per GiB (for volumes of at least 8 GiB, 4,000 IOPS or more, capping at 1,000 MiB/s).

If you were previously happy with your GP2 volume, it’s almost guaranteed your company will be just as happy with GP3, and you’ll save a decent percentage of the storage cost, even up to ~20%.

Assuming that you don’t need more than what GP2 provided you with, make the switch. To make it even easier to cut those costs down, AWS is now automatically setting the IOPS and throughput of your new GP3 volume to the exact same as your GP2. Less guessing, less messing around, more good stuff. Just click a few buttons and save some cash, no strings attached.

This is the storage class I would recommend for every “generic” or “typical” workload. It’s a good starting point for almost every business use case that doesn’t expect huge amounts of disk operations, but is ready to scale up when the dreaded flood of demand for disk power comes.

GP3 can easily handle the role of boot volume for EC2 instances. It can host both simple and a bit more complex applications, databases such as MySQL and PostgreSQL, or a blend of those in a containerized environment, especially in a multi-node configuration or as part of a Kubernetes cluster. Zesty Disk for Kubernetes handles this switch automatically if you have the CSI driver installed

When the time comes, it can scale in tandem with demand. In certain cases, it might even be cheaper to provision high-speed storage with GP3 than the IO1/IO2 types, which are dedicated to speed and next on our list.

IO Storage Class – Need for Speed The IO storage class was introduced to cater to the needs of businesses based on extreme durability, high read/write speeds, and the lowest latency available. It requires certain criteria to be met to show its true potential and costs significantly more. But, in turn, it provides unmatched speeds and reliability, as well as some features you won’t find in any other storage type.

This volume type is great if you seek a volume to provision for your high-speed disk-based database, for example, or to create a single MySQL database host for multiple large-scale e-commerce instances, although there are multiple options available for this from RDS to multi-instance DBs (e.g., MySQL Galera). It’s also a fit if you need to accept a lot of data fast to process with an ML algorithm.

IO1/IO2 Vs. IO2 Block Express – Which Type of “Fastest” do you Need? IO2 Block Express is pretty much IO2 on extremely potent steroids. You get limits that are four times as high for IOPS and throughput, latency of less than milliseconds, and surprisingly, no additional fee. Yes, you read that right, they’re priced the same. Plus, provisioning additional IOPS/throughput above the limit of a standard IO2 instance actually costs less.

If you decide to use IO2 and meet the requirements (a certain type of EC2 instance), you should definitely consider Block Express—if it’s not already running by default.

And when would you run IO2 anyway? Well, with its power and hefty price tag, this class is dedicated to the most disk-dependent, throughput-hungry, and IOPS-dependent monstrosities. For example, a data-crunching machine for a stock exchange, a global log-ingestion mainframe, or a multi-pod Kubernetes node prepared for very demanding workloads—the titans of storage, no doubt. It’s a go-to when you need a single machine that’s going to pack as much punch as you can get.

How about IO1 then? It’s a compromise between low latency, high IOPS, and price. A good choice when you need a bit more than general purpose, but still not enough to go for the fastest storage out there. Keep in mind there’s also one additional drawback—less durability. A 0.2% failure rate annually does not sound like a lot, but if you’re thinking about dozens or hundreds of drives, absolutely critical workloads, or both, it definitely makes a huge difference.

So, Which Should I Choose for my Workloads? As you can see, the world of EBS is quite complex, to say the least. So, I made a chart that should ease the selection process for your use case:

< CHART!>https://devops.com/wp-content/uploads/2022/12/Screen-Shot-2022-12-02-at-8.40.01-AM.png

Fantastic Disks and How to Tame Them Choosing the right storage class is a tough decision to make—but the benefits are worth it. With just one shot, you can both improve the performance of your instances and save on costs as soon as your next invoice.

This is one of the reasons I decided to compile a list of “DevOps engineering principles”. It was important for me to create a basic structure, something that could be shared, discussed, and reinforced periodically.

Most of it came from mentors I had or glorious failures I experienced over the years. This is the idea:

1. Take a stand and make a decision
2. Every point can be discussed and debated
3. Upon discussion and agreements, we use it as our guiding light
4. Observe, assess, make adjustments, keep moving

Document everything

The number of “DevOps questions” asked daily by internal teams is too high. Whether it’s newly hired engineers or instructions that have already been forgotten. By documenting things, you will be able to both link to the document and gradually engrain the habit of searching the docs before asking a question.

Our team tries to document everything it can:

• Whenever we experience a production failure, or something goes wrong, such as a breaking change of an external library hitting us or introducing new limitations by a provider (cough DOCKER cough), we write it up as a post mortem. Anyone reading it can understand what went wrong, even if they were not involved, and maybe learn something. In order to prevent a recurrence, a post-mortem should conclude with action items.
• There’s nothing like a good ol’ instruction page to explain something once and for all, whether it’s logging into a Docker registry or running localstack to mimic a serverless environment.
• Research - Being a DevOps team in a DevOps company involves a lot of research. It is easy to follow, understand, remind yourself and others of findings and lessons learned when it is documented, and equally important, it prevents rework and re-examination.
• How-tos - Oftentimes, we find ourselves instructing something very ‘simple’, in or two sentences. Even then, a document can be very helpful in providing additional context. Having a “search-the-docs” culture and a habit of documenting everything has a compounding effect.

In particular, we use Notion for all things documentation. There’s no way to say whether that’s objectively “the finest”. As readers and writers alike, we find it more inviting, easy to use and feature-rich.

Ask questions / always open a discussion

Experts working in their own bubbles and sharing information randomly miss the point. Creating a habit of discussing is beneficial to everyone; The one asking may learn something they did not know or consider, the responder shares their expertise, and passive readers gain enriched knowledge “for free”.

A culture of “there are no stupid questions”, is easy to decalre and hard to practice, but the benefits are incredible. I’d say the distance between something so un-measureable yet so valuable is amazing.

Principles:

• Don’t be afraid to ask questions
• Don’t be afraid to challenge anything
• Are you bringing an innovative idea to the table? Come prepared to be challenged
• Create a discussion everywhere - Slack groups (not private chats), notion comments, pull-request remarks. The discussion may be of interest to the team, but it’s happening far from their eyes (e.g. a Github PR)? Invite others to read and participate by linking it in the team’s channel

Think lean, speed and quality

The point here is not to do things quickly, but to create things that work quickly. Spend more time setting up processes, so that you can not only save time down the road, but also help others reap the rewards of your labor.

In the world of DevOps, this takes the form of

• Lean applications - Cut out unnecessary layers of complexity / attack vectors / loading of literally everything in the way (downloading, pulling, installing, packing, etc.). E.g. pulling a 50Mb JavaScript library to create random numbers.
• Lean packages - Zip packages uploaded to Lambda, for instance, can sometimes be compressed further, or unnecessarily packed with non-production content
• Lean containers - Can be based on alpine or :slim tags. Can use fewer Docker layers, and utilize multi-step build containers to only carry essential production components
• Lean CI - The list of examples is endless, but steps can use cache from other steps, use lighter images (see lean containers), or perhaps run in parallel to save time and speed up the production process. Consequently, other jobs and applications have more time to work, the build queue is shortened, and resources are used less. There is an exponential effect.
• Utilizing cache - Whether it be between CI steps, Docker layers, query results stored in Redis, pre-built images and, of course, using a CDN.Cache is a driver of speed and efficiency, apply wherever possible
• Before hacking your way through, do things in the native way. “Read the docs instead of writing another bash command” or “Search the knowledge base before spending a day creating your own environment” are two quotes we used recently. System features often include solutions for what most engineers need. There are too many reasons to list why “hacking as a standard” is counterproductive in large teams.

Think Security

Although this is a big one, here’s a short list:

• MFA - For everything. The effects of this security layer are mind-blowing yet it is surprisingly easy to implement
• Secrets management - Too often overlooked when it shouldn’t be. A dedicated manager should be responsible for managing secrets. Not in a Git repository, not as an encrypted text file, not on the CI server (although not as severe).
• Run resources on private networks. In a DevOps team, this goes without saying. With developers literally testing the cloud (did I mention we’re a DevOps company?) it’s hard to educate, let alone enforce. However, all resources should run privately and be accessed through a zero-trust proxy.
• Segregate environments - Good - separate VPCs, Better - unpeered networks, Best - separate accounts and use different access groups
• Using WAF as an example: use whatever deployable tool you have in your toolkit to make attackers’ lives difficult. Utilize zero-trust, limit network access, create complex password policies, and enforce multi-factor authentication. Maintain your devs’ sanity while limiting attack vectors

Networking structure

• HA - It is a principle in and of itself to ensure high availability. When designing an architecture, building production (and staging) environments, or running log aggregation systems, HA should be considered. You are guaranteed to spend time on a failure if you let something slip under the radar. The following is an example of a recent real issue - a self-managed ELK cluster hanging by a thread on an AWS spot instance serving the entire R&D department. Fortunately, we spotted the problem in time
• Cost efficiency - Terminology first: cost-efficient does not mean cost-reduction. The concept of efficiency can be viewed in a variety of ways, such as utilizing VPC endpoints for private, quick, and cost-effective solutions, or using spot instances for non-critical operations. A container can also be used over a serverless function that lasts a long time, or the opposite when the job runs in a few milliseconds, and a container/instance isn’t a good option

Everything as code

• Infrastructure as code. There are no words to describe how important this is. Just do it, as the saying goes.
• Meaningful commit messages - It is incredibly useful to have meaningful messages, for history searching, formatting, debugging, compiling change-logs, and so on.
• Use a Git system whenever possible. Is it time to update a WordPress plugin? Is it time to upgrade the infrastructure? (It’s all coded now, isn’t it?). Changing the permissions of a group? Is a secret being updated? USE GIT. The number of cases I’ve solved by comparing multiple Git histories is beyond my ability to count. It’s not for nothing that Git was once compared to a blockchain. The Git repository holds the entire (almost) history of the project, so use it wisely!

Backup everything!

It’s kind of a “no-s**t-sherlock” thing, but still, it’s amazing how often backups are not available when they’re most needed. As such, I consider it relevant to list it as a “principle” even though it has been a common best practice since ancient times.

By automating backups, hours of work, pain, and sorrow can be saved. Most cloud platforms offer backups as part of their products, so you just need to turn them on. As an example, let’s look at AWS:

• RDS will enable backups and require a timeframe and a retention periods
• S3 can have versioning turned on
• EC2 (unless used as cluster nodes) can be snapshotted
• EBS snapshots are just as straight forward It goes on and on with ECS task-definition versions, Lambda functions, DynamoDB tables, etc. All provide some form of data backup and restoration for point-in-time recovery. Don’t call a task “backlog” or “tech debt”, because by the time it’s needed, the Jira ticket will no longer be efficient. In the army, there is a saying that states “these instructions were written in blood” (softening the analogy: systems experienced downtime). In other words, they are there to teach a lesson learned from others’ unfortunate experiences, and they have a good basis for doing so.

Another AWS specific but has counterparts in other platforms: AWS Backup can help with the “heavy” lifting of most backup services.

One extremely important point to note here is that backups need to be regularly tested! It’s never enough to “know they’re there”. When the fit hits the shan 😉 they may actually not work, or just as bad - there’s no knowledge of how to apply them.

Build to last, self-heal, and think of future maintainers

For one minute, imagine you were fighting yet-another-fire caused by a hacky script, or a poor choice of technology, grunting quietly and cursing at an engineer leaving legacy code. Instead of becoming someone else’s future headache, try being the source of quality and standards. My belief is that this has a compounding effect, where others learn from engineers and apply, and their names are kept in a positive light long after they are gone. With that in mind, try being the source of quality and standards rather than a future headache. I honestly believe, this has a compounding effect, where others learn and apply, and engineers’ names are kept in positive aspects long after they’ve left.

Change management and reviews for everything

Keep changes managed. Those of you who have read The Phonix Project will likely have an image of sticky notes on walls describing a practically-non-existent change management system. Things don’t get missed when a change management process is in place, preferably with a chain of approval and quality review process. Even more importantly, the engineers evolve, and production tends not to burst into flames. There’s nothing like review for sharing ideas, concepts, conventions and standards! It can be applied to almost anything; from code to infrastructure to blog posts to landing pages.

Staging first

Despite staging (and other environments) being well-known for handling changes and testing them, they are too often ignored when it comes to operations-related systems. Exactly these types of systems should begin with a staging environment. Yes, they are usually not customer-facing, but they serve all others and ensure that they are safe, healthy, and functional. CI servers, alert systems, and container orchestrators are all essentials for the production environment. Even if they’re consumed as a service, in spite of the fact that they are all managed by a company you trust, they still suffer from many of the known pain points; versions can be upgraded, changes can break, retention can run out, and capacities can be reduced. “Staging First” is a concept that argues that everything should be staged in a separate environment before changes get to production. These should also be separated by accounts and user logins, for security, but also for human error and blast radius management.

Aftermath

This is by no means a comprehensive list. These are principles we developed as a team, agreed on, and adopted. You are invited to challenge them, add or suggest changes, and create a fruitful discussion around them. Let’s face it, we’re all here to learn, otherwise what’s the point? ;)

Thank you for reading 🖤
Feel free to reach out with questions or comments.

While both CloudFormation and Terraform are robust IaC platforms that offer efficient configuration and automation of infrastructure provisioning, there are a few key differences in the way they operate. CloudFormation is an AWS tool, making it ideal for AWS users looking for a managed service. Terraform, on the other hand, is an open-source tool created by Hashicorp which provides the full flexibility, adaptability, and community that the open-source ecosystem has to offer. These differences can be impactful depending on your specific environment, use cases, and several other key factors.

In this post, I’ll compare CloudFormation and Terraform based on important criteria such as vendor neutrality, modularity, state management, pricing, configuration workflow, and use cases to help you decipher which one is the best fit for you.

But first, I’ll provide a bit of background on each platform and highlight the unique benefits that each of them brings to the table.

What is CloudFormation?

AWS CloudFormation is an Infrastructure as Code (IaC) service that enables AWS cloud teams to model and set up related AWS and third-party resources in a testable and reproducible format.

The platform helps cloud teams focus on the application by abstracting away the complexities of provisioning and configuring resources. You also have access to templates to declare resources; CloudFormation then uses these templates to organize and automate the configuration of resources as well as AWS applications. It supports various services of the AWS ecosystem, making it efficient for both startups and enterprises looking to persistently scale up their infrastructure.

Key features of CloudFormation include:

• Declarative Configuration with JSON/YAML
• The ability to preview environment changes
• Stack management actions for dependency management
• Cross-regional account management

What is Terraform?

Key Takeaways While both CloudFormation and Terraform have the concept of modules, Terraform’s is better defined State storage in Terraform requires special care as the state file is needed to understand the desired state and can contain sensitive information CloudFormation excels at deploying AWS infrastructure whereas Terraform is best suited for dynamic workloads residing in multiple deployment environments where you want to control additional systems beyond the cloud Many organizations choose to use Terraform for databases and high-level infrastructure and CloudFormation for application deployment Since Terraform uses HCL, this can create beneficial segregation between Ops and Dev as Dev may not be familiar with this language. While both CloudFormation and Terraform are robust IaC platforms that offer efficient configuration and automation of infrastructure provisioning, there are a few key differences in the way they operate. CloudFormation is an AWS tool, making it ideal for AWS users looking for a managed service. Terraform, on the other hand, is an open-source tool created by Hashicorp which provides the full flexibility, adaptability, and community that the open-source ecosystem has to offer. These differences can be impactful depending on your specific environment, use cases, and several other key factors.

In this post, I’ll compare CloudFormation and Terraform based on important criteria such as vendor neutrality, modularity, state management, pricing, configuration workflow, and use cases to help you decipher which one is the best fit for you.

But first, I’ll provide a bit of background on each platform and highlight the unique benefits that each of them brings to the table.

What is CloudFormation? RELATED SPONSORED CONTENT SQL to NoSQL: Architecture Differences and Considerations for Migration Data Quality Fundamentals - Download the eBook (By O’Reilly) 2022 Gartner Market Guide for Container Management Foundations of Scalable Systems (By O’Reilly) Stop Using Databases! 3 Keys to Letting go of their Tyranny RELATED SPONSOR

Code, deploy, and scale Java your way. Microsoft Azure supports your workload with abundant choices, whether you’re working on a Java app, app server, or framework. Learn more.

AWS CloudFormation is an Infrastructure as Code (IaC) service that enables AWS cloud teams to model and set up related AWS and third-party resources in a testable and reproducible format.

The platform helps cloud teams focus on the application by abstracting away the complexities of provisioning and configuring resources. You also have access to templates to declare resources; CloudFormation then uses these templates to organize and automate the configuration of resources as well as AWS applications. It supports various services of the AWS ecosystem, making it efficient for both startups and enterprises looking to persistently scale up their infrastructure.

Key features of CloudFormation include:

Declarative Configuration with JSON/YAML The ability to preview environment changes Stack management actions for dependency management Cross-regional account management What is Terraform? Terraform is Hashicorp’s open-source infrastructure-as-code solution. It manages computing infrastructure lifecycles using declarative, human-readable configuration files, enabling DevOps teams to version, share, and reuse resource configurations. This allows teams to conveniently commit the configuration files to version-control tools for safe and efficient collaboration across departments.

Terraform leverages plugins, also called providers, to connect with other cloud providers, external APIs, or SaaS providers. Providers help standardize, provision, and manage the infrastructure deployment workflow by defining individual units of infrastructure as resources.

Key features of Terraform include:

• Declarative configurations via Hashicorp Configuration Language (HCL)
• The support of local and remote execution modes
• Default version control integration
• Private Registry
• Ability to ship with a full API

Comparing CloudFormation and Terraform

Vendor Neutrality

The most well-known difference between CloudFormation and Terraform is the association with AWS. While you can access both tools for free, as an AWS product, CloudFormation is only built to support AWS services. Consequently, it is only applicable for deployments that rely on the AWS ecosystem of services. This is great for users who run exclusively on AWS as they can leverage CloudFormation as a managed service for free and at the same time, get support for new AWS services once they’re released.

In contrast, Terraform is open-source and works coherently with almost all cloud service providers like Azure, AWS, and Google Cloud Platform. As a result, organizations using Terraform can provision, deploy, and manage resources on any cloud or on-premises infrastructure, making it an ideal choice for multi-cloud or hybrid-cloud users. Furthermore, because it is open source and modular, you can create a provider to wrap any kind of API or just use an existing provider. All of these capabilities have already been implemented by various vendors which offer users far more flexibility and convenience, making it ideal for a greater variety of use cases.

However, on the downside, Terraform often lags behind CloudFormation with regard to support for new cloud releases. As a result, Terraform users have to play catch-up when they adopt new cloud services.

Modularity

Modules are purposed for reusing and sharing common configurations, which renders complex configurations simple and readable. Both CloudFormation and Terraform have module offerings, however, CloudFormation’s is newer, and therefore not as mature as Terraform’s.

CloudFormation has always offered features to build modules using templates, and as of 2020, they now offer out-of-the-box support for modules as well. Traditionally, CloudFormation leverages nested stacks, which allow users to import and export commonly used configuration settings. Over the past few years, however, CloudFormation launched both public and private registries. As opposed to Terraform, CloudFormation offers its private registry right out-of-the-box which enables users to manage their own code privately without the risk of others gaining access to it. Furthermore, CloudFormation’s public registry offers a wide array of extensions such as MongoDB, DataDog, JFrog, Check Point, Snyk, and more.

Despite the fact that CloudFormation has come a long way in its modularity, Terraform has innately supported modularity from the get-go, making its registry more robust and easier to use. The Terraform registry contains numerous open-source modules that can be repurposed and combined to build configurations, saving time and reducing the risk of error. Terraform additionally offers native support for many third-party modules, which can be consumed by adding providers or plugins that support the resource type to the configuration.

State Management

One of the benefits of CloudFormation is that it can provision resources automatically and consistently perform drift detection on them. It bundles AWS resources and their dependencies in resource stacks, which it then uses to offer free, built-in support for state management.

In contrast, Terraform stores state locally on the disk by default. Remote storage is also an option, but states stored remotely are written in a custom JSON file format outlining the model infrastructure and must be managed and configured. If you do not manage state storage properly, it can have disastrous repercussions.

Such repercussions include the inability to perform DR because drifts have gone undetected, leading to extended downtime. This occurs if the state is impaired and can’t run the code, which means the recovery has to be done manually or started from scratch. Another negative repercussion is if the state file for some unexpected reason becomes public. Since state files often store secrets such as keys to databases or login details this information is quite dangerous to your organization if it gets into the wrong hands. If Hackers find state files it’s easier for them to attack your resources. This is an easy mistake to make since generally speaking, Terraform users that manage their own state on AWS store the files in an S3 bucket, which is one-way state files can be exposed publicly.

To combat this challenge, Terraform also offers an efficient deployment of self-managed systems that take care of the state by leveraging AWS S3 and DynamoDB. In addition, users can also purchase Terraform’s Remote State Management to automatically maintain state files as a service.

Pricing, License & Support

CloudFormation is a free service within AWS and is supported by all AWS pricing plans. The only cost associated with CloudFormation is that of the provisioned AWS service.

Terraform is open-source and free, but they also offer a paid service called Terraform Cloud which has several support plans like Team, Governance, and Business that enable further collaboration. Terraform Cloud offers additional features like team management, policy enforcement, a self-hosted option, and custom concurrency. Pricing depends on the features used and the number of users.

Language

CloudFormation templates are built using JSON/YAML, while Terraform configuration files are built using HCL syntax. Although both are human-readable, YAML is widely used in modern automation and configuration platforms, thereby making CloudFormation much easier to adopt.

On the other hand, HCL enables flexibility in configuration, but the language requires getting used to it.

It’s also worth mentioning some IaC alternatives that offer solutions for those who prefer to use popular programming languages. For example, in addition to CloudFormation, AWS offers CDK which enables users to provision resources using their preferred programming languages. Terraform users can enjoy the same benefits with Hashicorp’s [cdktf]*(https://github.com/hashicorp/terraform-cdk) which allows you to set HCL state files in Python, Typescript, Java, C#, and Go. Alternatively, Pulumi offers an open-source IaC platform that can be configured with a variety of familiar programming languages.

Configuration Workflow

With CloudFormation, the written code is stored by default locally or in an AWS S3 bucket. This configuration workflow is then used with the AWS CLI or the AWS Console to build the resource stack.

Terraform uses a straightforward workflow that only relies on the Terraform CLI tool to deploy resources. Once configuration files are written, Terraform loads these files as modules, creates an execution plan, and applies the changes once the plan is approved.

Use Cases

While both CloudFormation and Terraform can be used for most standard use cases, there are some situations in which one might be more ideal than the other.

Being a robust, closed-source platform that is built to work seamlessly with other AWS services, CloudFormation is considered most suitable in a situation where organizations prefer to run deployments entirely on AWS and achieve full state management from the get-go.

CloudFormation makes it easy to provision AWS infrastructure. Plus, you can more easily take advantage of new AWS services as soon as, or shortly after, they’re launched due to the native support, compliance, and integration between all AWS services. In addition, if you’re working with developers, the YAML language tends to be more familiar, making CloudFormation much easier to use.

In contrast, Terraform is best suited for dynamic workloads residing in multiple deployment environments where you want to control additional systems beyond the cloud. Terraform offers providers specifically for this purpose whereas CloudFormation requires you to wrap them with your own code. Hybrid cloud environments are also better suited for Terraform. This is because it can be used with any cloud (not exclusively AWS) and can, therefore, integrate seamlessly with an array of cloud services from various providers–whereas this is almost impossible to do with CloudFormation.

Furthermore, because it’s open source, Terraform is more agile and extendable, enabling you to create your own resources and providers for various technologies you work with or create.

Your given use case may also benefit from implementing both–for example, with multi-cloud deployments that include AWS paired with other public/private cloud services.

Another example where both may be used in tandem is with serverless architecture. For example at Zesty, we use Serverless which uses CloudFormation under the hood. However, we also use Terraform for infrastructure. Since we work with companies using different clouds, we want to have the ability to use the same technology to deploy infrastructure in multiple cloud providers, which makes Terraform the obvious choice for us. Another unplanned benefit this adds for us is the natural segregation between Dev and Ops. Because Ops tend to be more familiar with the HCL language, it creates barriers that make it more difficult for another team to make a mistake or leave code open to attacks.

In general, many organizations choose to use Terraform for databases and high-level infrastructure and CloudFormation for application deployment. They often do this because it helps to distinguish the work of Dev and Ops. It’s easier for developers to start from scratch with CloudFormation because it runs with YAML or JSON which are formats every developer knows. Terraform, on the other hand, requires you to learn a different syntax. The benefit of creating these boundaries between Dev and Ops is that one team cannot interfere with another team’s work, which makes it harder for human error or attacks to occur.

It’s worth noting that even if you don’t currently use both IaC platforms, it’s ideal to learn the syntax of each in case you wind up using one or the other in the future and need to know how to debug them.

Summary

As we have seen, both CloudFormation and Terraform offer powerful IaC capabilities, but it is important to consider your workload, team composition, and infrastructure needs when selecting your IaC platform.

Because I’m partial to open-source technologies, Terraform is my IaC of choice. It bears the Hashicorp name, which has a great reputation in the industry as well as a large and thriving community that supports it. I love knowing that if I’m not happy with the way something works in Terraform, I can always write code to fix it and then contribute it back to the community. In contrast, because CloudFormation is a closed system, I can’t even see the code, much less change something within it.

Another huge plus for Terraform is that it uses the HCL language which I prefer to work with over JSON/YAML. The reason for this is that HCL is an actual language whereas JSON and YAML are formats. This means when I want to run things programmatically, like running in a loop or adding conditionals, for example, I end up with far more readable and writable code. When code is easier to read, it’s easier to maintain. And since I’m not the only one maintaining this code, it makes everyone’s life easier.

Another reason I prefer to use Terraform is due to our extensive use of public modules, which we needed to leverage prior to CloudFormation’s public registry offering.

While CloudFormation may be quicker to adopt new AWS features and manage the state for you for free, all things considered, I prefer the freedom that comes with open source, making Terraform a better choice for my use case.

Hope you found this comparison helpful!

Kubernetes storage is useful for managing multiple forms of persistent and non-persistent storage in a cluster to cater to both stateful and stateless workloads in a containerized environment. Proper management of Kubernetes storage options allows us to dynamically provision the most suitable storage resources for multiple applications with minimal administration overhead.

There are various storage concepts you can leverage in Kubernetes, as well as typical use cases and pro tips for running Kubernetes on AWS from a storage perspective.

Introduction to Kubernetes Storage Concepts

The storage architecture for Kubernetes is based on volumes as the core abstraction. Volumes can be ephemeral (non-persistent) or persistent, depending on their intended use cases. Kubernetes also allows for the dynamic provisioning of storage resources using volume claims.

Volumes

Volumes are the basic storage entities in Kubernetes. A process in the container sees a filesystem view, which includes a root filesystem that matches the initial contents of a container image and the volumes mounted inside the container (if defined within the container specifications).

Volumes are mounted at specified paths within the container image, so you must independently specify the mount path of each volume used for each container in the pod. While volumes cannot be mounted within other volumes, a volume can be shared between multiple containers in a pod using sub-paths within the volume mount paths in the pod specifications.

Kubernetes supports multiple storage volumes, including local storage devices, network file systems (NFS), and cloud storage services like AWS Elastic Block Store (EBS) volumes. Developers can also create custom storage plugins to support specific storage systems deployed on Kubernetes clusters as extensions.

While support for cloud storage services is currently available within the core Kubernetes project, the Kubernetes Storage Special-Interest Group (k8s-sig-storage) is slowly shifting toward providing storage support through external Container Storage Interfaces (CSIs). For example, the native Amazon Elastic Block Store volume has been deprecated as of v1.17 in favor of the Amazon EBS CSI. 

Ephemeral Volumes

Unless specified within the container specifications, each container in a pod is created with an ephemeral volume by default in Kubernetes. This means there is a temporary storage directory on the machine that hosts the pod. Ephemeral volumes are removed after the pod ceases to exist.

The data in ephemeral volumes is safe in the case of a container crashing, as this does not remove a pod from a node. However, the pod with the crashed container may subsequently be deleted due to rescheduling or any reason that causes the pod to be evicted to another host, causing the data in the ephemeral volumes to be removed in the process.

Persistent Volumes (PVs) and Persistent Volume Claims (PVCs)

Kubernetes uses persistent volumes, and persistent volume claims to allow pods to use storage in a portable manner while abstracting its implementation away from how storage is consumed.

A persistent volume (PV) is a storage entity within a cluster that is either allocated manually by an administrator or dynamically allocated based on a storage class. PVs define the details of the storage implementation, such as capacity, access modes, storage class, and reclaim policy. As they are cluster resources, PVs are not portable between clusters.

A persistent volume claim is a storage request used by developers to describe an application’s storage requirements, for example, a container’s storage size and access mode. Since the storage request is separate from the storage creation, Kubernetes can enforce access control mechanisms based on the container or pod credentials and the available PVs in the cluster.

A PV’s lifecycle is independent of pods. The lifecycle of a PV and PVC consists of 5 stages:

• Provisioning: The cluster administrator creates a PV in static or dynamic modes based on a storage class.
• Binding: When the PVC is made, a PV that fulfills the requirements is assigned and bound to the PVC.
• Using: The container uses storage resources from a PV via a PVC.
• Releasing: The container releases the PV when the PVC is removed.
• Reclaiming: Kubernetes reclaims the storage resources used in the PVC based on the reclaim policy of the PV, unless the storage is set to be deleted when released.

For Kubernetes on AWS, EBS volumes are persistent volumes hosted in the same region and availability zone as the EC2 instance nodes running on the EKS cluster. When a pod is removed from an EC2 instance node on the EKS cluster, the data in the EBS volume mounted to the pod is persisted and the volume is unmounted.

Storage Classes

Cluster administrators can configure storage classes in the cluster and assign PVs to each class. Each class represents a particular type of storage that users can request in their PVCs, depending on varying workload requirements within the cluster.

Dynamic Volume Provisioning with Storage Classes

Dynamic volume provisioning is a feature in Kubernetes that lets you create storage volumes on-demand without requiring cluster administrators to create new ones manually.

Each storage class specifies a volume plugin called a provisioner and the parameters required for it to allocate storage volumes dynamically. When a user configures a storage class in their PVC, the provisioner automatically creates a storage volume based on the required specifications.

While some provisioners are internal and shipped alongside the Kubernetes project, you can also use external provisioners by following the specifications defined by Kubernetes.

Container Storage Interfaces (CSIs)

A Container Storage Interface (CSI) is a Kubernetes extension that provides an extensible plugin architecture for vendors to create compatible storage plugins. 

There are custom storage plugins in the form of CSI drivers for arbitrary storage systems external to your Kubernetes project (e.g., Amazon Elastic File System).

After you deploy the CSI driver on the Kubernetes cluster, you can use these CSI volumes with other Kubernetes storage API objects such as PVs and storage classes. For example, you can create a storage class that references the CSI provisioner, assign them to PVs, and reference the PVs in PVCs to mount the CSI volumes to a pod.

With the core Kubernetes project’s gradual shift toward distributing control over the provisioning of non-native storage to the respective providers, external vendor-based CSI drivers are currently the preferred approach for managing the lifecycle of the external storage system in Kubernetes clusters. As part of this gradual shift, numerous migration features are slowly being released within the core project to ease the transition from the in-tree storage plugin to the corresponding vendor-based CSI driver.

Use Cases for Kubernetes Storage

With various options available for Kubernetes storage, you need to know which type to choose for different use cases.

Persistent vs. Ephemeral Storage

For transient applications that require data storage (e.g., applications that extract read-only input data in files), you can use ephemeral volumes for storing data within the lifetime of the pod without being limited to the location and availability of some persistent volume.

Persistent volumes will store data beyond the lifetime of the pod for applications requiring data persistence across restarts (such as a database).

Varying Workload Requirements

For use cases that require managing storage for various workloads within the cluster, you can opt for storage classes with various storage levels, backup policies, or any arbitrary policies defined by cluster administrators.

5 Pro Tips for Running Kubernetes on AWS from a Storage Perspective

When it comes to running Kubernetes on AWS from a storage perspective, there are five pro tips you should keep in mind. 

1. Include PVCs in the Container Config

Using PVCs in the container config as part of the deployment IaC template lets users request persistent storage across clusters. This, in turn, enables storage configuration portability that is not tightly coupled with cluster resources. This tip applies to ephemeral Amazon EKS clusters created on-demand and Amazon EKS clusters deployed across multiple regions and availability zones.

2. Do Not Include PVs in the Container Config

Including PVs in the container, config is not recommended if you want to avoid tight coupling with a specific volume and prevent failure in binding storage volume when instantiating the container in the pod.

Instead, use PVCs as volumes while the cluster provisions storage by finding the PV bound to the claim and mounting that volume to the pod.

3. Define a Default Storage Class for PVCs

Cluster administrators can specify a default storage class for PVCs that do not have specific requirements for the storage class they need to bind to; they can also create separate storage classes that represent varying workload requirements. This allows users to request storage in PVCs based on the name of the storage class without manually specifying the volume within each pod’s specifications. Otherwise, PVCs without a specified storage class will fail to provide a PV.

4. Let Users Provide a Storage Class 

Cluster administrators can provide users with a storage class when instantiating the config template. If the user provides a storage class name, the value should be defined in the StorageClassName within the PVC specifications so that the PVC can match the correct storage class. Otherwise, a PV can be automatically provisioned for the user using the default storage class in the cluster.

5. Look Out for Unbound PVCs

When monitoring your Kubernetes cluster with Prometheus and configuring alerting rules via Alertmanager, keep an eye out for PVCs that remain unbound for a prolonged period. This could mean your cluster lacks dynamic storage support or a storage system, where users won’t be able to deploy configs requiring PVCs. If this is the case, the user should create a PV that matches the requirements defined in the PVC. 

Conclusion

Implementing best practices for Kubernetes storage enables you to apply optimal storage configurations and dynamically provision suitable storage resources to multiple containerized applications without significant administrative overhead.

To reap the benefits of Kubernetes storage options on AWS, you can gradually adopt these best practices by creating storage classes and enabling dynamic volume provisioning in the cluster. You can choose suitable storage types for your containerized workloads based on the given use case.

Give me six hours to chop down a tree and I will spend the first four sharpening the axe.

• A. Lincoln

Well I didn’t even know there was an axe… K8s resources, in that context, are one heck of a tree to chop. You better come ready to work. I hope that this information will find the axe for someone outthere and smooth out the process.

Controllers? Operators? CRDs?

Let’s bring some order to the chaos. K8s documents the notions of controllers and operators. The reader may be puzzled by the subtle differences between the two after reading them both.

An “Operator” is a pretty name coined by CoreOS back in 2016, to describe the concept of managing application infrastructure using controllers and custom resources.

K8s Custom Resource Definitions (CRDs) allow users to extend the system with the same tools used to create and manage Pods, ReplicaSets, StatefulSets, and ConfigMaps. This notion is discussed in the “kubebuilder” book - “Building a CronJob” tutorial. The creation of a CronJob component using an existing Job resource is an incredible example of an operator or controller. In a 2018 keynote in KubeCon, Maciej Szulik, the creator of CronJob actually claimed its not yet implemented with a full on controller features.

Described in the K8s docs, a “controller” is a component that utilizes a control loop to bring the “desired state” of the system to its actual state. An operator is, in that sense, a controller with a CRD, and a story.

Kubernetes is a Database

In a K8s podcast episode, Daniel Smith, Co-TL of SIG API, explains a powerful concept; “K8s is more like a database than an event-driven system”. He explains that instead of having different components communicating with one another, K8s holds information like a database. This information is the state of the cluster. An update or creation of a resource will result in a new desired state. Through a control loop, the controller takes care of achieving the desired state as events of its kind are received. Through iterations of the control loop, the desired state is reconciled with the existing state.

“Reconciliation” is noted here as the coined term for achieving equilibrium. You’ll find the Reconcile loop in most controllers as the heart of the logic, and where events start their way.

K8s is more like a database, then an event-driven system.

Start with Kubebuilder, read the book!

I wish someone had told me that. There are so many resources out there on how to build K8s operators and controllers, some like the Operator Framework automate things even further, helping the user construct operators based on native language, Helm, and others.

That said, Kubebuilder, does most of this work for you already, and IMHO, better. It comes with a (really incredible) piece of documentation: The Kubebuilder Book.

A must read, if you plan to build a K8s controller.

The book takes the reader from concept, through a real-life example and its step-by-step development process. It covers code generation, APIs, concepts of control loops and reconciliation, deployment and local development. Almost every piece of information needed for such a project.

The author has put a lot of care into writing the book and it’s easy to read. The concepts explored in the text are interesting and come with code snippets to underscore the points.

CRDs don’t create metadata by default

Now that’s a surprise, I had to double-check the data thoroughly to work out that metaData was never there. You can’t find it anywhere on the root-level and it’s seemed to be missing everywhere else.

My own case involves the creation of a statefulSet as part of my CRD, which I thought would be treated as a first-class citizen. On the contrary; nested meta objects will be ignored unless instructed specifically not to be. I don’t have answers to “why” (even though treated as a bug), only the “how”:

In its docs, the Code Gen CLI, sates the additional option to set crd:generateEmbeddedObjectMeta=true to allow nested meta objects. In the context of the makefile code generator, this would look something along the lines of:

.PHONY: manifests
manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
	$(CONTROLLER_GEN) rbac:roleName=manager-role \
                    crd:generateEmbeddedObjectMeta=true,maxDescLen=0 \
                    webhook paths="./..." \
                    output:crd:artifacts:config=config/crd/bases

You may also note the maxDescLen=0; During the development process (and afterwards), the amount of sheer text generated to every piece of the resource documentation is unbearable. Just try to kubectl describe your resource to find your terminal slowly losing its history. This doesn’t have to be “0”, but can help bring the lines of text to a manageable situation.

Interacting with CRDs outside the controller’s context is not straight forward

CRDs are great. You can create any kind (pun not intended) of object in K8s and manage it using a controller.

What about trying to interact with it outside of the controller’s context? You may be tempted to ask “why would anyone do that”. The way I see it:

1. Customers may want to interact with this production service using their own controller or informer.
2. Depending on the system built, other pieces of software may need access to the object produced. Exactly this was the usecase I had to deal with - a Daemonset pod that updated data on new types of CRDs based on node-centered events.

It is relatively straight-forward to access all native resources with client-go:

clientSet, _ := kubernetes.NewForConfig(config)
pods := clientSet.CoreV1().Pods("")

Custom resources, however, not only require you to obtain the relevant types for the object, once you do so, there is no client exported or generated for you. Additionally, once you finally reach the client you were looking for, you discover a full-on raw HTTP API with unstructured JSON requests and responses. Working this way isn’t fun (or safe).

Discovery number one: “The dynamic client”

The dynamic client for K8s is described well in this blog post. The TL;DR is that it allows direct access to any kind of object within the cluster, whether structured or not. Here is one of the main components of the dynamic client that deals with unstructured objects:

unstructured.Unstructured: This is a special type that encapsulates an arbitrary JSON while also complying with standard Kubernetes interfaces like runtime.Object

• The Kubernetes dynamic client

Provided a schema.GroupVersionResource (GVR), the dynamic client will fetch the CRD object and return it as an unstructured data object:

ctx := context.Background()
gvr := schema.GroupVersionResource{
  Group: "group.example.com",
  Version: "v1alpha1",
  Resource: "myresource",
}
returnedObj, err := c.Resource(gvr).
                    Namespace("default").
                    Get(ctx, "myresource-sample", metav1.GetOptions{})
if err != nil {
  return
}

It’s all fine and dandy, but what do you actually do with raw data if you need to do more than just parse it? According to the post mentioned above, you can modify nested data fields using functions like unstructured.NestedInt64, parsing and changing raw data in-place and returning it back to the cluster.

It felt like there was more to it:

Discovery number two: “The DefaultUnstructuredConverter”

The runtime library exposes a converter function with a FromUnstructured method for unmarshaling data into the known CRD type. This creates typed, mutable, and accessible information from the unstructured data:

var myResource MyResourceType
err = runtime.
      DefaultUnstructuredConverter.
      FromUnstructured(returnedObj.UnstructuredContent(), &myResource)
if err != nil {
  return err
}

From fetching the object to parsing and manipulating it away from the context of the controller, that’s a complete interaction with a CRD.

Since my use-case involves customers, I wanted to provide them the option to interact with the generated objects, so I exposed it through my own client using a few simple methods that handled CRUD logic:

func Get(c dynamic.Interface, nsn types.NamespacedName) (MyResource, error)
func List(c dynamic.Interface, nsn types.NamespacedName) (MyResource, error)
func Update(c dynamic.Interface, mr MyResource) (error)
func Delete(c dynamic.Interface, mr MyResource) (error)

Use Finalizers to terminate external resources

Finalizers are logic processes that are required before a K8s resource is deleted. The book says so, but it’s easy to skip. Perhaps you noticed a deletionTimestamp added to the metadata of an object when you tried to delete it. This is a finalizer that prevents garbage collection of the resource. For example, in AWS EBS, the internal logic backs the EBS controller attempts to delete the physical volumes before removing a PVC (removing the finalizer field in order to do so). As soon as a finalizer is removed, the object is automatically collected by the GC and removed from the cluster. With this method, you, the user, do not have to deal with waste and cluster leftovers.

Set Owner Reference on tracked objects

There is another very important concept, yet easy-to-skip section in the Kubebuilder “implementing a controller”: The K8s garbage collector knows to remove these objects when the parent object is deleted, just as the CronJob creates Jobs. If your CRD generates other resources in the cluster, like a Pod, or Deployment, then you must set an owner reference.

When constructing the object, make sure to include:

# r being the reconciler receiver
if err := ctrl.SetControllerReference(parentObj, childObj, r.Scheme); err != nil {
  return nil, err
}

Get your groups and crd name right

Changing them is quite the challenge, to say the least. Think about the group and resource names when you are creating the controller, CRD, API, etc. There are several reasons why these are important:

1. Imports are made throughout the project. Imports must be a. unambiguous, b. straightforward, and c. named meaningfully
2. They will be used in different places, such as yaml files holding the objects’ API Versions and Kinds. As well as in other areas such as the discussed GVR or GVK (GroupVersionKind) objects. If there is a meaningful group name other than “crd” or “apps”, use it. If not, keep in mind that it must make sense in the context of <group-name>.company.com as part of the apiVersion field.
3. Changing them is a pain in the bum, especially with custom resource objects. Besides being part of every generated function or kubebuilder generator markers, it is probably mentioned hundreds of times throughout the code. Refactoring isn’t necessary, but not doing so will reduce the project’s readability. Simply put: Do not change the name of the CRD, unless you must. You’ve been warned 😉

That’s all there is to it. Please let me know if you have any other dos and don’ts to add or remove. In any case, that’s all the things I wish I knew instead of spending hours figuring them out for myself.

Thanks for reading!

“Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers.”

• https://en.wikipedia.org/wiki/Serverless_computing

A recent uninformed discussion on Linkedin about container orchestration (wrongfully) compared to serverless, sparked a few thoughts in my mind:

1. How well do I understand what a serverless function is?
2. What does this mean for developers in my team that run code exclusively on AWS Lambda?
3. Does knowing the environment make any difference?

It is my intention in this post to answer these questions, given that (3) is obvious to me; I have to understand the environment in order to make informed decisions. In addition, I assume that in regards to question (2), if I didn’t have the time to dig deeper, many of my colleagues didn’t either, so this may be my way of sharing what I’ve learned with them.

The benefits and drawbacks of Serverless functions

Pros:

• Speed of development - using frameworks like serverless.com (deserves its own blog post), development speed is increased exponentially. All resources are easily deployed, connected to the network, protected by permissions and security groups, as well as a lot of other features. It can relieve DevOps / production engineers of the burden of managing moving parts, and allow developers to operate independently on their own terms, eliminating bottlenecks. Obviously, this isn’t magic, and “Managed correctly” is key here, templates and boilerplates, coupled with least privilege access will go a long way towards ensuring speed is not sacrificed for security or control.
• Scale - as Lambda is a service that runs packaged code, it can (theoretically) scale infinitely, only limited by AWS’s own limitations, which are considered close to infinite in most use-cases. It scales quickly and automatically. Even though it may come at a price, both in terms of service and cold starts, if the volume is monitored, it can be planned accordingly to maximize speed and efficiency (see “Handling cold starts” above).
• Security - The Lambda function, as noted, runs a number of seconds on its own infrastructure before going to “sleep”. In that sense, it is more secure because it can only be accessed through AWS’s API, and cannot be accessed in any other way. On the other hand, containers are usually accessed through an attached shell. When a malicious actor has access to an internal system, it becomes a possible attack vector. In this sense, Lambda functions are inaccessible, even when attached to a VPC. In addition, there isn’t an underlying infrastructure like K8s or ECS nodes to run, so one fewer layer needs to be protected.
• Efficiency - Lambda functions run for the required amount of time and then hibernate (or completely shut down the environment) until the next run. This provides maximum efficiency. Infrastructure is only paid for by the number of seconds it takes to process data, nothing more, nothing less. There is no billing for “dead” time. This drives constant improvements to reduce runtime. It can be achieved by using message queues, caching layers, or simply by improving the application’s performance. It is a pleasant side-effect with a positive impact.

Cons:

• Price - Lambda functions are not cheap. AWS offers a generous free tier, but lambdas can grow to be a major part of an AWS cloud bill after a certain point. There are many solutions to this problem, such as offloading certain components to containers or instances, and improving performance. However, this is one of the most important factors to consider when making the decision.
• Speed - also listed as a pro, speed the speed at which developers can deploy new infrastructure can get out of hand. It is something to consider when allowing the whole engineering team to use their networking and security components for production functions.
• Distribution can be an obstacle - Lambdas are best when they are distributed; when the code is divided into small parts, each with one logical purpose, rather than a few. This allows them to respond, scale and operate as fast as possible. At the same time, it can be difficult to keep everything under control. Monitoring, CI pipelines, and infrastructure become increasingly challenging to track, monitor and manage as the number of them grows.

What is it

Serverless functions are a way to run code in a distributed, scalable and efficient manner in a cloud environment.

A key feature is the ability to write code and “throw” it to the cloud, where some “magical” system runs it, preparing an environment and billing the user for the runtime seconds and resources used.

Under the hood

AWS Lambda runs code in a “runtime environment”, which is a containerized environment that holds everything the application needs. Dependencies, layers, and files are all prepared and loaded onto an AWS-shared infrastructure. This is done in an isolated, secured way so that neighboring functions are not exposed in any way to the other “environments”.

“Cold start” is the process of setting up a container and preparing it for a Lambda function, then running it for the first time. Imagine calling a function in a simple application you wrote, but each time it’s called, the application starts and then stops. The process is of course inefficient in the sense that it runs longer, but it is extremely more efficient considering the time a certain piece of code is not called, and therefore, is not billed.

This is not a straight forward process at all; certain questions must be asked:

• How often is this piece of code called on a minute, hour, or daily basis?
• How long does it take for the function to complete its run?
• Could the function end faster? (hint: yes, it sure can!)
• How crucial is responsiveness?
• Is latency something we’d like to pay extra for, or keep as low as possible?
• Assuming serverless is the way to go for our engineering team, but mainly in terms of speed, is the cloud the most suitable option for us?

Let’s try to answer a few of these questions to make architectural decisions:

What is the frequency of calling the function?

If the answer is thousands of times a minute, perhaps a provisioned concurrency  is worth considering. If the function is not being called all that often, perhaps it’s wise to use a lambda warmer to keep it warm - see dealing with cold starts below.

How long does it take for the function to run?

The best practice is to have functions operate on a single logical unit, having the smallest possible task to take care of. This will result in a short running time and a more distributed system that can scale and start up quickly. An option to consider is wrapping a “long running” function with supporting services to speed up its execution. A “long” run time can range from seconds to minutes. Examples for solutions are - utilizing queues and additional functions instead of sending a request and waiting for a response. Another option is the use of a cache mechanism to provide faster responses from data services. In the end, if a function still takes minutes to run, even with all the support and efficiency improvements, it may be a candidate for a live service. Maybe a job that runs in a container and terminates, or a full-fledged service running on K8s, or ECS. In either case, serverless is not a one-size-fits-all solution.

Is the cloud really the best choice?

When the goal of serverless is purely functional, i.e. making it easier for developers to work productively and deploy infrastructure quickly, a PaaS might not be the best option. Not because of features, but because of price, and in some cases, high latency when working against internal tools like databases and secret managers. Knative and Fission are two alternatives. As I don’t have real experience with these, I cannot comment on their capabilities. However, I can say they might hold some of our production serverless deployments in the future.

Key takeaway: “Serverless is not a one-size-fits-all solution”

A deeper look into networking

The functions run on an “AWS shared infrastructure”, as mentioned earlier. The shared environment here is an AWS VPC dedicated to Lambda. Functions are invoked through the Lambda API (and only through it) in the VPC, where they can access the public internet, but cannot access neighboring functions or the owners’ private network (that’s you).

A function must be connected to the internal VPC in order to gain access to private subnet resources, such as databases, internal API services, and other supporting components. This does not mean the function runs inside the VPC, rather it has access to it via:

1. Creating a cross-account attached interface
2. Attaching and using the network interface from the user’s VPC to access resources

A recent improvement to this network scheme

Lambda functions in a VPC used to be somewhat problematic, in that a user had to consider:

• Creating and attaching the network interface requires a longer cold start
• Keeping an eye on rate limits pertaining to the number of network interfaces in a user’s VPC
• Limits on the number of available IP addresses for attached interfaces should also be considered

The above has been dramatically improved in 2019, when AWS announced an improved networking for Lambdas. Since then (and gradually in additional regions), Lambda functions started reusing a “Hyperplane ENI”, essentially allowing network interface reuse.

Because the network interfaces are shared across execution environments, typically only a handful of network interfaces are required per function. Every unique security group:subnet combination across functions in your account requires a distinct network interface. If a combination is shared across multiple functions in your account, we reuse the same network interface across functions.

- AWS “Improved VPC networking for AWS Lambdas” announcement, 2019

Here’s a diagram from AWS, explaining the change visually, this is the “old” networking scheme for VPC-attached functions:

Vs the new networking model, where a shared ENI is used:

Dealing with cold starts

While you do not pay for the time it takes to start a container, it adds to the overall latency of the end user’s experience. This may be of little importance in some use cases, yet critical in others.

There are several ways to combat cold starts:

1. Warm up your lambdas with a lambda warmer. A warmer is essentially an external component that sends a ping to the function to initiate it and waits for a response. The key here is its repetitiveness and mimicking of customer-like behavior. It is recommended to do the following:
   1. Wait at least five minutes each time.
   2. Trigger the function directly to keep the same environment warm rather than trigger another one, which is typically the case when the trigger is coming through an API gateway
   3. Wait for a response, preferably one that’s prepared in the function specifically for warmers
   4. Send a test payload
   5. Further read this great post
2. Use provisioned concurrency - a way to pre-configure the number of desired environments that are kept warm for the customer. Despite being on the more expensive side, this solution “magically” solves the problem of cold starts to some extent. However, cold starts also occur when:
   1. Updated version is introduced (code and configuration are the same for Lambda)
   2. All provisioned concurrency is used and another incoming request is waiting
   3. The platform automatically re-balances availability zones deployments

3. Make sure functions attached to a VPC and using the same subnet share a security group. In this way, no latency is introduced when creating and attaching network interfaces, since they can use an existing ENI:

   Hyperplane ENIs are tied to a security group:subnet combination in your account. Functions in the same account that share the same security group:subnet pairing use the same network interfaces. This way, a single application with multiple functions but the same network and security configuration can benefit from the existing interface configuration.

   • AWS “Improved VPC networking for AWS Lambdas” announcement, 2019

Serverless isn’t always the best solution

The “cons” list above shows that running cloud functions can be challenging in some cases, and they may not always be the best solution. Aside from its usually high price, its features are good for some use cases and terrible for others. Combining functions with orchestrating containers seems to work well in my experience.

In addition to long-running services, containers can be used as standalone task containers (e.g. K8s cronjobs), triggered by functions, and used when the processing time exceeds a reasonable function run-time.

Serverless does not make “DevOps redundant”

It would be great if it did, but for the most part, it does the opposite. The serverless architecture leads to functions’ code being distributed across many repositories. The tracking, monitoring, maintaining, and other operational aspects become tedious and much more complex. This requires creative solutions, or more time-consuming manual labor. Both result in greater engineering effort. In that regard, serverless usually isn’t as promising as it sounds. Quite the contrary, in fact. In the future, a new tool or project may solve this complexity and provide a cross-functional solution. Serverless.com may carry this torch. We’re pretty much where we started in terms of hidden operations, and there’s still a long way to go until we can “abandon” Ops.

Thank you for reading

Serverless functions are an excellent platform if you take the considerations listed above into account. I hope this helps someone out there, and invite you to correct me if you find any inaccuracies.

Further reading material and references:

• AWS blog on Lambda performance optimization
• AWS announcement on Lambda VPC networking improvements