Krustlet v0.6.0 has several major new features, particularly around alpha level support for Container Storage Interface volumes. There were also some new SDK features (and a new crate!) that necessitated a few more breaking API changes. These changes are explained below. For more details on what isn't implemented yet, see the Known Issues section.

Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases.

Caveats

Please note that this is NOT production-ready software, but it is in a usable state. The WASI standard and wasmtime are still under heavy development, and because of this there are key features (like networking) that are missing; these will appear in the future. However, there is networking support available in wasCC.

Notable Features/Changes

• Container Storage Interface volume support is now available though PVCs 🎉 . Please note that this is ALPHA support and there are still some rough edges (namely around validation of some of the read access modes and other "advanced" configuration options). We plan on continuing to improve this during the 0.7 milestone.
• We have also broken out the state machine into its own SDK and crate called krator! This crate allows for generic reuse of the state machine logic to write any type of Kubernetes controller. Please read the introduction blog post for more information
• Lots of new doc updates. Thanks community members for all your help with that!
• Generic, reusable states. If you are a provider implementor, you're welcome. These are states that generally stay the same across provider implementations (like Error and Backoff states).
• A secondary state machine was introduced to manage individual containers. These utilize states defined in each of the two providers using Krator's API, a run_to_completion method provided by the Kubelet crate, and are spawned from within the Pod state machines of the providers. This is not required to implement a Provider, but we found that it significantly simplified our implementation.

Common State Implementations

We have implemented many common Pod states such that you can borrow one or more of these state handlers from the Kubelet crate rather than rewriting this boiler plate. These states currently include Registered, ImagePull, ImagePullBackoff, VolumeMount, CrashLoopBackoff, and Error.

If you would like to make use of these states, you must implement a few new traits:

• GenericProviderState for your shared state.
• GenericPodState for your object state.
• GenericProvider for your Provider type.

Please refer to either wascc-provider or wasi-provider for examples of how to implement these traits.

Breaking changes

Providers Trait

We had to make several small changes to the Provider trait to allow for generic state support.

The first is the addition of a new required method and associated type that captures the state of the provider itself (e.g. the container store and handles to running pods) that is shared across all pods:

type ProviderState: 'static + Send + Sync;

fn provider_state(&self) -> crate::state::SharedState<Self::ProviderState>;

SharedState<_> is simply a type alias for Arc<tokio::sync::RwLock<_>>, so you can refer to the Tokio documentation to understand its API.

The second is a change to the associated type for the PodState. This must now be something that implements the ObjectState type from the new krator crate:

type PodState: ObjectState<
    Manifest = Pod,
    Status = PodStatus,
    SharedState = Self::ProviderState,
>;

The last change is the addition of a new method that supports the Kubelet plugin registry (used for registering CSI plugins). This is an optional feature that has a default implementation returning None. If you want to opt in to CSI volumes, you can provide your own implementation of this function

fn plugin_registry(&self) -> Option<Arc<PluginRegistry>>;

All providers will need to implement these new fields and methods (with the exception of plugin_registry) upon upgrading to Krustlet 0.6. You can see an example of how these are implemented in the wasi-provider

Prelude

The prelude in kubelet::state has been moved to two separate preludes, kubelet::pod::state and kubelet::container::state, which export the same state machine API types but different status types, etc.

If you are not using the prelude, please be aware that a number of types were moved from kubelet::state to krator::state.

State

The State trait has changed in a few ways:

In the next method, a new argument has been introduced for accessing shared state: shared: SharedState<ProviderState>.

The next method now takes Manifest<Pod> instead of Pod. This wrapper allows access to a reflection of the Pod rather than a potentially out of date copy. At any time you can get a clone of the latest pod by calling pod.latest(). If you want to await updates to the manifest, Manifest implements Stream.

The status method now returns an arbitrary type which implements krator::ObjectState, rather than serde_json::Value. This allows you to wrap status-patch logic in an arbitrary type, rather than having to write JSON patches in every state handler.

AsyncDrop

The async_drop method implemented for PodState has been moved to a method on the newly introduced ObjectState trait, and the AsyncDrop trait has been removed.

Node Label

We have changed the node label kubernetes.io/os from linux to Provider::ARCH. The reason for this is that a number of vendors appear to use this label as an indication of the types of workloads that can run on the node, and not the host operating system. This is one of the culprits for frequent errors related to DaemonSets like kube-proxy being scheduled on Krustlet nodes. Unfortunately it does not completely eliminate this problem.

Known Issues/Missing Features

• Kubernetes networking support. The waSCC provider currently exposes the service on one of the node's ports, but there is nothing that updates Services or Endpoints. This is one of the major focuses of 0.7
• Unsupported workloads (such as those dropped automatically onto a node like kube-proxy) can enter an error loop. This is more of a nuisance that will cause some logging noise, but not impact the running of Krustlet. If you have any ideas or feedback, please check out #167
• Modifying a bare pod's image is not implemented. Nothing will error, but Krustlet will not restart the "container"
• TLS bootstrapping does not auto-renew certificates when they are close to expiry

What's next?

Our next anticipated version is 0.7.0 (although we will cut a 0.6.1 if necessary). Our main focus for 0.7 will be implementing networking and improving CSI support. During the next release cycle, we will also be moving out the waSCC provider to its own repo. Although we intended for this to occur during this release, we decided it would be better to do after we made these last changes to the provider trait. Full details will be in the 0.7 release notes.

Thanks

We want to express a huge thanks to all of those in the community who contributed to this release. We appreciate your efforts in making this project a success. As we mentioned before, there were a ton of doc updates from the community and we are very grateful.

Contributors to 0.6

• @kflansburg
• @itowlson
• @bacongobbler
• @thomastaylor312
• @DazWilkin
• @brooksmtownsend
• @jiayihu
• @willemneal

Installation

Download Krustlet 0.6.0:

• checksum file
• Linux amd64
• MacOS amd64
• Windows 64-bit

Check out our installation docs for information on how to install Krustlet.