In the first blog of this series, we took a looked at what Operator means in the Kubernetes context, as well as a working Hello World! example. In that example, we chose kubebuilder as our framework to implement the Operator pattern.
Nevertheless, pretty much like everything else in the cloud native world, there are many, many more other implementation of the Operator pattern in the market, alongside kubebuilder. In this post, we will take a look at some of the most prominent ones, and in turn, provide an analysis for you, in hope that it will help you choose the most suitable one for your case, should you find it necessary to create a customized controller in future.
Contenders for Kubernetes Operator
We hereby present the following six contenders to you in alphabetical orders:
Juju, formerly known as Charmed Operator Framework
kubebuilder
KUDO, a.k.a. Kubernetes Universal Declarative Operator
Metacontroller
Operator Framework, specifically, the Operator SDK
shell-operator
There are some other less significant players around that you are welcome to do your research on, but we will only discuss about these most prominent ones.
A table!
We have digested a table that will help in understanding and visualising features respectively.
NameOwner / Organisation behindStars on GitHubLanguage used to develop operatorsDifficulty to masterJujuCanonical1.9kPython, through Charmed Operator SDK (Formerly aptly named Python Operator Framework)MediumKubebuilderkubernetes-sig4.3kGolangMediumKUDOCNCF0.9kN/A, i.e. You don’t need to write code!EasyMetacontrollerNot obvious, but started by Google243 (here) or ~800 (here)BYOD, can be js, python, go, java, etc…HardOperator FrameworkRedHat5kGolang/Ansible/HelmMedium - Hardshell-operatorFlant1.4kBash script, as well as your trusty utilities, such as jq, awk, and so on.Easy (if you are a shell script guru!)
Remarks
Juju
Owned by Canonical, the company behind Ubuntu Linux, Juju is actually much more than an operator SDK. Dubbed as a Charmed Operator Framework, Juju consists of three logical parts: a Charmed Operator Lifecycle Manager (OLM), the Charmed Operator SDK, and the Charmhub.
Charmed OLM is a manager that adds an abstraction layer on top of the pre-existing Kubernetes cluster, baremetal machines, or micro/monolith services, and integrate them into models. Fully utilising model-driven operation, it’s ideal if your workload spans across vastly different architecture.
Charmed Operator SDK then defines what each model consists, and how would each model corresponds and translates to underlying Kubernetes resources. Written in Python, this is ideal if you are familiar with the language. After the model (a charm) is defined, you will be able to distribute it on Charmhub. You will also find a very extensive collection of charms on Charmhub, from observability, to data, to identity, and a lot more.
Pros:
Charmed OLM is quite capable and provides a quite intuitive way of mapping underlying resources to models, and in turn, charms. Since it’s quite standarised, it’s very easy to distribute your charms on the Charmhub.
Cons:
There doesn’t seem to be a way to use Charmed Operator SDK without using Charmed OLM. This added layer of abstraction can add quite a bit of complexity if your workload is fairly homogeneous. Installing Charmed OLM (itself as a controller) on your cluster can add resource overhead too.
kubebuilder
Kubebuilder is owned by one of Kubernetes SIGs (Special Interest Group). Written in Golang, kubebuilder aims to provide a complete solution, which includes code generation capability to scaffold your CRDs, a control loop to reconcile changes, a testing framework to test your CRDs/controllers against a cluster, and a scaffold Dockerfile to publish this controller. It uses controller-runtime, which is exactly on which the native controllers on Kubernetes are written.
The workflow when using kubebuilder is as follows:
Generate a project with the kubebuilder CLI.
Define a new API (in fact a CRD!)
Implementing a controller, in which the reconcilation logic is provided. kubebuilder provides a set of useful contexts and handles, so you can be fully equipped to make adjustments to the resources in order to meet what was defined in the API.
Implementing a webhook controller (if necessary)
Implementing testing strategy.
Profit!
Pros:
Kubebuilder is a Kubernetes SIG, which means it can be considered directly from Kubernetes. It has the most native feeling amongst contenders, and it’s probably the most well-known and mostly used operator framework. As mentioned before, since Kubernetes itself is written on top of controller-runtime, its stability and feature sets are timeproof and trialed.
Cons:
You’d better know Golang.
KUDO
KUDO is a CNCF Sandbox project. KUDO stands for Kubernetes Universal Declarative Operator. It differs from other implementations in its declarative approach, pretty much exactly like how you would write a GitHub Action or a Travis pipeline. No code is needed!
When you define a KUDO, you start with a task. A task can be further broken down into native Kubernetes resources such as Deployments or ConfigMaps. After the fulfilment of said task, an arbitrary set of Kubernetes resources can be created, updated or deleted.
You can also have plans to organise individual tasks into meaningful workflows.
Pros:
No code is needed!
Cons:
Because no code is needed, it’s sometimes hard to declaratively nominate a very complex workflow and thus breaking each pieces into rudimentary Kubernetes resources. This can bring about considerate boilerplate manifests. You also can’t define anything that couldn’t be represented by Kubernetes resources.
Metacontroller
Metacontroller was developed by Google, and hosted at GoogleCloudPlatform/metacontroller. However, it is recently changed to community-owned and migrated to metacontroller/metacontroller.
A very useful feature of Metacontroller is BYOD-ing your most familiar technology stack. Basically, you provide a CRD, a CompositeController or a DecoratorController, depending on whether your CRD is a collection of Kubernetes resources or a modification to a existing resource. Inside the controller resource, you define the scope of your CRD and a webhook address for Metacontroller to communicate to.
You can start implementing the webhook based on the specification provided by Metacontroller, in whatever the technology that sails your boat, be it Javascript, Golang, Java, or C++, as long as it correctly reports to the Metacontroller accordingly.
Pros:
Technology stack BYOD.
Cons:
Technology stack BYOD. Because Metacontroller doesn’t do as much as other contenders on market with its opinionated approach, you are on your own to implement the reconcilation logic and the testing logic, which could be pretty tricky. Your implementation is likely non-standard, since this really isn’t standarised by design.
Operator Framework/SDK
Operator Framework differs to the former name of Juju, Charmed Operator Framework by only a single word, and it shares a similar ideology. It was owned and maintained by RedHat, although the ownership is hidden in a shroud of mystery, since it’s donated to CNCF recently. In fact, you can’t even find a single occurence of Red Hat on its site.
An Operator Framework has three components: OLM, SDK, and OperatorHub. Now you see why I think of Juju. Thankfully, you are free to use Operator SDK without using its OLM.
The Operator SDK actually closely resembles kubebuilder, but you can define the reconcilation with Ansible or a Helm Chart, as well as through Golang.
Pros:
If GitHub stars correlates to popularity, Operator Framework is apparently the most popular one among all. Also, if you already have pre-existing helm charts or ansible playbook to define a set of resources, it is quite easy to migrate to Operator Framework.
Cons:
Although you can’t find Red Hat on operatorframework.io anymore, but the influence is uncanny. Stay away from it if you don’t like how Red Hat does things. This Twitter thread puts it pretty nicely.
shell-operator
The last one is an odd one. As Flant puts it,
Think of it as an operator-sdk, but for scripts.
Referring to a snippet of scripts provided here:
#!/usr/bin/env bash
if [[ $1 == "--config" ]] ; then
cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: v1
kind: Pod
executeHookOnEvent: ["Added"]
EOF
else
podName=$(jq -r .[0].object.metadata.name $BINDING_CONTEXT_PATH)
echo "Pod '${podName}' added"
fiYou basically provide hooks like the one above to react on various events in the cluster. The contract between hooks and shell-operator is quite simple: it should emit a YAML/JSON resource definition when called with --config. That piece of configuration should points to the namespace, or the particular piece of resource you’d like to watch on. And the else branch should be used to define what reconcilation logic should be performed when the match on the --config branch is successful.
In the example provided above, when a pod is added, shell-controller will run echo "Pod '${podName}' added".
Pros:
Bash script gurus will feel quite at home. You can also bring in your trusty linux toolkit to the party, such as yq, awk, sed, and much so on.
Cons:
Testing support is glaringly absent. I know, most bash script authors don’t test their scripts in the same way software engineers test their code, but I digress.
Conclusion
We have walked through six of the most common operator frameworks in market, and each of them is very opinionated. Some of them try to make you write as few lines of code as possible, whereas some of them try to stay out of your way so you’ll be able to control everything, albeit with a lot of code. Hopefully, this blog post will give you a direction of which specific implementation you should be looking at, and you will be able to make your own educated decision.
In the next blog post in this series, we will be looking at a much more complex and real-life use case that can benefit greatly from employing an operator framework, as well as providing a step-by-step tutorial on implementing said operator with kubebuilder. Stay tuned.
Innablr is a Kubernetes Certified Service Provider and leading consultancy for cloud native, Kubernetes, and serverless technologies. Frequently championing community events, delivering thought leadership and leading practices, Innablr is recognised in the Australian market as one of the most experienced providers of Kubernetes solutions.
Continuing our successful approach of building repeatable and extensible frameworks, Innablr has built a blueprint for Google Cloud and Amazon Web Services Kubernetes deployment whether it is Google Kubernetes Engine (GKE) or Elastic Kubernetes Service (EKS).
To learn more about how we’ve been helping businesses innovate with Kubernetes, see our Kubernetes Certified Solution Provider page.
Chuning Song, Engineer @ Innablr
Prateek Nayak, CTO @ Innablr