Are you finding that you and your fellow engineers are losing sleep over certificate management? It is 2022 and we still wake up with a headache because of this.
It’s Friday morning, you’ve just picked up your latte from the local cafe. You log into your emails to find you have been spammed by your PKI team that all of your workload certificates have expired. It’s now a scramble for you and your team to generate and upload new certificates to your workloads. We’ve all been there before. Do you find yourself asking why this is the case?
Why are certificates so important?
Digital certificates are an important part of internet security. They are used to authenticate the identity of websites, users, organisations and applications. You might remember in the early 2000’s when the DotCom boom first began, websites were popping up every day!
It was a common practice to serve your website using un-encrypted HTTP protocol (port 80). As more businesses started to leverage the power of internet to serve their customer, malicious activities like man-in-the-middle attacks, packet sniffing and identity spoofing became popular. This created a need to use encrypted communication channels when serving the customers via internet.
Fast forward to today, over 95% of websites now use HTTPS protocl (port 443). HTTPS uses a digital certificate with a unique public and private key pair. This key pair encrypts and verifies communication between the client and server.
Why is Certificate Management such a pain?
Encrypting messages between client and server sounds very straight forward and should not cause much headache. Unfortunately many engineering teams don’t fully understand how certificates work, how to manage them and how to create new certificates for their workloads. There is always that one person in the team that is the go to “certificate engineer”. Normally the process involves going to a portal → requesting certificate → storing private key material safely → uploading it to your workloads. This manual process of managing the lifecycle of certificates can lead to very poor developer experience, poor visibility and tracking of certificate. This can unfortunately in some cases lead outages due to expired certificates.
Certificates are rarely tracked in large workplaces
In my experience I have broadly observed three (not best) practises.
Certificate Lifecycle tracking via email - You receive an automated email from your PKI team that your certificates are expiring.
Someone has manually recorded down the certificates and expiry dates in a spreadsheet or confluence page somewhere.
That one person in your team just “knows” when they will expire.
Multiple certificates expiring all at once.
If you are lucky you may only have one certificate expiring at once, but almost all the time this is never the case. Most of the time the expiries don’t line up, as engineers we end up with multiple emails and messages about upcoming certificate expiries.
Large organisation are normally running lots of workloads with each of these require a certificate. This can lead to organisations needing to monitor, rotate and load 100s of certificates into their workloads.
Process to renew the certificates
Most large companies have an internal team that manages certificates. This process can require filling out multiple forms and waiting days for responses.
Deploying new certificates
Manually deploying certificates in a distributed environment where you have 100s of workloads running, can be a very cumbersome process in itself. This can become even more complicated when you are running your workloads in kubernetes where containers are ephemeral in nature.
What if a certificate is missed?
It some how always ends up happening! Oh no, We’ve missed a certificate!
Your workloads become “Not Secure” with some browsers completely denying users to access your pages.
Networking and Certificate management in Kubernetes
When running your workloads in kubernetes there are a few things required to give external traffic access to them. We can achieve this by using ingress, routing rules and services.
Ingress is used to exposes HTTP and HTTPS routes from outside the cluster to internal services in the cluster.
Traffic routing rules are defined on the ingress kind resource
Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.
Here we can define our TLS configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tls-example-ingress
spec:
tls:
- hosts:
- https-example.foo.com
secretName: testsecret-tls
rules:
- host: https-example.foo.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service1
port:
number: 80What is cert-manager?
Cert-manager is a kubernetes CRD (Custom resources) that expands the kube-api.
apiVersion: cert-manager.io/v1
This api can handle a number of different kinds (Kubernetes objects) ClusterIssuer, Issuer, Certificate, CertificateRequests
Cert-manager is made up of several components. These components are listed below
There are 2 types of issuers in cert-manager Issuer or ClusterIssuer
They represent certificate authorities available to you that are able to generate signed certificates.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: ca-issuer
namespace: mesh-system
spec:
ca:
secretName: ca-key-pairA CertificateRequest is a namespaced scoped resource in cert-manager that requests X.509 certificate from the Issuer. A successful CertificateRequest will generate a secret containing the certificate.
apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
name: my-ca-cr
spec:
request:
isCA: false
usages:
- signing
- digital signature
- server auth
duration: 2160h
issuerRef:
name: ca-issuer
kind: Issuer
group: cert-manager.ioA certificate is namespace scoped and is used to create a CertificateRequest. Cert-manager tracks this object which will be renewed and kept to automatically update the certificate.
Here we can define the secretName, dnsNames and issuer
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: example-crt
spec:
secretName: example-crt-secret
dnsNames:
- example.com
- foo.example.com
issuerRef:
name: example-ca-issuer
kind: Issuer
group: cert-manager.ioHow does cert-manager help?
As you can see from the sequence diagram below cert-manager takes away a lot of the manual steps an engineer is involved in when updating a certificate. Which removes any margin for mistakes that were listed above and most importantly reduces toil.
Conclusion
As the development and operation of software continues to be revolutionised by containers and serverless computing, cloud native platforms such as Kubernetes and Knative are increasingly being deployed across organisations.
While Kubernetes has proven to be the technology winner, it is still a new technology that has yet to enjoy the same level of deployment as other mission critical solutions organisations are familiar with. A complete and integrated Kubernetes platform that encompasses certificate management, CI/CD, security scanning, networking, storage, and monitoring is still a significant project for most organisations that simply do not have the skills and experience to undertake.
Innablr is a leading services provider for cloud native, and Kubernetes technologies. Frequently leading community events, delivering thought leadership and leading practices, Innablr is recognised in the Australian market as one of the most experienced providers of Kubernetes solutions.
In our experience building platforms powered by kubernetes we have found that cert-manager allows for easy installation, renewal and management of certificates in the cluster. By using cert-manager we can define our issuer, request our certificates and update them in our ingress configuration without any manual intervention.
Cert manager removes the need for any manual process around certificate lifecycle management. Lastly, Cert-manager allows us to expand the utilisation of issuers through plugins. Stay tuned for Part 2 where I dive deep into how to configure Cert-manager using google CAS and google CAS issuer plugin.