MicroK8s - Setup an nginx ingress with cert-manager

(Updated: )
Backend
, , ,

Sorry, what? microK8s, Ingress, cert-manager? 🤔

Disclaimer

I won’t explain every detail in this article, it’s mainly for my own note taking. If you read this article, I suppose you already know how to setup your DNS, setup some basic HTTP routes, deploy some deployments, services on a K8s cluster.

Why I use Kubernetes

Kubernetes (K8s) is a container orchestration tool, many cloud providers like Google, Amazon, DigitalOcean have their own proprietary implementation of Kubernetes and allow you to setup a Kubernetes cluster with a few clicks. A Kubernetes cluster offers many advantages, some of them target big companies or critical apps with High Availability, Replication, and so on. For my own projects, I’m more interested in using a tool that allows me to automatically orchestrate my own containers and automate their deployment.

Why I decided to try MicroK8s

I tried the cloud solutions, and loved them. One day, I had to update my cluster with the update process of DigitalOcean (automatic) and… BOOM 💥 all of my sites went down, the load balancer was down. I had to setup the load balancer again and I wasn’t happy with the idea of paying for the High Availability stuff, which wasn’t highly available AT ALL.

I found an article about MicroK8s: a lightweight K8s implementation that you can install almost anywhere. Ok, it means you need to install it and you don’t have the master node that comes for free. BUT, you can install it anywhere, and to be honest, I don’t think I need all of the DigitalOcean High Availability promises for my small apps.

So, I decided to try managing my own kubernetes cluster on a Vultr VPS instance. Their servers have better performances for the same price and people say their support is great (compared to the bad support at DO).

Ingress and cert-manager

An ingress controller lets you route your URLs to different services in your kubernetes cluster. It generally sits behind a Load Balancer (which sometimes manages your SSL certificates). In this article, I’ll install MicroK8s with some services and route some urls to them. I’ll use Cert-manager to manage the SSL certificate (with let’s encrypt).

Setting up MicroK8s

Install MicroK8s with snap

The instructions are available from the official website: https://microk8s.io/#get-started

I’ll use the guide for Ubuntu as I’m installing it on fresh new DigitalOcean droplet.

1
2
sudo snap install microk8s --classic --channel=1.18/stable
sudo microk8s status --wait-ready

Optional: get remote access via a SSH tunnel

SERVER
1
2
# Export the custom config
sudo microk8s kubectl config view --raw > $HOME/.kube/config
YOUR MACHINE
1
2
3
4
5
6
7
8
# Set a variable to your server IP
SERVERIP=xxx.xxx.xxx.xxx

# Copy the ~/.kube/config from the Server to your machine
scp root@$SERVERIP:~/.kube/config ~/.kube/config

# Open an SSH tunnel (everytime you need to use kubectl)
ssh -N -L localhost:16443:localhost:16443 root@$SERVERIP

Optional (2): get remote access without a SSH tunnel

⚠️ This method exposes the 16443 API port to the web, you might not want to expose it for production use cases ⚠️

SERVER
1
2
# Export the custom config
sudo ufw allow 16443
YOUR MACHINE
1
2
3
4
5
6
# Set a variable to your server IP
SERVERIP=xxx.xxx.xxx.xxx

# Copy the ~/.kube/config from the Server to your machine
scp root@$SERVERIP:~/.kube/config ~/.kube/config
sed -ie "s/127.0.0.1/$SERVERIP/g" ~/.kube/config

Test deployment

For the purpose of this article, I’ll deploy a basic nginx deployment and expose it.
I’ll also create another deployment/service called hello-deployment with the same method.

1
2
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
kubectl expose deployment nginx-deployment --port 80

Then type kubectl get all should display the following status:

1
2
3
4
5
6
7
8
9
10
11
12
13
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-6b474476c4-8vr85   1/1     Running   0          3m2s
pod/nginx-deployment-6b474476c4-skf77   1/1     Running   0          3m2s

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes         ClusterIP   10.152.183.1     <none>        443/TCP   6m53s
service/nginx-deployment   ClusterIP   10.152.183.136   <none>        80/TCP    15s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   2/2     2            2           3m2s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-6b474476c4   2         2         2       3m2s

Ingress

Installing an nginx ingress controller with microK8s is easy, just type the following command to enable the corresponding plugin:

1
microk8s.enable ingress

Then type kubectl get all --namespace ingress should return the following status, with a new daemonset and pod for the ingress controller.

1
2
3
4
5
6
NAME                                          READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-microk8s-controller-rjh6b   1/1     Running   0          9m22s

NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>          9m22s

Now, let’s create a basic ingress definition, this will route ilightshow.app/ to the nginx-deployment, and route ilightshow.app/hello to the hello-deployment. Please note that you need to setup your DNS records for your domain name.

ingress.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: default-ingress
spec:
  tls:
  - hosts:
    - ilightshow.app
    secretName: default-tls-secret
  rules:
  - host: ilightshow.app
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-deployment
          servicePort: 80
      - path: /hello
        backend:
          serviceName: hello-deployment
          servicePort: 80

Cert-manager

Setup your DNS

Don’t forget to setup the DNS record for your domain name, it needs to point to your server, otherwise, you won’t be able to reach the deployments by typing the URL (for me: http://ilightshow.app). AND, the Let’s Encrypt servers won’t be able to reach the cert-manager challenge (If you don’t understand this, please read this article).

Install cert-manager

First, run microk8s.inspect on the server, make sure you don’t have any warning. For instance, I had the following warning:

WARNING: IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT

This part is based from this tutorial.

1
2
3
4
5
# Create a new namespace for the cert-manager
kubectl create namespace cert-manager

# Apply the official yaml file (change the version to the latest release)
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml

Deploy a staging and prod issuer

Create the following files and apply them with kubectl apply -f

staging-issuer.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
 name: letsencrypt-staging
 namespace: cert-manager
spec:
 acme:
   # The ACME server URL
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   # Email address used for ACME registration
   email: <your-email-here>
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-staging
   # Enable the HTTP-01 challenge provider
   solvers:
   - http01:
       ingress:
         class:  nginx
production-issuer.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: <your-email-here>
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

Setup the ingress to work with cert-manager

Just add the following annotations, it will ask the cluster-issuer to issue a certificate.

ingress.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: default-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"

    # Add the following line (staging first for testing, then apply the prod issuer)
    cert-manager.io/cluster-issuer: "letsencrypt-staging" # "letsencrypt-prod" 
spec:
  tls:
  - hosts:
    - ilightshow.app
    secretName: default-tls-secret
  rules:
  - host: ilightshow.app
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-deployment
          servicePort: 80
      - path: /hello
        backend:
          serviceName: hello-deployment
          servicePort: 80

Then, check that the certificate is ready with kubectl get certificate. If everything worked fine, you can change the issuer to the prod issuer.