Bonus Post II - Tilt Tuesdays: SSL/TLS CAA Records w/ Let's Encrypt on Digital Ocean Kubernetes and Cert Manager

Bonus Post II - Tilt Tuesdays: SSL/TLS CAA Records w/ Let's Encrypt on Digital Ocean Kubernetes and Cert Manager

TL;DR: I got HTTPS working.

This is where I left off on last post:

Certificate issuance in progress. Temporary certificate issued.

So after waiting for at least three hours, the message was still the same: "Certificate issuance in progress. Temporary certificate issued."

On the events section, kubectl describe certificate letsencrypt-prod showed:

Events:
  Type     Reason        Age                  From          Message
  ----     ------        ----                 ----          -------
  Normal   OrderCreated  43m (x5 over 4h45m)  cert-manager  Created Order resource "letsencrypt-prod-3181568815"
  Warning  FailedOrder   43m (x5 over 4h45m)  cert-manager  Order "letsencrypt-prod-3181568815" failed. Waiting 1h0m0s before retrying issuance.

(On the last post I managed to get at least a basic, "fake" SSL/TLS cert going on in Digital Ocean Kubernetes using Cert Manager, installed via Helm. That's a mouthful.)

So to troubleshoot that, I had to drill down into the Kubernetes cert-manager objects to get more event logs and see what popped out. Note that on the previous command describe certificate we got an Order object of cert-manager  Order "letsencrypt-prod-3181568815".

Let's drill down then:

kubectl describe order letsencrypt-prod-3181568815
...
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  54m   cert-manager  Created Challenge resource "letsencrypt-prod-3181568815-0" for domain "ghost.callbackinsanity.com"
  Normal  Created  54m   cert-manager  Created Challenge resource "letsencrypt-prod-3181568815-1" for domain "cms.callbackinsanity.com"
  
kubectl describe Challenge letsencrypt-prod-3181568815-1
...
Spec:
  Authz URL:  https://acme-v02.api.letsencrypt.org/acme/authz/D--KluRrbfdSkbGNjdQA4qEbXsDCwJObEd-SeiAl7Zc
  Config:
    Http 01:
      Ingress Class:  nginx
  Dns Name:           cms.callbackinsanity.com
  Issuer Ref:
    Kind:    ClusterIssuer
    Name:    letsencrypt-prod
  Key:       RUVbRV5GdNUagRzkae-EzXRwUOp5yEnPJWYPk1VWbsI.CSGwMo2cCcWSifHCCh281opPJHYnJXcUXBVhdA-g_Es
  Token:     RUVbRV5GdNUagRzkae-EzXRwUOp5yEnPJWYPk1VWbsI
  Type:      http-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/challenge/D--KluRrbfdSkbGNjdQA4qEbXsDCwJObEd-SeiAl7Zc/16690103193
  Wildcard:  false
Status:
  Presented:   false
  Processing:  false
  Reason:      Error accepting authorization: acme: authorization for identifier cms.callbackinsanity.com is invalid
  State:       invalid
Events:
  Type     Reason     Age   From          Message
  ----     ------     ----  ----          -------
  Normal   Started    54m   cert-manager  Challenge scheduled for processing
  Normal   Presented  54m   cert-manager  Presented challenge using http-01 challenge mechanism
  Warning  Failed     54m   cert-manager  Accepting challenge authorization failed: acme: authorization for identifier cms.callbackinsanity.com is invalid

Well, that popped up for sure. The last command in the chain, kubectl describe Challenge letsencrypt-prod-3181568815-1, gives us the error:

Accepting challenge authorization failed: acme: authorization for identifier cms.callbackinsanity.com is invalid

As part of the Challenge log, we get a challenge URL:

https://acme-v02.api.letsencrypt.org/acme/challenge/D--KluRrbfdSkbGNjdQA4qEbXsDCwJObEd-SeiAl7Zc/16690103193

Clicking on that challenge URL gives us a lot more details from the certificate issuer, Let's Encrypt, about what's going on:

{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:caa",
    "detail": "CAA record for cms.callbackinsanity.com prevents issuance",
    "status": 403
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/D--KluRrbfdSkbGNjdQA4qEbXsDCwJObEd-SeiAl7Zc/16690103193",
  "token": "RUVbRV5GdNUagRzkae-EzXRwUOp5yEnPJWYPk1VWbsI",
  "validationRecord": [
    {
      "url": "http://cms.callbackinsanity.com/.well-known/acme-challenge/RUVbRV5GdNUagRzkae-EzXRwUOp5yEnPJWYPk1VWbsI",
      "hostname": "cms.callbackinsanity.com",
      "port": "80",
      "addressesResolved": [
        "206.189.254.228"
      ],
      "addressUsed": "206.189.254.228"
    }
  ]
}

This pops up:

"detail": "CAA record for cms.callbackinsanity.com prevents issuance",

Digital Ocean has a good post about what CAA records are and how to use them:

https://www.digitalocean.com/docs/networking/dns/how-to/caa/

Back on the DO control panel, create a new CAA record like this:

And then run:

kubectl describe order letsencrypt-prod-3181568815

Now there's a completely different output than from a couple of hours ago:

Events:
  Type    Reason      Age    From          Message
  ----    ------      ----   ----          -------
  Normal  Created     3m48s  cert-manager  Created Challenge resource "letsencrypt-prod-3181568815-0" for domain "ghost.callbackinsanity.com"
  Normal  Created     3m48s  cert-manager  Created Challenge resource "letsencrypt-prod-3181568815-1" for domain "cms.callbackinsanity.com"
  Normal  OrderValid  3m24s  cert-manager  Order completed successfully

Order Completed Successfully !!!

That's all we needed. Now we get a fully ready and operational https:// bar for our backend domain.

Connection is secure, courtesy of Let's Encrypt

On a side note, now static resources such as images should be coming from a valid https:// address on the backend, meaning that the front-end (Gatsby), should not be displaying mixed content HTTPS errors anymore, which break the address bar secure lock, like this:

And... that's all for today folks!