In my last post, I laid out why I’m building an AI Delivery Workflow instead of relying on a pile of disconnected AI tools. This article is the first practical step in that direction: getting OpenClaw running on Kubernetes.

For me, OpenClaw is not interesting as just another chat interface. It becomes useful when it lives inside a real delivery stack alongside GitHub, coding agents, and ArgoCD. That is why I want it running as part of real infrastructure, not sitting off to the side as a disconnected experiment.

This guide is for someone who already knows their way around a Kubernetes cluster and wants a clean, repeatable OpenClaw install. I’m assuming you already have a working cluster, basic kubectl access, Helm installed, and a general sense of how you handle storage, secrets, and service access in your own environment.

This is not a Kubernetes primer. The goal here is simpler than that: get OpenClaw running in a way that is understandable, debuggable, and easy to repeat.

What this guide covers

This guide walks through:

• getting a working provider token
• creating the Kubernetes secret
• choosing a storage path that matches your cluster
• creating a working Helm values file
• installing OpenClaw
• verifying that the deployment actually works

The goal is that someone with a Kubernetes cluster can come away with a running OpenClaw install.

Prerequisites

Before you begin, you should already have:

• a working Kubernetes cluster with kubectl access
• Helm 3 installed
• persistent storage available for workloads in your cluster
• a model provider you plan to use

For storage, I would treat one of these as a prerequisite:

• a default StorageClass that can dynamically provision a PVC for OpenClaw, or
• a PV/PVC arrangement you have already prepared for this cluster

I’m intentionally not turning this into a distro-specific storage setup guide. That varies too much across managed Kubernetes, bare metal, k3s, Talos, and everything in between. This article assumes you already know how storage is handled in your environment.

I’m using GitHub Copilot as the provider example in this guide because its authentication flow is the least obvious part of the setup. If you plan to use another provider, the overall install pattern is similar, but the secret values and model configuration will change.

Step 1: Create a namespace

Create a dedicated namespace for OpenClaw:

kubectl create namespace openclaw

Step 2: Get a GitHub Copilot OAuth token

If you are using GitHub Copilot, this is the part most likely to trip you up.

A GitHub Personal Access Token with the copilot scope is not enough for model access here. You need an OAuth token from GitHub’s device flow.

Request a device code:

curl -s -X POST https://github.com/login/device/code \
  -H "Accept: application/json" \
  -d "client_id=Iv1.b507a08c87ecfe98&scope=read:user"

This returns a device_code, user_code, and verification_uri.

Open https://github.com/login/device in your browser and enter the user_code.

Then poll for the access token:

curl -s -X POST https://github.com/login/oauth/access_token \
  -H "Accept: application/json" \
  -d "client_id=Iv1.b507a08c87ecfe98&device_code=<DEVICE_CODE>&grant_type=urn:ietf:params:oauth:grant-type:device_code"

Repeat that command until the response includes an access_token beginning with ghu_.

To verify the token works:

curl -s -o /dev/null -w "%{http_code}" \
  https://api.github.com/copilot_internal/v2/token \
  -H "Authorization: token ghu_YOUR_TOKEN"

A 200 response confirms the token is valid.

Step 3: Discover available model IDs

The model names you see in a UI do not always match the exact model IDs you need in configuration.

You can query the available models directly:

# Get a session token
COPILOT_SESSION=$(curl -s https://api.github.com/copilot_internal/v2/token \
  -H "Authorization: token ghu_YOUR_TOKEN")

# Extract the API endpoint and token
API_URL=$(echo "$COPILOT_SESSION" | jq -r '.endpoints.api')
SESSION_TOKEN=$(echo "$COPILOT_SESSION" | jq -r '.token')

# List models
curl -s "$API_URL/models" \
  -H "Authorization: Bearer $SESSION_TOKEN" \
  -H "Copilot-Integration-Id: vscode-chat" | jq '.data[].id'

When you configure OpenClaw, prefix the model ID with github-copilot/.

For example:

github-copilot/gpt-5.4

Step 4: Create the Kubernetes secrets

Create one secret for your provider token:

kubectl create secret generic openclaw-provider-secret -n openclaw \
  --from-literal=GITHUB_TOKEN='ghu_YOUR_OAUTH_TOKEN'

Then create a second secret for the OpenClaw gateway password:

kubectl create secret generic openclaw-auth-secret -n openclaw \
  --from-literal=OPENCLAW_GATEWAY_PASSWORD='YOUR_GATEWAY_PASSWORD'

In this example:

• GITHUB_TOKEN is the GitHub Copilot OAuth token from the device flow
• OPENCLAW_GATEWAY_PASSWORD is the password you will use to sign in to the OpenClaw control UI

If you are using another model provider, substitute the correct provider credential in the provider secret.

Step 5: Confirm your storage path

The key storage requirement is simple: OpenClaw needs writable persistent state under /home/node/.openclaw.

Treat storage as a prerequisite here, not a side quest. The practical question is whether your cluster can give OpenClaw a writable home directory cleanly and predictably.

For chart version openclaw/openclaw 1.5.10, the detail that matters most is that /home/node/.openclaw must be mounted writable.

If you miss that, the pod can crash during init because it tries to write /home/node/.openclaw/openclaw.json onto the read-only root filesystem.

Option A: Non-persistent smoke test

Use this if you want the fastest proven path to a working install.

This was the cleanest path I validated on lernaean-dev. The trick is that disabling app-template.persistence.data by itself is not enough for this chart version. You also need to mount a writable emptyDir at both /tmp and /home/node/.openclaw.

If you choose this path, you do not need to create a PV or PVC in this step.

Option B: Persistent storage

Use this if you want OpenClaw to keep its local state between restarts.

The exact implementation depends on your cluster. If your cluster has a StorageClass with dynamic provisioning, the storage driver handles directory creation and ownership — no extra steps needed. If you are using a static hostPath PV with DirectoryOrCreate, Kubernetes creates the directory as root:root on first use. Since OpenClaw runs as uid 1000, the init-config container will fail with Permission denied and the pod will crash-loop. In that case, add the fix-permissions init container shown in the values snippet below.

First check whether your cluster already has a storage class:

kubectl get storageclass

If your cluster has a default storage class, or a storage class you want to use, Helm may be able to create the PVC for you during install.

If your cluster does not have a usable storage class, create storage first. For example:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: openclaw
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  claimRef:
    namespace: openclaw
    name: openclaw
  hostPath:
    path: /var/openclaw-data
    type: DirectoryOrCreate

Apply it:

kubectl apply -f openclaw-storage.yaml

Then verify the PVC is ready:

kubectl get pvc openclaw -n openclaw

The success condition is simple: the PVC should show Bound.

Two practical caveats from testing:

• this chart version expected the PVC name to be openclaw when persistence stayed enabled
• existingClaim under app-template.persistence.data works, but requires --skip-schema-validation at install time
• if using a static hostPath PV with DirectoryOrCreate and a fresh directory, include the fix-permissions init container in the values snippet. If your storage class or an existing directory is already owned by uid 1000, you can omit it.

Step 6: Create the values file

Create a file named openclaw-values.yaml.

The goal of this values file is to keep the deployment explicit, simple, and easy to debug. Start with this shared base config for either storage path:

app-template:
  configMode: merge
  controllers:
    main:
      containers:
        chromium:
          enabled: false
        main:
          envFrom:
            - secretRef:
                name: openclaw-provider-secret
            - secretRef:
                name: openclaw-auth-secret
  configMaps:
    config:
      enabled: true
      data:
        openclaw.json: |
          {
            "gateway": {
              "port": 18789,
              "mode": "local",
              "auth": {
                "mode": "password"
              },
              "controlUi": {
                "dangerouslyAllowHostHeaderOriginFallback": true
              }
            },
            "browser": {
              "enabled": false
            },
            "agents": {
              "defaults": {
                "workspace": "/home/node/.openclaw/workspace",
                "model": {
                  "primary": "github-copilot/gpt-5.4"
                },
                "userTimezone": "UTC",
                "timeoutSeconds": 600,
                "maxConcurrent": 1
              },
              "list": [
                {
                  "id": "main",
                  "default": true,
                  "identity": {
                    "name": "OpenClaw",
                    "emoji": "🦞"
                  }
                }
              ]
            },
            "session": {
              "scope": "per-sender",
              "store": "/home/node/.openclaw/sessions",
              "reset": {
                "mode": "idle",
                "idleMinutes": 60
              }
            },
            "logging": {
              "level": "info",
              "consoleLevel": "info",
              "consoleStyle": "compact",
              "redactSensitive": "tools"
            },
            "tools": {
              "profile": "full",
              "web": {
                "search": {
                  "enabled": false
                },
                "fetch": {
                  "enabled": true
                }
              }
            }
          }

If you chose the non-persistent smoke-test path, use this storage section:

  persistence:
    data:
      enabled: false
    tmp:
      enabled: true
      type: emptyDir
      advancedMounts:
        main:
          init-config:
            - path: /tmp
            - path: /home/node/.openclaw
          init-skills:
            - path: /tmp
            - path: /home/node/.openclaw
          main:
            - path: /tmp
            - path: /home/node/.openclaw

If you chose the persistent path, use this storage section instead:

  controllers:
    main:
      initContainers:
        fix-permissions:
          image:
            repository: busybox
            tag: "1.36"
          command:
            - sh
            - -c
            - |
              set -eu
              mkdir -p /home/node/.openclaw
              chown -R 1000:1000 /home/node/.openclaw
          securityContext:
            runAsUser: 0
            runAsGroup: 0
            runAsNonRoot: false
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              drop:
                - ALL
              add:
                - CHOWN
  persistence:
    data:
      enabled: true
      type: persistentVolumeClaim
      accessMode: ReadWriteOnce
      size: 5Gi
      advancedMounts:
        main:
          fix-permissions:
            - path: /home/node/.openclaw
          init-config:
            - path: /home/node/.openclaw
          init-skills:
            - path: /home/node/.openclaw
          main:
            - path: /home/node/.openclaw
    tmp:
      enabled: true
      type: emptyDir

The tmp emptyDir is still needed even with persistent storage. OpenClaw writes temporary files outside of /home/node/.openclaw during init, so the root filesystem must have a writable /tmp even when the PVC is handling persistent state.

A few practical notes:

• OPENCLAW_GATEWAY_PASSWORD comes from openclaw-auth-secret
• the browser sidecar is disabled here to keep the install lighter
• if you use another model provider, update both the credentials and the configured model name
• the smoke-test path is ephemeral; pod recreation will discard local state

Step 7: Install OpenClaw

Add the Helm repo:

helm repo add openclaw https://serhanekicii.github.io/openclaw-helm
helm repo update

Then install the chart:

helm install openclaw openclaw/openclaw -n openclaw \
  -f openclaw-values.yaml \
  --skip-schema-validation

Step 8: Wait for the deployment

Wait for the rollout:

kubectl rollout status deployment openclaw -n openclaw --timeout=600s

You can also check the pods directly:

kubectl get pods -n openclaw

You want to see the OpenClaw pod reach Running.

If it does not, inspect the pod and recent events:

kubectl describe pod -n openclaw <pod-name>
kubectl get events -n openclaw --sort-by=.lastTimestamp

Step 9: Access the control UI

The simplest way to test a fresh install is with port forwarding:

kubectl port-forward -n openclaw deployment/openclaw 18789:18789

Then open:

http://localhost:18789

Sign in with the password you stored in the openclaw-auth-secret secret.

You can expose OpenClaw through ingress or a load balancer later if you want always-on remote access. For the first install, I would treat that as an operational choice, not a prerequisite. Port forwarding is the fastest way to verify that everything works.

Quick troubleshooting notes

If the install does not work the first time, these are the first places I would check:

1. Provider token: if you are using GitHub Copilot, make sure you used the OAuth device flow token, not a PAT.
2. Model ID: query the API instead of guessing the exact model name.
3. Writable home directory: for chart 1.5.10, make sure /home/node/.openclaw is mounted writable.
4. Storage: if your PVC stays pending, check whether your cluster has a default storage class, whether you need to set one explicitly, or whether you need to create a PV first.
5. Chart schema: the wrapper chart schema can reject values that look reasonable. If using existingClaim under app-template.persistence.data, add --skip-schema-validation at install time.
6. Cold start time: initial image pulls can take a while, especially on smaller or ARM64 systems.
7. Token expiry: GitHub Copilot OAuth tokens can expire and may need to be refreshed later.

Why this matters in the bigger workflow

Getting OpenClaw running is not the whole point. The point is to make it one reliable part of a larger delivery system.

In the workflow I’m building, GitHub holds the work queue, coding agents help implement changes, ArgoCD handles deployment, and OpenClaw acts as the operating layer tying those pieces together. Infrastructure only matters if it supports a repeatable way to move work from idea to production. That is the role I want OpenClaw to play.

This guide covers the first part of that path: getting OpenClaw running cleanly so the rest of the workflow has a solid base.

Conclusion

OpenClaw makes sense on Kubernetes when you want your AI operating layer to live alongside the rest of your delivery infrastructure instead of floating outside it.

Once provider credentials, storage, and chart configuration are sorted out, the deployment itself is fairly straightforward. The real friction is usually around authentication, model naming, and choosing the right storage path for your cluster.

If you are trying to move from scattered AI tooling toward a usable AI Delivery Workflow, this is where the system starts to become real. A clean OpenClaw install does not finish the job, but it gives you a solid operating base for everything that comes next.

AI tooling has gotten good enough to be useful, but for most technical builders it still does not feel like a real operating system.

You can open ChatGPT, Claude, Copilot, Codex, or another coding agent and get something helpful. You can ask for code, explanations, refactors, commands, and plans. You can connect pieces of your stack. You can automate parts of your workflow.

But most of the time, it still feels scattered.

That is the problem I care about right now.

I am not especially interested in AI as a toy, a gimmick, or a source of endless screenshots. I am interested in whether it can become a practical delivery layer for real work.

I want something that helps move a project from idea to issue, from issue to implementation, from implementation to review, and from review to deployment without turning the whole process into chaos.

That is what I mean by an AI Delivery Workflow.

Not “AI does everything”

When people hear language like this, it is easy to imagine a fully autonomous setup that replaces judgment, skips review, and magically runs the whole software lifecycle on its own.

That is not what I am building.

I do not think the goal is to hand over the keys and hope for the best. I think the goal is to build a workflow where AI is genuinely useful inside a system that still has structure, boundaries, and human decision points.

In practice, that means:

• issues still matter
• review still matters
• deployment boundaries still matter
• human approval still matters

The value is not that AI replaces the workflow. The value is that AI becomes productive inside the workflow.

The problem with the current tool landscape

Right now, there are a lot of individually impressive tools:

• coding agents that can implement real changes
• systems like OpenClaw that can act more like an operating layer than a chat box
• GitHub issues and pull requests that already provide a clean work queue
• GitOps tools like ArgoCD that create a sane deployment path

But if you are a technical builder, platform engineer, founder, or operator trying to actually use these tools together, the path is still fuzzy.

You can usually get one piece working. You can often get two or three pieces working.

What is harder is getting the overall system to feel coherent.

That is where most of the friction lives:

• What tool should do what?
• Where should work begin?
• How do you keep agents from becoming disconnected chat assistants?
• How do you make GitHub the queue instead of a side effect?
• How do you preserve review and deployment discipline?
• How do you make the whole thing feel usable instead of fragile?

That is the gap I want to close.

What I’m actually building

I am working toward a practical operating model built around a few core ideas:

• OpenClaw as the coordinating layer
• GitHub issues as the work queue
• coding agents as implementation helpers, not independent bosses
• pull requests and review as quality and control points
• ArgoCD and Kubernetes as the deployment path

That stack will not be right for everyone. It is opinionated. It assumes some technical comfort. It is not a beginner course and it is not trying to be one.

But for the kind of builder I care about here, it solves a real problem: how to turn a pile of promising AI and infrastructure tools into a workflow you can actually trust.

Why this matters to me

I do not want to spend my time bouncing between disconnected tools, each of which is impressive in isolation but awkward in combination.

I want a workflow that helps me do delivery work with more leverage and more clarity.

I want to be able to:

• capture work cleanly
• delegate parts of implementation to agents
• review changes with clear boundaries
• ship through a real deployment path
• understand what the system is doing and why

That last point matters a lot.

I am not trying to build a black box. I am trying to build a workflow that increases confidence.

What this series will cover

This is the frame for a broader set of writing and operator material I plan to publish.

Some of it will stay public. Some of the more structured playbooks, checklists, and deeper workflow material will eventually live as paid member content. But the goal is the same across all of it: make this stack more understandable, more usable, and more practical.

The first implementation article in that series will be about getting OpenClaw running on Kubernetes in a way that fits this broader workflow direction.

That matters because I do not want the install guide to feel like an isolated technical note. I want it to sit inside a more complete point of view:

AI becomes much more useful when it is part of a delivery workflow instead of just another chat window.

What this is not

To keep this grounded, it is worth saying what I am not trying to do.

• I am not promising full automation.
• I am not saying AI replaces engineering judgment.
• I am not building a generic prompt guide.
• I am not treating Kubernetes, GitHub, and agent tooling like magic.
• I am not trying to create a hypey “one weird trick” system.

I am trying to create a workflow that a technically capable person can actually operate with confidence.

The real goal

If this work goes well, the outcome is not just “I have OpenClaw installed.”

The outcome is something better than that:

• I understand the stack
• I know what each part is for
• I can move work through it with less friction
• I trust it enough to use it on real projects

That is the standard I care about.

That is why I am building an AI Delivery Workflow.

And that is the direction this series is going next.

Now that we have a running Kubernetes cluster, we need to make it a bit more usable. For my home cluster, I wanted to ensure it was fault-tolerant, so I decided to implement a load balancer, an ingress controller, and an application manager. For these components, I chose MetalLB, Nginx, and Kubeapps.

In this guide, we will use Helm to install all the core applications. We’ll start with MetalLB for load balancing, then move on to Nginx as the ingress controller, and finally install Kubeapps. Installing Kubeapps last ensures that we have a UI available once everything else is set up.

Another suggestion I would make is to get a UI-based kubectl IDE like Lens or Headlamp. This will give you an easy way to view and interact with your cluster. Be aware, though, that the CPU and memory usage information will be missing until we install metrics-server. Talos has an excellent tutorial on how to install metrics-server, but I’ll explain some of the basics here as well.

Preparing Your Cluster for Metrics-Server

To deploy metrics-server on a Talos-based Kubernetes cluster, you’ll need to make some changes to the machine configuration for each node. The Talos documentation goes into great detail about the reasons for these changes, so I won’t rehash that here. Instead, I’ll focus on how to make these updates.

If you have a configuration file saved for each of your nodes, you can simply update the file and reapply it using the apply-config command as we did earlier. If you don’t have a machine config file saved or prefer to edit it directly, you can use the following Talos CLI command:

talosctl edit machineconfig -e <IP_ADDRESS> -n <IP_ADDRESS>

This will open the configuration file in your preferred editor, allowing you to make the necessary changes. Specifically, you’ll need to enable certificate rotation for the kubelet by adding the following configuration:

machine:
  kubelet:
    extraArgs:
      rotate-server-certificates: "true"

Once the changes are saved, Talos will automatically apply the change to the node.

Automating Certificate Approval

With certificate rotation enabled, you’ll also need a mechanism to automatically approve the new certificates generated by the kubelets. The Kubelet Serving Certificate Approver automates this process. Deploy the Kubelet Serving Certificate Approver directly using the following command:

kubectl apply -f https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml

Deploying the Metrics Server

After enabling certificate rotation and setting up automatic approval, you can deploy the Metrics Server by applying its official manifest:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Alternatively, you can include both the Kubelet Serving Certificate Approver and the Metrics Server in your cluster’s bootstrap process by adding them to the extraManifests section of your cluster configuration:

cluster:
  extraManifests:
    - https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml
    - https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Installing MetalLB

We’ll start by installing MetalLB to provide load balancing capabilities to your cluster. Before doing so, you need to make some permission changes to the metallb-system namespace:

1. Create the namespace:

kubectl create --save-config -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
EOF

2. Install MetalLB using Helm:

helm repo add metallb https://metallb.github.io/metallb
helm install metallb --namespace metallb-system metallb/metallb

Once installed, you’ll need to configure a Layer 2 or BGP mode IP range by applying the following IPAddressPool and L2Advertisement resources:

kubectl apply -f - <<EOF
kind: IPAddressPool
apiVersion: metallb.io/v1beta1
metadata:
  name: default-lb-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.0.46-192.168.0.50 
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: metaladvertisement
  namespace: metallb-system
EOF

This enables MetalLB to assign IP addresses for load-balanced services.

Installing Nginx

Next, we’ll install Nginx as the ingress controller. Add the Helm repository and deploy it:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set publishService.enabled=true

Monitor the pods in the ingress-nginx namespace to ensure everything is running smoothly:

kubectl get pods -n ingress-nginx

Create Storage Class for the NVME drive

Run the following command

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

Installing Kubeapps - optional

Finally, let’s install Kubeapps to provide a UI for managing applications via Helm charts. First, we need to add a PVC that our Kubeapps application can use for storage.

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kubeapps-postgres-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
EOF

Next, we need to add the Helm repository for Kubeapps and install it in a dedicated namespace:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kubeapps --namespace kubeapps bitnami/kubeapps \
  --namespace kubeapps --create-namespace \
  --set ingress.enabled=true \
  --set postgresql.primary.persistence.enabled=true \
  --set postgresql.primary.persistence.existingClaim="kubeapps-postgres-pvc"

You then need to create a credential that can be used to access the dashboard.

kubectl create --namespace default serviceaccount kubeapps-operator
kubectl create clusterrolebinding kubeapps-operator --clusterrole=cluster-admin --serviceaccount=default:kubeapps-operator
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: kubeapps-operator-token
  namespace: default
  annotations:
    kubernetes.io/service-account.name: kubeapps-operator
type: kubernetes.io/service-account-token
EOF

You can now request access to the token we created to access the dashboard.

kubectl get --namespace default secret kubeapps-operator-token -o go-template='{{.data.token | base64decode}}'

For now you will need to add a host entry to your host file to access the UI through the ingress. You can find the IP address for your ingress by running the following command.

kubectl get -n kubeapps  ingress

You should get something that looks like this.

Then edit your /etc/hosts file and add an entry for your kubeapps ingress.

<ip-address>    kubeapps.local

Then you should be able to access Kubeapps by opening a browser and navigate to http://kubeapps.local

Conclusion

By installing MetalLB, Nginx, and Kubeapps, we’ve transformed our Kubernetes cluster into a fault-tolerant and user-friendly platform. These tools, along with a UI-based kubectl IDE and metrics-server, make managing and monitoring your cluster a breeze. Happy clustering!

This is part 2 of my Kubernetes Home Automation Cluster series. In the first part, we talked about how to assemble the hardware. In this part, we’ll go over how to get Kubernetes up and running.

Why Talos Linux?

As I said in Part 1, I really liked the Talos Linux project when I started searching for the best way to run Kubernetes on an SBC. I did a proof of concept on several different hardware/software combinations during my search for the best edge Kubernetes clusters for my home automation needs but ultimately settled on Talos. I won’t go into all my POCs right now because this article is going to be long enough already. I tried both purpose-built Kubernetes distributions and Kubernetes on standard Linux, but Talos stood out as the best fit for my needs.

Prerequisites

Before we begin, here’s what you’ll need:

• A Rock 4 SE SBC or compatible hardware
• An SD card (at least 16 GB)
• A computer with an SD card reader
• Ethernet cable and access to your network’s DHCP server
• A tool to flash disk images (I use Balena Etcher)

Downloading and Flashing the Talos Image

Let’s get started by visiting the Talos Image Factory:

1. Select the "Single Board Computer" option.
2. Choose the version of Talos you want to install. The current version, as of the time of writing, is 1.9.1.
3. Next, select your SBC board. In our case, that’s the Rock 4 SE.
4. Click "Next" on the System Extension and Kernel Arguments screens until you reach the download page.

Download the disk image file from the "Disk Image" link. Then, flash it to your SD card using your favorite image flashing tool. I personally like Balena Etcher because it makes flashing an image to an SD card easy.

Booting the Board

Once you’ve flashed the SD card with the Talos image, insert it into your Rock 4 SE. Plug in the Ethernet cable and power up the board. You should see the board’s lights come on, indicating activity.

Finding Your Device’s IP Address

Next, we need to find the IP address assigned to the board by your network’s DHCP server. How you do this depends on your router or network setup:

• Using Your Router: Log in to your router’s admin interface and look for connected devices.
• Using Network Tools: Tools like nmap or Angry IP Scanner can help you find the new device.
• Omada Users: I use TP-Link Omada networking and can see the device’s IP address in the interface.

Once you’ve located the IP, you can confirm it’s your Talos system by running the following command:

talosctl -e <IP_ADDRESS> -n <IP_ADDRESS> get disks --insecure

This command should list all the disks on the machine. You can also confirm your NVMe drive is detected. You should see something like this:

Configuring Talos

At this point, Talos is booted in maintenance mode and isn’t fully operational yet. The next step is to apply a configuration file to the node so it can boot into the appropriate role. We’re essentially following the Talos Getting Started guide.

Choosing the Cluster Endpoint

The cluster-endpoint is the IP address for your Talos API server. In an all-metal setup like this, you need a stable IP for the control plane node. Talos uses a VIP (Virtual IP) to manage this. A VIP is just an unassigned IP address on your network, and Talos ensures it’s always assigned to the active control plane node using etcd.

Then, run this command:

talosctl gen config <cluster-name> <cluster-endpoint>

Editing the Control Plane File

The gen config command generates several files with very good defaults. The one we’re currently concerned with is the control-plane.yaml file. We need to edit this file to configure our bare-metal cluster. Below are the changes we need to make:

Network Changes

We need to update the network configuration to ensure the node is set up correctly for the cluster. Refer to the Talos documentation for more details, but here are the sections to focus on: DeviceSelector, DHCP, and VIP.

Here’s an example of the network portion:

network:
  interfaces:
    - deviceSelector:
        driver: rk*
      dhcp: true
      vip:
        ip: 192.168.0.45 # Specifies the IP address to be used.
  nameservers:
    - 8.8.8.8
    - 1.1.1.1

Setting the Installation Disk

To ensure the OS installs on the correct media, we’ll use a diskSelector instead of specifying the disk by name. This makes the setup resilient to changes in disk naming conventions:

install:
  image: ghcr.io/siderolabs/installer:v1.9.0 # The image used for installation.
  wipe: false # Indicates if the installation disk should be wiped.
  diskSelector:
    type: sd

Mounting the NVMe Drive

Next, we’ll configure the NVMe drive to mount at a usable location. Update the disks section of the file:

disks:
  - device: /dev/nvme0n1 # The name of the disk to use.
    partitions:
      - mountpoint: /var/mnt/extra # Where to mount the partition.

This configuration ensures the NVMe drive is mounted at /var/mnt/extra.

Allowing Workloads on Control Plane Nodes

Finally, enable workloads to run on control-plane nodes by updating this setting:

allowSchedulingOnControlPlanes: true

With these changes, your control-plane file is ready to bootstrap your bare-metal cluster.

Control Plane Node Setup

The following steps are adapted from the Talos Getting Started guide:

Configuring the First Control Plane Node

First, we need to apply the configuration to our control plane node using the talosctl apply-config command. Replace <IP_ADDRESS> with the IP address of your node:

talosctl apply-config -e <IP_ADDRESS> -n <IP_ADDRESS> --talosconfig ./talosconfig --file controlplane.yaml --insecure

This step applies the settings from your controlplane.yaml file, including network, disk, and scheduling configurations, ensuring the node is properly set up to join the cluster.

Bootstrapping the Control Plane Node

Next, bootstrap the first node of the control plane cluster. This step sets up the etcd cluster and starts the Kubernetes control plane components:

talosctl bootstrap -e <IP_ADDRESS> -n <IP_ADDRESS> --talosconfig ./talosconfig

Verifying the Node

After a few moments, you should be able to download the Kubernetes client configuration and start using your cluster:

talosctl kubeconfig --nodes <IP_ADDRESS> --endpoints <IP_ADDRESS> --talosconfig=./talosconfig

This command merges the cluster configuration into your default Kubernetes configuration file. If you’d like to save it to a different file, you can specify an alternative filename:

talosctl kubeconfig alternative-kubeconfig --nodes <IP_ADDRESS> --endpoints <IP_ADDRESS> --talosconfig ./talosconfig

To make sure that everything is running corrctly, I like to issue the services command via talosctl like:

talosctl services -e <VIP_IP_ADDRESS> -n <NODE_IP_ADDRESS> --talosconfig ./talosconfig

You should get something like this:

You can also run the "health" command like:

talosctl health -e <VIP_IP_ADDRESS> -n <NODE_IP_ADDRESS> --talosconfig ./talosconfig

You should get something like this:

Now, you can connect to your Kubernetes cluster and verify that your nodes are up and running:

kubectl get nodes

You should see your control plane node listed and ready.

Adding Additional Nodes

Adding the remaining SBC boards to your cluster is as simple as scaling up. Ideally, you’ll want at least three control plane nodes for redundancy. To scale your cluster, simply apply your controlplane.yaml file to two additional nodes:

talosctl apply-config -e <VIP_IP_ADDRESS> -n <NODE_IP_ADDRESS> --talosconfig ./talosconfig --file controlplane.yaml --insecure

Again, a good way to check node health is to run the "talosctl services" command:

talosctl services -e <VIP_IP_ADDRESS> -n <NODE_IP_ADDRESS> --talosconfig ./talosconfig

You should get something that looks like this:

If you get something that looks like this:

If issues persist, rebooting the node typically resolves them and allows provisioning to complete.

You can do this with the following command:

talosctl reboot -e <NODE_IP_ADDRESS> -n <NODE_IP_ADDRESS> --talosconfig ./talosconfig  

Once you have configured the additional nodes, you can verify their status by running the following command:

kubectl get nodes

Adding Additional Worker Nodes

If you have more than three nodes to add to your cluster, you can make similar edits to the worker.yaml file as you did to the controlplane.yaml file. There are a few exceptions to note:

• You do not need to set the VIP in the worker configuration file.
• You also do not need to set the allowSchedulingOnControlPlanes property.

Conclusion

I had originally planned for this to be a two-article series, but after writing this, I feel this is a good stopping point. If you’re already familiar with Kubernetes, you can stop here. However, if you’re looking for a "batteries included" version of Kubernetes, I encourage you to check out my next article. There, I’ll explain how to set up an application manager, a load balancer, and an ingress controller.

So, I will be honest with you: this article right here is the whole reason I started my blog. I have a need to move all of my home automation components to a more reliable and scalable solution. Given my current background, I decided that would be a home Kubernetes cluster. This whole journey started when I realized that I needed to run InfluxDB for my home energy monitoring and then continued on when I realized my Raspberry Pi running Home Assistant was less fault-tolerant than I desired. With that in mind, I started building my own home Kubernetes (K8s) cluster.

When looking for an OS to run, I was really looking for something that was Kubernetes-specific. That’s all that I want to run—I don’t need the rest of a typical Linux system and didn’t want the headache of having to keep it patched. There are lots of options out there, but I landed on Talos Linux (https://www.talos.dev/). I don’t want to muddy this article with all the reasons for my decision, but what I will say is that Talos Linux is a very good, immutable, purpose-built Linux distribution.

Once I decided what Linux OS I wanted to run, it was time to decide what SBC board I wanted to use. Since I had decided to use Talos, I started with the boards that they had support for. Looking at the specs for the boards they support, I decided I really liked the Rock 4 line of boards. I settled on the Rock 4SE. They have the same footprint as a Raspberry Pi but offer better specs and are a bit more readily available. They also support POE with the addition of a simple hat and allow for an M.2 SSD. I wanted to stay with the footprint of a Raspberry Pi because it made selecting cases a bit easier.

When it came to case selection, there were lots of options, but I wanted something that I could easily swap boards in and that had a spot for a POE switch. I ended up settling on the Uctronics Raspberry Pi enclosure. For this build, I went with the desktop model, but they also make a rack-mount model.

For my Kubernetes cluster, I chose a PoE switch from TP-Link due to its reliable performance and compatibility with the setup's requirements. TP-Link also stood out for its support of Matter, a critical standard in smart home interoperability, which aligns with my focus on building systems that integrate seamlessly into modern connected environments. Additionally, I’ve found their smart light switches to be particularly effective in my own experience, offering functionality that fits well into a home automation ecosystem.

There are more components I needed, but once I selected the major components, the subcomponents were easier to choose, so I won’t go into detail about the reasoning behind those. Maybe in a future article, I will cover why I made some of these decisions, but if I did that here, this article would be way too long.

Let’s get to the good stuff and build a Kubernetes cluster. First, I’m going to list all the major components needed to build this. Then, I will get into the build details, and in a follow-up post, I will talk about the software installation and some of the ongoing issues that need to be addressed. Without further ado, here is the build list.

Product List

Note: You might also need some standoffs, like these: Link.

Now comes the time for assembly. The case is built for Raspberry Pis, so the Rock 4SE fits fairly well. Really, the only issue is that the case is designed for a standard 2.5" SSD on the back, not an M.2 SSD board. It is also a bit tight when finally assembled, it could use just a little more room to swap boards out easily. With that in mind, I proceeded to mount the Rock 4SE board with the M.2 extension attached so that I could measure where to put standoffs on the backside of the sled.

With the Rock 4SE board attached and the M.2 extension connected but free-hanging, I wrapped the M.2 extension around to the back and marked where to drill holes with a pencil. I don’t know exactly what size holes I drilled in the sled to mount the standoffs for the M.2 extension board because I have a whole box of drill bits and just used the one that was closest. Then I drilled the holes that I had marked.

Once those holes are drilled, assembly can begin. First, we need to install some standoffs for the M.2 extension board on the back of the sled. I’m not sure of the exact length of the standoffs because I had them in a kit that wasn’t labeled, but they were just slightly shorter than the lip of the sled, like this.

I used nuts on the back because the thickness of the sled is not enough to hold the standoffs on its own.

Now for the sled assembly. Start by securing the Rock 4SE board to the front of the sled with the M.2 board connected but hanging freely. Once you have the Rock 4SE board secured, you can then connect the M.2 extension to the back of the sled on the standoffs that were installed. After that, you can put the POE hat on the Rock 4SE. It should look something like this:

Build that 4 times and you will have a fully assembled K8s cluster similar to this:

In conclusion, the hardware assembly for my home Kubernetes cluster is now complete, and the setup is coming together nicely. In the next article, I’ll dive into the software installation process, ensuring everything is configured properly and running smoothly. Stay tuned for more details, and thanks for following along on this journey!

One of the things that I desired to create early on for my Home Assistant installation was a touchscreen control. I wanted to mount this touchscreen somewhere centrally in my house so that the whole family could easily interface with the smart home. After looking at many different touchscreens available online and not knowing exactly what I wanted, I was conversing with a friend of mine. I was telling him about this next home automation project I was working on when he mentioned that he had just removed a touchscreen from a kiosk he was servicing, and it just happened to still be in working order. Voilà, I had found my touchscreen.

This is an EFFINET 27'' TFT LCD touchscreen monitor. You can find similar monitors on eBay. They are typically used in the gaming and kiosk industries. Here is a link to one that sold on eBay. I was easily able to mount this to my wall with an LCD TV wall mount available at most electronics stores.

Problem number one was: how do I get my Home Assistant displayed on this wonderful monitor in my possession? Luckily, with an easy Google search, I realized I could use one of my gifted Raspberry Pi 3s to do exactly that. There is a great OS out there for Raspberry Pis called FullPageOS. Formatting an SD card for FullPageOS is very easy with the Raspberry Pi Imager. There is very good documentation on how to set up and configure the OS to load the webpage of your choice. In this instance, it would be our local Home Assistant that is being used. I won't go into a lot of detail about that because it is very well documented, and we have other problems to solve.

Problem number two was that I wanted to turn this monitor lengthwise. So, I had to figure out how to rotate the display 90 degrees. To make this even more difficult, since it’s a touchscreen, I also had to figure out how to rotate the touch interface 90 degrees. I did lots of Googling this time. There are many different ways to accomplish this through Linux commands, but I was lucky enough to stumble across a script that made it really easy. I have tried recently to find where I found this script so that I could give the author proper credit. Unfortunately, I was not able to find the posting for this particular script. It has a reference to the script and author it was created from, but it does not have the author who did the modifications. I will post the script here and publish it on my GitHub, but if the original author ever wants credit for this, I will gladly include that if they email me.

#!/bin/bash
#  Due to the new display driver in the pi 4 the /boot/config.txt method for screen rotation doesn't work this is a work around for the time being. 
#  Taken from this gist https://gist.github.com/mildmojo/48e9025070a2ba40795c#gistcomment-2694429
#  Adds the ability to rotate the screen with a single command or in a user created addition at build time
#
if [ -z "$1" ] ; then
  echo "Usage: $0 [normal|inverted|left|right]"
  echo " "
  exit 1
fi

function do_rotate
{
  xrandr --output $1 --rotate $2

  TRANSFORM='Coordinate Transformation Matrix'

  POINTERS=`xinput | grep 'slave  pointer'`
  POINTERS=`echo $POINTERS | sed s/↳\ /\$/g`
  POINTERS=`echo $POINTERS | sed s/\ id=/\@/g`
  POINTERS=`echo $POINTERS | sed s/\ \\\[slave\ pointer/\#/g`
  iIndex=2
  POINTER=`echo $POINTERS | cut -d "@" -f $iIndex | cut -d "#" -f 1`
  while [ "$POINTER" != "" ] ; do
    POINTER=`echo $POINTERS | cut -d "@" -f $iIndex | cut -d "#" -f 1`
    POINTERNAME=`echo $POINTERS | cut -d "$" -f $iIndex | cut -d "@" -f 1`
    #if [ "$POINTER" != "" ] && [[ $POINTERNAME = *"TouchPad"* ]]; then    # ==> uncomment to transform only touchpads
    #if [ "$POINTER" != "" ] && [[ $POINTERNAME = *"TrackPoint"* ]]; then  # ==> uncomment to transform only trackpoints
    #if [ "$POINTER" != "" ] && [[ $POINTERNAME = *"Digitizer"* ]]; then   # ==> uncomment to transform only digitizers (touch)
    #if [ "$POINTER" != "" ] && [[ $POINTERNAME = *"MOUSE"* ]]; then       # ==> uncomment to transform only optical mice
    if [ "$POINTER" != "" ] ; then                                         # ==> uncomment to transform all pointer devices
        case "$2" in
            normal)
              [ ! -z "$POINTER" ]    && xinput set-prop "$POINTER" "$TRANSFORM" 1 0 0 0 1 0 0 0 1
              ;;
            inverted)
              [ ! -z "$POINTER" ]    && xinput set-prop "$POINTER" "$TRANSFORM" -1 0 1 0 -1 1 0 0 1
              ;;
            left)
              [ ! -z "$POINTER" ]    && xinput set-prop "$POINTER" "$TRANSFORM" 0 -1 1 1 0 0 0 0 1
              ;;
            right)
              [ ! -z "$POINTER" ]    && xinput set-prop "$POINTER" "$TRANSFORM" 0 1 0 -1 0 1 0 0 1
              ;;
        esac      
    fi
    iIndex=$[$iIndex+1]
  done
}

XDISPLAY=`xrandr --current | grep primary | sed -e 's/ .*//g'`
if [ "$XDISPLAY" == "" ] || [ "$XDISPLAY" == " " ] ; then
  XDISPLAY=`xrandr --current | grep connected | sed -e 's/ .*//g' | head -1`
fi

do_rotate $XDISPLAY $1

Now that I had an easy way to rotate the display and interface, I had to find a way to make that happen when FullPageOS starts up. Luckily, there is a scripts directory in the home directory of FullPageOS located at /home/pi/scripts. This scripts directory has a file called startup_gui. This file configures and runs all the scripts needed by FullPageOS. To ensure that the screen is rotated the direction I want, all I had to do was create a file in the scripts directory for our script that rotates the display. I created a file called /home/pi/scripts/rotate.sh.

touch /home/pi/scripts/rotate.sh

Edit this file with your favorite editor and add the previous script.

 chmod +x rotate.sh 

Before you can execute this script, you will also need to install the xinput dependency and export your display as follows.

sudo apt-get install xinput
export DISPLAY=:0

The last thing to do is make sure that this script is run at startup. This is very easy by editing the /home/pi/scripts/startup_gui. The last line of this file runs the binary to start FullPageOS. I simply inserted a line right before that to run my rotate command.

...
/home/pi/scripts/rotate.sh right

/home/pi/scripts/run_onepageos

I rotated my screen 90 degrees to the right, but you can rotate it as you please.

Here are a few more pictures of my the monitor and my installation.

I still need to add a plug behind the monitor and frame it with some nice trim. Hopefully, I will have time to do that soon. The Raspberry Pi is attached to the back with some double-sided industrial tape.

I hope this article on how I accomplished my Home Assistant touchscreen control is helpful!

When I embarked on the journey of writing this blog, it was more than just a chronicle of my side projects; it was a pathway to share my passion for integrating technology into everyday life. The initial articles of this blog detailed the complexities of setting up Kubernetes, a foundational step that paved the way for my real passion: home automation.

Living in a country home, my venture into home automation was born out of a need for security and energy savings. The idea of keeping my outdoor lights on throughout the night always felt like a safeguard against the unseen 'boogeymen' lurking in the dark. However, letting these lights burn during the daylight hours was a clear waste of energy and resources. I yearned for a solution that offered more sophistication and control than the standard dusk-to-dawn lights could provide. It wasn’t just about turning the lights on at night and off during the day; I wanted a system that adapted to my lifestyle and offered me the flexibility and control I desired.

The heart of my home automation system is Home Assistant, chosen for its Python-based architecture and excellent performance on Raspberry Pi devices. This platform is not just a tool; it's an entire operating system designed for Raspberry Pi, ensuring seamless updates and a robust user experience. The open-source nature of Home Assistant, combined with its extensive community-driven integrations, allows for a high degree of customization. This flexibility is crucial for tailoring the system to fit specific needs and preferences.

To get started with Home Assistant on a Raspberry Pi, a few essential items are needed: a Raspberry Pi 4 or 3 Model B, a suitable power supply, a Micro SD Card (32 GB or larger), an SD card reader, and an Ethernet cable for a reliable connection during installation. I used a Raspberry Pi 3 because I was gifted some that were being cycled out and it has worked great. The setup process is straightforward and well-documented on the Home Assistant website. It involves preparing the Raspberry Pi, writing the Home Assistant OS image to the SD card, starting up the Raspberry Pi, and accessing Home Assistant through a browser. While simple, the process might require troubleshooting, especially if the Home Assistant page does not show up after installation​​.

The benefits of this journey into home automation are numerous. The most tangible is the cost savings – managing energy usage more intelligently means lower bills and a reduced environmental impact. More importantly, it's about the comfort and convenience of a home that understands and responds to your unique lifestyle. A home that doesn't just function smartly but also intuitively.

As I delve deeper into the world of home automation, future articles will explore specific projects and implementations. I'll share the practical applications, the challenges encountered, and the innovative solutions that have emerged. This journey is not just about technology; it's about creating a living space that is as intelligent as it is comfortable.

In my first part, I introduced how to set up a basic Linode Kubernetes(LKE) cluster using Argo CD autopilot and configuring Traefik ingress and ExternalDNS for routing using Linode Domains.  In this article, we are going to add automatic HTTPS for the ingress of our applications.  To do this we will be using cert-manager in conjunction with Let's Encrypt.  

Certificates with cert-manger

Kubernetes cert-manager adds the capability to manage certificates for your services in an automated way.  The easiest and cheapest way to get certificates for your services is to configure cert-manager to get certificates from Let's Encrypt.  Let's Encrypt provides free trusted certificates to websites to make encryption easy and cheap.  

Cert-manger uses the ACME protocol to verify domains.  You can verify domains in two different ways:

• HTTP request to a well-known URL
• DNS recorded created and verified

We will be using the second mechanism, DNS record created and verified.  This mechanism has the advantage of not requiring ingress to be configured and working for domain ownership to be verified.  Cert-manager has integration with many DNS providers to allow integration.  However, they DO NOT provide native integration with Linode Domains.  They do allow third parties to integrate through a webhook that is provided.  This is the mechanism that we will use and is documented on their site here with the implementation of the webhook here.

As we did in the previous article, we will need to add an application spec to Argo CD and the corresponding manifest for cert-manager.  In this instance, we are going to be loading a helm chart that will merge the cert-manager dependency with the cert-manager-webhook-linode dependency.

First, let's look at the ArgoCD application.  Again this will be created in the bootstrap dir and look something like this.

#./bootstrap/cert-manager.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: null
  labels:
    app.kubernetes.io/managed-by: argocd-autopilot
    app.kubernetes.io/name: cert-manager
  name: cert-manager
  namespace: argocd
spec:
  destination:
    namespace: cert-manager
    server: https://kubernetes.default.svc
  ignoreDifferences:
  - group: argoproj.io
    jsonPointers:
    - /status
    kind: Application
  project: default
  source:
    path: bootstrap/cert-manager
    repoURL: https://github.com/owner/repo.git
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - allowEmpty=true
    - CreateNamespace=true
status:
  health: {}
  summary: {}
  sync:
    comparedTo:
      destination: {}
      source:
        repoURL: ""
    status: ""

In this file, you will need to change the source repo to match your git repo.

repoURL: https://github.com/owner/repo.git

Next, we need to create a cert-manager directory to put the cert-manager HELM files in like this:

mkdir ./bootstrap/cert-manager

Then we need to create a master HELM chart to combine cert-manager and the Linode webhook.

#./bootstrap/cert-manager/Chart.yaml
apiVersion: v2
name: cert-manager
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"

dependencies:
- name: "cert-manager"
  version: 1.10.1
  repository: https://charts.jetstack.io
- name: "cert-manager-webhook-linode"
  version: 0.2.0
  repository: "file://./chart/cert-manager-webhook-linode"

The main thing to notice in this file is that it simply merges 2 dependencies.  It combines cert-manager and the cert-manager-webhook-linode.

Cert-manager is pulled from the jetstack helm chart repository.  This makes things very easy to update.  When jetstack releases a new version of the cert-manager helm chart you simply edit this file and update the version of the cert-manager dependency.

Unfortunately, cert-manager-webhook-linode is not stored in a HELM repo and is only accessible on GitHub as a code repository.  The only way to reference dependencies in HELM is to reference a repo or a subdirectory.  For this reason, we will need to copy version v0.2.0 of cert-manager-webhook-linode to our ./bootstrap/cert-manager/chart/cert-manager-webhook-linode dir.

mkdir ./bootstrap/cert-manager/chart

Now copy the cert-manager-webhook-linode dir from your downloaded resource to the chart dir.

Unfortunately, v0.2.0 of the cert-manager-wehbook-linode has a bit of an error in its helm chart.  We will need to fix this in order for the container to launch correctly.  Edit the file ./bootstrap/cert-manager/chart/cert-manager-webhook-linode/values.yaml and comment out the "logLevel: 6" line.  The deployment section will then look like this.

 #./bootstrap/cert-manager/chart/cert-manager-webhook-linode/values.yaml 
deployment:
  secretName: linode-credentials
  secretKey: token
  # logLevel: 6

Next, we need to create a values file to set the variables in our dependent HELM charts.

#./bootstrap/cert-manager/values.yaml
chartVersion:
keyID:

cert-manager:
  installCRDs: true

cert-manager-webhook-linode:
  api:
    groupName: acme.<your.domain>
  image:
    tag: v0.2.0
  deployment:
    secretName: linode-api-token

There are a few variables being set in this values file.  First is the cert-manager CRD install.  The CRDs will be needed for our use cases.

cert-manager:
  installCRDs: true

Next, a few variables for the cert-manager-webhook-linode process need to be set.  The api.groupName needs to be filled in with your domain name.  Then the image needs to be set to the same version as the download: v0.2.0. Also, the deployment secret needs to be set to our Linode token name: linode-api-token.

cert-manager-webhook-linode:
  api:
    groupName: acme.<your.domain>
  image:
    tag: v0.2.0
  deployment:
    secretName: linode-api-token

API Token

By default, secrets are only accessible within the same namespace in which they are created. This is because secrets are stored as Kubernetes API objects, which are scoped to a particular namespace. This means that secrets cannot be accessed by pods or services in other namespaces, even if those namespaces are part of the same cluster. Because of this, we need to copy our Linode API token to the cert-manager namespace.

kubectl create namespace cert-manager
kubectl get secret linode-api-token --namespace=external-dns -oyaml | grep -v '^\s*namespace:\s' | kubectl apply --namespace=cert-manager -f -

Cluster Issuers

For cert-manager to work properly with Let's Encrypt we need to configure a cert-manager ClusterIssuer.  We are going to create more than 1  ClusterIssuer to allow testing our setup before using production Let's Encrypt certificates. This is useful because there is a rate-limit on the production Let's Encrypt certificate issuer.

First, let's create the Let's Encrypt staging file.  This is the Let's Encrypt staging server.  It is not a trusted certificate, but the server is not rate-limited.  Use this configuration when you are testing sequences that will cause the issuance of more certificates than are allowed for Let's Encrypt trusted certificates.

#./bootstrap/cert-manager/templates/letsencrypt-staging.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: <your@email>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - dns01:
          webhook:
            solverName: linode
            groupName: acme.<your.domain>

The next file to create is the Let's Encrypt production ClusterIssuer file.  This is the configuration to use when you want to issue a production Let's Encrypt certificate.

#./bootstrap/cert-manager/templates/letsencrypt-prod.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: <your@email>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - dns01:
          webhook:
            solverName: linode
            groupName: acme.<your.domain>

In both of these files you will need to update the groupName to match the groupName define in the values file of the deployment.

groupName: acme.<your.domain>

Update our Ingress

The next thing we need to do to manage ingress certificates automatically is open up the ingress SSL port.  In our case, we are also going to automatically redirect all traffic to the appropriate SSL interface so that we can enforce secure traffic.

In my previous post, I showed how to configure Traefik to accept Kubernetes ingress traffic.  We must add the following image arguments to our traefik configuration in the file ./traefik/traefik.yaml.

- --entrypoints.web.http.redirections.entrypoint.to=websecure
- --entrypoints.websecure.address=:443
- --entrypoints.websecure.http.tls

We will need to open the SSL port on the container running the Traefik service.  We will need to add the following to the container.ports section of the yaml.

- name: websecure
  containerPort: 443

The SSL port for the Traefik service will also need to be opened for the ingress to work.  This will require adding the following to the traefik-web-service LoadBalancer.

    - name: websecure
      targetPort: websecure
      port: 443

The complete file will look like this.

./traefik/traefik.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-account
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-role

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-role-binding

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-role
subjects:
  - kind: ServiceAccount
    name: traefik-account
    namespace: traefik
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik-deployment
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-account
      containers:
        - name: traefik
          image: traefik:v2.9
          args:
            - --api.insecure
            - --api.dashboard=true
            - --providers.kubernetesingress
            - --providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik-web-service
            - --entrypoints.web.address=:80
            - --entrypoints.web.http.redirections.entrypoint.to=websecure
            - --entrypoints.websecure.address=:443
            - --entrypoints.websecure.http.tls
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-service
spec:
  type: LoadBalancer
  ports:
    - name: web
      targetPort: web
      port: 80
    - name: websecure
      targetPort: websecure
      port: 443
  selector:
    app: traefik

With all these edits in place, we will need to commit our code changes to our git repo like this:

git add .
git commit -m "adding cert-manager and updating traefik to force ssl"
git push origin

Let's test it!

Let's update our whoami application to use SSL with a certificate from Let's Encrypt.  We will use the test certificate store to make sure that everything is working correctly before we switch to the production certificate store.  To do that we will need to update our whoami ingress in the ./apps/whoami/base/install.yaml file to add the following annotations.

  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: websecure

We will also need to add TLS to the ingress definition.

  tls:
  - hosts:
      - whoami.mydomain.com
    secretName: secure-whoami-cert

The full file will look like this:

#./apps/whoami/base/install.yaml
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
  labels:
    app: whoami

spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  ports:
    - name: http
      port: 80

  selector:
    app: whoami
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami-ingress
  annotations:
    external-dns.alpha.kubernetes.io/hostname: whoami.yourdomain.com
    cert-manager.io/cluster-issuer: letsencrypt-staging
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  tls:
  - hosts:
      - whoami.yourdomain.com
    secretName: secure-whoami-cert
  rules:
  - host: whoami.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Exact
        backend:
          service:
            name:  whoami
            port:
              number: 80

Commit the changes:

git add .
git commit -m "update whoami ingress to use SSL"
git push origin

It may take a few seconds/minutes for the verification process and for the certificate to be issued.   You can check on the process by watching the cert-manager application in ArgoCD UI.  

You can also check the output for the web service using this curl command:

curl -k whoami.yourdomain.com

Success!! You should have a secure whoami service.  We have one problem though, we are using the untrusted staging Let's Encrypt certificate.  We need to make one last change to our ingress to use the trusted production Let's Encrypt certificate. We need to change the following line:

cert-manager.io/cluster-issuer: letsencrypt-staging

To the following:

cert-manager.io/cluster-issuer: letsencrypt-prod

Commit the change:

git add .
git commit -m "update whoami ingress to use production let's encrypt"
git push origin

Yay!! We are finally done!  You should now have cert-manager configured to provide your services with both staging and production certificates from Let's Encrypt.

TL;DR

This article is intended to show how to set up an LKE cluster, bootstrap it with Argo CD Autopilot and install ExternalDNS and Traefik.

I have been a customer of Linode for a while now and being that I am a platform engineer at my day job, I wanted to build myself a platform on LKE(Linode Kubernetes) for several of my home projects and side hustles. Doing what any engineer my age does first I "googled" it. On a side note, I want to mention that it looks like safari is technically switching to duck-duck-go for its default search engine. We can talk more about that later. While I did find many good articles and videos in my search, I did not find one that centers around GitOps and more specifically Argo CD. I am a huge fan of both GitOps and Argo CD so I wanted to start my platform with those in mind.

Wanting to start my platform with Argo CD and having used Argo CD before, I knew they had a project called Argo CD Autopilot. It is specifically intended for bootstrapping a new Kubernetes cluster with Argo CD and providing a good opinionated project code structure. I wanted to give the Argo CD Autopilot project a try to see if it would help get me moving quicker.

Deploying your LKE cluster

Linode has many good docs and guides. They have several on how to set up your LKE cluster and configure your local kubectl config. I will leave that up to them to explain as they will do a much better job than I will.

I used the following article, but they have many others as well: https://www.linode.com/docs/guides/lke-continuous-deployment-part-3

For the rest of this article, I will assume you have a running LKE cluster. The size of the cluster is not important as this should run on their minimum-sized cluster.

Bootstrapping with Argo CD Autopilot

Next, we will want to get our new Kubernetes cluster started right by bootstrapping all of the applications with Argo CD. Argo CD has a project called Autopilot with the specific goal of making GitOps easier.

Here is a link to their homepage: https://argocd-autopilot.readthedocs.io/en/stable/

To get started using Argo CD Autopilot you will need:

• a git repo (I use GitHub)
• a token for your git repo
• the argocd-autopilot command for your OS installed
• a kubernetes cluster (In our case LKE)
• kubectl configured to connect to your kubernetes cluster

Argo CD Autopilot does a good job of documenting their cli and we will be following their getting started guide here: https://argocd-autopilot.readthedocs.io/en/stable/Getting-Started/

First, you will need to export your git token:

export GIT_TOKEN=ghp_PcZ...IP0

Next, you will need to export the git repo you would like to store your code in.

export GIT_REPO=https://github.com/owner/name/some/relative/path

Then you will simply execute the bootstrap command to get your new LKE cluster up and running with Argo CD

argocd-autopilot repo bootstrap

Congratulations!! You should have Argo CD running on your cluster and connected to your git repository to deploy automatically based on your git commits. The code that was generated will be pushed to the git repository that was specified with the GIT_REPO environment variable. You should be able to connect to the Argo CD UI using a local forward like this:

kubectl port-forward -n argocd svc/argocd-server 8080:80

Using a browser to access the address: http://localhost:8080 and using the username: admin and password given during the bootstrap command, you will get the Argo CD UI.

If you missed the password at setup you should be able to use this command to retrieve it:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

If you look at the git repo used during the repo export command you should have a directory structure that looks like this:

Argo CD Autopilot gives a good starting code structure to organize your code. It has the following base dirs:

• app - This is where deployed application specifications live
• bootstrap - This is where all of the applications and manifest live to bootstrap the cluster with Argo CD, including Argo CD itself.
• projects - This is where all the Argo CD projects are defined.

git clone git@github.com:owner/name/some/relative/path.git

Adding ExternalDNS to the mix

Argo CD and GitOps give us a great start, but it would be nice if every service that is started on our platform was able to be accessed by name. Enter ExternalDNS! In their own words, "ExternalDNS makes Kubernetes resources discoverable via public DNS servers. Like KubeDNS". ExternalDNS has many integrations with DNS providers, but we are going to couple it with Linode Domains. Linode Domains is the DNS service that Linode provides that allows you to manage DNS for your domains and it's FREE!!!

To get started we need the following:

• Domain setup in Linode Domains
• Linode API key

The integration between Linode Domains and ExternalDNS is fairly well documented here but I will repost the RBAC deployment as I made some changes.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: external-dns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.k8s.io/external-dns/external-dns:v0.13.2
        args:
        - --source=ingress
        - --source=service
        - --provider=linode
        - --domain-filter=example.com # (optional) limit to only example.com 
        - --txt-prefix=xdns-
        env:
        - name: LINODE_TOKEN
          valueFrom:
            secretKeyRef:
              name: linode-api-token
              key: token

There are a few things I would like to note about my configuration as compared to the configuration in the Linode ExternalDNS example. If you look at the "args" section of the spec you will notice that I have added 2 more args. The first of which is another source arg.

This is done so that when we start creating ingresses, the ExternalDNS controller will watch ingress definitions also. I find this helpful so that I can place all my annotations related to ingress and DNS in one spot on the ingress definition.

This option is in the example but I think it's very important to mention here. If you are running multiple instances of ExternalDNS against a DNS provider you will want to set this filter. If you do not then the ExternalDNS instances will collide with each other and modify each other's DNS entries. This might happen for instance, if you are running multiple clusters.

The next additional arg added is to specify a prefix for the txt DNS records that are added. This is used so that there will not be collisions if a CNAME record is created in Linode because CNAME records and TXT records CAN NOT be named the same. ExternalDNS uses some logic to decide whether to create a CNAME or an A record depending if it detects a load balancer or not (This is explained in this article). CNAME and TXT collisions showed up in my log files when I was originally doing the integration but have since gone away. I believe they were related to this issue, which should have been fixed in version 12.2 but I was still seeing the issue. I need to investigate more.

To deploy the ExternalDNS manifest we are going to use Argo CD. In order to accomplish this we will need to add an Argo CD application specification along with the ExternalDNS manifest that we just looked at to the bootstrap directory of our Argo CD application. Argo CD Autopilot cli has many handy features for creating projects and applications but unfortunately, they do not have one for creating bootstrap applications so we will have to do this manually.

Create the following files in the locations specified.

#./bootstrap/external-dns.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: null
  labels:
    app.kubernetes.io/managed-by: argocd-autopilot
    app.kubernetes.io/name: external-dns
  name: external-dns
  namespace: argocd
spec:
  destination:
    namespace: external-dns
    server: https://kubernetes.default.svc
  ignoreDifferences:
  - group: argoproj.io
    jsonPointers:
    - /status
    kind: Application
  project: default
  source:
    path: bootstrap/external-dns
    repoURL: https://github.com/owner/repo.git
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - allowEmpty=true
    - CreateNamespace=true
status:
  health: {}
  summary: {}
  sync:
    comparedTo:
      destination: {}
      source:
        repoURL: ""
    status: ""

In this file, you will need to change the source repo to match your git repo.

repoURL: https://github.com/owner/repo.git

Then we will need to add the ExternalDNS manifest to a new directory we create under the bootstrap directory like this.

mkdir ./bootstrap/external-dns

#./bootstrap/external-dns/external-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: external-dns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.k8s.io/external-dns/external-dns:v0.13.1
        args:
        - --source=ingress
        - --source=service
        - --provider=linode
        - --domain-filter=example.com # (optional) limit to only example.com 
        - --txt-prefix=xdns-
        env:
        - name: LINODE_TOKEN
          valueFrom:
            secretKeyRef:
              name: linode-api-token
              key: token

You will notice in the external-dns manifest we specified the Linode API token needed by external-dns to access your Linode account like this:

env:
- name: LINODE_TOKEN
  valueFrom:
    secretKeyRef:
      name: linode-credentials
      key: token

The valueFrom:secretKeyRef: allows us to tell Kubernetes to pass in a secret key for this env value. We will use this mechanism so that we do not have to commit our secret Linode token to source control.

We will create our Linode secret using the kubectl command. To create a secret using kubectl issue this command with kubectl connected to the appropriate Kubernetes context. We will also issue a command to create the namespace. Normally argo-cd will do this, but in this instance, we need the secret before argo-cd has run gitops.

kubectl create namespace external-dns
kubectl create secret generic linode-api-token -n external-dns --from-literal=token='token_goes_here'

After this secret is created you are ready to commit the source code in your project created by Argo CD. You can issue these commands from the repository you cloned previously.

git add .
git commit -m "adding externalDNS"
git push origin

Argo CD will see the commit and perform all of its GitOps duties. Once deployed you should see something like this from localhost:8080 if you are still running the port-forward command to Argo CD's service.

Congratulations!! You now have a working externalDNS.

Adding Ingress

The next thing needed to have a good functioning platform is an ingress implementation. There are many Kubernetes ingress implementations out there but I have come to like Traefik.

Implementing Traefik is fairly simple and that is one of the main reasons that I like using it. Just like in our ExternalDNS implementation, we will need to add a few files to the Argo CD bootstrap directory. First, we will add the application specification for Traefik as so:

#./bootstrap/traefik.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: null
  labels:
    app.kubernetes.io/managed-by: argocd-autopilot
    app.kubernetes.io/name: traefik
  name: traefik
  namespace: argocd
spec:
  destination:
    namespace: traefik
    server: https://kubernetes.default.svc
  ignoreDifferences:
  - group: argoproj.io
    jsonPointers:
    - /status
    kind: Application
  project: default
  source:
    path: bootstrap/traefik
    repoURL: https://github.com/owner/repo.git
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - allowEmpty=true
    - CreateNamespace=true
status:
  health: {}
  summary: {}
  sync:
    comparedTo:
      destination: {}
      source:
        repoURL: ""
    status: ""

Again, In this file, you will need to change the source repo to match your git repo.

repoURL: https://github.com/owner/repo.git

Next, we need to create a traefik directory to put the traefik manifest and CRD in like this:

mkdir ./bootstrap/traefik

Then we need to create the traefik manifest.

#./bootstrap/traefik/traefik.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-account
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-role

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-role-binding

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-role
subjects:
  - kind: ServiceAccount
    name: traefik-account
    namespace: traefik
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik-deployment
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-account
      containers:
        - name: traefik
          image: traefik:v2.9
          args:
            - --api.insecure
            - --api.dashboard=true
            - --providers.kubernetesingress
            - --providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik-web-service
            - --entrypoints.web.address=:80
          ports:
            - name: web
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-service
spec:
  type: LoadBalancer
  ports:
    - name: web
      targetPort: web
      port: 80
  selector:
    app: traefik

The key things to notice in the traefik.yaml files are the args and the ports. Let's talk about the args first. Here is the one we want to pay special attention to:

- --providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik-web-service
- --entrypoints.web.address=:80

The arg in our list "publishedservice" is a namespace/service argument for the service that is responsible for publishing the ingresses.

The "entrypoints" args are used to enable access from the outside and associate it with a port. Currently, we will just open up port 80 we will address SSL in the future.

Again we will need to commit our code changes to our git repo like this:

git add .
git commit -m "adding Traefik ingress"
git push origin

Once you commit these files and Argo CD refreshes you should see something similar to this:

Let's Deploy an App

So that we can check all of our hard work let's deploy a simple app using Argo CD. As I said before Argo CD Autopilot has a good cli and we can use that to create our Argo CD project and application.

Let's first create a project list this:

argocd-autopilot project create test

Next, we will need to create the application specification that Argo CD will use to manage the application. Currently, Autopilot only supports creating applications from a Kustomization specification. Some people see this as a limitation, but as mentioned in this issue. Kustomize allows a native way to import HELM charts if that is what the application is specified.

Argo CD Autopilot cli will need a few things for it to create the application. One of the things it will need is the initial Kustomization file you want to use to create your application. It can get the Kustomization file from a git repo or a local file system. We are going to use a local file system to make things easier. You will need to create the following files on your local system somewhere outside of the Argo CD Autopilot repo.

#kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml

#deployment.yaml
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
  labels:
    app: whoami

spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  ports:
    - name: http
      port: 80

  selector:
    app: whoami
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami-ingress
  annotations:
    external-dns.alpha.kubernetes.io/hostname: whoami.yourdomain.com
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
  - host: whoami.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Exact
        backend:
          service:
            name:  whoami
            port:
              number: 80

Few things to mention here. You will need to change:

whoami.yourdomain.com

To a domain that you own and which is configured in Linode Domains.

Next, we will run the command to create the application in your Argo CD repo.

argocd-autopilot app create whoami --app ./path/to/kustomization/dir --project test

You should now be able to access HTTP//:whoami.yourdomain.com in your browser of choice.

Closing

Yay!! You did it. You should now have a platform that can:

• Deploy an application using GitOps
• Automatically create DNS records
• Automatically setup Ingress for a service

In my next post, I will show how to add automatic SSL to the mix with cert-manager and LetsEncrypt.

Here is a bit of a back story about why I created a blog in 2023.

TL;DR

I'm a tech junky with a computer science degree living in the country, full-time teleworking, that has been putting off creating a tech blog for 20 years.

Where do I start?  First I guess I will start by telling you a little about me.  I'm a software engineer/devops engineer/platform engineer and just general tech junky. I'm in my mid-forties, which means that I spent my youth in the best decade ever,  the 90s!!!  I grew up in a small town somewhere in middle America.  I grew up in the country; hunting, camping, going to the lake, and shooting guns are the things I did for fun.

Even though I lived in the country I still grew up as a part of the original Nintendo generation and I have always had a scientific mind.  When I was around 12 years old I got my first computer for Christmas.  I was so excited, some of my friends had a computer and I had wanted one for quite a while. It was a 286 and it was awesome!

I did my fair share of playing games as most kids did but I was also very interested in making the computer do things I wanted it to do.  One of the first things I did was to write a menu system for our 286.  At the time Windows did not exist, so you were stuck using ms-dos.  My mom would take me to the library and I would check out programming books. Yes, I had to check out actual books, the internet was not a thing for the common man yet.  With help from my library books, I was able to write a simple menu system that made it easier for my family to use the computer.  It was simple and the user just typed in a number to get to one of the programs that were loaded on the system.  That's all it took and I was hooked, I knew I wanted to be a software engineer or rock star.  We will talk more about that in a future post, maybe.

Fast forward a few years and I was in college pursuing my software engineering degree at a mid-sized state college and downloading illegal music off of Napster of course. I was working at a well-known internet company at the time doing tech support and that would lead me to meet my future wife online.   This was way before dating sites and the like, but we met in what would have been a "chat room" at the time.  I graduated with my computer science degree and worked for a few start-ups that allowed me to move out of my small town and travel around the country.  

A few years later and my now wife and I were looking to move back home to the state we were from and I was lucky enough to find a job at a government agency where the work seemed interesting.  I never really thought I would stay at a company for a long time as I like to move around and do different parts of the tech industry.  Nor had I ever really thought about working for the government and what that really meant and some of the connotations associated with it.  However, fast-forward about 2 decades and it turns out I am still happily working for that same government agency.  In my career at the agency, I have been fortunate enough to have moved around and been allowed to do many different interesting and cutting-edge things.  I have been a software engineer, application security SME, a DevOps engineer, and I am currently the lead platform engineer for our new Kubernetes platform.  The agency I work for also started letting IT people start teleworking many years ago and I was allowed to start teleworking full-time a few years before the 2020 covid lockdown.  I take pride in the work that I do for the government and enjoy my job and the people I work with.

I always knew I would move back to the country and a little over a decade ago my wife and I were able to purchase our dream property and build our dream home.  Many of the people I work with are fascinated that I am a tech guy living on 20 acres somewhere out in the country, but I wouldn't have it any other way.  I have used my tech knowledge to become a home automation junky and I am always trying to automate or modernize some aspect of my family's daily living.  

Ok long story short, I have always wanted to create a tech blog just never took the time to write one; Notice I registered my joshdmoore.com domain in 2002.  

Domain Name: JOSHDMOORE.COM
Registry Domain ID: 87627451_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.godaddy.com
Registrar URL: https://www.godaddy.com
Updated Date: 2022-06-21T14:37:59Z
Creation Date: 2002-06-17T21:17:45Z
Registrar Registration Expiration Date: 2024-06-17T21:17:45Z
Registrar: GoDaddy.com, LLC

My 2023 resolution is to do some of the things I have wanted to do but have not done for one reason or another.  This will mainly be a tech blog but will cover other things such as country living and a few other aspects of my life.