skip to content
Andrew Marder

Why did I do this!?

  1. I’ve been self-hosting an email newsletter from home. With power outages, internet outages, and home server outages, the total downtime was getting to be a bit much for my taste. I wanted to deploy to the cloud for better uptime.

  2. I’m cheap. I had spun up a two-node Kubernetes cluster with a load balancer on DigitalOcean, and that was costing about $36 a month. It didn’t feel like I was getting much for my money.

  3. I like declarative deployments.

When I learned I could run Kubernetes for $4.10 a month on Hetzner, I decided it checked all my boxes.

  • Better uptime (compared to my homelab) ✔️
  • Cheap ✔️
  • Declarative deployments ✔️

So, I decided to go down a rabbit hole and deploy Keila with Kubernetes on Hetzner. This post will walk through the key things I learned in the process. If you have any suggestions on how I can improve this setup, I’m all ears! I plan to add additional services like Uptime Kuma to my “cluster”.

Prerequisites

Here’s a quick list of everything I needed to set up:

For fellow macOS users, here are the commands I used to install these tools:

Terminal window
brew install vitobotta/tap/hetzner_k3s
brew install kubectl
brew install helm
brew install helmfile
export PATH="$PATH:/opt/homebrew/bin"


# helm-secrets has 3 components
helm plugin install https://github.com/jkroepke/helm-secrets/releases/download/v4.7.5/secrets-4.7.5.tgz --verify=false
helm plugin install https://github.com/jkroepke/helm-secrets/releases/download/v4.7.5/secrets-getter-4.7.5.tgz --verify=false
helm plugin install https://github.com/jkroepke/helm-secrets/releases/download/v4.7.5/secrets-post-renderer-4.7.5.tgz --verify=false


helm plugin install https://github.com/databus23/helm-diff --verify=false


curl -L -o sops https://github.com/getsops/sops/releases/download/v3.11.0/sops-v3.11.0.darwin.arm64
chmod +x sops
sudo mv sops /usr/local/bin/


brew install age

1. Create Hetzner API Token

  1. Log in to Hetzner Cloud Console
  2. Create a new project (e.g. k3s)
  3. Go to Security → API tokens → Generate API token
  4. Give it Read & Write permissions
  5. Save the token

2. Create the Cluster

Create a file called cluster.yaml:

cluster_name: k3s
kubeconfig_path: "./kubeconfig"
k3s_version: v1.35.1+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
      - ::/0
    api:
      - 0.0.0.0/0
      - ::/0
  public_network:
    ipv4: true
    ipv6: true
  private_network:
    enabled: true
    subnet: 10.0.0.0/16
    existing_network_name: ""
  custom_firewall_rules:
    - description: "Allow HTTP"
      direction: in
      protocol: tcp
      port: 80
      source_ips:
        - 0.0.0.0/0
        - ::/0
    - description: "Allow HTTPS"
      direction: in
      protocol: tcp
      port: 443
      source_ips:
        - 0.0.0.0/0
        - ::/0

schedule_workloads_on_masters: true

masters_pool:
  instance_type: cx23
  instance_count: 1
  locations:
    - nbg1

worker_node_pools: []

addons:
  csi_driver:
    enabled: false
  local_path_storage_class:
    enabled: true
  traefik:
    enabled: true
  servicelb:
    enabled: true

protect_against_deletion: true

A few things to note:

  • IPv4 included. I was tempted to save $0.60/month by going IPv6-only (public_network.ipv4: false), but IPv4 means you can SSH in from any network, all visitors can reach your services regardless of their ISP, and you don’t have to troubleshoot IPv6 connectivity issues. Overall, I think it’s money well-spent.
  • schedule_workloads_on_masters: true is required for a single-node cluster so pods can run on the master node.
  • CSI driver disabled. We’re using k3s’s built-in local-path-provisioner instead of Hetzner block storage. Data is stored directly on the node’s disk at /var/lib/rancher/k3s/storage/. This saves money but means data is lost if the node’s disk fails.
  • Traefik and ServiceLB enabled. Traefik handles ingress routing; ServiceLB exposes ports 80 and 443 on the node.
  • Firewall. Hetzner’s Cloud Firewall allows only SSH, HTTP, HTTPS, and the Kubernetes API. Everything else is blocked.

Note that hetzner_token is not in the config file. Instead, hetzner-k3s reads it from the HCLOUD_TOKEN environment variable.

Create the cluster:

Terminal window
export HCLOUD_TOKEN=<your-hetzner-api-token>
hetzner-k3s create --config cluster.yaml | tee create.log

This takes a few minutes. When finished, your kubeconfig is saved to ./kubeconfig:

Terminal window
export KUBECONFIG=./kubeconfig
kubectl get nodes

You can check the available k3s versions with hetzner-k3s releases.

hetzner-k3s deploys a cluster autoscaler, but it’s not needed for a single-node cluster with no worker pools. It will sit in CrashLoopBackOff indefinitely, so I chose to remove it:

Terminal window
kubectl delete deployment cluster-autoscaler -n kube-system

3. Create DNS Record on Cloudflare

Get your server’s IPv4 address:

Terminal window
kubectl get nodes -o wide

Look for the EXTERNAL-IP column.

In the Cloudflare dashboard:

  1. Select your domain
  2. Go to DNS → Records → Add Record
  3. Create a wildcard A record:
    • Type: A
    • Name: *.k3s
    • IPv4 address: your server’s IP address
    • Proxy status: DNS only (grey cloud)
  4. Click Save

Now any subdomain of k3s.andrewmarder.net resolves to your server. Adding a new service later is just creating a new Kubernetes Ingress — no DNS changes needed.

4. Project Structure

We use Helmfile to manage all cluster services declaratively. The project is organized as follows:

.
├── .sops.yaml                 # SOPS config (age public key)
├── cluster.yaml               # hetzner-k3s cluster config (no secrets)
├── helmfile.yaml              # declares all Helm releases
├── charts/
│   ├── cluster-tls/           # ClusterIssuer + wildcard cert + TLSStore
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   └── templates/
│   │       ├── cloudflare-secret.yaml
│   │       ├── cluster-issuer.yaml
│   │       ├── tls-store.yaml
│   │       └── wildcard-certificate.yaml
│   └── keila/                 # Keila app + Postgres
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
│           ├── ingress.yaml
│           ├── keila-deployment.yaml
│           ├── keila-secrets.yaml
│           ├── keila-service.yaml
│           ├── postgres-deployment.yaml
│           ├── postgres-pvc.yaml
│           ├── postgres-service.yaml
│           └── uploads-pvc.yaml
├── secrets/                   # SOPS-encrypted (safe to commit)
│   ├── cluster-tls.yaml
│   ├── hetzner.yaml
│   └── keila.yaml
└── values/                    # plain-text config (no secrets)
    ├── cluster-tls.yaml
    └── keila.yaml

The key ideas:

  • helmfile.yaml is the single source of truth for what’s deployed. Adding a service means adding a chart (or reusing an existing one), a values file, and a secrets file, then running helmfile apply.
  • secrets/ files are SOPS-encrypted with age. They’re safe to commit to git. The helm-secrets plugin decrypts them on the fly during helmfile apply.
  • values/ files are plain-text configuration (image versions, storage sizes, schedules) — no sensitive data.

Here’s my helmfile.yaml:

repositories:
  - name: jetstack
    url: https://charts.jetstack.io

releases:
  - name: cert-manager
    namespace: cert-manager
    createNamespace: true
    chart: jetstack/cert-manager
    version: v1.17.2
    values:
      - crds:
          enabled: true

  - name: cluster-tls
    namespace: cert-manager
    chart: ./charts/cluster-tls
    needs:
      - cert-manager/cert-manager
    values:
      - ./values/cluster-tls.yaml
    secrets:
      - ./secrets/cluster-tls.yaml

  - name: keila
    namespace: keila
    createNamespace: true
    chart: ./charts/keila
    needs:
      - cert-manager/cluster-tls
    values:
      - ./values/keila.yaml
    secrets:
      - ./secrets/keila.yaml

A few things to note:

  • needs ensures releases are installed in order: cert-manager first, then the ClusterIssuer and wildcard certificate, then Keila.
  • createNamespace: true creates the namespace automatically.
  • secrets references SOPS-encrypted files. Helmfile (via helm-secrets) decrypts them at deploy time and merges the values with the plain-text values files.
  • Pinned versions. The cert-manager chart version is pinned; image versions are pinned in the values files.

5. Encrypt Secrets with SOPS + age

All secrets are stored encrypted in the secrets/ directory using SOPS with age encryption. This means secrets live in git alongside your charts — encrypted at rest, decrypted on the fly during helmfile apply.

Generate an age key
Terminal window
age-keygen -o key.txt

This creates key.txt containing your private key and prints the public key (starting with age1...). Store key.txt somewhere safe (e.g. password manager, secure backup). Without it, you can’t decrypt your secrets.

Set the environment variable so SOPS can find your key:

Terminal window
export SOPS_AGE_KEY_FILE=$(pwd)/key.txt
Configure SOPS

Update .sops.yaml with your age public key:

creation_rules:
  - path_regex: secrets/.*\.yaml$
    age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Create the Cloudflare API Token

For certificate issuance, I chose DNS-01 challenges (supported by cert-manager) via Cloudflare. This lets us get a single wildcard certificate for all services and avoids some common routing issues associated with HTTP-01 challenges.

  1. Go to Cloudflare API Tokens
  2. Click Create Token → use the Edit zone DNS template
  3. Under Zone Resources, select your domain
  4. (Optional) I chose to set up Client IP Address Filtering, setting the Operator to “Is in” and adding my server’s public IP address to the Value list. This way, the API token only works from that one IP address.
  5. Click Continue to summary → Create Token
Fill in and encrypt the secrets files

Edit secrets/hetzner.yaml with your Hetzner API token:

hetzner_token: <your-hetzner-api-token>

Edit secrets/cluster-tls.yaml with your Cloudflare API token:

cloudflare:
  apiToken: <your-cloudflare-api-token>

Edit secrets/keila.yaml with generated passwords and your SMTP credentials:

secrets:
  postgresPassword: <generate-with-openssl-rand>
  secretKeyBase: <generate-with-openssl-rand>
  keilaUser: you@example.com
  keilaPassword: <generate-with-openssl-rand>
  mailerSmtpHost: <your-smtp-host>
  mailerSmtpPort: "587"
  mailerSmtpUser: <your-smtp-username>
  mailerSmtpPassword: <your-smtp-password>

You can generate random passwords with:

Terminal window
openssl rand -base64 24 | tr -d '/+='

Use a longer value for secretKeyBase — Keila requires at least 64 bytes:

Terminal window
openssl rand -base64 64 | tr -d '/+='

Encrypt all secrets files in place:

Terminal window
sops -e -i secrets/hetzner.yaml
sops -e -i secrets/cluster-tls.yaml
sops -e -i secrets/keila.yaml

The files are now safe to commit to git. To view or edit them later:

Terminal window
sops secrets/keila.yaml

6. Deploy Everything

A single command installs cert-manager, the wildcard TLS certificate, and Keila. Helmfile decrypts the secrets files on the fly via helm-secrets:

Terminal window
helmfile apply

This installs the releases in dependency order: cert-manager → ClusterIssuer + wildcard certificate → Keila.

7. Verify

Wait a minute or two for everything to start, then check:

Terminal window
# All pods should be Running
kubectl get pods -A


# Wildcard certificate should be Ready (may take a minute for DNS-01 validation)
kubectl get certificate -n default


# Check the ingress
kubectl get ingress -n keila

Open https://keila.k3s.andrewmarder.net in your browser and log in with the credentials from your secrets file. Although SMTP is configured at deploy time via secrets/keila.yaml, I still had to create a sender through the web interface.

8. Security

I’m hopeful the setup above covers the security basics. Here’s what’s in place:

  • Firewall. This setup allows SSH (22), HTTP (80), HTTPS (443), Kubernetes API (6443), and Kubernetes NodePorts (30000-32767). All other ports are blocked.
  • TLS everywhere. A wildcard certificate covers all *.k3s.andrewmarder.net subdomains.
  • Secrets encrypted at rest. All secrets are SOPS-encrypted with age and safe to commit to git.
  • No exposed databases. PostgreSQL is only reachable within the cluster via its ClusterIP service.
  • Registration disabled. DISABLE_REGISTRATION=true prevents strangers from creating Keila accounts.

If there’s additional hardening that’s important, please let me know!

9. Maintenance

Upgrading k3s

hetzner-k3s installs the System Upgrade Controller by default. To upgrade k3s, update k3s_version in cluster.yaml and run:

Terminal window
export HCLOUD_TOKEN=$(sops -d --extract '["hetzner_token"]' secrets/hetzner.yaml)
hetzner-k3s upgrade --config cluster.yaml
Updating Container Images

Update the pinned image versions in your values files (e.g. values/keila.yaml), then apply:

Terminal window
helmfile apply
Syncing After Changes

Any time you modify helmfile.yaml, chart templates, or values files, run helmfile apply to reconcile the cluster state. Use helmfile diff first to preview changes.

10. Cost

ItemMonthly Cost
CX23 (2 vCPU, 4 GB RAM, 40 GB disk)$3.50
Public IPv4 address$0.60
Local storage (instead of block storage)$0.00
Let’s Encrypt certificates$0.00
Total$4.10

Prices are for the nbg1 (Nuremberg, Germany) location, excluding VAT. Check Hetzner’s pricing page for current rates. You can list available instance types with:

Terminal window
curl -H "Authorization: Bearer $HCLOUD_TOKEN" 'https://api.hetzner.cloud/v1/server_types'

11. Adding New Services

The wildcard DNS record and wildcard TLS certificate mean adding a new service requires no DNS or certificate changes. For any new app:

  1. Create a Helm chart in charts/<service>/ (or use a public chart). Include Secret templates so Helm manages Kubernetes Secrets.
  2. Add a values file at values/<service>.yaml (plain-text config)
  3. Add a secrets file at secrets/<service>.yaml, fill in sensitive values, and encrypt it:
Terminal window
sops -e -i secrets/my-new-service.yaml
  1. Add a release to helmfile.yaml:
  - name: my-new-service
    namespace: my-new-service
    createNamespace: true
    chart: ./charts/my-new-service
    needs:
      - cert-manager/cluster-tls
    values:
      - ./values/my-new-service.yaml
    secrets:
      - ./secrets/my-new-service.yaml
  1. Deploy:
Terminal window
helmfile apply

Every service chart should include an Ingress referencing the shared wildcard-tls secret.

References

Links to GitHub projects: