etcd Backup (OpenShift Container Platform)

Assuming the Kubernetes cluster is set up through OpenShift Container Platform, etcd is part of the control plane. Its data is encrypted at rest. Red Hat supports only one method for backing up etcd: the cluster-backup.sh script, run on a control-plane (master) node. It produces both the etcd snapshot (snapshot_*.db) and the static_kuberesources_*.tar.gz archive. This archive contains the encryption keys and static pod resources that go with it. A snapshot taken any other way, or without that archive, cannot be restored.

This page implements etcd backup using a small, permanent keeper pod:

• The keeper is pinned to a control-plane node and runs cluster-backup.sh, writing its output to a permanent PersistentVolumeClaim.
• Before Veeam Kasten captures the restore point, it triggers the keeper to create a fresh backup. It then captures that PVC as a CSI snapshot. The snapshot serves directly as the restore point, the same as for any other PVC.
• After Veeam Kasten takes the snapshot, it verifies the backup to confirm a successful capture.
• This blueprint requires no separate upload or external backup-storage step and performs no restore or cleanup action.

An etcd backup is not an OpenShift backup

Take an etcd snapshot for what it is, and don't mistake it for a recovery strategy:

• It captures no application data — no PVC contents, no object storage, only cluster API objects at a point in time. Your databases and files are not in it.
• Granular restore is impossible. etcd restore is all-or-nothing — it rolls the entire cluster back to the snapshot, not a single namespace, workload, or PVC.
• It is disruptive and dangerous. cluster-restore.sh takes the API server down and rebuilds the control plane on every master — a full-cluster disaster operation.

The recommended OpenShift recovery model:

1. Rebuild the cluster and its infrastructure operators with a solid GitOps process (ArgoCD/Flux) — declaratively, reproducibly, not from an etcd snapshot.
2. Restore the applications onto it with Veeam Kasten — granular, non-disruptive, and with their PVC data and object storage. This is what etcd can never do.

Veeam Kasten can also back up the infrastructure-operator namespaces themselves. Treat that as a fail-over convenience, not the primary path (GitOps remains the source of truth for the platform).

Why use this blueprint? etcd backups still have their place: legacy practices or compliance requirements that mandate them, and forensics. Forensics means inspecting cluster state at a past point in time. This blueprint makes those etcd snapshots safe, scheduled, and snapshot-managed by Veeam Kasten.

Prerequisites

1. The VolumeSnapshotClass for your CSI driver must be annotated for Veeam Kasten, as described in Storage. Do not use a legacy in-tree storage class (for example gp2) — it does not support CSI snapshots.

2. The keeper pod mounts the control-plane node's /etc/kubernetes directory and runs as root with seLinuxOptions.type: spc_t, which requires the privileged Security Context Constraint. Granting it is part of the deploy step below and needs cluster-admin privileges.

3. The keeper image bundles a fixed etcdctl/etcdutl and kubectl. Make sure the image you use matches your cluster's etcd and Kubernetes versions. cluster-backup.sh skips downloading its own etcdctl when a compatible one is already on PATH. A version mismatch can also affect etcdutl snapshot status output.

Backup

Step 1: Deploy the Keeper

Create a file keeper.yaml with the following contents. It creates the etcd-backup namespace, a ServiceAccount, a ClusterRole / ClusterRoleBinding that lets the keeper read ClusterOperator status (used to confirm no control-plane rollout is in progress before a backup), the permanent PVC ocp-etcd-backup and the keeper Deployment itself, pinned to a control-plane node:

apiVersion: v1
kind: Namespace
metadata:
  name: etcd-backup
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: etcd-backup-keeper
  namespace: etcd-backup
---
# Allow the keeper to read ClusterOperator status, so the backupPrehook can verify
# no control-plane operator is mid-rollout before taking the etcd backup
# (the equivalent of cluster-backup.sh's own check, which we run with --force).
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: etcd-backup-keeper-co-reader
rules:
  - apiGroups: ["config.openshift.io"]
    resources: ["clusteroperators"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: etcd-backup-keeper-co-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: etcd-backup-keeper-co-reader
subjects:
  - kind: ServiceAccount
    name: etcd-backup-keeper
    namespace: etcd-backup
---
# Permanent backup PVC. Kasten snapshots THIS PVC; the snapshot is the restore
# point. It is also the volume an admin can inspect through the keeper pod.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ocp-etcd-backup
  namespace: etcd-backup
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ocp-etcd-backup-keeper
  namespace: etcd-backup
  labels:
    app: ocp-etcd-backup-keeper
    etcd-backup-keeper: "true"
spec:
  replicas: 1
  strategy:
    type: Recreate          # RWO PVC: never two pods mounting it at once
  selector:
    matchLabels:
      app: ocp-etcd-backup-keeper
  template:
    metadata:
      labels:
        app: ocp-etcd-backup-keeper
        etcd-backup-keeper: "true"
    spec:
      serviceAccountName: etcd-backup-keeper
      # Run on a control-plane node: cluster-backup.sh + the etcd static-pod
      # resources and certs only exist on the masters.
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-role.kubernetes.io/control-plane
                    operator: Exists
              - matchExpressions:
                  - key: node-role.kubernetes.io/master
                    operator: Exists
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      securityContext:
        seLinuxOptions:
          type: spc_t          # allow reads of host /etc/kubernetes under SELinux
        runAsUser: 0
        runAsGroup: 0
      containers:
        - name: keeper
          image: ghcr.io/kanisterio/ocp-etcd-backup:0.120.0
          imagePullPolicy: Always
          command: ["/bin/bash", "-c"]
          args: ["echo 'ocp-etcd-backup keeper ready'; sleep infinity"]
          volumeMounts:
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
            - name: backup
              mountPath: /backup
      volumes:
        - name: etc-kubernetes
          hostPath:
            path: /etc/kubernetes
            type: Directory
        - name: backup
          persistentVolumeClaim:
            claimName: ocp-etcd-backup

Note

The manifest above uses the ebs-sc storage class as an example. Replace it with a CSI storage class that supports snapshots on your cluster (for example managed-csi on Azure, standard-rwo on GKE), and size the PVC to comfortably fit your cluster's etcd snapshot.

Apply it, then grant the privileged SCC and restart the keeper so the pod is admitted with it:

$ oc apply -f keeper.yaml

$ oc adm policy add-scc-to-user privileged -z etcd-backup-keeper -n etcd-backup
$ oc label ns etcd-backup \
      pod-security.kubernetes.io/enforce=privileged \
      pod-security.kubernetes.io/warn=privileged \
      pod-security.kubernetes.io/audit=privileged --overwrite

$ oc rollout restart deployment/ocp-etcd-backup-keeper -n etcd-backup
$ oc rollout status  deployment/ocp-etcd-backup-keeper -n etcd-backup

Confirm the keeper is running on a master and can see the host script:

$ POD=$(oc get pods -n etcd-backup -l app=ocp-etcd-backup-keeper -ojsonpath='{.items[0].metadata.name}')
$ oc exec -n etcd-backup $POD -- \
    ls -l /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/cluster-backup.sh

Step 2: Deploy the Blueprint and BlueprintBinding

Create a file ocp-etcd-backup-bp.yaml with the following contents:

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: ocp-etcd-backup
  namespace: kasten-io
actions:
  backupPrehook:
    phases:
    - func: KubeExec
      name: etcdBackup
      args:
        namespace: "{{ .Deployment.Namespace }}"
        pod: "{{ index .Deployment.Pods 0 }}"
        container: keeper
        command:
        - /bin/bash
        - -c
        - |
          set -euo pipefail

          BACKUP_DIR=/backup/etcd-backup
          HOST_SCRIPT=/etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/cluster-backup.sh

          echo "=== OpenShift etcd backup pre-hook ==="
          echo "Node: $(cat /etc/hostname 2>/dev/null || echo unknown)"

          if [ ! -x "$HOST_SCRIPT" ]; then
            echo "ERROR: $HOST_SCRIPT not found. Is the keeper running on a control-plane node"
            echo "       with host /etc/kubernetes mounted read-only?"
            exit 1
          fi

          # Operator-progression safety check.
          # A reliable etcd backup requires that no control-plane rollout is in
          # progress, otherwise the captured static-pod resources may be inconsistent
          # with etcd. cluster-backup.sh enforces this itself ONLY when run without
          # --force, using `oc` + the node localhost.kubeconfig (server
          # https://localhost:6443) which this non-hostNetwork pod cannot reach.
          # So we run --force below and perform the equivalent check here against the
          # in-cluster API with kubectl + the keeper ServiceAccount token (which has
          # get/list on clusteroperators) — exactly as the upstream etcd-bp-azcopy
          # blueprint does in its checkOperatorProgression phase.
          echo "Checking control-plane operators are not progressing ..."
          for op in kube-apiserver kube-controller-manager kube-scheduler etcd; do
            status=$(kubectl get co "$op" -o jsonpath='{.status.conditions[?(@.type=="Progressing")].status}' 2>/dev/null || true)
            echo "  $op: Progressing=${status:-<unknown>}"
            if [ -z "$status" ]; then
              echo "ERROR: could not read Progressing status for $op. Check kubectl access / RBAC. Aborting!"
              exit 1
            fi
            if [ "$status" != "False" ]; then
              echo "ERROR: $op is progressing (rollout in flight). A reliable backup requires no rollout in progress. Aborting!"
              exit 1
            fi
          done
          echo "All control-plane operators are stable."

          # Idempotent: start each run from a clean backup dir so the restore point
          # contains exactly one consistent snapshot.
          rm -rf "$BACKUP_DIR"
          mkdir -p "$BACKUP_DIR"

          # --force because the script's built-in operator-progression check (above)
          # is unreachable from this pod; we already performed the equivalent check.
          # --force also marks output filenames __POSSIBLY_DIRTY__ (normalized below).
          echo "Running cluster-backup.sh --force ..."
          "$HOST_SCRIPT" --force "$BACKUP_DIR"

          # Normalize the __POSSIBLY_DIRTY__ marker out of the filenames.
          echo "Normalizing snapshot file names ..."
          cd "$BACKUP_DIR"
          shopt -s nullglob
          for f in *__POSSIBLY_DIRTY__*; do mv "$f" "${f/__POSSIBLY_DIRTY__/}"; done
          shopt -u nullglob

          echo "Backup directory contents:"
          ls -lh "$BACKUP_DIR"

          # EBS/Azure-disk snapshots are block-device snapshots that bypass the OS
          # page cache. Flush to disk so the 89 MB snapshot .db is fully persisted
          # before Kasten takes the PVC snapshot.
          echo "Flushing page cache (sync) ..."
          sync
          echo "etcd backup pre-hook completed."

  backupPosthook:
    phases:
    - func: KubeExec
      name: etcdVerify
      args:
        namespace: "{{ .Deployment.Namespace }}"
        pod: "{{ index .Deployment.Pods 0 }}"
        container: keeper
        command:
        - /bin/bash
        - -c
        - |
          set -euo pipefail
          BACKUP_DIR=/backup/etcd-backup

          echo "=== OpenShift etcd backup post-hook: verifying snapshot ==="
          SNAP=$(ls "$BACKUP_DIR"/snapshot_*.db 2>/dev/null | head -1 || true)
          if [ -z "$SNAP" ] || [ ! -s "$SNAP" ]; then
            echo "ERROR: no non-empty snapshot_*.db found in $BACKUP_DIR"
            ls -lh "$BACKUP_DIR" || true
            exit 1
          fi

          echo "Snapshot file: $SNAP ($(du -h "$SNAP" | cut -f1))"
          etcdutl snapshot status "$SNAP" -w table

          TARBALL=$(ls "$BACKUP_DIR"/static_kuberesources_*.tar.gz 2>/dev/null | head -1 || true)
          if [ -z "$TARBALL" ] || [ ! -s "$TARBALL" ]; then
            echo "ERROR: no static_kuberesources_*.tar.gz found in $BACKUP_DIR"
            exit 1
          fi
          echo "Static resources: $TARBALL ($(du -h "$TARBALL" | cut -f1))"
          echo "etcd backup post-hook verification passed."

  # NOTE: there is intentionally NO restore action.

Action	Function	What it does
backupPrehook	KubeExec (keeper)	Confirms kube-apiserver, kube-controller-manager, kube-scheduler and etcd are not Progressing, then runs cluster-backup.sh --force into the PVC and flushes it to disk.
backupPosthook	KubeExec (keeper)	Verifies a non-empty snapshot_*.db and static_kuberesources_*.tar.gz both exist, and runs etcdutl snapshot status on the snapshot.

There is intentionally no restore or delete action:

• No restore* action — restoring the keeper PVC does not require any preparation. The actual etcd restore is a manual control-plane procedure and should not be triggered automatically through a hook.

• No delete action — deleting the restore point also removes the underlying PVC snapshot, which contains the backup. Therefore, no additional backup data or resource needs cleanup.

Note

backupPrehook aborts the backup if any of the four control-plane operators is still Progressing. This is expected, not a sign that the blueprint is broken. Retry the backup once the rollout in progress completes.

Create a file ocp-etcd-bp-binding.yaml with the following contents to bind the Blueprint to the keeper Deployment:

apiVersion: config.kio.kasten.io/v1alpha1
kind: BlueprintBinding
metadata:
  name: ocp-etcd-backup-binding
  namespace: kasten-io
spec:
  blueprintRef:
    name: ocp-etcd-backup
    namespace: kasten-io
  resources:
    matchAll:
      # Bind to keeper Deployments labelled etcd-backup-keeper: "true".
      - type:
          operator: In
          values:
            - group: apps
              resource: deployments
      - labels:
          key: etcd-backup-keeper
          operator: In
          values:
            - "true"
      # Let an explicit kanister.kasten.io/blueprint annotation override this binding.
      - annotations:
          key: kanister.kasten.io/blueprint
          operator: DoesNotExist

$ oc --namespace kasten-io apply -f ocp-etcd-backup-bp.yaml
$ oc --namespace kasten-io apply -f ocp-etcd-bp-binding.yaml

Alternatively, use the Blueprints and Blueprint Bindings pages on the Veeam Kasten Dashboard to create these resources.

Step 3: Back up with Veeam Kasten

Create a policy targeting the etcd-backup namespace with a Location profile. An Infra profile is not enough — restore extracts the profile from the restore point automatically. Run the policy from the Dashboard or with:

$ oc apply -f - <<'EOF'
apiVersion: config.kio.kasten.io/v1alpha1
kind: Policy
metadata:
  name: etcd-backup-policy
  namespace: kasten-io
spec:
  comment: "OpenShift etcd backup via keeper blueprint"
  frequency: "@onDemand"
  actions:
    - action: backup
      backupParameters:
        profile:
          name: <YOUR_LOCATION_PROFILE>
          namespace: kasten-io
  selector:
    matchExpressions:
      - key: k10.kasten.io/appNamespace
        operator: In
        values: [etcd-backup]
EOF

$ oc create -f - <<'EOF'
apiVersion: actions.kio.kasten.io/v1alpha1
kind: RunAction
metadata:
  generateName: run-etcd-
  namespace: kasten-io
spec:
  subject:
    apiVersion: config.kio.kasten.io/v1alpha1
    kind: Policy
    name: etcd-backup-policy
    namespace: kasten-io
EOF

Confirm the hooks were fired:

$ oc get runaction -n kasten-io
$ oc logs -n kasten-io deploy/kanister-svc --tail=200 | grep -E "etcdBackup|etcdVerify"
$ oc get restorepoint -n etcd-backup

Restore

info

Two different things are called "restore" here, and Veeam Kasten automates only the first:

1. Restoring the backup artifact — a Veeam Kasten restore of the etcd-backup namespace that brings back the keeper PVC and Deployment with snapshot_*.db and static_kuberesources_*.tar.gz. This is what the steps below do.
2. Restoring the control-plane etcd onto the cluster means rolling the entire cluster back to that snapshot, using OpenShift's own Restoring to a previous cluster state procedure (cluster-restore.sh on a chosen master, then recovery of the remaining masters). It takes the API server down, and an administrator must drive it by hand — this blueprint deliberately has no restore action for this step.

1. Restore the Backup Artifact with Veeam Kasten

$ RP=$(oc get restorepoint -n etcd-backup -ojsonpath='{.items[0].metadata.name}')
$ oc create -f - <<EOF
apiVersion: actions.kio.kasten.io/v1alpha1
kind: RestoreAction
metadata:
  generateName: restore-etcd-
  namespace: etcd-backup
spec:
  subject:
    apiVersion: apps.kio.kasten.io/v1alpha1
    kind: RestorePoint
    name: $RP
    namespace: etcd-backup
  targetNamespace: etcd-backup
EOF

Or use the Dashboard to restore the etcd-backup namespace to itself. No profile needs to be set on the RestoreAction — Veeam Kasten extracts the Location profile from the restore point automatically.

Verify the artifact came back:

$ POD=$(oc get pods -n etcd-backup -l app=ocp-etcd-backup-keeper --field-selector=status.phase=Running -ojsonpath='{.items[0].metadata.name}')
$ oc exec -n etcd-backup $POD -- bash -c \
    'ls -lh /backup/etcd-backup; etcdutl snapshot status /backup/etcd-backup/snapshot_*.db -w table'

2. Restore the control-plane etcd onto the cluster (manual)

Before doing this, re-read the warning above about an etcd backup not being an OpenShift backup — this is a full-cluster disaster operation, and in most situations rebuilding via GitOps and restoring applications with Veeam Kasten is the right answer instead of rolling etcd back.

This blueprint does not automate this step. Follow Red Hat's documented, version-specific procedure, using the artifacts recovered above:

1. Copy snapshot_*.db and static_kuberesources_*.tar.gz off the keeper pod onto a chosen recovery master (for example under /home/core/assets/backup/):

   $ oc cp etcd-backup/$POD:/backup/etcd-backup ./etcd-backup-artifact

2. On the recovery master (via oc debug node/<master>, then chroot /host), run /usr/local/bin/cluster-restore.sh against the copied backup directory, as described in Restoring to a previous cluster state.

3. Restart the kubelet service and recover the remaining control-plane nodes as described in the same OpenShift documentation, then re-validate that all cluster operators report AllNodesAtLatestRevision.