How to Capture Database Dumps on Staging PVCs

Some applications need a point-in-time dump instead of a live volume snapshot. This pattern creates a staging PersistentVolumeClaim (PVC) in backupPrehook and writes the dump to it. It then declares the PVC for late capture, so Kasten captures it in the restore point.

This procedure uses a PostgreSQL StatefulSet as a concrete example. The dump and replay commands are specific to PostgreSQL. Adapt them to your application's tools.

Prerequisites

• A blueprint bound to the workload through an annotation or a BlueprintBinding.
• A CSI storage class that supports volume snapshots and the Filesystem volume mode. The cluster default applies unless you set one in step 1.
• A backup policy whose resource filters exclude the application's own data volume. The dump captures that data, so a snapshot of the volume is redundant.
• Database credentials the blueprint can read. This example reads a Secret named after the StatefulSet.
• A location profile for Kanister actions on the policy. This blueprint passes artifacts between actions, which requires one.

Procedure

The following steps build one blueprint. For the assembled result, see Complete Blueprint.

1. In backupPrehook, declare a lateCapture output artifact and create the staging PVC with a deterministic name.

   actions:
     backupPrehook:
       kind: StatefulSet
       outputArtifacts:
         lateCapture:
           keyValue:
             apiVersion: "{{ .Phases.createStagingPVC.Output.apiVersion }}"
             group: "{{ .Phases.createStagingPVC.Output.group }}"
             resource: "{{ .Phases.createStagingPVC.Output.resource }}"
             name: "{{ .Phases.createStagingPVC.Output.name }}"
             namespace: "{{ .Phases.createStagingPVC.Output.namespace }}"
       phases:
         - func: KubeOps
           name: createStagingPVC
           args:
             operation: create
             namespace: "{{ .StatefulSet.Namespace }}"
             spec: |-
               apiVersion: v1
               kind: PersistentVolumeClaim
               metadata:
                 name: "{{ .StatefulSet.Name }}-dump"
                 namespace: "{{ .StatefulSet.Namespace }}"
               spec:
                 accessModes: ["ReadWriteOnce"]
                 resources:
                   requests:
                     storage: 10Gi

   This PersistentVolumeClaim omits storageClassName, so it uses the cluster default. Set storageClassName when you need a specific class. Size the storage request to hold the uncompressed dump, which is roughly the logical size of the database.

   Prefer a deterministic name, such as {{ .StatefulSet.Name }}-dump. backupPosthook can then delete the PVC by name. If a crashed backup leaves the PVC behind, Kasten fails the next backup, so the leftover cannot go unnoticed. If your use case requires a generated name, see Dynamic PVC Names.

2. In the same action, use PrepareData to write the dump to the staging PVC and flush it to disk.

         - func: PrepareData
           name: dumpDatabase
           objects:
             pgSecret:
               kind: Secret
               name: "{{ .StatefulSet.Name }}"
               namespace: "{{ .StatefulSet.Namespace }}"
           args:
             namespace: "{{ .StatefulSet.Namespace }}"
             image: postgres:16
             volumes:
               "{{ .Phases.createStagingPVC.Output.name }}": /backup
             command:
               - bash
               - -c
               - |
                 export PGPASSWORD='{{ index .Phases.dumpDatabase.Secrets.pgSecret.Data "postgres-password" | toString }}'
                 pg_dumpall --clean --if-exists -h "{{ .StatefulSet.Name }}" -U postgres > /backup/full.sql
                 sync
             failOnError: true

   The sync call flushes the page cache before Kasten snapshots the PVC. Without it, the snapshot can capture a 0-byte file.

   The --clean --if-exists flags drop and recreate each object on replay. A restore then replaces the database contents and does not add to them. The result is the same whether you restore to the original namespace or a new one.

3. In backupPosthook, delete the staging PVC by name. Kasten already snapshotted it after step 2, so the data is safe.

     backupPosthook:
       kind: StatefulSet
       phases:
         - func: KubeOps
           name: deleteStagingPVC
           args:
             operation: delete
             objectReference:
               apiVersion: v1
               resource: persistentvolumeclaims
               name: "{{ .StatefulSet.Name }}-dump"
               namespace: "{{ .StatefulSet.Namespace }}"

   The backupPosthook must delete the staging PVC. A surviving PVC fails the backup. See Staging PVC Requirements.

4. In restorePosthook, declare the artifact as an input, mount the restored staging PVC and replay the dump.

     restorePosthook:
       kind: StatefulSet
       inputArtifactNames:
         - lateCapture
       phases:
         - func: PrepareData
           name: replayDump
           objects:
             pgSecret:
               kind: Secret
               name: "{{ .StatefulSet.Name }}"
               namespace: "{{ .StatefulSet.Namespace }}"
           args:
             namespace: "{{ .StatefulSet.Namespace }}"
             image: postgres:16
             volumes:
               "{{ .ArtifactsIn.lateCapture.KeyValue.name }}": /restore
             command:
               - bash
               - -c
               - |
                 export PGPASSWORD='{{ index .Phases.replayDump.Secrets.pgSecret.Data "postgres-password" | toString }}'
                 psql -h "{{ .StatefulSet.Name }}" -U postgres -f /restore/full.sql
             failOnError: true
   
         - func: KubeOps
           name: removeRestoredPVC
           args:
             operation: delete
             objectReference:
               apiVersion: v1
               resource: persistentvolumeclaims
               name: "{{ .ArtifactsIn.lateCapture.KeyValue.name }}"
               namespace: "{{ .StatefulSet.Namespace }}"

Verification

1. Run a backup and open the restore point details. Confirm that the staging PVC appears with a volume snapshot, like any other captured PVC.
2. Run kubectl get pvc -n <namespace>. Confirm the staging PVC is gone from the namespace.
3. Run a restore. Confirm the database contains the dumped data. Confirm the restored staging PVC is gone from the namespace.

Dynamic PVC Names

When your use case requires it, Kasten supports a generated name, such as one with a timestamp. Kasten fails the next backup only when a leftover PVC has the same name. A leftover PVC with a different name can go undetected. Kasten discovers and snapshots it like any other PVC until you delete it, which consumes storage unnecessarily.

Because the hook cannot reconstruct a generated name, add a label to the PVC so that backupPosthook can find it. Add a workload-scoping label as well when several workloads in the namespace each create one. Delete by the labels you set:

command:
  - bash
  - -c
  - |
    kubectl delete pvc -n "{{ .StatefulSet.Namespace }}" \
      -l example.com/staging-pvc=true,example.com/workload={{ .StatefulSet.Name }} \
      --ignore-not-found

Complete Blueprint

The following blueprint combines every step in this procedure.

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: postgres-staging-pvc
  namespace: kasten-io
actions:
  backupPrehook:
    kind: StatefulSet
    outputArtifacts:
      lateCapture:
        keyValue:
          apiVersion: "{{ .Phases.createStagingPVC.Output.apiVersion }}"
          group: "{{ .Phases.createStagingPVC.Output.group }}"
          resource: "{{ .Phases.createStagingPVC.Output.resource }}"
          name: "{{ .Phases.createStagingPVC.Output.name }}"
          namespace: "{{ .Phases.createStagingPVC.Output.namespace }}"
    phases:
      - func: KubeOps
        name: createStagingPVC
        args:
          operation: create
          namespace: "{{ .StatefulSet.Namespace }}"
          spec: |-
            apiVersion: v1
            kind: PersistentVolumeClaim
            metadata:
              name: "{{ .StatefulSet.Name }}-dump"
              namespace: "{{ .StatefulSet.Namespace }}"
            spec:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 10Gi

      - func: PrepareData
        name: dumpDatabase
        objects:
          pgSecret:
            kind: Secret
            name: "{{ .StatefulSet.Name }}"
            namespace: "{{ .StatefulSet.Namespace }}"
        args:
          namespace: "{{ .StatefulSet.Namespace }}"
          image: postgres:16
          volumes:
            "{{ .Phases.createStagingPVC.Output.name }}": /backup
          command:
            - bash
            - -c
            - |
              export PGPASSWORD='{{ index .Phases.dumpDatabase.Secrets.pgSecret.Data "postgres-password" | toString }}'
              pg_dumpall --clean --if-exists -h "{{ .StatefulSet.Name }}" -U postgres > /backup/full.sql
              sync
          failOnError: true

  backupPosthook:
    kind: StatefulSet
    phases:
      - func: KubeOps
        name: deleteStagingPVC
        args:
          operation: delete
          objectReference:
            apiVersion: v1
            resource: persistentvolumeclaims
            name: "{{ .StatefulSet.Name }}-dump"
            namespace: "{{ .StatefulSet.Namespace }}"

  restorePosthook:
    kind: StatefulSet
    inputArtifactNames:
      - lateCapture
    phases:
      - func: PrepareData
        name: replayDump
        objects:
          pgSecret:
            kind: Secret
            name: "{{ .StatefulSet.Name }}"
            namespace: "{{ .StatefulSet.Namespace }}"
        args:
          namespace: "{{ .StatefulSet.Namespace }}"
          image: postgres:16
          volumes:
            "{{ .ArtifactsIn.lateCapture.KeyValue.name }}": /restore
          command:
            - bash
            - -c
            - |
              export PGPASSWORD='{{ index .Phases.replayDump.Secrets.pgSecret.Data "postgres-password" | toString }}'
              psql -h "{{ .StatefulSet.Name }}" -U postgres -f /restore/full.sql
          failOnError: true

      - func: KubeOps
        name: removeRestoredPVC
        args:
          operation: delete
          objectReference:
            apiVersion: v1
            resource: persistentvolumeclaims
            name: "{{ .ArtifactsIn.lateCapture.KeyValue.name }}"
            namespace: "{{ .StatefulSet.Namespace }}"

---

See Late Capture for the full declaration contract. See Phase Lifecycle Reference for where late capture runs in the backup sequence.