SUSE Virtualization (Harvester) VM Backup and Restore Support Limitations

This document outlines the backup and restore support limitations for SUSE Virtualization Virtual Machines (VMs) in Veeam Kasten. Certain conditions should be kept in mind before initiating the backup or restore process:

1. DHCP-identifier

If using DHCP for network connectivity, the network interface configuration for VMs should have DHCP identifier configured to use MAC address, as otherwise the SUSE Virtualization DHCP server may fail to assign an IP address to the VM after it is restored via Kasten. An example of the cloud-init configuration is shown below:

    cloud-user@customer-test-vm:~$ sudo cat /etc/netplan/50-cloud-init.yaml
    network:
        ethernets:
            enp1s0:
                dhcp4: true
                dhcp-identifier: mac
        version: 2

Note

MAC addresses of virtual machines can be preserved during restore operations to maintain the original network identity. This helps ensure proper DHCP assignment after restore. For more details, see Preserving MAC Addresses During Restore.

2. QEMU Guest Agent

The QEMU guest agent should be activated within your VM to ensure the consistent backup of data. The guest agent can halt write functions during the backup procedure, thus preventing data manipulation or loss. For more information, refer to the Harvester instruction installing the QEMU guest agent.

3. Volume Snapshot Class Annotation

For successful operations with Harvester VMs, users need to annotate the Volume Snapshot Class (VSC) with a specific annotation. Currently, the support is limited to VSCs having the "parameters.type.snap" (same as the default "longhorn-snapshot"). One of its limitations is that users cannot restore the VM using a "local" restore point. Only 'Import and Restore' are feasible - this implies each Veeam Kasten VM backup needs to be exported to a location specified in a "Location Profile" before a VM restoration.

4. Backup of SSH Key Objects

If "SSH Key" objects - represented as Kubernetes Custom Resource Definition named "KeyPair" need to be backed up, additional Veeam Kasten policy configuration may be necessary. This is because "KeyPair" objects may be located in a different Kubernetes namespace than that of the VM.

5. Block Mode Export of Encrypted VM Images and VM Image Volumes

Currently, block mode export of encrypted/decrypted VM images and volumes created from such images is not supported.

6. VM Restore with Cloud-Init Volumes Referencing External Secrets

VM restore operations may fail when the VM specification uses a cloudInitNoCloud volume that references external secrets or ConfigMaps. During the restore process, scaling down the VM can delete the associated secrets, causing them to be missing when the VM is scaled back up, even if they were previously restored.

Example VM specification that may be affected:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: test-vm
  namespace: test-namespace
spec:
  runStrategy: RerunOnFailure
  template:
    spec:
      volumes:
      - name: disk-0
        persistentVolumeClaim:
          claimName: test-vm-disk-0-v5lau
      - cloudInitNoCloud:
          userDataSecretRef:
            name: test-vm-secret
          networkDataSecretRef:
            name: test-vm-secret
        name: cloudinitdisk

This issue is particularly evident when restoring into a namespace that already contains a running VM referencing the same secret. The VM will fail to boot with errors such as:

Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Warning  FailedScheduling        10m                  default-scheduler        0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled               10m                  default-scheduler        Successfully assigned test/virt-launcher-test-s4n6g to harvester-131-ci-1
  Normal   SuccessfulAttachVolume  9m56s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-28ad74fd-7ff5-4f30-826d-524271d04ed5"
  Normal   SuccessfulMountVolume   9m55s                kubelet                  MapVolume.MapPodDevice succeeded for volume "pvc-28ad74fd-7ff5-4f30-826d-524271d04ed5" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-28ad74fd-7ff5-4f30-826d-524271d04ed5/dev"
  Normal   SuccessfulMountVolume   9m55s                kubelet                  MapVolume.MapPodDevice succeeded for volume "pvc-28ad74fd-7ff5-4f30-826d-524271d04ed5" volumeMapPath "/var/lib/kubelet/pods/9d10849a-c1f2-4606-90ed-14ef4bb49961/volumeDevices/kubernetes.io~csi"
  Warning  FailedMount             112s (x12 over 10m)  kubelet                  MountVolume.SetUp failed for volume "cloudinitdisk-udata" : secret "test-vm-secret" not found
  Warning  FailedMount             112s (x12 over 10m)  kubelet                  MountVolume.SetUp failed for volume "cloudinitdisk-ndata" : secret "test-vm-secret" not found

Workarounds:

1. Manually recreate secrets or ConfigMaps: Ensure that any secrets or ConfigMaps referenced by cloudInitNoCloud volumes are manually recreated after the VM restore operation if they are missing.

2. Use inline cloud-init data: Alternatively, use inline cloud-init data instead of external secrets by specifying userData and networkData fields directly in the VM specification:

- cloudInitNoCloud:
    userData: |
      #cloud-config
      user: cloud-user
      password: mypassword
      ssh_authorized_keys:
        - ssh-rsa AAAAB3...
    networkData: |
      version: 2
      ethernets:
        enp1s0:
          dhcp4: true
          dhcp-identifier: mac
  name: cloudinitdisk