Immutable Backups Workflow

Veeam Kasten can leverage the object-locking capability available in object stores to make backups immutable. This guards against catastrophic disaster scenarios such as ransomware attacks and allows recovering the backups in those situations.

This feature is currently available for use with Azure, AWS S3 and any S3-compatible object store that supports object locking.

Warning

The generic storage and shareable volume backup and restore workflows are not compatible with the protections afforded by immutable backups. Use of a location profile enabled for immutable backups can be used for backup and restore, but the protection period is ignored, and the profile is treated as a non-immutability-enabled location. Please note that using an object-locking bucket for such use cases can amplify storage usage without any additional benefit. Please contact support for any inquiries.

Disaster Scenarios

Vulnerabilities can arise from many sources, such as lack of privilege separation due to credentials with permissive access, or from sophisticated attacks.

Consider a comprehensive breach of all secured systems in a Kubernetes cluster and ancillary infrastructure. Assume that a malicious agent has compromised all of the following:

  • the Kubernetes cluster - can inspect and control applications running in all namespaces, read secrets, manipulate snapshots, and locate backups.

  • the object store - can tamper with or destroy application backup data that had been exported by Veeam Kasten.

  • the Veeam Kasten deployment - can force retirement of Veeam Kasten snapshots and backups, compelling Veeam Kasten to delete the associated data and metadata, including application backups and Veeam Kasten Disaster Recovery self-backups.

  • the production application - can tamper with or encrypt vital data, demanding a ransom in exchange for resumed access.

In such a sophisticated scenario, an attacker might attempt to render restores from backups useless, before encrypting live application data and demanding a ransom.

Veeam Kasten Immutable Backups

In the face of such a comprehensive attack, Veeam Kasten has the ability to turn back the clock.

Veeam Kasten is capable of exporting backups to object store containers with object locking enabled. Doing so renders the data written there immutable. Even users with administrative privileges are prevented from deleting or tampering with the backup data. Each repository blob is immutable and secure for an extendable period of time.

Veeam Kasten uses these immutable blobs to go back in time, and retrieve the backups as they were prior to the time of an attack.

Protection Period

How far back in time Veeam Kasten can see is a tunable parameter called the protection period. The protection period is chosen by the user when creating a profile with immutable backups enabled.

The protection period represents the estimated amount of time required by your organization to recover following impact. Recovery must begin within that protection period window in order to guarantee a successful restoration from immutable backups. The longer the protection period, the more time can be afforded for the recovery from an impact event like ransomware.

The protection period can be any length of time greater than or equal to 1 day, with a granularity of days. Longer protection periods may have increased storage costs associated with them, as object versions must be kept around longer.

Active Monitoring

Veeam Kasten has a background service that monitors repositories containing immutable backups. Each time a restore point is exported to a repository in a locked bucket, some new blobs may be written, and some blobs may be reused. Therefore the retain-until date applied to each blob (the property that renders it immutable) may need to be refreshed, or pushed out to a later date, as time passes. This is because Veeam Kasten's backup solution makes use of incremental backups.

Once an immutable backups repository is created, Veeam Kasten will begin periodically querying its data blobs in the background. The frequency of this operation is determined by the chosen protection period; a longer protection period doesn't require checking as frequently, because the retain-until date can be pushed out further when needed. That said, the longest the service will wait without performing a refresh check is 2 days. This will only run on the cluster which made the original export; if a restore point is imported into another cluster, that cluster will not extend the protection period on the imported restore points.

In order for this background operation to run smoothly, you must maintain a profile referencing the location containing that repository. Additionally that profile must follow the criteria perscribed for an immutable backups profile.

An alert notification will appear in the upper right corner of the Veeam Kasten dashboard when any of the following conditions are encountered:

  • the profile no longer meets these criteria

  • the background monitoring service has trouble connecting to the store

  • the background monitoring service is otherwise unable to perform the refresh procedure

    ../_images/immutable_backups_alert_icon.png

Clicking the alert icon will open a side-pane containing a list of outstanding errors and warnings, each with a description of what went wrong. If you are unable to determine the appropriate fix from the information provided, please refer to Kasten Veeam Kasten Community Support or open a case via https://my.veeam.com.

../_images/immutable_backups_alert_side_pane.png

Comprehensive Protection

Individual applications can be protected with immutable backups by exporting to a location with object locking enabled. This ensures those application restore points are unable to be tampered with, and that Veeam Kasten knows how to access them at the appropriate points in time.

However if the cluster has been completely compromised, Veeam Kasten may also be susceptible to tampering. For example if an attacker manually retires all restore points, from Veeam Kasten's perspective, those backups no longer exist.

To comprehensively secure an application's backups even in the face of an attack on Veeam Kasten, it is necessary to activate a Veeam Kasten Disaster Recovery policy that also makes use of immutable backups. This means that all backups of Veeam Kasten itself are also stored immutably in an object-locked object store location.

By combining immutable backup policy runs for an application with a subsequent immutable Disaster Recovery policy run for Veeam Kasten, you preserve both the application's data and the ability to restore it.

Each time the Disaster Recovery policy runs, it will "lock in" the ability to recover from any of the active restore points at that point in time. Multiple applications may be simultaneously protected by different policies with differing scheduling frequencies. Therefore it is recommended to schedule the Disaster Recovery policy to run at least as frequently, if not more so, than the most frequent policy schedule that performs immutable backups. Additionally, the protection period chosen for the Disaster Recovery profile should be at least as long as the longest protection period in use by an immutable backups policy.

Setting Up Immutable Backup Protection for S3

Begin by setting up one or more location profiles, each pointing to an object store destination with object-locking enabled.

The selected object store destination must meet certain criteria:

  • Must be a bucket on AWS S3 or an S3-compatible object store.

  • Must be reachable with the credentials provided in the profile form.

  • Must already exist.

  • The region provided on the profile form must match the region in which the bucket resides.

  • Must have object locking enabled. Note: On some S3-compatible implementations, the object locking property of a bucket can only be enabled/configured at bucket creation time.

Select the checkbox for "Enable Immutable Backups".

../_images/enable_backups_button_checked_pre_validate.png

Click the "Validate Bucket" button. This will initiate a pre-flight check of the above criteria.

If all of the checks succeed, a slider bar will appear for selecting the desired protection period. The bounds of this bar are from 1 day to an upper value of 90 days.

../_images/enable_backups_button_checked_post_validate.png

Click the "Save Profile" button to submit this profile configuration.

Warning

Enabling immutability protection for an existing profile will only be effective if the backups have not been compromised prior to activation.

Profiles that reference object-locked object store locations and have immutable backups enabled will display this on its respective profile card: a field indicating "Object Immutability" is "enabled", and the chosen protection period.

../_images/profile_view_with_immutable_backups.png

Next follow the standard Backups workflow, selecting one of the immutable backup profiles for each policy. This will protect each covered application with immutable backups for all policy runs.

Finally follow the Veeam Kasten Disaster Recovery workflow, selecting the desired immutable backup profile for the "Cloud Location Profile" in the DR form.

Setting Up Immutable Backup Protection for Azure

To set up immutability in Azure take into account the following requirements:

  • Ensure that the container exists and is reachable with the credentials provided in the profile form.

  • Enable versioning on the related storage account.

  • Ensure support for version-level immutability on the container or related storage account.

Since Veeam Kasten ignores retention policies, it is not necessary to set one on the container. As an alternative, choose the desired protection period, and the files will be initially protected for that amount plus a safety buffer to ensure protection compliance.

Setting Up Immutable Backup Protection for Google

To set up immutability in Google take into account the following requirements:

  • Ensure that the bucket exists and is reachable with the credentials provided in the profile form.

  • Enable object versioning on the bucket.

  • Enable object retention lock on the bucket.

  • If using minimal permissions with the credentials, storage.objects.setRetention permission is also required.

Setting Up Immutable Backup Protection Using Veeam Repository

To use the Veeam Kasten backup immutability feature with VBR, make sure that immutability settings are configured consistently as follows:

  • Set up the immutability window length in VBR for the desired immutability period.

  • Configure a proper protected period in Veeam Kasten. This period must be less than or equal to the immutability window in VBR.

  • Adjust Veeam Kasten backup frequency and retention settings to prevent the deletion restore points inside the VBR immutability window. Failure to do so may lead to orphaned restore points that will not be automatically retired.

Note

The integrity of restore points outside the immutability window in VBR is not guaranteed. It is recommended to set up the retention policy to automatically delete such restore points when they are outside the immutability window.

This approach allows you to keep only the N last backups according to the chosen backup frequency. If you need to keep, for example, N daily backups and M yearly backups, create several policies with different backup frequencies and retention settings.

Example

A customer wants to guarantee the immutability for 30 days for their daily backups:

  • The immutability period for VBR must be set to 30 days

  • The protection period for Veeam Kasten must be set to 30 days (or lower)

The configuration of the S3 bucket is beyond the scope of this example. We assume that it has already been properly configured.

Setting up the Immutability Window Length in VBR

To set up the immutability window length in VBR, open the Repository settings and enter the desired value in the "Make recent backups immutable for" field:

../_images/vbr_immutability_02.png

Note

Allowed value for immutability window in VBR is 7 to 9999 days.

Here, we set it to 30 days because the customer wants to guarantee immutability for this period.

Additional information about immutable repositories can be found in the VBR documentation.

Setting up the Immutability Window Length in Veeam Kasten

The Veeam Kasten protection period should be set to any value up to and including 30 days, but not more, to align with the specified requirements.

../_images/enable_backups_button_checked_post_validate.png

Note

The Protection period set in Veeam Kasten should not exceed the immutability window set in VBR. If the protection period is configured incorrectly, the Policy validation will display the following error (as shown in the figure):

../_images/vbr_immutability_05.png

Please check, Setting Up Immutable Backup Protection for details about immutable profile creation.

Creating an Immutable Policy

Create a policy with a daily snapshot (one snapshot per day) and set the exported snapshot retention as shown in the figure:

../_images/vbr_immutability_03.png

Note

Weekly, monthly and yearly snapshots are intentionally set to 0.

If a user needs another backup frequency, it can be done via another (separate) policy with separate S3 and VBR repositories, with corresponding immutability settings.

Recovering from a Worst-Case Scenario Attack

Stage 1: Recover Veeam Kasten

In many disaster circumstances, it may be safer to assume that Veeam Kasten has been compromised if an attack has been detected. If so, Veeam Kasten may need to be recovered to a point-in-time prior to the attack. If it is absolutely certain that Veeam Kasten has not been affected, directly restoring the application from Veeam Kasten's current known restore points is still possible, in which case skip directly to Stage 2. When in doubt, discuss with your CISO if it makes sense to restore Veeam Kasten to a time before the attack.

Restoring Veeam Kasten is straightforward. Follow the standard Recovering Veeam Kasten From a Disaster workflow:

  1. Reinstall Veeam Kasten, deleting the Veeam Kasten namespace if reinstalling on the same cluster.

  2. Create a secret to contain the DR passphrase.

  3. Provide the external storage configuration by adding a location profile referencing the object store location where backups are stored. This should resemble the profile that was created when Setting Up Immutable Backup Protection, but it is not necessary to create it with "Immutable Backups" enabled.

  4. Ascertain when the disaster was likely to have occurred. Pick a timestamp corresponding to a point-in-time prior to the disaster event. The time chosen should be no earlier than the protection period (associated with the Veeam Kasten Disaster Recovery immutable profile) in the past.

  5. Install the helm chart that creates the Veeam Kasten restore job. Provide the source cluster ID, the name of the location profile just created, and the point-in-time chosen in the previous step, formatted as a RFC3339 time stamp.

# e.g. to restore Veeam Kasten to 15:04:05 UTC on Jan 2, 2022:
$ helm install k10-restore kasten/k10restore --namespace=kasten-io \
    --set sourceClusterID=<source-clusterID> \
    --set profile.name=<location-profile-name> \
    --set pointInTime="2022-01-02T15:04:05Z"

Upon successful recovery, Veeam Kasten will now be in the same state as it was at the last time the Disaster Recovery policy had been run prior to the chosen point-in-time. This includes references to restore points that had since been retired, even if that retirement happened as part of routine policy retention management.

Restore points referring to snapshots or non-immutable backups may or may not be recoverable in this state; local storage snapshots or non-immutable exported backup data may have been permanently deleted, either during an attack, or as part of routine operation. However any restore points corresponding to immutable backups should still be fully recoverable.

Stage 2: Restore Applications

The process for restoring applications from immutable backups is identical to the standard restoration workflow. Make sure to select the Exported restore point instance, referring to the backup residing in the object store locked bucket.

../_images/restore_point_instance_locked_bucket.png

Veeam Kasten will do all point-in-time management based on its knowledge of the time the backup took place. Initiating a restore is as simple as selecting the desired backup (exported) restore point and clicking "Restore".

Note

The point-in-time chosen for the restore will be different for each PVC, and is a function of the time when the data upload completed for each. The upload completion time can be viewed for each data artifact by querying the RestorePointContent Details. Veeam Kasten will use a point-in-time 30 seconds after the uploadEndTime timestamp for the restore.

When the restore action completes, the application will be running in the same state, with the same persistent data, as it was at the time the backup took place.