Immutable Backups Workflow

K10 can leverage the object-locking capability available in object stores to make backups immutable. This guards against catastrophic disaster scenarios such as ransomware attacks and allows recovering the backups in those situations.

This feature is currently available for use with Azure, AWS S3 and any S3-compatible object store that supports object locking.

Warning

The generic storage and shareable volume backup and restore workflows are not compatible with the protections afforded by immutable backups. Use of a location profile enabled for immutable backups can be used for backup and restore, but the protection period is ignored, and the profile is treated as a non-immutability-enabled location. Please note that using an object-locking bucket for such use cases can amplify storage usage without any additional benefit. Please contact support for any inquiries.

Disaster Scenarios

Vulnerabilities can arise from many sources, such as lack of privilege separation due to credentials with permissive access, or from sophisticated attacks.

Consider a comprehensive breach of all secured systems in a Kubernetes cluster and ancillary infrastructure. Assume that a malicious agent has compromised all of the following:

the Kubernetes cluster - can inspect and control applications running in all namespaces, read secrets, manipulate snapshots, and locate backups.
the object store - can tamper with or destroy application backup data that had been exported by K10.
the K10 deployment - can force retirement of K10 snapshots and backups, compelling K10 to delete the associated data and metadata, including application backups and K10 Disaster Recovery self-backups.
the production application - can tamper with or encrypt vital data, demanding a ransom in exchange for resumed access.

In such a sophisticated scenario, an attacker might attempt to render restores from backups useless, before encrypting live application data and demanding a ransom.

K10 Immutable Backups

In the face of such a comprehensive attack, K10 has the ability to turn back the clock.

K10 is capable of exporting backups to object store containers with object locking enabled. Doing so renders the data written there immutable. Even users with administrative privileges are prevented from deleting or tampering with the backup data. Each repository blob is immutable and secure for an extendable period of time.

K10 uses these immutable blobs to go back in time, and retrieve the backups as they were prior to the time of an attack.

Protection Period

How far back in time K10 can see is a tunable parameter called the protection period. The protection period is chosen by the user when creating a profile with immutable backups enabled.

The protection period represents the estimated amount of time required by your organization to recover following impact. Recovery must begin within that protection period window in order to guarantee a successful restoration from immutable backups. The longer the protection period, the more time can be afforded for the recovery from an impact event like ransomware.

The protection period can be any length of time greater than or equal to 1 day, with a granularity of days. Longer protection periods may have increased storage costs associated with them, as object versions must be kept around longer.

Active Monitoring

K10 has a background service that monitors repositories containing immutable backups. Each time a restore point is exported to a repository in a locked bucket, some new blobs may be written, and some blobs may be reused. Therefore the retain-until date applied to each blob (the property that renders it immutable) may need to be refreshed, or pushed out to a later date, as time passes. This is because K10's backup solution makes use of incremental backups.

Once an immutable backups repository is created, K10 will begin periodically querying its data blobs in the background. The frequency of this operation is determined by the chosen protection period; a longer protection period doesn't require checking as frequently, because the retain-until date can be pushed out further when needed. That said, the longest the service will wait without performing a refresh check is 2 days.

In order for this background operation to run smoothly, you must maintain a profile referencing the location containing that repository. Additionally that profile must follow the criteria perscribed for an immutable backups profile.

An alert notification will appear in the upper right corner of the K10 dashboard when any of the following conditions is encountered:

the profile no longer meets these criteria
the background monitoring service has trouble connecting to the store
the background monitoring service is otherwise unable to perform the refresh procedure

Clicking the alert icon will open a side-pane containing a list of outstanding errors and warnings, each with a description of what went wrong. If you are unable to determine the appropriate fix from the information provided, please refer to Kasten K10 Community Support or open a case via https://my.veeam.com.

Comprehensive Protection

Individual applications can be protected with immutable backups by exporting to a location with object locking enabled. This ensures those application restore points are unable to be tampered with, and that K10 knows how to access them at the appropriate points in time.

However if the cluster has been completely compromised, K10 may also be susceptible to tampering. For example if an attacker manually retires all restore points, from K10's perspective, those backups no longer exist.

To comprehensively secure an application's backups even in the face of an attack on K10, it is necessary to activate a K10 Disaster Recovery policy that also makes use of immutable backups. This means that all backups of K10 itself are also stored immutably in an object-locked object store location.

By combining immutable backup policy runs for an application with a subsequent immutable Disaster Recovery policy run for K10, you preserve both the application's data and the ability to restore it.

Each time the Disaster Recovery policy runs, it will "lock in" the ability to recover from any of the active restore points at that point in time. Multiple applications may be simultaneously protected by different policies with differing scheduling frequencies. Therefore it is recommended to schedule the Disaster Recovery policy to run at least as frequently, if not more so, than the most frequent policy schedule that performs immutable backups. Additionally, the protection period chosen for the Disaster Recovery profile should be at least as long as the longest protection period in use by an immutable backups policy.

Setting Up Immutable Backup Protection for S3

Begin by setting up one or more location profiles, each pointing to an object store destination with object-locking enabled.

The selected object store destination must meet certain criteria:

Must be a bucket on AWS S3 or an S3-compatible object store.
Must be reachable with the credentials provided in the profile form.
Must already exist.
The region provided on the profile form must match the region in which the bucket resides.
Must have object locking enabled. Note: On some S3-compatible implementations, the object locking property of a bucket can only be enabled/configured at bucket creation time.

Select the checkbox for "Enable Immutable Backups".

Click the "Validate Bucket" button. This will initiate a pre-flight check of the above criteria.

If all of the checks succeed, a slider bar will appear for selecting the desired protection period. The bounds of this bar are from 1 day to an upper value of 90 days.

Click the "Save Profile" button to submit this profile configuration.

Profiles that reference object-locked object store locations and have immutable backups enabled will display this on its respective profile card: a field indicating "Object Immutability" is "enabled", and the chosen protection period.

Next follow the standard Backups workflow, selecting one of the immutable backup profiles for each policy. This will protect each covered application with immutable backups for all policy runs.

Finally follow the K10 Disaster Recovery workflow, selecting the desired immutable backup profile for the "Cloud Location Profile" in the DR form.

Setting Up Immutable Backup Protection for Azure

To set up immutability in Azure, follow a process similar to S3, but take into account the following requirements:

Ensure that the container exists and is reachable with the credentials provided in the profile form.
Enable versioning on the related storage account.
Ensure support for version-level immutability on the container or related storage account.

Since K10 ignores retention policies, it is not necessary to set one on the container. As an alternative, choose the desired protection period, and the files will be initially protected for that amount plus a safety buffer to ensure protection compliance.

Setting Up Immutable Backup Protection Using Veeam Repository

To use the K10 backup immutability feature with VBR, make sure that immutability settings are configured consistently as follows:

Set up the immutability window length in VBR for the desired immutability period.
Configure a proper protected period in K10 (see the K10 immutability documentation for details). This period must be less than or equal to the immutability window in VBR.
Adjust K10 backup frequency and retention settings to prevent the deletion restore points inside the VBR immutability window. Failure to do so may lead to orphaned restore points that will not be automatically retired.

Note

The integrity of restore points outside the immutability window in VBR is not guaranteed. It is recommended to set up the retention policy to automatically delete such restore points when they are outside the immutability window.

This approach allows you to keep only the N last backups according to the chosen backup frequency. If you need to keep, for example, N daily backups and M yearly backups, create several policies with different backup frequencies and retention settings.

Example

A customer wants to guarantee the immutability for 30 days for their daily backups:

The immutability period for VBR must be set to 30 days
The protection period for K10 must be set to 30 days (or lower)

The configuration of the S3 bucket is beyond the scope of this example. We assume that it has already been properly configured.

Setting up the Immutability Window Length in VBR

To set up the immutability window length in VBR, open the Repository settings and enter the desired value in the "Make recent backups immutable for" field:

Note

Allowed value for immutability window in VBR is 7 to 9999 days.

Here, we set it to 30 days because the customer wants to guarantee immutability for this period.

Additional information about immutable repositories can be found in the VBR documentation.

Setting up the Immutability Window Length in K10

The K10 protection period should be set to any value up to and including 30 days, but not more, to align with the specified requirements.

Note

The Protection period set in K10 should not exceed the immutability window set in VBR. If the protection period is configured incorrectly, the Policy validation will display the following error (as shown in the figure):

Please check, Setting Up Immutable Backup Protection for details about immutable profile creation.

Creating an Immutable Policy

Create a policy with a daily snapshot (one snapshot per day) and set the exported snapshot retention as shown in the figure:

Note

Weekly, monthly and yearly snapshots are intentionally set to 0.

If a user needs another backup frequency, it can be done via another (separate) policy with separate S3 and VBR repositories, with corresponding immutability settings.

Recovering from a Worst-Case Scenario Attack

Stage 1: Recover K10

In many disaster circumstances, it may be safer to assume that K10 has been compromised if an attack has been detected. If so, K10 may need to be recovered to a point-in-time prior to the attack. If it is absolutely certain that K10 has not been affected, directly restoring the application from K10's current known restore points is still possible, in which case skip directly to Stage 2. When in doubt, discuss with your CISO if it makes sense to restore K10 to a time before the attack.

Restoring K10 is straightforward. Follow the standard Recovering K10 From a Disaster workflow:

Reinstall K10, deleting the K10 namespace if reinstalling on the same cluster.
Create a secret to contain the DR passphrase.
Provide the external storage configuration by adding a location profile referencing the object store location where backups are stored. This should resemble the profile that was created when Setting Up Immutable Backup Protection, but it is not necessary to create it with "Immutable Backups" enabled.
Ascertain when the disaster was likely to have occurred. Pick a timestamp corresponding to a point-in-time prior to the disaster event. The time chosen should be no earlier than the protection period (associated with the K10 Disaster Recovery immutable profile) in the past.
Install the helm chart that creates the K10 restore job. Provide the source cluster ID, the name of the location profile just created, and the point-in-time chosen in the previous step, formatted as a RFC3339 time stamp.

# e.g. to restore K10 to 15:04:05 UTC on Jan 2, 2022:
$ helm install k10-restore kasten/k10restore --namespace=kasten-io \
    --set sourceClusterID=<source-clusterID> \
    --set profile.name=<location-profile-name> \
    --set pointInTime="2022-01-02T15:04:05Z"

Upon successful recovery, K10 will now be in the same state as it was at the last time the Disaster Recovery policy had been run prior to the chosen point-in-time. This includes references to restore points that had since been retired, even if that retirement happened as part of routine policy retention management.

Restore points referring to snapshots or non-immutable backups may or may not be recoverable in this state; local storage snapshots or non-immutable exported backup data may have been permanently deleted, either during an attack, or as part of routine operation. However any restore points corresponding to immutable backups should still be fully recoverable.

Stage 2: Restore Applications

The process for restoring applications from immutable backups is identical to the standard restoration workflow. Make sure to select the Exported restore point instance, referring to the backup residing in the object store locked bucket.

K10 will do all point-in-time management based on its knowledge of the time the backup took place. Initiating a restore is as simple as selecting the desired backup (exported) restore point and clicking "Restore".

Note

The point-in-time chosen for the restore will be different for each PVC, and is a function of the time when the data upload completed for each. The upload completion time can be viewed for each data artifact by querying the RestorePointContent Details. K10 will use a point-in-time 30 seconds after the uploadEndTime timestamp for the restore.

When the restore action completes, the application will be running in the same state, with the same persistent data, as it was at the time the backup took place.