The ability to move an application across clusters is an extremely powerful feature that enables a variety of use cases including Disaster Recovery (DR), Test/Dev with realistic data sets, and performance testing in isolated environments.
In particular, the K10 platform is being built to support application migration and mobility in a variety of different and overlapping contexts:
Cross-Namespace: The entire application stack can be migrated across different namespaces in the same cluster (covered in restoring applications).
Cross-Cluster: The application stack is migrated across non-federated Kubernetes clusters.
Cross-Account: Mobility can additionally be enabled across clusters running in different accounts (e.g., AWS accounts) or projects (e.g., Google Cloud projects).
Cross-Region: Mobility can further be enabled across different regions of the same cloud provider (e.g., US-East to US-West).
Cross-Cloud: Finally, mobility can be enabled across different cloud providers (e.g., AWS to Azure).
Some additional infrastructure configuration is needed before migration between two clusters can be enabled.
Required: Object storage configuration
Use-Case Specific: Cross-account and Kanister configuration
For two non-federated clusters to share data, we need to create an
object storage bucket (e.g.,
k10-migrate) that will be used to
store data about the application restore points that have been
selected for migration between clusters. The source cluster needs to
have write permissions to this bucket, and the destination cluster
needs to have read permissions. The appropriate credentials can be
configured using Export Profile and Import Profile as described
The Kanister framework, described later, stores application-level artifacts in an object store. For Kanister-enabled applications, in addition to the profile configuration above, both clusters need to use a Kanister profile that refers to the same object storage bucket. This can be accomplished by creating a Kanister profile via the K10 dashboard or via the CRD-based Profiles API.
The requirement for both clusters to have access to the Kanister object storage bucket will go away in an upcoming release in favor of only using the object storage bucket configured above for mobility.
When migrating applications across different cloud providers, including hybrid environments, K10 should have support for storage system snapshots on the originating side.
Apart from the Import and Export profiles, a Kanister profile also needs to be created as the data belonging to cross-cloud portable snapshots is stored as Kanister artifacts.
Currently, data exported by K10 for cross-cloud snapshot migration
is encrypted via the use of the AES-256 algorithm. It is highly
recommended that you specify your own encryption key.
This can be done via the creation of a
Kubernetes secret in the
kasten-io namespace after install:
$ kubectl create secret generic k10-artifact-key \ --namespace kasten-io \ --from-literal=key=<key>
If the above key is modified after artifacts have been created, old data will not be readable without support intervention.
The following configuration is only required if cross-account migration within the same cloud-provider is needed.
If cross-account volume migration is desired, K10 needs an IAM user
created within the destination account (e.g.,
destination AWS account ID
is also needed. During the creation of the migration profile, the
username should be specified as
<AWS Account ID>/<User Name>
In an AWS environment, K10 grants access to snapshots stored in the source account to the destination account. If the destination cluster is in a different region, K10 will make a cross-region copy of the underlying EBS snapshots before sharing access.
To enable the export and import of applications, you are required to
create a profile on each side of the transfer. Profile creation can be
accessed from the
Settings icon in the top-right corner of
the dashboard or via the CRD-based Profiles API.
To move data to another cluster, simply click
on the profiles page and select
Export. We recommend using the
target cluster name as the profile name to make it easier to access
As mentioned earlier, exporting data requires an object storage location. You are therefore required to pick an object storage provider, a region for the bucket, and the bucket name. If a bucket with the specific name does not exist, it will be created in the specified region.
Further, if the source and destination clusters are in different regions of the same cloud provider, it is essential that the region selected in the profile is the same region as where the destination cluster is located, as K10 uses that to determine the region where the artifacts (e.g., snapshot copies) should be copied to.
If migration is desired across Kubernetes clusters in different clouds or between on-premises and public cloud environments, you need to enable the infrastructure portability option when creating an export profile.
If you are using an S3 compatible bucket that is not hosted by one of the supported cloud providers, you will need to also specify an S3 endpoint URL (accessible from both clusters) and optionally disable SSL verification. Disabling SSL verification is only recommended for test setups.
When you hit
Validate and Save, the config profile will be created
and you will see a profile similar to the following:
The workflow for exporting applications into another cluster is very
similar to the workflow followed for restoring applications. You simply go over to the Applications page, click on the
dropdown, and select
After a restore point is selected, you will be presented with information about the application snapshot being migrated (e.g., time, originating policy, artifacts) and, more importantly, the option to configure the destination by selecting a migration profile.
A migration profile, as defined above, specifies where the exported data and metadata will be stored. This profile contains the name of the cluster that we are migrating data to and the location of an object storage bucket (e.g., using AWS S3, Google Cloud Storage, or any other S3-compliant object store) to export the needed data.
Simply select any of the migration profiles you might have previously
created and then click on
Export. After confirming the export, you
will be presented with a confirmation dialog. It is essential that
you copy the text block presented here because it will be required to
securely create an import policy on the destination cluster.
Importing an application snapshot is again very similar to the policy
workflow for protecting applications. From the
Policies page, simply select
Create New Policy.
To import applications, you need to select
Import for the
action. While you need to also specify a frequency for the import,
this simply controls how often to check the shared object storage
location. If data within the object store is not refreshed at the same
frequency, no duplicate work will be performed.
You will also need to paste the text block presented by the destination cluster during export to allow us to create the data export/import relationship for this application across these clusters. Finally, similar to the export workflow, the permissions profile also need to be selected but, as mentioned earlier, the destination cluster only needs read and list permissions on all the data in the shared object store.
Create Policy button is clicked, the system will start
scheduling jobs based on the policy definition. If new data is
detected, it will be imported into the cluster, automatically
associated with the application stack already running, and be made
available as a restore point. The normal workflow for restoring
applications can be followed and, when given the choice,
simply select a restore point that is tagged with the policy name used
above to create the import policy.
While migrating applications across clusters, regions, and clouds is significantly simpler with K10, there are still other considerations to be aware of that might impact your migration workflow. The below sections cover these considerations in detail for a smoother migration process.
While the K10 platform will protect all resources that are discovered in the application namespace, it intentionally does not gather resources that are not part of an application namespace but only found at the cluster level. Examples of such resources include Custom Resource Definitions or CRDS (but not the actual Custom Resources or CRs), Storage Classes, or Cluster Role Bindings. We assume that these resources will be deployed as a part of cluster creation by the administrator or the cluster deployment process. This is of particular importance when migrating applications to a new cluster as the absence of these resources could cause application restore failures.
Related to the issue of cluster-wide resources, there are Kubernetes resources that are only found in a namespace but have a resource conflict with other applications in the same cluster. While generally discouraged for production usage, a commonly observed resource that falls in this category is the NodePort service. Once claimed by an application, a subsequent request for the same NodePort will conflict and be disallowed. Whenever possible, K10 attempts to work around such limitations by resetting such settings on restore to a cluster-provided non-conflicting value. For example, with NodePort, K10 will allow Kubernetes to pick a port from the default port range allocated to the NodePort service.
Finally, applications sometimes will have dependencies that are external to the cluster (e.g., DNS entries) that might not be visible to Kubernetes or K10. However, it is possible to work with such external resources through the use of Kanister blueprints.
For migrations across Kubernetes clusters that are spread across different availability (or failure) zones, K10, for stateful applications in the destination cluster, tries to maintain the same fault independence as the source cluster. However, this can be hard or not possible when either the destination cluster has fewer availability zones than the source cluster or does not have nodes deployed in all availability zones.
To help work around these issues, K10 adopts a number of techniques to simplify the redeployment process. As far as possible, K10 will attempt to only place volumes in availability zones where it has already discovered worker nodes belonging to the destination cluster.
In the case of migration in the same region, K10 will attempt to first place volumes in the same zones that the source cluster had selected. If that is not possible, it will assign volumes to other availability zones where it has found worker nodes.
For cross-region migration, K10 will first attempt to map availability zones between the source and destination regions and, assuming the worker nodes are found in those zones, provision volumes according to that mapping. If a mapping is not possible (e.g., the source cluster used more available zones than the destination cluster has available), it will provision volumes into one of the other discovered zones in the destination cluster.
In the unlikely case that K10 is unable to either find worker nodes or discover zones, it will fallback to assigning volumes to zones that it knows exist in that region.
Finally, note that the above approach can potentially run into scheduling issues where the destination cluster might have either insufficient or no compute resources in a given zone. If you run into this issue, the simplest solution is to provision more worker nodes across your desired available zones.