The ability to move an application across clusters is an extremely powerful feature that enables a variety of use cases including Disaster Recovery (DR), Test/Dev with realistic data sets, and performance testing in isolated environments.
In particular, the K10 platform is being built to support application migration and mobility in a variety of different and overlapping contexts:
Cross-Namespace: The entire application stack can be migrated across different namespaces in the same cluster (covered in restoring applications).
Cross-Cluster: The application stack is migrated across non-federated Kubernetes clusters.
Cross-Account: Mobility can additionally be enabled across clusters running in different accounts (e.g., AWS accounts) or projects (e.g., Google Cloud projects).
Cross-Region: Mobility can further be enabled across different regions of the same cloud provider (e.g., US-East to US-West).
Cross-Cloud: Finally, mobility can be enabled across different cloud providers (e.g., AWS to Azure).
Some additional infrastructure configuration is needed before migration between two clusters can be enabled.
Required: Object storage configuration
Use-Case Specific: Cross-account and Kanister configuration
For two non-federated clusters to share data, K10 needs access to an
object storage bucket (e.g.,
k10-migrate) that will be used to
store data about the application restore points that have been
selected for migration between clusters. The source cluster needs to
have write permissions to this bucket, and the destination cluster
needs to have read permissions. The appropriate credentials can be
configured using Export and
The Kanister framework, described later, stores application-level artifacts in an object store. For Kanister-enabled applications, in addition to the profile configuration above, both clusters need to use a Kanister profile that refers to the same object storage bucket. This can be accomplished by creating a Kanister profile via the K10 dashboard or via the CRD-based Profiles API.
The requirement for both clusters to have access to the Kanister object storage bucket will go away in an upcoming release in favor of only using the object storage bucket configured above for mobility.
When migrating applications across different cloud providers, including hybrid environments, K10 should have support for storage system snapshots on the originating side.
Apart from creating Import and Export profiles, a Kanister profile also needs to be created as the data belonging to cross-cloud portable snapshots is stored as Kanister artifacts.
The following per-cloud provider configuration is required when cross-account/project/subscription migration within the same cloud-provider is needed.
When creating an Export profile for AWS, please follow the directions specified in Export Settings for Migration first and, in particular, the region selection for the S3 bucket being used.
If cross-account volume migration is desired, K10 needs an IAM user
created within the destination account (e.g.,
destination AWS account ID
is also needed. During the creation of the migration profile, the
username should be specified as
<AWS Account ID>/<User Name>
In an AWS environment, K10 grants access to snapshots stored in the source account to the destination account. If the destination cluster is in a different region, K10 will make a cross-region copy of the underlying EBS snapshots before sharing access.
Finally, if you want to use IAM roles, you have one of two
options. First, if K10 was installed with an IAM role, that can be
used by selecting the
Authenticate With AWS IAM Role
option. Alternatively, if an IAM role needs to be specified at profile
creation time, the
Execute Operations Using an AWS IAM Role option
should be selected. As a result, K10 will generate and use temporary
security credentials for executing operations.
Currently, due to underlying provider limitations, optimized migration is only supported among different clusters in the same project. For migration across clusters running within different accounts, please follow the documentation for cross-cloud migration.
The workflow for exporting applications into another cluster is very
similar to the workflow followed for protecting applications. When a policy is created to capture an application, simply
also select the
Enable Backups via Snapshot Exports option shown above.
The export profile, as defined in the configuration section, specifies where the exported data and metadata will be stored. This profile contains the location of an object storage bucket (e.g., using AWS S3, Google Cloud Storage, Azure blob, or any other S3-compliant object store) to export the needed data.
Export Snapshot Data option stores the exported data and
metadata to the object store, but the
Export Snapshot References Only
option stores only metadata about the application restore points. Below
these options, options exist to specify storage classes that override the
Snapshot Data or
Snapshot References Only setting.
For example, if there is a cluster where multiple storage providers are
in use, and all of the storage classes in this cluster can be exported
by reference, then the
Export Snapshot References Only option is all
that is needed; but if one of the storage classes in this cluster is only
visible inside the cluster (e.g., a Rook Ceph setup with a StorageClass
Export Snapshot References Only option can still
be used, but that one particular StorageClass (
rc1) needs to be listed
as an exception to the portability setting, because its data needs to be
exported. The image above shows the settings for this example.
If migration is desired across Kubernetes clusters in different clouds
or between on-premises and public cloud environments, you need to enable
Export Snapshot Data option in order to migrate both the application
data and metadata. Without it, the application data will not be exported
and the subsequent import will fail.
When this application is imported into another cluster, as described
below, initial handshake configuration will be required. This can be
obtained from the policy by clicking
Show import details and will
result in the below encoded configuration being displayed.
Apart from policy-driven exports, it is also possible to manually
create a one-off application export. From the Applications page,
simply click on the
After a restore point is selected, you will be presented with information about the application snapshot being migrated (e.g., time, originating policy, artifacts) and, more importantly, the option to configure the destination by selecting a mobility profile.
The option to select export portability follows. Refer back to the above section for information on the settings relating to this option.
Finally, click on
Export. After confirming the export, the
encoded import configuration required by the destination cluster
will be presented.
Import policies are not needed for restoring from backups that use export profiles. They are only supported for importing applications into a cluster that is different than where the application was captured from. The protecting applications section has more details.
Importing an application snapshot is again very similar to the policy
creation workflow for protecting applications. From the
Policies page, simply select
Create New Policy.
To import applications, you need to select
Import for the
action. While you need to also specify a frequency for the import,
this simply controls how often to check the shared object storage
location. If data within the object store is not refreshed at the same
frequency, no duplicate work will be performed.
Care should be taken when auto-restoring the application during import. In particular, ensure that the newly restored application does not conflict with the application running in the source cluster. Examples of potential conflicts include accidental credential reuse, access to and use of external services, and services conflicting for exclusive ownership of shared resources.
It is also possible to select
Restore after Import to bring the
application up in this cluster after the metadata import is complete.
You will also need to paste the text block displayed by the destination cluster's export policy (or shown during a manual export) to allow K10 to create the data export/import relationship for this application across the clusters. Finally, similar to the export workflow, an import profile also need to be selected but the destination cluster usually only needs read and list permissions on all the data in the shared object storage system.
Create Policy button is clicked, the system will start
scheduling jobs based on the policy definition. If new data is
detected, it's metadata will be imported into the cluster,
automatically associated with the application stack already running,
and be made available as a restore point. Note that unless
after Import is selected, only metadata is brought into the
cluster. If the data volumes reside in an object store (e.g., after a
cross-cloud migration), they will not be converted into native volumes
until a restore operation is initiated.
The normal workflow for restoring applications can be followed and, when given the choice, simply select a restore point that is tagged with the import policy name.
While migrating applications across clusters, regions, and clouds is significantly simpler with K10, there are still other considerations to be aware of that might impact your migration workflow. The below sections cover these considerations in detail for a smoother migration process.
While the K10 platform will protect all resources that are discovered in the application namespace, it intentionally does not gather resources that are not part of an application namespace but only found at the cluster level. Examples of such resources include Custom Resource Definitions or CRDS (but not the actual Custom Resources or CRs), Storage Classes, or Cluster Role Bindings. K10 assumes that these resources will be deployed as a part of cluster creation by the administrator or the cluster deployment process. This is of particular importance when migrating applications to a new cluster as the absence of these resources could cause application restore failures.
Related to the issue of cluster-wide resources, there are Kubernetes resources that are only found in a namespace but have a resource conflict with other applications in the same cluster. While generally discouraged for production usage, a commonly observed resource that falls in this category is the NodePort service. Once claimed by an application, a subsequent request for the same NodePort will conflict and be disallowed. Whenever possible, K10 attempts to work around such limitations by resetting such settings on restore to a cluster-provided non-conflicting value. For example, with NodePort, K10 will allow Kubernetes to pick a port from the default port range allocated to the NodePort service.
Finally, applications sometimes will have dependencies that are external to the cluster (e.g., DNS entries) that might not be visible to Kubernetes or K10. However, it is possible to work with such external resources through the use of post-restore hooks available in K10 and Kanister blueprints.
For migrations across Kubernetes clusters that are spread across different availability (or failure) zones, K10, for stateful applications in the destination cluster, tries to maintain the same fault independence as the source cluster. However, this can be hard or not possible when either the destination cluster has fewer availability zones than the source cluster or does not have nodes deployed in all availability zones.
To help work around these issues, K10 adopts a number of techniques to simplify the redeployment process. As far as possible, K10 will attempt to only place volumes in availability zones where it has already discovered worker nodes belonging to the destination cluster.
In the case of migration in the same region, K10 will attempt to first place volumes in the same zones that the source cluster had selected. If that is not possible, it will assign volumes to other availability zones where it has found worker nodes.
For cross-region migration, K10 will first attempt to map availability zones between the source and destination regions and, assuming the worker nodes are found in those zones, provision volumes according to that mapping. If a mapping is not possible (e.g., the source cluster used more available zones than the destination cluster has available), it will provision volumes into one of the other discovered zones in the destination cluster.
In the unlikely case that K10 is unable to either find worker nodes or discover zones, it will fallback to assigning volumes to zones that it knows exist in that region.
Finally, note that the above approach can potentially run into scheduling issues where the destination cluster might have either insufficient or no compute resources in a given zone. If you run into this issue, the simplest solution is to provision more worker nodes across your desired available zones.