K10 Tools
The k10tools binary has commands that can help with validating
if a cluster is setup correctly before installing K10 and for
debugging K10's micro services.
The latest version of k10tools can be found
here.
It has binaries that are compatible with both Linux and MacOS.
Authentication Service
The k10tools debug auth sub command can be used to debug K10's
Authentication service when it is setup with Active Directory or
OpenShift based authentication. Provide -d openshift flag for
OpenShift based authentication. It verifies connection to the OpenShift
OAuth server and the OpenShift Service Account token. It also searches
for any error events in Service Account.
./k10tools debug auth
Dex:
OIDC Provider URL: https://api.test
Release name: k10
Dex well known URL:https://api.test/k10/dex/.well-known/openid-configuration
Trying to connect to Dex without TLS (insecureSkipVerify=false)
Connection succeeded - OK
./k10tools debug auth -d openshift
Verify OpenShift OAuth Server Connection:
Openshift URL - https://api.test:6443/.well-known/oauth-authorization-server
Trying to connect to Openshift without TLS (insecureSkipVerify=false)
Connection failed, testing other options
Trying to connect to Openshift with TLS but verification disabled (insecureSkipVerify=true)
Connection succeeded - OK
Verify OpenShift Service Account Token:
Initiating token verification
Fetched ConfigMap - k10-dex
Service Account for OpenShift authentication - k10-dex-sa
Service account fetched
Secret - k10-dex-sa-token-7fwm7 retrieved
Token retrieved from Service Account secrets
Token retrieved from ConfigMap
Token matched - OK
Get Service Account Error Events:
Searching for events with error in Service Account - k10-dex-sa
Found event/s in service account with error
{"type":"Warning","from":"service-account-oauth-client-getter","reason":"NoSAOAuthRedirectURIs","object":"ServiceAccount/k10-dex-sa","message":"system:serviceaccount:kasten-io:k10-dex-sa has no redirectURIs; set serviceaccounts.openshift.io/oauth-redirecturi.<some-value>=<redirect> or create a dynamic URI using serviceaccounts.openshift.io/oauth-redirectreference.<some-value>=<reference>","timestamp":"2021-04-08 05:06:06 +0000 UTC"} ({"message":"service account event error","function":"kasten.io/k10/kio/tools/k10primer/k10debugger.(*OpenshiftDebugger).getServiceAccountErrEvents","linenumber":224}) - Error
Catalog Service
The k10tools debug catalog size sub command can be used to obtain
the size of K10's catalog and the disk usage of the volume
where the catalog is stored.
# ./k10tools debug catalog size
Catalog Size:
total 380K
-rw------- 1 kio kio 512K Jan 26 23:57 model-store.db
Catalog Volume Disk Usage:
Filesystem Size Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-4acee649-5c24-4a79-955f-9d8fdfb10ac7 20G 45M 19G 1% /mnt/k10state
Backup Actions
The k10tools debug backupactions sub command can be used to obtain
the backupactions created in the respective cluster. Use the -o json
flag to obtain more information in the JSON format.
# ./k10tools debug backupactions
Name Namespace CreationTimestamp PolicyName PolicyNamespace
scheduled-6wbzw default 2021-01-29 07:57:08 +0000 UTC default-backup kasten-io
scheduled-5thsg default 2021-01-29 05:37:03 +0000 UTC default-backup kasten-io
Kubernetes Nodes
The k10tools debug node sub command can be used to obtain information
about the Kubernetes nodes. Use the -o json flag to obtain more
information in the JSON format.
# ./k10tools debug node
Name |OS Image
onkar-1-pool-1-3d1cf |Debian GNU/Linux 10 (buster)
onkar-1-pool-1-3d1cq |Debian GNU/Linux 10 (buster)
onkar-1-pool-1-3d1cy |Debian GNU/Linux 10 (buster)
Application Information
The k10tools debug applications sub command can be used
to obtain information
about the applications running in given namespace.
Use the -o json flag to obtain more
information in the JSON format
(Note: Right now, JSON format support is only provided for PVCs).
Use -n to provide the namespace.
In case the namespace is not provided, application information
will be
fetched from the default namespace.
e.g. -n kasten-io
# ./k10tools debug applications
Fetching information from namespace - kasten-io | resource - ingresses
Name |Hosts |Address |Ports |Age |
k10-ingress |* |138.68.228.199 |80 |36d |
Fetching information from namespace - kasten-io | resource - daemonsets
Resources not found
PVC Information -
Name |Volume |Capacity
catalog-pv-claim |pvc-4fc67966-aee7-493c-b2fd-c6251933875c |20Gi
jobs-pv-claim |pvc-cdda0458-6b63-48a6-8e7f-c1b947600c9f |20Gi
logging-pv-claim |pvc-36a92c5b-d018-4ce8-ba79-970d15554387 |20Gi
metering-pv-claim |pvc-8c0c6477-216d-4227-a6af-9725ce2a3dc1 |2Gi
prometheus-server |pvc-1b14f51c-5abf-45f5-8bd9-1a58d86d58ef |8Gi
K10 Primer for Pre-Flight Checks
The k10tools primer sub command can be used to run pre-flight checks.
before installing K10. Refer to the section about
Pre-Flight Checks for more details.
The code block below shows an example of the output when executed on a Kubernetes cluster deployed in Digital Ocean.
# ./k10tools primer
Kubernetes Version Check:
Valid kubernetes version (v1.17.13) - OK
RBAC Check:
Kubernetes RBAC is enabled - OK
Aggregated Layer Check:
The Kubernetes Aggregated Layer is enabled - OK
CSI Capabilities Check:
Using CSI GroupVersion snapshot.storage.k8s.io/v1alpha1 - OK
Validating Provisioners:
kube-rook-ceph.rbd.csi.ceph.com:
Is a CSI Provisioner - OK
Storage Classes:
rook-ceph-block
Valid Storage Class - OK
Volume Snapshot Classes:
csi-rbdplugin-snapclass
Has k10.kasten.io/is-snapshot-class annotation set to true - OK
Has deletionPolicy 'Retain' - OK
dobs.csi.digitalocean.com:
Is a CSI Provisioner - OK
Storage Classes:
do-block-storage
Valid Storage Class - OK
Volume Snapshot Classes:
do-block-storage
Has k10.kasten.io/is-snapshot-class annotation set to true - OK
Missing deletionPolicy, using default
Validate Generic Volume Snapshot:
Pod Created successfully - OK
GVS Backup command executed successfully - OK
Pod deleted successfully - OK
K10 Primer for Upgrades
The k10tools primer upgrade sub command can be used to find the recommended
upgrade path of your K10 version and to check there is adequate space to
perform the upgrades. It only provides commands for Helm deployments.
See Upgrading K10 for additional details.
This tool requires Internet access to http://gcr.io
# ./k10tools primer upgrade
Catalog Volume Disk Usage:
Filesystem Size Used Avail Use% Mounted on
/dev/sdf 20G 1.3G 19G 7% /mnt/k10state
Current K10 Version: 4.5.5
Latest K10 Version: 4.5.6
Helm Install: true
* To upgrade successfully you must have at least 50% free in catalog storage
Recommended upgrade path:
helm repo update && \
helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \
helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml --version=4.5.6
CSI Capabilities Check
The k10tools primer storage csi-checker command can be used to check
a specified CSI storage class is able to carry out snapshot and restoration
activities or report configuration issues if not. It creates a temporary
application to test this.
The -s flag specifies the storage class.
The -u flag specifies the user the pod runs as.
# ./k10tools primer storage csi-checker -s standard-rwo -u 1000
Starting CSI Checker. Could take up to 5 minutes
Creating application
-> Created pod (kubestr-csi-original-podr2rkz) and pvc (kubestr-csi-original-pvc2fx6s)
Taking a snapshot
-> Created snapshot (kubestr-snapshot-20220608113008)
Restoring application
-> Restored pod (kubestr-csi-cloned-podhgx57) and pvc (kubestr-csi-cloned-pvccfh8w)
Cleaning up resources
CSI Snapshot Walkthrough:
Using annotated VolumeSnapshotClass (my-snapshotclass)
Successfully tested snapshot restore functionality. - OK
Generic Volume Snapshot Capabilities Check
The k10tools primer gvs-cluster-check command can be used to check
if the cluster is compatible for K10 Generic Volume Snapshot.
K10 Generic backup commands are executed on a pod running
kanister-tools image and checked for appropriate output.
Use -n flag to provide namespace.
By default, kasten-io namespace will be used.
Use -s flag to provide a storageclass for the checks to be run against.
By default, no storage class will be used and the checks will be done using
temporary storage from the node the pod runs on.
Use --service-account flag to specify the service account to be used
by pods during GVS checks. By default, default service
account will be used.
Note
By default, the k10tools command will use the publicly available
kanister-tools image at ghcr.io/kanisterio/kanister-tools:<Kanister version>.
Since this image is not available in air-gapped environments, to
override the default image, set the KANISTER_TOOLS environment variable
to the kanister-tools image that is available in the air-gapped
environment's local registry. The kanister-tools version can be found by running
this command in a shell that has access to the public internet and with the
appropriate K10 version.
docker run --rm -it gcr.io/kasten-images/k10offline:<K10 version> list-images | grep kanister-tools.
- Example:
export KANISTER_TOOLS=<your local registry>/<your local repository name>/kanister-tools:k10-<Kanister version>
# ./k10tools primer gvs-cluster-check
Validate Generic Volume Snapshot:
Pod Created successfully - OK
GVS Backup command executed successfully - OK
Pod deleted successfully - OK
K10 Generic Storage Backup Sidecar Injection
The k10tools k10genericbackup can be used to make Kubernetes
workloads compatible for K10 Generic Storage Backup by injecting a
Kanister sidecar and setting the forcegenericbackup=true annotation
on the workloads.
Note
By default, the k10tools command will use the publicly available
kanister-tools image at ghcr.io/kanisterio/kanister-tools:<Kanister version>.
Since this image is not available in air-gapped environments, to
override the default image, set the KANISTER_TOOLS environment variable
to the kanister-tools image that is available in the air-gapped
environment's local registry. The kanister-tools version can be found by running
this command in a shell that has access to the public internet and with the
appropriate K10 version.
docker run --rm -it gcr.io/kasten-images/k10offline:<K10 version> list-images | grep kanister-tools.
- Example:
export KANISTER_TOOLS=<your local registry>/<your local repository name>/kanister-tools:k10-<Kanister version>
## Usage ##
# ./k10tools k10genericbackup --help
k10genericbackup makes Kubernetes workloads compatible for K10 Generic Storage Backup by
injecting a Kanister sidecar and setting the forcegenericbackup=true annotation on the workloads.
To know more about K10 Generic Storage Backup, visit https://docs.kasten.io/latest/install/generic.html
Usage:
k10tools k10genericbackup [command]
Available Commands:
inject Inject Kanister sidecar to workloads to enable K10 Generic Storage Backup
uninject Uninject Kanister sidecar from workloads to disable K10 Generic Storage Backup
Flags:
--all-namespaces resources in all the namespaces
-h, --help help for k10genericbackup
--k10-namespace string namespace where K10 services are deployed (default "kasten-io")
-n, --namespace string namespace (default "default")
Global Flags:
-o, --output string Options(json)
Use "k10tools k10genericbackup [command] --help" for more information about a command.
## Example: Inject a Kanister sidecar to all the workloads in postgres namespace ##
# ./k10tools k10genericbackup inject all -n postgres
Inject deployment:
Inject statefulset:
Injecting sidecar to statefulset postgres/mysql
Updating statefulset postgres/mysql
Waiting for statefulset postgres/mysql to be ready
Sidecar injection successful on statefulset postgres/mysql! - OK
Injecting sidecar to statefulset postgres/postgres-postgresql
Updating statefulset postgres/postgres-postgresql
Waiting for statefulset postgres/postgres-postgresql to be ready
Sidecar injection successful on statefulset postgres/postgres-postgresql! - OK
Inject deploymentconfig:
Skipping. Env is not compatible for Kanister sidecar injection
CA Certificate Check
The k10tools debug ca-certificate command can be used to check
if the CA certificate is installed properly in K10.
The -n flag can be used to provide namespace and it
defaults to kasten-io.
More information on
installation
process.
# ./k10tools debug ca-certificate
CA Certificate Checker:
Fetching configmap which contains CA Certificate information : custom-ca-bundle-store
Certificate exists in configmap - OK
Found container : aggregatedapis-svc to extract certificate
Certificate exists in container at /etc/ssl/certs/custom-ca-bundle.pem
Certificates matched successfully - OK
Installation of K10 in OpenShift clusters
The k10tools openshift prepare-install command can be used to
prepare an OpenShift cluster for installation of K10.
It extracts a CA Certificate from the cluster, installs it in
the namespace where K10 will be installed, and generates
the helm command to be used for installing K10.
The -n flag can be used to provide the namespace where K10
will be installed. The default namespace is kasten-io.
--recreate-resources flag recreates resources that
may have been created by previous execution of this command.
Set --insecure-ca flag to true if Certificate Issuing
Authority is not trusted.
# ./k10tools openshift prepare-install
Openshift Prepare Install:
Certificate found in Namespace 'openshift-ingress-operator' in secret 'router-ca' - OK
Checking if namespace 'kasten-io' exists
Namespace 'kasten-io' exists - OK
Created configmap 'custom-ca-bundle-store' with custom certificate in it - OK
Searching for Apps Base Domain Name in Ingress Controller
Found Apps Base Domain 'apps.test.aws.kasten.io' - OK
Created Service Account 'k10-dex-sa' successfully - OK
Please use below helm command to start K10 installation
--------------------------------------------------------------------
helm repo add kasten https://charts.kasten.io/
helm install k10 kasten/k10 --namespace=kasten-io \
--set scc.create=true \
--set route.enabled=true \
--set route.tls.enabled=true \
--set auth.openshift.enabled=true \
--set auth.openshift.serviceAccount=k10-dex-sa \
--set auth.openshift.clientSecret=<your key will be here automatically>\
--set auth.openshift.dashboardURL=https://k10-route-kasten-io.apps.test.aws.kasten.io/k10/ \
--set auth.openshift.openshiftURL=https://api.test.aws.kasten.io:6443 \
--set auth.openshift.insecureCA=false \
--set cacertconfigmap.name=custom-ca-bundle-store