K10 Tools

The k10tools binary has commands that can help with validating if a cluster is setup correctly before installing K10 and for debugging K10's micro services.

The latest version of k10tools can be found here. It has binaries that are compatible with both Linux and MacOS.

Authentication Service

The k10tools debug auth sub command can be used to debug K10's Authentication service when it is setup with Active Directory or OpenShift based authentication. Provide -d openshift flag for OpenShift based authentication. It verifies connection to the OpenShift OAuth server and the OpenShift Service Account token. It also searches for any error events in Service Account.

./k10tools debug auth

Dex:
  OIDC Provider URL: https://api.test
  Release name: k10
  Dex well known URL:https://api.test/k10/dex/.well-known/openid-configuration
  Trying to connect to Dex without TLS (insecureSkipVerify=false)
  Connection succeeded  -  OK

./k10tools debug auth -d openshift

Verify OpenShift OAuth Server Connection:
  Openshift URL - https://api.test:6443/.well-known/oauth-authorization-server
  Trying to connect to Openshift without TLS (insecureSkipVerify=false)
  Connection failed, testing other options
  Trying to connect to Openshift with TLS but verification disabled (insecureSkipVerify=true)
  Connection succeeded  -  OK

Verify OpenShift Service Account Token:
  Initiating token verification
  Fetched ConfigMap - k10-dex
  Service Account for OpenShift authentication - k10-dex-sa
  Service account fetched
  Secret - k10-dex-sa-token-7fwm7 retrieved
  Token retrieved from Service Account secrets
  Token retrieved from ConfigMap
  Token matched  -  OK

Get Service Account Error Events:
  Searching for events with error in Service Account - k10-dex-sa
  Found event/s in service account with error
  {"type":"Warning","from":"service-account-oauth-client-getter","reason":"NoSAOAuthRedirectURIs","object":"ServiceAccount/k10-dex-sa","message":"system:serviceaccount:kasten-io:k10-dex-sa has no redirectURIs; set serviceaccounts.openshift.io/oauth-redirecturi.<some-value>=<redirect> or create a dynamic URI using serviceaccounts.openshift.io/oauth-redirectreference.<some-value>=<reference>","timestamp":"2021-04-08 05:06:06 +0000 UTC"} ({"message":"service account event error","function":"kasten.io/k10/kio/tools/k10primer/k10debugger.(*OpenshiftDebugger).getServiceAccountErrEvents","linenumber":224})  -  Error

Catalog Service

The k10tools debug catalog size sub command can be used to obtain the size of K10's catalog and the disk usage of the volume where the catalog is stored.

# ./k10tools debug catalog size

 Catalog Size:
   total 380K
 -rw------- 1 kio kio 512K Jan 26 23:57 model-store.db
 Catalog Volume Disk Usage:
   Filesystem                                                                Size  Used Avail Use% Mounted on
 /dev/disk/by-id/scsi-0DO_Volume_pvc-4acee649-5c24-4a79-955f-9d8fdfb10ac7   20G   45M   19G   1% /mnt/k10state

Backup Actions

The k10tools debug backupactions sub command can be used to obtain the backupactions created in the respective cluster. Use the -o json flag to obtain more information in the JSON format.

# ./k10tools debug backupactions

Name                            Namespace     CreationTimestamp                           PolicyName      PolicyNamespace
scheduled-6wbzw                 default               2021-01-29 07:57:08 +0000 UTC     default-backup        kasten-io
scheduled-5thsg                 default               2021-01-29 05:37:03 +0000 UTC     default-backup        kasten-io

Kubernetes Nodes

The k10tools debug node sub command can be used to obtain information about the Kubernetes nodes. Use the -o json flag to obtain more information in the JSON format.

# ./k10tools debug node

  Name                 |OS Image
  onkar-1-pool-1-3d1cf |Debian GNU/Linux 10 (buster)
  onkar-1-pool-1-3d1cq |Debian GNU/Linux 10 (buster)
  onkar-1-pool-1-3d1cy |Debian GNU/Linux 10 (buster)

Application Information

The k10tools debug applications sub command can be used to obtain information about the applications running in given namespace. Use the -o json flag to obtain more information in the JSON format (Note: Right now, JSON format support is only provided for PVCs). Use -n to provide the namespace. In case the namespace is not provided, application information will be fetched from the default namespace. e.g. -n kasten-io

# ./k10tools debug applications

  Fetching information from namespace - kasten-io | resource - ingresses
  Name        |Hosts |Address        |Ports |Age |
  k10-ingress |*     |138.68.228.199 |80    |36d |

  Fetching information from namespace - kasten-io | resource - daemonsets
  Resources not found

  PVC Information -
  Name                |Volume                                     |Capacity
  catalog-pv-claim    |pvc-4fc67966-aee7-493c-b2fd-c6251933875c   |20Gi
  jobs-pv-claim       |pvc-cdda0458-6b63-48a6-8e7f-c1b947600c9f   |20Gi
  logging-pv-claim    |pvc-36a92c5b-d018-4ce8-ba79-970d15554387   |20Gi
  metering-pv-claim   |pvc-8c0c6477-216d-4227-a6af-9725ce2a3dc1   |2Gi
  prometheus-server   |pvc-1b14f51c-5abf-45f5-8bd9-1a58d86d58ef   |8Gi

K10 Primer for Pre-Flight Checks

The k10tools primer sub command can be used to run pre-flight checks. before installing K10. Refer to the section about Pre-Flight Checks for more details.

The code block below shows an example of the output when executed on a Kubernetes cluster deployed in Digital Ocean.

# ./k10tools primer

Kubernetes Version Check:
  Valid kubernetes version (v1.17.13)  -  OK

RBAC Check:
  Kubernetes RBAC is enabled  -  OK

Aggregated Layer Check:
  The Kubernetes Aggregated Layer is enabled  -  OK

CSI Capabilities Check:
  Using CSI GroupVersion snapshot.storage.k8s.io/v1alpha1  -  OK

Validating Provisioners:
kube-rook-ceph.rbd.csi.ceph.com:
  Is a CSI Provisioner  -  OK
  Storage Classes:
    rook-ceph-block
      Valid Storage Class  -  OK
  Volume Snapshot Classes:
    csi-rbdplugin-snapclass
      Has k10.kasten.io/is-snapshot-class annotation set to true  -  OK
      Has deletionPolicy 'Retain'  -  OK

dobs.csi.digitalocean.com:
  Is a CSI Provisioner  -  OK
  Storage Classes:
    do-block-storage
      Valid Storage Class  -  OK
  Volume Snapshot Classes:
    do-block-storage
      Has k10.kasten.io/is-snapshot-class annotation set to true  -  OK
      Missing deletionPolicy, using default

Validate Generic Volume Snapshot:
  Pod Created successfully  -  OK
  GVS Backup command executed successfully  -  OK
  Pod deleted successfully  -  OK

K10 Primer for Upgrades

The k10tools primer upgrade sub command can be used to find the recommended upgrade path of your K10 version and to check there is adequate space to perform the upgrades. It only provides commands for Helm deployments. See Upgrading K10 for additional details. This tool requires Internet access to http://gcr.io

# ./k10tools primer upgrade
Catalog Volume Disk Usage:
  Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf         20G  1.3G   19G   7% /mnt/k10state

Current K10 Version: 4.5.5
Latest K10 Version: 4.5.6
Helm Install: true

* To upgrade successfully you must have at least 50% free in catalog storage

Recommended upgrade path:
  helm repo update && \
    helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \
    helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml --version=4.5.6

Generic Volume Snapshot Capabilities Check

The k10tools primer gvs-cluster-check command can be used to check if the cluster is compatible for K10 Generic Volume Snapshot. K10 Generic backup commands are executed on a pod running kanister-tools image and checked for appropriate output. Use -n flag to provide namespace. By default, kasten-io namespace will be used.

# ./k10tools primer gvs-cluster-check
  Validate Generic Volume Snapshot:
    Pod Created successfully  -  OK
    GVS Backup command executed successfully  -  OK
    Pod deleted successfully  -  OK

K10 Generic Storage Backup Sidecar Injection

The k10tools k10genericbackup can be used to make Kubernetes workloads compatible for K10 Generic Storage Backup by injecting a Kanister sidecar and setting the forcegenericbackup=true annotation on the workloads.

## Usage ##
# ./k10tools k10genericbackup --help

k10genericbackup makes Kubernetes workloads compatible for K10 Generic Storage Backup by
injecting a Kanister sidecar and setting the forcegenericbackup=true annotation on the workloads.
To know more about K10 Generic Storage Backup, visit https://docs.kasten.io/latest/install/generic.html

Usage:
  k10tools k10genericbackup [command]

Available Commands:
  inject      Inject Kanister sidecar to workloads to enable K10 Generic Storage Backup
  uninject    Uninject Kanister sidecar from workloads to disable K10 Generic Storage Backup

Flags:
      --all-namespaces         resources in all the namespaces
  -h, --help                   help for k10genericbackup
      --k10-namespace string   namespace where K10 services are deployed (default "kasten-io")
  -n, --namespace string       namespace (default "default")

Global Flags:
  -o, --output string   Options(json)

Use "k10tools k10genericbackup [command] --help" for more information about a command.


## Example: Inject a Kanister sidecar to all the workloads in postgres namespace ##
# ./k10tools k10genericbackup inject all -n postgres

Inject deployment:

Inject statefulset:
  Injecting sidecar to statefulset postgres/mysql
  Updating statefulset postgres/mysql
  Waiting for statefulset postgres/mysql to be ready
  Sidecar injection successful on statefulset postgres/mysql!  -  OK
  Injecting sidecar to statefulset postgres/postgres-postgresql
  Updating statefulset postgres/postgres-postgresql
  Waiting for statefulset postgres/postgres-postgresql to be ready
  Sidecar injection successful on statefulset postgres/postgres-postgresql!  -  OK

Inject deploymentconfig:
  Skipping. Env is not compatible for Kanister sidecar injection

CA Certificate Check

The k10tools debug ca-certificate command can be used to check if the CA certificate is installed properly in K10. The -n flag can be used to provide namespace and it defaults to kasten-io. More information on installation process.

# ./k10tools debug ca-certificate
  CA Certificate Checker:
    Fetching configmap which contains CA Certificate information : custom-ca-bundle-store
    Certificate exists in configmap  -  OK
    Found container : aggregatedapis-svc to extract certificate
    Certificate exists in container at /etc/ssl/certs/custom-ca-bundle.pem
    Certificates matched successfully  -  OK

Installation of K10 in OpenShift clusters

The k10tools openshift prepare-install command can be used to prepare an OpenShift cluster for installation of K10. It extracts a CA Certificate from the cluster, installs it in the namespace where K10 will be installed, and generates the helm command to be used for installing K10. The -n flag can be used to provide the namespace where K10 will be installed. The default namespace is kasten-io. --recreate-resources flag recreates resources that may have been created by previous execution of this command. Set --insecure-ca flag to true if Certificate Issuing Authority is not trusted.

# ./k10tools openshift prepare-install
Openshift Prepare Install:
  Certificate found in Namespace 'openshift-ingress-operator' in secret 'router-ca'  -  OK
  Checking if namespace 'kasten-io' exists
  Namespace 'kasten-io' exists  -  OK
  Created configmap 'custom-ca-bundle-store' with custom certificate in it  -  OK
  Searching for Apps Base Domain Name in Ingress Controller
  Found Apps Base Domain 'apps.test.aws.kasten.io'  -  OK
  Created Service Account 'k10-dex-sa' successfully  -  OK

Please use below helm command to start K10 installation
--------------------------------------------------------------------
 helm repo add kasten https://charts.kasten.io/
 helm install k10 kasten/k10 --namespace=kasten-io \
 --set scc.create=true \
 --set route.enabled=true \
 --set route.tls.enabled=true \
 --set auth.openshift.enabled=true \
 --set auth.openshift.serviceAccount=k10-dex-sa \
 --set auth.openshift.clientSecret=<your key will be here automatically>\
 --set auth.openshift.dashboardURL=https://k10-route-kasten-io.apps.test.aws.kasten.io/k10/ \
 --set auth.openshift.openshiftURL=https://api.test.aws.kasten.io:6443 \
 --set auth.openshift.insecureCA=false \
 --set cacertconfigmap.name=custom-ca-bundle-store