Transforms

Transforms enable modifications to Kubernetes resources on restore. Restore actions may include transforms to modify resources prior to creation in the target environment.

Transforms are loosely modeled on JSON Patch (RFC-6902) which structures patches into patch documents containing patch operations. Transforms follow a similar structure to JSON Patch, with transform documents containing transform commands. Transforms deviates from JSON Patch in its support for regular expressions and path globbing.

Transforms Overview

Transform Structure
Transform Commands
- Test
- Add
- Remove
- Copy
- Move
- Replace
Transform Example

Transform Structure 

A transform document is composed of transformation commands to be performed on a subject. Each transformation command within the document is performed in sequence. Processing halts if a transform command results in an error. If the error is the result of a test failure processing will continue to the next document, otherwise the restore will fail.

The example below contains two transform documents that contains two commands. The first command, test, checks for the presence of the 'metadata/labels/release' element within a deployment resource. The copy command following the test will execute if the test succeeds.

The second document will be evaluated independent of the first document. The second document replaces the "/metadata/labels/component" key value with "webserver" only if the value is "nginx".

transforms:
- subject:
    # Group, version, resource and/or name of the resource for transformation
    resource: deployments
  # ``json`` key below indicates JSON patch document object used to perform
  # a JSON patch-like transform
  json:
  - op: test
    # reference for operation. mandatory.
    path: '/metadata/labels/release'
  - op: copy
    # source reference for operation. only valid for the 'move' and 'copy' operations
    from: '/metadata/labels/release'
    # target reference for operation. mandatory.
    path: '/spec/template/metadata/labels/release'
    # apply regex to match expression. Valid for 'remove', 'replace', 'move',
    # 'copy' and 'test'.
    regex: 'prod-v.*'
    # value is any json structure. only valid for 'add', 'replace'
    # and 'test' operation
    value: 'prod'
# Second transform document also applies transform to deployments.
# Transforms on a resource are guaranteed to be applied in the order
# specified in the restore action.
- subject:
    # Group, version, resource and/or name of the resource for transformation
    resource: deployments
  # ``json`` key below indicates JSON patch document object used to perform
  # a JSON patch-like transform
  json:
  - op: test
    # reference for operation. mandatory.
    path: '/metadata/labels/component'
  - op: replace
    # reference for operation. mandatory.
    path: '/metadata/labels/component'
    # use regex to test for presence of element value for replacement
    regex: 'nginx'
    # value is any json structure. only valid for 'add', 'replace'
    # and 'test' operation
    value: 'webserver'

Transforms supports six command operations test, add, remove, copy, move and replace:

test checks that an element exists.

add inserts a new element.

remove deletes an element.

copy duplicates an element, possibly altering the value.

move relocates a element, possibly altering the value.

replace replaces an existing element with a new element.

The subject property, as used in the example above, is used to transform a subset of resources on restore. Only those resources matching the group, version, resource or name will be selected from the restore-point. Any empty subject property will match all values. The subject above matches all deployment resources with any name, for any group or version.

Each command has an operation, op, and a path. op specifies the command operation to perform, path references the element within the resource to operate on.

copy and move operations contain two paths - from for the source element of the operation and path for the destination element of the operation.

value contains either a element for comparison or a new element for inclusion. Similarly regex can be used for comparing string elements or processing elements into new string elements.

Paths 

Every command has at least one path that references element(s) to operate on. path and from are the only command properties that contain paths. Paths consist of keys delineated by slashes (/). Keys can be object property names or array indexes. For example, "/spec/template/spec/containers/0" references the first container, nginx, within the Deployment resource in the example below.

When adding an element to an array, "-" can be used in the path to add an element at the and of the array.

Per RFC-6901, paths may contain character escapes that allow inclusion of / characters within path keys. For example "/metadata/annotations/k10~1injectKanisterSidecar" will reference the k10/injectKanisterSidecar key on the annotations object within the resource metadata. ~ characters may be included by using ~0.

Regex and Value 

The function of value and regex properties depend on the operation performed:

test with value compares the element at path to value for an exact match.

test with regex compares the string element at path to the regular expression in regex

copy and move that include a regex property operate only on from string elements that match the regular expression in regex.

copy and move that include a regex and a value property operate only on from string elements that match regex and replace destination path elements with expanded value strings. Capturing groups in value are replaced to produce new values from regex expressions.

replace that include a regex property operate only on path string elements that match the regular expression in regex.

replace that include a regex and a value property operate only on path string elements that match regex and replace destination elements with expanded value strings. Capturing groups in value are replaced to produce new values from regex expressions.

value and regex do not apply to the remove and add operations. A test operation with both value and regex properties will produce and error.

See re2 for the syntax accepted by regex.

Capturing groups are referenced in value with identifiers starting with a dollar-sign, "$", followed by the index of the capturing group.

The following example copies the mountPath from the first volumeMounts entry to the second using regex and value with capturing group replacement. The example uses the Deployment resource below. The parent path, /etc/nginx from mountPath is preserved and copied into the second volumeMounts as /etc/nginx/config.

op: copy
from: "spec/template/spec/containers/0/volumeMounts/0/mountPath"
path: "spec/template/spec/containers/0/volumeMounts/1/mountPath"
regex: "([a-z/]+)/ssl"
value: "$1/config"

Path Wildcards 

Path wildcards allow path or from to apply to sets of elements. A wildcard can be used in place of a path key to refer to all keys of an object or array. Valid wildcards are * and ** - * references all keys within one element, ** references all keys across multiple consecutive elements.

Using the deployment example below, all spec app labels can be accessed with "spec/**/app", expanding to two concrete paths:

[
  "/spec/selector/matchLabels/app"
  "/spec/template/metadata/labels/app"
]

In the example below, all metadata labels can be accessed with the path "**/metadata/labels/*" expanding to the four concrete paths below:

[
  "/metadata/labels/app"
  "/metadata/labels/version"
  "/metadata/labels/release"
  "/spec/template/metadata/labels/app"
]

Reference groups can be used in the path property of the copy and move operations to form new groups of destination paths. Reference groups allow for constructing new destination paths from source paths with wildcards. Reference groups have keys that start with $ followed by a numeric index of the wildcard. The index refers the wildcard ordinal, counting from one. During processing, the reference group will be replaced with concrete values of the indexed wildcard.

The following copy command example copies all selector matchLabels to the spec template metadata.

op: copy
from: "/**/selector/matchLabels/*"
path: "/spec/template/metadata/labels/$2"

Transform Commands 

All of the available command forms are listed below. Examples apply to the Deployment resource below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
    version: 1.7.9
    release: canary
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: containers.example.com/nginx:1.7.9
        ports:
        - containerPort: 443
        - containerPort: 80
        volumeMounts:
        - mountPath: /etc/nginx/ssl
          name: secret-volume
        - mountPath: /etc/nginx/conf.d
          name: configmap-volume

Test 

op: test
path: "/metadata/name"

Test that element at path exists. path may contain wildcards. If a test fails, the remainder of the current transform document will be skipped and processing of the next transform document will begin.

op: test
path: "/metadata/labels"
value: { app: nginx, release: "canary" }

Test that element at path equals element in value. Order of map keys is not significant. Otherwise, set of map keys and values must match exactly. Arrays must be the same size and elements are compared in order. If path contains wildcards, all elements referenced by path must equal value. If a test fails, the remaining commands in the current transform document will be skipped and processing of the next transform document will begin.

op: test
path: "/metadata/name"
regex: "nginx.*"

Test that string element at path exists and matches regular expression in regex. path may contain wildcards. If a test fails, the remaining commands in the current transform document will be skipped and processing of the next transform document will begin.

Add 

op: add
path: "/metadata/label/release"
value: canary

Add value at path. If path references a map, the key will be added or replaced. If path references an array, a new value will be inserted. value must be a value JSON/YAML element. path may contain wildcards.

The transform will fail if the path does not exist. The last key of the path may contain a new map key or an array index.

Remove 

op: remove
path: "/metadata/label/release"

Remove element at path. path may contain wildcards.

The transform will fail if path does not exist.

Copy 

op: copy
from: "/metadata/label"
path: "/spec/selector/matchLabel"

Copy element at from to path. If path references a map, element will be added/replaced as in add above. If path references an array, copy will insert element as in add above. If from contains wildcards, path may contain wildcard variables. If from is a concrete path, path may contain wildcards.

The transform will fail if from or path does not exist. The last key of path may contain a new map key or an array index.

op: copy
from: "metadata/labels/app"
path: "metadata/labels/release"
regex: "(.*)"
value: "prod-$1"

Copy string element at from to path. If path references a map, node will be added/replaced as in add above. If path references is array, copy will insert element as in add above. If from element matches regular expression in regex, perform copy of element with replacement of capturing groups in value. If from contains wildcards, path may contain wildcard variables. If from is a concrete path, path may contain wildcards.

The transform will fail if from element does not exist or is not a string. The transform will fail if path does not exist. The last key of path may contain a new map key or an array index.

Move 

op: move
from: "/spec/template/spec/containers/1/port"
path: "/spec/template/spec/containers/0/port"

Move element at from to path. If path references a map, element will be added/replaced as in add above. If path references an array, copy will insert element as in add above.

The transform will fail if from or path does not exist. The last key of path may contain a new map key or an array index.

op: move
from: "/metadata/labels/version"
path: "/spec/template/metadata/labels/version"
regex: "(.*)"
value: "v$1"

Move string element at from to path. If path references a map, element will be added/replaced as in add above. If path references an array, copy will insert element as in add above. If regular expression in regex matches element at from, perform replacement of capturing groups in value. If from contains wildcards, path may contain wildcard variables. If from is a concrete path, path may contain wildcards.

The transform will fail if from element does not exist or is not a string. Transform will fail if path does not exist. The last key of path may contain a new map key or an array index.

Replace 

op: replace
path: "/spec/replicas"
value: 5

Replace element at path with the element in value. path must reference an existing element. path may contain wildcards.

The transform will fail if path element does not exist.

op: replace
path: "/spec/template/spec/template/containers/0/image"
regex: ".*[/]([a-z/]+):([0-9.]+)"
value: "$2/$1"

Replace element at path with string element in value. If path references a map, node will be added/replaced as in add above. If path references is array, the element referenced will be replaced. If element at path matches regular expression in regex, replace element at path with value. Any capturing groups in value will be expanded value. path may contain wildcard variables.

The transform will fail if path element does not exist or is not a string.

Transform Example 

Transform to change the storage class on a persistent volume claim (pvc):

transforms:
- subject:
  resource: persistentvolumeclaims
  json:
  - op: replace
    path: /spec/storageClassName
    regex: ^ssd$
    value: gp2

Resource to operate on:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pic-gallery
  namespace: gallery-app
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: ssd
  volumeMode: Filesystem

Transformed resource:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pic-gallery
  namespace: gallery-app
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: gp2
  volumeMode: Filesystem