Configuring an HTTP(S) Proxy

Veeam Kasten can run in clusters where outbound traffic must pass through a corporate HTTP(S) proxy. Proxy settings are configured once through Helm and applied to every Veeam Kasten service and to the pods Veeam Kasten creates while it runs.

How It Works

Veeam Kasten does not implement proxy handling of its own. When a proxy is configured, the Helm chart sets the conventional HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables, in upper and lower case, on:

• every Veeam Kasten service container (the gateway, which only forwards requests to in-cluster services, is excluded),
• the bundled Prometheus server,
• the pods Veeam Kasten creates at run time, such as data movers, Kanister function pods, repository servers and the debug pods created by k10tools,
• the Kanister sidecar that Veeam Kasten injects into application workloads for generic storage backup.

All HTTP clients inside these pods honor the variables. Backups, exports, restores, imports, usage reporting and dashboard version checks go through the proxy. Exceptions are destinations that match NO_PROXY and the data transfer to Veeam Backup & Replication, which always connects directly. For more information, see Veeam Backup & Replication.

With no proxy configured, the chart sets none of these variables and nothing changes.

Configuring the Proxy

Set the following Helm values:

Helm value	Description
global.proxy.httpProxy	Proxy URL for plain HTTP requests.
global.proxy.httpsProxy	Proxy URL for HTTPS requests. Nearly all Veeam Kasten traffic is HTTPS, so this is the value that matters. The URL scheme describes how to reach the proxy, not the destination: http://proxy.example.com:3128 is the usual form, https:// is supported for proxies that accept TLS connections.
global.proxy.noProxy	Comma-separated list of hosts, domain suffixes, IP addresses and CIDRs that must bypass the proxy. Appended to the entries Veeam Kasten adds on its own, see Excluding Destinations from the Proxy.
global.proxy.openshift.caCertsAutoExtraction	OpenShift only, default false. When true, extracts the cluster's CA certificates, including the cluster-wide proxy's trustedCA bundle, into Veeam Kasten's trust bundle. See OpenShift.

Setting either httpProxy or httpsProxy enables proxy support.

proxy-values.yaml

global:
  proxy:
    httpProxy: http://proxy.example.com:3128
    httpsProxy: http://proxy.example.com:3128
    noProxy: 10.96.0.0/12,10.244.0.0/16,192.168.10.0/24

helm upgrade k10 kasten/k10 --namespace=kasten-io --reuse-values \
    -f proxy-values.yaml

Note

The preceding noProxy example is illustrative. It must contain the service and pod CIDRs of your cluster and the other cluster-specific destinations described in the next section. Without at least the service CIDR, Veeam Kasten pods send their Kubernetes API traffic to the proxy and fail to start.

To remove the proxy configuration, run helm upgrade ... --reuse-values --set global.proxy=null. If you upgrade with a complete values file instead of --reuse-values, delete the proxy values from the file. When passing noProxy with --set, escape the commas (\,) or use a values file.

Excluding Destinations from the Proxy

NO_PROXY lists the destinations that are reached directly. The final value that Veeam Kasten pods receive has two parts.

What Veeam Kasten adds. Veeam Kasten always adds the names of its own services and the .svc and .<cluster domain> suffixes. As a result, every in-cluster DNS name in any namespace bypasses the proxy. The cluster domain is the DNS domain of Kubernetes services, which is cluster.local on most clusters, including OpenShift. On clusters with a different domain, Veeam Kasten uses the cluster.domainName Helm value.

What you must add. Veeam Kasten adds nothing that is addressed by IP, because those values differ from cluster to cluster. Name-based entries and CIDR entries cover different connections and you always need both: Veeam Kasten reaches the Kubernetes API server by its cluster IP, and no name entry can match an IP address.

Entry	Why	Required
Service CIDR	The Kubernetes API server is reached by its cluster IP. Repository servers and other Veeam Kasten components are reached by service IP as well.	Always
Pod CIDR	Data movers and Kanister pods connect to other pods by pod IP.	Always
Node or VPC CIDR	Node-local endpoints and host-network services.	Recommended
Loopback addresses localhost, 127.0.0.1 and ::1	Veeam Kasten's own components never send loopback traffic through the proxy. Some tools that blueprints run, such as curl, send localhost to the proxy unless it is listed. This applies to tools in the blueprint's own images and in the injected Kanister sidecar.	When blueprints run such tools against localhost
Cloud metadata endpoint, 169.254.169.254 (AWS, Azure, GCP) and metadata.google.internal (GCP)	Instance-role and managed-identity credentials are fetched from a link-local address that the proxy cannot reach. GKE Workload Identity uses the same address, and EKS Pod Identity uses 169.254.170.23. Not needed with EKS IAM Roles for Service Accounts or Azure Workload Identity, whose token exchange goes through the proxy.	When instance or workload identity credentials are used
OpenShift internal API endpoint api-int.<cluster base domain> and the application domain .apps.<cluster base domain>	The internal API endpoint and Veeam Kasten's own dashboard route must not go through the proxy. The cluster base domain combines the cluster name and the base domain, for example mycluster.example.com. See Discovering the Values.	On OpenShift
Object storage that is reachable inside your network, for example an on-premises S3-compatible store	Otherwise exports leave through the proxy.	Site-specific
Veeam Backup & Replication server	The data transfer connects directly; excluding the server keeps the REST API connection direct as well, see Veeam Backup & Replication.	Recommended when Veeam Backup & Replication is used
Internal identity providers, container registries, OpenTelemetry collectors, multi-cluster peer ingress names	Only when they must not be proxied.	Site-specific

Entry Format

• Entries are separated by commas.
• A host name matches that host and its subdomains; an entry with a leading dot (.example.com) matches subdomains only.
• Single IP addresses and CIDR ranges are supported.
• A single * disables the proxy for all destinations, and *.example.com is treated like .example.com. Other wildcard patterns are not supported.
• A port can be appended to a host name or a single IP address, but not to a CIDR range.
• These rules apply to Veeam Kasten's own components. Tools that blueprints run in their own images, such as curl or the AWS CLI, may not support CIDR entries or *. prefixes.

Discovering the Values

The commands in this section depend on your Kubernetes version, network plugin and cloud service. If a command returns no value or fails, find the network ranges in the documentation of your Kubernetes service provider.

# Service CIDR: the API server's cluster IP lies inside it
kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}'
# Kubernetes 1.33 and later
kubectl get servicecidr

# Pod CIDR
kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}'

# Node addresses
kubectl get nodes -o wide

On OpenShift, the cluster network configuration contains both CIDRs. The cluster DNS configuration contains the cluster base domain. If a cluster-wide proxy is configured, its status.noProxy field contains a computed NO_PROXY list that you can reuse:

oc get network.config cluster -o jsonpath='{.spec.serviceNetwork} {.spec.clusterNetwork[*].cidr}'
oc get proxy cluster -o jsonpath='{.status.noProxy}'
oc get dns cluster -o jsonpath='{.spec.baseDomain}'            # cluster base domain
oc get ingresses.config cluster -o jsonpath='{.spec.domain}'   # add with a leading dot

Required Outbound Access

The proxy must allow CONNECT connections to port 443 for the HTTPS destinations Veeam Kasten uses:

• charts.kasten.io, for the Helm repository and the dashboard's version check,
• analytics.kasten.io, for usage reporting,
• the endpoints of your location profiles, including Veeam Vault and its service vdc-vault-integration-prod.veeamdatacloud.com,
• the APIs of your cloud provider used for snapshots and infrastructure profiles,
• your identity provider, container registries and other integrations.

Usage reporting and the dashboard's version check are turned off when metering.mode=airgap is set.

Destinations addressed with http://, such as an S3-compatible store without TLS, are sent through httpProxy as plain forwarded requests. They bypass the proxy when only httpsProxy is set.

Container images are pulled by the container runtime on the nodes, which follows the node's proxy configuration, not global.proxy.

If a location profile uses a non-standard port, the proxy must allow CONNECT to that port as well.

Platform Notes

OpenShift

• The OpenShift cluster-wide proxy (proxy/cluster) configures the platform itself, such as the nodes, image pulls and cluster operators. It does not set proxy environment variables in user workloads, including Veeam Kasten. Configure global.proxy.* in addition, normally pointing at the same proxy. For noProxy, reuse status.noProxy of the cluster proxy and add .apps.<cluster base domain>, which the platform's list does not include.
• Trust can be configured automatically. If the proxy's CA certificate is the trustedCA of the cluster-wide proxy, set global.proxy.openshift.caCertsAutoExtraction=true. Before each install and upgrade, Veeam Kasten then collects the trustedCA bundle and the cluster's own CA certificates. It stores them in the custom-ca-bundle-store ConfigMap, or in the ConfigMap named by cacertconfigmap.name. All Veeam Kasten pods mount this ConfigMap, so no cacertconfigmap values are needed. The same extraction runs when OpenShift authentication is enabled; see Using Trusted Root Certificate Authority Certificates for TLS and Extracting OpenShift CA Certificates. The extraction fails the install when the cluster exposes no certificates to extract. This is the case on hosted control plane clusters, such as ROSA with hosted control planes, unless the cluster-wide proxy has a trustedCA. Leave the setting at false there and configure cacertconfigmap manually if the proxy intercepts TLS.
• When Veeam Kasten is installed through the Operator, the K10 custom resource's spec holds the same values: set spec.global.proxy.*. The Operator applies changes to the custom resource by rolling out the affected pods.
• Changing httpProxy, httpsProxy or noProxy of the cluster-wide proxy makes OpenShift reboot the nodes; changing its trustedCA does not. Changing Veeam Kasten's own values is an ordinary Helm upgrade.

SUSE Virtualization (Harvester) and RKE2

Harvester's http-proxy setting applies to Harvester's system components only; there is no platform-level proxy for user workloads. Configure Veeam Kasten through global.proxy.*. noProxy needs the service and pod CIDRs, the node subnet and your internal DNS suffix; on-premises clusters normally do not need a cloud metadata entry.

Azure Kubernetes Service

AKS clusters created with the HTTP proxy feature inject proxy environment variables into all pods through an admission webhook of their own. If your cluster uses it, Veeam Kasten pods already receive the AKS proxy settings; setting global.proxy.* as well configures the same variables twice, so use one mechanism. The AKS trustedCa setting applies to the nodes only; pods need cacertconfigmap for a TLS-intercepting proxy.

Other Admission Policies

Cluster policies that add proxy environment variables to pods (for example a Kyverno policy or a MutatingAdmissionPolicy) modify the same variables the chart sets. Use one mechanism.

TLS-Intercepting Proxies

A proxy that inspects TLS traffic re-signs server certificates with its own CA. Veeam Kasten must trust that CA:

• On any Kubernetes distribution, add the proxy's CA certificate through the cacertconfigmap Helm values as described in Using Trusted Root Certificate Authority Certificates for TLS. The pods Veeam Kasten creates at run time receive the same CA bundle together with the proxy environment.
• On OpenShift, the certificates can be extracted from the cluster-wide proxy's trustedCA by setting global.proxy.openshift.caCertsAutoExtraction=true, see OpenShift.
• When the proxy URL uses the https:// scheme, the proxy's own server certificate must be trusted in the same way.

Note

The CA bundle is mounted into the pods at start. After you change the certificate, restart the Veeam Kasten pods, for example with kubectl --namespace=kasten-io rollout restart deployment.

Authenticated Proxies

Proxies that require Basic authentication are supported by putting the credentials in the proxy URL, for example http://user:pass%21word@proxy.example.com:3128. Reserved characters in the username and password must be percent-encoded.

warning

Credentials in the proxy URL are stored in plain text in the following places:

• The Helm release values, or the Veeam Kasten instance of an Operator install.
• The pod specifications of Veeam Kasten pods.
• The pod specifications of application workloads with an injected Kanister sidecar.

Veeam Kasten masks the password in its own logs and in the support bundle.

Application Workloads: Kanister Sidecar and Blueprints

The webhook of Veeam Kasten injects the Kanister sidecar for generic storage backup. The sidecar receives the proxy settings of the Veeam Kasten release each time the workload is created or updated. After you change or remove global.proxy.*, roll out the affected workloads again, for example with kubectl rollout restart, to apply the new settings. Because the settings are part of the pod template, the first update after a settings change rolls out the pods of the workload. This includes updates that otherwise do not restart pods, such as a label or annotation change. Scaling a workload does not trigger this. Sidecars added with k10tools k10genericbackup inject are updated by running the command again. Workloads restored or imported from another cluster receive the settings of the cluster they are restored into.

The pods that Kanister functions such as KubeTask and PrepareData create, and kando location push and pull running in them, honor the proxy settings as well. Pods created by MultiContainerRun and commands that KubeExec runs in application containers do not receive them.

warning

NO_PROXY keeps in-cluster traffic direct by matching the .svc and cluster domain suffixes, so a service must be addressed by a name that contains them. A blueprint that uses https://my-database.my-namespace.svc:9200 connects directly. A blueprint that uses the short name https://my-database:9200 matches no NO_PROXY entry, so the request goes to the proxy. The proxy cannot reach services inside the cluster. Use fully qualified service names in blueprints, or add the short name to global.proxy.noProxy.

Veeam Backup & Replication

Only the connection to the Veeam Backup & Replication REST API is able to use the proxy. The data transfer to Veeam Backup & Replication repositories cannot be proxied. It always connects directly from the data mover pods to the backup server, so the cluster needs a direct network path to the backup server regardless of the proxy.

The REST API listens on port 9419 by default. Many proxies do not allow CONNECT to this port. With Veeam Backup & Replication versions earlier than 13.2, a TLS-intercepting proxy also breaks the certificate check of the Veeam Repository Location Profile, because Veeam Kasten records the proxy's certificate instead of the server's. Because the backup server must be reachable directly anyway, add its address to global.proxy.noProxy so that the REST API connection also bypasses the proxy.

Veeam Kasten Primer and k10tools

• The Veeam Kasten Primer runs its pre-flight checks against the Kubernetes API only and needs no proxy.
• k10tools repository commands that use --create-pod, such as repository diagnose and repository connect, create a debug pod in the Veeam Kasten namespace. The pod receives the same proxy settings as every other Veeam Kasten pod. The proxy settings of the shell running k10tools only affect the connections k10tools makes from your machine. The debug pod is not removed automatically.
• k10tools k10genericbackup inject gives the injected sidecar the proxy settings of the Veeam Kasten installation, not those of the shell it runs in.

Troubleshooting

Errors are reported in the status of the failed action (for example the RunAction or ExportAction), including the underlying network or certificate error.

Symptom	Cause	Resolution
Most Veeam Kasten pods crash-loop after enabling the proxy; logs show Kubernetes API requests to the cluster IP failing with Forbidden, Service Unavailable or Bad Gateway, or with proxyconnect tcp: dial tcp <proxy>: i/o timeout	The service CIDR is missing from noProxy, so API server traffic goes to the proxy	Add the service CIDR to global.proxy.noProxy
Export fails within a few minutes with x509: certificate signed by unknown authority	The CA of a TLS-intercepting proxy is not trusted	Configure the CA, see TLS-Intercepting Proxies
Helm install or upgrade on OpenShift fails with pre-install hook failed or pre-upgrade hook failed for the k10-extract-ocp-ca-cert-job Job (prefixed with the release name)	global.proxy.openshift.caCertsAutoExtraction=true on a cluster that exposes no certificates to extract, for example a hosted control plane cluster without a cluster-wide proxy trustedCA	Set the value back to false and configure cacertconfigmap manually if needed
TLS error when connecting to the proxy address itself	The https:// proxy's own certificate is not trusted	Trust the proxy's certificate
Action stalls for a long time, then fails with i/o timeout	Direct egress is blocked and the pod has no proxy settings, or the proxy cannot reach the destination	Check the pod's environment and the proxy's access log
Errors contain Proxy Authentication Required, or the proxy log shows 407	Missing or wrongly encoded credentials	Fix the proxy URL
Proxy access log shows CONNECT requests to a single-label host name such as catalog-svc	Veeam Kasten's NO_PROXY entries were overridden, or a blueprint uses a short service name	Verify the environment; use fully qualified names
Veeam Repository Location Profile fails with Forbidden; proxy log shows a denied CONNECT to port 9419	The proxy does not allow CONNECT to the REST API port	Allow the port on the proxy, or add the backup server to global.proxy.noProxy
Location profile validation times out after about a minute; proxy log shows a 503 for the destination	The proxy cannot reach the destination	Add the destination to noProxy or fix the proxy's routing

Limitations

• Only the REST API connection to Veeam Backup & Replication is able to use the proxy. The data transfer to the backup server cannot be proxied and always connects directly.
• Only Basic authentication to the proxy is supported.
• Pods created by the Kanister MultiContainerRun function and commands that KubeExec runs in application containers do not receive the proxy settings.
• global.proxy.noProxy is not derived automatically; the cluster-specific entries must be provided.
• In a multi-cluster setup, the peer cluster's ingress must be reachable through the proxy or listed in noProxy, on both the primary and the secondary cluster.