Integrating Security Information and Event Management (SIEM) Systems
Integrating Security Information and Event Management (SIEM) Systems
Inhibiting data protection software and deleting backup data are examples
of actions that may be taken by a malicious actor before proceeding to the
next stage of an attack, such as file encryption. Prompt notification of such
potentially malicious behavior can help mitigate the impact of an attack.
To provide activity correlation and analysis, K10 can integrate with SIEM
solutions. SIEMs ingest and aggregate data from an environment,
including logs, alerts, and events, for the purpose of providing real-time
threat detection and analysis, and assisting in investigations.
As an application built upon Kubernetes CRDs and API Aggregation, K10 events
(e.g., creating a Location Profile resource) can be captured through the
Kubernetes audit log.
These events can then be ingested by a SIEM system. However, there are
situations where you may not have direct control over the Kubernetes audit
policy configuration for a cluster (or the kube-apiserver), especially when
using a cloud-hosted managed Kubernetes service. This limitation can impact the
detail available in Kubernetes API server responses that can be collected for
audit events and the customization of log transmission.
For this reason, K10 provides an extended audit mechanism to enable direct
ingestion of K10 events into a SIEM system, independently of Kubernetes cluster
audit policy configurations. Furthermore, this extended mechanism allows more
fine-tuned control over how to store these logs, including options like
file-based and cloud-based storage.
The audit policy applied to K10's aggregated-apiserver is the following:
This section provides documentation on configuring each of these mechanisms
and includes example rules that a SIEM system can enable. Currently, we only
support an integration with Datadog Cloud SIEM.
By default, K10 is deployed to write these new audit event logs to
stdout (standard output) from the aggregatedapis-svc pod. You can
ingest these logs from an agent installed in the cluster and an example for
Datadog is shown below.
To disable, configure the K10 deployment with
--setsiem.logging.cluster.enabled=false.
Many SIEM solutions support ingestion of stdout log data from Kubernetes
applications using an agent deployed to the cluster. If an agent-based approach
is not available or not preferred, K10 offers the option to send these audit
events to a Location Profile. SIEM-specific tools can then be used to ingest
the log data from the object store.
Note
Currently, only AWS S3 Location Profiles are supported as a target for
K10 audit events.
By default, K10 is deployed with the ability to send these new audit event
logs to available cloud object stores. However, enabling this feature is just
the first step. The action of sending the logs depends on the creation or
update of an applicable K10 AuditConfig that
points to a valid LocationProfile. An example for Datadog is shown
below.
To disable the sending of these logs to AWSS3, you can configure the K10
deployment with the following command:
--setsiem.logging.cloud.awsS3.enabled=false.
To begin, you should first determine the name of your target Location Profile.
Next, define and apply an AuditConfig manifest to your K10 namespace. In
the example below, make sure to replace the target values for
spec.profile.name and spec.profile.namespace before applying.
Note
If the spec.profile.namespace is left blank, the default value
will be the namespace of the AuditConfig.
# Create AuditConfig manifest with Location Profile name/namespace specific
# to your environment.
$ cat > k10-auditconfig.yaml <<EOF
apiVersion: config.kio.kasten.io/v1alpha1kind: AuditConfigmetadata: name: k10-auditconfig namespace: kasten-iospec: profile: name: <LOCATION PROFILE NAME> namespace: <LOCATION PROFILE NAMESPACE>EOF# Apply AuditConfig manifest
$ kubectl apply -f k10-auditconfig.yaml -n kasten-io
config.kio.kasten.io/k10-auditconfig created
K10 event logs will now be sent to the target Location Profile bucket under the
k10audit/ directory. If you wish to change the destination path of the logs
within the bucket, configure the K10 deployment with
--setsiem.logging.cloud.path=<DIRECTORYPATHWITHINBUCKET>.
K10 integrates with Datadog Cloud SIEM to provide high-fidelity signal data
that can be used to detect suspicious activity and support security operators.
Review each of the sections below to understand how K10 event data can be sent
to Datadog. Both methods can be configured per cluster.
Setting up the Datadog Agent on a Kubernetes Cluster
The Datadog Agent can be installed on the Kubernetes cluster and used to
collect application logs, metrics, and traces.
Refer to Datadog Kubernetes
documentation for complete instructions on installing the Agent on the cluster.
For Datadog to ingest K10 event logs, the Agent must be configured with
log collection enabled
and an include_at_match global processing rule to match the K10-specific
pattern, (?i).*K10Event.*.
Here is an example of a values.yaml file for installing the Datadog Agent
using Helm:
datadog:apiKey:<YOUR DD API KEY>appKey:<YOUR DD APP KEY>site:datadoghq.comlogs:enabled:truecontainerCollectAll:trueenv:-name:DD_LOGS_CONFIG_PROCESSING_RULESvalue:'[{"type":"include_at_match","name":"include_k10","pattern":"(?i).*K10Event.*"}]'
Refer to the Datadog
processing rules
documentation for instructions on alternative methods for configuring
processing rules.
Detection Rules define how Datadog analyzes ingested data and when to
generate a signal. Using these rules, K10 event data can be used to alert
organizations to specific activity that could indicate an ongoing security
breach. This section provides the details required to add example K10 rules to
Datadog Cloud SIEM.
Open the Datadog Cloud SIEM user
interface and select Detection Rules from the toolbar.
At the top right corner of the page, click the New Rule button.
Complete the form using the details below for each rule.
Note
Each rule should be configured to notify the appropriate services
and/or users. Since the specific configurations are unique to each
environment, they are not covered in the examples provided below.
The purpose of this rule is to detect deletions of K10
RestorePoint resources initiated by a user.
Typically, the removal of this type of resource would be the result of backup
data no longer being needed based on a policy's retention schedule and
performed directly by K10.
Note
Removal of a Kubernetes namespace containing RestorePoints may also
trigger this signal.
Rule Name
Kasten RestorePoints Manually Deleted
Rule Type
Log Detection
Detection Method
Threshold
Query
Trigger
deleted_k10_rps>0
Severity
Low
Tags
tactic:TA0040-impact
Use the following notification body to provide an informative alert:
## GoalDetect when Kasten RestorePoints (Kubernetes application backups) are beingmanually deleted by a user. This could be an indication that the environmenthas been compromised and Kasten backups are being deleted to prevent systemrecovery following an attack.## StrategyMonitor Kubernetes Audit logs to detect when a single user or non-K10service account deletes RestorePoint resources beyond the defined threshold.## Triage and response1. Determine if the user `{{@User.username}}` and IP address(es)`{{@network.ip.list}}` should be retiring backup data for`{{@ObjectRef.Namespace}}`. NOTE: This rule may also be triggered by thelegitimate deletion of the `{{@ObjectRef.Namespace}}` namespace to which theRestorePoint resources belong.2. If the action is legitimate, consider including the user in a suppressionlist. See [Best practices for creating detection rules with Datadog CloudSIEM][1] for more information.3. Otherwise, use the Cloud SIEM - User Investigation dashboard to see if theuser `{{@User.username}}` has taken other actions.4. If the results of the triage indicate that an attacker has taken theaction, begin your company's incident response process and investigation.[1]: https://www.datadoghq.com/blog/writing-datadog-security-detection-rules/#fine-tune-security-signals-to-reduce-noise
The purpose of this rule is to detect deletions of K10
RestorePointContent resources initiated by a
user. The removal of this type of resource should only be the result of backup
data no longer being needed based on a policy's retention schedule and
performed directly by K10.
Use the following notification body to provide an informative alert:
## GoalDetect when Kasten RestorePointContents (Kubernetes application backups)are being manually deleted by a user. This could be an indication that theenvironment has been compromised and Kasten backups are being deleted toprevent system recovery following an attack.## StrategyMonitor Kubernetes Audit logs to detect when a single user or non-K10service account deletes RestorePointContent resources beyond the definedthreshold.## Triage and response1. Determine if the user `{{@User.username}}` and IP address(es)`{{@network.ip.list}}` should be retiring backup data on cluster`{{cluster-name}}`.2. If the action is legitimate, consider including the user in a suppressionlist. See [Best practices for creating detection rules with Datadog CloudSIEM][1] for more information.3. Otherwise, use the Cloud SIEM - User Investigation dashboard to see ifthe user `{{@User.username}}` has taken other actions.4. If the results of the triage indicate that an attacker has taken theaction, begin your company's incident response process and investigation.5. If restoring applications from deleted RestorePointContents (RPCs) isrequired on this cluster, see [Recovering K10 From a Disaster][2] for moreinformation.[1]: https://www.datadoghq.com/blog/writing-datadog-security-detection-rules/#fine-tune-security-signals-to-reduce-noise[2]: https://docs.kasten.io/latest/operating/dr.html#recovering-k10-from-a-disaster
Note
Use of the cluster-name tag in both the query and notification
body requires capturing K10 event logs via Datadog Agent.
The purpose of this rule is to detect deletions of K10
ClusterRestorePoint resources initiated by a
user. The removal of this type of resource should only be the result of backup
data no longer being needed based on a policy's retention schedule and
performed directly by K10.
Use the following notification body to provide an informative alert:
## GoalDetect when Kasten ClusterRestorePoint (Kubernetes backups of cluster-scopedresources) are being manually deleted by a user. This could be an indicationthat the environment has been compromised and Kasten backups are beingdeleted to prevent system recovery following an attack.## StrategyMonitor Kubernetes Audit logs to detect when a single user or non-K10service account deletes ClusterRestorePoint resources beyond the definedthreshold.## Triage and response1. Determine if the user `{{@User.username}}` and IP address(es)`{{@network.ip.list}}` should be retiring backup data from cluster`{{cluster-name}}`.2. If the action is legitimate, consider including the user in a suppressionlist. See [Best practices for creating detection rules with Datadog CloudSIEM][1] for more information.3. Otherwise, use the Cloud SIEM - User Investigation dashboard to see if theuser `{{@User.username}}` has taken other actions.4. If the results of the triage indicate that an attacker has taken theaction, begin your company's incident response process and investigation.5. If restoring applications from deleted RPCs is required on this cluster,see [Recovering K10 From a Disaster][2] for more information.[1]: https://www.datadoghq.com/blog/writing-datadog-security-detection-rules/#fine-tune-security-signals-to-reduce-noise[2]: https://docs.kasten.io/latest/operating/dr.html#recovering-k10-from-a-disaster
Note
Use of the cluster-name tag in both the query and notification
body requires capturing K10 event logs via Datadog Agent.
