In this blog, we will be showcasing an integration between Kasten K10’s Kubernetes native data protection solution and Amazon GuardDuty, a threat detection solution that monitors your AWS accounts and workloads for malicious activity and delivers detailed security findings for visibility and remediation.
As the recognized leader in data protection for containerized environments, Kasten K10 ensures adequate protection of all your sensitive and mission-critical data in the event of unforeseen downtime. Kasten K10 new integrations with Amazon GuardDuty and Red Hat Advanced Cluster Security provide for additional layers of security by enabling early threat detection through wholistic monitoring and correlation of all relevant events across the environment. This makes it possible to counter potential attacks earlier than before, alerting SecOps teams and thereby buying them extra time to mount a response and keep the blast radius at a minimum.
Below we’ll cover the following areas:
- A perspective of cloud native security and Kasten K10’s role in data recovery
- What Amazon GuardDuty is and the benefits it provides
- The Kubernetes audit log and how Kasten K10 extends it
- A quick start guide to getting Amazon GuardDuty to monitor your Kasten K10-based cluster
Introduction
With the ever-increasing amount of data flowing through the cloud1, and the trope “data is king” becoming more true with each passing day, it’s more important than ever to protect against the rising wave of cyber attacks; 2021 had around 4100 security breaches with an estimated 22 billion records exposed2. The adoption of Security Information and Event Management (SIEM) systems aims to detect these threats in real-time and respond quickly in order to minimize any damage caused, but doesn’t protect against the data lost during these breaches.
Kasten K10 for Kubernetes makes backing-up and restoring data easy in the event of a security breach or unintended or unauthorized data manipulation. It’s a powerful cloud native application that automates application stack replication to a standby cluster for fast failovers, securely replicates backups to off-site storage, protects against broad infrastructure and hardware failures, and provides robust ransomware protection.
Kasten K10’s cloud native philosophy inherently makes it easy to integrate into security monitoring systems without much effort. Security monitoring is an important part of every cloud application, and any third-party application launched in a cloud environment should be able to be easily monitored for any possible threat, like picking up on any unauthorized activity – which could signify the first signs of an attack. Kasten K10 is no exception, and, as we’ll see in this article, integrates natively with security monitoring systems such as AWS GuardDuty by making use of the Kubernetes Audit.
We’ll walk through an introduction to AWS GuardDuty, discuss the Kubernetes Audit and how Kasten K10 natively integrates with it, see how to configure GuardDuty, and lastly find Kasten K10 events in CloudWatch logs which GuardDuty pulls from.
What Is GuardDuty?
GuardDuty is a security monitoring service provided by AWS that analyzes activity within your account from a multitude of data sources and provides security correlations based on a set of finding types for each of these. The one we’re the most concerned about in this context is the EKS control plane logs which gather the Kubernetes Audit logs to help protect and detect suspicious activity within your clusters.
When you start a cluster in EKS, it passes in an audit policy that gathers logs from all registered API groups and flows them directly into CloudWatch. From CloudWatch you can export your data to other third-party software if you wish to do so.
You can view the list of Kubernetes audit finding types that EKS protection for GuardDuty provides for threat detection within your clusters here. If one of these types is triggered, you’ll see a finding in your GuardDuty dashboard with more information on what caused it.
Kubernetes Audit and Kasten K10
All activity that is processed by the kube-apiserver can be logged into an audit event type which can be used for security monitoring. This means any calls to the core Kubernetes API, or to an extended API set up via the aggregation layer, could be logged.
The criteria for logging events is based on the audit policy that is supplied to the server on startup. In a managed service such as EKS, you cannot customize the server, however the generated audit policy file for EKS pulls all registered API groups and logs them at the metadata level. For more information on the Kubernetes Audit please refer to this document.
Configuring your clusters audit log will depend on your Kubernetes distribution. For example, when deploying with k3d, use the following flags to launch the kube-apiserver with a log audit backend:
k3d cluster create kube-audit-test \
--volume "PATH_TO_POLICY/audit-policy-minimal.yaml:/etc/kubernetes/audit/policy.yaml@server:0" \
--k3s-arg "--kube-apiserver-arg=audit-policy-file=/etc/kubernetes/audit/policy.yaml@server:0" \
--k3s-arg "--kube-apiserver-arg=audit-log-path=/etc/kubernetes/audit/audit.log@server:0" \
--k3s-arg "--kube-apiserver-arg=audit-log-maxsize=300@server:0" \
--k3s-arg "--kube-apiserver-arg=audit-log-maxbackup=3@server:0"There are two backend types for the kube audit: log and webhook. The log backend logs all the audit event locally and is ephemeral, while the webhook backend allows you to send the data to an external server. Extra options are provided for each type for flexible configuration but must be passed in as flags to the kube-apiserver on startup, something you may not have access to depending on where your clusters are deployed.
The kube audit is extendable, meaning you can write new backends if they implement the backend interface and pass them to extended API servers to add functionality as to where the audit data is sent.
The audit event type logs at four different levels: None, Metadata, Request, and RequestResponse. All of these add more information to the event object that’s logged, culminating in the full request and response body logged in the RequestResponse level. There is a potential scalability issue depending on how open the audit policy is, how big each audit event object is (Metadata vs. RequestResponse), and backend type.
Metadata provides the best balance between information for threat detection and scalability, with an example for getting a K10 passkey object shown:
{
"kind":"Event",
"apiVersion":"audit.k8s.io/v1",
"level":"Metadata",
"auditID":"ac1735bc-9713-407f-880f-6f6c28c88caf",
"stage":"ResponseComplete",
"requestURI":"/apis/vault.kio.kasten.io/v1alpha1/passkeys/k10MasterKey",
"verb":"get",
"user":{
"username":"system:admin",
"groups":[
"system:masters",
"system:authenticated"
]
},
"sourceIPs":[
"{IP_ADDRESS}"
],
"userAgent":"kubectl/v1.25.0 (darwin/arm64) kubernetes/a866cbe",
"objectRef":{
"resource":"passkeys",
"name":"k10MasterKey",
"apiGroup":"vault.kio.kasten.io",
"apiVersion":"v1alpha1"
},
"responseStatus":{
"metadata":{},
"code":200
},
"requestReceivedTimestamp":"2022-12-22T00:06:47.042962Z",
"stageTimestamp":"2022-12-22T00:06:47.047325Z",
"annotations":{
"authorization.k8s.io/decision":"allow",
"authorization.k8s.io/reason":""
}
}You can see the sourceIPs, userAgent, and user being provided. Some Kubernetes managed service providers will add extra information such as credentials, which we’ll show in the below section as applies to AWS EKS.
For a self-deployed Kubernetes cluster, a good policy to include all current Kasten K10 groups and resources would be the following:
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
rules:
- level: None
nonResourceURLs:
- /healthz*
- /version
- /openapi/v2*
- /timeout*
- level: Metadata
resources:
- group: "actions.kio.kasten.io"
resources: ["backupactions", "restoreactions", "exportactions", "importactions", "backupclusteractions", "restoreclusteractions", "retireactions", "runactions", "cancelactions", "reportactions", "upgradeactions"]
- group: "apps.kio.kasten.io"
resources: ["restorepointcontents", "clusterrestorepoints", "restorepoints", "ApplicationResource"]
- group: "vault.kio.kasten.io"
resources: ["passkeys"]
- group: "repositories.kio.kasten.io"
resources: ["restorepointrepositories", "storagerepositories"]
- group: "config.kio.kasten.io"
- group: "dist.kio.kasten.io"
- group: "auth.kio.kasten.io"
- group: "reporting.kio.kasten.io"
verbs: ["create", "update", "patch", "delete", "get"]
We don’t include the list verb as the UI makes many calls that can quickly overwhelm the logs. Some of the groups have the resources listed, and these can be easily changed to only show what you’re interested in.
They also all are shown at the Metadata level, but some could be added at the RequestResponse level to show the request and response body if interested in capturing that level of detail, such as in the upcoming Event custom resource.
How Kasten K10 Activity Gets Captured by the Kube Audit
There are many custom resources that power Kasten K10, created either via a Custom Resource Definition or the Aggregated API. The API groups and associated resources created by the CRD’s are:
Auth.kio.kasten.io
- k10clusterrolebindings, k10clusterroles Config.kio.kasten.io
- policies, policypresets, profiles, Dist.kio.kasten.io
- bootstraps, clusters, distributions Reporting.kio.kasten.io
- reports
And those created from the Aggregated API:
Actions.kio.kasten.io
- backupactions, restoreactions, exportactions, importactions, backupclusteractions,
- restoreclusteractions, retireactions, runactions, cancelactions, reportactions,
- upgradeactions
Apps.kio.kasten.io
- restorepointcontents, clusterrestorepoints, restorepoints, applicationresource
Vault.kio.kasten.io
- passkeys
Repositories.kio.kasten.io
- restorepointrepositories, storagerepositoriesYou can see a list of these by running
kubectl get apiservice | grep kio.kasten.iowhich will also show you the version and how they were created.
No matter how these custom resources are interacted with, the interaction is processed through the kube-apiserver. This flow creates an audit event based on the criteria set forth in the audit policy and flags passed to the kube-apiserver, such as which backend type to use, log or webhook, and configurable options for each.
This means that all external interactions with Kasten K10 natively leverage the Kubernetes Audit, and any security monitoring system that uses it as a data source can be used to monitor Kasten K10 for security.
Activity within Kasten K10 that does not relate to a custom resource will not be logged by the kube audit, since these were never processed by the kube-apiserver and thus were not related to the core Kubernetes API, or extended APIs via the aggregation layer.
Since this will be a custom resource created by the Aggregated API, all activity will flow through the kube-apiserver and thus will allow for the creation of audit events as per the audit policy. To get more specific information about each of these internal events, the audit event level would need to be Request or RequestResponse. This implies that managed services such as EKS will not log this information, and an extended backend will need to be built within Kasten K10 that provides this functionality.
See how to deploy Amazon GuardDuty in conjunction with Kasten by Veeam’s K10 to monitor your cluster’s posture in this technical how-to guide.
Conclusion
Data protection and management is at the core of Kasten K10 by Veeam, and its resulting architecture makes it a cloud-native application which natively integrates into GuardDuty. To achieve this it leverages the Kubernetes audit logs that are generated when requests come through the kube-apiserver; these requests are made on K10 custom resources that were created via CRD’s and the Aggregated API.
Out of the box, Kasten K10’s architecture enables customers to leverage GuardDuty for protection with as little as a few clicks. Try Kasten K10 free today!
