Execution Monitoring
At the core of Tetragon is the tracking of all executions in a Kubernetes cluster, virtual machines, and bare metal systems. This creates the foundation that allows Tetragon to attribute all system behavior back to a specific binary and its associated metadata (container, Pod, Node, and cluster).
Observe Tetragon execution events
Tetragon exposes the execution events over JSON logs and GRPC stream. The user can then observe all executions in the system.
Use the following instructions to observe execution events. These instructions assume you have deployed the Cilium demo application in your environment.
For a single node Kubernetes cluster, you can target the Tetragon DaemonSet with
a kubectl exec command:
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods xwing
This command runs tetra getevents -o compact --pods xwing in the single Pod
that is a member of the Tetragon DaemonSet. Because there is only a single node
in the cluster, it is guaranteed that the “xwing” Pod will also be running on
the same node and that Tetragon will be able to capture and report execution
events.
In a cluster with multiple nodes, you will need to ensure that the Tetragon Pod you use is located on the same node as the “xwing” Pod, so that it can capture the execution events.
You can use this command to get the name of the Tetragon Pod that is on the same Kubernetes node as the “xwing” Pod:
POD=$(kubectl -n kube-system get pods -l 'app.kubernetes.io/name=tetragon' -o name --field-selector spec.nodeName=$(kubectl get pod xwing -o jsonpath='{.spec.nodeName}'))
Once you have the identified the matching Pod, then target it with a kubectl exec to run the tetra getevents command.
kubectl exec -ti -n kube-system $POD -c tetragon -- tetra getevents -o compact --pods xwing
Because the Tetragon Pod where you are running tetra getevents in on the same
node as the “xwing” Pod, the command will return the execution events captured
by Tetragon.
docker exec tetragon tetra getevents -o compact
The tetra get-events -o compact command returns a compact form of the execution
events. To trigger an execution event, you will run a curl command inside the
“xwing” Pod/container.
kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'docker exec -ti starwars-xwing-1 curl https://ebpf.io/applications/#tetragonThe CLI will print a compact form of the event to the terminal similar to the following output. The example output below is from Kubernetes; the Docker output is very similar.
🚀 process default/xwing /bin/bash -c "curl https://ebpf.io/applications/#tetragon"
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 60
The compact execution event contains the event type, the pod name, the binary
and the args. The exit event will include the return code; in the case of the
curl command above, the return code was 60.
For the complete execution event in JSON format remove the -o compact option
from the tetra getevents command.
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents --pods xwingkubectl exec -ti -n kube-system $POD -c tetragon -- tetra getevents --pods xwingdocker exec -ti tetragon tetra geteventsThe complete execution event includes a lot more details related to the binary
and event. See below for a full example of the execution event generated by the
curl command used above. In a Kubernetes environment this will include the
Kubernetes metadata include the Pod, Container, Namespaces, and Labels, among
other useful metadata.
Full output of a process execution event
{
"process_exec": {
"process": {
"exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4Njc0MzIxMzczOjUyNjk5",
"pid": 52699,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/curl",
"arguments": "https://ebpf.io/applications/#tetragon",
"flags": "execve rootcwd",
"start_time": "2023-10-06T22:03:57.700327580Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-10-06T21:52:41Z",
"pid": 49
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
},
"workload": "xwing"
},
"docker": "551e161c47d8ff0eb665438a7bcd5b4",
"parent_exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5",
"tid": 52699
},
"parent": {
"exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5",
"pid": 52699,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"arguments": "-c \"curl https://ebpf.io/applications/#tetragon\"",
"flags": "execve rootcwd clone",
"start_time": "2023-10-06T22:03:57.696889812Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-10-06T21:52:41Z",
"pid": 49
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
},
"workload": "xwing"
},
"docker": "551e161c47d8ff0eb665438a7bcd5b4",
"parent_exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjQ1MjQ1ODM5OjUyNjg5",
"tid": 52699
}
},
"node_name": "gke-john-632-default-pool-7041cac0-9s95",
"time": "2023-10-06T22:03:57.700326678Z"
}
What’s next
Execution events are the most basic event Tetragon can produce. To see how to use tracing policies to enable file monitoring see the File Access Monitoring section of the Getting Started guide. To see a network policy check the Networking Monitoring section.