Detecting Attacks with Linux auditd

Home / Runtime Security / Detecting Attacks with Linux auditd

Detecting Attacks with Linux auditd

The Linux Audit Framework is one of the most underused tools in a defender's arsenal. Built directly into the kernel, it can log every syscall made by every process on the system — with full context: who ran the process, what file it touched, what arguments it passed. Unlike application-layer logging, auditd cannot be bypassed by userspace tricks. If a process makes a syscall, the kernel sees it.

This tutorial builds a complete hands-on lab around a vulnerable C application that sends ICMP echo requests. The binary has an intentional command-injection flaw on its error path — a system() call fed unsanitized user input. You will configure audit rules to watch the binary's behavior, run it in both normal and attack modes, then read the audit trail to see exactly how the kernel recorded the attack. By the end, you will understand how to write targeted audit rules, decode audit records, correlate events across a session, and persist rules for production use.

This is the detection layer that operates below Falco, below eBPF tools, and below any application — because it sits in the kernel itself.

How the Linux Audit Framework Works

The audit framework has three layers:

Kernel audit subsystem — a ring buffer inside the kernel that intercepts syscalls matching your rules. When a match fires, the kernel writes structured records into the buffer. No userspace daemon can suppress this; the records exist in kernel memory before auditd ever sees them.

auditd daemon — a userspace daemon that reads from the kernel buffer via a netlink socket and writes records to /var/log/audit/audit.log. It handles log rotation and rate limiting. If auditd is not running, the kernel still buffers events (up to the configured buffer size) and can optionally block syscalls when the buffer is full — configurable via auditctl -f.

Rule types:

Rule type Example What it watches
Syscall rule -S execve -F exe=/bin/bash Any call to a specific syscall matching field filters
File/dir watch -w /etc/passwd -p rwa Read, write, attribute changes on a file or directory
Exclude rule -a never,exit -F msgtype=CWD Suppress specific record types to reduce noise

Key tools:

Tool Purpose
auditctl Add/remove/list rules in the running kernel; changes are lost on reboot
ausearch Query /var/log/audit/audit.log by key, syscall, file, UID, time, and more
aureport Generate summary reports (exec events, file events, login events)
augenrules Merge .rules files from /etc/audit/rules.d/ into the persistent ruleset

Prerequisites

A Debian-based Linux host (Ubuntu 20.04 or 22.04 recommended). The lab uses cap_net_raw to allow raw ICMP sockets without running the binary as root.

# Install auditd, C compiler, and capability tools
sudo apt-get update
sudo apt-get install -y auditd audispd-plugins gcc libcap2-bin

# Verify auditd is running
sudo systemctl status auditd

# Confirm audit log exists
sudo ls -la /var/log/audit/audit.log

Create the lab working directory:

mkdir -p ~/auditd-icmp-lab
cd ~/auditd-icmp-lab
mkdir -p /tmp/icmp-audit-lab

Lab Overview: The Vulnerable ICMP Tool

The lab binary does two things:

  1. Normal path — resolves a hostname via getaddrinfo, opens a raw ICMP socket using cap_net_raw, sends an ICMP echo request, and receives the reply.
  2. Error path (vulnerable) — when getaddrinfo fails, calls vulnerable_resolution_log() which builds a shell command string with snprintf and passes it to system(). Because the hostname comes directly from argv[1] without sanitization, an attacker can inject shell metacharacters.

The attack payload is:

notarealhost; echo pwned >/tmp/icmp-audit-lab/owned #
  • notarealhost — a hostname that will fail DNS resolution
  • ; echo pwned >/tmp/icmp-audit-lab/owned — shell command appended after the semicolon
  • # — comments out the >/dev/null redirect that follows in the format string, so the injected command actually writes to the owned file

Why this is a good auditd teaching vehicle: A legitimate ICMP ping tool has a well-defined syscall profile — socket(AF_INET, SOCK_RAW, IPPROTO_ICMP), sendto, recvfrom, close. It has no reason to call execve or write to arbitrary files. When those events appear in the audit trail, they are unambiguous indicators of compromise.

Step 1: Build the Lab Environment

Write the source file:

cat > icmp_audit_demo.c <<'EOF'
#include <arpa/inet.h>
#include <errno.h>
#include <netdb.h>
#include <netinet/ip_icmp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

static unsigned short checksum(void *data, int len) {
    unsigned short *buf = data;
    unsigned int sum = 0;

    for (; len > 1; len -= 2) {
        sum += *buf++;
    }
    if (len == 1) {
        sum += *(unsigned char *)buf;
    }
    sum = (sum >> 16) + (sum & 0xFFFF);
    sum += (sum >> 16);
    return (unsigned short)(~sum);
}

static void vulnerable_resolution_log(const char *host) {
    char cmd[512];

    /*
     * Intentionally vulnerable: user-controlled input passed to system().
     *
     * The format string appends host directly into a shell command.
     * Example injection payload:
     *   notarealhost; echo pwned >/tmp/icmp-audit-lab/owned #
     *
     * The trailing # comments out the >/dev/null redirect, so the
     * injected echo command actually executes and writes to disk.
     */
    snprintf(cmd, sizeof(cmd), "echo failed_to_resolve_%s >/dev/null", host);

    fprintf(stderr, "[debug] resolution failed, running helper: %s\n", cmd);
    system(cmd);
}

static int send_icmp_echo(const char *host) {
    struct addrinfo hints;
    struct addrinfo *res = NULL;
    struct sockaddr_in dst;
    struct icmphdr icmp;
    unsigned char rxbuf[1500];
    struct timeval tv = { .tv_sec = 1, .tv_usec = 0 };
    int sock;
    ssize_t n;
    int rc;

    memset(&hints, 0, sizeof(hints));
    hints.ai_family   = AF_INET;
    hints.ai_socktype = SOCK_RAW;
    hints.ai_protocol = IPPROTO_ICMP;

    rc = getaddrinfo(host, NULL, &hints, &res);
    if (rc != 0) {
        vulnerable_resolution_log(host);
        return 2;
    }

    memcpy(&dst, res->ai_addr, sizeof(dst));
    freeaddrinfo(res);

    sock = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP);
    if (sock < 0) {
        perror("socket");
        return 1;
    }

    if (setsockopt(sock, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)) != 0) {
        perror("setsockopt");
        close(sock);
        return 1;
    }

    memset(&icmp, 0, sizeof(icmp));
    icmp.type           = ICMP_ECHO;
    icmp.code           = 0;
    icmp.un.echo.id     = (unsigned short)(getpid() & 0xFFFF);
    icmp.un.echo.sequence = htons(1);
    icmp.checksum       = checksum(&icmp, sizeof(icmp));

    if (sendto(sock, &icmp, sizeof(icmp), 0,
               (struct sockaddr *)&dst, sizeof(dst)) < 0) {
        perror("sendto");
        close(sock);
        return 1;
    }

    n = recvfrom(sock, rxbuf, sizeof(rxbuf), 0, NULL, NULL);
    if (n < 0) {
        perror("recvfrom");
        close(sock);
        return 1;
    }

    printf("sent ICMP echo to %s\n", host);
    close(sock);
    return 0;
}

int main(int argc, char **argv) {
    if (argc != 2) {
        fprintf(stderr, "usage: %s <host>\n", argv[0]);
        return 1;
    }

    return send_icmp_echo(argv[1]);
}
EOF

Compile and assign the raw socket capability:

gcc -Wall -Wextra -O2 -o icmp_audit_demo icmp_audit_demo.c

# Grant cap_net_raw so the binary can open raw sockets without running as root.
# +ep means: set in Effective and Permitted capability sets.
sudo setcap cap_net_raw+ep ./icmp_audit_demo

# Verify the capability was set
getcap ./icmp_audit_demo
# Expected: ./icmp_audit_demo = cap_net_raw+ep

Test that it runs cleanly on localhost:

./icmp_audit_demo 127.0.0.1
# Expected: sent ICMP echo to 127.0.0.1

Step 2: Understanding Audit Rule Syntax

Before applying rules, it helps to understand what each flag means. The general form of a syscall audit rule is:

auditctl -a <action>,<list> -F arch=<arch> -S <syscall> [-F <field>=<value> ...] -k <key>
Component Meaning
-a always,exit Always log this event; attach the rule to the exit filter (fires when a syscall returns)
-F arch=b64 Match 64-bit syscalls only (use b32 for 32-bit)
-S socket Match the socket syscall specifically; use -S all to match every syscall
-F exe=/path Only match if the process executable equals this path
-F a0=2 Match syscall argument 0 (first arg) equal to 2 — for socket(), that is AF_INET
-F a1=3 Match argument 1 equal to 3 — for socket(), that is SOCK_RAW
-F a2=1 Match argument 2 equal to 1 — for socket(), that is IPPROTO_ICMP
-F dir=/path/ -F perm=wa File/directory watch: log writes and attribute changes
-k icmp_demo_trace Tag matching events with this key — used to search with ausearch -k

The four rules for this lab:

Rule Purpose
-S all -F exe=$APP_PATH -k icmp_demo_trace Trace every syscall the binary makes
-S socket -F a0=2 -F a1=3 -F a2=1 -F exe=$APP_PATH -k icmp_demo_icmp_socket Specifically flag raw ICMP socket creation
-F dir=/tmp/icmp-audit-lab/ -F perm=wa -k icmp_demo_labdir Watch the lab directory for writes
-S execve -F exe=$SH_PATH -k icmp_demo_shell Alert on any shell spawned (the injection indicator)

Step 3: Apply the Audit Rules

Warning: auditctl -D clears the entire in-kernel audit ruleset. Only use this on a dedicated lab machine. On a host that already has a production audit policy, skip this step and apply only the four rules below without clearing existing ones.

# Resolve absolute paths — audit rules require exact executable paths
APP_PATH="$(readlink -f ./icmp_audit_demo)"
SH_PATH="$(readlink -f /bin/sh)"

echo "APP_PATH = $APP_PATH"
echo "SH_PATH  = $SH_PATH"

# Lab only: clear existing rules so we start clean
sudo auditctl -D

# Rule 1: Trace all syscalls made by the demo binary
sudo auditctl -a always,exit \
    -F arch=b64 \
    -S all \
    -F exe="$APP_PATH" \
    -k icmp_demo_trace

# Rule 2: Specifically watch for raw ICMP socket creation (AF_INET=2, SOCK_RAW=3, IPPROTO_ICMP=1)
sudo auditctl -a always,exit \
    -F arch=b64 \
    -S socket \
    -F a0=2 -F a1=3 -F a2=1 \
    -F exe="$APP_PATH" \
    -k icmp_demo_icmp_socket

# Rule 3: Watch the lab directory for writes and attribute changes
sudo auditctl -a always,exit \
    -F dir=/tmp/icmp-audit-lab/ \
    -F perm=wa \
    -k icmp_demo_labdir

# Rule 4: Alert on any shell execution (injection indicator)
sudo auditctl -a always,exit \
    -F arch=b64 \
    -S execve \
    -F exe="$SH_PATH" \
    -k icmp_demo_shell

Verify the active ruleset:

sudo auditctl -l
# You should see all four rules listed

Step 4: Baseline — Normal ICMP Run

Run the binary against localhost and collect the baseline audit trail:

./icmp_audit_demo 127.0.0.1
# Expected: sent ICMP echo to 127.0.0.1

Query each audit key to see what was recorded:

# All syscalls made by the binary
sudo ausearch -k icmp_demo_trace --start recent -i

# The specific raw socket syscall
sudo ausearch -k icmp_demo_icmp_socket --start recent -i

# Any shell executions (should be empty for a clean run)
sudo ausearch -k icmp_demo_shell --start recent -i

# Any writes to the lab directory (should be empty)
sudo ausearch -k icmp_demo_labdir --start recent -i

Anatomy of an Audit Record

The -i flag makes ausearch translate numeric values (UIDs, syscall numbers) into human-readable names. A typical output for the socket syscall looks like this:

----
time->Mon Jan 01 12:00:00 2025
type=SYSCALL msg=audit(1735689600.123:4567): arch=x86_64 syscall=socket success=yes exit=3
  a0=2 a1=3 a2=1 a3=0 items=0 ppid=1234 pid=5678 auid=kurtis uid=kurtis gid=kurtis
  euid=kurtis suid=kurtis fsuid=kurtis egid=kurtis sgid=kurtis fsgid=kurtis tty=pts0
  ses=1 comm="icmp_audit_demo" exe="/home/kurtis/auditd-icmp-lab/icmp_audit_demo"
  subj=unconfined key="icmp_demo_icmp_socket"
type=PROCTITLE msg=audit(1735689600.123:4567): proctitle=icmp_audit_demo 127.0.0.1

Record types that appear together in one event:

Record type What it contains
SYSCALL The syscall name, return value, PID/PPID, UID/AUID, executable path, arguments
EXECVE Full command line arguments when an execve syscall fires
PATH File path involved in the syscall (for open, write, rename, etc.)
CWD Current working directory at the time of the syscall
PROCTITLE The full command line of the process (hex-encoded if it contains spaces)

Key SYSCALL fields:

Field Meaning
auid Audit UID — the user who originally logged in; survives su/sudo switches
uid / euid Real and effective UID at syscall time
success Whether the syscall succeeded (yes/no)
exe Full path of the executable making the syscall
comm Short process name (first 16 chars of the executable name)
a0-a3 First four syscall arguments (shown as hex or decoded when -i is used)
ppid Parent process ID — useful for tracing process lineage

What normal looks like: For a clean run against 127.0.0.1, ausearch -k icmp_demo_shell returns no results, and ausearch -k icmp_demo_labdir returns no results. The only hits are on icmp_demo_trace and icmp_demo_icmp_socket, showing the expected socket(AF_INET, SOCK_RAW, IPPROTO_ICMP) call.

Step 5: Trigger the Attack

The injection payload exploits the format string in vulnerable_resolution_log:

snprintf(cmd, sizeof(cmd), "echo failed_to_resolve_%s >/dev/null", host);
system(cmd);

With the payload notarealhost; echo pwned >/tmp/icmp-audit-lab/owned #, the rendered command becomes:

echo failed_to_resolve_notarealhost; echo pwned >/tmp/icmp-audit-lab/owned # >/dev/null

The shell interprets this as two commands separated by ;: 1. echo failed_to_resolve_notarealhost — harmless 2. echo pwned >/tmp/icmp-audit-lab/owned — writes to the watched directory

The trailing # comments out the >/dev/null redirect that the format string appends, ensuring the second echo actually executes.

Run the attack:

# Clean the target file first so we can confirm it was created by the attack
rm -f /tmp/icmp-audit-lab/owned

./icmp_audit_demo 'notarealhost; echo pwned >/tmp/icmp-audit-lab/owned #'

You will see stderr output showing the injected command being built:

[debug] resolution failed, running helper: echo failed_to_resolve_notarealhost; echo pwned >/tmp/icmp-audit-lab/owned #  >/dev/null

Confirm the attack succeeded:

cat /tmp/icmp-audit-lab/owned
# Expected: pwned

Step 6: Read the Audit Trail

Detecting the Shell Spawn

sudo ausearch -k icmp_demo_shell --start recent -i

You will see two records: a SYSCALL showing execve of /bin/sh, and an EXECVE record with the full argument list:

----
type=SYSCALL msg=audit(...): arch=x86_64 syscall=execve success=yes exit=0
  ppid=5678 pid=5679 auid=kurtis uid=kurtis
  comm="sh" exe="/bin/sh"
  key="icmp_demo_shell"
type=EXECVE msg=audit(...): argc=3
  a0="sh" a1="-c"
  a2="echo failed_to_resolve_notarealhost; echo pwned >/tmp/icmp-audit-lab/owned #  >/dev/null"
type=PROCTITLE msg=audit(...): proctitle=sh -c "echo failed_to_resolve_notarealhost; ..."

The smoking gun: ppid of the shell process matches the pid of icmp_audit_demo. A raw ICMP binary spawning a shell via system() is not expected behavior. The EXECVE record contains the full injected command — the payload is fully visible in the audit log.

Detecting the File Write

sudo ausearch -k icmp_demo_labdir --start recent -i

This fires on the write to /tmp/icmp-audit-lab/owned:

----
type=SYSCALL msg=audit(...): arch=x86_64 syscall=openat success=yes exit=4
  ppid=5678 pid=5679 auid=kurtis uid=kurtis
  comm="sh" exe="/bin/sh"
  key="icmp_demo_labdir"
type=PATH msg=audit(...): item=0 name="/tmp/icmp-audit-lab/owned"
  inode=123456 dev=08:01 mode=0100644 ouid=kurtis ogid=kurtis
  nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0
type=CWD msg=audit(...): cwd="/home/kurtis/auditd-icmp-lab"

You can also query directly by file path:

sudo ausearch -f /tmp/icmp-audit-lab/owned --start recent -i

Correlating the Events

The auid field (audit UID) and pid/ppid chain tie the events together. The shell's ppid matches the icmp_audit_demo process's pid. Use ausearch with a time window to pull both events at once:

# Pull all events tagged with either the shell key or the directory key in the last hour
sudo ausearch -k icmp_demo_shell -k icmp_demo_labdir --start recent -i

Note: When you pass multiple -k flags to ausearch, it performs an OR search — it returns events matching any of the keys. This gives you the full attack sequence in chronological order.

The detection signal is simple: A tool whose entire legitimate syscall profile is socket → sendto → recvfrom → close has no business calling execve or writing to arbitrary paths. When both appear in the audit trail, the compromise is unambiguous.

Persistent Rules with augenrules

auditctl rules are lost on reboot. For production, rules live in /etc/audit/rules.d/ as .rules files. augenrules merges all .rules files from that directory and applies them — it is the recommended path over editing /etc/audit/audit.rules directly.

Write a rules file for this lab:

sudo tee /etc/audit/rules.d/icmp-demo.rules <<EOF
# auditd rules for the ICMP audit demo lab
# Generated for: $APP_PATH

# Trace all syscalls made by the demo binary
-a always,exit -F arch=b64 -S all -F exe=$APP_PATH -k icmp_demo_trace

# Watch specifically for raw ICMP socket creation
-a always,exit -F arch=b64 -S socket -F a0=2 -F a1=3 -F a2=1 -F exe=$APP_PATH -k icmp_demo_icmp_socket

# Watch the lab directory for writes and attribute changes
-a always,exit -F dir=/tmp/icmp-audit-lab/ -F perm=wa -k icmp_demo_labdir

# Alert on any shell spawned on this host
-a always,exit -F arch=b64 -S execve -F exe=$SH_PATH -k icmp_demo_shell
EOF

Apply the persistent rules:

sudo augenrules --load

# Confirm they are active
sudo auditctl -l

When to use auditctl vs augenrules:

Use Tool
Temporary testing and iteration auditctl — instant, no reboot required
Production deployment augenrules — survives reboots, managed as files
Viewing the current active ruleset auditctl -l — always shows what is currently loaded

Summary Reports with aureport

While ausearch lets you dig into individual events, aureport generates summary statistics — useful for high-level triage across a host's full audit history.

# Summary of all execve events (processes that were run)
sudo aureport --exec

# Summary of file access events
sudo aureport --file

# Summary of syscall events by frequency
sudo aureport --syscall

# Summary of failed events (syscalls that returned an error)
sudo aureport --failed

# Overview report (logins, failed auths, processes, files)
sudo aureport

aureport is the starting point for incident response: it tells you the what (which categories of events are elevated), then ausearch tells you the who, when, and exactly how.

Remediating the Vulnerability

The root cause is passing user-controlled data to system(). The safe fix is to never use system() when the input comes from an untrusted source. Use the execv family instead, which passes arguments as an array without invoking a shell:

/* Safe alternative — no shell interpretation, argv is an array */
char *args[] = { "logger", "--", failed_message, NULL };
execv("/usr/bin/logger", args);

Detection takeaway: auditd caught this attack because the binary's legitimate behavior is well-defined — raw socket, send, receive. The moment it spawned a shell and wrote to a watched directory, the kernel recorded both events with full process lineage, the injected command string, and the file path created. This is the core value proposition of auditd: behavioral deviation is visible at the syscall layer, regardless of what the application logs say.

Lab Cleanup

# Clear in-kernel rules
sudo auditctl -D

# Remove the persistent rules file
sudo rm -f /etc/audit/rules.d/icmp-demo.rules
sudo augenrules --load

# Remove lab files
rm -rf /tmp/icmp-audit-lab/
rm -rf ~/auditd-icmp-lab/

Next Steps