Auditing System Events in Linux
Original publication date: September 22, 2015.
The audit subsystem is used to raise the level of security in Linux systems. Although it doesn’t offer additional security per se, it’s used to retrieve detailed information on system events. This provides detailed information on system violations, which can be used to implement additional targeted security measures. We’ll be taking a deeper look at the audit subsystem in this article.
Kernel Auditing: Architecture and Principles
The audit subsystem was first introduced to the Linux kernel in version 2.6. Its initial purpose was to track critical system events related to security. These kinds of events include the following (although this list is far from full):
- the launch and completion of system jobs
- reading, writing, and changing a file’s access rights
- initiating network connections
- failed authorization attempts
- changing network configurations
- changing user and group information
- launching and halting applications
- executing system calls
None of these events can occur without a system call from the kernel. Tracking requires the system calls be intercepted, and that is just what the kernel’s audit subsystem does:
After receiving a call from an application in the user space, the audit subsystem passes it through one of the following filters: user, task, or exit (we’ll discuss this further on). Afterwards, the call passes through the exclude filter, which according to the audit rules, transfers it to the auditd daemon for further processing.
This simple layout lets you effectively track any aspect of the operating system and in the event the system is compromised, identify suspicious events and determine their cause.
Installation
To use the audit subsystem, the auditd package must first be installed (the examples in this article use Ubuntu, but instructions for other distributions are similar):
$ sudo apt-get install auditd
The following auxiliary tools will be installed along with the audit daemon:
- auditctl — utility for managing the auditd daemon; returns information on the audit subsystem’s current status and can be used to add and delete rules
- autrace — utility for auditing events caused by processes (the same as strace)
- ausearch — utility for searching for events in log files
- aureport — utility for generating reports on the audit system
Configuration
Before using the audit subystem, we open the configuration file etc/audit/auditd.conf. This includes the following parameters (among others):
- log_file — the file audit subsystem logs will be saved to
- log_format — the format logs will be saved in
- freq — the maximum number of records that can be saved in the buffer
- flush — the mode for synchronizing the buffer with the disk (none – do nothing; incremental – transfers data from the buffer to the hard disk at the frequency set in the freq parameter; data – constant synchronization of data; sync – synchronize both data and metadata files when writing to the disk)
- max_log_file — the maximum log file size in megabytes
- max_log_file_action — the action to be taken when the maximum log file size is reached
- space_left — the minimum amount of free disk space in megabytes; this amount will trigger the following parameter
- space_left_admin — the action to be taken when the minimum allotted disk space is left (ignore – do nothing; syslog – issues a warning to syslog; email – sends notification via email; suspend – stops writing records to the disk; single – switch to single-user mode; halt – shuts down the system)
- disk_full_action — the action to be taken when the disk is full (this parameter can take the same values as space_left_admin)
Creating Rules
We can add and configure rules with the auditctl command. Possible options are:
- -l — print out a list of existing rules
- -а — add a new rule
- -d — delete an existing rule
- -D — delete all existing rules
To create a new rule, we have to run a command with the following syntax:
$ auditctl -a , -S -F
After the -a option, we enter the list we wish to add rules to. There are five kinds of lists:
- task — events related to creating new processes
- entry — events that take place on system call entrance
- exit — events that take place on system call exit
- user — events that use user space parameters
- exclude — used for excluding events
Then we enter the action to be taken when the event occurs. Here we have two options: always (enter the event in the log) or never (don’t enter the event).
After the -S option, we enter the name of the system call the event needs to be intercepted for (open, close, etc.).
We can set additional filters after the -F option. For example, if we need to audit references to files from the /etc catalog, the rule would like like the following:
$ auditctl -a exit,always -S open -F path =/etc/
We can add an extra filter:
$ auditctl -a exit,always -S open -F path =/etc/ -F perm = aw
The aw abbreviation can be broken down as: а – attribute change, w -write. The formula perm = aw means that all attribute changes and writes in the /etc directory need to be tracked.
When configuring tracking for individual files, we can retract the -S option, for example:
$ auditctl -a exit,always -F path =/etc/ -F perm = aw
Rule Files
Rules may be assigned outside of the command line by making entries to the etc/audit/audit.rules file.
The file starts with what are called meta-rules, which assign general logging parameters:
# delete all previously created rules -D # set the number of buffers where messages will be saved - b 320 # action to take in critical situations (for example, when a buffer gets full): 0 - do nothing; 1 - send a message to dmesg, 2 - set the kernel to panic -f 1
Then we have user rules. Their syntax is extremely simple and requires we just list the appropriate auditctl command options. Let’s look at an example of a typical configuration file:
#track the unlink () and rmdir() system calls -a exit,always -S unlink -S rmdir #track the open () system calls for the user with UID 1001 -a exit,always -S open -F loginuid=1001 #track access to password and group files and attempts to changes them: -w /etc/group -p wa -w /etc/passwd -p wa -w /etc/shadow -p wa -w /etc/sudoers -p wa #track access to the following directories: -w /etc/secret_directory -p r # close access to configuration files to prevent changes -e 2
After changes have been made to the configuration file, the auditd daemon needs to be restarted:
$ sudo service auditd restart
Analyzing Logs: aureport
All logs are saved in the /var/log/audit directory in machine readable format. They can be converted to a human readable format using special utilities. Among these, we should firstly name aureport, which generates reports from log files.
If we run aureport without arguments, general system stats will be printed to the console (the number of system users, general number of system calls, number of open terminals, etc.):
$ sudo aureport Summary Report ====================== Range of time in logs: 07/31/2015 14:04:23.870 - 08/04/2015 09:37:13.200 Selected time for report: 07/31/2015 14:04:23 - 08/04/2015 09:37:13.200 Number of changes in configuration: 0 Number of changes to accounts, groups, or roles: 3 Number of logins: 0 Number of failed logins: 0 Number of authentications: 0 Number of failed authentications: 61205 Number of users: 2 Number of terminals: 5 Number of host names: 73 Number of executables: 6 Number of files: 0 Number of AVC's: 0 Number of MAC events: 0 Number of failed syscalls: 0 Number of anomaly events: 0 Number of responses to anomaly events: 0 Number of crypto events: 0 Number of keys: 0 Number of process IDs: 17858 Number of events: 61870
These statistics don’t have any particular practical value. Specialized reports may be much more interesting. We can look at information on all system calls as an example:
$ sudo aureport -s Syscall Report ======================================= # date time syscall pid comm auid event ======================================= 1. 08/03/2015 15:45:03 313 10285 modprobe -1 52501 2. 08/03/2015 15:45:03 313 10290 modprobe -1 52502 3. 08/03/2015 15:45:03 54 10296 iptables -1 52503 4. 08/03/2015 15:45:03 54 10302 iptables -1 52504 5. 08/03/2015 15:45:03 54 10305 iptables -1 52505 6. 08/03/2015 15:45:03 54 10313 iptables -1 52506 7. 08/03/2015 15:45:03 54 10325 iptables -1 52507 8. 08/03/2015 15:45:03 54 10329 iptables -1 52508 9. 08/03/2015 15:45:03 54 10343 iptables -1 52509 10. 08/03/2015 15:45:03 54 10345 iptables -1 52510 11. 08/03/2015 15:45:03 54 10349 iptables -1 52511
Using the -au (or –auth) option, we can view information on all attempts to enter the system:
$ sudo aureport -au Authentication Report ============================================ # date time acct host term exe success event ============================================ 1. 08/31/2015 18:00:19 ubnt static-166-6-249-80.stalowa.pilicka.pl ssh /usr/sbin/sshd no 333 2. 08/31/2015 18:01:38 root 59.63.188.31 ssh /usr/sbin/sshd no 334 3. 08/31/2015 18:01:41 root 59.63.188.31 ssh /usr/sbin/sshd no 335 4. 08/31/2015 18:01:45 root 59.63.188.31 ssh /usr/sbin/sshd no 336 5. 08/31/2015 18:01:53 root 59.63.188.31 ssh /usr/sbin/sshd no 337 6. 08/31/2015 18:01:57 root 59.63.188.31 ssh /usr/sbin/sshd no 338 7. 08/31/2015 18:01:59 root 59.63.188.31 ssh /usr/sbin/sshd no 339
Using the –success and –failed flags, we can print out information on successful and failed authorization attempts.
Aureport allows filtering by date and time:
$ sudo aureport -s --start 07/31/15 12:00 --end 07/31/15 13:00
You can enter both a specific date and time as well as some intuitive constructions:
- now — the current moment
- yesterday — yesterday
- recent — 10 minutes ago
- this-week (or this-month, this-year) — the current week (month, year)
With aureport, it’s also possible to view information on any system user’s actions. To do this, the user id first has to be identified:
$ id user uid=1000(user) gid=1000(andrei) groups=1000(andrei),27(sudo)
and then we run the following command:
$ sudo ausearch -ui 1000 --interpret
Scanning and Analyzing Events: ausearch
To view detailed event information, we use the ausearch command:
$ ausearch -a <event number>
The output of this command will look as follows:
type=SYSCALL msg=audit(1364481363.243:24287): arch=c000003e syscall=2 success=no exit=-13 a0=7fffd19c5592 a1=0 a2=7fffd19c4b50 a3=a items=1 ppid=2686 pid=3538 auid=500 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts0 ses=1 comm="cat" exe="/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="sshd_config"
Let’s take a closer look at its structure. In the type field, we have the record type; type = syscall means the records were made after a system call. The msg field contains the event time in Unix timestamp format and its unique identification number.
The arch field contains information on the system architecture (c000003e means x86_84), which is presented in encoded hexadecimal format. To print this out in a more comprehensible format, we use the -i or –interpret option.
The system call type is indicated in the syscall field. In our case, we have a 2, indicating an open call. The success parameter indicates if a call was successfully processed or not. In our example, the call was not processed (succes = no).
Individual parameters are also listed in the report for each choice; more details can be read in the official manual. Information on any parameter can be printed out in the console in an intuitive format using the abovementioned -i or –interpret option, for example:
$ ausearch --interpet --exit -13
The -sc option lets us include events related to a specific system call event, for example:
$ ausearch -sc ptrace
The -ui option scans events by user ID:
$ ausearch -ui 33
To search for events by filename, we use the -x option:
$ ausearch -x /usr/bin/docker
And by daemon with the -tm option:
$ ausearch -x -tm cron
Keys can also be used to scan for particular events, for example:
$ auditctl -k root-actions
This command prints a list of all the events performed by the root-user.
You can also filter by date and time.
A list of failed events can be printed using the –failed option.
Analyzing Processes Using autrace
In some cases, it may be useful to gather information on events related to one particular process. We can do this with the autrace utility. Let’s say we need to track the date process and find out which system calls and files it uses. We’ll run the command:
$ sudo autrace /bin/date
The following text will be printed in the console:
Waiting to execute: /bin/date Mon Aug 31 17:06:32 MSK 2015 Cleaning up... Trace complete. You can locate the records with 'ausearch -i -p 29234'
Pay attention to the last line of the output: it shows a command that can be used to get even more detailed information. We run this command and send the printout to aureport, which converts it to a human readable format:
$ ausearch -p 29215 --raw | aureport -f -i
As a result, we receive the following report:
File Report =============================================== # date time file syscall success exe auid event =============================================== 1. 08/31/2015 16:52:16 /bin/date execve yes /bin/date root 25 2. 08/31/2015 16:52:16 /etc/ld.so.nohwcap access no /bin/date root 27 3. 08/31/2015 16:52:16 /etc/ld.so.preload access no /bin/date root 29 4. 08/31/2015 16:52:16 /etc/ld.so.cache open yes /bin/date root 30 5. 08/31/2015 16:52:16 /etc/ld.so.nohwcap access no /bin/date root 34 6. 08/31/2015 16:52:16 /lib/x86_64-linux-gnu/libc.so.6 open yes /bin/date root 35 7. 08/31/2015 16:52:16 /usr/lib/locale/locale-archive open yes /bin/date root 52 8. 08/31/2015 16:52:16 /etc/localtime open yes /bin/date root 56
Centralized Log Storage
To send audit subsystem logs to centralized storage, we use the audisp-remote plugin. It is included in the audisp-plugins package, which is installed separately:
$ sudo apt-get install audisp-plugins
The configuration files of all the plugins are saved in the directory /etc/audisp/plugins.d.
Remote logging settings are written in the configuration file /etc/audisp/plugins.d/audisp-remote.conf. By default, this file looks as follows:
active = no direction = out path = /sbin/audisp-remote type = always #args = format = string
To enable sending logs to remote storage, we replace the value of the active parameter to yes. We then open the file etc/audisp/audisp-remote.conf and for the remote_server parameter value, we enter the alphabetical or IP address of the server where logs will be saved.
To accept logs from remote hosts and save them on the server, we need to set the following parameters to the file /etc/audit/auditd.conf:
tcp_listen_port = 60 tcp_listen_queue = 5 tcp_max_per_addr = 1 ##tcp_client_ports = 1024-65535 #optional tcp_client_max_idle = 0
Conclusions
In this article, we presented the fundamental uses of the Linux audit system. We’ve looked at the the principles behind the audit system, learned how to write rules, read logs, and used auxiliary utilities.
For those interested in learning more about the audit system, you can visit the following links:
- a complete list of audit record types
- a complete list of possible events
- an article about using audit capability to investigate kernel attacks
If you have any questions, please feel free to ask in the comments below.