auditbeat Linux: Complete Guide to Shipping Audit Data to Elasticsearch

Introduction to auditbeat

auditbeat is a lightweight shipper from the Elastic Beat family that collects audit data from the Linux Audit Framework and ships it to Elasticsearch or other outputs. It provides comprehensive security monitoring by tracking file integrity, process execution, network connections, and system calls.

This powerful tool helps organizations maintain security compliance, detect unauthorized changes, and monitor system activities in real-time. auditbeat transforms raw audit logs into structured data that can be easily analyzed and visualized in Kibana.

Key Features of auditbeat

• File Integrity Monitoring (FIM): Tracks changes to critical files and directories
• Process Monitoring: Captures process execution and termination events
• Network Monitoring: Records network socket activities
• System Call Auditing: Monitors system calls for security analysis
• User Session Tracking: Logs user login/logout activities
• Compliance Support: Helps meet PCI DSS, HIPAA, and SOX requirements

Prerequisites

Before installing auditbeat, ensure your system meets these requirements:

• Linux distribution with kernel 2.6.30 or later
• Elasticsearch cluster (version 6.8 or later recommended)
• Root or sudo privileges for installation
• Network connectivity to Elasticsearch cluster
• Sufficient disk space for temporary log storage

Installing auditbeat on Linux

Method 1: Using Package Managers

Ubuntu/Debian Installation

# Add Elastic repository
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Update package list and install
sudo apt update
sudo apt install auditbeat

CentOS/RHEL Installation

# Add Elastic repository
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# Create repository file
cat << EOF | sudo tee /etc/yum.repos.d/elastic.repo
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

# Install auditbeat
sudo yum install auditbeat

Method 2: Direct Download and Installation

# Download auditbeat
curl -L -O https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-8.10.0-linux-x86_64.tar.gz

# Extract the archive
tar xzvf auditbeat-8.10.0-linux-x86_64.tar.gz

# Move to appropriate directory
sudo mv auditbeat-8.10.0-linux-x86_64 /opt/auditbeat

# Create symbolic link
sudo ln -s /opt/auditbeat/auditbeat /usr/local/bin/auditbeat

Basic Configuration

The main configuration file is located at /etc/auditbeat/auditbeat.yml. Here’s a basic configuration example:

# Basic auditbeat configuration
auditbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

auditbeat.modules:
- module: auditd
  audit_rules: |
    -w /etc/passwd -p wa -k identity
    -w /etc/group -p wa -k identity
    -w /etc/shadow -p wa -k identity
    -a always,exit -F arch=b64 -S execve -k exec

- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc

- module: system
  datasets:
    - package
    - host
  period: 10m

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "your_password"

setup.kibana:
  host: "localhost:5601"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/auditbeat
  name: auditbeat
  keepfiles: 7

Module Configuration

Auditd Module

The auditd module captures events from the Linux Audit Framework:

- module: auditd
  resolve_ids: true
  failure_mode: silent
  backlog_limit: 8196
  rate_limit: 0
  include_raw_message: false
  include_warnings: false
  audit_rules: |
    # Monitor file access
    -w /etc/passwd -p rwxa -k passwd_changes
    -w /etc/shadow -p rwxa -k shadow_changes
    -w /etc/group -p rwxa -k group_changes
    
    # Monitor system calls
    -a always,exit -F arch=b64 -S chmod,fchmod,chown,fchown -k perm_mod
    -a always,exit -F arch=b64 -S unlink,rmdir -k delete
    
    # Monitor network connections
    -a always,exit -F arch=b64 -S socket,connect -k network

File Integrity Module

Monitor file changes with detailed configuration:

- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  exclude_files:
  - '(?i)\.sw[nop]$'
  - '~$'
  - '/\.git($|/)'
  scan_at_start: true
  scan_rate_per_sec: 50 MiB
  max_file_size: 100 MiB
  hash_types: [sha1]
  recursive: true

System Module

Collect system information periodically:

- module: system
  datasets:
    - package    # Installed packages
    - host       # Host information
    - login      # User logins
    - process    # Running processes
    - socket     # Network sockets
    - user       # System users
  period: 10m
  state.period: 12h
  user.detect_password_changes: true

Elasticsearch Output Configuration

Basic Elasticsearch Configuration

output.elasticsearch:
  hosts: ["elasticsearch1:9200", "elasticsearch2:9200", "elasticsearch3:9200"]
  protocol: "https"
  username: "auditbeat_writer"
  password: "${ELASTICSEARCH_PASSWORD}"
  ssl.certificate_authorities: ["/etc/auditbeat/ca.crt"]
  ssl.certificate: "/etc/auditbeat/auditbeat.crt"
  ssl.key: "/etc/auditbeat/auditbeat.key"

Advanced Elasticsearch Settings

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "auditbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  template.name: "auditbeat"
  template.pattern: "auditbeat-*"
  template.settings:
    index.number_of_shards: 1
    index.number_of_replicas: 1
    index.refresh_interval: "5s"
  
  # Bulk settings
  bulk_max_size: 1600
  worker: 1
  compression_level: 0
  escape_html: false

Starting and Managing auditbeat

Service Management

# Enable and start auditbeat service
sudo systemctl enable auditbeat
sudo systemctl start auditbeat

# Check service status
sudo systemctl status auditbeat

# View logs
sudo journalctl -u auditbeat -f

# Stop and restart service
sudo systemctl stop auditbeat
sudo systemctl restart auditbeat

Testing Configuration

# Test configuration
sudo auditbeat test config

# Test output connectivity
sudo auditbeat test output

# Run in foreground for debugging
sudo auditbeat -e -c /etc/auditbeat/auditbeat.yml

Common Use Cases and Examples

Monitoring Critical System Files

Create audit rules to monitor critical system files:

# Add to auditd module configuration
audit_rules: |
  # Monitor critical system files
  -w /etc/passwd -p wa -k passwd_file
  -w /etc/shadow -p wa -k shadow_file
  -w /etc/sudoers -p wa -k sudoers_file
  -w /etc/ssh/sshd_config -p wa -k ssh_config
  
  # Monitor system binaries
  -w /bin/su -p x -k privileged_commands
  -w /usr/bin/sudo -p x -k privileged_commands
  -w /bin/login -p x -k login_commands

Process Execution Monitoring

# Monitor process execution
audit_rules: |
  # Track all command executions
  -a always,exit -F arch=b64 -S execve -k command_exec
  -a always,exit -F arch=b32 -S execve -k command_exec
  
  # Monitor specific dangerous commands
  -w /usr/bin/wget -p x -k network_tools
  -w /usr/bin/curl -p x -k network_tools
  -w /bin/nc -p x -k network_tools

Network Activity Monitoring

# Network connection monitoring
audit_rules: |
  # Monitor network socket creation
  -a always,exit -F arch=b64 -S socket -F a0=2 -k network_socket_ipv4
  -a always,exit -F arch=b64 -S socket -F a0=10 -k network_socket_ipv6
  
  # Monitor network connections
  -a always,exit -F arch=b64 -S connect -k network_connect
  -a always,exit -F arch=b64 -S bind -k network_bind

Performance Tuning

Optimizing Resource Usage

# Performance settings in auditbeat.yml
queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# Limit file integrity monitoring
- module: file_integrity
  paths:
  - /etc
  scan_rate_per_sec: 10 MiB
  max_file_size: 50 MiB

Managing Audit Log Volume

# Control audit rule scope
audit_rules: |
  # Exclude noisy system calls
  -a never,exit -F arch=b64 -S adjtimex -F auid=unset
  -a never,exit -F arch=b64 -S clock_settime -F auid=unset
  
  # Rate limiting
  -r 500  # Limit to 500 messages per second

Security Considerations

Secure Communication

# SSL/TLS configuration for secure communication
output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  ssl.verification_mode: full
  ssl.certificate_authorities: ["/etc/ssl/certs/ca.pem"]
  ssl.certificate: "/etc/ssl/certs/auditbeat.pem"
  ssl.key: "/etc/ssl/private/auditbeat.key"

Authentication and Authorization

# API key authentication
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  api_key: "id:api_key"

# Or username/password with keystore
# Store password securely
sudo auditbeat keystore create
sudo auditbeat keystore add elasticsearch.password

Troubleshooting Common Issues

Permission Issues

# Check auditbeat user permissions
sudo ls -la /var/log/auditbeat/
sudo chown -R auditbeat:auditbeat /var/log/auditbeat/

# Verify audit system status
sudo auditctl -s

# Check if audit daemon is running
sudo systemctl status auditd

Connection Problems

# Test Elasticsearch connectivity
curl -X GET "elasticsearch:9200/_cluster/health?pretty"

# Check auditbeat logs for errors
sudo tail -f /var/log/auditbeat/auditbeat

# Verify network connectivity
telnet elasticsearch_host 9200

High Resource Usage

# Monitor auditbeat resource usage
sudo top -p $(pgrep auditbeat)

# Check disk usage
sudo du -sh /var/log/auditbeat/
sudo df -h /var/log

# Reduce monitoring scope if needed
# Edit /etc/auditbeat/auditbeat.yml to exclude unnecessary paths

Integration with Kibana

Setting Up Dashboards

# Load Kibana dashboards
sudo auditbeat setup --dashboards

# Setup index patterns and templates
sudo auditbeat setup --index-management --dashboards

Custom Visualizations

Create custom Kibana visualizations for:

• File modification timeline
• Process execution frequency
• Network connection patterns
• User activity heat maps
• Compliance dashboard views

Best Practices

Configuration Management

• Version control your auditbeat configurations
• Use centralized configuration management tools
• Test configurations in staging environments
• Document custom audit rules and their purposes
• Implement configuration validation processes

Security Hardening

• Run auditbeat with minimal required privileges
• Secure configuration files with appropriate permissions
• Use encrypted communication channels
• Implement log rotation and retention policies
• Monitor auditbeat’s own security events

Maintenance and Monitoring

• Regular updates and security patches
• Monitor auditbeat performance metrics
• Set up alerts for service failures
• Periodic review of audit rules effectiveness
• Backup critical configuration files

Conclusion

auditbeat provides a robust solution for shipping Linux audit data to Elasticsearch, enabling comprehensive security monitoring and compliance tracking. By properly configuring modules, optimizing performance, and following security best practices, organizations can maintain effective oversight of their Linux systems.

The combination of file integrity monitoring, process tracking, and network activity surveillance makes auditbeat an essential tool for security-conscious environments. Regular maintenance, proper tuning, and integration with visualization tools like Kibana ensure maximum value from your audit data collection efforts.

Remember to start with basic configurations and gradually expand monitoring scope based on your specific security requirements and compliance needs. This approach helps maintain system performance while providing the security visibility your organization requires.