Event Forwarding – using rsyslog

Nowadays all major Linux distributions are using systemd-journald to handle locally generated system events, but you may still need a syslog agent if you want to forward them to a remote location – like a SIEM.

Rsyslog is still installed in many (Red Hat based) distribution by default, the reason for this however questionable, as it is just traditionally(?) writing the local events to separate files. Those files however still reside on a local disk, and needs to be rotated and manage it’s retention periods in some way.

This method might be ‘good enough’ for a single server, however as soon as you managing multiple (like hundreds or thousands) ones, you just need a central log collector and/or a SIEM.

In this case, you need to setup rsyslog to:

• Fetch events from journald

module(load="imjournal"             # provides access to the systemd journal
       StateFile="imjournal.state") # File to store the position in the journal

• Use the RFC 5424 compatible message format,

# Use the RFC5424 compatible format
module(load="builtin:omfile" Template="RSYSLOG_SyslogProtocol23Format")
# instead of the default
#module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")

Yes, this is affecting the local files only, but it is a good practice to have your local logs in a same format as you forward them.

• Optionally enable the MARK message capability

This may helps on a low volume server by sending ‘keep alive’ messages, to see if the host is still logging properly.

module(load="immark") # provides --MARK-- message capability

• Optionally use FQDN instead of short hostname in syslog headers

This may helps if your hostnames are not unique in your environment.

global(PreserveFQDN="on")

• Filter for your selected events,

It is really up to your goals, but to provide a working example for my previous article:

ruleset( name="filter_QRadar" )
{
    # authpriv.*
    if ( $syslogfacility-text == 'authpriv' )
    then
    {
        call QRadar
    }
    # audit logs
    if ( $syslogfacility-text == 'local6' )
    then
    {
        call QRadar
    }
    # Error messages
    if ( $syslogseverity < 4 )
    then
    {
        call QRadar
    }
    # Syslog related messages - including MARK
    if ( $syslogfacility-text == 'syslog' )
    then
    {
        call QRadar
    }
}

• Send those to the remote Syslog Collector and/or SIEM server via TCP,
• With local disc caching to not loose events in case of a (short) network outage,
• Use an encrypted and authenticated TLS encapsulation:

global(
       DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca-bundle.crt")
       DefaultNetstreamDriverKeyFile="/etc/ssl/rsyslog.key"
       DefaultNetstreamDriverCertFile="/etc/ssl/rsyslog.pem")

ruleset( name="QRadar" )
{
    action( type="omfwd"
        target  = "qradar.security.lab"
        port    = "6514"
        protocol= "tcp"
	KeepAlive= "on"
	Template="RSYSLOG_SyslogProtocol23Format"

        # and queue to disk if needs be
        queue.spoolDirectory="/var/lib/rsyslog"
        queue.filename="tls_fwd"
        queue.type="LinkedList"
        queue.saveonshutdown="on"
        queue.maxdiskspace="1g"

        action.resumeretrycount="-1"
        action.reportsuspension="on"

	StreamDriver="gtls"
	StreamDriverMode="1"
	StreamDriverAuthMode="x509/name"
        StreamDriverPermittedPeers="qradar.security.lab"
        )
}

call filter_QRadar

Keep in mind, that for TLS authentication, you need to have proper certificates on booth sides, but how to generate these are out of scope for this article.

If you don’t have proper certificates and/or just don’t need authentication, you just need to change the AuthMode parameter:

StreamDriverAuthMode="anon"

In this case, you can also omit the StreamDriverPermittedPeers, DefaultNetstreamDriverKeyFile and the DefaultNetstreamDriverKeyFile settings.

 

After restarting rsyslogd, you can check if the connection is successfully established:

[root@centos-8 ~]# netstat -tpn

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 172.16.10.102:50616     172.16.30.75:6514       ESTABLISHED 28009/rsyslogd      

Journald

You may also need to modify some journald related configuration, to tune your system for remote event forwarding. So make sure you have the following in /etc/systemd/journald.conf

Storage=volatile
RateLimitIntervalSec=0
RateLimitBurst=0
RuntimeMaxUse=250

Linux Audit

The Linux Audit system provides a way to track security-relevant information on your system. Based on pre-configured rules, Audit generates log entries to record as much information about the events that are happening on your system as possible. This information is crucial for mission-critical environments to determine the violator of the security policy and the actions they performed.

However this is usually not configured and/or enabled by default. So make sure if it is:

• Installed and enabled on your servers.
• Syslog plugin also installed and configured:

active = yes
direction = out
path = /sbin/audisp-syslog
type = always 
args = LOG_LOCAL6
format = string

---

 As I’m currently working for IBM, please read my disclaimer.