To enable SYSLogs TLS end-to-end encryption

In my professional journey, I’ve gained experience with logging forwarding through ELK and EFKK stacks, both of which were installed on virtual machines and Kubernetes. In our current organization, we’re transitioning to using syslogs to forward logs to external observability tools such as NewRelic, Datadog, Splunk, and others. Admittedly, working with syslogs is a new venture for me, and I still have limited knowledge in areas like configuration and troubleshooting.

The procedure involves setting up a virtual machine (VM) and installing syslogs as a container within it. Subsequently, logs will be forwarded to external observability tools.

syslogs tcp architecture

The depicted diagram illustrates the process of initially forwarding logs to a syslogs server, which then subsequently transmits the logs to an external party. From a security standpoint, I am mandated to refrain from utilizing a TCP connection between the application and the syslogs server, as well as between the syslogs server and the external party. Consequently, I am required to implement TLS end-to-end encryption between both entities, as shown in the following diagram.

syslogs tls architecture

In order to employ syslogs as a container with TLS encryption, the initial step involves generating a private key. The choice of tools for this task, such as openssl or certtool, can be determined based on project requirements. However, to streamline the process and avoid adding complexity to the pipeline, I have opted to include the private key generation command directly in the Dockerfile, as illustrated below:

FROM ubuntu:23.04
RUN apt-get update -y
RUN apt-get install -y rsyslog-gnutls ca-certificates

WORKDIR /src

RUN openssl req -x509 -nodes -newkey rsa:2048 -keyout /src/key.pem -out /src/cert.pem -days 7300 -subj '/CN=localhost' -addext "keyUsage = digitalSignature, keyEncipherment, dataEncipherment, cRLSign, keyCertSign" -addext "extendedKeyUsage = serverAuth, clientAuth"

ADD ./src/myconfig.conf /etc/rsyslog.d/myconfig.conf
RUN cp /src/cert.pem /etc/rsyslog.d/cert.pem
RUN cp /src/key.pem /etc/rsyslog.d/key.pem

EXPOSE 6514

ENTRYPOINT [ "rsyslogd", "-n", "-f", "/etc/rsyslog.d/myconfig.conf" ]

The first three lines entail the installation of syslogs and the generation of CA certificates within the system.

RUN openssl req -x509 -nodes -newkey rsa:2048 -keyout /src/key.pem -out /src/cert.pem -days 7300 -subj '/CN=localhost' -addext "keyUsage = digitalSignature, keyEncipherment, dataEncipherment, cRLSign, keyCertSign" -addext "extendedKeyUsage = serverAuth, clientAuth"

The provided command is a straightforward way to generate a private key, with the inclusion of keyUsage options, which are essential for certain endpoints. The subsequent lines involve copying syslogs configuration files and private keys into the container.

As you’re aware, some individuals opt for separate configuration files for server and client setups to enhance clarity and visibility. However, in my case, I prefer consolidating everything within a single configuration file, addressing both logs input and output configurations in one place for simplicity.

global(
     DefaultNetstreamDriver="gtls"
     DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca-certificates.crt"
     DefaultNetstreamDriverCertFile="/etc/rsyslog.d/cert.pem"
     DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/key.pem"
)

The provided code is for configuring the certificate file. While a self-signing certificate is an option for the CA certificate, there are instances where a TLS handshake between the client and server is necessary. Hence, my preference is to utilize ca-certificates, a tool that generates the CA certificate for us.

# load TCP listener
module(
     load="imtcp"
     StreamDriver.Name="gtls"
     StreamDriver.Mode="1"
     StreamDriver.Authmode="anon"
)

# start up listener at port 6514
input(
     type="imtcp"
     port="6514"
     ruleset="sendToExternal"
)

The provided configuration pertains to TLS input. To accommodate numerous applications streaming logs into our syslogs container within the VM, we opt for StreamDriver.Authmode=”anon” to eliminate the need for verifying each application’s certificate. Applications will utilize port 6514 to stream logs to our syslogs endpoint. After receiving logs from applications, it is imperative to forward these logs to an external party for monitoring and alerting purposes.

action(
    type="omfwd"
    target="central.mylogging.net"
    protocol="tcp"
    port="6514"
    StreamDriver="gtls"
    StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="central.mylogging.net"
)

If our organization has its own logstash server, the recommended configuration involves using StreamDriverAuthMode=”x509/name” since we employ the same CA certificate and private key for both the client and server. However, when utilizing external tools such as NewRelic, Datadog, and Splunk, it becomes necessary to modify the configuration by setting StreamDriverAuthMode=”anon” to overlook TLS verification issues.

  action(type="omfwd"
     Target="newrelic.syslog.eu.nr-data.net"
     Port="6514"
     Protocol="tcp"
     Template="newrelic-rfc5424"
     ResendLastMSGOnReconnect="on"
     StreamDriver="gtls"
     StreamDriverAuthMode="anon"
     StreamDriverMode="1"
  )

I trust that this information will assist you in configuring TLS end-to-end encryption for syslogs, both in on-premises environments and public clouds like AWS, Azure, and others.