To enable SYSLogs TLS end-to-end encryption

In my professional journey, I’ve gained experience with log forwarding using both the ELK and EFKK stacks, which were deployed on virtual machines and Kubernetes. Currently, our organization is transitioning to using syslogs to forward logs to external observability tools such as New Relic, Datadog, Splunk, and others. Admittedly, working with syslogs is new to me, and I have limited knowledge in areas like configuration and troubleshooting.

The process involves setting up a virtual machine (VM) and installing syslogs as a container within it. Subsequently, logs will be forwarded to the external observability tools.

syslogs tcp architecture

The diagram illustrates the process of forwarding logs to a syslogs server, which then transmits the logs to an external party. From a security standpoint, I am required to avoid using a TCP connection between the application and the syslogs server, as well as between the syslogs server and the external party. Consequently, I must implement TLS end-to-end encryption between both entities, as depicted in the diagram.

syslogs tls architecture

To use syslogs as a container with TLS encryption, the first step is generating a private key. The choice of tools for this task, such as OpenSSL or Certtool, depends on the project’s requirements. However, to streamline the process and avoid adding complexity to the pipeline, I have opted to include the private key generation command directly in the Dockerfile, as shown below:

FROM ubuntu:23.04
RUN apt-get update -y
RUN apt-get install -y rsyslog-gnutls ca-certificates

WORKDIR /src

RUN openssl req -x509 -nodes -newkey rsa:2048 -keyout /src/key.pem -out /src/cert.pem -days 7300 -subj '/CN=localhost' -addext "keyUsage = digitalSignature, keyEncipherment, dataEncipherment, cRLSign, keyCertSign" -addext "extendedKeyUsage = serverAuth, clientAuth"

ADD ./src/myconfig.conf /etc/rsyslog.d/myconfig.conf
RUN cp /src/cert.pem /etc/rsyslog.d/cert.pem
RUN cp /src/key.pem /etc/rsyslog.d/key.pem

EXPOSE 6514

ENTRYPOINT [ "rsyslogd", "-n", "-f", "/etc/rsyslog.d/myconfig.conf" ]

The first three lines entail the installation of syslogs and the generation of CA certificates within the system.

RUN openssl req -x509 -nodes -newkey rsa:2048 -keyout /src/key.pem -out /src/cert.pem -days 7300 -subj '/CN=localhost' -addext "keyUsage = digitalSignature, keyEncipherment, dataEncipherment, cRLSign, keyCertSign" -addext "extendedKeyUsage = serverAuth, clientAuth"

The provided command is a straightforward way to generate a private key, including keyUsage options that are essential for certain endpoints. The subsequent lines involve copying the syslogs configuration files and private keys into the container.

As you’re aware, some individuals prefer separate configuration files for server and client setups to enhance clarity and visibility. However, in my case, I prefer consolidating everything into a single configuration file, addressing both log input and output configurations in one place for simplicity.

global(
     DefaultNetstreamDriver="gtls"
     DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca-certificates.crt"
     DefaultNetstreamDriverCertFile="/etc/rsyslog.d/cert.pem"
     DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/key.pem"
)

The provided code is for configuring the certificate file. While a self-signed certificate is an option for the CA certificate, there are instances where a TLS handshake between the client and server is necessary. Therefore, I prefer to use `ca-certificates`, a tool that generates the CA certificate for us.

# load TCP listener
module(
     load="imtcp"
     StreamDriver.Name="gtls"
     StreamDriver.Mode="1"
     StreamDriver.Authmode="anon"
)

# start up listener at port 6514
input(
     type="imtcp"
     port="6514"
     ruleset="sendToExternal"
)

The provided configuration pertains to TLS input. To accommodate numerous applications streaming logs into our syslogs container within the VM, we use `StreamDriver.Authmode=”anon”` to eliminate the need for verifying each application’s certificate. Applications will use port 6514 to stream logs to our syslogs endpoint. After receiving logs from the applications, it is essential to forward these logs to an external party for monitoring and alerting purposes.

action(
    type="omfwd"
    target="central.mylogging.net"
    protocol="tcp"
    port="6514"
    StreamDriver="gtls"
    StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="central.mylogging.net"
)

If our organization has its own Logstash server, the recommended configuration involves using `StreamDriverAuthMode=”x509/name”`, as we use the same CA certificate and private key for both the client and server. However, when utilizing external tools such as New Relic, Datadog, or Splunk, it becomes necessary to modify the configuration by setting `StreamDriverAuthMode=”anon”` to bypass TLS verification issues.

  action(type="omfwd"
     Target="newrelic.syslog.eu.nr-data.net"
     Port="6514"
     Protocol="tcp"
     Template="newrelic-rfc5424"
     ResendLastMSGOnReconnect="on"
     StreamDriver="gtls"
     StreamDriverAuthMode="anon"
     StreamDriverMode="1"
  )

I trust that this information will assist you in configuring TLS end-to-end encryption for syslogs, both in on-premises environments and public clouds like AWS, Azure, and others.