Docker Syslog

My objective is to get logging output from my container applications consolidated under /var/log, with logfile names matching their origin container. Logfile cohabitation will allow me to incorporate them in the housekeeping rotation ritual of the rest of my system, and help with speedy fault diagnosis.

1. Docker Syslog Log Driver
2. More Videos For Docker Syslog »
3. Docker Syslog Server

We need to restart the docker to apply these changes. First stopped the docker daemon and then started it again. If we check now using the ‘docker info’ command, we can see the default logging driver has been changed to ‘syslog’. Code: sudo systemctl stop docker sudo systemctl start docker docker info. Docker run -v /tmp/socket/:/tmp/ -p 9000:9000 usman/go-auth -t syslog -l debug run -p 9000 -db-host MYSQLHOSTNAME The sample application requires a running instance of MySQL which can be setup inside a Docker container with the following command. Versions used: Ubuntu 16.04 Docker version 17.03.1-ce, build c6d412e Docker-Compose version 1.12.0-rc2, build 08dc2a4 Install Docker CE Install a few basics that we need: sudo apt-get install apt.

The environment: Traefik v1, Docker, Docker-Compose, Nginx, Ubuntu 18.04, rsyslogd, logrotate. This is a basic setup presenting the 'Welcome to nginx!' page.

Some Docker logging basics

With some standard containers running, some simple logging can be seen using the docker logs command at the prompt, specifying a container name. Here we just see a bunch of simple HTTP 200 responses for the Nginx basic 'Welcome to nginx!'.

I am using a very basic Docker config with docker-compose serving up Traefik and Nginx containers, with the default Welcome screen.

Configuring a Syslog catcher for custom log files

Docker Syslog Log Driver

For maximum flexibility, I would like to be able to specify a tag in the compose definition that determines the eventual log file name, and I would like to group all of the container based application logs under a new directory, /var/log/containers. A subdirectory is optional.

In /etc/rsyslog.d I create a new template file, the name is arbitrary, here 40-docker.conf, but check the below text about duplicates when naming. With your favourite text editor add the contents:

I will prefix any container logs with the string 'docker-' for filtering and grouping purposes. Run the following, if you are going to also use a logging subdirectory strategy:

Restart rsyslogd with sudo systemctl restart rsyslog (Ubuntu 18.04).

About duplicate log file entries

Syslog will read the files in /etc/rsyslog.d in order. Our sample .conf file has a tilde at the end, which indicates a stop to further processing. Therefore, if you create a .conf file such as '40-docker.conf' that is processed before the 50-default.conf file, it will be logged to the appropriate custom container log, then processing for that entry will stop.

If you create a .conf file such as '60-docker.conf' it will be processed after the 50-default.conf, so the entry will first be logged in the main default syslog, then in the custom docker file. You will end up with the same entry in both places.

Setting up Docker Compose for the logging

I want to use the syslog format, so my logging entry in the compose file for Nginx looks like this.

There is more info on the syslog driver syntax and options here: https://docs.docker.com/config/containers/logging/syslog/

After adding, and doing a docker-compose up -d, then an Nginx page reload, I am rewarded with a custom log file. I can quickly see from the file name which container it's from, which will be great for debugging.

Set up log rotation for the docker logs

This is straight up Linux stuff. The example here is just a starter rotation config applying to all files in the containers log directory. There are a ton of options, see the man page for logrotate.

With your favourite text editor, create a file in /etc/logrotate.d called, say, docker. Add these contents, change as you see fit. Wildcards should be used with caution.

I've checked, and logrotate works with files owned by syslog or root, so probably the logfile owner doesn't matter.

Apply custom logging to all the container things

Once proven, the 'logging' option can be - cautiously - applied to all of your docker-compose services. Verify the log rotation after a day or so.

Having the logs on hand, all in one place, will allow you to really fine tune your configuration. Often there are many small errors that won't stop an application from working, but could potentially in future, and might indicate server errors or security issues.

To illustrate the point, when I enabled logging for my Traefik container, I discovered an 'Error renewing certificate from LE' error that had been going on the whole life of the server. It was functioning, but constantly faulting. It took less than a minute to fix.

If you are not looking at Nginx or Trafik logging specifics, the next couple of chapters can be skipped, feel free to jump forward to 'It Works!'.

A note about Nginx application logging

This chapter is about more advanced stacks, and not about the demo build example.

If you have followed some of my previous posts, or it just is, you might have local Nginx configuration, not using syslog. In that situation, you might have a volume mapping in your docker compose a bit like: /data/logs/nginx-wp1:/var/log/nginx in combination with an Nginx config file containing something like:

In my case I would like to keep the separation between access and error logs, but still benefit from both log rotation and cohabitation - having the logs coexist with the rest of my containers' logs. So rather than stdout and rsyslog, I am going to map the log volumes directly.

This is optional, the default stdout logging also works fine as per the example compose file at the end - feel free to skip this section.

Map the container log directory to the Nginx container

Volume map before:

Volume mapping after:

Tweak the log file names in the Nginx default.conf

Now that the log files will be cohabiting with the other container logs, I will modify the file names a little.

Log file names before:

Log file names after:

After the mods, I regenerate with compose, and note that the server has gone into a bit of an out of memory death spiral. At this point I remember it's just a 1GB server running double Bloatpress, and there is just a few hundred MB memory headroom. I reboot, then go for it again, successfully this time.

I have another post on reducing memory use.

Nginx Log File permissions

The Nginx logs have been created as root, so rather than wait and see if log rotate breaks, I run a sudo chown syslog:adm on them.

A note about Traefik logging

In order for the traefik logging to work in this way, it needs to be logging to stdout. I found that my config initially was not. Check your traefik.toml file (or equivalent) and verify that the [traefikLog] parameter is unset / blank.

If the parameter is not there at all, add it with nothing in it, then restart the container. This channel is pretty quiet if your config is clean, so if you don't see a traefik log file generated after setting it up, it might not be a problem.

If you would like ACCESS logs, do the same but add the [accessLog] parameter. No further values will enable with default settings. These logs will of course be super noisy, so watch out for disk fill risk. More info here: https://docs.traefik.io/configuration/logs/

Having both seems to munge both into one file - as you would expect with stdout logging.

It works!

I check after a day, and the log files, and rotation looks good. Your reward will be files cycling and compressing daily.

I have to confess that before I established this, it felt like I was flying blind a bit with containerised apps. Now I have control! It's also given me a huge amount of info to browse through and look for opportunities to enhance the configuration.

Example full docker-compose file, with logging

• This is a simplified Nginx config with no volume mapping or custom config files
• I have chosen the 'maroilles-alpine' traefik image to fix this at V1. You could simply have image: traefik:alpine or even just image: traefik, but at some point that will become V2 which is not the config I have based this on.

Barely relevant log pile photo courtesy of Pär Pärsson on Unsplash

You are welcome to comment anonymously, but bear in mind you won't get notified of any replies! Registration details (which are tiny) are stored on my private EC2 server and never shared. You can also use github creds. Marvel vs capcom play.

More Videos For Docker Syslog »

Estimated reading time: 5 minutes

The journald logging driver sends container logs to thesystemd journal.Log entries can be retrieved using the journalctl command, through use of thejournal API, or using the docker logs command.

In addition to the text of the log message itself, the journald log driverstores the following metadata in the journal with each message:

Usage

To use the journald driver as the default logging driver, set the log-driverand log-opt keys to appropriate values in the daemon.json file, which islocated in /etc/docker/ on Linux hosts orC:ProgramDatadockerconfigdaemon.json on Windows Server. For more aboutconfiguring Docker using daemon.json, seedaemon.json.

The following example sets the log driver to journald:

Restart Docker for the changes to take effect.

To configure the logging driver for a specific container, use the --log-driverflag on the docker run command.

Options

Use the --log-opt NAME=VALUE flag to specify additional journald loggingdriver options.

If a collision occurs between label and env keys, the value of the env takesprecedence. Each option adds additional fields to the attributes of a loggingmessage.

Below is an example of the logging options required to log to journald.

This configuration also directs the driver to include in the payload the labellocation, and the environment variable TEST. If the --env 'TEST=false'or --label location=west arguments were omitted, the corresponding key wouldnot be set in the journald log.

Note regarding container names

Docker Syslog Server

The value logged in the CONTAINER_NAME field is the name of the container thatwas set at startup. If you use docker rename to rename a container, the newname is not reflected in the journal entries. Journal entries continueto use the original name.

Retrieve log messages with journalctl

Use the journalctl command to retrieve log messages. You can apply filterexpressions to limit the retrieved messages to those associated with a specificcontainer:

You can use additional filters to further limit the messages retrieved. The -bflag only retrieves messages generated since the last system boot:

The -o flag specifies the format for the retried log messages. Use -o jsonto return the log messages in JSON format.

View logs for a container with a TTY enabled

If TTY is enabled on a container you may see [10B blob data] in the outputwhen retrieving log messages.The reason for that is that r is appended to the end of the line andjournalctl doesn’t strip it automatically unless --all is set:

Retrieve log messages with the journal API

This example uses the systemd Python module to retrieve containerlogs: