We've recently moved to using an ELK stack for capturing our application logs, however prior to this we needed an interim logging solution, and papertrail provided the ideal solution, as it was quick to setup, and provides a free tier of 100MB per month.

Our web application is deployed as a containerised service; we build the application into a Docker image, and then use Amazon's ECS to run up a cluster of containers behind a load balancer.

We wanted to be able to capture logging information from our containers so we could diagnose problems, however as we can't be sure which container in the cluster is used to serve a given request, we have to capture the logs of all deployed containers, and then be able to filter amongst these as necessary.

However, there's a few pieces that need to be glued together to make things work, and we cover this below.

package.json

To keep things simple, in our package.json we simply tee the server STDOUT / STDERR output to a file on disk, which we can then watch with remote_syslog and ship to papertrail. This means that STDOUT / STDERR still can be used for other purposes, such as being captured to our new ELK stack.

  "scripts": {
    ...
    "start:app": "DEBUG_COLORS=yes node lib/server.js 2>&1 | tee -a /var/log/app/app.log",
    ...
  },

papertrail_log_files.yml

The papertrail_log_files.yml controls how we talk to papertrail, and is detailed in the papertrail destinations for your account. It should look somewhat like the following, though your files, host, and port details will differ:

files:  
  - /var/log/app/app.log
destination:  
  host: logs1.papertrailapp.com
  port: 46123
  protocol: tls

remote_syslog

We also need a remote_syslog, which is the default environment for the remote_syslog service.
We'll typically override this file at runtime, but if not, we'll report to 'development'

ENVIRONMENT=development  

logrotate.app

We use logrotate to rotate our log files, and therefore we provide the details of our logfile to logrotate using logrotate.app:

/var/log/app/app.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    copytruncate
}

papertrail/remote_syslog.init.d

The configuration of Papertrail's remote_syslog is controlled by papertrail/remote_syslog.init.d, most of which is boilerplate, but which also specifies both the location of the main config, and the hostname to group logs under:

# The location of the papertrail config
config="/etc/papertrail_log_files.yml"

# Set the hostname to the environment (e.g. "production") so that log streams
# from different machines across the cluster are grouped to a single papertrail stream.
[ -f /etc/default/$prog ] && . /etc/default/$prog

if [ -z "$ENVIRONMENT" ] ;  then  
    echo "ENVIRONMENT is not set, please set it in /etc/default/$prog" >&2
    exit 1
fi

EXTRAOPTIONS="--hostname=$ENVIRONMENT"  

The full papertrail/remote_syslog.init.d is shown below:

#!/bin/bash

### BEGIN INIT INFO
# Provides: remote_syslog
# Required-Start: $network $remote_fs $syslog
# Required-Stop: $network $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start and Stop
# Description: Runs remote_syslog
### END INIT INFO

#       /etc/init.d/remote_syslog
#
# Starts the remote_syslog daemon
#
# chkconfig: 345 90 5
# description: Runs remote_syslog
#
# processname: remote_syslog

prog="remote_syslog"

# The location of the papertrail config
config="/etc/papertrail_log_files.yml"  
pid_dir="/var/run"

# Set the hostname to the environment (e.g. "production") so that log streams
# from different machines across the cluster are grouped to a single papertrail stream.
[ -f /etc/default/$prog ] && . /etc/default/$prog

if [ -z "$ENVIRONMENT" ] ;  then  
  echo "ENVIRONMENT is not set, please set it in /etc/default/$prog" >&2
  exit 1
fi

EXTRAOPTIONS="--hostname=$ENVIRONMENT"

pid_file="$pid_dir/$prog.pid"

PATH=/sbin:/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH

RETVAL=0

is_running(){  
  [ -e $pid_file ]
}

start(){  
    echo -n $"Starting $prog: "

    unset HOME MAIL USER USERNAME
    $prog -c $config --pid-file=$pid_file $EXTRAOPTIONS
    RETVAL=$?
    echo
    return $RETVAL
}

stop(){  
    echo -n $"Stopping $prog: "
    if (is_running); then
      kill `cat $pid_file`
      RETVAL=$?
      echo
      return $RETVAL
    else
      echo "$pid_file not found"
    fi
}

status(){  
    echo -n $"Checking for $pid_file: "

    if (is_running); then
      echo "found"
    else
      echo "not found"
    fi
}

reload(){  
    restart
}

restart(){  
    stop
    start
}

condrestart(){  
    is_running && restart
    return 0
}


# See how we were called.
case "$1" in  
    start)
    start
    ;;
    stop)
    stop
    ;;
    status)
    status
    ;;
    restart)
    restart
    ;;
    reload)
    reload
    ;;
    condrestart)
    condrestart
    ;;
    *)
    echo $"Usage: $0 {start|stop|status|restart|condrestart|reload}"
    RETVAL=1
esac

exit $RETVAL  

Dockerfile

We then tie everything together using our Dockerfile to set up papertrail remote_syslog2 daemon

...

# Install logrotate
RUN apt-get update -q && \  
    apt-get install -y -qq logrotate

# Set the default environment variables for the remote_syslog service
ADD papertrail/remote_syslog /etc/default/remote_syslog

# Setup logrotate for the app log
ADD papertrail/logrotate.app /etc/logrotate.d/logrotate.app

# Pull in our papertrail config
ADD papertrail/remote_syslog.init.d etc/init.d/remote_syslog  
ADD papertrail/papertrail_log_files.yml /etc/papertrail_log_files.yml

# Install remote_syslog2 so we can have papertrail logging
RUN wget -q -O - https://github.com/papertrail/remote_syslog2/releases/download/v0.17/remote_syslog_linux_amd64.tar.gz \  
  | tar -zxf - \
  && cp remote_syslog/remote_syslog /usr/local/bin \
  && chmod +x /etc/init.d/remote_syslog \
  && update-rc.d remote_syslog defaults \
  && mkdir -p /var/log/app

...

CMD ["sh", "-c", "echo \"ENVIRONMENT=$ENVIRONMENT\" > /etc/default/remote_syslog && service remote_syslog start && npm version && npm start:app"]