Logging messages

The OpenWrt system logging facility is an important debugging/monitoring capability. The standard logging facility is implemented using logd, the ubox log daemon. This is implemented as a ring buffer with fixed sized records stored in RAM. The ring buffer records can be read using logread on the router, streamed to a file or sent to a remote system through a TCP/UDP socket.

# List syslog
logread
 
# Write a message with a tag to syslog
logger -t TAG MESSAGE
 
# List syslog filtered by tag
logread -e TAG

Usage: logger [OPTIONS] [MESSAGE]

Write MESSAGE (or stdin) to syslog

        -s      Log to stderr as well as the system log
        -t TAG  Log using the specified tag (defaults to user name)
        -p PRIO Priority (numeric or facility.level pair)

Examples of using priority and tag values:

logger "example"
logger -p notice -t example_tag "example notice"
logger -p err -t example_tag "example error"
# Fri May  8 00:23:26 2020 user.notice root: example
# Fri May  8 00:23:31 2020 user.notice example_tag: example notice
# Fri May  8 00:23:40 2020 user.err example_tag: example error

The message format differs based on the destination (local logread, local file, remote socket). Roughly it can be viewed as:

<time stamp> <router name> <subsystem name/pid> <log_prefix>: <message body>

The logging message facility and priority are roughly equivalent to syslog implementations (see linux /usr/include/sys/syslog.h). The local 'logread' executable puts the facility.priority after the time stamp. Logging to a remote socket puts a numeric value before the time stamp.

For some common OpenWrt messages see log.messages. - the log.messages reference is way out of date but a useful placeholder.

logd is configured in /etc/config/system. After changing the file, run

/etc/init.d/log restart
/etc/init.d/system restart

to read in the new configuration and restart the service.

There are three basic destinations for log messages: the RAM ring buffer (the default), a local persistent file, a remote destination listening for messages on a TCP or UDP port.

The full set of log_* options for /etc/config/system are defined in System Configuration

This is the default interface and the simplest. It is a local executable that will read the ring buffer records and display them chronologically.

In order to log to a local file on the router, one needs to set the following options:

config system 
...
   option log_file '/var/log/mylog'
   option log_remote '0'

In order to log remotely one needs to set the following options in /etc/config/system

config system
...
   option log_ip <destination IP>
   option log_port <destination port>
   option log_proto <tcp or udp>

For the destination port, if you'll be manually reading the logs on the remote system as an unprivileged user (such as via the netcat command given below), then specify a high port (e.g. 5555). If you're sending to a syslog server, use whatever port the syslog server is listening on (typically 514).

Additionally, the firewall3 default is to ACCEPT all LAN traffic. If the router blocks LAN-side access, add the following firewall3 rule to /etc/config/firewall to ACCEPT tcp/udp traffic from the router to the LAN-side.

config rule
      option target 'ACCEPT'
      option dest 'lan'
      option proto 'tcp udp'
      option dest_port '5555'
      option name 'ACCEPT-LOG-DEVICE-LAN'

and then reload the rules using /etc/init.d/firewall restart.

For the LAN-side station/client, there are a large number of mechanisms to listen for log messages. One of the simplest is ncat:

# TCP
ncat -4 -l 5555
 
# Read UDP logs with ncat or python3
ncat -u -4 -l 5555
python3 -c "import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('0.0.0.0', 5141))
while True:
   print(s.recvfrom(4096)[0].decode('utf-8'))"

Log messages are in traditional syslog format (RFC 3164 / 5424), beginning with a priority number in angle brackets (e.g., <30>) and lacking a terminating newline. The above netcat method will therefore yield somewhat messy output. The python log reader above will most of the time get the line breaks into the right spots. A cleaner solution is to send messages to a remote machine's syslog daemon, in which case they will appear in the remote system's logs. See Receiving Messages from a Remote System for server configuration instructions for rsyslog.

The advantage to using TCP is reliability - it logs every event. The disadvantage is it can cause some performance degradation on the router if the logging level is high. There is a section on iptable event logging which can cause a noticable latency in traffic throughput using TCP socket logging.

If you want to test the logging out, just run a command like

logger testLog "Blah1"

and it should be written to the configured destination. If an event is not logged, check:

* /sbin/logd is running; it should have an argument of -S <log_size> indicating the size of the ring buffer, * logd is configured correctly in /etc/config/system, * restart it using /etc/init.d/log restart and check for warnings/errors

To automatically manage large collections of daily, weekly, or monthly logs, you may want to use logrotate.

opkg install logrotate

Here's a working example that rotates a persistent log to a USB mount each night:

uci set system.@system[0].log_file="/mnt/sda1/logs/system.log"
uci set system.@system[0].log_remote="0"
uci commit system
service system restart 

Then in /etc/logrotate.conf:

logrotate.conf

include /etc/logrotate.d
 
/mnt/sda1/logs/system.log {
    # Rotate log files daily.
    daily
 
    # Keep 1 week worth of logs.
    rotate 1
 
    missingok
    notifempty
    postrotate
        service log restart
        sleep 1
        logger -p warn -s "Log rotation complete."
    endscript
}

Then in /etc/crontabs/root (crontab -e):

58 23 * * * logrotate /etc/logrotate.conf

To debug your configuration, try running:

logrotate --verbose --debug /etc/logrotate.conf

See rsyslog - to e.g. route all or specific logs to a (central) rsyslog receiver

opkg install rsyslog

With the config file: /etc/rsyslog.conf

*.info;mail.none;authpriv.none;cron.none;kern.none  /var/log/messages
..
kern.*					  @192.168.1.119:514

You can support logging direct to a cloud ELK provider like Logz.io by adding a few lines to your rsyslog.conf.

Replace codecodecode with your unique Logz.io identifier, it's 32 characters. And will appear in help manuals when you're logged in, reference the guide here.

$template logzFormatFileTagName,"[codecodecodecode] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=TYPE] %msg%\n"
*.* @@listener.logz.io:5000;logzFormatFileTagName

Confirm you have the right config with:

rsyslogd -N1

The logging mechanism discussed here uses logd. There are other packages that provide the same functionality.

See syslog-ng (log.syslog-ng3). - the syslog-ng page appears very out-of-date.