Logging messages

The OpenWrt system logging facility is an important debugging/monitoring capability. The standard logging facility is implemented using logd, the ubox log daemon. This is implemented as a ring buffer with fixed sized records stored in RAM. The ring buffer records can be read using logread on the router, streamed to a file or sent to a remote system through a TCP/UDP socket.

# List syslog
logread
 
# Write a message with a tag to syslog
logger -t TAG MESSAGE
 
# List syslog filtered by tag
logread -e TAG
Usage: logger [OPTIONS] [MESSAGE]

Write MESSAGE (or stdin) to syslog

        -s      Log to stderr as well as the system log
        -t TAG  Log using the specified tag (defaults to user name)
        -p PRIO Priority (numeric or facility.level pair)

Examples of using priority and tag values:

logger "example"
logger -p notice -t example_tag "example notice"
logger -p err -t example_tag "example error"
# Fri May  8 00:23:26 2020 user.notice root: example
# Fri May  8 00:23:31 2020 user.notice example_tag: example notice
# Fri May  8 00:23:40 2020 user.err example_tag: example error

The message format differs based on the destination (local logread, local file, remote socket). Roughly it can be viewed as:

<time stamp> <router name> <subsystem name/pid> <log_prefix>: <message body>

The logging message facility and priority are roughly equivalent to syslog implementations (see linux /usr/include/sys/syslog.h). The local 'logread' executable puts the facility.priority after the time stamp. Logging to a remote socket puts a numeric value before the time stamp.

For some common OpenWrt messages see log.messages. FIXME - the log.messages reference is way out of date but a useful placeholder.

logd is configured in /etc/config/system. After changing the file, run

/etc/init.d/log restart
/etc/init.d/system restart

to read in the new configuration and restart the service.

There are three basic destinations for log messages: the RAM ring buffer (the default), a local persistent file, a remote destination listening for messages on a TCP or UDP port.

The full set of log_* options for /etc/config/system are defined in System Configuration

This is the default interface and the simplest. It is a local executable that will read the ring buffer records and display them chronologically.

In order to log to a local file on the router, one needs to set the following options:

config system 
...
   option log_file '/var/log/mylog'
   option log_remote '0'

In order to log remotely one needs to set the following options in /etc/config/system

config system
...
   option log_ip <destination IP>
   option log_port <destination port>
   option log_proto <tcp or udp>

For the destination port, if you'll be manually reading the logs on the remote system as an unprivileged user (such as via the netcat command given below), then specify a high port (e.g. 5555). If you're sending to a syslog server, use whatever port the syslog server is listening on (typically 514).

Additionally, the firewall3 default is to ACCEPT all LAN traffic. If the router blocks LAN-side access, add the following firewall3 rule to /etc/config/firewall to ACCEPT tcp/udp traffic from the router to the LAN-side.

config rule
      option target 'ACCEPT'
      option dest 'lan'
      option proto 'tcp udp'
      option dest_port '5555'
      option name 'ACCEPT-LOG-DEVICE-LAN'

and then reload the rules using /etc/init.d/firewall restart.

For the LAN-side station/client, there are a large number of mechanisms to listen for log messages. One of the simplest is ncat:

# TCP
ncat -4 -l 5555
 
# Read UDP logs with ncat or python3
ncat -u -4 -l 5555
python3 -c "import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('0.0.0.0', 5141))
while True:
   print(s.recvfrom(4096)[0].decode('utf-8'))"

Log messages are in traditional syslog format (RFC 3164 / 5424), beginning with a priority number in angle brackets (e.g., <30>) and lacking a terminating newline. The above netcat method will therefore yield somewhat messy output. The python log reader above will most of the time get the line breaks into the right spots. A cleaner solution is to send messages to a remote machine's syslog daemon, in which case they will appear in the remote system's logs. See Receiving Messages from a Remote System for server configuration instructions for rsyslog.

The advantage to using TCP is reliability - it logs every event. The disadvantage is it can cause some performance degradation on the router if the logging level is high. There is a section on iptable event logging which can cause a noticable latency in traffic throughput using TCP socket logging.

If you want to test the logging out, just run a command like

logger testLog "Blah1"

and it should be written to the configured destination. If an event is not logged, check:

* /sbin/logd is running; it should have an argument of -S <log_size> indicating the size of the ring buffer, * logd is configured correctly in /etc/config/system, * restart it using /etc/init.d/log restart and check for warnings/errors

To automatically manage large collections of daily, weekly, or monthly logs, you may want to use logrotate.

opkg install logrotate

Here's a working example that rotates a persistent log to a USB mount each night:

uci set system.@system[0].log_file="/mnt/sda1/logs/system.log"
uci set system.@system[0].log_remote="0"
uci commit system
service system restart 

Then in /etc/logrotate.conf:

logrotate.conf
include /etc/logrotate.d
 
/mnt/sda1/logs/system.log {
    # Rotate log files daily.
    daily
 
    # Keep 1 week worth of logs.
    rotate 1
 
    missingok
    notifempty
    postrotate
        service log restart
        sleep 1
        logger -p warn -s "Log rotation complete."
    endscript
}

Then in /etc/crontabs/root (crontab -e):

58 23 * * * logrotate /etc/logrotate.conf

To debug your configuration, try running:

logrotate --verbose --debug /etc/logrotate.conf

See rsyslog - to e.g. route all or specific logs to a (central) rsyslog receiver

opkg install rsyslog

With the config file: /etc/rsyslog.conf

*.info;mail.none;authpriv.none;cron.none;kern.none  /var/log/messages
..
kern.*					  @192.168.1.119:514

You can support logging direct to a cloud ELK provider like Logz.io by adding a few lines to your rsyslog.conf.

Replace codecodecode with your unique Logz.io identifier, it's 32 characters. And will appear in help manuals when you're logged in, reference the guide here.

$template logzFormatFileTagName,"[codecodecodecode] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=TYPE] %msg%\n"
*.* @@listener.logz.io:5000;logzFormatFileTagName

Confirm you have the right config with:

rsyslogd -N1

The logging mechanism discussed here uses logd. There are other packages that provide the same functionality.

See syslog-ng (log.syslog-ng3). FIXME - the syslog-ng page appears very out-of-date.

This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website.More information about cookies
  • Last modified: 2023/03/29 15:55
  • by bigwave