Logging messages
Introduction
The OpenWrt system logging facility is an important debugging/monitoring capability.
The standard logging facility is implemented using logd
, the ubox log daemon.
This is implemented as a ring buffer with fixed sized records stored in RAM.
The ring buffer records can be read using logread
on the router, streamed to a file or sent to a remote system through a TCP/UDP socket.
# List syslog logread # Write a message with a tag to syslog logger -t TAG MESSAGE # List syslog filtered by tag logread -e TAG
Usage: logger [OPTIONS] [MESSAGE] Write MESSAGE (or stdin) to syslog -s Log to stderr as well as the system log -t TAG Log using the specified tag (defaults to user name) -p PRIO Priority (numeric or facility.level pair)
Examples of using priority and tag values:
logger "example" logger -p notice -t example_tag "example notice" logger -p err -t example_tag "example error" # Fri May 8 00:23:26 2020 user.notice root: example # Fri May 8 00:23:31 2020 user.notice example_tag: example notice # Fri May 8 00:23:40 2020 user.err example_tag: example error
Messages format
The message format differs based on the destination (local logread, local file, remote socket). Roughly it can be viewed as:
<time stamp> <router name> <subsystem name/pid> <log_prefix>: <message body>
The logging message facility and priority are roughly equivalent to syslog implementations (see linux /usr/include/sys/syslog.h
).
The local 'logread' executable puts the facility.priority after the time stamp.
Logging to a remote socket puts a numeric value before the time stamp.
For some common OpenWrt messages see log.messages. - the log.messages reference is way out of date but a useful placeholder.
logd
logd
is configured in /etc/config/system
. After changing the file, run
service log restart service system restart
to read in the new configuration and restart the service.
There are three basic destinations for log messages: the RAM ring buffer (the default), a local persistent file, a remote destination listening for messages on a TCP or UDP port.
The full set of log_*
options for /etc/config/system
are defined in
System Configuration
logread
This is the default interface and the simplest. It is a local executable that will read the ring buffer records and display them chronologically.
Local file logging
In order to log to a local file on the router, one needs to set the following options:
config system ... option log_file '/var/log/mylog' option log_remote '0'
Network logging
In order to log remotely one needs to set the following options in /etc/config/system
config system ... option log_ip <destination IP> option log_port <destination port> option log_proto <tcp or udp>
For the destination port, if you'll be manually reading the logs on the remote system as an unprivileged user (such as via the netcat command given below), then specify a high port (e.g. 5555). If you're sending to a syslog server, use whatever port the syslog server is listening on (typically 514).
Additionally, the firewall3 default is to ACCEPT all LAN traffic. If the router blocks LAN-side access, add the following firewall3 rule to /etc/config/firewall
to ACCEPT tcp/udp traffic from the router to the LAN-side.
config rule option target 'ACCEPT' option dest 'lan' option proto 'tcp udp' option dest_port '5555' option name 'ACCEPT-LOG-DEVICE-LAN'
and then reload the rules using service firewall restart
.
For the LAN-side station/client, there are a large number of mechanisms to listen for log messages. One of the simplest is ncat:
# TCP ncat -4 -l 5555 # Read UDP logs with ncat or python3 ncat -u -4 -l 5555 python3 -c "import socket s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.bind(('0.0.0.0', 5141)) while True: print(s.recvfrom(4096)[0].decode('utf-8'))"
Log messages are in traditional syslog format (RFC 3164 / 5424), beginning with a priority number in angle brackets (e.g., <30>) and lacking a terminating newline. The above netcat method will therefore yield somewhat messy output. The python log reader above will most of the time get the line breaks into the right spots. A cleaner solution is to send messages to a remote machine's syslog daemon, in which case they will appear in the remote system's logs. See Receiving Messages from a Remote System for server configuration instructions for rsyslog.
The advantage to using TCP is reliability - it logs every event. The disadvantage is it can cause some performance degradation on the router if the logging level is high. There is a section on iptable event logging which can cause a noticable latency in traffic throughput using TCP socket logging.
Test runtime logging support
If you want to test the logging out, just run a command like
logger testLog "Blah1"
and it should be written to the configured destination. If an event is not logged, check:
* /sbin/logd
is running; it should have an argument of -S <log_size>
indicating the size of the ring buffer,
* logd
is configured correctly in /etc/config/system
,
* restart it using service log restart
and check for warnings/errors
Logrotate
To automatically manage large collections of daily, weekly, or monthly logs, you may want to use logrotate. Here's an example that rotates a persistent log on a USB storage each night keeping it for 1 week.
# Install packages opkg update opkg install logrotate # Configure logging uci set system.@system[0].log_file="/mnt/sda1/logs/system.log" uci set system.@system[0].log_remote="0" uci commit system service system restart # Configure logrotate cat << "EOF" > /etc/logrotate.conf include /etc/logrotate.d /mnt/sda1/logs/system.log { daily rotate 1 missingok notifempty postrotate service log restart sleep 1 logger -p warn -s "Log rotation complete" endscript } EOF # Configure cron cat << "EOF" >> /etc/crontabs/root 58 23 * * * logrotate /etc/logrotate.conf EOF service cron restart # Debugging logrotate --verbose --debug /etc/logrotate.conf
Alternative implementations
See rsyslog - to e.g. route all or specific logs to a (central) rsyslog receiver
opkg install rsyslog
With the config file: /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none;kern.none /var/log/messages .. kern.* @192.168.1.119:514
If you add to the rsyslog receiver's /etc/rsyslog.conf e.g. this template:
$template DynamicFile,"/mnt/sda1/logs/%HOSTNAME%/forwarded-logs.log" *.* -?DynamicFile
you get the messages separated from every sender in a own folder.
rsyslog and Logz.io
You can support logging direct to a cloud ELK provider like Logz.io by adding a few lines to your rsyslog.conf
.
Replace codecodecode
with your unique Logz.io identifier, it's 32 characters.
And will appear in help manuals when you're logged in, reference the guide here.
$template logzFormatFileTagName,"[codecodecodecode] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=TYPE] %msg%\n" *.* @@listener.logz.io:5000;logzFormatFileTagName
Confirm you have the right config with:
rsyslogd -N1
Archive
The logging mechanism discussed here uses logd
. There are other packages that
provide the same functionality.
See syslog-ng
(log.syslog-ng3).
- the syslog-ng
page appears very out-of-date.