Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revisionBoth sides next revision
docs:guide-user:base-system:log.essentials [2020/08/19 02:14] – [Introduction] comments vgaeteradocs:guide-user:base-system:log.essentials [2024/06/02 06:30] – [logread] stokito
Line 1: Line 1:
-====== Runtime Logging in OpenWrt ====== +====== Logging messages ====== 
-The openwrt system logging facility is an important debugging/monitoring capability. +{{section>meta:infobox:howto_links#basic_skills&noheader&nofooter&noeditbutton}}
-This document describes common support for the LEDE 17 implementations. +
-It appears there have been some changes in (recently) released Openwrt 18 branch. +
  
 ===== Introduction ===== ===== Introduction =====
 +The OpenWrt system logging facility is an important debugging/monitoring capability.
 The standard logging facility is implemented using ''logd'', the ubox log daemon. The standard logging facility is implemented using ''logd'', the ubox log daemon.
-This is implemented as an in-core ring buffer with fixed sized records.  +This is implemented as a [[wp>Circular_buffer|ring buffer]] with fixed sized records stored in [[wp>Random-access_memory|RAM]]
-The ring-buffer records can be read using ''logread'' on the router, streamed to a file or sent to a remote system through a TCP/UDP socket.+The ring buffer records can be read using ''logread'' on the router, streamed to a file or sent to a remote system through a TCP/UDP socket.
  
 <code bash> <code bash>
Line 13: Line 12:
 logread logread
  
-# Write a message to syslog+# Write a message with a tag to syslog
 logger -t TAG MESSAGE logger -t TAG MESSAGE
  
 # List syslog filtered by tag # List syslog filtered by tag
 logread -e TAG logread -e TAG
 +</code>
 +
 +<code>
 +Usage: logger [OPTIONS] [MESSAGE]
 +
 +Write MESSAGE (or stdin) to syslog
 +
 +        -s      Log to stderr as well as the system log
 +        -t TAG  Log using the specified tag (defaults to user name)
 +        -p PRIO Priority (numeric or facility.level pair)
 +</code>
 +
 +Examples of using priority and tag values:
 +<code bash>
 +logger "example"
 +logger -p notice -t example_tag "example notice"
 +logger -p err -t example_tag "example error"
 +# Fri May  8 00:23:26 2020 user.notice root: example
 +# Fri May  8 00:23:31 2020 user.notice example_tag: example notice
 +# Fri May  8 00:23:40 2020 user.err example_tag: example error
 </code> </code>
  
Line 36: Line 55:
  
 ===== logd ===== ===== logd =====
-''logd'' is configured in ''/etc/config/system''. After changing the file, run +''logd'' is a default OpenWrt logging daemon provided by [[https://github.com/openwrt/ubox|ubox]] package. 
 +It also listens for ''/dev/log'' and records syslog messages. 
 +It'configured in ''/etc/config/system''. After changing the file, run 
  
-<code> +<code bash
-/etc/init.d/log restart +service log restart 
-/etc/init.d/system restart+service system restart
 </code> </code>
  
 to read in the new configuration and restart the service. to read in the new configuration and restart the service.
  
-There are three basic destinations for log messages: the RAM ring-buffer (the default), a local persistent file, a remote destination listening for messages on a TCP or UDP port.+There are three basic destinations for log messages: the RAM ring buffer (the default), a local persistent file, a remote destination listening for messages on a TCP or UDP port.
  
 The full set of ''log_*'' options for ''/etc/config/system'' are defined in  The full set of ''log_*'' options for ''/etc/config/system'' are defined in 
Line 51: Line 72:
  
 ==== logread ==== ==== logread ====
-This is the default interface and the simplest. +This is the default interface to read log messages. It's provided by the [[https://github.com/openwrt/ubox|ubox]] package.
-It is a local executable that will read the ring-buffer records and display them chronologically.+
  
-==== Local File Logging ====+It is a local executable in ''/sbin/logread'' that will read the ring buffer records and display them chronologically. 
 + 
 +To show all log messages that contains a specific text (like a daemon name) and follow (like in ''tail -f'') use: 
 +<code bash> 
 +logread -fe firewall 
 +</code> 
 + 
 +Options: 
 +<code> 
 +-s <path> Path to ubus socket 
 +-l <count> Got only the last 'count' messages 
 +-e <pattern> Filter messages with a regexp 
 +-r <server> <port> Stream message to a server 
 +-F <file> Log file 
 +-S <bytes> Log size 
 +-p <file> PID file 
 +-h <hostname> Add hostname to the message 
 +-P <prefix> Prefix custom text to streamed messages 
 +-z <facility> handle only messages with given facility (0-23), repeatable 
 +-Z <facility> ignore messages with given facility (0-23), repeatable 
 +-f Follow log messages 
 +-u Use UDP as the protocol 
 +-t Add an extra timestamp 
 +-0 Use \0 instead of \n as trailer when using TCP 
 +</code> 
 + 
 +Please note that if you install the [[:docs:guide-user:perf_and_log:log.syslog-ng3|syslog-ng]] then the logread command will be overridden with it's own ''/usr/sbin/logread'' that has less options. 
 + 
 + 
 + 
 +==== Local file logging ====
 In order to log to a local file on the router, one needs to set the following options: In order to log to a local file on the router, one needs to set the following options:
  
Line 64: Line 114:
 </code> </code>
  
-==== Network Logging ====+==== Network logging ====
 In order to log remotely one needs to set the following options in ''/etc/config/system'' In order to log remotely one needs to set the following options in ''/etc/config/system''
  
Line 88: Line 138:
 </code> </code>
  
-and then reload the rules using ''/etc/init.d/firewall restart''.+and then reload the rules using ''service firewall restart''.
  
 For the LAN-side station/client, there are a large number of mechanisms to listen for log messages. For the LAN-side station/client, there are a large number of mechanisms to listen for log messages.
Line 97: Line 147:
 ncat -4 -l 5555 ncat -4 -l 5555
  
-# UDP+Read UDP logs with ncat or python3
 ncat -u -4 -l 5555 ncat -u -4 -l 5555
 +python3 -c "import socket
 +s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
 +s.bind(('0.0.0.0', 5141))
 +while True:
 +   print(s.recvfrom(4096)[0].decode('utf-8'))"
 </code> </code>
  
 Log messages are in [[https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/|traditional syslog format (RFC 3164 / 5424)]], beginning with a priority number in angle brackets (e.g., <30>) and lacking a terminating newline. Log messages are in [[https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/|traditional syslog format (RFC 3164 / 5424)]], beginning with a priority number in angle brackets (e.g., <30>) and lacking a terminating newline.
-The above netcat method will therefore yield somewhat messy output. +The above netcat method will therefore yield somewhat messy output. The python log reader above will most of the time get the line breaks into the right spots. A cleaner solution is to send messages to a remote machine's syslog daemon, in which case they will appear in the remote system's logs.
-A cleaner solution is to send messages to a remote machine's syslog daemon, in which case they will appear in the remote system's logs.+
 See [[https://www.rsyslog.com/receiving-messages-from-a-remote-system/|Receiving Messages from a Remote System]] for server configuration instructions for rsyslog. See [[https://www.rsyslog.com/receiving-messages-from-a-remote-system/|Receiving Messages from a Remote System]] for server configuration instructions for rsyslog.
  
Line 110: Line 164:
 There is a section on iptable event logging which can cause a noticable latency in traffic throughput using TCP socket logging. There is a section on iptable event logging which can cause a noticable latency in traffic throughput using TCP socket logging.
  
-====== Test Runtime Logging Support =====+===== Test runtime logging support ====
 If you want to test the logging out, just run a command like  If you want to test the logging out, just run a command like 
  
-''logger testLog "Blah1"''+<code>logger testLog "Blah1"</code>
  
 and it should be written to the configured destination. and it should be written to the configured destination.
 If an event is not logged, check: If an event is not logged, check:
  
- * ''/sbin/logd'' is running; it should have an argument of ''-S <log_size>'' indicating the size of the ring-buffer,  + * ''/sbin/logd'' is running; it should have an argument of ''-S <log_size>'' indicating the size of the ring buffer,  
  * ''logd'' is configured correctly in ''/etc/config/system'',  * ''logd'' is configured correctly in ''/etc/config/system'',
- * restart it using ''/etc/init.d/log restart'' and check for warnings/errors+ * restart it using ''service log restart'' and check for warnings/errors 
 + 
 +===== Logrotate ===== 
 +To automatically manage large collections of daily, weekly, or monthly logs, you may want to use [[packages:pkgdata:logrotate]]. 
 +Here's an example that rotates a persistent log on a USB storage each night keeping it for 1 week. 
 + 
 +<code bash> 
 +# Install packages 
 +opkg update 
 +opkg install logrotate 
 + 
 +# Configure logging 
 +uci set system.@system[0].log_file="/mnt/sda1/logs/system.log" 
 +uci set system.@system[0].log_remote="0" 
 +uci commit system 
 +service system restart  
 + 
 +# Configure logrotate 
 +cat << "EOF" > /etc/logrotate.conf 
 +include /etc/logrotate.d 
 +/mnt/sda1/logs/system.log { 
 +    daily 
 +    rotate 1 
 +    missingok 
 +    notifempty 
 +    postrotate 
 +        service log restart 
 +        sleep 1 
 +        logger -p warn -s "Log rotation complete" 
 +    endscript 
 +
 +EOF 
 + 
 +# Configure cron 
 +cat << "EOF" >> /etc/crontabs/root 
 +58 23 * * * logrotate /etc/logrotate.conf 
 +EOF 
 +service cron restart 
 + 
 +# Debugging 
 +logrotate --verbose --debug /etc/logrotate.conf 
 +</code>
  
 ===== Alternative implementations ===== ===== Alternative implementations =====
-See **rsyslog** - to e.g. rout all or specific logs to a (central) rsyslog receiver+See **rsyslog** - to e.g. route all or specific logs to a (central) rsyslog receiver
  
 <code bash> <code bash>
Line 137: Line 232:
 </code> </code>
  
 +If you add to the rsyslog receiver's /etc/rsyslog.conf e.g. this template:
 +
 +<code>
 +$template DynamicFile,"/mnt/sda1/logs/%HOSTNAME%/forwarded-logs.log"
 +*.* -?DynamicFile
 +</code>
 +you get the messages separated from every sender in a own folder.
 ===== rsyslog and Logz.io ===== ===== rsyslog and Logz.io =====
-You can support logging direct to a cloud ELK provider like Logz.io by adding a few lines to your rsyslog.conf :+You can support logging direct to a cloud ELK provider like Logz.io by adding a few lines to your ''rsyslog.conf''.
  
-replace "codecodecodewith your unique Logz.io identifier, it's 32 characters. +Replace ''codecodecode'' with your unique Logz.io identifier, it's 32 characters. 
-And will appear in help manuals when you're logged in, reference the guide here : [[https://app.logz.io/#/dashboard/data-sources/rsyslog]]+And will appear in help manuals when you're logged in, reference the guide [[https://app.logz.io/#/dashboard/data-sources/rsyslog|here]].
  
 <code> <code>
Line 148: Line 250:
 </code> </code>
  
-Confirm you have the right config with :+Confirm you have the right config with:
  
 <code bash> <code bash>
  • Last modified: 2024/07/27 17:26
  • by stokito