Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revisionBoth sides next revision
docs:guide-user:base-system:log.essentials [2018/08/03 20:14] dturvenedocs:guide-user:base-system:log.essentials [2024/06/02 06:30] – [logread] stokito
Line 1: Line 1:
-====== Runtime Logging in OpenWrt ====== +====== Logging messages ====== 
-The openwrt system logging facility is an important debugging/monitoring capability.  This document describes common support for the LEDE 17 implementations.  It appears there have been some changes in (recently) released Openwrt 18 branch. +{{section>meta:infobox:howto_links#basic_skills&noheader&nofooter&noeditbutton}}
  
 ===== Introduction ===== ===== Introduction =====
-The standard logging facility is implemented using ''logd'', the ubox log daemon.  This is implemented as an in-core ring buffer with fixed sized records.  +The OpenWrt system logging facility is an important debugging/monitoring capability. 
-The ring-buffer records can be read using ''readlog'' on the router, streamed to a file or sent to a remote system through a TCP/UDP socket.+The standard logging facility is implemented using ''logd'', the ubox log daemon. 
 +This is implemented as a [[wp>Circular_buffer|ring buffer]] with fixed sized records stored in [[wp>Random-access_memory|RAM]]
 +The ring buffer records can be read using ''logread'' on the router, streamed to a file or sent to a remote system through a TCP/UDP socket. 
 + 
 +<code bash> 
 +# List syslog 
 +logread 
 + 
 +# Write a message with a tag to syslog 
 +logger -t TAG MESSAGE 
 + 
 +# List syslog filtered by tag 
 +logread -e TAG 
 +</code> 
 + 
 +<code> 
 +Usage: logger [OPTIONS] [MESSAGE] 
 + 
 +Write MESSAGE (or stdin) to syslog 
 + 
 +        -s      Log to stderr as well as the system log 
 +        -t TAG  Log using the specified tag (defaults to user name) 
 +        -p PRIO Priority (numeric or facility.level pair) 
 +</code> 
 + 
 +Examples of using priority and tag values: 
 +<code bash> 
 +logger "example" 
 +logger -p notice -t example_tag "example notice" 
 +logger -p err -t example_tag "example error" 
 +# Fri May  8 00:23:26 2020 user.notice root: example 
 +# Fri May  8 00:23:31 2020 user.notice example_tag: example notice 
 +# Fri May  8 00:23:40 2020 user.err example_tag: example error 
 +</code>
  
 ===== Messages format ===== ===== Messages format =====
-The message format differs based on the destination (local logread, local file, remote socket).  Roughly it can be viewed as:+The message format differs based on the destination (local logread, local file, remote socket). 
 +Roughly it can be viewed as:
    
-''<time stamp> <router name> <subsystem name/pid>: <message body>''+<code> 
 +<time stamp> <router name> <subsystem name/pid> <log_prefix>: <message body> 
 +</code>
  
-The logging message facility and priority are roughly equivalent to syslog implementations (see linux  +The logging message facility and priority are roughly equivalent to syslog implementations (see linux ''/usr/include/sys/syslog.h''). 
-''/usr/include/sys/syslog.h'').  logread puts the facility.priority after the time stamp.  Remote socket puts a numeric value before the time stamp+The local 'logread' executable puts the facility.priority after the time stamp. 
- +Logging to a remote socket puts a numeric value before the time stamp.
-FIXME - the log messages example is way out of date but a useful placeholder ...+
  
 For some common OpenWrt messages see [[docs:guide-user:perf_and_log:log.messages]]. For some common OpenWrt messages see [[docs:guide-user:perf_and_log:log.messages]].
 +FIXME - the log.messages reference is way out of date but a useful placeholder.
  
-===== Alternative implementations ===== +===== logd ===== 
-The logging mechanism discussed here uses ''logd''There are other packages that  +''logd'' is a default OpenWrt logging daemon provided by [[https://github.com/openwrt/ubox|ubox]] package
-provide the same functionality+It also listens for ''/dev/log'' and records syslog messages. 
 +It's configured in ''/etc/config/system''. After changing the file, run 
  
-See ''syslog-ng'' ([[docs:guide-user/perf_and_log/log.syslog-ng3]]). +<code bash> 
-FIXME - the ''syslog-ng'' page appears very out-of-date.+service log restart 
 +service system restart 
 +</code> 
 + 
 +to read in the new configuration and restart the service. 
 + 
 +There are three basic destinations for log messages: the RAM ring buffer (the default), a local persistent file, a remote destination listening for messages on a TCP or UDP port. 
 + 
 +The full set of ''log_*'' options for ''/etc/config/system'' are defined in  
 +[[docs:guide-user:base-system:system_configuration|System Configuration]] 
 + 
 +==== logread ==== 
 +This is the default interface to read log messages. It's provided by the [[https://github.com/openwrt/ubox|ubox]] package. 
 + 
 +It is a local executable in ''/sbin/logread'' that will read the ring buffer records and display them chronologically. 
 + 
 +To show all log messages that contains a specific text (like a daemon name) and follow (like in ''tail -f'') use: 
 +<code bash> 
 +logread -fe firewall 
 +</code> 
 + 
 +Options: 
 +<code> 
 +-s <path> Path to ubus socket 
 +-l <count> Got only the last 'count' messages 
 +-e <pattern> Filter messages with a regexp 
 +-r <server> <port> Stream message to a server 
 +-F <file> Log file 
 +-S <bytes> Log size 
 +-p <file> PID file 
 +-h <hostname> Add hostname to the message 
 +-P <prefix> Prefix custom text to streamed messages 
 +-z <facility> handle only messages with given facility (0-23), repeatable 
 +-Z <facility> ignore messages with given facility (0-23), repeatable 
 +-f Follow log messages 
 +-u Use UDP as the protocol 
 +-t Add an extra timestamp 
 +-0 Use \0 instead of \n as trailer when using TCP 
 +</code> 
 + 
 +Please note that if you install the [[:docs:guide-user:perf_and_log:log.syslog-ng3|syslog-ng]] then the logread command will be overridden with it's own ''/usr/sbin/logread'' that has less options. 
 + 
 + 
 + 
 +==== Local file logging ==== 
 +In order to log to a local file on the router, one needs to set the following options: 
 + 
 +<code bash> 
 +config system  
 +... 
 +   option log_file '/var/log/mylog' 
 +   option log_remote '0' 
 +</code> 
 + 
 +==== Network logging ==== 
 +In order to log remotely one needs to set the following options in ''/etc/config/system'' 
 + 
 +<code bash> 
 +config system 
 +... 
 +   option log_ip <destination IP> 
 +   option log_port <destination port> 
 +   option log_proto <tcp or udp> 
 +</code> 
 + 
 +For the destination port, if you'll be manually reading the logs on the remote system as an unprivileged user (such as via the netcat command given below), then specify a high port (e.g. 5555). If you're sending to a syslog server, use whatever port the syslog server is listening on (typically 514). 
 + 
 +Additionally, the firewall3 default is to ACCEPT all LAN traffic. If the router blocks LAN-side access, add the following firewall3 rule to ''/etc/config/firewall'' to ACCEPT tcp/udp traffic from the router to the LAN-side. 
 + 
 +<code bash> 
 +config rule 
 +      option target 'ACCEPT' 
 +      option dest 'lan' 
 +      option proto 'tcp udp' 
 +      option dest_port '5555' 
 +      option name 'ACCEPT-LOG-DEVICE-LAN' 
 +</code> 
 + 
 +and then reload the rules using ''service firewall restart''
 + 
 +For the LAN-side station/client, there are a large number of mechanisms to listen for log messages. 
 +One of the simplest is ncat: 
 + 
 +<code bash> 
 +# TCP 
 +ncat -4 -l 5555 
 + 
 +# Read UDP logs with ncat or python3 
 +ncat -u -4 -l 5555 
 +python3 -c "import socket 
 +s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 
 +s.bind(('0.0.0.0', 5141)) 
 +while True: 
 +   print(s.recvfrom(4096)[0].decode('utf-8'))" 
 +</code> 
 + 
 +Log messages are in [[https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/|traditional syslog format (RFC 3164 / 5424)]], beginning with a priority number in angle brackets (e.g., <30>) and lacking a terminating newline. 
 +The above netcat method will therefore yield somewhat messy output. The python log reader above will most of the time get the line breaks into the right spots. A cleaner solution is to send messages to a remote machine's syslog daemon, in which case they will appear in the remote system's logs. 
 +See [[https://www.rsyslog.com/receiving-messages-from-a-remote-system/|Receiving Messages from a Remote System]] for server configuration instructions for rsyslog. 
 + 
 +The advantage to using TCP is reliability - it logs every event. 
 +The disadvantage is it can cause some performance degradation on the router if the logging level is high. 
 +There is a section on iptable event logging which can cause a noticable latency in traffic throughput using TCP socket logging.
  
-====== logd and logread ======+===== Test runtime logging support ====
 If you want to test the logging out, just run a command like  If you want to test the logging out, just run a command like 
  
-''logger testLog "Blah1"''+<code>logger testLog "Blah1"</code>
  
-And it should appear in memory (run ''logread''), over the network, or in your local file...if not check to see that ''logread'' and ''logd'' are running.+and it should be written to the configured destination. 
 +If an event is not logged, check:
  
-Both of these (logd and logread) are started, stopped and restarted from /etc/init.d/log and are configured from /etc/config/systemfor example:+ * ''/sbin/logd'' is running; it should have an argument of ''-S <log_size>'' indicating the size of the ring buffer,   
 + * ''logd'' is configured correctly in ''/etc/config/system'', 
 + * restart it using ''service log restart'' and check for warnings/errors
  
-==== File Logging =====+===== Logrotate ====
 +To automatically manage large collections of daily, weekly, or monthly logs, you may want to use [[packages:pkgdata:logrotate]]. 
 +Here's an example that rotates a persistent log on a USB storage each night keeping it for 1 week.
  
-<WRAP center round important 60%+<code bash
-This configuration of logging is not robust for logging systems errors.+# Install packages 
 +opkg update 
 +opkg install logrotate
  
-Noted when trying to capture causes of the kernel oom-killer being called.  +# Configure logging 
-</WRAP>+uci set system.@system[0].log_file="/mnt/sda1/logs/system.log" 
 +uci set system.@system[0].log_remote="0" 
 +uci commit system 
 +service system restart 
  
 +# Configure logrotate
 +cat << "EOF" > /etc/logrotate.conf
 +include /etc/logrotate.d
 +/mnt/sda1/logs/system.log {
 +    daily
 +    rotate 1
 +    missingok
 +    notifempty
 +    postrotate
 +        service log restart
 +        sleep 1
 +        logger -p warn -s "Log rotation complete"
 +    endscript
 +}
 +EOF
  
-Note that if you are file logging locally [[https://forum.openwrt.org/viewtopic.php?pid=340647#p340647|you may have to change order in which system is started]] to after fstab is started to write out the logs to the external drive.+# Configure cron 
 +cat << "EOF" >> /etc/crontabs/root 
 +58 23 * * * logrotate /etc/logrotate.conf 
 +EOF 
 +service cron restart
  
-    config system  +# Debugging 
-        option hostname 'OpenWrt'  +logrotate --verbose --debug /etc/logrotate.conf 
-        option zonename 'America/New York'  +</code>
-        option timezone 'EST5EDT,M3.2.0,M11.1.0'  +
-        option conloglevel '8'  +
-        option cronloglevel '8'  +
-        option log_file '/mnt/logging/messages' # On an fstab mounted external drive some where... +
-        option log_type 'file'  +
-        option log_size 100000  +
-        option log_buffer_size 2048  +
-        option log_remote '0'  # Need to turn this off to log locally.+
  
-==== Network Logging ====+===== Alternative implementations ===== 
 +See **rsyslog** - to e.g. route all or specific logs to a (central) rsyslog receiver
  
-In ''/etc/config/system'' or via LuCI:+<code bash> 
 +opkg install rsyslog 
 +</code>
  
-  config system +With the config file: /etc/rsyslog.conf
-  (...) +
-      option log_ip <destination IP> +
-      option log_port <something not needing root, e.g. 5555> +
-      option log_proto <tcp or udp> +
-      +
  
-more log_options https://openwrt.org/docs/guide-user/base-system/system_configuration+<code> 
 +*.info;mail.none;authpriv.none;cron.none;kern.none  /var/log/messages 
 +.. 
 +kern.*   @192.168.1.119:514 
 +</code>
  
-==== File Logging ==== +If you add to the rsyslog receiver's /etc/rsyslog.conf e.g. this template:
-TODO, I tried to use '"log_file"' but the file was not written There was a warning to change the+
  
-Set ''option log_remote 0''.+<code> 
 +$template DynamicFile,"/mnt/sda1/logs/%HOSTNAME%/forwarded-logs.log" 
 +*.* -?DynamicFile 
 +</code> 
 +you get the messages separated from every sender in a own folder. 
 +===== rsyslog and Logz.io ===== 
 +You can support logging direct to a cloud ELK provider like Logz.io by adding a few lines to your ''rsyslog.conf''.
  
-==== RAM Logging ==== +Replace ''codecodecode'' with your unique Logz.io identifier, it's 32 characters. 
-TODO+And will appear in help manuals when you're logged in, reference the guide [[https://app.logz.io/#/dashboard/data-sources/rsyslog|here]].
  
 +<code>
 +$template logzFormatFileTagName,"[codecodecodecode] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=TYPE] %msg%\n"
 +*.* @@listener.logz.io:5000;logzFormatFileTagName
 +</code>
 +
 +Confirm you have the right config with:
 +
 +<code bash>
 +rsyslogd -N1
 +</code>
 +
 +===== Archive =====
 +The logging mechanism discussed here uses ''logd''. There are other packages that 
 +provide the same functionality. 
 +
 +See ''syslog-ng'' ([[docs:guide-user/perf_and_log/log.syslog-ng3]]).
 +FIXME - the ''syslog-ng'' page appears very out-of-date.
  
  • Last modified: 2024/07/27 17:26
  • by stokito