Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revisionBoth sides next revision
docs:guide-user:troubleshooting:log.essentials [2018/03/04 02:21] – [Network Logging] add notes on networking logging guifipedrodocs:guide-user:base-system:log.essentials [2024/06/03 07:13] – rsyslog moved to a separate page stokito
Line 1: Line 1:
-====== System log in OpenWrt ======+====== Logging messages ====== 
 +{{section>meta:infobox:howto_links#basic_skills&noheader&nofooter&noeditbutton}}
  
 ===== Introduction ===== ===== Introduction =====
-In Unix it is common to use centralized logging systems using a deamon and ''/dev/log'' socket for writing messagesThere are also few helper functions like ''syslog'' & family defined in ''syslog.h''.+The OpenWrt system logging facility is an important debugging/monitoring capability. 
 +The standard logging facility is implemented using ''logd'', the ubox log daemon. 
 +This is implemented as a [[wp>Circular_buffer|ring buffer]] with fixed sized records stored in [[wp>Random-access_memory|RAM]]. 
 +The ring buffer records can be read using ''logread'' on the router, streamed to a file or sent to a remote system through a TCP/UDP socket.
  
-While many normal distributions use Syslog (with its ''syslogd'' deamon) on embedded systems there are usually some replacements used. Older OpenWrt releases (AA and earlier oneswere using BusyBox'''syslogd'' (and ''logread''while the newer ones (BB and lateruse [[docs:techref:ubox]]'s ''logd'' and ''logread''.+<code bash> 
 +# List syslog 
 +logread 
 + 
 +# Write a message with a tag to syslog 
 +logger -t TAG MESSAGE 
 + 
 +# List syslog filtered by tag 
 +logread -e TAG 
 +</code> 
 + 
 +<code> 
 +Usage: logger [OPTIONS] [MESSAGE] 
 + 
 +Write MESSAGE (or stdinto syslog 
 + 
 +        -     Log to stderr as well as the system log 
 +        -t TAG  Log using the specified tag (defaults to user name) 
 +        -p PRIO Priority (numeric or facility.level pair) 
 +</code> 
 + 
 +Examples of using priority and tag values: 
 +<code bash> 
 +logger "example" 
 +logger -p notice -t example_tag "example notice" 
 +logger -p err -t example_tag "example error" 
 +# Fri May  8 00:23:26 2020 user.notice root: example 
 +# Fri May  8 00:23:31 2020 user.notice example_tag: example notice 
 +# Fri May  8 00:23:40 2020 user.err example_tag: example error 
 +</code>
  
 ===== Messages format ===== ===== Messages format =====
-An example message that can be read from system log looks like this:+The message format differs based on the destination (local logread, local file, remote socket). 
 +Roughly it can be viewed as: 
 + 
 <code> <code>
-Feb 28 23:12:57 router user.notice kernelthe barmaid is the most beautiful woman in earth +<time stamp> <router name> <subsystem name/pid> <log_prefix><message body>
 </code> </code>
-The format includes date, hostname, facility & severity (both defined in [[http://tools.ietf.org/html/rfc3164|RFC3164]]) and the message itself. 
  
-For some common OpenWrt messages see [[docs:guide-user:troubleshooting:log.messages]].+The logging message facility and priority are roughly equivalent to syslog implementations (see linux ''/usr/include/sys/syslog.h''). 
 +The local 'logread' executable puts the facility.priority after the time stamp. 
 +Logging to a remote socket puts a numeric value before the time stamp.
  
-===== ubox ===== +For some common OpenWrt messages see [[docs:guide-user:perf_and_log:log.messages]]. 
-As said earlier, OpenWrt currently uses its own system log implementation which is implemented as part of ''ubox'' project. It consists of: +FIXME the log.messages reference is way out of date but useful placeholder.
-  - ''logd'' -- a deamon creating ''/dev/log'' socket, forwarding kernel messages & providing [[docs:techref:ubus]] ''log'' object (with ''read'' and ''write'' commands) +
-  - ''logread'' -- a tool for reading messages using ''ubus'', see help messages for its usage+
  
-===== BusyBox ===== +===== logd ===== 
-So far the vanilla firmwares offered on OpenWrt utilize the ''busybox-syslogd''Usually you can configure the syslog in ''/etc/syslogd.conf'' but this busybox ignores this[[doc:howto:log.overview]]+''logd'' is a default OpenWrt logging daemon provided by [[https://github.com/openwrt/ubox|ubox]] package. 
 +It listens on ''/dev/log'' unix socket to record syslog messages. 
 +It's configured in ''/etc/config/system''After changing the file, run 
  
-^  Name            ^   Size  ^  Description  ^ +<code bash> 
-| busybox klogd    |  242620 | Kernel logger   | +service log restart 
-| busybox syslogd  |  242620 | System logging utility  |+service system restart 
 +</code>
  
-<code> +to read in the new configuration and restart the service.
- klogd +
-           klogd [-c n] [-n]+
  
-           Kernel logger.  Options:+There are three basic destinations for log messagesthe RAM ring buffer (the default), a local persistent file, a remote destination listening for messages on a TCP or UDP port.
  
-                   -c n    Sets the default log level of console messages to n +The full set of ''log_*'' options for ''/etc/config/system'' are defined in  
-                   -n      Run as foreground process+[[docs:guide-user:base-system:system_configuration|System Configuration]] 
 + 
 +Additionally it sends log messsages to UBUS and you can listen them with ''ubus subscribe log''
 + 
 + 
 +==== logread ==== 
 +This is the default interface to read log messages. It's provided by the [[https://github.com/openwrt/ubox|ubox]] package. 
 + 
 +It is local executable in ''/sbin/logread'' that will read the ring buffer records and display them chronologically. 
 + 
 +To show all log messages that contains a specific text (like a daemon name) and follow (like in ''tail -f'') use: 
 +<code bash> 
 +logread -fe firewall
 </code> </code>
  
 +Options:
 <code> <code>
-syslogd +-s <path> Path to ubus socket 
-    syslogd [OPTIONS] +-l <count> Got only the last 'count' messages 
-    System logging utility. Note that this version of syslogd ignores /etc/syslog.conf. +-e <pattern> Filter messages with a regexp 
-    Options: +-r <server> <port> Stream message to a server 
-            -n              Run in foreground +-F <file> Log file 
-            -O FILE         Log to given file (default:/var/log/messages) +-S <bytes> Log size 
-            -l n            Set local log level +-p <file> PID file 
-            -S              Smaller logging output +-h <hostname> Add hostname to the message 
-            -s SIZE         Max size (KB) before rotate (default:200KB, 0=off+-P <prefix> Prefix custom text to streamed messages 
-            -b NUM          Number of rotated logs to keep (default:1, max=99, 0=purge+-z <facility> handle only messages with given facility (0-23), repeatable 
-            -R HOST[:PORT]  Log to IP or hostname on PORT (default PORT=514/UDP) +-Z <facility> ignore messages with given facility (0-23), repeatable 
-            -L              Log locally and via network (default is network only if -R) +-f Follow log messages 
-            -D              Drop duplicates +-u Use UDP as the protocol 
-            -C[size(KiB)]   Log to shared mem buffer (read it using logread)+-t Add an extra timestamp 
 +-0 Use \0 instead of \n as trailer when using TCP
 </code> </code>
  
-The "shared mem buffer" or ringbuffer is not a file on a tmpfs partition but just data in RAMTo read it, you have to use ''logread''. +Please note that if you install the [[:docs:guide-user:perf_and_log:log.syslog-ng3|syslog-ng]] then the logread command will be overridden with it's own ''/usr/sbin/logread'' that has less options.
-you probably have syslogd running ''ps aux | grep syslog'':+
  
-''  381 root      1356 S    syslogd -C16'' 
  
-16KB is a busybox default value. To change it, set ''log_size'' option in ''/etc/config/system'' (remember that the number must be in KB, **not** bytes). The buffer size must be at least 4KB, otherwise ''syslogd'' fails to start. 
  
-Who logs? The syslogd acts as the server and any program can act as the client and send log messages to it. For example ''logger'' can be used to manually write messages to the system log. Some scripts in ''/etc/init.d/'' actually use this.+==== Local file logging ==== 
 +In order to log to a local file on the router, one needs to set the following options:
  
-Any program can act as the client and the syslogd acts as the serverCommunication is prone to the //syslog communications protocol//.+<code bash> 
 +config system  
 +... 
 +   option log_file '/var/log/mylog' 
 +   option log_remote '0' 
 +</code>
  
 +==== Network logging ====
 +In order to log remotely one needs to set the following options in ''/etc/config/system''
  
-==== Output ==== +<code bash> 
-Syslogd writes the log messages it receives into a file or into the RAM ringbuffer (option ''-C'')The file is a file, it can be accessed with ''cat'', ''less'', ''vi'', etcThe data in the RAM ringbuffer should be accessed with ''logread''You can of course use pipes, like ''logread -f | nc 192.168.1.1 514'' or ''logread -f >> /mnt/share/logfile'' (reasonable on non-flash media, see [[docs:guide-user:storage:usb-drives]] or [[docs:guide-user:services:nas:nfs.client]]) or pretty much whatever you want.+config system 
 +... 
 +   option log_ip <destination IP> 
 +   option log_port <destination port> 
 +   option log_proto <tcp or udp> 
 +</code>
  
-==== Aufbau einer Syslog-Meldung (max 1024 bytes==== +For the destination port, if you'll be manually reading the logs on the remote system as an unprivileged user (such as via the netcat command given below), then specify a high port (e.g. 5555). If you're sending to syslog server, use whatever port the syslog server is listening on (typically 514).
-=== The Header (?? Byte)=== +
-The header contains [[wp>timestamp]] and a hostname (max 64 Byteor an ip address.+
  
-The timestamp is set by the receiver of the log-message, the syslogd, not by the sender (for example ''logger'') and marks the //Empfangszeitpunkt//+Additionally, the firewall3 default is to ACCEPT all LAN traffic. If the router blocks LAN-side accessadd the following firewall3 rule to ''/etc/config/firewall'' to ACCEPT tcp/udp traffic from the router to the LAN-side.
-The hostname or the ip address belong to the sender of the message.+
  
-====== logd and logread ======+<code bash> 
 +config rule 
 +      option target 'ACCEPT' 
 +      option dest 'lan' 
 +      option proto 'tcp udp' 
 +      option dest_port '5555' 
 +      option name 'ACCEPT-LOG-DEVICE-LAN' 
 +</code>
  
-In Chaos Calmer 15.01, ''logd'' and ''logread'' replace syslogd.  The logd daemon runs and logread reads from it and redirects its output either to memory, over the network, or to a local file +and then reload the rules using ''service firewall restart''.
  
-If you want to test the logging outjust run command like +For the LAN-side station/clientthere are large number of mechanisms to listen for log messages. 
 +One of the simplest is ncat:
  
-''logger testLog "Blah1"''+<code bash> 
 +# TCP 
 +ncat -4 -l 5555
  
-And it should appear in memory (run ''logread'')over the network, or in your local file...if not check to see that ''logread'' and ''logd'' are running.+# Read UDP logs with ncat or python3 
 +ncat -u -4 -l 5555 
 +python3 -c "import socket 
 +s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 
 +s.bind(('0.0.0.0', 5141)) 
 +while True: 
 +   print(s.recvfrom(4096)[0].decode('utf-8'))" 
 +</code>
  
-Both of these (logd and logread) are started, stopped and restarted from /etc/init.d/log and are configured from /etc/config/systemfor example:+Log messages are in [[https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/|traditional syslog format (RFC 3164 5424)]], beginning with a priority number in angle brackets (e.g., <30>) and lacking a terminating newline. 
 +The above netcat method will therefore yield somewhat messy output. The python log reader above will most of the time get the line breaks into the right spots. A cleaner solution is to send messages to a remote machine's syslog daemon, in which case they will appear in the remote system's logs. 
 +See [[https://www.rsyslog.com/receiving-messages-from-a-remote-system/|Receiving Messages from a Remote System]] for server configuration instructions for rsyslog.
  
-==== File Logging =====+The advantage to using TCP is reliability - it logs every event. 
 +The disadvantage is it can cause some performance degradation on the router if the logging level is high. 
 +There is a section on iptable event logging which can cause a noticable latency in traffic throughput using TCP socket logging.
  
-Note that if you are file logging locally [[https://forum.openwrt.org/viewtopic.php?pid=340647#p340647|you may have to change order in which system is started]] to after fstab is started to write out the logs to the external drive.+===== Test runtime logging support ==== 
 +If you want to test the logging out, just run a command like 
  
-    config system  +<code>logger testLog "Blah1"</code>
-        option hostname 'OpenWrt'  +
-        option zonename 'America/New York'  +
-        option timezone 'EST5EDT,M3.2.0,M11.1.0'  +
-        option conloglevel '8'  +
-        option cronloglevel '8'  +
-        option log_file '/mnt/logging/messages' # On an fstab mounted external drive some where... +
-        option log_type 'file'  +
-        option log_size 100000  +
-        option log_buffer_size 2048  +
-        option log_remote '0'  # Need to turn this off to log locally.+
  
-==== Network Logging ====+and it should be written to the configured destination. 
 +If an event is not logged, check:
  
-  config system + * ''/sbin/logd'' is running; it should have an argument of ''-S <log_size>'' indicating the size of the ring buffer,   
-  (...) + * ''logd'' is configured correctly in ''/etc/config/system'', 
-      option log_ip <rsyslog IP>+ * restart it using ''service log restart'' and check for warnings/errors
  
-more log_* options https://openwrt.org/docs/guide-user/base-system/system_configuration+===== Logrotate ===== 
 +To automatically manage large collections of daily, weekly, or monthly logs, you may want to use [[packages:pkgdata:logrotate]]. 
 +Here's an example that rotates a persistent log on a USB storage each night keeping it for 1 week.
  
-general guide http://bredsaal.dk/debian-rsyslog-server-with-openwrt-rsyslog-client+<code bash> 
 +# Install packages 
 +opkg update 
 +opkg install logrotate 
 + 
 +# Configure logging 
 +uci set system.@system[0].log_file="/mnt/sda1/logs/system.log" 
 +uci set system.@system[0].log_remote="0" 
 +uci commit system 
 +service system restart  
 + 
 +# Configure logrotate 
 +cat << "EOF"/etc/logrotate.conf 
 +include /etc/logrotate.d 
 +/mnt/sda1/logs/system.log { 
 +    daily 
 +    rotate 1 
 +    missingok 
 +    notifempty 
 +    postrotate 
 +        service log restart 
 +        sleep 1 
 +        logger -p warn -s "Log rotation complete" 
 +    endscript 
 +
 +EOF 
 + 
 +# Configure cron 
 +cat << "EOF" >> /etc/crontabs/root 
 +58 23 * * * logrotate /etc/logrotate.conf 
 +EOF 
 +service cron restart 
 + 
 +# Debugging 
 +logrotate --verbose --debug /etc/logrotate.conf 
 +</code>
  
-note: I recommend adding this line in /etc/rsyslog.conf+===== Alternative implementations ===== 
 +The logging mechanism discussed here uses ''logd''. 
 +There are other packages that provide the same functionality:
  
-    # separate log files by host name of sending device +  * [[:docs:guide-user:perf_and_log:log.syslog-ng3|syslog-ng]] - is better supported in OpenWrt and used by default by some manufactures like Turris
-    # src http://www.rsyslog.com/article60/ +  [[:docs:guide-user:perf_and_log:log.rsyslog|rsyslog]]
-    $template DynaFile,"/var/log/system-%HOSTNAME%.log" +
-    *.* -?DynaFile+
  
-==== RAM Logging ==== 
-TODO 
  
  
  • Last modified: 2024/07/27 17:26
  • by stokito