Name

cyclog — safely log standard input to status files and log directories

Synopsis

cyclog [--max-file-size max-file-size] [--max-total-size max-total-size] [--margin margin] {directories...}

Description

cyclog is a simple logging utility that logs everything that it receives on its standard input to a set of log file directories named as its command arguments. It automatically timestamps logs, produces time-sortable log file names, rotates log files, and caps log directory sizes.

At startup, cyclog attempts to open all of the directories. If they are directories (or symbolic links to the same), it treats them as log directories. Otherwise, if they don't exist or are other types of file such as regular files, device files, or FIFOs, it aborts startup without reading from standard input.

It keeps a file descriptor open to each log directory for its lifetime, and accesses files relative to that open file descriptor. Log directories can thus be renamed as it is running, and it will continue to use the original directory, wherever it is renamed to.

Log directory

A log directory is compatible with the log directories employed by daemontools' and daemontools-encore's multilog(1), s6's s6-log(1), and runit's svlog(1). It contains:

  • a lock file (compatible with setlock(1)) that prevents two logging processes from sharing an active log directory;

  • a set of old log files, named as @ followed by a TAI64N timestamp in external form (16 hexadecimal digits of seconds and 8 hexadecimal digits of nanoseconds) and either .s (safely written to disc) or .u (aborted whilst writing to disc and possibly incomplete); and

  • a current file where incoming log data are currently being appended.

An active current file, or an old log file or a current file that was not written to disc by cyclog properly shutting down, has permissions 0644. A current file or an old log file where cyclog properly shut down has permissions 0744. (See the Compatibility section.)

cyclog shuts down when either it sees EOF on its standard input, or it receives one of the "good" termination signals (SIGTERM, SIGPIPE, SIGINT, or SIGHUP). At shut down, cyclog writes all pending data that it has already read from standard input, flushes current to disc, and changes current's permissions. At start up, it changes current's permissions and carries on.

Reliable piped input

In the normal case, the standard input of cyclog will be a pipe; and both ends of the the pipe will be open in a long-lived supervisor process such as service-manager(1). cyclog, as aforementioned, ensures that it writes all data read from its standard input, even if it is terminated by a (good) signal.

Thus no log data are lost, even if cyclog is shut down and restarted.

Automatic log rotation

cyclog performs automatic log rotation when it writes a linefeed within margin bytes of the maximum size (max-file-size) of the current file, when it is about to exceed that size, or when it receives a SIGALRM. It flushes current to disc, changes its permissions and renames it to a timestamped name, and then creates a new current file. The TAI64N timestamp of an old log file is the timestamp of when cyclog rotated current to that file.

At log rotation, and also at startup, it checks to ensure that the total size of all log files in the directory does not exceed the maximum total size (max-total-size). (It only totals the sizes of current and old log files. Other files, not managed or created by cyclog, are ignored.) If the total exceeds that maximum, it deletes each old log file with the numerically lowest name until either the total is less than the maximum or there is only the current file left.

Thus the maximum size of all log files at any time is max-total-size (the total size after the last rotation) plus max-file-size (the data written since that rotation). The default max-file-size is 16MiB and the default max-total-size is 1GiB.

cyclog totals file sizes rather than space allocated on disc. The amount of space allocated to all files may be, depending from the filesystem type and the maxima chosen, higher or lower than the space usage calculated by cyclog.

Timestamps

cyclog writes a timestamp at the beginning of every line written to current, which is the time when it processes the beginning of a line. This is slightly after it has read the beginning of that line. The timestamp is in TAI64N external form (16 hexadecimal digits of seconds and 8 hexadecimal digits of nanoseconds), which can be converted to human-readable form using tai64nlocal(1).

cyclog uses the CLOCK_REALTIME clock of the clock_gettime(2) system call. On many systems this has nanosecond resolution.

cyclog attempts to generate real TAI timestamps. On a Linux system where it detects an Olson "right" timezone currently in use, it knows that the system clock is TAI seconds since the Epoch and performs a simple conversion. On other Linux systems, and on BSDs, it assumes that the system clock is UTC seconds since the Epoch and attempts to correct for (known) UTC leap seconds in order to determine TAI.

Compatibility

A cyclog(1) utility was part of daemontools version 0.53. This utility is a workalike, with some minor additions to automatic log rotation and changes to log directory semantics.

daemontools later replaced cyclog(1) with multilog(1), which treats its command arguments as a script and has the abilities to do pattern matching and run external commands. cyclog is intended for the commonest, simple, use cases of logging services where there is just one plain logging directory with no bells and whistles. As such, it does not implement a script syntax, does not have pattern matching, and does not invoke other programs at all.

The version 0.53 cyclog(1) used permissions 0644 and 0444 to flag files that had not been written to disc. cyclog adopts the convention employed by the later multilog(1), and thence s6-log(1) and svlog(1). This convention changed to using the owner execute permission bit rather than the owner write permission bit as the flag. cyclog does not make use of the execute permission bit itself, but it sets and clears it as appropriate for interoperability with those tools. multilog(1) altered the convention in order to cope with scenarios where it would end up trying to open a read-only file for writing.

The version 0.53 cyclog(1) used the TAI64 timestamp of when a log file was started as its timestamp. Again cyclog adopts the convention employed by later tools.

daemontools' and daemontools-encore's multilog(1) uses a timer with only microsecond resolution, and for the same events will thus produce different timestamps to cyclog.

See the timestamps section of the Nosh Guide for detailed information on the differences from daemontools, daemontools-encore, and other toolsets in how cyclog handles TAI.

Author

Jonathan de Boyne Pollard