py.log documentation and musings


This document is an attempt to briefly state the actual specification of the py.log module. It was written by Francois Pinard and also contains some ideas for enhancing the py.log facilities.

NOTE that py.log is subject to refactorings, it may change with the next release.

This document is meant to trigger or facilitate discussions. It shamelessly steals from the Agile Testing comments, and from other sources as well, without really trying to sort them out.

Logging organisation

The py.log module aims a niche comparable to the one of the logging module found within the standard Python distributions, yet with much simpler paradigms for configuration and usage.

Holger Krekel, the main py library developer, introduced the idea of keyword-based logging and the idea of logging producers and consumers. A log producer is an object used by the application code to send messages to various log consumers. When you create a log producer, you define a set of keywords that are then used to both route the logging messages to consumers, and to prefix those messages.

In fact, each log producer has a few keywords associated with it for identification purposes. These keywords form a tuple of strings, and may be used to later retrieve a particular log producer.

A log producer may (or may not) be associated with a log consumer, meant to handle log messages in particular ways. The log consumers can be STDOUT, STDERR, log files, syslog, the Windows Event Log, user defined functions, etc. (Yet, logging to syslog or to the Windows Event Log is only future plans for now). A log producer has never more than one consumer at a given time, but it is possible to dynamically switch a producer to use another consumer. On the other hand, a single log consumer may be associated with many producers.

Note that creating and associating a producer and a consumer is done automatically when not otherwise overriden, so using py logging is quite comfortable even in the smallest programs. More typically, the application programmer will likely design a hierarchy of producers, and will select keywords appropriately for marking the hierarchy tree. If a node of the hierarchical tree of producers has to be divided in sub-trees, all producers in the sub-trees share, as a common prefix, the keywords of the node being divided. In other words, we go further down in the hierarchy of producers merely by adding keywords.

Using the py.log library

To use the py.log library, the user must import it into a Python application, create at least one log producer and one log consumer, have producers and consumers associated, and finally call the log producers as needed, giving them log messages.


Once the py library is installed on your system, a mere:

import py

holds enough magic for lazily importing the various facilities of the py library when they are first needed. This is really how py.log is made available to the application. For example, after the above import py, one may directly write py.log.Producer(...) and everything should work fine, the user does not have to worry about specifically importing more modules.

Creating a producer

There are three ways for creating a log producer instance:

  • As soon as py.log is first evaluated within an application program, a default log producer is created, and made available under the name py.log.default. The keyword default is associated with that producer.
  • The py.log.Producer() constructor may be explicitly called for creating a new instance of a log producer. That constructor accepts, as an argument, the keywords that should be associated with that producer. Keywords may be given either as a tuple of keyword strings, or as a single space-separated string of keywords.
  • Whenever an attribute is taken out of a log producer instance, for the first time that attribute is taken, a new log producer is created. The keywords associated with that new producer are those of the initial producer instance, to which is appended the name of the attribute being taken.

The last point is especially useful, as it allows using log producers without further declarations, merely creating them on-the-fly.

Creating a consumer

There are many ways for creating or denoting a log consumer:

  • A default consumer exists within the py.log facilities, which has the effect of writing log messages on the Python standard output stream. That consumer is associated at the very top of the producer hierarchy, and as such, is called whenever no other consumer is found.

  • The notation py.log.STDOUT accesses a log consumer which writes log messages on the Python standard output stream.

  • The notation py.log.STDERR accesses a log consumer which writes log messages on the Python standard error stream.

  • The py.log.File() constructor accepts, as argument, either a file already opened in write mode or any similar file-like object, and creates a log consumer able to write log messages onto that file.

  • The py.log.Path() constructor accepts a file name for its first argument, and creates a log consumer able to write log messages into that file. The constructor call accepts a few keyword parameters:

    • append, which is False by default, may be used for opening the file in append mode instead of write mode.
    • delayed_create, which is False by default, maybe be used for opening the file at the latest possible time. Consequently, the file will not be created if it did not exist, and no actual log message gets written to it.
    • buffering, which is 1 by default, is used when opening the file. Buffering can be turned off by specifying a 0 value. The buffer size may also be selected through this argument.
  • Any user defined function may be used for a log consumer. Such a function should accept a single argument, which is the message to write, and do whatever is deemed appropriate by the programmer. When the need arises, this may be an especially useful and flexible feature.

  • The special value None means no consumer at all. This acts just like if there was a consumer which would silently discard all log messages sent to it.

Associating producers and consumers

Each log producer may have at most one log consumer associated with it. A log producer gets associated with a log consumer through a py.log.setconsumer() call. That function accepts two arguments, the first identifying a producer (a tuple of keyword strings or a single space-separated string of keywords), the second specifying the precise consumer to use for that producer. Until this function is called for a producer, that producer does not have any explicit consumer associated with it.

Now, the hierarchy of log producers establishes which consumer gets used whenever a producer has no explicit consumer. When a log producer has no consumer explicitly associated with it, it dynamically and recursively inherits the consumer of its parent node, that is, that node being a bit closer to the root of the hierarchy. In other words, the rightmost keywords of that producer are dropped until another producer is found which has an explicit consumer. A nice side-effect is that, by explicitly associating a consumer with a producer, all consumer-less producers which appear under that producer, in the hierarchy tree, automatically inherits that consumer.

Writing log messages

All log producer instances are also functions, and this is by calling them that log messages are generated. Each call to a producer object produces the text for one log entry, which in turn, is sent to the log consumer for that producer.

The log entry displays, after a prefix identifying the log producer being used, all arguments given in the call, converted to strings and space-separated. (This is meant by design to be fairly similar to what the print statement does in Python). The prefix itself is made up of a colon-separated list of keywords associated with the producer, the whole being set within square brackets.

Note that the consumer is responsible for adding the newline at the end of the log entry. That final newline is not part of the text for the log entry.