Drupal's log system
To understand the limitations of the Drupal log and the improvements introduced by Monolog, it is first necessary to know what both systems consist of.
What does Drupal provide us with?
The most commonly used logging system that Drupal offers is dblog, a tool that logs events and errors that happen on the site, storing these logs directly in the system's database. Although dblog is useful for its accessibility and simplicity of integration, it can present significant performance issues when dealing with a high amount of traffic or frequent and massive logs.
On the other hand, there is syslog, a more robust alternative for log management. It allows system administrators to collect and store log messages on a dedicated server, thereby centralizing logs from multiple systems in a single location. However, it can be more complex to configure and is less accessible for users with limited system knowledge.
What if I don't want everything to go to the same place?
Drupal natively only allows you to choose one way to manage all logs. Sometimes, we need logs to be stored differently depending on their source. For this purpose, Monolog allows the management of each channel, enabling the processing of messages, selection of their format, and their destination.
In the context of this article, we want to keep dblog for critical errors to facilitate tracking certain logs, while setting a limit to prevent filling up the database.
Logs related to certain actions, informational messages, or external APIs should be separated into log files, as this information can be useful for troubleshooting. However, because it stores a lot of information and occurs frequently, it can easily saturate the database and obscure other important errors.
One Monolog to rule them all
Monolog is a widely used PHP library for log management. Through its modular approach, it allows developers to send log messages to a variety of destinations, such as files, databases, and monitoring services. In Drupal, the Monolog module integrates this powerful tool to extend and improve the native logging system, offering greater flexibility and options for managing system logs.
How to Use Monolog in Drupal
The module's readme provides excellent examples and insights into what can be achieved with it.
Some of the benefits of using it include:
- Configurable logging levels
- Multiple handlers, formatters, and processors
- The full power and flexibility of Monolog
The Monolog module doesn't have a graphical interface; all configuration is done through "*.services.yml" files. You must create a service file named monolog.services.yml and load it from the settings.php file:
$settings['container_yamls'][] = 'sites/default/monolog.services.yml';The Monolog module for Drupal will override Drupal's log management behaviour. To continue using Drupal's log channels (services with the 'logger' tag) such as dblog or others, it is necessary to add them to the module's configuration.
- For example, to use dblog on the default channel, you should have the following in the Monolog service file loaded in
settings.php:
parameters:
monolog.channel_handlers:
default:
handlers:
- name: 'drupal.dblog'
Handlers
In Monolog, a handler is the component responsible for determining what to do with each log entry. Essentially, handlers define how and where log messages are stored or sent. For example, a handler can write logs to a file, send them to an external service like Slack, or insert them into a database.
Each handler can be configured with different parameters, such as the minimum log level it will handle (for example, DEBUG, INFO, WARNING, ERROR, etc.) and other specific behaviours related to the log's destination. Monolog comes with a variety of predefined handlers to handle logs in different ways, and also allows developers to create custom handlers if their specific needs are not covered by the existing ones.
The flexibility of handlers allows Monolog to adapt to different environments and logging requirements, providing the ability to manage logs efficiently and effectively across a wide range of applications.
In the following example, we'll use the RotatingFileHandler. We'll see the simplest configuration that allows Monolog to log entries in a "rotating" file:
parameters:
monolog.channel_handlers:
default:
handlers:
- name: 'default_rotating_file'
services:
monolog.handler.default_rotating_file:
class: Monolog\Handler\RotatingFileHandler
arguments: ['private://logs/debug.log', 10, 'DEBUG']
The name property configured in the parameters/monolog.channel_handlers section is arbitrary but must match the name in the services/monolog.handler.[HANDLER_NAME] section.
This configuration will log each message with a log level greater than (or equal to) debug in a file named debug.log located in the logs folder in the private file system. The files will rotate daily, and the maximum number of files to keep will be 10.
The Monolog module automatically registers a handler for each enabled Drupal logger.
Example for sending errors from the cron log channel to Slack:
parameters:
monolog.channel_handlers:
cron:
handlers:
- name: 'cron_slack'
services:
monolog.handler.cron_slack:
class: Monolog\Handler\SlackHandler
arguments: ['slack-token', 'monolog', 'Drupal', true, null, 'ERROR']
Formatters
Monolog can alter the message format using formatters. A formatter needs to be registered as a service in the Drupal Service Container. The Monolog module provides a set of predefined formatters like the line formatter and the json formatter.
Example for sending all specific PHP logs to a separate file in JSON format:
parameters:
monolog.channel_handlers:
php:
handlers:
- name: 'php_rotating_file'
formatter: 'json'
services:
monolog.handler.php_rotating_file:
class: Monolog\Handler\RotatingFileHandler
arguments: ['private://logs/php.log', 10, 'DEBUG']
If no formatter is specified, the module will default to the line formatter. It's possible to send a log to multiple handlers, each with its formatter.
Processors
Monolog can alter the messages written in a logging setup using processors. The module provides a set of predefined processors to add information such as the current user, request URI, client IP, etc. Processors are defined as services under the namespace monolog.processor.
To edit the list of processors used, you need to override the monolog.processors parameter in monolog.services.yml and set the ones you need:
parameters:
monolog.processors:
- 'message_placeholder'
- 'current_user'
- 'request_uri'
- 'ip'
- 'referer'
- 'filter_backtrace'
Example configuration using both formatters and processors:
parameters:
monolog.channel_handlers:
php:
handlers:
- name: 'rotating_file'
formatter: 'json'
processors: ['current_user']
default:
handlers: ['drupal.dblog']
services:
monolog.handler.rotating_file:
class: Monolog\Handler\RotatingFileHandler
arguments: ['private://logs/debug.log', 10, 'DEBUG']
This will send the PHP logs to the RotatingFileHandler with a JSON formatter and only with the processor identified by the current_user.
Monolog Extra, Extending Monolog in Drupal
At Metadrop, we have developed the module Monolog Extra, extending the capabilities of the Monolog module and providing advanced configurations and custom handlers. It is used to address specific logging needs, such as file rotation.
It currently provides a processor to limit the number of characters in the log message called TruncateMessagesProcessor. It also includes a rotating type handler that prevents errors in log creation, creates the folder if it doesn't exist, and logs errors if any occur during the process.
Example configuration in monolog.services.yml:
parameters:
monolog.channel_handlers:
mymodule_file_log:
handlers:
- name: 'mymodule.safe_rotating_handler'
processors: ['mymodule.truncate_messages']
Example definition in a custom module, for example mymodule.services.yml:
services:
monolog.handler.mymodule.safe_rotating_handler:
class: Drupal\monolog_extra\Logger\Handler\SafeRotatingFileHandler
arguments: ['@file_system', '@logger.channel.mymodule_file_handler', 'private://mymodule/logs/mymodule.log', 14]
monolog.processor.mymodule.truncate_messages:
class: Drupal\monolog_extra\Logger\Processor\TruncateMessagesProcessor
arguments: [1000]
logger.channel.mymodule_file_handler:
class: Drupal\Core\Logger\LoggerChannel
factory: logger.factory:get
arguments: ['mymodule_file_handler']
logger.channel.mymodule_file_log:
class: Drupal\Core\Logger\LoggerChannel
factory: logger.factory:get
arguments: ['mymodule_file_log']
Based on the above example, a log with a file limit of 14 is defined, logs will be saved in the private://mymodule/logs/ directory. If there's an error while creating the log, it will be registered in the mymodule_file_handler channel, and the log message will be processed and limited to 1000 characters.
Problems that we have detected
At Metadrop, we have identified several errors and limitations when using Monolog.
Incompatibility with Stream Wrappers in Drupal
When defining main directories, such as private, public, or temporary ones, Drupal uses Stream Wrappers to represent paths more easily, without needing to use the actual server path, for example, "private://" or "public://".
Monolog uses the glob() function from the RotatingFileHandler to capture logs from a directory but glob() does not support Drupal's Stream Wrappers, causing errors in obtaining and managing these logs.
In the issue proposed by Metadrop Monolog glob does not support stream wrappers, it is proposed to extend the SafeRotatingFileHandler class from the Monolog Extra module and add the getGlobPattern method to convert Drupal's stream wrappers into actual paths before using them. This is achieved by using Drupal's file system function realpath.
The implementation of getGlobPattern is as follows:
protected function getGlobPattern(): string {
$pattern = parent::getGlobPattern();
if (preg_match('#^(\w+://)(.+)#', $pattern, $matches)) {
$stream_wrapper_prefix = $matches[1];
$rest_of_pattern = $matches[2];
$real_path = $this->fileSystem->realpath($stream_wrapper_prefix);
if ($real_path) {
return $real_path . DIRECTORY_SEPARATOR . $rest_of_pattern;
}
}
return $pattern;
}
The issue has only been resolved in the Monolog Extra module.
Reference: PHP StreamWrapper support for glob, Drupal Monolog Issue #3326496, PHP Monolog Issue.
Issue with Log File Rotation
Using the Monolog Extra module, old logs are not deleted because the logFileCanBeCreated method creates an empty file to check its writability, which collides with the rotation mechanism.
public function logFileCanBeCreated() {
$url = $this->getUrl();
@fopen($url, 'a');
set_error_handler([$this, 'customErrorHandler']);
$stream = fopen($url, 'a');
restore_error_handler();
if (is_resource($stream)) {
$can_be_created = TRUE;
}
else {
$can_be_created = FALSE;
$this->loggerChannel->critical($this->t('The stream or file :file could not be opened: :message', [
':file' => $url,
':message' => self::$errorMessage,
]));
}
return $can_be_created;
}
RotatingFileHandler::write method:
protected function write(LogRecord $record): void
{
// on the first record written, if the log is new, we rotate (once per day) after the log has been written so that the new file exists
if (null === $this->mustRotate) {
$this->mustRotate = null === $this->url || !file_exists($this->url);
}
...
Since the logFileCanBeCreated method creates the file but doesn't delete it, the write method does not set the mustRotate property because the condition that the file does not exist is not met. Consequently, the rotate method, where the number of files is checked and existing ones are deleted, is not called.
The RotatingFileHandler::rotate method is as follows:
protected function rotate(): void
{
....
// skip GC of old logs if files are unlimited
if (0 === $this->maxFiles) {
return;
}
$logFiles = glob($this->getGlobPattern());
if (false === $logFiles) {
// failed to glob
return;
}
if ($this->maxFiles >= \count($logFiles)) {
// no files to remove
return;
}
// Sorting the files by name to remove the older ones
usort($logFiles, function ($a, $b) {
return strcmp($b, $a);
});
foreach (\array_slice($logFiles, $this->maxFiles) as $file) {
if (is_writable($file)) {
// suppress errors here as unlink() might fail if two processes
// are cleaning up/rotating at the same time
set_error_handler(function (int $errno, string $errstr, string $errfile, int $errline): bool {
return false;
});
unlink($file);
restore_error_handler();
}
}
}
It is proposed to delete the file once its creation is verified, allowing Monolog to properly identify new files and execute rotation.
The implementation of logFileCanBeCreated after the correction is as follows:
public function logFileCanBeCreated() {
$url = $this->getUrl();
$can_be_created = FALSE;
set_error_handler([$this, 'customErrorHandler']);
try {
$file_exist_before = file_exists($url);
$stream = fopen($url, 'a');
if (is_resource($stream)) {
fclose($stream);
if (!$file_exist_before) {
unlink($url);
}
$can_be_created = TRUE;
}
} catch (\Exception $e) {
self::$errorMessage = $e->getMessage();
}
restore_error_handler();
if (!$can_be_created) {
$this->loggerChannel->critical($this->t('The stream or file :file could not be opened: :message', [
':file' => $url,
':message' => self::$errorMessage,
]));
}
return $can_be_created;
}
The issue has only been resolved in the Monolog Extra module.
Reference: The method logFileCanBeCreated avoids the rotate method
How to Read Logs
There are multiple tools for reading logs. The simplest method is displaying the output in real time, although interpreting and finding the information can be more challenging.
In most cases, logs are stored in GNU/Linux systems, so knowing how to read files from the console is essential.
The tail -f command allows you to display the latest lines of logs in real time.
tail -f private-files/my_module/logs/my_module_api-202*Using grep, you can search for a specific word. It also supports regular expressions, allows you to choose how many lines to show, and can be combined with the previous command using a pipe (the "|" character).
tail -f /var/log/auth.log | grep "session opened" There are also other tools like lnav or goaccess which allow filtering, sorting, and conducting more intensive searching and tracking of logs.
Other external systems can also be integrated like ELK (Elasticsearch / Logstash / Kibana) or Grafana Loki to set up a system monitor based on logs.
Conclusion
The integration of Monolog along with Monolog Extra provides Drupal with advanced and flexible logging capabilities. However, it's crucial to be aware of potential incompatibilities and issues that may appear, such as handling Stream Wrappers and log file rotation. With the solutions proposed, developers can optimize log management and ensure reliable and efficient performance. Getting the most out of these tools involves not only using them properly but also engaging with and learning from the active Drupal community.
