--- url: 'https://metadrop.net/en/articles/improve-log-drupal-system-monolog' title: "How to extend and improve Drupal's log system using Monolog" author: 'Eduardo Morales' date: '2025-01-03T13:04:13+00:00' updated: '2025-01-23T15:57:10+00:00' type: article summary: "Find out why Drupal's log system can sometimes produce performance issues or be limited, and discover how Monolog can solve this problem." tags: - 'Drupal Planet' - 'Drupal modules' - Drupal published: true og: determiner: Automatic site_name: Metadrop 'image:alt': "How to extend and improve Drupal's log system using Monolog" street_address: 'Calle Manuel Luna, 12, 3 Dcha' locality: Madrid region: Madrid postal_code: '28020' country_name: España email: hola@metadrop.net phone_number: '910053180' schema: '@context': 'https://schema.org' '@graph': - '@type': Article '@id': 'https://metadrop.net/en/articles/improve-log-drupal-system-monolog#article' name: "How to extend and improve Drupal's log system using Monolog" headline: "How to extend and improve Drupal's log system using Monolog" description: "Find out why Drupal's log system can sometimes produce performance issues or be limited, and discover how Monolog can solve this problem." about: - 'Drupal modules' - Drupal datePublished: '2025-01-23T14:04:13+0100' dateModified: '2025-01-23T16:57:10+0100' author: '@type': Person name: 'Eduardo Morales' publisher: '@type': Organization '@id': 'https://metadrop.net/#organization' mainEntityOfPage: 'https://metadrop.net/en/articles/improve-log-drupal-system-monolog' - '@type': Organization '@id': 'https://metadrop.net/#organization' url: 'https://metadrop.net/' name: Metadrop sameAs: - 'https://www.drupal.org/metadrop' - 'https://twitter.com/metadrop' - 'https://asociaciondrupal.es/partner/metadrop' - 'https://www.linkedin.com/company/metadrop' logo: '@type': ImageObject url: 'https://metadrop.net/themes/custom/mdrop_radix/logo-metadrop-500-500.jpg' width: '500' height: '500' - '@type': ItemPage '@id': 'https://metadrop.net/en/articles/improve-log-drupal-system-monolog' breadcrumb: '@type': BreadcrumbList itemListElement: - '@type': ListItem position: 1 name: Home item: 'https://metadrop.net/en' - '@type': ListItem position: 2 name: 'Expert Drupal & Tech Articles' item: 'https://metadrop.net/en/articles' publisher: '@type': Organization '@id': 'https://metadrop.net/#organization' --- 1. [Articles](https://metadrop.net/en/articles) # How to extend and improve Drupal's log system using Monolog Thursday, January 23, 2025 Find out why Drupal's log system can sometimes produce performance issues or be limited, and discover how Monolog can solve this problem. ## Drupal's log system To understand the limitations of the Drupal log and the improvements introduced by ***Monolog***, it is first necessary to know what both systems consist of. ### What does Drupal provide us with? The most commonly used logging system that Drupal offers is *dblog*, a tool that logs events and errors that happen on the site, storing these logs directly in the system's database. Although *dblog* is useful for its accessibility and simplicity of integration, it can present significant performance issues when dealing with a high amount of traffic or frequent and massive logs. On the other hand, there is *syslog*, a more robust alternative for log management. It allows system administrators to collect and store log messages on a dedicated server, thereby centralizing logs from multiple systems in a single location. However, it can be more complex to configure and is less accessible for users with limited system knowledge. ### What if I don't want everything to go to the same place? Drupal natively only allows you to choose one way to manage all logs. Sometimes, we need logs to be stored differently depending on their source. For this purpose, ***Monolog*** allows the management of each channel, enabling the processing of messages, selection of their format, and their destination. In the context of this article, we want to keep *dblog* for critical errors to facilitate tracking certain logs, while setting a limit to prevent filling up the database. Logs related to certain actions, informational messages, or external APIs should be separated into log files, as this information can be useful for troubleshooting. However, because it stores a lot of information and occurs frequently, it can easily saturate the database and obscure other important errors. ## One Monolog to rule them all [Monolog](https://github.com/Seldaek/monolog "Monolog Library") is a widely used **PHP** library for log management. Through its modular approach, it allows developers to send log messages to a variety of destinations, such as files, databases, and monitoring services. In Drupal, [the Monolog module](https://drupal.org/project/monolog "Monolog") integrates this powerful tool to extend and improve the native logging system, offering greater flexibility and options for managing system logs. ### How to Use Monolog in Drupal The [module's readme](https://git.drupalcode.org/project/monolog/blob/HEAD/README.md "Readme") provides excellent examples and insights into what can be achieved with it. Some of the benefits of using it include: - Configurable logging levels - Multiple handlers, formatters, and processors - The full power and flexibility of Monolog The **Monolog** module doesn't have a graphical interface; all configuration is done through "\*.services.yml" files. You must create a service file named `monolog.services.yml` and load it from the `settings.php` file: ```php $settings['container_yamls'][] = 'sites/default/monolog.services.yml'; ``` The **Monolog** module for Drupal will override Drupal's log management behaviour. To continue using Drupal's log channels (services with the '*logger*' tag) such as *dblog* or others, it is necessary to add them to the module's configuration. - For example, to use *dblog* on the default channel, you should have the following in the Monolog service file loaded in `settings.php`: ```php parameters: monolog.channel_handlers: default: handlers: - name: 'drupal.dblog' ``` ### Handlers In **Monolog**, a *handler* is the component responsible for determining what to do with each log entry. Essentially, handlers define how and where log messages are stored or sent. For example, a handler can write logs to a file, send them to an external service like *Slack*, or insert them into a database. Each handler can be configured with different parameters, such as the minimum log level it will handle (for example, *DEBUG*, *INFO*, *WARNING*, *ERROR*, etc.) and other specific behaviours related to the log's destination. **Monolog** comes with a variety of predefined handlers to handle logs in different ways, and also allows developers to create custom handlers if their specific needs are not covered by the existing ones. The flexibility of handlers allows **Monolog** to adapt to different environments and logging requirements, providing the ability to manage logs efficiently and effectively across a wide range of applications. In the following example, we'll use the *RotatingFileHandler*. We'll see the simplest configuration that allows *Monolog* to log entries in a "rotating" file: ```yaml parameters: monolog.channel_handlers: default: handlers: - name: 'default_rotating_file' services: monolog.handler.default_rotating_file: class: Monolog\Handler\RotatingFileHandler arguments: ['private://logs/debug.log', 10, 'DEBUG'] ``` The *name* property configured in the `parameters/monolog.channel_handlers` section is arbitrary but must match the name in the `services/monolog.handler.[HANDLER_NAME]` section. This configuration will log each message with a log level greater than (or equal to) *debug* in a file named *debug.log* located in the **logs** folder in the private file system. The files will rotate daily, and the maximum number of files to keep will be **10**. The **Monolog** module automatically registers a handler for each enabled Drupal logger. Example for sending errors from the *cron* log channel to Slack: ```yaml parameters: monolog.channel_handlers: cron: handlers: - name: 'cron_slack' services: monolog.handler.cron_slack: class: Monolog\Handler\SlackHandler arguments: ['slack-token', 'monolog', 'Drupal', true, null, 'ERROR'] ``` ### Formatters **Monolog** can alter the message format using *formatters*. A *formatter* needs to be registered as a service in the [Drupal Service Container](https://www.drupal.org/docs/8/api/services-and-dependency-injection/services-and-dependency-injection-in-drupal-8). The **Monolog** module provides a set of predefined formatters like the *line formatter* and the *json formatter*. Example for sending all specific **PHP** logs to a separate file in **JSON** format: ```yaml parameters: monolog.channel_handlers: php: handlers: - name: 'php_rotating_file' formatter: 'json' services: monolog.handler.php_rotating_file: class: Monolog\Handler\RotatingFileHandler arguments: ['private://logs/php.log', 10, 'DEBUG'] ``` If no *formatter* is specified, the module will default to the *line formatter*. It's possible to send a log to multiple *handlers*, each with its *formatter*. ### Processors **Monolog** can alter the messages written in a logging setup using *processors*. The module provides a set of predefined processors to add information such as the current user, request URI, client IP, etc. Processors are defined as services under the namespace `monolog.processor.` To edit the list of processors used, you need to override the **monolog.processors** parameter in `monolog.services.yml` and set the ones you need: ```yaml parameters: monolog.processors: - 'message_placeholder' - 'current_user' - 'request_uri' - 'ip' - 'referer' - 'filter_backtrace' ``` Example configuration using both *formatters* and *processors*: ```yaml parameters: monolog.channel_handlers: php: handlers: - name: 'rotating_file' formatter: 'json' processors: ['current_user'] default: handlers: ['drupal.dblog'] services: monolog.handler.rotating_file: class: Monolog\Handler\RotatingFileHandler arguments: ['private://logs/debug.log', 10, 'DEBUG'] ``` This will send the PHP logs to the `RotatingFileHandler` with a JSON formatter and only with the processor identified by the *current\_user*. ### Monolog Extra, Extending Monolog in Drupal At Metadrop, we have developed the module [Monolog Extra](https://www.drupal.org/project/monolog_extra "Monolog Extra"), extending the capabilities of the [Monolog module](https://www.drupal.org/project/monolog "Monolog") and providing advanced configurations and custom *handlers*. It is used to address specific logging needs, such as file rotation. It currently provides a processor to limit the number of characters in the log message called `TruncateMessagesProcessor`. It also includes a rotating type handler that prevents errors in log creation, creates the folder if it doesn't exist, and logs errors if any occur during the process. Example configuration in `monolog.services.yml`: ```yaml parameters: monolog.channel_handlers: mymodule_file_log: handlers: - name: 'mymodule.safe_rotating_handler' processors: ['mymodule.truncate_messages'] ``` Example definition in a custom module, for example `mymodule.services.yml`: ```yaml services: monolog.handler.mymodule.safe_rotating_handler: class: Drupal\monolog_extra\Logger\Handler\SafeRotatingFileHandler arguments: ['@file_system', '@logger.channel.mymodule_file_handler', 'private://mymodule/logs/mymodule.log', 14] monolog.processor.mymodule.truncate_messages: class: Drupal\monolog_extra\Logger\Processor\TruncateMessagesProcessor arguments: [1000] logger.channel.mymodule_file_handler: class: Drupal\Core\Logger\LoggerChannel factory: logger.factory:get arguments: ['mymodule_file_handler'] logger.channel.mymodule_file_log: class: Drupal\Core\Logger\LoggerChannel factory: logger.factory:get arguments: ['mymodule_file_log'] ``` Based on the above example, a log with a file limit of **14** is defined, logs will be saved in the `private://mymodule/logs/` directory. If there's an error while creating the log, it will be registered in the `mymodule_file_handler` channel, and the log message will be processed and limited to **1000 characters**. ## Problems that we have detected At Metadrop, we have identified several errors and limitations when using **Monolog**. ### Incompatibility with Stream Wrappers in Drupal When defining main directories, such as private, public, or temporary ones, Drupal uses *Stream Wrappers* to represent paths more easily, without needing to use the actual server path, for example, "*private://*" or "*public://*". **Monolog** uses the `glob()` function from the `RotatingFileHandler` to capture logs from a directory but `glob()` does not support Drupal's *Stream Wrappers*, causing errors in obtaining and managing these logs. In the issue proposed by Metadrop [Monolog glob does not support stream wrappers](https://www.drupal.org/project/monolog_extra/issues/3497134 "Issue Monolog glob does not support stream wrappers"), it is proposed to extend the `SafeRotatingFileHandler` class from the [Monolog Extra](https://www.drupal.org/project/monolog_extra "Monolog Extra") module and add the `getGlobPattern` method to convert Drupal's stream wrappers into actual paths before using them. This is achieved by using Drupal's file system function `realpath`. The implementation of getGlobPattern is as follows: ```php protected function getGlobPattern(): string { $pattern = parent::getGlobPattern(); if (preg_match('#^(\w+://)(.+)#', $pattern, $matches)) { $stream_wrapper_prefix = $matches[1]; $rest_of_pattern = $matches[2]; $real_path = $this->fileSystem->realpath($stream_wrapper_prefix); if ($real_path) { return $real_path . DIRECTORY_SEPARATOR . $rest_of_pattern; } } return $pattern; } ``` *The issue has only been resolved in the **Monolog Extra** module.* Reference: [PHP StreamWrapper support for glob](https://github.com/php/php-src/issues/9224), [Drupal Monolog Issue #3326496](https://www.drupal.org/project/monolog/issues/3326496), [PHP Monolog Issue](https://github.com/Seldaek/monolog/issues/204). ### Issue with Log File Rotation Using the [Monolog Extra](https://www.drupal.org/project/monolog_extra "Monolog Extra") module, old logs are not deleted because the `logFileCanBeCreated` method creates an empty file to check its writability, which collides with the rotation mechanism. ```php public function logFileCanBeCreated() { $url = $this->getUrl(); @fopen($url, 'a'); set_error_handler([$this, 'customErrorHandler']); $stream = fopen($url, 'a'); restore_error_handler(); if (is_resource($stream)) { $can_be_created = TRUE; } else { $can_be_created = FALSE; $this->loggerChannel->critical($this->t('The stream or file :file could not be opened: :message', [ ':file' => $url, ':message' => self::$errorMessage, ])); } return $can_be_created; } ``` `RotatingFileHandler::write` method: ```php protected function write(LogRecord $record): void { // on the first record written, if the log is new, we rotate (once per day) after the log has been written so that the new file exists if (null === $this->mustRotate) { $this->mustRotate = null === $this->url || !file_exists($this->url); } ... ``` Since the `logFileCanBeCreated` method creates the file but doesn't delete it, the write method does not set the `mustRotate` property because the condition that the file does not exist is not met. Consequently, the `rotate` method, where the number of files is checked and existing ones are deleted, is not called. The `RotatingFileHandler::rotate` method is as follows: ```php protected function rotate(): void { .... // skip GC of old logs if files are unlimited if (0 === $this->maxFiles) { return; } $logFiles = glob($this->getGlobPattern()); if (false === $logFiles) { // failed to glob return; } if ($this->maxFiles >= \count($logFiles)) { // no files to remove return; } // Sorting the files by name to remove the older ones usort($logFiles, function ($a, $b) { return strcmp($b, $a); }); foreach (\array_slice($logFiles, $this->maxFiles) as $file) { if (is_writable($file)) { // suppress errors here as unlink() might fail if two processes // are cleaning up/rotating at the same time set_error_handler(function (int $errno, string $errstr, string $errfile, int $errline): bool { return false; }); unlink($file); restore_error_handler(); } } } ``` It is proposed to delete the file once its creation is verified, allowing **Monolog** to properly identify new files and execute rotation. The implementation of `logFileCanBeCreated` after the correction is as follows: ```php public function logFileCanBeCreated() { $url = $this->getUrl(); $can_be_created = FALSE; set_error_handler([$this, 'customErrorHandler']); try { $file_exist_before = file_exists($url); $stream = fopen($url, 'a'); if (is_resource($stream)) { fclose($stream); if (!$file_exist_before) { unlink($url); } $can_be_created = TRUE; } } catch (\Exception $e) { self::$errorMessage = $e->getMessage(); } restore_error_handler(); if (!$can_be_created) { $this->loggerChannel->critical($this->t('The stream or file :file could not be opened: :message', [ ':file' => $url, ':message' => self::$errorMessage, ])); } return $can_be_created; } ``` *The issue has only been resolved in the **Monolog Extra** module.* Reference: [The method logFileCanBeCreated avoids the rotate method](https://www.drupal.org/project/monolog_extra/issues/3497117 "The method logFileCanBeCreated avoids the rotate method") ## How to Read Logs There are multiple tools for reading logs. The simplest method is displaying the output in real time, although interpreting and finding the information can be more challenging. In most cases, logs are stored in **GNU/Linux** systems, so knowing how to read files from the console is essential. The `tail -f` command allows you to display the latest lines of logs in real time. ```bash tail -f private-files/my_module/logs/my_module_api-202* ``` Using `grep`, you can search for a specific word. It also supports regular expressions, allows you to choose how many lines to show, and can be combined with the previous command using a pipe (the "|" character). ```bash tail -f /var/log/auth.log | grep "session opened" ``` There are also other tools like *lnav* or *goaccess* which allow filtering, sorting, and conducting more intensive searching and tracking of logs. Other external systems can also be integrated like *ELK (Elasticsearch / Logstash / Kibana)* or *Grafana Loki* to set up a system monitor based on logs. ## Conclusion The integration of **Monolog** along with **Monolog Extra** provides Drupal with advanced and flexible logging capabilities. However, it's crucial to be aware of potential incompatibilities and issues that may appear, such as handling *Stream Wrappers* and log file rotation. With the solutions proposed, developers can optimize log management and ensure reliable and efficient performance. Getting the most out of these tools involves not only using them properly but also engaging with and learning from the active Drupal community. [Drupal modules](https://metadrop.net/en/articles?text=Drupal%20modules) [Drupal](https://metadrop.net/en/articles?text=Drupal) - Eduardo Morales Senior Drupal developer [Drupal maintenance & support services](https://metadrop.net/en/services/drupal/maintenance " See Drupal maintenance & support services") Independent, EU-based Drupal maintenance and support keeping mission-critical platforms secure, compliant and evolving — preventive, corrective and evolutive care under one governance model with monthly written reporting. See more