diff --git a/README.md b/README.md index 4fe8a4c..1cbddf4 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,184 @@ # Logs monitoring -Provides endpoint that indicate about errors in the log files. +Provides endpoints that indicate about errors in the log files, about cron that +stopped running and about Drush that stopped working from the CLI. ## Configuration In the form /admin/config/development/logs-monitoring-settings should be set: 1) paths to logs 2) a count of rows in the end where a search will be performed -3) a list of the words that indicate errors. +3) a list of the words that indicate errors +4) the cron max age, in seconds +5) the Drush heartbeat max age, in seconds. ### Usage + +#### Logs The module provide REST endpoint /admin/reports/logs-monitoring where a list of logs with their statuses will be provided. It can be used by Uptimerobot to detect errors. Returns HTTP status codes: - 200 - no errors found - 500 - some errors found + +#### Cron +The REST endpoint /admin/reports/cron-monitoring reports how long ago cron last +completed. Point a second Uptimerobot monitor at it. +Returns HTTP status codes: +- 200 - cron completed within the configured max age +- 503 - cron has not completed within the configured max age, or never ran + +Example response: + +```json +{ + "check": "cron", + "status": "ok", + "last_run": 1753600000, + "age_seconds": 312, + "threshold": 3600 +} +``` + +`status` is one of `ok`, `stale` or `never`. When it is `never` there is no +`system.cron_last` timestamp yet and `age_seconds` is `null`. + +Drupal writes `system.cron_last` only after all cron handlers and all queues +have finished, so a cron run that dies half way through (fatal error, PHP +timeout, killed worker) does not advance it. The endpoint therefore detects a +silently failing cron, not just a cron that was never triggered. + +#### Drush + +The REST endpoint /admin/reports/drush-monitoring reports how long ago Drush +last completed a health check from the CLI. Point a third Uptimerobot monitor +at it. + +Returns HTTP status codes: +- 200 - the heartbeat was recorded within the configured max age +- 503 - the heartbeat is older than the configured max age, or was never + recorded + +Example response: + +```json +{ + "check": "drush", + "status": "ok", + "last_run": 1753600000, + "age_seconds": 312, + "threshold": 3600 +} +``` + +`status` is one of `ok`, `stale` or `never`. When it is `never` no heartbeat has +been recorded yet and `age_seconds` is `null`. Users with the *administer site +configuration* permission also get a `meta` object holding the CLI PHP version +and SAPI, the Drush version, the hostname, the system user and how long the +check took - the values needed to work out why a heartbeat went stale. It is +withheld from everyone else so an anonymous uptime monitor is not handed the +host's internals. + +##### How it works + +A web request cannot test Drush. Shelling out from the web server would run +under the wrong PHP SAPI, the wrong user and the wrong environment, which is +exactly where CLI-only breakage hides. So the check is inverted: the CLI does +the work and the endpoint only reads the result. + +`drush logs-monitoring:heartbeat`, scheduled in the system crontab, bootstraps +Drupal, runs a real database query, runs an entity query and only then records a +timestamp in Drupal State. The write is last on purpose - any failure above it +leaves the previous timestamp in place, the endpoint goes stale and your monitor +alerts. Nothing has to push a failure anywhere, which is what makes this a dead +man's switch rather than a report that can itself go missing. + +Anything that stops scheduled Drush from completing is caught: a broken +`vendor/` directory, a CLI PHP upgrade that no longer matches the codebase, a +`settings.php` the cron user cannot read, database credentials that only work +over the web server's socket, a PHP memory limit that differs between SAPIs, or +a crontab entry that was removed during a deploy. + +Note what that last one implies: a stale heartbeat means *scheduled Drush +execution* is broken, which is not the same claim as "the Drush binary is +broken". Whoever gets paged should check the crontab first. A dead crontab will +also make the cron endpoint go stale, so expect both monitors to fire together. + +##### Setup + +Requires Drush 12 or newer, which is what discovers commands from +`src/Drush/Commands`. + +1. Verify the command runs at all: + + ```bash + cd /var/www/example.com + vendor/bin/drush logs-monitoring:heartbeat + ``` + + On a multisite, add `--uri=example.com`. + +2. Copy and edit the wrapper's config file: + + ```bash + cp scripts/drush-healthcheck.conf.example /etc/drush-healthcheck.conf + $EDITOR /etc/drush-healthcheck.conf + ``` + + Set `DRUPAL_ROOT`, `DRUSH_BIN` and `DRUSH_HC_SITES` explicitly. The script + can auto-detect them, but for anything scheduled you want the paths pinned: + a wrong guess would report a different site as healthy. + +3. Schedule it as the user that owns the codebase - not root, or it will leave + root-owned files in the Drupal file system and caches: + + ```cron + */15 * * * * /var/www/example.com/web/modules/contrib/logs_monitoring/scripts/drush-healthcheck.sh -q + ``` + + `-q` keeps cron silent on success and still mails you the stderr of a + failure. + +4. Set the *Drush heartbeat max age* on the settings form comfortably above the + crontab interval, so one missed run does not raise an alert. With the + 15-minute schedule above, 3600 seconds tolerates three consecutive failures + before alerting. + +5. Add the Uptimerobot monitor for /admin/reports/drush-monitoring. + +The wrapper is optional - the crontab line can call the Drush command directly. +What the wrapper adds is a lock so overlapping runs cannot pile up, a timeout so +a hung database cannot wedge cron, a loop over the sites of a multisite, a +`drush version` pre-flight that produces a readable error when the binary itself +is broken, and an optional outbound ping. + +Run `scripts/drush-healthcheck.sh --help` for all options. Exit codes: 0 +healthy, 1 a check failed, 2 bad usage or configuration, 3 another instance is +running. + +##### Optional: external dead man's switch + +Setting `DRUSH_HC_PING_URL` makes the script curl a URL after a site passes +every check, for use with Healthchecks.io, Cronitor or an Uptimerobot heartbeat +monitor. That takes Drupal out of the alert path entirely, so a broken web stack +cannot mask a Drush failure. It needs outbound network access from the server +and keeps the monitoring configuration outside the repository, so treat it as a +complement to the endpoint rather than a replacement. + +##### Inspecting and testing + +```bash +# What was last recorded. +drush state:get logs_monitoring.drush_last_ok --format=yaml + +# Simulate a dead Drush: delete the heartbeat, endpoint answers 503 "never". +drush state:delete logs_monitoring.drush_last_ok + +# Simulate a stale heartbeat: backdate it beyond the threshold. +drush state:set logs_monitoring.drush_last_ok 1 --input-format=integer + +# Run the checks without touching state, e.g. against a read-only replica. +drush logs-monitoring:heartbeat --no-state +``` + +A bare integer is accepted as a heartbeat, which is why the `state:set` line +above works for testing. diff --git a/config/install/logs_monitoring.settings.yml b/config/install/logs_monitoring.settings.yml index 5f0383b..be9fa81 100644 --- a/config/install/logs_monitoring.settings.yml +++ b/config/install/logs_monitoring.settings.yml @@ -1 +1,3 @@ log_configs: [] +cron_max_age: 3600 +drush_max_age: 3600 diff --git a/config/install/rest.resource.logs_monitoring_cron.yml b/config/install/rest.resource.logs_monitoring_cron.yml new file mode 100644 index 0000000..54d0247 --- /dev/null +++ b/config/install/rest.resource.logs_monitoring_cron.yml @@ -0,0 +1,17 @@ +langcode: en +status: true +dependencies: + module: + - serialization + - user + - logs_monitoring +id: logs_monitoring_cron +plugin_id: logs_monitoring_cron +granularity: resource +configuration: + methods: + - GET + formats: + - json + authentication: + - cookie diff --git a/config/install/rest.resource.logs_monitoring_drush.yml b/config/install/rest.resource.logs_monitoring_drush.yml new file mode 100644 index 0000000..dda21cb --- /dev/null +++ b/config/install/rest.resource.logs_monitoring_drush.yml @@ -0,0 +1,17 @@ +langcode: en +status: true +dependencies: + module: + - serialization + - user + - logs_monitoring +id: logs_monitoring_drush +plugin_id: logs_monitoring_drush +granularity: resource +configuration: + methods: + - GET + formats: + - json + authentication: + - cookie diff --git a/config/schema/logs_monitoring.schema.yml b/config/schema/logs_monitoring.schema.yml new file mode 100644 index 0000000..b5f5753 --- /dev/null +++ b/config/schema/logs_monitoring.schema.yml @@ -0,0 +1,29 @@ +logs_monitoring.settings: + type: config_object + label: 'Logs monitoring settings' + mapping: + log_configs: + type: sequence + label: 'Log configs' + sequence: + type: mapping + label: 'Log config' + mapping: + path_to_logs: + type: string + label: 'Path to the log' + search_words: + type: text + label: 'Error words to search' + exclude_words: + type: text + label: 'Words to exclude' + lines_count: + type: integer + label: 'Count of lines to read' + cron_max_age: + type: integer + label: 'Cron max age in seconds' + drush_max_age: + type: integer + label: 'Drush heartbeat max age in seconds' diff --git a/logs_monitoring.info.yml b/logs_monitoring.info.yml index 1d1fd9b..8d84652 100644 --- a/logs_monitoring.info.yml +++ b/logs_monitoring.info.yml @@ -2,7 +2,7 @@ name: Logs monitoring type: module description: Provides functionality for logs monitoring. core: 8.x -core_version_requirement: ^8 || ^9 || ^10 +core_version_requirement: ^8 || ^9 || ^10 || ^11 package: Development dependencies: - drupal:rest diff --git a/logs_monitoring.install b/logs_monitoring.install index c0e275d..3fc4a97 100644 --- a/logs_monitoring.install +++ b/logs_monitoring.install @@ -5,6 +5,9 @@ * Contains install and update routines. */ +use Drupal\Core\Serialization\Yaml; +use Drupal\logs_monitoring\Heartbeat; +use Drupal\logs_monitoring\Plugin\rest\resource\CronMonitoring; use Drupal\user\Entity\Role; use Drupal\user\RoleInterface; @@ -14,18 +17,99 @@ use Drupal\user\RoleInterface; function logs_monitoring_install($is_syncing) { // Add permissions to anonymous and authenticated roles. if (!$is_syncing) { - $permissions = [ + _logs_monitoring_grant_permissions([ 'restful get logs_monitoring', - ]; - $roles = Role::loadMultiple([ - RoleInterface::ANONYMOUS_ID, - RoleInterface::AUTHENTICATED_ID + 'restful get logs_monitoring_cron', + 'restful get logs_monitoring_drush', ]); - foreach ($roles as $role) { - foreach ($permissions as $permission) { - $role->grantPermission($permission); - } - $role->save(); + } +} + +/** + * Grants permissions to the anonymous and authenticated roles. + * + * @param array $permissions + * The permission names to grant. + */ +function _logs_monitoring_grant_permissions(array $permissions) { + $roles = Role::loadMultiple([ + RoleInterface::ANONYMOUS_ID, + RoleInterface::AUTHENTICATED_ID, + ]); + foreach ($roles as $role) { + foreach ($permissions as $permission) { + $role->grantPermission($permission); } + $role->save(); + } +} + +/** + * Creates a REST resource config entity from the module's install config. + * + * @param string $id + * The rest_resource_config entity ID, which is also the plugin ID. + */ +function _logs_monitoring_install_rest_resource(string $id) { + // The "restful get " permission is derived from the REST resource config + // entity, so both the plugin definition and the entity have to exist before + // the permission is granted - Role::calculateDependencies() silently strips + // permissions it cannot resolve. + \Drupal::service('plugin.manager.rest')->clearCachedDefinitions(); + + $storage = \Drupal::entityTypeManager()->getStorage('rest_resource_config'); + if (!$storage->load($id)) { + $module_path = \Drupal::service('extension.list.module')->getPath('logs_monitoring'); + $file = DRUPAL_ROOT . '/' . $module_path . '/config/install/rest.resource.' . $id . '.yml'; + $storage->create(Yaml::decode(file_get_contents($file)))->save(); + } + + _logs_monitoring_grant_permissions(['restful get ' . $id]); + + // Make sure the route of the new endpoint becomes available. + \Drupal::service('router.builder')->setRebuildNeeded(); +} + +/** + * Adds the cron monitoring endpoint and its staleness threshold. + */ +function logs_monitoring_update_10001() { + // The "restful get logs_monitoring_cron" permission is derived from the REST + // resource config entity, so both the plugin definition and the entity have + // to exist before the permission is granted - Role::calculateDependencies() + // silently strips permissions it cannot resolve. + \Drupal::service('plugin.manager.rest')->clearCachedDefinitions(); + + $storage = \Drupal::entityTypeManager()->getStorage('rest_resource_config'); + if (!$storage->load('logs_monitoring_cron')) { + $module_path = \Drupal::service('extension.list.module')->getPath('logs_monitoring'); + $file = DRUPAL_ROOT . '/' . $module_path . '/config/install/rest.resource.logs_monitoring_cron.yml'; + $storage->create(Yaml::decode(file_get_contents($file)))->save(); } + + $config = \Drupal::configFactory()->getEditable('logs_monitoring.settings'); + if (!$config->get('cron_max_age')) { + $config->set('cron_max_age', CronMonitoring::DEFAULT_CRON_MAX_AGE)->save(); + } + + _logs_monitoring_grant_permissions(['restful get logs_monitoring_cron']); + + // Make sure the route of the new endpoint becomes available. + \Drupal::service('router.builder')->setRebuildNeeded(); +} + +/** + * Adds the Drush monitoring endpoint and its staleness threshold. + */ +function logs_monitoring_update_10002() { + _logs_monitoring_install_rest_resource('logs_monitoring_drush'); + + $config = \Drupal::configFactory()->getEditable('logs_monitoring.settings'); + if (!$config->get('drush_max_age')) { + $config->set('drush_max_age', Heartbeat::DEFAULT_MAX_AGE)->save(); + } + + // The endpoint stays at 503 until the heartbeat is recorded for the first + // time, which cannot happen until the crontab entry exists. + return (string) t('Schedule "drush logs-monitoring:heartbeat" in the system crontab, otherwise /admin/reports/drush-monitoring will keep reporting "never". See the module README.'); } diff --git a/logs_monitoring.services.yml b/logs_monitoring.services.yml new file mode 100644 index 0000000..9d21f48 --- /dev/null +++ b/logs_monitoring.services.yml @@ -0,0 +1,8 @@ +services: + # Registered under its class name so it can be autowired, which is how Drush + # command classes resolve their constructor arguments. + Drupal\logs_monitoring\Heartbeat: + arguments: ['@state', '@datetime.time'] + logs_monitoring.heartbeat: + alias: Drupal\logs_monitoring\Heartbeat + public: true diff --git a/scripts/drush-healthcheck.conf.example b/scripts/drush-healthcheck.conf.example new file mode 100644 index 0000000..d6c04cd --- /dev/null +++ b/scripts/drush-healthcheck.conf.example @@ -0,0 +1,31 @@ +# Example configuration for drush-healthcheck.sh. +# +# Copy to one of the locations the script looks at (see --help) and edit: +# +# cp drush-healthcheck.conf.example /etc/drush-healthcheck.conf +# +# This file is sourced by bash, so it is shell syntax, not ini. +# +# Setting these explicitly is strongly recommended for scheduled runs: the +# script's auto-detection is a convenience for interactive use and can only +# guess. A wrong guess would report a different site as healthy. + +# Composer root of the project (the directory containing vendor/). +DRUPAL_ROOT="/var/www/example.com" + +# Drush binary. Usually $DRUPAL_ROOT/vendor/bin/drush. +DRUSH_BIN="/var/www/example.com/vendor/bin/drush" + +# Sites to check, comma or space separated. Use "-" for a single-site install +# (no --uri is passed). For a multisite, list the URIs Drush expects: +# DRUSH_HC_SITES="example.com, fr.example.com, de.example.com" +DRUSH_HC_SITES="-" + +# Per-command timeout in seconds, so a hung database cannot wedge cron. +DRUSH_HC_TIMEOUT=120 + +# Optional external dead man's switch (Healthchecks.io, Cronitor, an +# UptimeRobot heartbeat monitor). Curled only after a site passes every check. +# "{site}" is replaced with the site URI, which is useful on a multisite where +# each site has its own monitor. +# DRUSH_HC_PING_URL="https://hc-ping.com/your-uuid-here" diff --git a/scripts/drush-healthcheck.sh b/scripts/drush-healthcheck.sh new file mode 100755 index 0000000..6bfa490 --- /dev/null +++ b/scripts/drush-healthcheck.sh @@ -0,0 +1,332 @@ +#!/usr/bin/env bash +# +# drush-healthcheck.sh - proves scheduled Drush execution works on this site. +# +# Runs "drush logs-monitoring:heartbeat" for every site. That command performs +# the actual checks (bootstrap, database query, entity query) and records a +# timestamp in Drupal State; this script only handles the things bash is good +# at: locking, timeouts, iterating over a multisite and reporting exit codes. +# +# The heartbeat write is the proof. If Drush is broken the timestamp never +# updates, /admin/reports/drush-monitoring starts answering 503 and your uptime +# monitor alerts. Nothing needs to push a failure anywhere. +# +# Exit codes: +# 0 all checks passed +# 1 a check failed (see stderr) +# 2 bad usage / configuration +# 3 another instance is already running +# +# Usage: +# drush-healthcheck.sh [options] +# +# Options: +# -r, --root PATH Drupal project root (composer root). Default: auto-detect. +# -d, --drush PATH Drush binary. Default: auto-detect. +# -s, --sites LIST Comma/space separated site URIs. Default: "-". +# "-" means a single-site run with no --uri. +# -p, --ping-url URL Dead-man's-switch URL, curled only on success. +# "{site}" in the URL is replaced by the site URI. +# -t, --timeout SECONDS Per-command timeout. Default: 120 +# -c, --config FILE Config file to source. Default: auto-detect. +# --no-state Run the checks without recording the heartbeat. +# -q, --quiet Only output on failure. +# -h, --help Show this help. +# +# Environment variables: +# DRUPAL_ROOT, DRUSH_BIN, DRUSH_HC_SITES, DRUSH_HC_PING_URL, DRUSH_HC_TIMEOUT +# +# Config file (first found wins, unless -c is given): +# ./drush-healthcheck.conf +#