Feat: Sentry rate limiting

documentation/changelog.rst

@@ -58,6 +58,7 @@ New features
 
 Infrastructure / Support
 ----------------------
+* Add a ``FLEXMEASURES_SENTRY_DAILY_RATE_LIMIT`` setting for spreading a host's Sentry error allowance across the month with a fail-open daily Redis counter, and send the startup error about the database schema not being at the Alembic head revision to Sentry at most once per UTC calendar day per pair of current and expected revisions (it is still logged in full on every start) [see `PR #2366 <https://www.github.com/FlexMeasures/flexmeasures/pull/2366>`_]
 * Shrink the scheduler's mixed-integer program for one-way devices: where a device can only consume or only produce, its power-sign binaries and their big-M constraints are dropped, as simultaneous consumption and production is already ruled out by the power bounds [see `PR #2412 <https://www.github.com/FlexMeasures/flexmeasures/pull/2412>`_]
 * The database migration for this release splits each stored flex-context's ``inflexible-device-sensors`` field into ``inflexible-consumption``/``inflexible-production`` sensor references, classifying each sensor by its ``consumption_is_positive`` attribute (behavior-preserving; sensor attributes themselves are kept). Downgrading merges them back into bare sensor IDs, dropping any source filters added in the meantime [see `PR #2358 <https://www.github.com/FlexMeasures/flexmeasures/pull/2358>`_]
 * Speed up listing assets: eager-load each asset's sensors instead of lazy-loading them one query per asset during serialization, and skip loading sensors entirely for field-filtered responses that do not include them [see `PR #2363 <https://www.github.com/FlexMeasures/flexmeasures/pull/2363>`_]

documentation/configuration.rst

@@ -819,6 +819,25 @@ so without this filter, 404 errors can inflate Sentry error budgets unnecessaril
 Default: ``True``
 
 
+FLEXMEASURES_SENTRY_DAILY_RATE_LIMIT
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Set a positive integer to limit the number of error events sent to Sentry per
+UTC calendar day. The event count is shared between FlexMeasures processes
+through Redis. If Redis is unavailable, events are sent without rate limiting.
+
+.. note::
+   This limit is applied before Sentry's error sampling. If ``sample_rate`` or
+   ``error_sampler`` is configured through ``FLEXMEASURES_SENTRY_CONFIG``,
+   events that are later sampled out still count towards the limit. The number
+   of events actually sent to Sentry may therefore be lower than the configured
+   limit.
+
+Default: ``None`` (no rate limit)
+
+.. note:: This setting is also recognized as environment variable.
+
+
 FLEXMEASURES_TASK_CHECK_AUTH_TOKEN
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

documentation/host/error-monitoring.rst

@@ -5,7 +5,7 @@ Error monitoring
 
 When you run a FlexMeasures server, you want to stay on top of things going wrong. We added two ways of doing that:
 
-- You can connect to Sentry, so that all errors will be sent to your Sentry account. Add the token you got from Sentry in the config setting :ref:`sentry_access_token` and you're up and running! 
+- You can connect FlexMeasures to Sentry, an error-monitoring service that collects and groups application errors, by setting :ref:`sentry_access_token`. To keep your Sentry plan from maxing out too fast, FlexMeasures filters 404 (Not Found) events by default, can apply a shared daily event limit, and reports repeated database-schema mismatches only once per UTC day; see :ref:`configuration` for details and configurable settings.
 - Another source of crucial errors are things that did not even happen! For instance, a (bot) user who is supposed to send data regularly, fails to connect with FlexMeasures. Or, a task to import prices from a day-ahead market, which you depend on later for scheduling, fails silently.
 
 

flexmeasures/data/__init__.py

@@ -15,6 +15,7 @@
 
 from flexmeasures.data.config import configure_db_for, db
 from flexmeasures.data.transactional import after_request_exception_rollback_session
+from flexmeasures.utils.sentry_utils import SENTRY_DEDUPLICATION_KEY_ATTRIBUTE
 
 ma: Marshmallow = Marshmallow()
 
@@ -25,6 +26,16 @@ def _is_running_db_upgrade_command() -> bool:
     return any(args[i : i + 2] == ["db", "upgrade"] for i in range(len(args) - 1))
 
 
+def _schema_mismatch_deduplication_key(revision_status) -> str:
+    """Identify a schema mismatch to Sentry by the revisions involved.
+
+    Mismatches against other revisions are a new problem, so they get their own key and are reported right away.
+    """
+    current_heads = ",".join(revision_status.current_heads) or "unknown"
+    expected_heads = ",".join(revision_status.expected_heads) or "unknown"
+    return f"database-schema-mismatch:{current_heads}:{expected_heads}"
+
+
 def _add_vacuum_option_to_db_upgrade(app: Flask):
     """Extend `flexmeasures db upgrade` to vacuum-analyze the database afterwards.
 
@@ -85,10 +96,17 @@ def register_at(app: Flask):
                     f"Details: {revision_status.inspection_error}"
                 )
             else:
+                # Every process logs this while starting up, and hosts run FlexMeasures CLI commands often,
+                # so we ask Sentry to report it only once a day per pair of revisions.
                 app.logger.error(
                     "Database schema is not at the Alembic head revision "
                     f"({format_database_schema_revision_status(revision_status)}). "
-                    "Run `flexmeasures db upgrade` before starting the app."
+                    "Run `flexmeasures db upgrade` before starting the app.",
+                    extra={
+                        SENTRY_DEDUPLICATION_KEY_ATTRIBUTE: _schema_mismatch_deduplication_key(
+                            revision_status
+                        )
+                    },
                 )
 
     global ma

flexmeasures/data/tests/test_utils.py

@@ -1,12 +1,19 @@
 from __future__ import annotations
 
+import logging
+
 from sqlalchemy.exc import OperationalError, ProgrammingError
 
-from flexmeasures.data import db
+from flexmeasures.data import db, register_at
 from flexmeasures.data.utils import (
+    DatabaseSchemaRevisionStatus,
     format_database_schema_revision_status,
     get_database_schema_revision_status,
 )
+from flexmeasures.utils.sentry_utils import (
+    SENTRY_DEDUPLICATION_KEY_ATTRIBUTE,
+    _make_sentry_daily_deduplicator,
+)
 
 
 class _DummyConnection:
@@ -33,6 +40,54 @@ def get_heads(self) -> tuple[str, ...]:
         return self._heads
 
 
+def test_schema_mismatch_log_record_is_deduplicated(
+    app, clean_redis, monkeypatch, caplog
+):
+    """The schema check and Sentry filter share the LogRecord marker contract."""
+    revision_status = DatabaseSchemaRevisionStatus(
+        current_heads=("current-a",), expected_heads=("head-a",)
+    )
+    monkeypatch.setattr("flexmeasures.data.configure_db_for", lambda app: None)
+    monkeypatch.setattr("flexmeasures.data.Migrate", lambda *args, **kwargs: None)
+    monkeypatch.setattr(
+        "flexmeasures.data._add_vacuum_option_to_db_upgrade", lambda app: None
+    )
+    monkeypatch.setattr(
+        "flexmeasures.data._is_running_db_upgrade_command", lambda: False
+    )
+    monkeypatch.setattr("flexmeasures.data.ma.init_app", lambda app: None)
+    monkeypatch.setattr(app, "teardown_request", lambda function: None)
+    monkeypatch.setattr(app, "testing", False)
+    monkeypatch.setitem(app.config, "FLEXMEASURES_ENV", "production")
+    monkeypatch.setattr(
+        "flexmeasures.data.utils.get_database_schema_revision_status",
+        lambda app: revision_status,
+    )
+
+    with caplog.at_level(logging.ERROR):
+        register_at(app)
+
+    schema_mismatch_record = next(
+        record
+        for record in caplog.records
+        if record.message.startswith(
+            "Database schema is not at the Alembic head revision"
+        )
+    )
+    assert (
+        getattr(schema_mismatch_record, SENTRY_DEDUPLICATION_KEY_ATTRIBUTE)
+        == "database-schema-mismatch:current-a:head-a"
+    )
+
+    deduplicate = _make_sentry_daily_deduplicator(
+        app, redis_connection=app.redis_connection
+    )
+    event = {"message": schema_mismatch_record.getMessage()}
+    hint = {"log_record": schema_mismatch_record}
+    assert deduplicate(event, hint) is event
+    assert deduplicate(event, hint) is None
+
+
 def test_database_schema_is_migrated_to_head_when_revisions_match(app, monkeypatch):
     monkeypatch.setattr(db.engine, "connect", lambda: _DummyConnection())
     monkeypatch.setattr(

flexmeasures/utils/app_utils.py

@@ -11,10 +11,15 @@
 import sentry_sdk
 from sentry_sdk.integrations.flask import FlaskIntegration
 from sentry_sdk.integrations.rq import RqIntegration
-from werkzeug.exceptions import NotFound
 
 from flexmeasures import __version__ as fm_version
 from flexmeasures.app import create as create_app
+from flexmeasures.utils.sentry_utils import (
+    _make_sentry_daily_deduplicator,
+    _make_sentry_daily_rate_limiter,
+    _make_sentry_redis_connection,
+    _sentry_filter_notfound,
+)
 
 
 def provision_default_template_assets_on_startup(app: Flask) -> None:
@@ -57,40 +62,6 @@ def flexmeasures_cli():
     pass
 
 
-# For the Sentry integration, a crucial task is to filter out noise before it reaches Sentry.
-# Limiting what gets sent to Sentry (by 95%) keeps your costs to what you are interested in.
-# We want to filter out 404s (also those who in addition use untrusted-host request headers),
-# which are common probes in the wild.
-# Note: errors may reach Sentry twice - as raised Exception plus if FlexMeasures logs the error (e.g. during handling it)
-#       With verbose=False, Sentry might only see the  logging event, not an Exception, as it is only visible in the LogRecord message rather than in hint["exc_info"].
-
-
-def _sentry_filter_notfound(event, hint):
-    """Filter out noisy handled web errors to avoid inflating Sentry error budgets."""
-    if "exc_info" in hint:
-        _exc_type, exc_value, _tb = hint["exc_info"]
-        if isinstance(exc_value, NotFound):
-            return None
-    # FlexMeasures logs handled 404s with verbose=False to keep automated
-    # scans for hackable URLs from overwhelming log files. Sentry receives
-    # those as logging events, so the NotFound exception is only visible in
-    # the LogRecord message rather than in hint["exc_info"].
-    # We also filter out handled SecurityErrors that are logged when untrusted-host
-    # request headers are used.
-    log_record = hint.get("log_record")
-    if log_record is not None:
-        message = log_record.getMessage()
-        if message.startswith("NotFound - URL was: "):
-            return None
-        if (
-            message.startswith("SecurityError - URL was: ")
-            and " - \"Host '" in message
-            and message.endswith("' is not trusted.\"")
-        ):
-            return None
-    return event
-
-
 def init_sentry(app: Flask):
     """
     Configure Sentry.
@@ -105,11 +76,38 @@ def init_sentry(app: Flask):
         return
     app.logger.info("[FLEXMEASURES] Initialising Sentry ...")
 
-    before_send = (
-        _sentry_filter_notfound
-        if app.config.get("FLEXMEASURES_DO_NOT_SEND_NOTFOUND_TO_SENTRY")
-        else None
-    )
+    # The filters run in order, so events dropped by an earlier filter do not count towards the daily rate limit.
+    filters = []
+    if app.config.get("FLEXMEASURES_DO_NOT_SEND_NOTFOUND_TO_SENTRY"):
+        filters.append(_sentry_filter_notfound)
+
+    redis_connection = _make_sentry_redis_connection(app)
+    filters.append(_make_sentry_daily_deduplicator(app, redis_connection))
+
+    daily_rate_limit = app.config.get("FLEXMEASURES_SENTRY_DAILY_RATE_LIMIT")
+    if daily_rate_limit is not None:
+        if (
+            isinstance(
+                daily_rate_limit, bool
+            )  # reject True/False explicitly (instances of int)
+            or not isinstance(daily_rate_limit, int)
+            or daily_rate_limit <= 0
+        ):
+            app.logger.warning(
+                "FLEXMEASURES_SENTRY_DAILY_RATE_LIMIT must be a positive integer "
+                "or None. Sentry events will be sent without rate limiting."
+            )
+        else:
+            filters.append(
+                _make_sentry_daily_rate_limiter(app, daily_rate_limit, redis_connection)
+            )
+
+    def before_send(event, hint):
+        for event_filter in filters:
+            event = event_filter(event, hint)
+            if event is None:
+                break
+        return event
 
     sentry_sdk.init(
         dsn=sentry_dsn,
@@ -118,7 +116,7 @@ def init_sentry(app: Flask):
         release=f"flexmeasures@{fm_version}",
         send_default_pii=True,  # user data (current user id, email address, username) is attached to the event.
         environment=app.config.get("FLEXMEASURES_ENV"),
-        before_send=before_send,
+        before_send=before_send if filters else None,
         **app.config["FLEXMEASURES_SENTRY_CONFIG"],
     )
     sentry_sdk.set_tag("mode", app.config.get("FLEXMEASURES_MODE"))

flexmeasures/utils/config_defaults.py

@@ -118,6 +118,7 @@ class Config(object):
     # you probably want to adjust this.
     FLEXMEASURES_SENTRY_CONFIG: dict = dict(traces_sample_rate=0.33)
     FLEXMEASURES_DO_NOT_SEND_NOTFOUND_TO_SENTRY: bool = True
+    FLEXMEASURES_SENTRY_DAILY_RATE_LIMIT: int | None = None
     FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS: list[str] = (
         []
     )  # Deprecated. Use FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS instead.

flexmeasures/utils/config_utils.py

@@ -245,6 +245,7 @@ def read_env_vars(app: Flask):
             "LOGGING_LEVEL",
             "MAPBOX_ACCESS_TOKEN",
             "SENTRY_DSN",
+            "FLEXMEASURES_SENTRY_DAILY_RATE_LIMIT",
             "FLEXMEASURES_PLUGINS",
             "FLEXMEASURES_JSON_COMPACT",
             "SECURITY_TWO_FACTOR",
@@ -256,6 +257,11 @@ def read_env_vars(app: Flask):
             continue
         if isinstance(getattr(DefaultConfig, var, None), bool):
             value = parse_bool_env(value)
+        elif var == "FLEXMEASURES_SENTRY_DAILY_RATE_LIMIT":
+            try:
+                value = int(value)
+            except ValueError:
+                pass
         app.config[var] = value
 
     # Sentry is initialized from SENTRY_DSN, but we long documented the typo SENTRY_SDN,