Relax soc-targets along with the other SoC constraints

documentation/api/introduction.rst

@@ -124,8 +124,8 @@ The response body will contain a JSON message with a ``status`` field set to ``"
 .. note::
 
     FlexMeasures' built-in storage scheduler no longer computes a fallback schedule for infeasible problems.
-    Instead, ``soc-minima`` and ``soc-maxima`` are relaxed by default (setting ``relax-soc-constraints`` or ``relax-constraints`` to ``false`` keeps them hard, with an explicitly set ``relax-soc-constraints`` taking precedence).
-    The hard constraints that remain even after constraint relaxation are ``soc-min``, ``soc-max``, ``soc-targets`` and ``power-capacity`` in the ``flex-model``, and ``site-power-capacity`` in the ``flex-context``.
+    Instead, ``soc-minima``, ``soc-maxima`` and ``soc-targets`` are relaxed by default (setting ``relax-soc-constraints`` or ``relax-constraints`` to ``false`` keeps them hard, with an explicitly set ``relax-soc-constraints`` taking precedence).
+    The hard constraints that remain even after constraint relaxation are ``soc-min``, ``soc-max`` and ``power-capacity`` in the ``flex-model``, and ``site-power-capacity`` in the ``flex-context``.
     The device ``consumption-capacity`` and ``production-capacity`` are not covered by ``relax-constraints`` at all; they stay hard unless you relax them by name, either by setting ``relax-capacity-constraints`` or by setting ``consumption-breach-price`` or ``production-breach-price`` yourself.
     If hard constraints cannot be satisfied, the scheduling job fails and clients receive the failure reason when requesting the schedule.
 

documentation/changelog.rst

@@ -42,6 +42,7 @@ New features
 * Floor off-clock API datetimes to a non-instantaneous sensor's resolution by default when ingesting sensor data, uploading sensor data, and handling scheduler flex-model timed events; configurable with the ``floor_datetimes_to_resolution`` sensor attribute [see `PR #2146 <https://www.github.com/FlexMeasures/flexmeasures/pull/2146>`_ and `PR #2194 <https://www.github.com/FlexMeasures/flexmeasures/pull/2194>`_]
 * In the UI, asset and sensor charts now render with Apache ECharts (canvas) by default, for much faster drawing and interaction on dense time series, while staying visually and functionally equivalent to the previous Vega-Lite charts, which remain available as a fallback via a toggle [see `PR #2234 <https://www.github.com/FlexMeasures/flexmeasures/pull/2234>`_ and `PR #2399 <https://www.github.com/FlexMeasures/flexmeasures/pull/2399>`_]
 * Breaking behaviour change: the top-level flex-context's ``relax-constraints`` field now defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False``. ``relax-constraints`` no longer covers the device ``consumption-capacity`` and ``production-capacity`` at all, which stay hard unless relaxed by name, either by setting ``relax-capacity-constraints`` or by setting the device breach prices yourself. A directional device capacity may state a physical impossibility (a heat pump that cannot produce) rather than an economic preference, and a default should not make that breachable at a price. Explicitly given device breach prices are also respected now, instead of being overwritten by the defaults [see `PR #2172 <https://www.github.com/FlexMeasures/flexmeasures/pull/2172>`_ and `PR #2398 <https://www.github.com/FlexMeasures/flexmeasures/pull/2398>`_]
+* Breaking behaviour change: ``soc-targets`` are now relaxed along with the other state-of-charge constraints, instead of always being enforced as hard equality constraints. A target that cannot be reached is breached at a price (falling short is priced like a ``soc-minima`` breach, overshooting like a ``soc-maxima`` breach) and reported among the unresolved constraints, rather than making the whole scheduling job fail as infeasible. Set ``relax-constraints`` (or both ``relax-constraints`` and ``relax-soc-constraints``) to ``False`` to keep targets hard [see `PR #2390 <https://www.github.com/FlexMeasures/flexmeasures/pull/2390>`_]
 * Support for creating new assets by using another asset as a template from the UI. [see `PR #2195 <https://www.github.com/FlexMeasures/flexmeasures/pull/2195>`_ and `PR #2268 <https://www.github.com/FlexMeasures/flexmeasures/pull/2268>`_
 * In the UI, asset and sensor lists can be filtered by ID prefix through API-backed search fields [see `PR #2231 <https://www.github.com/FlexMeasures/flexmeasures/pull/2231>`_]
 * Support configurable lower and upper bounds and snapping for forecast post-processing [see `PR #2273 <https://www.github.com/FlexMeasures/flexmeasures/pull/2273>`_]

documentation/features/scheduling.rst

@@ -673,8 +673,10 @@ The ``violation`` values tell you how much shortfall exists:
 
 If ``unresolved`` and ``resolved`` are both empty, no state-of-charge constraints were set.
 
-.. note:: Hard constraints (``soc-targets``) are never reported in results because the scheduler enforces them strictly by definition.
-          If a hard constraint cannot be met, the entire scheduling job will fail, not produce results with violations.
+.. note:: ``soc-targets`` are reported under ``unresolved`` only, and only while constraint relaxation is on.
+          A target is a two-sided constraint, so its reported violation is the absolute deviation from the target, in either direction,
+          and there is no headroom to report when a target is met.
+          With relaxation off, a target that cannot be met makes the entire scheduling job fail instead of producing results with violations.
 
 Work on other schedulers
 ---------------------------------------

flexmeasures/api/v3_0/tests/test_jobs_api.py

@@ -398,6 +398,9 @@ def test_get_job_status_failed_infeasible_schedule_includes_exc_info(
 ):
     charging_station = add_charging_station_assets["Test charging station"].sensors[0]
     message = message_for_trigger_schedule(with_targets=True, realistic_targets=False)
+    # Unreachable SoC constraints only yield an infeasible problem while relaxation is off;
+    # by default they are breached at a price instead.
+    message["flex-context"] = {"relax-constraints": False}
 
     with app.test_client() as client:
         trigger_response = client.post(

flexmeasures/data/models/planning/storage.py

@@ -1399,6 +1399,106 @@ def device_list_series(
                 # soc-maxima will become a soft constraint (modelled as stock commitments), so remove hard constraint
                 soc_maxima[d] = None
 
+            # A soc target is a two-sided constraint: falling short of it is a shortage
+            # (priced like a soc-minima breach) and overshooting it is a surplus (priced
+            # like a soc-maxima breach). We therefore relax targets only when both breach
+            # prices are available, which is the case whenever SoC relaxation is on, since
+            # the two default prices are filled in as a pair.
+            if (
+                self.flex_context.get("soc_minima_breach_price") is not None
+                and self.flex_context.get("soc_maxima_breach_price") is not None
+                and soc_targets[d] is not None
+                and soc_at_start[d] is not None
+                and self._soc_relaxation_applies_to(device_stock_key.get(d), sensor_d)
+            ):
+                soc_minima_breach_price = self.flex_context["soc_minima_breach_price"]
+                soc_maxima_breach_price = self.flex_context["soc_maxima_breach_price"]
+                any_soc_target_shortage_price = (
+                    get_continuous_series_sensor_or_quantity(
+                        variable_quantity=soc_minima_breach_price,
+                        unit=self.flex_context["shared_currency_unit"] + "/MWh",
+                        query_window=(start + resolution, end + resolution),
+                        resolution=resolution,
+                        beliefs_before=belief_time,
+                        fill_sides=True,
+                    ).shift(-1, freq=resolution)
+                )
+                all_soc_target_shortage_price = (
+                    get_continuous_series_sensor_or_quantity(
+                        variable_quantity=soc_minima_breach_price,
+                        unit=self.flex_context["shared_currency_unit"]
+                        + "/MWh*h",  # from EUR/MWh² to EUR/MWh/resolution
+                        query_window=(start + resolution, end + resolution),
+                        resolution=resolution,
+                        beliefs_before=belief_time,
+                        fill_sides=True,
+                    ).shift(-1, freq=resolution)
+                )
+                any_soc_target_surplus_price = get_continuous_series_sensor_or_quantity(
+                    variable_quantity=soc_maxima_breach_price,
+                    unit=self.flex_context["shared_currency_unit"] + "/MWh",
+                    query_window=(start + resolution, end + resolution),
+                    resolution=resolution,
+                    beliefs_before=belief_time,
+                    fill_sides=True,
+                ).shift(-1, freq=resolution)
+                all_soc_target_surplus_price = get_continuous_series_sensor_or_quantity(
+                    variable_quantity=soc_maxima_breach_price,
+                    unit=self.flex_context["shared_currency_unit"]
+                    + "/MWh*h",  # from EUR/MWh² to EUR/MWh/resolution
+                    query_window=(start + resolution, end + resolution),
+                    resolution=resolution,
+                    beliefs_before=belief_time,
+                    fill_sides=True,
+                ).shift(-1, freq=resolution)
+                # Set up commitments DataFrame
+                # soc_targets_d is a temp variable because add_storage_constraints can't deal with Series yet
+                soc_targets_d = get_continuous_series_sensor_or_quantity(
+                    variable_quantity=soc_targets[d],
+                    unit="MWh",
+                    query_window=(start + resolution, end + resolution),
+                    resolution=resolution,
+                    beliefs_before=belief_time,
+                    as_instantaneous_events=True,
+                    resolve_overlaps="first",
+                )
+                # shift soc targets by one resolution (they define a state at a certain time,
+                # while the commitment defines what the total stock should be at the end of a time slot,
+                # where the time slot is indexed by its starting time)
+                soc_targets_d = soc_targets_d.shift(-1, freq=resolution) * (
+                    timedelta(hours=1) / resolution
+                ) - soc_at_start[d] * (timedelta(hours=1) / resolution)
+
+                commitment = StockCommitment(
+                    name="any soc targets",
+                    quantity=soc_targets_d,
+                    # negative price because breaching in the downwards (shortage) direction is penalized
+                    downwards_deviation_price=-any_soc_target_shortage_price,
+                    # positive price because breaching in the upwards (surplus) direction is penalized
+                    upwards_deviation_price=any_soc_target_surplus_price,
+                    index=index,
+                    _type="any",
+                    device=d,
+                    stock=device_stock_key.get(d),
+                )
+                commitments.append(commitment)
+
+                commitment = StockCommitment(
+                    name="all soc targets",
+                    quantity=soc_targets_d,
+                    # negative price because breaching in the downwards (shortage) direction is penalized
+                    downwards_deviation_price=-all_soc_target_shortage_price,
+                    # positive price because breaching in the upwards (surplus) direction is penalized
+                    upwards_deviation_price=all_soc_target_surplus_price,
+                    index=index,
+                    device=d,
+                    stock=device_stock_key.get(d),
+                )
+                commitments.append(commitment)
+
+                # soc-targets will become a soft constraint (modelled as stock commitments), so remove hard constraint
+                soc_targets[d] = None
+
             # only apply SOC constraints to the first device of a shared stock
             apply_soc_constraints = True
             for stock_id, devices in self.stock_groups.items():
@@ -2773,6 +2873,52 @@ def _build_soc_schedule(  # noqa: C901
 
         return soc_schedule, soc_schedule_mwh
 
+    def _soc_target_violations(
+        self,
+        soc_targets,
+        soc_mwh: pd.Series,
+        start: datetime,
+        end: datetime,
+        resolution: timedelta,
+        precision: int,
+        most_relevant_only: bool,
+    ) -> list[dict]:
+        """Report time slots where the scheduled state of charge misses a soc target.
+
+        A target is a two-sided constraint, so a violation is the absolute deviation from
+        the target, in either direction. There is no headroom to report when a target is
+        met, so targets never produce a "resolved" entry.
+        """
+        if soc_targets is None:
+            return []
+        soc_targets_series = get_continuous_series_sensor_or_quantity(
+            variable_quantity=soc_targets,
+            unit="MWh",
+            query_window=(start + resolution, end + resolution),
+            resolution=resolution,
+            beliefs_before=self.belief_time,
+            as_instantaneous_events=True,
+            resolve_overlaps="first",
+        )
+        defined_targets = soc_targets_series.dropna()
+        if len(defined_targets) == 0:
+            return []
+        deviations = (soc_mwh.reindex(defined_targets.index) - defined_targets).abs()
+        # Ignore deviations that would round away at the reporting precision.
+        violations = deviations[deviations.mul(1000).round(precision) > 0]
+        if violations.empty:
+            return []
+        violation_times = (
+            [violations.index[0]] if most_relevant_only else violations.index
+        )
+        return [
+            {
+                "datetime": t.tz_convert("UTC").isoformat(),
+                "violation": f"{round(float(violations[t]) * 1000, precision)} kWh",
+            }
+            for t in violation_times
+        ]
+
     def _compute_unresolved_targets(
         self,
         flex_model: list[dict],
@@ -2943,6 +3089,18 @@ def _compute_unresolved_targets(
                             for t in margin_times
                         ]
 
+            target_violations = self._soc_target_violations(
+                soc_targets=flex_model_d.get("soc_targets"),
+                soc_mwh=soc_mwh,
+                start=start,
+                end=end,
+                resolution=resolution,
+                precision=precision,
+                most_relevant_only=most_relevant_only,
+            )
+            if target_violations:
+                device_violations["soc-targets"] = target_violations
+
             if device_violations:
                 violation_entry = {"asset": asset_id}
                 violation_entry.update(device_violations)

flexmeasures/data/models/planning/tests/test_solver.py

@@ -1251,7 +1251,12 @@ def test_validate_constraints(
 
 
 def test_infeasible_problem_error(db, add_battery_assets):
-    """Try to create a schedule with infeasible constraints. soc-max is 4.5 and soc-target is 8.0"""
+    """Try to create a schedule with infeasible constraints. soc-max is 4.5 and soc-target is 8.0
+
+    Note that this only yields an infeasible problem when constraint relaxation is off;
+    with relaxation on (the default), an unreachable target is breached at a price. See
+    ``test_unreachable_soc_target_is_relaxed_by_default``.
+    """
 
     # get the sensors from the database
     _epex_da, battery = get_sensors_from_db(db, add_battery_assets)
@@ -1269,6 +1274,7 @@ def compute_schedule(flex_model):
             end,
             resolution,
             flex_model=flex_model,
+            flex_context={"relax-constraints": False},
         )
         schedule = scheduler.compute()
 
@@ -1299,6 +1305,44 @@ def compute_schedule(flex_model):
         compute_schedule(flex_model)
 
 
+def test_unreachable_soc_target_is_relaxed_by_default(db, add_battery_assets):
+    """An unreachable soc-target no longer makes the problem infeasible.
+
+    Same setup as ``test_infeasible_problem_error`` (soc-max 4.5, soc-target 8.0), but
+    with the default ``relax-constraints``. The target becomes a stock commitment, so
+    the scheduler charges as far as the (hard) soc-max allows and breaches the target
+    at a price, instead of failing to produce a schedule at all.
+    """
+    _epex_da, battery = get_sensors_from_db(db, add_battery_assets)
+
+    tz = pytz.timezone("Europe/Amsterdam")
+    start = tz.localize(datetime(2015, 1, 2))
+    end = tz.localize(datetime(2015, 1, 3))
+    resolution = timedelta(hours=1)
+
+    soc_at_start = battery.get_attribute("soc_in_mwh")
+    scheduler = StorageScheduler(
+        battery,
+        start,
+        end,
+        resolution,
+        flex_model={
+            "soc-at-start": soc_at_start,
+            "soc-min": 0.5,
+            "soc-max": 4.5,
+            "soc-targets": [{"datetime": "2015-01-02T16:00:00+01:00", "value": 8.0}],
+        },
+    )
+    schedule = scheduler.compute()
+    soc_schedule = integrate_time_series(schedule, soc_at_start, decimal_precision=3)
+
+    # The hard soc-max still holds, and the scheduler gets as close to the target as it can.
+    assert soc_schedule.max() <= 4.5 + TOLERANCE
+    assert soc_schedule.loc[pd.Timestamp("2015-01-02T16:00:00+01:00")] == pytest.approx(
+        4.5, abs=1e-3
+    )
+
+
 def test_numerical_errors(app_with_each_solver, setup_planning_test_data, db):
     """Test that a soc-target = soc-max can exceed this value due to numerical errors in the operations
     to compute the device constraint DataFrame.
@@ -1350,6 +1394,9 @@ def test_numerical_errors(app_with_each_solver, setup_planning_test_data, db):
             ],
             "soc-unit": "MWh",
         },
+        # This test is about numerical error in the hard "equals" constraint, so opt
+        # out of the relaxation that would turn soc-targets into stock commitments.
+        flex_context={"relax-constraints": False},
     )
 
     (
@@ -2704,7 +2751,14 @@ def test_add_storage_constraint_from_sensor(
     ]
 
     scheduler: Scheduler = StorageScheduler(
-        battery, start, end, resolution, flex_model=flex_model
+        battery,
+        start,
+        end,
+        resolution,
+        flex_model=flex_model,
+        # This test inspects the hard "equals" constraint, so opt out of the
+        # relaxation that would turn soc-targets into stock commitments instead.
+        flex_context={"relax-constraints": False},
     )
 
     scheduler_info = scheduler._prepare()