docs: forced-subtag fix resolution (#1 proto artifact, #2 fixed)

docs/plans/2026-06-15-forced-subtag-mode-fix.md

@@ -0,0 +1,116 @@
+# ForceSubtag Mode Fix Implementation Plan
+
+> Fixes the two defects surfaced by the D3 live validation (2026-06-15): forced-subtag
+> doesn't actually run subtag (#1), and the gateway never reflects a forced provider mode
+> into the gauge/feed (#2).
+
+**Goal:** Make `MxGateway:Alarms:Fallback:Mode=ForceSubtag` actually serve degraded subtag
+alarms AND have the gateway advertise `provider_mode=2` / degraded badge.
+
+**Evidence:** Live ForceSubtag run returned alarmmgr-sourced active alarms (May raise
+timestamps, `degraded=false`) and `provider_mode` stuck at 1, despite ForceSubtag binding
+(proven by invalid-value crash) and the deployed worker containing the ForcedMode routing
+(`3f5e5fc` ∈ `5976770`, worker dated 2026-06-14).
+
+---
+
+## Defect #2 (CONFIRMED code defect) — gateway never reflects forced mode
+
+**Root cause:** `GatewayAlarmMonitor.RunMonitorAsync` hard-baselines `_providerMode=Alarmmgr`
+and sets the gauge to 1, ignoring `_options.Fallback.Mode`. `_providerMode` only advances on a
+worker `OnAlarmProviderModeChanged` event, which is raised ONLY by `FailoverAlarmConsumer`
+(Auto mode). Forced-subtag builds `SubtagAlarmConsumer` directly → no event → gauge/feed stay
+Alarmmgr forever.
+
+### Task 2: Seed provider mode from configured forced mode (gateway, net10)
+**Classification:** small · **Parallelizable with:** none (precedes the diagnostic build)
+**Files:**
+- Modify: `src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs` (`RunMonitorAsync`, ~lines 160-172; add a permanent observability log in `SubscribeAlarmsAsync` ~line 257)
+- Test: `src/ZB.MOM.WW.MxGateway.Tests/Alarms/GatewayAlarmMonitorProviderModeTests.cs`
+
+**Change:** in `RunMonitorAsync`, compute `initialMode = MapForcedMode(_options.Fallback.Mode)`
+mapped as `Subtag→Subtag`, `Alarmmgr→Alarmmgr`, `Unspecified→Alarmmgr` (Auto starts on the
+alarmmgr primary). Set `_providerMode/_providerDegraded(=Subtag)/_providerReason/_providerSince`
+and `_metrics.SetAlarmProviderMode(ModeToInt(initialMode))` — using the existing no-switch gauge
+seam so `provider_switches` does NOT increment. Add a log in `SubscribeAlarmsAsync`:
+`"Alarm subscribe forcedMode={ForcedMode} (configMode={ConfigMode}) watchList={Count}"`.
+
+**Tests (fake-worker, no MXAccess):** with `Fallback:Mode=ForceSubtag` assert (a) first
+`StreamAlarms` message is `ProviderStatus{Mode=Subtag,Degraded=true}`; (b) gauge==2; (c)
+`provider_switches`==0. Add `ForceAlarmManager`→gauge 1 and `Auto`→gauge 1 baseline cases.
+
+**Verify:** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter FullyQualifiedName~GatewayAlarmMonitorProviderMode`.
+
+---
+
+## Defect #1 (runtime bug — needs diagnosis) — ForceSubtag runs alarmmgr
+
+The HEAD source path (gateway `MapForcedMode` → `SubscribeAlarmsCommand.ForcedMode=Subtag` →
+IPC → worker `AlarmCommandHandler.BuildConsumer` → `SubtagAlarmConsumer`) is statically correct,
+yet the runtime ran alarmmgr. A runtime diagnostic must locate where `forcedMode` becomes
+`Unspecified`.
+
+### Task 1: Add worker BuildConsumer observability log (worker, net48 x86)
+**Classification:** small (net48 — no init-only props) · **Parallelizable with:** Task 2
+**Files:**
+- Modify: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/AlarmCommandHandler.cs` (`BuildConsumer`, ~line 206)
+
+**Change:** log at entry of `BuildConsumer`:
+`"BuildConsumer forcedMode={ForcedMode} watchList={Count}"`. This is permanent observability.
+
+**Verify (windev only):** `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86`.
+
+### Task 3: Diagnostic run — capture the real forcedMode
+**Classification:** standard (ops/diagnostic on windev) · **Parallelizable with:** none
+**Steps:** rebuild worker (x86) + server from the branch, stand up the D3-style temp ForceSubtag
+instance (alt ports 5122/5132, isolated `.D3` worker name, WMI-detached, Http2, Development env
+for file logs), trigger the always-on monitor, and read the two new log lines:
+- Gateway `Alarm subscribe forcedMode=...` — what the gateway SENDS.
+- Worker `BuildConsumer forcedMode=...` — what the worker RECEIVES.
+
+Decision matrix:
+- Gateway logs `Subtag`, worker logs `Unspecified` → IPC/serialization drops the enum → fix the
+  send/translation path (likely a worker-proto vs gateway-proto `SubscribeAlarmsCommand` mismatch
+  in the named-pipe envelope).
+- Worker logs `Subtag` but alarmmgr data appears → bug in `BuildStandby`/`SubtagAlarmConsumer`/
+  `AlarmDispatcher` snapshot path.
+- Gateway logs `Unspecified` despite config ForceSubtag → gateway config/options read.
+
+### Task 4: Fix #1 per Task 3 diagnosis
+**Classification:** standard/high-risk (depends on where the defect is) · the exact change is
+determined by Task 3. Add a regression test at the identified layer (worker unit test for
+BuildConsumer→SubtagAlarmConsumer, or a contract/IPC round-trip test if the enum is dropped).
+
+---
+
+## Final: build, test, redeploy, re-validate
+- Build gateway (macOS) + worker (windev x86); run gateway + worker test suites.
+- Redeploy windev Server (and Worker if Task 4 changed it) per `project_deploy_mechanics`,
+  preserving appsettings.
+- Re-validate live with the temp ForceSubtag instance: active alarms `degraded=true` /
+  `source_provider=SUBTAG` with recent timestamps, `provider_mode 2`. Tear down temp instance;
+  production untouched.
+
+## Execution note
+Branch off `main`. #2 is the clean confirmed fix; #1 is diagnose-then-fix. Net48 worker
+constraints apply (no init-only props/positional records). Do NOT increment `provider_switches`
+on an initial forced-mode set.
+
+---
+
+## Resolution (2026-06-15)
+
+**#1 was NOT a bug — it was a grpcurl proto-mismatch artifact.** End-to-end instrumentation
+proved: the gateway sends `forcedMode=Subtag`, the worker's `BuildConsumer` builds the
+`SubtagAlarmConsumer`, and the worker `QueryActive` + gateway `ApplyReconcile`/`StreamAlarms`
+all carry `degraded=true` / `source_provider=SUBTAG`. The original "degraded=false" observation
+came from running grpcurl against the host checkout's proto (branch `feat/lazy-browse-children`),
+which predates the feature and lacks the `degraded`/`source_provider` fields — grpcurl silently
+dropped them. With the matching proto, every active alarm shows `degraded=true`. No code change.
+Tasks 1/3/4 (worker diagnostics + #1 fix) were dropped; the temporary diagnostics were reverted.
+
+**#2 was a real defect — fixed.** Gateway now seeds `_providerMode`/gauge/feed from the configured
+forced mode (`fix: gateway reflects configured forced provider mode`). Verified live:
+`provider_mode=2`, first `ProviderStatus` = `Mode=Subtag, degraded=true,
+reason="Forced subtag mode (configuration)"`. Auto mode unchanged → windev production (Auto)
+unaffected; no redeploy required. Gateway tests: 163 passed.

src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs

@@ -157,19 +157,48 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
     private async Task RunMonitorAsync(string subscription, CancellationToken stoppingToken)
     {
         _state = GatewayAlarmMonitorState.Starting;
+
+        // Derive the lifecycle baseline from the configured forced mode so a
+        // ForceSubtag / ForceAlarmManager start advertises the correct mode even
+        // though no OnAlarmProviderModeChanged event is raised in those modes
+        // (only Auto/failover produces that event). ForceSubtag starts degraded.
+        AlarmProviderMode initialMode;
+        bool initialDegraded;
+        string initialReason;
+        switch (MapForcedMode(_options.Fallback.Mode))
+        {
+            case AlarmProviderMode.Subtag:
+                initialMode = AlarmProviderMode.Subtag;
+                initialDegraded = true;
+                initialReason = "Forced subtag mode (configuration)";
+                break;
+            case AlarmProviderMode.Alarmmgr:
+                initialMode = AlarmProviderMode.Alarmmgr;
+                initialDegraded = false;
+                initialReason = string.Empty;
+                break;
+            default:
+                // Unspecified (Auto): the failover consumer starts on the
+                // alarm-manager primary and only degrades to subtag on failure.
+                initialMode = AlarmProviderMode.Alarmmgr;
+                initialDegraded = false;
+                initialReason = string.Empty;
+                break;
+        }
+
         lock (_sync)
         {
             // Re-baseline the provider status for this lifecycle so a restarted
-            // monitor advertises alarm-manager/not-degraded until told otherwise.
-            _providerMode = AlarmProviderMode.Alarmmgr;
-            _providerDegraded = false;
-            _providerReason = string.Empty;
+            // monitor advertises the configured mode until told otherwise.
+            _providerMode = initialMode;
+            _providerDegraded = initialDegraded;
+            _providerReason = initialReason;
             _providerSince = DateTimeOffset.UtcNow;
         }
 
-        // Align the observable gauge with the Alarmmgr baseline without recording
+        // Align the observable gauge with the lifecycle baseline without recording
         // a switch — the gauge was 0 (unknown) from construction until now.
-        _metrics.SetAlarmProviderMode(ModeToInt(AlarmProviderMode.Alarmmgr));
+        _metrics.SetAlarmProviderMode(ModeToInt(initialMode));
 
         GatewaySession session = await _sessionManager.OpenSessionAsync(
                 new SessionOpenRequest(BackendName, MonitorClientName, Guid.NewGuid().ToString("N"), CommandTimeout: null),
@@ -256,6 +285,10 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
 
         AlarmProviderMode forcedMode = MapForcedMode(_options.Fallback.Mode);
 
+        _logger.LogInformation(
+            "Alarm subscribe: forcedMode={ForcedMode} configMode={ConfigMode} watchList={WatchListCount}.",
+            forcedMode, _options.Fallback.Mode, watchList.Count);
+
         // When the forced mode is Unspecified (the "Auto" case) and the resolved
         // watch-list is empty — the common alarmmgr-only deployment — the command
         // is identical-in-effect to the historical SubscribeAlarms (wnwrap only):

src/ZB.MOM.WW.MxGateway.Tests/Alarms/GatewayAlarmMonitorProviderModeTests.cs

@@ -156,6 +156,151 @@ public sealed class GatewayAlarmMonitorProviderModeTests
         await monitor.StopAsync(CancellationToken.None);
     }
 
+    [Fact]
+    public async Task ForceSubtagConfig_BaselinesProviderStatusToSubtagDegraded_WithoutSwitch()
+    {
+        using GatewayMetrics metrics = new();
+        long switchCount = 0;
+        int gaugeValue = -1;
+        using MeterListener listener = new();
+        listener.InstrumentPublished = (instrument, meterListener) =>
+        {
+            if (ReferenceEquals(instrument.Meter, metrics.Meter)
+                && (instrument.Name == "mxgateway.alarms.provider_switches"
+                    || instrument.Name == "mxgateway.alarms.provider_mode"))
+            {
+                meterListener.EnableMeasurementEvents(instrument);
+            }
+        };
+        listener.SetMeasurementEventCallback<long>(
+            (instrument, measurement, _, _) =>
+            {
+                if (ReferenceEquals(instrument.Meter, metrics.Meter)
+                    && instrument.Name == "mxgateway.alarms.provider_switches")
+                {
+                    Interlocked.Add(ref switchCount, measurement);
+                }
+            });
+        listener.SetMeasurementEventCallback<int>(
+            (instrument, measurement, _, _) =>
+            {
+                if (ReferenceEquals(instrument.Meter, metrics.Meter)
+                    && instrument.Name == "mxgateway.alarms.provider_mode")
+                {
+                    Interlocked.Exchange(ref gaugeValue, measurement);
+                }
+            });
+        listener.Start();
+
+        FakeSessionManager sessions = new();
+        using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, "ForceSubtag");
+
+        using CancellationTokenSource cts = new();
+        await monitor.StartAsync(cts.Token);
+        await sessions.WaitForSubscribeAsync(WaitTimeout);
+
+        using CancellationTokenSource streamCts = new();
+        AlarmFeedMessage? first = null;
+        Task reader = Task.Run(async () =>
+        {
+            await foreach (AlarmFeedMessage message in monitor.StreamAsync(null, streamCts.Token))
+            {
+                first = message;
+                break;
+            }
+        });
+
+        await WaitUntilAsync(() => first is not null, WaitTimeout);
+
+        Assert.NotNull(first);
+        Assert.Equal(AlarmFeedMessage.PayloadOneofCase.ProviderStatus, first!.PayloadCase);
+        Assert.Equal(AlarmProviderMode.Subtag, first.ProviderStatus.Mode);
+        Assert.True(first.ProviderStatus.Degraded);
+
+        // The observable gauge must read subtag (2) after start.
+        listener.RecordObservableInstruments();
+        Assert.Equal(2, Volatile.Read(ref gaugeValue));
+
+        // The initial set must not record a provider switch.
+        Assert.Equal(0, Interlocked.Read(ref switchCount));
+
+        await streamCts.CancelAsync();
+        await reader;
+        await cts.CancelAsync();
+        await monitor.StopAsync(CancellationToken.None);
+    }
+
+    [Fact]
+    public async Task ForceAlarmManagerConfig_BaselinesProviderStatusToAlarmmgr_WithoutSwitch()
+    {
+        using GatewayMetrics metrics = new();
+        long switchCount = 0;
+        int gaugeValue = -1;
+        using MeterListener listener = new();
+        listener.InstrumentPublished = (instrument, meterListener) =>
+        {
+            if (ReferenceEquals(instrument.Meter, metrics.Meter)
+                && (instrument.Name == "mxgateway.alarms.provider_switches"
+                    || instrument.Name == "mxgateway.alarms.provider_mode"))
+            {
+                meterListener.EnableMeasurementEvents(instrument);
+            }
+        };
+        listener.SetMeasurementEventCallback<long>(
+            (instrument, measurement, _, _) =>
+            {
+                if (ReferenceEquals(instrument.Meter, metrics.Meter)
+                    && instrument.Name == "mxgateway.alarms.provider_switches")
+                {
+                    Interlocked.Add(ref switchCount, measurement);
+                }
+            });
+        listener.SetMeasurementEventCallback<int>(
+            (instrument, measurement, _, _) =>
+            {
+                if (ReferenceEquals(instrument.Meter, metrics.Meter)
+                    && instrument.Name == "mxgateway.alarms.provider_mode")
+                {
+                    Interlocked.Exchange(ref gaugeValue, measurement);
+                }
+            });
+        listener.Start();
+
+        FakeSessionManager sessions = new();
+        using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, "ForceAlarmManager");
+
+        using CancellationTokenSource cts = new();
+        await monitor.StartAsync(cts.Token);
+        await sessions.WaitForSubscribeAsync(WaitTimeout);
+
+        using CancellationTokenSource streamCts = new();
+        AlarmFeedMessage? first = null;
+        Task reader = Task.Run(async () =>
+        {
+            await foreach (AlarmFeedMessage message in monitor.StreamAsync(null, streamCts.Token))
+            {
+                first = message;
+                break;
+            }
+        });
+
+        await WaitUntilAsync(() => first is not null, WaitTimeout);
+
+        Assert.NotNull(first);
+        Assert.Equal(AlarmFeedMessage.PayloadOneofCase.ProviderStatus, first!.PayloadCase);
+        Assert.Equal(AlarmProviderMode.Alarmmgr, first.ProviderStatus.Mode);
+        Assert.False(first.ProviderStatus.Degraded);
+
+        listener.RecordObservableInstruments();
+        Assert.Equal(1, Volatile.Read(ref gaugeValue));
+        Assert.Equal(0, Interlocked.Read(ref switchCount));
+
+        await streamCts.CancelAsync();
+        await reader;
+        await cts.CancelAsync();
+        await monitor.StopAsync(CancellationToken.None);
+    }
+
     [Fact]
     public async Task SubscribeAlarms_SendsForcedModeAndWatchList_FromConfiguration()
     {
@@ -259,6 +404,22 @@ public sealed class GatewayAlarmMonitorProviderModeTests
             NullLogger<GatewayAlarmMonitor>.Instance);
     }
 
+    private static GatewayAlarmMonitor CreateMonitor(FakeSessionManager sessions, GatewayMetrics metrics, string mode)
+    {
+        AlarmsOptions options = new()
+        {
+            Enabled = true,
+            SubscriptionExpression = @"\\NODE\Galaxy!Area",
+            Fallback = new AlarmFallbackOptions { Mode = mode },
+        };
+        return new GatewayAlarmMonitor(
+            sessions,
+            new StubWatchListResolver([]),
+            metrics,
+            Microsoft.Extensions.Options.Options.Create(new GatewayOptions { Alarms = options }),
+            NullLogger<GatewayAlarmMonitor>.Instance);
+    }
+
     private static async Task<AlarmFeedMessage> WaitForAsync(
         List<AlarmFeedMessage> received,
         Func<AlarmFeedMessage, bool> predicate,