Files
mxaccessgw/docs/WorkerConversion.md
T
Joseph Doherty 9ef6fc8489
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m28s
ci / portable (push) Failing after 16m55s
ci / java (push) Failing after 17m47s
fix(worker): send the Write2/WriteSecured2 time argument as box-local time (#228)
MXAccess reads the VT_DATE time argument of Write2, WriteSecured2 and their
bulk forms as box-local time and converts it to UTC itself. The worker passed
the proto Timestamp's UTC clock fields (the COM marshaler ignores
DateTime.Kind), so on vd03 (UTC-5) a Write2 sent at 01:33:00Z was stored at
06:33:00Z.

VariantConverter.ConvertToComTimestamp converts a timestamp_value instant with
TimeZoneInfo.ConvertTimeFromUtc into the worker's zone (TimeZoneInfo.Local by
default, DST included; an injectable zone for tests). The executor uses it for
the time argument only: value conversion is unchanged, and the read direction
(MXAccess reports UTC; ToTimestamp treats it as UTC) is unchanged. The one
residual loss is MXAccess-side: the repeated fall-back hour cannot be told
apart in an offset-less VT_DATE.

Tests: VariantConverterTimestampTests (UTC-5, US-Eastern DST both edges,
UTC+05:30, UTC, fall-back ambiguity, value path unchanged); the four executor
forwarding tests inject a fixed UTC-5 zone; the windev live Write2 smoke now
sends a past whole-second timestamp and asserts the stored one when applied.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:49:41 -05:00

16 KiB

Worker Conversion Layer

The conversion layer in ZB.MOM.WW.MxGateway.Worker.Conversion projects COM VARIANT payloads, HRESULT codes, and MXSTATUS_PROXY records into the protobuf wire types in ZB.MOM.WW.MxGateway.Contracts.Proto. The design is parity-first: every projection preserves enough raw metadata that the original COM observation can be reconstructed downstream.

Overview

gateway.md (sections "Value Model" and "Status Model") requires that the wire format use a value union capable of representing COM VARIANT values and arrays, that lossy conversions retain both the typed projection and raw diagnostic metadata, and that MXSTATUS_PROXY arrays never collapse to a single success flag. The types in src/ZB.MOM.WW.MxGateway.Worker/Conversion/ are the worker-side enforcement of those rules.

The layer is split into three concerns:

  • Value projection: VariantConverter
  • HRESULT projection: HResultConverter, HResultConversion
  • Status projection: MxStatusProxyConverter, MxStatusDetailText, MxStatusConversionException

VariantConverter

VariantConverter projects scalar and array VARIANT payloads delivered by COM interop into MxValue and MxArray. It accepts an optional expectedDataType so that an MXAccess attribute hint (for example MxDataType.Time for a 64-bit FILETIME) overrides the default CLR-driven projection.

Scalar projection

Scalars dispatch on Type.GetTypeCode and populate the matching field in the MxValue oneof. Each branch records both the typed value and the source VARIANT tag in VariantType, so consumers can distinguish a short from an int even after both project to MxDataType.Integer.

case TypeCode.Boolean:
    return new MxValue
    {
        DataType = MxDataType.Boolean,
        VariantType = variantType,
        BoolValue = (bool)value,
    };

case TypeCode.Byte:
case TypeCode.SByte:
case TypeCode.Int16:
case TypeCode.UInt16:
case TypeCode.Int32:
    return new MxValue
    {
        DataType = MxDataType.Integer,
        VariantType = variantType,
        Int32Value = System.Convert.ToInt32(value, CultureInfo.InvariantCulture),
    };

Decimal and 64-bit unsigned integers that exceed long.MaxValue cannot project losslessly. The converter still emits a typed best-effort value but attaches a RawDiagnostic so the loss is recorded on the wire rather than silently absorbed.

case TypeCode.Decimal:
    return new MxValue
    {
        DataType = MxDataType.Double,
        VariantType = variantType,
        DoubleValue = System.Convert.ToDouble(value, CultureInfo.InvariantCulture),
        RawDiagnostic = "Decimal value projected to double.",
    };

Time-typed integers use the expectedDataType hint. ConvertInt64Scalar interprets a long as a Windows FILETIME when the caller asks for MxDataType.Time, otherwise it stays an integer. ConvertUInt64Scalar falls through to a raw projection if the value exceeds long.MaxValue.

Null and missing values

VARIANT distinguishes VT_EMPTY from VT_NULL. The converter preserves both by inspecting whether the input is DBNull or a CLR null reference, and tags the result with IsNull = true so consumers do not misread a default value as data.

private static MxValue CreateNullValue(
    object? value,
    MxDataType expectedDataType)
{
    return new MxValue
    {
        DataType = expectedDataType == MxDataType.Unspecified ? MxDataType.NoData : expectedDataType,
        VariantType = value is DBNull ? "VT_NULL" : "VT_EMPTY",
        IsNull = true,
    };
}

Sparse array expansion (write path, gateway only)

MxSparseArray — the sparse_array_value arm on MxValue — is a write-only shorthand. The worker never produces or receives it; the gateway expands it into a full MxArray before the command reaches the named pipe. Expansion allocates a complete array of total_length slots, initializes every slot to the element type's default (bool → false; numeric → 0; string → ""; time/timestamp → Unix epoch), then writes each MxSparseElement at its declared index. The resulting array_value is an ordinary MxArray that passes through the conversion layer unchanged. The worker therefore still performs a single whole-array COM write, preserving MXAccess parity. Unmentioned indices are reset to their type default, not preserved from prior state — there is no read-modify-write merge.

Array projection

ConvertArray records the rank and per-dimension lengths so multi-dimensional SAFEARRAY shapes survive the round trip. The element type is resolved from the caller-supplied hint or the CLR element type via ResolveArrayElementDataType, then dispatched to the matching typed builder (ConvertBoolArray, ConvertInt64Array, ConvertTimestampArray, and so on).

for (int dimension = 0; dimension < array.Rank; dimension++)
{
    mxArray.Dimensions.Add((uint)array.GetLength(dimension));
}

System.Type? elementType = array.GetType().GetElementType();
MxDataType elementDataType = ResolveArrayElementDataType(elementType, expectedElementDataType);
mxArray.ElementDataType = elementDataType;

When the element type cannot be classified, ConvertArray does not throw. It downgrades the result to MxDataType.Unknown, records the original expected type in RawElementDataType, and serializes each element via ConvertRawArray as a UTF-8 byte string. This satisfies the gateway.md requirement to keep both the best typed projection and the raw diagnostic metadata.

default:
    mxArray.ElementDataType = MxDataType.Unknown;
    mxArray.RawElementDataType = (int)expectedElementDataType;
    mxArray.RawDiagnostic = CreateRawDiagnostic(array);
    mxArray.RawValues = ConvertRawArray(array);
    return mxArray;

Variant type names

GetVariantTypeName maps CLR types to canonical VT_* strings (VT_BOOL, VT_I4, VT_BSTR, VT_DATE, and so on). Unmapped CLR types fall back to CLR:<FullName> so the wire format never silently invents a VT_* tag it cannot justify. Array element tags are wrapped as SAFEARRAY(<element>) by CreateArrayVariantType.

Timestamps: the two directions differ

MXAccess does not use one time convention for both directions, so the worker does not either.

  • Read (OnDataChange source timestamps, VT_DATE values). MXAccess hands the worker UTC clock fields. ToTimestamp treats an unspecified-kind DateTime as UTC and converts a local-kind one with ToUniversalTime. A plain Write on vd03 (UTC-5) reports the current UTC time as its source timestamp, which confirms this.
  • Write time argument (Write2, WriteSecured2, Write2Bulk, WriteSecured2Bulk). MXAccess reads the VT_DATE time argument as box-local time and converts it to UTC itself. A VT_DATE has no offset field, and the COM marshaler ignores DateTime.Kind, so passing the UTC clock fields stored the source timestamp shifted by the box's UTC offset. On vd03 a Write2 sent at 01:33:00Z was stored at 06:33:00Z (#228). ConvertToComTimestamp therefore converts the timestamp_value instant with TimeZoneInfo.ConvertTimeFromUtc into the worker box's zone (TimeZoneInfo.Local, DST rules included) before the COM call. The command executor uses it for the time argument only; ConvertToComValue still renders a timestamp value as UTC, because whether a Time-typed attribute value write follows the same rule has not been proven.

UTC to local is always defined: every UTC instant has exactly one local wall-clock time, and the conversion never produces a local time that does not exist (the skipped spring-forward hour). The one loss is on the MXAccess side. In the hour that repeats when DST ends, two UTC instants an hour apart share one local wall-clock time, and the offset-less VT_DATE cannot say which one is meant, so MXAccess may store an instant from that hour 1 h off. Run the worker box on UTC if exact timestamps across the fall-back hour matter. On a UTC box the conversion is the identity, which is why #228 could not be seen there.

A time argument of another kind (a string or a number) carries no zone and is passed through unchanged; its meaning is the caller's. Consumers must send timestamp_value as a UTC instant, as the proto Timestamp already is.

Why lossless preservation matters

The MXAccess engine returns values whose semantic type only fully resolves after consulting the engine's own attribute metadata. Clients that round-trip these values through the gateway (replay, parity fixtures, diagnostics) need the original VT_* tag, the engine-declared MxDataType, and any conversion diagnostic; otherwise edge cases such as decimal-to-double rounding, ulong overflow, or an unknown SAFEARRAY element type become invisible bugs. Storing both the typed projection and the raw fields in the same MxValue/MxArray lets cross-language clients recover the original observation byte-for-byte where possible and detect lossy cases where it is not.

HResultConverter and HResultConversion

HResultConverter.Convert wraps any Exception thrown across the COM boundary. It prefers COMException.ErrorCode over Exception.HResult because the runtime sometimes overwrites Exception.HResult while marshalling, and the ErrorCode field is the value the COM call actually returned.

public HResultConversion Convert(Exception exception)
{
    if (exception is null)
    {
        throw new ArgumentNullException(nameof(exception));
    }

    int hresult = exception is COMException comException
        ? comException.ErrorCode
        : exception.HResult;

    return new HResultConversion(
        hresult,
        CreateProtocolStatus(hresult, exception),
        CreateSafeDiagnosticMessage(exception));
}

CreateProtocolStatus maps 0 (S_OK) to ProtocolStatusCode.Ok and any non-zero HRESULT to ProtocolStatusCode.MxaccessFailure. The status Message includes the exception type name and the formatted HRESULT (HRESULT 0x<8 hex digits>), giving downstream operators the same identifier they would see in COM event logs.

HResultConversion is an immutable carrier with three fields:

Field Purpose
HResult Raw signed 32-bit HRESULT, suitable for inclusion in a wire reply
ProtocolStatus The ProtocolStatus projection used in command replies
DiagnosticMessage A safe, fully-qualified diagnostic string for logs

The diagnostic message is built from the exception type's FullName and the formatted HRESULT only. It deliberately omits Exception.Message so user-supplied or localized strings do not leak into worker log channels that may be shipped to less trusted destinations.

MxStatusProxyConverter

MxStatusProxyConverter projects MXAccess MXSTATUS_PROXY records into the MxStatusProxy proto message. Because the COM struct is exposed via interop reflection rather than a typed binding, the converter reads the success, category, detectedBy, and detail fields by name through FieldInfo.

public MxStatusProxy Convert(object status)
{
    if (status is null)
    {
        throw new ArgumentNullException(nameof(status));
    }

    Type statusType = status.GetType();
    int success = ReadInt32Field(status, statusType, "success");
    int rawCategory = ReadInt32Field(status, statusType, "category");
    int rawDetectedBy = ReadInt32Field(status, statusType, "detectedBy");
    int detail = ReadInt32Field(status, statusType, "detail");

    return new MxStatusProxy
    {
        Success = success,
        Category = MapCategory(rawCategory),
        DetectedBy = MapSource(rawDetectedBy),
        Detail = detail,
        RawCategory = rawCategory,
        RawDetectedBy = rawDetectedBy,
        DiagnosticText = MxStatusDetailText.Lookup(detail),
    };
}

MapCategory and MapSource translate the integer codes documented for MXSTATUS_PROXY (for example 0 = Ok, 3 = CommunicationError, 0 = RequestingLmx, 5 = RespondingAutomationObject) into typed enums. The original integers are preserved alongside the typed projection in RawCategory and RawDetectedBy, so any code the runtime emits outside the documented range is still observable.

Status arrays

ConvertMany walks an inbound Array of status structs and emits IReadOnlyList<MxStatusProxy>. Null entries become explicit Unknown/Unknown placeholders with a diagnostic text rather than being dropped, so the index of each status still aligns with the index of the value or item handle it describes.

foreach (object? status in statuses)
{
    if (status is null)
    {
        converted.Add(new MxStatusProxy
        {
            Category = MxStatusCategory.Unknown,
            DetectedBy = MxStatusSource.Unknown,
            DiagnosticText = "Null MXSTATUS_PROXY entry.",
        });
        continue;
    }

    converted.Add(Convert(status));
}

Why arrays are not collapsed

A single MXAccess command (notably Read, Write, and event callbacks) can return one status per item handle. gateway.md requires that the wire format represent each entry independently, because collapsing them to a Boolean success flag hides partial failures: a 50-item write where one item fails would be indistinguishable from a 50-item write where every item failed. Preserving the array per-position lets clients correlate each MxStatusProxy with its item handle and MxValue.

Completion-only status fallback

When MXAccess returns a status payload that is not a recognizable MXSTATUS_PROXY struct (for example a completion-only byte buffer from older runtimes), PreserveCompletionOnlyStatusBytes hex-encodes the raw bytes so they survive transport even though they cannot be typed.

public string PreserveCompletionOnlyStatusBytes(byte[] statusBytes)
{
    if (statusBytes is null)
    {
        throw new ArgumentNullException(nameof(statusBytes));
    }

    return $"completion_only_status_hex={BitConverter.ToString(statusBytes).Replace("-", string.Empty)}";
}

MxStatusDetailText

MxStatusDetailText is an internal lookup that maps known MXSTATUS_PROXY.detail codes to short human-readable strings (for example 28 = "Index out of range", 42 = "Unable to convert string", 8017 = "Object must be offscan to modify attributes that have an MxSecurityConfigure security classification"). MxStatusProxyConverter.Convert calls Lookup and writes the result to DiagnosticText. Unknown codes return string.Empty, leaving the numeric Detail field as the authoritative identifier.

The mapping covers the engine-error range documented for MXAccess (16-50, 56-61, 541-542, 8017). Adding entries here is the supported way to enrich wire-level diagnostics without changing the proto schema.

MxStatusConversionException

MxStatusConversionException is the worker-internal signal that a status struct could not be projected at all - for example, the interop type is missing one of the required fields, or the field is null. MxStatusProxyConverter.ReadInt32Field raises it with a message that names both the offending CLR type and the missing field.

private static int ReadInt32Field(
    object value,
    Type valueType,
    string fieldName)
{
    FieldInfo? field = valueType.GetField(fieldName, BindingFlags.Instance | BindingFlags.Public);
    if (field is null)
    {
        throw new MxStatusConversionException(
            $"Status object type '{valueType.FullName}' does not expose required field '{fieldName}'.");
    }

    object? fieldValue = field.GetValue(value);
    if (fieldValue is null)
    {
        throw new MxStatusConversionException(
            $"Status object field '{fieldName}' on type '{valueType.FullName}' is null.");
    }

    return System.Convert.ToInt32(fieldValue, CultureInfo.InvariantCulture);
}

The exception is distinct from COMException so the worker's command pipeline can route it through a dedicated handler. Callers typically translate it into a ProtocolStatus with ProtocolStatusCode.MxaccessFailure and a synthetic HRESULT, then attach the exception message as a RawDiagnostic rather than letting the failure surface as a generic worker error.