From Beta to Production: Building Detection Rules with Confidence

A rule that catches a test event has passed its first test. It has not yet shown that it can distinguish an attacker from a maintenance script, handle missing fields, or behave consistently across customer environments. Those questions require observation and investigation.

SiemCore provides separate Active and Beta controls so rule authors can develop a rule, evaluate it against live logs, review findings, and decide when it should feed mysoc.ai. Understanding those controls is the starting point for a useful release process.

Understand what Beta changes

New rules in the Pro editor start disabled. Enable Active and Beta together when you are ready to observe them. Clearing Beta on a disabled rule does not activate it. Likewise, switching Active off does not erase its Beta setting. Review both controls when changing a rule’s lifecycle.

Write the detection hypothesis before the conditions

Describe the behavior, the evidence required, and the benign explanations you expect to encounter. For an after-hours access rule, the hypothesis might be that an interactive login outside a customer’s permitted schedule deserves review. That is more precise than “any activity at night is malicious.”

Record the fields the rule needs: user identity, event time, device, login outcome, and relevant customer context. Confirm those fields in received events. Choose the appropriate source categories and check the normalized field values; a Windows event number and an action such as login_success are different concepts.

Build a small test set with contrasting outcomes

• Expected match: an authorized test login outside the agreed schedule with the required identity and timing fields present.
• Expected non-match: an ordinary login within that schedule.
• Benign exception: an approved maintenance login whose context should change handling.
• Incomplete evidence: a record without a usable username or timestamp, to establish how the rule handles the gap.
• Tenant contrast: comparable activity in another customer with a different operating schedule.

These are illustrative tests, not customer results. Keep the test inputs, expected behavior, and rule version together. For Windows logons, Microsoft’s event 4624 reference helps explain why logon type and account context matter when constructing examples.

Investigate matches instead of counting them

In Findings, use the Beta view to review the rule’s stored findings and supporting evidence. A useful review asks why each match occurred: which condition matched, which entity was grouped, what surrounding activity explains it, and whether a responder would have had enough context to decide what to do.

Separate false positives caused by broad logic from those caused by missing context. A service account creating routine activity is different from a parser assigning the wrong username. Fixing the latter in the rule with a blanket exclusion can conceal the underlying data problem.

Also review expected matches that never appeared. Zero findings can mean a quiet environment, a disabled rule, the wrong source category, absent telemetry, or an incorrect condition. A negative test set alone cannot demonstrate detection coverage.

Tune narrowly and repeat the tests

Suppose the after-hours rule finds a scheduled backup account. Decide whether the relevant exception is the account, a particular host and time window, or a documented maintenance pattern. Prefer the narrowest exception that reflects the operational reason. Retest the original suspicious example so the exception does not silently remove the intended coverage.

Review aggregation, grouping, and cooldown settings as well as the match conditions. A rule may identify the right behavior yet create an impractical number of findings, or group unrelated users together. Check results per customer before treating an MSSP-wide definition as ready for everyone.

Make promotion an explicit decision

1. Verify prerequisites. Required events and fields are present in representative customer environments.
2. Review expected and unexpected results. Positive tests match, negative tests behave as intended, and remaining exceptions are understood.
3. Assign ownership. Name who will monitor the rule and investigate problems after release.
4. Check the destination. Understand which mysoc workflow and notifications will receive findings when Beta is cleared.
5. Activate in scope. Keep Active on, clear Beta for the intended scope, and verify a new controlled finding through the downstream path.

Distinguish moving a rule out of Beta from promoting a customer-owned rule to an MSSP Global rule. One changes forwarding behavior; the other changes ownership and the customers that inherit its definition. Shared-rule defaults and customer overrides deserve their own review.

After release, compare actual findings with the reviewed expectations. If the rule needs further observation without forwarding to mysoc, re-enable Beta; if evaluation itself must stop, disable Active. The next article follows the finding into playbook automation, where an eligible workflow and a completed action are also separate states.