Why alert feedback should be structured first

Alert quality cannot be guessed from delivery logs

Delivery logs prove that an alert moved through the machinery. The candidate was created, the writer ran, the request succeeded, the payload was rejected, or a retry eventually got the message into the downstream tool.

That matters. It just stops too early.

From the system’s side, everything can look clean. The scheduled job ran. The candidate was processed. The writer returned success. The alert exists in the business system. Every transport-level check says the machine did its part. The plumbing is green while the work being created is still bad.

The recipient may have seen something else entirely. The alert arrived two days late. It repeated a case already handled. It pointed at stale data. It gave too little context to act without opening three other systems. The message was delivered, but the alert still created low-quality work.

Closure time, expiry, writeback status, and repeated dedupe keys all help. They show movement around the alert. They still leave the important part unanswered: whether the alert was useful enough to deserve the interruption.

That answer has to become part of the data model. If the only record after delivery is transport data, tuning depends on reading comments later, remembering complaints from meetings, and treating the loudest recent example as evidence. Very scientific. Somewhere, a spreadsheet is already preparing to lie.

Free text is useful, but not enough

Free text has a place. A recipient can explain an edge case, add local context, describe why the current rule failed, or point at a condition the model does not understand yet. A note is especially useful when the alert is wrong in a way the existing categories do not cover.

The problem starts when free text becomes the primary feedback model. It’s hard to aggregate, hard to compare across alert types, and easy to leave empty. Even when the note is useful, someone has to interpret the language before the rule can change. Quality review becomes manual reading, weak classification, or straight up guesswork.

The first response should answer the part that can be measured. Was the alert relevant? Was the timing right? Was it a duplicate? Was the data wrong? Did it create work worth doing?

Those should be fields, not guesses extracted later from paragraphs.

A note can carry nuance after that. It can explain why the category was selected, describe the exception, or give the rule owner something specific to check. It should not carry the whole measurement job. When the category comes first, the system gets a usable signal even if the recipient has no time to write anything else.

Structured feedback creates a tuning dataset

Once responses have stable categories, delivery and usefulness become separate measurements. A rule can be delivered successfully and still be marked as bad timing. A technically correct query can still create work recipients reject. A variant can look harmless in volume and still generate most of the duplicate feedback inside one segment.

That gives rule review a better starting point than complaint volume. If recipients say alerts are noisy, the feedback can show where the noise comes from. If the query looks correct, feedback can show whether correctness is turning into useful action.

Most alert rules fail in boring ways. The threshold is slightly too low. The rule fires before the useful context arrives. The same case comes back after it was handled. The message reaches the right route on the wrong day. That needs evidence attached to the rule.

Feedback makes changes testable. A threshold change, new variant, different schedule, or revised message can be compared against prior response patterns. The system doesn’t have to wait for the loudest complaint to decide whether the change helped.

The categories should reflect real failure modes

Feedback categories should be boring. They should name failures the system can actually repair.

A positive category is still needed. “Relevant and well timed” sounds dull, but it tells the system the alert reached the right place at a useful moment and carried enough context for a response. Without that option, the feedback model only collects complaints.

Bad timing deserves its own answer because it usually points somewhere other than detection. The signal may be valid while the dispatch policy is wrong. Maybe the alert arrived too early, before the account had enough activity to judge. Maybe it arrived too late, after the useful window closed. Maybe Friday afternoon is a stupid time to create that kind of work. The repair may be in scheduling, freshness, gating, due-date policy, or suppression rules.

Duplicates need their own category too. When a recipient says the alert was already handled, the system likely failed to remember something. The dedupe key may be too narrow. The cooldown may be missing. Reminder logic may be pretending to be a new alert. Writeback may not be closing the loop. That feedback points toward state and repetition controls, not better message wording.

Low business value is a different failure. The rule may be detecting a real condition that does not deserve intervention. The signal might belong in a dashboard, a weekly review, or nowhere. Some true facts are not worth turning into work.

Bad data needs its own path because it damages trust in the alert contract. The issue may be source quality, late-arriving records, wrong joins, stale enrichment, weak identity, or a calculation that does not match the business definition. That should not be mixed with low value. A bad signal and a low-value signal need different repairs.

“Other with note” still belongs in the model, but it should stay an escape hatch. If it becomes the default response, the categories are wrong or the alert is asking a question the workflow has not modeled yet.

Feedback should stay attached to the alert that created it

When someone marks an alert as duplicate, the repair path needs the dedupe key, alert type, variant, recipient route, dispatch time, payload snapshot, and current state. Without that, “duplicate” is only a complaint.

Bad timing needs a different trail. The useful facts are the signal timestamp, schedule, due date, source freshness, owner availability if that exists, and any suppression or gating rule that ran before dispatch. The timing problem might be in the schedule. It might be in the data. It might be in the assumption that the alert should be sent at all.

Bad-data feedback needs the source fields, calculation snapshot, join identity, enrichment output, and the payload that was actually sent. If the visible message says one thing and the warehouse now says another, the snapshot keeps review from turning into archaeology.

The human-facing and machine-facing facts both matter. Wording affects interpretation. The snapshot explains the decision. The selected answer shows what happened at the workflow edge. The writeback result shows whether the answer completed the loop or died in the side-effect layer.

Attached context keeps feedback from drifting into opinion. It also prevents the usual review failure where everyone agrees there’s noise, but no one can tie the noise to the exact rule, payload, state transition, and dispatch decision that created it.

What you can measure over time

Once feedback is structured and attached to context, alert quality becomes visible over time.

The measurements should follow the repair path. Bad timing should sit next to dispatch hour, weekday, route, source freshness, and due windows. Duplicate feedback should sit next to dedupe keys, cooldown settings, reminder rules, and closure writeback. Bad-data feedback should point back to source models, enrichment dependencies, threshold families, identity rules, and the snapshot that produced the payload.

Response behavior becomes visible too. Some alerts get accepted quickly. Some sit open until expiry. Some close without enough answer detail. Some trigger writeback but never reach a closed state because the response options do not match the actual work. Some repeatedly need notes, which usually means the category model is missing a real case.

The uncomfortable measurements are often the useful ones. A rule may have a good relevance rate and still create too much work for one route. Another may look noisy overall and perform well inside one segment. A schedule may look reasonable on paper and fail during the part of the week where recipients are actually able to act.

Inspection matters as well. If rule quality cannot be measured, tuning becomes a personality contest between whoever built the rule and whoever receives the noise. Noise tends to win those arguments because it appears every day, while the reason for the rule is usually reconstructed after the damage is already visible.

How feedback changes the rules

Relevant and well-timed alerts can mostly stay as they are. They may still need clearer wording, a better payload section, or a more useful writeback option, but the rule is creating work that recipients accept as valid.

Bad timing should push changes in schedules, blackout windows, freshness gates, due windows, owner availability handling, or dispatch cadence. The signal may be fine while the timing policy is wrong.

Duplicate or already handled feedback should push changes in dedupe keys, cooldowns, open-alert checks, reminder idempotency, closure writeback, or parsed history. The system had enough evidence to know better and failed to use it.

Low-value feedback should push threshold changes, variant changes, segment exclusions, or removal from the alerting workflow. A signal can be true, interesting, and still not worth interrupting anyone over.

Incorrect-data feedback belongs near source models, enrichment queries, identity logic, late-arriving data, and payload snapshots. If the system cannot explain the calculation, the alert isn’t trustworthy enough to carry action.

Other with note should be reviewed periodically. If the same note pattern keeps appearing, the category model is missing something or the rule is creating work the current workflow cannot describe.

Feedback should eventually change thresholds, schedules, dedupe behavior, payloads, response options, writeback actions, or rule scope. When the same responses keep appearing and the rule does not move, the system is only recording the failure more neatly.