Predictive Maintenance With Machine Learning: Choosing a Failure Target and Validating Early Warnings

An industrial software provider wants to add failure warnings to equipment dashboards that already collect telemetry. Its CTO and the customer's reliability lead need to agree on what the feature should predict and what evidence would justify using it. Predictive maintenance uses equipment condition and history to anticipate maintenance needs. A useful warning also gives someone enough time to investigate and act.

This guide follows a hypothetical project monitoring conveyor motor bearing failures. It shows how to choose a failure target, establish usable sensor and event history, and compare machine learning with an existing threshold rule. The worked example uses invented data. By the end, you can define a bounded pilot and a reproducible way to evaluate its early warnings.

Key takeaways

• A specific failure mode determines the sensor history, event labels, and operating context needed to evaluate a model.
• The time required to investigate and act defines which warnings are useful. Detection close to failure needs its own category.
• Count individual failures and grouped alert episodes under explicit rules. Repeated notifications can increase workload while referring to the same event.
• Compare ML with the current threshold rule on the same history. Use the resulting detection and investigation tradeoff to choose a limited next step.

Choose a failure target and a useful warning window

Match the task to the available failure history

Start with one component and a failure mode that maintenance specialists can confirm. “Bearing failure” needs an agreed operational definition, including the evidence that distinguishes it from another cause of a conveyor stop.

MathWorks documentation for R2026b, reviewed on November 13, 2026, distinguishes anomaly detection, fault detection, and remaining useful life estimation. For pilot scoping, separate these practical questions:

An anomaly score establishes a deviation. Its connection to the target failure still needs validation. RUL estimation also requires evidence that the available histories support estimating progression toward an outcome.

Set the time needed to investigate and act

In the conveyor scenario, assume the target is a confirmed bearing failure within the next 72 hours. Assume maintenance needs at least 24 hours to investigate and arrange action. A first warning 24–72 hours before failure is timely; one arriving eight hours before failure is late.

These are scenario assumptions. Giannoulidis et al., published March 28, 2024, discuss prediction horizons and a buffer for action in a cold-forming press study. Set your own boundaries with the reliability lead using actual response constraints.

Build a trustworthy sensor and event history

Reconcile failures, maintenance, and operating context

Join telemetry and maintenance records through stable asset and component identifiers. Track component replacements so that measurements from successive bearings remain distinguishable. Normalize timestamps and preserve both event time and the time a record became available.

A maintenance report completed after a breakdown can confirm the outcome used for labeling. Its diagnosis was unavailable to an earlier model score. Historical features must contain only information available at that scoring time. Otherwise, replay gives the model knowledge it would lack during operation.

Give confirmed target failures, planned replacements, and stops caused by other faults separate statuses. Ask a maintenance specialist to review ambiguous event descriptions before they become labels. Record uncertainty when the evidence cannot establish the selected failure mode.

Observation ending without failure provides limited information about what happens next. Keep a prediction window unresolved when follow-up ends before its outcome can be established. A planned replacement can also end observation of the component's natural progression.

Preserve chronology and operating episodes

Use earlier periods for training, a later validation period for configuration, and a final untouched test period. Account for feature lookback and future outcome windows at each boundary: a training label must be knowable before the next evaluation phase begins. Fit preprocessing using the training data.

Keep overlapping windows associated with their underlying failure event and operating episode. Randomly distributing near-identical windows across splits can expose the same event during training and testing. If deployment includes unseen equipment, hold out assets separately to test that transfer.

The split should reproduce the intended deployment. Giannoulidis et al. demonstrate the importance of chronological, episode-aware evaluation under changing operating conditions. Their reported performance belongs to their press dataset.

Establish the baseline and select a first model

Preserve the current threshold rule as the baseline, including its persistence requirement and reset behavior. Also document the existing maintenance process: who sees a warning, how investigations happen, and what triggers intervention. This is the operational control throughout the pilot.

With confirmed failures and useful preceding measurements, start by testing a simple supervised model on interpretable sensor summaries and operating context. With sparse labels, an anomaly detector trained on reviewed normal-operation data can supply investigation candidates. Its evaluation still needs confirmed outcomes. Deep learning and RUL become candidates when the available history and target justify their requirements.

Give the baseline and ML the same available inputs, test period, eligibility rules, and response assumptions. Tune thresholds, persistence, and repeat grouping on validation data, then freeze them before testing. Record the configurations so another engineer can reproduce both alert streams. Select a candidate by the maintenance tradeoff it produces on validation data.

External development can include this broader delivery work. As an illustration of commissioned scope and deployment scale, DataArt's conveyor project covered platform design, development, and deployment, including data collection, component models, and warnings. DataArt reports four operational North American facilities after a year. This supports the delivery example; it supplies no evidence for the comparison calculated below.

Replay warnings against individual failure events

Convert repeated notifications into alert episodes

For this article's illustrative protocol, group consecutive signals for the same component and failure mode when gaps are at most six hours. Preserve the episode's start time. A repair or confirmed failure closes it; a later signal starts a new episode. Apply the same configuration to baseline and ML.

Match each episode to the next confirmed failure of the same component and type within 72 hours of its start. Assign each episode to at most one failure and count each failure once. If several episodes match a failure, use the earliest start within that window to classify its warning. Retain the raw notifications so reviewers can trace how grouping changed investigation workload.

An episode beginning before the useful window keeps its original start. Repeats inside the window cannot make that long-running episode timely. A genuinely new episode after a gap exceeding six hours receives its own start time. This distinction prevents persistent warnings from gaining credit through repeated delivery.

Report timely detection and investigation burden

Use these explicitly defined measures:

• Timely failure recall: failures with a qualifying first episode warning 24–72 hours before failure, divided by all eligible confirmed failures.
• Late-only detection: failures whose earliest matched episode starts less than 24 hours before failure, with no timely warning.
• False alert episodes: episodes with no selected failure in the following 72 hours, complete follow-up, and no intervention that could have changed the outcome.
• False episodes per monitored asset-week: false episodes divided by total eligible equipment observation time in weeks.

Exclude incomplete follow-up from confirmed false alarms. Record episodes beginning more than 72 hours before a subsequent failure separately; they earn no detection credit for that failure and may qualify as false episodes under this protocol. A later, distinct episode can qualify for that failure. Keep unresolved and intervention-affected episodes visible in reporting.

An inspection or planned replacement after a warning can prevent observation of a later failure. Review findings and maintenance feedback before classifying that warning. Absence of a subsequent failure alone cannot establish a false alert.

Giannoulidis et al. examine event-aware evaluation, timing boundaries, and KPI limitations. The grouping rule and formulas here are the article's illustrative protocol, independently specified for the conveyor example.

Work through the conveyor example and choose the next step

Compare the same six failures under two alert rules

Illustrative example with invented data.

The held-out history contains 20 eligible monitored asset-weeks and six confirmed bearing failures with complete follow-up. Every number below is fixed for this arithmetic example. The table gives the first episode warning associated with each failure; there are no earlier episodes for these failures.

ML catches F1–F4 within the useful window: 4/6 = 66.7% timely failure recall. The baseline catches F1 and F4 within that window: 2/6 = 33.3%. F5 is late-only for ML, and F6 is missed. The baseline has two late-only detections, F2 and F5, and two missed failures, F3 and F6.

The late warnings remain visible because they show detection with insufficient response time under the agreed assumptions. Counting them as timely would change the question the pilot was designed to answer. Repeat notifications for F1 or any already counted event add no further detected failures.

Separately, ML produces 14 raw false notifications. The frozen grouping rule combines these into six false episodes. The baseline produces four false episodes. All these episodes have complete follow-up, no target failure in the following 72 hours, and no intervention that could have altered the outcome.

ML therefore produces 6/20 = 0.30 false episodes per monitored asset-week. The baseline produces 4/20 = 0.20. Both use the same eligible observation denominator, excluding periods where monitoring or outcome determination is incomplete. The comparison shows additional timely detections accompanied by additional false-episode investigation workload. Raw notification counts would describe a different burden.

Resolve the scenario with a bounded decision

The conveyor team chooses an observational shadow pilot on the same component type. The current maintenance process continues. An operator reviews the signals, records inspection findings, and documents any intervention under that process.

The reason is concrete: the replay suggests useful additional warnings alongside extra investigation work. Six failures leave substantial uncertainty about future performance. Before expanding coverage, the team needs live evidence on timeliness, alert volume, confirmed findings, and review capacity against agreed acceptance criteria. These six events establish the arithmetic; further observations are required to establish stable superiority or savings.

Connect each warning to an owned maintenance response

Deliver a reviewable warning record

A warning should carry the asset and component, target failure mode, scoring timestamp, input freshness, episode history, relevant readings, model version, and alert-rule configuration. Assign an owner and an expected review deadline. Those details let the reviewer reconstruct what triggered the episode and how much response time remains.

Keep the progression explicit: warning → reviewed → inspected → outcome recorded. Each transition needs a timestamp and a responsible person. Separate scoring frequency and notification delivery latency from warning lead time: frequent scores provide no assurance that the underlying failure can be predicted early enough.

Expose telemetry gaps, stale scores, and failed delivery to the process owner. Keep automatic equipment shutdown outside this analytical pilot's scope.

Microsoft Fabric documentation, reviewed on November 13, 2026, illustrates connecting equipment telemetry, maintenance context, and notifications. It is an integration reference; the pilot's actual capabilities depend on its implementation.

Capture outcomes and monitor changes

Log inspection findings, repairs, replacements, and reasons for closing warnings. Preserve uncertain outcomes for later review. Track operating modes, input quality, alert volume, timely detections, and confirmed findings as outcomes become available. Mad Devs' guide to monitoring models in production explains the broader monitoring disciplines.

Agree a prospective evaluation design and maintenance outcomes separately if the next question concerns intervention effectiveness. Retrospective replay evaluates warnings. Live maintenance-effectiveness claims require evidence about actions and outcomes under the agreed design.

Conclusion

The conveyor team chooses a shadow pilot to investigate additional timely warnings while measuring the extra review burden. That bounded decision follows from an explicit target, usable history, and a reproducible comparison.

For your project, choose one failure mode, reconcile event records with an equipment specialist, and agree on a useful warning window. Then specify how episodes, failures, incomplete follow-up, and interventions will be counted against the baseline. The result is a limited pilot scope with an acceptance method. Companies needing joint work on data readiness, validation, and workflow integration can discuss that scope with Mad Devs' machine learning development team.