Start with a quantity and an error cost
Suppose a repair business wants to estimate labor hours when a job is booked. Define whether the target includes travel, waiting, rework, or only active repair time. Specify when the prediction is made and which booking fields are known then. A model trained on completed-job diagnostics would solve a different, easier problem.
Ask how estimates will be used. Underestimating a long job may disrupt the schedule; overestimating may leave unused capacity. Record the units and practical tolerance. An error of two hours has a different meaning for a ten-minute task and a two-day repair. This context should guide evaluation before you choose an algorithm.
Build constant predictions first
Fit a mean and a median baseline using training targets only. The mean is the best constant for minimizing training squared error, while the median minimizes training absolute error. Neither claim guarantees the better result on unseen jobs. Scikit-learn's DummyRegressor provides both strategies and can also predict a specified quantile or constant. [1](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html)
Evaluate these predictions on the same validation records as every later model. Keep their results visible in your experiment table. If a complex model cannot beat a constant prediction, first inspect target quality, feature availability, and split design. More hyperparameter searching may not address the real limitation.
Add the rule a practitioner already uses
A useful baseline can encode existing practice: median labor hours by repair category, with a global median fallback for unseen categories. Estimate category medians from training data, not the full dataset. Track how often the fallback is used so a strong average result does not hide poor coverage of new work.
If jobs arrive over time, compare on a later period. Seasonal demand and changing procedures may make a random split optimistic. Write the rule precisely enough for another person to reproduce it: which categories, minimum sample requirements, and fallback order. Avoid adding dozens of exceptions after examining validation failures, since that turns a baseline into an undocumented fitted model.
Use a small linear model as a diagnostic
Try a linear or regularized linear model using a few booking-time features, such as job category, equipment age, and reported symptom count. Handle missing numeric values and categorical encoding within a pipeline. The official common-pitfalls guide explains why transformations must be fitted on training data and applied consistently to held-out records. [2](https://scikit-learn.org/stable/common_pitfalls.html)
Check whether predictions can become negative and decide how to handle that limitation. Clipping may be operationally reasonable, but it changes the evaluated procedure and should be documented. Do not treat a coefficient as a causal effect: correlated features and the booking process can strongly affect its interpretation.
Read MAE, RMSE, and residuals together
Mean absolute error summarizes the average absolute miss in labor-hour units. Root mean squared error also uses those units but weights large errors more strongly. Report both when large scheduling failures matter. R-squared can provide another view, but it is not a percentage accuracy score and can be negative on held-out data.
Plot residuals against predicted hours and inspect errors by repair category and job size. A model that performs well overall may systematically underestimate complex repairs. Avoid percentage errors when targets are zero or close to zero without explicitly addressing the denominator problem. Include the number of evaluated jobs, time range, and a few representative misses alongside aggregate metrics.
Decide whether the improvement is worth operating
Create a comparison table containing the constant baselines, category rule, linear model, evaluation period, MAE, RMSE, and practical limitations. Use paired resampling or repeated appropriate splits when feasible to examine how sensitive the difference is to sample selection. A tiny average improvement may not justify extra dependencies or harder explanations.
Finish by replaying a sample day's scheduling decisions using each method. Ask whether predictions improve planning and where a human override remains necessary. Save the training recipe and a versioned evaluation dataset reference. Your next experiment should respond to a diagnosed failure, such as long-job underestimation, rather than merely escalating to a more fashionable model family.
Sources & further reading
Last editorial review: 27 October 2026. Examples are educational, not live employer data or guaranteed outcomes.