Serving and release

Required

• Answer with a fallback when the model fails or is slow

  MLO-28

  Give every model call a timeout and a written fallback answer, such as the rule-based baseline or a safe default. The feature keeps working when the model server is down, slow or rejects the input.

Recommended

• Compute features in one place for training and serving

  MLO-23

  Define each feature once, in one module or feature store, and use that definition both to build training data and to answer live requests. Never write the same feature twice in two languages or two code paths.

• Check every input against what the model was trained on

  MLO-24

  Validate each request against a schema saved with the model version. It lists the type, the allowed range or categories and whether null is allowed, for every feature. Send a request that fails to the fallback, and count it.

• Log the features and prediction of every request

  MLO-25

  For each prediction, log a prediction ID, the model version, the feature values, the score, the decision and the time. Store the log apart from app logs, under the retention set for it.

• Run a new model in shadow before it decides

  MLO-26

  Before a new model makes any decision, run it next to the current one on live traffic for a set period, such as seven days. Log both scores and act only on the current model's.

• Release models through a canary that compares predictions

  MLO-27

  Send a new model a small share of traffic first, and widen it in steps. Roll back by itself when the error rate, the latency, the share of positive decisions or an early business metric moves past its limit.