Skip to content
LearnMLByDoing
Free

6.3 Training pipelines, feature stores and serving

30 minProintermediateGCP-PMLEAI-103MLA-C02CompendiumHOML

A model in production is a pipeline that rebuilds it reproducibly, a feature store that feeds training and serving the same numbers, and a serving tier sized so that queueing does not eat the latency budget.

01From notebook to pipeline

A notebook trains a model once. Production needs to train it again next week, on new data, and get a model you can explain. That requires every input to be pinned: the data (an immutable snapshot, here ds-0412), the code (a commit, git a1f3c9), the environment (a lockfile or container image digest) and the randomness (seeds for splits, initialisation and sampling). Change none of them and the run reproduces; change one and you know exactly what changed.

The back row of the viewport is the pipeline as a directed acyclic graph: ingest, validate, transform, train, evaluate, gate, register, deploy. Each stage reads artifacts and writes new ones; the cubes travelling along the edges are those artifacts (cyan for data and features, emerald for the model, amber for metrics). An orchestrator runs the stages in dependency order, retries failures, and records every input and output.

Watch two runs. Run 1 goes all the way to the gate and is rejected (rose). Run 2 is triggered by a training-config change: the first three stages finish instantly in amber because their inputs did not change, so the orchestrator reuses their cached outputs. Only train onwards re-executes, the gate promotes, and the model is registered as v4 and deployed to the replicas on the right.

cache key(s)=hash⁡(codes, paramss, hash⁡(inputss))\text{cache key}(s)=\operatorname{hash}\big(\text{code}_s,\ \text{params}_s,\ \operatorname{hash}(\text{inputs}_s)\big)
A stage is skipped when its key matches a previous run. A changed learning rate alters the key of train and everything downstream, not of ingest.

Professional Machine Learning Engineer exam guide, Google Cloud, Automating and orchestrating ML pipelines

Hands-On Machine Learning, Aurélien Géron, Ch. 19

02Validation and evaluation gates

A pipeline that runs unattended needs automatic stop conditions. There are two families. Data validation runs before training: the schema (columns, types), value ranges, null rates and category sets are checked against expectations, and the new snapshot's distributions are compared with a reference. A failed check stops the run before it wastes compute or trains on a broken join.

The evaluation gate runs after training and decides promotion. The candidate is scored on a frozen evaluation set and compared with the champion, the model now serving. The score here is the AUC, the area under the ROC curve from lesson 2.8: the probability that the model ranks a random positive example above a random negative one (0.5 is guessing, 1 is perfect). Here the rule is

promote  ⟺  AUCcand ≥ AUCchamp+δ=0.874+0.005=0.879\tok{gate}{\text{promote}}\iff \mathrm{AUC}_{\text{cand}}\ \ge\ \mathrm{AUC}_{\text{champ}}+\delta = 0.874+0.005 = 0.879
Plus absolute floors (no slice below its minimum) and serving constraints such as model size and latency.

Run 1's candidate scores 0.861 and is rejected: its cube drops off the gate and nothing downstream runs. Run 2's candidate scores 0.889 and is promoted. The margin δ\delta guards against promoting noise: two models whose AUCs differ by less than the evaluation set's sampling error are not meaningfully different, and every swap carries operational risk.

AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services, Deployment and orchestration of ML workflows

Maths, CS & AI Compendium, Henry Ndubuaku, ML systems: testing and validation

03Feature stores and point-in-time joins

Features are computed once by a shared transformation f(x)f(x) (the emerald node, front left) and written to two stores. The offline store keeps history: every value with the timestamp at which it became known, in a columnar layout built for scans and joins when assembling training sets. The online store keeps only the latest value per entity key, for millisecond lookups at request time. In the viewport the offline block is entities (columns) by days (rows); the online store is one cell per key.

training set from the offline store[,]

One row per labelled event, assembled by joining each event to feature values as they were at the event time. The online store holds only an [entities × F] table of latest values.

Start with one customer before any notation. In the offline store, customer u2 has tickets_7d rows stamped on days 1, 3, 4, 5 and 8, with values 0, 0, 1, 1 and 6. u2 cancels on day 6, so day 6 is the event whose label (churned) we train on. The model must learn from what was known on day 6: the latest row at or before day 6 is day 5, value 1. A plain join on the customer key returns the latest row in the table today, day 8, value 6, and those extra tickets are the cancellation calls u2 made after churning. The feature now contains the label.

Loading figure
Three customers from the viewport's offline store; the dots are feature rows with their tickets_7d value, the cyan bars the event times (drag them). Emerald is the joined row, rose a row from after the event. Switch the rule to naive: latest and all three customers join a future row (leaked rows: 3). Switch to strict τ < t: u3's event on day 7 skips the row stamped on day 7 itself and falls back to day 6, so it loses the newest legitimate value.

In general, a labelled event ii has an entity eie_i, a time tit_i and a label yiy_i. The correct training feature is the one the model would have seen if it had been asked at tit_i:

Why a naive join leaks the label1/4
  1. 1
    xi=f(ei, τ∗),τ∗=max⁡{τ∈Tei: τ≤ti}x_i = f\big(e_i,\ \tau^{*}\big),\qquad \tau^{*}=\max\{\tau\in T_{e_i}:\ \tok{pit}{\tau\le t_i}\}

    T_e is the set of timestamps of entity e's feature rows. Point-in-time correctness: only rows known at or before the event.

Hover τ≤t\tok{pit}{\tau\le t}: an as-of plane sweeps up through the offline store. For each entity column the joined row (emerald) is the highest one at or below the plane; rows above it are the future and turn rose. Equality counts: a row stamped exactly at tit_i was available at tit_i, provided the timestamp records when the value became available, not when the underlying activity happened.

Professional Machine Learning Engineer exam guide, Google Cloud, Data preparation and feature engineering

Maths, CS & AI Compendium, Henry Ndubuaku, ML systems: feature stores

The rest of this lesson is in Pro

  • 04Training-serving skew
  • 05Serving patterns: batch, online, streaming, edge
  • 06Capacity, Little's law and tail latency
  • 07Rollouts, CI/CD/CT and lineage
  • Python challenge: Leak-free features and a capacity model
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is training-serving skew?
Training-serving skew is a difference between the feature values a model was trained on and the values it receives in production. It usually comes from computing a feature twice with two implementations, from stale online values, or from preprocessing that lives in a notebook instead of the model artifact. It does not cause errors, only wrong predictions, so it has to be detected by logging serving features and comparing them with the training path.
What is a point-in-time join in a feature store?
A point-in-time join, or as-of join, attaches to each labelled event the latest feature value for the same entity whose timestamp is at or before the event time. It reproduces what the model would have known when the prediction was needed. Joining the current value instead lets information from after the event leak into training, which inflates offline metrics and fails in production.
When should I use batch prediction instead of an online endpoint?
Use batch prediction when predictions can be computed ahead of time and the set of keys is known, for example nightly scores for every customer, and when a few hours of staleness is acceptable. Use an online endpoint when the inputs only exist at request time or must be seconds fresh, and size it for peak load with headroom because latency grows sharply as utilisation approaches one.

Next in Production MLOps and Certification Prep

6.4LLM serving: KV cache, batching and speculative decodingWhy decoding is memory-bound, how the KV cache grows, and how paged memory, continuous batching and draft models raise throughput.

Sources

  • Professional Machine Learning Engineer exam guide, Google Cloud
  • Developing AI Apps and Agents on Azure (study guide), Microsoft
  • AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Hands-On Machine Learning, Aurélien Géron