Machine Learning Engineering

Real-world Production ML System

The common architecture of the real-world production machine learning system is shown below:

real-world production ML system

Types of the Training Process

Static Model – Trained Offline

  • We train the model exactly once and then use that trained model for a while.
  • Easy to build and test - use batch train & test, iterate until good.
  • Still requires monitoring of inputs.
  • Easy to let this grow stale.

Dynamic Model – Trained Online

  • Continue to feed in training data over time, regularly sync out updated version.
  • Use progressive validation rather than batch trining & test.
  • Needs monitoring, model rollback & data quarantine capabilities.
  • Will adapt to changes, staleness issues avoided.
  • Data is continually entering the system and we’re incorporating that data into the model through continuous updates.


  • Static models are easier to build and test.
  • Dynamic models adapt to changing data. The world is a highly changeable place.

Types of Inference Process

Offline Inference

  • Meaning that you make all possible predictions in a batch, using a MapReduce or something similar. You then write the predictions to an SSTable or Bigtable, and then feed these to a cache/lookup table.
  • Upside: don’t need to worry much about cost of inference.
  • Upside: can likely use batch quota.
  • Upside: can do post-verification on predictions on data before pushing.
  • Downside: can only predict things we know about – bad for long tail.
  • Downside: update latency likely measured in hours or days.

Online Inference

  • Meaning that you predict on demand, using a server.
  • Upside: can predict any new item as it comes in – great for long tail.
  • Downside: compute intensive, latency sensitive – may imit model complexity.
  • Downside: monitoring needs are more intensive.

Data Dependencies

  • Reliability
  • Versioning
  • Necessity
  • Correlations
  • Feedback Loops


Types of Bias

  • Reporting Bias
  • Automation Bias
  • Selection Bias
    • Coverage bias
    • Non-response bias
    • Sampling bias
  • Group Attribution Bias
    • In-group bias
    • Out-group bias
  • Implicit Bias

Effective Machine Learning Guidelines

  • Keep your first model simple.
  • Focus on ensuring data pipeline corrections.
  • Use a imple, observable metric for training & evaluation.
  • Own and monitor your input features.
  • Treat your model configuration as code: review it, check it in.
  • Write down the results of all experiments, especially “failures”.

Colab: Intro to Fairness

Note: Cover Picture