Designing a Machine Learning Pipeline

An ML pipeline automates the flow from raw data to production predictions with monitoring and feedback loops.

ML Pipeline Stages

Data Collection
    ↓
Data Validation
    ↓
Feature Engineering
    ↓
Model Training
    ↓
Model Evaluation
    ↓
Model Registry
    ↓
Model Serving
    ↓
Monitoring + Feedback

Feature Store

Centralized repository for ML features:

Offline Store (historical):
  Feature: user_purchase_count_30d
  Source: Spark job on data warehouse
  Storage: Parquet on S3

Online Store (low latency):
  Same feature, precomputed for active users
  Storage: Redis (<5ms lookup)

Feature consistency: Train + serve use same feature definitions

Training Pipeline

# MLflow experiment tracking
with mlflow.start_run():
  # Load features
  X_train, y_train = feature_store.get_training_data()

  # Train
  model = XGBClassifier(n_estimators=100)
  model.fit(X_train, y_train)

  # Evaluate
  auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
  mlflow.log_metric('auc', auc)
  mlflow.sklearn.log_model(model, 'model')

Model Registry

Models versioned and staged:
Staging → Challenger (tested against production)
Production → Champion (serving live traffic)
Archived → Old versions

Promotion requires:
  AUC improvement > 0.5%
  No regression on key segments
  Shadow testing passed

Model Serving

Online inference: REST API, <100ms SLA
  → Triton Inference Server / TorchServe
  → Horizontal scaling + load balancing

Batch inference: Nightly score all users
  → Spark + model → Predictions to DB

Model Monitoring

Data drift:    Input feature distribution shifted?
Concept drift: Relationship between features and target changed?
Prediction drift: Output distribution changed?

Alert if drift detected → Trigger retraining

Conclusion

Feature stores solve train-serve skew. MLflow handles experiment tracking. Continuous drift monitoring triggers retraining to keep models fresh in production.