Machine Learning Pipeline - Multi-Agent MLOps Orchestration

Design and implement a complete ML pipeline for: $ARGUMENTS

Use this skill when

• Working on machine learning pipeline - multi-agent mlops orchestration tasks or workflows

• Needing guidance, best practices, or checklists for machine learning pipeline - multi-agent mlops orchestration

Do not use this skill when

• The task is unrelated to machine learning pipeline - multi-agent mlops orchestration

• You need a different domain or tool outside this scope

Instructions

• Clarify goals, constraints, and required inputs.

• Apply relevant best practices and validate outcomes.

• Provide actionable steps and verification.

• If detailed examples are required, open resources/implementation-playbook.md.

Thinking

This workflow orchestrates multiple specialized agents to build a production-ready ML pipeline following modern MLOps best practices. The approach emphasizes:

• Phase-based coordination: Each phase builds upon previous outputs, with clear handoffs between agents

• Modern tooling integration: MLflow/W&B for experiments, Feast/Tecton for features, KServe/Seldon for serving

• Production-first mindset: Every component designed for scale, monitoring, and reliability

• Reproducibility: Version control for data, models, and infrastructure

• Continuous improvement: Automated retraining, A/B testing, and drift detection

The multi-agent approach ensures each aspect is handled by domain experts:

• Data engineers handle ingestion and quality

• Data scientists design features and experiments

• ML engineers implement training pipelines

• MLOps engineers handle production deployment

• Observability engineers ensure monitoring

Phase 1: Data & Requirements Analysis

Deliverables:

-

Data source audit and ingestion strategy:

• Source systems and connection patterns

• Schema validation using Pydantic/Great Expectations

• Data versioning with DVC or lakeFS

• Incremental loading and CDC strategies

-

Data quality framework:

• Profiling and statistics generation

• Anomaly detection rules

• Data lineage tracking

• Quality gates and SLAs

-

Storage architecture:

• Raw/processed/feature layers

• Partitioning strategy

• Retention policies

• Cost optimization

Provide implementation code for critical components and integration patterns.

Deliverables:

-

Feature engineering pipeline:

• Transformation specifications

• Feature store schema (Feast/Tecton)

• Statistical validation rules

• Handling strategies for missing data/outliers

-

Model requirements:

• Algorithm selection rationale

• Performance metrics and baselines

• Training data requirements

• Evaluation criteria and thresholds

-

Experiment design:

• Hypothesis and success metrics

• A/B testing methodology

• Sample size calculations

• Bias detection approach

Include feature transformation code and statistical validation logic.

Phase 2: Model Development & Training

Build comprehensive training system:

-

Training pipeline implementation:

• Modular training code with clear interfaces

• Hyperparameter optimization (Optuna/Ray Tune)

• Distributed training support (Horovod/PyTorch DDP)

• Cross-validation and ensemble strategies

-

Experiment tracking setup:

• MLflow/Weights & Biases integration

• Metric logging and visualization

• Artifact management (models, plots, data samples)

• Experiment comparison and analysis tools

-

Model registry integration:

• Version control and tagging strategy

• Model metadata and lineage

• Promotion workflows (dev -> staging -> prod)

• Rollback procedures

Provide complete training code with configuration management.

Focus areas:

-

Code quality and structure:

• Refactor for production standards

• Add comprehensive error handling

• Implement proper logging with structured formats

• Create reusable components and utilities

-

Performance optimization:

• Profile and optimize bottlenecks

• Implement caching strategies

• Optimize data loading and preprocessing

• Memory management for large-scale training

-

Testing framework:

• Unit tests for data transformations

• Integration tests for pipeline components

• Model quality tests (invariance, directional)

• Performance regression tests

Deliver production-ready, maintainable code with full test coverage.

Phase 3: Production Deployment & Serving

Implementation requirements:

-

Model serving infrastructure:

• REST/gRPC APIs with FastAPI/TorchServe

• Batch prediction pipelines (Airflow/Kubeflow)

• Stream processing (Kafka/Kinesis integration)

• Model serving platforms (KServe/Seldon Core)

-

Deployment strategies:

• Blue-green deployments for zero downtime

• Canary releases with traffic splitting

• Shadow deployments for validation

• A/B testing infrastructure

-

CI/CD pipeline:

• GitHub Actions/GitLab CI workflows

• Automated testing gates

• Model validation before deployment

• ArgoCD for GitOps deployment

-

Infrastructure as Code:

• Terraform modules for cloud resources

• Helm charts for Kubernetes deployments

• Docker multi-stage builds for optimization

• Secret management with Vault/Secrets Manager

Provide complete deployment configuration and automation scripts.

Kubernetes-specific requirements:

-

Workload orchestration:

• Training job scheduling with Kubeflow

• GPU resource allocation and sharing

• Spot/preemptible instance integration

• Priority classes and resource quotas

-

Serving infrastructure:

• HPA/VPA for autoscaling

• KEDA for event-driven scaling

• Istio service mesh for traffic management

• Model caching and warm-up strategies

-

Storage and data access:

• PVC strategies for training data

• Model artifact storage with CSI drivers

• Distributed storage for feature stores

• Cache layers for inference optimization

Provide Kubernetes manifests and Helm charts for entire ML platform.

Phase 4: Monitoring & Continuous Improvement

Monitoring framework:

-

Model performance monitoring:

• Prediction accuracy tracking

• Latency and throughput metrics

• Feature importance shifts

• Business KPI correlation

-

Data and model drift detection:

• Statistical drift detection (KS test, PSI)

• Concept drift monitoring

• Feature distribution tracking

• Automated drift alerts and reports

-

System observability:

• Prometheus metrics for all components

• Grafana dashboards for visualization

• Distributed tracing with Jaeger/Zipkin

• Log aggregation with ELK/Loki

-

Alerting and automation:

• PagerDuty/Opsgenie integration

• Automated retraining triggers

• Performance degradation workflows

• Incident response runbooks

-

Cost tracking:

• Resource utilization metrics

• Cost allocation by model/experiment

• Optimization recommendations

• Budget alerts and controls

Deliver monitoring configuration, dashboards, and alert rules.

Configuration Options

• experiment_tracking: mlflow | wandb | neptune | clearml

• feature_store: feast | tecton | databricks | custom

• serving_platform: kserve | seldon | torchserve | triton

• orchestration: kubeflow | airflow | prefect | dagster

• cloud_provider: aws | azure | gcp | multi-cloud

• deployment_mode: realtime | batch | streaming | hybrid

• monitoring_stack: prometheus | datadog | newrelic | custom

Success Criteria

-

Data Pipeline Success:

• < 0.1% data quality issues in production

• Automated data validation passing 99.9% of time

• Complete data lineage tracking

• Sub-second feature serving latency

-

Model Performance:

• Meeting or exceeding baseline metrics

• < 5% performance degradation before retraining

• Successful A/B tests with statistical significance

• No undetected model drift > 24 hours

-

Operational Excellence:

• 99.9% uptime for model serving

• < 200ms p99 inference latency

• Automated rollback within 5 minutes

• Complete observability with < 1 minute alert time

-

Development Velocity:

• < 1 hour from commit to production

• Parallel experiment execution

• Reproducible training runs

• Self-service model deployment

-

Cost Efficiency:

• < 20% infrastructure waste

• Optimized resource allocation

• Automatic scaling based on load

• Spot instance utilization > 60%

Final Deliverables

Upon completion, the orchestrated pipeline will provide:

• End-to-end ML pipeline with full automation

• Comprehensive documentation and runbooks

• Production-ready infrastructure as code

• Complete monitoring and alerting system

• CI/CD pipelines for continuous improvement

• Cost optimization and scaling strategies

• Disaster recovery and rollback procedures

Limitations

• Use this skill only when the task clearly matches the scope described above.

• Do not treat the output as a substitute for environment-specific validation, testing, or expert review.

• Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.