AI-operations

Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/

A high-throughput and memory-efficient inference and serving engine for LLMs

Open-Source Personal Cloud OS for Always-On Agents

Workflow Engine for Kubernetes

Machine Learning Pipelines for Kubeflow

The Open Source Feature Store for AI/ML

An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

Always know what to expect from your data.

🐢 Open-Source Evaluation & Testing library for LLM Agents

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

Label Studio is a multi-type data labeling and annotation tool with standardized output format

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

A framework for efficient model inference with omni-modality models

The absolute trainer to light up AI agents.

Python SDK for healthcare AI — typed, validated FHIR tools for agents, real-time EHR connectivity, production deployment ✨ 🏥

Dynamic, resilient AI orchestration. Coordinate data, models, and compute as you build AI workflows.

Ship AI Agents to Google Cloud in minutes, not months. Production-ready templates with built-in CI/CD, evaluation, and observability.

Weaviate is an open-source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database​.

Community maintained hardware plugin for vLLM on Huawei Ascend

OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

🏕️ Reproducible development environment for humans and agents

🔥 Real-time NVIDIA GPU dashboard

LUPINE is a GPU over IP bridge allowing GPUs on remote machines to be attached to CPU-only machines.

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

The simplest way to serve AI/ML models in production

Deploy any AI model, agent, database, RAG, and pipeline locally or remotely in minutes

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Open Source Continuous Inference Benchmark Research Platform — Kimi K3 2.8T, MiniMax M3, DeepSeekv4, GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 & soon™ TPUv6e/v7/Trainium2/3 | 开源持续推理基准研究平台 — Kimi K3-Code、MiniMax M3、DeepSeekv4、GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72，即将推出™ TPUv6e/v7/Trainium2/3

A personal research and development (R&D) lab that facilitates the sharing of knowledge.

Open Source Data Annotation & Labeling Tools

Serverless LLM Serving for Everyone.

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

Learn to build your Second Brain AI assistant with LLMs, agents, RAG, fine-tuning, LLMOps and AI systems techniques.

✨ Build a machine learning model from a prompt

LightlyStudio - The Unified Data Platform for Multimodal ML

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.

skops is a Python library helping you share your scikit-learn based models and put them in production

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

Agent traces you can run, not just read.

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind one endpoint.

SmarterRouter: An intelligent LLM gateway and VRAM-aware router for Ollama, llama.cpp, and OpenAI. Features semantic caching, model profiling, and automatic failover for local AI labs.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

Historical Ultralytics HUB repository — HUB shut down on July 31, 2026 and was replaced by Ultralytics Platform.

An AI-native OS. Models run on your own hardware, nodes federate peer-to-peer with no broker, and the API is OpenAI-compatible. Boots, has its own Wayland compositor, and runs on 8GB. Apache 2.0.

Run more RL experiments. Wait less for GPUs.

The missing bridge between your ML models and your AI agents.

Hyperparameter optimization and feature selection for scikit-learn using evolutionary algorithms. A modern alternative to GridSearchCV and RandomizedSearchCV.

Extensible Python SDK for developing Flyte tasks and workflows. Simple to get started and learn and highly extensible.

Additional packages (components, document stores and the likes) to extend the capabilities of Haystack

Build MLOps Pipelines in Minutes

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

Serve the home! Inference stack for your Nvidia DGX Spark aka the Grace Blackwell AI supercomputer on your desk. Mostly vLLM based for now and single-spark. For the not-so-rich buddies. If you want latest/in-testing, look at the branches

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.

ZKE（Z Kubernetes Engine）：AI 原生的 Kubernetes 云操作环境，桌面式多集群控制台加受控 AIOps Agent，基于 Server + Agent 与 QUIC/mTLS，适用于私有云、混合云及边缘环境

BentoDiffusion: A collection of diffusion models served with BentoML

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

Free MLOps course from DataTalks.Club. Register here 👇🏼 to get notified about the next cohort

Retired Python SDK for the shut-down Ultralytics HUB API, kept for reference; build new integrations on the Ultralytics Platform REST API

An orchestration runtime for multi-agent AI systems. Declare agents, tools, and policies as YAML; Orloj schedules, executes, routes, and governs them for production-grade operation.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

OpenMLDB is an open-source machine learning database that provides a feature platform computing consistent features for training and inference.

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式

🤖 MLE-Agent: Your intelligent companion for seamless AI engineering and research. 🔍 Integrate with arxiv and paper with code to provide better code/research plans 🧰 OpenAI, Anthropic, Gemini, Ollama, etc supported. :fireworks: Code RAG

A framework for generating realistic LLM serving workloads

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

Immersive workshop showcasing the remarkable potential of integrating SoTA foundation models to enhance product experiences and streamline backend workflows. Leverages Microsoft's Copilot stack, Microsoft Agent Framework and Azure primitives to offer an engaging and comprehensive introduction to AI-infused app development and deployment

Fast, concurrent inference for YuE2. Same model and recipe: 1.68× faster per song, 3.31× more songs per GPU.

An open-source virtual AI Infra team that discovers, benchmarks, and safely upgrades local LLM inference—with independent quality gates, a stable OpenAI-compatible API, and automatic rollback.

Machine Learning In Production (MLOps)

cubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台，算法全链路流程，多租户，算力租赁平台，token中转，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务，VGPU虚拟化，云边端协同，边缘计算，自动化标注平台，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库llmops智能体，AI模型市场，支持国产异构算力调度,昇腾/寒武纪/海光/摩尔/沐曦等，支持ib/roce/RDMA，信创支持

A kedro-plugin for integration of mlflow capabilities inside kedro projects (especially machine learning model versioning and packaging)

Route inference across providers.

Browser-based model and inference management for NVIDIA DGX Spark - inventory local and Hugging Face models, manage Ollama and LiteLLM, generate Docker Compose deployments for vLLM, SGLang, llama.cpp, LocalAI, and ComfyUI, with multi-user access, diagnostics, and multi-node support.

NucliaDB, The AI Search database for RAG

CLI for Physical AI Specialization

Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks.

BharatMLStack is an open-source, end-to-end machine learning infrastructure stack built at Meesho to support real-time and batch ML workloads at Bharat scale

MONAI Deploy App SDK offers a framework and associated tools to design, develop and verify AI-driven applications in the healthcare imaging domain.

🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. 🗃️ Llama3, Mistral, etc. 🧑‍💻 Video Tutorials.

Deprecated historical repo. Superlinked now develops SIE, a self-hosted inference engine for embeddings, reranking, OCR, extraction, and document processing.

Examples for Cerebrium Serverless GPUs

MLOps for DevOps Engineers - A hands-on, project-based guide to Machine Learning Operations

A highly optimized LLM inference acceleration engine for Llama and its variants.

Superduper: End-to-end framework for building custom AI applications and agents.

Frouros: an open-source Python library for drift detection in machine learning systems.

Transform your pythonic research to an artifact that engineers can deploy easily.

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

FastAPI Skeleton App to serve machine learning models production-ready.

GenAIOps with Prompt Flow is a "GenAIOps template and guidance" to help you build LLM-infused apps using Prompt Flow. It offers a range of features including Centralized Code Hosting, Lifecycle Management, Variant and Hyperparameter Experimentation, A/B Deployment, reporting for all runs and experiments and so on.

The Machine Learning project including ML/DL projects, notebooks, cheat codes of ML/DL, useful information on AI/AGI and codes or snippets/scripts/tasks with tips.

A throughput-oriented high-performance serving framework for LLMs

Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

The Virtual Feature Store. Turn your existing data infrastructure into a feature store.

Next Generation Experimental Tracking for Machine Learning Operations

Amazon Web Services Operator Interface

Open source AI platform for rapid development of advanced AI and AGI pipelines.

Fraudfinder: A comprehensive lab series on how to build a real-time fraud detection system on Google Cloud

Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

Open-source, low-code AutoML platform for Python. PyCaret 4.0: sklearn-native engine + React control plane.

Hopsworks - Data-Intensive AI platform with a Feature Store

Large-scale Auto-Distributed Training/Inference Unified Framework | Memory-Compute-Control Decoupled Architecture | Multi-language SDK & Heterogeneous Hardware Support

Runtime for building and managing AI agents and Workflows. Easy to learn, fast to build, High Performance, Reliable by design, Intuitive UI, Production Ready.

Fast, high-quality forecasts on relational and multivariate time-series data powered by new feature learning algorithms and automated ML.

CLIP as a service - Embed image and sentences, object recognition, visual reasoning, image classification and reverse image search

ML/AI meta-model, used in MLRun/Iguazio/Nuclio, see qgate-sln-<MLRun | solution>

🧠 Motorhead is a memory and information retrieval server for LLMs.

Kraken CI is a continuous integration and testing system.

MLRun/Iguazio/Nuclio quality gate solution. The solution checks a quality of MLRun implementation/delivery.

PostgreSQL vector database extension for building AI applications

This repository contains guidance related to SageMaker AI Projects. SageMaker Projects help organizations set up and standardize developer environments for data scientists and CI/CD systems for MLOps engineers.

BentoML Example Projects 🎨

The Fuzzy Labs guide to the universe of open source MLOps

A standalone inference server for trained Rubix ML estimators.

Local inference, optimized for your hardware

An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

Turn any OCR models into online inference API endpoint 🚀 🌖

Designing end-to-end weekly stock report generation using LSTM and Agentic AI. Deploying on AWS with MLOps practices.

Making data lake work for time series

부스트캠프 AI Tech - Product Serving 자료

An open-source ML pipeline development platform

mRMR (minimum-Redundancy-Maximum-Relevance) for automatic feature selection at scale.

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.

Starter app for fastai v3 model deployment on Render

MLOps examples

🤖Self-Modifying Framework from the Future 🔮 World's First AMS

Feathr – A scalable, unified data and AI engineering platform for enterprise

Deploy a ML inference service on a budget in less than 10 lines of code.

Train and Deploy an ML REST API to predict crypto prices, in 10 steps

CDF SIG MLOps

mlops template

🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

Aqueduct is no longer being maintained. Aqueduct allows you to run LLM and ML workloads on any cloud infrastructure.

Coarse-grained lineage and tracing for machine learning pipelines.

deployKF builds machine learning platforms on Kubernetes. We combine the best of Kubeflow, Airflow†, and MLflow† into a complete platform.

🍫 Example code for a basic ML Platform based on Pulumi, FastAPI, DVC, MLFlow and more

flink-jpmml is a fresh-made library for dynamic real time machine learning predictions built on top of PMML standard models and Apache Flink streaming engine

An easy way to host your own AI API and expose alternative models, while being compatible with "open" AI clients.

Slides, scripts and materials for the Machine Learning in Finance Course at NYU Tandon, 2022

A Data Engineering & Machine Learning Knowledge Hub

☁️ Terraform plugin for machine learning workloads: spot instance recovery & auto-termination | AWS, GCP, Azure, Kubernetes

A Survey of AI startups

fastText model serving service

CodeProject.AI Server is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI.

Smoothly Manage Multiple LLMs (OpenAI, Anthropic, Azure) and Image Models (Dall-E, SDXL), Speed Up Responses, and Ensure Non-Stop Reliability.

Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)

Just enough Kubernetes for you to fly

an MLOps/LLMOps platform

FeatHub - A stream-batch unified feature store for real-time machine learning

A Beautiful Flask Web API for Yolov7 (and custom) models

A scalable, high-performance serving system for federated learning models

Crane is a easy-to-use and beautiful desktop application helps you build manage your container images.

A wannabe Ollama equivalent for Apple MlX models

Serving PyTorch models with TorchServe :fire:

A beginner's project on automating the training, evaluation, versioning, and deployment of models using GitHub Actions.

A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.

A booklet on machine learning systems design with exercises. NOT the repo for the book "Designing Machine Learning Systems", which is `dmls-book`

Scalable Machine Learning and Deep Learning, Final Project, 2023/2024

Joining the modern data stack with the modern ML stack

Deploy, serve, and run a Hugging Face model on a Raspberry Pi with just a few lines of code

ModelFox makes it easy to train, deploy, and monitor machine learning models.

Solutions on Practical Data Science Specialization on Coursera (offered by deeplearning.ai)

All the available resources to master MLOPS from scratch

🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)

A detailed summary of "Designing Machine Learning Systems" by Chip Huyen. This book gives you and end-to-end view of all the steps required to build AND OPERATE ML products in production. It is a must-read for ML practitioners and Software Engineers Transitioning into ML.

'Deploying machine learning models with a Flask API' tutorial, written for HyperionDev

This project employs emotion detection in textual data, specifically trained on Twitter data comprising tweets labeled with corresponding emotions. It seamlessly takes text inputs and provides the most fitting emotion assigned to it.