[SIGCOMM 2023] Lightning: A Reconfigurable Photonic-Electronic SmartNIC for Fast and Energy-Efficient Inference
-
Updated
Nov 17, 2023 - Verilog
#
inference-serving
Here are 34 public repositories matching this topic...
[Long Term Support] [SIGCOMM 2023] Lightning: A Reconfigurable Photonic-Electronic SmartNIC for Fast and Energy-Efficient Inference
verilog fpga-soc smartnic machine-learning-systems photonic-computing rfsoc inference-serving in-network-computing
-
Updated
Sep 20, 2024 - Verilog
The official repo for the paper "Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching"
-
Updated
Mar 17, 2025
Official repo for the ACSOS 2021 paper on how to manage many deep learning models at the edge!
-
Updated
Sep 29, 2021 - Python
local model & cloud models collab to solve tasks, no private data is sent to the cloud
inference openai routing-engine inference-server routing-algorithm inference-engine problem-solving-algorithms inference-optimization routing-security inference-acceleration inference-serving local-llm local-ai offline-ai anthropic-api inference-gateway model-router inference-time-scaling model-routing melious
-
Updated
Nov 12, 2026 - Rust
NexusKV is a model-state intelligence layer for inference systems.
python rust golang distributed-systems inference radix-tree model-state kv-cache inference-serving llm-inference ai-infrastructure state-reuse
-
Updated
Aug 21, 2026 - Python
KV Cache Aware LLM Inference Router : Inferrail is a KV cache aware router and control plane for self hosted LLM inference. It sits between OpenAI compatible clients and a fleet of vLLM or SGLang engines, and sends every request to the replica that already holds most of its prompt in cache, without letting any replica overload.
rust distributed-systems load-balancing kv-cache inference-serving llm-inference sglang llm-router gpu-infrastructure
-
Updated
Oct 28, 2026 - Rust
Local AI, without the setup
gpu inference inference-server local-first inference-serving llm llama-cpp vllm local-llm llm-inference llm-tools local-ai vllm-serve inference-gateway
-
Updated
Nov 14, 2026 - Rust
Local inference serving with adaptive batching, benchmark sweeps, and regression gates for batching tradeoffs.
python benchmark machine-learning real-time latency websocket load-testing inference regression-testing performance-testing model-serving quality-gates mlops fastapi inference-serving llm-inference adaptive-batching
-
Updated
May 20, 2026 - Python
Locus is an engine-neutral inference control plane for compute and model-state placement.
rust distributed-systems scheduling inference orchestration control-plane load-balancing model-serving inference-serving llm-inference ai-infrastructure workload-placement state-aware-scheduling
-
Updated
Nov 4, 2026 - Rust
Reactive LLM inference.
inference incremental-computation inference-optimization kv-cache inference-serving llm llm-inference kvcache-reuse recurrent-memory
-
Updated
Oct 2, 2026 - Python
A multi-tenant LLM serving system with fair queuing, admission control, prefix-aware routing, and reproducible H100 benchmarks.
kubernetes distributed-systems gpu prometheus ai-agents ml-systems inference-serving vllm llm-inference h100
-
Updated
Oct 11, 2026 - Python
Reproducible Apple-Silicon (Apple MLX) deployment and one-token verification kit for the FreeToken edge-native MoE serving API — pinned, loopback-only, schema-validated evidence.
macos benchmark reproducibility mlx mixture-of-experts research-reproducibility apple-silicon inference-serving llm-inference freetoken
-
Updated
Aug 29, 2026 - Python
Prepaid asynchronous LLM inference platform: FastAPI, PostgreSQL-backed queues, deadline-aware scheduling, modular GPU workers, vLLM integration, token metering, ledger billing, Stripe test payments, Next.js dashboard, Python SDK, and Docker. Includes a mock mode and recorded walkthrough; real GPU validation pending.
docker machine-learning multi-tenant stripe asynchronous gpu nextjs scheduler postgresql nvidia developer-tools python-sdk job-queue model-serving fastapi inference-serving usage-based-billing vllm llm-inference inference-platform
-
Updated
Nov 14, 2026 - Python
Reproducible benchmark harness for MoFlux admission control, with deterministic paired seed sweeps measuring latency, goodput, overload protection, token accounting, and policy trade-offs across Tyr and Latchflo.
nodejs docker distributed-systems typescript latency grafana reliability concurrency load-testing rate-limiting prometheus observability performance-testing backpressure platform-engineering admission-control inference-serving llm-benchmarking ai-infrastructure llm-infrastructure
-
Updated
Oct 6, 2026 - JavaScript
GSM artifact for exact online GNN inference serving with degree-based scheduling and memory-aware batch division.
scheduling cuda memory-management graph-neural-networks gnn dgl inference-serving online-inference graph-inference
-
Updated
Jun 15, 2026 - Python
Machine-readable companion to the IEEE OJ-CS survey 'Semantic Caching and Response Reuse for Large Language Model Services: A Survey' (Chukkapalli, Mishra, Naik, 2026): 21-work evidence matrix, systematic-search log, proposed benchmark trace schema, stdlib-only contract validator, and CPU pilot. Code MIT; data CC-BY-4.0.
-
Updated
Jul 11, 2026 - Python
Technical report for a cluster-shared L3 KV-cache pool for Agnes inference.
-
Updated
Sep 3, 2026
Embeddding Reranker Pools is the RAG inference microservice for semantic search and retrieval, featuring GPU-isolated embedding and reranking pools, Triton-based model serving, low-latency online inference, and high-throughput batch ingestion for search pipelines.
python kubernetes machine-learning microservice backend cpp high-throughput embedding-models batch-processing reranking ai-search rag vector-search kserve inference-serving gpu-inference retrieval-augmented-generation enterprise-ai llm-ops ml-platform-workflow
-
Updated
Nov 7, 2026 - Python
Add this topic to your repo
To associate your repository with the inference-serving topic, visit your repo's landing page and select "manage topics."