⚡ Production Engineering — YOLOv8 API

Production-Ready Asynchronous Object Detection API

Scaling YOLOv8 Deep Learning Inference Beyond Single-Threaded Toy Scripts

GitHub Repository → Live API Docs ↗
FastAPI YOLOv8 Docker Asyncio Python OpenCV
📥
Ingest
🔍
Validate
📋
Queue
🧠
Infer
📊
Poll
🧹
GC Cleanup

🎯 Try the API Live

Upload any image below. It goes straight to the production API on Hugging Face Spaces — you'll see the real async queue at work, then the annotated result returned with bounding boxes.

⚠️ Cold-start notice: Hugging Face Spaces pause after inactivity. If the API hasn't been used recently, the first request may take 20–40 seconds to wake the container. Subsequent requests are fast. A warning will appear automatically if this happens.
🖼️
Click to select an image, or drag & drop
JPEG or PNG · any size
⏳ The API might be waking up from sleep — Hugging Face Spaces pause after inactivity. First request can take 20–40 seconds. Hang tight!
Uploading...
01 — EXECUTIVE SUMMARY

📌 From Local Script to Production System

In computer vision engineering, moving a deep learning model from a local testing script (model.predict()) into a production cloud environment introduces severe compute bottlenecks. Deep learning inference is extremely CPU/GPU intensive. When multiple clients send high-resolution image payloads concurrently to a standard synchronous API, the server threads freeze — resulting in socket timeouts, thread/RAM exhaustion, and container crashes.

To resolve this, I engineered a Production-Ready Asynchronous Object Detection API using FastAPI and YOLOv8. By leveraging an asynchronous queueing pattern, the API decouples long-running model inference into non-blocking sequential background tasks, protecting cloud hardware from resource starvation while maintaining high service availability.

3
Concurrent Requests Handled
9.01s
Full Stress Test Duration
0
Dropped Requests or Crashes
02 — SYSTEM ARCHITECTURE

🏛️ Non-Blocking Background Queue Pattern

The application decouples client requests from heavy model execution using a non-blocking background queue and an asynchronous state polling pattern. Each component has a single, clear responsibility.

1
The Ingestion Shield — POST /predict/ Accepts multi-part file uploads and validates MIME headers instantly to reject invalid formats (PDFs, executables) before consuming any downstream RAM or CPU resources.
2
The Asynchronous Queue Generates a unique UUID task_id, initializes a task record in the in-memory state database, schedules the heavy processing onto a background thread pool, and immediately responds with 201 Created — freeing the main socket thread instantly.
3
The Worker Loop Dynamically decodes the binary image stream into raw OpenCV matrices via cv2.imdecode, feeds it into a pre-loaded YOLOv8 instance (zero cold-start latency), plots bounding box annotations, and encodes the output as a Base64 JPEG data URL.
4
The Polling State Machine — GET /result/{task_id} The client queries this endpoint on a loop. The server reads the internal task state without blocking incoming request traffic. Each task transitions through a well-defined lifecycle:
queued → processing → completed / error
03 — ENGINEERING CHALLENGES

🧹 Production Hardening

1. Eliminating Container OOM (Out Of Memory) Crashes

The Problem Storing image results — massive Base64 string structures — in an in-memory dictionary (tasks_db = {}) causes RAM usage to scale linearly with traffic. Under automated high-concurrency loads, the container quickly hits memory limits, triggering the OS Kernel's OOM Killer to terminate the FastAPI process.
The Solution I engineered a high-efficiency background garbage collection daemon using a low-overhead Daemon Thread (threading.Thread(..., daemon=True)). It wakes every 60 seconds, calculates the delta between the current timestamp and task completion records, and cleanly scrubs tasks older than 5 minutes from RAM. Because it runs as a daemon thread, its lifecycle is bound directly to the master FastAPI process — preventing zombie memory leaks during system restarts. 
def fastapi_queue_cleaner():
    """Background loop: wakes every 60s, purges tasks older than 5 minutes."""
    while True:
        current_time = time.time()
        expired_tasks = [
            task_id for task_id, task_info in list(tasks_db.items())
            if current_time - task_info.get("updated_at", 0) > 300  # 5 minutes
        ]
        for task_id in expired_tasks:
            del tasks_db[task_id]
        time.sleep(60)

threading.Thread(target=fastapi_queue_cleaner, daemon=True).start()

2. Defensive Exception Interception

The Problem OpenCV's cv2.imdecode returns a quiet None object when an uploaded image byte-stream is corrupted or invalid. Without an explicit check, this silently crashes the YOLOv8 inference pipeline mid-execution with an unhandled exception.
The Solution An intermediate array verification layer intercepts corrupted uploads before they reach the model. Low-level C++ decoding failures are translated into clean HTTP error responses instead of server-side crashes. 
nparr = np.frombuffer(file_bytes, np.uint8)
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

if img is None:
    raise ValueError("Failed to decode image: corrupted or invalid format")

3. Concurrency Stress-Testing & Performance Validation

The Approach To prove the robustness of the decoupled queuing architecture, I built an asynchronous multi-connection benchmarking tool using asyncio and httpx. The script simulates high concurrent request spikes by firing parallel high-resolution image payloads at the ingestion queue and polling for all results simultaneously.
--- INICIANDO ENVÍO EN PARALELO ---
🚀 [Tarea 1] Enviando imagen a la cola...
🚀 [Tarea 2] Enviando imagen a la cola...
🚀 [Tarea 3] Enviando imagen a la cola...
📥 [Tarea 2] Aceptada. ID: 185f9844-31f4-42ea-ba30-d36c5db6177b
📥 [Tarea 1] Aceptada. ID: 93445b71-12c8-479c-b4ba-115f088195a8
📥 [Tarea 3] Aceptada. ID: 8fb6c623-fa14-411a-8c90-93a8d11c5f0a
 
--- INICIANDO MONITOREO EN PARALELO ---
🔄 [Tarea 3] Verificando... Estado: processing
🔄 [Tarea 2] Verificando... Estado: processing
✨ [Tarea 2] ¡Éxito! Guardada como stress_output_2_185f9844.jpg
✨ [Tarea 1] ¡Éxito! Guardada como stress_output_1_93445b71.jpg
✨ [Tarea 3] ¡Éxito! Guardada como stress_output_3_8fb6c623.jpg
 
⏱️ Tiempo total de la prueba con alta concurrencia: 9.01 segundos
Key Takeaway Even under burst concurrent traffic, the API never drops requests or times out. It enqueues workloads safely, keeps socket threads open for incoming connections, and processes detection tasks sequentially without any loss in server stability.
04 — CONTAINERIZATION

🐳 Docker & Cloud Deployment

A key production challenge was packaging the API with all low-level graphics dependencies — specifically solving the classic libGL.so.1 missing library error that occurs when running OpenCV on a headless Linux Docker container.

The solution is a custom multi-stage Dockerfile that installs system-level graphics bindings before the Python layer, ensuring cv2.imshow and matrix rendering work correctly inside the container without a display server. The resulting image is deployed to Hugging Face Spaces as a Docker space, exposing a persistent public API endpoint. 

# Build the image
docker build -t yolo-queue-api .

# Run locally
docker run -d -p 7860:7860 --name yolo-api yolo-queue-api
# → API live at http://localhost:7860/docs
05 — REPOSITORY STRUCTURE

📂 Codebase Overview

yolo-queue-api/
├── images/ # Sample images for local testing and validation
├── main.py # FastAPI server — non-blocking background queue
├── client.py # Single-request polling client script
├── stress_test.py # Async concurrent stress-test benchmarker
├── requirements.txt # Python library dependencies
├── Dockerfile # Multi-stage production Docker build
└── README.md # Project documentation
06 — TECH STACK

🛠️ Key Tooling

Layer Technology & Role
Web Framework FastAPI — ASGI Python server built on Starlette and Pydantic, handling async routing and background task dispatch
Inference Engine Ultralytics YOLOv8 — pre-loaded at startup for zero cold-start latency on each individual request
Image Processing OpenCV — byte-stream decoding via cv2.imdecode, NumPy matrix transformations, bounding box annotation rendering
Concurrency Python asyncio + threading — daemon-thread garbage collector, sequential background worker queue
Stress Testing HTTPX + asyncio — asynchronous multi-connection benchmarking client firing parallel high-res image payloads
Containerization Docker — custom multi-stage Linux runtime resolving libGL.so.1 system graphics dependencies for headless OpenCV
Cloud Deployment Hugging Face Spaces — Dockerized deployment with persistent public API endpoint
07 — KEY TAKEAWAYS

💡 Engineering Decisions That Matter

This project demonstrates engineering judgment, not just model execution — every design decision was driven by a real production constraint.

🏗️

System-Level Thinking

Designed a complete asynchronous task architecture with decoupled ingestion, processing, and polling layers — not just a model wrapper.

🧠

Resource Management

Identified a real RAM constraint under load and engineered an automated daemon-thread cleanup mechanism to prevent container OOM kills.

🛡️

Production Hardening

Wrote defensive validation layers intercepting corrupted payloads before they crash the inference pipeline — translating C++ failures into clean HTTP errors.

← Back to Computer Vision Projects