Ultralytics YOLO

Ultralytics YOLO provides state-of-the-art computer vision for object detection, pose estimation, and segmentation. The plugin enables real-time pose detection with skeleton overlays.

Vision Agents uses Stream Video for real-time WebRTC transport by default. External WebRTC transports are supported as well. Most AI providers offer free tiers to get started.

Installation

uv add "vision-agents[ultralytics]"

Quick Start

from vision_agents.core import Agent, User
from vision_agents.plugins import ultralytics, gemini, getstream

agent = Agent(
    edge=getstream.Edge(),
    agent_user=User(name="Fitness Coach", id="agent"),
    instructions="Analyze the user's form and provide feedback.",
    llm=gemini.Realtime(fps=10),
    processors=[
        ultralytics.YOLOPoseProcessor(
            model_path="yolo11n-pose.pt",
            conf_threshold=0.5,
            enable_hand_tracking=True,
        )
    ],
)

YOLO models download automatically on first use.

Parameters

Name	Type	Default	Description
model_path	str	"yolo11n-pose.pt"	YOLO pose model
conf_threshold	float	0.5	Keypoint confidence threshold
device	str	"cpu"	Device ("cpu" or "cuda")
enable_hand_tracking	bool	True	Draw hand skeleton connections
enable_wrist_highlights	bool	True	Highlight wrist positions

Model Sizes

Model	Speed	Use Case
yolo11n-pose.pt	Fastest	Real-time on CPU
yolo11s-pose.pt	Fast	Real-time on GPU
yolo11m-pose.pt	Medium	Quality-focused
yolo11l-pose.pt	Slower	Maximum accuracy

Next Steps

Build a Voice Agent

Get started with voice

Build a Video Agent

Add video processing