How VSL works
Video Search Local turns continuous, hard-to-search surveillance footage into structured events that can be reviewed and retrieved with natural language.
End-to-end pipeline
1. Ingest a camera stream
FFmpeg reads RTSP, RTSPS, RTMP, RTMPS, HTTP-FLV, HLS, MJPEG, SRT, UDP, or TCP sources. Most common IP cameras and NVRs expose an RTSP main stream and a lighter sub-stream.
2. Detect objects locally
VSL samples frames at the configured rate, resizes them for YOLO26s or D-FINE-S, and runs ONNX Runtime on CPU or NVIDIA CUDA. The detector returns classes, confidence scores, and bounding boxes.
3. Track objects with ByteTrack
ByteTrack associates detections over time and assigns a stable short ID to each object. VSL renders IDs with distinct colors and saves trajectories. A track ID is local to one camera session and is not a real-world identity.
4. Evaluate rules and record events
Users group model labels into rules. A vehicle rule, for example, can include car, truck, and bus. VSL creates an event only after the same tracked object matches a rule for the configured number of inference frames.
Each event may include an annotated MP4, a JPEG cover, compressed frame-level trajectory metadata, and structured SQLite fields.
5. Review the event with a VLM API
After recording, an asynchronous worker sends sampled frames or the native clip to the configured vision-language API. The expected JSON describes the summary, activities, risks, keywords, matched rules, object IDs, and timeline. API failures are persisted and retried without blocking local detection.
6. Store and search
VSL converts labels, rules, summaries, and keywords into a search vector stored in Qdrant. A natural-language query returns similar events with their time, description, objects, score, and playable clip.
Inputs and outputs
| Type | Data |
|---|---|
| Inputs | Camera URL, detection model, rules, VLM API, Qdrant settings |
| Local processing | Decode, sample, detect, track, evaluate, annotate, record |
| Optional external step | Semantic event review by a VLM API |
| Outputs | MP4, JPG, trajectory JSON, SQLite events, Qdrant vectors |
| User interface | Live monitor, events, smart search, and settings |
Why two inference stages?
Local detectors are fast and predictable, but mainly answer “what objects are visible?” Vision-language models are better at relationships, activities, and risk, but are costly and inappropriate for uploading every live frame.
VSL uses a detector and rules to select meaningful events, then asks a VLM “what happened?” This balances latency, privacy, bandwidth, and API cost.