Special thanks to Sarvesh Talele for his meaningful contributions, support, and wisdom that helped shape this work.
Traffic cameras at intersections and along highways record the circumstances of a great many collisions, but the methods that read those recordings automatically are usually trained on annotated footage from the same camera. Move the camera and the method has to be retrained. This work asks a narrower question: how much of an accident can be recovered from a recording when there is no labelled real-world data at all, and nothing may be fine-tuned. The answer is a pipeline of three independent modules, one for when the collision happened, one for where in the frame it happened, and one for what kind of collision it was, running end to end on pre-trained weights alone.
This is joint work with Sarvesh Talele, written for the ACCIDENT @ CVPR 2026 challenge and available as an arXiv preprint, 2604.09685.
Introduction
Road traffic crashes kill over one million people each year [1]. Surveillance cameras already record much of what happens, and if that footage could be read automatically, dispatchers would be alerted sooner and investigators would have an objective account of the scene. The obstacle is that most detection methods are supervised, and supervised on video from the deployment site itself [2] [3]. Each new installation brings a different viewpoint, a different lens, different lighting and different traffic, and the annotation bill is paid again.
The ACCIDENT @ CVPR 2026 competition [4] removes that option deliberately. Development material is synthetic, rendered in the CARLA simulator [5]. The test set is real CCTV, and annotating it by hand is prohibited. Anything that scores has to arrive at the real footage carrying only what it learned somewhere else.
Chronological sampled frames from a synthetic CARLA traffic incident in the ACCIDENT @ CVPR 2026 development split. Most clips record approximately 18 seconds of motion at 20 frames per second.
Three predictions are required for every video:
- the accident time in seconds,
- the normalized image coordinates of the point of impact, and
- the collision type, drawn from five categories: head-on, rear-end, sideswipe, single-vehicle and t-bone.
We answer each with a separate module. Timing comes from statistical anomaly detection on differences between consecutive frames. Location comes from accumulated dense optical flow. Type comes from CLIP [6], a vision-language model trained on 400 million image-text pairs, compared against written descriptions of each category. Because the three share no parameters, any one of them can be replaced without disturbing the others.
The Competition
Scoring is the part that shapes the design. A submission is graded on the harmonic mean of three quantities.
The temporal component 𝒯 measures how close the predicted time is to the truth through a Gaussian kernel with σt = 2.0 seconds:
𝒯 = exp( −½ ( (tpred − tgt) / σt )2 )
The spatial component 𝒮 applies the same form, with σs = 0.1, to the Euclidean distance between the predicted and true impact points in normalized coordinates. The classification component 𝒞 is top-1 accuracy, so it is either 1 or 0. The final score is
ℋ = 3 / ( 1/𝒯 + 1/𝒮 + 1/𝒞 )
The harmonic mean is unforgiving by construction. A zero anywhere sends the whole score to zero, which means a method cannot buy a good result by being excellent at the easy component and hopeless at the hard one. That property matters a great deal to how the results below should be read.
Dataset
The development split holds 2,211 synthetic CCTV-style videos rendered in CARLA, each annotated with accident time, impact coordinates and collision type. The test split holds 2,027 real surveillance recordings gathered from public traffic camera feeds, varying in resolution, frame rate and lighting, and carrying the compression artefacts, lens distortion and partial occlusion that real installations produce.
All synthetic videos are rendered at 1920 × 1080 at a fixed 20 frames per second. Clip length runs from 5.8 to 32.2 seconds, with a mean of 17.7 seconds and a standard deviation of 3.9 seconds.
Distribution of ground-truth accident times across the 2,211 synthetic videos. Most incidents occur within the first ten seconds of the clip.
The ground-truth accident falls at a median of 6.9 seconds into the clip, with an interquartile range of 5.2 to 9.8 seconds, which places most collisions in the first half of the recording.
The five categories are far from balanced.
| Collision type | Videos | Share of split |
|---|---|---|
| Rear-end | 794 | 35.9% |
| Head-on | 588 | 26.6% |
| Sideswipe | 405 | 18.3% |
| T-bone | 358 | 16.2% |
| Single-vehicle | 66 | 3.0% |
Collision type frequency in the synthetic development split. The rear-end category holds twelve times as many samples as the single-vehicle category.
Impact coordinates are normalized to the unit square and sit close to the middle of the frame. Both cx and cy have means near 0.50, with standard deviations of 0.13 and 0.18 respectively.
Ground-truth impact point distribution across 2,211 synthetic videos, coloured by collision type. Points cluster near the frame centre, with head-on and sideswipe events showing the widest spatial spread.
That concentration is worth holding on to. It means a spatial prediction that simply guesses the centre is already not terrible, and it sets a floor that any real method has to beat before its spatial score means anything.
Method
Temporal Localization
A collision produces a sudden change in image intensity. The temporal module turns that observation into a one-dimensional signal and then looks for statistical outliers in it.
Let It be the grayscale frame at index t, resized to 180 × 320 for speed. The mean absolute difference between adjacent frames is
dt = (1 / HW) Σu,v | It+1(u,v) − It(u,v) |
This series carries the collision, and it also carries camera shake and ordinary traffic. A centred rolling mean over a window of w = 5 frames suppresses the short-lived noise, and the smoothed values are then turned into z-scores against the mean and standard deviation of the whole series:
zt = ( dt − μ ) / ( σ + ε )
with ε = 10−8 to keep the denominator away from zero
Any frame whose zt exceeds a threshold τ = 1.5 becomes a candidate, and the candidate with the strongest score wins. If nothing crosses the threshold, the module falls back to the global maximum, so it always returns an answer. The predicted time in seconds is the winning frame index divided by the frame rate.
Temporal localization on a synthetic CARLA video. Above, the mean absolute frame difference across all frames. Below, the z-score anomaly series after rolling-mean smoothing, with the dashed line marking the detection threshold.
The pair of charts shows why the smoothing step earns its place. The raw signal mixes a slow drift, produced by vehicles moving through the scene, with sharp transients. After smoothing and normalization the shape of the event survives while the isolated single-frame spikes do not.
Spatial Impact Localization
A collision concentrates high-magnitude motion into a small part of the image. The spatial module finds that part by accumulating dense optical flow over a short window and taking the weighted centroid of the result.
When the temporal module has produced a time, a 30-frame window is centred on the corresponding frame. Otherwise the window starts at frame ⌊N/3⌋. Each consecutive pair inside the window is passed through the Farnebäck dense optical flow algorithm [7] at 320 × 180, which estimates per-pixel displacement using quadratic polynomial expansions over a multi-scale Gaussian pyramid. The displacement magnitudes are summed across the window:
M(u,v) = Σt √( fx2(u,v,t) + fy2(u,v,t) )
Everything below the 90th percentile of M is then set to zero. This is the step that separates a collision from busy traffic: diffuse motion spread across the frame is discarded, and only the dense high-magnitude cluster survives. The impact point is the weighted centroid of what remains, normalized to the unit square:
cx = (1/W) · Σ v · M(u,v) / Σ M(u,v) cy = (1/H) · Σ u · M(u,v) / Σ M(u,v)
If the total falls below 10−6, the module returns the frame centre. The calculation is a special case of the image moment framework of Hu [8], applied to flow magnitudes rather than pixel intensities.
Cumulative Farnebäck optical flow magnitude after 90th-percentile thresholding. The bright region corresponds to the collision area, and diffuse background motion has been suppressed.
Collision Type Classification
CLIP [6] learns a shared embedding space for images and text by contrastive training on 400 million image-text pairs. At test time an image embedding can be compared against text embeddings of candidate class names by cosine similarity, which gives a classifier with no task-specific training data behind it.
For each of the five types we write five short descriptions of the collision as a bystander would put it. Every prompt is encoded, L2-normalized and averaged into a single vector for that class, which reduces the influence of any one wording [6] [9]. These vectors are computed once and cached before any video is read.
| Type | Example prompt |
|---|---|
| Head-on | “two cars colliding head-on from opposite directions” |
| Rear-end | “a car colliding into the back of another car” |
| Sideswipe | “two vehicles scraping alongside each other” |
| Single-vehicle | “a single car crashing into a wall or obstacle” |
| T-bone | “a car hitting the side of another car at an intersection” |
At inference, eight frames centred on the predicted accident time are passed through the CLIP visual encoder (ViT-B/32), L2-normalized and averaged into one representation v. The predicted type is the class whose text vector gives the highest cosine similarity with v.
Implementation
Inference runs on a single NVIDIA T4 GPU on the Kaggle platform. No weights are trained or fine-tuned on either split. Processing all 2,027 test videos takes roughly two hours.
The hyperparameters were chosen by inspecting a handful of synthetic videos and then held fixed for every test prediction.
| Component | Parameter | Value |
|---|---|---|
| Temporal | Smoothing window w | 5 |
| Temporal | Z-score threshold τ | 1.5 |
| Spatial | Start frame | centred on t* |
| Spatial | Context window | 30 frames |
| Spatial | Pyramid scale | 0.5 |
| Spatial | Pyramid levels | 3 |
| Spatial | Window size | 15 |
| Spatial | Flow percentile threshold | 90th |
| Classification | CLIP backbone | ViT-B/32 |
| Classification | Peak-region frames | 8 |
Results
The pipeline scores 0.2523 on the public leaderboard, computed on approximately 25% of the real CCTV test set. The final ranking uses the remaining 75%, so the standing can still move.
The more useful number is the breakdown. On a ten-video calibration subset drawn from the synthetic split:
| Component | Mean score | Best individual |
|---|---|---|
| Temporal 𝒯 | 0.438 | 0.94 |
| Spatial 𝒮 | 0.168 | 0.96 |
| Classification 𝒞 | 0.0 | 0.0 |
Two things are worth separating here. The best individual temporal score of 0.94 and best spatial score of 0.96 show that when the pipeline locks on to the right event, both estimates can be accurate. The composite on this subset is nevertheless zero, because every one of those ten calibration videos is a head-on collision and CLIP answers t-bone for all ten. Under a harmonic mean, one component at zero settles the matter.
That is a property of the calibration subset rather than a claim about the whole test set, and it should not be read as the pipeline scoring zero in general. It does, however, point straight at where the loss is concentrated.
Predicted collision type distribution across the 2,027 real test videos. Sideswipe and single-vehicle dominate the predictions, while rear-end is almost never selected, inverting the synthetic distribution.
The predicted distribution inverts the training distribution almost exactly. Sideswipe is chosen for 770 of 2,027 videos and single-vehicle for 687, while rear-end, the most common category in the synthetic split at 794 videos, is chosen 23 times.
| Collision type | Synthetic split | Predicted on test |
|---|---|---|
| Rear-end | 794 | 23 |
| Head-on | 588 | 122 |
| Sideswipe | 405 | 770 |
| T-bone | 358 | 425 |
| Single-vehicle | 66 | 687 |
A shift of that size is not a small calibration error. It says that CLIP’s similarity scores here are responding to viewing angle and scene geometry rather than to the dynamics of the collision itself.
Error Analysis
Three failure patterns account for most of the loss.
Temporal. The frame-difference signal cannot tell a collision from any other sudden change in the image. Swaying vegetation, cloud shadows crossing the road and camera shake all produce spikes of comparable magnitude, and the module has no way to prefer one over another.
Spatial. The centroid is an average, so it drifts whenever several vehicles are moving at once: the weighted mean spreads across every active region instead of settling on one. The central clustering of true impact points, visible in the scatter above, is what keeps these drifted predictions from being worse than they are.
Classification. This is the bottleneck. CLIP was trained on internet photographs taken at roughly eye level, and CCTV views are overhead or steeply oblique. The model is being asked to recognise a geometry it has hardly seen, and the inverted prediction distribution is the visible symptom.
What Would Improve It
The modular structure is what makes the next step cheap, because each part can be replaced on its own.
Two directions look most promising. Replacing Farnebäck with a learned estimator such as RAFT [10] should improve displacement accuracy at the low resolutions the pipeline runs at. More importantly, fine-tuning the CLIP visual encoder on the synthetic split would attack the domain gap between internet imagery and surveillance stills directly, and that gap is the largest single source of loss identified above.
Conclusion
The pipeline detects, localizes and classifies traffic accidents in CCTV footage without a single fine-tuned weight, reaching a public leaderboard score of 0.2523. Timing comes from z-score peak detection on frame differences, location from thresholded Farnebäck optical flow reduced to a weighted centroid, and type from CLIP embeddings matched against multi-prompt text descriptions.
Read as a whole, the result is a fair account of what general-purpose pre-training currently supplies for free. The two components built from classical signal processing behave, and can be accurate when they lock on to the right event. The component that leans on a large pre-trained model is the one that fails, and it fails in a structured, legible way that names its own remedy.
Preprint
Additional Resources
Preprint, Code, and Competition
The preprint, the notebook that produced every number above, and the competition itself:
Citation
Please cite this work as:
Thakur, Amey, and Sarvesh Talele. "A Modular Zero-Shot Pipeline for Accident Detection, Localization, and Classification in Traffic Surveillance Video". arXiv preprint arXiv:2604.09685 (Apr 2026). https://arxiv.org/abs/2604.09685.Or use the BibTex citation:
@article{thakur2026accident,
title = "A Modular Zero-Shot Pipeline for Accident Detection, Localization, and Classification in Traffic Surveillance Video",
author = "Thakur, Amey and Talele, Sarvesh",
journal = "arXiv preprint arXiv:2604.09685",
year = "2026",
month = "Apr",
url = "https://arxiv.org/abs/2604.09685"
}