Pixel Tracking Analysis

Senior Forensic Animator, David Hessel discusses pixel tracking techniques used to determine velocities of moving objects within a video.

Video Transcription

I wanted to try and find a quick, but accurate way of getting speeds of vehicles.

Pixel tracking is a technique where you track a pattern of pixels across an image. So, the software allows you to find a window, which chooses your pattern. It could be something like a taillight, a logo, a license plate that the software can just lock onto. Once it gets that pattern it tries to find that same pattern in consequent frames and once it does it track that feature throughout that sequence. I can use this tracking software to just track headlights and license plates, or emblems on vehicles and since I already know the camera motion from having tracked it before I can use that information along with some information about the vehicle to determine location of that tracking point in 3D space over time, which would give me speed.

When you have the camera and you have the pixel tracking it gives you this vector that just keeps going out forever in space. You have to have some way of stopping it. So, I figure out where exactly it is. You know it’s along that line. but figuring out where it is on that line you need one other piece of information. So, the thing that I write about in this paper and the thing that I did in that particular example was look for the height off the ground. If the headlight is supposed to be two feet off the ground you find the point where it’s two feet off the ground, stop it, solve that point, and that’s where that point is in 3D space and time. Do that for multiple frames and you’ve got your speeds.

We actually had a train derailment we did in Montana. We wanted to recreate it, so William actually went out there and derailed a train just like how it happened in the accident. And basically, what we did was we set up two high speed cameras so I could triangulate points. So, we had those set up and I tracked points all about the train. I think it was, like 12 of them, in both cameras, using the same technique. And I got 3D points from for all of those and then using those 3D points I was able to see how they all moved together and solve for a single transform from it, which gave me position and rotation and from that I was able to recreate the exact motion of the train. You can see the actual footage and you can see the reconstructed footage that I did using that technique and they’re identical.

Mainly, the software we use for tracking is PF Track. That’s what’s used for tracking the video and the 2D tracking as well. I have also used After Effects for that. This particular software we also use does also have the ability to load the LiDAR data, and you can take your tracking points and assign it a point in that LiDAR data. You can tell it, “this point is actually this scan point.” And you can do it for multiple points.

There’s nothing in existence that actually takes that 2D data and transforms it into a 3D coordinate like we do here. That was all custom written stuff. So, that’s all scripts we’ve developed.