Forensic Acquisition of Vehicle Infotainment Systems Data
Last week, Nathan Rose sat down with Will Bortles of Kineticorp to discuss the research he just published at the 2017 Society of Automotive Engineers World Congress. Here is a transcript of their discussion and what this research means to accident reconstruction. This research relates to methods for acquiring data from passenger vehicle infotainment systems. Data from these systems is beginning to be used by accident investigators and reconstructionists, particularly in criminal case settings
Nathan: My intent today is to give other accident reconstructionists a snapshot of what you’ve been working on. This, I think, is an area that’s going to be potentially big in the future. We’re talking about data that could be mined off of infotainment and telematics systems on vehicles.
Will: Yeah, that’s correct.
Nathan: So, what vehicles specifically should people be thinking about that might have data on them?
Will: Vehicles that would really scream at me that this technology’s available, are vehicles with high end trim packages. So, if you have the touch button navigation, the integration of the Bluetooth – those are the types of systems that we see more advanced telematics systems, more advanced infotainment systems where you get the navigation.
Nathan: With event data recorders, we tend to think of it in terms of manufacturers.
Will: Manufacturers and models.
Nathan: Yeah, right. Is this similar? Or is it just, it’s not so much a manufacturer thing, it’s more of a, if one of these infotainment systems is on there that has these capabilities, and hey if it does we can give it a shot and see if we can get data off of it.
Will: It’s both. First, it has to do with whether or not you have the sophisticated infotainment systems. Second, Berla, the manufacturer of the hardware and software system that we use for the data acquistions, has only been able to reverse engineer or to extract data from certain makes and models. So it’s a two-fold thing, first you’ve got to be in the right generation of vehicles. You usually start to see this data becoming available in about model year 2008. That is when the iPhone really caught on, so the automotive manufacturers collectively started to make their vehicles more compatible with smartphones. Second, you’ve got to have the right hardware in the vehicle, so that becomes vehicle make and model specific.
Nathan: Right, and that brings up an interesting point. This capability is being developed by Berla…
Nathan: It’s not authorized by the auto manufacturers, right?
Nathan: So, what is the stance of the auto manufacturers towards this? Are they against it? Are they okay with it?
Will: I don’t know one way or another. They certainly don’t promote it. We’re actually going through some pretty extraordinary, physically the data acquisition process is pretty extraordinary, it’s pretty invasive. I’m thinking specifically of the GM modules where you have to go in, disassemble the dash, take the module out, take the motherboard out of the actual module. And then you have a little fiberglass pen, and you have to scratch away the solder mask just to be able to make contact with the wires to extract the data. So, it’s certainly not plug-in-play, it’s very invasive…
Nathan: Is there a list of say, three, four, five things you could give us that this is the type of data we would typically be able to access?
Will: The most prevalent data is cell phone stuff, that pretty much goes across all vehicles, for BMW, for Fiat Chrysler vehicles, GM vehicles, Ford vehicles that – you sync up your phone to the vehicle. It helps to think about what you’re doing inside the vehicle and that helps you understand the data that’s being collected. If you plug in your phone, you allow it to download your contact list in order to do a speed dial, well now you realized that you’ve pushed that contact information to the vehicle. That information is now being stored on your car because now you have your phone list where you could just say, “Call home”, “Call the wife”, “Call the office”…
Nathan: So, could you get a call history out of the…?
Nathan: Now tell me about the GPS data, is it only if the navigation system is being utilized?
Will: In my testing it was opposite actually. So, for some vehicles they’ll have – you predefine certain locations or routes. For instance, you may have your office addressed predefined, you have your home address predefined, you have certain stores or places that you go and those are stored as ‘Locations.’ When you program those addresses into the vehicle it knows that and it stores it that data. For example, you tell your vehicle: here’s where daycare is, here’s where my kid’s school is, here’s where my office is. That’s one of the locations that you can put in there, then there’s another data set called a ‘Route.’ Which is, every morning you stop at Starbucks and then you go drop a kid off, and then you go to work, and if you program that as a route, that’ll be stored in the vehicle that way. So ‘Locations’ and ‘Routes’ are listing of addresses.
The other GPS data element, what I think is the more exciting, or the more applicable GPS data for an investigation is what they call ‘Track Logs.’ Which is, on such and such a date, you were here, and then you were here, and then you were here, so it’s just a little breadcrumb trail of GPS coordinates– a data sample of about one per second. Based on our research, I havent fully figured out the data arbitration on that. I don’t know why certain ‘Track Logs’ are being saved and certain ‘Track Logs’ are not saved. The paper that I’m going to present, we recovered several ‘Track Logs’ from the vehicle. However, some of our test runs were not stored or recovered.
We did a whole battery of testing, which we were testing the doors, the lights, the phone activity. We were testing gear shift events – so we were just out in the parking lot, driving forward, stopping, putting it in reverse. Driving forward, putting it in second gear, driving in reverse. We were able to recover several ‘Track Logs’ of that driving, just back and forth in the office while we tested the gear shifts. Ultimately, we instrumented the vehicle with the V-Box Sport and drove around to emulate normal day-to-day driving. We drove first down towards to the Park Meadows mall, so we were doing mixed highway and surface street driving. Then we stopped and then did some surface driving down County Line Road, and then we starting coming and making our way back towards the office.
For that testing, we did five total runs. The first four runs, we were using the GPS – hey “take us to Park Meadows Mall”, hey “take us to another waypoint that we set” – “came back to the office.”For the last run, we just drove around freestyle, or without the navigation. What was notable is that the freestyle driving was actually the only ‘Track Log’ that the vehicle recorded. Of the five runs that we were testing, we only got data for the last run. Which is counterintuitive. I was thinking that since the first four runs, the vehicle was monitoring where we were. It was monitoring where it thinks you should go, and telling you hey make a left, make a right, oops you messed up – turn around. The morning of our testing, there was a detour related to construction, so we had to deviate from the suggested path and turn around. So the vehicle was recalculating the suggested route. I thought that all that time, those were going to be the most likely GPS ‘Track Logs’ that we were going to get from. Because the GPS in the vehicle was figuring out where we were at, figuring out where we wanted to go, calculating the ETA, all that stuff – I thought that type of data retention was going to be the Track Log that we got, and it turns out it was the opposite.
Nathan: Interesting. So talk about what case issues should trigger in our minds that we should think about mining this type of data.
Will: The beauty of this thing is that it doesn’t necessarily need that same crash trigger to record data, or you don’t necessarily need a crash event at all. If the traditional event data recorders require a pretty good impact to trigger the system to record data. I think the new standard requires more than five mile an hour delta-v to recognize a nondeployment event. If the vehicle is not exposed to that 5 mph speed change, you might not get traditional EDR data. Supposing you have one of those crash events, you get EDR data and it’s only five or ten seconds back. Suppose you wanted to maybe see if there’s something further beyond that window…
Nathan: Ok, that’s interesting. Well, so how long is this data stored?
Will: We were using volunteer’s vehicles to perform the testing, essentially anyone with the right vehicle that would allow us to hack into their vehicle. From these vehicles, we were finding ‘Track Logs’ that were really old – about a year or so prior. But several of our runs of testing weren’t recorded for whatever reason. Again, I don’t know how the some data is being stored while some is not. It does not appear to be the typical “first in, first out” data retention system.
Will: And it could be highly vehicle specific, so I don’t really know. This technology is so new, we still have a lot of work to do with other vehicles.
Nathan: Have you done this yet on a vehicle that has been involved in a crash?
Will: No. Not a crash vehicle. We’re hoping to have an iVe-supported vehicle at the ARC-CSI conference.
Nathan: Are you hoping to mine some data at ARC-CSI live?
Will: It depends on the vehicle. Some of these vehicles after you disassemble the dash and take the board out, take up to twelve hours to download.
Will: For we did our testing for the paper, the 2015 Ford F-350, it took more than four hours to do the data acquisition. We did one baseline download on a Saturday afternoon before the testing. Then came back and ran all of our testing on Sunday, and did the post-test download on Sunday night.
Nathan: What is Berla’s long term plan with this? Are they wanting to package this like CDR where eventually they would be putting this out to all reconstruction companies would have this capability? Or is going to remain a more specialized area for people who like digging circuit boards out of cars?
Will: I get the feeling that they’re trying to break into the accident reconstruction community, but I think their foothold is in national defense, police, DOD, Homeland Security. So, I think the law enforcement side of this technology is probably going to help this get more established and recognized, before it falls in to the civil realm – for use by the accident reconstruction community looking into crashes. Some of the cases that have already been using this technology is coming from law enforcement. It is these cases that are allowing us to see the potential that this has.
Nathan: Has any of that data been admitted in a trial?
Will: I think that several cases are heading towards trial. That was a topic for discussion at the this year’s EDR Summit down in Houston – how many times has this been used in court?
Nathan: Is there anything else you think needs to be said about this technology at this point – where we’re at with it, how people should be thinking about it, what our clients need to know, whatever…
Will: Well certainly, obviously your first step in using this technology is to determine whether any specific vehicle in question is supported. To do that, it’s a very vehicle specific thing. We would need the VIN of the vehicle so we can check it with the iVe software to see if we’re able to recover the data. For us in the reconstruction community, if you think that it may be helpful in any of your crashes, or in any of your cases you want to look for a bigger time window prior to a crash, it’s worth looking into. And conversely, it doesn’t necessarily have to involve a crash at all. It could be any investigation you’re conducting involving a vehicle.
Nathan: Awesome. Alright Will, thanks for taking the time.
Will: Of course, thank you.