DIY Street View Panoramas

I have added street view support to the Yerevan Tree Map. I bought a DJI Osmo 360 camera and built a quick-and-dirty pipeline to synchronize the video footage with an external GPX track. Additionally, I process the results through OpenSfM to improve spatial accuracy. Finally, I upload everything to Mapillary and use it via their API.

Why this matters

The ultimate goal is the automatic recognition of trees, calculating their coordinates through triangulation, and placing them on the map. We are still far from that: I need to improve data accuracy, build a manual triangulation interface, and then automate the process through image recognition.

However, even now, this solves several problems for our application. First, it allows us to quickly document the state of a street if construction or other works that could affect trees are planned or underway. Second, once this tool is fully operational, there will be no need to take individual photos of trees to document their general appearance.

Encountered problems

(1) Synchronizing video with GPS. I use standard consumer equipment, specifically a mobile phone to record coordinates (using the GPS Logger app). This means, first, an accuracy of ±6 m in the city, and second, the phone uses some data approximation, resulting in a 2-3 second lag. Consequently, no matter how hard you try to synchronize the device timers, it is futile.

I stopped trying to sync timers and came up with something like a manual GCP (Ground Control Point): I find a segment of the recording where I am driving at a constant speed (usually around 40 km/h), find the nearest landmark (an intersection or a building corner), take the road coordinates perpendicular to this landmark, and based on the current frame number and coordinates, I can accurately synchronize the video sequence with the external GPX file. This is manual work, but not too difficult, and it translates well into a web service. I haven't yet figured out how to automate this completely.

(2) Coordinate accuracy. The phone records coordinates with ±6 meters accuracy, which is insufficient for triangulation. Furthermore, there is no data on the camera's orientation during recording. While the direction of travel can be calculated from the track, the initial camera orientation and how it changed after hitting a speed bump remain unknown.

I solved this by integrating OpenSfM, which accurately reconstructs the scene and calculates these parameters for each specific frame. However, an additional UI is still needed to verify the results.

(3) Mapillary quirks. It refuses to accept some datasets, complaining about high pixel density. I haven't solved this yet, but in general, after adding OpenSfM to the pipeline, Mapillary could be discarded, as it is somewhat cumbersome for file hosting, and I don't use any other of its features.