deep learning - Obtaining matching image frames from videos with two different frame rates

Question

I am in the process of converting videos to images in python3.6 (i.e. cut videos to get images)

I have two types of videos The first one is a RGB video recorded from Real sense D435i with a frame rate of 30 (i.e. fps= 30) The second one is a thermal-IR video recorded from Flir Adas camera with a frame rate of 9 (It was originally a frame stream file which i converted to images and formulated it into a video with fps of 9 using python3-cv2). The video formats are in mp4 and avi respectively (though I have converted the avi to mp4 and tested it out as well). They are equal in length.

I am trying to create a matched image pair from the thermal and IR videos. However, when I seem to cut them using the same frame rate, it seems like they don't match (but differ by 5-6 images).

I have about 200 ish videos so it is very time consuming and difficult for me to track them down one by one.

Any ideas on how I can get this dataset to make it a paired one?

Many thanks

Answer 1

You might be able to recreate the 30 fps video as a matching 9 fps video by matching up the timestamps of the two videos. OpenCV lets you specify the framerate in the VideoWriter (I think you knew this, but just to be sure). OpenCV reports the timestamp of the current video with

cv2.VideoCapture.get(cv2.CAP_PROP_POS_MSEC);

Grab one frame and timestamp from the FLIR camera and then keep grabbing frames and timestamps from the color camera until the timestamp catches up or passes the FLIR timestamp. Then write the color frame to the vidwriter. So long as both of the videos had a consistent framerate and they both started/stopped recording at the same time then this should see both videos matched up as closely as possible.