So your observed speed difference of the video is due to Pr's interpretation of the imported footage. One solution is to make a copy of the 120fps clip in the Project panel, and interpret it to 120fps. Leave the original unchanged. Use the audio from the original and the video from the copy and you should be good to go.
Jeff