Did you unlink the audio and video track for the clips that you expect not to move? Unless they are unlinked any trim that changes position or duration of the video track will affect the linked audio track
I believe that this is expected behavior becuase why would you usually want to take the vidio and audio out-of sync. if you want to do something different, then you need to lock the track (but if the tracks are liniked, both the audio and video track will lock, I believe.