After remux with MKVMerge to add subs, scanning is slow

I added subtitles to some mkv files using MKVMerge, however now when I scan through the files with VLC, it takes 15 seconds before it starts playing again. I can skip to any part of the original mkv files and VLC plays without any lag. Any ideas?