I recently tried to play a mkv file over SMB on my quest. I found that mkv files containing H.264 video, AAC audio and subt subtitles play fine, but when I try a file with H.265 (HEVC) video and ssa subtitles, I only get audio, with no video or subtitles. The H.265 video is only 720p, so it is not a file size problem, I think. Possibly just a subtitle formatting issue, But I don't know.
The HEVC file does show some video stuttering and subtitle issues in VLC (while still at least showing video), though it plays fine in MPC-HC.