It's your eyes.
The total video resolution is wrapped around your head either in 180 or 360 degrees. What you view in your line-of-sight is essentially only a part of the whole video (cropped or zoomed in if you will), which effectively has a lower visible resolution than the total resolution of the video.
If you had a 4k or 5k video source, that visible portion can then reach closer to the full resolution of the display. As for 4k vs. 5k, I wouldn't know, but there is a clear difference between 3.2k and 4k.
