The scenario was the same, but what I met was the entire skybox environment shifted about a meter to the left. After put the headset on, I need to recenter it or click the power button to sleep and wakeup again to get the center back.
I feel like something remembered the center position at the moment of going into sleep state. And then recalled the center position back after you put on the headset again.