This $200 open-source VR headset - https://www.relativty.com/ seems to be promising, although I don't need VR, the hardware might still be used to create a wearable low latency video glasses. I'm planning to explore it further, suggestions are welcome.