Last week I finished the most exciting project of my professional life (so far): I got to develop a 180° projection surface virtual reality system, more catchily coined a VR tank. It features:
- a curved projection surface measuring more than 4 meters in width and 2 meters height
- a HD stereo rear-projection system consisting of 4 projectors
- a 10-camera VICON motion capture system
- optionally, a treadmill to give users an infinite walk-space
My part in the project focussed on developing a VR application framework in C++, integrating the open-source Horde3D engine, interfacing with the real-time Vicon data stream and developing a new perspective-based rendering pipeline. It was most satisfying!
Below you can see a video of the system in action (filmed in infrared). The bright things atop the screen are Vicon cameras, their bright glare is infrared only and usually not visible. You can see how the user perspective is dynamically calculated based on head position: the black areas outside his field of view are not visible to the user.
The objects that the user is interacting with appear to him as floating in front of the screen. The Vicon system does full-body skeleton reconstruction, and we’re tracking the user’s hand position to detect object interactions: the spheres “bounce” and disappear when he touches them. They seem blurry in the video because the two stereo-separated views are superimposed.
How does it work?
We’re using the Vicon system to implement a 6-degrees-of-freedom headtracker, with which we can approximate the user’s field of view in the virtual world. Using standard OpenGL cameras, we then render two fixed-fov views, one for each eye. The “magical ingredient” is then a GLSL shader to raytrace these world views onto a theoretical cylinder, representing our projection surface. These final images are then drawn (quadbuffered) on a fullscreen quad spanning the projection screen.
The video unfortunately shows only a rather boring environment view – it’s amazing how real and immersive more complex environments feel when experienced through our setup.
I’d like to thank my esteemed colleague Martin Löffler for the productive collaboration and all the fun we had!