Prototyping in Virtual Reality is quite difficult – and more so in Augmented and Mixed Reality (MR) where the physical and virtual realities unify. In this project, we propose a novel approach and system to create Mixed Reality experiences rapidly, for anyone who has a typical MR headset. Called SpatialProto, the system allows users to directly record and “scratch out” objects in their live reality. These recordings can be used to create any kind of interactive scene.
SpatialProto combines a video see-through AR headset with a depth-sensing 3D camera to accomplish a novel, immersive easy-to-use UI. Directly record what you see, in motion, e.g. a dancing person or an object that you hold and move – no need to pre-make components. This is complemented with features for background removal, placing the recording in the environment, and chaining multiple recordings to an interactive experience.
The concept of SpatialProto consists of three stages.
In our paper, we (1) demonstrate how rapid prototyping in mixed reality can become much faster and efficient, (2) report artifacts designed by novice users within a short period of time, and (3) show how one can create advanced immersive experiences such as a mixed reality game or AR user interface prototypes.
The novel, direct way for prototyping in 3D opens rich features for many people to get creative with MR experiences and contributes to the vision of the MR design app of the future. A few examples we build:
Real world objects can be recorded as shown in the top row. Then, an MR designer can integrate recordings directly into the reality, for example a virtual ground, a box with a flying coin, a Rubic’s cube animation, or even a real user that is seemingly punching you! But don’t worry – this is virtual, you won’t feel the punch. Though this will be supported in SpatialProto 2.0. Hang on until then, and check out the current version:
L. Müller, K. Pfeuffer, J. Gugenheimer, S. Prange, B. Pfleging, and F. Alt. SpatialProto: Using Real-World Captures for Rapid Prototyping of Mixed Reality Experiences. In Proceedings of the 2021CHI Conference on Human Factors in Computing Systems (CHI ’21), Association for Computing Machinery, New York, NY, USA, 2021. doi:10.1145/3313831.3376840 , PDF
Ken Pfeuffer 