Professional Documents
Culture Documents
Mixed Fantasy: An Integrated System For Delivering MR Experiences
Mixed Fantasy: An Integrated System For Delivering MR Experiences
University of Central Florida Orlando, FL 32816 {ceh@cs.ucf.edu, cstaplet@ist.ucf.edu, moshell@cs.ucf.edu, pmicikev@cs.ucf.edu, darin@cs.ucf.edu, peter@dm.ucf.edu} Abstract
This paper describes the underlying science and technologies employed in a system we have developed for creating and delivering multimodal Mixed Realities. We also present several aspects of a number of experiences that we have developed and delivered, when such aspects help understanding of algorithmic requirements and design decisions. The technical contributions include a unique application of retro -reflective material that results in nearly perfect chroma-key, even in the presence of changing illumination, and a component-based approach to scenario delivery. The user interface contribution is the integration of 3-D audio, hypersonic sound and special effects with the visuals typical of an MR experience. realistic visuals required of a multimodal MR entertainment experience [5]. Section 2 presents the interesting aspects of our MR Graphics Engine, focusing on our contributions to the areas of object selection and registration. Section 3 describes the audio systems we have developed, looking at our 3D audio design and delivery system and our experiments with hypersonic sound. Section 4 describes the technologies of show control devices (called macrostimulators here) and the creative uses we have of them in MR experiences. Section 5 describes our scenario authoring and delivery system. Section 6 ends the paper with a brief discussion of directions that our current research efforts are taking.
2.1. Rendering
Rendering is done in three stages. In the first stage, images are captured by the HMD cameras, scaled, and
ISMAR 2003
processed to correct for lens distortion. Since the cameras provide only RGB components for each pixel, alpha channel is added by chroma-keying, as described in the subsection below. Alpha-blending and multi-pass rendering are used to properly composite the virtual objects with the captured images. For example, if a bluescreen indicates an opening in the wall, virtual characters are rendered realistically when they are behind the portal and infrared vision effect may be used whenever the characters are behind the wall. Detection of a laser-dot, described in the next subsection, is also performed during the first stage. In the second stage, the virtual objects are rendered and combined with the output of the first stage. Depending on the application, multiple passes and alphablending are utilized as indicated above. In the third stage, the output of the second stage is used to determine whether the laser-beam intersects any virtual or real objects. More details on the process are given in the following subsections.
intersection are computed, the list of the virtual objects is traversed to determine which object has been hit. Note that this approach automatically selects the first object hit by the laser, while tracker-based techniques require an additional step to sort the objects. However, the laserbased approach requires the user to be looking at the intersecting object, while tracker-based methods do not. We extended the above method to also detect intersection between the laser beam and real objects. For example, in several scenarios we developed, a macrostimulator is invoked to flip a real object when it is hit by the laser. To achieve this functionality, a transparent polygon is rendered in the same global 3-D position as the real object. Thus, while the RGB values of the image are not affected, the corresponding position in the depth buffer is modified. Since before every rendering step the depth of all pixels is set to infinity, laser-object intersection is detected by a change in the depth value. Global coordinates for the intersection are then computed by unprojection and used to identify a real-world object to be affected by the macrostimulator system.
ISMAR 2003
uniformly distributing the LED directions across the view frustum. The maximum distance is determined by the LED intensity. The four images below were captured using the HMD and our MR software. Figure 1 shows the environment and the retro-reflective screen while Figure 2 shows the same scene illuminated by the camera-mounted LEDs.
The best and worst results after chroma-keying are displayed in Figure 3 and Figure 4, respectively. Each pixel determined to have the color emitted by the LEDs appears as red; all others are unchanged. Chroma-keying is done by our MR software in real-time (rather than coloring the pixels red, our software sets the alpha bits for appropriate blending when rendering virtual objects; here the virtual object is a red polygon).
ISMAR 2003
The MR Sound Engine combines these attributes to provide the ability for real-time spatialization of both moving and stationary sound sources. For stationary 3-D sounds, the source can be placed anywhere along the x, y, and z axes. Moving sounds must have a start and end point and a timeline for that movement to take place. Simulation of emitter size, sound orientation, and rotating sound sources are supported. Emitter size can only be altered for sounds that are spatialized. Sounds that do not have spatialization are called ambient sounds and play at an even volume in all speakers. The orientation of a sound is defined by an inside cone and an outside cone. Parameters may be adjusted to change the size and volume of these cones. Rotating sounds, both those that rotate around the user and those that rotate around their own axis in space are supported. Standard features such as simultaneous playback of multiple sounds (potentially hundreds using software buffers), ramping, looping, and sound scaling are supported.
We provide a flexible approach in which sound types and their parameters are predefined by the positioning engine, but most of the parameters can be changed by the Scenario Engine in real-time (Parameters, for which runtime change doesnt make sense, like class type, ID, filename are excluded.) The MR sound engine provides realistic spatialization of sound suited for Mixed Reality applications. By using a multi-speaker system instead of headphones, real world audio is mixed with simulated sound. This feature enables a true MR experience without the cumbersome task of head tracking with real-time response, or HRTF calculation.
ISMAR 2003 Virtual sound sources are also compelling, but are more difficult to apply in practice. When the hearer is behind the plane of the face of the HSS unit, the hearer almost always correctly localizes the virtual sound source to within ten degrees of its actual azimuth. However, if the hearer is in front of the HSS unit, there is substantial potential for confusion between the direct and virtual sources. There is some evidence for differences in the angular dispersion of virtual sound sources, depending on the material against which the beam is directed. This is to be expected, based on the physics of the system.
lighting signifies the start of a scenario, change the mood as it is dimmed, and reveal new parts of the set on demand. Effect lighting allows us to illuminate smaller areas where another macro-stimulator event might be happening, or simply act as an interactive element. Heat lamps let the user to feel the temperature of an explosion and flickering LED lighting can simulate a fire. In one of our simulations, users are instructed to shoot out the lights to provide cover for fellow teammates. A motion tracked shot allows us to flicker out the light (over a door or window), while adjusting parameters of virtual adversaries to make their shots less effective (this latter adjustment is done within the Scenario Engine). Compressed air and smoke We have incorporated pneumatic actuators to move pieces of sets. The actions we use include flinging a door open to reveal a virtual character, toppling a pile of rocks, etc. Possible uses of compressed air include simulating an explosion, a rocket taking off, or (combined with lighting) simulating an underwater environment.
4. Macrostimulators
Computer controlled macro stimulators are special effects applied in a new way, freed from the linear storylines of most traditional show control systems and applied to the next generation of interactive simulations. Macro-stimulators in a mixed reality allow for rich enhancement of the real space in response to actions by either real or virtual characters. When coupled with our Scenario Engine software, this system creates the next level of mixed reality one in which the real world knows and is affected by items in the virtual, and vice versa.
5. The System
The MR Fantasy system encompasses both a scenario authoring and a scenario delivery system. These are integrated in the sense that the authoring system produces an XML document that is imported into the delivery system, resulting in the instantiation of all objects needed to run the scenario.
4.1. Effects
We apply traditional special effects, the same that might enrich a theme park attraction, theater show, or museum showcase. The following is a breakdown of just a few of the types of effects we use. Figure 5 depicts a scene in one of our scenarios with labels showing places where we have inserted effects.
5.1. Authoring
Scenario authoring is presently done by selecting actors that reflect the virtual and active real objects of your world. For instance, a scenario might contain a virtual helicopter, several virtual people, a number of real lights, a real soda can, an abstract object that is acting like a story director and, of course, the real person who is experiencing the MR world. An actor component, when brought into the authoring system, will have a set of default behaviors, some complex enough that they will have been pre-programmed. However, a finite state machine, consisting of states and transitions, is the primary means of expressing an actors behavior. Each transition emanates from a state, has a set of trigger mechanisms (events) that enable the transition, a set of actions that are started when the transition is selected, and a new state that is entered as a consequence of effecting the transition. A state can have many transitions, some of which have overlapping conditions. If multiple transitions are simultaneously enabled, one is selected randomly. Weighting can be achieved by
Lighting We lace our exp eriences with lighting to illuminate the area of the experience and also as localized effects. Area
ISMAR 2003
repeating a single transition many times (the cost is just one object handle per additional copy). States and transitions may have associated listeners (attached via hookups in the authoring environment) causing transitions to be enabled or conditions to be set for other actors. Actors, finite state machines (FSMs) and transitions can be archived so that completely specified actors can be chosen, or new actors can be composed from the FSM behaviors stored with other actors (that serve as prototypes) or from behaviors stored independently or even from transitions stored independently. Totally new behaviors can be created by starting with a blank slate, adding states and transitions, and the actions that are associated with these transitions. All components are Java beans and can, consequently be assembled without the need (in most cases) of looking inside their code. The exception is when it is convenient (or perhaps necessary) to add a new Java service that is invoked as part of an action associated with a transition. Our intention is to alleviate this (somewhat) in the future by supporting services written in JPython. At present, our library of actors and behaviors is limited by the scenarios we have already created, but this will grow substantially as we and others develop new scenarios.
effected. For instance, an audio request goes to the MR Sound Engine, requesting it to play a ricochet sound emanating from the spot where the bottle was located; a special effects request goes (via DMX) to the Macrostimulators to activate a servomechanism that causes the bottle to jump; and a request goes back to the graphics engine to deactivate the bottles virtual stand-in. One critical aspect of the above is that only the scenario engine understands the semantics of events. All other parts of the system are either purely reactive (audio and special effects) or can also perform some specialized action, such as the graphics engine analyzing a scene, but do not determine the consequences of this analysis.
6. Future directions
Recent publications on rendering have demonstrated offline integration of real and virtual objects [1]. However, extending such work for real-time computation does not seem straightforward [3]. Our approach, based on a hardware-assisted algorithm, creates accurate and near interactive rendering of virtual scenes. Extending this algorithm to render virtual objects with proper illumination in real scenes is a goal that now appears attainable with
ISMAR 2003
the aid of a simple cluster of workstations, each with programmable graphics cards. This same cluster approach, planned for interactive illumination, has already shown great promise in occlusion-based level-of-detail management. Our goal is to apply this research, currently focused on real-time walkthroughs of complex, virtual forest scenes, to achieve such performance in mixed realities involving augmentation with complex and evolving vegetation. This is especially desirable in informal science MR exhibits that will allow visitors to change environmental conditions and then experience the resulting augmentation to a real world environment. Our current delivery system, while effective, will not scale to massively shared MR worlds, populated with large numbers of virtual objects. Our design for a future system is being informed by Enterprise environments that have made effective use of scalable middleware, managing shared services based on the balance between global efficiency and individual quality of service criteria.
audio software), Tim Eck, Felix Hamza-Lup, Scott Malo, Chris Murray, Matthew OConnor, Preet Rawat, Eileen Smith and Ron Weaver for their dedicated work on this project.
8. References
[1] P. Debevec, Image Based Lighting. IEEE Computer Graphics and Applications, pp: 26-34, Mar/ Apr 2002. [2] D. Gelernter, Generative Communication in Linda, ACM Transactions on Programming Languages and Systems, 7(1), pp. 80-112, January 1985. [3] S. Gibson. and A. Murta, Interactive Rendering with Real-World Illumination, Rendering Techniques 2000 (Proceedings of 11th EG Rendering Workshop), pp. 365375, June 2000, Brno, Czech Republic. [4] C. B. Stapleton, C. E. Hughes, E. Smith, S. Malo and C. Murray, Where Realities Collide and Imagination Soars, submitted to IEEE and ACM International Symposium on Mixed and Augmented Reality, October 2003, Tokyo, Japan. [5] C. B. Stapleton, C. E. Hughes, J. M. Moshell, P. Micikevicius and M. Altman, Applying Mixed Reality to Entertainment, IEEE Computer 35(12), pp. 122-124, December 2002. [6] S. Uchiyama, K. Takemoto, K. Satoh, H. Yamamoto, and H. Tamura, MR Platform: A Basic Body on Which Mixed Reality Applications Are Built, Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality, September/October, 2002, Darmstadt, Germany.
7. Acknowledgements
The authors gratefully acknowledge the inspiration and guidance provided by Dr. Hideyuki Tamura, Ritsumeikan University. This work was supported in part by the Canon Mixed Reality Systems Laboratory, the US Army Research, Development & Engineering Command (RDECOM), the Office of Naval Research, and the National Science Foundation (under grant EIA 9986051). We are indebted to Nick Beato, James Burnett, Alex Cook, Daniel Dobler (the creator of the first version of the 3D