CS 6770 Virtual Reality

Dr S Semwal

Irving Rynning 3rd May 2012

Note bene: I have used the several words of each of the papers in quotes as an abbreviation, and each is defined in italics following the title.

PART A
Question 1: (10 points) (Can you help me concentrate Room by Adi and Roberts Can you.. Auto calibration of Cylindrical multi-projection systems by Sajiti and Majumder Auto Cal.. The effect of Tiled display on performance in Multi-Scree Immersive virtual environments by Agana et al Tiled Display .. We consider several large projections displays such as walls, screen, or tiled displays

(a)(5 points) what are the two main advantages and disadvantages of each one of the three systems Although this begs the question of to (or from) whom the advantage is directed, we will presume that both realism and cost are being evaluated. Can you... , where being in an environment in which the perceived reality is changing independently of what one is doing was the most surprising I would be distracted by the view, as so often I am anyway. Costs were not mentioned, but as this was recent (2010), and it seems CAVE has matured, I presume that unlike the Cruz-Neira days (1993) they are no longer extravagant, especially if the researchers invest their own funds. I know I would. A disadvantage would be the complexity of projecting a five sided environment, but again is the presumption of an

already developed methodology so that the evocation of the images is a given the experiment was not to develop such, but to test the interaction with the subjects. Auto Cal.. is a different sort of horse: here we are directing our effort towards the development of a suitable procedure to meld projection systems, and are not concerned with what is being projected, though a CAVE application was mentioned. Like auto-focus I have found such methods arcane, meaning beyond my initial comprehension, and I see it as a magic of sorts. What I got from the work is that, given the geometry of the projection surface, one can with a suitable camera minimise distortion and boundary errors. An error of one pixel out of a thousand is good~! As the costs were born by sponsoring companies (Canon, Epson) and The Disney Corp., I presume they received some benefit, especially in that they got to watch. This point of view is continued in Tiled Display... where costs are considered in developing a substitute for the expensive, single pane displays in a matrixed one, i.e., several used as tiles in the projection. Not unlike looking to the world through a mullioned window or an automobile the paper barely scratched the issue, proposing several studies in testing the discernment and the drawbacks of the borders. Advantages: low cost, portability; disadvantages: probable distortion, calibration, and the viewer accepting that he is not quite a part of the created world.

(b) (5 points) Discuss one new (not described in the paper) application of a system where the walls come alive. Assume that full 3D graphics, sound, tracking, and haptic interaction are at our disposal as well if needed. (two pages).

By walls comes alive we will accept the paradigm that a viewer / subject / participant accepts the surroundings as part of his world: an extreme test would be the evocation of fright or the perception of threat, comfort or sanctuary being more difficult to observe, though the former may raise ethical questions. The American Psychological Association (APA) frowns upon such invasive techniques. The most obvious measure of such an environment is a game, in which the subject and his world interact, with score being kept. It is suggestive of the later paper on AstroJumping, in which the user gets feedback, an indeed one could ask for nothing better than to reward and punish the subject. As well as a CAVE 5 sided space including the floor, one would like at least a 8-12 channel audio system the four corners of the cube, plus top and bottom, presuming the cube-like shape; tracking should be bi-optic, so that the attitude of the head is known; and as for any haptic, a special chair, stool or platform, implements in a rocking motions, pitch/roll/yaw, or ejection note that a more complex environment would add air temperature, air currents, heat/cold sources, illumination and smell. But I am not sure what we would achieve, other than to immerse the subject the test parameters would need to measure how involved he was, and such personalization variables would need to ascertained a priori, though a learning process is inevitable. 4

Question 2: (20 points) (Surround-Screen Projection-based VR: The Design and Implementation of the CAVE by Cruz-Neira et al. P135-Cruz-Neira.pdf CAVE VR Movie with scents paper (MovieWithScents01667625.pdf); Scents Camera parameter Estimation method using infrared markers for Live TV Production by Mitsumine et al (LiveTV2008[1].pdf )) Live TV

(a)(5 points) What is the significance of the CAVE for VR today? Explain.

As Dr Cruz' paper was done almost 20 years ago, several presumptions must be made as we noted in question one, where CAVE-like environments, now seemingly a de facto standard, are still being used with success 1) that it is a useful, accurate, creation; 2) that it is economically feasible; 3) given the magnitude of the change of the speed and computational power in processors, many roadblocks have been overcome. and that at the least is the significance.

And I presume Dr Cruz has her excessive green phosphor persistence issue solved. Colour televisions did not quite have the concern, since their vertical refresh rates were slower. 5

(b) (5 points) Explore the connection between sense of smell and emotions/memory? How can the aroma be used for movie presentation based on the olfactory paper for movie presentation? Unfortunately I never seen the Japanese movie Spirited Away...

The concept has quite a bit of merit, as I noted in the classroom discussion. The reaction times, as well as the clear all function required on our olfactory sense, preclude any transient, dynamic use of scents, but overall the authors did an outstanding job in interpreting the movie with their odorous contribution. I, too, have the association of memories with scents, but I think that the re-learning of associations may be difficult, especially ones where there is a strong emotional attachment cookies, ex-lovers, or for soldiers, blood and death. For presenting a movie with augmented scents (other than stale popcorn, dried, spilled soda pop, or worse..), one needs a thorough review of the scenes and an aesthetic judgement as to when and how much and what type of scent. The four minute example in the paper seemed rather short, and too is the added concern of over-stimulation, clearing the air and the attendant noise and cooling effect and anyone's memories.

(c) (5 points) What is the use and limitation of the LiveTV paper idea? As noted in the mid-term exam the limitation is the cumbersome hardware required: two lenses of differing sensitivity parameters, and skilled operators. The advantages are the use of infrared markers which are not visible on the set, and their independence to environmental conditions.

(d)(5 points) Using the above three system, suggest one application where all there will be useful systems to have. Explain. (Page limit: three pages).

What we have is at least five walls of projective, interactive displays, accompanied by dinosaurs popping up whenever, and the scent of daisies, barnyards roasting chickens, and (off topic) is barking dogs, bird calls and a melody.

I walk outside except I am outside and a breeze comes up, a little bit of ozone and a thunderstorm: the sky darkens, a flash, I tun my head to just catch the bolt of lightening, and a resounding roar assaults my ears. My pet theocant nods his head by my arm, and I catch his heady, musky breath. Across the field of corn pops up his friend, and disappears as I notice him, turning my head in a vain attempt. I notice growing shadows, and the air turns to crisp, wispy mountain scents of columbines and Indian paintbrush, and I walk a bit to pat a dog's head a good day, I think, and head for home as a baking pie reminds me of my childhood.

I am immersed.

PART B: RESEARCH SECTION

These questions would be graded on the basis of the approach you would take. The answers must reflect your analysis. You should not look for some articles in the library. Your answer should not be more than five pages long (six when counting the figures).

Question 3: (35 points) (Simulation of Genetic Inheritance in the Generation of Virtual Characters Viera et al (GeneticFacespdf); Twinkle: Interacting with Physical Surfaces using handheld projector by Yoshida et al; Upponrukka: An inexpensive Immersive display for public VR Installation; ARToolKit).

Over the years, we have performed several experiments using ARToolKit and found out that ARToolKit can detect simple patterns most of the time accurately. Sometimes if the angle of the camera is very acute then detection of markers could be a problem. In addition, the distance from the pattern can also be a limitation as that is inherent problem of camerabased systems. Further away the marker from the camera, less robust is the position estimation of the pattern. Some benefits of ARToolKit are: cross platform availability, cheap pattern creation, and possibility of mobile systems with simple camera activation. It will be interesting to extend ARToolKit for measuring accurate distances in mixed reality virtual environment.

(a) (15 points) Suggest a system to use ARToolkit to assist in mixed-reality (AR/VR applications, so that genetically varying faces can be generated easily. Describe the system. If needed, assume that we can recognize several (as much needed) unique patterns in an ARToolKit environment.

We will presume that the AR tool kit is the software required to recognise and decode the marker patterns of the black squares, and project a virtual image replacing the marker as the markers themselves usually use a variation of the Hamming code, with modified parity bits so there is no entry with an all-black or all-white encoding. The squares themselves are such, and thus any camera angle deviation can be corrected. To generate the faces one needs only a 11 map to the set I have seen the AR toolkit used to create a virtual jigsaw puzzle, where one clicks and drags the markers' projected image. The encoding may take on any form, and the faces need not be correlated in any sort of order. One could map the marker identifier with a genetic code of sort, with isomorphisms for skin colour, eye colour, and for width of nose, ears, eye pupils, et cetera. A more intense coding may be necessary, for the lower end markers are of order 321024: one would need thousands at least. One might use smaller squares, or go to equilateral triangles or circles it is important to be able to correct for camera angles and illumination variances like shadows and reflections.

(b) (10 points) Use ARToolKit in Twinkle system and suggest an accurate estimation game by combining markers in twinkle environment. In a Twinkle system there is a flat surface to which we will first add static markers; the added marker induced images would virtually interact with the environment so when they are activated, they realise they can only be so large, or twist and bend themselves to accommodate with the surface; and with camera/projector movement changing, moving themselves, bouncing off, and perhaps crying out ouch. Taking the question's estimation game literally say, making stabs at the dimensions of an object which object will we then guess ? If we move our little guy from marker to marker, he will change size perhaps, or grow horns or turn green. We could have a death marker, and so the problem becomes a maze to avoid the trap. We could use the motion to guess at the camera angle, and as we get closer, our displayed friend changes colour, grows larger, or like the children's game, one gets warmer or colder, or the computer might pick a marker at random, and our friend then indicates how close we are. In combining markers we require a function that maps into the added complexity of both a projected creature with the marker-induced images which might interact another game, avoiding the dangerous ones, embracing the friendly ones, and perhaps some sot of goal. Another way to to have the markers moveable on the planar surface while a second user operates the Twinkle hardware. Since the squares are of a fixed external circumference, the system knows how far one is relative to another, and upon detection of the moved marker, can project the image and compensate for its size correctly, larger or smaller. 10

(c) (10 points) Upponrukka system can be considered as outside in system, whereas ARToolKit could be thought as inside-out system. Suggest strategies to combine the two systems facilitating accurate measurement of distances in a virtual environment. State your assumptions, if any. (Page limit: six pages).

The presumptuous assumption that we are measuring the movement of markers on a subject using the markers placed upon said subject then in the display projected objects would grow or shrink, and even explode were one to get too close. Given the geometric rigidity of the AR Toolkit markers, any distance can be extrapolated accordingly. The original Upponrukka exhibit used coloured balls mounted on the viewer, giving gross estimates of the viewer's position, which may not have given accurate distance measurements, let alone the angle of the head by substituting ones with known dimensions such inaccuracies can be thwarted, possibly enabling the subject to nearly circumnavigate scene (up to the limit of the projectors). The downside is that coloured balls look the same from any angle, especially from over the subject's head, as in Upponrukka the markers do not, so if one is occluded, tracking may be lost.

11

Question 4: (35 points) (Way-finding and Navigation Semwal et al (BackupVRandVisually ImpairedpartOne.ppt); Navigation System for the Blind (NavigationAssistive.pdf); Astrojumper: Designing Virtual Reality Exergame to motivate children with Autism to Exercise by Finkelstein et al.; Lightweight bleeding and Smoke effect for Surgical Simulators by Halic et al)

(a) (15 points) Describe use of VR for visually impaired based on Wayfinding and Navigation, autism care, and surgical simulation what is the limiting factor or disadvantage of using VR for these three applications. The use of VR for the above for the blind suggests a technologically advanced method of solving not only the algorithm, but in producing a social value through one's talents ditto for the autistic child. As I have read on the use of the Senasable Phantom in dental surgery, a well executed simulation can be priceless and I think there is a program for veterinarians as well. The use of VR for the blind essentially involves creating a space inside the subject's head, and mapping it. Of course one cannot move his body to any extreme unless he were strapped in a robotic throne of sorts but using a haptic device and sonarlike sounds one might be able to feel one's way around. Of course the obvious GPS-style turn left in one hundred meters... in a pleasant voice that never gets angry when one misses the turn can be used, presuming one has the voice. If I had a navigator in my ears, either I would love it, or hate it... no middle ground. 12

For a game playing autistic child, nothing could be more fun than that a little exercise, especially with obvious rewards like exciting sounds, and pretty lights. I do not intend to sound trivial, but I know I would like that. The game is totally immersive in a CAVE environment, and the subject then can engage himself, like in real life, of which for the autistic child there may not be much. I can envision a similar style for boxing (punching the spaceships), or soccer (running in place, of course). The surgical VR was an effort aimed at providing visual feedback for the operator using the graphics processor, as opposed to the CPU. Bleeding (intersection of the scalpel and tissue) and smoke (visual distortion caused by air currents, lighting, and other internal fluids) was developed using software ported to the GPU using algorithms previously developed. Regarding any limitations, the first thing that comes to mind is the sense of balance, or one's spatial orientation e.g., the Astrojumper might find his jumps out of bounds, and certainly any surgeon might enjoy seeing even more blood, overtaxing the processors. The blind depending on how long their disability has been an issue might find being led by a navigational device disconcerting: a similar effect is produced when one is in an automobile being driven by an erratic or inattentive driver panic. Another disadvantage is the cost a CAVE setup is not cheap, nor are haptic devices. The navigation tools require blind subjects upon whom to test, and of course the Astrojumper is not particularly portable. Again there may be ethical issues if the simulation of flying asteroids is too realistic.

13

(b) (15 points) Add one point force feedback (haptic) device to the surgical simulation paper proposed in the paper Device a strategy so that blood can be felt by such a device.

The haptic feedback for blood could easily be implemented by a slight dragging counter force as one draws the scalpel or tool, but that might be too similar to having a variable sharpness or edge (and dullness) on the virtual tool. I would propose perhaps an audible signal in conjunction with a slight vibration at an appropriate frequency, say, 20 30 Hz. A shuddering movement orthogonal to the incision might also be effective. Of course the visual cue must be implemented. This is all based on the premise that bleeding is to be avoided, as any haptic cue which affects the user's hand might tend to disrupt the operation.

14

(c)(5 points) What other features can be added to the Astrojumper to make the experiment better for autistic children? State your assumptions, if any. (Page limit: six pages).

I would think features that make the experience even more immersive would be valuable: lights, colours, sounds, and easily discernible scoring rules tracking markers which are transparent in the sense of being unrestrictive to the player, as well as a pre and post game routine which minimizes any anxiety. Of course we are talking about the setup for the child. For the experimenter to develop the experience, one needs access to lots of subjects, meaning a convenient location and all-hours accessibility, as well as enthusiastic researchers , integrated briefing/debriefing sessions, and recording hardware and monitors which allow essential data to be observed and recorded. One needs not any double-blind or sophisticated theories to assess any benefits, but to justify the costs such an attitude may be indispensable.

15