Isqua Istari

The Wise Wizards

PS Cast: Oculus and VR

Posted in PSCast:S3 by Ziggy Friday February 20, 2015 at 22:06

You’re in for it now! I’ve heard of some new technology and I’ve got all sorts of little ideas!
Listen to them all repeated over and over again in today’s blathering about the Oculus and VR!


[marginally musical vocal intro]
Thank you for joining me today. This is the Paul Spooner Podcast.
Luke, Dan, and Ryan bring you this podcast via Patreon.

So, here’s some arm-chair musings on the Oculus Rift. I don’t own one, I haven’t ordered one. I’m not planning on getting one. I think the VR technology, while neat, is kind of gimmicky at the moment. I don’t think it’s going to be valuable until we have the technology to produce worlds that behave realistically (which goes into the whole Fledgeling thing, and artificial intelligence) but they are cool. I don’t know what all people are doing with them, but here are some things that I’ve considered and thought about based on the stuff I’ve heard.

One of the things I’ve heard is that VR has a very nauseating effect when the head tracking suddenly turns off. So basically, when you’ve got this thing on your head, it’s got to track where your head is pointing because it’s got to update the images properly. If that’s working fine, your brain starts thinking of it as a real place. It’s like “Alright, I’m looking around, I’m moving my head, everything moves.” And if, suddenly, the head tracking turns off and it freezes the image, so that it no longer moves when you move your head, then your brain thinks that you’re dying. It basically says, “Oh. I think I’m moving my head. My neck sensors tell me that my head is moving. My inner ear tells me that my head is moving… but my eyes tell me that my head is not moving. So, clearly, there is something terribly terribly wrong, and you need to sit down and be sick for a while, while we figure this out… while we figure out what just happened.” So you get incredibly ill from this.

So I was thinking about this, and I was like, “Okay. Sure! That makes sense. Your brain is trying to protect you and it’s like ‘Hey, there’s something terribly wrong. You’re going to feel really bad. You need to sit down and relax for a little bit, you know, while we do some calibration or something.’ But what if you did something different?”

What if you did it intentionally?

Because most of the time it’s an error, right? It’s a problem where the machine freezes or the software can’t quite keep up so it stutters, or there’s a loading screen so it doesn’t have anything to display. Things like that. So usually it’s not intentional. But what if you did it intentionally? And I don’t just mean like freezing the screen. I mean like intentionally messing with the head tracking.

So, one thing that I thought of that you could do is, head tracking is just a vector. It gives you the vector, and then you have to move the camera in-game to move it properly to get the correct update. So, what if you took that vector, and instead of taking it at face value, you changed the amplitude very slightly, so that instead of rotating, you know, you rotate your head ten degrees and it rotates it [the vector] ten degrees plus point one degree. Very small! And you do this incrementally so that it’s really slow, so your brain doesn’t notice. But what it will be doing is it will be training your brain to re-synchronize, to have a different synchronization between your eyes and your inner ear, and your eyes and your neck muscles.

And then, when you take the headset off… You’ll get that crazy, punch to the gut, “Oh no, something’s terribly wrong” feeling. And so it will actually cause you to feel sick from stopping using the VR headset. So this could be really useful if developers want people to stay in VR for a long time, they just do this thing where it slowly de-tunes your neck muscles so then if you take it off early or you take it off in the middle of an experience you feel really ill. It’s going to train players to not do that. And then at the end of the experience it tunes it back to normal, so that when you take it off at that point, then everything feels fine. So you could be very subtle about it and probably do some really subtle training on people to make them play your game or stay in your VR experience, even if they don’t want to, because they’ll be trained that they just feel horrible when they take the VR headset off before they’ve finished the experience.

So that’s something. Watch out for that. It’s possible and I think people will try it, probably, if people aren’t aware of it.

Another thing you could do is de-synch the eyes. So you’ve got this dual head-tracking thing where the tracking occurs on both your eyes at once, usually. I mean, if you want it to be normal, that’s how you do it. So you could de-synch the eyes. You could have the vectors slightly different. Maybe when there’s an explosion or something, there’s a camera shake, have the eyes shake independently so that it’ll feel like your eyes are rattling around in your skull. It would give you a really unnerving feeling of actually getting shaken without actually feeling anything. You might get some synthsensia (EDITORS NOTE: I meant to say synesthesia, but if you know the word then you probably knew what I meant, and if you didn’t the slightly incorrect pronunciation probably didn’t throw you too far off.), though, I don’t know if it’ll work, but it’s worth a shot.

Another thing you could do is… Oh, hang on, I’m coming up on the toll booth. [exchange in Japanese] Okay, so, head tracking. Another thing you could do is make players feel like they are changing scale, like growing or shrinking, by moving the eyes closer together or further apart. That’s an easy thing to do, just changing the parallax. And another thing you could do, for combining these effects, you have, I don’t know what this would do, but you have the really weird effect of changing the amount that you’re applying the direction vector between each eye… de-synching the eye rotation. So basically it would cause you to go cross-eyed, or go wall-eyed, if you de-synched the head rotation from the two eyes. So, that could be interesting. If you were trying to train yourself to be able to go cross-eyed or be able to go wall-eyed or do eye muscle excercises then you could do that where even if the player is holding their head still, the image slowly moves in and out so that your eyes have to track it to keep track of where that thing is. So that could be an interesting thing. I don’t know if it would be worth it.

Certainly you could make someone feel tired or drowsy. Like just waking up, or just falling asleep. Because I know this happens to me a lot. Right when you wake up, it’s difficult to get your eyes to focus on the same spot. Usually one will drift higher or lower than the other… usually you can get them laterally to synch up, but a lot of the time it will vertically drift out. So you could do that to make the player feel like they’re really drowsy or feel like they’ve just woken up, by de-synching the two eye vectors vertically, so that they’re pointing in different directions vertically, so your eyes can’t quite synch up vertically. It’ll make you feel like “Woah, I’m not feeling…” It won’t make you feel sick, it’ll just make you feel kind of drowsy, like, “Oh man, I’m really tired. I’m really fatigued.” Because I know that’s happened to me a lot. I’ve observed that in my own experience when I wake up, and your eyes don’t quite match up. And it’s okay, as long as it’s slow and not sudden, I don’t think it would make you feel sick, just make you feel drowsy.

I’m not sure if there’s anything that could cause you to feel alert? And all of these are of course going to be temporary. It’s not going to be a permanent feeling. Because your brain is very good at adapting and changing the way it behaves. But maybe changing the light levels?

Or changing the peripheral? In computer games if you grey out the edges of the screen or red out the edges of the screen that gives you the feeling of your vision closing in. But you could really do it with VR. Especially if you had eye tracking, but even, I think, without eye tracking, if you just kind of give a tunnel vision kind of effect, you could make the player feel like they are under a lot of stress. Or feeling dehydrated, or whatever it is that cause the blood flow to constrict to your eyes… or reduce blood flow to your eyes. So that could be interesting.

(8:42) So people talk about enhanced reality, and being able to do things where you’re in a real place and you can see real things, but then you put on your VR goggles and it’s got a camera in it and it overlays virtual stuff onto real stuff. And that’s… I mean, that’s clever and interesting but I think one of the real values in VR will be telepresence, where you can put on a VR headset and have a set of cameras in some other location and then feel as if you’re there. You know, move your head around and look through the cameras as if you were in that spot. But it’s going to require a little bit of finesse, because you have to do head tracking to make you really feel like you’re there, or else it will feel like your head is locked in place, and it won’t feel quite right. So you could put a chair in and so then your head doesn’t move. And you have a head-rest where you set your head and then you can rotate your head but not move your head. And so then it will give you more a sense of being there because the cameras aren’t going to be moving in real time to suit your head motion. Although you could do some of that, I suppose. Have it on a gimbal or something. You’d need a really tight control loop.

But one of the things that I thought of that would be very useful (Really handy. I think it would be really advantageous.) is:

Using VR to replace airplane windows.

And there’s several reasons for this. If you’re trying to do enhanced reality, and you’re standing on the street… you know what the street looks like! So you’re not going to be fooled by any kind of glitches, any kind of errors. It’s going to be like “Oh, that looks totally fake.” But people rarely fly in airplanes, and the outside of an airplane is not an experience that someone is used to having, so it will be much easier to have a lower quality experience that will still feel real and convincing.

So what you could do is, instead of having windows all along the side of the airplane, which reduces the structural strength and is hard to manufacture, all this kind of thing… You instead break all those out, make the airplane solid, don’t put windows in, and then put blisters of cameras on the outside, (so you’ve got a whole bunch of cameras, kind of like a bug’s eye, that are all pointing in different directions.) And then you can interpolate between them and use that image data to allow the user to do head tracking and look around inside this spherical image. And so then, if they move their head a little bit [spatially] it doesn’t really matter because everything is so far away that the paralax isn’t going to have an effect… or enough effect that it’s going to take them out of the experience. And also, you can have everyone looking out of any position they want, so you put a blister on the nose of the airplane, then everyone can be looking out the nose if they want, or not! They can be looking out the side, or out the roof, or the bottom of the airplane. Off the tip of the wing! Anywhere!

So it would really give you a great advantage to people who are feeling claustrophobic, because they could be really outside the airplane instead of being trapped behind this tiny little window. It would give a great advantage to people who were acrophobic because you wouldn’t have any windows in the entire airplane, you could just feel like you’re inside, you can’t see anything, it’s all blocked off. So if people wanted to sleep, no one is opening the windows, it’s all done on VR headsets.

So I think, absolutely, that would be an extremely valuable thing, to put VR headsets and blisters on the airplane. And then if there was some problem, or if there was some sort of thing where you didn’t want people looking out the windows for whatever reason, you just turn it off! And then people can’t see outside! There’s nothing they can do about it! You could have all kinds of other experiences. You could even have recorded footage of the outside of the airplane, so people would feel like they’re looking outside but they’re actually not, they’re actually not seeing things outside, if, for whatever reason, you didn’t want the passengers to see something. You were flying over or flying by.

So I think this would cause it to be easier for airlines to fly through restricted areas, because they could say, “Look, only our pilots are going to be able to see outside. All the passengers will think they are seeing outside but they’re actually not, they’re seeing recorded footage, or doctored footage of some kind.” And it wouldn’t be that hard to do, to get footage and doctor it so that every frame was changed so that it looked different Because you only have a few of these recorded positions, you don’t have to record positional head tracking, all that stuff. You just have a spherical image that you’re looking out of and that’s it, Bob’s your uncle.

So it’d be really… really… it’s be awesome. I think… Definately! We should do this! And if no one is thinking about it already, we should be. I think the cost of buying a bunch of these VR headsets and then giving them to passengers for the flight, and then taking them back would be less than the cost of designing and building windows into the aircraft. Both from an engineering standpoint, from a manufacturing standpoint, all of those things it would be way simpler to not have windows, and buy everyone VR headsets and just put some camera blisters on the outside.

So think about it, aircraft guys. Think about it.

(13:34) So, getting back to more game-related and computer-related stuff. Another thing you could do is make players feel like they are being threatened. And one of the things you can do here is… if you shove something in someone’s face, it’s shocking, it’s an aggressive action. And there’s lots of game. Like, in HalfLife, the face-huggers will jump up onto the screen. And it’s startling, even when you’re just on a screen. So I think it would be even more startling in VR where you’ve actually got this thing on your face, and then you turn your head and it’s still stuck to your head. You’d probably have to have some physics so that it felt like you were shaking a physical object around, but it could be very startling. And I think even better would be (for a sense of claustrophobia, for a sense of being closed [in] or trapped) having something covering just one eye. Instead of covering both eyes, the whole face? Just cover one eye. Just have something in front of you. So then it feels like you’re peeking out from behind something. You could use it in a multitude of situations, where you feel like you’re peeking out from behind something or something’s obscuring your vision. It will feel dream-like. A lot of the time in dreams you can only see out of one eye, I’ve noticed. So you can do a lot of stuff with that, where you are dealing with each eye independently. You put something there, or you put a bunch of blades of grass or something so that you can’t really see through with both eyes at once. You can only see with one eye at a time, or thin slits or something like that, where you can only get one eye lined up to see through it. I think it would be really interesting for gameplay purposes, and for the psychological effect of VR on people.

So, de-synching head tracking from actual tracking data and in-game tracking. De-synching the eyes, doing things with one eye or the other eye. And then some enhanced reality stuff where people are always talking about driving your car or walking on the street. And those experiences, people are already familiar with them. It’s not going to be effective to try to convince them that they are in some other place in those situations. Much better to convince them that they are in some other place when they’re in an airplane. And then you could do some amazing stuff with enhanced reality on airplanes! Because you don’t have to do the head tracking, you don’t have to do paralax. So you could have it feel like you were flying through space.

Or flying through some other environment, where you’ve got it rendered in real-time based on the airplane telemetry so that when the airplane moves your inner ear will tell you that it’s moving, and so then you can move it in the game or in this virtual experience, and so then the entire airplane turns into a VR room. A motion chamber. You’ve got Star Tours or whatever, this giant motion chamber. Then the airplane turns into, instead of turbulence and movement feeling unnerving, it can be exhilarating or interesting or fascinating! Where the airplane is flying through this fantasy landscape, and it banks to the left, and in the game it banks! So, that’d be awesome.

So I think airplanes are really ideal for this kind of thing. You can save on structure, you can save on airplane design, you can help people have the experience that they want. If they don’t want to see outside, they don’t have to. If they want to see outside they can, and they can really be outside with a VR headset. And, you can use the entire airplane as this kind of holo-deck thing, this virtual space that moves and banks and tilts, and you’ll be able to feel that and sense it and really feel like you’re there. And! You’ve got a large number of people all in one object, so you only have to gather telemetry for one object! You only have to generate visuals for one object, and then you’ve got it to disperse to all these people. So it’s much better than, in a car, say, if you had a VR headset on in a car, you’d have to gather telemetry data for the car and its motions and it’s accelerations, and get that really nailed down, but you wouldn’t have that economy of scale, because you can only fit so many people in a car. You can only fit, you know, ten people, maybe, at max, and probably only usually four or three or two or one. And the driver probably shouldn’t be wearing a VR headset because if it fails then he won’t be able to see. But all the passengers could.

But then in an aircraft, the passengers could definately see all over the place, it would be really useful! And then it would be useful for the pilots too, of course, if they wanted to see around them, see next to them while they’re taxiing, they can look out on the wing, the tip of the wing or the back of the airplane, the front of the airplane or something, really have a great awareness around them. And of course you’d want wind-shields still on airplanes because pilots need to be able to see. Even if all their equipment goes down, they still want to be able to view outside the airplane.

But eventually, of course, you’re going to have telepresence for pilots too, and then you won’t need cockpits like that. You’ll just have camera bubbles, or camera blisters on the outside and it will all be flown remotely from a cockpit deep inside the airplane instead of out along the shell, or up in the nose. But for the present at least we’re still going to have cockpits in the nose, we’re still going to have windshields. The technology isn’t proven enough to be able to fly airplanes with it, but I think it will definately be enough to convince passengers of the value of the technology. And the whole airplane moving is going to be really beneficial for virtual experiences.

If you did it properly, if you did it with some intelligence, then this virtual experience (if you’ve got a rendered scene of some kind) could take advantage of the airplane’s movements. So, say the airplane is flying along somewhere, you could have a bunch of asteroids around, say. You’re in space and you’ve got a bunch of asteroids around. The airplane is the space-ship of course, or maybe you’re flying second seat on this thing, and so you’ve got control of the guns and the pilot up in front is flying the airplane. So most of the time you’ll just be going straight forward. And if the airplane turns (and so then you’re going to actually incentivize the pilots to make mid-course maneuvers that are unnecessary for navigation but are important for passenger enjoyment, because if you’re just flying in a straight line and you don’t have any movement it won’t feel like you’re doing anything. But maybe you could do something where it’s really slow. A slow bank or a slow turn where the passengers wouldn’t feel it so much so it wouldn’t be bad for your inner ear, it wouldn’t de-convince you of it. But if the airplane did make sudden motions or real motions where it was banking or turning, or going through turbulence especially, you’d want to have some in-game symbol of that, to convince the player that what they are feeling is important and real. So you’d have something like that where an asteroid suddenly flies in from the side and crashes into the space-ship so turbulence happens. Or you have the airplane turns and it’s in order to miss this large object or in order to miss enemy fire that’s coming in or whatever it is, whatever the environment is that you’re creating.

Maybe it’s a really peaceful environment and it’s just they’re flying around the trunks of gargantuine trees and floating like a leaf on the wind through these huge environments. And then if there’s turbulence it’s just the wind coming by and just shaking the leaf gently as it flutters around or if it’s banking then it’s just a breeze that blows by, a cross-breeze or something like that. So the whole thing could be very natural, could be very convincing, could be very interesting, could be very effective, it could be very economical, so, definately airplanes, I think, are a huge application for enhanced reality and fixing a lot of problems that we have right now with airplane design and with passenger entertainment.

Thank you for joining me on the Paul Spooner Podcast, and allow me to play you out.
[much more musical piano outro]
Well it’s so nice to be singing and playing my piano.

No Comments »

No comments yet.

Leave a comment

RSS feed for comments on this post. TrackBack URI

Powered by Wordpress, theme by neuro