The iTV Doctor Is In!: Walk-in Theater
I walked into my office this morning and saw somebody riding a bicycle through a village--apparently in Japan. Then I heard a sound to my left and turned towards it. There was a storefront--a florist--with customers, and a bird chirping. From behind me I hear water dripping and turn around to encounter a young woman dipping a drink from a bamboo spout--it reminded me of the scene from the end of "Kill Bill." And then I hear a car pull up beside me and turn towards it.
Of course I wasn't really in Japan at all. I was using an amazing iPad app called "Walk-in Theater." And my head has been spinning with the possibilities all day.
Walk-in Theater (free download for iPads and iPhones at the App Store) is the brainchild of Eric Gould Bear, Rachel Strickland, and Jim McKee, who have been incubating new media concepts together for 18 years. Go to the App Store now and start the download. It should be done by the time you finish this column.
You may remember reading about Eric in The iTV Doctor's column from 6/10/10. Eric is CEO of MONKEYmedia, and is a user interface designer and inventor. He's filed for over 100 patents, yet I can usually understand what he's talking about.
Jim McKee's experience includes mixing, engineering and sound design for television, radio, commercials and drama, as well as research and production for interactive media.
Rachel Strickland, according to her bio, is "a documentary filmmaker, architect, and time-based media designer, whose research and art practice of the past 30 years has focused on cinematic dimensions of the sense of place, animate and ephemeral dimensions of architectural space, and new paradigms for narrative construction in digital media."
You can see that I'm already having trouble with the vocabulary of Walk-in Theater. But just wait: when you put these folks together, this is what comes out (from the Walk-in Theater site):
"Walk-in Theater is an experiment with peripatetic perspective, engaging participants' proprioceptors and spatial memory to orient themselves as they navigate among multiple video streams in a 3D sound field."
WHOA! I think Neo said that in "The Matrix." But it might have been from "Bill and Ted's Excellent Adventure." Ol' Keanu says that a lot.
But wait, there's more:
"Departing from one-way linear cinema played on a single rectangular screen, this multi-channel virtual environment pursues a cinematic paradigm that undoes habitual ways of framing things, employing architectural concepts in a polylinear video/sound construction to create a kind of experience that lets the world reveal itself and permits discovery on the part of visitors.
"Improvising on techniques of polylinear perspective long explored in Japanese painting and landscape design, Rachel Strickland's 'Emptiness Can Hold Things' establishes a new vocabulary for the cinematic construction of a sense of place. 'Cycle By' is the first of six Kyoto location studies in this interactive video series. Amidst intervals and thresholds of an urban landscape, at the confluence of two rivers there is a parallel universe perceived by bicyclists. The scene on any spring weekend unrolls like a picture scroll."
OK--let's step back into THIS universe now. What I THINK they just said was that we experience life on all sides, in all angles, and with full surround sound. And most of us tend to be the center of our respective universes. So why should we limit our video experience to a backseat linear perspective? Sit anywhere from mid-audience forward in a real Imax theater (not one of the shrunken multiplex versions of Imax), and you'll be looking up, down, left and right just to take everything in. And with Pixar/Dolby 7.1 the sound comes at you from ALL sides.
Walk-in Theater puts you in the center of your video universe--a virtual theater where the video comes at you from left, right, up or down; up close or far away; and you can actually walk THROUGH to see something the artist/director/cinematographer/sound designer has put BEHIND the video.
That's what polylinear means, at least to me. And the medical definition of proprioceptors is "neuromuscular receptors that register stimuli, such as stretch, tonicity, and movement within muscles."
So you don't only SEE the video images come at you from all directions, you HEAR the sounds from each video from all directions, and you FEEL what's going on because the device in your hands is your controller, your steering wheel. You literally rotate your body around in the real world to change which way you're facing in the virtual space.
I spoke to Eric and Rachel the other day, and I've played with Walk-in Theater on both the New iPad and iPhone 4S. (Note: the processing power required for Walk-in Theater is beyond the capabilities of older iPhones and the first-generation iPad). And I'll give you a couple clues before you start: 1) Use good headphones. Earbuds are OK, but headphones give you a more vivid 3D experience of sounds coming from scenes that surround you. 2) There is a little amber dot in a compass on the screen. KEEP THE AMBER DOT IN THE CENTER OF THE COMPASS! At least when you get started. Think of it as the bubble in a carpenter's level and you'll be OK.
Then just follow the on-screen instructions. You'll get the hang of it. And you'll have more fun if you're either standing on your feet or sitting in a swivel chair (just shut the door to your office before you start, because other folks in your office will think you've lost your marbles).
The special sauce of Walk-in Theater's user experience includes:
- Playing multiple videos simultaneously in a navigable 3D space.
- A patent-pending combination of sensor mappings for intuitive interactivity--e.g. spinning around a plumb-line for virtual camera aim, PLUS x-axis pivoting to move forward/back in virtual space, PLUS z-axis tipping to move sideways; all with natural sensor thresholds and acceleration curves. You know, the details that users shouldn't ever notice except when they don't work well. Their white paper covers these details in depth.
- Seeing and hearing a particular composition of polylinear videos--e.g. what do participants notice about the world portrayed in the content? What catches their attention/piques their curiosity?
If the heady Japanese esthetic isn't up your alley, what content might you like to experience in this manner? A multi-way telepresence party? Immersive street-view maps? Direct control of the Mars rover? Once you've tried Walk-in Theater, I would REALLY like to hear your suggestions of how it could be used--send me your ideas directly and I'll publish them in the column. Send to firstname.lastname@example.org, or email@example.com.
Here's a taste of some of the ideas that popped up while Eric, Rachel and I were talking about Walk-in Theater:
Rachel is an architect (among other things), and she conceives of Walk-in Theater as a way to sketch and experience physical and imaginary architectures in a virtual environment. Think about being able to prototype an exhibit designed for the lobby of the Louvre without having to route visitors outside this space for weeks ahead of time.
Eric sees multi-camera recordings of musical groups (with separate cameras on each musician) playing in the Walk-in Theater. Spatialized sound would emanate from the virtual locations of each instrument or voice. As you get closer to any player, their sound gets more pronounced, relative to the other musicians. But you can still hear them all. And if you turn and move towards another source (by tilting the iPad or iPhone), the sound-mix changes dynamically.
That made me think about a string quartet playing Bach, and the music from each instrument getting louder as you "walk" towards it, and then you walk THROUGH the video image of the musician to see what's in her mind as she's playing.
Imagine a live concert shot with Walk-in Theater in mind. Step into the middle of Woodstock, and watch Soul Sacrifice from the center of the stage:
And then there's the NFL. But maybe we'll leave that for another time.
Ping me if you have questions, or ideas.
And you can reach Rachel, Eric and Jim via firstname.lastname@example.org
The iTV Doctor is Rick Howe, who provides interactive video consulting services to programmers, advertisers and technology providers. He is the recipient of a CTAM Tami Award for retention marketing and this year was inducted into The Cable Pioneers. He is also the co-author of a patent for the use of multiscreen mosaics in EPG's. Endorsed by top cable and satellite distributors, "Dr" Howe still makes house calls, and the first visit is always free. His services include product development, distribution strategy and the development of low-cost interactive applications for rapid deployment across all platforms.
Have a question for the iTV Doctor? Email him at email@example.com