Learning by doing: Seeking best practices for immersive journalism

Ernest Wilson, the dean of the Annenberg School of Communications and Journalism, put it like this: What if, after receiving the home and garden section in the morning, the reader could walk right into the section and visit a garden? This bucolic vision reflects one potential scenario for what we are calling at Annenberg “immersive journalism,” a new genre that utilizes gaming platforms and virtual environments to convey news, documentary and non-fiction stories. As a senior research fellow, I am prototyping immersive journalism stories, hoping to discover and create best practices for a burgeoning filed that can capture audiences increasingly accustomed to experiencing digital worlds.

The fundamental idea of immersive journalism is to allow the audience to actually enter a virtually recreated scenario representing the news story. The pieces can be built in online virtual worlds, such as Second Life, or produced using a head-tracked head-mounted display system, or HMD. An HMD is a lightweight helmet that has screens covering the eyes and tracks head movement so ensure digital imagery on the screens stays in perspective to create a sensation of having a virtual body in a virtual location. Immersive journalism can also be constructed in a Cave, which uses full body-tracking technologies in a small room so that individuals can move their bodies around the space.

Visual and audio primary source material from the physical world reinforce the concept that participants are experiencing a nonfiction story, with the video, sounds or photographs acting on the narrative. For example, video triggers at key points in the virtual landscape to remind a participant that the computer generated environment is grounded in the physical world. Scripted events that create a first person interaction with the reportage can also help create a feeling of “being there.” Also, participants can query or interact with the elements around them to learn more about the details or context of the news story.

In general, participants travel through the story as a digital representation of themselves, or as one of the subjects in the news piece. Whether visiting the space as oneself or as a subject in the narrative, immersive journalism aims to afford the participant unprecedented access to the sights and sounds, and possibly, the feelings and emotions that accompany the news.

Well-crafted journalism always aims to elicit a connection between the audience and the news story. Creating that connection via different kinds of ‘immersion’ has long been considered ideal. Describing her reporting during World War II, Martha Gellhorn called it “the view from the ground.” Writer George Plimpton famously joined the Detroit Lions American football team in order to give his readers the most intimate sense of playing on this team. Television news correspondent Walter Cronkite made a series of documentaries recreating historical events in which he would offer a brief introduction before an announcer would give the date and the event, proclaiming, “You Are There!” More recently, attempts to combine audio, video and photographs on the Internet have created what some journalists call “immersive storytelling.” As technology editor at MSNBC, Jonathan Dube (now Vice President at ABCNews.com) said that he believes this can bring the reader or viewer “closer to the truth.”

In collaboration with digital media designer Peggy Weil, we have built several prototypes which reflect my interest in covering human rights issues. Gone Gitmo, a virtual Guantanamo Bay prison built in Second Life, allows participants to explore a place that is inaccessible to the average American citizen and press. (In fact, the Pentagon just expelled four reporters who have been covering the prison for years.) Gone Gitmo includes an experience on what it might be like to be detained, hooded and then imprisoned in Camp X-Ray. It also examines the ramifications of losing habeas corpus rights.

Another Second Life prototype, Cap & Trade, is a news report on the carbon market that sends people on a journey to follow the money in order to try to better understand the complexities and human consequences of trading carbon credits. Cap & Trade was built in partnership with the Center for Investigative Reporting and Frontline World and is particularly reliant on the excellent reporting by Mark Schapiro that appears on Frontline and in Mother Jones and Harpers Magazine.

A third prototype is based on the interrogation logs of Detainee 063, Mohammed Al Qahtani, who had been declared tortured by the Bush administration. Built at the Event Lab in Barcelona with Mel Slater and his team, we use an HMD to put participants into the virtual body of a detainee who is held in what is referred to as a “stress position.” When participants look around, they see a virtual mirror with a digital figure in that mirror who looks like a detainee and moves in unison with the participant. Participants also wear a breathing strap that programs the avatar to breathe at the same time as they do, further enhancing the sense of virtual body ownership. Throughout, the sounds of the Al Qahtani interrogation play as if coming from the next room. While research data was not collected on this particular prototype, every participant anecdotally reported that their body was hunched over in a stress position when in fact they were sitting upright.

Immersive journalism is distinct from news games in that news games embrace gaming protocols. With news games, the player undertakes a task or pursues a goal, voluntarily constrained by agreed upon rules, and must take action to advance position. Progress is often measured by indicators such as levels or points.  In contrast, a participant in immersive journalism isn’t playing a game, but is put into an experience where she is participating and affected by events but may or may not have agency to change a situation. Immersive journalism also parallels a news narrative playing out in the physical world much like a piece in a newspaper or segment on television and while one might experience the story from different starting points, the story itself should not shift.

When the record industry refused to consider experience, i.e. how their audience was going to interact with music, they gave Apple the right of way to build iTunes. The result was an extremely successful and robust environment that offers an entertaining, multilayered way to access music while also supporting Apple’s iPod music device. No doubt immersive journalism is nascent, but we hope to learn from the mistakes of the music industry which, unfortunately, legacy media seems well on its way to repeating. With iTunes as our model, we are concentrating on experience, and hope that in the near future we will support an offline platform as well.

You can see videos about the prototypes mentioned in this piece and learn more about this burgeoning avenue of journalism at www.ImmersiveJournalism.com.

The news of the future

The problem of veracity and realism in digital graphics has challenged Web editors and designers since the outset of online journalism. Where do we draw the line between fact and fantasy? How much latitude can we give the audience to create its own realities?

One answer has been to define Virtual Reality and create immersive applications that meet journalists’ notions of epistemology – the grounding of knowledge in verifiable facts and information. In contrast to artists, online journalists do not put a high value on illusion. We are not in the deception business. Nor are we gamers.

On the other hand, digital technology gives online journalists a chance to experiment with multisensory presentations, and we have long favored giving the audience opportunities to participate in storytelling. Harking back to MSNBC’s baggage checking exercise and other early versions of hypothetical scenarios, we have given the audience increasing latitude to explore the possibilities of digital landscapes from a first-person point of view.

Over the last several years, more effort has been put into elaborate calculators, civic games and hypothetical scenarios. The goal has been to use the immersive techniques of gamers “as an amplifier of thought,” to use the phrase of one design theorist, Brenda Laurel. For journalists, this requires creating a new vocabulary, a new metalanguage. Another theorist, art historian Jonathan Crary, describes it as “a radically different practice about the possibility of presence within perception.” To the print newsroom, it may seem more like Web journalists playing with dangerous toys.

A fresh example of where to draw the line in using Virtual Reality to tell the news has been created by the National Geographic in its documentary “Six Degrees.” It is based on a book, has a Web version, appeared in mid-February on cable and satellite TV and is set to be released in IMAX theaters in a 3-D version.

Each of us will come away from seeing the various versions of “Six Degrees” with our own opinions. But here, for the sake of discussion, and in no particular order, are my thoughts about a high-minded and expensive effort to put the audience into a hypothetical alternative world of global climate change. What do we see?

  • Mixed realities to create an appearance of the real
  • A topic that is large and complex has been reduced to the representation of a natural force, the rise in temperature due to greenhouse gas emissions
  • A point of view from outer space – a metaphor of the space voyager looking down on Earth
  • The application opens with the expectation that something will happen – the beginning of a plot – with an ominous sound reminiscent of the opening of “Jaws.”
  • The presentation Is not linear but has a design structure – the possible perspectives are not infinite
  • The ‘AS IF’ possibilities have been limited for the purposes of logical and affective clarity
  • It purposefully dissolves fixed limits on both time and space
  • It creates an ephemeral reality with an ontology that is founded on the process of global warming
  • The images are transient and malleable – they play upon memories and reinforce our experience (Memories of camping vs. civilization being reduced to tents on the Arctic Circle.)
  • The premise assumes shared information and a common ground – this is not a debate over whether human activities have provoked global climate change
  • It investigates problems but offers no solutions
  • The interface both enables and represents – it emphasizes action, raises alarms
  • The representations involve direct sensing and cognition (sounds of whale songs, melting ice, violent crowds)
  • Scenes are selected, arranged and represented so as to both intensify emotion and condense time (But are they hokey, especially the newscasts?)
  • The design has implicit restraints, but they arise naturally from our growing knowledge of the context
  • The explicit restraints – the temperature scale and Lighthouse Buttons – frame our actions
  • The multisensory experience creates empathy – we vicariously experience what the characters are experiencing
  • The overall impact is to give us a vision that changes our beliefs – our ways of doing things must change (or else…)
  • The application is built upon the storage and retrieval of information in a variety of media types to provide an organic experience that involves the whole sensorium.

    For what it’s worth, my favorite scene is the sidewalk café in Paris (Degree Four). It is reminiscent of “Last Year at Marienbad.”

  • Journalism can be welcome in 'smart homes'

    News organizations seem to be throwing open many doors, if not windows, hoping the right one or combination yields the secrets of audience attraction. Maybe the search should be more reflective – not based on science, necessarily, but at least on principles more closely tied to what’s going on in the information ecology of homes, offices, schools, libraries, cars, trains and buses.

    Ideally, the equipment or products used to spread words and images would create micro-environments where news can flourish. Wired-up homes, a big opportunity for online journalists, create a space where news undergoes cognitive processing, to use research talk. Studies indicate that audiences prefer content when both the media environment and delivery mechanism match the natural or biological capabilities of the consumers.

    Evolution is not necessarily a remote theory. Whether news is sent via print or through TV or computer screens, earplugs, baby-faced mobile devices or trained-pigeon messages, its success may depend how the delivery engages human sense organs and minds formed by millennia of biological development.

    The TV screen, floating in a space where humans can either focus or interact among themselves and ignore it, has had a lot going for it. The movie theater, a dark space redolent of unhealthy candy, butter and popcorn, inhabited by strangers who seem more and more to get on each other’s nerves, has been waning in a culture absorbed by interaction. IM fits the needs of teen organisms. Radio hitches itself to the fertile Internet and draws a global audience seeking to escape broadcast boundaries. Music lovers migrate from CDs to the mobility and flexibility of iPods and cell phones. Each medium makes use of heritable human traits, like curiosity, mate searching and a preference for mobility. The losers are anachronistic.

    This train of thought brings into question assumptions about the usefulness of displays, formats and delivery systems that have decidedly non-evolutionary origins – like engineering compromises chosen to get commercial products off the ground at a set deadline. Many news websites were pushed into the public sphere by senior editors and executives out of panic, not planning or calculated resource allocation. What success journalism has had in new media often seems more accident than design.

    As the wonders of the “smart home” unfold, this might be a good time to re-examine assumptions about how electronically delivered news is used. The developing space for news in the home represents the opposite of what happened when Macs and PCs came through front doors in the mid-‘80s. The devices went into dens and separate bedrooms for single-station use; personal privacy and segmented content within the home became the norm. Gender, marital and age gaps were allowed to squelch domestic discourse –sometimes with unfortunate results.

    That may not be true now. The creation of info-nodes, wide screens and hybrid TV-Internet-games platforms for the home has the potential to change news consumption from isolation and segmentation into a more communal dynamic where family members actually talk about what they learn, are amused by or share in common. They can interact among themselves, as well as within the virtual worlds they enter. It is not just mixed but multiple use, where the physical and virtual co-mingle.

    Game designers are putting these tools to good use. Military and aerospace trainers have developed extraordinary simulators. Medical innovators are making important use of telepresence in surgery and remotely monitored therapy. But news providers are only beginning to offer access to space imagery, panoramic perspectives, 3-D and immersive delivery formats. Even though the ability to combine virtual reality technology and interactive immersive environments has been available since Howard Rheingold brought it to public attention in the early 1990s, the potential of virtual reality (VR) as a news conveyor has lagged behind developments in other fields.

    And like phylogenic trees, media trees tend to diverge and channel. Military and entertainment needs moved VR development in directions that haven’t been much help to journalists. Bendable worlds, exotic avatars and complex battle scenes don’t fit well with journalists’ need for accurate, timely, verifiable and in-depth information. But delivery tools need to be thought of separately from content.

    As wired-up homes open their doors, journalists will have to give more thought to how they want to be received and how they will make use of not just wide screens but of local wireless networks, surround sound systems, laptops and games technology – haptic devices that convey physical feed-back, intuitive controllers (think Wii news) and sophisticated head-mounted displays, which are also becoming cheaper, more powerful and accommodating (even for adults).

    Taken together, these developing technologies have the possibility of making broadband news delivery a different experience, more like the ‘60s family gathered before the TV, but with 21st Century feistiness and a taste for global connections based on common interests. News content can be shared both within the physical group in the home – on networked devices, if not physically together – and within alternative worlds.

    Take a crass example: The smiling guy with a beer behind home plate who’s waving to the family at home and talking with them on a cell phone. “Right, I’ll buy that team shirt on the way out, don’t worry.” This is a complex event, involving media crossover and telepresence or intervention in a parallel reality. Crass, but a mustard seed. Computer scientists and engineers are working to develop layers of reality and multiple paths for audience intervention.

    Here are examples of barriers that threaten to keep news websites in a state of perpetual anachronism. They are drawn from current discussions in the computer science, communications and media design literature:

    1. As VR technology becomes more accessible, the tendency is to think of it as a “home theater.” That’s not a good metaphor, since theaters are dying. Journalists should be moving away from the lights-out, no-talking tradition of passive theater experience. If what is happening in research labs is an indication, the future lies in cooperative tasking in mixed reality.

    2. VR’s capability to simulate real environments, which should be a plus for journalists, can also inhibit graphic imagination. News and information can sometimes be best told in non-literal ways, as designers at the more forward-looking print publications have discovered. If print has become comfortable with abstractions, why does online journalism, with its vast capacity for animation and collage, lag behind?

    3. VR worlds tend to function in isolation from one another, a legacy of audience segmentation. It’s hard to traverse multiple worlds, when scale, navigation, sound and avatar portrayals have little standardization. No rules exist on what may be accurate or authentic vs. fanciful content. This may be keeping news organizations from thinking creatively about VR news applications that would fit into the ecology of the smart home. Indeed, coming to grips with VR technology could be an expensive, difficult task, but think of the alternatives.

    4. Screen-based personal computer displays have a fixed field of view and a concrete frame that limits interactions. It’s hard to collaborate when you have to do the electronic equivalent of peering through a key-hole. But that is changing, if the width of screens at Best Buy check-out lines are any clue. Expect homes to have multiple wide high-definition screens, panoramas of at least 180 degrees and user controls of perspectives that can free up the human eye to rove and make full use of peripheral vision.

    5. Commercial virtual settings, including Google Earth, provide spaces where interaction can take place, but then what? It’s nice to navigate cities, buildings, landscapes, pyramids, veins and arteries, molecules, etc., and fiddle with mash-up information. But critics of shared virtual environments argue that often “there’s nothing to talk about within them.”

    What an opportunity for bright, entrepreneurial journalists – converting sterile spaces into human places.