An augmented future for multimedia producers

In the dying moments of 2009, the US issue of Esquire magazine was published amid a certain amount of fanfare.

For this was no ordinary magazine. It used augmented reality technology to turn the magazine itself into an interactive toy. Hold it up in front of your computer’s webcam, and magical things would happen. None other than Robert Downey Jnr would pop into life on your screen, welcoming you to the magazine and offering a menu of interactive extras. Here’s one reader’s instant reaction:

“Flipping amazing!” the reader remarks. Right, but the magic didn’t happen without a certain amount of preparation. Before holding the magazine up in front of the webcam, that reader (and all the others wanting to see the fun stuff) had to go to the Esquire web site and download a small chunk of software for their computer. Webcams don’t understand augmented reality by default; they have to be told what to look for.

Yes, it was a gimmick. (Note the number of times people say “Wow, that’s so cool!” in the video above.) And who knows, maybe it bumped the sales figures up a bit. But it was also a pointer, pointing in the direction of new and amazing ideas that have a direct impact on magazines, media, and photography.

Those of you who have ever worked for print magazines will know the kind of planning that goes into each issue. You’ll have an understanding of the workflow, of the process used to turn raw content (words, text, and advertising) into a product on paper. It’s by no means a trivial operation.

Now consider the added complexity of augmented reality. Imagine getting hold of a Hollywood superstar like Downey Jnr, and pre-planning not only the photography and not only the video, but also the precise manner in which they will interlock. Consider the forethought used to create print images that directly and clearly reference the video content available via computer screen.

Oh, and the advertising. The Esquire mag included animated AR adverts from Lexus. That needed some pulling together too.

David MacDonald is technical director at Digitati, a UK-based interactive solutions company. His experiences with AR and similar technologies to date have taught him a thing or two about the changing production process.

At the moment, most AR experiments require the user to hold up a symbol – a printed sheet like the front of Esquire magazine.

“You have to have something that’s easy for people to share in order for them to interact with the content,” says MacDonald.

“Downloading additional software is a barrier. It’s easier to get a symbol to the user via the web or in print, although the iPhone screen works well at displaying interactive symbols.”

Another aspect the AR producer has to consider is the environment in which the symbol will be held up in front of the computer.

“You need to think in advance about how you’re going to use your symbol. Will the lighting be at an acceptable level? You have no control at all over the environment the users will be in, so you need to work around that.

“Symbols on paper are easy for users to print out at home, but a floppy sheet of paper can bend and twist very easily in front of the webcam, making it much harder to recognise.”

Where is AR going next? The people at Kooaba are working to make it more ubiquitous. This Swiss company makes an app for iPhone or Android, with which users can create connections between the digital and real worlds. So far it has been used for quick recognition of CDs, DVDs and book covers, but an experiment by Swiss newspaper Blick took things a little further.

By taking a photo of an article in Blick, users could pull up extra content. More photos, links to relevant web sites, controls for passing on a virtual clipping to friends via email or social networks.

The printed product remains what it is, but is enhanced – augmented – by the ability to call up additional information. You don’t need a Cuecat or even any visible codes (as were used in the Esquire experiment). The Kooaba system simply recognises what it sees, which in this case is the content as it was printed.

What’s coming soon, says MacDonald, is a move away from printed symbols.

“We’re on the cusp of being able to use no symbol at all,” he says. “Can the computer recognise something like a hand or a face held up to the computer? That’s not far off.

“It’s interesting that Esquire used Robert Downey Jnr for their AR issue,” he continues. “In the first Iron Man film, there was a scene in which he designed a mechanical glove for his Iron Man suit, using a sort of 3D AR interface that was almost holographic. That sort of technology isn’t as far fetched as you might think – it’s almost possible to create something like that for real. It will happen soon.”

So how does all this change the work of a photographer? It opens up a great deal of potential for new and additional work.

The modern augmented reality photographer needs to record metadata everywhere. The location of a photo, web data and links that are connected to the subject, perhaps unique reference codes for every item of clothing worn by a model.

In addition, the photographer now has to consider the role of “extra” content. If these four images are to be used in the print edition, we must also choose another eight images that will be shown to AR readers. Should the “best” images be saved for print? And what sort of payment and contractual implications are there? What is the value of an AR image in comparison to a print image or a web image? Some will argue that AR content equates directly to web content, but some might disagree (particularly editors who are paying for it).

As ever your thoughts – and real-world experiences, if you have any – are welcomed.