This article reflects on curating and mediating art and culture using Augmented Reality (AR) and its role in creating “holistic museum experiences.” The context for this discussion is the concept of paraverse, which is a short form for parallel universe. According to Mersmann and Ohls, this can be a hypothetical universe outside of what we know, and also multiple parallel worlds.[1] This terminology initially suggests a dichotomy of the physical and the digital world or experience, where the digital would exist in parallel with and separated from the physical or non-digital. But does this still hold true when it comes to Augmented Reality as a medium? And what are the unique characteristics and possibilities of this medium for curation, mediation of art and artistic expression?

Augmented Reality is, alongside lifelogging, mirror worlds and virtual worlds, part of the metaverse, which according to Smart, Cascio & Paffendorf is defined “not as virtual space but as junction or nexus of our physical and virtual world.”[2] In Augmented Reality, the physical world is technologically enhanced for an individual user by the use of location-aware systems. The interfaces overlay networked information and media (such as visuals or sound) on top of the perception of the world or space around the user.[3] In opposition to the virtual world, which is defined as a simulated world generated entirely by a computer (such as a game world), and where the user is represented and navigating the world as an avatar, Augmented Reality is rooted in the physical world and navigated by physically moving through the space, deeply intertwining physical and digital space. Or, to paraphrase Kit Gilbert from Niantic: “the real world becomes a canvas for XR-experiences.”[4]

Niantic is the company that brought Augmented Reality and location-based technologies on mobile devices to the general public’s radar in 2016 with their game “Pokémon go,” where people could find, catch and train fictitious Pokémon-characters located at “places of interest” in the real world. “Every corner, every street becomes a stage for interaction and magic.”[5] Moreover, it is a social game that rewards social interaction and visiting historical sites. With this it follows the mission of Niantic to “inspire people to explore the world together,” [6] where technology is employed “to deepen our connection to the world around us and to each other.”[7] Besides fostering technological developments in spatial computing, visual positioning systems and augmentation, the game also inspired museums and cultural heritage institutions to take a closer look at Augmented Reality for mediation and visitor experience design. But what does it entail to curate and mediate art and culture in Augmented or Extended Reality? In the following, the article will highlight some examples of strategies using AR in an art and mediation context.

 

Enabling the exploration of an information space

Artscan Feature in the guiding app of the Louvre Abu Dhabi, 2023. Photograph by Florian Wiencek © Florian Wiencek.

As a first application, Augmented Reality can be used to explore the information space e.g. of a museum. A great example is the Art Scan feature in the mobile guiding app of the Louvre Abu Dhabi, produced by NOUS.[8] By utilizing computer vision in the form of image and object recognition, the user is able to snap a picture of an object in the museum and access the information space around it — from the rudimentary label information to more in-depth multimedia content. This is a further development of the multimedia guide technology of entering a number or touching a beacon or NFC tag to retrieve information within the walled garden of the application. But now the physical object truly acts as information and content hub. It also fosters the free exploration of an exhibition where further layers of information can be accessed through taking a “photo” or “scanning” the physical exhibition objects themselves on demand, without having to worry too much about the navigation or data structures of a mobile or web-application. The exhibition as information and knowledge structure is enhanced by a digital information layer that is directly linked to the objects.

 

Enhancing / overlaying cultural objects

Screenshot of the Artivive overlay over Bildnis der Frau des Künstlers, Edith Schiele by Egon Schiele at Belvedere Museum, 2018. Screenshot by Florian Wiencek. Phoneframe by vector_corp on freepik.com. © Florian Wiencek

Whereas information access in the example of the Louvre Abu Dhabi is taking place in conventional multimedia guide interfaces, a different approach is represented by enhancing recognized objects inside an exhibition directly in the view of the visitor, e.g. by overlaying and digitally enhancing the cultural object with audiovisual layers, 3D objects or sound on a mobile device, as done by applications such as Artivive.[9] This adds new dimensions of experience to an object, such as bringing it to life or showing hidden layers of a painting, as well as telling stories directly involving the object. A good example is the mediation of the Egon Schiele exhibition at the Belvedere Museum in Vienna in 2018. By holding the smartphone in front of the painting Bildnis der Frau des Künstlers, Edith Schiele by Egon Schiele, the moving image overlay zooms in and pans over technical images created during a research process on the painting, such as infrared reflectography. These images unveil previous overpainted versions of the portrait showing corrections on the dress of Edith Schiele, which were demanded by the buyer of the painting. This particular story about the painting is told directly in relation to the original artwork, enabling a direct comparison and contrasting of the two versions of the painting. The same application is also used by artists to create digitally extended artworks that go beyond their physical form and include a digital overlay as part of the original work. These hybrid artworks enable the artists to bridge analog and digital forms of visual creation and storytelling in one work. The same technology is used for augmented city tours called “Linz Augmented,” where defined features of landmarks, statues, or buildings are used as unique anchor-points for location-based animations, re-imagining historic material from the collections of the state-museum as well as combining them with purpose-created drawings and animation of found elements of the places or their histories. In these three examples the mobile phone or tablet becomes a lens into parallel worlds and acts as a tool for exploring and uncovering ephemeral digital layers of physical objects.

 

Enabling spatial storytelling & spatial experiences

Screenshot of the AugartenAR app by Sonic Traces, 2024. Screenshot by Florian Wiencek. Phoneframe by vector_corp on freepik.com. © Florian Wiencek

Augmented Reality can also be used for storytelling and staging in hybrid spaces to create meaningful experiences. One example for this application of AR is the Augarten AR app by Sonic Traces,[10] which utilizes the concept of sound scenography and audio-AR to transform the park in front of the Augarten Porcelain Museum in Vienna into a storyscape. Sound scenography is basically defined as staging spaces and environments through sound to present artistic, historical, scientific content or to establish atmospheres and moods.[11] Sound is utilized to create a sensory and emotional entry point to topics, places and objects. The sonic experiences open up the possibility to immerse the visitor in a story or in other times or spaces. Contrary to a guided audio tour, a story that can be explored by moving through space is not necessarily linear, but consists of narrative and sonic modules that are located at specific points in space or related to specific objects in an exhibition. They can run on a timeline but mostly they are used to be recombined in a non-linear fashion as part of the spatial exploration or the interaction of the user with interactive stations in an exhibition or experience. This is a powerful tool that combines different modes of engagement from simple entering a specific area in space to complex technological interactions with location-based 3D audio, depending of course on the utilized technology or system. The mentioned Augarten AR app opens with a minimalistic interface that, after the calibration of the headphones and language selection, shows mainly a black screen with a positioning marker in the middle while nature sounds are setting a relaxed atmosphere. When coming closer to story-points, larger blurred circles appear on the screen as orientation. When entering a narrow circle by walking into the area in the physical world — determined by a combination of GPS sensors and a visual positioning system utilizing the camera of the phone — the story starts to play. The listeners virtually meet several persons in the park, hearing stories from historic figures, narratives introducing historical facts about the place as well as a series of stories from employees of the nearby Augarten Porcelain Museum and their exhibitions, providing a segway into entering the museum and exploring the galleries. The app utilizes spatial audio in order to provide an auditive sense of space and direction. You can listen to birds above you, hear (and nearly feel) people passing by through their footsteps located relative to your standing position. Together with atmospheric sounds, this heightens the immersion into the scene, place and story, that the visitor becomes virtually a part of.

 

Photograph in the exhibition Evidence by the Soundwalk Collective and Patti Smith at MAC/CCB in Lisbon. It shows some of the seating circles in the exhibition space. © Photograph by Florian Wiencek.

 

Where Augarten AR works with a historic space and uses spatial and location-based audio to augment the space with narrative layers, another important aspect of the concept of sound scenography is the close intertwining of spatial sound and scenography or design of the space. A good example of this is the exhibition and installation Evidence by the Soundwalk Collective and Patti Smith. The exhibition uses a headphone-based 3D audio system and “takes a sonic dimension, inviting the public to pass through acoustic landscapes: visitors are given headphones that geolocate them in space and react to their movements,”[12] as it is described in the exhibition catalog. The sound material used in the exhibition for the sonic landscapes stems from the album triptych Perfect Vision that was recorded as a collaboration of the Soundwalk Collective with Patti Smith, “in which each of the albums is inspired by the journeys made by Antonin Artaud in Mexico, Arthur Rimbaud in Abyssinia and René Daumal in India,”[13] retracing the footsteps of these poets. The retracing was done literally by the artists through travelling to the places described by the poets, collecting sounds and video materials, as well as reading and researching their work to develop poetic texts and interpretations. The musical and sound compositions, along with the poetic voiceovers by Patti Smith on these albums, were the starting point for the site-specific exhibition that, in a “poetic and immersive quest […] presents sound, film, abstract imagery, objects, and found art collected from their [Soundwalk Collective’s] travels, leading the visitor into a large investigative installation that juxtaposes photography, text, and original artworks by Patti Smith,” as the exhibition text inside the re-installation in the Museum of Contemporary Art and Architecture Centre (MAC/CCB) in Lisbon in 2024 reads. The main exhibition area is divided into three circular structures with possibilities for sitting around a central sculptural element. Each of the three circles represents the intellectual and story world of one of the three authors, and at the same time, the country they traveled to. When entering the exhibition space, one is immediately drawn into a soundscape of the room. Moving towards the circles, one enters the sonic world of one of the three albums where a story or poem narrated by Patti Smith, combined with experimental music and soundscapes, is placed at each seat within the circle, so that one could either stand or sit down at this place to listen to it. This resembles the concept of a fireplace, where people gather to listen to and exchange stories — and also relates to the cultures of the countries we are virtually visiting in this installation. It becomes evident that the layout and scenography of the room guides the exploration of the visitors through the relation to sound and objects within the space, as it can be learned very quickly and is consistently applied. Through the fireplace-like setup it feels natural to gather in the circles and to sit down to deeply listen to the content, thus supporting a specific way of interaction with the installation. The richness of content —seemingly changing over time when revisiting a spot one has already listened to, or in relation to other media items on display — the installation invites meandering and slowing down, moving physically through the exhibition space and deeply immersing oneself into these sonic and poetic worlds, investigating and uncovering them piece by piece. This highlights the importance of designing the space and the sonic experience together to support each other and to form a coherent and holistic experience. Otherwise, the sound just becomes another redundant layer on top of an exhibition space. Moreover, the location-based headphone system leaves the visual and haptic senses of the visitors open to foster a full immersion into the combination of physical installation, projections and sound as a holistic experience.

 

Enabling spatial gamification

Group photo with the Fantastic Palastics in the garden of the Belvedere Museum. Photo: David Payr. © David Payr, Belvedere Wien.

An additional way to engage users in exploring spaces and cultural learning is gamification. A good example is the newly launched game The Fantastic Palastics at Belvedere Museum Vienna, that engages with the history of the UNESCO World Heritage site in a playful way. The goal of this web-based AR-game is to combine the museum’s history and educational goals into an engaging experience for children. It is a scavenger-hunt type game, where the players need to find the lost animals of Prince Eugen in the palace garden. About 300 years ago under Prince Eugen of Savoy the garden was one of the earliest zoos in Europe, where the prince collected animals from all over the world. The game begins at one of the stone sculptures at the Upper Belvedere, where a Sphinx statue comes to life and awakens the animals, who lived there for 300 years, from their long hiatus. With the help of the Sphinx’s knowledge and a map of the gardens the players need to find all the animals. On their mission they explore the historic gardens and learn about art and culture. “Players search for clues and scan statues to find and collect the garden’s lost animals, feeding them as part of their mission. Once all animals are collected, players can take a celebratory photo or selfie with the creatures amidst the beautiful surroundings of the Palace Garden.”[14] The exploration is driven by a fictive story that builds upon the actual history of the site. A reward system and engaging storytelling keeps the motivation up to play and explore further. However, the game in the gardens is maybe only the starting point, as the characters of The Fantastic Palastics have the potential to be developed into a full scale ecosystem and can act as drivers for the children’s educational program of the Belvedere, according to Peter Meere from The Brand Father:[15] This includes merchandise to take home, and a coloring book solution for the monthly kids magazine of the museum, where children can color a character on paper, scan it on a smart phone, and bring it to life in 3D with AR-technology. The goal is not only to create a game, but to involve the characters outside of the museum and have a positive impact on the lives of the children who engage with them. And as a next step, the group is preparing a concept for a curated AR tour inside the museum with The Fantastic Palastics. The goal of this tour will be to unleash the inner creativity of the children, exploring how art resonates with them, as they search for the “secret artist.” The team wants to achieve this goal by combining different types of physical and digital exercises utilizing a combination of mobile devices and a physical game map, to minimize the usage of the phone in the museum and concentrate on the art and atmosphere by: a) watching art and introduction videos; b) listen to the characters talking about artworks; c) playing mini games about specific artworks or answering questions about artworks or the architecture in the museum; d) drawing a self-portrait and enable a moment of self-expression.[16] This also highlights the potential beyond the fun factor of a scavenger hunt game to actually lead to a closer engagement with cultural objects and artworks and facilitate a more introspective view on the individual perception of art and the emotions they can elicit. It also emphasizes the fact that thinking about augmentation does not mean to only perceive the world and the museum through the screen of a phone or smart glasses.

Placing cultural objects in new spatial contexts
One of the earliest concepts of mediating and curating with Augmented Reality is placing (cultural) objects in other spaces or contexts. This allows museums to extend their collection beyond the gallery walls. An early example is ARTours by the Stedelijk Museum in Amsterdam, which consist of a series of prototypes launched while the museum was closed for renovations. One of the prototypes took the museum collection to the Lowlands music festival and established an “ARthotheque” (2010), where festival visitors could borrow artworks from the collection and position them on the grounds of the festival. The festival area therefore became the temporary AR gallery space of a co-curated exhibition. Another prototype was a walk through the city of Amsterdam, where the users were able to experience design objects in meaningful contexts and could learn about the relation of the object and the place.[17] A pattern that is often repeated in such applications is the placement of historic photographs at the places where they were taken, so that they ideally overlay the current view of this location. That way they provide a window into the past of the respective location.

 

A visitor interacts with a piece of the Hexahedron Exhibition by Imvizar & Maser (2024) outside of the gallery building, that followed the initial exhibit at RHA Gallery in 2023. Photo: Imvizar. © Imvizar.

 

Instead of focusing solely on the recontextualization of natively digital or digitized cultural objects, one can also adopt an artistic strategy or particular style in order to extend the gallery experience to the space outside the building. A great example for this is the exhibition Maser — Around the Block, which took place in the RHA Gallery in Dublin, Ireland in 2023. Maser is an artist working between street art, murals and gallery artworks. For the show at the RHA gallery, Imvizar[18] created amongst other elements, an external display on the facade of the gallery that transformed the outside of the building into a canvas for Maser’s art. Drawing inspiration from the optical illusions of billboard signs, Imvizar “created 3D animations that echoed Maser's signature style, incorporating elements of movement, sound, and his distinctive approach to painting. These animations transcended traditional static displays, morphing into animated objects and shapes that moved transiently, engaging viewers in a multi-sensory experience.”[19] With this digital extension of the building, they on the one hand paid homage to the artist’s roots in street art, making his gallery show expand into the streetscape. On the other hand, the AR billboard provided an engaging experience with his works and artistic strategies and connected with the audience outside the gallery, inviting people inside, where the augmentations continued with a second AR display as part of the exhibition.

 

Annotations and interaction with the Rosetta Stone in the Civilisations AR app by BBC. Screenshot by Florian Wiencek. Phoneframe by vector_corp on freepik.com. © Florian Wiencek

 

Beyond permanently situating cultural objects in contexts outside of their collection institution, Augmented Reality can also be used to place cultural objects in any space around the user. An often-cited example is the BBC Civilisations AR app, which is conceptualized as a companion to a BBC TV series called Civilisations, covering a vast number of artworks across human history from 31 countries. The AR application features 40 artefacts submitted by collections from the UK, which fit the themes of the TV series.[20] The artefacts were purposefully 3D-scanned and can virtually be placed anywhere through the app — from a user’s living room to a classroom in school. The focus of this application is the interaction with and exploration of these objects as 3D data. One can not only twist and turn all the objects and inspect them closely but also get information through direct annotations, look inside the objects (such as a sarcophagus) through technical imaging, or receive further insights such as translations of inscriptions, as with the Rosetta Stone. Similar techniques can also be used in group guiding settings, as done in the Humboldt Forum in Berlin, to give the visitors the opportunity to explore objects on display in more depth, instead of just showing flat 2D image slides of details or context. Thus, besides making the objects more accessible to the interested public, this technology is enabling a (rudimentary) hands-on experience and meaningful interaction with objects that under normal circumstances could not be touched inside a museum. This provides an experience that goes beyond what is possible in a traditional exhibition setting.

Summary

The article showed examples for five strategies of using Augmented Reality for curating or mediating art and culture:

– Exploration of an information space turns cultural objects into information hubs but can also act as catalyst and anchor point for dialogues between visitors and other stakeholders.

– Enhancing or overlaying cultural artifacts result in the emergence of extended artworks or hybrid artforms. Technical devices such as phones or smart glasses act as a lens into hidden data worlds, stories, objects or creatures invisible to the bare eye.

– Spatial storytelling and experiences transform exhibitions as well as public or historical places into stages for immersive stories through a 6-degrees-of-freedom system, where paradigms like sound scenography are used to create holistic experiences bridging the physical and digital world. Users physically interact with stories in space, including moving through virtual sound fields and imaginary data worlds horizontally, but this can also include the z-axis through sitting / lying down or climbing up. This intensifies the embodied experience and exploration of the physical space itself and can transport the visitors to different places or times.

– Spatial gamification gives the users tasks and purposes to explore a space and keep engaging with cultural objects and histories of the space. Game creatures can become a brand of their own and serve as a basis for further engaging mediation tools beyond screen-based AR.

– Placing cultural objects in new spatial contexts recontextualizes cultural objects or artworks and serves as an outreach tool for cultural institutions. Every place can become a place for learning or meaningful interaction with cultural objects / data.

It becomes clear that Augmented Reality changes the way we perceive and interact with spaces and localized information or cultural data in different spatial contexts. It acts as a bridge between the physical and the digital, where the experiences cannot be conceptualized and designed separately as they are perceived as integrated experience by the visitors. In our post-digital times, the museum and cultural experiences need to approach both distinct types of interfaces — or “in-betweens” bridging cultural objects / data and the visitors — in order to create a holistic experience, instead of thinking of the potentially multiple digital layers within an augmented space as a “parallel universe.”

Florian Wiencek (B.Sc. Digital Media, M.A. Mediation of Art and Culture, PhD Visual Studies) is an expert at the interface of digital media and cultural education. He is the founder and CEO of Musealisten — Studio for digital mediation, and consults museums and cultural institutions with regard to digital mediation and learning from concept to implementation. Moreover, Wiencek develops sound experiences for the museum as well as public spaces as part of the Initiative for Sound Scenography (in collaboration with Extraplan) and works as media composer and sound artist, creating multi-layered sonic spaces. At LIT Open Innovation Center of the Johannes Kepler University Linz he coordinates the Interreg project “KreATivita&InovaCZe”, networking creative industries, research and industry in Upper Austria and Southern Bohemia.

At Fluxguide, he was previously responsible for Digital Concepts and R&D and designed digital mediation and learning opportunities for museums and cultural institutions, including the Deutsches Museum, the Deutsches Bergbaumuseum Bochum, the Württemberg State Museum and the Arvo Pärt Center. At the Austrian Center for Digital Humanities & Cultural Heritage, he worked in the field of knowledge transfer and headed the CLARIAH-AT working group “Tools & Methods.” Since 2014, Wiencek teaches at the University of Continuing Education Krems and regularly publishes articles on the digital mediation of art. He is particularly interested in the question of how digital media and cultural data (with their specific characteristics), are used in the mediation of art, culture and cultural learning, and how they enable museums to master the challenges of the 21st century.