Recipes for tangible and embodied visit experiences

Massimo Zancanaro, FBK, Italy, Elena Not, Fondazione Bruno Kessler, Italy, Daniela Petrelli, Sheffield Hallam University, UK, Mark Marshall, Sheffield Hallam University, UK, Taco van Dijk, Waag Society, The Netherlands, Martin Risseeuw, Waag Society, The Netherlands, Dick van Dijk, Waag Society, The Netherlands, Adriano Venturini, ECTRL SOLUTIONS SRL, Italy, Dario Cavada, eCTRL Solutions, ITALY, Thomas Kubitza, University of Stuttgart, Germany

Abstract

Keywords: Tangible and Embodied Interaction, Authoring Tool

1. Bridging the gap between digital and material through tangible and embodied experiences

There is a notable need for museums and cultural heritage sites to engage visitors at multiple levels. The use of digital technology often interferes with the direct engagement of visitors with the exhibition and the place (vom Lehn & Heath, 2003). At the same time, digital technology has the potential to create museum experiences that offer multiple, possibly contrasting, interpretations for the same exhibit; engage visitors at a multi-sensory level; and stimulate a longer and deeper interaction in place. What is crucial is the possibility of embedding digital content into physical objects and places that react to visitors in order to offer a seamless experience.

Within the Material Encounters with Digital Cultural Heritage (meSch) project (Petrelli et al., 2013; http://mesch-project.eu), we are working on technology to support the whole visiting experience: materiality complements and completes cognition (Dudley, 2010), and a personally meaningful and rich sensory experience with museum exhibits and places can greatly improve both the visit and also visitors’ appreciation for the museum’s cultural values. Taking objects out of their cases to feel their shape and weight while learning about them (figure 1, right) or examining the details of a small inscription on a fragile exhibit with a technologically augmented loupe (figure 2), are examples of experiences that engage the visitors at a physical level and put information in context without the distractions induced by interacting with extraneous devices, apps, and touch screens.

Figure 1: the indirect experience of objects in cases mediated by personal technological devices (left) contrasted with a hands-on experience (right) at the Museo Storico Italiano della Guerra (Rovereto, Italy).

A number of prototypes have been designed, implemented and evaluated to investigate the creation process and the potential of tangible, embedded, and embodied experiences in a cultural heritage space. Such prototypes include:

• An augmented plinth that triggers a projection when visitors approach or when an object is placed on top of it; the projection is dynamic, changes with the proximity of the visitors to the plinth, and can be personalised by using an NFC wristband (Kubitza & Schmidt, 2014)
• A multi-point auditory narrative system distributed in an outdoor historical site for the narration of multiple situated stories that amplify the emotional experience of feeling the past through natural interactions with a book or a belt (Petrelli et al., 2014a)
• A wooden magnifying glass that takes the visitor through the museum in a “treasure hunt” style and reveals details and layers of information when the target exhibit is found (Petrelli et al., 2014b)
• Interactive cases that measure visitors’ interest, run a competition between exhibits, and “enable” the exhibit to have a Twitter conversation with the visitors (Petrelli et al., 2014b)

Figure 2: unveiling hidden stories of objects at the Allard Pierson Museum (Amsterdam) in a non-intrusive way. The visitor is not aware he is using the optical zoom of a smartphone encased within the wooden magnifying glass.

The variety is notable: both stationary and portable devices; different media (text and images, animation, video, and sound); unconstrained visit and guided discovery; and reaction by presence (plinth and cases) or by touch (book, belt, loupe). Despite this variety, all prototypes were designed with the intention of engaging and surprising the visitors.

However, meSch does not just aim to experiment with exemplars of experiences, but to design a platform by which such a variety of mixed material-digital experiences can be created by domain experts in a simple way without requiring the assistance of computer scientists or electronic engineers. For this aim to be achieved, a number of questions must first be answered: Do these different experiences have something in common? What is it? How should the process of creating them be segmented and represented into elementary steps to make them easily replicable? Is it possible to generalize the approach and provide cultural heritage professionals with understandable instructables and blueprints to enable a do-it-yourself experimentation?

These questions have been the subject of much deliberation within the meSch interdisciplinary research team of interaction designers, cultural heritage professionals, experts in personalization and evaluation, and scientists in technology for ubiquitous computing. Through the creation of storyboards and prototypes, we have been able to elicit reactions from cultural heritage professionals and heritage audiences, and we now have a good understanding of what visitors engage most with. We also appreciate the complexity and flexibility behind the process of imagining, designing, and creating an exhibition.

We propose the metaphor of an “Experience Recipe” aimed at indicating all that is needed for the creation of a specific technology-augmented exhibit or space. A recipe is a combination of ingredients and instructions, including a description of which digital information is attached to which exhibit and/or locations, as well as a detailed specification of how visitors’ movements and actions will activate the contextualized presentation of that information, by means of hardware devices with input sensors and output capabilities.

2. The basic ingredients of a recipe

The recipe metaphor has been developed into a rich formal structure, which we call the “Experience Schema,” that permits the separate representation of the various aspects that make up a tangible and embodied experience. In particular, we separate the content from the context: the multiple stories from the specific setting in which a single story is going to be told. The schema has been designed to facilitate the management of the specification of points of interest, content, rules for interaction, and devices.

From the logical point of view, an experience schema is structured into four main components:

• The narrative: a set of curated digital content items, annotated into alternative thematic threads and levels of detail that revolve around the objects of an exhibition or places of a heritage site (e.g. text and images providing alternative descriptions for the same exhibit; different personal stories from people who lived in the same place);
• The appliance: a declarative specification of the capabilities of the technology embedded in the museum’s premises (e.g. the ability of the plinth to detect the current exhibit on display; visitors’ proximity to the plinth, both in terms of distance and direction of approach; the ability to project information on its different sides);
• The interaction script: the rules that govern when presentations should be started or how the system behaviour adjusts to personalize the experience to the visitors’ context (e.g. a set of rules that selects the most appropriate content for a visitor with a certain wristband: for example, the wristband for the Spanish language);
• The device: the description of the actual hardware required for deploying the system and its technical requirements (e.g. proximity sensors with proximity thresholds, NFC reader and tags, projector).

The goal of this approach is to separate the content (the narrative) from the technology (the device) and specify how the content is controlled and delivered (the interaction script) on the basis of the features of the device (captured in the appliance). This schema was explicitly created with the purpose of implementing the following properties:

• Reduction of complexity: authors can concentrate on the preparation of the necessary content and on grouping it according to the narrative dimensions exploited in the experience, ignoring all the details related to technology. The rules for putting content in context are represented separately and can be edited by experienced interaction designers.
• Abstraction: the declarations contained in the appliance specify in a formal way the type of elementary interactions and behaviour events that are available to shape the experience. This allows an author who is interested in the content to ignore the actual hardware details and the low-level sensor logs, whereas it enables interaction designers to experiment with different technological solutions for the same narratives.
• Reusability: the structure of an existing experience can be reused with different content and in different physical settings to create different instances. For example, the recipe for the Narrative in Place experience implemented with the book device in figure 3 was used to create one experience instance for the Sheffield General Cemetery (Ciolfi et al., 2013, http://mesch-project.eu/the-companion-novel-recorded-at-the-sheffield-general-cemetery/) and one for the World War I trenches at Nagià Grom in Italy (http://mesch-project.eu/going-to-the-trenches-with-mesch-technology/). To reuse it, only the content items in the narrative component had to be changed; that is to say, only the sound files and the printed pages of the book had to be replaced, while the rest, from the software to the device, stayed the same.
• Modularity: parts of an experience schema can be replaced to create a new experience. For example, the interaction script can be changed to introduce different rules for firing the presentation of the same content under different conditions. For example, the Narrative in Place instantiation for the World War I trenches at Nagià Grom was easily transformed into a more meaningful physical experience by replacing the augmented book device with an augmented belt with pockets that can accommodate thematic cards identified by NFC tags: two different physical devices, but represented by the same appliance.

3. Narrative in Place: A multi-point auditory narrative with a tangible set-up

In this section, we give an example of how the formal structure described above can be used in practice. For this we use the concept of the Narrative in Place, a visitor-aware personalised multi-point auditory narrative system that automatically plays ambient sounds and stories depending on a combination of features such as location, visitor position, and visitor preferences. We have experimented with the Narrative in Place concept in two different outdoor heritage settings for which different devices have been designed, implemented, and tested, namely a book (figure 3, left) and a belt (figure 3, right). The book is our first design, originally created for the Sheffield General Cemetery (Ciolfi et al., 2013) and reused in the World War I fortified camp (http://mesch-project.eu/going-to-the-trenches-with-mesch-technology/), for which the belt was then added as a means of freeing the hands of visitors trekking in the Alps.

Figure 3: cultural heritage professionals testing the Narrative in Place prototype, book version (left) and belt version (right), at the World War I trenches of Nagià Grom (Italy).

From the interaction point of view, the experience can be described as follows. The visitor moves freely in a historical open space with an augmented book in his  or her hands. The book has only a few pages, each outlining a different theme. For the cemetery (which is also a nature reserve) the themes selected were: nature (to provide an easy entry to visiting the cemetery); social history (the stories of the people buried there); amusement (with “weird and wonderful” facts about the cemetery); and finally, the favourite spots of volunteers who look after the site. A magnetic bookmark placed on a page selects the theme that the visitor wants to listen to. When he is in the area of a hotspot, a sound is played through loudspeakers that are placed throughout the site. This acts to attract attention and invite the visitor to move closer. When he is near the hotspot, a voice recording of a story is played; which sound and story is played depends on the theme currently selected. Next the visitor can listen to another story by moving the bookmark to another page or else walk to another hotspot. In this way, the visitor has multiple stories available at any time and can dynamically choose what to listen to. Purposefully, there are no screens, and all interaction takes place using non-digital objects (the book and bookmark and, in the following instantiation, the belt and cards). This works to bring the attention back to the heritage site and its experience.

The experience schema for the Narrative in Place splits the description of the experience into the following elements:

• Narrative: free-movement story discovery for a list of points of interest (POIs), with multiple narrative themes and two engagement levels. A list of POIs is associated with audio narrations. For each POI, alternative stories are prepared according to a range of thematic threads. Each POI has two specific content items for each thematic thread: one ambient sound to attract visitors to the POI and one that contains the actual story.
  Parameters: number of alternative thematic threads
• Appliance: multi-point audio. The technological setup includes a facilitating object carried by the visitor and POIs distributed in space. Audio presentations are played at the POIs; a physical selector on the facilitator object enables the visitor to switch between settings. The appliance implements the following interaction abilities: detection of which POI is in the vicinity; proximity of the visitor from POI; detection of selector position; and ability to play audio files.
  Parameters: number of POIs; number of selector positions; distance ranges
• Interaction script: free movement in space and theme selection. The visitor moves in the space freely and finds augmented POIs. The position of the selector can be changed by the visitor at any point in time to indicate to the system the preferred theme. When the visitor is in range of a POI, the attraction sound for the selected theme is played; when the visitor comes closer to the POI, the audio narration for the selected theme and that specific POI is played.
• Device: the book with the bookmark (or, later, the belt with cards) and a set of Bluetooth loudspeakers. Each Bluetooth loudspeaker represents a POI, where it will be finally placed. The augmented book is powered by a Raspberry Pi computing unit; it recognizes the position on a page of a magnetic bookmark used as selector and detects Bluetooth signals that identify the loudspeakers. The book uses the strength of the Bluetooth signal to infer its distance from the POIs and sends commands for audio-file playback by loudspeakers.

Figure 4: the device setup based on an augmented book and Bluetooth loudspeakers of the Narrative in Place experience schema, as tested at Sheffield General Cemetery.

This division of the overall experience into elementary descriptors of specific aspects provides flexibility in the composition of different elements in different ways.

The Narrative in Place concept initially created for an historical cemetery was then ported to a different outdoor heritage, the archaeological remains of the trenches and fortified camp from World War I in the Italian Alps. From a technical point of view, the direct porting of the full setting from the cemetery to the trenches required only the sound files to be changed and the pages printed on the book (figure 4). However, in the preparation of the multiple narratives for the trenches, the curators of the Museo della Guerra created personal accounts (as opposed to objective and detached descriptions) that included explicit references to the place (the narratives had names of villages and mountains nearby). This aimed to trigger empathy and understanding while leaving the visitors free to interpret the content and the place on the basis of their own understanding and sensibilities. Indeed, having experienced the Narrative in Place via the book enabled the curators to see the potential of the technology, and it pushed them to be creative and explore ways of engaging with the visitors at an affective level.

When porting the Narrative in Place to the trenches, we also decided to redesign the device visitors carry around: as visitors had to trek along mountain tracks, a device that left the hands free was considered more suitable. A belt-like device inspired by the clothes of World War I was then designed and implemented (figure 5); the selection of the theme is via an NFC-enabled card inserted in the belt. In switching from the book to the belt, only the description of the device had to be changed in the experience schema.

Figure 5: the device setup based on an augmented replica of a military belt used to deploy and evaluate a variant of the Narrative in Place recipe for the historical site of World War I trenches at Nagià Grom (Italy).

It should be noted that the same schema can be applied in very different situations. We are currently reusing it to deliver multiple content items at fixed locations indoors: in this scenario, the visitor carries an NFC augmented object (figure 6) that represents a theme, while the NFC reader is built into the display case. When the visitor places the object on the reader, an historical video specific to the narrative thread represented by the object is projected onto the glass of the case.

Figure 6: an NFC-augmented object, in this case a replica World War II travel pass, that activates content for a specific theme. The setup has been codesigned with Museon (Den Haag, The Netherlands).

4. Providing authoring facilities for tangible and embodied experiences

Starting from the concept of experience schema described above and its multilayer division, the meSch project is developing an online platform aimed at helping cultural heritage professionals create their own interactive, tangible exhibitions through (i) a user-friendly graphical interface to specify the declarations and input data and rules that make an experience schema, and (ii) a software platform that facilitates the assembly and initialization of the hardware components that actuate the experience in context (Kubitza et al., 2013). By easing the creation of software and hardware prototypes, we expect to enable a paradigm change in Cultural Heritage: from interactives created for museums to interactives created by museums; and from complex applications created ad hoc for one exhibition to shareable and portable experiences generated by a do-it-yourself community that learns from each other and contributes to the collaborative evaluation of technology deployed in museums.

In order to design such a toolkit and enable heritage professionals to compose digital content to be embedded into smart objects and spaces without the need for specialized technical knowledge, the meSch consortium have extensively explored the process by which exhibitions are created by both primary research on interactive exhibitions and co-design activities (McDermott et al., 2013, 2014). Through a series of co-design workshops, the ideas of technologists, interaction designers, and cultural heritage professionals were confronted and enriched, and converged toward the definition of a creation process that supports functionalities such as:

• being inspired by existing examples of recipes;
• customizing an existing schema by uploading your own digital content and adapting it to your physical setting; or creating a new experience schema for an augmented exhibit(ion) by composing the narratives and the interactive components;
• putting in practice the necessary steps for the technical assembly and its initialization; and
• sharing the result with the community of exhibition makers.

The meSch tool allows the creation of the narrative by providing a template to reference media elements (video, audio, text) and annotating them with thematic keywords. The tool also facilitates the browsing of media assets from several internal and external repositories (e.g., Europeana) by providing a recommendation facility to locate other assets relevant to the ones already selected.

With regard to appliances and devices, the only responsibility of the author is to choose the ones best suited for the experience s/he is going to create. For example, the Narrative in Place experience requires a multi-point audio appliance, which is compatible with either the book device or belt device. The conceptual level of the appliance simplifies the decisions that the authors have to take regarding both the content organization and the logic of interaction. Furthermore, it is very useful when the specific goal is to port one experience to a new scenario but the author wants to reuse the same interaction logic, such as in the case described above of porting the book from the cemetery to the trenches, and then replacing the book with the belt device. In this respect, the authoring tool facilitates the browsing and choice from a list of devices (that the museum owns or that can be fabricated or rented) or from a list of appliances. It then allows the author to set the parameters that fit the specific application setting (for example, the number of POIs to be supported).

At present, only limited graphical support has been implemented to define the interaction script. Although the conceptual level of appliances may facilitate the definition of the rules of behaviour (by hiding many technical details of the devices, such as Bluetooth connection in the example), it is still a task that requires cultural heritage professionals to be assisted by technical staff. We are currently investigating two possible methods of simplifying this task: (1) reducing the rules to very simple interactions managed by a language similar to Scratch (a very successful didactic programming language for children, http://scratch.mit.edu/), or (2) focusing on a parameter-setting task to personalize some default behaviour that is linked to specific appliances.

5. Conclusion

In this paper, we have discussed the current efforts in the meSch project (http://mesch-project.eu) towards the definition of a general formalism, inspired by co-design activities with cultural heritage professionals, for specifying “recipes” and experience schemas for creating tangible and embodied engagement with objects and places at cultural heritage sites. The formalism optimally supports the implementation of authoring facilities that would help the creation of a do-it-yourself community of exhibition makers that share their creations and learn from each other. The main idea is to reduce the complexity of the authoring task by providing abstraction and encouraging a modular approach that fosters reusability. An authoring tool based on these principles is currently in a pre-alpha version and will be demonstrated at the Museums and the Web 2015 conference.

Acknowledgments

The research described in this paper is part of a shared effort carried out in meSch, Material Encounters with Digital Cultural Heritage. The interdisciplinary team includes: Sheffield Hallam University (UK); University of Limerick (IR); Waag Society (NL); University of Strathclyde (UK); eCTRL Solutions (I); DEN (NL); University of Stuttgart (D); University Carlos III Madrid (E); Museo Storico Italiano della Guerra (I); University of Amsterdam – Allard Pierson Museum (NL); MUSEON (NL); and Fondazione Bruno Kessler (I).

meSch (2013 to 2017) receives funding from the European Community’s Seventh Framework Programme “ICT for access to cultural resources” (ICT Call 9: FP7-ICT-2011-9) under the Grant Agreement 600851. More information is available at http://www.mesch-project.eu.

References

Ciolfi, L., D. Petrelli, F. Caparrelli, N. Dulake, R. Goldberg, & M. Willox. (2013). “Exploring Historical, Social and Natural Heritage: Challenges for Tangible Interaction Design at Sheffield General Cemetery.” In Proceedings of NODEM 2013 Nordic Digital Excellence in Museums Conference, Stockholm, Sweden, December 2013.

Dudley, S. (2010). “Museum materialities: Objects, sense and feeling.” In S. Dudley (ed.). Museum Materialities: Objects, Engagements, Interpretations. Routledge.

Kubitza, T., & A. Schmidt. (2014). “A First Set: A Physical Components for the Creation of Interactive Exhibits.” In Proceedings of TEI ’14, 8th International Conference on Tangible, Embedded and Embodied Interaction.

Kubitza, T., N. Pohl, T. Dingler, S. Schneegass, C. Weichel, & A. Schmidt. (2013). “Ingredients for a New Wave of Ubicomp Products.” IEEE Pervasive Computing 12(3), 5–8.

McDermott, F., L. Clarke, G. Avram, & E. Hornecker. (2013). “The Challenges and Opportunities Faced by Cultural Heritage Professionals in Designing Interactive Exhibits.” In Proceedings of NODEM 2013 Nordic Digital Excellence in Museums Conference, Stockholm, Sweden, December 2013.

McDermott, F., L. Maye, L., & G. Avram. (2014). “Co-designing a Collaborative Platform with Cultural Heritage Professionals.” In Proceedings of iHCI2014 the Irish Human Computer Interaction Conference.

Petrelli, D., L. Ciolfi, D. van Dijk, E. Hornecker, E. Not, & A. Schmidt. (2013). “Integrating Material and Digital: a New Way for Cultural Heritage,” Interactions 20(4), 58–63.

Petrelli, D., N. Dulake, M. Marshall, M. Willox, F. Caparrelli, & R. Goldberg. (2014a). “Prototyping tangibles : exploring form and interaction.” In Proceedings of TEI ’14, 8th International Conference on Tangible, Embedded and Embodied Interaction.

Petrelli, D., E. Not, A. Damala, D. van Dijk, & M. Lechner. (2014b). “meSch – Material Encounters with Digital Cultural Heritage.” In Proceedings of EUROMED 2014.

vom Lehn, D., & C. Heath. (2003). “Displacing the object: mobile technology and interpretive resources.” In Proceedings of ICHIM International Cultural Heritage Informatics Meeting.