A new look at an old friend: Reevaluating the Met’s audio-guide service

Laura Mann, Frankly, Green + Webb USA, USA, Grace Tung, The Metropolitan Museum of Art, USA

Abstract

Keywords: audio guide, mobile, user experience, research, evaluation, service design

1. Introduction: Audio guide as service

Audio guides have been around for more than fifty years, and while the technology has changed considerably, the basic concept has not: museums have thought of audio guides as a “product” that visitors carry with them to enhance their visit.

There is a growing tendency for museums, perhaps due to their longevity, to see their audio guides as old-fashioned. When take-up rates for the guide are lower than expected, this “outdated” technology is often identified as the culprit. New functionality, more content choices, and sexy new hardware are seen as the solution and means to a successful guide program. But this approach hasn’t delivered results: there is no evidence that new technology either inspires visitors to pick up the audio guide or improves the quality of the guide experience. For example, visitor satisfaction rates are approximately the same at the National Gallery London, which offers more antiquated mp3 players, and the Metropolitan Museum of Art, which offers the guide on iPod Touch players.

Museums should be wary of continuing to ask the same questions about hardware and technology platforms while expecting a different answer. But what if we thought less about “products” and more about “services”? What if we stepped back from the specificities of the latest hardware features and the amount of content, and focused more on the services our visitors need? Designing a service rather than a product will prompt us to look at this long-standing issue from a new perspective.

Service design allows museums to understand—and design for—the visitor experience as a whole. While museums often identify “digital” as a discrete entity (especially on the institution’s organizational chart), when viewed from the visitor’s perspective, digital is only one part of a larger continuum of experience that combines the digital and the non-digital. This interpretation of digital service from the UK Government Digital Service (2012) could equally apply to the museum experience: “Our service doesn’t begin and end at our website. It might start with a search engine and end at the post office.”

Service design as a methodology is well established in the commercial sector (Polaine, Løvlie, & Reason, 2013) but still relatively new to the museum field. In addition, service design is a particularly useful framework for audio guides and other mobile experiences. Service design allows museums to shift the focus from specific hardware platforms towards the needs of their audiences.

Service design is a way of designing experiences in order to better meet visitor needs. This requires an understanding of visitor needs across the entire user journey from initial awareness to the impact of the guide on the overall museum experience.

Service design allows museums to better understand the user experience as a whole. Visitor access to the service could then be thought of as a series of barriers or hurdles that need to be overcome for the service to function as a cohesive whole. If one of these barriers can’t be crossed, then anything else that follows is irrelevant (Figure 1).

Figure 1: audio-guide service design: a series of barriers for a visitor to overcome

The Metropolitan Museum of Art and Frankly, Green + Webb USA used this service design model in a recent evaluation of the Met’s audio-guide program. This paper will explore the evaluation approach and our key findings in terms of insights for the museum field writ large.

2. Evaluating the Met’s audio guide

In September 2013, the Met launched a new audio guide complete with a redesigned interface and repackaged content. The launch of this new guide provided an opportunity—and a need—to take a more strategic look at this long-standing service.

The Met has used audio guides for more than fifty years. Historically, the Museum captured data about sales and quantity of content produced. Greater amounts of content and more choices for the visitor were seen as inherently better. However, these measures tell us very little about the impact of the service on visitors. These measures express the success of the audio guide in relation to how well the guide is serving the needs of the institution. Questions like “how many audio guides are being sold?” and “how much content is being produced annually?” do not reflect the needs and goals of visitors. In order to answer questions like “who is using the audio guide and why?” and “what is the guide’s impact on the visitor experience?” the Museum is learning to view its audio guide through the eyes of its visitors.

The research presented here was designed to look at the audio-guide service as a whole—from initial awareness to educational impact—in order to understand what was and not working. Specifically, the evaluation was designed to provide the following:

• Actionable recommendations
• An evidence base for future decision making around the guide and other mobile offerings
• Tools for ongoing evaluation

Methods: Quantitative and qualitative

The evaluation used a wide range of methodologies, both quantitative and qualitative:

1. Quantitative survey

• Surveyed 492 audio-guide users and non-users at the main building and the Cloisters
• Conducted in English, Spanish, and Mandarin (Mandarin and Spanish are the most popular foreign languages on the audio guide)

2. Qualitative interviews

• Interviewed twenty audio-guide users at the main building and the Cloisters
• Conducted in English with visitors who used the guide in English, French, Spanish, and German

3. In-gallery usability testing

• Seventeen pre-recruited participants
• Structured (users asked to perform specific tasks) and unstructured (unscripted visits to the galleries) usability testing (Figure 2)
• Used the “think aloud” method
• Recorded sessions with a video camera mounted to an audio-guide player (Figure 3)

4. Service audit

• Observation
• Data review
• Stakeholder interviews

Figure 2: in-gallery usability testing session

Figure 3: in-gallery usability testing rig: GoPro® video camera and audio-guide player mounted to an acrylic sled

What does the audio guide offer?

When Frankly, Green + Webb and the Met conducted the evaluation of the audio-guide service in the spring and summer of 2014, the audio guide included the following:

• English-language content
  • More than 2,600 stops, including:
    • The Director’s Tour
    • More than twenty themed and collections tours
    • Six to eight special exhibition tours annually

• Foreign language content
  • In the Met’s main building: forty-four stops in nine languages
  • At the Cloisters: ninety-two stops in six languages

Figure 4: Metropolitan Museum of Art’s new audio guide, launched in September 2013

3. What’s working well? Overall impact of the guide

The audio guide is successful based on some standard measures. Visitors reported that they were satisfied with the guide: 90 percent would recommend it to a friend, and 81 percent rated their audio-guide experience as good or very good. However, satisfaction rates by themselves aren’t very useful. To understand what these figures are really telling us, we need to understand why visitors reported positively. Visitors clearly articulated how the guide made their visit to the Met more meaningful (Metropolitan Museum of Art, 2014):

It draws your attention to things you might not have noticed yourself.

… It’s like getting to know someone rather than just finding them pretty.

Gives you a better experience—you’re not just wandering around—you have more of an itinerary or agenda. You can get lost in these places. 

These comments reflect what many visitors said: the guide prompted close looking and fostered a deeper sense of connection and engagement with the art works. More than eight out of ten visitors reported that as a result of using the guide, they learned more about the art and noticed details in the art works that they would not otherwise have noticed on their own (Figure 5).

Figure 5: audio-guide impact

We also see the impact of the guide in areas that might not be expected from a mobile guide: 63 percent of visitors said that they “discussed what [they] saw and learned” more as a result of using the guide. Conventional wisdom says that audio guides are socially isolating, but this suggests the opposite—that the content fuels conversation and provides a real social element to the impact of the guide.

Visitors who use the audio guide are also more likely to report a better overall museum experience compared to non-users (Figure 6).

Figure 6: impact of audio guide on overall museum visit

This finding reinforces the positive impact of the audio guide but also suggests that increasing audio-guide take-up rates is likely to improve visitor experience of the Museum. A museum might want to raise take-up rates for many reasons, like meeting educational objectives or increasing revenue. But this finding indicates that improving visitor experience is an equally important reason and should be considered as a goal for mobile projects.

4. Awareness: Who are the users of the service, and what are their needs?

Historically, the Met envisioned the audio guide as a product that should and could serve all museum visitors. The audience for the guide was therefore seen as homogeneous. The advent of digital hardware platforms that could accommodate larger amounts of content reinforced this idea: expanding content, choice, and flexibility would allow the audio guide to be all things to all people.

However, the research has shown that two broad audiences actually currently use the guide at the Met. International and domestic tourists are the larger, primary audience, and New York/Tri-State residents represent a smaller, secondary audience.

Primary audience: Domestic and international tourists

The primary audience for the audio-guide service is domestic and international tourists:

• They make up 81 percent of audio-guide users
• Most are first-time visitors
• Most come to the Museum because of its reputation as a popular destination in New York

Unfamiliar with the Museum, these visitors are making a special trip, and they want to get the most out of their visit. From this point of view, the audio guide offers them clear benefits:

• It helps them navigate the Museum, conceptually and physically
• It helps them find the “treasures” in the Museum’s collection
• It provides content in the visitor’s own language

This audience is the “natural” market for the guide, and it also constitutes the majority of overall visitors to the Met. Understanding the needs of the primary audience for the audio-guide service is now enabling the Museum to be more responsive to this audience. For example, the Museum is working on making the audio guide easier to buy by aligning its on-site and online marketing efforts with the primary audience’s touchpoints, as well as implementing a new, more targeted content strategy for the future.

In particular, the Museum is focusing on refining the audio guide offer for foreign language speakers who represent a significant minority of all audio-guide users: 40 percent of guides are taken in a language other than English.

Overall, foreign language speakers have reported a less positive experience compared to those who take the guide in English: where 90 percent of the visitors who took the guide in English described their experience as good or very good, only 73 percent of those who took the guide in a foreign language felt the same way.

Foreign-language speakers are an important and expanding audience for the Met. The Museum is focused on designing mobile services for this audience that meet its particular needs and goals. By refining the foreign-language offer on the audio guide (and other mobile projects), the Museum sees a clear opportunity to improve the visitor experience for a growing number of foreign visitors.

For example, Mandarin is the most popular language on the guide, after English, and the growth in Mandarin-speaking visitors is representative of the rapidly changing nature of international audiences. The Museum’s Mandarin-speaking audience has grown and evolved dramatically in the past ten years, as a result of increased numbers of visitors from Mainland China. This is a timely reminder that international visitors are not monolithic or static. The needs of specific audiences must be considered when initially designing the service and in refining the service over time.

The Museum is now taking a more strategic approach to expanding the foreign-language offer; the approach reflects the needs of international visitors, guided by tangible use patterns on the guide as tracked in analytics. The Museum has also developed a new approach to increase the number of translated materials produced annually in nine languages (in addition to English) by considering the patterns of foreign visitation at both the Museum’s Main Building and at the Cloisters museum and gardens. Finally, the Museum is also building internal capacity to support expanded foreign-language mobile offerings by revamping its digital media translation process so that it is more detailed and more efficient.

Secondary audience: Local and regional visitors

The secondary audience for the guide consists of residents of New York City and the Tri-State Area (New York, New Jersey, and Connecticut).

• They make up 19 percent of guide users
• They are more likely to be repeat visitors and members of the Museum
• They are more likely to be motivated to use the guide for special exhibitions
• It is a smaller and more diverse audience overall

These visitors are more likely to think about audio guides in the context of special exhibitions and to think of the permanent collection guide as being for “tourists.” The secondary audience knows that the guide offers information but—with the exception of special exhibitions—they don’t perceive that it meets their particular needs or identity.

This audience also has more varied touchpoints—or points of interaction—with the Museum. For example, they may visit the website to check current exhibitions, but because they’re familiar with the Museum they would not explore the site in depth and find where the audio guide is usually promoted.

Designing a service that would truly meet the needs of this audience would require greater insight into its diverse needs and motivations and may need separate branding. It’s clear that one size will not fit all.

What does this mean for service design in other museums?

Many programs that museums produce for visitors are inherently service oriented. And services need to be designed for specific audiences. Understanding audiences and their needs, motivations, and touchpoints with the museum allows you to shape the service: building awareness, communicating in ways that resonate with the audience, and reaching visitors at the right moments in their journey. Who is your service targeting? How and where is it promoted? Does the promotion of the service align with the actual touchpoints of the target audience? Answering these questions requires a cross-departmental team that goes well beyond digital and education. It’s crucial to the success of the service to involve teams from marketing, visitor services, and retail (among others) early in the design process.

5. Understanding: Is the service easy to understand?

The Met’s audio guide is large and complex, with multiple ways to access content and more than twenty-five tours for highlights, collections, themes, kids, and special exhibitions.

Describing such a complex offer is—not surprisingly—a challenge for the audio-guide staff. As a result, staff focuses on the keypad (for English tour-takers) and the Director’s Tour (for foreign language tour-takers) when distributing the guide to visitors.

The result is that many visitors do not always understand the scope of the offer on the guide and cannot take full advantage of it, and the complexity of the offer means that visitors are unable to make informed choices. In usability testing, for example, users often had difficulty choosing a “highlights” tour from the multiple options on the “Highlights” screen.

Supported by the visitor insights revealed in the evaluation, the Museum is refining its audio-guide offer and adjusting the service to meet specific visitor needs. For example, a redesign of the user interface is currently being considered, and several of the tour options will be revised and carefully condensed.

What does this mean for service design in other museums?

More is not necessarily better when it comes to a mobile offer. Museums need to give visitors a basis on which to make informed choices, which should reflect visitor needs and behaviors in the museum, and also use language that visitors can easily understand.

The offer must be easy for museum staff to explain clearly (this is especially important for foreign-language visitors) during a brief transaction time. Museums need to more clearly identify whom the service can help and explain the benefits: for example, “The guide is specially designed for first-time visitors. It will help you make the most of your time here.” Ultimately, audio and mobile guides in museums must have a short learning curve. Visitors need to be able to pick up the guide and understand how to use it almost immediately. Visitor focus is on navigating the physical space of the museum; there is little opportunity for them to explore the mobile guide before using it.

6. Usability: Is the service easy to use?

Most visitors (76 percent) in this evaluation indicated that they found the guide “very easy” to use. Interviews suggested that these positive ratings for usability reflect the widespread use of the keypad to select content (versus the various tour options), which visitors singled out as easy to use: “I just type in the number.” In fact, Google Analytics and survey data tell us that the vast majority of visitors accessed content by entering numbers into the device’s keypad.

However, what visitors say and what they do is often different, and in-gallery usability testing revealed a richer story about how visitors actually use the guide.

What is a “tour”? Opportunities and challenges of “tours”

Every collection tells a multitude of stories, and museums around the world, filled with creative staff, are keen to use the capacity of newer technology to offer visitors more choices and more information. The Met is no different. The current audio guide is ambitious; it includes more than twenty-five tours of the permanent collection, including highlights, themes (music, architecture, fashion), and tours for children. The evaluation looked closely at each of these packaged tours and revealed both opportunities and challenges with this wealth of content, which the Museum is now actively addressing.

For example, the Director’s Tour is the most popular tour on the audio guide. It is a tour of approximately forty objects narrated by Director Thomas Campbell. This tour includes directions and is accompanied by a paper map, with each object on the tour identified by a thumbnail image. The Director’s Tour is also the main offer for foreign-language speakers, as it is available in nine languages in addition to English.

The Director’s Tour is widely used and valued by tourist visitors because it meets some of their most important needs:

• It helps them see the highlights of the Museum
• It provides a curated experience and an itinerary through the Museum
• It provides orientation and directions for physically navigating the Museum (in the audio and through the map)

In the case of the other permanent collection tours, visitors listen to the tour stops, but they access them as individual stops through the keypad. Visitor interviews suggested that visitors were interested in and enjoyed the content of the individual tour stops. They appreciated the variety of voices and additional perspectives offered by the tour stops, such as stories about the history of the museum and explanations of conservation techniques.

But visitors don’t follow the tours through the Museum, in part because the tours currently don’t reflect the physical realities of navigating the Museum space. In other words, the tours do not answer these questions: Where does the tour start relative to where I am now? Where are the other objects on the tour? How long will this take? Most visitors appear to expect a tour to move them through the Museum and a story (albeit a very loose one), but the current tours (aside from the Director’s) function as thematic groupings of stops rather than guided linear experiences.

The answer isn’t simply to provide linear directions for each tour. Rather, the Museum has used the results of this evaluation to think in new ways about how to use mobile to balance the visitor’s need for a curated, structured itinerary of “highlights” (like the Director’s Tour), along with the need for flexibility and variety. Thinking about how to refine the offer in these veins and adjust the service to allow these needs to cohere for visitors is an opportunity for the Museum to experiment with new approaches to storytelling and experience design.

Disconnect between the physical space and what is on the guide

Navigation and wayfinding are a tremendous challenge for the Met, and the issue of physical context adds another layer of complexity to the visitor experience of the audio guide. The evaluation revealed that the in-gallery, in-ear, and on-screen spaces don’t necessarily correlate or reference each other, making it difficult for visitors to move between the platforms.

The Met saw this challenge clearly when assessing the audio-guide labels. During usability testing, visitors had no trouble locating the labels for audio-guide stops, but they often had difficulty understanding what they meant. The large number of tours translates to a complex lettering system on stop labels that can be confusing. Many visitors misinterpret their meaning. Based on these findings, the Museum is revising the labels so that they are simpler, with clear references back to the content on the guide device. The labels are a reminder that designing for mobile with physical context in mind is a critical aspect of providing a successful audio-guide service. Understanding the guide as a service in this way ensures that museums look at—and design for—the areas where the digital and the non-digital meet.

What does this mean for service design in other museums?

Our desire to be “innovative” and the user’s need to quickly understand the offer can be at odds. Innovation should be in the service of user needs: fix a problem or grasp an opportunity that will make things easier for users, not harder.

Mobile guides in a museum need to reflect the physical realities of that museum and the experience that museum is trying to present. To do this, mobile guides are responsible for joining up the “in-ear” and “on-screen” with the “in-gallery.” A development process that “starts mobile and stays mobile”; ideating and testing the experience and content in the gallery space can help ensure that the digital and physical experiences remain connected.

7. Value: Does the service deliver value?

Visitors see the audio guide as a learning product (Figure 7); they expect the guide will support them to learn and—as we saw earlier—reported that the guide was helping them learn (Figure 5).

Figure 7: why did users take the audio guide?

But visitors also want the audio guide to be easy and pleasurable. Their desire to learn is in the context of a leisure visit (Figure 8).

Figure 8: why did users take the audio guide?

What does learning mean for the visitor? What resonates with visitors?

When visitors described what they remembered most vividly from their tour, three themes were consistent:

• Meaning rather than just information
• The importance of looking closely
• The story of the Museum and how objects became part of the collection

Visitors recalled specific visual details that they would not have otherwise noticed, like the flour on the heads of small Egyptian figurines, or the brilliant color that a now-white sandal would have been painted. Visitors also enjoyed hearing about why and how objects came to be in the Met’s collections. Being in the Museum’s collection conferred value on an object, and visitors were keen to understand that value. They wanted to see the Museum as a character in the larger story told by the audio guide.

In addition, visitors value their time in front of real objects. This came across when people told us that video on the guide was not a high priority for them: the majority (64 percent) said that video was either “not at all important” or “slightly important.” Video was only important if it could deliver a special insight that helped visitors understand the object more fully.

What does this mean for service design in other museums?

Audio and mobile guides are learning products, and they should have specific learning objectives that reflect the museum and its mission. What do you want your visitors to leave thinking? Feeling? Remembering? Creating a small number of key learning objectives based on these questions will:

• Enable the museum to communicate the value of the guide in ways that appeal to the visitor
• Support the organization to choose and create appropriate content
• Support the organization in evaluating the effectiveness of the guide

Content in this context isn’t simply information. Audio is the perfect medium to convey emotion, excitement, character; it is one person speaking to another. Find the voice of your institution. Not the voice where one specialist speaks to another or the neutral “objective” public service broadcast, but the one where that specialist seeks to excite and enthuse a smart friend and share the stories, challenges, attitudes, and quirks of the institution.

8. Conclusions

This evaluation of the Met’s audio guide has provided a strong evidence base for future decision making for the audio guide and other mobile initiatives. These findings support the Museum’s new strategic approach to the audio-guide service, investing financial and human resources to meet the needs of specific audiences and to better support specific learning objectives.

Thinking about the audio guide as a service can allow museums to see their mobile projects within the larger continuum of the visitor experience that includes the digital and the non-digital. A service-design model also supports an ongoing iterative process of identifying the audience need, adjusting the offer, and refining the service. Designing successful services requires the cross-departmental collaboration of teams from digital media, curatorial, education, visitor services, facilities, and marketing. By designing digital services, museums can maximize the impact of their mobile offers and more successfully serve an ever-broadening audience base.

Acknowledgements

We’re grateful to Alyson Webb, Loic Tallon, and Paco Link for their contributions to this paper, and to the entire audio-guide team at the Met for their support of the evaluation process.

References

Metropolitan Museum of Art. (2014). Audio Guide Evaluation and Usability Testing. Frankly, Green + Webb. Unpublished.

Polaine, A., L. Løvlie, & B. Reason. (2013). Service Design: From Insight to Implementation. New York: Rosenfeld Media.

UK Government Digital Service. (2012). Design Principles. Item 8, “Build digital services, not websites.” Last updated July 2, 2012. Consulted January 23, 2015. Available https://www.gov.uk/design-principles#eighth