The Baumann Marionette Project: Virtual marionettes take the stage
Mimi Roberts, New Mexico Department of Cultural Affairs, USA, Jonathan Lee, NMHU, USA, Elizabeth Starks, New Mexico Highlands University, USA, Joey Montoya, New Mexico Highlands University, USA, Brandi Daw, New Mexico Highlands University, USA
AbstractThis paper describes how a rare museum collection fueled a broad and diverse partnership and a new entity (the Cultural Technology Development Lab) and showcases some emerging technologies and processes ideally suited to preserving and providing access to fragile three-dimensional (3D) objects in museum collections and cultural heritage sites. The Gustave Baumann marionettes were ideal subjects for interactivity. Because of their fragility, visitors are unlikely to ever see them in motion as they were designed. Being able to embody the models gives new life to these objects while giving visitors a new way of interacting with them in the museum, a space that is often considered hands-off. The process of 3D scanning and modeling opens up a wealth of new opportunities for surfacing all kinds of objects in collections, not limited to marionettes. It can be applied to everything from sculpture and basketry to fossils and natural history specimens. Once the 3D models are available, they can be deployed throughout the institution, behind the scenes for research and conservation, on the exhibit floor, for enhanced visitor engagement and accessibility, and beyond museum walls, for education and outreach.
Keywords: cultural technology, non-profit R&D, virtual marionettes, exhibit technologies
The year 2015 marks the tenth anniversary of the partnership between the New Mexico Department of Cultural Affairs (DCA) and New Mexico Highlands University’s Department of Media Arts and Technology (NMHU); the fifth anniversary of our AmeriCorps Cultural Technology (ACT) internship program; and the third anniversary of our Community Technology (makerspace) program and Cultural Technology Development Lab (CTDL). Collectively, these programs comprise the Center for Cultural Technology (CCT), a center of excellence in the domain of cultural technology, the emerging field at the intersection of computer networks, digital media, and cultural content.
CCT is grounded in an academic department at a regional, Hispanic-serving public university in the northeastern quadrant of New Mexico that offers degrees in media arts and in software systems design. Students who opt for an emphasis on cultural technology do so through the selection of elective courses and paid internships in cultural institutions and non-profit organizations. The majority of the students come from low-income and underserved-minority backgrounds (http://www.cctnewmexico.org).
DCA is a state agency that umbrellas a system of museums, historic sites, the New Mexico State Library, and other divisions. It also serves the broader statewide constituency of museums and libraries. The CCT partnership offers NMHU students access to opportunities for paid internships, through which they receive mentorship, opportunities to hone their professional skills, and exposure to career paths. The CCT partnership offers DCA and New Mexico’s cultural community practical and affordable access to expertise, fresh ideas, and new technologies.
The students who have worked on the Baumann Marionettes Project represent the diversity of creative and technical talent for which CCT has become known. Daniela De Angeli was an international student who came to the graduate program at NMHU after an introduction at Museums and the Web 2007, which she attended with the delegation from the Universita della Svizzera italiana. She is currently studying for her engineering doctorate at the University of Bath and continuing her research on the marionettes. Paige Hicks, a native of New Mexico, received her master’s degree from NMHU after interning at the Balboa Park Online Collaborative and is currently working at Blizzard Entertainment as a game programmer. Elizabeth Starks, a native of Zuni, New Mexico, began her museum career as a designer at the Natural History Museum in San Diego. While there, she met Michael Jackson, who was a Highlands intern at the Balboa Park Online Collaborative, and who recruited her. She was returning to New Mexico to enroll in the museum studies certificate program at the Institute for American Indian Arts in Santa Fe, after which she enrolled in the NMHU SSD graduate program. Rianne Trujillo, a native of Las Vegas, New Mexico, received her undergraduate degree in media arts from NMHU prior to enrolling in the SSD graduate program.
The next iteration of the project, described herein, is being spearheaded by exhibit curator Joey Montoya, currently a media arts instructor, who received his master’s degree in media arts from NMHU and participated in internships at the Georgia O’Keeffe Museum and New Mexico History Museum. He is assisted by AmeriCorps Cultural Technology intern Brandi Daw, a Media Arts graduate student. They will work closely with Dave Morgan, the director of the Carlsbad Museum, and faculty advisor Jonathan Lee to showcase new technologies and how they are being developed for use in museums. Newly produced multimedia and interactive components and graphics panels will lead visitors through the sections of the exhibit, each featuring a different new technology.
2. The Baumann marionettes
Last year, CCT received an exciting invitation to tell our story in the form of an exhibition from the Carlsbad Museum and Art Center, a cultural hub founded in 1931 that serves Carlsbad and the rural surrounding communities in southeastern New Mexico. Their exhibits focus on the art, history, and culture of the region and the greater American Southwest. As a municipal community art center, the Carlsbad Museum also hosts exhibitions from schools and local arts and crafts organizations. After some thought, we decided that we would let the CCT story be told to the Carlsbad audience by virtual marionettes.
Several years ago, in 2012, CCT had received an invitation from the New Mexico Museum of Art in Santa Fe that introduced us to the Baumann marionettes. In the basement of the museum is a room where collections are stored. Among the paintings and sculptures lives a rare collection: over seventy wooden marionettes hand-carved by artist Gustave Baumann during the 1920s and 1930s.
Baumann was a German immigrant to the United States. In 1918, he found his way to Santa Fe, where he became active in the art colony of that time for the next fifty years. While he is primarily known professionally as a printmaker, he also carved wooden marionettes for the amusement of family and friends.
This fanciful wooden array of people and animals, once the center of attention on a puppet stage in the Baumann living room, is now hidden away in various states of disrepair, strings tangled and limbs disconnected, in the dark safety of museum storage. Their age and fragility means that they are rarely displayed and can never again be used as designed. Annual Christmas performances at the museum use replicas.
3. The “Virtual Marionettes”
Figuring out a way to surface this collection and encourage an interactive experience has long been a goal for the collection’s curators. After visiting the marionettes in storage and getting to know a few of them (plus their caretakers), we thought this would be a perfect project for CCT.
The Baumann Marionettes Project was designed to demonstrate the possibilities of bringing this rare, rarely exhibited museum collection to the public through the use of emerging technologies. Luckily, around the same time, the Microsoft Kinect had come along (which allows sophisticated motion sensing) and made a project viable. We decided that the Kinect (and now the Leap Motion as well) would help us build the best interactive experience. The code for this project would be open, with documentation, and designed to be shared with other museums with an interest in surfacing a fragile collection of this nature. The first phase of the research and development project began in February 2013 and was completed in July 2013.
Also around this time in 2012, CCT received grant funding from LANS LLC, the consortium operating Los Alamos National Lab, and we were able to use some of the money to pilot two projects that would launch the CT Development Lab, a program where university faculty and students, museum professionals, and other partners work together on design and technology solutions for cultural institutions. The first project was a mobile pneumatic tubes display, and the second project would be the “Virtual Marionettes.” Our puppet workshop would be in the new CCT Museum Classroom, a satellite instructional and project space at the New Mexico Museum of Natural History and Science in Albuquerque.
The technologies involved allowed a team of faculty and students to create digital three-dimensional (3D) models using the computational photography technique of photogrammetry, work with the Kinect SDK (software development kit), write custom programming to control the marionettes, and create printable 3D models. The interactive is now the centerpiece of a small mobile exhibit. Visitors have a chance to play with the marionettes, learn about the history and craft behind this incredible collection, and connect with the disappearing skill of puppeteering.
The Baumann Marionettes Interactive was included in the exhibit “Gustave Baumann: A Santa Fe Legend,” which opened at the Las Cruces Museum of Art in February 2014. On February 20, 2014, they had their international debut when they were presented at Museums and the Web Florence, as part of the panel on using 3D in museums. (Langer & De Angeli, 2014). In the summer of 2014, they added pizazz to an exhibition of Baumann’s prints in the Governor’s Gallery at the New Mexico State Capitol (http://www.nmartmuseum.org/site/exhibitions/past/past-exhibitions-2014/prints-of-gustave-baumann-governors-gallery.html).
The first step in the process of creating the virtual marionettes was to create digital 3D models of some of them using photogrammetry, a computer process that uses still photographs. This allows for accurate measurements to be taken from the model for a variety of uses. Fortunately, we already had gained some experience with the technique during the summer of 2012, when AmeriCorps Cultural Technology members Joey Montoya and Greg Williamson were invited to intern with Georgia O’Keeffe Museum Senior Conservator Dale Kronkright on a grant-funded summer project to test and document photogrammetry and Reflectance Transformation Imaging on their efficacy for the preservation of historic structures and collections. The faculty advisor was NMHU Professor Megan Jacobs (see “Applications of Digital Photogrammetric Methods for Preservation Documentation of Historic Homes” (Jacobs, 2012)).
Most of the models created were areas of Georgia O’Keeffe’s home in Abiquiu, New Mexico. Photogrammetry was being tested for monitoring change over time in the structures, and also for the ability to archive 3D models. Photographs taken of exterior parts of the home were stitched together to form the models; however, not all of them were stitched together, because the computer power necessary for this task was more than expected.
In January 2013, Kronkright and Jacobs led a group of six students in Jacobs’ advanced photography course to Cuba for a cultural exchange that included creating 3D photographic models of architectural treasures to aid preservation. The partnership continued during the fall semester, when Jacobs’ class collaborated with the Georgia O’Keeffe Museum on a research project that refined the photogrammetry workflow (Daw, 2013).
Figure 1: example photogrammetry setup
In November, the culmination of the year’s work was a presentation at the Museum Computer Network conference (Grassie et al., 2014). In April 2014, students Joey Montoya and Brandi Daw demonstrated photogrammetry at the joint Texas Association of Museums/New Mexico Association of Museums Conference in Lubbock, Texas, as part of the Horizon Report panel, as an example of NUI (natural user interfaces) (Daw & Montoya, 2014).
5. Natural user interface
What was learned from the collaboration with the Georgia O’Keeffe Museum is that photogrammetry is very useful for preservation and archival purposes. The next step would be to combine it with interactive technology to provide access to historic structures or fragile treasures that would otherwise be outside the public eye.
In February 2013, with the wonderful cooperation of the staff at the New Mexico Museum of Art, Professor Jacobs and graduate student Daniela De Angeli photographed five marionettes: Pecos Bill, Nambe Nell, Lord L, Gus Baumann, and the Rooster and Pinocchio. There was a lot of experimentation in creating a structural/lit environment to get good data, all of which is outlined in the documentation. (Gallagher, 2013).
Once Jacobs and De Angeli finished their part, the project was turned over to the software team to be stitched and cleaned. Professors Miriam Langer and Jonathan Lee served as project designers. The team included Lead Developer Paige Hicks, a recent graduate of the CCT program, and Marionette modelers Elizabeth Starks and Rianne Trujillo, both graduate students in Software Systems Design.
Figure 2: rendering of Rooster marionette in 3D software Blender
Scans often have extraneous artifacts or holes in them and need to be smoothed or patched. The scans were cleaned up and posed in the open-source 3D modeling software Blender and made interactive by having a virtual skeleton applied and linked to the Kinect sensor via the 3D interactive programming environment Unity.
Of the five scans, all but Pinocchio were useable. Pinocchio’s problems were due to a few things, including the darkness of his costume and even the difficulty of getting frontal focus on his huge puppet nose! The others looked great. Because of the limitations of the actual puppet of the rooster, and how much of the virtual skeleton would have to be manipulated and frozen in Unity, it was decided that that model should be left out of the initial version of the exhibit. Some 360-degree pdfs were given to the museum to put on their website.
Once the models were functional and able to move as the original marionettes, the Kinect was used to detect the user’s movements and to match the user with the marionette. The open-source solutions OpenNI and NITE were used to allow skeletal rigging of the models. Model articulations were prepared in Blender, before importing the .obj files in Unity.
Figure 3: users controlling marionettes via the Kinect
The models were programmed to move as actual marionettes do, with each virtual marionette having the same joints and same potential movements as the actual marionettes. This allows the visitor to learn about the complexity and limitations of manipulating a marionette for use on stage with hand and body motion—perhaps even creating an original play along the way. Users see themselves embodied as Nambe Nell, Pecos Bill, Gustave Baumann, or Lord L on a projected or screen-based stage set with Baumann’s props and painted backdrops. Up to two users and two marionettes can perform on the stage at any time, acting independently or together.
6. Creative digitization for audience engagement
Once the virtual marionettes had been brought to life, we were eager to see what else could be done with the 3D images and other technologies, some of which were new and just coming onto the market. For the Carlsbad exhibit (November 2015 to January 2016), the goal is to expand the current exhibit from the single interactive by adding new sections that extend the mission of creative digitization for audience engagement.
The exhibit team for the Carlsbad Museum and Art Center, including curator Joey Montoya, intern Brandi Daw, museum director Dave Morgan, and faculty advisor Jonathan Lee, will produce graphics and new multimedia and interactive components that demonstrate new technologies being developed for use in museums. In addition to the current interactive, the expanded exhibit will include the following components:
- Leap Motion Controller, developed by Rianne Trujillo as a class project, is similar to the Kinect controller, but instead of skeletal sensing, it senses ten fingertips and tracks their movement. This will give users finer control of the marionettes in a way that feels more like a puppeteer’s experience (http://lpb-riannetrujillo.com/marionettes/).
- Pepper’s Ghost, a technique for creating ghostly images that has been used by magicians and in haunted houses since John Pepper popularized it in the 1880s. It works by reflecting an image off of a sheet of glass or plexi and allows users to see what appears to be the marionette but is actually a holograph (i.e., a 3D image that changes as the position and orientation of the viewer changes, as if the object were present).
- Pico Projectors, inexpensive handheld devices that project video. In a museum-exhibit context, we hope to use them to create one or more tiny 8-inch by 8-inch single-viewer dioramas, like looking through a viewfinder or a keyhole.
- AR will allow visitors to use their own devices (smartphones/tablets) to see additional exhibit content by using AR applications (Layar, Junaio, Aurasma, etc.) when viewing objects or printed materials. One idea is to have the graphic panels themselves be AR triggers (e.g., to see animated versions of the marionettes, such as recorded puppet show vignettes). Another idea would be to add this feature to promotional postcards.
7. Makerspace programming ideas
The Baumann marionettes are beloved for many reasons. Like most people, we are attracted by their obvious charms, but there was an additional reason. In a parallel development, CCT has been developing our own version of a museum maker program, one that we call Community Technology. In fact, Miriam Langer organized the first museum conference panel on the topic of museum makerspaces at the Museum Computer Network Conference in Seattle, Washington, on November 9, 2012 (MCN, 2012).
The Baumann Marionettes are the perfect embodiment of CCT’s commitment to honoring handwork and reclaiming New Mexico’s maker heritage while introducing new technologies to museum audiences. Because of their appeal to all ages, the marionettes also embody our focus on intergenerational learning experiences, which finds particular cultural relevance in New Mexico, where knowledge and the mastery of practical skills have traditionally been passed down from generation to generation.
CCT is committed to museums and libraries as environments for maker programs because they provide cultural and historical context. The Community Technology program trains students and interns to offer maker programs in cultural institutions and offer professional development opportunities for museum personnel and librarians through the New Mexico Makerstate Initiative, a partnership with the New Mexico State Library (http://themakerstate.com), and MakeCations, immersive experiences including workshops, community events, and discussion (http://www.makecation.org).
Along with all of the other reasons museums choose to offer maker activities, the opportunity to use them as a way of reinterpreting and deepening a sense of connection to our collections remains largely untapped. In an effort to explore the potential for this, we will be working with the Carlsbad Museum to develop some maker activities themed to the exhibition.
Visitors will be invited to assemble their own marionettes from 3D printed parts. The hardest part will be keeping the printer running and repairing it when it (inevitably) breaks. Ideally, the museum will be able to recruit one or more community volunteers who would be trained to staff this station and make printer repairs and provide demonstrations and workshops for the public.
We would like to reproduce a photogrammetry studio setup for use in demonstrations and workshops to teach visitors to create their own 3D photographs using photogrammetry techniques. It will include a box of models or a marionette replica on a rotating base for visitors to scan, with step-by-step instructions on how to shoot the photos, clean them up, save them, etc.
8. The CT Development Lab
Over the past decade that we have been building CCT, museums across the United States have developed in-house media labs or teams capable of producing exhibit technologies and online presences, including the Indianapolis Museum of Art, Metropolitan Museum, and Smithsonian’s Cooper Hewitt. The effort to integrate technologists into the museum workplace, however, has not been easy. People from museum and IT backgrounds do not always to speak the same language. Those schooled as museum professionals can be resistant to change and do not always comprehend the amount of time and skill involved in production. Those schooled as technologists don’t always fully understand deeply embedded institutional cultural norms and values.
In our situation, working in a poor state (New Mexico), cultural heritage is preserved by mainly small- to moderate-sized institutions spread over a geographic expanse populated by rural villages and towns—museums like the Carlsbad Museum and Art Center. No single institution has the financial resources or staff for developing technology; sharing costs and pooling resources makes sense. Necessity has been the mother of invention for our media lab model. The CT Development Lab is able to serve multiple institutions statewide and nationally. Integration with the academic and internship programs and our cost-sharing model maximize cost effectiveness.
The management of the CT Development Lab consists of NMHU professors Miriam Langer, Jonathan Lee, and Stanley Cohen. Undergraduate and graduate students work on projects alongside faculty members, museum professionals, and other partners. Since we began in 2012, students have worked on apps for SITE Santa Fe, Colorado National Monument, Acadia National Park, an e-paper labeling system for a national consortium of museums, and the Museduino, an in-house solution for exhibit developers: an open-source hardware kit for creating robust public exhibits using the Arduino IDE.
On the down side, working at the intersection of university, state, and federal bureaucracies can make many days feel like a perfect storm. But the frustrations are bureaucratic rather than work related. There are three advantages to our model that might find relevance to other cultural institutions:
- Consortia, partnerships, and collaborations and the ability to serve multiple institutions provide affordable access to technology for cultural institutions
- Involvement of students, interns, and faculty to work alongside museum staff provides access to new technologies and fresh ideas for institutions that lack in-house expertise
- Commitment to community and the opportunity to contribute help build strong development teams with shared goals
The Carlsbad Museum and Art Center is representative of New Mexico museums. They need to adopt new technologies to attract tourists whose expectations for their visitor experience have been raised by visits to museums in other parts of the country where exhibit technology is more common. Without in-house IT support, it is important that these technologies are simple, robust, and affordable so that they stay in good working order.
In addition, these museums need new ways to attract local residents, for whom visits tend to be event driven. Opportunities to participate in hands-on activities centered on new technologies can be used to make new connections with locals, many of whom have skill sets that they are eager to share, and others who might be hesitantly trying their hands at a new skill for the first time.
On the most basic level for the CT Development Lab, the Baumann Marionette Project is about solving technological problems. On a deeper level, it is also about modeling a collaborative process that brings together university faculty and students, museum professionals, and other partners. And at its deepest level, it is about connecting museums and communities through a shared love of cultural heritage collections and historic sites, and appreciation for the art of making.
When the virtual marionettes take center stage at the Carlsbad Museum and Art Center, they are sure to delight new audiences with their silly antics, just as their wooden forebears did in their performances in the Baumanns’ Santa Fe living room. The physical collection can be preserved, safely in storage, while new virtual versions will put them once again in the spotlight. And residents of southeastern New Mexico and visitors to Carlsbad will be inspired by a little-known piece of New Mexico art, history, and culture as wondrous as the new technologies that make it all possible.
This paper brings together and updates a series of previous presentations and papers by the project participants cited herein. Thanks are owed to Mary Kershaw, director of the New Mexico Museum of Art, and Michelle Gallagher Roberts, head of Registration and Collections; Patsy Jackson-Christopher, City of Carlsbad, director of Arts and Culture; and LANS LLC, our funding partner. A wonderful book about the marionettes is available from Museum of New Mexico Press (Zieselman, 1999).
Daw, Brandi. (2013). “Goals for the Semester.” GOK blog. December 12. Available https://bdaw2004.wordpress.com/2013/12/12/gok-blog
Daw, Brandi, & Joey Montoya. (2014). “Computational Imaging: Photogrammetry.” TAM2014 Photogrammetry Presentation. Available https://presentationtam2014.wordpress.com/2015/02/15/2/
Georgia O’Keeffe Museum. (n.d.). “Photogrammetry | 3D: Photo Imaging for Condition Detection, Monitoring, and Documentation.” Available http://www.gokmconservation.org/scientific-methods/photogrammetry
Grassie, Billie, Megan Jacobs, Jonathan Lee, & Mariano Ulibarri. (2013). “Re]envisioning the Future: Photogrammetry for Museums: From Super Geek to Super Easy.” Available http://www.slideshare.net/MeganJacobs/photogrammetry-for-museums
Jacobs, Megan. (2012). Applications of Digital Photogrammetric Methods for Preservation Documentation of Historic Homes. Available http://ncptt.nps.gov/wp-content/uploads/2012-11.pdf
Langer, Miriam, & Daniela De Angeli. (2014). “The Gustave Baumann Marionettes- Surfacing a Rare Collection through Interaction.” PowerPoint. Museums and the Web. February.
Museum Computer Network (MCN). (2012.) “MCN 2012: What’s the Point of a Museum Maker Space.” November 9. Seattle, Washington. https://www.youtube.com/watch?v=LzsNg1avK5c
Roberts, Michelle Gallagher. (2013). “The Baumann Marionettes Go 3D! – 2/27/13.” Available http://www.nmartmuseum.org/cypher-space/the-baumann-marionettes-go-3d.html
Zieselman, Ellen. (1999). “The Hand-Carved Marionettes of Gustave Baumann, Share Their World.” Museum of New Mexico Press.
. "The Baumann Marionette Project: Virtual marionettes take the stage." MW2015: Museums and the Web 2015. Published February 27, 2015. Consulted .