From physical to digital: Recent research into the discovery, analysis, and use of museums resources by classroom educators and students

Darren Milligan, Smithsonian Institution, USA, Melissa Wadman, Smithsonian Institution, USA

Abstract

Keywords: education, learning, digital, metadata, teacher, student

1. Introduction

In the second half of the twentieth century, museums found themselves obligated to respond to new pressures to demonstrate impact. This change, now often referred to as new museology, is a period (arguably one in which we are still) in which museums recognized the need to respond to their audiences’ desire for museums to have an impact in their educations and lives (both locally and globally) (Falk & Dierking, 2000). One way museums addressed this demand was to experiment with the use of digital outreach tools, with most, including the Smithsonian, going online in the mid-1990s. What followed in the age of Web 2.0 were museum audiences’ new expectations for participation. Silverman (2014) has called this transformation a “third age of museums.”

One that is defined not only by their important and impressive collections … and not just by their roles as educational institutions, museums have entered an age of social service. One that emphasizes the use of their collections … to foster change in our communities, our neighbors, and our world.

What follows is an examination of how the Smithsonian, and in particular its central education and outreach agency, has taken a research-based approach to understanding and responding to the needs of audiences in one specific environment: the classroom.

Museums have long identified classroom educators and their students as primary audiences. Through object loans, lesson plans, and more recently digital resources, our institutions have recognized the potential for the narratives told through exhibitions to reach students outside the museum walls. Many of these traditional models no longer reflect the realities of the contemporary classroom, one being rapidly transformed by budget shortcomings, evolving teaching standards, and the potential of new technologies. Museums are uniquely positioned to play a greater role in formal education, but to meet this potential, they must understand the needs of these audiences and the capacity of their institutions to address them.

Within the Institution, the Smithsonian Center for Learning and Digital Access (SCLDA) is a center of excellence focused on establishing the Smithsonian as a learning laboratory for everyone. As new technologies make it possible for audiences to connect with Smithsonian researchers, collections, and educational programs like never before, SCLDA’s mission is to coordinate among the museums and offices within the Smithsonian, provide models and methods that enable learners to access everything the Smithsonian has to offer, and empower learners to explore their own interests and engage with others (http://smithsonianeducation.org/tools/about.html).

SCLDA has a long history of both developing and providing unified access to Smithsonian learning resources. Currently, on the central education website for the Institution, Smithsonian Education (http://smithsonianeducation.org), SCLDA provides a searchable database that catalogs more than two thousand learning resources (lesson plans, research databases, interactives, games, etc.) for the classroom from more than thirty Smithsonian museums, research centers, and programs. This database search feature is used more than one hundred thousand times a year to locate the rich and diverse learning resources that educators from across the Smithsonian have produced.

Since 2010, SCLDA has conducted a series of research projects to better understand how classroom educators and their students access and use museum learning resources. This paper shares the results of these five independent, yet progressional, studies and offers best practices garnered from a meta-analysis of their results.

Remedial Evaluation of the Materials Distributed at the Smithsonian Institution’s Annual Teachers’ Night (2010)
Previously unpublished, this literature review and evaluation produced generalizable guidelines for the design and development of museum-based lesson plans and investigated classroom educator methodologies for incorporating museum-based lessons into classrooms.

Capturing the Voice of Customer, a Satisfaction Insight Review of SmithsonianEducation.org (2011)
Collected from more than seven thousand surveys completed by visitors to the central Smithsonian Education website, the makeup of this audience, their motivations for site visitation, their activities while on the website, and their sources of dissatisfaction were explored.

Digital Learning Resources Project (2012)
Data analyzed in the first two studies indicated that educators were not using instructional materials as published, but were rebuilding them to fit their curricular and student needs. To better understand this and provide a road map for future digital resource development, this previously unpublished evaluation and iterative prototyping project was conducted.

Piloting Tools to Enable Active and Participatory Learning for Middle School Students: Facilitating Digital Learning with Smithsonian Digital Resources (2014)
Throughout these studies, consistently, classroom educators have indicated that relevance to students’ needs are a top priority. Recently, we have begun testing directly with students to better understand how they use digital museum assets (specifically digitized collection objects) and to document the types of scaffolds necessary to enable active and participatory learning using them.

Increasing the Discoverability of Smithsonian Digital Resources: Learning Resource Metadata Initiative (2014)
Key to having the intended impact with educators and students is the need to ensure that our existing resources are highly discoverable. Findings from a detailed evaluation of the Smithsonian’s LRMI project are shared, including a relevance assessment by target users of search-result-page and museum-website-page content.

For the purposes of brevity and utility, the reader will find below both summaries and at times brief excerpts of the methodologies and results from the original, often lengthy, reports. For a fuller picture of this work, it is recommended that the complete reports (links to most can be located in the References section) be examined. It should be noted that many contributors participated in the research design, data collection, analysis, and writing of the original reports (some of which appears in these summaries). Those contributions are recognized in the Acknowledgments section.

2. Remedial Evaluation of the Materials Distributed at the Smithsonian Institution’s Annual Teachers’ Night

Background

Teachers’ Night is an annual event hosted by the Smithsonian Institution and coordinated by the SCLDA. The event provides free teaching resources from the Smithsonian’s nineteen museums and nine research centers (as well as additional libraries, archives, and the National Zoo) to thousands of primarily K–12 classroom educators. The event includes booths staffed by museum educators from each Smithsonian agency, presented in a different Smithsonian museum each year. The event last approximately three hours and attracts primarily local (Washington, D.C., Maryland, and Virginia) classroom educators and administrators.

The goals of Teachers’ Night are to:

1. Raise awareness about the educational resources offered by the Smithsonian, and
2. Help classroom educators incorporate Smithsonian collections and resources into the classroom.

The purpose of the Teachers’ Night remedial evaluation was to determine if the Smithsonian Institution’s second goal for the annual event—to help classroom educators incorporate Smithsonian collections and resources into the classroom—was being realized.

Methodology/sample

The evaluation consisted of three phases.

Phase one consisted of an extensive literature review to explore the following questions:

1. What methods do classroom educators use to find and use lesson plans?
2. Do lesson plans from museums meet classroom educators’ needs?

Phase two was an analysis of comments by visitors to SmithsonianEducation.org on specific Smithsonian lesson plans (from five Smithsonian agencies: the Anacostia Community Museum, National Museum of Natural History, National Museum of the American Indian, National Science Research Center, and Smithsonian Center for Education and Museum Studies). The analysis reviewed 132 public comments classroom educators posted in 2008. The purpose of the content analysis of these comments on Smithsonian lesson plans was to identify themes in alignment with the literature review.

Finally, phase three consisted of focus groups and in-depth interviews with classroom educators. Two focus groups and four in-depth interviews were conducted to address two questions:

1. What design elements are most important to classroom educators?
2. What extent and in what ways do classroom educators make use of materials provided by the Smithsonian?

Figure 1: the distribution of classroom resources at Smithsonian Teachers’ Night 2012 at the Smithsonian’s Donald W. Reynolds Center. Photo © Bruce Guthrie. Used with permission.

Findings

Phase one

The literature review confirmed that the usability and navigability of websites are important. The research suggests that there are common usability problems for visitors (including classroom educators) who are non-museum professionals when using museum websites:

• Frustration with overloading of content
• Distracting graphical user interfaces
• Browsing not conducive to understanding specific topics
• Difficulties for non-museum professionals with certain terminology
• Disconnect of museum websites to the physical museums

Regarding the suitability of lesson plans for classroom use, the literature review presented several key requirements classroom educators need for museum material to be incorporated into their teaching. Material must be:

• Aligned to curriculum standards
• Updated
• Interdisciplinary
• Related to big concepts
• Educational
• Not dependent on museum visits

Phase two

The content analysis of classroom educators’ comments on lesson plans on SmithsonianEducation.org reinforced what was found in the literature review. Important aspects for classroom educators looking to use museum resources are that the material being enjoyable for their students (in other words, fun to use), its user-friendliness, interdisciplinary nature, adaptability, alignment with curriculum standards, and flexibility to accommodate a diversity of students.

Phase three

The focus groups and in-depth interviews demonstrated the importance of the design of Smithsonian materials to how classroom educators use them and whether classroom educators are using Smithsonian materials as intended. Educators came to the focus groups and interviews with a several preformed paradigms, including the needs and personalities of their students, the multilayered administration in which they work, and the brand trust of the Smithsonian. These ideas are highlighted in a passage from the evaluation report:

Students are deemed the biggest reward but also the biggest challenge for classroom educators; the diversity of students and learning gaps (e.g., a child whose first language is not English learning amongst English speaking children) entails difficulties for classroom educators in their lesson plans. Expectations and requirements from the federal and state level as well as from administration within schools also present a challenge to teachers, one that is not conducive to adapting to the diversity of their students.

Educators reported that they used Smithsonian materials acquired at Teachers’ Night as resource starting points for their teaching. They expressed that materials need to be visually appealing, useful right out of the box, durable, and multipurpose. Feedback suggested that classroom educators are selective when using Teachers’ Night materials. Museum materials do not always fit into their teaching (museum materials must be deconstructed and then reconstructed). Educators expressed the desire for the Smithsonian materials to be more readily usable in the classroom. There was no definite indication that participants from the focus groups used the materials as intended by the Smithsonian.

The outcome of the evaluation of Smithsonian Teachers’ Night indicates success in that classroom educators appreciated Smithsonian services and materials from the event. It also suggests more work needs to be done to make Smithsonian materials more usable for classroom educators by making them easier to adapt (to be able to easily pick apart salient information and resources) and by aligning materials more closely to the curriculum and standards that classroom educators have to follow.

3. Capturing the Voice of Customer, a Satisfaction Insight Review of SmithsonianEducation.org

Background

SmithsonianEducation.org was launched in 2003 by the SCLDA (then called the Smithsonian Center for Education and Museum Studies). The core objectives for the SCLDA website were to serve as the gateway to Smithsonian educational resources; tailor content to serve three distinct audiences: educators, families, and students; promote the understanding and use of museums; and emphasize inquiry-based learning using primary sources. During the early 2000s, websites were moving into a new era by acknowledging and addressing the unique needs of different user groups. This approach also supported the shift in museology in which museums moved from being “about something” to being “for somebody” (Weil, 1999).

Throughout the past twelve years, the site has grown to house SCLDA-published materials and to serve as the gateway to the Institution’s more than two thousand digital educational resources (from thirty-two museums, libraries, archives, and research centers) indexed and searchable by national and Common Core State Standards. In 2007 and 2008, the site was selected as the People’s Voice winner for Best Cultural Institution Website in the Webby Awards.

While the site remained popular throughout its first half decade, little was known about its users. In order to understand how best to properly serve this specific audience (what was assumed to be classroom educators accessing digital museum resources), SCLDA found it crucial to first deeply understand their usage patterns.

Methodology/sample

Beginning in 2009 and running continuously for two years, a user satisfaction survey (using the methodology of the American Customer Satisfaction Index) was conducted via SmithsonianEducation.org. The study was originally intended to highlight areas of improvement to the existing website information architecture and content, but it more importantly provided us with a deep look into who our audience was, their motivations for coming to the site, and their activities while there.

A total of 7,470 surveys were completed during the twenty-four-month span of data collection through a popup window presented to website visitors. The survey consisted of twenty-one numerically scored model questions, as well as ten multiple choice and five open-ended custom questions developed by SCLDA. The survey invitation was presented at random to 30 percent of visitors who had viewed two or more pages of the site. Once accepted, the actual survey was presented as visitors left the site while a session-level cookie prevented re-invitations to complete the survey (during the same session).

Figure 2: SmithsonianEducation.org educators homepage as it appeared in 2014.

Findings

It probably comes to little surprise that a site called “Smithsonian Education” would primary appeal to classroom educators. About half of visitors (48 percent, n=3,578) to the site identified themselves as “teachers,” by far the largest audience segment. However, if you add in other formal education audiences (Librarian, Curriculum Developer, School Administrator), we find that this segment increases to approximately 56 percent.

To dive deeper into the dominant visitor segment to the site, we then asked the subject area taught by those self-identifying as a teacher (the survey here allowed multiple selections). The data points to a fairly equal distribution between Language Arts (34 percent, n=915) , Science (34 percent, n=915), and Social Studies (33 percent, n=888). Again, it was not a surprise that more generalized subjects are dominant (Language Arts and Social Studies), as 65 percent of teacher survey respondents taught in grades PreK–8.

Inherent to understanding satisfaction (the original intention of the survey) was a need to identify the motivations that lead to users arriving at the website. Almost 60 percent (congruent with the number of visitors in formal educational roles) came to the site with aspirations of finding educational resources.

Now that we understood the makeup of the visitors and their reasons for arriving at the website, we looked into the actual activities they performed while on the site to correlate intention with real action. Most came to access the content: to read it online (48 percent, n=3,344), to search for it (47 percent, n=3,274), to download it (25 percent, n=1,742), and to share it with others (22 percent, n=1,533).

Diving even deeper still, we asked the question: “What were you primarily looking for today?” We see here a connection between visitor motivation pre-visit and the actions performed on the site. Fifty-nine percent came to the site to “find educational resources,” and again the majority end up performing this type of search, in the form of teaching resources, lesson-plan downloads, or content information (clarified in the survey to mean specific information, such as oceanography).

This data, while very useful to the development of an understanding of the site’s users (and instrumental in SCLDA’s next study, the Digital Learning Resources Project) may have shifted since the original survey closed in 2011. Recently, a similar survey has begun to collect user information again on SmithsonianEducation.org, but has yet to be analyzed.

4. Digital Learning Resources Project (DLRP)

Background

The impetus for the Digital Learning Resources Project was to assist the Smithsonian to better understand the educational uses of Smithsonian digital resources and provide a road map for future digital development. Building on research findings from the two previous studies, the specific research objectives focused on classroom educators’ ability to identify, analyze, and extract digital content, with the ultimate goal of enabling all users to achieve their own personal learning objectives through the Smithsonian’s resources.

The project had three sets of intended outcomes for the short, medium, and long term:

• Short-term outcomes were to increase classroom educators’ skills in identifying specific Smithsonian digital learning content, analyzing specific Smithsonian digital learning content, and extracting specific content from Smithsonian digital learning resources.
• Medium-term outcomes were to increase skills to make strategic use of digital media and visual displays of data to express information and enhance understanding; and creativity.
• The long-term outcome is to foster online users who are active creators of digital resources personalized for learning in their own classroom.

The project was documented and is archived in total on the project wiki site (http://smithsonian-digital-learning.wikispaces.com).

Methodology/sample

The work (conducted between April and October 2012) was organized across four phases. Phases one and two involved focus groups with twenty classroom educators in Northern California combined with user analytics drawn from Brokers of Expertise (now known as the Digital Chalkboard; see https://www.mydigitalchalkboard.org/), a resource repository containing metadata describing Smithsonian digital learning resources. In addition, during these early phases a literature review and an environmental scan were conducted to further refine goals and research questions. During phase three, initial prototypes were developed and tested by a larger group of sixty-nine classroom educators over a three-week period in the summer of 2012. In Phase four the prototype was finalized based on the findings of the testing and a full set of requirements for the eventual build of the prototype was developed.

Figure 3: a teacher prototyping session held in Washington, D.C.

Findings

Phases one and two

The results of the research in phases one and two yielded recommendations that fall into three categories: Improved search and visualization; Content that is engaging, standards-aligned, and learner-centered; and Instructional tool suggestions.

Search and Visualization Tools
The literature suggests that museums need to make resources more findable and to generate assets that are personalized and accessible anytime, anywhere, and on multiple platforms. Classroom educators also asked for:

• Search results with thumbnails, previews, tag clouds, and rating systems that allow them to easily identify what is useful and what is not
• Personalized search hints
• Search capabilities that can be either highly filtered or extremely broad

Engaging, Standards-aligned, Learner-centered Content
Previous findings suggest that classroom educators put student interest and engagement at the top of their list and need content that aligns with learning goals and standards. When analyzing resources, classroom educators want content that will:

• Engage students
• Allow for student interaction and adaptation
• Afford accessibility for various learning styles and levels
• Offer coherence with the lesson and multidisciplinary opportunities
• Support problem-based learning goals
• Support standards-based teaching goals
• Convey a virtual museum experience

Instructional Tools
Deeper exploration answered new questions about how classroom educators use museum digital content in their classrooms. When extracting resources, classroom educators want:

• Flexible technologies for a diversity of devices and delivery methods
• Tools to assess learning
• Tools to adjust reading level of text
• Ways for classroom educators to upload their self-authored components into a saved file, or resources from other sites or collections
• Specific pedagogical tools ( such as Graphic organizers, Vocabulary/glossary builders, and Discussion and question area)

Despite this consistency, there is some diversity of opinion reflected across these data that should be noted. While the majority of classroom educators in the SmithosnianEducation.org survey and the DLRP prefer lesson-planning ideas over fully packaged lesson plans, a small percentage (22 percent in the survey and 38 percent in the DLRP) prefer fully crafted lesson plans and materials.

Phase three

When taken together, the three weeks of classroom-educator workshops enabled the research team to confirm a set of behaviors across the participant groups. If we use the lens of the project goals of identifying, analyzing, and extracting content to become more creative in classroom instructional use, we can summarize the prototype testing findings as follows.

Search and Visualization (Identifying)
Classroom educators commonly:

• Search by entering a general search term, then filtering further if needed. Educators also preferred the gallery view to review their search results. Participants want more intelligence in their searches and results to guide them toward the most valuable resources. This intelligence included auto-complete typing, auto-correct spelling, and similar items for returns that bear few results. (We will also see these suggestions from the student testers in the next study.)
• Use a diversity of locations to find what they need and have little loyalty to one site in particular, although they go to educational sites more frequently than non-educational sites.
• Use the Facebook Share option that was provided, but the most popular method of sharing was emailing the link to themselves or a colleague.

Authentication, Saving, and Storing (Analyzing)
Classroom educators prefer:

• To save resources that they find useful. They will use whatever means available to do it, even if the site does not provide this function.
• The flexibility to organize and annotate resources according to their own schemas.
• Flexibility in the types of viewing methods available: one for whole-class interaction (where site order is emphasized and only one site is viewed at a time), and one for individual interaction (where student selection is emphasized and all sites are easily accessed).
• The ability to allow students to use the site and its tools as much as the classroom educator.
• Content that is aligned with Common Core State Standards.

Instructional Tools (Extracting)
Classroom educators:

• Were excited about the use of “interactives” with the resources found in the Smithsonian collection.
• Appreciated the search functionality of the site but want better visibility of the tools, including prompts and explanations for their use.
• Liked being able to upload resources from other sources to augment their collections and appreciated being given tools that make this easier to accomplish within the site.

Phase four

The final phase of the DLRP was to synthesize the findings into a concrete set of technical requirements geared toward the eventual build and implementation of the prototype tools. A detailed technical development and implementation plan has been drafted and is available on the project wiki (http://smithsonian-digital-learning.wikispaces.com/Teacher+Toolkit+%28Technical+Requirements%29).

One of the additional recommendations of the study pointed towards a need to acknowledge that current technology trends point to users moving away from accessing content in traditional ways, that is via institutional websites. As mobile applications and integrated systems of shared data becoming the norm, developing on a platform that values data portability and a separation of services from traditional Web display will ensure greater flexibility and long-term success for these resources. One of these is the Learning Resource Metadata Initiative discussed in the final study presented in this paper.

5. Piloting Tools to Enable Active and Participatory Learning for Middle School Students: Facilitating Digital Learning with Smithsonian Digital Resources

Background

The DLRP study concluded that “a new examination of the student user as a growing audience for digital museum resources” was needed. In response to this, the DLRP continued to explore the role of digital museum resources in learning. Specifically, SCLDA wanted to focus on middle school students’ ability to become active creators of digital resources personalized for learning. Middle school students were chosen in particular because they are the largest student audience for current SCLDA programming.

SCLDA and the University of Maryland College of Education faculty worked with middle school students during the 2013–2014 academic year to explore development and use of the DLRP prototype (http://scems.navnorth.com/). In essence, the prototype replicates the process educators use in preparing a lesson (i.e., locating resources, reviewing and selecting them, sequencing and editing them, layering them with instructional activities, and sharing them with their students and colleagues). The prototype, in this study, enables students to produce digital projects guided by their own interests. Students can use the prototype, for the first time, to search for and view items in the database of educational resources, the Smithsonian collections, videos, blog posts, podcasts, etc., all via one unified search interface (currently not available from the Smithsonian).

Methodology/sample

The prototype was presented to middle school students in June 2014 in College Park, Maryland, and in October 2014 in Chico, California. Testing occurred in four classrooms with sixth and seventh graders both during and after school with twenty to thirty students in each class. Students worked primarily independently, but occasionally in small groups, on supplied laptops.

Testing was conducted using a predefined set of instructions and tasks. Each testing session included an activity where students were first introduced to the Smithsonian, the research project, and their role as testers. Students were then shown the prototype and walked through how to search, save, and edit collections. Individually, they were then asked to conduct the same search. Finally they were instructed to search, create, and edit a collection based on their own interests. The group was then interviewed for suggestions of improvements.

Data were collected through:

• Observations
• Group interview
• Analysis of students’ collections (saved in the prototype)

The findings are based on the notes of the data collectors and an analysis of the collections created by the participants.

Figure 4: students explore the Smithsonian collections and build a personalized collection using the prototype.

Findings

According to the observational data, participants did not struggle with search or collection creation; in fact, some of the students were creating collections before instructions were provided. The observers noted that students were working collaboratively—they shared their searches and showed each other what they were finding, and some made similar collections to each other.

Though the students were able to easily use the prototype, there were challenges that participants encountered, including:

• Unintelligible descriptions: some of the students found that the descriptions were difficult to understand (the information provided on collection items is often museum cataloging information rather than education information specifically designed for students)
• Spelling limitations: some of the students could not find the items they were looking for because they could not spell the search terms correctly
• Loss of authority: some of the participants thought that the information in their collections was unreliable if they (or anyone else) were able to change the information from the original Smithsonian descriptions (a feature of the prototype)

Participants wanted to see features that are familiar from other search engines/software, such as:

• Auto-correct to assist with spelling
• Predictive searches
• Recommended searches
• Filters for inappropriate material
• Ability to adjust fonts
• Microphone speech for text searching

Participants wanted the ability to personalize their collection by being able to:

• Draw on objects
• Put external images into their collection
• Share collections through social media and email
• Export their collection
• Create unique collages from multiple images

Participants wanted additional content, such as:

• Modern photos
• Specific information about the items (such as whether or not certain animals are endangered or where the animal lives)
• Child-accessible collection descriptions

As students spent more time with the prototype, they found that they wanted to do more with it than the prototype could do (such as delete items, search using predictive searching, etc.). Students expected basic functionality to be available like that found in familiar software (such as Google and social-media platforms).

6. Increasing the Discoverability of Smithsonian Digital Resources: Learning Resource Metadata Initiative (LRMI)

Background

The Learning Resource Metadata Initiative (LRMI) emerged from a partnership between the Association of Educational Publishers and the Creative Commons (funded by the Bill & Melinda Gates Foundation and the William and Flora Hewlett Foundation). Simply put, the goal of the LRMI is to make online educational resources searchable, more transparent, and easier to locate and analyze. This is accomplished through the development of a standard way of tagging (the process of creating metadata) online content. For a complete background on LRMI and the Learning Registry, see Connecting Learners and Museums through Educational Metadata Initiatives, published by the authors (along with James Collins) in the proceedings of the 2014 Museums and the Web Conference (http://mw2014.museumsandtheweb.com/paper/connecting-learners-and-museums-through-educational-metadata-initiatives/).

In 2014, SCLDA began an LRMI and Learning Registry (a technology infrastructure for the distribution of metadata) initiative in collaboration with the National Museum of American History. The goals of the Smithsonian LRMI project are to:

• Develop and publish metadata to fully describe 2,500 existing Smithsonian learning resources
• Evaluate the impact of LRMI metadata on the discoverability, analysis, and use of Smithsonian learning resources
• Build capacity at the Smithsonian for developing metadata for new resources

Methodology/sample

To assess the effects of the project, SCLDA has established an ongoing LRMI Impact Evaluation Study to understand if LRMI metadata enables educators to better discover, analyze, and use Smithsonian resources to create personalized learning resources for their students. To address these questions, fifty-six classroom educator participants (from across the United States and recruited through an external firm) in this study completed the following activities:

Phase one: Online survey

The survey was designed to understand participants’ current attitudes and expectations regarding digital resource discoverability, analysis, and extraction. The survey was based upon one developed by the LRMI Survey Report (Winter Group, 2013).

Phase two: Web-page analysis

To understand participants’ tag or attribute preferences in the analysis of digital resources, participants were asked to view and comment upon four different Web environments, each showing metadata describing the same SCLDA-authored digital resource, World War II on the Home Front: Civic Responsibility (http://www.smithsonianeducation.org/educators/lesson_plans/civic_responsibility/), but each having a slightly different user interface as determined by the inclusion (or absence) of LRMI tags.

Website markup and explanation
The markup and explanation activity collected similar information as the survey, but through qualitative data. Participants were shown screenshots of metadata describing the Smithsonian digital resource as it could be displayed in two search result environments: 1) Google search results, and 2) California Brokers of Expertise website (note that Brokers of Expertise is now known as the Digital Chalkboard; see https://www.mydigitalchalkboard.org/). Participants were asked to identify and place a virtual pin on two aspects of each environment that supported their analysis. Follow-up questions asked the participant to explain their pin placement. Pin placement served as a visual representation of analysis, while participant explanations provided qualitative support.

Website surf and explanation
Participants were asked to click on live links to two environments, each hosting a landing page describing the Smithsonian digital resource, review the information on the page for no more than two minutes, and then answer follow-up questions. As with the website markup activity above, participants were asked to identify two aspects of each landing page that would support their analysis of the digital resource. Participants were asked for deeper explanations about the characteristics of the pages they identified.

Phase three: Web analytics

Further assessment on the quantitative impact of LRMI on discovery, analysis, and extraction will be examined through comparative Web analytics in the next phase of this study in mid-2015.

Figure 5: heat map results from California Brokers of Expertise Web environment.

Findings

Phase one

Responses indicated that only a handful of the participants were familiar with LRMI, having heard about it from “our school librarian” or “at a teachers’ conference.” Not surprisingly, almost all of the participants projected that educator-specific tags would increase their search satisfaction. When presented with a list of educator-specific tags, a large majority of the participants ranked “grade level” and “content/subject area” as the most important tags, followed by “alignment to standards.” The majority also indicated that educator-specific tags would increase the likelihood they would use the materials.

Phase two

Participants analyzed the possible usefulness of a resource by a “trusted” or “familiar” source (e.g., Smithsonian, PBS, and the National WWII Museum) or by content/subject area. Per the Web environment, participants found being able to search by “topics” and “grades” to be particularly useful. Per the sources themselves, participants preferred attributes similar to the Google search above, including possible usefulness of a resource because it is a “trusted” or “familiar” source (e.g., Smithsonian) or by content/subject area.

The vast majority of the participants use Google to search for digital resources and search for online instructional resources several times a week, if not almost every day. Digital resource searches occur both on home computers and on classroom computers. Barely one half of educators consider their searches successful. When asked what makes searches unsuccessful or frustrating, participants cited irrelevant results, lack of educator-specific filters, and time consumption.

The evaluation study has shown that the addition of LRMI tagging to its resources will have a positive impact on the discoverability, analysis, and extraction of Smithsonian digital resources. Specifically, participants strongly assert through both quantitative and qualitative data that grade level, content/subject matter, and connection to standards are among their top priorities in discoverability and analysis (both part of the LRMI specification). These findings support the conclusions of a previous study conducted for the Learning Resource Metadata Initiative (Winter Group, 2013).

7. Conclusions

The studies, when examined together with recent literature, lead us toward a framework for developing best practices in the creation and distribution of digital learning resources. The authors’ intent is that others can identify reusable methodologies and findings in these specific studies that may match situations at their institutions, and therefore be of use.

To reach these conclusions, the authors examined each study and consulted both the literature analyzed in the individual literature reviews referenced above, as well as more recent literature, to identify common conclusions. These are organized below and reflect educational preferences for both searching for learning resources and analyzing their usefulness: 1) educator search preferences, 2) factors that contribute to difficult analysis and resistance to use learning resources, 3) educators’ preferences for learning resources, and 4) educators’ preferences for platform content and functionality. The specific studies mentioned in this paper (as well as relevant literature) are cited following each of the specific conclusions. For brevity, the five studies are indicated using a single number that follows their order as presented in this report and, at times, an indication of which portion of the study lent evidence to the conclusion (e.g., 3-survey):

1. Remedial Evaluation of the Materials Distributed at the Smithsonian Institution’s Annual Teachers’ Night
2. Capturing the Voice of Customer, a Satisfaction Insight Review of SmithsonianEducation.org
3. Digital Learning Resources Project
4. Piloting Tools to Enable Active and Participatory Learning for Middle School Students: Facilitating Digital Learning with Smithsonian Digital Resources
5. Increasing the Discoverability of Smithsonian Digital Resources: Learning Resource Metadata Initiative

Educator Search Preferences

Enabling effective search for educators reflects both standard website-search best practices and also recognizes the need for and providing for descriptive metadata relevant and understandable to this audience, specifically grade level and subject alignment. This audience does not appears to have any specific loyalty to particular providers of content, nor platforms where learning resources are provided. However, our recent analysis has shown that clear indications of the source of the item coming from a trusted provider (such as the Smithsonian) are useful. In general, educators look for the following in searching for learning resources for the classroom:

1. Searching assist via autocomplete and/or spelling assist
   (3-prototyping, 4-prototyping)
2. Search results that allow for both browsing and filtering
   (3-interviews, 3-prototyping; Borgman, 2005; Green et al., 2007; Harley, 2007; Lindquist & Long, 2011; Pattuelli, 2011)
3. Scannable grade-level and subject information (to quickly analyze usefulness)
   (5-survey, 5-analysis; Green et al., 2007; Lindquist & Long, 2011; Sotto, 2012; Winter Group, 2014)
4. Resources from a wide variety of sources (no loyalty to specific resource providers)
   (3-prototyping, 5-analysis; Lindquist & Long, 2011)

Factors that Contribute to Difficult Analysis and Resistance to Use Learning Resources

Educators report that the number of search results encountered, further hampered by both the website user interface design and the information provided with museum-based content, detract from their ability to effectively determine if the resources are appropriate for their pedagogical needs. Specifically, we found these factors to be the most detrimental:

1. Too many search results; too difficult to browse quickly for relevant content
   (1-literature review, 3-prototyping, 3-literature review, 5-survey, 5-analysis; Lindquist & Long, 2011; Purcell et al., 2013; Sotto, 2012)
2. Distracting user interfaces
   (1-literature review; Eamon, 2006; Harley, 2007; Johnson, 2008; Sotto, 2012)
3. Unfamiliar terminology and/or lack of contextual information
   (1-literature review, 4-prototyping; Baker, 2009; Lee & Clarke, 2003; Lindquist & Long, 2011; Sotto, 2012; Swan & Locascio, 2008; Goldenberg & Tally, 2005)

Educators’ Preferences for Learning Resources

Educators express some consistent needs when explaining the types of digitally accessible resources they look for from museums. Many express the desire that the focus of the resources be inter- or multidisciplinary. This may stem from preconceived notions of the types of knowledge contained within museums, but the studies did not offer enough information to firmly conclude this. Secondly, a strong preference was noticed for content that connects with both the interests of the students and accepted teaching standards, such as Common Core State Standards. In terms of format, resources should be adaptable, in how they are used either pedagogically (for various learning or reading styles/abilities) or functionally (flexible to various presentation, sharing, and export formats). Specifically educators are looking for resources that meet these criteria:

1. Interdisciplinary and/or multidisciplinary
   (1-literature review, 1-reviews, 3-interviews; Baker, 2009)
2. Connection to students’ interests
   (1-reviews, 3-interviews; Lindquist & Long, 2011; Purcell et al., 2013; Sotto, 2012)
3. Alignment to teaching standards and/or relationship to big concepts
   (1-literature review, 1-reviews, 3-interviews, 3-prototyping, 5-survey, 5-analysis; Lindquist & Long, 2011; Sotto 2012)
4. Adaptable
   (1-reviews, 1-interviews, 3-literature review, 3-interviews, 3-prototyping, 4-prototyping; Baker, 2009; Green et al., 2007; Leftwich & Bazley, 2009; Marty et al., 2011; Sotto 2012)
5. Downloadable format
   (2-survey; Green et al., 2007; Sotto, 2012)

Educators’ Preferences for Platform Content and Functionality

When considering the platforms on which educators access educational resources for their classrooms, they seem to prefer ones that consolidate content from a variety of sources (not just the platform provider, for example). They also look for tools to make the content more useful, for both their students (such as discussion areas, annotation tools, etc.) and themselves. They prefer a variety of sharing options (although the SCLDA research indicates that email continues to be the preferred method of sharing resource with other colleagues) and ways to save and sort content for later use. Specifically, the research and literature indicates these preferences:

1. Content available from more than one producer/supplier
   (3-interviews, 3-prototyping, 4-prototyping; Baker, 2009; Green et al., 2007; Marty et al., 2011)
2. Tools available within the platform for student interaction with the resources
   (3-interviews, 3-prototyping, 4-prototyping; Baker, 2009; Sotto, 2012)
3. Variety of sharing options
   (3-prototyping, 4-prototyping; Marty et al., 2011)
4. Ability to save and structure resources within the platform for later review/use
   (3-prototyping; Baker, 2009; Lindquist & Long, 2011)

Acknowledgements

The authors wish to thank Stephanie Norby, Michelle Smith, and Pino Monaco of the Smithsonian Center for Learning and Digital Access for their support and assistance in producing this paper and in the development of the projects it summarizes. The authors also express their gratitude to Claudine Brown, the Smithsonian assistant secretary for Education and Access, as well as Karen Garrett, the Youth Access Grant program, the Pearson Foundation, Brokers of Expertise of the California Department of Education, the Council of Chief State School Officers, and inBloom for financial and in-kind support.

This paper summarizes, and at times excerpts from, five original research reports conducted or commissioned by the Smithsonian Center for Learning and Digital Access from 2010 to 2014. The following individuals, in addition to the authors, made significant contributions to the research design, data collection, analysis, and writing of the original reports and should be acknowledged and credited:

Remedial Evaluation of the Materials Distributed at the Smithsonian Institution’s Annual Teachers’ Night
For the Smithsonian Center for Learning and Digital Access: Jennie Ito, Lesley Langa, and Pino Monaco.

Capturing the Voice of Customer, a Satisfaction Insight Review of SmithsonianEducation.org
For Foresee Results: Sara Dorn-Havlik.

Digital Learning Resources Project
For the Smithsonian Center for Learning and Digital Access: Pino Monaco and Michelle Smith; for Navigation North Learning Solutions, LLC: Brian Ausland, Joe Hobson, Daniel Kreiger, and MaryRose Lovgren; for Cross & Joftus, LLC: Christopher Cross, John Ittelson, Dilan Maherdran, Virginia McMunn, Griffith Montgomery, Clark Quinn, Jillian Ryan, and Virginia Adams Simon.

Increasing the Discoverability of Smithsonian Digital Resources: Learning Resource Metadata Initiative (LRMI)
For the Smithsonian Center for Learning and Digital Access: James Collins; for Oberg Research, LLC: Caren Oberg and Heidi Kartchner.

Piloting Tools to Enable Active and Participatory Learning for Middle School Students: Facilitating Digital Learning with Smithsonian Digital Resources
For the Smithsonian Center for Learning and Digital Access: Pino Monaco; for Oberg Research, LLC: Caren Oberg and Heidi Kartchner; for Navigation North Learning Solutions, LLC: Brian Ausland, Jodi Ausland, and MaryRose Lovgren.

References

Baker, R. (2009). “Museums & Education Digital Content Exchange Final Report.” The Learning Federation.

Borgman, C.L., L.J. Smart, K.A. Millwood, J.R. Finley, L. Champeny, A.J. Gilliland, & G.H. Leazer. (2005). “Comparing faculty information seeking in teaching and research: Implications for the design of digital libraries.” Journal of the American Society for Information Science and Technology 56(6): 636–657.

Eamon, M. (2006). “A ‘Genuine Relationship with the Actual’: New Perspectives on Primary Sources, History and the Internet in the Classroom.” The History Teacher 39(3): 297–314.

Falk, J.H., & L. Dierking. (2000). Learning from Museums: Visitor Experiences and the Making of Meaning. Walnut Creek: Altamira, 229.

Goldenberg, L.B., & B. Tally. (2005). “Fostering historical thinking with digitized primary sources.” Journal of Research on Technology in Education 38(1+).

Green, D., R. Lancefield, D. Harley, S. Hsi, G. Waibel, S. Chun, & L. White. (2007). “Museum Images On-line: Meeting the Needs of Educators.” In J. Trant & D. Bearman (eds.). Museums and the Web 2007: Proceedings, Toronto: Archives & Museum Informatics. Published March 1, 2007.

Harley, D. (2007). “Why study users? An environmental scan of use and users of digital resources in humanities and social sciences undergraduate education.” First Monday 12(1).

Johnson, A. (2008). “Users, use and context: supporting interaction between users and digital archives.” In Craven, L. (ed.). What are archives?: cultural and theoretical perspectives: a reader. Burlington, VT; Aldershot, England: Ashgate.

Lee, J., & G. Clarke. (2003). “High school social studies students’ uses of online historical documents related to the Cuban Missile Crisis.” Society for Information Technology & Teacher Education International Conference 2003. Albuquerque, New Mexico: Association for the Advancement of Computing in Education (AACE).

Leftwich, M., & M. Bazley. (2009). “Pedagogy and Design: Understanding Teacher Use of On-line Museum Resources.” In J. Trant & D. Bearman (eds.). Museums and the Web 2009: Proceedings. Toronto: Archives & Museum Informatics. Published March 31, 2009.

Lindquist, T., & H. Long. (2011). “How can educational technology facilitate student engagement with online primary sources?” Library Hi Tech 29(2): 224–241.

Marty, P., S. Sayre, & S. Filippini Fantoni. (2011). “Personal Digital Collections: Involving Users in the Co-Creation of Digital Cultural Heritage.” In S. Georgios, K. Dimitrios, & L. Fotis (eds.). Handbook of Research on Technologies and Cultural Heritage: Applications and Environments. Hershey, PA: IGI Global, 285–304.

Milligan, D., M. Wadman, & J. Collins. (2014). “Connecting Learners and Museums through Educational Metadata Initiatives.” In N. Proctor & R. Cherry (eds.). Museums and the Web 2014, Silver Spring, MD: Museums and the Web. Published January 31, 2014.

Pattuelli, M.C. (2011). “Modeling a domain ontology for cultural heritage resources: A user-centered approach.” Journal of the American Society for Information Science and Technology 62(2): 314–342.

Purcell, K., A. Heaps, J. Buchanan, & L. Friedrich. (2013). “How Teachers Are Using Technology at Home and in Their Classrooms.” Pew Research Center’s Internet & American Life Project.

Silverman, L. (2011). Social Work of Museums. Online video. Bunker Ljubljana. Consulted October 14, 2014. Available http://vimeo.com/19213313

Sotto, T. (2012). Summary Report of Survey Findings, Teacher Programs: Assessing the Getty Museum’s Online Resources for K–12 Teachers. J. Paul Getty Trust.

Swan, K., & D. Locascio. (2008). “Evaluating alignment of technology and primary source use within a history classroom.” Contemporary Issues in Technology and Teacher Education 8(2): 175–186.

Weil, S. (1999). “From Being about Something to Being for Somebody: The Ongoing Transformation of the American Museum.” Daedalus 128(3), America’s Museums (Summer, 1999), 229–258.

Winter Group. (2013). LRMI Survey report August 2013 update, Ease and discoverability: Educators and publishers on the search for educational content. Association of Educational Publishers.

Winter Group. (2014). LRMI Survey Report July 2014 Update: Momentum. Opportunity. Outreach. Association of Educational Publishers.

For the complete studies summarized in this paper, including background, methodologies, and full references, see the following:

Remedial Evaluation of the Materials Distributed at the Smithsonian Institution’s Annual Teachers’ Night
http://smithsonian-digital-learning.wikispaces.com/file/view/Remedial%20Evaluation%20November%202010.doc/323385242/Remedial%20Evaluation%20November%202010.doc

Capturing the Voice of Customer, a Satisfaction Insight Review of SmithsonianEducation.org
For more information on the study, please contact the authors and see http://learninglab.si.edu/news/2014/10/who-is-our-audience-smithsonianeducation-org/

Digital Learning Resources Project
Literature Review: http://smithsonian-digital-learning.wikispaces.com/Teacher+Toolkit+%28Literature+Review%29

Environmental Scan: http://smithsonian-digital-learning.wikispaces.com/Teacher+Toolkit+%28Environmental+Scan%29

Teacher Prototyping Report: http://smithsonian-digital-learning.wikispaces.com/Teacher+Toolkit+%28Research+Findings%29

Technical Requirements Report: http://smithsonian-digital-learning.wikispaces.com/Teacher+Toolkit+%28Technical+Requirements%29

Final Report and Conclusions: http://smithsonian-digital-learning.wikispaces.com/Teacher+Toolkit+%28Final+Report%29

Piloting Tools to Enable Active and Participatory Learning for Middle School Students: Facilitating Digital Learning with Smithsonian Digital Resources
http://smithsonian-digital-learning.wikispaces.com/file/view/DLRP_PilotingTools_FinalReport.pdf/536630252/DLRP_PilotingTools_FinalReport.pdf

Increasing the Discoverability of Smithsonian Digital Resources: Learning Resource Metadata Initiative (LRMI)
For more information on the ongoing evaluation study, please contact the authors.