Preparing to Observe Integrated STEM Education in K-12 Classrooms

Introduction
As science, technology, engineering, and mathematics education – often collectively referred to as STEM education – moves to an integrated approach in primary and secondary schooling, there is a need to better understand what STEM education is in order to implement it in practice. A review of the literature reveals a wide variety of approaches to STEM education that include: STEM as a replacement term for science and mathematics (Breiner, Harkness, Johnson & Koehler, 2012; Sanders, 2009), STEM as a pedagogical shift toward an integrated approach (Breiner et al., 2012; English, 2016; Honey, Pearson, & Schweingruber, 2014; Kelley & Knowles, 2016), curriculum changes that reflect the work of STEM professionals (Breiner et al., 2012; Labov, Reid, & Yamamoto, 2010; Sanders, 2009), and curricula that emphasize an engineering design challenge (Bryan, Moore, Johnson, & Roehrig, 2015). Despite these variations in definitions, there are some common elements, including: the inclusion of an engaging, real-world context (e.g., Breiner et al., 2012; Brown, Brown, Reardon, & Merrill, 2011); explicit connections between science, technology, engineering, and mathematics and modelling them as they would be used in STEM careers (e.g., English, 2016; Herschbach, 2011; Honey et al., 2014; Kelley & Knowles, 2016); the intentional development of 21st century competencies (e.g., Bryan et al., 2015; Honey et al., 2014); and an emphasis of student-centred pedagogies (e.g., Bryan et al., 2015; Breiner, et al., 2012; Labov et al., 2010; Sanders, 2009). In short, integrated STEM education is a complex combination of content and pedagogy, making it difficult to capture and define.
While defining STEM education has proved challenging, assessing the quality of integrated STEM instruction in classrooms proves even more elusive. This is most likely due to the lack of a protocol designed specifically for such teaching. Protocols that measure inquiry-based teaching, such as the Reformed Teaching Observation Protocol (Sawada et al., 2002), have been used in lieu of a STEM-specific protocol, but this choice comes with challenges and limitations (Ellis, Dare, Roehrig, & Ring-Whalen, in review). Current observation protocols tend to focus on one discipline, which is problematic when STEM instruction addresses multiple disciplines. The work presented here is part of a project whose end goal is to disseminate an integrated STEM observation protocol for use in K-12 science and engineering classrooms.

Workshop Focus
As integrated STEM education becomes increasingly prevalent in primary and secondary education, there is a need for an observation protocol designed specifically to understand the implementation of integrated STEM in K-12 classrooms. Despite the existence of various definitions STEM education, as noted above, the literature related to STEM education suggests overlaps in these definitions which can be used to develop such an instrument. With the support of a four-year NSF DRK-12 grant, we have developed an observation protocol for integrated STEM instruction that can be used by both educational researchers and educators at various levels; at the end of the project, this protocol will be a valid and reliable instrument for research purposes. This workshop is designed to inform attendees from the science teacher education community about the development of our STEM Observation Protocol (STEM-OP) and provide them with an opportunity to learn to use the protocol by watching and scoring classroom video of integrated STEM lessons. Additionally, attendees will have opportunities to provide the presenters with feedback on the instrument.

Relevance and Benefits of Attending
Science teaching and learning in K-12 education is changing and the work described above is helping to address the gaps that exist. The STEM-OP has been designed to be used in science and engineering classrooms that integrate STEM disciplines in lessons and units, making it a viable instrument for use in states that have adopted the Next Generation Science Standards (NGSS Lead States, 2013) or other standards that incorporate engineering. The instrument was designed with different stakeholders in mind and for use in a variety of educational contexts to increase the quality of K-12 STEM education and education research. To this end, the STEM-OP was designed not only for summative and evaluative purposes, but also for formative and reflective purposes. For instance, educational researchers may benefit from having an instrument available for them to carry out research and continue to improve science and STEM education in a variety of ways armed with an instrument designed to assess integrated instruction. Science teacher educators may benefit from using the instrument as a way to teach their pre-service teachers about STEM education and as a tool to use in professional development to guide in-service teachers in STEM education. District science coaches may also use this tool to engage in reflective practice and cognitive coaching with science teachers. Because of the variety of uses that this instrument is targeting, this workshop will be of interest to ASTE members working in a variety of contexts. In short, this workshop will be of interest to ASTE members who are interested in learning more about what integrated STEM education is and how to observe it in K-12 classrooms. This workshop would be suitable to methods instructors, field experience supervisors, education researchers, school administrators, and more.

Presenters’ Expertise
All of the workshop presenters are principal investigators on the collaborative NSF-funded grant that supports this work and each has extensive experience in the area of STEM integration. Dr. Emily Dare is an Assistant Professor of Science Education at Florida International University. Dr. Joshua Ellis is an Assistant Professor of Science Education at Florida International University. Dr. Elizabeth Ring-Whalen is an Assistant Professor of Education, Coordinator for the EcoSTARS and Elementary STEM Certificate programs, and Director of the National Center for STEM Elementary Education at St. Catherine University. Dr. Gillian Roehrig is a Professor of STEM Education at the University of Minnesota – Twin Cities. Collectively, we have hosted numerous grant-support professional development workshops focused on supporting in-service K-12 science teachers in learning and implementing integrated STEM practices and have also taught pre-service science methods courses that address integrated STEM. As a result, we have shared curricula not only with teachers we have worked with, but with other educators through practitioner-oriented journals (e.g., Dare, Ellis, & Tyrrell, 2018; Ellis, Dare, Voigt, & Roehrig, 2015). Our research has also covered a variety of areas related to integrated STEM education including: teacher conceptions of STEM education (e.g., Ring, Dare, Crotty, & Roehrig, 2017); the development of K-12 learning environments (e.g., Roehrig, Wang, Moore, & Park, 2012); teacher experiences in implementing STEM curricular units (e.g, Dare, Ellis, & Roehrig, 2018); the intersection of teacher beliefs and practices of STEM education (e.g., Ring-Whalen, Dare, Roehrig, Titu, & Crotty, 2018); the intersection of teacher beliefs, practices, and student achievement in STEM education (e.g., Crotty, Guzey, Roehrig, Glancy, Ring-Whalen, & Moore, 2017); and gender-equity in STEM education (e.g., Dare, 2015).

Description of Workshop Activities
The following sections outline the agenda for this 2-hour workshop. The actual length of time proposed for each activity may vary according to how many individuals attend the session. The aim of this workshop is to not only inform those in the science education community about the development of our observation protocol, but to provide workshop attendees an opportunity to use and discuss the protocol. Further, by attending this workshop, attendees will:
● Identify their own definition of STEM education
● Understand that defining STEM is challenging
● Evaluate their own thinking of STEM education
● Critically examine the items of the STEM-OP
● Reflect on the use of the STEM-OP in their own work
The workshop activities are designed to first engage attendees in their own thinking of what STEM education is, identify common features across different models of STEM education, examine the STEM-OP items, use the STEM-OP while watching a portion of classroom video, and share implications that this instrument might have on their practice. Each activity allows us to formatively assess attendees’ progress towards the learning objectives and attendees’ summative learning will be assessed through an online survey at the end of the session.

What is STEM education? (approx. 30 minutes). The session will begin by problematizing STEM education, beginning by asking attendees to share their conceptual model of STEM education as a visual representation. In anticipating that there will be various models shared by attendees, attendees will then share their models with others in the room and identify common features among the models. Attendees will then work in small groups and use large Post-it papers to list what they would want to see in a classroom that claims to be implementing integrated STEM instruction. After sharing these lists with the whole group, the presenters will then discuss the challenges in the development of this instrument, which included a full literature review, discussion and consensus among the project team related to protocol items and levels, piloting of the instrument with classroom observation videos, and factor analysis.

Sharing and using the protocol (approx. 60 minutes). The STEM-OP will be shared in paper format with attendees for their review and use in the workshop. In its current form, the STEM-OP is 13-items and uses a 5-point (0-4) Likert-scale. Each item includes a brief description of the item, a set of guidelines for scoring the item, and detailed descriptions for each level of the scale. Assuming a group of 30 attendees, four groups of 7-8 attendees will be assigned 3-5 items to become “experts” on. Each group will be joined by one of the presenters and, through discussion, the groups will work together to understand how to score the items they have been assigned. In order to further attendees’ understanding of the items and how to score them, several example classroom scenarios will be shared as prompts for which the group needs to score. Once attendees have familiarized themselves with their assigned items and the their associated levels, a segment of classroom video (20-30 minutes) will be played. During this time, attendees will focus on their assigned items and use the STEM-OP to score the video segment.

Reflection and discussion (approx. 30 minutes). After using the STEM-OP to score classroom video, attendees will share how they might use this protocol in their own research and teaching to improve understanding and implementation of integrated STEM education. Because the protocol was designed for a variety of purposes in mind, including as an aid to assist teachers and schools in better understanding what integrated STEM education is, it is important to address the protocol as non-evaluative.

Continued Learning
At the end of the workshop, attendees will be asked to complete a short online survey via Google Forms to provide feedback related to the protocol and its potential use in their own work. Attendees will also be provided with contact information of the presented so that they may follow up with the protocol as it continues to evolve within the four-year project.