A Network Analytic Approach to Gaze Coordination during a Collaborative Task
S. Andrist, A. R. Ruis, & D. Williamson Shaffer
Computers in Human Behavior 89 (2018): 339-348
A critical component of collaborative learning is the establishment of intersubjectivity, or the construction of mutual understanding. Collaborators coordinate their understanding with one another across various modes of communication, including speech, gesture, posture, and gaze. Given the dynamic, interdependent, and complex nature of coordination, this study sought to develop and test a method for constructing detailed and nuanced models of coordinated referential gaze patterns. In the study, 13 dyads participated in a simple collaborative task. We used dual mobile eye tracking to record each participant’s gaze behavior, and we used epistemic network analysis (ENA) to model the gazes of both conversational participants synchronously. In the model, the nodes in the network represent gaze targets for each participant, and the connections between nodes indicate the likelihood of gaze coordination. Our analyses indicate: (a) properties and patterns of how gaze coordination unfolds throughout an interaction sequence; and (b) differences in gaze coordination patterns for interaction sequences that lead to breakdowns and repairs. In addition to contributing to the growing body of knowledge on the coordination of gaze behaviors in collaborative activities, this work suggests that ENA enables more effective modeling of gaze coordination.
Modeling Processes of Enculturation in Team Training
A. R. Ruis, A. J. Hampton, B. S. Goldberg, & D. Williamson Shaffer
Design Recommendations for Intelligent Tutoring Systems: Volume 6 – Team Tutoring, eds. R. Sottilare, A. C. Graesser, X. Hu, & A. M. Sinatra (Orlando: U.S. Army Research Laboratory, 2018), 45–51.
Both military and civilian work is increasingly organized around small, dynamic teams rather than large bureaucratic frameworks. Such teams combine individuals with highly specific skill sets and extensive training to solve complex, non-standard problems, often under extreme pressure. Importantly, these teams are not typically composed of interchangeable members but are formed and trained as expert teams to function semi-autonomously. To accomplish mission objectives in the face of complex challenges, the United States military needs to develop teams that consistently exhibit high levels of taskwork and teamwork skills. Implementing principles of team training and maintaining team efficacy are critical in the armed forces, where teams are often widely dispersed and consequences for underperformance can be severe, though many civilian teams—such as hospital trauma teams or flight crews—face similar challenges. Establishing and maintaining high levels of team performance in these contexts requires the creation of practical and effective team development interventions, including team training, as well as systems for ongoing assessment of team function. We argue that one critical component of training, monitoring, and maintaining high-functioning teams is the ability to model team performance.
Using Epistemic Network Analysis to Identify Targets for Educational Interventions in Trauma Team Communication
S. A. Sullivan, C. Warner-Hillard, B. R. Eagan, R. Thompson, A. R. Ruis, K. Haines, C. M. Pugh, D. Williamson Shaffer, & H. S. Jung
Surgery 163, no. 4 (2018): 938–943.
Epistemic Network Analysis (ENA) is a technique for modeling and comparing the structure of connections between elements in coded data. We hypothesized that connections among team discourse elements as modeled by ENA would predict the quality of team performance in trauma simulation.
The Modified Non-technical Skills Scale for Trauma (T-NOTECHS) was used to score a simulation-based trauma team resuscitation. Sixteen teams of 5 trainees participated. Dialogue was coded using Verbal Response Modes (VRM), a speech classification system. ENA was used to model the connections between VRM codes. ENA models of teams with lesser T-NOTECHS scores (n = 9, mean = 16.98, standard deviation [SD] = 1.45) were compared with models of teams with greater T-NOTECHS scores (n = 7, mean = 21.02, SD = 1.09).
Teams had different patterns of connections among VRM speech form codes with regard to connections among questions and edifications (meanHIGH = 0.115, meanLOW = −0.089; t = 2.21; P = .046, Cohen d = 1.021). Greater-scoring groups had stronger connections between stating information and providing acknowledgments, confirmation, or advising. Lesser-scoring groups had a stronger connection between asking questions and stating information. Discourse data suggest that this pattern reflected increased uncertainty. Lesser-scoring groups also had stronger connections from edifications to disclosures (revealing thoughts, feelings, and intentions) and interpretations (explaining, judging, and evaluating the behavior of others).
ENA is a novel and valid method to assess communication among trauma teams. Differences in communication among higher- and lower-performing teams appear to result from the ways teams use questions. ENA allowed us to identify targets for improvement related to the use of questions and stating information by team members.
In Search of Conversational Grain Size: Modeling Semantic Structure using Moving Stanza Windows
A. L. Siebert-Evenstone, G. Arastoopour, W. Collier, Z. Swiecki, A. R. Ruis, & D. Williamson Shaffer
Journal of Learning Analytics 4, no. 3 (2017): 123–139.
Analyses of learning based on student discourse need to account not only for the content of the utterances but also for the ways in which students make connections across turns of talk. This requires segmentation of discourse data to define when connections are likely to be meaningful. In this paper, we present an approach to segmenting data for the purposes of modelling connections in discourse using epistemic network analysis. Specifically, we use epistemic network analysis to model connections in student discourse using a temporal segmentation method adapted from recent work in the learning sciences. We compare the results of this study to a purely conversation-based segmentation method to examine the affordances of temporal segmentation for modelling connections in discourse.
Eating to Learn, Learning to Eat: The Origins of School Lunch in the United States
A. R. Ruis
New Brunswick: Rutgers University Press, 2017.
This book explores the origins of the National School Lunch Program, the most extensive and longest-running children’s nutrition program in U.S. history. In doing so, it reveals how and why school meals came to have the form that is so familiar to most Americans. While the National School Lunch Program was in part a response to specific concerns about children’s access to sufficient and sufficiently nourishing food during the Great Depression and World War II, many of the issues that shaped school meal policies and practices arose decades earlier. By exploring the origins of school meal initiatives, this book endeavors to explain why it was (and to some extent, has continued to be) so difficult to establish meal programs that satisfy the often competing interests of children, parents, schools, health authorities, politicians, and the food industry. This is accomplished through careful studies of several regions and detailed analysis of the policies and politics that governed the development of school meal programs, first at the local level, and then at the federal level. In doing so, the book shows how the early history of school meal program development helps us understand contemporary debates over changes to school lunch policies, including those inaugurated by the passage of the Healthy, Hunger-Free Kids Act in 2010.
Automating the Detection of Reflection-on-Action
J. Saucerman, A. R. Ruis, & D. Williamson Shaffer
Journal of Learning Analytics 4, no. 2 (2017): 207–234.
Learning to solve complex problems—problems whose solutions require the application of more than basic facts and skills—is critical to meaningful participation in the economic, social, and cultural life of the digital age. In this paper, we use a theoretical understanding of how professionals use reflection-in-action to solve complex problems to investigate how students learn this critical 21st-century skill and how we can develop and automate learning analytic techniques to assess that learning. We present a preliminary study examining the automated detection of reflective discourse during collaborative, complex problem solving. We analyze student reflection-on-action in a virtual learning environment, focusing on both reflection in individual discourse and collaborative reflection among students. Our results suggest that it is possible to detect student reflection on complex problems in virtual learning environments, but that different models may be appropriate depending on students’ prior domain experience.
Can We Rely on Reliability? Testing the Assumptions of Inter-Rater Reliability
B. R. Eagan, B. Rogers, R. Serlin, A. R. Ruis, G. Arastoopour, & D. Williamson Shaffer
Making a Difference: Prioritizing Equity and Access in CSCL: 12th International Conference on Computer-Supported Collaborative Learning, eds. B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (2017), II:529–532.
Researchers use Inter-Rater Reliability (IRR) to measure whether two processes—people and/or machines—identify the same properties in data. There are many IRR measures, but regardless of the measure used, there is a common method for estimating IRR. To assess the validity of this common method, we conducted Monte Carlo simulation studies examining the most widely used measure of IRR: Cohen’s kappa. Our results show that the method commonly used by researchers to assess IRR produces unacceptable Type I error rates.
Epistemic Network Analysis: A Worked Example of Theory-Based Learning Analytics
D. Williamson Shaffer & A. R. Ruis
Handbook of Learning Analytics, eds. C. Lang, G. Siemens, A. Wise, & D. Gašević (Society for Learning Analytics Research, 2017), 175–187.
In this chapter, we provide a worked example of a theory-based approach to learning analytics in the context of an educational game. We do this not to provide an ideal solution for others to emulate, but rather to explore the affordances of a theory-based—rather than data-driven—approach. We do so by presenting (1) epistemic frame theory as an approach to the conceptualization of learning; (2) data from an epistemic game, an approach to educational game design based on epistemic frame theory; and (3) epistemic network analysis (ENA), a technique for analyzing discourse and other data for evidence of complex thinking based on the same theory. We describe ENA through a specific analytic result, but our aim is to explore how this result exemplifies what we consider a key “best practice” in the field of learning analytics.
Annals and Analytics: The Practice of History in the Age of Big Data
A. R. Ruis & D. Williamson Shaffer
Medical History 61, no. 2 (2017): 336-339.
The practice of historical research stands to benefit considerably from, and in a growing number of cases requires, a mixed-methods approach that combines the qualitative and the quantitative and incorporates the analytic strengths of human interpretation and computational processing. In this brief reflection, we explore one approach to mixed-methods history using network analysis: various statistical techniques with which the structure of connections among entities—people, places, concepts, and so forth—can be modeled.
Procedural Simulations and Reflective Practice: Meeting the Need
S. A. Sullivan, A. R. Ruis, & C. M. Pugh
Journal of Laparoendoscopic and Advanced Surgical Techniques 27, no. 5 (2017): 455-458.
The medical profession continues to learn about the ways in which simulations can help to improve education and training. In addition, simulation is constantly evaluated regarding the many ways it can help to improve the abilities of surgeons throughout their course of training and practice. The goal of this study is to review the importance of reflection on clinical practice in situating surgeons as lifelong learners and explore the role that simulation can play in that process. As surgical trainees work to acquire basic necessary skills, simulations are often used to help new learners with such things as knowledge of procedural steps and development of psychomotor techniques. However, the following is an important question: can simulated procedures still play a role for more advanced learners to continue their professional development beyond the basics?
A Tutorial on Epistemic Network Analysis: Analyzing the Structure of Connections in Cognitive, Social, and Interaction Data
D. Williamson Shaffer, W. Collier, & A. R. Ruis
Journal of Learning Analytics 3, no. 3 (2016): 9-45.
This paper provides a tutorial on epistemic network analysis (ENA), a novel method for identifying and quantifying connections among elements in coded data and representing them in dynamic network models. Such models illustrate the structure of connections and measure the strength of association among elements in a network, and they quantify changes in the composition and strength of connections over time. Importantly, ENA enables comparison of networks both directly and via summary statistics, so the method can be used to explore a wide range of qualitative and quantitative research questions in situations where patterns of association in data are hypothesized to be meaningful. While ENA was originally developed to model cognitive networks—the patterns of association between knowledge, skills, values, habits of mind, and other elements that characterize complex thinking—ENA is a robust method that can be used to model patterns of association in any system characterized by a complex network of dynamic relationships among a relatively small, fixed set of elements.
Teaching Health Care Workers to Adopt a Systems Perspective for Improved Control and Prevention of Health Care-Associated Infections
A. R. Ruis, D. Williamson Shaffer, D. K. Shirley, & N. Safdar
American Journal of Infection Control 44, no. 11 (2016): 1360–1364.
- Education and training of health care workers is fundamental to infection prevention.
- A systems approach is needed for optimal health care–associated infection prevention interventions.
- A cognitive simulation, allowing the health care workers to adopt a multidimensional perspective, should be tested for efficacy in health care settings.
Teaching and Assessing Engineering Design Thinking with Virtual Internships and Epistemic Network Analysis
G. Arastoopour, D. Williamson Shaffer, Z. Swiecki, A. R. Ruis, & N. C. Chesler
International Journal of Engineering Education 32, no. 3B (2016): 1492-1501.
One of the goals of engineering education is providing students with authentic and meaningful design experiences as well as assessing the development of design thinking. In this paper, we review virtual internships, online simulations of 21st-century engineering design practice, as one method for providing students with authentic experiences. To assess the development of design thinking in virtual internships, we used epistemic network analysis (ENA), a tool for measuring complex thinking as it develops over time. We provide an example of how ENA can be used to measure quantitatively student teams’ qualitative discourse in a virtual internship program in order to assess performance in engineering. The combination of virtual internships and ENA provides opportunities for students to engage in authentic engineering design, receive concurrent feedback on their design thinking, and develop the identity, values, and ways of thinking of 21st-century professional engineers.
Should Trainees be Involved in the Direct Care of Patients with Ebola Virus Disease?
N. Safdar, A. R. Ruis, D. K. Shirley, & N. Fost
AMEE MedEdPublish 5, no. 1 (2016).
With the world in the throes of the largest outbreak of Ebola Virus Disease (EVD) in history, many institutions in the United States have developed institutional policies and procedures for the screening, diagnosis and management of patients with EVD. Healthcare workers caring for these seriously ill patients are at high risk of contracting EVD through contact with blood and body fluids. With the exception of experimental therapeutics, treatment is largely supportive, requiring healthcare workers to have regular close contact with patients. Given the proportion of healthcare workers who have contracted EVD from patients and the high mortality rate, a central consideration has been whether healthcare trainees should be exempted from the direct care of patients with EVD. Educational needs and ethical issues are discussed. We provide arguments for and against this approach, and present a multifaceted framework that may be useful to institutions grappling with this issue.
Local versus Global Connection Making in Discourse
W. Collier, A. R. Ruis, & D. Williamson Shaffer
Transforming Learning, Empowering Learners: The International Conference of the Learning Sciences (ICLS) 2016, eds. C.-K. Looi, J. Polman, U. Cress, & P. Reimann (2016), I:426–433.
This paper examines techniques for modeling relationships among domain concepts and practices in discourse to assess learning in a CSCL environment. We compare two approaches: a traditional psychometric approach, which models the global correlation structure of student discourse markers across the learning intervention, and a model that accounts for the local correlation structure of discourse markers within activities. We investigate whether: (a) analysis of local correlation structure can identify significant differences between novices and relative experts; (b) these differences reflect meaningful differences the discourse; and (c) analysis of global correlation structure can identify significant differences between novices and relative experts. We assess whether an approach that models local relationships among concepts in a domain provides useful information beyond what might be extracted from a more traditional modeling approach. Our results indicate that techniques that account for local correlation structure can identify patterns in discourse not reflected in global correlation structure.
Technology and the New Professionalization of Teaching
D. Williamson Shaffer, P. Nash, & A. R. Ruis
Teachers College Record 117, no. 12 (2015): 1-30.
By 2009, 99% of U.S. classrooms had access to computers, with an average ratio of 1.7 students per computer, and 40% of teachers report using computers often in their classrooms. However, while K-12 schools are investing more heavily in digital technologies, only a small fraction of this investment is going to instructional software (7%) and digital content (5%). Education policy leaders have called for increased investment in and use of digital learning technologies in K-12 education, which has significant professional implications for the 40% of teachers who use computers often and, perhaps more importantly, for the 60% who do not. This article explores for a broad audience the changing landscape of education in the digital age, the changing roles of teachers in a technology-rich education system, and the skills, knowledge, values, and ways of thinking that teachers will need to have to support students’ social, emotional, and intellectual development in a digital learning environment. This analytic essay reviews and synthesizes research on learning in a digital environment, providing a theoretical framework for understanding the changing landscape of learning in technology-rich environments and the consequent changes in teacher preparation that this may entail. We explore the influence of educational technologies on teaching and teacher preparation by looking at three kinds of learning technology: digital workbooks that help students learn basic skills through routine practice; digital texts, such as ebooks, virtual museums, and learning games, that provide students with mediated experiences; and digital internships that simulate real-world practices, helping students learn how to solve problems in the ways that workers, scholars, and artists in the real world do. We examine the extent to which these technologies can assume different aspects of teachers’ traditional functions of assessment, tutoring, and explication. We argue that increased use of these and other digital learning technologies could allow teachers to provide more nuanced curricula based on their students’ individual needs. In particular, teachers will likely assume a new role, that of a coordinator who provides guidance through and facilitation of the learning process in individual students’ social, intellectual, and emotional contexts. We suggest this may require changes to teacher preparation and in-service professional development to help both new and experienced teachers succeed in an ever-changing digital learning environment, as well as new methods of evaluating teacher performance that account for more than student achievement on standardized tests.
“The Penny Lunch Has Spread Faster Than the Measles”: Children’s Health and the Debate over School Lunches in New York City, 1908-1930
A. R. Ruis
History of Education Quarterly 55, no. 2 (2015): 190-217.
A few days before Thanksgiving in 1908, the home economist Mabel Hyde Kittredge initiated a school lunch program at an elementary school in Hell’s Kitchen, serving soup and bread to hungry children in the infamous Manhattan neighborhood. The following year, she founded the School Lunch Committee (SLC), a voluntary organization composed of home economists, educators, physicians, and philanthropists dedicated to improving the nutritional health and educational prospects of schoolchildren. By 1915, the SLC was serving 80,000 free or low-price lunches a year to children at nearly a quarter of the elementary schools in Manhattan and the Bronx. Sparse but compelling evidence indicated that the program had reduced malnourishment among the children who partook, and teachers and principals at participating schools reported reductions in behavioral problems, dyspepsia, inattentiveness, and lethargy. With the hope of expanding the service and making it a permanent function of New York City’s public schools, the SLC transferred control to the Board of Education in 1919. Despite the success of the pilot program and the availability of public funding earmarked to maintain and even expand school lunch provision, the Board drastically reduced meal service. What had been a carefully planned and executed school health initiative was mostly replaced by a for-profit concessionaire system with no public health or educational mandate, no nutritional requirements, no food safety inspections, no reduced-price or free meals for poor children, and virtually no oversight of any kind. It is overly simplistic to regard the Board’s abdication of a popular health, education, and social welfare program as a government agency’s callous indifference to the needs of the poor. Because school meals were a matter of public policy in numerous domains, including health, education, labor, law, and social welfare, what the SLC regarded as a simple transfer from private charity to public entitlement was in fact a socially and politically charged negotiation of responsibility for children’s nutritional health and the proper role of the public school.
Pomegranate and the Mediation of Balance in Early Medicine
A. R. Ruis
Gastronomica: The Journal of Critical Food Studies 15, no. 1 (2015): 22-33.
Different elements of the pomegranate, both tree and fruit, had a wide range of uses in pre-modern therapeutics. Pomegranate also had a rich symbolic role in the art, literature, and religion of numerous cultures. In nearly every part of the globe where the pomegranate grew, it came to represent fundamental dualities: life and death, inside and out, many and one. The medicinal purposes for which healers recommended pomegranate at times reflected broader symbolic associations, and those associations are an important part of the therapeutic tradition. The dualistic symbolism that attended the pomegranate in various cultural traditions synergized with dualistic medical concepts, reinforcing the therapeutic power of pomegranate in otherwise diverse contexts. Reflecting this duality, pomegranate was both an astringent and a laxative, an emmenagogue and an antimenorrhagic, an expectorant and an antiemetic, a pyrogen and an febrifuge, a restorative and a soporific. In both literary and medical traditions, the pomegranate mediated transitions—or maintained balance—between opposing states. This essay provides an overview of the rich and sundry uses of pomegranate in pre-modern therapeutics, revealing how cultural associations both reflected and informed medical practices.
A Novel Paradigm for Engineering Education: Virtual Internships with Individualized Mentoring and Assessment of Engineering Thinking
N. C. Chesler, A. R. Ruis, W. Collier, Z. Swiecki, G. Arastoopour, & D. Williamson Shaffer
Journal of Biomechanical Engineering 137, no. 2 (2015): 024701-1–8.
Engineering virtual internships are a novel paradigm for providing authentic engineering experiences in the first year curriculum.They are both individualized and accommodate large numbers of students. As we describe in this report, this approach can (a) enable students to solve complex engineering problems in a mentored, collaborative environment; (b) allow educators to assess engineering thinking; and (c) provide an introductory experience that students enjoy and find valuable. Furthermore, engineering virtual internships have been shown to increase students’—and especially women’s—interest in and motivation to pursue engineering degrees. When implemented in first-year engineering curricula more broadly, the potential impact of engineering virtual internships on the size and diversity of the engineering workforce could be dramatic.
Authoring Networked Learner Models in Complex Domains
D. Williamson Shaffer, A. R. Ruis, & A. C. Graesser
Design Recommendations for Intelligent Tutoring Systems: Volume 3 – Authoring Tools and Expert Modeling Techniques, eds. R. Sottilare, A. C. Graesser, X. Hu, & K. Brawner (Orlando: U.S. Army Research Laboratory, 2015), 179–191.
Developing cost-effective authoring tools that enable non-programmers to create sophisticated virtual learning environments has been described as the “holy grail” of authorware design. Such tools must account for essential components, such as conversation management, semantic representations, production rules, and pedagogical strategies. The content produced must conform to theory-driven constraints, discourse processes, cognitive science, and computer science, along with practical constraints such as state standards, education policies, and standardized tests. Can all of this occur without requiring the user to have expertise in computer programming or educational software development? While progress has been made toward this goal, most sophisticated authoring systems (there are many for intelligent tutoring systems alone) are used primarily in research contexts. Those that have received broader usage, such as Cognitive Tutor Authoring Tools (CTAT) and Authoring Software Platform for Intelligent Resources in Education (ASPIRE), primarily support the development of modules that help students learn to solve well-formed problems, such as those common in basic mathematics, computer science, or language acquisition. In this chapter, we discuss the potential to develop authorware for learning environments in which students solve complex, ill-formed problems. In particular, we explore the design parameters of authoring tools for Syntern virtual internships, online learning environments that simulate professional practica in complex domains such as engineering design and urban planning. Given the relatively small body of research on the processes with which curriculum developers design content, we argue that a key element of developing authorware is to develop a science of the pedagogical authoring process.
“Children with Half-Starved Bodies” and the Assessment of Malnutrition in the United States, 1890-1950
A. R. Ruis
Bulletin of the History of Medicine 87, no. 3 (2013): 380-408.
Malnutrition was one of the most significant children’s health issues of the early twentieth century, but it also engendered considerable controversy. Just how many children were truly malnourished, and how could they be reliably identified? Despite the failures of numerous diagnostic methods—even the definition of malnutrition defied consensus—health authorities remained convinced that malnutrition was a serious and widespread problem. Indeed, the imprecision that surrounded the condition allowed it to be used metaphorically to advance a broad range of professional, social, and public health agendas. By the 1940s, due in part to the lack of reliable diagnostic methods, public health nutrition policy shifted abruptly from one of assessment, based on mass surveillance and individualized care, to one of management, based on a universal program of nutrition education, fortification of foods, and food security that treated all children as in need of nutritional assistance.
Nutrition Classes and Clinics
A. R. Ruis
The Oxford Encyclopedia of Food and Drink in America, 2nd Edition, ed. Andrew F. Smith (Oxford: Oxford University Press, 2012), 723-25.
The nutrition class, also know as the nutrition clinic, helped undernourished children to achieve and to maintain good health through a combination of routine medical examination and care, supplemental feeding, instruction in foods and nutrition, and social work. Along with other public health nutrition initiatives developed during the Progressive Era, such as school meal programs, anthropometric assessment of nutritional health, and extension work in foods and nutrition, nutrition classes were a response to public and professional concern about malnutrition in the first decades of the 20th century.
The Schism Between Medical and Public Health Education: A Historical Perspective
A. R. Ruis & R. N. Golden
Academic Medicine 83, no. 12 (2008): 1153-57.
The separation of “medicine” and “public health” in academic institutions limits the potential synergies that an integrated educational model could offer. The roots of this separation are deeply imbedded in history. During the past two centuries, there have been repeated efforts to integrate public health education into the core training of physicians, usually in response to a perceived short-term crisis, and without widespread, lasting success. The cost of additional public health instruction and the “overcrowding” of the medical curriculum have been cited as obstacles for creating an integrated medical/public health curriculum for more than a century. Several thoughtful and prescient proposals for integration were developed at a conference convened by the Rockefeller Foundation in the early 20th century, but not all were implemented. Today, there is growing recognition of the considerable value afforded by the integration of medicine and public health education. Many schools have responded to a national call for a renewed relationship between medicine and public health by increasing the availability of MD/MPH programs and/or by incorporating one or more public health courses into the basic medical curriculum. A few schools have created more substantial and innovative changes. Review and consideration of the history and politics of past efforts may serve as a guide for the development of successful new approaches to creating a clinical workforce that incorporates the principles of both clinical medicine and public health.