Simplification of Epistemic Networks Using Parsimonious Removal with Interpretive Alignment
Y. Wang, Z. Swiecki, A. R. Ruis, & D. Williamson Shaffer
Advances in Quantitative Ethnography: Second International Conference, ICQE 2020, Malibu, CA, USA, February 1-3, 2021, Proceedings, eds. A. R. Ruis & S. B. Lee (Springer, 2021), 137–151.
A key goal of quantitative ethnographic (QE) models, and statistical models more generally, is to produce the most parsimonious model that adequately explains or predicts the phenomenon of interest. In epistemic network analysis (ENA), for example, this entails constructing network models with the fewest number of codeswhose interaction structure provides sufficient explanatory power in a given context. Unlike most statistical models, however, modification of ENA models can affect not only the statistical properties but also the interpretive alignment between quantitative features and qualitative meaning that is a central goal in QE analyses. In this study, we propose a novel method, Parsimonious Removal with Interpretive Alignment, for systematically identifying more parsimonious ENA models that are likely to maintain interpretive alignment with an existing model. To test the efficacy of the method, we implemented it on a well-studied dataset for which there is a published, validated ENA model, and we show that the method successfully identifies reduced models likely to maintain explanatory power and interpretive alignment.
Directed Epistemic Network Analysis
A. Fogel, Z. Swiecki, C. L. Marquart, Z. Cai, Y. Wang, J. Brohinsky, A. L. Siebert-Evenstone, B. R. Eagan, A. R. Ruis, & D. Williamson Shaffer
Advances in Quantitative Ethnography: Second International Conference, ICQE 2020, Malibu, CA, USA, February 1-3, 2021, Proceedings, eds. A. R. Ruis & S. B. Lee (Springer, 2021), 122–136.
Quantitative ethnographers across a range of domains study complex collaborative thinking (CCT): the processes by which members of a group or team develop shared understanding by making cognitive connections from the statements and actions of the group. CCT is difficult to model because the actions of group members are interdependent—the activity of any individual is influenced by the actions of other members of the group. Moreover, the actions of group members engaged in some collaborative tasks may need to follow a particular order. However, current techniques can account for either interdependence or order, but not both. In this paper, we present directed epistemic network analysis (dENA), an extension of epistemic network analysis (ENA), as a method that can simultaneously account for the interdependent and ordered aspects of CCT. To illustrate the method, we compare a qualitative analysis of two U.S. Navy commanders working in a simulation to ENA and dENA analyses of their performance. We find that by accounting for interdependence but not order, ENA was not able to model differences between the commanders seen in the qualitative analysis, but by accounting for both interdependence and order, dENA was able to do so.
Trajectories in Epistemic Network Analysis
J. Brohinsky, C. L. Marquart, J. Wang, A. R. Ruis, & D. Williamson Shaffer
Advances in Quantitative Ethnography: Second International Conference, ICQE 2020, Malibu, CA, USA, February 1-3, 2021, Proceedings, eds. A. R. Ruis & S. B. Lee (Springer, 2021), 106–121.
While quantitative ethnographers have used epistemic network analysis (ENA) to model trajectories that show change in network structure over time, visualizing trajectory models in a way that facilitates accurate interpretation has been a significant challenge. As a result, ENA has predominantly been used to construct aggregate models, which can obscure key differences in how network structures change over time. This study reports on the development and testing of a new approach to visualizing ENA trajectories. It documents the challenges associated with visualizing ENA trajectory models, the features constructed to address those challenges, and the design decisions that aid in the interpretation of trajectory models. To test this approach, we compare ENA trajectory models with aggregate models using a dataset with previously published results and known temporal features. This comparison focuses on interpretability and consistency with prior qualitative analysis, and we show that ENA trajectories are able to represent information unavailable in aggregate models and facilitate interpretations consistent with qualitative findings. This suggests that this approach to ENA trajectories is an effective tool for representing change in network structure over time.
Exploring the Effects of Segmentation on Semi-Structured Interview Data with Epistemic Network Analysis
S. Zörgő, Z. Swiecki, & A. R. Ruis
Advances in Quantitative Ethnography: Second International Conference, ICQE 2020, Malibu, CA, USA, February 1-3, 2021, Proceedings, eds. A. R. Ruis & S. B. Lee (Springer, 2021), 78–90.
Quantitative ethnographic models are typically constructed using qualitative data that has been segmented and coded. While there exist methodological studies that have investigated the effects of changes in coding on model features, the effects of segmentation have received less attention. Our aim was to examine, using a dataset comprised of narratives from semi-structured interviews, the effects of different segmentation decisions on population- and individual-level model features via epistemic network analysis. We found that while segmentation choices may not affect model features overall, the effects on some individual networks can be substantial. This study demonstrates a novel method for exploring and quantifying the impact of segmentation choices on model features.
How We Code
D. Williamson Shaffer & A. R. Ruis
Advances in Quantitative Ethnography: Second International Conference, ICQE 2020, Malibu, CA, USA, February 1-3, 2021, Proceedings, eds. A. R. Ruis & S. B. Lee (Springer, 2021), 62–77.
Coding data defining concepts and identifying where they occur in data is a critical aspect of qualitative data analysis, and especially so in quanti-tative ethnography. Coding is a central process for creating meaning from data, and while much has been written about coding methods and theory, relatively little has been written about what constitutes best practices for fair and valid coding, what justifies those practices, and how to implement them. In this paper, our goal is not to address these issues comprehensively, but to provide guidelines for good coding practice and to highlight some of the issues and key questions that quantitative ethnographers and other re-searchers should consider when coding data.
Assessing Individual Contributions to Collaborative Problem Solving: A Network Analysis Approach
Z. Swiecki, A. R. Ruis, C. Farrell, & D. Williamson Shaffer
Computers in Human Behavior 104 (2020): 105876.
Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. However, current methods for measuring the CPS processes of individuals, such as coding and counting, treat these processes as sets of isolated and independent events. In contrast, Epistemic Network Analysis (ENA) models how the contributions of a given individual relate to the contributions of others. This article examines the communications of air defense warfare teams from an experiment comparing two different computer-based decision support systems, using this data to ask whether ENA provides a more ecologically valid quantitative model of CPS than coding and counting. Qualitative analysis showed that commanders using one system asked questions to understand the tactical situation, while commanders using an experimental system focused more on actions in response to the tactical situation. Neither of the coding and counting approaches we tested corroborated these findings with statistically significant results. In contrast, ENA created models of the individual contributions of commanders that (a) showed statistical differences between commanders using the two systems to corroborate the qualitative analysis, and (b) revealed differences in individual performance. This suggests that ENA is a more powerful tool for CPS assessment than coding and counting approaches.
Evaluating How Residents Talk and What It Means for Surgical Performance in The Simulation Lab
A. D. D’Angelo, A. R. Ruis, W. Collier, D. Williamson Shaffer, & C. M. Pugh
American Journal of Surgery 220, no. 1 (2020): 37–43.
Background: This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate skills and knowledge through discourse.
Methods: Senior residents (N = 11) were recorded while performing a simulated laparoscopic ventral hernia (LVH) repair. Audio transcripts were coded for five discourse elements related to knowledge, skills, and operative independence. Epistemic network analysis was used to model the ordered integration of the five discourse elements.
Results: Participants with poorer hernia repair outcomes had stronger connections between the discourse elements operative planning and asking for information or advice (Operative planning), while participants with better hernia repair outcomes had stronger connections between the discourse elements giving assistant instructions and identifying errors (Operative management): (p = .006; Cohen’s d = 2.79).
Conclusion: Participants with better hernia repair outcomes engaged in more operative management communication during the simulated procedure. This ability to integrate multiple operative steps and verbally communicate them significantly correlated with better operative outcomes.
Multiple Uses for Procedural Simulators in Continuing Medical Education Contexts
A. R. Ruis, A. A. Rosser, J. N. Nathwani, M. V. Beems, S. A. Jung, & C. M. Pugh
Advances in Quantitative Ethnography: First International Conference, ICQE 2019, Madison, WI, USA, October 20–22, 2019, Proceedings, eds. B. Eagan, M. Misfeldt, & A. Siebert-Evenstone (Springer, 2019), 211–222.
Simulators have been widely adopted to help surgical trainees learn procedural rules and acquire basic psychomotor skills, and research indicates that this learning transfers to clinical practice. However, few studies have explored the use of simulators to help more advanced learners improve their understanding of operative practices. To model how surgeons with different levels of experience use procedural simulators, we conducted a quantitative ethnographic analysis of small-group conversations in a continuing medical education short course on laparoscopic hernia repair. Our research shows that surgeons who had less experience with laparoscopic surgery tended to use the simulators to learn and rehearse the basic procedures, while more experienced surgeons used the simulators as a platform for exploring a range of hernia presentations and operative approaches based on their experiences. Thus simple, inexpensive simulators may be effective with both novice and more experienced learners.
Designing an Interface for Sharing Quantitative Ethnographic Research Data
Z. Swiecki, C. Marquart, A. Sachar, C. Hinojosa, A. R. Ruis, & D. Williamson Shaffer
Advances in Quantitative Ethnography: First International Conference, ICQE 2019, Madison, WI, USA, October 20–22, 2019, Proceedings, eds. B. Eagan, M. Misfeldt, & A. Siebert-Evenstone (Springer, 2019), 334–341.
Recently, there have been growing calls to make research data more widely available. While the potential benefits of sharing research data are many, there are also many challenges, including the interpretability, attendability, and complexity of the data. These challenges are particularly salient for research data associated with quantitative ethnographic analyses, which often use relatively novel and sophisticated techniques. In this paper, we explore design considerations for an interface for sharing research data that attempts to address these challenges for quantitative ethnographic analyses. These considerations include: (a) maintaining the consistency of the interpretive space, (b) simplifying model details, (c) including example results and interpretations, and (d) highlighting key affordances in the user interface. To explore these considerations, we describe the design of an interactive visualization of the thematic networks present in the HBO television series, Game of Thrones.
Understanding When Students Are Active-in-Thinking through Modeling-in-Context
Z. Swiecki, A. R. Ruis, D. Gautam, V. Rus, & D. Williamson Shaffer
British Journal of Educational Technology 50, no. 5 (2019): 2346–2364.
Learning-in-action depends on interactions with learning content, peers and real world problems. However, effective learning-in-action also depends on the extent to which students are active-in-thinking, making meaning of their learning experience. A critical component of any technology to support active thinking is the ability to ascertain whether (or to what extent) students have succeeded in internalizing the disciplinary strategies, norms of thinking, discourse practices and habits of mind that characterize deep understanding in a domain. This presents what we call a dilemma of modeling-incontext: teachers routinely analyze this kind of thinking for small numbers of students in activities they create or customize for the needs of their students; however, doing so at scale and in real-time requires some automated processes for modeling student work. Current techniques for developing models that reflect specific pedagogical activities and learning objectives that a teacher might create require either more expertise or more time than teachers have. In this paper, we examine a theoretical approach to addressing the problem of modeling active thinking in its pedagogical context that uses teachercreated rubrics to generate models of student work. The results of this examination show how appropriately constructed learning technologies can enable teachers to develop custom automated rubrics for modeling active thinking and meaning-making from the records of students’ dialogic work.
Analysing Computational Thinking in Collaborative Programming: A Quantitative Ethnography Approach
B. Wu, Y. Hu, A. R. Ruis, & M. Wang
Journal of Computer Assisted Learning 35, no. 3 (2019): 421–434.
Computational thinking (CT), the ability to devise computational solutions for real‐life problems, has received growing attention from both educators and researchers. To better improve university students’ CT competence, collaborative programming is regarded as an effective learning approach. However, how novice programmers develop CT competence through collaborative problem solving remains unclear. This study adopted an innovative approach, quantitative ethnography, to analyze the collaborative programming activities of a high‐performing and a low‐performing team. Both the discourse analysis and epistemic network models revealed that across concepts, practices, and identity, the high‐performing team exhibited CT that was systematic, whereas the CT of the low‐performing team was characterized by tinkering or guess‐and‐check approaches. However, the low‐performing group’s CT development trajectory ultimately converged towards the high‐performing group’s. This study thus improves understanding of how novices learn CT, and it illustrates a useful method for modeling CT based in authentic problem‐solving contexts.
Finding Common Ground: A Method for Measuring Recent Temporal Context in Analyses of Complex, Collaborative Thinking
A. R. Ruis, A. L. Siebert-Evenstone, R. Pozen, B. R. Eagan, & D. Williamson Shaffer
A Wide Lens: Combining Embodied, Enactive, Extended, and Embedded Learning in Collaborative Settings: 13th International Conference on Computer-Supported Collaborative Learning (CSCL) 2019, eds. K. Lund, G. Niccolai, E. Lavoué, C. Hmelo-Silver, G. Gweon, & M. Baker (2019), I:136–143.
Complex, collaborative thinking is often conceptualized as a process of developing cognitive connections among the contributions of different participants. A central problem in modeling collaboration in this way is thus determining, for any contribution to a discussion, the appropriate context for modeling the connections being made—that is, for determining the appropriate recent temporal context. Recent temporal context is typically defined using a moving window of fixed length. However, that length is dependent on the setting, and there are no existing methods for reliably determining an appropriate window length. This paper presents an empirical method for measuring recent temporal context, and thus for defining an appropriate window length to be used in analyses of complex, collaborative thinking. Importantly, the method we describe minimizes the need for human annotation while providing both qualitative and quantitative warrants for choosing a particular window length.
Reading for Breadth, Reading for Depth: Understanding the Relationship between Reading and Complex Thinking using Epistemic Network Analysis
H. Sung, S. Cao, A. R. Ruis, & D. Williamson Shaffer
A Wide Lens: Combining Embodied, Enactive, Extended, and Embedded Learning in Collaborative Settings: 13th International Conference on Computer-Supported Collaborative Learning (CSCL) 2019, eds. K. Lund, G. Niccolai, E. Lavoué, C. Hmelo-Silver, G. Gweon, & M. Baker (2019), I:376–383.
This paper examines whether and to what extent long and short readers make different contributions to collaborative design discussions in a CSCL environment—that is, we investigate whether a simple measure of reading behavior based on clickstream data is a good proxy for engagement with readings. Our approach to addressing this question is multimodal, involving two sources of data: (a) a record of students’ online conversations, and (b) the frequency and duration with which documents were open on each student’s screen. This study suggests that in this specific case, relatively thin data about reading frequency and mean reading duration can be used to make inferences about students’ reading behavior in a CSCL context where it is impossible to observe directly. It also shows the power of a multimodal approach to the data—here, we are using one mode of data (discussion) to get a better understanding of another mode (clickstream).
Does Order Matter? Investigating Sequential and Cotemporal Models of Collaboration
Z. Swiecki, Z. Lian, A. R. Ruis, & D. Williamson Shaffer
A Wide Lens: Combining Embodied, Enactive, Extended, and Embedded Learning in Collaborative Settings: 13th International Conference on Computer-Supported Collaborative Learning (CSCL) 2019, eds. K. Lund, G. Niccolai, E. Lavoué, C. Hmelo-Silver, G. Gweon, & M. Baker (2019), I:112–119.
Many researchers have argued that models of collaborative processes should account for temporality, but there exist different approaches for doing so. We compared two specific approaches to modeling collaborative processes in a CSCL context: Epistemic Network Analysis, which models events cotemporally (unordered and temporally proximate), and Sequential Pattern Mining, which models events sequentially (ordered and temporally proximate). Our results suggest that in this context cotemporal models constructed with Epistemic Network Analysis outperform sequential models constructed with Sequential Pattern Mining in terms of (a) explanatory power, (b) efficiency, and (c) interpretability.
“Trois Empreintes d’un Même Cachet”: Toward a Historical Definition of Nutrition
A. R. Ruis
Viral Networks: Connecting Digital Humanities and Medical History, eds. E. T. Ewing & K. Randall (Blacksburg: VT Publishing, 2018), 185–216.
In this chapter, I attempt to model the concept of “nutrition” in English-language sources from the 19th and 20th centuries using epistemic network analysis (ENA), a set of techniques for measuring, visualizing, and comparing patterns of association among conceptual elements. In doing so, I argue that conceptual networks can help us understand macrohistorical patterns in discourses&emdash;in this case, discourses of nutrition&emdash;without sacrificing microhistorical rigor. Specifically, I will describe an approach in which microhistorical analyses inform the development of macrohistorical models that in turn suggest new avenues for microhistorical investigation.
Thinking about Sources as Data: Reflections on Epistemic Network Analysis as a Technique for Historical Research
M. DiMeo & A. R. Ruis
Viral Networks: Connecting Digital Humanities and Medical History, eds. E. T. Ewing & K. Randall (Blacksburg: VT Publishing, 2018), 113–135.
In this chapter, we reflect on the use of epistemic network analysis (ENA) as a tool for modeling conceptual networks. Because there are a number of resources that explain ENA in great detail as a technique and a tool, we will not discuss how to use ENA but rather explore why and how a historian might find the approach useful. Following this, we explore some of the issues with which the historian must engage in order to move from a strictly human, qualitative methodology to a mixed-methods approach that includes ENA. While digital humanities papers commonly include a methods section, these final products tend not to reflect on the complexity of the methodological process that got the authors to that stage, to talk openly about which data models failed, or to reflect on the limitations of tools they previously considered and rejected. This paper is intentionally focused on this “work in progress” stage that all historians go through, and which newcomers to the digital humanities can find isolating. Using a case study approach&emdash;applying ENA to a seventeenth-century archival collection of letters known as the Hartlib Papers&emdash;we will consider the kinds of intellectual and theoretical challenges historians may grapple with as they try to think about their source materials as a dataset and supplement their qualitative analyses with quantitative models.
The Hands and Head of a Surgeon: Modeling Operative Competency with Multimodal Epistemic Network Analysis
A. R. Ruis, A. A. Rosser, C. Quandt-Walle, J. N. Nathwani, D. Williamson Shaffer, & C. M. Pugh
American Journal of Surgery 216, no. 5 (2018): 835-840.
This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate psychomotor, procedural, and cognitive skills to manage errors.
Audio-video data were collected from general surgery residents (N = 45) performing a simulated laparoscopic ventral hernia repair. Errors were identified using a standard checklist, and speech was coded for elements related to error recognition and management. Epistemic network analysis (ENA) was used to model the integration of error management skills.
There was no correlation between number or type of errors committed and operative outcome. However, ENA models showed significant differences in the integration of error management skills between high-performing and low-performing residents.
These results suggest that error checklists and surgeons’ speech can be used to model the integration of psychomotor, procedural, and cognitive aspects of intraoperative performance. Moreover, ENA can identify and quantify this integration, providing insight on performance gaps in both individuals and populations.
A Network Analytic Approach to Gaze Coordination during a Collaborative Task
S. Andrist, A. R. Ruis, & D. Williamson Shaffer
Computers in Human Behavior 89 (2018): 339-348.
A critical component of collaborative learning is the establishment of intersubjectivity, or the construction of mutual understanding. Collaborators coordinate their understanding with one another across various modes of communication, including speech, gesture, posture, and gaze. Given the dynamic, interdependent, and complex nature of coordination, this study sought to develop and test a method for constructing detailed and nuanced models of coordinated referential gaze patterns. In the study, 13 dyads participated in a simple collaborative task. We used dual mobile eye tracking to record each participant’s gaze behavior, and we used epistemic network analysis (ENA) to model the gazes of both conversational participants synchronously. In the model, the nodes in the network represent gaze targets for each participant, and the connections between nodes indicate the likelihood of gaze coordination. Our analyses indicate: (a) properties and patterns of how gaze coordination unfolds throughout an interaction sequence; and (b) differences in gaze coordination patterns for interaction sequences that lead to breakdowns and repairs. In addition to contributing to the growing body of knowledge on the coordination of gaze behaviors in collaborative activities, this work suggests that ENA enables more effective modeling of gaze coordination.
Modeling Processes of Enculturation in Team Training
A. R. Ruis, A. J. Hampton, B. S. Goldberg, & D. Williamson Shaffer
Design Recommendations for Intelligent Tutoring Systems: Volume 6 – Team Tutoring, eds. R. Sottilare, A. C. Graesser, X. Hu, & A. M. Sinatra (Orlando: U.S. Army Research Laboratory, 2018), 45–51.
Both military and civilian work is increasingly organized around small, dynamic teams rather than large bureaucratic frameworks. Such teams combine individuals with highly specific skill sets and extensive training to solve complex, non-standard problems, often under extreme pressure. Importantly, these teams are not typically composed of interchangeable members but are formed and trained as expert teams to function semi-autonomously. To accomplish mission objectives in the face of complex challenges, the United States military needs to develop teams that consistently exhibit high levels of taskwork and teamwork skills. Implementing principles of team training and maintaining team efficacy are critical in the armed forces, where teams are often widely dispersed and consequences for underperformance can be severe, though many civilian teams—such as hospital trauma teams or flight crews—face similar challenges. Establishing and maintaining high levels of team performance in these contexts requires the creation of practical and effective team development interventions, including team training, as well as systems for ongoing assessment of team function. We argue that one critical component of training, monitoring, and maintaining high-functioning teams is the ability to model team performance.
Using Epistemic Network Analysis to Identify Targets for Educational Interventions in Trauma Team Communication
S. A. Sullivan, C. Warner-Hillard, B. R. Eagan, R. Thompson, A. R. Ruis, K. Haines, C. M. Pugh, D. Williamson Shaffer, & H. S. Jung
Surgery 163, no. 4 (2018): 938–943.
Epistemic Network Analysis (ENA) is a technique for modeling and comparing the structure of connections between elements in coded data. We hypothesized that connections among team discourse elements as modeled by ENA would predict the quality of team performance in trauma simulation.
The Modified Non-technical Skills Scale for Trauma (T-NOTECHS) was used to score a simulation-based trauma team resuscitation. Sixteen teams of 5 trainees participated. Dialogue was coded using Verbal Response Modes (VRM), a speech classification system. ENA was used to model the connections between VRM codes. ENA models of teams with lesser T-NOTECHS scores (n = 9, mean = 16.98, standard deviation [SD] = 1.45) were compared with models of teams with greater T-NOTECHS scores (n = 7, mean = 21.02, SD = 1.09).
Teams had different patterns of connections among VRM speech form codes with regard to connections among questions and edifications (meanHIGH = 0.115, meanLOW = −0.089; t = 2.21; P = .046, Cohen d = 1.021). Greater-scoring groups had stronger connections between stating information and providing acknowledgments, confirmation, or advising. Lesser-scoring groups had a stronger connection between asking questions and stating information. Discourse data suggest that this pattern reflected increased uncertainty. Lesser-scoring groups also had stronger connections from edifications to disclosures (revealing thoughts, feelings, and intentions) and interpretations (explaining, judging, and evaluating the behavior of others).
ENA is a novel and valid method to assess communication among trauma teams. Differences in communication among higher- and lower-performing teams appear to result from the ways teams use questions. ENA allowed us to identify targets for improvement related to the use of questions and stating information by team members.
In Search of Conversational Grain Size: Modeling Semantic Structure using Moving Stanza Windows
A. L. Siebert-Evenstone, G. Arastoopour, W. Collier, Z. Swiecki, A. R. Ruis, & D. Williamson Shaffer
Journal of Learning Analytics 4, no. 3 (2017): 123–139.
Analyses of learning based on student discourse need to account not only for the content of the utterances but also for the ways in which students make connections across turns of talk. This requires segmentation of discourse data to define when connections are likely to be meaningful. In this paper, we present an approach to segmenting data for the purposes of modelling connections in discourse using epistemic network analysis. Specifically, we use epistemic network analysis to model connections in student discourse using a temporal segmentation method adapted from recent work in the learning sciences. We compare the results of this study to a purely conversation-based segmentation method to examine the affordances of temporal segmentation for modelling connections in discourse.
Eating to Learn, Learning to Eat: The Origins of School Lunch in the United States
A. R. Ruis
New Brunswick: Rutgers University Press, 2017.
This book explores the origins of the National School Lunch Program, the most extensive and longest-running children’s nutrition program in U.S. history. In doing so, it reveals how and why school meals came to have the form that is so familiar to most Americans. While the National School Lunch Program was in part a response to specific concerns about children’s access to sufficient and sufficiently nourishing food during the Great Depression and World War II, many of the issues that shaped school meal policies and practices arose decades earlier. By exploring the origins of school meal initiatives, this book endeavors to explain why it was (and to some extent, has continued to be) so difficult to establish meal programs that satisfy the often competing interests of children, parents, schools, health authorities, politicians, and the food industry. This is accomplished through careful studies of several regions and detailed analysis of the policies and politics that governed the development of school meal programs, first at the local level, and then at the federal level. In doing so, the book shows how the early history of school meal program development helps us understand contemporary debates over changes to school lunch policies, including those inaugurated by the passage of the Healthy, Hunger-Free Kids Act in 2010.
Automating the Detection of Reflection-on-Action
J. Saucerman, A. R. Ruis, & D. Williamson Shaffer
Journal of Learning Analytics 4, no. 2 (2017): 207–234.
Learning to solve complex problems—problems whose solutions require the application of more than basic facts and skills—is critical to meaningful participation in the economic, social, and cultural life of the digital age. In this paper, we use a theoretical understanding of how professionals use reflection-in-action to solve complex problems to investigate how students learn this critical 21st-century skill and how we can develop and automate learning analytic techniques to assess that learning. We present a preliminary study examining the automated detection of reflective discourse during collaborative, complex problem solving. We analyze student reflection-on-action in a virtual learning environment, focusing on both reflection in individual discourse and collaborative reflection among students. Our results suggest that it is possible to detect student reflection on complex problems in virtual learning environments, but that different models may be appropriate depending on students’ prior domain experience.
Can We Rely on Reliability? Testing the Assumptions of Inter-Rater Reliability
B. R. Eagan, B. Rogers, R. Serlin, A. R. Ruis, G. Arastoopour, & D. Williamson Shaffer
Making a Difference: Prioritizing Equity and Access in CSCL: 12th International Conference on Computer-Supported Collaborative Learning, eds. B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (2017), II:529–532.
Researchers use Inter-Rater Reliability (IRR) to measure whether two processes—people and/or machines—identify the same properties in data. There are many IRR measures, but regardless of the measure used, there is a common method for estimating IRR. To assess the validity of this common method, we conducted Monte Carlo simulation studies examining the most widely used measure of IRR: Cohen’s kappa. Our results show that the method commonly used by researchers to assess IRR produces unacceptable Type I error rates.
Epistemic Network Analysis: A Worked Example of Theory-Based Learning Analytics
D. Williamson Shaffer & A. R. Ruis
Handbook of Learning Analytics, eds. C. Lang, G. Siemens, A. Wise, & D. Gašević (Society for Learning Analytics Research, 2017), 175–187.
In this chapter, we provide a worked example of a theory-based approach to learning analytics in the context of an educational game. We do this not to provide an ideal solution for others to emulate, but rather to explore the affordances of a theory-based—rather than data-driven—approach. We do so by presenting (1) epistemic frame theory as an approach to the conceptualization of learning; (2) data from an epistemic game, an approach to educational game design based on epistemic frame theory; and (3) epistemic network analysis (ENA), a technique for analyzing discourse and other data for evidence of complex thinking based on the same theory. We describe ENA through a specific analytic result, but our aim is to explore how this result exemplifies what we consider a key “best practice” in the field of learning analytics.
Annals and Analytics: The Practice of History in the Age of Big Data
A. R. Ruis & D. Williamson Shaffer
Medical History 61, no. 2 (2017): 336-339.
The practice of historical research stands to benefit considerably from, and in a growing number of cases requires, a mixed-methods approach that combines the qualitative and the quantitative and incorporates the analytic strengths of human interpretation and computational processing. In this brief reflection, we explore one approach to mixed-methods history using network analysis: various statistical techniques with which the structure of connections among entities—people, places, concepts, and so forth—can be modeled.
Procedural Simulations and Reflective Practice: Meeting the Need
S. A. Sullivan, A. R. Ruis, & C. M. Pugh
Journal of Laparoendoscopic and Advanced Surgical Techniques 27, no. 5 (2017): 455-458.
The medical profession continues to learn about the ways in which simulations can help to improve education and training. In addition, simulation is constantly evaluated regarding the many ways it can help to improve the abilities of surgeons throughout their course of training and practice. The goal of this study is to review the importance of reflection on clinical practice in situating surgeons as lifelong learners and explore the role that simulation can play in that process. As surgical trainees work to acquire basic necessary skills, simulations are often used to help new learners with such things as knowledge of procedural steps and development of psychomotor techniques. However, the following is an important question: can simulated procedures still play a role for more advanced learners to continue their professional development beyond the basics?
A Tutorial on Epistemic Network Analysis: Analyzing the Structure of Connections in Cognitive, Social, and Interaction Data
D. Williamson Shaffer, W. Collier, & A. R. Ruis
Journal of Learning Analytics 3, no. 3 (2016): 9-45.
This paper provides a tutorial on epistemic network analysis (ENA), a novel method for identifying and quantifying connections among elements in coded data and representing them in dynamic network models. Such models illustrate the structure of connections and measure the strength of association among elements in a network, and they quantify changes in the composition and strength of connections over time. Importantly, ENA enables comparison of networks both directly and via summary statistics, so the method can be used to explore a wide range of qualitative and quantitative research questions in situations where patterns of association in data are hypothesized to be meaningful. While ENA was originally developed to model cognitive networks—the patterns of association between knowledge, skills, values, habits of mind, and other elements that characterize complex thinking—ENA is a robust method that can be used to model patterns of association in any system characterized by a complex network of dynamic relationships among a relatively small, fixed set of elements.
Teaching Health Care Workers to Adopt a Systems Perspective for Improved Control and Prevention of Health Care-Associated Infections
A. R. Ruis, D. Williamson Shaffer, D. K. Shirley, & N. Safdar
American Journal of Infection Control 44, no. 11 (2016): 1360–1364.
- Education and training of health care workers is fundamental to infection prevention.
- A systems approach is needed for optimal health care–associated infection prevention interventions.
- A cognitive simulation, allowing the health care workers to adopt a multidimensional perspective, should be tested for efficacy in health care settings.
Teaching and Assessing Engineering Design Thinking with Virtual Internships and Epistemic Network Analysis
G. Arastoopour, D. Williamson Shaffer, Z. Swiecki, A. R. Ruis, & N. C. Chesler
International Journal of Engineering Education 32, no. 3B (2016): 1492-1501.
One of the goals of engineering education is providing students with authentic and meaningful design experiences as well as assessing the development of design thinking. In this paper, we review virtual internships, online simulations of 21st-century engineering design practice, as one method for providing students with authentic experiences. To assess the development of design thinking in virtual internships, we used epistemic network analysis (ENA), a tool for measuring complex thinking as it develops over time. We provide an example of how ENA can be used to measure quantitatively student teams’ qualitative discourse in a virtual internship program in order to assess performance in engineering. The combination of virtual internships and ENA provides opportunities for students to engage in authentic engineering design, receive concurrent feedback on their design thinking, and develop the identity, values, and ways of thinking of 21st-century professional engineers.
Should Trainees be Involved in the Direct Care of Patients with Ebola Virus Disease?
N. Safdar, A. R. Ruis, D. K. Shirley, & N. Fost
AMEE MedEdPublish 5, no. 1 (2016).
With the world in the throes of the largest outbreak of Ebola Virus Disease (EVD) in history, many institutions in the United States have developed institutional policies and procedures for the screening, diagnosis and management of patients with EVD. Healthcare workers caring for these seriously ill patients are at high risk of contracting EVD through contact with blood and body fluids. With the exception of experimental therapeutics, treatment is largely supportive, requiring healthcare workers to have regular close contact with patients. Given the proportion of healthcare workers who have contracted EVD from patients and the high mortality rate, a central consideration has been whether healthcare trainees should be exempted from the direct care of patients with EVD. Educational needs and ethical issues are discussed. We provide arguments for and against this approach, and present a multifaceted framework that may be useful to institutions grappling with this issue.
Local versus Global Connection Making in Discourse
W. Collier, A. R. Ruis, & D. Williamson Shaffer
Transforming Learning, Empowering Learners: The International Conference of the Learning Sciences (ICLS) 2016, eds. C.-K. Looi, J. Polman, U. Cress, & P. Reimann (2016), I:426–433.
This paper examines techniques for modeling relationships among domain concepts and practices in discourse to assess learning in a CSCL environment. We compare two approaches: a traditional psychometric approach, which models the global correlation structure of student discourse markers across the learning intervention, and a model that accounts for the local correlation structure of discourse markers within activities. We investigate whether: (a) analysis of local correlation structure can identify significant differences between novices and relative experts; (b) these differences reflect meaningful differences the discourse; and (c) analysis of global correlation structure can identify significant differences between novices and relative experts. We assess whether an approach that models local relationships among concepts in a domain provides useful information beyond what might be extracted from a more traditional modeling approach. Our results indicate that techniques that account for local correlation structure can identify patterns in discourse not reflected in global correlation structure.
Technology and the New Professionalization of Teaching
D. Williamson Shaffer, P. Nash, & A. R. Ruis
Teachers College Record 117, no. 12 (2015): 1-30.
By 2009, 99% of U.S. classrooms had access to computers, with an average ratio of 1.7 students per computer, and 40% of teachers report using computers often in their classrooms. However, while K-12 schools are investing more heavily in digital technologies, only a small fraction of this investment is going to instructional software (7%) and digital content (5%). Education policy leaders have called for increased investment in and use of digital learning technologies in K-12 education, which has significant professional implications for the 40% of teachers who use computers often and, perhaps more importantly, for the 60% who do not. This article explores for a broad audience the changing landscape of education in the digital age, the changing roles of teachers in a technology-rich education system, and the skills, knowledge, values, and ways of thinking that teachers will need to have to support students’ social, emotional, and intellectual development in a digital learning environment. This analytic essay reviews and synthesizes research on learning in a digital environment, providing a theoretical framework for understanding the changing landscape of learning in technology-rich environments and the consequent changes in teacher preparation that this may entail. We explore the influence of educational technologies on teaching and teacher preparation by looking at three kinds of learning technology: digital workbooks that help students learn basic skills through routine practice; digital texts, such as ebooks, virtual museums, and learning games, that provide students with mediated experiences; and digital internships that simulate real-world practices, helping students learn how to solve problems in the ways that workers, scholars, and artists in the real world do. We examine the extent to which these technologies can assume different aspects of teachers’ traditional functions of assessment, tutoring, and explication. We argue that increased use of these and other digital learning technologies could allow teachers to provide more nuanced curricula based on their students’ individual needs. In particular, teachers will likely assume a new role, that of a coordinator who provides guidance through and facilitation of the learning process in individual students’ social, intellectual, and emotional contexts. We suggest this may require changes to teacher preparation and in-service professional development to help both new and experienced teachers succeed in an ever-changing digital learning environment, as well as new methods of evaluating teacher performance that account for more than student achievement on standardized tests.
“The Penny Lunch Has Spread Faster Than the Measles”: Children’s Health and the Debate over School Lunches in New York City, 1908-1930
A. R. Ruis
History of Education Quarterly 55, no. 2 (2015): 190-217.
A few days before Thanksgiving in 1908, the home economist Mabel Hyde Kittredge initiated a school lunch program at an elementary school in Hell’s Kitchen, serving soup and bread to hungry children in the infamous Manhattan neighborhood. The following year, she founded the School Lunch Committee (SLC), a voluntary organization composed of home economists, educators, physicians, and philanthropists dedicated to improving the nutritional health and educational prospects of schoolchildren. By 1915, the SLC was serving 80,000 free or low-price lunches a year to children at nearly a quarter of the elementary schools in Manhattan and the Bronx. Sparse but compelling evidence indicated that the program had reduced malnourishment among the children who partook, and teachers and principals at participating schools reported reductions in behavioral problems, dyspepsia, inattentiveness, and lethargy. With the hope of expanding the service and making it a permanent function of New York City’s public schools, the SLC transferred control to the Board of Education in 1919. Despite the success of the pilot program and the availability of public funding earmarked to maintain and even expand school lunch provision, the Board drastically reduced meal service. What had been a carefully planned and executed school health initiative was mostly replaced by a for-profit concessionaire system with no public health or educational mandate, no nutritional requirements, no food safety inspections, no reduced-price or free meals for poor children, and virtually no oversight of any kind. It is overly simplistic to regard the Board’s abdication of a popular health, education, and social welfare program as a government agency’s callous indifference to the needs of the poor. Because school meals were a matter of public policy in numerous domains, including health, education, labor, law, and social welfare, what the SLC regarded as a simple transfer from private charity to public entitlement was in fact a socially and politically charged negotiation of responsibility for children’s nutritional health and the proper role of the public school.
Pomegranate and the Mediation of Balance in Early Medicine
A. R. Ruis
Gastronomica: The Journal of Critical Food Studies 15, no. 1 (2015): 22-33.
Different elements of the pomegranate, both tree and fruit, had a wide range of uses in pre-modern therapeutics. Pomegranate also had a rich symbolic role in the art, literature, and religion of numerous cultures. In nearly every part of the globe where the pomegranate grew, it came to represent fundamental dualities: life and death, inside and out, many and one. The medicinal purposes for which healers recommended pomegranate at times reflected broader symbolic associations, and those associations are an important part of the therapeutic tradition. The dualistic symbolism that attended the pomegranate in various cultural traditions synergized with dualistic medical concepts, reinforcing the therapeutic power of pomegranate in otherwise diverse contexts. Reflecting this duality, pomegranate was both an astringent and a laxative, an emmenagogue and an antimenorrhagic, an expectorant and an antiemetic, a pyrogen and an febrifuge, a restorative and a soporific. In both literary and medical traditions, the pomegranate mediated transitions—or maintained balance—between opposing states. This essay provides an overview of the rich and sundry uses of pomegranate in pre-modern therapeutics, revealing how cultural associations both reflected and informed medical practices.
A Novel Paradigm for Engineering Education: Virtual Internships with Individualized Mentoring and Assessment of Engineering Thinking
N. C. Chesler, A. R. Ruis, W. Collier, Z. Swiecki, G. Arastoopour, & D. Williamson Shaffer
Journal of Biomechanical Engineering 137, no. 2 (2015): 024701-1–8.
Engineering virtual internships are a novel paradigm for providing authentic engineering experiences in the first year curriculum.They are both individualized and accommodate large numbers of students. As we describe in this report, this approach can (a) enable students to solve complex engineering problems in a mentored, collaborative environment; (b) allow educators to assess engineering thinking; and (c) provide an introductory experience that students enjoy and find valuable. Furthermore, engineering virtual internships have been shown to increase students’—and especially women’s—interest in and motivation to pursue engineering degrees. When implemented in first-year engineering curricula more broadly, the potential impact of engineering virtual internships on the size and diversity of the engineering workforce could be dramatic.
Authoring Networked Learner Models in Complex Domains
D. Williamson Shaffer, A. R. Ruis, & A. C. Graesser
Design Recommendations for Intelligent Tutoring Systems: Volume 3 – Authoring Tools and Expert Modeling Techniques, eds. R. Sottilare, A. C. Graesser, X. Hu, & K. Brawner (Orlando: U.S. Army Research Laboratory, 2015), 179–191.
Developing cost-effective authoring tools that enable non-programmers to create sophisticated virtual learning environments has been described as the “holy grail” of authorware design. Such tools must account for essential components, such as conversation management, semantic representations, production rules, and pedagogical strategies. The content produced must conform to theory-driven constraints, discourse processes, cognitive science, and computer science, along with practical constraints such as state standards, education policies, and standardized tests. Can all of this occur without requiring the user to have expertise in computer programming or educational software development? While progress has been made toward this goal, most sophisticated authoring systems (there are many for intelligent tutoring systems alone) are used primarily in research contexts. Those that have received broader usage, such as Cognitive Tutor Authoring Tools (CTAT) and Authoring Software Platform for Intelligent Resources in Education (ASPIRE), primarily support the development of modules that help students learn to solve well-formed problems, such as those common in basic mathematics, computer science, or language acquisition. In this chapter, we discuss the potential to develop authorware for learning environments in which students solve complex, ill-formed problems. In particular, we explore the design parameters of authoring tools for Syntern virtual internships, online learning environments that simulate professional practica in complex domains such as engineering design and urban planning. Given the relatively small body of research on the processes with which curriculum developers design content, we argue that a key element of developing authorware is to develop a science of the pedagogical authoring process.
“Children with Half-Starved Bodies” and the Assessment of Malnutrition in the United States, 1890-1950
A. R. Ruis
Bulletin of the History of Medicine 87, no. 3 (2013): 380-408.
Malnutrition was one of the most significant children’s health issues of the early twentieth century, but it also engendered considerable controversy. Just how many children were truly malnourished, and how could they be reliably identified? Despite the failures of numerous diagnostic methods—even the definition of malnutrition defied consensus—health authorities remained convinced that malnutrition was a serious and widespread problem. Indeed, the imprecision that surrounded the condition allowed it to be used metaphorically to advance a broad range of professional, social, and public health agendas. By the 1940s, due in part to the lack of reliable diagnostic methods, public health nutrition policy shifted abruptly from one of assessment, based on mass surveillance and individualized care, to one of management, based on a universal program of nutrition education, fortification of foods, and food security that treated all children as in need of nutritional assistance.
Nutrition Classes and Clinics
A. R. Ruis
The Oxford Encyclopedia of Food and Drink in America, 2nd Edition, ed. Andrew F. Smith (Oxford: Oxford University Press, 2012), 723-25.
The nutrition class, also know as the nutrition clinic, helped undernourished children to achieve and to maintain good health through a combination of routine medical examination and care, supplemental feeding, instruction in foods and nutrition, and social work. Along with other public health nutrition initiatives developed during the Progressive Era, such as school meal programs, anthropometric assessment of nutritional health, and extension work in foods and nutrition, nutrition classes were a response to public and professional concern about malnutrition in the first decades of the 20th century.
The Schism Between Medical and Public Health Education: A Historical Perspective
A. R. Ruis & R. N. Golden
Academic Medicine 83, no. 12 (2008): 1153-57.
The separation of “medicine” and “public health” in academic institutions limits the potential synergies that an integrated educational model could offer. The roots of this separation are deeply imbedded in history. During the past two centuries, there have been repeated efforts to integrate public health education into the core training of physicians, usually in response to a perceived short-term crisis, and without widespread, lasting success. The cost of additional public health instruction and the “overcrowding” of the medical curriculum have been cited as obstacles for creating an integrated medical/public health curriculum for more than a century. Several thoughtful and prescient proposals for integration were developed at a conference convened by the Rockefeller Foundation in the early 20th century, but not all were implemented. Today, there is growing recognition of the considerable value afforded by the integration of medicine and public health education. Many schools have responded to a national call for a renewed relationship between medicine and public health by increasing the availability of MD/MPH programs and/or by incorporating one or more public health courses into the basic medical curriculum. A few schools have created more substantial and innovative changes. Review and consideration of the history and politics of past efforts may serve as a guide for the development of successful new approaches to creating a clinical workforce that incorporates the principles of both clinical medicine and public health.