Publications

In this paper, we describe iPlan, a web-based software platform for constructing localized, reduced-form models of land-use impacts, enabling students, civic representatives, and others without specialized knowledge of land-use planning practices to explore and evaluate possible solutions to complex, multi-objective land-use problems in their own local contexts.

This chapter briefly explores several dimensions along which gender was operant in the emergence of public health nutrition in the early twentieth century, a formative period in the development of nutrition science, pediatrics, and school-based health services more generally in the United States. The nutrition class—a comprehensive program for undernourished children offered in dispensaries, schools, and other contexts during the 1910s and 1920s—forms a focal case study for exploring how public health nutrition initiatives engendered conceptions of nutrition differently for boys and girls and how such programs revealed gender-based fault lines in the professionalization of public health nutrition.

A key goal of quantitative ethnographic (QE) models, and statistical models more generally, is to produce the most parsimonious model that adequately explains or predicts the phenomenon of interest. In epistemic network analysis (ENA), for example, this entails constructing network models with the fewest number of codeswhose interaction structure provides sufficient explanatory power in a given context. Unlike most statistical models, however, modification of ENA models can affect not only the statistical properties but also the interpretive alignment between quantitative features and qualitative meaning that is a central goal in QE analyses. In this study, we propose a novel method, Parsimonious Removal with Interpretive Alignment, for systematically identifying more parsimonious ENA models that are likely to maintain interpretive alignment with an existing model. To test the efficacy of the method, we implemented it on a well-studied dataset for which there is a published, validated ENA model, and we show that the method successfully identifies reduced models likely to maintain explanatory power and interpretive alignment.

While quantitative ethnographers have used epistemic network analysis (ENA) to model trajectories that show change in network structure over time, visualizing trajectory models in a way that facilitates accurate interpretation has been a significant challenge. As a result, ENA has predominantly been used to construct aggregate models, which can obscure key differences in how network structures change over time. This study reports on the development and testing of a new approach to visualizing ENA trajectories. It documents the challenges associated with visualizing ENA trajectory models, the features constructed to address those challenges, and the design decisions that aid in the interpretation of trajectory models. To test this approach, we compare ENA trajectory models with aggregate models using a dataset with previously published results and known temporal features. This comparison focuses on interpretability and consistency with prior qualitative analysis, and we show that ENA trajectories are able to represent information unavailable in aggregate models and facilitate interpretations consistent with qualitative findings. This suggests that this approach to ENA trajectories is an effective tool for representing change in network structure over time.

Quantitative ethnographic models are typically constructed using qualitative data that has been segmented and coded. While there exist methodological studies that have investigated the effects of changes in coding on model features, the effects of segmentation have received less attention. Our aim was to examine, using a dataset comprised of narratives from semi-structured interviews, the effects of different segmentation decisions on population- and individual-level model features via epistemic network analysis. We found that while segmentation choices may not affect model features overall, the effects on some individual networks can be substantial. This study demonstrates a novel method for exploring and quantifying the impact of segmentation choices on model features.

Coding data defining concepts and identifying where they occur in data is a critical aspect of qualitative data analysis, and especially so in quanti-tative ethnography. Coding is a central process for creating meaning from data, and while much has been written about coding methods and theory, relatively little has been written about what constitutes best practices for fair and valid coding, what justifies those practices, and how to implement them. In this paper, our goal is not to address these issues comprehensively, but to provide guidelines for good coding practice and to highlight some of the issues and key questions that quantitative ethnographers and other re-searchers should consider when coding data.

Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. However, current methods for measuring the CPS processes of individuals, such as coding and counting, treat these processes as sets of isolated and independent events. In contrast, Epistemic Network Analysis (ENA) models how the contributions of a given individual relate to the contributions of others. This article examines the communications of air defense warfare teams from an experiment comparing two different computer-based decision support systems, using this data to ask whether ENA provides a more ecologically valid quantitative model of CPS than coding and counting. Qualitative analysis showed that commanders using one system asked questions to understand the tactical situation, while commanders using an experimental system focused more on actions in response to the tactical situation. Neither of the coding and counting approaches we tested corroborated these findings with statistically significant results. In contrast, ENA created models of the individual contributions of commanders that (a) showed statistical differences between commanders using the two systems to corroborate the qualitative analysis, and (b) revealed differences in individual performance. This suggests that ENA is a more powerful tool for CPS assessment than coding and counting approaches.

Background: This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate skills and knowledge through discourse.
Methods: Senior residents (N = 11) were recorded while performing a simulated laparoscopic ventral hernia (LVH) repair. Audio transcripts were coded for five discourse elements related to knowledge, skills, and operative independence. Epistemic network analysis was used to model the ordered integration of the five discourse elements.
Results: Participants with poorer hernia repair outcomes had stronger connections between the discourse elements operative planning and asking for information or advice (Operative planning), while participants with better hernia repair outcomes had stronger connections between the discourse elements giving assistant instructions and identifying errors (Operative management): (p = .006; Cohen’s d = 2.79).
Conclusion: Participants with better hernia repair outcomes engaged in more operative management communication during the simulated procedure. This ability to integrate multiple operative steps and verbally communicate them significantly correlated with better operative outcomes.

Simulators have been widely adopted to help surgical trainees learn procedural rules and acquire basic psychomotor skills, and research indicates that this learning transfers to clinical practice. However, few studies have explored the use of simulators to help more advanced learners improve their understanding of operative practices. To model how surgeons with different levels of experience use procedural simulators, we conducted a quantitative ethnographic analysis of small-group conversations in a continuing medical education short course on laparoscopic hernia repair. Our research shows that surgeons who had less experience with laparoscopic surgery tended to use the simulators to learn and rehearse the basic procedures, while more experienced surgeons used the simulators as a platform for exploring a range of hernia presentations and operative approaches based on their experiences. Thus simple, inexpensive simulators may be effective with both novice and more experienced learners.

Learning-in-action depends on interactions with learning content, peers and real world problems. However, effective learning-in-action also depends on the extent to which students are active-in-thinking, making meaning of their learning experience. A critical component of any technology to support active thinking is the ability to ascertain whether (or to what extent) students have succeeded in internalizing the disciplinary strategies, norms of thinking, discourse practices and habits of mind that characterize deep understanding in a domain. This presents what we call a dilemma of modeling-incontext: teachers routinely analyze this kind of thinking for small numbers of students in activities they create or customize for the needs of their students; however, doing so at scale and in real-time requires some automated processes for modeling student work. Current techniques for developing models that reflect specific pedagogical activities and learning objectives that a teacher might create require either more expertise or more time than teachers have. In this paper, we examine a theoretical approach to addressing the problem of modeling active thinking in its pedagogical context that uses teachercreated rubrics to generate models of student work. The results of this examination show how appropriately constructed learning technologies can enable teachers to develop custom automated rubrics for modeling active thinking and meaning-making from the records of students’ dialogic work.

Complex, collaborative thinking is often conceptualized as a process of developing cognitive connections among the contributions of different participants. A central problem in modeling collaboration in this way is thus determining, for any contribution to a discussion, the appropriate context for modeling the connections being made—that is, for determining the appropriate recent temporal context. Recent temporal context is typically defined using a moving window of fixed length. However, that length is dependent on the setting, and there are no existing methods for reliably determining an appropriate window length. This paper presents an empirical method for measuring recent temporal context, and thus for defining an appropriate window length to be used in analyses of complex, collaborative thinking. Importantly, the method we describe minimizes the need for human annotation while providing both qualitative and quantitative warrants for choosing a particular window length.

Many researchers have argued that models of collaborative processes should account for temporality, but there exist different approaches for doing so. We compared two specific approaches to modeling collaborative processes in a CSCL context: Epistemic Network Analysis, which models events cotemporally (unordered and temporally proximate), and Sequential Pattern Mining, which models events sequentially (ordered and temporally proximate). Our results suggest that in this context cotemporal models constructed with Epistemic Network Analysis outperform sequential models constructed with Sequential Pattern Mining in terms of (a) explanatory power, (b) efficiency, and (c) interpretability.

In this chapter, I attempt to model the concept of “nutrition” in English-language sources from the 19th and 20th centuries using epistemic network analysis (ENA), a set of techniques for measuring, visualizing, and comparing patterns of association among conceptual elements. In doing so, I argue that conceptual networks can help us understand macrohistorical patterns in discourses&emdash;in this case, discourses of nutrition&emdash;without sacrificing microhistorical rigor. Specifically, I will describe an approach in which microhistorical analyses inform the development of macrohistorical models that in turn suggest new avenues for microhistorical investigation.

In this chapter, we reflect on the use of epistemic network analysis (ENA) as a tool for modeling conceptual networks. Because there are a number of resources that explain ENA in great detail as a technique and a tool, we will not discuss how to use ENA but rather explore why and how a historian might find the approach useful. Following this, we explore some of the issues with which the historian must engage in order to move from a strictly human, qualitative methodology to a mixed-methods approach that includes ENA. While digital humanities papers commonly include a methods section, these final products tend not to reflect on the complexity of the methodological process that got the authors to that stage, to talk openly about which data models failed, or to reflect on the limitations of tools they previously considered and rejected. This paper is intentionally focused on this “work in progress” stage that all historians go through, and which newcomers to the digital humanities can find isolating. Using a case study approach&emdash;applying ENA to a seventeenth-century archival collection of letters known as the Hartlib Papers&emdash;we will consider the kinds of intellectual and theoretical challenges historians may grapple with as they try to think about their source materials as a dataset and supplement their qualitative analyses with quantitative models.

Background
This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate psychomotor, procedural, and cognitive skills to manage errors.

Methods
Audio-video data were collected from general surgery residents (N = 45) performing a simulated laparoscopic ventral hernia repair. Errors were identified using a standard checklist, and speech was coded for elements related to error recognition and management. Epistemic network analysis (ENA) was used to model the integration of error management skills.

Results
There was no correlation between number or type of errors committed and operative outcome. However, ENA models showed significant differences in the integration of error management skills between high-performing and low-performing residents.

Conclusion
These results suggest that error checklists and surgeons’ speech can be used to model the integration of psychomotor, procedural, and cognitive aspects of intraoperative performance. Moreover, ENA can identify and quantify this integration, providing insight on performance gaps in both individuals and populations.

Both military and civilian work is increasingly organized around small, dynamic teams rather than large bureaucratic frameworks. Such teams combine individuals with highly specific skill sets and extensive training to solve complex, non-standard problems, often under extreme pressure. Importantly, these teams are not typically composed of interchangeable members but are formed and trained as expert teams to function semi-autonomously. To accomplish mission objectives in the face of complex challenges, the United States military needs to develop teams that consistently exhibit high levels of taskwork and teamwork skills. Implementing principles of team training and maintaining team efficacy are critical in the armed forces, where teams are often widely dispersed and consequences for underperformance can be severe, though many civilian teams—such as hospital trauma teams or flight crews—face similar challenges. Establishing and maintaining high levels of team performance in these contexts requires the creation of practical and effective team development interventions, including team training, as well as systems for ongoing assessment of team function. We argue that one critical component of training, monitoring, and maintaining high-functioning teams is the ability to model team performance.

Analyses of learning based on student discourse need to account not only for the content of the utterances but also for the ways in which students make connections across turns of talk. This requires segmentation of discourse data to define when connections are likely to be meaningful. In this paper, we present an approach to segmenting data for the purposes of modelling connections in discourse using epistemic network analysis. Specifically, we use epistemic network analysis to model connections in student discourse using a temporal segmentation method adapted from recent work in the learning sciences. We compare the results of this study to a purely conversation-based segmentation method to examine the affordances of temporal segmentation for modelling connections in discourse.

This book explores the origins of the National School Lunch Program, the most extensive and longest-running children’s nutrition program in U.S. history. In doing so, it reveals how and why school meals came to have the form that is so familiar to most Americans. While the National School Lunch Program was in part a response to specific concerns about children’s access to sufficient and sufficiently nourishing food during the Great Depression and World War II, many of the issues that shaped school meal policies and practices arose decades earlier. By exploring the origins of school meal initiatives, this book endeavors to explain why it was (and to some extent, has continued to be) so difficult to establish meal programs that satisfy the often competing interests of children, parents, schools, health authorities, politicians, and the food industry. This is accomplished through careful studies of several regions and detailed analysis of the policies and politics that governed the development of school meal programs, first at the local level, and then at the federal level. In doing so, the book shows how the early history of school meal program development helps us understand contemporary debates over changes to school lunch policies, including those inaugurated by the passage of the Healthy, Hunger-Free Kids Act in 2010.

Researchers use Inter-Rater Reliability (IRR) to measure whether two processes—people and/or machines—identify the same properties in data. There are many IRR measures, but regardless of the measure used, there is a common method for estimating IRR. To assess the validity of this common method, we conducted Monte Carlo simulation studies examining the most widely used measure of IRR: Cohen’s kappa. Our results show that the method commonly used by researchers to assess IRR produces unacceptable Type I error rates.

In this chapter, we provide a worked example of a theory-based approach to learning analytics in the context of an educational game. We do this not to provide an ideal solution for others to emulate, but rather to explore the affordances of a theory-based—rather than data-driven—approach. We do so by presenting (1) epistemic frame theory as an approach to the conceptualization of learning; (2) data from an epistemic game, an approach to educational game design based on epistemic frame theory; and (3) epistemic network analysis (ENA), a technique for analyzing discourse and other data for evidence of complex thinking based on the same theory. We describe ENA through a specific analytic result, but our aim is to explore how this result exemplifies what we consider a key “best practice” in the field of learning analytics.

The practice of historical research stands to benefit considerably from, and in a growing number of cases requires, a mixed-methods approach that combines the qualitative and the quantitative and incorporates the analytic strengths of human interpretation and computational processing. In this brief reflection, we explore one approach to mixed-methods history using network analysis: various statistical techniques with which the structure of connections among entities—people, places, concepts, and so forth—can be modeled.

This paper provides a tutorial on epistemic network analysis (ENA), a novel method for identifying and quantifying connections among elements in coded data and representing them in dynamic network models. Such models illustrate the structure of connections and measure the strength of association among elements in a network, and they quantify changes in the composition and strength of connections over time. Importantly, ENA enables comparison of networks both directly and via summary statistics, so the method can be used to explore a wide range of qualitative and quantitative research questions in situations where patterns of association in data are hypothesized to be meaningful. While ENA was originally developed to model cognitive networks—the patterns of association between knowledge, skills, values, habits of mind, and other elements that characterize complex thinking—ENA is a robust method that can be used to model patterns of association in any system characterized by a complex network of dynamic relationships among a relatively small, fixed set of elements.

• Education and training of health care workers is fundamental to infection prevention.
• A systems approach is needed for optimal health care–associated infection prevention interventions.
• A cognitive simulation, allowing the health care workers to adopt a multidimensional perspective, should be tested for efficacy in health care settings.

One of the goals of engineering education is providing students with authentic and meaningful design experiences as well as assessing the development of design thinking. In this paper, we review virtual internships, online simulations of 21^st-century engineering design practice, as one method for providing students with authentic experiences. To assess the development of design thinking in virtual internships, we used epistemic network analysis (ENA), a tool for measuring complex thinking as it develops over time. We provide an example of how ENA can be used to measure quantitatively student teams’ qualitative discourse in a virtual internship program in order to assess performance in engineering. The combination of virtual internships and ENA provides opportunities for students to engage in authentic engineering design, receive concurrent feedback on their design thinking, and develop the identity, values, and ways of thinking of 21^st-century professional engineers.

This paper examines techniques for modeling relationships among domain concepts and practices in discourse to assess learning in a CSCL environment. We compare two approaches: a traditional psychometric approach, which models the global correlation structure of student discourse markers across the learning intervention, and a model that accounts for the local correlation structure of discourse markers within activities. We investigate whether: (a) analysis of local correlation structure can identify significant differences between novices and relative experts; (b) these differences reflect meaningful differences the discourse; and (c) analysis of global correlation structure can identify significant differences between novices and relative experts. We assess whether an approach that models local relationships among concepts in a domain provides useful information beyond what might be extracted from a more traditional modeling approach. Our results indicate that techniques that account for local correlation structure can identify patterns in discourse not reflected in global correlation structure.

By 2009, 99% of U.S. classrooms had access to computers, with an average ratio of 1.7 students per computer, and 40% of teachers report using computers often in their classrooms. However, while K-12 schools are investing more heavily in digital technologies, only a small fraction of this investment is going to instructional software (7%) and digital content (5%). Education policy leaders have called for increased investment in and use of digital learning technologies in K-12 education, which has significant professional implications for the 40% of teachers who use computers often and, perhaps more importantly, for the 60% who do not. This article explores for a broad audience the changing landscape of education in the digital age, the changing roles of teachers in a technology-rich education system, and the skills, knowledge, values, and ways of thinking that teachers will need to have to support students’ social, emotional, and intellectual development in a digital learning environment. This analytic essay reviews and synthesizes research on learning in a digital environment, providing a theoretical framework for understanding the changing landscape of learning in technology-rich environments and the consequent changes in teacher preparation that this may entail. We explore the influence of educational technologies on teaching and teacher preparation by looking at three kinds of learning technology: digital workbooks that help students learn basic skills through routine practice; digital texts, such as ebooks, virtual museums, and learning games, that provide students with mediated experiences; and digital internships that simulate real-world practices, helping students learn how to solve problems in the ways that workers, scholars, and artists in the real world do. We examine the extent to which these technologies can assume different aspects of teachers’ traditional functions of assessment, tutoring, and explication. We argue that increased use of these and other digital learning technologies could allow teachers to provide more nuanced curricula based on their students’ individual needs. In particular, teachers will likely assume a new role, that of a coordinator who provides guidance through and facilitation of the learning process in individual students’ social, intellectual, and emotional contexts. We suggest this may require changes to teacher preparation and in-service professional development to help both new and experienced teachers succeed in an ever-changing digital learning environment, as well as new methods of evaluating teacher performance that account for more than student achievement on standardized tests.

A few days before Thanksgiving in 1908, the home economist Mabel Hyde Kittredge initiated a school lunch program at an elementary school in Hell’s Kitchen, serving soup and bread to hungry children in the infamous Manhattan neighborhood. The following year, she founded the School Lunch Committee (SLC), a voluntary organization composed of home economists, educators, physicians, and philanthropists dedicated to improving the nutritional health and educational prospects of schoolchildren. By 1915, the SLC was serving 80,000 free or low-price lunches a year to children at nearly a quarter of the elementary schools in Manhattan and the Bronx. Sparse but compelling evidence indicated that the program had reduced malnourishment among the children who partook, and teachers and principals at participating schools reported reductions in behavioral problems, dyspepsia, inattentiveness, and lethargy. With the hope of expanding the service and making it a permanent function of New York City’s public schools, the SLC transferred control to the Board of Education in 1919. Despite the success of the pilot program and the availability of public funding earmarked to maintain and even expand school lunch provision, the Board drastically reduced meal service. What had been a carefully planned and executed school health initiative was mostly replaced by a for-profit concessionaire system with no public health or educational mandate, no nutritional requirements, no food safety inspections, no reduced-price or free meals for poor children, and virtually no oversight of any kind. It is overly simplistic to regard the Board’s abdication of a popular health, education, and social welfare program as a government agency’s callous indifference to the needs of the poor. Because school meals were a matter of public policy in numerous domains, including health, education, labor, law, and social welfare, what the SLC regarded as a simple transfer from private charity to public entitlement was in fact a socially and politically charged negotiation of responsibility for children’s nutritional health and the proper role of the public school.

Different elements of the pomegranate, both tree and fruit, had a wide range of uses in pre-modern therapeutics. Pomegranate also had a rich symbolic role in the art, literature, and religion of numerous cultures. In nearly every part of the globe where the pomegranate grew, it came to represent fundamental dualities: life and death, inside and out, many and one. The medicinal purposes for which healers recommended pomegranate at times reflected broader symbolic associations, and those associations are an important part of the therapeutic tradition. The dualistic symbolism that attended the pomegranate in various cultural traditions synergized with dualistic medical concepts, reinforcing the therapeutic power of pomegranate in otherwise diverse contexts. Reflecting this duality, pomegranate was both an astringent and a laxative, an emmenagogue and an antimenorrhagic, an expectorant and an antiemetic, a pyrogen and an febrifuge, a restorative and a soporific. In both literary and medical traditions, the pomegranate mediated transitions—or maintained balance—between opposing states. This essay provides an overview of the rich and sundry uses of pomegranate in pre-modern therapeutics, revealing how cultural associations both reflected and informed medical practices.

Engineering virtual internships are a novel paradigm for providing authentic engineering experiences in the first year curriculum.They are both individualized and accommodate large numbers of students. As we describe in this report, this approach can (a) enable students to solve complex engineering problems in a mentored, collaborative environment; (b) allow educators to assess engineering thinking; and (c) provide an introductory experience that students enjoy and find valuable. Furthermore, engineering virtual internships have been shown to increase students’—and especially women’s—interest in and motivation to pursue engineering degrees. When implemented in first-year engineering curricula more broadly, the potential impact of engineering virtual internships on the size and diversity of the engineering workforce could be dramatic.

Malnutrition was one of the most significant children’s health issues of the early twentieth century, but it also engendered considerable controversy. Just how many children were truly malnourished, and how could they be reliably identified? Despite the failures of numerous diagnostic methods—even the definition of malnutrition defied consensus—health authorities remained convinced that malnutrition was a serious and widespread problem. Indeed, the imprecision that surrounded the condition allowed it to be used metaphorically to advance a broad range of professional, social, and public health agendas. By the 1940s, due in part to the lack of reliable diagnostic methods, public health nutrition policy shifted abruptly from one of assessment, based on mass surveillance and individualized care, to one of management, based on a universal program of nutrition education, fortification of foods, and food security that treated all children as in need of nutritional assistance.

The nutrition class, also know as the nutrition clinic, helped undernourished children to achieve and to maintain good health through a combination of routine medical examination and care, supplemental feeding, instruction in foods and nutrition, and social work. Along with other public health nutrition initiatives developed during the Progressive Era, such as school meal programs, anthropometric assessment of nutritional health, and extension work in foods and nutrition, nutrition classes were a response to public and professional concern about malnutrition in the first decades of the 20^th century.

The separation of “medicine” and “public health” in academic institutions limits the potential synergies that an integrated educational model could offer. The roots of this separation are deeply imbedded in history. During the past two centuries, there have been repeated efforts to integrate public health education into the core training of physicians, usually in response to a perceived short-term crisis, and without widespread, lasting success. The cost of additional public health instruction and the “overcrowding” of the medical curriculum have been cited as obstacles for creating an integrated medical/public health curriculum for more than a century. Several thoughtful and prescient proposals for integration were developed at a conference convened by the Rockefeller Foundation in the early 20^th century, but not all were implemented. Today, there is growing recognition of the considerable value afforded by the integration of medicine and public health education. Many schools have responded to a national call for a renewed relationship between medicine and public health by increasing the availability of MD/MPH programs and/or by incorporating one or more public health courses into the basic medical curriculum. A few schools have created more substantial and innovative changes. Review and consideration of the history and politics of past efforts may serve as a guide for the development of successful new approaches to creating a clinical workforce that incorporates the principles of both clinical medicine and public health.