Research Methods and Design


The “big tent” of SoTL (Hutchings & Huber in Chick, 2014) suggests that the field is welcoming of scholars coming from where they are, bringing with them the epistemological frameworks and methodologies that define their disciplines. Any one example of published SoTL might look radically different from another: employing different methods (Felten, 2013), building on different theoretical frames (Hutchings & Huber, 2008), and conversing with different foundational literature (Chick et al., 2021). That diversity does mean, however, that while scholars agree that SoTL should be “methodologically sound” (Felten, 2013), defining what is meant by ‘sound-ness’ is more challenging.

Bernstein (2018) sought to define a “unifying methodology” for SoTL by examining what internal and external validity should mean in the field. He raised the question many scientists and social scientists entering the field ask: “does this work require a control group?” (Bernstein, 2018, p. 121). He argued that “where such designs can be implemented ethically and effectively, we would be fools not to use them” (p. 122) but that requiring control groups for all “good” SoTL would be akin to using a hammer in place of all tools. Instead, he recommended focusing on the questions:

  1. Is what I am doing having an impact?
  2. and “what can others learn from the work I am doing?” (Bernstein, 2018, p. 123)

Offering similarly foundational questions about process, Bloch-Schulman et al. (2016) advocated for a wider range of studies in the field—especially those that focus on the transformative potential of teaching and learning and include the “representational and interpretive tools rooted in the arts and humanities” alongside the more experimental or observational approaches common in the social sciences (p. 113).

So, what does this mean for actually conducting a SoTL study?

You might employ methods like content and discourse analysis, arts-based research, broad qualitative methods, mixed-methods, basic descriptive statistics, and comparative quantitative methods such as ANOVA. Chick’s (Ed.; 2018) SoTL in Action includes specific chapters on design alignment (Meuller), questionnaires (Maurer), classroom observations (Cerbin), interviews (Miller-Young), close reading (Manarin), and think-alouds (Calder). Friberg’s (2018) blog post on “Sources, Types, and Analysis of Data in SoTL” is another excellent resource. As we continue to develop this resource page, we plan to add SoTL study examples of each of these methods and more.


1

Case Studies

SoTL researchers might use case studies to nuance theories about learning or offer a suggestion for future practice.


Di Lauro, F., & Johinke, R. (2017). Employing Wikipedia for good not evil: Innovative approaches to collaborative writing assessment. Assessment & Evaluation in Higher Education, 42(3), 478–491. https://doi.org/10.1080/02602938.2015.1127322

This essay presented two case studies of Wikipedia-based student assessments to suggest how Wikipedia can be a valuable tool when incorporated thoughtfully in the classroom.


Guberman, D. (2021). Teaching intercultural competence through heavy metal music. Arts and Humanities in Higher Education, 20(2), 115–132. https://doi.org/10.1177/1474022220903403

This essay reflected on how AAC&U’s VALUE rubric for Intercultural Knowledge and Competence aligned with the author’s experiences teaching heavy metal music and in the arts and humanities more broadly.


2

Validated Measures

SoTL researchers might use or adapt pre-existing validated measures to assess student learning or experiences.


Calder, L., & Williams, R. (2021). Must History Students Write History Essays? Journal of American History, 107(4), 926–941. https://doi.org/10.1093/jahist/jaaa464

This essay used the Stanford-developed “History Assessments of Thinking” (HATS) to ask students to respond to historical sources. HATS provided a rubric to analyze those responses for characteristics of historical thinking.


Stover, S., & Holland, C. (2018). Student Resistance to Collaborative Learning. International Journal for the Scholarship of Teaching and Learning, 12(2). https://doi.org/10.20429/ijsotl.2018.120208

This essay used two different measures to quantitatively assess students’ sense of the classroom as a community of inquiry and to assess their course satisfaction in response to collaborative learning.


3

Observation

SoTL researchers might employ classroom observation approaches to gather direct evidence of student and/or instructor behaviors or interactions.


Narayanan, D., & Abbot, S. (2020). Increasing the participation of underrepresented minorities in STEM classes through student-instructor partnerships. In L. Mercer-Mapstone & S. Abbot (Eds.), The Power of Partnership: Students, Staff, and Faculty Revolutionizing Higher Education (pp. 181–195). Center for Engaged Learning Open Access Book Series. https://www.centerforengagedlearning.org/books/power-of-partnership/section-three/chapter-12/

This chapter used classroom observations to understand impacts of small pedagogical shifts on students’ participation in an astrophysics class.


Cruz, L., & Lindemann, J. (2022). Classroom Mapping: New Perspectives on Capturing Student Engagement in the Classroom. In E. Sengupta & P. Blessinger (Eds.), Innovations in Higher Education Teaching and Learning (pp. 43–59). Emerald Publishing Limited. https://doi.org/10.1108/S2055-364120220000044004

This chapter explored what classroom observation mapping can teach us about student engagement and inclusion.

Peter Felten (Elon University), Ketevan Kupatadze (Elon University), and Mathilde van der Merwe (University of Cape Town) share examples of Scholarship of Teaching and Learning Projects.

References

Bernstein, J. L. (2018). Unifying SoTL Methodology: Internal and External Validity. Teaching & Learning Inquiry, 6(2), 115–126. https://doi.org/10.20343/teachlearninqu.6.2.9

Bloch-Schulman, S., Conkling, S. W., Linkon, S. L., Manarin, K., & Perkins, K. (2016). Asking bigger questions: An invitation to further conversation. Teaching and Learning Inquiry, 4(1), Article 1. https://doi.org/10.20343/teachlearninqu.4.1.12

Chick, N. L. (2014). ‘Methodologically sound’ under the ‘big tent’: An ongoing conversation. International Journal for the Scholarship of Teaching and Learning, 8(2). https://doi.org/10.20429/ijsotl.2014.080201

Chick, N. L. (Ed.)(2018). SoTL in Action: Illuminating critical moments of practice. Stylus.*
*George Mason University instructors can access the text via Mason Libraries here.

Chick, N. L., Abbot, S., Mercer-Mapstone, L., Ostrowdun, C. P., & Grensavitch, K. (2021). Naming is power: Citation practices in SoTL. Teaching and Learning Inquiry, 9(2), Article 2. https://doi.org/10.20343/teachlearninqu.9.2.2

Felten, P. (2013). Principles of Good Practice in SoTL. Teaching & Learning Inquiry, 1(1), 121–125. https://doi.org/10.20343/teachlearninqu.1.1.121

Friberg, J. C. (2018, October 1). Sources, Types, and Analysis of Data in SoTL. The SoTL Advocate. https://illinoisstateuniversitysotl.wordpress.com/2018/10/01/sources-types-and-analysis-of-data-in-sotl/

Hutchings, P., & Taylor Huber, M. (2008). Placing theory in the scholarship of teaching and learning. Arts and Humanities in Higher Education, 7(3), 229–244. https://doi.org/10.1177/1474022208094409