Not least because writing, as a subject of research, straddles various disciplines – including linguistics, educational science and psychology – its empirical study is complex. Challenges become particularly apparent when high standards regarding psychometric and (inference) statistical quality criteria clash with limited feasibility in the context of (higher) education. The same applies, however, when theoretical models lose their relevance and interpretability due to insufficient integration into subject-specific didactics. Accordingly, these challenges can be systematised using a four-quadrant matrix that combines two axes: (1) methodological rigour versus feasibility, and (2) theoretical core versus subject-specific didactic integration.
In the methodological rigour × theoretical core quadrant, the focus is on developing conceptually precise and psychometrically robust operationalisations of central writing constructs (Steinhoff, Grabowski, & Becker-Mrotzek, 2017). The guiding question here is which aspects of writing remain difficult to assess whilst maintaining high standards of quality, and where quantitative and qualitative methods reach their limits. By way of example, a distinction is made between ‘hard’ factors (e.g. writing fluency, lexical or syntactic features, text structure) and ‘soft’ factors (e.g. motivation, self-efficacy, creativity), which can often only be measured indirectly and consequently raise issues of validity and comparability.
In the methodological rigour × subject-specific didactic embedding quadrant, the focus is on the application of sophisticated measurement and evaluation frameworks to didactically meaningful task formats and classroom arrangements (Jost, 2017). The key question is how core empirical quality criteria can be ensured without under-specifying theoretical constructs or compromising didactic fit (see, for example, Wild, 2023).
The ‘feasibility × theoretical core’ quadrant addresses the consequences of institutional and organisational constraints within the higher education sector for theory-based research. The central question is which research designs are viable under real-world conditions and how theoretical precision can be ensured despite pragmatic limitations (Grabowski, 2017; Rhoads, 2016). Typical challenges include, for example, a limited number of measurement time points, restricted rater resources or a short intervention period.
In the feasibility × subject-specific didactic embedding quadrant, the focus is on the development and evaluation of practice-oriented interventions under everyday conditions, whilst simultaneously specifying the underlying mechanisms of action. The key question is how measures that are compatible with subject-specific didactics can be implemented and evaluated without the assumptions regarding their effects remaining vague (cf. Jeuk, Schilcher, & Stöger et al., 2017). Relevant challenges arise, amongst other things, from adherence to the intervention protocol, treatment fidelity and the context-dependence of effects.
The aim is to develop a jointly supported framework that articulates the key challenges in writing research in line with the four-field matrix and derives from this criteria for research designs that are evidence-based, suitable for practical application and compatible with subject-specific didactics.
Grabowski, J. (2017). Requirements for research designs. In M. Becker-Mrotzek, J. Grabowski & T. Steinhoff (Eds.), Research Handbook of Empirical Writing Pedagogy (pp. 315–334). Waxmann.
Jeuk, S., Schilcher, A., Stöger, H. et al. (2017): Challenges and opportunities in collaboration between research and educational institutions. On the difficulty of collecting data in schools. In Mercator Institute for Language Promotion and German (eds.), << LOOKING BACK TO MOVE FORWARD >>. Perspectives on language education in teacher training and research (pp. 21–24). Mercator Institute.
Jost, J. (2017). Principles and methods of writing environments that promote learning. In M. Becker-Mrotzek, J. Grabowski, & T. Steinhoff (Eds.), Research Handbook on Empirical Writing Didactics (pp. 173–186). Waxmann.
Rhoads, C. (2016). The Implications of Contamination for Educational Experiments With Two Levels of Nesting. Journal of Research on Educational Effectiveness, 9(4), 531–555. doi.org/10.1080/19345747.2015.1086912
Steinhoff, T., Grabowski, J., & Becker-Mrotzek, M. (2017). Challenges in empirical writing didactics. In M. Becker-Mrotzek, J. Grabowski & T. Steinhoff (Eds.), Research Handbook on Empirical Writing Pedagogy (pp. 9–24). Waxmann.
Wild, J. (2023). Assessing written narrative. Results of the empirical validation of an instrument for assessing the quality of narrative written texts within the framework of the RESTLESS project. In N. Kruse, V. Lemke, T. Steinhoff, & A. Sturm (Eds.), Writing Instruction: Studies and Discourses on Writing and Text Production (pp. 271–289). Waxmann.