Task difficulty is an important but complex phenomenon in Applied Linguistics, for which there is relatively little empirical research. This article discusses approaches to defining task difficulty ...and focuses on objective task difficulty derived from ratings of performances and on difficulty derived from an error count in the performances, thus taking errors as indicators of writing task difficulty. Errors are described in terms of the Scope–Substance error taxonomy in writing performances from the Slovene General Matura examination in English. The most frequent errors are located at word and phrase level. Generally, error frequency decreases as writing proficiency increases, but some error types do not conform to this trend. This is the case for punctuation errors, which gain prominence at higher levels of mastery. The results of this study are relevant for assessment, particularly for rating scale development or revision, and for rater training. They are equally relevant for teaching, since knowing sources of difficulty in tasks is a prerequisite for effective pedagogical action. More generally, if applied to performances based on a wider range of tasks, viewing errors as indicators of difficulty can lead to a better understanding of difficulty‐generating task features.
Povzetek
Težavnost nalog je pomemben, a zapleten pojav v uporabnem jezikoslovju, o katerem je razmeroma malo empiričnih raziskav. Prispevek obravnava pristope k opredeljevanju zahtevnosti nalog pri preverjanju pisnega sporočanja. Osredinjen je na njihovo objektivno zahtevnost, ki izhaja iz točkovnih ocen pisnih izdelkov in iz števila napak v izdelkih, kar pomeni, da napake obravnava kot kazalec zahtevnosti nalog. Napake opredeli po taksonomiji obseg‐vsebina in jo uporabi za ocenjevanje pisnih izdelkov na izpitu iz angleščine na slovenski splošni maturi. Najpogostejše napake se pojavljajo na ravni besed in besednih zvez; večinoma se njihova pogostnost zmanjša z naraščanjem pisne zmožnosti, vendar nekatere vrste napak ne sledijo temu trendu. Tak je primer napak v rabi ločil, ki so pogostejše na višjih ravneh znanja. Rezultati študije so pomembni za ocenjevanje, predvsem za razvoj in prenovo ocenjevalnih lestvic, pa tudi za usposabljanje ocenjevalcev. Dognanja so pomembna tudi za poučevanje, saj je razumevanje razlogov za zahtevnost predpogoj za učinkovito pedagoško delo. Širše gledano lahko napake uporabimo kot kazalec zahtevnosti tudi pri drugih vrstah nalog in s tem izboljšamo prepoznavanje in razumevanje tistih lastnosti nalog, ki prispevajo k njihovi zahtevnosti.
The aim of this pilot study is to examine how feedback type and error type influence the revision process of undergraduate writers. For the purpose of this study, text feedback was provided for eight ...essays, which were produced in a writing class for students who study English as a foreign language. Directive and non-directive feedback, differentiated by either providing or not providing students with suggestions for improvement, was distributed to approximately equal extents. The revised texts were then compared to the draft versions in order to analyse the students’ reactions to the feedback instances. Errors which triggered feedback were described in terms of the Scope – Substance taxonomy (Sigott, Cesnik, & Dobrić 2016). In this study, the focus was on Error Substance, i.e. the units of text that need to be modified, namely, text, paragraph, sentence, or word. The quantitative analysis of the feedback instances (n=90) has shown that written text feedback is generally highly effective, and that non-directive feedback leads to a more positive effect than directive feedback, particularly as far as lowerlevel errors (sentence or word) are concerned. The reasons for non-responses to feedback cannot be attributed to feedback type or error substance, which shows a distinct need for further qualitative analysis.
At secondary level, great importance is put in teaching and assessment on the ability to write texts that meet the conventions of text types. For the Austrian school system, descriptions of text ...types for German, English, French, Italian and Spanish are available. However, the text types differ across languages. This study focuses on English. It uses a set of 23 descriptive statements, derived from text linguistic theory, to empirically investigate their suitability to characterise the constructs held by practicing teachers for eight text types for L2 English. This approach has already been applied to German as the language of instruction (Sigott et al. 2020) and can be applied to all other languages. The results for English show that the suitability of the 23 statements varies considerably across text types. A subset of statements turns out to be suitable as dimensions for comparing all the English text types. In addition, statements are identified which are suitable to characterise individual text types and statements which should not be considered in describing text types and, therefore, not used in formulating assessment criteria. This is valuable information for teaching and relevant for assessment practice. These results are presented in a polarity profile for each text type.
Towards an error taxonomy for student writing Nikola Dobrić; Guenther Sigott
Zeitschrift für interkulturellen Fremdsprachenunterricht,
03/2015, Letnik:
19, Številka:
2
Journal Article
This article describes first attempts to identify sources of C-Test item difficulty. A distinction between test format factors and content factors is made. Ways of adjusting C-Test difficulty by ...changing the format factors are described. Three content factors are provisionally identified as sources of item difficulty. The article is intended to illustrate the construct identification approach to language test validation.
This article discusses problems inherent in the development of a language proficiency measure yielding scores which are comparable across languages. It first asks the question of what it means to ...know a language and discusses the relationship between the notion of "language proficiency" and the competence — performance dichotomy. Two frameworks for the description of communicative competence are presented, which reveal the complexity of the proficiency tester's task. Then two approaches to quantifying language proficiency are described and discussed under the headings of "discrete point tests" and "integrative tests". The concept of proficiency test equivalency across languages is analysed and problems in establishing equivalence are identified. More specifically, the psychometric behaviour of cloze as a crosslinguistic proficiency measure is examined. The main problem in the development of a proficiency measure with crosslinguistic applicability is to establish both content equivalence and equivalence with regard to difficulty. Test equivalence across languages can be achieved through translation and sampling. Tests of the discrete point type are not easily translatable. Establishing equivalence by attempting to set up a stratified random sample is complicated by the absence of a competence model which provides information on the weights of the individual subcompetences, which may well have to be different in different languages. Tests of the integrative type are generally translatable but it is uncertain to what extent translation of an integrative test guarantees equivalence with regard to difficulty. Tests of reduced redundancy such as cloze seem to have potential as proficiency measures with crosslinguistic applicability although test difficulty is affected differentially in English and French by both the point of onset of deletion and the scoring method used.
PDF
