ELI at Kanda University of International Studies

ELI lecturers Erik Fritz and Rachael Ruegg, and former ELI lecturer Jenn Holland will have their article, “Rater Sensitivity to Qualities of Lexis in Writing” published in the distinguished journal, TESOL Quarterly. The following video explains the story behind the article, from the origins of the research, to writing, and finally to possible implications. Dennis Koyama, ELI lecturer and co-coordinator of the Kanda English Proficiency Test (KEPT), also shares his thoughts on the trio’s project.

Vodpod videos no longer available.

In addition, here is the abstract of their article:

Distinguishing lexis from grammar is a particularly thorny issue. Because the distinction is so difficult to make, many scholars suggest that we should not try to distinguish the two but rather should accept the inextricable interwovenness which exists. However, many assessment criteria involve assessment of lexis and grammar. Assessing both lexis and grammar presupposes that the two can be evaluated independently of each other. Therefore, if such criteria are to be used, it is necessary to investigate just how raters are to distinguish lexis from grammar. On the other hand, if we are to accept that the two are inextricably interwoven then the assessment criteria must reflect this by assessing lexicogrammar as a single criterion.

Various researchers have discussed lexis in relation to speaking but there has been very little research relating directly to the distinction of lexis and grammar in writing. Research by Batty in 2006 found no correlation between examinees’ vocabulary knowledge as measured by the Depth of Vocabulary Knowledge (DVK) Test and their vocabulary scores in speaking section of the Kanda English Proficiency Test (KEPT). It was concluded that it is too difficult to assess vocabulary knowledge in a group discussion format. As a result of Batty’s research the vocabulary scale and grammar scale were collapsed into a single lexicogrammar scale for the speaking section of the KEPT.

The present collaborative study was conducted in relation to the writing section of the KEPT. This study was carried out to ascertain what raters are sensitive to when rating writing using the ‘lexis’ scale, one of four analytic rating scales. The lexis scale is intended to evaluate lexical usage in terms of both accuracy and range. It has been considered by some of the administrators of the KEPT that the addition of low frequency words may be sufficient to artificially inflate an examinee’s score on the lexis scale even when, overall, the lexis is insufficient in terms of accuracy and range. The lexical content of 140 essays was analysed. Particular attention was paid to accuracy and range of lexical usage, average frequency of the words used and overall lexical sophistication. In addition, the correlation between scores on the lexis scale and those on the grammar scale was considered. The collaborative research project will be explained in detail, results and implications of the study, as well as directions for much-needed future research will be discussed.