Adding a fourth rater to three had little impact in pre-linguistic outcome classification

Research output: Contribution to journalJournal articlepeer-review

Documents

  • Fulltext

    Final published version, 1.64 MB, PDF document

  • Christina Persson
  • Elizabeth J Conroy
  • Carrol Gamble
  • Anna Rosala-Hallas
  • William Shaw
  • Elisabeth Willadsen
The consequence of differing levels of agreement across raters is rarely studied. Subsequently, knowledge is limited on how number of raters affects the outcome. The present study aimed to examine the impact on pre-linguistic outcome classifications of 12-month-old infants when using four raters compared to three. Thirty experienced Speech and Language Therapists (SLTs) from five countries assessed 20 minute video recordings of four 12-month-old infants during a play session with a parent. One recording was assessed twice. A naturalistic listening method in real time was used. This involved: (1) assessing, each syllable as canonical or non-canonical, and (2) following the recording, assessing if the infant was babbling canonically and listing the syllables the infant produced with command. The impact that four raters had on outcome, compared to three, was explored by classifying the outcome based on all possible combinations of three raters and determining the frequency that the outcome assessment changed when a fourth assessor was added. Results revealed that adding a fourth rater had a minimal impact on canonical babbling ratio assessment. Presence/absence of canonical babbling and size of consonant inventory showed a negligible impact on three out of four recordings, whereas the size of syllable inventory and presence/absence of canonical babbling was minimally affected in one recording by adding a fourth rater. In conclusion, adding a forth rater in assessment of pre-linguistic utterances in 12-month-old infants with naturalistic assessment in real time does not affect outcome classifications considerably. Thus, using three raters, as opposed to four, is recommended.
Original languageEnglish
JournalClinical Linguistics & Phonetics
Volume35
Issue number2
Pages (from-to)138-153
ISSN0269-9206
DOIs
Publication statusPublished - 2021

ID: 240739885