Adding a fourth rater to three had little impact in pre-linguistic outcome classification
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Adding a fourth rater to three had little impact in pre-linguistic outcome classification. / Persson , Christina; Conroy, Elizabeth J; Gamble, Carrol; Rosala-Hallas, Anna; Shaw, William; Willadsen, Elisabeth.
In: Clinical Linguistics & Phonetics, Vol. 35, No. 2, 2021, p. 138-153.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Adding a fourth rater to three had little impact in pre-linguistic outcome classification
AU - Persson , Christina
AU - Conroy, Elizabeth J
AU - Gamble, Carrol
AU - Rosala-Hallas, Anna
AU - Shaw, William
AU - Willadsen, Elisabeth
PY - 2021
Y1 - 2021
N2 - The consequence of differing levels of agreement across raters is rarely studied. Subsequently, knowledge is limited on how number of raters affects the outcome. The present study aimed to examine the impact on pre-linguistic outcome classifications of 12-month-old infants when using four raters compared to three. Thirty experienced Speech and Language Therapists (SLTs) from five countries assessed 20 minute video recordings of four 12-month-old infants during a play session with a parent. One recording was assessed twice. A naturalistic listening method in real time was used. This involved: (1) assessing, each syllable as canonical or non-canonical, and (2) following the recording, assessing if the infant was babbling canonically and listing the syllables the infant produced with command. The impact that four raters had on outcome, compared to three, was explored by classifying the outcome based on all possible combinations of three raters and determining the frequency that the outcome assessment changed when a fourth assessor was added. Results revealed that adding a fourth rater had a minimal impact on canonical babbling ratio assessment. Presence/absence of canonical babbling and size of consonant inventory showed a negligible impact on three out of four recordings, whereas the size of syllable inventory and presence/absence of canonical babbling was minimally affected in one recording by adding a fourth rater. In conclusion, adding a forth rater in assessment of pre-linguistic utterances in 12-month-old infants with naturalistic assessment in real time does not affect outcome classifications considerably. Thus, using three raters, as opposed to four, is recommended.
AB - The consequence of differing levels of agreement across raters is rarely studied. Subsequently, knowledge is limited on how number of raters affects the outcome. The present study aimed to examine the impact on pre-linguistic outcome classifications of 12-month-old infants when using four raters compared to three. Thirty experienced Speech and Language Therapists (SLTs) from five countries assessed 20 minute video recordings of four 12-month-old infants during a play session with a parent. One recording was assessed twice. A naturalistic listening method in real time was used. This involved: (1) assessing, each syllable as canonical or non-canonical, and (2) following the recording, assessing if the infant was babbling canonically and listing the syllables the infant produced with command. The impact that four raters had on outcome, compared to three, was explored by classifying the outcome based on all possible combinations of three raters and determining the frequency that the outcome assessment changed when a fourth assessor was added. Results revealed that adding a fourth rater had a minimal impact on canonical babbling ratio assessment. Presence/absence of canonical babbling and size of consonant inventory showed a negligible impact on three out of four recordings, whereas the size of syllable inventory and presence/absence of canonical babbling was minimally affected in one recording by adding a fourth rater. In conclusion, adding a forth rater in assessment of pre-linguistic utterances in 12-month-old infants with naturalistic assessment in real time does not affect outcome classifications considerably. Thus, using three raters, as opposed to four, is recommended.
U2 - 10.1080/02699206.2020.1758793
DO - 10.1080/02699206.2020.1758793
M3 - Journal article
C2 - 32372661
VL - 35
SP - 138
EP - 153
JO - Clinical Linguistics & Phonetics
JF - Clinical Linguistics & Phonetics
SN - 0269-9206
IS - 2
ER -
ID: 240739885