OBJECTIVE: The objective of this study was to assess the performance and reproducibility of the two currently used ocular surface staining scores in the assessment of keratoconjunctivitis sicca in Sjögren syndrome (SS) research classification. METHODS: In a multidisciplinary clinic for the evaluation of sicca, we performed all tests for the American European Consensus Group (AECG) and the American College of Rheumatology (ACR)/European League Against Rheumatism (EULAR) classification criteria, including the van Bijsterveld score (vBS) and the Ocular Staining Score (OSS), in 994 participants with SS or with non-SS sicca. We analyzed the concordance between the scores, the diagnostic accuracy and correlation with clinical variables, and interrater and intrasubject reproducibility. RESULTS: A total of 308 (31.1%) participants had a discordant vBS and OSS that was due to extra corneal staining points in the OSS. The presence of one or more of the additional points was highly predictive of SS classification (odds ratio = 3.66; P = 1.65 × 10e-20) and was associated with abnormal results of all measures of autoimmunity and glandular dysfunction. Receiver operating characteristic curves showed optimal cutoff values of four for the vBS (sensitivity = 0.62; specificity = 0.71; Youden's J = 0.33) and five for the OSS (sensitivity = 0.56; specificity = 0.75; Youden's J = 0.31). Notably, there was very poor consistency in interobserver mean scores and distributions (P < 0.0001) and in intrasubject scores after a median of 5.5 years (35% changed status of the ocular criterion). CONCLUSION: Ocular surface staining scores are useful for SS research classification; however, they are subject to significant interrater and intrasubject variability, which could result in changes in classification in 5%-10% of all subjects. These results highlight the need for objective and reproducible markers of disease that have thus far remained elusive for SS.