Validity of scoring systems for the assessment of technical and non-technical skills in ophthalmic surgery – a systematic review

Authors: Wood TC, Maqsood S, Nanavaty MA, Rajak S.

Geographical coverage: Not reported

Sector: Biomedical

Sub-sector: Treatment

Equity focus: None

Study population: Review examines methods for assessing skills in ophthalmic surgery, rather than individuals. 

Review type: Other review

Quantitative synthesis method: Narrative synthesis

Qualitative synthesis method: Not applicable

Background: There is an increasing focus on competency-based methods of surgical training and assessment. The safe and effective surgeon must demonstrate NTS (non-technical skills) in addition to TS (technical skills). Appropriate, valid and reliable scoring systems are required for accurate surgical skills assessments.

Objectives: To outline the scoring systems for TS and NTS assessments specific to ophthalmic surgery, present the validity and reliability statuses of each scoring system, and make informed recommendations based on these factors.

Main findings:

A total of 27 articles were included in this review consisting of 19 assessment tools for TS and five assessment tools for NTS.

Authors the availability of TS scoring systems for various types of surgeries including cataract surgery, ptosis, strabismus, lateral tarsal strip, vitrectomy, and intraocular surgery. NTS scoring systems are applicable to cataract surgery or ophthalmic surgery in general. However, no single scoring system has met all validity and reliability criteria. The recommended TS scoring systems include the ‘International Council of Ophthalmology’s Ophthalmology Surgical Competency Assessment Rubrics’ (ICO-OSCAR) for phacoemulsification, strabismus, and paediatric cataract surgery, and the ‘Objective Structured Assessment of Cataract Surgical Skill’ (OSACSS). For NTS, the recommended systems are Non-Technical Skills for Surgeons (NOTSS), Observational Teamwork Assessment for Surgery (OTAS), and Anaesthetists Non-Technical Skills (ANTS). There is a noticeable scarcity of NTS scoring systems.

Authors identified limitations of the assessment tools including infrequent quantification of face and content validity, and inconsistency in terminology and statistical methods between studies. Despite being present occasionally, bias was not found to be widespread in the studies overall, therefore having a negligible impact on overall outcomes. Authors note the need of further research to validate all scoring systems to consistent standards.

Methodology:

This study included empirical research that developed or validated scoring systems for technical skills (TS) or non-technical skills (NTS) in ophthalmic surgery. The scoring systems needed to be printable and manually completable, facilitating real-time assessments in simulated or live settings, and allowing surgeons to use the forms for learning and reflection. Exclusions were computer-based systems, non-ophthalmic specialties, tools for clinical patient assessment, and exclusive validation of simulation models. Also excluded were non-English articles, previous reviews, books, and presentations. However, letters and editorials were included if they detailed their study’s methods and results.

Authors conducted a search of the English language literature on PubMed, ScienceDirect and Cochrane Library was performed between 20 December 2019 and 27 May 2020. No limits were applied for publication dates. Each article selected for full text review underwent a reference review; relevant articles that had not previously been elicited by the search terms were included until study saturation occurred.

One reviewer performed the searches and data extraction. The primary outcome measure was the validity and reliability status for each scoring system, which was evaluated in accordance with pre-set definitions. The secondary outcome measure included recommendation based on formal criticism in accordance with modified Oxford Centre for Evidence-Based Medicine guidelines. Levels of recommendation were provided based on the guideline’s levels of evidence. The methodology of each study was critiqued to reveal strengths and limitations. Risk of bias assessments were conducted for all studies; recognised forms of study bias were stated wherever they were identified. A formal risk of bias assessment tool was not utilised for this review, given the heterogenous nature of studies elicited.

Data from all included studies was tabulated. Articles were classified according to their emphasis on TS or NTS. Data extracted from each study included the scoring system used, analysis in a simulated or live setting, participant numbers and their training levels, and the validity and reliability statuses obtained. The heterogenous nature of the development and validation of these scoring systems meant that direct statistical comparisons and meta-analyses were neither appropriate nor applicable.

Applicability/external validity: The authors point out certain shortcomings (as mentioned earlier) in the current evaluations of scoring systems, implying that any comparisons made may lack thoroughness.

Geographic focus: Not reported

Summary of quality assessment:

There were a number of limitations to the approaches to identify, include and critically appraise studies. The search was restricted to English and to published material only. Articles were reviewed by one author only, in most instances. While the approach to the analysis of the data using narrative review was generally robust and appropriate, again, data extraction was only undertaken by one author in most instances. For these reasons, low confidence was attributed in the findings of this review.

Publication Source:

Wood, T.C., Maqsood, S., Nanavaty, M.A. et al. Validity of scoring systems for the assessment of technical and non-technical skills in ophthalmic surgery – a systematic review. Eye 35, 1833–1849 (2021).

source