Establishing Validation Methods: Measuring Progress (Measuring Teaching Effectiveness) – Global Rating Scales


Study

Tool

Description

Conclusion

Howells et al. [10]

Modified orthopaedic competence assessment project (OCAP)

Intraoperative 14 point arthroscopic checklist and OSATS GRS

Tool demonstrated improved performance in operating room for simulator trained individuals compared to untrained control

Insell et al. [7]

The basic arthroscopic knee scoring system (BAKSS)

Combined task-specific checklist and GRS for diagnostic arthroscopy and partial meniscectomy

System able to discriminate people with different levels of arthroscopic experience

Elliott et al. [8]

Arthroscopic skill assessment form (Knee)

100 points score, 75 for structure identification, 25 for time completion and deductions for cartilage damage

Can distinguish between novice, experienced and expert surgeon in cadaveric knee

Schantz et al. [12]

The objective assessment of arthroscopic skills (OAAS)

Global skills domains with 5 skill level option combines with 13 point anatomical area checklist

Discriminates between various skill level of training, high level of consistency and test-retest reliability

Tuijthof [13]

Arthroscopic surgery skill evaluation tool (ASSET)

Global skills domain with 8 skill level option additional 1 domain for complexity using procedural video

Discriminates between novice, competent and expert level of training


Derived from various sources



Analysing these GRS, one can conclude that a certain level of consensus exists on arthroscopic skills that a resident should be able to demonstrate in the operating theatre and the required level to qualify as competent. OCAP is not specifically tested, but its items are derived from the well-established OSATS GRS, which has been validated extensively. The four other GRS have been validated for construct, content and concurrent validity as well as internal consistency, interrater and test-retest reliability. These results are summarised in Table 9.2.


Table 9.2
All GRS that are suggested for rating of arthroscopic skills based on Hodgins and Veillette [14]




































Acronyms of global rating scales

Description

Validation

OCAP

9 items, scored on a 1–5 point Likert scale

Based on OSATS validation protocols

BAKSSS

10 items, scored on a 1–5 point Likert scale

Construct validity level of experience (p < 0.05)

Concurrent validity with year of residency (r = 0.93)

Concurrent validity with motion analysis (r = 0.58) (Alvand et al. [15])

Internal consistency (Cronbach’s α = 0.88) (Alvand et al. [15])

Interrater reliability (kappa = 0.543) (Olson et al. [16])

ASA

100-point score, 75 for structure identification, 25 for time to completion and penalties for cartilage damage

Construct validity level of experience (p < 0.001)

OAAS

7 items, scored on a 1–5 point Likert scale, complexity of procedure

Construct validity level of experience (p < 0.0001)

Only gold members can continue reading. Log In or Register to continue

Stay updated, free articles. Join our Telegram channel

Apr 7, 2017 | Posted by in ORTHOPEDIC | Comments Off on Establishing Validation Methods: Measuring Progress (Measuring Teaching Effectiveness) – Global Rating Scales

Full access? Get Clinical Tree

Get Clinical Tree app for offline access