The Use of Milestones in Physical Medicine and Rehabilitation Residency Education

Teresa L. Massagli

1: The Use of Milestones in Physical Medicine and Rehabilitation Residency Education

COMPETENCY-BASED MEDICAL EDUCATION

The Accreditation Council of Graduate Medical Education (ACGME) is a private nonprofit organization that sets the standards for and evaluates and accredits allopathic residency and fellowship programs in the United States. In 2002, the ACGME Outcome Project identified and endorsed six general competencies to assess resident competence: medical knowledge, patient care (PC), practice-based learning and improvement (PBLI), professionalism, interpersonal and communication skills, and systems-based practice (1). Although the goal of the Outcome Project was to increase accountability for individual resident competence, no national definitions of the competencies were developed, nor were uniform sets of assessments adopted (2). Residents were presumed to be competent because they had successfully completed the process of training in their residency program. To address these problems, the ACGME adopted a new accreditation system. As of July 2014, all residency programs will need to ensure that individual residents are achieving progressive milestones of competence during residency. The milestone narratives enhance transparency of residency education for residents, faculty, and ultimately the public (3).

THE PHYSICAL MEDICINE AND REHABILITATION MILESTONES

The Physical Medicine and Rehabilitation (PM&R) milestones were developed by a group of physician educators with periodic input from the field of PM&R in the years 2010 to 2012. Milestone narratives were developed for each of the six competencies. The milestones do not represent the entire scope of each competency but were constructed to capture the most important knowledge, skills, and attitudes residents should be developing in each of the six competencies. The six general competencies are part of the core program requirements that are revised by the ACGME approximately every 10 years. The milestones are not part of the core requirements and may be revised more frequently, depending on changes in the practice of PM&R and on whether national data analysis shows they demonstrate true progression of skill development. Currently, the ACGME requires PM&R residency programs to report seven PC milestones, one medical knowledge milestone, two interpersonal and communication milestones, and three milestones each in professionalism, PBLI, and systems-based practice. There are nine additional medical knowledge milestones that programs may use for developing curriculum, structuring clinical rotations, and evaluating residents. These nine medical knowledge milestones are for spinal cord disorders, brain disorders, stroke, amputation, neuromuscular disorders, musculoskeletal disorders, pain, pediatrics, and spasticity.

Each milestone set has five levels of observable behaviors arrayed from less to more advanced. The PM&R milestone levels are not postgraduate year (PGY) level specific because PM&R programs structure the timing of clinical experiences in diverse ways. However, each resident is expected to demonstrate progress over time. Level 1 represents the knowledge, skills, and attitudes of an entry-level resident. For some PM&R programs, entry-level residents are PGY1, and for others, they are PGY2. Because the milestones are PM&R specific, PGY1 residents may not have opportunities for learning or performing Level 1 milestones; in this case, the program would “score” that milestone as “has not achieved Level 1.” Levels 2 and 3 represent progressively more advanced skills. Level 4 is the target for a graduating resident, and Level 5 represents an aspirational goal that might be achieved by a few residents in some milestones. Level 4 is a target, not a requirement for resident graduation. The milestones are a framework for evaluation that the program director will use to determine if a resident is competent to enter practice without direct supervision (3). The milestones are also not intended to be used to accelerate completion of residency because their reliability and validity have not been established for use in high-stakes decisions. They do not represent the entirety of the dimensions of competence, and programs may have other additional requirements for scholarly work or clinical training that are not encompassed by the milestones. Finally, the duration of training that allows a resident to become eligible for Board certification is specified by the American Board of PM&R, not by the ACGME.

To be able to evaluate each resident using the milestones, programs will need an integrated mix of assessment tools. Table 1.1 summarizes the milestones, gives an example of some of the narratives in each milestone, and suggests what assessment tools may be helpful to determine an individual resident’s milestone attainment. For each milestone, several tools could be used together to assess a level, and the best tool varies depending on what is being measured. Choosing the best method of assessment depends on several different variables. Criteria for good assessment include the following:

Validity: Coherence, or does the method really assess the described behavior?

Reliability: Reproducibility or consistency, so are measurements consistent across evaluators or repeatable over time?

Educational effect: Does preparing to do well on the assessment motivate and educate the resident in the most relevant way?

Feasibility: Is the assessment method practical, realistic, sensible, efficient, and affordable given the circumstances and context?

Acceptability: Will faculty use it, and will residents and faculty find it credible and trust the results?

Equivalence: Will different versions of an assessment yield equivalent scores or decisions?

Catalytic effect: Does the assessment provide results and feedback in a fashion that creates, enhances, and supports education? (4)

On the surface, choosing the best method may seem daunting, but by closely reading the milestones, it often becomes obvious what tools could evaluate the skills, and then each program, and, in some cases, individual faculty within a program, can determine what method is feasible for their situation, the timing of when to evaluate each milestone, and how progress will be measured.

In addition to evaluation tools, sometimes additional opportunities will need to be created or adjusted to meet certain milestones, such as committee work or learning plans. For example, with PBLI milestone 3 (PBLI3), quality improvement (QI), residents are expected to understand basic QI principles and identify specific care processes that need improvement (Level 2) and then demonstrate active involvement in processes aimed at improving the care of patients (Level 3). These levels would best be assessed by residents learning about the QI process, and then participating in a QI project. To reach Level 4, the graduation target, they would then be expected to identify opportunities for process improvement in everyday work.

TABLE 1.1 Methods to Assess Milestones

MILESTONE (CORE COMPETENCIES IN BOLD)	EXAMPLE OF PORTION OF A MILESTONE	POSSIBLE WAYS TO ASSESS
Patient Care (PC)
PC1: History appropriate for age and impairment	“Documents and presents in a complete and organized manner”	Direct observation and workroom discussion Chart review OSCEs
PC2: Physiatric physical examination	“Identifies and correctly interprets atypical physical findings”	Direct observation and discussion OSCEs
PC3: Diagnostic valuation	“Orders appropriate diagnostic studies”	Case discussion Chart review Written and oral examinations
PC4: Medical management	“Manages patients with complex medical comorbidities”	Direct observation (counseling patients and families) Case discussion Chart review
PC5: Rehabilitation/functional management	“Prescribes commonly used prostheses”	Case discussion Chart review Written and oral examinations
PC6: Procedural skills	“Performs injections with direct supervision”	Direct observation Simulations
PC7: Electrodiagnostic procedures	“Identifies sites of EMG needle insertion in muscles commonly studied”	Direct observation Case discussion Chart review Written tests OSCEs
Medical Knowledge (MK)	“Predicts functional outcome and prognosis”	Case discussion Oral examinations Written tests
Systems-Based Practice (SBP)
SBP1: Systems thinking	“Has learned to coordinate care across a variety of settings”	Observation of patient case management Case discussion
SBP2: Team approach	“Leads the interdisciplinary team”	Observation of patient case management
SBP3: Patient safety	“Identifies health system factors that increase risk for errors”	Patient safety committee work Participation in practice improvement project Observation of patient case management
Practice-Based Learning and Improvement (PBLI)
PBLI1: Self-directed learning and teaching	“Develops and follows a learning plan”	Written learning plan Individual mentoring discussions Lecture attendance and participation Presentations
PBLI2: Evidence from scientific studies	“Effectively appraises evidence for its validity and applicability to individual patient care”	Journal club participation Presentations of case reports
PBLI3: Quality improvement (QI)	“Demonstrates active involvement in processes aimed at improving patient care”	QI committee work Practice improvement projects
Professionalism (PROF)
Prof 1: Compassion, integrity, and respect for others	“Exhibits compassion, integrity, and respect in challenging interaction with patients and families”	Direct observation Patient surveys 360° evaluations
Prof 2: Ethical principles	“Analyzes common ethical issues and seeks guidance when appropriate”	Case discussions Observation of patient case management
Prof 3: Professional behaviors and accountability	“Demonstrates that the responsibility of patient care supersedes self-interest”	Observation of patient case management Discussions with mentor
Interpersonal and Communication Skills (ICS)
ICS 1: Relationship management	“Effectively educates and counsels patients and families”	Patient survey 360° evaluations Direct observation OSCEs
ICS 2: Information gathering and sharing	“Ensures medical records are accurate and complete”	Chart review Observation of case management