Chapter 5 Assessing and Improving the Teaching and Learning Processes in Academic Settings

Did you ever express or hear comments like these?

Well, you have to be careful about putting too much time into your teaching. Good teaching takes time away from the things you need to do to get tenure and promotion at this institution. What really matters are research, grant writing, and publications.

Isn’t program assessment the function of the curriculum committee? Besides, I was hired as a neuroscientist, not an educator. I know what the students need to know; that’s why I need one more semester hour in my course to cover the material. Whatever you do, don’t volunteer for the self-study task force; leave those accreditation activities to the educators. Sure, I evaluate my course. All students fill out an evaluation form the last day of class. I have good assessment data for my teaching, just look at my final student grades. They all achieved the learning outcomes as demonstrated by their grades.

Assessment techniques are used to assess one’s teaching with the goal of improving student learning (curriculum) and the overall program. Essentially, there are two types of assessments: student learning outcomes and program effectiveness.¹ Assessment helps answer the question, How well do we do what we say we’re going to do? Classroom educators purposefully focus on designing, evaluating, and revising the particular courses they teach. Intent on the subject matter they teach, educators often disconnect their teaching roles from the ongoing assessment and thoughtful scholarship of how their teaching actually influences student knowledge, understanding, attitudes, values, and behaviors. In this chapter, we explore practical mechanisms and tools that all educator-scholars can use for assessment and improvement of their teaching and resultant improvement in student learning.

Chapter outline

Assessment

Student Learning Problems

Pillars for Transformative Learning

Assessment of Student Learning

Classroom Assessment

Teaching Goals Inventory

Classroom Assessment Techniques

Small Group Instructional Diagnosis

Quality Circles

Peer Coaching

End-of-Course Student Ratings

Program Accountability

Scholarship of Teaching

After completing this chapter, the reader will be able to:

1. Define and give examples of the three main learning problems that students experience: amnesia, fantasia, and inertia.

2. Distinguish between assessment and evaluation.

3. Describe the relationships between assessment and student learning across educational settings, including classroom, program, and institution.

4. Identify the four pillars of transformative assessment and describe how and why they undergird program assessment activities.

5. Articulate principles of best practice for assessing student learning, including didactic and clinical student learning and performance, professional behaviors, and clinical reasoning.

6. Recognize the utility of the Teaching Goals Inventory and its relationship with the preactive teaching grid. (See Appendix 5-A.)

7. Describe processes that are intended to improve teaching: classroom assessment techniques (CATs), small group instructional diagnosis (SGID), quality circles, peer coaching, and end of course student ratings.

8. Describe how curriculum mapping can provide an alignment structure for the assessment process.

9. Distinguish between scholarly teaching and scholarship of teaching.

Assessment

What is assessment? Is it that periodic bother that comes once every so many years when a program goes through a self-study and accreditation process? Is it the course evaluations that students fill out at the end of the semester? Is it a formal program review that institutions and regional accreditors require of each department or program every few years or when a new president or provost is hired? Is it collecting data from graduates and alumni? Assessment pertains to all of these major activities. But notice that all these activities are summative—that is, they occur after a teaching-learning experience has been completed. However, formative assessment activities, which occur during a teaching-learning experience, have equal or more importance in improving one’s teaching skills and resultant student learning. It is these formative assessment techniques that are the focus of this chapter.

The terms evaluation and assessment are often used interchangeably. However, for the purposes of this chapter, we make the following distinction: Evaluation refers to an end point, determining whether an action is right or wrong (e.g., answers to questions on a classroom examination). Assessment refers to looking closely at teaching events with a central emphasis on student learning and considerations of how to improve student outcomes.²

The assessment movement in education has been stimulated at the federal level by both institutional and specialized (professional) accreditation agencies. This movement is having a powerful influence on higher education. The central focus is on accountability of the educational program related to student learning outcomes.²^–⁴ The Association for the Assessment of Learning in Higher Education (AALHE; http://www.aalhe.org) is an organization that focuses on using effective assessment practice for purposes of improving student learning. AALHE offers members opportunities to join in discussions about assessment practices and hosts an annual assessment conference. Publications from the Council on Higher Education Accreditation (CHEA;http://www.chea.org) should be regular reading in physical therapy and physical therapist assistant programs for those interested in assessment. Why is ongoing educational assessment, which helps us determine ways to improve our teaching and resultant student outcomes, so important to physical therapy educators that we allocate an entire chapter to address it? The answer lies in the learning problems students bring with them when they enter physical therapy programs.

Student learning problems

Students come to our physical therapy programs with well-ingrained learning problems that hinder both the acquisition and retention of information. Lee Shulman ⁵ identifies three of these common problems as amnesia, fantasia, and inertia. For those of us teaching the next generation of health care professionals, these problems with student learning are an anathema.

Amnesia refers to students’ learning material for an examination but forgetting it immediately afterward. This happens when our physical therapy programs are packed with courses with little time built in for students to reflect, integrate, and apply the information they are receiving. Students learn to survive such programs by focusing on whatever examination is next. Thus, for instance, cramming for an anatomy examination is followed by “dumping” the information to cram for a pathology examination. This practice might not have severe consequences in an undergraduate liberal arts program in which a biology course is not connected to an English course. However, amnesia obviously has serious consequences for students in health professional programs, in which inter-relating and building on information is a well-imbedded assumption of the faculty.

Fantasia refers to illusions students have about how things in the world work and how people behave. Cognitive psychologists have long argued that new learning grows out of prior learning; that is, learning will occur only when new information is linked to existing knowledge. Thus, Shulman points out that the misconceptions students hold hinder the appropriate reception of new information. For example, if a student believes in “no pain, no gain” from her or his past athletic experiences, how will that belief influence her or his understanding of pain management and ability to work with patients who have painful, chronic, debilitating conditions?

Inertia is the student’s inability to apply what is known. This occurs when students are taught piecemeal facts that they cannot organize or use in a problem-solving situation. For example, students who are asked to memorize origins and insertions of muscles without a functional movement context are very likely to forget this anatomic information by the time they are asked to assess gait deviations.

Seasoned faculty are familiar with these problems of learning. For physical therapy faculty, these problems must be dealt with in order for students to successfully attain the knowledge, skills, and values of a professional practitioner. Each new group of students enters our programs having done well in their prior academic endeavors despite exhibiting amnesia, fantasia, and inertia. Students may even believe they performed well because of amnesia, fantasia, and inertia! How do we deal with these three common learning problems, as well as the unique learning problems individual students bring with them (e.g., fear of asking questions in the classroom, competitiveness and the related unwillingness to share knowledge with peers, and the student-versus-teacher mentality)? A concerted, programmatic effort must be in place to diminish as much as possible these students’ learning problems and give students the insights and skills they need to guide their own lifelong learning. In simple terms, we want to have further insight and understanding of how students are thinking and learning—we want to “get the inside out.” Use of the assessment ideas and tools presented in this chapter is a powerful method to help teachers improve their teaching so that students learn and, equally important, find enjoyment in learning.

Pillars for transformative learning

Angelo states, “[D]o assessment as if learning matters most.”⁶ Assessment affects all levels of education, from individual classrooms and laboratories to programs and the institution in which the programs reside. Angelo argues that there are four essential components, or pillars, for transformative assessment (Figure 5-1). The term transformative is used with specific intent. It means that something actually will happen within an institution that leads to a change in the institutional culture. This change is facilitated through individuals who share common beliefs and values, work together to develop guidelines, and act on those guidelines. An example of transformative change in higher education is the current emphasis on student learning, which is visible in many venues (e.g., highlighted in conference topics, discussed and debated in the research literature, and emphasized in accreditation documents).

Figure 5-1 Angelo’s four pillars of transformative assessment.

(Adapted from Angelo T. Doing academic development as though we value learning most: transformative guidelines from research and practice. In James R, Milton J, Gabb R. Research and Development in Higher Education 22. Melbourne, Victoria: HERDSA, 2000.)

The four pillars for transformative assessment are as follows:

1. Build shared trust. This is done through the faculty’s building a productive learning community in which the faculty involved in assessment trust one another. All persons must feel respected, valued, and safe to share their experiences.

2. Build shared motivation. There have to be collective goals worth working toward and problems worth solving. This means a shift for some faculty because faculty often focus on what they will teach rather than on what students will learn; students, in turn, often focus on getting through the program.

3. Build a shared language. Faculty need to develop a collective understanding of new concepts needed for transformative changes. Although assessment may mean only student course evaluations or standardized testing—and time wasted to some faculty members—a collaborative model would focus on assisting the entire faculty to see the broader conception of assessment that focuses on formative assessments and student learning and successful achievement of program outcomes.

4. Build shared guidelines. Faculty will benefit from a short list of research-based guidelines that can be used for assessment to promote student learning. An example of a research-based guideline in physical therapy or physical therapist assistant education might include gathering formative assessment data on the ability of students to demonstrate critical professional behaviors or ability-based outcomes. For example, do students demonstrate empathy across the curriculum? What evidence do students provide in support of their clinical decision-making skills?

Angelo’s assessment vision includes the formation of learning communities that include groups of faculty and students working toward shared, significant learning goals. He proposes that assessment should be seen as less of a technical data collection process and more of a monitoring and problem-solving process. Assessment is an ongoing process of quality improvement. Moving from a focus on summative evaluation to formative assessment requires the faculty to think about and distinguish between these two concepts. Think about the last time your department went through an accreditation process. What type of approach did your department use in providing information?

Nine principles of good practice for assessing student learning (Box 5–1) serve as a good starting point for thinking about your current assessment activities.^{7(p 23)} You might want to examine these nine principles and see which of them are included in the assessment activities in your classroom, program, or institution. For example, do your assessment activities attend equally to the teaching-learning experience and the learning outcomes?

Box 5–1 Nine Principles of Good Practice for Assessing Student Learning

1. Assessment of student learning begins with educational values.

2. Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time.

3. Assessment works best when the program has clearly, explicitly stated purposes.

4. Assessment requires equal attention to outcomes and to the experiences that lead to those outcomes.

5. Assessment works best when it is ongoing and not episodic.

6. Assessment fosters wider improvement when representatives from across the educational community are involved.

7. Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about.

8. Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change.

9. Through assessment, educators meet responsibilities to students and to the public.

Adapted from AAHE Assessment Forum. Principles of Good Practice for Assessing Student Learning. In Maki PL. Assessing for Learning: Building a Sustainable Commitment Across the Institution. Sterling, VA: Stylus, 2004.

Assessment of student learning

Classroom Assessment

Learning can and often does take place without the benefit of teaching—and sometimes even despite it—but there is no such thing as effective teaching in the absence of learning. Teaching without learning is just talking.^{8(p 3)}

All of us in higher education, regardless of whether we are in a professional school or a graduate department, aim to produce graduates who achieve the highest possible quality of learning. As educators, our greatest reward is the success of our graduates. As we teach, we are constantly engaged in an “informal” process of classroom assessment—that is, determining what students know, do not know, need to learn, do, and become.⁹ To do this, we ask students questions, observe and react to body language that depicts confusion or boredom, and listen carefully to students’ comments. In response to this input, we may speed up, slow down, review material, or change in other ways to react to student learning needs.

We infrequently, however, undertake systematic and formal classroom assessment. As previously stated, classroom assessment techniques are well suited to formative assessment—that is, getting specific feedback from the entire class as we move through the course, thus gaining insight into student learning so that changes can be made to enhance the learning process before the end of the course.

Angelo and Cross’s well-known classic book on classroom assessment techniques is an excellent resource for faculty.⁸ This book is a practical guide for designing and implementing classroom assessment techniques for any faculty member, regardless of his or her background. Their model of classroom assessment is built on the following assumptions about teaching and learning: The quality of student learning is directly (not exclusively) related to the quality of the teaching. One of the most promising ways to improve learning is to improve teaching.^4,^8,⁹ To improve effectiveness, teachers first must make their goals and objectives explicit and then obtain feedback to determine the extent to which students are achieving such goals. (See the preactive teaching grid in Chapter 2.)

• To improve learning, students need to receive appropriate and focused feedback early and often. They also need to learn how to assess their own learning.

• The type of assessment most likely to lead to improvement of teaching and learning is that conducted by faculty themselves, in which they formulate the questions specific to their own teaching concerns.

• The processes of systematic inquiry are an intellectual challenge and an important source of motivation for faculty. Classroom assessment provides this kind of challenge.

• Classroom assessment does not require specialized training. It can be carried out by any dedicated teacher within any discipline.

• Through collaborating with colleagues and actively involving students in classroom assessment efforts, faculty can enhance learning and personal satisfaction.

Teaching Goals Inventory

A useful way to initiate classroom assessment planning is for each faculty member to complete the Teaching Goals Inventory (TGI).⁸ The TGI is a questionnaire designed to assist faculty in identifying and ranking the relative importance of their teaching goals for any class. With one particular class in mind, the faculty member rates the importance of 52 teaching goals across six clustered areas: higher-order thinking skills, basic academic success skills, discipline-specific knowledge and skills, liberal arts and academic values, work and career preparation, and personal development. The complete TGI is available in Appendix 5-A. You will see that there are many parallels between the TGI and the philosophical orientations to curriculum discussed in Chapter 2.

The primary role of the TGI for classroom assessment is for teachers to identify what goals they view as important so that they can target their assessment efforts toward those teaching goals. Secondarily, the collective faculty can share its teaching goals across classes within a semester or across the entire curriculum to assess where its collective teaching and learning focus lies. As with sharing one’s philosophical orientation and learning theory emphasis, sharing teaching goals is often an eye-opening experience.