Virtual Coach Technology for Supporting Self-Care

“Virtual Coach” refers to a coaching program or device aiming to guide users through tasks for the purpose of prompting positive behavior or assisting with learning new skills. This article reviews virtual coach interventions with the purpose of guiding rehabilitation professionals to comprehend more effectively the essential components of such interventions, the underlying technologies and their integration, and example applications. A design space of virtual coach interventions including self-monitoring, context awareness, interface modality, and coaching strategies were identified and discussed to address when, how, and what coaching messages to deliver in an automated and intelligent way. Example applications that address various health-related issues also are provided to illustrate how a virtual coach intervention is developed and evaluated. Finally, the article provides some insight into addressing key challenges and opportunities in designing and implementing virtual coach interventions. It is expected that more virtual coach interventions will be developed in the field of rehabilitation to support self-care and prevent secondary conditions in individuals with disabilities.

The term virtual coach appeared in the 1950s and was used to refer to an athletic coach leading the team behind the scenes or over distance. The earliest virtual coach was reported in 1997 when an electronic device (ie, a laptop) was used to guide a rare surgical procedure in an operating room. Now virtual coach is used to refer to a coaching program or device aimed at guiding users through tasks for the purpose of prompting positive behavior or assisting with learning new skills. Cognitive orthosis, virtual trainer, occupational enabler, and prompting device are some of the terms used to name the devices or programs having similar functions as a virtual coach. Virtual coach also shares similar features as persuasive technology defined as interactive computing systems intentionally designed to change people’s attitudes and behaviors. Virtual coach, however, influences people by placing more emphasis on providing instructions on how to complete the target activity correctly than simply motivating them to start doing the activity if they have not done so.

One of the greatest areas of innovation for virtual coach is to support preventative health management and self-care. Healthy People 2010 emphasizes that a primary focus of health promotion for people with disabilities is on preventing or reducing secondary conditions. The Institute of Medicine also identifies the use of assistive technologies for managing and preventing secondary conditions as one of the major areas in need of future research in the field of rehabilitation. The current state of practice regarding preventing or managing secondary conditions among people with disabilities is to provide patient education during the rehabilitation process or service. With increased cost containment, however, the rehabilitation process or service continues to be compressed, and limited time is available to provide such education. Information overload or feeling the information may not apply to them can interfere with retention or application of the education materials in patients’ daily living environment. Also forgetfulness, complexity of the regimen, and disruption of daily routines may contribute to poor adherence to clinical recommendations. Many people, especially those having cognitive deficits, rely on caregivers or family members to provide verbal reminders or instructions to complete target activities. Virtual coach interventions ranging from simple time-based reminders to interactive Web-based applications have been used to address these issues. Most of these interventions, however, are designed to be used in a fixed space and cannot provide assistance as needed. They also generally operate open loop or rely on self-report, and the information delivered is more generic and not suited to an individual user’s context or situation.

This article focuses on virtual coach interventions that incorporate sensor technologies, context-aware computing, and adaptive coaching strategies. Such interventions are able to infer elements of a person’s context and activity and reason about when, how, and what messages to deliver based on the person’s performance, progress, and context. They rely on extensive pervasive sensing of the person and his or her environment and include powerful software engines that can process, mine, and send coaching messages tailored to the person’s condition and context in an automated way.

This article reviews virtual coach interventions with the purpose of guiding rehabilitation researchers to comprehend the essential components of such interventions, the underlying technologies and their integration, and exemplar applications. The authors’ goal is also to provide insight by addressing key challenges and opportunities in designing and implementing virtual coach interventions, and promote such interventions in rehabilitation to support self-care and prevent secondary conditions in individuals with disabilities.

Components of virtual coach interventions

To systematically understand virtual coach interventions, the authors have identified four components that define the design space of such interventions and address when, how, and what message to deliver in an automated and intelligent way ( Fig. 1 ).

Self-monitoring

Self-monitoring refers to observing and recording a target activity, such as medication usage, activity levels, and calorie intake, or an outcome of the target activity, such as weight. Self-monitoring often is achieved through self-report, which could be tedious and inaccurate. Additionally, it relies on user cooperation. With the advance of sensor technologies and wearable computing, self-monitoring in virtual coach interventions can be achieved automatically and in real time using various sensors such as heart rate monitors, accelerometers, global positioning system (GPS), and cameras. The sensors collect, store, and share relevant data with users and make it easier for them to know how well they are performing the target activity. Psychology research has shown that self-monitoring feeds the natural human drive for self-understanding and is an important strategy to motivate desired behaviors or avoid undesired behaviors. More importantly, sensor-based self-monitoring enables the system to track user performance and progress over time, and makes sure the coaching messages will be triggered only when necessary and that the message contents are more relevant to the user. Ijsselsteijn and colleagues used heart rate monitors to measure training intensity when participants cycled in a virtual environment with a stationary home exercise bike. The heart rate information was used to provide feedback to the participants via a virtual agent who could encourage participants to do better or tell them to slow down if the heart rate became too high. Mihailidis and colleagues developed a hand washing coach device for people with moderate-to-severe dementia, where a ceiling-mounted video camera in the washroom and computer vision algorithm were used to track hands and task objects to monitor hand washing steps and guide users through the steps as necessary.

Context Awareness

Location, time, identity, and activity have been proposed as the primary elements of context. People usually take a great deal of context into account when they communicate. Previous research has shown that intervening at the right context likely will increase the chance of getting responses. Studies in task interruption also have shown that responsiveness to an interruption depends crucially on what the user is doing at the time the interruption occurs, in addition to many other factors such as the emotional state of the user and the modality of the interruption. Sensor technologies and machine learning algorithms could enable machines to perceive certain elements of context such as location and user activity to determine levels of user interruptibility, and deliver prompting messages at the opportune moment when users are likely to respond. For example, if the user happens to be sitting idly when the prompting messages are delivered, these messages may result in a relatively high response rate. Siewiorek and colleagues designed a context-aware mobile phone that can modify its behavior such as ringer volume and vibration based on its user’s state and surrounding environment using motion, light, and microphone sensors in a device worn on the body. Kaushik and colleagues evaluated a proximity-based reminder system for home medical tasks in a 10-day case study where proximity to the location where the task must be completed was used to delay the reminder until execution of the task was convenient based on location. The study showed that 96% of the proximity-triggered reminders were acted upon within 5 minutes of being acknowledged compared with only 8% of timer-triggered reminders, and 25% of the proximity-triggered messages received the most favorable rating compared with 9% of the timer-triggered ones.

Interface Modality

Various modalities such as light, auditory, visual, and tactile interfaces can be used to present coaching messages to users. Within an interface modality, there are also different stimuli such as multiple sounds and text, graphics, or animation-based displays. Research has shown that different modalities/stimuli or combinations have different effects on user perception, performance, and acceptance. Mayer and colleagues found that participants performed better on a learning transfer test when the interface agent had a standard accent compared with foreign accent, and when the agent’s voice was human rather than machine synthesized. Bickmore and colleagues evaluated the impact of four different types of audio alert tones on the compliance of the target wrist rest behavior and found that compliance dropped off quickly with the very impolite alert tone. Mihailidis used the prerecorded voice of a professional male actor to prompt the correct sequence of hand washing for people with moderate-to-severe dementia. Some of the subjects, however, did not like the male voice because it reminded them of “being in the Army,” and one subject even became agitated by the male voice.

Animated agents recently have become one of the most popular interface modalities with the advance in graphical user interfaces (GUI) and computing technology. These agents range from animated shapes to human-like entities including cartoon-like shapes and characters, talking animals, or various other forms. A current trend is to design animated human-like agents with various features such as visual appearance, fidelity, expressiveness, and social–emotional skills. There is well documented research showing the effect of an animated agent on user performance, perception, and acceptance. Atkinson suggested that the dual mode of presentation (animated agent with narrated instructions) enhanced learning outcomes. King and Ohya found that a dynamic three-dimensional human agent whose eyes blinked was rated more intelligent than other forms such as caricatures and geometric shapes. Walker and colleagues found that people who interacted with a talking face spent more time on an online questionnaire, made fewer mistakes, and wrote more comments than those who answered a text questionnaire. Bickmore and colleagues conducted a study comparing an animated health counseling agent on a mobile device with equivalent agents that had text only or text and static image representations, and found that the animated versions led to significantly better social bonding with users. The same group of scientists also researched the role of emotional and relational skills in an animated human agent to support a behavioral intervention for physical activity adoption. They found that there was no significant difference between the amount of physical activity performed by subjects working with the agent equipped with emotional and relational skills versus those working with a nonrelational agent. Subjects working with a relational agent, however, reported a significantly greater desire to continue working with the relational agent. There are also negative effects reported with animated agents such as distraction and discomfort. Moreno and colleagues found that visual presence of the agent was no more effective than a voice-only condition. Given the mixed results, it appears that a virtual coach intervention should consider the combined elements of physical characteristics of the user (such as personality, gender, background knowledge, and capability), attributes of the agent (such as appearance, fidelity, speech quality, or expressiveness), and the type of target task (such as intent, difficulty, or complexity).

Coaching Strategy

Coaching strategies determine what content is presented to the user, such as general versus specific messages, and incorporating different effects such as positive or emotional effects into the messages. Steege and colleagues evaluated two prompt strategies for teaching common household tasks to students with severe disabilities. One of the strategies used the least-to-most restrictive sequence, where students received instructional prompts in the order of nonspecific verbal prompt, specific verbal prompt, gesture and verbal prompt, partial physical and verbal prompt, and total physical and verbal prompt if they failed to initiate a response to the prompt with 5 seconds. The other prescriptive strategy was based on user progress, where the prompt was given at a level just above the prompt that had induced a correct response in the previous trial. The results showed both strategies were equally effective in increasing independent task acquisitions; however, the prescriptive strategy was more efficient. Most-to-least restrictive sequence is another coaching strategy where the most specific message is provided at the beginning and the message specificity gradually decreases. This strategy holds the most control over the user’s behavior, but has a risk of message dependency. Usually the most-to-least restrictive sequence is used to help establish the confidence in performing the target activity when a user just starts to learn the new task. When the error rate decreases, the least-to-most restrictive strategy can take over to help retain the learning outcome by gradually removing message dependency. Ideally, a coaching strategy dynamically adapts to the need, performance, and progress made by the user through self-monitoring and context awareness, which will help avoid a nagging effect and enable a trustworthy relationship between the user and the system. The hand washing coach for older adults with dementia developed by Mihailidis and colleagues used coaching messages with three levels of assistance including low-guidance verbal prompt, high-guidance verbal prompt, and verbal prompt with a video demonstration of the action. The level of assistance was determined based on factors such as the error committed, sensory and cognitive status of the user, and past responsiveness to the previous coaching messages. This strategy gave the coach the adaptive ability to select the most appropriate support for each individual’s stage of dementia and overall responsiveness.

Coaching strategies also involve adding different affects to the messages. Coaching messages can be designed to minimize affect, or adopt a positive or negative effect. Positive messages may help reinforce the desired performance and increase comfort and interest in using the system. A study found that people who received computer praise after they played an experimental game felt the interaction was more engaging and were more willing to work with the system again. Also, coaching messages that respond empathetically and encouragingly to users rather than state cold hard facts may appeal to the emotion of reaching a goal and help promote a trustworthy relationship between the user and the system. Negative messages, on the other hand, should be used with caution and are generally not appropriate even when the purpose is to decrease the instances of errors or undesired behaviors.

Examples of virtual coach interventions

The use of technologies to coach activities for supporting preventive health management and self-care has been an active domain for research. Table 1 summarizes some typical virtual coach interventions including those mentioned in previous sections. The authors also discuss their preliminary work in developing a wheelchair seating virtual coach in this section.

Table 1

Summary of some typical virtual coach interventions

Reference	Target Activity	Portability	Interface Modality	Coaching Strategy	Evaluation	Outcome
The coach prompting system to assist older adults with dementia through hand washing: an efficacy study	Hand washing	Fixed station with a video camera to monitor hand washing performance	Prerecorded male voice, video demonstration	Three levels of instructions determined based on the error committed, user conditions, and past responsiveness	Pilot test with six older adults with dementia using single- subject design	More steps completed independently, fewer interactions with caregivers, 23% false alarm or miss detection
Virtual fitness: stimulating exercise behavior through media technology	Physical exercise	Fixed bicycle station with virtual environment and heart rate was monitored	Female virtual agent with prerecorded voice and synchronized lip movements, text in a cartoon-like text balloon	Feedback based on heart rate	Test with 24 subjects who do not exercise regularly using within- subject design	Significantly lowered the perceived control and pressure, but did not affect enjoyment
Bringing mobile guides and fitness activities together: a solution based on an embodied virtual trainer	Physical exercise	PocketPC with GPS that monitors the trial and user speed	Annotated trail map with prerecorded voice to support navigation, and embodied virtual trainer to provide motivation support and exercise demonstration	Positive feedback based on user speed on the trail; location aware exercise demonstration	Test with 12 users using within subject design	More useful for navigation than trail maps and more effective to teach how to correctly perform exercise
Lifelogging memory appliance for people with episodic memory impairment (EMI)	Memory	Wearable camera, audio recorder, and GPS logger to record personal experiences	Visual and audio cues in a slideshow narrative on a Tablet PC	Self-guided review based on automated heuristics to exact meaningful memory cues	Pilot test with three subjects with EMI and their caregivers, compared with two other strategies	Support the memory of people with EMI better and reduce the burden placed on their caregivers
Context awareness in a handheld exercise agent	Walk	PDA with an integrated two-dimensional accelerometer to detect if walking at a moderate intensity	Animated agent in a close-up shot with a range of nonverbal behaviors such as head nod or eyebrow raise and text display (no speech)	Positive feedback based on accelerometer readings	Pilot test with eight subjects using within- subject design (sensor-based feedback on and off)	More perceived awareness and closer relationship with the agent, but less walking when feedback based on accelerometer

Wheelchair Seating Virtual Coach

Power seating functions (PSFs), including tilt-in-space, backrest recline, and elevating leg rests ( Fig. 2 ), usually are prescribed for power wheelchair users who are often unable to adjust their body positions independently, and therefore predisposed to secondary conditions related to prolonged sitting such as pressure ulcers, spasticity, edema, orthostatic hypotension, and chronic pain. Power wheelchair users who have these seating functions may not use them as prescribed, however, and frequently come back to clinics for problems such as fatigue, pain, or pressure sores that could be improved or solved by changing seating positions. Although client education on proper use of PSFs is part of the clinical evaluation process in assistive technology clinics, these clinics have limited time and resources to devote to extended periods of client training. Users may know that using PSFs is good for their conditions, but simply may not remember when they should use them, and how to operate them. In a previous survey study, most individuals used the PSFs for comfort rather than the specific medical reasons for which they are provided. The authors also have tracked actual PSFs use of 11 subjects with a suite of sensors and a datalogger for 2 weeks. They found that subjects did not adjust seating positions frequently and seldom used large tilt angles (greater than 30°), which was found to provide sufficient weight shift in previous laboratory studies. Given the fact that a full complement of PSFs can double the cost of a power wheelchair, and many wheelchair users are not using the technology in an appropriate and effective way, it is important to develop interventions that address this problem. The authors are currently developing a wheelchair seating virtual coach ( Fig. 3 ). Fig. 4 shows the system diagram of the seating coach, which is comprised of

A suite of sensors that monitor the actual PSF use and elements of a user’s context such as location and activity
A single board computer that synthesizes sensor information to determine the appropriate coaching protocol to assist wheelchair users to use PSFs for effective pressure relief and other activities of daily living such as transfers
A touch screen that delivers the coaching messages to the user via a multi-modal interface.