Tools: Provider Behaviour Change

It is widely known that one of the biggest challenges of training is transferring knowledge gained from a training workshop to behaviour change in the workplace. This section focuses on how to evaluate whether or not your clinical training program is making a difference in how trainees care for their patients. Research suggests that the gap between knowledge and practice in the provision of clinical services is quite large in low resource settings. Therefore, the application of cost-effective and politically feasible methods to ensure that training activities are associated with improvements in clinical practice is vital to ensuring that training dollars are well spent.

Provider behaviour change can be measured through observation, self-reports, patient exit interviews, and patient record reviews.

  • Participant Observation Tool Instructions
    • File Type:
    Word Document
    Pages:
    3

    This tool was developed to advise international clinical mentors on conducting their clinical mentoring visit and using the observation tool.

  • MD Clinical Skills Checklist for Participant Observation
    • File Type:
    Word Document
    Pages:
    3

    The Skills Checklist is both a training and an evaluation tool. Its purpose is to provide some structure for observation and interaction during the on-site training component of a visit, and to provide structure for documenting and reporting on how much of the training to date has been transferred to the clinic. The particular skills or competencies listed on this tool were based on the national training curriculum for ART. The list was kept simple so that a clinical trainer could keep the competencies or major skills in their minds, rather than continually referring to the form while conducting the mentoring visit. The tool assumes an experienced mentor will be able to use their judgment to assess appropriate care within a particular context (i.e. when there are many patients waiting, severity of the case observed, labs available, etc.) The tool offers space for mentors to comment on why physician may not have executed skills as expected and to record suggestions given to the physician for improvement. This tool can be used to both assess physician skills and to guide teaching and feedback.

  • Nursing Skills Checklist
    • File Type:
    PDF*
    Pages:
    2

    This document is a checklist for observing nurses delivering patient care. This tool was designed for brevity to minimize the amount of time the mentor would be focused on the tool. It assumes that an experienced mentor will be able to assess appropriate care within a particular context (i.e. when there are many patients waiting, severity of the case observed, labs available, etc.) The checklist includes skills for interviewing, history taking, physical examination, assessment, health and illness care, and attitude and interpersonal skills. The tool offers space to record reasons why nurses may not implement skills as expected and for recommendations for nursing skills improvement. This tool can be used both to assess nursing skills and to guide teaching and feedback.

  • HIV/AIDS Nurse Specialist - Practicum Clinical Evaluation
    • File Type:
    Word Document
    Pages:
    4

    This form was used to evaluate participants on the clinical practicum component of a 3-week Advanced HIV/AIDS Nursing course in Ethiopia. Participants spent two days at three different practicum sites for a total of six-days of practicum. The evaluation form was completed by the preceptor at each site.

Observation

Direct observation is a fairly common research and evaluation method. Barriers to its implementation can be the cost (in most cases, it needs to be carried out by a trained clinician in order for the observations to be accurate) and the awkwardness training participants feel in being "observed." Clinical mentorship provides a unique opportunity to assess transfer of learning through direct observation at the same time as providing customized training to a particular provider according to demonstrated need.

The examples below describe several different contexts in which I-TECH has successfully used observation as an assessment or evaluation method. The tools used varied depending on the context.

Ethiopia: Clinicians as Participant Observers

As part of rapid ART scale-up in Ethiopia, I-TECH's training program included on-site training following classroom based training. The hospital or medical directors gave consent for clinical mentor visits from international I-TECH staff, but the mentors had no supervisory authority over the trainees. The clinical mentors were sometimes known to the medical staff from previous trainings, but due to high turnover at the facilities, this was not always the case.

The first step in creating a successful training experience in this setting was to establish a relationship of trust between the mentor and clinical staff. Without this, trainees would not be receptive to mentor suggestions or provide explanations for certain practices. Similarly, if mentors were faced with resentment and distrust, they would not feel comfortable asking questions or making suggestions about provider practice.

In this context, I-TECH developed an evaluation approach and tools in line with a methodology called "participant observation." In this methodology, the mentor is both participating in how patient care is being provided as a trainer, and observing how the trainee is practicing, and asking questions about that practice. The mentor takes notes on his or her observations and shares those at an appropriate point with the trainee. This is quite different from the role of the pure observer who stands on the sidelines, clipboard in hand, and doesn't interfere with or comment on the clinical practice. The primary role of the mentor is to teach; the mentor's role in observing and documenting is to inform that teaching, pinpoint areas where more work is needed and track the progress of trainees as a reflection of the effectiveness of the training.

Botswana: Observation with Minimum Guidance

As part of a formal evaluation of the effectiveness of a new curriculum for STI case management, I-TECH developed an observation checklist to assess health worker performance following training. Each health worker in the sample was observed five times. The observer was instructed to prompt healthcare providers where necessary to assist them in following the protocol taught in the training. These tools and methods could be adapted to assess on-going clinical mentoring.

  • Botswana STI Observation Checklist
    • File Type:
    Word Document
    Pages:
    4

    The checklist emphasizes patient-centered STI screening and syndromic management and includes a numeric scoring guide for evaluation. The observer was instructed to prompt healthcare providers where necessary to assist them in following the protocol taught in the training.

India: No Guidance

In India, I-TECH worked with a local organization to develop a quality of care study. The study included a baseline assessment of HIV care based on direct observations of provider practice, knowledge tests, patient exit interviews, and facility assessments. The plan is to repeat the baseline after a period of time to see if training and other program efforts have resulted in improvements in quality of care. Observers were recently graduated physicians and their role was to observe and take notes, and not to interfere or assist with provider practice in any way. Therefore, the observers needed to be able to recognize certain aspects of clinical practice but did not need to be able to advise or teach practitioners.

  • Physician Observation India Baseline
    • File Type:
    Word Document
    Pages:
    3

    Developed originally in tandem with IndiaCLEN, CDC and I-TECH, the physician observation tool is used in training needs assessment at an outpatient department.

Namibia: M&E with On-Site Clinical Teaching

In Namibia, "In-service tutors" (nurse-trainers who conduct in-service training for healthcare workers in PMTCT and VCT) have been conducting site visits to facilities in their region to both assess transfer of learning and to provide on-site clinical teaching, motivation, and support. Tutors observe and interact with healthcare workers in the antenatal clinic, labour and delivery ward, and postnatal ward, assess the facility to identify obstacles/barriers to service delivery, interview supervisors/managers, and conduct client exit interviews. A set of data collection tools was developed to standardize and enhance these visits and to facilitate assessment of transfer of learning.

India: Fellowship Observation

These checklists were designed to assess physician practice during a year-long HIV clinical fellowship at a host site. The use of the checklist and expectations of demonstrated skills is discussed with fellows at the beginning of the program. The checklists are to be used in various HIV/AIDS treatment settings during clinical rotations. Each document is very specific and useful for both initial skills assessment and ongoing monitoring. Though the forms are comparatively long, they were feasible in this setting where the clinical mentors were recognized instructors for the fellowship program and were thus expected to be carrying out some evaluative activities. In addition, since the same mentors were used continuously, the mentors became familiar with the instrument relatively early on and its length did not present significant problems.

Location Specific Skills Monitoring Checklists

    File Type:
    Word Documents
    Pages:
    Various

    These skills checklists are designed to be used in the various HIV/AIDS treatment settings during clinical rotations. Each document is very specific and useful for both initial skills assessment and ongoing monitoring. The forms include a description of the skills that should be demonstrated, an evaluation guide, and a place for scoring. Useful for on-going monitoring of changes in specific skills.

Variation on observation: An alternative approach to observation-based monitoring and evaluation would be to begin the clinical visit with providers or directors at the facility, asking them to identify key skills they'd like to work on, and have these skills be the basis of a checklist. The advantage of this approach is that it is more participatory. The disadvantage is that there will be less ability to compare change in practice over time on the same skills or across sites.

Self-Report

One way to find out whether providers have changed anything in their practice as a result of training is to ask them. The obvious drawback of this approach is one of reliability – trainees may be more inclined to reply politely or wishfully rather than objectively. Nonetheless, there are ways to make interviewing clinicians about their practice a more reliable method. For example, if you are wondering whether clinicians followed a certain treatment protocol, you can ask them to think about the last patient they cared for who presented the symptoms relevant to that protocol, and ask them to describe the process of diagnosing and treating that patient. Another way might be to ask participants what, if anything, they are doing differently as a result of the training.

Patient Exit Interviews

Patient exit interviews can be used to assess clinician communication skills and some practices. It is important to only ask those questions that the patient can reliably be expected to answer. For example, patients can answer questions about their experience of the patient/clinician encounter, whether they felt comfortable and/or have faith in the providers' recommendations, or what they understood those recommendations to be. Generally, patients are not in a position to provide information on the accuracy of the diagnosis or appropriateness of the treatment plan. Patient exit interviews can be difficult because of the need for patient confidentiality. Utilizing this method often requires clearance through a country level review board.

  • Patient Exit Interview
    • File Type:
    Word Document
    Pages:
    4

    This patient exit interview tool was designed to be used in conjunction with direct observation and a facility assessment in order to assess provider HIV care in Andrha Pradesh, India.

  • Patient Exit Interviews STI Care
    • File Type:
    Word Document
    Pages:
    15

    This interview protocol was designed to assess patient satisfaction with care and patient comprehension of key messages in Botswana. For patients attending for follow-up visits, the interview will assess self-reported health outcomes and adherence to therapy.

  • Client Exit Interview Form
    • File Type:
    Word Document
    Pages:
    3

    This form was developed to interview clients receiving PMTCT services at antenatal clinics in Namibia. It is one component of a comprehensive site visit that includes healthcare worker observation, interviews with supervisors/managers, and a facility assessment.

Patient Record Reviews

Patient record review is another way to assess some provider behaviours. The feasibility of using patient records depends on whether patient record-keeping meets some minimum standard.

  • Patient Record Abstraction
    • File Type:
    Word Document
    Pages:
    3

    This Tool was used in Botswana to collect information on the content of patient visits and assess health workers' documentation of STI cases.

[Top of Page]

*Software capable of displaying a PDF is required for viewing or printing this document. We have included a copy of Adobe Reader on this CD (in software folder) or you may download it from the Adobe website at http://www.adobe.com/products/acrobat/readstep2.html.