Wellness Self-Management Initiative

Performance Indicators Guidebook

The New York State Office of Mental Health Version 5 SEPTEMBER 2009

Table of Contents
Performance Indicators Guide 3 Performance Indicators... 3 Completing Performance Indicators.. 6 Submitting Performance Indicators... 7 Internal CQI Reports 18 Quick Guide.. 19 Performance Indicator Forms. 20 Indicator #1: Attendance. 21 Indicator #2: Discontinuation.. 23 Indicator #4: Fidelity. 24 Indicator #5: Personal Progress Checklist... 26 Indicator #6: Group Leader Ratings.. 27 Additional Resources 28 Key Concepts of Educational Supervision Lesson by Lesson Helpfulness Scale Participant Feedback Survey Group Leader Feedback Survey

INTRODUCTION The New York State Office of Mental Health (OMH) recently partnered with the newly created Center for Practice Innovations (CPI) at Columbia University. This center is a resource available to mental health agencies, consumers and family members as well as members of the Practice Improvement Network (PIN). The following guide and development of the web based data entry system has been developed by the CPI to support our collective efforts to better understand and evaluate important aspects of the Wellness Self Management Initiative. The information on performance indicators and a description of the web based system currently in development (http://nyebptac.networkofcare.org) is meant to support the gathering of practical and reliable information that enables agencies to determine the value of WSM service and identify challenges affecting the starting, sustaining, and spreading of WSM services. For OMH, your experiences implementing WSM helps us better understand the system and programmatic issues that influence the adoption of innovations. It also helps us better understand the usefulness of the training approach and resource materials employed in this initiative. In light of OMHs goal to promote and support the availability of evidence based practices, it is critical that agencies have the tools to adopt, implement, and sustain quality practices. The data your agency provides will be critical to furthering this goal. Reporting the data described in this guidebook will enable your agency to 1) keep track of key performance indicators and 2) examine your data in relation to data from other members of your collaborative as well as regional and statewide data. It is very important to emphasize that collecting data on specific performance indicators is meant to support YOUR efforts to improve the quality of your services and OUR efforts to determine effective methods to promote quality services. It will not be used to judge quality of a specific agency or program. As you begin using the web based system to enter data we very much want your feedback and suggestions.

PERFORMANCE INDICATORS As part of the Quality Improvement plan for this initiative, each participating agency will report data related to six performance indicators at the intervals listed below (see Table 1, pg 5). We have developed several tools to assist you in gathering information on these performance indicators. Table 2 (pg 5) lists the persons responsible for completing the forms. The tools are included at the end of this guidebook.
Indicator #1: Attendance. This indicator provides a measure of how many consumers participated in WSM across the State. The QI representative of each agency/site should submit monthly attendance reports on the CPI secure website for each WSM group.

Indicator #2: Discontinuation. As part of our ongoing QI efforts, it is important for programs to know why consumers are discontinuing participation in the WSM group. The QI representative of each agency/site should submit monthly discontinuation reports on the CPI secure website for each WSM group, which includes the number of consumers who discontinued participation during that month and the primary reason for discontinuation. If no one discontinued during the month enter 0. Indicator # 3: Total number of lessons completed by each participant. We have streamlined this indicator by including it as an item in indicator # 6 (Group leader Ratings: Item 1). For this reason, you may notice that there is no separate indicator #3. Indicator #4: Fidelity. This indicator provides a measure of the degree to which WSM groups were implemented based on the specific WSM techniques and values imbedded in the program and the training provided. Optimally, clinical supervisors complete a fidelity rating at least three times over the course of the group (Time 1: during the first 8 lessons Time 2: at the midpoint of the group around lesson 34 and Time 3: near the end of the group around lesson 50) .The clinical supervisor meets with the group leader(s) before the session, attend the session, and mark the check boxes based on their observations of how often they observe the techniques being used by the group leader(s) and to what degree the group leader(s) incorporate WSM values into the session. The QI representative of each agency/site should submit a fidelity report on the CPI secure website in after each fidelity review conducted by the supervisor. Indicator #5: Personal Progress Checklist. This indicator measures consumers perception of their progress in achieving their personal wellness goals. This information will not be entered on the web, but rather, the QI representative of each agency/site should mail the white copy of the checklist to the CPI; this is voluntary and should be done with the verbal consent of the consumer. Agencies/sites should black out consumers names or other identifying information before mailing to maintain the consumers confidentiality. Indicator #6: Group Leader Ratings: This indicator measures the group leader(s) assessment of the following for each consumer: number of lessons completed; level of group participation; percentage of lessons for which action steps were completed; level of involvement of others; level of health checkups; and level of progress on their goals. This assessment should be completed after completion of lesson 34 and lesson 57. The QI representative of each agency/site should submit a fidelity report on the CPI secure website upon completion of lesson 34 and lesson 57.

Table 1. Schedule for Reporting the Six Performance Indicators th Monthly (due the 10 After Indicator of each month for the By end of Lesson Number Indicator Name previous month) Lesson 8 34 1 2 4 Attendance Discontinuation Fidelity (supervisor ratings) Personal Progress Checklist (the checklist should be filled out at Lesson 7, Lesson 34 and Lesson 57, and submitted at the end of the group) Group Leader Ratings of each participant across 6 items X X X X

After Lesson 57

5 6

Table 2. Schedule for Who Fills Out the Six Performance Indicators Indicator Number Indicator Name Group Leader Supervisor 1 2 4 5 6 Attendance Discontinuation Fidelity Personal Progress Checklist Group Leader Ratings X X X X

Consumer

COMPLETING PERFORMANCE INDICATORS The CPI has setup a website to assist sites with the collection and entry of data for the six performance indicators. The process for collecting and entering data is as follows: 1. Each QI team should determine how best to collect the data requested, either using agency forms or the forms provided in this guide. 2. Forms for collecting data for each indicator should be provided to each group leader with the instructions for completing each form and timelines for submitting these forms to the QI representative on the team. (See Tables 1 and 2, pg 5) 3. The QI representative and one other backup person (Users) will request a unique ID and password for accessing the CPI website. If user does not have an ID and password, please contact Melissa Hinds-Martinez and she will get one assigned. 4. The User(s) will access the data-entry website at http://nyCPI.networkofcare.org The User(s) will enter all required data on the CPI website monthly and/or specific times outlined above (see Table 1, pg 5) for each performance indicator. 5. All of the information that is reported through the website will be in reference to specific groups that are operating. Some sites may have multiple groups running simultaneously. Each agency/site may choose to enter data on the first group started or on two or more groups. Data from other groups operating will provide additional data to assist OMH and the CPI in spreading WSM and assisting agencies in adopting, implementing, and sustaining other evidence based practices. Each time a new group is started, the User must assign a unique name to that group (the User should store these group names, with information that distinguishes the groups, in a safe place for future reference). 6. Once a group has been assigned a name, the User selects which performance indicator to access. Note that those indicators that are due monthly should be submitted by th the 10 of each month for the previous month. CPI will send monthly reminders to all agency QIT members. 7. In addition to entering data, the website will eventually allow agencies to print summary reports showing data for their site relative to the group, region, and State as a whole. These reports include rates of attendance over time, a report of the number discontinuing and completing the program over time, a summary of the fidelity scores, and a summary of progress consumers made towards their goals. 8. If you have any questions about the data-entry, please contact: Melissa Hinds-Martinez at 212-543-5941 or hindsma@pi.cpmc.columbia.edu .

SUBMITTING PERFORMANCE INDICATORS

When the User is ready to enter the data, the User will access the website (located at http://nyCPI.networkofcare.org) and enter their User ID and password as shown here:

If this is the first time the User has accessed the website, the User will be brought to the section of the Home page where they can create a new Group Name. Because all ratings are attached to a specific group, a unique name will need to be assigned to each group each time a new group starts. The figure below illustrates what information is needed, including the type of program in which the group is taking place, the number of group leaders, whether one is a peer specialist, and the date that the group began. In the example below, the User created a group name that was meaningful to them to help them keep track of their groups going forward.

Once the User has assigned a name to each group, the User can begin entering the information for each respective indicator. The Existing Groups section of the Home page will show the User all of the available indicators for which they may enter data. This page can always be accessed by clicking the Home button. Typically, the User will begin by entering the Attendance data for the month. This is accessed by clicking on the Attendance link on the Home page.

Performance Indicator #1: Attendance When entering information regarding Enrollment and Attendance, report the data for each week during the reporting month. Most months will only have 4 weeks, but for those with 5, a field is available for reporting data. The website automatically fills in the weeks for each respective month, by showing the date which falls on the Monday for each week. Note that if the last week of the month spills over into the next month, depending on what day of the week the group runs, the User might need to enter the first weeks data in the last week of the previous month. There are spaces available for those agencies which run more than one session per week; most agencies will only run one session per week, and so all of their attendance data should be entered under Session #1 Attendance. Also, note that a person is no longer considered enrolled when they clearly indicate that they will no longer be attending the group, or if their circumstances change such that it is apparent that the person will not be returning to the group. In many cases, determining enrollment will be a judgment call on the part of the group leader or the programs policies and procedures. If there are any weeks during the month when the group is not conducting a session (e.g., starting partway through the month, vacations, holidays), please just leave the enrollment column blank (this tells us that no session was expected to occur during that week). At any time, you can review and edit previous entries by clicking on the Existing Entries tab.

10

A note about enrollment and discontinuation: Every group has its own unique circumstances, and agencies are open to implement the program in the way that makes the most sense for the consumers being served. Additionally, we recognize that attendance can be irregular as a new group is starting to form. New people may join the group late in the program, or existing members may decide the program isnt right for them. Occasionally, two groups may merge into one group, or one group may split if the group membership gets too big, and some groups may become inactive for a period and then reconvene with a few new members. We are most interested in how many consumers are receiving this intervention, why some consumers decide to discontinue their involvement, and what impact the program has on the lives of the consumers who complete the program. Some things to keep in mind: The number enrolled in a group should always equal or exceed the number attending each week. Consumers are considered enrolled when they state their intention to join the group and attend at least one session; a consumer who never attends the group should not be considered enrolled. Only report a consumer as having discontinued when they leave the WSM altogether, and have no intention of returning; consumers who move to another group should not be considered discontinued. If a consumer switches to another group, that should be reflected in the enrollment numbers (enrollment should go up for one group and down for another group), but the person should not be considered discontinued. Because enrollment and attendance data can not be tied to identifiable consumers, we have no way of knowing that a person who leaves one group is the same as the person who shows up in another group. Consequently, please avoid reporting data that incorrectly suggests large changes in enrollment which are, in reality, reorganization of group memberships for very good clinical and practical reasons. Cases where group enrollment converts to zero and then stops completely, should be reserved for those cases where the group truly ended and was not moved elsewhere. Although individual circumstances may vary, generally, if one group splits into two, the agency should report the data as if there were two groups running simultaneously from the beginning. Generally, if two groups merge into one, the agency should treat both groups as one single group from the beginning (please contact Melissa Hinds-Martinez at 212-543-5941 for questions about enrollment and discontinuation).

11

Once the data have been entered, click on the Save button, and if the data were entered correctly, the statement, Record Inserted Successfully will appear at the top.

12

Performance Indicator #2: Discontinuation The number of participants (if any) who discontinue enrollment during the month should be recorded on Indicator #2 along with the reasons for discontinuation (this will be particularly helpful for OMH to understand how to make WSM even better for participants). We anticipate that for many months, there will be no consumers discontinuing. In those cases, simply enter zero (0) for the number discontinuing during that month. As much as possible, try to use one of the pull-down options unless the situation is so unusual as to warrant a response of other. You may still use the specify box to elaborate any reason for discontinuing, even if you select one of the pull-down options other than other, and being as descriptive as possible can be useful for helping OMH understand the reasons for discontinuing. Note that moving to another group is not considered discontinued and a consumer who never attended the group should neither be considered enrolled nor discontinued. At any time, you can review and edit previous entries by clicking on the Existing Entries tab.

13

Performance Indicator #4: Fidelity The supervisor should complete the fidelity ratings for each group and hand the form to the User (Note: the supervisor should only provide one rating at each of the 3 intervals, and the ratings that are entered on the website should be for the group as a whole; although individualized supervision is encouraged, the fidelity ratings must be about the group as a whole and not the individual group leaders). The User will then access the website and enter the information as shown below. At any time, you can review and edit previous entries by clicking on the Existing Entries tab.

14

15

Performance Indicator #5: Personal Progress Checklist The Personal Progress Checklist should not be entered on the web. Rather, after the last lesson is completed, the consumer should tear off the white sheet and hand it to the group leader. The group leader will give them to the agency QI person who will mail them to: Melissa Hinds-Martinez Administrative Assistant Center for Practice Innovations Department of Mental Health Services and Policy Research New York State Psychiatric Institute Room 2739, Box 100 1051 Riverside Drive New York, New York 10032 Office: (212) 543-5941 Fax: (212) 543-6535 Email: hindsma@pi.cpmc.columbia.edu Because this is voluntary, it is important to obtain verbal consent from the consumer. A suggested script follows: Our agency and OMH are interested in knowing how the Wellness program is helping consumers who participate in the groups. Over the past year, you have been recording your progress on the Personal Progress Checklist. This checklist provides a lot of information about how the program has specifically helped you. Would it be okay with you if I mailed a copy of your completed form to OMH? I will not put your name anywhere on the form. If your name is on the form, please cross it out to insure confidentiality. This is completely voluntary. Would this be alright with you?

16

Performance Indicator #6: Group Leader Ratings The group leader(s) of each group should evaluate the progress of each member of the group twice a year (at midpoint and endpoint). At midpoint, the group leader should fill out the st 1 set of ratings and hand the form to the User, and the User will enter the information on st the web as shown below (note that only the 1 section should be entered at midpoint; at the nd end of the program, the group leader should fill out the 2 set of ratings and hand it back to the User to enter the endpoint data; for this indicator, it is very important to make sure that st the consumer who is listed as #1 for the 1 set of ratings is the same person who is #1 for nd the 2 set of ratings, and so forth for all of the group members):

17

INTERNAL CQI REPORTS Below are some sample reports which may be generated from the data that is entered online. Each agency will be able to see how their site is performing relative to the rest of the State, Region and similar Program Types. This functionality will be made available at a later date. In the meantime, the Center will continue to provide summary reports by email on a regular basis.

Attendance Rates Over Time For Your Agency Compared to the Rest of the State
100 90
Percent Attendance

80 70 60 50 40 30 20 10 0
May Jun 2008 Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr 2009 *Hypothetical Data

Statewide Your Agency

Time to Discontinuation From WSM Group

Proportion Still Enrolled in WSM Group 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 012345678910 Months Since Group Began
11 12
*Hypothetical data!

Statewide Your Agency

18

Wellness Self-Management Program Quick Guide for Submitting Wellness Indicators

Monthly (by the 10 of each Month): Group leaders fill out the forms for Indicators #1 and #2 and hand them to the QI person responsible for entering the data (see below). By the end of December 2009: The supervisor makes the first fidelity rating for Indicator #4. The QI member of the Quality Improvement Team enters the data from the Core Competencies Checklist (see below). Midway through the program (Lesson 34): The group leader makes the first set of ratings for Indicator #6. The supervisor makes the second fidelity rating for Indicator #4. All data entry forms are handed to the QI person who will enter this data along with the Monthly indicators (see below). When the group is nearing completion (lessons 54-57): The group leader completes the second set of ratings for Indicator #6 (Group Leader Ratings). The supervisor completes the third fidelity rating for the group. The group leader will also invite participants to tear out the white copy of the Personal Progress Checklist and hand it in to the group leader who will mail it to the CPI. The group leader will also invite participants to complete the feedback surveys as well as the lesson by lesson helpfulness checklist that is also included in the workbook. Participants are asked to tear these forms from the workbook and hand in to the group leader. NO IDENTIFIABLE INFORMATION SHOULD BE ON ANY OF THE FORMS HANDED IN BY GROUP PARTICIPANTS All data entry forms are handed to the QI person who will enter this data along with the Monthly indicators (see below). For entering data: 1) The QI person accesses the website. (http://nyCPI.networkofcare.org). 2) The QI person enters his/her user ID and password to view the secure dataentry screens for his/her agency. 3) The QI person selects the screen that is needed: a) If a new group has started, the QI person needs to assign a group name b) If data is being reported for an existing group that already has a group name, the QI person will select the appropriate indicator. 4) Follow the instructions on the screen and submit the data. NOTE: Make sure you have the correct Group Name listed for the group being reported.
th

Questions? Contact Melissa Hinds-Martinez at 212-543-5941

Performance Indicator Forms

Performance Indicator #1: Attendance

Agency _____________________ Program _________________Group Leader(s) _________________ Date Group Began ____________ Group Name_______

Submit data on a monthly basis that will provide information regarding the number of consumers enrolled and the actual number who attended their Wellness SelfManagement group. Report separately for each group that is running. For those groups which run more than one session per week, there are spaces for you to report attendance for up to three sessions during each week. For those groups that only operate one session per week, only fill in the attendance for Session #1. If no sessions were conducted during a particular week (e.g., group hadnt started yet, vacations, holidays), please leave the enrollment column blank (a blank in the enrollment column tells us that no group was expected to attend that week). Weeks 1-5 refer to the weeks of the month for which you are reporting.

Month Reporting (e.g. 6/08) ______________

# enrolled each week Week 1 Week 2 Week 3 Week 4 Week 5 (if applicable)

# attending each week Session 1 Session2 Session3

21

ATTENDANCE TRACKING FORM: DATA TO ASSIST IN COMPLETING PERFORMANCE INDICATOR # 1 AND # 6 (Item 1: total number of lessons completed)

Program ______________Group Leader(s) ______________ Date Group Began _________ Group Name___________________________________ Initials # Lessons Completed: Circle the number for each lesson that the person attended in a scheduled group or completed in a make-up session. 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
25 47 26 48 27 49 28 50 29 51 30 52 31 53 32 54 33 55 34 56 35 57 36 37 38 39 40 41 42 43 44 45 46

2 3 4 5 6 7 8 9 10 11 12

1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49 1 2 3 4 25 26 27 47 48 49

5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52 5 6 7 8 28 29 30 50 51 52

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 53 54 55 56 57

22

Performance Indicator #2: Number and Reasons for Discontinuation

Agency _____________________ Program _________________Group Leader(s) _________________ Date Group Began ____________ Group Name_______ Month Reporting __________ # of consumers who discontinued this month _______

For each person who discontinued this month, please indicate the primary reason for discontinuing participation in the group (where possible, based on self report)
Person # 1 Primary Reason for Discontinuation (please check one box) Work schedule School schedule Family/Home Demands Relapse/De-compensation Hospitalized Incarcerated Loss of contact Deceased Work Schedule School Schedule Family/Home Demands Relapse/De-compensation Hospitalized Incarcerated Loss of contact Deceased Work Schedule School Schedule Family/Home Demands Relapse/De-compensation Hospitalized Incarcerated Loss of contact Deceased Work Schedule School Schedule Family/Home Demands Relapse/De-compensation Hospitalized Incarcerated Loss of contact Deceased Work Schedule School Schedule Family/Home Demands Relapse/De-compensation Hospitalized Incarcerated Loss of contact Deceased Work Schedule School Schedule Family/Home Demands Relapse/De-compensation Hospitalized Incarcerated Loss of contact Deceased Didnt get along with group members Did not follow ground rules Discharged from the agency/program Schedule conflict with other services Workbook is too difficult Group not helpful Physically unable to attend Didnt get along with group members Did not follow ground rules Discharged from the agency/program Schedule conflict with other services Workbook is too difficult Group not helpful Physically unable to attend Didnt get along with group members Did not follow ground rules Discharged from the agency/program Schedule conflict with other services Workbook is too difficult Group not helpful Physically unable to attend Didnt get along with group members Did not follow ground rules Discharged from the agency/program Schedule conflict with other services Workbook is too difficult Group not helpful Physically unable to attend Didnt get along with group members Did not follow ground rules Discharged from the agency/program Schedule conflict with other services Workbook is too difficult Group not helpful Physically unable to attend Didnt get along with group members Did not follow ground rules Discharged from the agency/program Schedule conflict with other services Workbook is too difficult Group not helpful Physically unable to attend Other, specify: ________________ ________________ ________________

Other, specify: ________________ ________________ ________________

Other, specify: ________________ ________________ ________________

Other, specify: ________________ ________________ ________________

Other, specify: ________________ ________________ ________________

Other, specify: ________________ ________________ ________________

23

PERFORMANCE INDICATOR # 4: Core Competencies Checklist for use by group facilitators and supervisors
Agency _____________________ Program _________________Group Leader(s) _________________ Date Group Began ____________ Group Name_______ Person who completed ratings __________________________ Date ratings completed ____________ One aspect of the implementation of WSM relates to the clinical competencies demonstrated by group facilitators. Four core competencies have been identified: Engagement and motivational techniques Educational/teaching techniques Cognitive-behavioral techniques Group skills utilizing the ROPES format This checklist will be utilized by supervisors and group facilitators to support the continuous improvement of skills in implementing the WSM group program. It focuses on the four core competencies plus the four WSM values, and allows supervisors to provide feedback to group facilitators after observing the implementation of group session(s). Furthermore, group facilitators may use this checklist to assess their knowledge and skills and identify areas they may want to improve. For each skill component listed below, supervisors and/or group facilitators indicate, by checking the box next to each item, whether the component was Not observed, Sometimes observed or Frequently observed.
Not Observed Sometimes Observed Frequently Observed

I. Engagement and Motivational Techniques Connects topic to members goals and values Uses reflective listening and empathic responding (avoids judgmental and critical comments) Emphasizes the benefits of learning the topic area Makes inspiring comments that promote hope Expresses appreciation for participants efforts

Not Observed

Sometimes Observed

Frequently Observed

II. Educational/Teaching Techniques Engages people in reading out loud Asks questions about main points to increase comprehension Respectfully assists member to stay on topic Clarifies ideas via examples relevant to participants lives Asks questions to check on participants comprehension

24

Not Observed

Sometimes Observed

Frequently Observed

III. Cognitive-Behavioral Techniques Reframes ideas or beliefs that are self-defeating Breaks information down into small segments to shape understanding Models behavior via demonstrations and/or self disclosure Provides specific feedback to participants Provides positive reinforcing comments to participants

Sometimes Observed

Frequently Observed

Not Observed

IV. Group Skills Utilizing the ROPES Format Prior to group session, describes the plan for the session with specific goals that can be observed Prior to the group session, describes the progress made by each participant Checks on the outcome of Action Steps from previous session Reviews participants experiences with and knowledge about todays topic Presents an Overview of todays session Leads a discussion about the importance of todays topic for participants Leads a discussion focusing on the Important Information presented in todays lesson Assists participants with the completion of Personalized Worksheets In session, assists members planning for Action Steps Encourages participants to Summarize the important points of the lesson

Sometimes Observed

Frequently Observed

Not Observed

V. WSM Values (Please rate how often the group leader incorporated the 4 values of the WSM Program) Hope (emphasizes positive possibilities, acknowledges setbacks without a sense of defeat) Choice (emphasizes options, encourages informed decision making, helps people identify pros and cons) Involvement (emphasizes learning from others, encourages action about getting needs met) Recovery (emphasizes personal strengths, values and goals)

25

PERFORMANCE INDICATOR 5: Participant perception of progress in self identified goal areas QUALITY OF LIFE PROGRESS CHECKLIST TIME 1 (Lesson 7) TIME 2 (Lesson 34)
Quality of Life Areas I want to improve Mid Point in the WSM

TIME 3 (End)
At the end of the WSM

Working at a paid job (part or full time) that I like Contributing to my community in a useful way(e.g., volunteering, joining a self help or peer advocacy group) Working towards an academic degree, continuing my education or learning a trade in school Living in a place that I like and can manage successfully Having a good relationship with one or more family members Socializing with friends (spending enjoyable time with others) Enjoying hobbies, leisure and recreational activities Engaging in creative activities (music, art, writing, dance etc.) Being confident that I can handle my mental health problems and not relapse Being hopeful about my future (confident that I will find success and satisfaction in important areas of my life) Stopping or reducing my use of alcohol, drugs or cigarettes Using medication in a way that works for me. Improving my physical health( healthy eating, exercise, getting regular checkups for medical, dental and vision problems) Improving my spiritual/religious side( e.g., being part of a supportive spiritual community) Add your own:

No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this

No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this No Improvement Improvement Ive achieved this

26

PERFORMANCE INDICATOR # 6: WSM GROUP LEADER RATINGS:

RATING GUIDELINES : To answer the questions accurately, you may want to meet individually with the participant. Place numbers in the corresponding boxes for each member of the WSM group across two (2) points of time: lesson 34 and the end.
Agency _____________________ Program _________________Group Leader(s) _________________ Date Group Began ____________ Group Name_______ Group 1st Ratings at midpoint (lesson 34) 2nd Ratings at completion of WSM program GROUP st Member Date 1 Rating Completed ___/___/___ Date 2nd Rating Completed ___/___/___ MEMBER Number
1) LESSONS COMPLETED

INITIALS
(for local use, DO NOT ENTER INTO WEBBASED DATABASE

1) LESSONS COMPLETED

2) PARTICIPATION

3) ACTION STEPS

4) INVOLVEMNT OF OTHERS

5) HEALTH CHECKUP

6) GOAL PROGRESS

4) INVOLVEMNT OF OTHERS

6) GOAL PROGRESS

2) PARTICIPATION

3) ACTION STEPS

5) HEALTH CHECKUP

1 2 3 4 5 6 7 8 9 10 11 12 1) LESSONS COMPLETED: Please indicate the number of lessons completed by each member of the group for two time-points (Lesson 34 and lesson 57). This is the total number completed since the group began and include all lessons completed during the group session, as well as lessons completed in individual meetings. 4) INVOLVEMENT OF OTHERS: Indicate the extent to which
each consumer involved other people (e.g., family /friends/peers) in getting the most out if the WSM program.

1 = No involvement 2 = Low level of involvement (discuss few of the lessons with others) 3 = Moderate level of involvement (e.g. discuss many lessons with others, received support such as help with reading some lessons) 4 = High level of involvement (discuss most lessons and received help with reading and completing action steps; others met with group leader to discuss their involvement 5) HEALTH CHECK UP: How many times did the person visit a physical healthcare professional (e.g., doctor, dentist, nurse, optometrist) in the past 6 months? 0 = never 1 = once 2 = twice 3 = three or more times 6) GOAL PROGRESS 1 = Person is doing worse in many of their goal areas 2 = Little or no progress made in any goal area 3 = Significant progress made in one or more goal areas 4 = Achieved one or more goals

2) PARTICIPATION 1 = Infrequent participation in group discussion even when prompted 2 = Usually participates only when prompted 3 = Often participates without prompting 4 = Very active participation throughout in group discussion throughout the program without prompting 3) ACTION STEPS: the person completed Action Steps for: 1 = less than 25% of the lessons he/ she completed 2 = 26-50% of the lessons he/ she completed 3 = 51-75% of the lessons he/ she completed 4 = 76-100% of the lessons he/she completed

27

Additional Resources
Educational Supervision information WSM Participant Feedback Survey WSM Group Leader Feedback Survey Lesson by Lesson Helpfulness Scale

28

Key Concepts of Educational Supervision

The outcome of a successful educational supervisory process is that the supervisee finds it
helpful

Supervisor and supervisee engage in a professional development process designed to reinforce

and enhance the skills and knowledge of both parties. parties choose to make it so.

Educational supervision is NOT part of a formal performance evaluation process unless both The supervisor is NOT assumed to have superior expertise in conducting WSM groups. In fact,
the supervisee may have greater expertise. Nonetheless, the supervisor typically has the responsibility to support and promote the professional development of those he/she supervises.

Educational supervision creates a time and place for colleagues to review the standards of

practice associated with conducting a WSM group. In this way, fidelity to the WSM service is reinforced.

The information from educational supervision assists the NYS OMH to evaluate the

effectiveness of the training provided as well as evaluating the learning collaborative method of disseminating WSM.

The educational supervisory method emphasizes the importance of supervisees engaging in a

Self-Assessment process. Consequently, educational supervision begins with the supervisee reviewing, completing and sharing his/her self assessment using the Core Competencies Checklist.

The supervisor shares his/her observations in a manner that emphasizes strengths. The supervisor
engages the supervisee in a discussion, The following questions may be considered: o What are your thoughts about todays group? o What were you pleased with? What would you have wanted to do differently? o In what way were you able to accomplish what you set out to accomplish? o What challenges did you face in todays group? o How helpful do you find the workbook and the use of the ROPES framework? o In what way could I be helpful in addressing areas you are concerned about?

29