POSITIVELY DYNAMIC EVALUATORS BABYS BEST START GROUP PROGRAM EVALUATION

Evaluation Tools

Evaluability Assessment Research Questions Data Collection Tools

Focus Groups Survey Retrospective Chart Review Program Documentation BORN Ontario Data Registry

Data Analysis

Evaluability Assessment Babys Best Start Group Program The health unit recognized the potential of using a group program service delivery to enhance not only program efficiencies but to strengthen knowledge, capacity and relationship building of participants through the group model. Moving to the group model represents a major change for the program. With this pilot project the health unit is very interested in evaluating the group program delivery model. In public health utilization of evaluations is very much an expectation for government funding models. Evaluatbilty assessments help program planners identify gaps and make changes to program before going too far into a project plan. An evaluability assessment approach has demonstrated increase utilization of evaluations (Thurston, Graham, & Hatfield (2003); Patton, 2008). The goal of an evaluability assessment is to ensure a program problem to be addressed is clearly described and program theory is evident and makes sense. As part of the evaluation plan an evaluability assessment will be conducted based on the framework of Thurston (1991) which includes seven elements: a) b) c) d) e) f) g) Identifying program goals, objectives and activities Review of program documents Review of program logic model Observation of the program Is there a program model in place Identification of evaluation users and stakeholders Assessing assessment to proceed with an evaluation

Reference: Patton, M.Q. (2008). Utilization focused evaluation (4th ed.). Thousand Oaks, CA: Sage Publications Inc.
Thurston, W.E., Graham, J., Hatfield, J.(2003). Evaluability Assessment: A Catalyst for Program change and Improvement. Evaluation and the Health Profession, 26(2), 206- 221.

Evaluation Research Questions

1. To what extent are program participants satisfied

with the group service delivery model?

2. To what extent does the group model influence

healthy lifestyle choices during pregnancy? 3. To what extent does the group service delivery model impact program efficiency?

1. To what extent are program participants satisfied with the group service delivery model? To answer this research question a few different approaches will be used. Participants will be able to provide feedback to the program coordinators by completing a satisfaction survey at the end of the 12 week group program session. Focus Groups with participants icipants will also be used to inform program evaluation and quality improvement. Data on participant satisfaction will be obtained from: Participant satisfaction isfaction survey at the end of 12 week group program Participant focus group

2. To what extend does the e group model influence healthy lifestyle choices during pregnancy? Answering this question through evaluation requires assessment of participants behavior change from participating in the group program. The group program provides participants with connections tions and potential support networks that one on one counselling ling does not provide. Assessment of the value of connections and knowledge shared and support for behavior change is key to answering this evaluation question. BBS Resources required to analysi analysis this question: Dietician Program staff

To answer this research question the following data and analysis will include: Participant Survey following completion of the program Focus Group with BBS Group Program participants Dietician chart review of intake interview and nutritional assessment and post delivery discussion and assessment Number of referrals to external community partner resources for extra support

3. To what extent does the group service delivery model impact program efficiency? The data collection choices impact the evaluator and program managers time. Chart reviews and population health data may be cumbersome to obtain and time intensive. Using a mixed method approach helps to provide depth and context to the evaluation. Focus group sessions provide an opportunity for participants to share ideas and provide broad insight that may help improve the problem being evaluated (THUC, 2002.) The disadvantage of focus groups is the potential for influencing between participants and the special skill set and experience required for skilled analysis of focus group themes. To evaluate the service delivery model impact on program efficiency a review of financial reports and program monthly reports. To answer this research question the following data and analysis will include:
1) 2) 3) 4) 5)

Rate of healthy term weight babies from BORN Ontario data registry Focus Group with Community Partners Number of FTE devoted to program delivery Program financial statements Program monthly reports

Answering this question through evaluation requires a careful assessment of the costs of program implementation in relation to the program outcomes

Resources for Implementing the BBS program People - dietitian(s) - public health nurse(s) - program manager - evaluator

-staff - volunteers Economic Resources - salaries - venue (in-kind) - stationary and supplies - computer and software - food and cooking equipment - refreshments for participants

Information Resources - literature sources - research - Best Start Network Advisory Committee - participants feedback - participant attendance logs - information delivery methods (projector / pamphlets / bulletins / posters)

Competencies - staff training and education

Time - time required for program implementation in group format (participants reached per time allocation)

Obtaining the Data

Data on resource needs and use are obtained from: - financial and budget statements (costs incurred for staffing and delivery of services) - staff scheduling and hours worked - hours recorded for implementing group program - training hours and cost

Reference: The Health Unit Communication Unit, (2002). Using Focus Groups http://www.thcu.ca/infoandresources/publications/Focus_Groups_Master_Wkbk_Complete_v2_content_06.30.00_ format_aug03.pdf

FOCUS GROUPS
A focus group is a formal discussion lead by an expert facilitator to facilitate discussion and information gathering from a small group of identified participants. Focus groups ideally should be kept under 12 people with at least 6 people participating. According to The Health Communication Unit (2002), Focus group sessions provide an opportunity for participants to share ideas and provide broad insight that may help improve the problem being evaluated. Focus group moderators must be skilled in group process ocess as well as how to guide the discussion to keep participants focused on the topic yet allow for some flexibility to explore key themes. When considering the use of focus groups in evaluation facilitators must keep in mind how they are going to recruit participants and ensuring they have venue and timing of focus group that meets potential participants. In evaluation focus groups is a qualitative approach that requires experience and practice to become skilled in analysis of key themes that the discussions discussi provide. Advantages of Focus Groups Reasonable cost and allows for a group of participations to provide information at one time Variety of opinions can be expressed at one time compared to one on one indepth interviews Provides a depth of insight and personal stories that may assist in assessment of the problem being investigated.

Disadvantages of Focus Groups The potential for influencing between participants and the special skill set and experience required for skilled analysis of focus group t themes. Challenging to analysis the data based on the subjective views of participants and requires skill in theme and quantitative analysis Faciliator skills influence the quality of the discussion

The Health Communication Unit, (2002). Using Focus Group Groups

http://www.thcu.ca/infoandresources/publications/Focus_Groups_Master_Wkbk_Complete_v2_content_0 6.30.00_format_aug03.pdf

Community Partners & Participant Focus Groups Moderators Guide Introduction Thank you for taking time out of your day to participate in this focus group. We have scheduled 1 hour for this focus group. Your feedback and comments will assist us with improving delivery of the Baby's Best Start program in Cornwall. The pilot project to test providing this service in a group model would not have been successful without your support and contribution to this program. This focus group will inform the overall evaluation of the BBS group program. This session will be recorded and there are no wrong or right answers. Your responses are confidential and will be coded so they cannot be traced back to you. Participation in this focus group is voluntary. Your honest responses or refusal to participate will not have any impact on our program. Key Content Section Three one hour community partner focus groups will be held during the 18 month evaluation. The community partners consist of: 7 Grocery stores that provide food coupons 2 Community Health Centers 1 Ontario Early Years Center 4 Obstetricians

Three one hour participant focus groups will be held during the 18 month evaluation. The participants will be invited to be part of the focus group. Community Partners Focus Group Questions: How would you describe your communication with the BBS Program Team? Do you know who your key contact is for the BBS Program? Were you consulted in the program planning for the Group BBS Program? Are you able to clearly identify the BBS Team expectations of your contribution to the Group Program? 5) Has the Group Program service delivery increased your interaction and contribution with the BBS Team? 6) Have you identified increased access to your programs/services by former BBS Group participants? 1) 2) 3) 4)

Participant Focus Group Questions: 1. How did the program team involve you in identifying themes for the weekly sessions? 2. What is your understanding of factors that contribute to a healthy pregnancy and baby? 3. Could you describe how these sessions have or have not impacted your lifestyle choices? 4. How would you describe your favorite or least favorite activities of the program? 5. What is your overall satisfaction with the various program components such as weekly education sessions, food preparation activities, free vitamins and food and bus coupons? 6. If you were to describe this program to a friend what would you say?

Summary Moderator will ensure time is provided to check in with each participant to ensure their voice is added to the focus group. This will be a time for further comments and feedback from participants.

Closing: Moderator will thank participants for sharing their valuable opinions and suggestions for improvement to the BBS group program. Inform participants that the recorded sessions will be analyzed for common themes to inform the final evaluation. A draft of the themes will be compiled and emailed to the participants for review and they can provide changes or input to the information back to the focus group lead. Focus group participants will receive a copy of the final evaluation plan

SURVEY RESEARCH

A survey is a systematic method of collecting data from program participants or a population of interest. It is generally quantitative in nature Data is usually collected through the use of structured and standardized questionnaires The purpose of conducting a survey for the BBS evaluation is to determine participants satisfaction with the group model of prenatal nutrition education The BBS participant satisfaction survey will address the perceptions of pregnant women in Cornwall who are attending the BBS group program. Participants will be presented with the survey at the completion of the 12week program.

Internal resources are required for the survey implementation. These include funding, staff, equipment (telephones), computer for data entry, and software for data analysis Issues being addressed include: - program venue - timeliness and appropriateness of services - community resource information and usefulness - coupons and vitamin usefulness - nutritional counselling and follow-up usefulness - knowledge acquisition - overall satisfaction with the BBS program The survey findings will be used to determine participants perceptions, opinions, and attitudes toward the BBS program The survey findings will be used to evaluate the effectiveness of the BBS program from the participants perspective The survey does not replace qualitative techniques but, rather, is used in conjunction with qualitative methods Community partners have an interest in the results. They include Public Health administrators, BBS program administrators, BBS program providers, BBS program funders, and volunteers. The results are used to contribute to decisions regarding future implementation of the BBS group program Advantages of Surveys Can gather a large amount of data in a short amount of time Can be completed in person or by mail, telephone, e-mail, or fax It is quantifiable and generalizable Can take less time to analyze than qualitative data

Disadvantages of Surveys Can be expensive Requires statistical and other specialized knowledge and skills to analyze and interpret results Difficult to collect in-depth information of respondents perspectives The Health Commission Unit (1999). Conducting survey research. Retrieved from http://www.thcu.ca/infoandresources/publications/Surveys_Master_Wkbk_ V2_formatting%2008.09.03_Content%2003.31.99.pdf

BBS Participant Satisfaction Survey

Participant Satisfaction Survey Babys Best Start (BBS) Group Program
We invite you to complete this survey in order to assist us with improving delivery of the Babys Best Start (BBS) Program. Your responses are confidential and cannot be traced back to you. Participation in this survey is voluntary, and honest responses or refusal to participate will not have any impact on your care in our program. Please feel free to ask your dietitian or nurse if you have any questions about this survey. Please put your completed survey in the envelope provided and drop it in the surveys' box. If you have any question about how this information will be used, you can contact the BBS program manager, Patti Gauley at 613-123-4567 or at pgauley@abcd.ca. Thank you for your valuable feedback! 1. Is the BBS program site convenient for you?
Always Sometimes Rarely Never Not Applicable

2. Do you feel welcomed by the program staff?

Always Sometimes Rarely Never Not Applicable

3. Do you think the program provides appropriate and timely services for pregnant women?
Always Sometimes Rarely Never Not Applicable

4. Were you asked to provide input/ideas for the weekly group sessions?
Always Sometimes Rarely Never Not Applicable

5. Have you learned about healthy food choices and balanced meals preparation?
Always Sometimes Rarely Never Not Applicable

6. Do the program activities meet your learning needs?

Always Sometimes Rarely Never Not

Applicable

7. Were you informed about the community resources that support a healthy pregnancy?
Always Sometimes Rarely Never Not Applicable

8. Were you able to access these community resources and did you find them helpful?
Always Sometimes Rarely Never Not Applicable

9. Did you find the program vitamins, food coupons and bus tickets useful?
Always Sometimes Rarely Never Not Applicable

10. Did you find the nutritional counselling by the dietitian useful?
Always Sometimes Rarely Never Not Applicable

11. Has the program increased your awareness of healthy lifestyle choices during pregnancy?
Always Sometimes Rarely Never Not Applicable

12. Do you feel that you are making healthier lifestyle choices as a result of the program?
Always Sometimes Rarely
Done

Never

Not Applicable

Thank you for taking the time to complete this survey. Your feedback is valuable for the program.

Retrospective Chart Reviews

Chart reviews are used as a qualitative tool in evaluation and research projects. A retrospective chart review involves the secondary use of data collected in chart documentation. Information and data contained in charts/records was collected for a specific purpose is later examined for a purpose other than the reason it was initially collected. Development of a data capture sheet with clear instructions on how to collect and code the data assists in the process and analysis of chart information. Retrospective chart review is an important methodology with distinct advantages and some disadvantages. To address the disadvantages Gearing et al, 2006 recommend a nine step method to maximize the benefits of chart reviews. This method can be used by evaluators who value the information from chart reviews. Benefits Valuable source of data Helps to inform evaluation and research questions Inexpensive method to obtain rich source of existing data

Disadvantages Confidentiality issues ( storage of charts when being reviewed, access by how many people) Less control over predictability of casual relationships Incomplete documentation Missing charts Challenges in interpreting information in the documentation Internal validity: o selection bias o Researcher bias External Validity: o Generalizability of results Reference:

Reference: Gearing, E., Mian, I, Barber, J., & Ickowicz, A. (2006). A Methodology for Conducting Retrospective Chart Review Research in Child and Adolescent Psychiatry. Journal of the Canadian Academy of Child and Adolescent Psychiatry. 15(3), 126- 134

Better Outcomes Registry Network (BORN) Ontario

BORN Ontario is a pregnancy, birth and childhood registry and network. Data is collected, interpreted and shared about all births occurring across the Province. Reports are compiled for Local Health Integration Networks and public health regions. Some of the data available include teen pregnancy rates, multiple births, delivery type, prenatal class attendance, rate of preterm births, and rate of low birth weight babies, infant feeding and smoking rates during pregnancy (BORN, n.d.) Benefits Data sharing is available to a broad range of stakeholders in real time (BORN, n.d.) All hospitals/birthing sites across Ontario enter data into this registry/network (BORN,n.d.).

Figure 4.10 Rate of low birth weight (<2,500 grams), by public health unit Eastern Region, 2008

Data source BORN Ontario (Niday Perinatal Database), 2008 Definition of indicator The number of live births with a birth weight of <2,500 grams, expressed as a percentage of the total number of live births (in a given place and time).

Limitations Data provided in the reports are descriptive statistics. No statistical tests have been completed on these data; therefore interpretation of data over time or across subgroups should be done cautiously (BORN,n.d.). Although data is to be available real-time, this may not always be the case because of staffing constraints.

Figure 5.1 Rate of preterm birth (<37 weeks of gestation) among singleton live births, by maternal smoking during pregnancy Eastern Region, 2008

Data source BORN Ontario (Niday Perinatal Database), 2008 Definition of indicator The number of live births with a gestational age of <37 completed weeks of gestation, expressed as a percentage of the total number of live births (in a given place and time). For this figure, the analysis is restricted to singleton live births.

Program Documents
Various program documents will be reviewed in the process of collecting data for the Babys Best Start Group Evaluation. These documents include daily time tracking records, financial statements, monthly reports and participant charts. Benefits These data sources are already available. Program staff and manager use these tools on a regular basis (Wholey, Hatry & Newcomer, 2004). These tools are useful in providing a picture of what is happening within a program (Wholey, Hatry & Newcomer, 2004).

Limitations These tools alone often do not provide the necessary data to measure outcomes in an evaluation (Wholey, Hatry & Newcomer, 2004). Data errors can occur during record keeping thus affecting the quality of the data (Wholey, Hatry & Newcomer, 2004).

Better Outcomes Registry Network [BORN} Ontario (n.d.). Perinatal health report 2008 Eastern Ontario public health region. Retrieved from https://www.bornontario.ca/en/publications/reports/publichealth-region-reports/

DATA ANAYLSIS FOR BABY BEST START GROUP PROGRAM PILOT PROJECT
The purpose of this evaluation is to assess the effectiveness and efficiency of the twelve-week prenatal group service delivery model in meeting the Baby Best Start program outcomes. According to MacLellan, Bradley & Brimacombe (2001), epidemiological evidence has shown that adequate prenatal nutrition care is associated with improved birth outcomes and a supportive environment assists pregnant women experiencing multiple stressors; however, the relative importance of a specific program care components is difficult to assess. This program follows a population health framework for gathering and analyzing information about conditions that affect the health of high-risk pregnant women. Population health is a structured approach to health promotion and disease prevention that looks at how lifestyles and living conditions affect the health of individuals and a population group. This framework incorporates public health initiatives and community development to maintain and improve the health of this population and to reduce inequities in health status among this group (Thompson 2010). A mixed-methods approach capitalizes on the strength of both qualitative and quantitative evaluation methods. Mixing methods in evaluation validates the accuracy of findings, allows greater comprehensiveness, and provides insight into the program processes linked to outcomes (Bowers, Cohen, Elliot, Grabowski, Fishman, Sharkey, Zimmerman, Horn & Kemper (2013). According to Patton (2008), widespread agreement now exists that a combination of evaluation methodologies is required to understand the mechanism under which programs can effect change in peoples lives. QUANTITATIVE DATA The quantitative data collection method used to evaluate participants satisfaction is a survey comprised of thirteen close-ended questions (five-step Likert scale) that was pretested for clarity, comprehension and content. The most important thing is that we need to keep our original purpose of the evaluation and the research questions in mind. The participant satisfaction survey results measure our evaluation questions so it is important to organize the results according to the original research questions and use the results to answer these questions for comprehensive survey findings.

(Google Images, 2014)

o A participant satisfaction survey hard copy will be handed to participants at the end of the 12-week program to solicit their feedback and input regarding their experience in the program. They will be asked to put the completed survey in the envelope provided and drop it in the surveys box. o They will also be encouraged to contact the program manager, dietitian or public nurse if they have any questions about the survey or how the survey information will be used. They will be ensured that their responses are confidential and cannot be traced back to them. o The program administrative staff will collect the surveys from the box and enter them into SurveyMonkey statistical package for descriptive statistics to be generated. o The analysis of survey results will provide question summaries presented in spreadsheet graphs. o The statistical package reports include data trends, summary data, responses data and individual responses which will also provide profiles of full survey responses. o For the participant survey analysis, only a descriptive statistical report is needed to interpret the survey results as all the questions are closeended. The data analysis is required to answer the research questions and draw conclusions. o Statistically analyzing the results is one of the most crucial steps in the process of ensuring useful findings that will accurately reflect the opinions and views of the program participants involved. o The statistical analysis provides a way to determine the repeatability of any differences observed. QUALITATIVE DATA Focus groups with participants are 3 over the 18 month evaluation period Focus groups with community partners are 3 over the 18 month evaluation period The analysis of focus groups data consists of objectively reviewing transcripts and identifying the main themes which answer the evaluation questions. The focus group facilitator presents the transcribed information to the focus groups for validation and further discussion. Affinity diagrams will be used by the focus group to interpret the data collected. The Affinity diagram is a brainstorming visual tool that allows an individual or a team to group a large number of ideas, issues, observations or items into categories for further analysis (Quality Progress, 2012). Ideas of a natural relationship or relevance are placed together in the same category.

Benefits of an Affinity Diagram data analysis method are: Actively involves participants and community partners in the specifics of the group program delivery that requires their understanding, experience, knowledge and support. Generates creative associations and breakthrough insightful thinking Further analyzing data, ideas or observations for hypothesis testing, prioritization and decision making Gain consensus among participants and community partners regarding issues, ideas, observations or items subject for discussion

Data collected analysis through the mixed method should address program evaluation questions. The data analysis results are a means to implement program changes as suggested by these results. It is critical to take action and implement changes in order to make improvements to the program. The key point for utilization-focused data analysis is to involve participants, staff and community partners in interpreting and taking action on result (Patton, 2008). Prioritizing program changes is based on the most important and feasible to implement. The in-depth analysis of both the quantitative and qualitative evaluation methods will directly feed into recommendations for the program improvements. For the dissemination process, it is crucial to provide the participants and community partners feedback results to program participants, staff and community partners through a written report and a presentation to keep everyone on the same page. This will establish buy-in and facilitate the implementation of the evaluation report recommendations in the future. In the interpretation of data analysis results, the evaluation team needs to communicate limitation of the data collected e.g. possible biases in addition to validity, reliability and generalizability of the results.

References: Affinity Diagrams Basic Quality, Quality Progress (2012). American Society for Quality, Inc. 2021. Bowers, B., Cohen, L. W., Elliot, A. E., Grabowski, D. C., Fishman, N. W., Sharkey, S. S., Zimmerman, S., Horn, S. D. & Kemper, P. (2013). Creating and supporting a mixed methods health services research team, Health Services Research, 48(6), 2125-2208. MacLellan, D., Bradley, D., & Brimacombe, M. (2001). Stakeholder Evaluation of a high-risk prenatal nutrition intervention program in Prince Edward Island, Canadian Journal of Dietetic Practice and Research, 62(4), 182-187.

Patton, M. Q. (2008). Utilization Focused Evaluation (4th ed.). Thousand Oaks, CA: SAGE Publications, Inc. Thompson, V. (2010). Health and Health Care Delivery in Canada (1st ed.). Elsevier Canada Publications, Ltd.