Evaluation and Program Planning 45 (2014) 18

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier.com/locate/evalprogplan

Using implementation evaluation to uncover a child neglect

prevention program
Annie Berube a,*, Vicky Lafantaisie b, Diane Dubeau a, Sylvain Coutu a,
Josee Caron c, Annie Devault a
a

Universite du Quebec en Outaouais, Canada

Universite du Quebec a` Trois-Rivie`res, Canada
c
Centre de services sociaux et de la sante de Gatineau, Canada
b

A R T I C L E I N F O

A B S T R A C T

Article history:
Received 26 October 2012
Received in revised form 8 January 2014
Accepted 1 March 2014
Available online 11 March 2014

Child neglect is an ecosystemic problem with a great variety of risk factors to consider and, therefore, it
requires a multimodal and individualized intervention. Although such an intervention is better for the
families, it represents a great challenge for the evaluation process.
Objectives: The purpose of this study is to document, using Dane and Schneiders model (1998), the
differences between the services received by parents participating in a parental group designed to
prevent the presence or the recurrence of child neglect.
Methods: Quantitative program implementation data was collected from 50 families who took part in a
four-module program over a two-year period.
Results: The results demonstrate uniformity with regard to the programs central elements despite the
differences in the services each family received. Adherence to the program was mainly respected despite
slight variations in the number of sessions offered and in the group sizes. On the other hand, dosage
varied greatly, with families attending from one to four offered modules. For each module, attendance
varied from participation in one group session to participation in all ten group sessions. Moreover, for
families who participated in at least two modules, attendance signicantly increased between the rst
and second module. The families level of participation also differed, with families being rated from low
to highly engaged at the end of each group session.
Conclusions: Interventions must be adjusted to the specic needs of the clientele and to the
characteristics of the environment in which they will be implemented. These variations could have
important impacts on the effects of the intervention on the families. Therefore, evaluating these
programs requires the consideration of these variations and of their repercussions on the programs
effects.
2014 Elsevier Ltd. All rights reserved.

Keywords:
Program evaluation
Child neglect
Implementation

1. Introduction
The evaluation of programs advocating multimodal intervention is a daunting task due to the variety of services offered. A
single program may offer the possibility of participating in a great
range of services such as individualized and group activities or
home visits that tackle various themes. This is the case with a vast

This article received nancial support from ministe`re de la Sante et des Services
Sociaux, as well as from the Groupe de recherche et daction sur la victimisation des
enfants (GRAVE).
* Corresponding author at: Departement de psychologie et de psychoeducation,
Universite du Quebec en Outaouais, C.P. 1250, succ. Hull, Gatineau, Quebec, Canada
J8X 3X7. Tel.: +1 819 595 3900x2547.
E-mail address: annie.berube@uqo.ca (A. Berube).

http://dx.doi.org/10.1016/j.evalprogplan.2014.03.001
0149-7189/ 2014 Elsevier Ltd. All rights reserved.

majority of programs aiming to support families where child

neglect is an issue.
Two meta-analyses on interventions targeting child neglect
concluded that family interventions often lead to minor changes
that can be observed through small, or very small, effect sizes
(MacMillan et al., 2009; Pinquart & Teubert, 2010). Indeed, the
MacMillan et al. (2009) study found that there were no programs
capable of preventing the reoccurrence of neglectful behavior.
Furthermore, the results shown in the study performed by
Barlow, Johnston, Kendrick, Polnay, and Stewart-Brown (2006)
demonstrated the difculty of modifying the trajectory of
families when it came to preventing the reoccurrence of child
neglect.
This article aims to highlight the possible complexity and
variability of an intervention program instated to meet the various

A. Berube et al. / Evaluation and Program Planning 45 (2014) 18

needs of families living in a context of neglect. What each family

was exposed to should be measured before to conduct an impact
evaluation. Doing otherwise could lead to great difculties in
detecting statistically signicant changes when measuring effects.
This article follows the special issue recently published by
Administration Policy in Mental Health (2011, 38), in which many
authors studied the methodology adopted in research surrounding
the evaluation of services provided to children under child welfare
services (Horwitz & Landsverk, 2011; Landsverk, Brown, Reutz,
Palinkas, & Horwitz, 2011; Schoenwald et al., 2011).
1.1. Child neglect
Child neglect is a very serious problem that modern society has
not yet succeeded in stemming. In fact, current data shows that
child neglect is still the most prevailing form of child maltreatment
(Association des Centres jeunesse du Quebec, 2010). According to
the most recent U.S. National Incidence Study of Child Abuse and
Neglect (NIS-4), when applying the Harm Standard, neglect
accounts for 61% of all forms of maltreatment, affecting roughly
770 000 children in the U.S. Whereas child abuse has been
declining since it reached its peak in 1993s NIS-3, child neglect has
stayed at the same level. Moreover, when the more inclusive
Endangerment standard is applied, taking into account injured
children as well as those considered to be in danger, the data show
an important increase in emotional neglect. Indeed, the number of
children in that situation has more than doubled (Sedlak et al.,
2010).
The consequences of child neglect on children can be severe,
ranging from physical and mental health problems to an increased
risk of developing patterns of behavior leading to aggression and
criminality (Gilbert et al., 2009). For instance, a history of either
childhood emotional or physical neglect increases the odd ratio of
illicit drug abuse in early adolescence by about 150% (OR of 2.4 and
2.5 for emotional and physical abuse respectively), and the lifetime
odd ratio by 3080% (OR 1.31.8) (Dube et al., 2006; see also Dunn
et al., 2002; Topitzes, Mersky, & Reynolds, 2010). Moreover,
children living in a neglectful environment appear to show
difculties mostly regarding their social and emotional development. Findings show that, after controlling for the effect of
maternal depression and socio-demographic risk factors, psychologically neglectful environments are linked to internalizing and
externalizing problem behaviors in children as young as three
years old (Dubowitz, Pitts, & Black, 2004; Dunn et al., 2002).
Furthermore, children having experienced neglect during early
childhood are four times more likely to show delinquency in
adulthood than children living in similar socio-demographic
conditions but not exposed to this form of maltreatment
(Kazemian, Spatz Widom, & Farrington, 2011).
Growing up in a neglectful environment also impacts childrens
emotional development. Children raised in such a context exhibit a
weaker understanding of others negative emotions as well as a
decreased capacity for emotional self-regulation (Shipman,
Edwards, Brown, Swisher, & Jennings, 2005). Similarly, neglect
strongly inuences the intellectual capacities of the child, as
demonstrated by results nearly a standard deviation below
average in reading, mathematics, and intelligence (Lounds,
Borkowski, & Withman, 2006). As a matter of fact, neglects
repercussions on the cognitive, social and emotional development
are more severe than those caused by physical abuse (Hildyard &
Wolfe, 2002).
Over the years, researchers have attempted to understand the
phenomenon by which neglect is formed and perpetuated within
families. There seems to be a consensus regarding the necessity of
adopting an ecosystemic and developmental approach focused on
the environmental components that inuence the answer to the

developmental needs of the child rather than simply focusing on

inadequate parental behavior (Brown, Cohen, Johnson, & Salzinger,
1998; Dubowitz & Bennet, 2007; Tanner & Turney, 2003).
According to this point of view, child neglect is dened more as
the absence of behaviors that are benecial to the child than by
toxic parental behavior (Ethier, Bourassa, Klapper, & Dionne, 2006).
Supporting this afrmation are the ndings that neglect is linked to
many social factors such as single parenting, unemployment, and
the lack of attendance to childcare much more so than any other
forms of abuse (Mersky, Berger, Reynolds, & Gromoske, 2009).
According to Trocme et al. (2005), neglect occurs as a result of a
double disruption: (a) a disruption of the parentchild relationship
characterized by the parents signicant difculty in demonstrating an emotional availability for the child, thus jeopardizing his/
her physical integrity and development on many levels, whether it
be physical, cognitive, emotional, or social; and (b) a disruption of
relationships between the family and the community, characterized by a functional and social isolation of members of the family,
which leads to the absence of adequate alternatives for answering
the childs needs.
1.2. Evaluating interventions targeting child neglect
The scope of the problem of neglect and its consequences on
children motivates the creation of interventions targeting affected
families and, the effectiveness of these interventions is at the
center of a growing body of research. Many studies concentrate on
maltreatment in general, thus including abusive behavior and
neglect. The meta-analyses of Pinquart and Teubert (2010) and of
Mikton and Butchart (2009) compile the evaluations of more than
a hundred studies on the matter. Studies evaluating programs
particularly aimed at neglect are rarer despite many authors
insisting on the necessity of answering this problem specically
(MacMillan et al., 2009; Mersky et al., 2009).
Amid the studies examining neglect, the one of DePanllis and
Dubowitz (2005) evaluated the effectiveness of home visits paired
with collective activities with families identied as being at risk of
neglect. Results indicated an attenuation of many risk factors
associated with neglect such as parental stress, parental depression, and daily stress, as well as an increase in protective factors
(i.e., parental competencies, social support, and family functioning). Overall, this resulted in an improvement of the physical and
psychological care and security of the child and in a decrease of
external and internal behavioral disorders in children.
A similar study was completed by Barlow et al. (2007). An
intensive home visit program was compared to a control program
where home visits were less frequent (an average of 41.2 compared
to 9.2 home visits over 18 months). The results showed no
difference between the two groups for quality of the home
environment and with regard to protective issues concerning the
children. However, mothers level of sensibility signicantly
increased over the intervention period for the more intensive
intervention.
The study performed by Ethier, Couture, Lacharite, and Gagnier
(2000), which focused on families facing many risk factors but
never having benetted from services of child welfare services,
drew the same conclusion. The study compared the effects of two
services offered to neglectful families, one psychosocial intervention embedded in regular services to at-risk families and one
multimodal program offering individualized and group interventions to parents and their children. Both interventions showed
positive results by improving parentchild relationships and
decreasing parental stress, depression and potential for abuse
and neglect. However, only the multimodal program was
associated with an improvement in social support and marital
relationships.

A. Berube et al. / Evaluation and Program Planning 45 (2014) 18

On the other hand, other reviews of child neglect interventions

highlight the fact that few studies have demonstrated solid results
and that most are based on a small sample size (Allin, Wathen, &
MacMillan, 2005; Barlow et al., 2006; MacMillan et al., 2009). The
results of these studies tend to demonstrate the difculty of
inuencing the trajectory of families already living in a context of
neglect.
Given the importance of the problem and the severity of its
consequences on those that experience neglect, it is important to
reach a consensus on the practices that lead to a noticeable
improvement of the childs situation. Two issues require further
research. First, little attention has been given to the delity of the
interventions received by participants and to how it inuences the
programs effects (Horwitz & Landsverk, 2011; Mildon & Shlonsky,
2011; Pinquart & Teubert, 2010).
Many authors underline the importance of taking into account
the implementation delity when it comes to analyzing services
offered to children living in a context of great vulnerability
(Landsverk et al., 2011; Pinquart & Teubert, 2010; Schoenwald
et al., 2011). In fact, individualized programs show a wide variety
in the kind of intervention each family actually receives. For
example, in their meta-analysis, Pinquart and Teubert (2010)
reviewed 142 papers documenting the effects of different
prevention interventions. The average length of an intervention
was of 15 months, ranging from one day to 60 months, with
participants attending an average of 29 meetings, with a range of
one to 421 meetings. This example illustrates how widely different
the interventions received by the parents of the study sample could
be. Despite these differences, the data are all agglomerated and
presented together in the results section. The authors recommend
verifying elements such as the duration of the intervention and the
qualications of the worker as well as the type of intervention used
in order to better interpret the changes observed in the families.
Mildon and Shlonsky (2011) share the same line of thinking when
they underline the little attention that is given to the implementation of services offered to clients of child protective services.
Implementation data allows a better understanding of specic
program components and a better interpretation of the effects of
the intervention in a context where uniformity of services is
neither wanted nor desirable. Indeed, interventions targeting child
neglect require a swift and situation-specic intervention. When
such an approach is adopted, matters of dosing, quality, and
participation in the program are not pre-established but rather
measured in order to understand their contribution to the
repercussions of the program on its participants.
The World Health Organization recently released a report that
highlighted the necessity of documenting the effectiveness of
interventions targeting child neglect (Butchart, Harvey, Mian, &
Furniss, 2006). What researchers have yet to determine is which
approach is best suited to evaluating programs whose goal is to
prevent or to treat the problem of neglect while numerous risk
factors push parents to focus on their own needs rather than on
those of their children (Brown et al., 1998; Dubowitz & Bennet,
2007). Given the adverse consequences of neglect and the
contradictory conclusions to which researchers have arrived so
far, we believe the evaluation of interventions targeting child
neglect needs to be rethought. An evaluation combining a study of
the programs implementation and of its effects could provide
answers to still unanswered questions as well as provide a deeper
understanding of the characteristics of the core components of
those interventions.
1.3. Implementation delity evaluation
A growing number of studies demonstrate that it is easier to
determine changes linked to a program by adopting an approach

that analyses its effects and takes into account the contextual
elements to which participants are exposed (Domitrovich &
Greenberg, 2000; Durlak & DuPre, 2008; Fagan, Hanson, Hawkins,
& Arthur, 2008; Lee et al., 2008). Durlak and DuPre (2008) conclude
in their meta-analysis that effect sizes are two to three times larger
when analyses account for program delity.
Gradually, researchers have reached a consensus on a denition
of the elements that characterize what many refer to as an
implementation delity evaluation. Dane and Schneider (1998)
have elaborated a conceptual model that aims to document the
factors of a program that are likely to affect its outcome. This model
has been used in other studies, such as those by Durlak and DuPre
(2008), Domitrovich and Greenberg (2000), Fagan et al. (2008), Lee
et al. (2008), and Paquette, Tourigny, and Joly (2009), among
others. The model is composed of the following ve aspects: (1)
adherence or consistency between practices suggested by the
program and practices actually implemented; (2) quality of the
intervention (e.g., the degree of conversion of the program by the
worker); (3) dosage or exposure of the participants to the program;
(4) participation, such as the response of participants to the
program; (5) differentiation or the difference between what is
suggested by the program and what has already been implemented. The advantage of this model is that it allows for a
categorization of the majority of the information gathered in the
context of the implementation evaluation and that it helps
establish links between the way the intervention has been
implemented and the observed changes.
The purpose of this study is to document, using Dane and
Schneiders model (1998), the differences between the services
received by parents participating in a parental group designed to
prevent the presence or the recurrence of child neglect. The
research question is therefore to identify what were the programs
variations over two years regarding adherence, dosage, quality,
parent participation, and differentiation. More specically, we will
study the program adherence in terms of frequency and duration of
the meetings, number of parents in each group and themes
broached during the meetings. For dosage, we will look at the
number of sessions attended by participants and whether the
attendance is related to the parents characteristics, the length of
participation in the program or programs topic. Regarding quality,
we will examine if the quality of the group atmosphere, as well as
participants appreciation of the program were similar for each
group and whether parents consider that the program responded
to their needs. Concerning the degree of participation, the idea is to
identify if it varies from one participant to another and if this
variability is related to parents characteristics, length of participation in the program or its topic. Finally, in terms of differentiation, we will examine if participants were similarly involved in
other social services. This is an exploratory study conducted in
order to verify the pertinence of documenting these dimensions
when considering a broader evaluation of the programs effects on
participants.
2. Method
2.1. Participants
Data collection was completed between the fall of 2008 and the
spring of 2010. In total, 49 parents (45 mothers and 4 fathers)
agreed to take part in at least one parent group. Four inclusion
criteria and one exclusion criterion were observed to recruit
families willing to participate in the program. First, participants
must be parental gures to children aged 012 years old. The
second criterion stipulates that families must be admitted on the
basis of neglectful behavior that compromises the development or
security of their children. Both of these criteria were respected by

A. Berube et al. / Evaluation and Program Planning 45 (2014) 18

all of the families enrolled. According to the third inclusion

criterion, the situations of neglect must be enduring within the
family despite previous interventions. It appears that all
families, except for one, were currently receiving or had
recently received services from the Child protection services.
The nal eligibility criterion concerns a factor of exclusion
allowing stopping the group component of the program for
parents whose issues were problematic in parent groups. During
the two years the study was conducted, no parent was excluded
from a group.
The socio-demographic portrait of the families reveals that
38.7% of the families were headed by a single parent. About threequarters of the parents had custody of their child (69.4%). The data
on the socio-demographic portrait completed by the intervenors
revealed that all participants but two were considered as having
nancial difculties. Drug use was prevalent in more than half of
the participants (51.0%), whereas alcohol problems were present
for 18.4% of parents. Symptoms of depression were also present in
more than one third of the sample (36.1%). Moreover, 26.5% of
participants showed signs of slow learning or intellectual
disability. Finally, 85.7% of parents were considered as having a
weak social network.

program philosophy, components and content are described in the

program manual by Lacharite et al. (2005).
The current study looks only at parent group activities. These
activities encourage parents to look at their familys situation
through four different modules: the parentchild relationship, the
parent as a parent, the parentenvironment relationship, and the
relationship with themselves. According to the manual, the program
is based on a series of three to ve modules that each includes 820
weekly meetings, lasting 2 or 2 h, and gathering 812
participants. Themes and activities are scheduled for four modules
parents (Lacharite et al., 2005). However, the program advocates
exibility, thus allowing parents individual and collective
experiences to be expressed.
The parent groups are co-facilitated by two psychosocial
intervenors, one from a rst line service (social services system)
and the other from a second line service (Child protection services).
Each facilitator has experience with neglecting families and
received an initial program training provided by the program
designer, which is followed by three to four annual training and
supervision meetings. The groups take place in a community
organization, putting parents into contact with three family
services in their community.

2.2. The program

2.3. Measures

The Programme daide personnelle, familiale et communautaire

deuxie`me generation (PAPFC2) (Personal, family, and community
assistance program, second generation) was launched by Carl
Lacharite and his team in 2005.
The rst generation of this program was evaluated by Ethier
et al. (2000). The results obtained at the time were modest yet
positive: following their participation in the PAPFC, parents felt
more competent in their roles; their support system was less
restricted; and their potential for abuse and high stress levels
decreased slightly.
Many changes were effectuated following this evaluation,
which led to the creation of the PAPFC2, now implemented in many
regions of Quebec. The PAPFC2 differs from other programs devised
to treat neglect. Rather than being centered on the education of the
parents, it is ecosystemic-based (Ethier et al., 2000). The
ecosystemic approach is often mentioned to explain the occurrence of situations of neglect, suggesting that individually
observed parental characteristics are not the cause of neglect
but rather that the interaction and the accumulation of various
personal, interpersonal, and social factors (e.g., drug addiction,
spousal abuse and poverty) are at the root of neglect (Lacharite,
Ethier, & Nolin, 2006; Tanner & Turney, 2003). Considering the fact
that deciencies are possible at different levels within these
families, the PAPFC2 suggests a multi-modal approach and targets
many different dimensions (e.g., the child, the parents, the parent
child relationship, the family, and the community).
The program is aimed at children between the ages of 0 and 12
years and at their parents. These families are at a high risk of facing
or are already facing personal, relationship, and social difculties
stemming directly from situations of neglect. In these cases, the
development or safety of the participating children is either
considered a concern or is compromised, and previous access to
services for the children or their families did not contribute to
improving their situation.
The PAPFC2 consists of a global approach through which various
partners (i.e., rst line services, child care services, community
organizations, etc.) are involved with the child and his/her family.
Five main activities compose this program: (1) analysis of the
childs needs and planning of services and interventions; (2)
professional, customized support; (3) direct actions aimed at the
child; (4) parental group; and (5) extra-curricular support. The

Family information sheet. The programs information sheet is

lled out by the familys main intervenor in order to refer the
family to the program. The intervenor has to identify which of the
childrens needs are not being met using a list (e.g., sufcient
nutrition, home security, attention, affection, stimulation. . .).
Intervenors also have to indicate which family risk factors
characterize the childs environment. They can chose as many
factors as applicable among a list of the parents personal
difculties (e.g., depression, alcohol abuse, drug abuse, slow
learning, intellectual disabilities, others) or social difculties (e.g.,
unemployment, nancial difculties, weak social support, others).
The childrens needs and the list of family risk factors were
elaborated using Ward and Roses (2002) model of needs
assessment in children. The information sheet also contains the
program inclusion checklist and a history of previous services
provided to the family. This last piece of information gives an
overview of the challenges related to differentiating the current
program from the previous services received.
Log books. Log books were lled out by the group facilitators
at the end of each meeting. They contained data about themes
discussed during the workshop. The information was collapsed to
form program adherence indicators. For each of the four proposed
modules, 59 themes are suggested. The facilitators have to
indicate which themes were tackled during the meeting and
which related activities were done. For each parent group, a score
was established by calculating the proportion of themes
pertaining to each module. The log books also indicated the
frequency and the duration of the family groups, which were then
compared to the manuals recommended frequency and duration
(a measure of adherence). The presence or absence of each
participant was also noted in the logbook, providing an individual
measure of dosage. Finally, the log books were used to measure
parents participation, a dimension included in Dane and
Schneiders model (1998). Facilitators indicated in the log books
the names of participating parents and their individual level of
participation, evaluated according to a three point scale (0
representing a lack of participation in the group, 1 representing
minimal participation, and 2 being considered an excellent
participation in the group). Scores were combined for each
participant in order to calculate an individual participation score
for each module attended.

A. Berube et al. / Evaluation and Program Planning 45 (2014) 18

Parents Workshop Appreciation Scale. Using a brief questionnaire at the end of each meeting, facilitators asked parents to
rate their appreciation of the meeting. On a scale of 110,
illustrated at both ends by a happy and an unhappy face, they
rated: their appreciation of the relationship they had with the
facilitators during the activity; the way the facilitators lead the
meeting; the content of the meeting; and their global appreciation.
This scale measures the quality of the activity from the parents
perspective.
Parents Global Appreciation Scale. This questionnaire was
lled out by the parents at the end of their group module. In this
questionnaire, parents rated their degree of satisfaction regarding
the parent group. On a scale of 110, they rated their level of
satisfaction regarding the workshops attended and whether or not
their expectations concerning the parent groups were met for
themselves and for their children. This scale provides a more global
appreciation of the programs quality.
2.4. Data analysis
In total, eight groups of parents received the intervention
during this data collection period. We conducted a series of
descriptive analyses in order to obtain a description of the
programs level of adherence, by analysing the themes broached
during each session as well as the frequency and duration of the
meetings. We also performed descriptive analyses on the parents
appreciation scales to account for the quality of the activities
included in each session.
Finally, we used inferential statistics (correlations and mean
comparison) to verify if the dosage and participation level were
related to the parents personal difculties, to time exposure to the
program or to the modules themes.

one module. In the two years during which data was collected,
parents participated in an average of two out of four workshop
modules offered (M = 1.98, SD = 0.99). The average number of
meetings parents attended during a workshop module was 5.39
(SD = 2.470), with individual scores ranging from attendance to a
single meeting to attendance to all eight to ten meetings proposed
within the module. Correlation analyses revealed that dosage is
neither related to participants personal difculties nor to
depression, alcohol or drug abuse, slow learning or intellectual
disabilities. Variance analyses were undertaken to verify if
attendance to workshops increased with parents participation
in the modules. It appears that the attendance of parents
participating for the rst time in the program was not as good
as that of those participating in a second module. The analyses
were performed on 30 participants that attended at least two
modules. During their rst participation, parents attended an
average of 3.77 (SD = 2.28) meetings, while those that signed up for
a second module attended an average of 6.33 (SD = 2.60) meetings
(T(29) = 4.53; p < .00). This signicant variation in dosage of
services offered to the participants reiterates the importance of the
integration of data in the implementation delity evaluation and of
their consideration when analyzing the effects of the program.
Other analyses were performed to verify whether or not the
topics broached during a module had an effect on the number of
workshops parents attended. The one-way analyses of variance
indicated a signicant difference in the participation of parents
depending on the module they were exposed to (F(3,94) = 3.29;
p = .024). Post hoc analyses using the Tukey HSD test indicated that
the only difference occurred between participation in workshop 1,
where the delicate subject of parentchild relationships was
broached (M = 4.28, SD = 2.29), and in workshop 4, which
concerned parents interactions with their environment and the
services available to them (M = 6.30, SD = 2.60).

3. Results
3.3. Quality
3.1. Program adherence
For this study, few components were observed for adherence:
the frequency and the duration of the meetings, the number of
parents per groups and the themes broached during the meetings.
The program advocates the importance of offering a exible
intervention that allows for adaptations in response to the parents
characteristics. The Outaouais region offers parents 4 types of
modules. In the past years, parents have participated in 810
meetings per module. That uctuation remains between the
numbers of meetings recommended. The program duration also
conforms to the program prescription as every meeting lasted two
hours or two hours and a half. The number of participants recruited
for these various models uctuated between 8 and 16. Five of the
eight groups included more than the recommended maximum of
12 participants. However, when looking at the number of
participants actually present during the meetings, only one group
had more than 12 parents that showed up to more than one
meeting.
Each module proposed between four and eight themes. The
proportion of themes tackled during a module was stable between
groups. However, for the two modules where the number of
themes was the highest, not all the themes were tackled. In one
module, eight or nine themes were presented, whereas for the
other module, the groups discussed only ve of the eight suggested
themes.
3.2. Dosage
Data related to dosage was collected from a group comprised of
49 parents (45 mothers and 4 fathers) who participated in at least

Facilitators evaluation of the global atmosphere during the

meetings varied from one group to another. For three groups, the
facilitators rated the atmosphere as excellent, with a mean of 2 out
of 2. The mean atmosphere for the ve other groups varied from
1.67 to 1.80. Although based on a small sample of groups (n = 8),
facilitators perception of the atmosphere was not related to the
parents appreciation of the activities.
The relationship with the facilitators was evaluated by
participants as being greatly appreciated, with scores ranging
from 9.10 to 9.75 on a 10-point scale. The participants also scored
their appreciation of the facilitators attitude as really positive,
ranging from 9.22 to 9.77.
Still under the umbrella of quality, the appropriateness of the
services offered in relation to the needs of the family must be
considered. Thus, a questionnaire aiming to address this issue was
distributed to the parents at the end of the workshops and the
results indicate that all of the parents except for one considered
that the workshops met their needs as parents as well as the needs
of their children.
3.4. Participation
Results indicate that the parents average level of participation
hovered at 1.62 (.38). For 16% of the sample, the average global
participation level was below 1 whereas for 20% of the parents, the
score was at the complete opposite of the spectrum, with an
average score of 2 out of 2. No signicant difference was noticeable
when comparing parents level of participation during the rst and
second workshop. Similarly, it appears that the parents level of
participation was not inuenced by the topic of the module they

A. Berube et al. / Evaluation and Program Planning 45 (2014) 18

were participating in. Correlation analyses were performed in

order to examine the relationship between parents participation
and certain personal characteristics. The degree of parental
participation during the group meetings was not related to
depression, alcohol or drug abuse, slow learning or intellectual
disabilities.
3.5. Differentiation
We looked at services received by the families prior to the
intervention. More than two thirds of the sample had received
services from a rst line services (68%). A majority of participants
(53.1%) had received services from both the social services system
and the Child protection services before taking part in the program.
Only two participants had not used one service or the other.
Finally, facilitators reported the presence of a community
organization in nearly a quarter of the families (26%). However,
current data does not allow for an in-depth study on the difference
between what these programs offer or have offered to participants
and the specic components of the program.
4. Discussion
Neglect is an important societal problem, whether in terms of
prevalence or in terms of the consequences for the children. Many
different initiatives attempt to answer this worrisome problem.
However, it seems to be particularly difcult to inuence parental
response once neglect is a reality, as is demonstrated by the failure
of existing programs in producing signicant or meaningful
changes in these families (Allin et al., 2005; Barlow et al., 2006;
MacMillan et al., 2009). This situation highlights the importance of
reconsidering the way interventions are evaluated, thus leading to
a better understanding of which conditions are required in order to
make a difference in the lives of children and of their parents.
The evaluation must consider exactly what is offered to
participants, which is referred to by many authors as program
implementation delity (Dane & Schneider, 1998; Domitrovich &
Greenberg, 2000; Durlak & DuPre; Fagan et al., 2008; Paquette
et al., 2009). This type of analysis answers many questions such as:
the impact of dosage or level of families participation on the
programs effectiveness; the minimal quality required of an
intervention and what it is composed of; and how interventions
offered through so-called innovative programs differ from rather
traditional services.
The results of the current study demonstrate a uniformity of the
programs central elements despite variances occurring with
regard to the services each participant benetted from. We based
our analysis on Dane and Schneiders model (1998) in order to
examine how much variation in the program could be disregarded
by a traditional effect evaluation that would compare participants
results before and after the intervention.
We rst looked at programs adherence, which requires a
comparison between the various elements of the program as they
were implemented in order to verify if they conform to the
programs theoretical description. Despite all of the changes that
can occur when the program is disseminated by a variety of
workers in a diverse range of settings, adherence is rarely
documented (Horwitz & Landsverk, 2011; Mildon & Shlonsky,
2011; Pinquart & Teubert, 2010) as are the impacts of variations in
the program implementation on the interventions effects. In the
case of the program presented herein, many factors reduced the
level of variation, such as planning that clearly stipulated which
elements were central to the program. This program structure
promotes exibility, thus allowing parents individual and collective experiences to be expressed (Lacharite et al., 2005). Results
demonstrate a uctuation of certain components of the program,

such as the number of workshops and the topics addressed. Far

from being problematic, this exibility translates into a fair respect
of the programs philosophy. However, when considering an
evaluation of the programs effects in order to verify the relation
between those factors and the program efcacy, it should be noted
that families were not exposed to the same number of themes and
that some groups were larger than others.
Dosage is another important component of program delity.
The effect of dosage variation on the effectiveness of a program is
the subject of a growing number of studies. Interestingly, Pinquart
and Teubert (2010) concluded from their meta-analysis that early
parenting education programs lasting 36 months have larger
positive effects on parenting skills than shorter interventions or
longer ones. However, DePanllis and Dubowitz (2005) found no
signicant difference between families having participated in a
program for 3 or 9 months, noting that even an intervention
spanning over a short period of time can bring about signicant
changes in families at risk of neglect. Data from the current sample
demonstrates important variations in the number of workshops
parents were exposed to. On average, the parents were present at
half of the meetings. Their presence was more related to their
experience with the program and to the topic of the module than to
their personal characteristics. These results can be regarded as
encouraging since the facilitators can exert a greater inuence on a
program topic and on the construction of a relationship with
participants than on families characteristics. Nevertheless, the
signicant variation in dosage of services offered to participants
reiterates the importance of the integration of dosage in the
implementation delity evaluation and of its consideration when
analyzing the effects of the program.
The quality of the intervention concerns the attitude of the staff
in charge of the implementation, their ability to use the methods
recommended by the program or to deliver the prescribed content
(Paquette et al., 2009). The effects of the quality of the intervention
on the changes in the families are supported by studies on
prevention programs among others. An experimental study
conducted by Olds et al. (2002) demonstrated that the program
is benecial to parents and to their child when the follow-up is
monitored by a qualied nurse. Subsequently, it is noted that, in
these families, pregnancies are more spaced, women return to
work more quickly and social, cognitive, and language development in the children is improved. These effects are not signicant
in families where the follow-up is monitored by paraprofessionals.
The evaluation of the PAPFC2 is simplied by the stability of the
facilitators, which ensures a reduction in uctuations in attitudes
and in program facilitation knowledge. Furthermore, the quality of
this operation is conrmed based on parents opinions collected via
workshop evaluations. Parents highly appreciated the relationship
they established with the intervenors and the attitude of those
professionals. The group atmosphere was also reported as being
highly positive by the facilitators. A more neutral evaluation of
quality provided, for example, by a structured observation of the
activity could be integrated in further studies as a third measure of
the program quality and later be used as a control measure in an
impact evaluation of the program.
The association between the parents participation and the
effect of an intervention on neglectful parenting has rarely been
taken into consideration despite the seemingly important consequences it could have. Studies on child problem behavior, for
example, have demonstrated that parents participation is related
to important variations in major parenting outcomes such as the
use of physical punishment, parental warmth and parental
implication in school (Nix, Berman, & McMahon, 2009). This
aspect must be considered in subsequent studies aiming to
document the effects of the PAPFC2, given the important
differences noted in the parents level of commitment to the

A. Berube et al. / Evaluation and Program Planning 45 (2014) 18

workshops. Our results show that parental participation varies

greatly from one parent to another and that it is relatively stable
within a participant. Parents who truly engaged in the program
tended to be evaluated as highly participative during each meeting.
Other participants, however, stayed disengaged during the whole
program. These families did not engage more over time or
depending on the modules topic. Surprisingly, parents dosage and
participation level are not related to their personal difculties.
Many parents displayed characteristics that could result in
difculty learning throughout the program, like depression,
substance dependence or slow learning. These factors did not
affect parental implication during the program.
Finally, differentiation describes how the program differs
compared to the ones parents were previously exposed to. This
subject has not been given much attention and it brings about
serious questions when offering a program to families known for
years by youth care services. Data for the current study does not
allow for an in-depth study on the difference between what these
programs offer or have offered to participants and the specic
components of the PAPFC2. A review of literature pertaining to the
particularities of each service is essential. Indeed, only a
comparative study would allow for a better understanding of
how the program differs from other services and whether or not its
specic elements result in signicant changes in the families
functioning.
4.1. Limitations
In the present study, we looked at only one aspect of the
program, being the parents group. It would be interesting to have
an implementation study that would integrate the other components of the program (individualized services to parents and
children, as well as collective activities). Such an evaluation would
represent a greater challenge, because of the amount of data to
collect. However, it is necessary to eventually better understand
the effect of a multimodal program. Furthermore, an implementation evaluation would have to incorporate self-reported measures,
as well as more objective measures like observations or
standardized measures. The use of standardized measures is a
great challenge since each program has its own components.
However, there is a need to develop more uniform way of
measuring adherence or quality in order to better generalize the
results for implementation studies. Finally, the data did not allow
for a true evaluation of differentiation. Measuring the novelty of a
program could be a key element in changing families used to social
services for many years.
4.2. Implications
Incidentally, insuring the programs delity represents a great
challenge, in particular when it comes to transferring efcacy
studies into a real-world context. This is the reason why many
well-documented studies are reevaluated in the context of an
effectiveness study once introduced into the eld (Klimes-Dougan
et al., 2009; Letarte, Normandeau, & Allard, 2010). Intervention
delity, however, heeds a fair warning: interventions must adjust
to the specic needs of its clientele and to the characteristics of the
environment in which it is implemented. It is the evaluation that
has to document these variations and to look at their repercussions
on the programs effects. The PAPFC2 is a program that could be
qualied as a`-la-carte since, although many activities are prescheduled, services are later personalized to suit the families
characteristics. This situation entails particular challenges when
comes the time to evaluate the effects of the program. A
compilation of the changes observed in the families is insufciently informative as it then becomes impossible to separate which

conditions bring about the most signicant changes. Needless to

say, such evaluation must be considered for interventions related
not only to neglect, but to various issues, particularly those that
require a personalized and multimodal approach.

References
Allin, H., Wathen, C. N., & MacMillan, H. (2005). Treatment of child neglect: A
systematic review. Canadian Journal of Psychiatry, 50, 497504.
Association des Centres jeunesses du Quebec. (2010). La negligence: Faites-lui face.
Montreal: Association des Centres jeunesses du Quebec.
Barlow, J., Davis, H., McIntosh, E., Jarrett, P., Mockford, C., & Stewart-Brown, S. (2007).
Role of home visiting in improving parenting and health in families at risk of abuse
and neglect: Results of a multicentre randomised controlled trial and economic
evaluation. Archive of Disease in Childhood, 92, 229233.
Barlow, J., Johnston, I., Kendrick, D., Polnay, L., & Stewart-Brown, S. (2006). Individual
and group-based parenting programmes for the treatment of physical child abuse
and neglect. Cochrane Database of Systematic Reviews, 3, CD005463.
Brown, J., Cohen, P., Johnson, J. G., & Salzinger, S. (1998). A longitudinal analysis of risk
factors for child maltreatment: Findings of a 17-year prospective study of ofcially
recorded and self-reported child abuse and neglect. Child Abuse and Neglect, 22,
10651083.
Butchart, A., Harvey, A. P., Mian, M., & Furniss, T. (2006). Preventing child maltreatment.
A guide to taking action and generating evidence. Geneva: World Health Organization.
Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary
prevention: Are implementation effects out of control? Clinical Psychology Review,
18, 2345.
DePanllis, D., & Dubowitz, H. (2005). Family connections: A program for preventing
child neglect. Child Maltreatment, 10, 108123.
Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current
ndings from effective programs that prevent mental disorders in school-aged
children. Journal of Educational and Psychological Consultation, 11, 193221.
Dube, S. R., Miller, J. W., Brown, D. W., Giles, W. H., Felitti, V. J., Dong, M., et al. (2006).
Adverse childhood experiences and the association with ever using alcohol and
initiating alcohol use during adolescence. Journal of Adolescent Health, 38, 444.e1
444.e10.
Dubowitz, H., & Bennet, S. (2007). Physical abuse and neglect of children. Lancet, 369,
18911899.
Dubowitz, H., Pitts, S. C., & Black, M. M. (2004). Measurement of three major subtypes of
child neglect. Child Maltreatment, 9, 344356.
Dunn, M. G., Tarter, R. E., Mezzich, A. C., Vanyukov, M., Kirisci, L., & Kirillova, G. (2002).
Origins and consequences of child neglect in substance abuse families. Clinical
Psychology Review, 22, 10631090.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review on the inuence
of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327350.
Ethier, L. S., Bourassa, L., Klapper, U., & Dionne, M. (2006). Levolution des familles
negligentes: Chronicite et typologie. Etat de suivi 1992 a` 2005. Rapport de recherche
du Fond Quebecois de la Recherche de la Societe et de la Culture, Gouvernement du
Quebec.
Ethier, L. S., Couture, G., Lacharite, C., & Gagnier, J.-P. (2000). Impact of a multidimensional intervention program applied to families at risk for child neglect. Child Abuse
Review, 9, 936.
Fagan, A. A., Hanson, K., Hawkins, D. J., & Arthur, M. W. (2008). Bridging science to
practice: Achieving prevention program implementation delity in the Community Youth Development Program. American Journal of Community Psychology, 41,
249253.
Gilbert, R., Widom, C. S., Browne, K., Fergusson, D., Webb, E., & Janson, S. (2009). Burden
and consequences of child maltreatment in high-income countries. Lancet,
373(9657), 6881.
Hildyard, K. L., & Wolfe, D. A. (2002). Child neglect: Developmental issues and outcomes. Child Abuse and Neglect, 26, 679695.
Horwitz, S. M., & Landsverk, J. (2011). Methodological issues in child welfare and
childrens mental health implementation research. Administration and Policy in
Mental Health and Mental Health Services Research, 38, 13.
Kazemian, L., Spatz Widom, C., & Farrington, P. (2011). A prospective examination of
the relationship between childhood neglect and juvenile delinquency in the
Cambridge study in delinquent development. International Journal of Child, Youth
and Family Studies, 1 & 2, 6582.
Klimes-Dougan, B., August, G. J., Lee, C.-Y.S., Realmuto, G. M., Bloomquist, M. L.,
Horowitz, J. L., et al. (2009). Practitioner and site characteristics that relate to
delity of implementation: The Early Risers prevention program in a going-toscale intervention trial. Professional Psychology: Research and Practice, 40, 467475.
Lacharite, C., Ethier, L. S., & Nolin, P. (2006). Vers une theorie ecosystemique de la
negligence envers les enfants. Bulletin de psychologie, 59, 381394.
Lacharite, C., Fafard, G., Bourassa, L., et al. (2005). Le programme daide personnelle,
familiale et communautaire: nouvelle generation. Guide de programme. Trois-Rivie`res: Groupe de recherche et dintervention sur la negligence.
Landsverk, J., Brown, H. C., Reutz, J. R., Palinkas, L., & Horwitz, S. M. (2011). Design
elements in implementation research: A structured review of child welfare and
child mental health studies. Administration and Policy in Mental Health and Mental
Health Services Research, 38, 5463.

A. Berube et al. / Evaluation and Program Planning 45 (2014) 18

Lee, C.-Y.S., August, G. J., Realmuto, G. M., Horowitz, J. L., Bloomquist, M. L., & KlimesDougan, B. (2008). Fidelity at a distance: Assessing implementation delity of the
Early Risers Prevention Program in a going-to-scale intervention trial. Prevention
Science, 9, 215229.
Letarte, M.-J., Normandeau, S., & Allard, J. (2010). Effectiveness of a parent training
program Incredible Years in a child protection service. Child Abuse and Neglect,
34, 253261.
Lounds, J. J., Borkowsky, J. G., & Withman, T. L. (2006). The potential for child neglect:
The case of adolescent mothers and their children. Child Maltreatment, 11, 281
294.
MacMillan, H. L., Wathen, N. C., Barlow, J., Fergusson, D. M., Leventhal, J. M., & Taussig,
H. N. (2009). Interventions to prevent child maltreatment and associated impairment. Lancet, 373, 250266.
Mersky, J. P., Berger, L. M., Reynolds, A. J., & Gromoske, A. N. (2009). Risk factors for child
and adolescent maltreatment: A longitudinal investigation of a cohort of inner-city
youth. Child Maltreatment, 14, 7388.
Mikton, C., & Butchart, A. (2009). Child maltreatment prevention: A systematic review
of reviews. Bulletin of the World Health Organization, 87, 353369.
Mildon, R., & Shlonsky, A. (2011). Bridge over troubled water: Using implementation
science to facilitate effective services in child welfare. Child Abuse and Neglect, 35,
753756.
Nix, R. L., Bierman, K. L., & McMahon, R. J. (2009). How attendance and quality of
participation affect treatment response to parent management training. Journal of
Consulting and Clinical Psychology, 77, 429438.
Olds, D. L., Robinson, J., OBrien, R., Luckey, D. W., Pettitt, L. M., & Henderson, et al.
(2002). Home visiting by paraprofessionals and by nurses: a randomized, controlled trial. Pediatrics, 110(3), 486496.
Paquette, G., Tourigny, M., & Joly, J. (2009). Programme dintervention de groupe pour
des adolescentes agressees sexuellement: Etude des effets speciques et exploration du lien avec limplantation. In M. Tardif (Ed.), Lagression sexuelle: Transformations et paradoxes, Cifas 2009. Textes choisis (pp. 2536). Montreal: Cifas-Institut
Philippe-Pinel de Montreal.http://www.cifas.ca/.
Pinquart, M., & Teubert, D. (2010). Effects of parenting education with expectant and new parents: A meta-analysis. Journal of Family Psychology, 24,
316327.
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. L., & SouthamGerow, M. A. (2011). Toward the effective and efcient measurement of implementation delity. Administration and Policy in Mental Health and Mental Health
Services Research, 38, 3243.
Sedlak, A. J., Mettenburg, J., Basena, M., Petta, I., McPherson, K., Greene, A., et al. (2010).
Fourth National Incidence Study of Child Abuse and Neglect (NIS-4): Report to Congress,
Executive Summary. Washington, DC: US Department of Health and Human Services Administration for Children and Families.
Shipman, K., Edwards, A., Brown, A., Swisher, L., & Jennings, E. (2005). Managing
emotion in a maltreating context: A pilot study examining child neglect. Child
Abuse and Neglect, 29, 10151029.
Tanner, K., & Turney, D. (2003). What do we know about child neglect? A critical review
of the literature and its application to social work practice. Child and Family Social
Work, 8, 2534.

Topitzes, J. D., Mersky, J. P., & Reynolds, A. J. (2010). Child maltreatment and adult
cigarette smoking: A long-term developmental model. Journal of Pediatric Psychology, 35, 484498.
Trocme, N., MacLaurin, B., Fallon, B., Daciuk, J., Felstiner, C., Black, T., et al. (2005). Etude
canadienne sur lincidence des signalements de cas de violence et de negligence envers
les enfants 2003, Donnees principales. Ottawa, ON: Ministre des Travaux publics et
des Services gouvernementaux du Canada.
Ward, H., & Rose, W. (Eds.). (2002). Approaches to Needs Assessment in Childrens Services.
London: Jessica Kingsley Publishers.
Annie Berube is a Professor in the Department of Psychoeducation and Psychology at
Universite du Quebec en Outaouais. Her research interests focus on the evaluation of
implementation and effect of program targeting children and their family. She concentrates particularly on parental involvement in their child development. She holds a
Ph.D. in Community Psychology from the Universite du Quebec a` Montreal (UQA`M).
Vicky Lafantaisie is a Ph.D. student in Psychoeducation at the Universite du Quebec a`
Trois-Rivie`res and a research coordinator for the project Evaluation de programmes en
negligence. Her master degree looks at social isolation of neglecting families. Her actual
work concerns the sociocultural dimensions of parental neglect, as well as to ways to
encourage empowerment in those families.
Diane Dubeau is a Professor in the Department of Psychoeducation and Psychology at
Universite du Quebec en Outaouais (Campus de Saint-Jerome). Her projects are
articulated around two main topics, program evaluation and paternity. She holds a
master in Child development and a Ph.D. in Psychology (UQA`M). She has published in
2009 a book on paternity in the XXth century: La paternite au XXIe`me sie`cle published at
Presses de lUniversite Laval. This volume is a synthesis of her work over the 15 last years
on an intervention aimed at supporting paternal involvement.
Sylvain Coutu is a Professor in the Department of Psychoeducation and Psychology at
Universite du Quebec en Outaouais (Campus de Saint-Jerome) since 1988. He holds a
master in Psychoeducation (Universite de Montreal) and a Ph.D in Psychology (UQA`M).
He is an associate researcher for lEquipe de recherche sur la Qualite educative des services
de garde et petite enfance. His work concentrates on the socioemotional development of
preschool children. He also has an interest for programs targeting children from
vulnerable families.
Josee Caron is working with neglected population for more than 10 years. She
developped a detailed understanding of that population. She is currently the coordinator of a program helping neglecting families in the Outaouais region.
Annie Devault is a Professor at Social work Department at the Universite du Quebec en
Outaouais. Since the last ten years, her work explores paternal involvement in
vulnerable contexts. She also has an interest for the intervention with neglect families
Dr. Devault holds a Ph.D. in community psychology from UQA`M.