2013-2014

Ontario’s Enhanced 18-Month

Well-Baby Visit: Modified Dillman
Method with Primary Care

Final Report August 2014

Niagara Region Public Health
Table of Contents

Background ................................................................................................................................................................ 1
Methods ..................................................................................................................................................................... 2
Sample characteristics ............................................................................................................................................ 3
Results ........................................................................................................................................................................ 6
Primary care awareness and use of EWBV fee codes ............................................................................................ 6
Primary care awareness and use of the RBR and NDDS......................................................................................... 6
Primary care referral to early childhood services in Niagara ................................................................................. 7
Primary care comfort level with EWBV elements .................................................................................................. 8
Follow-up cards from primary care ........................................................................................................................ 8
Discussion................................................................................................................................................................. 10
Limitations ............................................................................................................................................................ 10
Recommendations and Next Steps ......................................................................................................................... 12
Appendix A: Traditional Dillman Method and Modified Dillman Method ............................................................ 13
Appendix B: 18-month EWBV Package Components ............................................................................................. 15
Cover letter: week 1 ............................................................................................................................................. 15
Cover letter: week 3 ............................................................................................................................................. 16
Cover letter: week 7 ............................................................................................................................................. 17
Survey: week 1, 3 (if necessary) and 7 (if necessary) ........................................................................................... 18
Follow-up card: week 1, 3 (if necessary), and 7 (if necessary) ............................................................................. 21
Reminder post card: week 2 ................................................................................................................................. 22
Thank you card ..................................................................................................................................................... 23
Works Cited .............................................................................................................................................................. 24
Contact Information ................................................................................................................................................ 25

i
Background

Well-baby visits are part of comprehensive primary care for infants and their families, particularly in
the first two years of life. In an effort to promote healthy child development, the Province of Ontario
has funded a longer, more in-depth Enhanced 18-Month Well-Baby Visit (EWBV) since 2009.1 The
timing of this visit occurs when early detection and intervention can make the difference for many
children in order to optimize their development and prevent further delay. In addition, the EWBV is
one of the last points when primary care will see all young children in their practice to administer
scheduled vaccinations before school entry. 2

The Ministry of Children and Youth Services outlines the process of the EWBV to include the use of
validated standardized tools such as the Rourke Baby Record (RBR) and the Nipissing District
Developmental Screen (NDDS).3 In addition to the use of the screening tools, primary care will:

• Have a discussion with parents on healthy child development, as it pertains to their child
and addressing any concerns they may have
• Provide information on parenting, literacy and community programs, such as library
programs and the Ontario Early Years Centres
• Make referrals for children with potential developmental issues to specialized community
services, such as Speech Services and Healthy Babies Healthy Children 4

Previous research has focused exclusively on primary care’s use of the RBR in Ontario, with
questionnaires being mailed to primary care in London, Toronto and Ottawa. 5 Utilization of the RBR,
NDDS and 18-Month EWBV fee code is unknown when looking specifically at primary care in Niagara.

To support the EWBV locally in Niagara, Niagara Region Public Health (NRPH) sought to reach out to
local primary care in order to:

1. Understand primary cares’ current practices and experiences with the RBR, NDDS and
knowledge of where to refer families with children in the Niagara community
2. Follow-up with primary care if necessary, in the method requested on the follow-up card
3. Obtain feedback from primary care about their experiences with NRPH

This report will focus on the first of the three objectives: understanding primary cares’ current
practices, experiences, and knowledge.

1|Page
Methods

The Dillman Method is regarded as the standard for mail surveys with professional groups.
Throughout the years, many iterations of the Dillman Method have been used. Most recently, the
Tailored Design Method (TDM) has been used to receive high levels of response rates from primary
care. 6 The current version of the TDM involves five elements for achieving high response rates
including:

• A respondent-friendly questionnaire
• Four contacts by registered mail, with an additional ‘special’ contact (e.g. telephone call)
• Use of return envelopes with real first-class stamps
• Personalized correspondence
• A token financial incentive that is sent with the initial questionnaire 6

After a review of the literature for the Dillman Method, it was identified that it has been researched
extensively and recent studies recommend utilizing a modified Dillman approach to survey primary
care.6 Therefore, NRPH used these recommendations with the primary care providers, with careful
attention being paid to the effectiveness of specific components within the recommended
methodology. Some of the most important components included the use of first class mail and a token
financial incentive.6 In October 2013, NRPH implemented a modified Dillman methodology with its
primary care providers over the course of seven weeks (refer to Appendix A for a more detailed
comparison of the Dillman Method and NRPH’s modifications).

Within this study, a variety of information was collected from primary care in two different ways-
through a survey and a follow-up card. These two tools collected different pieces of information from
primary care and were both sent out in the first package. Primary care could respond to one or both.
If no information was received after the week one mail-out, the entire package was sent out in week
three. If the survey, the follow-up card, or both were received, no subsequent packages were sent out.

The survey primary care was asked to complete contained questions pertaining to their knowledge and
use of the EWBV fee codes, the RBR, the NDDS, community-based and private services for referrals,
and their comfort levels with other EWBV elements (see Appendix B for the survey). In addition to
completing the survey, primary care were also asked to complete a follow-up card and return via mail.
This follow-up card had pre-defined ways in which they would like to receive ongoing support from
NRPH with regards to the EWBV.

During the first week, a total of 312 primary care providers were sent the EWBV survey via registered
mail, along with a letter of instructions for completing the survey, a paid postage envelope to return

2|Page
the survey, a follow-up card, and a $25 Chapters gift card. This resulted in 21 surveys being returned
before the next mail-out. Thank you cards were sent to those who responded as responses were
received. In week two, a reminder post card and the NRPH Physician Newsletter were sent by regular
mail to those who had not responded to the week one survey. During this week, 60 additional surveys
were returned to NRPH. During week three, regular mail was utilized again to send out a new letter
with instructions, another copy of the survey with the paid postage envelope, and a follow-up card.
Through weeks three and four, another 56 surveys were returned. After the week 3 mail-out, NRPH
did not contact primary care until week five or six. During these weeks, a reminder phone call was
made to primary care offices, with 20 more surveys being returned during this time. Finally, in week
seven, one last piece of registered mail was sent out, with the cover letter, the survey, a paid postage
envelope, and the follow-up card. From this mail-out until data analysis began, 25 more surveys were
returned to NRPH. Three surveys did not have dates of return recorded. This methodology is
summarized in Figure 1.

Sample characteristics

Among the 312 primary care providers mailed an EWBV survey, the sample was reduced to 284, as
some primary care providers had either moved or retired. Of the 284 who received a survey, 185
returned the survey, with a response rate of 65.1%, and 152 returned a follow-up card, with a response
rate of 53.5%. Overall, 168 of the 185 primary care providers (90.8%) identified that they saw children
under the age of two in their practice each month. The following analysis presented in this report is
based on the 168 primary care providers who see children. See Figure 2 for an outline of how the final
sample size was obtained utilizing the Modified Dillman Method.

3|Page
Figure 1. Modified Dillman Method utilized by NRPH to survey primary care

•Registered mail
•Cover letter with instructions
•Survey with paid postage envelope
Week One •Follow-up card
October 16 •$25 Chapters gift card

•Regular mail
•Reminder post card
Week Two •Insert in NRPH's Physician Newsletter
October 23

•Regular mail
•New cover letter with instructions
Week Three •Survey with paid postage envelope
•Follow-up card
October 30

Results •Reminder phone call

Week Five and Six
Results
November 13 and
20

Results
•Registered mail
•Cover letter with instructions
•Survey with paid postage envelope
Week Seven
•Follow-up card
November 27

Results

4|Page
Figure 2. Respondent sample size
312 packages sent to primary care in
the Niagara Primary Care Practictioner
Database
28 primary care moved or
retired

284 primary care successfully received

the package

99 primary care did not

return the survey

185 primary care returned the survey

168 primary care responded that they 17 primary care did not meet the
see children on a monthly basis inclusion criteria

149 family 15 4 nurse

practictioners paediatricians practictioners

5|Page
Results

Among survey respondents who saw children under two (N=168), the majority were family
practitioners (88.7%), followed by paediatricians (8.9%), and nurse practitioners (2.4%). When
respondents were asked who provides well-baby care in their office, 85.1% identified a family
physician, 23.8% a nurse, 14.9% a nurse practitioner, and 8.3% a paediatrician. A small number
reported others providing well-baby care. Just under 60% (n=100) of primary care are using an
electronic platform in their office (e.g., OSCAR, Physician Solutions, Practice Solutions).

Primary care awareness and use of EWBV visit fee code

Overall, the majority of primary care in Niagara is aware of a variety of the EWBV elements. In total,
88.7% (n=149) of primary care are already aware of the EWBV fee code. In addition to this, 77.4%
(n=130) of primary care indicated they are already using the EWBV fee code in their practice.

Primary care awareness and use of the RBR and NDDS

Two tools that are used at the EWBV include the RBR and the NDDS. The vast majority of respondents
are aware of and use the RBR at the EWBV. In total, 97.0% (n=163) of primary care were aware of this
tool. Most commonly, the RBR was used as a guide, with 81.5% of primary care using this guide often
or always during the EWBV (Table 1). In addition, it is also used often or always for charting purposes
(80.3%).

Table 1. Primary care use of RBR during 18-month visit (N=168)

Never Seldom Sometimes Often Always Missing
(%) (%) (%) (%) (%) (%)
As a guide 5.4 3.6 7.1 8.9 72.6 2.4
For charting purposes 7.7 5.4 6.0 7.1 73.2 0.6
At other well-baby visits
9.5 4.8 6.0 9.5 69.0 1.2
(e.g., 12-months)

The other tool that the majority of respondents were aware of and use is the NDDS. In total, 91.7%
(n=154) of respondents were aware of this checklist. Three-quarters of primary care (75.0%) are using
this checklist as a basis of discussion with parents often or always during the course of the visit, while a
similar proportion (73.8%) are using this resource often or always to screen for a problem (Table 2).

6|Page
 Table 2. Primary care use of NDDS during 18-month visit (N=168)
Never Seldom Sometimes Often Always Missing
(%) (%) (%) (%) (%) (%)
As a basis of discussion with
13.1 4.2 5.4 11.3 63.7 2.4
parents
To screen for a problem 13.7 1.8 7.7 7.7 66.1 3.0
At other well-baby visits
25.6 11.9 18.5 8.3 33.3 2.4
(e.g., 12-months)

Although their functions are different, primary care has been more aware of and used the RBR more
often in comparison to the NDDS. At other well-baby visits, the RBR was used always or often by more
than three-quarters (78.5%) of respondents, while the NDDS was used always or often by less than half
of respondents (41.6%).

Primary care referral to early childhood services in Niagara

Throughout Niagara, there are many different early childhood services and programs that primary care
can refer parents to. Although primary care was aware of most of the services listed, a higher
proportion of primary care was unaware of the Niagara Regional Native Centre (15.5%) and library
programs (10.7%). In addition, community-based services such as parenting workshops, early year
centers, and library programs are less commonly recommended by primary care, compared with
specialized services (e.g., Contact Niagara, Speech Services Niagara) (Table 3).

Table 3. Primary care referral to early childhood services in Niagara Region (N=168)
Never Seldom Sometimes Often Very Often Unaware Missing
(%) (%) (%) (%) (%) (%) (%)
Ontario Early
15.5 22.6 29.8 10.7 11.9 4.2 5.4
Years Centres
Public Health
16.1 22.6 29.8 14.3 9.5 4.2 3.6
services
Public Health
19.0 29.8 29.2 7.1 6.0 4.8 4.2
parenting classes
Niagara Regional
52.4 19.6 7.1 0.0 0.0 15.5 5.4
Native Centre
Speech Services
4.2 7.7 43.5 23.8 16.1 0.6 4.2
Niagara
Niagara
9.5 20.2 38.7 14.3 10.7 2.4 4.2
Children’s Centre
Contact Niagara 6.5 14.3 40.5 25.0 9.5 1.2 3.0
Library programs 35.1 23.8 11.9 2.4 4.8 10.7 11.3
Note: Other programs were mentioned in very small numbers and included FACS visiting home nurse, Pathstone,
paediatrician, or YMCA.

7|Page
Primary care comfort levels with EWBV elements

Overall, almost the entirety of Niagara primary care indicated they are either comfortable or very
comfortable discussing child development with parents (94.0%) or identifying at-risk children early
(90.5%). Although at least half of primary care responded to feeling comfortable or very comfortable
with EWBV elements, just under one-fifth of primary care (19.6%) indicated they were uncomfortable
or very uncomfortable using these resources (Table 4).

Table 4. Primary care comfort levels with EWBV elements (N=168)

Very Very
Uncomfortable Comfortable Missing
Uncomfortable Comfortable
(%) (%) (%)
(%) (%)
Identifying at-risk
1.2 7.1 64.9 25.6 1.2
children early
Talking with parents
about their child’s 0.6 4.8 59.5 34.5 0.6
development
Referring parents to
services in the 0.6 8.9 55.4 33.3 1.8
community
Using the electronic
10.7 8.9 29.2 41.7 9.5
RBR and NDDS

Follow-up cards from primary care

In addition to the surveys, 152 follow-up cards were received to assess how primary care would like to
receive support from NRPH. Of these 152 cards, 9 were left blank and were excluded from the
analysis. Overall, the most commonly requested ways to administer support are through mailing a
resource package with follow-up by the public health nurse (36.4%) and CME credit education (29.4%).

Table 5 provides a breakdown of how primary care indicated they would like to receive follow-up,
stratified by EWBV fee code use. For those who are not yet billing for the EWBV, the majority of
respondents requested the mailing of a resource package with a follow-up by the public health nurse
(60.4%). While some primary care indicated they were already billing for the EWBV (N=90), these
individuals were still able to check off ways they would like support for EWBVs. In total, 41.1% of those
who were already billing had still requested additional support from NRPH to assist with this process.

8|Page
Table 5. Primary care preference for follow-up
Follow-up requested by Follow-up requested by
those who are not yet using those who are using the
the EWBV fee code (N=53) EWBV fee code (N=90)
(%) (%)
10-minute detailing visit at your practice from a
18.9 5.6
public health nurse
10-minute information phone call from a public
15.1 4.4
health nurse
Mailed 18-month resources package with follow-
60.4 22.2
up by a public health nurse
30-minute detailing presentation for you and
17.0 7.8
your team over lunch hour
30-minute presentation for yourself and other
5.7 1.1
providers at Ontario Early Years Centre
Opportunity to participate in education
39.6 23.3
associated with a CME credit
Other 1.9 2.2
Do not want more follow-up 0.0 58.9

9|Page
Discussion

With the majority of primary care in Niagara aware of and billing for the EWBV Visit, the results of the
follow-up cards show that primary care would still like to receive support for the EWBV. A number of
primary care would like to receive this information through a mailed resource package, with follow-up
from a public health nurse.

Throughout EWBVs, the RBR is more consistently used compared to the NDDS, especially at visits other
than the 18-month visit. The majority of primary care in Niagara is able to use these elements
throughout their 18-month visit, in addition to any other visits where the RBR and NDDS could be used.
Although this comfort level exists with these resources, a level of discomfort still exists for using these
tools electronically.

Although levels of comfort for various EWBV elements are high within Niagara, there is still room for
improvement. With approximately 10% of primary care not comfortable with identifying at-risk
children, work remains to be done in this area.

In regard to community referrals, primary care is more aware of and referring more often to
specialized services (i.e., Speech Services Niagara) than to community based programs such as the
Ontario Early Years Centres or library programs. For some of these services, awareness is still an issue,
with primary care not yet aware that some of these services are available for parents.

Limitations

One of the main limitations of this project was related to receiving details as to why primary care did
not respond to this survey. From the first mail-out, some packages were returned to NRPH because
the primary care at this location had moved or retired. After this occurred, these individuals were not
mailed subsequent packages and were excluded from analysis of the response rate. On the other
hand, there is no way of tracking the reason for non-response for those who did not respond to the
survey and did not have their packages returned to NRPH. This issue was dealt with by calling the
offices at the five and six week mark. NRPH recorded information on the status of primary care during
phone calls when possible.

In addition, with some of the survey packages being distributed numerous times in such a short time
frame, duplicate responses had occurred. Although this happened in a very small proportion of cases,
these respondents were excluded from the analysis.

10 | P a g e
Another limitation of this research project is that the survey was not sent directly to other forms of
primary care (other than doctors), as NRPH does not have a database for other types of primary care.
Although some nurse practitioners filled out the responses on the questionnaire instead of the doctor,
this is not a true sample of these individuals, and as a result, no stratified analysis occurred by each
type of primary care.

11 | P a g e
Recommendations and Next Steps

This research study has identified some ways in which NRPH can follow-up with primary care in order
to move forward with the EWBV strategy. In response to the lower use of the NDDS, NRPH may have a
role in working with primary care office support staff to encourage the dissemination of the guide to
parents for completion prior to their EWBV and all other well-baby visits. This may save the provider
time as the parent will have recorded their child’s developmental progress prior to the appointment
and will be prepared to discuss it at the time of their appointment.

In regard to referral to early childhood services in Niagara, NRPH should work with its partners to
promote community based programs, such as the Ontario Early Years Centres, so that primary care are
able to better understand how these programs can benefit their clients.

NRPH’s plan was to engage primary care thoroughly, and then target parents. After this survey and the
subsequent follow up that is ongoing, the primary care population in Niagara can be considered fully
engaged around the EWBV.

12 | P a g e
Appendix A: Dillman Total Design Survey and Modified Dillman Method
Dillman Total Design Survey 7,8
Week 1:
• Initial mail-out, consisting of the
survey, instructions, cover letter
(signed in blue ink), a return
envelope signed by hand
• Optional return post card
containing their name (to ensure
anonymity), or can be pre-
numbered
Week 2:
• Reminder postcard
Week 3:
• Duplicate package is sent out,
consisting of the survey,
Dillman Tailored Design Method 9
A few days before the survey is mailed
out:
• Pre-notice letter, advising the
recipients that they will receive a
survey within the next week, sent
by first class mail
Week 1:
• Initial mail-out by first class mail,
consisting of the survey, a cover
letter, a token incentive, and a
return envelope that was hand-
stamped
Week 2:
• Reminder/thank you postcard
Week 3:
• No mail out
Modified Dillman Method
Week 1:
• Initial mail-out by registered mail,
consisting of the survey, a cover
letter with instructions, a return
envelope with paid postage, a
follow-up card, and a $25 Chapters
gift card
• Surveys were pre-numbered, while
follow-up cards had the physician’s
name and assigned number
Week 2:
• Reminder postcard
• Thank you postcard
• Insert in NRPH’s Physician
Newsletter
Week 3:
• Duplicate package is sent out,
consisting of the survey, a new
13 | P a g e
 instructions, cover letter (signed in
blue ink), a return envelope
Week 4: Week 4:
• No mail-out • Duplicate package is sent out by
first class mail, consisting of the
survey, a follow-up letter, and a
return envelope with a stamp
Week 5 and 6: Week 5 and 6:
• No mail-out • No mail-out
Week 7: Week 7:
• Duplicate package is sent out by • Duplicate package is sent out by
registered mail, consisting of the registered mail or courier,
survey, instructions, cover letter consisting of the survey, a follow-
(signed in blue ink), a return up letter, and a return envelope
envelope with a stamp
cover letter with instructions, a
return envelope with paid postage,
and a follow-up card
Week 4:
• No mail-out
Week 5 and 6:
• No mail-out
• Reminder phone calls
Week 7:
• Duplicate package is sent out by
registered mail, consisting of the
survey, a new cover letter with
instructions, a return envelope
with paid postage, and a follow-up
card

14 | P a g e
Appendix B: 18-month EWBV Package Components

Cover letter: week 1

15 | P a g e
Cover letter: week 3

16 | P a g e
Cover letter: week 7

17 | P a g e
Survey: week 1, 3 (if necessary) and 7 (if necessary)

18 | P a g e
19 | P a g e
20 | P a g e
Follow-up card week 1, 3 (if necessary), and 7 (if necessary)

21 | P a g e
Reminder post-card: week 2

22 | P a g e
Thank you card

23 | P a g e
 Works Cited
1
Williams, R., & Clinton, J. Getting it right at 18 months: in support of an enhanced well-baby visit.
Paediatrics and Child Health. 16. 10 (2011).
2
Dahinten, V.S., Ford, L. Validation of the Nipissing District Developmental Screen For Use With
Infants and Toddlers- Working Paper. 2004.
3
Currie, L., Dodds, L., Shea, S., Flowerdew, G., McLean, J., Walker, R., & Vincer, M. Investigation of
test characteristics of two screening tools in comparison with a gold standard assessment to detect
developmental delay ay 36 months: a pilot study. Journal of Paediatrics and Child Health. 17. 10
(2012).
4
Getting it right at 18 months… Making it right for a lifetime. The Expert Panel on the 18 Month
Well Baby Visit. http://www.children.gov.on.ca. 2005.
5
Rourke, L., Godwin, M., Rourke, J., Pearce, S., & Bean, J. The Rourke Baby Record Infant/Child
Maintenance Guide: do doctors use it, do they find it useful, and does using it improve their well-
baby visit records? BMC Family Practice. 10. 28 (2009).
6
Thorpe, C., Ryan, B., McLean, S.L., Burt, A., Stewart, M., Brown, J.B., Reid, G.J., & Harris, S. How to
obtain excellent response rates when surveying physicians. Family Practice. 26. (2008).
7
Dillman, D.A. The design and administration of mail surveys. Annual Review of Sociology. 17. (1991).
8
Hoddinott, S.N., & Bass, M.J. The Dillman total design survey method: a sure fire way to get high
survey return rates. Canadian Family Physician. 32. (1986).
9
Dillman, D.A. (2007) Mail and internet surveys: the tailored design method (2nd ed.) Hoboken, NJ:
Wiley.

24 | P a g e
Contact Information

For more information about this report, please contact:

Sandy Richardson, BScN, RN

Manager Healthy Babies Healthy Children &Infant & Child Developmental Services
Family Health, Public Health
Niagara Region
Phone: 905-688-8248 ext. 7503 Toll-free: 1-888-505-6074
Email: sandra.richardson@niagararegion.ca
Website: www.niagararegion.ca

25 | P a g e