Running head: EVALUATION OF DUOLINGO LESSON

Evaluation of Duolingo Portuguese Basic Level 1 Lesson

Tina Indalecio
California State University Monterey Bay

IST622 Assessment and Evaluation

Dr. Bude Su
June 20, 2015

EVALUATION OF DUOLINGO LESSON

Table of Contents
I. Introduction......................................................................................................................3
II. Methodology...................................................................................................................3
a. Prototype.....................................................................................................................3
b. Learners.......................................................................................................................4
c. Tryout Process.............................................................................................................5
d. Tryout Conditions........................................................................................................6
III. Results............................................................................................................................6
a. Entry Conditions..........................................................................................................6
b. Instruction...................................................................................................................7
c. Outcomes.....................................................................................................................7
d. Recommendations........................................................................................................9
IV. Summary and Conclusion..............................................................................................9
V. Appendices....................................................................................................................10
A. Instructions and Survey Questions............................................................................10
B. Pre-Survey Responses...............................................................................................18
C. Post-Survey Responses.............................................................................................20
D. Pre-test and Post-test Scores....................................................................................22
E. Paired Two Sample T-Test for Pre-test and Post-test Scores...................................23
F. Two Sample T-Test Assuming Equal Variances for Post-test Scores and Help........23
G. Graphs for Scaled Questions...................................................................................24

EVALUATION OF DUOLINGO LESSON

Evaluation of Duolingo Portuguese Basic Level 1 Lesson

I. Introduction
The purpose of this evaluation was to determine the effectiveness of the Duolingo online
language e-learning tool. In addition a simple usability evaluation was conducted to help identify
areas needing improvement. In effort to ensure the learners did not have prior knowledge of the
language lesson, Portuguese was selected due to the very limited use within this sample of
learners. In general, the goal of this evaluation was to determine if a learner with no prior
knowledge of a language, could take a short basic lesson and then be able to correctly remember
and apply key concepts from the lesson. In addition, was there a noticeable difference between
the scores of the learners who took the product tour and utilized the tips and notes section during
the lesson, versus those who skipped the tour and who did not use the tips section.
II. Methodology
a. Prototype
The prototype used for evaluation was Duolingo.com, an online free language e-learning
site. The company offers several languages and claims that 34 hours of instruction on their
website is equivalent to one university level semester, or 11 weeks of language instruction
(duolingo.com, 2015). The site does not specify an age minimum to use their site and the lessons
start very basic, it appears that the site is geared for middle school to adult learners. Each lesson
module can be completed as quickly as 15 minutes or over 30 minutes. All lessons are self-paced
and if a learner answers the terms correctly they progress through the lesson quickly, if they miss
questions the lesson will continue to teach the concept in various ways until they get the correct
answer.
The site is mobile responsive and allows the user to take the lesson on a phone, tablet or
computer. Only a basic web browser and microphone is needed. The user also has the ability to

EVALUATION OF DUOLINGO LESSON

skip questions that require the microphone if needed. Before the learner begins they are
prompted to take the product tour, they can skip if and proceed directly to the lesson. In addition
within the lesson there is a tips & notes section that the learner can click on if they need
assistance. However, this section is difficult to see on the site. The site does not allow the learner
to skip or jump lessons, they must progress through the lessons in a sequence. For the purpose of
this evaluation, the learners only completed the basic level 1 lesson in Portuguese.
b. Learners
The learners for this evaluation all came from various professional occupations, and
education levels. In addition, the learners lived in various locations, with the majority of them
residing in California, Oregon and Washington. The group was comprised of 14 individuals, 21%
male and 79% female. Ages ranged from 29 to 58 years old, with a median age of 47. The
majority of the learners only spoke English, 14% indicated that they spoke one other language
and it was Spanish. Only 14% said they had never studied a foreign language before, everyone
else in the group had studied at least one foreign language. The main languages previously
studied included Spanish, French, German and Italian. All of the learners, except one, indicated
that they wanted to learn a foreign language. No one in the group knew how to speak or
understand Portuguese, and no one in the group had ever tried learning a language online.
However 21% of the group stated they had heard of Duolingo. For the purpose of this evaluation
having no prior knowledge of Portuguese was ideal. In addition, there was not a need for the
learner to have studied a foreign language previously.
c. Tryout Process
Due to the learners various locations the process included an emailed list of instructions,
remote observation via text support, a pre-survey with pre-test questions included, and a post-test
survey with post-test and usability questions. The instructions included detailed information on

EVALUATION OF DUOLINGO LESSON

what they would be learning, how long it would take and the links to get started. The surveys
(see appendix A) were hosted on google and upon completion of the pre-survey the learner was
given the direct link to begin the lesson. Once they finished the lesson, they completed the postsurvey piece.
During the process it was not possible to visually observe the learner taking the lesson,
due to location variances, therefore additional questions were added to the post-survey to
determine if they asked anyone for help, if they were frustrated, if they completed the lesson or
quit, etc. In addition, as data posted from the surveys, it was possible to see who was currently
taking the lesson, and that made it simple to do text check-ins with the learner to gauge progress
and issues. Two learners had issues with the duolingo site and needed to update items on their
computers prior to proceeding, this resulted in them having to re-do the pre-survey to get the
lesson link again. This created duplicate entries, however because the data being posted was
observed in real time, and because it was easy to see the duplicated based on the learners age, the
duplicate data lines were easily removed so as not to damage the sample results. All learners
completed the pre-survey, lesson and post-survey. In addition, no one asked for help and
managed to complete the lesson un-aided. There was only one user who stated they quit before
they finished the lesson.
The pre-test questions and post-test questions were exactly the same and covered the
basic topics of the lesson. The learner needed to be able to remember and recognize how to say
and write basic terms related to boy, man, woman and girl. The pre-survey general questions
gathered basic demographic and prior learning experience with foreign languages, which was
stated previously. The post-survey in addition to the test questions, included basic usability

EVALUATION OF DUOLINGO LESSON

questions to determine how well the learner liked the lesson and the interface, as well as how
effective they felt the lesson was and if they would use Duolingo again.
d. Tryout Conditions
Duolingo lessons are designed for the user to take the lessons from any location.
Therefore the learners used their own devices to complete the tryout. The majority of learners
used their computer, two learners used their mobile phones and one learner used an ipad. There
were no complaints regarding accessibility of the lesson. Two users did not have a microphone
on their work computers, but were informed they could skip those questions in the lesson and it
would teach the information in an alternative way. Some users took the lesson from home, while
others took the lesson while at work.
III. Results
a. Entry Conditions
As mentioned previously, knowledge of a foreign language was not a requirement for this
evaluation. In addition, all learners were familiar with how to use a computer, access the internet,
navigate the lesson and respond to the surveys. Only two learners had to update their computer
settings before accessing the lesson, and two others did not have access to a microphone. In all
cases the issues were easily resolved and there were not any entry condition issues that impacted
the evaluation.
b. Instruction
Due to the simplicity of the instructions and lesson, the majority of learners did not
encounter any problems. Only one learner continued on to take other lessons, because she did not
read the instructions carefully. She was stopped before she started lesson 3, and then went on to
complete the post-survey questions. An issue that can impact how long some people spent on the
lesson versus others was that the nature of the lesson does make it easy to make educated guesses
to progress through the lessons faster. Therefore some learners may have been able to complete

EVALUATION OF DUOLINGO LESSON

the lessons faster than others by guessing. In addition, the lesson has a tour the user can take to
familiarize them selves with how the product works. The user can skip this or take a minute or
two and go through it. There were 50% of the learners who skipped the tour, which may or may
not have impacted the speed of progressing through the lesson, their perceptions of usability and
their post-test scores.
Everyone finished the lesson, except one individual who quit before completing it. This is
not a major concern and giving the learner flexibility to complete the lesson at their own pace,
and navigate the lesson as they choose provides them with control over their learning.
c. Outcomes
There were two areas statistically evaluated based on the data acquired. The first was to
determine if the lesson impacted the pre-test and post-test scores significantly (see appendices D
and E). The second area of statistical evaluation was to determine if there was a difference in
post-test scores of those who took the tour, or used the tips & notes section versus those who did
not (see appendix F). A paired two sample t-test was performed to see if the lesson contributed to
significant learning gains of the Portuguese language. The p-value was less than .05 (.P < 0.000)
and the absolute value of t-stat was greater than the t critical value (|t|=7.2111 > 1.77093),
therefore we reject the null hypothesis and there is evidence to suggest that the alternative
hypothesis may be true. The result is statistically significant and it appears the lesson has created
increased learning gains.
The second statistical evaluation was a two sample t-test assuming equal variances. When
evaluating the post-test scores with those who took the tour versus those who did not, the p-value
was greater than .05 (.P > .24177) there was no statistical significance found and therefore we
fail to reject the null hypothesis and would not accept the alternative hypothesis. The evaluation
of the post-test scores and if the learner used the tips & notes section versus those who did not,

EVALUATION OF DUOLINGO LESSON

yielded similar results. The p-value was greater than .05 (.P > .38377) again there was no
statistical significance found. In addition, the median scores for both items were the same. An
important item to note is that on the survey, for the question regarding the tips & notes section,
the learners were given the option to indicate yes, no or no I did not see it. Everyone who
indicated no, selected the no I did not see it option. Therefore, its important to consider that if
they had seen the option for help, would they have used it and would that have positively
impacted their post-test score. Additional research is needed to understand what impact if any
would have been made if all users took the tour and utilized the tips & notes section.
In addition to the pre and post test questions the learners were also asked questions, on a
5 point scale, to evaluate the usability and effectiveness of the lesson. The results (see appendix
G) indicated that the majority of learners felt the lesson was easy to understand, easy to navigate,
and was visually appealing. Scores for these three areas were 86% to 93% positively rated. In
addition, 71% of the learners felt that the graphics and sounds helped them learn the terms, and
that the lesson was effective in teaching them basic words in Portuguese. The majority of
learners felt the level of difficulty was just right. However when asked how confident they felt
about being able to remember the terms, the answered varied with 50% stating they were
confident to very confident and the other 50% stating they were neutral to not confident. In
closing 79% said they felt the lesson was fun and that they would use Duolingo again.
d. Recommendations
For the majority of learners they were able to complete the lesson without help or issues.
The user interface is very clean and easy to navigate. One suggestion would be to make the tips
& notes section more visible. In addition, unless the learner takes the tour, and notices the tips &
notes section, there are no instructions to guide the learner through the lesson. This issue could
potentially lead to frustration and the learner quitting the lesson before completing it. To help

EVALUATION OF DUOLINGO LESSON

ensure all learners understand how to use the product, making the tour required prior to
beginning the lesson may help with increasing learning outcomes and decreasing time spent
trying to figure out how to progress through the lessons. It is difficult to suggest other
improvements at this time, because only the basic first course was evaluated.
IV. Summary and Conclusion
The evaluation of pre and post-test scores as well as the usability feedback, indicated that
not only did the lesson produce noticeable learning outcomes, the learners also enjoyed the
lesson and would continue to use Duolingo in the future. There were only a few minor issues
with two users encountering a need to update their settings to take the lesson, which resulted in
duplicate data on the pre-survey. However, that issue was quickly corrected and did not
negatively impact the data collected. The main area for improvement is to make the tour required
prior to completing a lesson and to make the tips & notes section more visible, to ensure all
learners understand how to use the product.
Limitations regarding this evaluation include two issues, one is related to the observation
process used and the other is related to the limited lessons evaluated. It would have been ideal to
observe each learner in person to evaluate how quickly or slowly they progressed, what their
body language communicated and to control the testing environment better. In addition, if time
had permitted, having the learner progress through three lessons would have allowed them more
time to work in the product to gain a better understanding of how it works and how to navigate
through the lessons. In addition, it would have allowed for more substantial language practice,
which could have in-turn created better learning outcomes.
V. Appendices
A. Instructions and Survey Questions
(These instructions were sent to the learner, and the surveys were hosted on google. Click
on the links to see the live survey forms.)

EVALUATION OF DUOLINGO LESSON

10

INSTRUCTIONS:
Thank you for agreeing to help me with a school project! This questionnaire and lesson
should take you about 30 minutes to 45 minutes to complete. There is a short pre-survey, the
actual lesson and then a short post-survey. Please be sure to do everything in one sitting,
otherwise the results will not be accurate.
You will need access to a computer, the internet, a web browser and your built in
microphone on your ipad, tablet, or computer. You can also use your phone headset that has a
built in microphone.
The short lesson is on learning a new language. Thank you so much for your time and
help!
To begin please click here:
http://goo.gl/forms/2ohLFZVl3G
When you are done with the pre-survey and the lesson, please click here to take the
Post-Survey:
http://goo.gl/forms/6co4Ngyrjb

STEP 1 The Pre-Survey:

Please list your age:

Please list the gender you identify with:

Female
Male

EVALUATION OF DUOLINGO LESSON

Do you speak any other language besides English?
Yes
No

If yes, what other languages do you speak?

Have you ever studied a foreign language?

Yes
No

If yes, what other languages have you studied?

Do you want to learn a foreign language?

Yes
No

Have you ever used the internet to learn a language?

Yes
No

If yes, what websites have you used?

Have you ever heard of the website duolingo.com?

11

EVALUATION OF DUOLINGO LESSON

Yes
No

Do you know how to speak Portuguese?

Yes, I am fluent
Yes, I know a little
No

Can you understand Portuguese?

Yes, I am fluent
Yes, I know a little
No

What does o menino mean?

What does uma mulher mean?
-

one woman
a man
one boy
a girl
I do not know
What does Eu sou uma menina mean?

What does Eu sou um homem mean?

-

I am a woman
I am a man
There is a boy

12

EVALUATION OF DUOLINGO LESSON

-

There is one girl

I do not know

STEP 2 The Lesson:

Thank you for completing the pre-survey, to begin the lesson please click here:
https://www.duolingo.com/course/pt/en/Learn-Portuguese-Online
You DO NOT have to sign up or register, you can simply complete the lesson.
When ready - Select Start Course

Note you will have the opportunity to take a short tour of how to use Duolingo and
then be prompted to begin the lesson, select Basic 1 and then Lesson 1 - You only need to
complete that one mini lesson.

STEP 3 The Post-Survey:

What are you using to complete these surveys and lesson?
-

a computer
a tablet/ipad
a mobile phone
Did you take the Tour or did you skip the Tour before beginning the lesson?

I took the tour

I skipped the tour
Did you use the Tips & Notes section during the lesson?

yes
no, I did not need it
no, I did not see it

13

EVALUATION OF DUOLINGO LESSON

14

Did you ask anyone for help while taking the lesson?
-

yes
no
Did you experience any frustration while taking the lesson?

never
occasionally but I managed to figure it out on my own
occasionally and I used the tips & notes section for help
occasionally and I asked someone for help
often and I had problems completing with the lesson
Did you finish the entire lesson or did you quit before finishing?

I finished
I quit before finishing
Please rate the following statements

The lesson instructions were easy to understand:

1

strongly

strongly

disagree

agree

The lesson navigation was easy to use:

1

strongly

strongly

disagree

agree

The lesson layout and design was visually appealing:

EVALUATION OF DUOLINGO LESSON

1

15
4

strongly

strongly

disagree

agree

The graphics were helpful when learning how to translate the terms properly:
1

strongly

strongly

disagree

agree

Being able to hear the terms spoken in Portuguese helped me understand the terms:
1

strongly

strongly

disagree

agree

The lesson helped me learn new Portuguese terms:

1

strongly

strongly

disagree

agree

How would you rate the difficulty level of the lesson?

Too easy / somewhat easy/ just right / somewhat difficult / too difficult

How confident do you feel that you can remember these Portuguese terms:

EVALUATION OF DUOLINGO LESSON

1

16
4

not

very

confident

confident

The lesson was fun to complete:

1

strongly

strongly

disagree

agree

I would use Duolingo again to learn a language:

1

strongly

strongly

disagree

agree

What does o menino mean?

What does uma mulher mean?

-

one woman
a man
one boy
a girl
I do not know
What does Eu sou uma menina mean?

What does Eu sou um homem mean?

EVALUATION OF DUOLINGO LESSON

-

I am a woman
I am a man
There is a boy
There is one girl
I do not know

B. Pre-Survey Responses

17

EVALUATION OF DUOLINGO LESSON

18

EVALUATION OF DUOLINGO LESSON

C. Post-Survey Responses

19

EVALUATION OF DUOLINGO LESSON

20

EVALUATION OF DUOLINGO LESSON

D. Pre-test and Post-test Scores

21

EVALUATION OF DUOLINGO LESSON

22

E. Paired Two Sample T-Test for Pre-test and Post-test Scores

F. Two Sample T-Test Assuming Equal Variances for Post-test Scores and Help
This statistic shows post scores variances based on if the learner took the tour or skipped
the tour.

EVALUATION OF DUOLINGO LESSON

This statistic shows post scores variances based on if the learner used the tips and notes
section versus those who did not.

G. Graphs for Scaled Questions

23

EVALUATION OF DUOLINGO LESSON

24

EVALUATION OF DUOLINGO LESSON

25