Mobile​ ​App​ ​and​ ​Query​ ​Evaluation​ ​for​ ​Google​ ​Searches

Version:​ ​2016-11-08
In​ ​this​ ​task,​ ​you​ ​will​ ​research​ ​a​ ​mobile​ ​app​ ​and​ ​a​ ​query​ ​entered​ ​in​ ​Google​ ​Search.​ ​You​ ​will​ ​then
answer​ ​a​ ​question​ ​about​ ​how​ ​well​ ​the​ ​app​ ​fulfills​ ​what​ ​the​ ​user​ ​is​ ​looking​ ​for.

Terms
● You​ ​already​ ​know​ ​what​ ​a​ ​query​ ​is:​ ​it​ ​is​ ​a​ ​word​ ​or​ ​phrase​ ​that​ ​a​ ​real​ ​person​ ​typed​ ​into​ ​a
Google​ ​search.
● A​ ​mobile​ ​device​ ​is​ ​a​ ​small,​ ​handheld​ ​computing​ ​device,​ ​such​ ​as​ ​a​ ​smartphone​ ​or​ ​tablet.
● A​ ​mobile​ ​app​ ​is​ ​a​ ​smartphone​ ​or​ ​tablet​​ ​application​.

How​ ​to​ ​rate​ ​this​ ​task

Step​ ​1:​ ​Review​ ​and​ ​understand​ ​the​ ​user​ ​query.
You​ ​will​ ​see​ ​a​ ​query​ ​that​ ​a​ ​user​ ​entered.​ ​Just​ ​as​ ​in​ ​normal​ ​search​ ​ads​ ​rating,​ ​it​ ​will​ ​be​ ​your
responsibility​ ​to​ ​do​ ​your​ ​best​ ​to​ ​understand​ ​what​ ​the​ ​user​ ​was​ ​looking​ ​for.​ Y ​ ou​ ​will​ ​be​ ​asked​ ​to
assume​ ​that​ ​the​ ​user​ ​entered​ ​the​ ​query​ ​in​ ​a​ ​regular​ ​search​ ​on​ ​Google.com​ ​on​ ​a​ ​mobile
device.​ ​Although​ ​this​ ​evaluation​ ​is​ ​concerning​ ​the​ ​relevance​ ​of​ ​mobile​ ​apps,​ ​do​ ​not​ ​assume
that​ ​if​ ​a​ ​user​ ​entered​ ​a​ ​search​ ​into​ ​Google​ ​that​ ​they​ ​are​ ​searching​ ​for​ ​an​ ​app.​ ​Instead,​ ​consider
the​ ​user​ ​to​ ​be​ ​an​ ​average​ ​person​ ​entering​ ​the​ ​query​ ​into​ ​Google​ ​search.​ ​For​ ​more​ ​information
on​ ​how​ ​this​ ​is​ ​applicable​ ​to​ ​the​ ​task​ ​at​ ​hand,​ ​please​ ​see​ ​examples​ ​in​ ​the​ ​table​ ​below​ ​Step​ ​3.

Unrateable​ ​Query
If​ ​the​ ​query​ ​is​ ​in​ ​a​ ​foreign​ ​language,​ ​is​ ​incomprehensible,​ ​or​ ​otherwise​ ​can’t​ ​be​ ​rated,​ ​select​ ​the
“Unrateable​ ​query”​ ​flag​ ​and​ ​submit​ ​the​ ​task.

Step​ ​2:​ ​Familiarize​ ​yourself​ ​with​ ​the​ ​mobile​ ​app.

Click​ ​the​ ​button​ ​labeled​ ​“research​ ​mobile​ ​app”​ ​to​ ​open​ ​the​ ​app’s​ ​Google​ ​Play​ ​page,​ ​then​ ​review
its​ ​content.​ ​You​ ​should​ ​consider​ ​all​ ​the​ ​information​ ​available​ ​to​ ​you​ ​in​ ​order​ ​to​ ​inform​ ​your
understanding​ ​of​ ​the​ ​app,​ ​including​ ​the​ ​text​ ​that​ ​describes​ ​the​ ​app,​ ​ratings,​ ​any​ ​relevant
comments​ ​from​ ​users,​ ​and​ ​even​ ​your​ ​overall​ ​impression​ ​of​ ​how​ ​professional​ ​and​ ​trustworthy​ ​the
app​ ​looks.​ ​Some​ ​additional​ ​notes:

● Do​ ​not​ ​download​ ​the​ ​app.

● The​ ​Play​ ​Store​ ​page​ ​may​ ​include​ ​some​ ​additional​ ​content,​ ​such​ ​as​ ​links​ ​to​ ​other​ ​parts​ ​of
the​ ​Play​ ​Store,​ ​or​ ​lists​ ​of​ ​“Similar”​ ​mobile​ ​apps.​ ​Please​ ​ignore​ ​this​ ​and​ ​only​ ​focus​ ​on​ ​the
content​ ​that​ ​is​ ​related​ ​to​ ​the​ ​main​ ​app​ ​on​ ​the​ ​page.
● You​ ​may,​ ​however,​ ​follow​ ​links​ ​and​ ​do​ ​additional​ ​research​ ​outside​ ​of​ ​the​ ​Play​ ​Store​ ​if​ ​it
helps​ ​you​ ​understand​ ​the​ ​app.

Unrateable​ ​Apps
If​ ​the​ ​Play​ ​Store​ ​page​ ​or​ ​the​ ​app​ ​it​ ​describes​ ​is​ ​in​ ​a​ ​foreign​ ​language,​ ​is​ ​incomprehensible,​ ​or
otherwise​ ​can’t​ ​be​ ​rated,​ ​select​ ​the​ ​“Unrateable​ ​app”​ ​flag​ ​and​ ​submit​ ​the​ ​task.
Personal​ ​Knowledge​ ​of​ ​Apps
If​ ​you've​ ​personally​ ​installed​ ​and​ ​used​ ​the​ ​app,​ ​or​ ​believe​ ​you​ ​have​ ​additional​ ​knowledge​ ​about
the​ ​app​ ​that​ ​would​ ​affect​ ​your​ ​ratings​ ​in​ ​this​ ​task,​ ​select​ ​the​ ​flag​ ​labeled​ ​"I​ ​am​ ​personally​ ​familiar
with​ ​this​ ​app."​ ​You​ ​will​ ​still​ ​be​ ​required​ ​to​ ​complete​ ​the​ ​task.

Step​ ​3:​ ​Answer​ ​the​ ​following​ ​question:​ ​Would​ ​installing​ ​and​ ​using​ ​the​ ​app​ ​satisfy​ ​the
user​ ​intent?
Think​ ​about​ ​what​ ​the​ ​mobile​ ​app​ ​is​ ​used​ ​for​ ​and​ ​what​ ​the​ ​user​ ​was​ ​looking​ ​for.​ ​For​ ​example,​ ​if
the​ ​app’s​ ​purpose​ ​is​ ​to​ ​deliver​ ​food​ ​and​ ​the​ ​provided​ ​query​ ​contains​ ​the​ ​word​ ​“fruit​ ​ninja”,​ ​then
the​ ​user​ ​intent​ ​is​ ​unlikely​ ​to​ ​be​ ​satisfied​ ​by​ ​the​ ​user​ ​installing​ ​and​ ​using​ ​the​ ​app.​ ​Pay​ ​attention​ ​to
the​ ​app’s​ ​quality​ ​as​ ​well.​ ​A​ ​good​ ​quality​ ​app​ ​looks​ ​trustworthy,​ ​has​ ​a​ ​well-written​ ​description​ ​that
makes​ ​the​ ​app’s​ ​purpose​ ​clear,​ ​and​ ​gets​ ​positive​ ​reviews​ ​from​ ​users.​ ​The​ ​user​ ​intent​ ​is​ ​less
likely​ ​to​ ​be​ ​satisfied​ ​by​ ​a​ ​poor​ ​quality​ ​app.

Taking​ ​all​ ​of​ ​these​ ​factors​ ​into​ ​consideration,​ ​select​ ​a​ ​rating​ ​from​ ​[-100]​ ​to​ ​[100]​ ​to​ ​indicate​ ​the
likelihood​ ​that​ ​a​ ​specific​ ​user​ ​would​ ​be​ ​satisfied​ ​by​ ​installing​ ​and​ ​using​ ​the​ ​app.
Rating Description Examples

Satisfaction​ ​Likely
The​ ​user​ ​is​ ​looking​ ​for​ ​an​ ​app,​ ​and
this​ ​one​ ​provides​ ​exactly​ ​the
functionality​ ​that​ ​the​ ​user​ ​is​ ​looking
for.​ ​The​ ​app​ ​seems​ ​to​ ​be​ ​of​ ​high
quality​ ​(i.e.,​ ​it's​ ​not​ ​scammy).
If​ ​the​ ​user​ ​is​ ​looking​ ​for​ ​a​ ​specific
app​ ​but​ ​is​ ​presented​ ​with​ ​a​ ​plausible
substitute,​ ​you​ ​should​ ​select​ ​this
option.
User​ ​query:​​ ​[flashlight​ ​app]
App:​​ S
​ uper-Bright​ ​LED​ ​Flashlight
The​ ​user​ ​can​ ​turn​ ​their​ ​phone​ ​into​ ​a
flashlight​ ​with​ ​this​ ​app.
User​ ​query:​​ ​[caviar​ ​food​ ​delivery]
App:​​ D
​ oorDash
While​ ​the​ ​user​ ​is​ ​looking​ ​for​ ​a
specific​ ​food​ ​delivery​ ​app​ ​-​ ​Caviar​ ​-
the​ ​DoorDash​ ​food​ ​delivery​ ​app​ ​is​ ​a
plausible​ ​substitute.

Satisfaction​ ​Possible
The​ ​user​ ​might​ ​install​ ​and​ ​use​ ​the
app​ ​because​ ​the​ ​query​ ​has​ ​multiple
equally​ ​likely​ ​user​ ​intents,​ ​namely
the​ ​user​ ​could​ ​be​ ​looking​ ​for​ ​a
physical​ ​good​ ​or​ ​app.​ ​The​ ​app​ ​can
only​ ​satisfy​ ​one​ ​of​ ​the​ ​plausible
intents.
User​ ​query:​​ ​[flashlight]
App:​​ S​ uper-Bright​ ​LED​ ​Flashlight
The​ ​user​ ​is​ ​equally​ ​likely​ ​looking​ ​for
a​ ​flashlight​ ​app​ ​as​ ​s/he​ ​is​ ​looking​ ​for
a​ ​physical​ ​flashlight​ ​for​ ​sale.

Dissatisfaction The​ ​user​ ​probably​ ​won't​ ​install​ ​the
Possible app​ ​because​ ​the​ ​user​ ​intent​ ​is
unlikely​ ​to​ ​be​ ​met​ ​by​ ​the​ ​app,
and/or​ ​because​ ​the​ ​app​ ​appears​ ​to
be​ ​of​ ​low​ ​quality.
User​ ​query:​​ ​[airline]
App:​​ K ​ ayak
There​ ​is​ ​no​ ​app​ ​that​ ​could​ ​possibly
receive​ ​"Satisfaction​ ​Likely"​ ​for​ ​the
query​ ​[alrline]​ ​-​ ​it​ ​is​ ​simply​ ​too​ ​vague
of​ ​a​ ​query.​ ​However,​ ​a​ ​subset​ ​of
users​ ​might​ ​find​ ​Kayak,​ ​a​ ​flight​ ​deal
app,​ ​useful.​ ​(​Note:​ ​ ​other​ ​airline​ ​apps
like​ ​Southwest​ ​or​ ​Delta​ ​would​ ​also
be​ ​Satisfaction​ ​Possible).
User​ ​query:​​ ​[facebook]
App:​ ​Facebook
The​ ​user​ ​may​ ​be​ ​looking​ ​for​ ​the
Facebook​ ​app,​ ​but​ ​it's​ ​also​ ​likely​ ​that
the​ ​just​ ​want​ ​to​ ​log​ ​in​ ​through
Facebook's​ ​website​ ​as​ ​quickly​ ​as
possible.
User​ ​query:​​ ​[install​ ​facebook]
App:​​ S​ napchat
The​ ​user​ ​is​ ​looking​ ​for​ ​a​ ​specific
app.​ ​In​ ​some​ ​ways,​ ​Snapchat​ ​is
similar​ ​to​ ​Facebook,​ ​but​ ​it's​ ​not​ ​close
enough​ ​to​ ​be​ ​a​ ​true​ ​plausible
substitute.
User​ ​query:​​ ​[board​ ​games]
App:​​ C ​ heckers​ ​Elite
The​ ​user​ ​intent​ ​is​ ​much​ ​more​ ​likely
to​ ​find​ ​physical​ ​board​ ​games​ ​rather
than​ ​the​ ​app-related​ ​version,​ ​but​ ​it​ ​is
still​ ​possible​ ​that​ ​the​ ​user​ ​might
download​ ​the​ ​app.
User​ ​query:​​ ​[veterinarian​ ​game​ ​app]
App:​​ V
​ et​ ​Clinic
The​ ​user​ ​likely​ ​won’t​ ​install​ ​this​ ​app
because​ ​it​ ​is​ ​very​ ​low​ ​quality:​ ​it​ ​has
bad​ ​reviews​ ​in​ ​the​ ​Play​ ​store​ ​and​ ​the
app​ ​description​ ​is​ ​riddled​ ​with
spelling​ ​and​ ​grammatical​ ​errors.
Dissatisfaction​ ​Likely The​ ​user​ ​won’t​ ​install​ ​the​ ​app
because​ ​the​ ​app​ ​is​ ​unrelated​ ​to
what​ ​they​ ​are​ ​looking​ ​for,​ ​because
the​ ​app​ ​is​ ​of​ ​obviously​ ​low​ ​quality,
or​ ​because​ ​the​ ​query​ ​has​ ​multiple
possible​ ​user​ ​intents​ ​and​ ​the​ ​query
is​ ​unlikely​ ​referring​ ​to​ ​an​ ​app.
User​ ​query:​​ ​[bicycle]
App:​​ B ​ ike​ ​Mayhem​ ​Free
If​ ​a​ ​user​ ​enters​ ​[bicycle]​ ​into​ ​a
Google​ ​search​ ​query,​ ​it​ ​is​ ​highly
unlikely​ ​that​ ​the​ ​user​ ​is​ ​searching​ ​for
an​ ​app.