Triggerin Knowledge Panels Patent

USO10922326B2
United States Patent

(12) Provine ( 10 ) Patent No .: US 10,922,326 B2
(45 ) Date of Patent : Feb. 16 , 2021
( 54 ) TRIGGERING KNOWLEDGE PANELS 2005/0096897 A1 *
2007/02 14122 A1 *
5/2005 Ando et al .
9/2007 Bala
704/4
707/3
( 71 ) Applicant: Google LLC , Mountain View , CA (US) 2008/0027888 A1 * 1/2008 Azzam et al . 706/45
2009/0241018 Al 9/2009 Costello et al .
2010/0076972 A1 * 3/2010 Baron G06F 17/3071
( 72 ) Inventor: John R. Provine , Berkeley, CA (US ) 707/736
2012/0059838 A1 * 3/2012 Berntson G06F 17/30864
( 73 ) Assignee : Google LLC , Mountain View , CA (US ) 707/759
2013/0159272 A1 * 6/2013 Donmez et al . 707/706
( * ) Notice: Subject to any disclaimer, the term of this 2014/0059042 A1 * 2/2014 Vinay et al . 707/723
patent is extended or adjusted under 35 * cited by examiner
U.S.C. 154 ( b ) by 1071 days.
( 21) Appl . No .: 13 /830,526 Primary Examiner Boris Gorney
(22) Filed : Mar. 14 , 2013 Assistant Examiner . James E Heffern
(74 ) Attorney, Agent, or Firm — Fish & Richardson P.C.
( 65 ) Prior Publication Data
US 2015/0161131 A1 Jun . 11 , 2015 ( 57) ABSTRACT
Related U.S. Application Data Methods, systems, and apparatus, including computer pro
( 60 ) Provisional application No. 61 /730,115 , filed on Nov. grams encoded on a computer storage medium , for provid
27 , 2012 . ing knowledge panels with search results . In one aspect , a
method includes receiving a query. Search results that are
(51 ) Int. Ci. responsive to the received query are obtained . A first set of
GO6F 16/9535 ( 2019.01 ) factual entities referenced by the received query are identi
G06F 16/248 ( 2019.01 ) fied . A particular factual entity is selected from the first set .
( 52 ) U.S. CI. It is determined that a knowledge panel for the particular
CPC GO6F 16/248 (2019.01 ) ; GO6F 16/9535 factual entity is to be provided with the search results based ,
( 2019.01 ) at least in part, on content of the knowledge panel and
( 58 ) Field of Classification Search characteristics of the search results. The knowledge panel
CPC G06F 16/248 ; G06F 16/9535 can include at least one content item , for the selected factual
See application file for complete search history. entity , received from a first resource and at least one content
( 56 ) References Cited item , for the selected factual entity, received from a second
resource different than the first resource . The knowledge
U.S. PATENT DOCUMENTS panel is provided .
8,032,519 B2 10/2011 Groeneveld et al.
8,285,699 B2 10/2012 Groeneveld et al . 22 Claims , 11 Drawing Sheets
100
106
User
104 Device
Search Knowledge
Query Panel
Web Site 109 113
Resource (s) Search
105
Results
111
Network
102
Search System Knowledge Panel Apparatus

120 130
Search Historical Content Items Knowledge Entity Index

Index Data Panels 136
122 124 1132 134
Indexed
Resources
U.S. Patent Feb. 16 , 2021 Sheet 1 of 11 US 10,922,326 B2
100
} 106
?
User
Device
1047 Search
Query
Knowledge
Panel
Web Site 109 113
Resource (s)
105 Search
Results
111
Network
102
3
Search System Knowledge Panel Apparatus

120 130
Search Historical Knowledge Entity Index

Index Data Content Items Panels
122 124 132 134 136
1 1
1 1
1 Indexed
i Resources :
FIG . 1
200
So
DOX 226
2
228
DY 222
FAisanacmtours aAmcenrtiocdarn InfoMores.inger 1985 :
27
June
:
Born
, $30mNetwiolritohn
AFacmtours FAacmtours Image

Movies Music Related
2207 224 230 232 234
234
+
X
207
-SCexampli.co/vhstfeaudrml e
C
S
- x a m p l i
. c o / u sht e f a m c
d r
o / l e-SCxampl.ico/sntwrkhgfeaudo/ml- 2
.
FIG
205
APnoathger -SeCxamipl.co/fhlusaetord/m
Ehtp:/wD.examlCcosrAngi
209
MA-FSEcxaoitmvrpiu'les FsmPAtuoacrvtihoaeunsrg –FSMEAxacuimtsopiulercs AFpsPbyeuacormftnhrogaumserd

E
N
S
F
-
A e
xo
at
ciw
m o
r
tpka
ui
len
r
s g
?
AFacmtours siteofFtheoV...Afaicmstociuatrsl
FAosnpacfetmwiocrguk'aiselng
S-OAFfacimtoiuearsl
SPeargche Searcho
21 a 21 b 21 0 21 0
300
so
QOX youDo 322 324
inCUPnauilvbefrositncea UPinCnarilvefsoartnie
1University 3University 4University U2niversity5University 6University
320320
3A
.
FIG
307
X1 305
?
APnoathger 309 -SCexamipl.co/unhivlersaty1d/hml S-Cexamipl.co/unhversiaty2dml e-SCxampil.co/unhvlersiaty3d/m -SeCxamipl.co/unhvlersiaty4d/ml

?
SPeargceh
Ehtp:/w.exCamlcosArngi CUanilvefrostnia
Searcho 1University
31 a
Visit
osite
the
U1
of
...
fnivecrisatly
31 b
2University
oUsite
the
Visit
of
...nfivecrsiatly
2
31 0
U3niversity
Visit
the
osite
U4
of
.
fnivecrisatly
31 0
U4niversity
osite
the
Visit
U4
of
...
nfiviecrsiatly
350
QOX 376
2378 Famous Explore
DO FAisanacmtours aAmcenrtiocdarn InfoMore.singer 1985 Net
27
June
:
Born
, mil ion
30
$
:
worth
Famous Chef
372
Famous Golfer
AFacmtours FAacmtours Image arSebseoults
Movies Famous Singer
370
Z 374 380 382
e
C
S
- x a m p l .
i c o / s n t w r khg e f a m u cdo / l 3B
.
FIG
htp:/w.examlcosrngi S-Cexampli.co/vhstfeaudrml e
C
S
- x a m p l i
. c o / u s ht e f a m cdr
o / l
307
X
305
APnoatgher
?
309 -SCexampil.co/fhulsaetrdm
by
P
s
p
F
A e
u
ao
crmfn
th
rg
oa
u
m s
e
r d E
N
S
F
-
A e
xo
a
tc
iw
m t
o
rp
ka
ui
l
e n
r
sg
?
PFearmsoun AofFstheoV...faicmstociueatrsl
MA-FSEcxaoitmvrpiu'les AFsmPtuaocrvhioauensrg -FSMEAxacuimtsopiulercs FAosnpacfetmwiocrguka'iselng
S-OAFfacimtoiuearsl
SPeargceh E*DCA Searcho
361a 361b 3610 3610
400 402
Receive a query
404
Identify search results for the received query
406
Identify a factual entity referenced by the received query
408
Yes No
Provide knowledge panel ?
410 2 5 412
Provide knowledge panel and optionally Provide search results
search results
FIG . 4
5003 Receive data specifying a query and search results for the
query
502
504
Identify a first set of factual entities that each have an alias
that matches a portion of a received query
506
Identify a second set of factual entities that are topical to the
search results
Identify factual entitiesandthatthe aresecond

included
set
in both the first set 508
Select at least one of the factual entities that is included in 510

the both the first set and the second set for reference in the
knowledge panel
FIG . 5
600 602
Receive data specifying a query and search results for the
query
604
Identify a set of factual entities referenced by the search
results
606
Identify a proper subset of the higher ranked search results
For each factual entity , determine a partial topicality score for so608
the entity with respect to each search result of the proper
subset
610
Selecta proper subset of the factual entities
partial topicality scores
based on the 56
FIG . 6
003
700
Determine an overall topicality score for each factual entity of
15702
a set of factual entities based on partial topicality scores for
the factual entities
Select a proper subset of the factual entities

overall topicality scores
based on the 5704
FIG . 7
800
0} Receive data specifying a query
query and search results for the 5802
Identify a setmatches
of factual entities that each have an alias that 5 804
a portion of a received query
For eachsearchresult, determine a partialtopicality scores 806

for the search result with respect to each factual entity of the
set of factual entities
Identify propersubset ofpartialtopicality scores that satisfya5 808

topicality threshold
Determine which one of the proper subset of partial topicality

scores corresponds to the highest ranking search result
S8810
812
Select the factual entity that corresponds to the one topicality
score as the factual entity referenced by the query
FIG . 8
900
Receivedataspecifying aquery
query and search results for the 5902
904
Identify click through rates ("CTRs") for each search results
with respect to the received query
906
Yes Determine whether No

to provide knowledge panel
based on CTRS
9087 5910
Provide knowledge panel Do not provide knowledge panel
FIG . 9
1000 1002
Identify an entity referenced by a search query
Identify content related to entity for knowledge panel 5 1004
1006
Yes No
Sufficient content ?
1008
2 s 1010
FIG . 10
1100 1102
Receive data specifying query
Identify entity referencedpanelby received

for entity
query and a knowledge 511104
1106
Yes Knowledge panel No

include query refinements
for received query ?
1108
z S 1110
FIG . 11
US 10,922,326 B2
1 2
TRIGGERING KNOWLEDGE PANELS Identifying a second set of factual entities, each factual
entity of the second set being a factual entity determined to
CROSS REFERENCE TO RELATED be topical to the obtained search results, can include iden
APPLICATIONS tifying a third set of factual entities, each factual entity of the
5 third set being a factual entity referenced in at least one of
This application claims the benefit under 35 U.S.C. § the obtained search results, identifying a proper subset of the
119 (e ) of U.S. Patent Application No. 61 /730,115 , entitled obtained search results , the proper subset including a par
“ TRIGGERING KNOWLEDGE PANELS,” filed Nov. 27 , ticular number of higher ranked obtained search results ; for
2012 , which is incorporated herein by reference in its 10 each factual entity of the third set of factual entities, deter
entirety. mining, relative to the factual entity, a partial topicality score
BACKGROUND for each search result in the proper subset of obtained search
results , the partial topicality score for each search result
This specification relates to presenting data with search being a measure of topical relatedness between the search
results and determining when to present the data with the 15 result and the factual entity; and selecting, from the third set
search results . of factual entities, the second set of factual entities based on
The Internet provides access to a wide variety of theSelecting partial topicality scores.
the second set of factual entities based on the
resources , such as image files, audio files , video files, and partial topicality scores can include, for each factual entity
web pages . A search system can identify resources in
response to queries submitted by users and provide infor- 20 of the third set, determining an overall topicality score for
mation about the resources in a manner that is useful to the the factual entity based on the partial topicality scores for the
users . The users can then navigate through , e.g. , click on the factual entity ; and selecting the second set of factual entities
search results to acquire information of interest to the users . based on the overall topicality scores. Determining the
Users of search systems are often searching for informa- overall topicality score for the factual entity based on the
tion regarding a specific entity. For example, users may want 25 partial topicality scores for the factual entity can include
to learn about a singer that they just heard on the radio . determining a weighted sum of the partial topicality scores
Conventionally, the user would submit a search query iden for the factual entity.
tifying the singer and select from a list of search results Selecting, from among the first set of factual entities, a
determined to be relevant to the search query . particular factual entity for which a knowledge panel is
30 eligible to be provided with the search results can include for
SUMMARY each obtained search result , determining a topicality score
for the search result with respect to each factual entity of the
In general, one innovative aspect of the subject matter first set of factual en es , the topicality score for a search
described in this specification can be embodied in methods result with respect to a factual entity being a measure of
that include the actions of receiving a query ; obtaining 35 topical relatedness of the search result with respect to the
search results that are responsive to the received query ; factual entity; comparing each topicality score to a topicality
identifying a first set of factual entities referenced by the threshold to identify a proper subset of the topicality scores
received query ; selecting , from among the first set of factual that exceed satisfy the topicality threshold ; determining
entities, a particular factual entity for which a knowledge which one of the proper subset of topicality scores corre
panel is eligible to be provided with the search results , the 40 sponds to a highest ranking search result for the proper
selection being based , at least in part, on a measure of subset of topicality scores; and selecting the factual entity
topicality between each identified factual entity in the first that corresponds to the one partial topicality score as the
set and one or more of the obtained search results ; deter- particular factual entity.
mining that the knowledge panel for the particular factual Determining that the knowledge panel for the particular
entity is to be provided with the search results , the deter- 45 factual entity is to be provided with the search results can
mination being based , at least in part, on content of the include identifying a click - through rate for each obtained
knowledge panel and characteristics of the search results , the search result with respect to the received query ; determining
knowledge panel including at least one content item , for the that the identified click - through rates do not satisfy a thresh
selected factual entity, received from a first resource and at old click - through rate ; and determining to provide the
least one content item , for the selected factual entity, 50 knowledge panel in response to determining that the iden
received from a second resource different than the first tified click -through rates do not satisfy the threshold click
resource ; and providing the knowledge panel. Other through rate .
embodiments of this aspect include corresponding systems, Determining that the knowledge panel for the particular
apparatus, and computer programs, configured to perform factual entity is to be provided with the search results can
the actions of the methods, encoded on computer storage 55 include identifying a top ranked search result for the
devices. received query ; identifying a click -through rate for the top
These and other embodiments can each optionally include ranked search result with respect to the received query ;
one or more of the following features . Selecting , from determining that the identified click - through rate does not
among the first set of factual entities, a particular factual exceed satisfy a click - through rate threshold ; and determin
entity for which a knowledge panel is eligible to be provided 60 ing to provide the knowledge panel in response to deter
with the search results can include identifying a second set mining that the identified click - through rate does not exceed
of factual entities, each factual entity of the second set being satisfy the click -through rate threshold .
a factual entity determined to be topical to the obtained Determining that the knowledge panel for the particular
search results , identifying one or more factual entities factual entity is to be provided with the search results can
included in the set of factual entities and in the second set of 65 include identifying content for inclusion in the knowledge
factual entities ; and selecting one of the one or more factual panel; determining that the identified content includes at
entities as the identified factual entity. least a threshold amount of content; and determining to
US 10,922,326 B2
3 4
provide the knowledge panel in response to determining that Like reference numbers and designations in the various
the identified content includes at least a threshold amount of drawings indicate like elements.
content.
Determining that the knowledge panel for the particular DETAILED DESCRIPTION
factual entity is to be provided with the search results can 5
include determining whether the knowledge panel includes responseA system can provide one or more knowledge panels in
links to query refinements for the received query ; and to a received search query. A knowledge panel is a
determining to provide the knowledge panel in response to user interface element that provides a collection of infor
determining that the knowledge panel includes links to 10 mation or other content related to a particular entity refer
enced by a search query. For example, the entity may be a
query refinements for the received query .
Particular embodiments of the subject matter described in person, place, ,business
country, landmark , animal, historical event,
this specification can be implemented so as to realize one or organization
song , album , game ,
, sports team , sporting event, movie ,
work of art, or any other entity. In some
more of the following advantages. Knowledge panels can implementations, the entity may be a concept, subject, or
improve users' search experiences, in particular for queries 15 topic.
directed to learning, browsing, or discovery. For example, In general, a knowledge panel provides a summary of
the knowledge panel supplies users with basic factual infor information for the entity . For example , a knowledge panel
mation from multiple different sources or a summary of for a singer may include the name of the singer, an image of
information about a particular entity referenced in a search the singer, a description of the singer, one or more facts
query. Knowledge panels can assist users in navigating to 20 about the singer, content that identifies songs and albums
multiple different network locations that each includes recorded by the singer, and / or links to searches related to the
related content in a seamless and natural way. Knowledge singer . Other types of information and content can also be
panels can pply ew content that may not otherwise be presented in the knowledge panel.
encountered by a user without the user selecting several The knowledge panel for a particular entity typically
different search results and visiting multiple different net- 25 includes content obtained from multiple disparate sources ,
work locations . Knowledge panels can also help users obtain e.g. , multiple different domains . For example , a knowledge
information faster than they would if the users were required panel for a person may include an image obtained from an
to click through multiple search results to obtain the infor- official web site for the person and facts obtained from an
mation from multiple different network locations . online encyclopedia that is not affiliated with the official web
The details of one or more embodiments of the subject 30 site for the person .
matter described in this specification are set forth in the When a search query is received , the system can deter
accompanying drawings and the description below . Other mine whether to provide a knowledge panel with search
features, aspects , and advantages of the subject matter will results for the search query. This determination may be
become apparent from the description, the drawings, and the based , in part, on whether the received query is determined
claims . 35 to reference a known entity. In some implementations, the
system determines whether the received query references a
BRIEF DESCRIPTION OF THE DRAWINGS known entity based on a comparison of the received query ,
or terms of the received query , to an entity index that
FIG . 1 is a block diagram of an example environment in identifies known entities and their aliases . If a match is
which a search system provides search services. 40 determined to exist based on the comparison , the received
FIG . 2 is a screen shot of an example search interface in query is determined to reference the known entity, and the
which a knowledge panel is presented with search results . system may provide a knowledge panel for the matching
FIG . 3A is a screen shot of an example search interface in entity.
which a disambiguation knowledge panel is presented with Some queries may reference multiple entities or one or
search results . 45 more terms that can be considered an alias of multiple
FIG . 3B is a screen shot of an example search interface in entities. For example, the term “ phoenix ” may refer to a city
which a dominant entity knowledge panel is presented with in Arizona or a mythical bird . For queries that match, or are
search results . otherwise relevant to , multiple known entities, the system
FIG . 4 is a flow chart of an example process for providing can determine whether to provide a knowledge panel for one
a knowledge panel and / or search results for a search query. 50 of the matching entities, for example if one entity is the most
FIG . 5 is a flow chart of an example process for identi- likely entity for the query, or to provide a knowledge panel
fying a factual entity for providing a knowledge panel. that includes content for multiple entities. For example, if a
FIG . 6 is a flow chart of an example process for identi- query includes “ Phoenix , Ariz.” or the query was received
fying a subset of factual entities for a received search query . from a device located in Arizona, the system may determine
FIG . 7 is a flow chart of an example process for selecting 55 that the city in Arizona is the most likely entity being
a subset of factual entities based on topicality scores. referenced by the search query and provide a knowledge
FIG . 8 is a flow chart of an example process for identi- panel for the city of Phoenix . If the query includes only
fying a factual entity for providing a knowledge panel. “ phoenix ” the system may provide a knowledge panel that
FIG . 9 is a flow chart of an example process for deter- includes content for the city of Phoenix and the mythical
mining whether to provide a knowledge panel for a received 60 bird.
search query. One example of a knowledge panel that includes content
FIG . 10 is a flow chart of another example process for for multiple entities is a disambiguation knowledge panel .
determining whether to provide a knowledge panel for a The disambiguation knowledge panel is a knowledge panel
received search query. that includes content for two or more different known
FIG . 11 is a flow chart of another example process for 65 entities and / or user interface elements that enable the user to
determining whether to provide a knowledge panel for a specify which of the known entities is of interest to the user.
received search query. For example, the " phoenix ” disambiguation knowledge
US 10,922,326 B2
5 6
panel could include content items including an artistic example environment 100 includes a network 102 , e.g. , a
rendering of the mythical bird and a map representing local area network (LAN ), wide area network (WAN ), the
Phoenix , Ariz . Each of these content items can be linked to Internet, or a combination of them , connects web sites 104 ,
more information about the respective entities and / or user devices 106 , and the search system 120. The network
include instructions that upon user interaction with one of 5 102 can be accessed over a wired and / or a wireless com
the content items requests a knowledge panel for the entity munications link . For example , mobile computing devices,
represented by the content item . such as smartphones can utilize a cellular network to access
Typically, a disambiguation knowledge panel includes the the network . The environment 100 may include millions of
same or a similar amount of content for each entity refer- web sites 104 and user devices 106 .
enced by the disambiguation knowledge panel. For example , 10 A web site 104 is one or more resources 105 associated
a disambiguation knowledge panel may include, for each with a domain name and hosted by one or more servers . An
entity, a title for the entity, a brief description of the entity, example web site is a collection of web pages formatted in
and a link that, when selected , requests a resource having hypertext markup language ( HTML ) that can contain text,
additional information or initiates a search for the entity . The images , multimedia content, and programming elements,
area of the knowledge panel designated for each entity may 15 e.g. , scripts. Each web site 104 is maintained by a publisher,
also be the same or similar in size . e.g. , an entity that manages and / or owns the web site .
Another example of a knowledge panel that includes A resource 105 is data provided over the network 102 and
content for multiple entities is a dominant entity knowledge that is associated with a resource address, e.g. , a uniform
panel. A dominant entity knowledge panel includes content resource locator. Resources 105 that can be provided by a
for multiple entities, but has a greater amount of content for 20 web site 104 include HTML pages , word processing docu
one or more of the matching entities . For example, although ments, and portable document format (PDF ) documents ,
multiple entities may match the search query , a particular images , video , and feed sources, to name just a few . The
entity may more closely match the search query, may be resources 105 can include content, e.g. , words , phrases,
more topical to search results for the search query , or may be images and sounds and may include embedded information,
more relevant to the search query than the other entities. In 25 e.g. , meta information and hyperlinks, and / or embedded
such a case , the system may provide a dominant entity instructions, e.g. , scripts.
knowledge panel with more content for the particular query , A user device 106 is an electronic device that is under
and less content for each of the other matching entities. For control of a user and is capable of requesting and receiving
example, a dominant entity knowledge panel may include a resources 105 over the network 102. Example user devices
title , image , description, set of facts, links to resources 30 106 include personal computers , mobile computing devices,
related to the particular entity, and additional content for the e.g. , smartphones and / or tablet computing devices , that can
particular entity. The same dominant entity knowledge panel send and receive data over the network 102. As used
may include , for each other matching entity, only a title , throughout this document the term mobile computing device
image , and link to additional content for the entity. (“ mobile device " ) refers to a user device that is configured
In some implementations, the system considers how well 35 to communicate over a mobile communications network . A
an entity matches the query and / or search results identified smartphone, i.e. , a phone that is enabled to communicate
for the query to determine which of multiple matching over the Internet, is an example of a mobile device . A user
entities for which to provide a knowledge panel. For device 106 typically includes a user application, e.g. , a web
example, the system may select an entity that closely browser, to facilitate the sending and receiving of data over
matches the query and that is well represented by the search 40 the network 102 .
results for the query . To facilitate searching of resources 105 , the search system
In some implementations, the system identifies a first set 120 identifies the resources 105 by crawling and indexing
of entities that each matches the query. The system also the resources 105 provided on web sites 104. Data about the
identifies a second set of entities that are considered topical resources 105 can be indexed based on the resource to which
to the search results. The system may then select between 45 the data corresponds. The indexed and, optionally, cached
entities that are included in both the first and second sets . copies of the resources 105 are stored in a search index 122 .
This helps ensure that the selected entity for which to The user devices 106 submit search queries 109 to the
provide a knowledge panel both matches the search query search system 120. In response , the search system 120
and is also on topic for the query. Several operations for accesses the search index 122 to identify resources 105 that
selecting between entities are described in detail below. 50 are relevant to , e.g. , have at least a minimum specified
The system may also consider other factors when deter- relevance score for, the search query 109. The search system
mining whether to provide a knowledge panel for a received 120 identifies the resources 105 , generates search results 111
query, such as click - through rates or other performance that identify the resources 105 , and returns the search results
measures of search results for the received query and / or 111 to the user devices 106. A search result 111 is data
content available to include in knowledge panel for an entity 55 generated by the search system 120 that identifies a resource
referenced by the received query. For example , the system 105 that is responsive to a particular search query, and
may consider whether search results for the query have a low includes a link to the resource 105. An example search result
engagement rate, e.g. , a low click -through rate , whether the 111 can include a web page title , a snippet of text or a portion
query is non -navigational, e.g. , the click -through rate of a of an image extracted from the web page , and the URL of the
high ranked search result for the query is high , whether there 60 web page .
is sufficient content, for the entity referenced by the query, Data for the search queries 109 submitted during user
to be included in a knowledge panel, and / or whether the sessions are stored in a data store, such as the historical data
knowledge panel for the entity includes links to common store 124. For example, the search system 120 can store
search refinements that users would otherwise have to enter received search queries in the historical data store 124 .
manually . 65 Selection data specifying actions taken in response to search
FIG . 1 is a block diagram of an example environment 100 results 111 provided in response to each search query 109 are
in which a search system 120 provides search services. The also stored in the historical data store 124 , for example , by
US 10,922,326 B2
7 8
the search system 120. These actions can include whether a area consumed by two or more of the search results pre
search result 111 was selected or otherwise interacted with sented in the search results area .
at the user device . The selection data can also include, for The knowledge panel apparatus 130 includes or is com
each selection of a search result 111 , data identifying the municably coupled to one or more data storage units that
search query 109 for which the search result 111 was 5 include a content item store 132 , a knowledge panel store
provided. 134 , and an entity index 136. The content item store 132
The selection data can be used to determine a click- stores content items that can be inserted into a knowledge
through rate or other measure of performance of each search panel. In general, the content items are discrete units of
result 111 with respect to each search query 109. The content, e.g. , data files, and can be in the form of text,
click - through rate for a particular search result with respect 10 images , videos , graphics, audio , tables, or other types of
to a particular search query can be determined by dividing content.
the number of times the search result was selected from The knowledge panel store 134 stores knowledge panels
search results provided for the search query by the number 113 for entities and / or knowledge panel templates that can
of times the search result was presented with search results be populated with content to generate a knowledge panel . In
for the search query . The click -through rates can also be 15 general, a knowledge panel template specifies types of
stored in the historical data store 124 . content items to include in the knowledge panel 113 and
The example environment 100 also includes a knowledge includes placeholders for content items of the specified type .
panel apparatus 130 communicably coupled to the search For example, a knowledge panel template may include
system 120 , e.g. , directly coupled or coupled over a network placeholders for a title , one or more images , a description,
such as network 102. The search system 120 can interact 20 a set of facts, and / or other types of content items .
with the knowledge panel apparatus 130 to determine The entity index 136 includes an index of known entities.
whether to provide a knowledge panel 113 for a search query For each entity, the entity index 136 can include data
109. If it is determined that a knowledge panel 113 should identifying aliases of the entity. For example, a famous
be provided, the knowledge panel apparatus 130 can gen- celebrity may have several nicknames or stage names for
erate the knowledge panel 113 and provide the generated 25 which a user may enter as a query for the celebrity. Each of
knowledge panel 113 to the search system 120. In turn , the these nicknames or stage names can be included as aliases
search system 120 provides the knowledge panel 113 , for the entity in the entity index 136 .
optionally with search results 111 , to the user device 106 that The entity index 136 can also include an index of content
submitted the search query 109. In some implementations, items for each entity. In some implementations, the knowl
the knowledge panel apparatus 130 provides the knowledge 30 edge panel apparatus 130 identifies content items to include
panel 113 to the user device 106 , bypassing the search in knowledge panels 113 for each entity and stores the
system 120 . content items in the content item store 132. The knowledge
A knowledge panel 113 is typically provided for queries panel apparatus 130 can also index the content items in the
that have been determined to reference a particular factual entity index 136 for use in subsequent retrieval of the
entity, such as a person , place , country, landmark , animal, 35 content items .
historical event, organization , business , sports team , sport- FIG . 2 is a screen shot of an example search interface 200
ing event, movie , song , album , game, work of art, or any in which a knowledge panel 220 is presented with search
other entity. In some implementations, a factual entity is a results 211a - 211d . The search interface 200 includes a query
concept, subject, or topic . entry field 205 in which a user can enter a search query 209
In general, a knowledge panel 113 for a particular entity 40 and a search initiation element 207 with which a search
includes content that is obtained from multiple disparate query 209 can be submitted to the search system 120. The
resources and that is related to the particular entity. In some search interface 200 also includes search results 211a - 211d
implementations, a knowledge panel 113 includes a set of and a knowledge panel 220. In this example search interface
factual information deemed relevant to the particular entity. 200 , the knowledge panel 220 and the search results 211a
For example, a knowledge panel 113 for an actor may 45 211d have been received from the search system 120 and / or
include biographical information for the actor, as well as the knowledge panel apparatus 130 in response to the
content associated with movies or television shows that the submission of the search query “ Famous Actor.”
actor has appeared . In some implementations, the knowl- The example knowledge panel 220 includes a title 222 , an
edge panel 113 includes a summary of information related to image of Famous Actor 224 , a description of Famous Actor
the particular entity. For example, a knowledge panel 113 for 50 226 , and a set of facts about Famous Actor 228. This
a nation may include a map of the nation , the flag of the example set of facts 228 includes the date of birth of Famous
nation , the official language of the nation , and / or other facts Actor and the net worth of Famous Actor. Other facts could
and content related to the nation , and each of these discrete also be presented in a “ person ” template and can vary based
units of information may be obtained from a different on the person . For example, if the person is deceased , the
resource than the other discrete units of information. 55 date of death may be included in the set of facts.
In some implementations, a knowledge panel 113 is much In this example, Famous Actor is credited with movie
larger and consumes more area of a search interface than appearances and music . To present information regarding
standard search results 111. For example, the knowledge the movie appearances and the music of Famous Actor, the
panel 113 may span the length or height of three or more knowledge panel 220 includes a set of images 230 of movie
standard search results 111 to accommodate the content 60 posters for movies in which Famous Actor appeared . The set
items in the knowledge panel 113 and to draw attention to of images 230 may be obtained , for example, from an online
the knowledge panel 113 . movie database and /or include a link to the online movie
In some implementations, a knowledge panel 113 is database. The knowledge panel also includes a set of images
displayed in a knowledge panel area . The knowledge panel 232 for albums released by Famous Actor. The set of images
area may be presented with or alongside a search results area 65 232 may be obtained , for example, from an online music
that presents search results 111. In some implementations, database or an online music service and include a link to the
the knowledge panel area consumes a larger area than the source of the images 232. In some implementations, a table
US 10,922,326 B2
9 10
may be used to provide information about movies , albums, particular, the knowledge panel 370 includes a title for
or songs related to Famous Actor. The knowledge panel 220 Famous Actor 372 , an image of Famous Actor 374 , a
also includes a set of images 234 that includes images for description of Famous Actor 376 , a set of facts about
related search queries. In some implementations, the images Famous Actor 378 , and a set of images 380 of movie posters
230 , 232 , and 234 can each include a link that, when 5 for movies in which Famous Actor appeared . The knowl
selected , initiates a search for the subject of the image . edge panel 370 includes a set of images 382 for the other
FIG . 3A is a screen shot of an example search interface entities. The set of images 382 includes an image for each of
300 in which a disambiguation knowledge panel 320 is the other entities. The image for an entity can include a link
presented with search results 311a - 311d . As described that initiates a search for the image's corresponding entity .
above , a disambiguation knowledge panel can be provided 10 FIG . 4 is a flow chart of an example process 400 for
for queries that reference multiple entities. For example, a providing a knowledge panel and / or search results for a
disambiguation knowledge panel may be provided if the search query . The process 400 can , for example, be imple
knowledge panel apparatus 130 determines that it is unclear mented by the search system 120 and / or the knowledge
which of the multiple entities the user is referencing with the panel apparatus 130. In some implementations, the opera
received query. For example, the query “ California Univer- 15 tions of the example process 400 can be implemented as
sity ” can be considered to reference many universities and instructions stored on a non - transitory computer readable
colleges in the state of California . medium , where the instructions cause a data processing
Similar to the search interface 200 , the search interface apparatus to perform operations of the example process 400 .
300 includes a query entry field 305 in which a user can enter Query data specifying a search query is received (402 ) .
a search query 309 and a search initiation element 307 with 20 For example , with reference to FIG . 1 , the search system 120
which a search query 309 can be submitted to the search may receive query data specifying a user query submitted
system 120. The search interface 300 also includes search from a user device 106. In response to receiving the query
results 311a - 311d and an example disambiguation knowl- data , the search system 120 may identify search results 111
edge panel 320. In this example search interface 300 , the responsive to the search query 109 specified by the query
knowledge panel 320 and the search results 311a - 311d have 25 data (404 ) .
been received from the search system 120 in response to the A factual entity referenced by the search query 109 is
submission of the search query “ California University . ” identified (406 ) . In some implementations, the knowledge
As there are many universities in California , it may be panel apparatus 130 identifies a factual entity referenced by
difficult to determine an appropriate university for which to a search query 109 by comparing one or more terms of the
provide a knowledge panel. Thus, the knowledge panel 30 search query 109 to a list of known factual entities and / or
apparatus 130 may provide the disambiguation knowledge aliases of the known factual entities. For example, if the
panel 320 , which includes content about multiple universi- search query is “ Ima Singer,” the knowledge panel apparatus
ties in California . In particular, the knowledge panel 320 130 may determine whether “ Ima Singer” is a factual entity
includes content for several public universities located in by comparing “ Ima” and / or “ Singer ” to a list of identifiers
California 322 and content for several private universities 35 for known factual entities that are indexed in the entity index
located in California 324. The content for each university 136. If, based on the comparison , an identifier for a single
can include a link that, when selected, initiates a search for factual entity is determined to match the search query 109 ,
that university , or that links to the official web page or the single factual entity is identified as the factual entity
another resource for the university. being referenced by the search query 109 .
FIG . 3B is a screen shot of an example search interface 40 If another entity, in addition to the single factual entity,
350 in which a dominant entity knowledge panel 370 is has an alias of “ Ima” or “ Singer , ” then the knowledge panel
presented with search results 361a - 361d . As described apparatus 130 may also identify that entity as a factual entity
above , a dominant entity knowledge panel may be provided referenced by the search query “ Ima Singer.” For example,
when multiple entities match a search query, but a particular the knowledge panel apparatus 130 may identify “ Ima
entity more closely matches , or is more topical to the search 45 Dancer ” as a factual entity referenced by the search query
query than the other matching queries. The dominant entity “ Ima Singer . ” By way of another example, the knowledge
knowledge panel may include more content for the particu- panel apparatus 130 may identify the entities “ Ima Singer ”
lar entity than the other entities . and “ Ima Dancer” for a search query that includes the terms
Similar to the search interface 300 , the search interface “ videos of Ima, " as both entities have an alias with the term
300 includes a query entry field 305 in which a user can enter 50 “ Ima ” of the search query.
a search query 309 and a search initiation element 307 with If multiple factual entities match or have an alias that
which a search query 309 can be submitted to the search matches the search query 109 or a portion of the search
system 120. The search interface 350 also includes search query 109 , the knowledge panel apparatus 130 may select
results 361a - 361d and an example dominant entity knowl- one of the matching entities as the identified factual entity or
edge panel 370. In this example search interface 350 , the 55 determine to provide a disambiguation knowledge panel, or
knowledge panel 370 and the search results 361a - 361d have a dominant entity knowledge panel, that includes content for
been received from the search system 120 in response to the multiple factual entities. In some implementations, the
submission of the search query “ Famous Person . ” knowledge panel apparatus 130 considers a measure of
In this example, the knowledge panel apparatus 130 has topicality between each of the entities and one or more of the
determined that the entities “ Famous Actor,” “ Famous 60 identified search results when selecting the entity. For
Singer,” “ Famous Golfer, ” “ Famous Chef,” and “ Famous example, the knowledge panel apparatus 130 may select the
Explorer , ” each match the search query Famous Person . The entity that is most topical to the identified search results or
knowledge panel apparatus 130 has also determined that the the entity that is topical to the highest ranked identified
entity Famous Actor more closely matches, or is more search result for the search query 109 .
topical, to the search query Famous Person than the other 65 In general, the topicality between an entity and a search
entities. Thus, the knowledge panel 370 includes more result 109 , referred to herein as a partial topicality score, is
content for Famous Actor than for the other entities . In determined based on the contents of the resource to which
US 10,922,326 B2
11 12
the search result is linked . In some implementations, the search query. If the click -through rate for a high ranked
partial topicality score for an entity and a search result is search result is relatively high, e.g. , greater than a threshold
based on a quantity , placement, and/ or type of references to or at least a threshold amount greater than each of the other
the entity that are included in the resource for the search search results, this may indicate that the resource to which
result . The references can be in the form of text, e.g. , text 5 the search result links satisfies the informational needs of the
identifying the entity, images , e.g. , an image having meta users that submit the search query . In such cases , it may not
information identifying the entity, audio , video or other be advantageous to provide a knowledge panel 113 in
content that references the entity . As an example, a web page
that includes multiple references to a particular landmark response to the search query. If the click -through rate for the
may have a higher topicality score for the landmark than a 10 a threshold orsearch
high ranked result is not relatively high , e.g. , less than
within a threshold amount of another search
web page that includes a single reference to the landmark . result, then the knowledge panel apparatus 130 may deter
In some implementations, the partial topicality score for
an entity and a search result is based on the location of the mine entity
to provide a knowledge panel 113 for the identified
in response to the search query.
references to the entity in the resource for the search result. In some implementations, the knowledge panel apparatus
Different resource areas , e.g. , title , main text, tags , and 15 130 considers the amount of content available to include in
toanchor text, may
determine thebepartial
given topicality
different weights in a relationship
score . For example, a a knowledge panel for the entity when determining whether
reference to the entity included in the title of the resource to provide a knowledge panel in response to a search query .
may be assigned a greater weight than a reference to the For example, if the content item store 132 does not contain
entity included in the footnote of the resource . By way of 20 sufficient content to populate a knowledge panel 113 for the
another example, references that are located near a begin- entity, e.g. , fewer than a threshold number of content items,
ning of a resource, e.g. , within a threshold quantity of words the knowledge panel apparatus 130 may determine that a
of an article or a threshold quantity of pixels of a top of a knowledge panel 113 for the entity will not be provided in
web page , may be assigned a higher weight than a reference response to the search query .
located near an end of the resource , e.g. , within a footnote 25 In some implementations, the knowledge panel apparatus
or within a threshold quantity of words or pixels of a bottom 130 bases the determination of whether to provide a knowl
of a web page . Processes for selecting a factual entity from edge panel, at least in part, on the type of content that is
multiple matching entities based on topicality are described available for inclusion in a knowledge panel 113 for the
in more detail below. entity . For example, the knowledge panel apparatus 130 may
A determination is made whether to provide a knowledge 30 have knowledge panel templates that include placeholders
panel for the identified entity (408 ) . If a single entity is for certain types of content to include in the knowledge
identified , or selected from multiple entities that match the panels. If a particular type of content specified by the
search query , the knowledge panel apparatus 130 may knowledge panel template is not available for the entity, then
determine to provide a knowledge panel 113 for the entity in the knowledge panel apparatus 130 may determine to not
response to identifying the entity . 35 provide a knowledge panel 113 for the entity.
The knowledge panel apparatus 130 may also consider In some implementations, the knowledge panel apparatus
additional criteria in determining whether to provide a 130 bases the determination of whether to provide a knowl
knowledge panel 113 for the entity. In some implementa- edge panel, at least in part, on the content of a knowledge
tions , the knowledge panel apparatus 130 considers criteria panel 113 that has been generated for the entity. For
regarding content for the knowledge panel 113 and / or char- 40 example , the knowledge panel apparatus 130 determines
acteristics of the identified search results . whether the knowledge panel 113 for the entity includes
In some implementations, the knowledge panel apparatus links to refinements, e.g. , common refinements , of the
130 considers performance measures , e.g. , click - through received search query . If the knowledge panel 113 includes
rates, for one or more of the identified search results withrefinements for the received search query , the knowledge
respect to the search query when determining whether to 45 panel apparatus 130 may determine to provide a knowledge
provide a knowledge panel 113 for the identified entity . For panel 113 for the entity.
example, if the higher ranked search results for the received A combination of the aforementioned criteria can be used
search query have low click - through rates, e.g., less than a to determine whether to provide a knowledge panel 113. For
threshold click - through rate or a lower click -through rate example, the knowledge panel apparatus 130 may consider
than a specified portion of all search queries, with respect to 50 click - through rates for the identified search results with
the received search query, this may indicate that the search respect to the search query in combination with the amount
results do not satisfy users ' informational needs . In such of content available for the entity. Many other combinations
cases , it may be beneficial to provide a knowledge panel 113 are also possible . These criteria can also be used to deter
with content relevant to the identified entity or a knowledge mine the type of knowledge panel to provide, for example if
panel that includes content for multiple entities referenced 55 the search query matches multiple entities.
by the received search query and / or referenced by the If a determination is made to provide a knowledge panel
identified search results . It may also be beneficial to provide 113 , a knowledge panel 113 and optionally search results 111
a knowledge panel 113 that includes links to query revisions are provided ( 410 ) . For example, the knowledge panel
for the received search query, for example query revisions apparatus 130 may identify a knowledge panel 113 for the
that are often received subsequent to the received query . 60 identified entity, or entities, and provide the knowledge
Such a knowledge panel 113 may assist a user in locating the panel 113 , e.g. , data identifying a network location of the
information for which the user is searching. knowledge panel , to the search system 120. In turn , the
In some implementations, the knowledge panel apparatus search system 120 may provide the knowledge panel 113 to
130 considers the click -through rate for a high ranked the user device 106 from which the search query was
identified search result , e.g. , top ranked search result , or one 65 received . The search system 120 may also provide search
of the top ten search results , for determining whether to results 111 for the received search query 109 to the user
provide a knowledge panel in response to the received device 106. The user device 106 may present the knowledge
US 10,922,326 B2
13 14
panel 113 and the search results 111 in a search interface, by comparing the factual entities of the first set to the factual
such as the search interface 200 of FIG . 2 or the search entities of the second and include the matching factual
interface 300 of FIG . 3 . entities in the third set .
If a determination is made to not provide a knowledge At least one of the factual entities of the third set is
panel 113 , search results 111 are provided (412 ) . For 5 identified for reference in a knowledge panel (510 ) . In some
example, the knowledge panel apparatus 130 may provide to implementations, the knowledge panel apparatus 130 con
the search system 120 data indicating that a knowledge siders the measures of topicality between the entities in the
panel 113 should not be provided . In turn , the search system third set and the search results 111 to identify an entity for
120 can provide search results 111 without a knowledge a knowledge panel. For example, the knowledge panel
panel 113 to the user device 106 from which the search 10 apparatus 130 may consider the overall topicality score for
each entity in the third set when identifying one or more
query 109 was received .
FIG . 5 is a flow chart of an example process 500 for entities to reference in the knowledge panel.
identifying a factual entity for providing a knowledge panel. apparatus 130, may
In addition or in the alternative, the knowledge panel
consider click - through rates of search
The process 500 can , for example , be implemented by the 15 results that reference each entity. For example, the knowl
knowledge panel apparatus 130. In some implementations, edge panel apparatus 130 may interact with the search
the operations of the example process 500 can be imple system 120 to identify a click -through rate for search results
mented as instructions stored on a non - transitory computer identified for the search query 109. The click -through rate
readable medium , where the instructions cause a data pro- for each search result is measured with respect to the search
cessing apparatus to perform operations of the example 20 query 109. The search results may be limited to a particular
process 500 . number, e.g. , 10 , of the highest ranked search results . The
Data specifying a search query 109 and search results 111 knowledge panel apparatus 130 may determine an overall
for the search query 109 are received (502 ) . For example , the click -through rate for each entity based on the click - through
search system 120 may receive the search query 109 from a rate of each search result that references the entity. For
user device 106. In response to receiving the search query 25 example, if two search results reference the entity, or the
109 , the search system 120 may identify search results 111 resource referenced by the search results reference the entity,
for the search query 109 and provide data specifying the the click - through rates for those two search results can be
search query 109 and the search results 111 to the knowledge used tosome determine an overall click - through rate for the entity.
panel apparatus 130. This search query 109 and search 30 130Indetermines implementations, the knowledge panel apparatus
results 111 may be provided to the knowledge panel appa The entity scoreanforentity score for each entity of the third set.
ratus 130 along with a request to provide a knowledge panel topicality score for theentity
can may be based on the overall
113 or to determine whether a knowledge panel 113 is through rate for the entity. entity For
and / or the overall click
example , the entity score for
appropriate for the search query 109 . an entity may be the sum or average of the overall topicality
A first set of factual entities that match or have an alias 35 score and the overall click -through rate for the entity.
that matches the search query 109 are identified ( 504 ) . In The knowledge panel apparatus 130 can select one or
some implementations, the knowledge panel apparatus 130 more of the entities for reference in the knowledge panel
compares one or more terms of the search query 109 to a set based on the entity scores . For example, the knowledge
of known entities, for example a set of known entities panel apparatus 130 may select the entity having the highest
indexed in the entity index 136. The knowledge panel 40 entity score. However, in some cases , two or more distinct
apparatus 130 can include, in the first set of factual entities, entities may have similar entity scores . Thus, it may be
any known factual entity that matches or includes an alias beneficial to consider a difference in the entity scores
that matches one or more terms of the search query 109 . between the entity having the highest entity score and one or
A second set of factual entities that are topical to the more entities having a lower entity score , e.g. , second and or
search results are identified (506 ) . In some implementations, 45 third highest scores . If there is a substantial difference
the knowledge panel apparatus 130 identifies entities refer- between the highest entity score and the second highest
enced by the search results 111 or a proper subset of the entity score , e.g. , the difference is greater than a first
search results 111. For example, the knowledge panel appa- threshold, the knowledge panel apparatus 130 may select
ratus 130 may identify each entity referenced by a certain only the entity having the highest entity score for reference
number of the highest ranked search results for the search 50 in the knowledge panel.
query 109. For each identified entity, the knowledge panel If the difference between the highest entity score and the
apparatus 130 determines a measure of topicality between second highest entity score is small , e.g. , the difference is
the entity and one or more of the search results. For example , less than a second threshold, the knowledge panel apparatus
the knowledge panel apparatus 130 may determine an over- 130 may determine to provide a disambiguation knowledge
all topicality score for each entity as described below with 55 panel that references the entity having the highest entity
reference to FIG . 7. Using the overall topicality scores , the score and the entity having the second highest entity score .
knowledge panel apparatus 130 can rank the identified The knowledge panel apparatus 130 can also reference
entities and select entities for the second set based on the additional entities in the disambiguation knowledge panel .
ranking. Example processes for selecting the second set of For example, if the difference between the highest entity
factual entities are illustrated in FIGS . 6 and 7 and described 60 score and the third highest entity score is less than the
below. second threshold , the knowledge panel apparatus 130 may
A third set of factual entities is identified ( 508 ) . The third reference the entity having the third highest entity score in
set of factual entities can include factual entities that are the knowledge panel .
included in both the first set and the second set . Thus, the If the difference between the highest entity score and the
third set can include factual entities that both match the 65 second highest entity score is between the first threshold and
search query 109 and are topical to the search results 111 . the second threshold , the knowledge panel apparatus 130
The knowledge panel apparatus 130 can identify the third set may determine to provide a dominant entity knowledge
US 10,922,326 B2
15 16
panel that includes content for the entity having the highest the respective factual entity and the resource to which the
entity score and content for the entity having the second respective search result 111 links. For example, an official
highest entity score . The knowledge panel may include more web page or online encyclopedia web page for a particular
content for the entity having the highest entity score than the celebrity may have a higher topicality score for the celebrity
entity having the second highest entity score . Additional 5 than a web page that includes content about many different
entities may also be referenced by the dominant entity celebrities.
knowledge panel if the difference between the highest entity In some implementations, the knowledge panel apparatus
score and the entity score for the additional entity is within 130 determines the partial topicality scores based on the
the first and second thresholds. content of the resources to which the search results link and
In some implementations, rather than consider the differ- 10 the factual entities, as described above. In some implemen
ence between entity scores to determine whether to select
multiple entities , and therefore to select the type of knowl tations , the knowledge panel apparatus 130 obtains the
edge panel, the knowledge panel apparatus 130 may con topicalitytopicality
partial
index or
score from another source , such as a
a topicality score apparatus.
sider the ratio between the entity scores . For example, if the A proper subset of the factual entities is selected based on
ratio between the highest entity score and the second highest 15 the partial topicality scores ( 610 ) . In some implementations,
entity score exceeds a first threshold , the knowledge panel the knowledge panel apparatus 130 ranks the factual entities
apparatus 130 may determine to provide a knowledge panel
for the entity having the highest entity score . If the ratio is based on the partial topicality scores and selects a certain
less than a second threshold, the knowledge panel apparatus number of the higher ranked factual entities or those having
130 may determine to provide a disambiguation knowledge 20 a partial topicality score that satisfies a threshold topicality
panel for the entity having the highest entity score and one score for the proper subset . For example, a partial topicality
or more additional entities. If the ratio is between the first score may satisfy the threshold topicality score by meeting
and second thresholds, the knowledge panel apparatus 130 or exceeding the threshold topicality score .
may determine to provide a dominant entity knowledge In some implementations, the knowledge panel apparatus
panel for the entity having the highest entity score and one 25 130 determines an overall topicality score for each factual
or more additional entities. entity using the partial topicality scores for the factual entity
FIG . 6 is a flow chart of an example process 600 for and selects the entities for the proper subset based on the
identifying a subset of factual entities for a received search overall topicality scores . FIG . 7 is a flow chart of an example
query. The process 600 can , for example , be implemented by process 700 for selecting a subset of factual entities based on
the knowledge panel apparatus 130. In some implementa- 30 overall topicality scores . The process 700 can, for example ,
tions , the operations of the example process 600 can be be implemented by the knowledge panel apparatus 130. In
implemented as instructions stored on a non - transitory com- some implementations, the operations of the example pro
puter readable edii where the instructions cause a data cess 700 can be implemented as instructions stored on a
processing apparatus to perform operations of the example non - transitory computer readable medium , where the
process 600 . 35 instructions cause a data processing apparatus to perform
Data specifying a search query 109 and search results 111 operations of the example process 700 .
for the search query 109 are received ( 602 ) . For example , the An overall topicality score is determined for each factual
knowledge panel apparatus 130 may receive the data from entity of a set of factual entities based on partial topicality
the search system 120 in a manner similar to that described scores for the factual entity (702 ) . In some implementations,
above with reference to FIG . 5 . 40 the knowledge panel apparatus 130 determines the overall
A set of factual entities referenced by the search results topicality score for a particular factual entity by computing
111 is identified ( 604 ) . In some implementations, the knowl- a sum or weighted sum of the partial topicality scores for the
edge panel apparatus 130 compares terms found on the particular factual entity . In some implementations, the
resources to which the search results 111 link to a list of knowledge panel apparatus 130 considers only the partial
known entities and their aliases . Each known entity that 45 topicality scores for a particular number of the higher ranked
matches one or more of the terms may be included in the set search results when determining the overall topicality
of factual entities. scores . For example, the knowledge panel apparatus 130
A proper subset of the higher ranked search results 111 is may determine the overall topicality score for a factual
identified ( 606 ) . In some implementations, the knowledge entity by determining the sum of the partial topicality scores
panel apparatus 130 identifies a particular number of the 50 for the factual entity with respect to the top “ n ” search
higher ranked search results 111. For example, the knowl- results, where “ n ” is a non -zero number.
edge panel apparatus 130 may be configured to identify, A proper subset of the factual entities are selected based
based on the relative placements among the search results , on the overall topicality scores (704 ) . In some implementa
the top 10 , top 20 , or top 50 search results . In some tions , the knowledge panel apparatus 130 selects a certain
implementations, the knowledge panel apparatus 130 may 55 number of the factual entities having the higher overall
receive rank or relevance scores for each search result 111 topicality scores . In some implementations, the knowledge
with respect to the search query 109 and use this information panel apparatus 130 selects each factual entity that has a
to identify the proper subset of search results . For example , topicality score that satisfies a threshold topicality score .
the knowledge panel apparatus 130 may identify for the FIG . 8 is a flow chart of an example process 800 for
proper subset the search results 111 that have a rank or 60 identifying a factual entity for providing a knowledge panel.
relevance score that satisfies a particular threshold score . A The process 800 can, for example , be implemented by the
rank or relevance score may satisfy the threshold score by knowledge panel apparatus 130. In some implementations,
meeting or exceeding the threshold score . the operations of the example process 800 can be imple
For each factual entity of the set of factual entities, a mented as instructions stored on a non - transitory computer
partial topicality score is identified for the entity with respect 65 readable medium , where the instructions cause a data pro
to each search result of the proper subset ( 608 ) . Each partial cessing apparatus to perform operations of the example
topicality score is a measure of topical relatedness between process 800 .
US 10,922,326 B2
17 18
Data specifying a search query 109 and search results 111 A click -through rate is identified for each search result 111
for the search query 109 are received (802 ) . For example , the with respect to the search query 109. For example, the search
knowledge panel apparatus 130 may receive the data from system 120 may provide the click - through rates to the
the search system 120 as described above with reference to knowledge panel apparatus 130 along with the data speci
FIG . 5 . 5 fying the search query 109 and the search results 111 .
A set of factual entities that match or have an alias that A determination is made whether to provide a knowledge
matches the search query 109 are identified ( 804 ) . In some on one113or more
panel in response to the received search query 109 based
of the identified click -through rates (906 ) . In
implementations, the knowledge panel apparatus 130 com some implementations , the knowledge panel apparatus 130
pares one or more terms of the search query 109 to a set of
known entities, for example a set of known entities indexed 10 evaluates the click
search results 111 , -e.g.
through rates of atnumber
, a particular least aofsubset of the
the highest
in the entity index 136. The knowledge panel apparatus 130 ranked
can include any known factual entity that matches or through search rates
results , to determine whether the click
indicate that the search results 111 have low
includes an alias that matches one or more terms of the engagement for the search query 109. For example, if the
search query 109 in the set of factual entities.
A partial topicality score is determined for each search less than a threshold click -throughresults
15 click - through rates for the search
rate ,
111 are low , e.g. ,
this may indicate
result 111 with respect to each factual entity of the set of that the search results 111 do not satisfy the users ' informa
factual entities (806 ) . These partial topicality scores can be tional needs . In such cases , the knowledge panel apparatus
the same or substantially similar to the partial topicality 130 may determine to provide a knowledge panel 113 with
scores described above with respect to FIG . 6. In general, the 20 content relevant to the identified entity, a disambiguation
partial topicality score for a particular search result with knowledge panel that includes content for multiple entities
respect to a particular factual entity is a measure of topical referenced by the received search query , or a knowledge
relatedness between the particular search result and the panel 113 that includes links to query revisions for the
particular factual entity. received search query.
A proper subset of the partial topicality scores that satisfy 25 In some implementations, the click - through rate for one
a threshold topicality score is identified (808 ) . For example, or more high ranked search results for the search query 109
the knowledge panel apparatus 130 may compare each are evaluated to determine whether to provide a knowledge
partial topicality score to the threshold topicality score to panel 113 in response to the search query 109. For example,
identify the partial topicality scores that exceed the threshold if the click - through rate for the highest ranked search result
topicality score . 30 is high, e.g. , greater than a threshold or at least a threshold
A determination is made as to which of the proper subset amount greater than each of the other search results, this
of partial topicality scores corresponds to the highest rank- may indicate that the resource to which the search result
ing search results 111 for the search query 109 (810 ) . Each links satisfies the informational needs of the users that
partial topicality score corresponds to a particular search submit the search query. In such cases , it may not be
result and a particular factual entity . The knowledge panel 35 advantageous to provide a knowledge panel 113 in response
apparatus 130 can identify the search result to which each to the search query since the user's informational need may
topicality score included in the proper subset corresponds be satisfied by visiting a single site .
and determine which of the identified search results is the If the knowledge panel apparatus 130 determines to
highest ranked with respect to the other search results to provide a knowledge panel 113 , the knowledge panel appa
which the topicality scores of the proper subset correspond . 40 ratus 130 can provide a knowledge panel 113 to the search
This search result may or may not be the highest ranked system 120 or the user device 106 from which the search
search result overall as the partial topicality scores for that query 109 was received (908 ) . If the knowledge panel
search result may not have satisfied the threshold topicality apparatus 130 determines to not provide a knowledge panel
score . 113 , the knowledge panel 113 apparatus 130 can send data
The factual entity that corresponds to the partial topicality 45 to the search system 120 indicating that a knowledge panel
score determined to correspond to the highest ranking search 113 should not be provided for the search query 109 .
result is selected as the factual entity to reference in a FIG . 10 is a flow chart of another example process 1000
knowledge panel 113. This example process 800 helps for determining whether to provide a knowledge panel for a
increase the likelihood that the factual entity chosen for the received search query . The process 1000 can, for example,
knowledge panel 113 matches the highest ranked search 50 be implemented by the knowledge panel apparatus 130. In
result that has a clear topic . As the knowledge panels 113 are some implementations, the operations of the example pro
often displayed alongside the top results , this approach cess 1000 can be implemented as instructions stored on a
offers a more visually consistent experience for users. non - transitory computer readable medium , where the
FIG . 9 is a flow chart of an example process 900 for instructions cause a data processing apparatus to perform
determining whether to provide a knowledge panel for a 55 operations of the example process 1000 .
received search query. The process 900 can , for example, be An entity referenced by a search query 109 is identified
implemented by the knowledge panel apparatus 130. In ( 1002 ) . For example , the knowledge panel apparatus 130
some implementations, the operations of the example pro- may identify the entity as described above . Content related
cess 900 can be implemented as instructions stored on a to the entity is identified ( 1004 ) . In some implementations,
non - transitory computer readable medium , where the 60 the knowledge panel apparatus 130 accesses the content
instructions cause a data processing apparatus to perform items store 132 to obtain content items for the entity. In some
operations of the example process 900 . implementations, the knowledge panel apparatus 130
Data specifying a search query 109 and search results 111 obtains a knowledge panel template for the entity .
for the search query 109 are received (902 ) . For example, the A determination is made whether there is sufficient con
knowledge panel apparatus 130 may receive the data from 65 tent to provide in a knowledge panel 113 for the entity
the search system 120 as described above with reference to ( 1006 ) . In some implementations, a knowledge panel tem
FIG . 5 . plate includes placeholders for particular types of content to
US 10,922,326 B2
19 20
include in the knowledge panel. For example, a knowledge 109 , the knowledge panel apparatus 130 determines to not
panel template may include a placeholder for a canonical provide a knowledge panel for the entity ( 1110 ) .
image and a set of facts regarding an entity. The knowledge Although FIGS . 9-11 are illustrated and described as
panel apparatus 130 can compare the type of content separate processes, the knowledge panel apparatus 130 can
required by the knowledge panel template to the content 5 consider the criteria of each process in determining whether
available for the entity to determine whether there is content to provide a knowledge panel 113 for a received search
available for each content type , for at least one particular query 109. For example, if the click - through rates for the
type of content, or for at least a minimum number of content search results indicate that a knowledge panel 113 should be
types. For example , if the knowledge panel template provided, but there is not sufficient content available for the
includes a placeholder for a canonical image , but the knowl- 10 identified entity, then the knowledge panel apparatus 130
edge panel apparatus 130 does not have access to a canonical may elect to not provide a knowledge panel.
image for the entity, e.g. , a particular type of content, the Embodiments of the subject matter and the operations
knowledge panel apparatus 130 may determine that there is described in this specification can be implemented in digital
not sufficient content to provide in a knowledge panel 113 . electronic circuitry, or in computer software , firmware, or
If the knowledge panel apparatus 130 determines that 15 hardware, including the structures disclosed in this specifi
there is sufficient content to provide in a knowledge panel cation and their structural equivalents, or in combinations of
113 for the entity , the knowledge panel apparatus 113 one or more of them . Embodiments of the subject matter
determines to provide a knowledge panel 113 for the entity described in this specification can be implemented as one or
( 1008 ) . If the knowledge panel apparatus 130 determines more computer programs, i.e. , one or more modules of
that there is not sufficient content to provide in a knowledge 20 computer program instructions, encoded on computer stor
panel 113 for the entity, the knowledge panel apparatus 113 age medium for execution by, or to control the operation of,
determines to not provide a knowledge panel 113 for the data processing apparatus. Alternatively or in addition, the
entity ( 1010 ) . The knowledge panel apparatus 130 may also program instructions can be encoded on an artificially
send data to the search system 120 indicating that a knowl- generated propagated signal , e.g. , a machine -generated elec
edge panel 113 should not be provided for the search query 25 trical, optical , or electromagnetic signal, that is generated to
109 . encode information for transmission to suitable receiver
FIG . 11 is a flow chart of another example process 1100 apparatus for execution by a data processing apparatus. A
for determining whether to provide a knowledge panel for a computer storage medium can be , or be included in, a
received search query . The process 1100 can , for example , computer -readable storage device , a computer -readable stor
be implemented by the knowledge panel apparatus 130. In 30 age substrate, a random or serial access memory array or
some implementations, the operations of the example pro- device, or a combination of one or more of them . Moreover,
cess 1100 can be implemented as instructions stored on a while a computer storage medium is not a propagated signal ,
non - transitory computer readable edium , where the a computer storage medium can be a source or destination of
instructions cause a data processing apparatus to perform computer program instructions encoded in an artificially
operations of the example process 1100 . 35 generated propagated signal. The computer storage medium
Data specifying a search query 109 are received ( 1102 ) . can also be , or be included in , one or more separate physical
For example, the knowledge panel apparatus 130 may components or media , e.g. , multiple CDs , disks , or other
receive the data from the search system 120 as described storage devices.
above with reference to FIG . 5 . The operations described in this specification can be
An entity referenced by the search query 109 and a 40 implemented as operations performed by a data processing
knowledge panel for the entity are identified ( 1104 ) . For apparatus on data stored on one or more computer - readable
example, the knowledge panel apparatus 130 may identify storage devices or received from other sources .
the entity referenced by the search query 109 , as described The term “ data processing apparatus” encompasses all
above . After identifying the entity, the knowledge panel kinds of apparatus , devices , and machines for processing
apparatus 130 can , for example, access the knowledge 45 data, including by way of example a programmable proces
panels store 134 to identify a knowledge panel 113 for the sor , a computer, a system on a chip , or multiple ones , or
identified entity. combinations, of the foregoing. The apparatus can include
A determination is made whether the knowledge panel for special purpose logic circuitry, e.g. , an FPGA ( field pro
the identified entity includes links to common search refine- grammable gate array ) or an ASIC (application -specific
ments for the received search query 109 ( 1106 ) . In some 50 integrated circuit ) . The apparatus can also include , in addi
implementations, the knowledge panel apparatus 130 deter- tion to hardware, code that creates an execution environment
mines whether the knowledge panel includes links to other for the computer program in question, e.g. , code that con
resources . If so , the knowledge panel apparatus 130 can stitutes processor firmware, a protocol stack , a database
compare the resources to which the links are directed to a set management system , an operating system , a cross -platform
of common query refinements for the search query 109. If 55 runtime environment, a virtual machine , or a combination of
there is at least one match , then the knowledge panel one or more of them . The apparatus and execution environ
apparatus 130 may determine that the knowledge panel for ment can realize various different computing model infra
the identified entity includes links to common search refine- structures, such as web services, distributed computing and
ments for the received search query 109 . grid computing infrastructures.
If the knowledge panel apparatus 130 determines that the 60 A computer program , also known as a program , software,
knowledge panel for the identified entity includes links to software application, script, or code , can be written in any
common search refinements for the received search query form of programming language , including compiled or
109 , the knowledge panel apparatus 130 determines to interpreted languages, declarative or procedural languages,
provide a knowledge panel for the entity ( 1108 ) . If the and it can be deployed in any form , including as a stand
knowledge panel apparatus 130 determines that the knowl- 65 alone program or as a module , component, subroutine,
edge panel for the identified entity does not include links to object, or other unit suitable for use in a computing envi
common search refinements for the received search query ronment. A computer program may, but need not , correspond
US 10,922,326 B2
21 22
to a file in a file system . A program can be stored in a portion server, or that includes a front - end component, e.g. , a client
of a file that holds other programs or data , e.g. , one or more computer having a graphical user interface or a Web browser
scripts stored in a markup language document, in a single through which a user can interact with an implementation of
file dedicated to the program in question , or in multiple the subject matter described in this specification, or any
coordinated files, e.g. , files that store one or more modules , 5 combination of one or more such back - end, middleware, or
sub - programs, or portions of code . A computer program can front - end components. The components of the system can be
be deployed to be executed on one computer or on multiple interconnected by any form or medium of digital data
computers that are located at one site or distributed across communication , e.g. , a communication network . Examples
multiple sites and interconnected by a communication net- of communication networks include a local area network
work . 10 ( “ LAN ” ) and a wide area network ( “ WAN ”) , an inter
The processes and logic flows described in this specifi- network , e.g. , the Internet, and peer - to -peer networks, e.g. ,
cation can be performed by one or more programmable ad hoc peer -to - peer networks.
processors executing one or more computer programs to The computing system can include clients and servers. A
perform actions by operating on input data and generating client and server are generally remote from each other and
output. The processes and logic flows can also be performed 15 typically interact through a communication network. The
by, and apparatus can also be implemented as , special relationship of client and server arises by virtue of computer
purpose logic circuitry, e.g. , an FPGA ( field programmable programs running on the respective computers and having a
gate array ) or an ASIC ( application - specific integrated cir- client - server relationship to each other . In some embodi
cuit) . ments, a server transmits data , e.g. , an HTML page , to a
Processors suitable for the execution of a computer pro- 20 client device, e.g. , for purposes of displaying data and
gram include, by way of example , both general and special receiving user input from a user interacting with the client
purpose microprocessors, and any one or more processors of device . Data generated at the client device, e.g. , a result of
any kind of digital computer. Generally, a processor will the user interaction, can be received from the client device
receive instructions and data from a read -only memory or a at the server.
random access memory or both . The essential elements of a 25 While this specification contains many specific imple
computer are a processor for performing actions in accor- mentation details , these should not be construed as limita
dance with instructions and one or more memory devices for tions on the scope of any inventions or of what may be
storing instructions and data . Generally, a computer will also claimed, but rather as descriptions of features specific to
include , or be operatively coupled to receive data from or particular embodiments of particular inventions. Certain
transfer data to , or both, one or more mass storage devices 30 features that are described in this specification in the context
for storing data , e.g. , magnetic, magneto -optical disks , or of separate embodiments can also be implemented in com
optical disks. However, a computer need not have such bination in a single embodiment. Conversely, various fea
devices. Moreover, a computer can be embedded in another tures that are described in the context of a single embodi
device , e.g. , a mobile telephone, a personal digital assistant ment can also be implemented in multiple embodiments
( PDA) , a mobile audio or video player, a game console , a 35 separately or in any suitable subcombination . Moreover,
Global Positioning System (GPS ) receiver, or a portable although features may be described above as acting in
storage device, e.g. , a universal serial bus (USB ) flash drive, certain combinations and even initially claimed as such , one
to name just a few . Devices suitable for storing computer or more features from a claimed combination can in some
program instructions and data include all forms of non- cases be excised from the combination, and the claimed
volatile memory , media and memory devices, including by 40 combination may be directed to a subcombination or varia
way of example semiconductor memory devices, e.g. , tion of a subcombination .
EPROM , EEPROM , and flash memory devices; magnetic Similarly, while operations are depicted in the drawings in
disks , e.g. , internal hard disks or removable disks; magneto- a particular order, this should not be understood as requiring
optical disks ; and CD - ROM and DVD - ROM disks . The that such operations be performed in the particular order
processor and the memory can be supplemented by, or 45 shown or in sequential order, or that all illustrated operations
incorporated in , special purpose logic circuitry. be performed , to achieve desirable results . In certain cir
To provide for interaction with a user, embodiments of the cumstances, multitasking and parallel processing may be
subject matter described in this specification can be imple- advantageous. Moreover, the separation of various system
mented on a computer having a display device , e.g. , a CRT components in the embodiments described above should not
( cathode ray tube ) or LCD ( liquid crystal display ) monitor, 50 be understood as requiring such separation in all embodi
for displaying information to the user and a keyboard and a ments , and it should be understood that the described
pointing device , e.g. , a mouse or a trackball, by which the program components and systems can generally be inte
user can provide input to the computer. Other kinds of grated together in a single software product or packaged into
devices can be used to provide for interaction with a user as multiple software products.
well ; for example, feedback provided to the user can be any 55 Thus, particular embodiments of the subject matter have
form of sensory feedback , e.g. , visual feedback , auditory been described . Other embodiments are within the scope of
feedback , or tactile feedback ; and input from the user can be the following claims . In some cases , the actions recited in
received in any form , including acoustic , speech , or tactile the claims can be performed in a different order and still
input. In addition, a computer can interact with a user by achieve desirable results. In addition , the processes depicted
sending documents to and receiving documents from a 60 in the accompanying figures do not necessarily require the
device that is used by the user; for example, by sending web particular order shown, or sequential order, to achieve
pages to a web browser on a user's client device in response desirable results . In certain implementations, multitasking
to requests received from the web browser. and parallel processing may be advantageous.
Embodiments of the subject matter described in this What is claimed is :
specification can be implemented in a computing system that 65 1. A method performed by data processing apparatus, the
includes a back - end component, e.g. , as a data server, or that method comprising:
includes a middleware component, e.g. , an application receiving a query from a client device;
US 10,922,326 B2
23 24
obtaining search results that are responsive to the received subset of search results , the partial topicality score for
query ; each search result being a measure of topical related
identifying two different factual entities that each corre ness between the search result and the factual entity ;
spond to the received query and that are both identified and
using a same word ; 5
selecting , from the third set of factual entities , the second
determining, for each of the two identified factual entities , set of factual entities based on the partial topicality
a topicality score that specifies a measure of topical scores .
relatedness between the factual entity and content pre 4. The method of claim 3 , wherein selecting the second set
sented by resources linked to by the obtained search of factual entities based on the partial topicality scores
results ; 10
comprises:
selecting, from among the two different factual entities, a for each factual entity of the third set , determining an
particular factual entity for which a knowledge panel is overall topicality score for the factual entity based on a
to be provided with the obtained search results based , combination of the partial topicality scores for the
at least in part, on the topicality score for each of the factual entity ; and
two factual entities, the knowledge panel for the par- 15 selecting the second set of factual entities based on the
ticular factual entity including a plurality of different
types of content for the particular factual entity , the overall topicality scores.
plurality of different types of content including at least 5. The method of claim 4 , wherein determining the overall
one content item , for the particular factual entity, topicality score for the factual entity based on the partial
received from a first resource and at least one content 20 topicality scores for the factual entity comprises determining
item , for the particular factual entity, received from a a weighted sum of the partial topicality scores for the factual
second resource different than the first resource ; entity.
determining that the knowledge panel for the particular 6. The method of claim 1 , wherein selecting , from among
factual entity is to be provided with the obtained search the two different factual entities, a particular factual entity
results, the determination being based, at least in part, 25 for which a knowledge panel is to be provided with the
on content of the knowledge panel and characteristics search results comprises :
of the obtained search results, the characteristics for each search result , determining a topicality score for
including a performance of one or more resources the search result with respect to each of the two
referenced by the identified search results when search different factual entities, the topicality score for a
results that reference the one or more resources were 30 search result with respect to a factual entity being a
provided in response to the received query ; and measure of topical relatedness of the search result with
providing, to the client device, data that presents the respect to the factual entity ;
knowledge panel at the client device . determining which one of the topicality scores corre
2. The method of claim 1 , wherein : sponds to a highest ranking search result
the two different factual entities are included in a first set 35 selecting , as the particular factual entity, the factual entity
of factual entities that are each identified using the that corresponds to the one topicality score .
same word ; and 7. The method of claim 1 , wherein determining that the
selecting , from among the two different factual entities, a knowledge panel for the particular factual entity is to be
particular factual entity for which a knowledge panel is provided with the obtained search results comprises:
to be provided with the obtained search results com- 40 identifying a click - through rate for each search result with
prises: respect to the received query ;
identifying a second set of factual entities, each factual determining that the identified click - through rates do not
entity of the second set being a factual entity deter- satisfy a threshold click -through rate; and
mined to be topical to the obtained search results , the determining to provide the knowledge panel in response
second set of factual entities including at least one 45 to determining that the identified click -through rates do
factual entity that is not included in the first set of not satisfy the threshold click -through rate .
factual entities; 8. The method of claim 1 , wherein determining that the
identifying the two different factual entities as being knowledge panel for the particular factual entity is to be
included in both the first set of factual entities and in provided with the obtained search results comprises:
the second set of factual entities ; and 50 identifying a top ranked search result for the received
selecting from only among the two different factual query ;
entities in response to the two different factual enti- identifying a click -through rate for the top ranked search
ties being included in both the first set of factual result with respect to the received query ;
entities and in the second set of factual entities. determining that the identified click -through rate does not
3. The method of claim 2 , wherein identifying a second 55 satisfy a click - through rate threshold ; and
set of factual entities, each factual entity of the second set determining to provide the knowledge panel in response
being a factual entity determined to be topical to the to determining that the identified click -through rate
obtained search results , comprises: does not satisfy the click - through rate threshold .
identifying a third set of factual entities , each factual 9. The method of claim 1 , wherein determining that the
entity of the third set being a factual entity referenced 60 knowledge panel for the particular factual entity is to be
in at least one of the obtained search results; provided with the obtained search results comprises :
identifying a proper subset of the obtained search results , identifying content for inclusion in the knowledge panel ;
the proper subset including a particular number of determining that the identified content includes at least a
higher ranked search results ; threshold amount of content; and
for each factual entity of the third set of factual entities, 65 determining to provide the knowledge panel in response
determining, relative to the factual entity, a partial to determining that the identified content includes at
topicality score for each search result in the proper least a threshold amount of content .
US 10,922,326 B2
25 26
10. The method of claim 1 , wherein determining that the results when search results that reference the one or
knowledge panel for the particular factual entity is to be more resources were provided in response to the
provided with the obtained search results comprises : received query ; and
determining whether the knowledge panel includes links providing, to the client device, data that presents the
to query refinements for the received query ; and 5 knowledge panel at the client device.
determining to provide the knowledge panel in response 14. The system of claim 13 , wherein :
to determining that the knowledge panel includes links the two different factual entities are included in a first set
to query refinements for the received query . of factual entities that are each identified using the
11. The method of claim 1 , wherein selecting the particu same word ; and
lar factual entity comprises: 10 selecting , from among the two different factual entities, a
determining that the topicality score for the particular particular factual entity for which a knowledge panel is
factual entity is at least a threshold amount greater than to be provided with the obtained search results com
the topicality score for another factual entity of the two prises:
factual entities; and identifying a second set of factual entities, each factual
determining to provide the knowledge panel for the 15 entity of the second set being a factual entity deter
particular factual entity rather than a disambiguation mined to be topical to the obtained search results , the
knowledge panel that includes content for both factual second set of factual entities including at least one
entities in response to determining that the topicality factual entity that is not included in the first set of
score for the particular factual entity is at least a factual entities;
threshold amount greater than the topicality score for 20 identifying the two different factual entities as being
another factual entity of the two factual entities . included in both the first set of factual entities and in
12. The method of claim 1 , wherein determining that the the second set of factual entities; and
knowledge panel for the particular factual entity is to be selecting from only among the two different factual
provided with the obtained search results comprises deter entities in response to the two different factual enti
mining to provide the knowledge panel for the particular 25 ties being included in both the first set of factual
factual entity in response to a performance of a resource entities and in the second set of factual entities.
referenced by a highest ranked search result of the identified 15. The system of claim 14 , wherein identifying a second
search results being less than a threshold . set of factual entities, each factual entity of the second set
13. A system comprising: being a factual entity determined to be topical to the
a data processing apparatus ; and 30 obtained search results , comprises:
a memory storage apparatus in data communication with identifying a third set of factual entities, each factual
the data processing apparatus, the memory storage entity of the third set being a factual entity referenced
apparatus storing instructions executable by the data in at least one of the obtained search ults ;
processing apparatus and that upon such execution identifying a proper subset of the obtained search results ,
cause the data processing apparatus to perform opera- 35 the proper subset including a particular number of
tions comprising : higher ranked search results ;
receiving a query from a client device ; for each factual entity of the third set of factual entities,
obtaining search results that are responsive to the determining, relative to the factual entity, a partial
received query ; topicality score for each search result in the proper
identifying two different factual entities that each cor- 40 subset of search results , the partial topicality score for
respond to the received query and that are both each search result being a measure of topical related
identified using a same word ; ness between the search result and the factual entity ;
determining, for each of the two identified factual and
entities, a topicality score that specifies a measure of selecting , from the third set of factual entities, the second
topical relatedness between the factual entity and 45 set of factual entities based on the partial topicality
content presented by resources linked to by the scores .
obtained search results; 16. The system of claim 15 , wherein selecting the second
selecting, from among the two different factual entities , set of factual entities based on the partial topicality scores
a particular factual entity for which a knowledge comprises :
panel is to be provided with the obtained search 50 for each factual entity of the third set , determining an
results based , at least in part, on the topicality score overall topicality score for the factual entity based on a
for each of the two factual entities, the knowledge combination of the partial topicality scores for the
panel for the particular factual entity including a factual entity; and
plurality of different types of content for the particu- selecting the second set of factual entities based on the
lar factual entity, the plurality of different types of 55 overall topicality scores.
content including at least one content item , for the 17. The system of claim 13 , wherein selecting , from
particular factual entity , received from a first among the two different factual entities, a particular factual
resource and at least one content item , for the par- entity for which a knowledge panel is to be provided with
ticular factual entity, received from a second the search results comprises:
resource different than the first resource ;
determining that the knowledge panel for the particular
60
fortheeachsearch result,with
searchresult determining a topicality score for
respect to each of the two
factual entity is to be provided with the obtained different factual entities, the topicality score for a
search results, the determination being based, at least search result with respect to a factual entity being a
in part, on content of the knowledge panel and measure of topical relatedness of the search result with
characteristics of the obtained search results , the 65 respect to the factual entity;
characteristics including a performance of one or determining which one of the topicality scores corre
more resources referenced by the identified search sponds to a highest ranking search result
US 10,922,326 B2
27 28
selecting , as the particular factual entity, the factual entity results, the determination being based , at least in part,
that corresponds to the one topicality score . on content of the knowledge panel and characteristics
18. The system of claim 13 , wherein determining that the of the obtained search results , the characteristics
knowledge panel for the particular factual entity is to be including a performance of one or more resources
provided with the obtained search results comprises : 5
referenced by the identified search results when search
identifying a click - through rate for each search result with results that reference the one or more resources were
respect to the received query ; provided in response to the received query ; and
determining that the identified click - through rates do not providing , to the client device , data that presents the
satisfy a threshold click - through rate; and knowledge panel at the client device.
determining to provide the knowledge panel in response 10 21. The computer storage medium of claim 20 , wherein :
to determining that the identified click - through rates do the two different factual entities are included in a first set
not satisfy the threshold click - through rate. of factual entities that are each identified using the
19. The system of claim 13 , wherein determining that the same word; and
knowledge panel for the particular factual entity is to be selecting , from among the two different factual entities, a
provided with the obtained search results comprises: 15
particular factual entity for which a knowledge panel is
identifying a top ranked search result for the received
query ; to be provided with the obtained search results com
identifying a click -through rate for the top ranked search prises :
result with respect to the received query ; identifying a second set of factual entities, each factual
determining that the identified click - through rate does not 20 entity of the second set being a factual entity deter
satisfy a click -through rate threshold ; and mined to be topical to the obtained search results , the
determining to provide the knowledge panel in response second set of factual entities including at least one
to determining that the identified click - through rate factual tity that is not included in the first set of
does not satisfy the click -through rate threshold . factual entities;
20. A computer storage medium encoded with a computer 25 identifying the two different factual entities as being
program , the program comprising instructions that when included in both the first set of factual entities and in
executed by data processing apparatus cause the data pro the second set of factual entities; and
cessing apparatus to perform operations comprising: selecting from only among the two different factual
receiving a query from a client device; entities in response to the two different factual enti
obtaining search results that are responsive to the received 30 ties being included in both the first set of factual
query ; entities and in the second set of factual entities.
identifying two different factual entities that each corre 22. The computer storage medium of claim 21 , wherein
spond the received query and that are both identified identifying a second set of factual entities , each factual
using a same word ; entity of the second set being a factual entity determined to
determining, for each of the two identified factual entities, 35 be topical to the obtained search results , comprises:
a topicality score that specifies a measure of topical identifying a third set of factual entities, each factual
relatedness between the factual entity and content pre entity of the third set being a factual entity referenced
sented by resources linked to by the obtained search in at least one of the obtained search results ;
results ;
selecting , from among the two different factual entities, a 40 identifying
the
a proper subset of the obtained search results ,
proper subset including a particular number of
particular factual entity for which a knowledge panel is higher ranked search results ;
to be provided with the obtained search results based , for each factual entity of the third set of factual entities ,
at least in part, on the topicality score for each of the determining, relative to the factual entity , a partial
two factual entities, the knowledge panel for the par topicality score for each search result in the proper
ticular factual entity including a plurality of different 45 subset of search results , the partial topicality score for
types of content for the particular factual entity, the each search result being a measure of topical related
plurality of different types of content including at least ness between the search result and the factual entity ;
one content item , for the particular factual entity, and
received from a first resource and at least one content selecting , from the third set of factual entities, the second
item , for the particular factual entity, received from a 50 set of factual entities based on the partial topicality
second resource different than the first resource ; scores .
determining that the knowledge panel for the particular
factual entity is to be provided with the obtained search

Triggerin Knowledge Panels Patent

Uploaded by

Copyright:

Available Formats

You might also like

Triggerin Knowledge Panels Patent

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Triggerin Knowledge Panels Patent

Uploaded by

Copyright:

Available Formats

USO10922326B2

United States Patent

Search System Knowledge Panel Apparatus

Search Historical Content Items Knowledge Entity Index

Search System Knowledge Panel Apparatus

Search Historical Knowledge Entity Index

AFacmtours FAacmtours Image

MA-FSEcxaoitmvrpiu'les FsmPAtuoacrvtihoaeunsrg –FSMEAxacuimtsopiulercs AFpsPbyeuacormftnhrogaumserd

APnoathger 309 -SCexamipl.co/unhivlersaty1d/hml S-Cexamipl.co/unhversiaty2dml e-SCxampil.co/unhvlersiaty3d/m -SeCxamipl.co/unhvlersiaty4d/ml

Identify factual entitiesandthatthe aresecond

Select at least one of the factual entities that is included in 510

Select a proper subset of the factual entities

For eachsearchresult, determine a partialtopicality scores 806

Identify propersubset ofpartialtopicality scores that satisfya5 808

Determine which one of the proper subset of partial topicality

Yes Determine whether No

Identify content related to entity for knowledge panel 5 1004

Identify entity referencedpanelby received

Yes Knowledge panel No

You might also like