Machine Learning and Human

Capital: Experimental Evidence on

Productivity Complementarities
Prithwiraj Choudhury Evan Starr
Rajshree Agarwal

Working Paper 18-065

 Machine Learning and Human Capital:
Experimental Evidence on Productivity
Complementarities
Prithwiraj Choudhury
Harvard Business School

Evan Starr
University of Maryland

Rajshree Agarwal
University of Maryland

Working Paper 18-065

Copyright © 2018 by Prithwiraj Choudhury, Evan Starr, and Rajshree Agarwal

Working papers are in draft form. This working paper is distributed for purposes of comment and discussion only. It may
not be reproduced without permission of the copyright holder. Copies of working papers are available from the author.
 MACHINE LEARNING AND HUMAN CAPITAL: EXPERIMENTAL EVIDENCE ON PRODUCTIVITY
COMPLEMENTARITIES

Prithwiraj Choudhury
Harvard Business School
pchoudhury@hbs.edu

Evan Starr
Robert H. Smith School of Business
University of Maryland
estarr@rhsmith.umd.edu

Rajshree Agarwal
Robert H. Smith School of Business
University of Maryland
rajshree@rhsmith.umd.edu

August 30 2018
Abstract

Machine learning process technologies usher new questions regarding potential complementarity with human
capital. We examine productivity differentials between machine learning and older vintage technology based
on interactions with two important human capital attributes: domain-specific expertise and vintage-specific
human capital. Our experimental setting simulates the US Patent and Trademark Office examination process.
We randomize assignment of novice patent examiners to each process technology, and provision of domain-
specific expertise. Our analysis of examiner accuracy and speed in patent adjudication reveals, irrespective of
process technology, expertise is necessary for identifying prior art that invalidates the patent being examined,
and productivity in the machine learning process technology requires computer science and engineering
(CS&E) skills. We discuss implications for artificial intelligence and strategic management of the pace of
technological substitution.

1
 Artificial intelligence (AI) and machine learning—where algorithms learn from existing patterns in

data to conduct statistically driven predictions and facilitate decisions (Kleinberg et al., 2017; Brynjolfsson and

McAfee 2014)—may well transform future production and work. For example, Bughin et al., (2017) predict

machine learning, when combined with robotics, may increase global productivity growth by 0.8-1.4%, while

automating nearly 50% of current work performed by humans. Accordingly, AI has ushered debate on the

substitution of humans by intelligent machines, and on who may benefit or lose (Autor 2015; Benzell,

Kotlikoff, LaGarda, and Sachs, 2015; Bessen 2016; Brynjolffson and McAfee 2014). Budding empirical

research compares decision making and productivity of machine learning vs. humans, and the wage and

employment effects for vertically differentiated—skilled vs. unskilled—labor (Acemoglu and Restrepo 2017;

Aghion, Bergeaud, Blundell and Griffith, 2017; Cowgill 2017). At least in the foreseeable future, however,

workplaces will require humans to interface with algorithms, especially in the case of AI-related process

technologies which incorporate humans as part of the production process. This raises an important question:

How does the underlying skill composition of human capital influence productivity differentials between a

machine learning technology and the prior-vintage? The answer is critical to determining the pace of

technological substitution (Adner and Kapoor, 2016; Adner and Snow, 2010; Christensen, 1997), aligning

potentially productive technologies with existing workforce capabilities (Bloom, Sadun and Van Reenan,

2012; Henderson 2006), and assessing social and individual welfare effects on individuals’ skill obsolescence

and refurbishment (Bekman, Bound and Machin, 1998; Fernandez, 2001).

We study complementarities between machine learning and older process technologies and human

capital, focusing on two key human capital attributes. Domain-specific expertise—the skills and knowledge

accumulated through prior learning investments within a knowledge domain (Simon, 1991)—is particularly

salient since it is unclear whether and to what extent domain-specific expertise may retain value, given

potentially superior predictions from machine learning technologies. Vintage-specific human capital—the

skills and knowledge specific to the technology accumulated through prior familiarity of tasks with the

technology (Chari and Hopenhayn, 1991)—poses questions regarding the requisite baseline skills for interfacing

with machine learning process technologies. Given machine learning represents an enhancement in prediction

technologies (Kleinberg et al., 2017), we theorize productivity enhancements of machine learning

2
technologies relative to older vintages will be contingent on complementarities between the technology

vintage, domain-specific expertise, and vintage-specific skills.

This framework maps well to the empirical context of patent prior art search conducted in the U.S.

Patent and Technology Office (USPTO), in which a newly developed machine learning process technology

may supplant to the older vintage of Boolean search technology. Specifically, patent examiners within

specialized knowledge domains (e.g. biomedical, chemical, computer, mechanical and natural sciences and

engineering; also law and business) utilize search technologies to generate predictions of relevant prior art as a

critical input when adjudicating novelty of patent application claims, and the scope of intellectual property

should a patent be granted. Domain expertise of patent examiners is critical in determining the search strategy

(i.e. what keywords to search for in available databases while adjudicating claims). For vintage-specific human

capital related to interacting with machine learning technologies and manipulating user interfaces of such

technologies, we focus on individuals with prior experience in computer science and engineering (CS&E),

because it enables higher absorptive capacity (Cohen and Levinthal, 1990).

Our experimental framework permits this examination through random assignment of each process

technology (machine learning and Boolean search), and random provision of domain-specific expertise, to

novice patent examiners who differ in their prior specialized knowledge bases (CS&E vs. non CS&E).

Specifically, we are interested in which combination of skills and technology most improves the patent

examination process, both in terms of time and accuracy of the adjudication process. Accordingly, with the

help of USPTO personnel, we devised the task of adjudicating a patent application with five claims, which

would all be rejected if the examiner identifies just one prior patent (the “silver-bullet”) upon conducting a

search using the randomly assigned technology. We then asked 221 MBA students at a top-tier business

school, stratified by CS&E backgrounds, to simulate a novice patent examiner’s role for claim adjudication.

All participants received general training on the examination process, and a subset were randomly provided

with expert advice (codified domain-specific expertise), in the form of communication from a senior USPTO

patent examiner to the novice examiners.

Our first main finding is that domain-specific expertise is critical to productivity: regardless of the

technology, only when participants are provided with expert advice did they locate the silver bullet. The

3
second main result is that machine learning is typically less likely to find the silver bullet. However, there are

important interaction effects with vintage-specific human capital. Among those with the machine learning

technology who also received expert advice, CS&E examiners were better able to find the silver bullet, while

none of the non-CS&E examiners were able to do so.

To better understand these patterns, we investigated the distribution of patents identified by the

examiners, and their textual similarity to both the focal patent application and the silver bullet. We find the

distribution of patents identified by examiners using the machine learning technology was both narrower (i.e.

more precise) and showed higher similarity with the focal patent application relative to examiners using

Boolean technology, regardless of the provision of expert advice. This indicates machine learning technology

is superior to Boolean technology in “reading” the focal patent and identifying relevant prior art based on

keywords contained in it. Productivity requires identification of patents similar to the silver bullet, however,

rather than patents similar to the focal patent application. Here we find that expert advice uniformly causes

examiners, regardless of technology, to cite patents that are more similar to the silver bullet.

Our deeper examination of activity logs of individual novice examiners revealed reasons for

differences in productivity due to vintage-specific human capital. Within examiners assigned to machine

learning, CS&E examiners were better able to handle the complexities of the user interface than non-CS&E

examiners. Their superior implementation of expert advice when using machine learning technology interface

enabled them to identify more precisely patents similar to the silver bullet, and the silver bullet itself—thus

leading to their greater accuracy. In contrast, on the Boolean technology, while both groups are equally able

to implement the expert advice to identify patents similar to the silver bullet, we found, somewhat to our

surprise, non-CS&E examiners were more effective in identifying the silver bullet itself. We speculate, rather

than conclusively posit, two potential explanations for this result: On Boolean, either CS&E examiners exert

less effort for identification of the silver bullet within the patents yielded by the technology relative to non-

CS&E examiners, or they are relatively worse at reading and sifting through the identified patents and locating

the silver bullet. We hope future work can disentangle these explanations.

In identifying the complementarities between process technologies and human capital attributes of

domain-specific expertise and vintage-specific human capital, the paper contributes to several literature

4
streams at the intersection of innovation and human capital. First, amidst fierce debate about future

replacement of humans at work in eras ushered by artificial intelligence and machine learning (Autor 2015;

Benzell et al., 2015; Bessen 2016; Brynjolffson and McAfee 2014), there is scant attention to how productivity

gains from substituting machine learning for older technology vintages may be conditioned by

complementary human capital. Specifically, theories for artificial intelligence/machine learning’s effect on

productivity and workplace practices (Aghion et al., 2017; Agrawal, Gans and Goldfarb, 2018, Autor, 2015;

Benzell et al., 2015), and theories of technology diffusion (Chari and Hopenhayn, 1991; Jovanovic, 2009)

assume away horizontal differentiation in prior knowledge bases. In highlighting the importance of domain-

specific expertise in ex ante framing the search strategy prior to using any process technology, our study shows

that even within the high-skilled labor force, horizontal differentiation due to specialization of human capital

accounts for significant productivity differentials when interfacing with different technology vintages. Our

results on productivity differentials due to human capital interfaces with old and new technologies also

contributes to the strategic management of innovation literature stream on pace of technology substitution

(Adner and Snow, 2010; Adner and Kapoor, 2016; Agarwal and Prasad, 1999; Christensen, 1997). Here, we

add to specialized (domain-specific and vintage specific) human capital as an important factor that may result

in failures for new technology diffusion in spite of their potential relative superiority. For human capital and

career specialization literature (Becker and Murphy, 1992; Rosen, 1983; Zuckerman et al., 2003), we reaffirm

beneficial effects of human capital investments in developing expertise (e.g. Castanias and Helfat, 1991).

Additionally, we highlight the need to develop skills for interfacing with a focal technology, to the extent

newer technology vintages shift the knowledge base requirements for productivity gains.

The findings of our study are replete with important managerial implications, given the firm’s pre-

existing workforce is its most important capability (Bloom et al., 2012; Hatch and Dyer 2004), and also the

most difficult to change. The prior knowledge bases of the workforce will interact differently with new

technologies, requiring firms and workers to re-think existing complementarities between specialized

knowledge domains, even as they consider the complementarities between workers and technologies. We

suggest one reason firms may fail following adoption of a seemingly profitable technology, despite training to

update skill sets of their workers, is the lack of concomitant revaluation of tasks and complementary human

5
capital. Our finding that machine learning technologies cannot substitute for domain-specific expertise has

two implications. One, domain experts may benefit more from machine learning than novices, because of

the complementarity between the new technology and their (tacit) knowledge gained through years of

practice. Two, domain experts may have a critical role in diffusing new technologies to novices. To unleash

the productivity potential of machine learning, managers should consider a staged introduction of new

technologies, and in doing so, leverage both the knowledge and the status of domain experts for more

widespread acceptance in the rest of the workforce. Our finding that machine learning technologies require

prior CS&E skills for use implies that even if a firm may invest in significant technology-specific training to

help workers adapt to a new technology, such training will not help all firms, or all workers, equally. Our

study offers a cautionary note against a “one size fits all” training regimen that is technology specific, and calls

for more nuanced programs where the training takes into account the absorptive capacity related to prior

knowledge base and skills of individuals. Similarly, given hiring, retention, innovation management, and

compensation are ultimately tied to productivity differentials based on available technologies, the implications

of our study extend to a careful re-thinking of these practices too.

LITERATURE REVIEW AND THEORY

Our study builds upon insights from two literature streams—implications of AI for productivity and

workplace practices, and vintages of technology and human capital. We provide a brief overview of each and

then integrate the two to develop a guiding framework for our study’s examination of productivity

differentials of process technologies, as contingent on their use by individuals with different human capital.

Artificial Intelligence and Implications for Productivity and Workplace Practices

Artificial intelligence (AI) is defined as “the capability of a machine to imitate intelligent behavior”

(Merriam-Webster, 2017). In its current state, AI builds on machine learning algorithms that are trained to

improve their learning over time autonomously, through programs which utilize additional observational data

and information from real-world interactions (Faggella, 2017). From a socio-economic perspective, AI is

bringing back to fore age-old issues regarding implications of technological breakthroughs for productivity

and management practices, as well as changing human capital and skills composition for individuals at work

(Brynjolffson and McAfee 2014, Autor 2015, Bessen 2016 Hounshell, 1985; Mokyr, 1990). Within this debate,

6
some scholars have provided reasons why, at least in the foreseeable future, AI’s substitution for humans at

work is overstated (Agrawal et al., 2018; Autor, 2014; Jones et al., 2017). These reasons rely on both theories

regarding whether AI is capital or labor augmenting, and on the comparative advantage of humans vs.

machines in different types of tasks. For example, Autor (2014) builds on codified vs. tacit knowledge to

highlight complementarities between machines’ comparative advantage in routine, codifiable tasks, and

humans’ comparative advantage on tasks requiring tacit knowledge, flexibility, judgment and creativity.

Two limitations arise in the current discourse on AI, where theoretical models make different

assumptions about machines and humans in the underlying production function, task structure, and even

rates at which past investments by “dead high-tech workers” can replace effort by current period high-tech

workers (Aghion et al., 2017; Autor, 2015; Benzell et al., 2015). One, notwithstanding a budding set of

empirical papers (Acemoglu and Restrepo, 2017; Cowgil, 2017; Graetz & Michaels, 2015), there is a dearth of

empirical evidence on the effects of machine learning on productivity, relative to current alternative

technologies. Two, the models ignore the literature on technology diffusion—which we turn to next—that

notes new technologies may take prolonged periods of time for large scale adoption even in the presence of

newer regimes (Adner and Kapoor, 2016; Chari and Hopenhayn, 1991; Mansfield, 1968).

Vintages of Technology and Human Capital Heterogeneity

Endogenous models of technological change highlight a new technology’s diffusion relative to

existing vintages depends not only to its “objective” superiority, but also on the technology’s interaction with

complementary resources and capabilities. In particular, emergence challenges to new technologies relate to

bottlenecks posed in existing ecosystems in the form of complementary products, resources and capabilities

(Adner and Kapoor, 2016), and extension opportunities for old technologies are enabled by demand

heterogeneity and user preferences for the old technology (Adner and Snow, 2010). Similarly, models of

endogenous technological change provide mechanisms underpinning prolonged periods of technology

diffusion, relating the time lag to its relative superiority to existing technology vintages and the distribution of

human capital familiar with each technology vintage (Chari and Hopenhayn, 1991; Jovanovic, 2009).

Empirically, scholars examining technology diffusion have implicated differences in individuals and

7
workplace practices as critical to the technology’s acceptance and productivity (Agarwal and Prasad, 1999;

Black and Lynch, 2001; Greenwood, Agarwal, Agarwal and Gopal, 2018).

A salient difference of diffusion models from models in the preceding section where AI and machine

learning may (exogenously) complement or substitute for human capital, is the focus on not only the vintage

of the technology (new vs. old), but also the vintage of the human capital interfacing with the technology.

Given each technology ushers with it different tasks, and additionally changes the composition of tasks

performed by machines or humans (Autor, et al., 2003), Chari and Hopenhayn (1991) define vintage human

capital as technology-specific skills which relate to both task-specific human capital (Gibbons and Waldman,

2004) accumulated through learning-by-doing, and the evolution of tasks across technology vintages.

A limitation of the literature on endogenous technology diffusion due to vintage-specific human

capital, however, is its abstraction away from within technology vintage heterogeneity in human capital.

Productivity of different vintages of technology may not only entail complementarities between higher vs.

lower levels of human capital and skills (Jovanovic, 2009), but also complementarities with heterogeneous

human capital that is qualitatively distinct because of domain expertise (rather than vertically differentiated),

given specialization and division of labor (Becker and Murphy, 1992; Schultz, 1961; Simon, 1991). Even

within a high skilled labor force, it is accordingly important to incorporate interactions of horizontal

differences in prior knowledge bases and successive technology vintages for differences in productivity.

Production Complementarities between Process Technology and Specialized Knowledge Bases

Incorporating the insight that current machine learning technologies are geared towards statistically

driven prediction, we focus on theorizing for contexts wherein process technologies are used to enhance

prediction, and depict our stylized framework in Figure 1. The “anatomy of a prediction task” from existing

data requires devising a search strategy, utilizing the process technology to identify the relevant and narrower

data that enhances prediction, and arriving at an action based on the information signals from the process

(Agarwal, Gans, and Goldfarb 2018). We posit the importance of two human capital attributes—domain-

specific expertise and vintage-specific human capital—for using prediction process technologies.

[Insert Figure 1 here]

8
 Domain-specific expertise is important to productivity, because of its role in determining the search

strategy for identifying the relevant data through use of a process technology, and for later interpretation of

these data or prediction signals to arrive at an action. Dasgupta and David (1994) highlight the importance of

“context which makes focused perception possible, understandable and productive” (p. 493). Large literatures

on expertise in cognitive psychology (Chase and Simon, 1973; Ericsson, Krampe, Tesch-Romer, 1993; Simon,

1991) and specialized human capital in strategy and economics (Becker, 1962; Harris and Helfat, 1997;

Castanias and Helfat, 1991) have shown productivity benefits from increased specialization. Such gain from

domain-specific expertise need not only accrue to those who develop knowledge through personal

experience. Scholars have also documented that non-expert or novices benefit through vicarious learning

from domain experts, through transfer mechanisms including demonstration, personal instruction and by

provision of expert services such as advice, and consultation (Coleman, Katz and Menzel, 1957; Dasgupta

and David, 1994; Greenwood et al., 2017; Thornton and Thompson, 2001). Empirical research substantiates

the increases in productivity of a new technology when workplace practices include provision of technology

and task specific information (Bapna et al., 2013; Bartel, 1994; Black and Lynch, 2001). This is because the

codification and transmission of tacit know-how and know-what increases the speed at which followers learn

from pioneers’ learning by doing and trial and error processes with the new technology (Edmondson,

Winslow, Bohmer and Pisano, 2003). Consistent with this large literature and in the context of novice users

of prediction technology, we expect the provision of expert knowledge—which we define as transfer of

domain-specific expertise, from experts to novice workers—will increase the productivity of novice workers

when interfacing with either machine learning or older process technology.

Second, Figure 1 depicts vintage-specific human capital is important because it impacts how humans

interface with the technology. As noted in the literature review, the same tasks may need to be performed

differently across two different technologies, and further, the composition of tasks (and skills required) may

also be different across technologies (Chari and Hopenhayn, 1991). Accordingly, the potential of machine

learning technology’s productivity improvement can be realized contingent on individuals interfacing with the

technology having vintage-specific human capital. Here, we note our focus is on workers not as developers, but

users of machine learning technologies. We posit workers with computer science and engineering (CS&E)

9
knowledge bases will be more productive with machine learning technologies relative to those with non-

CS&E knowledge bases because the user-interface of machine learning technologies will be more familiar to

workers with prior CS&E knowledge bases, than for those with non CS&E knowledge. In their seminal

article on absorptive capacity, Cohen and Levinthal (1990) note an individual’s prior related knowledge is key

to their ability to absorb new knowledge. They remark, “students who … mastered the principles of algebra find it

easier to grasp advanced work in mathematics such as calculus; …Pascal (computer programming language) students learned

LISP (another programming language) much better because they better appreciated the semantics of various programming

concepts” (p. 130). Similarly, prior computer programming knowledge will provide CS&E workers a better

skill-set for interfacing with machine learning tools. Prior knowledge gained from working on computer

software tools that involve manipulating user interfaces should enhance their productivity when interacting

with the tool, relative to non-CS&E workers who may lack these skills and expertise.

To summarize the insights from Figure 1, productivity differentials between new (machine learning)

and the older vintage process technology hinges on two important human capital attributes of the individuals

using the process technology. Domain-specific expertise matters in devising effective prediction strategies and

interpreting results, and vintage-specific human capital matters for effective interfacing with the process

technology, conditional on having the same prediction strategy.

RESEARCH CONTEXT AND EXPERIMENTAL DESIGN

Our study examines productivity differentials across technology vintages based on their interface

with heterogeneous human capital within the context of the US Patent and Trademark Office examination

process. 1 The review of patent applications requires the use of prediction process technologies—the existing

(old) vintage relies on Boolean keyword search, while the newer vintage utilizes machine learning. The

context represents an ideal setting for our study, for reasons explicated below as we provide brief descriptions

1 Patents protect inventors’ intellectual property rights by “exclud[ing] others from making, using, offering for sale, or
selling the invention throughout the United States or importing the invention into the United States.”
(https://www.uspto.gov/learning-and-resources/electronic-data-products/issued-patents-patent-grants-data-products,
accessed July 2017). Anyone who wishes to secure the rights to an invention in the U.S. must apply for a patent from the
USPTO. Patent applications include an abstract, which describes the patent filing, and a list of specific claims, which
specify the precise components of the invention that the applicant wishes to patent.

10
of the patent examination process and each of these techniques below (For more detailed information, please

refer to Appendix B, Exhibit B1 and B2, GAO (2016) and the USPTO website at www.uspto.gov).

The Patent Examination Process

To meet the legal standards for a patent, claims must be both novel (i.e., a new, unique invention)

and non-obvious (i.e., sufficiently inventive). USPTO examiners review patent applications, compare it to

prior art (i.e., all existing patents, patent applications, and published materials), and determine whether an

inventor’s claims meet these two criteria. Upon receipt, an application is classified based on technical subject

matter to one or more of classifications among thousands of different technologies. This classification routes

the application to one of eleven technology centers which collectively cover electrical, chemical and

mechanical technology fields. The eleven technology centers are further subdivided into smaller “art units”

consisting of patent examiners who are highly skilled and clustered based on their domain expertise on related

technologies within the unit. Within each art unit, the assignment to patent examiner is based on examiner

workload rather than complexity of patent. 2

Upon an application being assigned, the patent examiner adjudicates on novelty and non-

obviousness of the application by comparing it to prior art. This process maps well to Figure 1. Specifically,

various databases—including the Patent Full-Text and Image Database (PatFT) and Patent Applications Full

Text (AppFT) provide the repository of prior patents and their claims. Prediction about novelty of claims

requires the examiner to devise a search strategy and use available technologies to identify a list of potentially

relevant prior art for the specific patent application (See Appendix B, Exhibit B3). The examiner then reviews

the results to arrive at a “signal” regarding the novelty of the claims. This signal is an important input,

combined with domain expertise, to arrive at a decision of whether or not to invalidate the claims made in the

application, and determine intellectual property scope if a patent is granted.

Need and Efforts for Productivity Improvements in the USPTO Patent Examination Process

In maintaining a balance between expediency and patent examination quality, the USPTO aims to

provide at least a first office action within 10 months of receipt (Brinkerhoff, 2014). However, increases in

2U.S. Patent and Trademark Office, “Prior Art Frequently Asked Questions,” July 28 2014,
https://www.uspto.gov/sites/default/files/patents/init_events/prior_art_faq20140728.pdf accessed November 2017

11
the number of patent applications received by the USPTO (e.g. an 18 % increase in the five year period

between 2010 and 2014)3 have created significant delays with over half a million backlogged applications, and

current processing of first office action at more than sixteen months.4 This is in spite of USPTO investing

significant resources to increase the number of examiners by 28% from 2010 to 2014 (Crouch, 2014;

Choudhury, Khanna and Mehta, 2017). The Government Accountability Office report on intellectual

property (GAO, 2016) notes these delays are compounded by recent increases in disputes over patent validity,

raising concerns “the USPTO examiners do not always identify the most relevant prior art, which has resulted

in granting some patents that may not meet the statutory requirements for patentable inventions” (p. 2).

The concerns regarding time and accuracy of patent examination can be traced to underlying

challenges in the various important and laborious steps in the process (GAO, 2016). For example, to identify

prior art, examiners often search beyond PatFT and AppFT in other databases to locate prior art stored

elsewhere (e.g., published articles, trade journals). The complexity arises in part because of the characteristics

of prior art and patent applications: patent examiners often deal with non-uniform language, intentionally

vague or broad patent claims, and missing relevant references, which increase prior art search costs.

Additional challenges relate to the process technologies utilized when conducting prior-art search.

The current mainstream technology for prior art search tools available to patent examiners (and accessible to

others in public versions) are named EAST (Examiners’ Automated Search Tool) and WEST (Web-based

Examiners Search Tool) and rely on Boolean logic.5 Patent examiners use Boolean operators to conduct

structured searches using keywords and strings with the available databases to yield potential prior art

citations. As Alcacer and Gittelman (2006) document, two-thirds of prior art citations on the average patent are

inserted by examiners to define appropriate scope. When evaluating the current search tools and capabilities,

GAO (2016) noted 48% of the patent examiners found these search tools made identification of prior art

3 U.S. Patent and Trademark Office, “U.S. Patent Statistics Chart Calendar Years 1963-2015,” June 15, 2016,
https://www.uspto.gov/web/offices/ac/ido/oeip/taf/us_stat.htm, accessed July 2017.
4 U.S. Patent and Trademark Office, “First Office Action Pendency,” June 2017,

https://www.uspto.gov/corda/dashboards/patents/main.dashxml?CTNAVID=1004, accessed July 2017 and U.S.

Patent and Trademark Office, “June 2017 Patents Data, At a Glance,”
https://www.uspto.gov/dashboards/patents/main.dashxml, accessed July 2017.
5 U.S. Patent and Trademark Office, “Prior Art Frequently Asked Questions,” July 28 2014,

https://www.uspto.gov/sites/default/files/patents/init_events/prior_art_faq20140728.pdf accessed November 2017

12
easier, but 17% found them to make it more difficult. More than half of the patent examiners thought the

tools available to them in EAST/WEST as “less advanced and more reliant on keyword searching than other

available tools” (p 24). The report stated the desire expressed by the majority for search engines for increased

automation: both in search for concepts and synonyms related to keywords entered by the examiners, and in

identification of prior art based on information already included in the patent application (claims and

specification) without any reliance on patent examiner entered keywords.

In 2016, the USPTO invested resources in developing a machine learning tool called Sigma (Krishna

et al., 2016). Sigma increases automation in the search for existing patents (see Appendix B Exhibit B4).

Examiners manipulate the search algorithm by selecting patent application components (e.g., the abstract,

title, or description) they believe are most relevant, and also enter the most important search terms into a

“word cloud.”6 Sigma, in turn, adjusts the algorithm accordingly and retrieves a more refined set of search

results. Over time, Sigma learns the examiner’s habits to suggest searches based on the examiner’s prior

behavior. Sigma is currently being tested and refined for adoption. Importantly, while the first and critical

stakeholder group are the patent examiners whose productivity depends on the search technologies, as with

the earlier technology vintages, the USPTO intends to make these tools available to the general public, to

benefit the population of potential inventors and the personnel they employ and facilitate their patent

application process. Accordingly, the larger goal when investing in machine learning technologies relates to

their ease of use for all relevant stakeholders.

Productivity of Process Technology Contingent on Domain and Vintage-specific Human Capital

Returning to Figure 1, it is useful to think through the differences in each search technology, and the

steps in the process where domain expertise and vintage-specific human capital is required of examiners who

use each technology. Within the context of prediction, domain expertise is important in determining the

content of the search. The appropriate content of search may need to expand beyond the information

included in a focal patent application, and domain expertise provides the requisite knowledge of what

additional information is critical to identify prior art that may be relevant to the claims in the focal patent.

Also, each process technology uses different user interfaces, and the search strategies need to be customized

6 Word clouds visualize text data through increases in size of search terms deemed more important than others.

13
to the process technology. As an example, Boolean search for potentially relevant prior art requires the use of

appropriate search strings and operands, while machine learning requires the manipulation of word clouds.

Accordingly, each process technology requires knowledge and skills specific to its vintage, and we expect

examiners who have prior CS&E knowledge and experience to be more adept at interfacing with machine

learning technology. Based on the search for identical content, the machine learning and Boolean search

technologies produce different outputs. Unlike the machine learning process technology, which produces a

relevance-ranked list based on reading of the documents, the Boolean search tool yields an unranked list of

potential prior art. Also, as noted above, the machine learning technology uses natural language processing

algorithms to update itself based on examiner choices of what is relevant. As a result, well-performed

searches on the machine learning has the potential to result in a narrower distribution of relevant prior art.

In sum then, while machine learning technology has the potential to deliver on the need for

productivity improvements, the realization of that potential will hinge on its use by patent examiners (and

other stakeholders) who represent a diversity of domain expertise and prior experience and skills.

Accordingly, the interaction between their human capital and the process technology will determine the

extent of potential productivity enhancements of new technologies, relative to the existing options.

Description of Experiment

The ideal design to examine productivity differentials of process technology based on interactions

with heterogeneous human capital is one in which (1) subjects do not self-select into the experiment, (2)

technology vintage type is randomly assigned across individuals with and without CS&E knowledge, and

domain expertise is randomly manipulated (3) there are ways to assess potential contamination between

treatment and control group and (4) it is possible to not only causally identify the relationships of interest, but

also shed light on the underlying mechanisms. We briefly describe the experiment before describing how our

research design allows us to address each of these concerns (See Appendix B, Experimental Design Section).

The participants in this experiment were 221 graduate students enrolled in three sections of a course

at a top tier business school, who were asked to “examine” a patent with five claims over a period of five

days. We chose experimental subjects, i.e. MBA students, rather than patent examiners for two reasons. The

use of patent examiners would translate into an inability to control for unobservable differences across

14
subjects in human capital (domain and vintage-specific), which could bias results and create concerns

regarding our cross-technology comparisons. The use of MBA students mitigates this concern; all subjects

had similar (lack of) experience in patent examination processes and search technologies used within them, so

they better simulate new workers (with differences in prior CS&E knowledge bases) being assigned to

different technologies. Secondly, as noted above, the USPTO intends both technologies to be available for

general public use; the heterogeneity in prior backgrounds of MBA students may be more representative of

the skilled labor force at large and thus informative of productivity differences for larger stakeholder groups.

The research team simulated conditions encountered by patent examiners in designing the patent

examination exercise with invaluable help from USPTO officials. First, we determined the time period to be

allotted for examination of the patent claims, and also ensured task manageability given expectations of time

investments by the subjects participating in the experiment. In consultation with the USPTO officials, we

determined 0.6 hours per claim to be reasonable; the additional 33% of time allotment per claim provided a

cushion given the newness of the task to the participants.7

Second, USPTO officials helped identify a past patent application of average complexity for use in

the experiment. The specific patent application had been examined in 2010, and was rejected based on a single

prior art citation that invalidated its novelty claims (Appendix B, Exhibits B5 and B6). Given the applicant

had abandoned the process after the claims were rejected, no record of the patent examination results were

available to our experimental participants (e.g. as could be identified through a Google search). To pare down

the time investments to manageable levels, the USPTO officials identified 5 claims from the 39 claims made

in the original application which were deemed most relevant for the outcome of the examination.

Third, to enable random provision of “domain expertise,” an experienced USPTO examiner who

worked in the art-unit where the patent was originally examined in 2010 created the content of the emails

which were shared as expert knowledge. This expert advice included keywords to be used to search for

relevant prior art. Analysis of the emails from the expert (that codifies domain-specific expertise, Appendix B,

Exhibits B8 and B9) reveals that the expert examiner highlighted several words from the text of the patent

7As Lemley and Sampat (2012) document, while the average patent prosecution duration at the USPTO is three to four
years, the examiner spends on average 18 hours examining the claims. Also, Alison and Lemley (2000) document an
average patent has around 14.9 claims, indicating a USPTO examiner takes around 0.4 hours to examine a single claim.

15
being examined (e.g. “data”, “program”, “watch”, “watching”, etc.). Most importantly, the expert examiner

suggested keywords not present in the text of the patent being examined be added to the search (e.g. “viewer”,

“monitor”, “recording”, “tuning”, etc.). Our field interviews indicate that patent lawyers drafting patent text,

often using “strategic language” to frame claims with the intent of securing “as broad a claim as possible;”

additionally patent law allows patent assignees to “be their own lexicographers.”8 To overcome these

constraints, expert examiners often consider keywords not present in the text of the patent being examined

while searching for prior art. The expert advice was additionally tailored to the process technology, and

ensured the following the steps outlined in the advice resulted in the best performing search strategy for each

technology. Thus, the expert advice was both task and technology specific, and recommended explicit search

strategies on manipulating a word cloud to insert examiner added keywords (for the machine learning group)

and composing the search string with the examiner added keywords (for the Boolean search group).

Fourth, post-search adjudicating of claims was simplified such that identification of the “silver

bullet”—one prior patent which would invalidate all claims in the focal patent application—would result in

the correct decision, i.e. that the application be rejected.9 Accordingly, the expert advice provided for search

was sufficient for this later step of arriving at a decision. Finally, the business school’s information technology

department worked with USPTO officials to ensure smooth functioning of both technologies—Boolean and

Machine Learning, and capture the work-logs of participants as they interfaced with the technology.

The actual experiment proceeded over two weeks, as follows. In Week 1, participants were trained

on details of the patent examination task by officials from USPTO, and provided information similar to new

patent examiner hires. Here, they were introduced to both search technologies, but at this point, participants

were unaware of which technology would be assigned to them. At the end of Week 1, participants were

randomly assigned to a technology—Boolean or Machine learning (see details in Appendix B) using a random

8 See: https://www.ipmvs.com/news/the-patentee-is-free-to-act-as-his-own-lexicographer
9 Our field interviews reveal that patent examination often entails the identification of a “silver-bullet” to invalidate
claims being examined. Expert examiners often identified the silver-bullet through a process described to us as
“rummaging”, i.e. using different relevant keywords to identify relevant prior art. Once the examiner was able to identify
a silver-bullet, he or she often searched for other prior art references that might invalidate the claims. This was done by
searching for prior art in the relevant technology sub-class. The examiner was assured of the existence of a silver-bullet
(this is a term used by patent examiners in the field interviews) if all relevant search pathways led to a single patent (i.e.
the silver-bullet) that could invalidate the claims being examined.

16
number generator. Each participant received additional online documentation and general training materials

to ensure familiarity with the technology and task and had five days to complete the exercise. The participants

then initiated the exercise, searching for prior patent references. Half of the participants (randomly chosen

again) on each technology received the email containing expert advice from the USPTO patent examiner in

the midst of this exercise. Correctly inputting these strategies resulted in a list of the prior art potentially

relevant to the specific patent application; however, it was not sufficient for task completion. The subject had

to read through the output, determine which prior art references were relevant, and use this information

when adjudicating the claims. The adjudication element was simplified: the signal—identification of the

“silver bullet” prior art patent—would lead to rejection. At the end of the exercise, we assessed performance,

measured by both search accuracy (reference of or similarity to the “silver bullet” citation) and speed.

As it relates to the four criteria for an “ideal experiment,” first, participants did not self-select into this

experiment. All participants were students enrolled in a semester-long, second-year elective MBA course, and

they were unaware at time of enrollment of their potential participation in the experiment. While

participation in the experiment was voluntary, two of the three sections of the course had a 100%

participation rate for students physically on campus that week10, and the third had a participation rate of 81%.

In addition, participants signed consent forms prior to knowing the technology they were assigned, alleviating

the concern that participation in the experiment might be endogenous to being assigned to a technology.

Second, to prevent across group contamination, participants were asked to sign an academic declaration of

honesty, stating they would work on the exercise on their own and would not collaborate with anyone during

the exercise. While such contamination would only attenuate our results, this alleviates some concern that

individuals in one technology group accessed the search tool assigned to the other group. Moreover, access to

the electronic activity logs enables us to test how many of the individuals in the group that did not receive

expert advice ended up using the advice as suggested by the e-mail: we find that this occurred in only three

cases, and the results are robust to these cases being excluded from the analysis. Finally, the activity logs for

10For these two sections, when counting all students including those not physically on campus, the participation rate was
97% and 94% respectively. The students who were not on campus were traveling out of town for job interviews.

17
each participant also help shed light on the underlying mechanisms driving the causal relations of interest,

including whether or not participants integrated the expert advice into their search activities.

Table 1 presents descriptive statistics for the sample. Overall, the average participant spent 22.6

minutes on the tool; the time spent ranged from a minimum of 0 minutes, to a maximum of 93 minutes.11 It

is important to note that the minutes spent on the tool only include when the examiner was logged in.

Examiners were logged out automatically after 90 seconds of inactivity on the tool, and had to re-enter their

search queries when they logged back in.

[Insert Table 1 here]

Variable Definitions

Dependent Variables: Success in the task is based on whether the patent application being reviewed was rejected

for all five claims due to participants identifying the silver bullet. To measure an examiner's success, we count

the number of times they cited the silver bullet in their evaluation of each claim. To measure an examiner's

productivity, we compute the ratio of silver bullet citations to time spent on the platform. In mechanisms tests,

we also examine dependent variables that reflect Levenshtein similarity scores (Thoma et al., 2010) between

the patents cited by the examiners a) the focal patent, and b) the silver bullet patent. In robustness checks, we

use as a dependent variable an indicator for citing the silver bullet patent at all.

Independent Variables: We manipulated two variables in this study: First, 𝑀𝐿𝑖 is a dummy variable set to 1 if the

examiner 𝑖 was assigned to use the machine-learning based process technology, Sigma, (and 0 otherwise).

Second, 𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 , is a dummy variable set to 1 if the examiner 𝑖 received the expert advice email (and

0 otherwise). The prior knowledge base of examiners is captured by the indicator variable 𝐶𝑆&𝐸𝑖 , set to 1 if

examiner 𝑖 has a degree in computer science and engineering (and 0 otherwise). Based on the framework

outlined in Figure 1, the variable 𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 encapsulates the construct of domain expertise; on the

other hand, the variable 𝐶𝑆&𝐸𝑖 reflects vintage-specific human capital for machine learning technology.

1123 participants spent no time working on the tool, and they seem to be randomly distributed across groups. Also,
neither technology, nor CS&E background, nor expert advice are predictive of the time spent on the time tool.

18
Controls: Although randomization ensures unbiased estimates, we include several pre-treatment control

variables in our model to reduce residual variation and increase the precision of treatment effect estimates.

Controls include indicators for section, gender, whether the individual has a partner, and is a US citizen.

Balance Tests and Empirical Approach

As noted above, each examiner was randomly assigned to one of four treatment groups (Boolean vs.

Machine Learning; Received Expert Advice vs. not), stratified by examiner background (CS&E skills vs. not)

for each class section. The sampling method comprised three steps. First, we sorted student IDs by section

and CS&E background. Second, we matched subjects in the four treatment groups to the six strata (3

sections* CS&E background dummy). This ensured each of the six strata have an approximately equal

number of subjects assigned to each treatment group. In the final step, within each stratum, we shuffled the

treatment assignments using a random number generator. We verified our randomized treatment assignment

generated roughly equal numbers of each treatment within each stratum. Table 2 presents means and

standard errors of various pre-treatment variables by treatment group. The stratification and matching seems

to have worked as designed, because the distribution of CS&E and section are evenly distributed between the

four treatment groups. Other pre-treatment variables are also balanced across the groups.

[Insert Table 2 here]

To examine the causal effects of machine learning and the random provision of expert advice, we

estimate the following set of equations using OLS:12

𝑌𝑖 = 𝛾0 + 𝛾1 𝑀𝐿𝑖 + 𝐵𝑿 + 𝜀𝑖 (1)

𝑌𝑖 = 𝛼0 + 𝛼1 𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 + 𝐵𝑿 + 𝜀𝑖 (2)

𝑌𝑖 = 𝛽0 + 𝛽1 𝑀𝐿𝑖 + 𝛽2 𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 + 𝛽3 𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 ∗ 𝑀𝐿𝑖 + 𝐵𝑿 + 𝜀𝑖 (3)

where 𝑿 includes the controls noted above and 𝑌𝑖 is either the number of times the silver bullet is referenced,

the examiner’s productivity (silver bullet citations per minute), or the similarity between the patents cited and

the silver bullet or focal patent applications. Equation (1) estimates the causal effect of machine learning

12 We use OLS as opposed to count models such as Poisson because many of the observations are perfectly predicted, as
is readily apparent in Figures 3 and 4. We also perform robustness checks using linear probability models and find results
that are quantitatively consistent, both in terms of magnitude and statistical significance.

19
relative to the Boolean technology, while equation (2) examines the causal effect of the provision of expert

advice, and equation (3) examines the interaction between expert advice and machine learning.

RESULTS

Table 3 reports the results from estimating equations (1)-(3). Columns (1) and (2) of Panel A show

that machine learning is associated with 0.12 fewer citations of the silver bullet. Machine learning is not

necessarily less efficient in terms of time (columns (7) and (8) of Panel B), indicating the gains in finding the

relevant prior art on the Boolean technology (from Panel A) came at the cost of more time. Columns (3) and

(4) of Panel A show that, as expected, expert advice causes the silver bullet to be cited 0.21 more times,

increasing productivity by 0.008 (column (10) of Panel B), an increase of nearly 200% relative to the sample

mean of 0.004 citations of the relevant prior art per minute. Considering the interaction of expert advice and

machine learning, the point estimates suggest that expert advice is strongly associated with finding the

relevant prior art on the Boolean technology (0.327 in column (5)), but is relatively less effective with machine

learning (-0.236 in column (5)), though the negative interaction is not robust to including controls.

[Insert Table 3 about here]

We next turn to the efficacy of the search process using each technology in terms of data output of

prior art potentially relevant to the focal application. We examine whether the use of machine learning based

process technology leads to more precise (i.e. narrower) distributions of relevant prior art, relative to Boolean

technology. Table 4 provides the total number of prior art patents, and validates this proposition: in the two

Boolean treatment groups, there were 204 (without expert advice) and 158 (with expert advice) unique

patents cited, versus 122 and 107 in the two machine learning groups respectively. The wider distributions in

the Boolean vs. machine learning groups are also depicted in Figure 2. Figure 2 provides their distribution

(number of times each prior art patent was cited) across all examiners in each treatment groups. Importantly,

the green bar in Figure 2 represents the “silver bullet” prior art citation, while the red bars represent patents

not necessary to invalidate the claims of the focal patent. Evident in Figure 2 is the critical role of expert

advice (i.e. domain-specific expertise): regardless of search technology used, the “silver bullet” was identified

only in the presence of expert advice.

20
 Moreover, of all the citations made by the Boolean group, 75% came from 92 unique patents in the

sample without expert advice, and 64 unique patents in the sample ith expert advice. In contrast, 75% of the

patents cited from the machine learning groups came from just 27 (no expert advice) and 33 unique patents

(with expert advice). We performed two-sample Kolmogorov-Smirnov tests to compare these distributions

and reject the null that the distributions are not different from each other (p<0.05). Similarly, chi-squared

tests to see if the same patents were cited in each treatment group also reject the null (p < 0.05). These

patterns validate that ML based technologies lead to a more precise (narrower) distributions, which in the

presence of expert advice, includes the silver bullet necessary for accurate predictions.

[Insert Table 4 and Figure 2 here]

In Table 5 and Figures 3-4, we address the extent to which productivity differentials between using

new (machine learning) and older (Boolean) process technology is contingent on domain-specific expertise

and vintage-specific human capital through a two-way interaction of technology and CS&E background, and

a triple interaction with expert advice. Both Table 5 and Figure 3 indicate the only combinations of

knowledge background and technologies able to identify the relevant prior citation were non-CS&E

examiners working with the Boolean technology, and CS&E examiners working with machine learning

technology. Surprisingly, we find that no CS&E examiners found the relevant prior art while using the

Boolean tool. The triple interaction results in Table 5 further suggest that the ability to find the relevant prior

art is entirely contingent on receiving expert advice. As shown in Figure 4, none of the examiners found the

relevant prior art if they did not receive expert advice. Moreover, within the sample that received expert

advice, we find that the only combinations yielding relevant prior art are non-CS&E examiners working with

Boolean technology, and CS&E examiners working with the machine learning technology (and the difference

between these two groups is not statistically significant). Given that CS&E examiners are only 22% of the

sample, the negative main effect of machine learning appears to be a consequence of the weighted average

across two unequal groups. In other words, because non-CS&E examiners are much better on Boolean than

machine learning, and because non-CS&E examiners make up 78% of the sample, we find that on average

machine learning underperforms Boolean.

[Insert Table 5, and Figures 3-4 here]

21
Mechanisms and Robustness Checks

In this section, we perform two additional analyses to probe the mechanisms underlying the results

documented above. The first analysis examines how the machine learning technology, in conjunction with the

expert advice, changes where (in the distribution of prior art) examiners search for prior art. We previously

documented that those using the machine learning technology cite a narrower set of patents, but what is

unclear is whether this narrower set of patents are actually similar to the silver bullet. The second analysis uses

information in the examiner’s search activity logs to try to better understand why expert advice has

heterogeneous productivity effects based on the process technology and CS&E background.

Our first analysis sheds light on how expert advice and machine learning change where examiners are

searching for prior art. To perform this analysis, we calculate the Levenshtein distance between each patent

cited by each examiner and both the silver bullet patent and the focal patent application. This similarity

measure is bounded between 0 and 1, with 1 being the most similar and 0 being the least similar.13 Table 6

repeats our baseline specifications from Table 3 with these similarity scores as the dependent variables, where

Panel A examines similarity between the cited patens and the silver bullet and Panel B examines similarity

between the cited patents and the focal patent application. Figure 5 summarizes the results from Table 6,

where we plot the average similarity between the cited patents and the focal patent application on the x-axis,

and the similarity between the cited patents and the silver bullet on the y-axis. The results of panel A, Table 6

are similar to those of Table 3: while the main effect (a weighted average) reveals examiners using machine

learning tend to cite patents that are slightly less similar to the silver bullet (columns 1-2), the interaction

effects show that what matters is expert advice. Expert advice increases the extent to which respondents cite

patents similar to the silver bullet (columns 3-4), and columns 5-6 indicate that, as before, expert advice has a

similarly-sized positive effect on both Boolean and machine learning technologies.

Panel B of Table 3 exhibits some interesting new patterns: examiners using the machine learning

technology are more likely to cite patents similar to the focal patent application (columns 7-8). This implies

the machine learning technology, utilizing natural language processing, ‘reads’ the text of the focal patent

better than Boolean, to generate predictions based on what prior art may be most relevant to focal patent’s

13 Since examiners cite multiple patents, we use the within-examiner average similarity across all cited patents.

22
content. However, the cues from the focal patent do not provide machine learning process what additional

keywords are necessary to identify the silver bullet; such knowledge is often tacit and is only possessed by

human experts. Expert advice reduces the similarity between cited patents and the focal patent (columns 9-

10), and it has a greater negative effect on this account for those on machine learning (columns 11-12). Taken

together, the analysis in Table 6 and Figure 5 show that machine learning enables examiners to identify

patents that are similar to the focal application, and expert advice enables examiners to identify patents similar

to the silver bullet. When both are present, examiners are able to identify patents with the highest joint

similarity to both the focal patent and silver bullet.

[Insert Table 6 and Figure 5 here]

We next turn an examination of examiner activity logs in search of an explanation for the

productivity differences among CS&E and non-CS&E examiners who interfaced with machine learning

technology and received expert advice. In that subsample, there were three particular expert tips that needed

to be followed to have a high probability of finding the prior art. The first two tips were relatively

straightforward to integrate into the user interface: they included setting the weights and checking the

“Claims” box. We call these the simple expert tips. The third tip, which is somewhat more complicated,

involved manipulating the word cloud within the user interface—we call this the complex tip. Panel B and C

of Figure 6 examine how well CS&E and non-CS&E examiners are able to incorporate the simple and

complex tips into their searches per minute on the tool. Panel B of Figure 5 presents a binned scatter plot of

the number of times the search on the machine learning tool included the simple tips per minute spent on the

tool. Visually, the slopes are quite similar: Columns (7)-(8) of Table 7 performs the empirical analysis and

confirms that there is no statistical discrepancy between the slopes. However, Panel C of Figure 6 shows

clearly that the non-CS&E examiners have much more difficulty incorporating the complex tip of

manipulating the word cloud relative to the CS&E examiners. Columns (11) and (12) of Table 6 confirm that

the visual differences in the slopes between the CS&E and non-CS&E examiners is statistically significant.

This analysis suggests one possible explanation for why the non-CS&E do worse on the machine

learning technology, even with expert advice: They do not properly implement the expert advice into the

machine learning technology, which subsequently reduces the likelihood that the silver bullet will surface in

23
their search. Indeed, those who found the silver bullet in the CS&E population using the machine learning

technology averaged 11.5 expert searches. In contrast, the non-CS&E examiners completed on average only

0.62 expert searches, with a maximum of just seven. Table 8, columns (7), (8), and (11) and (12) confirm the

heterogeneity between CS&E and expert searches matters for finding the silver bullet: columns (11) and (12)

show that only CS&E examiners are able to translate expert searches into finding the silver bullet. Table A3

and A4 confirm the results are robust to examining simply finding the prior art and to productivity.

Our next examination looks at the Boolean search tool, to see how effectively the examiners were

able to manipulate the user interface to implement the search tips recommended in the expert advice. On the

Boolean technology, this amounts to a simple copy and paste of the exact expression of the e-mail. Panel A

of Figure 6 shows a binned scatter plot of the number of these “expert searches” performed based on the

total number of minutes spent on the tool, broken out into CS&E and non-CSE&E categories.14 The figure

shows that the lines are essentially parallel, indicating that per-minute spent on the tool, CS&E and non-

CS&E were equally effective in translating the expert advice into searches. Columns (1)-(4) of Table 7 report

the tests of this comparison, confirming that there is no statistically significant difference in the slopes.

Despite being able to integrate the expert advice on the Boolean technology at the same rates, we see

in columns (3)-(4) of Table 8 that non-CS&E examiners are much more effective in turning those searches

into finding the relevant prior art (Table A4 confirms that these results hold when we examine productivity

differences as compared to just finding the silver bullet). We find this result somewhat puzzling, given that we

would expect CS&E examiners to have equally good or better Boolean search skills than non-CS&E

examiners. We posit two potential theories for why we observe this distinction, though we are unable to

further test them given data limitations: First, it may be the case that CS&E examiners, who were aware of the

alternative machine learning technology, were discouraged that they were randomly assigned the Boolean

search tool, and thus put forth less effort. A second possibility is that CS&E examiners are less effective at

parsing through large sums of reported text.

14Note that the respondents were automatically logged out after 90 seconds and when they logged back in they had to
repeat their search. Minutes in the tool is the total amount of minutes the respondent spent logged in.

24
 Whatever the explanation may be for why CS&E are worse on Boolean when they receive expert

advice, one potential concern is that these results are driven by some spurious outcomes. We can provide

some evidence that this is not the case by showing that the same patterns related to technology and CS&E

background are observed in both the sample of expert advice and in the sample that did not receive expert

advice. Table 9 uses as a dependent variable the similarity measure between the cited patents and the silver

bullet,15 and examines heterogeneous effects by technology and CS&E background, breaking out the effects

separately by the receipt of expert advice. Although the magnitudes of the effects are different across the two

samples, the directionality and statistical significance remains the same: Relative to a non-CS&E examiner on

Boolean, CS&E examiners on Boolean cite patents that are less similar to the silver bullet, both in the sample

with expert advice and in the sample without it. Similarly, non-CS&E examiners on machine learning cite less

similar patents to the silver bullet than non-CS&E examiners on Boolean, in both the sample with and

without expert advice, while CS&E on machine learning cite relatively more similar patents to the silver bullet

(again, in both subsamples).

[Insert Table 7, 8, 9, and Figure 6 here]

Given our experimental design randomizes technology vintage and expert knowledge, we believe the

results to be robust to many alternative explanations and model specifications. Nonetheless, one potential

concern is with the count measure as the dependent variable. We re-ran all our results with an indicator for

whether an individual found the relevant prior art citation at all, and the results, presented in the Appendix A,

are qualitatively and quantitatively very similar (see Tables A1-A4 in Appendix A). Another potential concern

is that voluntary participation in the study created a nonrandom selection into the study, thus creating

generalizability issues due to sample selection. As noted above, two of the three sections had a 100%

participation rate, while the other had an 81% rate; thus we do not expect nonrandom selection into the

sample to bias our results.

DISCUSSION AND CONCLUSION

15We cannot use the number of silver bullets as a dependent variable for this analysis since no examiner found the silver
bullet without expert advice, resulting in zero variation to explain in the outcome variable for this subsample.

25
 Our study is motivated by technological advances in machine learning—the current state of

development in AI—and the quest to examine production complementarities between machine learning

process technologies, domain-specific knowledge and vintage-specific human capital. These issues are

theoretically and practically important: complementarities between existing human capital and the process

technologies with which individuals interface affect not only the pace of technological substitution, but also

workplace practices and socio-economic outcomes for individuals at work.

Based on our theoretical framework outlined in Figure 1, we interpret our results as follows. Firstly,

as theorized, we find machine learning process technology generated a narrower signal (i.e. narrower

distribution) of prior art. However, we observe heterogeneous productivity effects based on the provision of

prior domain-specific knowledge and vintage-specific human capital. Prior domain-specific knowledge

(codified as expert advice provided in the email to help novice examiners frame the search strategy) is crucial

to identifying the relevant prior art. We also observe heterogeneous productivity effects based on

complementarity of process technology and prior vintage-specific human capital. As expected, we find

examiners with a background in CS&E were more productive while using the machine learning process

technology; we ascribe this effect to their superior ability (relative to non-CS&E examiners) in manipulating

the user interface of the machine learning tool to incorporate the relatively complex tips provided in the email

outlining the expert search strategy. We also find, compared to the CS&E examiners, non-CS&E examiners

were relatively more productive while using the Boolean tool. While counterintuitive and surprising, we

provide evidence that CS&E examiners were less able to utilize the expert advice to identify the silver bullet

while working on Boolean technology, compared to their non-CS&E counterparts. Collectively, we interpret

our findings as evidence of complementarities between process technology and two dimensions of human

capital: domain-specific expertise and vintage-specific human capital.

Of course, the generalizability of our study is subject to important limitations, some of which require

additional research to address. First, our focus in this study is on the window where workers first interface

with technologies, rather than after a prolonged association. This boundary condition is both a feature and a

limitation. Initial conditions are important to study, as they have path dependent outcomes for the future,

both in terms of adoption decisions, and the strategies which will enable better diffusion of adopted

26
technologies. However, it is a limitation inasmuch as we are unable to examine performance improvements

over longer periods of time. While the provision of expert advice and vintage-specific human capital increases

the initial productivity, what is unclear is whether constant exposure and learning-by-doing by workers would

cause the relative differences between the groups to grow or shrink over time.

Second, our reliance on an experimental design and choice of MBA students as subjects was

motivated by the need for causal inferences in a sample of highly skilled but heterogeneously specialized labor

force of “novice users” of process technology. Further, as noted above, this choice of experimental subjects

permitted us to control for prior familiarity with both task and technology, so we could isolate the effects of

prior knowledge domains, and the provision of domain expertise to randomly distributed subjects. However,

as is true for all laboratory experiments, generalizability of the results are limited by the abstractions from

reality, and applications to other relevant sub-populations of the labor force. While we expect our sample

population of MBA students at a top tier business school to be similar to the target sector of highly skilled

individuals interested in intellectual property rights, the results of our experiments need to be replicated in

similar contexts for confirmation. Also, we deliberately abstracted away from vertical differentiation (high vs.

low skilled labor), but widespread use of new technologies may well require an inquiry expanding to this

dimension as well.

Finally, our research context and technology vintages are very specific—applicable to USPTO’s

development of the machine learning tool Sigma relative to Boolean search. Machine learning technologies

are themselves heterogeneous, and their development for tasks may render user interfaces more or less

complex, which itself has implications for complementarities with human capital. Accordingly, while our

results should be interpreted with caution, we urge scholars to add to the budding empirical research

examining productivity of all machine learning technologies, and their contingencies.

Limitations notwithstanding, our study contributes to several literature streams. Recent theoretical

models and empirical findings related to artificial intelligence and machine learning have stressed substitution

effects of technologies on human capital, particularly when related to prediction (Agrawal et al. 2018; Autor

2015; Benzell et al. 2015; Jones et al., 2017). Human capital in this literature stream is largely incorporated

either in the form of vertical differentiation (high vs. low skilled workers), or because of comparative

27
advantage in complementary tasks related to judgment, new discoveries, or tacit knowledge (Aghion et al.,

2017; Agrawal et al., 2018; Autor, 2015). We extend this literature in two ways—first, we integrate insights

from the related literatures on technology diffusion (Chari and Hopenhayn, 1991; Jovanovic, 2009) and pace

of technology substitution (Adner and Kapoor, 2016; Adner and Snow, 2010; Christensen, 1997) to explicitly

compare the productivity of machine learning technologies with earlier vintages. Second, and contributing to

all three literature streams above, we highlight that productivity differentials between new (machine learning)

and older vintage process technology hinges on two important human capital attributes of individuals using

the process technology: vintage-specific human capital, and domain-specific expertise.

The strategic management literature on strategic renewal within the context of disruptive

technologies acknowledges both imperatives and challenges of change (Agarwal and Helfat, 2009;

Christensen, 1997; Tushman and Anderson, 1990). Moreover, scholars have noted the rate at which new

technologies substitute old technologies relate to complementarities with existing resources and capabilities,

in addition to endogenous strategies for extending the relevance of older vintages (Adner and Snow, 2010;

Agarwal and Prasad, 1999; Bloom et al., 2012; Bapna et al., 2013; Hatch and Dyer 2004; Henderson 2006).

Inasmuch as specialized knowledge domains embodied in heterogeneous human capital will continue to

retain value for judgment, and given that humans will be relevant at work in the foreseeable future, our study

highlights the importance of incorporating these contingencies in theories and empirical work examining

productivity of new (machine learning) technologies, rather than conceptualizing main effects alone. Doing so

enables the identification of real productivity costs incurred because of within-vintage differences in human

capital caused by differential rates of absorptive capacity (Cohen and Levinthal, 1990).

Finally, our study also has implications for related literature streams on human capital and career

specialization (Becker and Murphy, 1992; Lazear, 2004; Rosen, 1983; Teodoridis, 2017; Zuckerman et al.,

2003). While this literature has focused on life cycle effects of human capital investments, and generally found

beneficial effects of developing specialized knowledge (Becker and Murphy, 1992; Castanias and Helfat, 1991;

Greenwood et al., 2017), but also that benefits of specialization are contingent on market or institutional

factors (Castanias and Helfat, 1991; Merluzzi and Phillips, 2016). We extend this literature by highlighting the

differential effects of specialized domains for absorptive capacity to interface with entirely new technologies.

28
New technological advances such as AI exhibit skill-bias not only in the vertical realm (i.e. high vs. low skill),

but also to the extent they privilege CS&E knowledge relative to other specialized domains. Accordingly,

benefits and costs of career specialization must also be evaluated in the context of what skills and knowledge

are considered foundational for complementarities with the latest technology vintages.

The study has several implications for practice. First, and immediately applicable to USPTO, our

study unearths human capital complementarities as critical to productivity of alternative technologies. One

insight may be a sequential rather than simultaneous roll-out of machine learning technology. While GAO

(2016) has highlighted the hurdles posed by existing search technologies for patent examiners’ productivity

given significant public concern about backlogs in the patent approval process, our study shows that these

may nonetheless retain value for examiners who have non-CS&E knowledge bases. Overall productivity,

both in the short and long run, may be enhanced through CS&E examiners’ immediate use of machine

learning, with the search strategy being composed with the help of expert knowledge. Their cumulative learning-by-doing

insights can then be transformed into a codified library of task and technology specific knowledge for a larger

scale diffusion among examiners and stakeholders for intellectual property rights in the general population.

Using these tools, potential innovators may be better able to assess the likely viability of a patent application.

Second, at a broader level, our study highlights the need for technology-skill specific training, rather

than technology-specific alone. Given that new technologies require re-composition of tasks relative to old

technologies, training programs often focus on the provision of technology -pecific skills. Our study calls for

a more nuanced approach—one where firms need to account for prior vintage-specific human capital in its

workforce while designing technology training programs. Third, the implications of our study extend beyond

training of the existing workforce to other human resource management practices, including recruitment,

retention, and innovation management, especially for firms operating in dynamic industries. For example, the

literature on recruitment outlines conditions for when firms should hire externally vs. internally, including

“acqui-hiring” (Chatterji and Patro, 2014) to gain simultaneous access to technologies and complementary

employees, or for positions with strong firm-specific or industry-specific components (Bidwell and Keller,

2014; Bidwell and Mollick, 2015; Williams, Chen and Agarwal, 2017). Our results indicate firms investing in

new technologies need to assess relative value of skills when making decisions on whether to hire externally,

29
or promote internal talent. Also, for employee retention, perceptions of fairness and inequality may be

exacerbated by underlying productivity differentials from working with different technologies. Thus, in

addition to factors such as organizational culture, employee voice and work practices (Sheridan 1992; Spencer

1986; Guthrie 2001), our study highlights job satisfaction and incentive alignment will also depend on the

attention firms implementing new technologies pay to whether employees have skills that lend to higher

productivity while using new process technologies, and appropriate alignment of compensation mechanisms.

Another practical implication relates to our study’s focus on workers not as developers of machine

learning technologies, but as users of the process technology. While the development and integration of

machine learning and artificial intelligence into commercially viable technologies and products is performed

by those who possess computer science and engineering (CS&E) knowledge, their enhanced productivity will

be contingent on by a widespread usage by heterogeneous workforce specializing in different knowledge

contexts, and these other knowledge bases retain their importance for other task components such as

decision making and judgment. For example, in the more immediate term, many of the AI based process

technologies will be used by lawyers, radiologists, accountants, supply chain managers, i.e. individuals without

a background in computer science. Accordingly, the results of our study suggest that user-interfaces of

machine learning based process technologies need to be simple to use and manipulate from the perspective of

such “non-CS&E” users. This may also explain why several AI developers have argued for voice recognition

based interfaces for their tools, compared to graphic user interfaces.16

Finally, and at the larger levels of innovation and education policy, our study highlights the

importance of computer programming training to increase familiarity and develop these as generalized skills

for the knowledge economy, even as subsequent human capital investments continue to focus on specialized

knowledge domains. Recently, Apple CEO, Tim Cook remarked to primary school students at a French

school that learning to code may be more important than learning English as a second language (Clifford,

2017). Elaborating on the comment further, Cook discussed the benefits of computer coding not just in

terms of higher salaries, but as a critical “backseat” complement to creativity in the “front seat,” because “the

blend with both of these, you can do such powerful things now.”

16 Source: https://www.motocms.com/blog/en/voice-recognition-technology/

30
References

Artificial Intelligence. Merriam-Webster.com. Accessed January 3, 2018, https://www.merriam-

webster.com/dictionary/artificial intelligence.
Acemoglu, Daron, and Pascual Restrepo (2017) Robots and Jobs: Evidence from US labor markets. NBER
Working Paper 23285.
Adner, Ron, and Daniel Snow (2010) Old technology responses to new technology threats: demand
heterogeneity and technology retreats. Industrial and Corporate Change 19(5): 1655-1675.
Adner, Ron, and Rahul Kapoor (2016) Innovation ecosystems and the pace of substitution: Re‐examining
technology S‐curves. Strategic Management Journal 37(4): 625-648.
Agarwal, R., & Prasad, J. (1997). The role of innovation characteristics and perceived voluntariness in the
acceptance of information technologies. Decision Sciences, 28(3): 557-582.
Agarwal, Rajshree, and Constance E. Helfat (2009) Strategic renewal of organizations. Organization
Science 20(2): 281-293.
Agarwal, Ritu, and Jayesh Prasad (1999) Are individual differences germane to the acceptance of new
information technologies? Decision Sciences 30(2): 361-391.
Aghion, Philippe, Antonin Bergeaud, Richard Blundell, and Rachel Griffith (2017) Innovation, Firms and
Wage Inequality.
Aghion, Philippe, Benjamin F. Jones, and Charles I. Jones (2017) Artificial Intelligence and Economic
Growth. National Bureau of Economic Research. W23928.
Agrawal, A., Gans, J., & Goldfarb, A. (2016) The simple economics of machine intelligence. Harvard Business
Review, 17.
Agrawal, A., J. Gans, and A. Goldfarb (2018) Prediction, Judgement, and Uncertainty. NBER Working Paper
24243.
Alcacer, Juan, and Michelle Gittelman (2006) Patent citations as a measure of knowledge flows: The influence
of examiner citations. The Review of Economics and Statistics 88(4):: 774-779.
Allison, John R., and Mark A. Lemley (2000) Who’s Patenting What-An Empirical Exploration of Patent
Prosecution. Vand. L. Rev. 53: 2099.
Anderson, Philip, and Michael L. Tushman (1990) Technological discontinuities and dominant designs: A
cyclical model of technological change. Administrative Science Quarterly: 604-633.
Argote, Linda, and Dennis N. Epple (1990) Learning curves in manufacturing. Carnegie Mellon University,
Graduate School of Industrial Administration.
Autor, David H. (2015) Why Are There Still So Many Jobs? The History and Future of Workplace
Automation. Journal of Economic Perspectives, 29(3): 3-30.
Autor, David H., Frank Levy, and Richard J. Murnane (2003) The skill content of recent technological
change: An empirical exploration. The Quarterly Journal of Economics 118(4): 1279-1333.
Autor, David (2014) Polanyi’s paradox and the shape of employment growth.: National Bureau of Economic
Research 20485.
Bapna, Ravi, Nishtha Langer, Amit Mehra, Ram Gopal, and Alok Gupta (2013) Human capital investments
and employee performance: an analysis of IT services industry. Management Science 59(3): 641-658.
Bartel, A. P. (1994) Productivity gains from the implementation of employee training programs. Industrial
relations: a journal of economy and society, 33(4), 411-425.
Becker, G. S., & Murphy, K. M. (1992) The division of labor, coordination costs, and knowledge. The
Quarterly Journal of Economics, 107(4): 1137-1160.
Bekman, Eli, John Bound, and Stephen Machin (1998) Implications of skill-biased technological change:
international evidence. The Quarterly Journal of Economics 113(4)): 1245-1279.
Benzell, Seth G., Laurence J. Kotlikoff, Guillermo LaGarda, and Jeffrey D. Sachs (2015) Robots are us: Some
economics of human replacement. National Bureau of Economic Research, w20941.
Bessen, James E. (2016) How computer automation affects occupations: Technology, jobs, and skills. Boston
Univ. School of Law: 15-49.
Bidwell, Matthew, and Ethan Mollick (2015) Shifts and ladders: Comparing the role of internal and external
mobility in managerial careers. Organization Science 26.6: 1629-1645.

31
Bidwell, Matthew, and J. R. Keller (2014) Within or without? How firms combine internal and external labor
markets to fill jobs. Academy of Management Journal 57.4: 1035-1055.
Black, S. E., & Lynch, L. M. (2001) How to compete: the impact of workplace practices and information
technology on productivity. The Review of Economics and Statistics, 83(3), 434-445.
Bloom, N., Sadun, R., & Van Reenen, J. (2012) Americans do IT better: US multinationals and the
productivity miracle. The American Economic Review, 102(1): 167-201.
Brinckerhoff, Courtenay C. (2014) How Long Is Too Long (Or Too Short) For Patent Application
Pendency? Accessed July, 2017, https://www.pharmapatentsblog.com/2014/07/18/how-long-is-too-
long-or-too-short-for-patent-application-pendency/.
Brynjolfsson, Erik, and Andrew McAfee (2014) The second machine age: Work, progress, and prosperity in a
time of brilliant technologies. WW Norton & Company, 2014.
Bughin, J., E. Hazan, S. Ramaswamy, M. Chui, T. Allas, P. Dahlström, N. Henke, and M. Trench (2017)
Artificial intelligence–the next digital frontier. McKinsey Glob Institute https://www. mckinsey.
de/files/170620_studie_ai. Pdf.
Castanias, Richard P., and Constance E. Helfat (1991) Managerial resources and rents. Journal of
Management 17(1): 155-171.
Chari, Varadarajan V., and Hugo Hopenhayn (1991) Vintage human capital, growth, and the diffusion of new
technology. Journal of Political Economy 99(6): 1142-1165.
Chase, W.G., Simon, H.A. 1973 Perception in Chess. Cognitive Psychology 4(1): 55-81.
Chase, William G., and Herbert A. Simon (1973) The mind's eye in chess. Visual Information Processing:
215-281.
Chatterji, Aaron, and Arun Patro (2014) Dynamic capabilities and managing human capital. The Academy of
Management Perspectives 28(4): 395-408.
Choudhury, P., Khanna, T., and Mehta, S. (2017) Future of Patent Examination at the USPTO Harvard
Business School Case Study. https://hbr.org/product/The-Future-of-Patent-Exam/an/617027-PDF-
ENG.
Christensen, Clayton (1997) Patterns in the evolution of product competition. European Management Journal
15(2): 117-127.
Clifford, Catherine. Apple CEO Tim Cook: Learn to code, it’s more important than English as a second
language. Accessed December, 2017, https://www.cnbc.com/2017/10/12/apple-ceo-tim-cook-learning-
to-code-is-so-important.html.
Cohen, Wesley M., and Daniel A. Levinthal (1990) Absorptive capacity: A new perspective on learning and
innovation. Administrative Science Quarterly, 128-152.
Coleman, J., Katz, E., Menzel, H. (1957) The Diffusion of an Innovation among Physicians. Sociometry. 20(4):
253-270.
Cowgill, Bo (2017) The Labor Market Effects of Hiring through Machine Learning Working Paper.
Crouch, Dennis (2014) USPTO's Swelling Examiner Rolls. Accessed July 2017,
https://patentlyo.com/patent/2014/11/usptos-swelling-examiner.html.
Edmondson, A. C., Winslow, A. B., Bohmer, R. M., & Pisano, G. P. (2003) Learning how and learning what:
Effects of tacit and codified knowledge on performance improvement following technology
adoption. Decision Sciences, 34(2): 197-224.
Ericsson, K.A., Krampe, R.T., Tesch-Römer, C. 1993. The Role of Deliberate Practice in the Acquisition of
Expert Performance. Psychological Review. 100(3) 363.
Faggella, Daniel (2017) What is Machine Learning Accessed November 2017,
https://www.techemergence.com/what-is-machine-learning/.
Fernandez, R. M. (2001) Skill-biased technological change and wage inequality: Evidence from a plant
retooling. American Journal of Sociology, 107(2), 273-320.
Government Accountability Office (2016) Intellectual Property: Patent Office Should Strengthen Search
Capabilities and Better Monitor Examiner’s Work. Report to the Chairman, Committee on the Judiciary, House of
Representatives.
Gibbons, R., & Waldman, M. (2004) Task-specific human capital. The American Economic Review, 94(2), 203-
207.

32
Goos, Maarten, and Alan Manning (2007) Lousy and lovely jobs: The rising polarization of work in
Britain. The Review of Economics and Statistics 89(1): 118-133.
Graetz, Georg, and Guy Michaels (2015) Robots at work. Center for Economic Performance, 2042-2695.
Greenwood, B., Agarwal R., Agarwal R., and Gopal A. (2017) The Role of Individual and Organizational
Expertise in the Adoption of New Practices.
Guthrie, James P. (2001) High-involvement work practices, turnover, and productivity: Evidence from New
Zealand. Academy of Management Journal 44.1): 180-190.
Hatch, Nile W., and Jeffrey H. Dyer (2004) Human capital and learning as a source of sustainable competitive
advantage. Strategic Management Journal 25(12): 1155-1178.
Henderson, Rebecca (2006) The innovator's dilemma as a problem of organizational competence. Journal of
Product Innovation Management 23(1): 5-11.
Hoffman, Mitchell, Lisa B. Kahn, and Danielle Li (2015) Discretion in hiring. National Bureau of Economic
Research, w21709.
Hounshell, David (1985) From the American system to mass production, 1800-1932: The development of
manufacturing technology in the United States. JHU Press, 4.
Jovanovic, Boyan (2009) The technology cycle and inequality. The Review of Economic Studies 76(2): 707-729.
Krishna, Arthi M., Brian Feldman, Joseph Wolf, Greg Gabel, Scott Beliveau, and Thomas Beach (2016) User
Interface for Customizing Patents Search: An Exploratory Study. In International Conference on
Human-Computer Interaction, 264-269.
Lazear, Edward P. (2004) Balanced skills and entrepreneurship. The American Economic Review 94(2): 208-211.
Lemley, Mark A., and Bhaven Sampat (2012) Examiner characteristics and patent office outcomes. Review of
Economics and Statistics, 94.3: 817-827.
MacCrory, F., Westerman, G., Alhammadi, Y., & Brynjolfsson, E. (2014). Racing with and against the
machine: changes in occupational skill composition in an era of rapid technological advance.
Mansfield, E. (1968) Industrial research and technological innovation; an econometric analysis. London:
Longmans.
Merluzzi, Jennifer, and Damon J. Phillips (2016) The specialist discount: Negative returns for MBAs with
focused profiles in investment banking. Administrative Science Quarterly 61(1): 87-124.
Mokyr, Joel (1990) Twenty five centuries of technological change: an historical survey. Taylor & Francis, 35.
Partha, Dasgupta, and Paul A. David (1994) Toward a new economics of science. Research Policy 23(5): 487-
521.
Rosen S. (1983) Specialization and human capital. Journal of Labor Economics 1(1): 43–49.
Schultz, T.W. (1961) Investment in Human Capital. The American Economic Review, 1-17.
Sheridan, John E. (1992) Organizational culture and employee retention. Academy of Management Journal, 35.5:
1036-1056.
Spencer, Daniel G. (1986) Employee voice and employee retention." Academy of Management Journal, 29.3: 488-
502.
Teodoridis, Florenta (2017) Understanding Team Knowledge Production: The Interrelated Roles of
Technology and Expertise. Management Science.
Thoma, G., Torrisi, S., Gambardella, A., Guellec, D., Hall, B. H., & Harhoff, D. (2010). Harmonizing and
combining large datasets-An application to firm-level patent and accounting data (No. w15851). National
Bureau of Economic Research.
Thornton, R. A., & Thompson, P. (2001) Learning from experience and learning from others: An exploration
of learning and spillovers in wartime shipbuilding. American Economic Review, 1350-1368.
Williams, C., Chen, P. L., & Agarwal, R. (2017) Rookies and seasoned recruits: How experience in different
levels, firms, and industries shapes strategic renewal in top management. Strategic Management
Journal, 38(7), 1391-1415.
Zeira, Joseph (1998) Workers, machines, and economic growth. The Quarterly Journal of Economics 113(4): 1091-
1117.
Zuckerman EW, Kim T, Ukanwa K, von Rittmann J. (2003) Robust identities or nonentities? Typecasting in
the feature‐film labor market. American Journal of Sociology, 108(5): 1018–1074.

33
 Figures
Figure 1. Stylized Framework of Process Technologies to Enhance Predictions

Note: This figure describes our theoretical framework. We outline two sets of complementarities between
process technology and human capital. (1) Domain-specific expertise is important in framing the search
strategy and interpreting the prediction signal (2) Vintage-specific human capital is important for the user
interface of the process technology.

34
Figure 2. Distribution of Patents Cited by Technology Type and Expert advice

35
 Figure 3. Number of Silver Bullet Citations by Technology and CS&E Background

Figure 4. Silver Bullet Citations, CS&E Background, and the Receipt of Expert advice

36
Figure 5. Mean Similarity to Focal Patent Application and Silver Bullet by Process Technology, Expert advice

37
Figure 6. Incorporating Expert Search Tips Per Minute, by CS&E Background

38
39
40
41
42
43
44
45
46