1

2
Characterizing Snapshot Testing Patterns: Emerging Results 59
60
3
4
Placeholder Author 1 Placeholder Author 2 61
62
5
University / Company 1 University / Company 2 63
6
City 1, Country 1 City 2, Country 2 64
7 65
8 Placeholder Author 3 Placeholder Author 4 66
9 University / Company 3 University / Company 67
10 City 3, Country 3 City 4, Country 4 68
11 69
12 ABSTRACT Snapshot testing has seen widespread adoption among major 70
13 Snapshot testing has emerged as an intriguing technique in soft- enterprises such as Microsoft, Meta, Amazon, and Google. It has 71
14 ware characterization testing, garnering attention from prominent rapidly expanded adoption in software development communities. 72
15 companies. Despite its wide adoption, there is little scientific anal- However, despite its growing popularity, few researchers have con- 73
16 ysis regarding it’s usage in software projects. This paper aims to ducted studies on this technique, and consequently, there are few 74
17 improve the developer experience in writing snapshot tests by papers specifically addressing this topic [5, 8]. We believe that a 75
18 analyzing practical implementations of this technique through a characterization study of snapshot test methods can enhance devel- 76
19 characterization study of test methods based on public repositories. oper experience raising awareness to new developers by classify- 77
20 The goal of this study is to provide emerging insights, identifying ing what to embrace, assess or avoid as they begin to incorporate 78
21 patterns and edge cases present in the current state of snapshot snapshot tests into their test suites. Thus, in this paper, we aim to 79
22 tests. Our research indicates among the projects that adopt snap- identify similarities, patterns, and edge cases in regard to the usage 80
23 shot testing with Jest, there are common characteristics and strong of snapshot testing in projects by manually classifying a sample of 81
24 patterns despite the diversity of tools. However, we also identi- snapshot tests. More specifically, we used a dataset [5] composed of 82
25 fied cases where the usage of snapshot test needs to be further snapshot tests from GitHub repositories employing the Jest frame- 83
26 addressed as some may end up as a test smell. Additionally, we work (a popular JavaScript testing framework). Subsequently, we 84
27 identify potential avenues for future research, such as examining selected a random sample of 380 snapshot tests for our manual 85
28 and analyzing project CI/CD pipelines. analysis. This sample, drawn from 108 different repositories, was 86
29 thoroughly read, reviewed, and analyzed. 87
30 CCS CONCEPTS Our analysis revealed that snapshot tests are typically designed 88
to encapsulate state logic for functions within different architectural
31 • Software and its engineering → Software testing and debug- 89
32 components inside applications. Additionally, we found that most 90
ging;
33 snapshot tests aim to verify an expected use case of the application, 91
34 KEYWORDS with the final snapshots most commonly stored in HTML, followed 92
35 by JSON formats. We categorized the tests into patterns such as 93
Snapshot Testing, Empirical Study, Patterns regular and inline snapshots, with regular snapshots predominating.
36 94
37
ACM Reference Format: Additionally, we investigated edge cases to discover interesting 95
Placeholder Author 1, Placeholder Author 2, Placeholder Author 3, and Place- tests we deemed noteworthy. Some edge cases, like long methods
38 96
holder Author 4. 2024. Characterizing Snapshot Testing Patterns: Emerging and non-linear logic, can be considered test smells. However, we
39 97
Results. In Proceedings of 18th ACM/IEEE International Symposium on Empir-
40 found cases implementing good programming practices like the 98
ical Software Engineering and Measurement (ESEM ’2024). ACM, New York,
Error Snapshot. Nevertheless, by presenting these cases we bring
41 NY, USA, 6 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn 99
42 awareness to their existence and encourage future discussions. 100
43 1 INTRODUCTION The structure of this paper is summarized as follows: Section 2 101
44 provides background information on snapshot testing and reviews 102
As applications become more complex and expansive, innovative
45 recent studies on the topic. Section 3 outlines the data collection 103
testing methodologies are developed to ensure thorough coverage
46 method. Section 4 details the sample characterization of our dataset. 104
and minimize regressions, thus preserving software integrity [6].
47 Section 5 shows emerging patterns of our analysis. Section 6 shows 105
Snapshot testing is an example of an emerging technique, designed
48 edge cases of our analysis. Section 7 addresses potential threats 106
to streamline the testing process and preventing regressions.
49 to validity. Related work is presented in Section 8, and Section 9 107
50
Permission to make digital or hard copies of all or part of this work for personal or concludes the paper by presenting our discussions and proposing 108
classroom use is granted without fee provided that copies are not made or distributed directions for future research.
51 for profit or commercial advantage and that copies bear this notice and the full citation 109
52 on the first page. Copyrights for components of this work owned by others than the 110
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or 2 SNAPSHOT TESTING
53 111
republish, to post on servers or to redistribute to lists, requires prior specific permission
54 and/or a fee. Request permissions from permissions@acm.org. Snapshot testing is a methodology that verifies the output of a 112
55 ESEM ’2024, October 20–25, 2024, Barcelona, Spain software unit (typically a function or component) by creating and 113
© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
56
ACM ISBN 978-1-4503-XXXX-X/18/06 saving a reference output, known as a snapshot when the test is first 114
57 https://doi.org/10.1145/nnnnnnn.nnnnnnn executed. In subsequent tests, the current output of the component 115
58 1 116
 ESEM ’2024, October 20–25, 2024, Barcelona, Spain Anonymous et al.

117 under test (CUT) is compared against this stored snapshot. The test To our knowledge, there are few scientific studies that have 175
118 passes if the outputs match; it fails if they differ, as shown in the exemplified and examined the practice of snapshot testing. How- 176
119 workflow specified by Figure 1. ever, one study [5] is particularly noteworthy. In their research, the 177
120 authors conducted an empirical analysis highlighting key character- 178
121 istics of adoption within open-source projects in regard to snapshot 179
122 testing use and compiled an initial dataset of projects that utilize 180
123 Jest. Another research on snapshot test [8] highlights that one of 181
124 the primary advantages of snapshot testing is its ability to capture 182
125 complex states efficiently, reducing the time required for writing 183
126 assertion logic while also pointing out potential drawbacks, such 184
127 as the risk of over-relying on snapshots, which can lead to false 185
128 positives if the snapshots are not carefully reviewed and updated. 186
129 A significant advancement in snapshot testing frameworks, par- 187
130 ticularly Jest, is the introduction of new auxiliary assertion types 188
131 like toThrowErrorMatchingSnapshot(). This assertion enables 189
132 developers to capture and compare error messages thrown during 190
133 tests, facilitating consistent error handling throughout the applica- 191
134 tion’s lifecycle [2]. As highlighted in the Jest documentation, this 192
135 feature simplifies the validation of error outputs, aiding in tracking 193
136 changes to error handling and expediting debugging. By allowing 194
137 snapshots of error messages, test suites enhance the versatility of 195
138 snapshot testing, as a vital component in modern testing. 196
139 In conclusion, snapshot testing marks a pivotal evolution in the 197
140 Figure 1: Diagram highlighting the steps of snapshot testing. software testing landscape. As evidenced by various studies and 198
141 the ongoing enhancement of frameworks like Jest, snapshot testing 199
142
An example of how a snapshot test can be written using Jest is provides a comprehensive solution for maintaining application 200
143
presented in Listing 1. Unlike traditional unit tests, which require stability and reliability amidst the continual evolution of codebases. 201
144 202
developers to write specific assertions for each expected output,
145 203
snapshot tests automatically generate these assertions by compar-
146 204
ing the current output to the saved snapshot. When a snapshot test
147 205
fails, the testing framework highlights the differences between the 3 STUDY DESIGN
148 206
stored snapshot and the current state as shown in Figure 2.
149 To the best of our knowledge, comprehensive scientific studies on 207
1 const output = cut . toJSON () ;
150 snapshot testing are scarce. Nonetheless, we contend that conduct- 208
2 expect ( output ). toMatchSnapshot () ;
151 ing a detailed manual analysis and qualitatively characterizing a 209
152 Listing 1: Snapshot test example representative sample of snapshot tests can significantly deepen re- 210
153 searchers and developers understanding of the nature and behavior 211
154 of these tests. Our aim is to provide insights that enhance develop- 212
155 ing experience by highlighting the commonalities and disparities 213
156 in these tests across various software systems. This also involves 214
157 forming a consensus on what typically constitutes a coding pattern 215
158 for most projects, while also identifying instances where the usage 216
159 of this testing practice may have been unconventional. 217
160 Figure 2: Example of snapshot comparison. Based on the research by Fujita et al. [5], the first author ex- 218
161 amined the study’s replication package to explore its dataset, as 219
162 These differences might indicate a bug in the code (a true pos- it provides 1,487 GitHub projects adopting Jest. The data encom- 220
163 itive) or result from intentional changes to meet new or updated passed various attributes such as name, last commit hash, total files, 221
164 requirements (a false positive). In the case of a false positive, devel- lines of code, test methods, snapshot test methods and other rele- 222
165 opers should update the snapshot to reflect the current state, thus vant information. Based on this dataset, we have used a subset (our 223
166 setting a new snapshot version and resolving the failure. Frame- object of study) filtering only repositories with at least one snapshot 224
167 works like Jest can store snapshots in a file or embed them directly test, yielding 569 repositories as also reported by the original. 225
168 within the assertion method for easier verification in future runs. In an effort to replicate the experiment and verify metrics for 226
169 The roots of snapshot testing can be traced back to broader test- test methods and assertions, including their snapshot counterparts, 227
170 ing practices outlined in Michael Feathers’ seminal work, Working the first author sought to duplicate the study design outlined by the 228
171 Effectively with Legacy Code [4]. It suggests that characterization previous study. This process involved downloading all repositories 229
172 testing strategies, including automated tests, are crucial for miti- at their last commit SHA provided, filtering for test files, generating 230
173 gating the risks associated with modifying codebases. their AST (Abstract Syntax Trees), traversing nodes representing 231
174 2 232