Professional Documents
Culture Documents
Introducing SafeTest - A Novel Approach To Frontend Testing
Introducing SafeTest - A Novel Approach To Frontend Testing
Become a member
by Moshe Kolodny
In this post, we’re excited to introduce SafeTest, a revolutionary library that offers a
fresh perspective on End-To-End (E2E) tests for web-based User Interface (UI)
applications.
For instance, when using react-testing-library, a unit testing solution, you maintain
complete control over what to render and how the underlying services and imports
should behave. However, you lose the ability to interact with an actual page, which
can lead to a myriad of pain points:
Testing something like dark mode entails clicking the theme switcher or
knowing the localStorage mechanism to override.
Inability to test segments of the app, for example if a component is only visible
after clicking a button and waiting for a 60 second timer to countdown, the test
will need to run those actions and will be at least a minute long.
Recognizing these challenges, solutions like E2E Component Testing have emerged,
with offerings from Cypress and Playwright. While these tools attempt to rectify the
shortcomings of traditional integration testing methods, they have other limitations
due to their architecture. They start a dev server with bootstrapping code to load the
component and/or setup code you want, which limits their ability to handle complex
enterprise applications that might have OAuth or a complex build pipeline.
Moreover, updating TypeScript usage could break your tests until the
Cypress/Playwright team updates their runner.
Welcome to SafeTest
SafeTest aims to address these issues with a novel approach to UI testing. The main
idea is to have a snippet of code in our application bootstrapping stage that injects
hooks to run our tests (see the How Safetest Works sections for more info on what
this is doing). Note that how this works has no measurable impact on the regular
usage of your app since SafeTest leverages lazy loading to dynamically load the
tests only when running the tests (in the README example, the tests aren’t in the
production bundle at all). Once that’s in place, we can use Playwright to run regular
tests, thereby achieving the ideal browser control we want for our tests.
Access to all the DX features that come with Playwright (excluding the ones that
come with @playwright/test).
Video recording of tests, trace viewing, and pause page functionality for trying
out different page selectors/actions.
await expect(page.getByText('Admin')).not.toBeVisible();
});
it('has an admin mode', async () => {
const { page } = await render(<Header admin={true} />);
await expect(page.getByText('Admin')).toBeVisible();
});
await page.getByText('logout').click();
expect(await spy).toHaveBeenCalledWith();
});
});
Leveraging Overrides
SafeTest utilizes React Context to allow for value overrides during tests. For an
example of how this works, let’s assume we have a fetchPeople function used in a
component:
Now, in our test, we can override the response for this call:
describe('People', () => {
it('has a loading state', async () => {
const { page } = await render(
<FetchPerson.Override with={() => () => pending}>
<People />
</FetchPerson.Override>
);
await expect(page.getByText('Loading')).toBeVisible();
});
The render function also accepts a function that will be passed the initial app
component, allowing for the injection of any desired elements anywhere in the app:
it('has a people loaded state', async () => {
const { page } = await render(app =>
<FetchPerson.Override with={() => async () => resolved}>
{app}
</FetchPerson.Override>
);
await expect(page.getByText('User: Foo, name: 23')).toBeVisible();
});
With overrides, we can write complex test cases such as ensuring a service method
which combines API requests from /foo , /bar , and /baz , has the correct retry
mechanism for just the failed API requests and still maps the return value correctly.
So if /bar takes 3 attempts to resolve the method will make a total of 5 API calls.
Overrides aren’t limited to just API calls (since we can use also use page.route ), we
can also override specific app level values like feature flags or changing some static
value:
describe('Admin', () => {
it('works with admin flag', async () => {
const { page } = await render(
<UseIsAdmin.Override with={oldHook => {
const oldFlags = oldHook();
return { ...oldFlags, isAdmin: true };
}}>
<MyComponent />
</UseIsAdmin.Override>
);
await expect(page.getByText('Permission error')).not.toBeVisible();
});
});
describe('Language', () => {
it('displays', async () => {
const { page } = await render(
<Language.Override with={old => 'abc'}>
<MyComponent />
</Language.Override>
);
Overrides are a powerful feature of SafeTest and the examples here only scratch the
surface. For more information and examples, refer to the Overrides section on the
README.
Reporting
SafeTest comes out of the box with powerful reporting capabilities, such as
automatic linking of video replays, Playwright trace viewer, and even deep link
directly to the mounted tested component. The SafeTest repo README links to all
the example apps as well as the reports
SafeTest in Corporate Environments
Many large corporations need a form of authentication to use the app. Typically,
navigating to localhost:3000 just results in a perpetually loading page. You need to go
to a different port, like localhost:8000, which has a proxy server to check and/or
inject auth credentials into underlying service calls. This limitation is one of the
main reasons that Cypress/Playwright Component Tests aren’t suitable for use at
Netflix.
However, there’s usually a service that can generate test users whose credentials we
can use to log in and interact with the application. This facilitates creating a light
wrapper around SafeTest to automatically generate and assume that test user. For
instance, here’s basically how we do it at Netflix:
beforeAll(async () => {
createTestUser(options.extraUserOptions)
});
};
After setting this up, we simply import the above package in place of where we
would have used safetest/setup.
Beyond React
While this post focused on how SafeTest works with React, it’s not limited to just
React. SafeTest also works with Vue, Svelte, Angular, and even can run on NextJS or
Gatsby. It also runs using either Jest or Vitest based on which test runner your
scaffolding started you off with. The examples folder demonstrates how to use
SafeTest with different tooling combinations, and we encourage contributions to
add more cases.
At its core, SafeTest is an intelligent glue for a test runner, a UI library, and a
browser runner. Though the most common usage at Netflix employs
Jest/React/Playwright, it’s easy to add more adapters for other options.
Conclusion
SafeTest is a powerful testing framework that’s being adopted within Netflix. It
allows for easy authoring of tests and provides comprehensive reports when and
how any failures occurred, complete with links to view a playback video or
manually run the test steps to see what broke. We’re excited to see how it will
revolutionize UI testing and look forward to your feedback and contributions.
Search
Follow
Learn more about how Netflix designs, builds, and operates our systems and engineering organizations
1.3K 23
Netflix Technology Blog in Netflix TechBlog
551 6
1.5K 11
1K 11
Artturi Jalli
9.6K 126
Zhimin Zhan
303 8
Lists
ChatGPT
21 stories · 473 saves
Danny Moerkerke in ITNEXT
1K 20
Akhilesh Mishra
2.4K 22
551 6
Jacob Bennett in The Atomic Engineer
526 6