Retry, Rerun, Repeat

If at first you don't succeed, then dust yourself off and try again.                                                                                          - American R&B singer Aaliyah (1979-2001)

The modern Internet is built on retries. Software and hardware failures are normal and expected. Every action can fail, and thus every software system includes code to retry the action several times.

During testing, the application might be slower than expected, and thus the test runner needs to retry its assertions to avoid flakiness. A test might fail sometimes, and we might retry running it again, just to make sure the failure is real, and not an accidental fluke. Finally, even starting a browser can fail—thus, we need to repeat the test run before stopping the build. In this blog post I will show the three types of retries built into Cypress Test Runner and around it.

Retry

First, let's talk about retrying individual commands and assertions. Since Cypress does not care about the implementation details of the site under test, the only way the test can "know" when the application code does something is by retrying the command again and again - until the built-in assertions and the user-specified assertions pass.

The application below adds a new Todo item to the list after a two second delay (maybe it is sending the new item to the server first). The test enters the todo text and then asserts there is exactly one DOM element with class "todo".

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
  cy.get('.todo') // command
    .should('have.length', 1) // assertion
})

The code cy.get('.todo').should('have.length', 1) is executed again and again, until the assertion passes or times out. You can see the blue spinner for both the command and the assertion in the Command Log. Again and again the test command queries the DOM for all elements with class "todo", and then runs the assertion "there should be 1 element in the list of found elements". The assertion throws, and it goes back to selecting the elements again.

This command retry-ability is built into every command that is safe to retry, like cy.get, cy.its, etc. and is the first line of defense against flaky tests. For more examples, read our Retry-ability guide.

Rerun

But the single command retry-ability is not enough to get rid of flaky tests. Sometimes the test legitimately fails because our the backend server is slower than usual or even temporarily unavailable. The test fails—yet if we re-run the test, it passes because the failing condition has been resolved. After all—the Internet has been built to heal itself.

A good example of the test that might fail and needs to be re-run is a test that validates how long the server takes to respond to a HTTP request. I have described such tests in the Asserting Network Calls from Cypress Tests blog post. The source code is available in our "XHR Assertions" recipe.

describe('XHR', () => {
  it('is faster than ?ms', () => {
    cy.visit('index.html')

    // before the request goes out we need to set up spying
    // see https://on.cypress.io/network-requests
    cy.server()
    cy.route('POST', '/posts').as('post')

    cy.get('#load').click()
    cy.wait('@post').its('duration')
  })
})

We have the Ajax call's duration cy.wait('@post').its('duration') but what should the assertion compare the returned value to? Let's run the test and click on the Command Log to print the value in the DevTools' console.

Running the test several times shows numbers like 278, 205, 121, and 878ms. So typically this network call takes less than 300ms, yet sometimes it might take much much longer. A safe assertion would be

cy.wait('@post').its('duration').should('be.lessThan', 1000)

It certainly passes

Yet, it is a bad test. First, it still can fail. Maybe there is a network hiccup, maybe the CI is slower than the development machine, but very rarely, the test fails:

The second problem with this test is that it does not tell us much. The typical duration is less than 300ms, yet we are using much much larger value just to be safe. Notice that in this case, retrying the command does not help, since the duration value is already there. If we want a different duration value, we need to re-run the entire test!

We have introduced Test Retries in Cypress v5, and I think it is a very useful feature to tighten the assertions, yet avoiding constantly failing CI builds. In our test, we can use the "typical" duration limit of 300ms, and just re-run the test up to 3 times.

it('is faster than 300ms', { retries: 3 }, () => {
  cy.visit('index.html')

  // before the request goes out we need to set up spying
  // see https://on.cypress.io/network-requests
  cy.server()
  cy.route('POST', '/posts').as('post')

  cy.get('#load').click()
  cy.wait('@post').its('duration').should('be.lessThan', 300)
})

Most of the times pass on the first attempt. And if it fails, then it re-runs the entire test and (hopefully) passes. You can see this case happening in the GIF below. The first test attempt failed, because the server took almost 900ms to respond, but the second test attempt is green, because the server responded in just 89ms.

If the test really fails three times, if the target server really is slow and unresponsive, then the test fails, and we know we should investigate—this is not fluke.

Repeat

Sometimes we want to stress test (pun intended) our test commands, to see if there is any flake. We can repeat the same test again and again by creating multiple it blocks. I like using the bundled Lodash to write the code like this:

Cypress._.times(100, (k) => {
  it(`typing hello ${k + 1} / 100`, () => {
    cy.visit('/')
    cy.get('#enter-text').type('hello').should('have.value', 'hello')
  })
})

The above code snippet runs 100 identical tests in a row. Of course you can vary the test a little bit depending on the input data, see the Dynamic tests recipe.

Update: I have published cypress-grep plugin that allows you to find and repeat a test by its title or by a tag without modifying the spec's source code. The above test could be "burnt" 100 times from the command line by running:

$ npx cypress run --env grep="typing hello",burn=100

For more, read the blog post Burning Tests with cypress-grep.

Finally, something can go wrong on the larger scale, like the database connection might fail when starting the tests. In this case, we need to repeat the entire cypress run execution, and not just a single test. Using Cypress NPM module API we have written a Cypress wrapper called cypress-repeat. It is a tool that replaces the usual command cypress run ... <arguments> with the command cypress-repeat run -n <N> ... <arguments>. In addition, you can pass a command line argument to repeat the entire cypress run execution if there are any failures:

$ npm i -D cypress-repeat
$ npx cypress-repeat run -n 5 --until-passes --record ...

The above command performs Cypress run, then if all tests passed, it exits. But if there are any failed tests, the entire test run repeats up to 5 times.

Typically, we use cypress-repeat to find flaky tests, by running the same project again and again, or until it fails. We omit the --until-passes parameter and leave just -n <N>.

$ npx cypress-repeat run -n 5 --record ...

We could extend cypress-repeat in multiple ways, like re-running just the failed specs. If you have ideas please open a GitHub issue.

You can find all examples of repeating and retrying in the repository cypress-repeat-retry.

Happy Retries!

Share this blog