Watch Out For Fixtures in cypress.io

July 22, 2020

This blog was originally published on Dev.to

Today I would like to tell you a story about a bug that cost me two days of searching and debugging sessions. It turned out to be a trivial thing, and with a better error message, it could have taken seconds instead of days. Let's go!

Hey Przemek! Could you help me?

A few days ago, I noticed that our VRT (Visual Regression Tests) suite started to fail for one case. I've asked my colleague, Monica, to check it. She accepted the challenge. After a long day of searching for the root cause, she told me that she doesn't have any idea why the test is failing. On the local machine, it has been passing all the time, but on our GitlabCI, we got an error. Weird thing, isn't it? Monica felt resigned and asked me for help. After two days of trying, committing, pushing, waiting, we've finally found it.

Fake server

We use a lot of tools in our tests. For unit testing, we use [jest](https://jestjs.io/). In E2E, we use [py.test](https://docs.pytest.org/en/stable/) with webDriver bindings. We also have UI tests that check our app on a higher level (interactions between components, pages, or views). Recently we introduced another test suite - VRT (Visual Regression Tests). The last two (UI and VRT) are based on [cypress.io](https://www.cypress.io/). It is an excellent tool for writing tests - from unit to full E2E.The backend of our app is very complicated, and it is difficult to set up a local environment. Because of that, for UI and VRT tests, we use a killer feature from cypress.io - network stubbing. Cypress can plug in between our app and a network request giving us a possibility to decide about the response from the API endpoint.More info about stub responses can be found in official Cypress documentation.

Fixtures

Fixtures are another feature from cypress.io that we use a lot, especially in our VRT suite. A fixture is a simple file that holds the data. We can reuse this file in many places. It helps us in organizing tests and managing the common responses from stubbed network requests. To load a fixture, we use a cy.fixture command. It expects a path to the file that we want to load. The path should be relative to a folder specified to hold fixtures (cypress/fixtures by default). Let's assume that we have the following file structure:And now let's look at code which loads fixtures:

Authors of Cypress took care of reducing a boilerplate needed to use a fixture in stubbing network requests. The cy.route command can take a shortcut to fixture as a response argument:

In this way, we stubbed a network request with data kept in reusable fixture files. Great job!

Where is the hero of the story?

Ok, but where did our bug go?I've created a simple app to visualize the issue. In the beginning, the app displays the Loading… message, then makes a request and replaces the text with a downloaded response.Fetching the data in old, good XHR way.I've also written a test:And created a fixture file fixtures/exampleFixture.json:Have you noticed a bug yet?In my case, the screenshot from the failed test was very helpful. Cypress takes them by default for failing tests, which is neat!

And now...Have you noticed a bug yet?A message about the status from the stubbed request caught my attention. It was 400 instead of 200. That was a clue.

The typo and file systems

Our bug, which we've been trying to solve with Monica, was a simple typo. The name of the fixture file was in camelCase, and we tried to load it via shortcut without the same naming convention.exampleFixture vs examplefixtureOk, but why does it work on the local machine and doesn't on CI?99% of our frontend team works on MacBooks. Our CI runs the tests in the docker container (Linux). You can think - "so what?". The default file system on Linux is case sensitive. On the other hand, the default file systems on Mac or Windows are not. What does it mean in practice?On Linux you can create two files with the "same" name (different letter case):

myAwesomeFile.js
myawesomefile.js

Linux treats them as separate files. Try to do the same on Mac or Windows - you can't do it. It has also impact on the way how you load the files, for example in nodejs. On Mac, there is no difference in load file by "myFixture" or "mYFiXtURe" names - the file will be loaded. On Linux, we will get an error - file not found.

Let's check it:

The test is always green on Mac. On Linux we get a 400 status for stubbed network request and an error message in console.

Wait, wait, wait...WAT? The following error originated from your application code, not from Cypress. Are you sure Cypress?Let's try to load the fixture without a shortcut:The error message for this code is quite different:And this is the error message that I've been counting on. We know right the way where we should start looking.

Summary

There are two takeaways from this story:

small typo could make you cry for two days of debugging session
you are as good as the error message from your test runner ;)

I think that Cypress could return the better message about missing fixtures than CypressError. That's why I've created an issue in cypress GitHub repository - here you can check the status.Thank you for your attention. I am going to try to solve the issue that I've created. Maybe I will be able to add something to the OpenSource community to make cypress.io even better.

‍Photo by Vincent Ghilione on Unsplash