Discover the missing piece of E2E test results

This article is the first chapter of the Sidekick Stack-based Testing Series. Tools mentioned in this article (Cypress & Sidekick) are both open-source, and after reading it you will learn:

• How you can include backend exceptions in your Cypress E2E test results

Exceptions occur in both backend & frontend, and many of them are handled to provide users with trouble-free sessions. Having these error-handling mechanisms is healthy for users’ experience, but they can prevent us, developers, from observing the exceptions and backend-related anomalies in E2E tests.

E2E tests ensure that the application performs as expected by running the test cases based on the users’ workflows. Tools such as Cypress, Playwright & Selenium enable us to easily define and apply those tests. In this article, I’ll explain backend exceptions, a missing piece of E2E test results, and how we can collect them in a test run in Cypress.

Well-known E2E testing tools mentioned above allow us to define tests for each action, and there are countless add-ons to improve their abilities. Since those tests are based on users’ behaviors they are mainly focused on frontend experience, and there are barely any significant resources for backend-related test definitions. Most methods/plugins we have are limited to making requests to backend and defining tests based on responses. Although those might be more than enough for mocking users’ perspective, these tests lack depth about backend’s state. Primarily when exceptions are handled, and there is no trace left of anomalies. In the end, even passing all E2E tests might not mean a system with zero issues.

A way to collect exceptions & stacks from the backend can help us understand the missing pieces in E2E tests. Sidekick’s Test Module comes in handy for this use case. Sidekick’s Test Module and it’s upcoming test framework plugins enable developers to bring Sidekick actions (logpoints, tracepoints & error stack collection) to E2E tests, eventually giving developers ability to collect errors as they happen and define stack based tests to ensure reliability of backend’s state. This article focuses on the first one, collecting backend errors as they occur in a test run.

Sidekick’s Error Stack Collection gathers real-time stack data from backends whenever they have an exception. You can read more about it here:

Error Stack Collection is already helpful for problems in production. With Sidekick’s new Test Module it will also be a literal sidekick to developers for discovering root causes of errors in tests. Such data is nifty for finding neglected issues & root causes of some flaky tests.

Setting up Error Stack Collection within our E2E tests in Cypress :

Our setup will be consisted of a simple Node.js To-Do application powered by Express framework. We will define tests in Cypress as we normally do and observe stacks of exceptions thrown in the backend as a result of those tests.

You can find To-do app’s repo here:

Requirements:

• Cypress (this can be applied in any other test framework)
• Having a Sidekick SaaS account or a Sidekick Open-Source Broker up and running: https://www.runsidekick.com/
• A Node.js backend configured to work with Sidekick agent (Sidekick Error Stack Collection in supported in both Node.js & Python runtimes)
  • Check the example setup here: https://www.youtube.com/watch?v=tc9TupvV-TM
  • Related docs: https://docs.runsidekick.com/installation/agents/node.js/error-stack-collection
• Sidekick Test Module up and running:
  • An open-source project that can be found here: https://github.com/runsidekick/sidekick-test-mode

Setup:

Before starting our test, we need to setup the Sidekick agent & enable error stack collection.

1- Installing Sidekick agent to our backend

You can find extra info here: https://docs.runsidekick.com/installation/agents/node.js/installation

//add the code below block to top of your project
const SidekickDebugger = require('@runsidekick/sidekick-agent-nodejs');

2- Enable Error Stack Collection mode

//after importing Sidekick Agent, initialize it with the given configuration  
SidekickDebugger.start({   
  applicationName: 'Todo - Express',  
  applicationStage: 'prod',  
  applicationVersion: '1.0.0',  
  apiKey: '<Sidekick-api-key>',  
  errorCollectionEnable: true,  
  errorCollectionEnableCaptureFrame: true  
});

Now your backend is ready for our tests.

3- Configure & Start Sidekick Test Module (check its repo for more info)

Now we are good to go!

Test definition:

The user will login to the system before each test.

First, it will add a new item “Feed the cat” and check if it is added.
Secondly, it will try to add a new item with an empty string and expect nothing to change. Submitting an empty string results in an error in our backend, but we are catching those errors so that user is not inconvenienced.

UI test results in Cypress

Finally, in the last test, we will make a request to Error Stack endpoint of Sidekick Test Module to get collected errors within the error. The expected result is an empty array, and anything else means that we had an issue in our backend.

You can find the example definition below:

  it("Check backend errors", () => {  
    const getErrorStacks = {  
      url: "http://localhost:8081/api/events/errorstack",  
      headers: {  
        Authorization: "Bearer apikey",  
      },  
    };  
    cy.request(getErrorStacks)  
          .its("body")  
          .should("be.an", "array")  
          .and("have.length", 0);  
      });

Results:

As our test completes, we can see that although our frontend-related tests pass, our E2E test fails as Sidekick had collected errors when it tried to submit an empty string in the test run.

Overall test results

If there are errors occur in the backend, Sidekick Test Module should return a list of collected errors and their stacks, as seen below.

Failing test case and collected error information

Now we can click on the related line to get info about the collected error or check the error from any Sidekick client (Web IDE, VSCode, or JetBrains IDEs). Soon we will bring a better view of those errors with our upcoming Cypress plugin.

Detailed information about collected error’s stack

In conclusion, we have used Sidekick agent & Test Module to bring backend errors to our E2E test, and we were able to capture the snapshot of the state of our backend as it threw an exception.

Where to go from here:

With our current setup, we got in-depth error information from our backend apps. In further chapters, we will be bringing Sidekick’s tracepoint (non-breaking breakpoint) feature to Cypress that will allow us to define tests based on state and stack information of our backend. Right now, we are also working on a Cypress plugin that will make using Sidekick Actions within Cypress as easy as it gets, and your support and feedback will guide us through our road. After Cypress, supporting other test frameworks is on our roadmap. You can find us at Discord or contact us via our website https://runsidekick.com if you want to participate or become a part of our community