Regression Test Framework

The WEM Regression Test Framework is currently being developed for Internal Use first, to use specific WEM Projects with specific testable flows and pages that can be compared on different environments specifically to check for Regression. It uses Selenium to enable testing with "headless browser sessions" (no user interaction - optimal for browser-test-automation [wiki]).

Because WEM has its own specific elements and ways, that need to be taken into account when WEM Projects need to be tested automatically, the WEM Test DSL (Domain Specific Language, [wiki]) has been developed, providing specific script-commands to execute tests and perform specific actions on a WEM Project's Page in Runtime. Internally this is called "Flint-Script" after the person who created this DSL 😉. The next page provides details on this script.

WEM Regression Test Framework Parts

The WEM Regression Test Framework consists of the following parts:

• WEM Projects, available in staging/live runtimes, with specific test-able features that can be performed and checked using test-automation;

• Collections of test-cases, that may be organized and managed in suites, groups, written in WEM-Test-Script;

• Test Execution Server, that can accept collections of test-cases, execute them, perform comparison checks (based on screenshots) and return the results.

The Test Execution Server (TES) is at the time of writing only available for WEM Internal Use as we are still developing and improving this WEM Test Framework, but we intend to make this available in future. The TES hosts the Selenium Hubs, running on Kubernetes nodes. It can receive Test Jobs (containing all information about the Test Scripts to be executed against which runtimes) and executes them on Chrome, Edge and Firefox browsers (using the Selenium Drivers), and provides results. It provides a REST/JSON API to exchange the information.

Test and Reference Hosts

When a Test (a collection of test scripts) is being executed as a Test Job on the TES, the Test Job will have 2 specific URLs: Test and Reference, to compare the test cases on the Test Environment (the environment that is being tested/checked for changes) to the situation on the Reference Environment (the unchanged / current environment that holds the situation as it is expected to be). These hosts are provided on Test-Job level as parameters and can be to compare:

• Test-Runtime against Live-Runtime (what we usually do when Runtime Changes are about to be deployed);

• Staging against Live (to check certain changes before publishing a project);

• Public Runtime (staging/live) against Private Runtime (staging/live).

This approach requires that the tested project is available on both (test and reference) runtime environment, published and containing the same datasets (which may require additional manual work, unless within the project a specific data-initialization flow is available).

Perform tests using online portal

During Q3-Q4 2023 we will be working on a specific projects in WEM in which the Test Suites/Groups/Cases can be managed and organized, the test-scripts can be written in a user-friendly way, and ultimately can be sent to the TES for execution and test-results can be reviewed. The plan is to make this available at some point (when ready) via MyWEM or WEM DevOps Portals so WEM Users with the proper access can use this solution online (in runtime) to create and execute their own automated tests against their own WEM projects.