Create and Automate Grafana K6 API load testing through GitHub Action

 If you are new to Grafana K6 API load testing then I would recommend going through my initial post on API load testing using k6.

In this article here, I’m going to talk about automating your K6 load test APIs GitHub Action pipeline and integrating with Grafana cloud to publish the test run result.

Create your Grafana K6 Cloud Account

First visit Grafana K6 Cloud and create your free account which we will use for monitoring our test results on the cloud. Once you registered, grab an API Cloud Token from the setting under Testing & synthetics=> Performance blade from the left-hand side menu.

Next, let's create a project structure in an organized way to automate it through the pipeline, here I’m going to use my favorite GitHub Action. Follow the steps to do so.

Create K6 Project

Step 1: Create your project folder and two folders as env and tests and a javascript file main.js using your favorite IDE tool. I’m using again my favourite Microsoft Visual Studio Code.

Step 2: First we will set up env-related settings. Let’s create a folder as ‘test’ inside k6=>env and add two JSON files for settings related to tests, here I would define load and stress tests and then a file as settings.json which will have Tests related settings:

1. config.load.json: In this file, we will add settings for required VUs, duration, tags, etc for our tests.

{
    "vus": 1,
    "duration": "60s",
    "tags": {
      "test_type": "load",
      "api_name": "Test API"
    }
}

2. config.stress.json

{
    "vus": 10,
    "duration": "60s",
    "tags": {
      "test_type": "stress",
      "api_name": "Test API"
    }
}

3. settings.json

  {
    "SETTINGS": {    
      "baseUrl": "#{APIURL}#"
    },
    "TEST_FILTERS": {
      "enabled": false,
      "startsWith": null,
      "endsWith": null,
      "contains": null,
      "regex": null
    },
    "ENVIRONMENT": {
      "execution": "test",
      "optionsSet": "load",
    },
    "AZUREAD": {
      "tenantId":"#{TENANTID}#",
        "clientId": "#{CLIENTID}#",
        "clientSecret":"#{CLIENTSECRET}#", 
        "userName":"#{USERNAME}#", 
        "password":"#{PASSWORD}#"
    }
  }

in settings.json, I have defined four categories as URL settings, test filters (to filter tests to exclude/include in the run), environment ( to choose between load/test, and based on that one of the above two files config.load/stress.json will be loaded for VUs related settings and in last AzureAD category to define settings required to pull Azure AD token.

Step 3: create a folder ‘çommon’ inside k6=>tests and add two utility methods to get the token and another one to create an HTTP object for API calls.

1. getAuthToken.js.

import http from 'k6/http';
import { check } from 'k6';
import exec from 'k6/execution';

export function GetToken(tenantId, clientId, clientSecret, userName, password)
{   
  const authUrl = `https://login.microsoftonline.com/${tenantId}/oauth2/v2.0/token`;

  const payload = {
    client_id: clientId,
    client_secret: clientSecret,
    scope:`api://${clientId}/api.scope`,
    grant_type: 'password',
    username: userName,
    password: password
  };

  let options = {
    vus: 5,
    duration: "10s",
    thresholds: {  'http_req_duration{scenario:default}': [   `max>=0`,     ],    },
    }
   const headers = { 'Content-Type': 'application/x-www-form-urlencoded' };    

  const res = http.post(authUrl, payload, { headers, },  { options });
   if (!check(res, { 'Token Fetched': r => r.status == 200, }))
   { 
       console.log("Getting Auth token failed:",res);
        exec.test.abort("Abort Tests: Error! Getting Auth Token", res);
       return '';
   }
   
   console.log("your token is ---------- ", res.json()); 
   console.log("your token is ---------- ", res.json()["access_token"]); 
  
  return  res.json()["access_token"]; 
};

2. utility.js

//import modules
import { Httpx } from 'https://jslib.k6.io/httpx/0.0.1/index.js';
import { check } from "k6";
import { Rate } from 'k6/metrics';
//Error Rate is an object for representing a custom metric keeping track of 
//the failures
export const errorRate = new Rate('errors');

export function GetHttpObject(data) {
    //create httpx session
    const session = new Httpx({
        baseURL:  data.SETTINGS.baseUrl,
        timeout: 10000 // 10s timeout. 
    });
    //set headers
    session.addHeaders(
        {
            'Authorization': `Bearer ${data.JWT_TOKEN}`, 
            'User-Agent': 'My k6 custom user agent', 
            'Content-Type': 'application/json',
        }
    );
    return session;
}

Step 4: We will add a test by creating a module module-wise folder inside k6=> tests. For example, I added a folder called ‘user’ and user-tests.js inside the user folder. i.e. k6=>tests=>user=>user-tests.js with below code:

import http from 'k6/http';
import { check, sleep, group } from 'k6';

export let UserTests = [
    GetUserAccess, GetUserInterest
];

function GetUserAccess(httpsession) {
    group('GET User Access', function () {
        let res = httpsession.get(`/user/access`);

        check(res, {
            'status was 200': (r) => r.status == 200,
            "Got response in 2 seconds": (r) => r.timings.duration < 2000,
        });
    });
}

function GetUserInterest(httpsession) {
    group('Get User Interest', function () {
        let res = httpsession.get(`/user/interest`);
        check(res, {
            'status was 200': (r) => r.status == 200,
            "Got response in 2 seconds": (r) => r.timings.duration < 2000,
        });
    });
}

The above code, has first import statement for the dependencies and then a variable with an array of all defined method-test names followed by test code for methods/API where we have two assert statements, one to check the successful response and another for response time as less than 2 secs). You can add more assert statements based on your needs.

The variable ‘UserTests’ we define here will be used in the main file to list and run defined tests in this file.

Step 5: Now we add the main.js file, which is the starting/source file to be executed through the test command, to the project folder. i.e. K6=>main.js with below code.

import { sleep } from 'k6';
import {GetHttpObject} from './tests/common/utility.js';
import {GetToken} from './tests/common/getAuthToken.js';
//import tests
import { UserTests } from './tests/user/user-tests.js';

let ENVIRONMENT = {};
ENVIRONMENT.execution = "local";//default
if (__ENV.EXECUTION) {
    ENVIRONMENT.execution = __ENV.EXECUTION;
}

ENVIRONMENT.optionsSet = "load";//default option
if (__ENV.OPTIONS_SET) {
    ENVIRONMENT.optionsSet = __ENV.OPTIONS_SET;
}

//load tests
 let TESTS = [...UserTests];


// Load k6 Run Options
let optionsFile = `./env/${ENVIRONMENT.execution}/config.${ENVIRONMENT.optionsSet}.json`;
export let options = JSON.parse(open(optionsFile));

// Load test settings
let DATA = JSON.parse(open(`./env/${ENVIRONMENT.execution}/settings.json`));
DATA.ENVIRONMENT = ENVIRONMENT;

// Filter the tests to run
let TESTS_TO_RUN = [];

if (__ENV.TEST_FILTERS) {
    DATA.TEST_FILTERS.enabled = true;
    let tokens = __ENV.TEST_FILTERS.split('|');
    DATA.TEST_FILTERS[tokens[0]] = tokens[1];
}

if (DATA.TEST_FILTERS.enabled) {
    TESTS_TO_RUN = TESTS.filter(t => {

        console.debug(`Test filters. Probing ${t.name}`);
        // starts with
        if (DATA.TEST_FILTERS.startsWith != null &&
            t.name.startsWith(DATA.TEST_FILTERS.startsWith))
            return true;
        // ends with
        if (DATA.TEST_FILTERS.endsWith != null &&
            t.name.endsWith(DATA.TEST_FILTERS.endsWith))
            return true;
        // contains
        if (DATA.TEST_FILTERS.contains != null &&
            t.name.indexOf(DATA.TEST_FILTERS.contains) != -1)
            return true;
        // regex
        if (DATA.TEST_FILTERS.regex != null &&
            t.name.match(DATA.TEST_FILTERS.regex))
            return true;
        return false;
    });
}
else {
    TESTS_TO_RUN = [...TESTS];
}

export function setup() {
    if (__ENV.K6_CLOUD_TOKEN) {
        console.log('Reading from environment variable');
        DATA.JWT_TOKEN = __ENV.K6_CLOUD_TOKEN; //reading from environment variable.
    }
    else { //fetch token through Microsoft graph api.
        console.log('Fetching token from Microsoft Graph API call');
        DATA.JWT_TOKEN = GetToken(DATA.AZUREAD.tenantId, DATA.AZUREAD.clientId, DATA.AZUREAD.clientSecret, DATA.AZUREAD.userName, DATA.AZUREAD.password);
    }
    return DATA;
}
export default function (data) {
    const httpsession = GetHttpObject(data);
    TESTS_TO_RUN.forEach(t => { t(httpsession); sleep(1); });
    sleep(1);
}

export function teardown(data) {
    // teardown code
}

In the above file here, after the import statements where we imported the dependencies and the tests variable created in the user-tests.js file, we define the environment local/test/prod (the way you name it, in this case, we created only one environment as ‘test’) and test options as load/stress.

In the next steps, we read the options for load/stress tests and variables from the settings file which will be de-tokenized through the pipeline before execution, and then we have a logic to apply test filters to include/exclude tests if we define any such test filter token in settings.json file inside TEST_FILTERS section.

Next, we have defined the setup() method to perform the necessary setup for the test run. for example, setting up an auth token, etc. Note, that all variables are available on the DATA variable which includes settings for the settings.js file as well.

Next, we have a default method that will initiate API calls and tests one by one for each API module defined or filtered and available as part of the TESTS_TO_RUN variable.

Till here our project is fully ready and next, we will create a GitHub action pipeline to automate this.

Step 6: Create a GitHub action pipeline ‘k6_api_tests.yml’ in your .github folder for your GitHub project location.

name: K6 API performance test automation. 

on:
  push:
    branches:
      - main
    paths:
      - 'API/**'
  workflow_dispatch:
  
env:
  WORKING_DIRECTORY: './k6'

jobs:
  k6-test-execution:
    runs-on: ubuntu-latest
    environment: testenv
    defaults:
      run:
        working-directory: ./k6

    steps:
      - uses: actions/checkout@v4

      - name: Test env Settings Variable Substitution
        uses: microsoft/variable-substitution@v1
        with:
          files: "**/settings.json"
        env:
          SETTINGS.baseUrl: ${{secrets.API_PATH}}
          AZUREAD.tenantId: ${{ secrets.MSALTenantID}}
          AZUREAD.clientId: ${{ secrets.MSALClientID}}
          AZUREAD.clientSecret: secrets.MSALClientSecret}}
          AZUREAD.userName: ${{ secrets.AutomationTestUserName}}
          AZUREAD.password: ${{ secrets.AutomationTestUserPassword}}

      - name: Run k6 local test
        uses: grafana/k6-action@v0.3.1
        with:
          filename: ./k6/main.js
          flags: -e OPTIONS_SET="load" -e EXECUTION="test"
          cloud: true
          token: ${{ secrets.K6_CLOUD_API_TOKEN }}
          

In the above file, we are first de-tokenizing the variables in the settings.js file in the step ‘Test env Settings Variable Substitution’.

Finally, we have step ‘Run k6 local test’ where we use the ‘grafana/k6-action@v0.3.1’ GitHub action which is from Grafana. This action has four params as:
filename: your main.js file path,
flags: any flags like env variables input from the command i.e. OPTIONS_SET and EXECUTION in our case.
cloud: boolean variable, true means it will expect the K6 cloud API token to publish the result.
token: your K6 Cloud API Token which you grabbed from the Grafana K6 Cloud portal.

With all this, you are done. Now you run the pipeline and you will see the result in the Grafana K6 cloud dashboard as below:

Hope you enjoyed the content, follow me for more like this, and please don’t forget to LIKE it. Happy programming.

End-to-End test with cypress

 Before we start on E2E test writing using cypress, let’s know the cpress.

What is Cypress?
Cypress is a next-generation front-end testing tool built for modern web applications. In short, it enables any browser-based web application to write unit, integration, or end-to-end tests.

Cypress also has a companion product to record your test runs and make it available anywhere anytime with Cypress Cloud enterprise application.

Here in this article, we will learn to setup the E2E project for our web based application.

Installing Cypress
First we need to setup cypress and we can do it as explained on the link below based on your OS enviornment.

Seamless Cypress Installation Guide | Cypress Documentation

After you are done with installing, you will see a basic structure added in your designated location with some config files and a folder called ‘cypress/e2e’ under which we will be writing E2E tests.

From here, we will certain changes to organize our structure by following some best practices for our E2E test coverage.

1. Run your tests always in incognito mode to avoid browser’s cached data and to do this, we need add the following setting in the ‘cypress.config.js’ file available in your base project location (where cypress folder is created). * if the file is not created then add it.

`const { defineConfig } = require("cypress");
const cucumber = require('cypress-cucumber-preprocessor').default
const createBundler = require("@bahmutov/cypress-esbuild-preprocessor");
const preprocessor = require("@badeball/cypress-cucumber-preprocessor");
const createEsbuildPlugin = require("@badeball/cypress-cucumber-preprocessor/esbuild");

const setupNodeEvents = async (on, config) => {
 on("before:browser:launch", (browser = {}, launchOptions) => {
    if (browser.family === 'chromium' && browser.name !== 'electron') {
    launchOptions.args.push("--incognito");
    return launchOptions;
    }

    if (browser.name === 'electron') {                
        launchOptions.preferences.incognito = true;
        return launchOptions;
    }

 });

 await preprocessor.addCucumberPreprocessorPlugin(on, config);
 on(
  "file:preprocessor",
  createBundler({
   plugins: [createEsbuildPlugin.default(config)],
  }),
 );
 return config;
};

2. Add the below setting to avoid issues with Microsoft SSO login in the config file after the above code.

module.exports = defineConfig({
      viewportWidth: 1920,
      viewportHeight: 1080,
      defaultCommandTimeout: 20000,
      chromeWebSecurity: false,
      fixturesFolder: false,
      video: true,
      
    
      e2e: {
        projectId: "mywebapp",
        hideXHRInCommandLog: true,
        //to avoid "Infinite redirection - iframe-request-id" issue with Azure AD SSO login issue in cypress
        experimentalSessionAndOrigin: true,
        experimentalModifyObstructiveThirdPartyCode : true,

        setupNodeEvents,
        specPattern: "**/*.feature",
        excludeSpecPattern: ["*.js"],
      }
    });

To know more about this go through my previous article here as: https://binodmahto.blogspot.com/2024/03/infinite-redirection-iframe-request-id.html

3. To avoid hardcoding any sensitive or common values required for tests, add an environment config file in the base folder (where cypress.config.js is there) as ‘cypress.env.json’.

{
    "web_url": "https://mywebapp.com",
    "username":"*****",
    "password":"*****"
}

4. Add all your E2E test scenarios by creating *.feature file inside cypress/e2e/features. For example, I’m adding a file as home.feature with the below code.

Feature: Welcome User Test
@smoke
Scenario: verify if user is successfully able to login
    Given User logs in through azure ad sso
    Then User can view the welcome home page

Based on the above code, basically, I created a test scenario where a user will be able to log in to my web application with Microsoft SSO login and then be redirected to the welcome home page after successful login.

Note: What you mentioned in Given and Then is your test steps and the cypress will look for the exact match of these defined steps here in all the step definitions files in crpress/e2e/step_definitions folder by scanning through all *.cy.ts files here.

5. Add/create step definitions files. Based on above steps our test sceanrio has two steps as given and then so lets create fisrt step definition file for login as ssologinpage.cy.ts inside crpress/e2e/step_definitions folder with below code.

import { Given, Then, When } from "@badeball/cypress-cucumber-preprocessor";

Given("User logs in through azure ad sso", () => {
  try{
    cy.visit(Cypress.env('web_url'));

    cy.origin('login.microsoftonline.com', () => {
      cy.get('input[type="email"]').type(Cypress.env('username'));
      cy.get('input[type="submit"]').click();
      
      cy.get('input[type="password"]').type(Cypress.env('password'), {
        log: false,
      });
      cy.get('input[type="submit"]').click();
      cy.get('#idBtn_Back').click();

    });
  }
  catch(exception){
      throw new SyntaxError( "Failed to login",);
  }
});

Above code define the script for login steps where SSO login will be performed using configure user credentials in cypress.env.json file.

Note: If your application uses your custom login form then change the code here accordingly.

6. Next add the step for Then steps for welcome home vaildation as homepage.cy.ts inside crpress/e2e/step_definitions folder with below code.

import { Given, Then, When } from "@badeball/cypress-cucumber-preprocessor";


Then("User can view the welcome home page", () => {
 
    try{
        //Validate Title
        cy.get('.page-title').contains('Welcome User');
    }

    catch(exception)
    {
        throw new SyntaxError(
   "Homepage details not displayed",
  );
    }
});

7. cypress.env.json file has been created as best practice to separate environment specific variables and to avoid repetitive common values hence it is must to add this file in .gitignore file to avoid any push for this file by any user.

As a result, your folder structure will look like this:

8. Now to run your tests, type the below command in your VS terminal window as

npx cypress open

The above command will open the Cypress tool, which help you to execute your tests.

Bonus
Automate this through the GitHub Action pipeline and integrate it with Cypress Cloud.

Before we start the pipeline you need to acquire Cypress cloud, please follow this link to acquire the Cypress record key.

Here are the steps we need to write in the pipeline based on the above cypress example mentioned in this article.

Step 1: Replace or de-tokenize the setting from cypress.env.json.

 - name: Cypress env Settings Variable Substitution
   uses: microsoft/variable-substitution@v1
   with:
      files: "**/cypress.env.json"
   env:
      web_url: ${{ secrets.WEBAPPLICATION_URL }}
      username: ${{ secrets.AutomationTestUserName}}
      password: ${{ secrets.AutomationTestUserPassword}}

Step 2: Install project dependencies.

- name: npm login & Install
  shell: bash
  run: |
     npm install --legacy-peer-deps
  working-directory: uitests

Step 3: Finally, run cypress tests with the cypress record key.

 - name: Running E2E tests on Chrome
   shell: bash
   run: |
     npx cypress run --record --key ${{ secrets.CypressRecordKey }}
   working-directory: uitests

and with this pipeline execution, your results will be on the Cypress cloud dashboard.

Hope you enjoyed the content, follow me for more like this, and please don’t forget to like it. Happy programming.