Create and Automate Grafana K6 API load testing through GitHub Action

 If you are new to Grafana K6 API load testing then I would recommend going through my initial post on API load testing using k6.

In this article here, I’m going to talk about automating your K6 load test APIs GitHub Action pipeline and integrating with Grafana cloud to publish the test run result.

Create your Grafana K6 Cloud Account

First visit Grafana K6 Cloud and create your free account which we will use for monitoring our test results on the cloud. Once you registered, grab an API Cloud Token from the setting under Testing & synthetics=> Performance blade from the left-hand side menu.

Next, let's create a project structure in an organized way to automate it through the pipeline, here I’m going to use my favorite GitHub Action. Follow the steps to do so.

Create K6 Project

Step 1: Create your project folder and two folders as env and tests and a javascript file main.js using your favorite IDE tool. I’m using again my favourite Microsoft Visual Studio Code.

Step 2: First we will set up env-related settings. Let’s create a folder as ‘test’ inside k6=>env and add two JSON files for settings related to tests, here I would define load and stress tests and then a file as settings.json which will have Tests related settings:

1. config.load.json: In this file, we will add settings for required VUs, duration, tags, etc for our tests.

{
    "vus": 1,
    "duration": "60s",
    "tags": {
      "test_type": "load",
      "api_name": "Test API"
    }
}

2. config.stress.json

{
    "vus": 10,
    "duration": "60s",
    "tags": {
      "test_type": "stress",
      "api_name": "Test API"
    }
}

3. settings.json

  {
    "SETTINGS": {    
      "baseUrl": "#{APIURL}#"
    },
    "TEST_FILTERS": {
      "enabled": false,
      "startsWith": null,
      "endsWith": null,
      "contains": null,
      "regex": null
    },
    "ENVIRONMENT": {
      "execution": "test",
      "optionsSet": "load",
    },
    "AZUREAD": {
      "tenantId":"#{TENANTID}#",
        "clientId": "#{CLIENTID}#",
        "clientSecret":"#{CLIENTSECRET}#", 
        "userName":"#{USERNAME}#", 
        "password":"#{PASSWORD}#"
    }
  }

in settings.json, I have defined four categories as URL settings, test filters (to filter tests to exclude/include in the run), environment ( to choose between load/test, and based on that one of the above two files config.load/stress.json will be loaded for VUs related settings and in last AzureAD category to define settings required to pull Azure AD token.

Step 3: create a folder ‘çommon’ inside k6=>tests and add two utility methods to get the token and another one to create an HTTP object for API calls.

1. getAuthToken.js.

import http from 'k6/http';
import { check } from 'k6';
import exec from 'k6/execution';

export function GetToken(tenantId, clientId, clientSecret, userName, password)
{   
  const authUrl = `https://login.microsoftonline.com/${tenantId}/oauth2/v2.0/token`;

  const payload = {
    client_id: clientId,
    client_secret: clientSecret,
    scope:`api://${clientId}/api.scope`,
    grant_type: 'password',
    username: userName,
    password: password
  };

  let options = {
    vus: 5,
    duration: "10s",
    thresholds: {  'http_req_duration{scenario:default}': [   `max>=0`,     ],    },
    }
   const headers = { 'Content-Type': 'application/x-www-form-urlencoded' };    

  const res = http.post(authUrl, payload, { headers, },  { options });
   if (!check(res, { 'Token Fetched': r => r.status == 200, }))
   { 
       console.log("Getting Auth token failed:",res);
        exec.test.abort("Abort Tests: Error! Getting Auth Token", res);
       return '';
   }
   
   console.log("your token is ---------- ", res.json()); 
   console.log("your token is ---------- ", res.json()["access_token"]); 
  
  return  res.json()["access_token"]; 
};

2. utility.js

//import modules
import { Httpx } from 'https://jslib.k6.io/httpx/0.0.1/index.js';
import { check } from "k6";
import { Rate } from 'k6/metrics';
//Error Rate is an object for representing a custom metric keeping track of 
//the failures
export const errorRate = new Rate('errors');

export function GetHttpObject(data) {
    //create httpx session
    const session = new Httpx({
        baseURL:  data.SETTINGS.baseUrl,
        timeout: 10000 // 10s timeout. 
    });
    //set headers
    session.addHeaders(
        {
            'Authorization': `Bearer ${data.JWT_TOKEN}`, 
            'User-Agent': 'My k6 custom user agent', 
            'Content-Type': 'application/json',
        }
    );
    return session;
}

Step 4: We will add a test by creating a module module-wise folder inside k6=> tests. For example, I added a folder called ‘user’ and user-tests.js inside the user folder. i.e. k6=>tests=>user=>user-tests.js with below code:

import http from 'k6/http';
import { check, sleep, group } from 'k6';

export let UserTests = [
    GetUserAccess, GetUserInterest
];

function GetUserAccess(httpsession) {
    group('GET User Access', function () {
        let res = httpsession.get(`/user/access`);

        check(res, {
            'status was 200': (r) => r.status == 200,
            "Got response in 2 seconds": (r) => r.timings.duration < 2000,
        });
    });
}

function GetUserInterest(httpsession) {
    group('Get User Interest', function () {
        let res = httpsession.get(`/user/interest`);
        check(res, {
            'status was 200': (r) => r.status == 200,
            "Got response in 2 seconds": (r) => r.timings.duration < 2000,
        });
    });
}

The above code, has first import statement for the dependencies and then a variable with an array of all defined method-test names followed by test code for methods/API where we have two assert statements, one to check the successful response and another for response time as less than 2 secs). You can add more assert statements based on your needs.

The variable ‘UserTests’ we define here will be used in the main file to list and run defined tests in this file.

Step 5: Now we add the main.js file, which is the starting/source file to be executed through the test command, to the project folder. i.e. K6=>main.js with below code.

import { sleep } from 'k6';
import {GetHttpObject} from './tests/common/utility.js';
import {GetToken} from './tests/common/getAuthToken.js';
//import tests
import { UserTests } from './tests/user/user-tests.js';

let ENVIRONMENT = {};
ENVIRONMENT.execution = "local";//default
if (__ENV.EXECUTION) {
    ENVIRONMENT.execution = __ENV.EXECUTION;
}

ENVIRONMENT.optionsSet = "load";//default option
if (__ENV.OPTIONS_SET) {
    ENVIRONMENT.optionsSet = __ENV.OPTIONS_SET;
}

//load tests
 let TESTS = [...UserTests];


// Load k6 Run Options
let optionsFile = `./env/${ENVIRONMENT.execution}/config.${ENVIRONMENT.optionsSet}.json`;
export let options = JSON.parse(open(optionsFile));

// Load test settings
let DATA = JSON.parse(open(`./env/${ENVIRONMENT.execution}/settings.json`));
DATA.ENVIRONMENT = ENVIRONMENT;

// Filter the tests to run
let TESTS_TO_RUN = [];

if (__ENV.TEST_FILTERS) {
    DATA.TEST_FILTERS.enabled = true;
    let tokens = __ENV.TEST_FILTERS.split('|');
    DATA.TEST_FILTERS[tokens[0]] = tokens[1];
}

if (DATA.TEST_FILTERS.enabled) {
    TESTS_TO_RUN = TESTS.filter(t => {

        console.debug(`Test filters. Probing ${t.name}`);
        // starts with
        if (DATA.TEST_FILTERS.startsWith != null &&
            t.name.startsWith(DATA.TEST_FILTERS.startsWith))
            return true;
        // ends with
        if (DATA.TEST_FILTERS.endsWith != null &&
            t.name.endsWith(DATA.TEST_FILTERS.endsWith))
            return true;
        // contains
        if (DATA.TEST_FILTERS.contains != null &&
            t.name.indexOf(DATA.TEST_FILTERS.contains) != -1)
            return true;
        // regex
        if (DATA.TEST_FILTERS.regex != null &&
            t.name.match(DATA.TEST_FILTERS.regex))
            return true;
        return false;
    });
}
else {
    TESTS_TO_RUN = [...TESTS];
}

export function setup() {
    if (__ENV.K6_CLOUD_TOKEN) {
        console.log('Reading from environment variable');
        DATA.JWT_TOKEN = __ENV.K6_CLOUD_TOKEN; //reading from environment variable.
    }
    else { //fetch token through Microsoft graph api.
        console.log('Fetching token from Microsoft Graph API call');
        DATA.JWT_TOKEN = GetToken(DATA.AZUREAD.tenantId, DATA.AZUREAD.clientId, DATA.AZUREAD.clientSecret, DATA.AZUREAD.userName, DATA.AZUREAD.password);
    }
    return DATA;
}
export default function (data) {
    const httpsession = GetHttpObject(data);
    TESTS_TO_RUN.forEach(t => { t(httpsession); sleep(1); });
    sleep(1);
}

export function teardown(data) {
    // teardown code
}

In the above file here, after the import statements where we imported the dependencies and the tests variable created in the user-tests.js file, we define the environment local/test/prod (the way you name it, in this case, we created only one environment as ‘test’) and test options as load/stress.

In the next steps, we read the options for load/stress tests and variables from the settings file which will be de-tokenized through the pipeline before execution, and then we have a logic to apply test filters to include/exclude tests if we define any such test filter token in settings.json file inside TEST_FILTERS section.

Next, we have defined the setup() method to perform the necessary setup for the test run. for example, setting up an auth token, etc. Note, that all variables are available on the DATA variable which includes settings for the settings.js file as well.

Next, we have a default method that will initiate API calls and tests one by one for each API module defined or filtered and available as part of the TESTS_TO_RUN variable.

Till here our project is fully ready and next, we will create a GitHub action pipeline to automate this.

Step 6: Create a GitHub action pipeline ‘k6_api_tests.yml’ in your .github folder for your GitHub project location.

name: K6 API performance test automation. 

on:
  push:
    branches:
      - main
    paths:
      - 'API/**'
  workflow_dispatch:
  
env:
  WORKING_DIRECTORY: './k6'

jobs:
  k6-test-execution:
    runs-on: ubuntu-latest
    environment: testenv
    defaults:
      run:
        working-directory: ./k6

    steps:
      - uses: actions/checkout@v4

      - name: Test env Settings Variable Substitution
        uses: microsoft/variable-substitution@v1
        with:
          files: "**/settings.json"
        env:
          SETTINGS.baseUrl: ${{secrets.API_PATH}}
          AZUREAD.tenantId: ${{ secrets.MSALTenantID}}
          AZUREAD.clientId: ${{ secrets.MSALClientID}}
          AZUREAD.clientSecret: secrets.MSALClientSecret}}
          AZUREAD.userName: ${{ secrets.AutomationTestUserName}}
          AZUREAD.password: ${{ secrets.AutomationTestUserPassword}}

      - name: Run k6 local test
        uses: grafana/k6-action@v0.3.1
        with:
          filename: ./k6/main.js
          flags: -e OPTIONS_SET="load" -e EXECUTION="test"
          cloud: true
          token: ${{ secrets.K6_CLOUD_API_TOKEN }}
          

In the above file, we are first de-tokenizing the variables in the settings.js file in the step ‘Test env Settings Variable Substitution’.

Finally, we have step ‘Run k6 local test’ where we use the ‘grafana/k6-action@v0.3.1’ GitHub action which is from Grafana. This action has four params as:
filename: your main.js file path,
flags: any flags like env variables input from the command i.e. OPTIONS_SET and EXECUTION in our case.
cloud: boolean variable, true means it will expect the K6 cloud API token to publish the result.
token: your K6 Cloud API Token which you grabbed from the Grafana K6 Cloud portal.

With all this, you are done. Now you run the pipeline and you will see the result in the Grafana K6 cloud dashboard as below:

Hope you enjoyed the content, follow me for more like this, and please don’t forget to LIKE it. Happy programming.

API load testing using k6

 Grafana k6 is an open-source load testing tool. K6 allows you to test the reliability and performance of your APIs and catch performance regressions and problems earlier.

By reading this article, you will learn how to perform load testing on RESTful APIs using K6.

Setup
K6 has a standalone installer/package for Linux, macOS, and Windows and also supports a docker container.

Windows
For Windows users, please use the Choco (chocolatey package manager) or Winget (Window package manager, which default comes with Windows OS).
To install open a command prompt and run the command as

--for choco
choco install k6

--for winget
winget install k6

Alternatively, you can download and run the latest official installer.

If you are using Visual Studio Code, you can use the extension “k6 for Visual Studio Code” to execute your script directly from visual studio code.

Linux
For Linux Users, use the command below to install the k6.

--for Debian/Ubuntu
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6

--for Fedora/CentOS
sudo dnf install https://dl.k6.io/rpm/repo.rpm
sudo dnf install k6

MacOS
For MacOS users, install K6 using the Homebrew package installer.

brew install k6

For Window users, After installing the k6 package open a command prompt and type k6. If you get below output then you are ready.

Test Lifecycle
K6 test lifecycle has four stages and each stage runs in the same order.

1. init context: The init context runs once per VU and is generally used for tasks such as, loading local files, importing modules, declaring lifecycle functions etc. It is required.

//1. init code

2. setup function: setup function runs once in the entire lifecycle of the test and is generally used for data setups for tests. It is optional.

export function setup() {
  // 2. setup code
}

3. default function: default function depicts the test scenario, in short this is your test method. It is required.

export default function (data) {
  // 3. VU code
}

4. teardown function: Like setup, teardown function also runs once in the entire lifecycle and is generally used for cleanup or postprocessing data stuff. It is optional.

If the Setup function ends abnormally (e.g throws an error), the teardown() function isn’t called. Consider adding logic to the setup() function to handle errors and ensure proper cleanup.

export function teardown(data) {
  // 4. teardown code
}

Now let's write a test for REST API. For my tests here, I’m using Httpx library which is available here: https://jslib.k6.io/httpx/0.0.1/index.js

HTTP Get Request
Create your first script to test a Get REST request. To do this create a js file, for example k6-testscripts.js

//import modules
import { Httpx } from 'https://jslib.k6.io/httpx/0.0.1/index.js';
import { check } from "k6";
import { Rate } from 'k6/metrics';
//Error Rate is an object for representing a custom metric keeping track of 
//the failures
export const errorRate = new Rate('errors');
//create httpx session
const session = new Httpx({ 
                        baseURL: {baseurl i.e. http://localhost:80/},
                        timeout: 10000 // 10s timeout. 
                    });
//set headers
session.addHeaders(
  {
        'Authorization': 'Bearer {token}', //auth token
        'User-Agent': 'My k6 custom user agent', //user agent, any suitable name
        'Content-Type': 'application/json',
      }
);

//test method
export default function getUserPermission() {
    //get request
    let res = session.get("user/permission?userid=binod");

    //check response for 200 and response within 1000ms
    check(res, {
        "Is Status 200": (r) => r.status === 200,
        "Getting User Permission response time": (r) => r.timings.duration < 1000,
    }) || errorRate.add(1);

    if (res.status === 200) {
        let resp = JSON.parse(res.body);
        console.log("response:", JSON.stringify(resp));//prints the result, optional
    }
    else
        console.log(res.body);//this will help to know the error details on console. optional
}

Now let’s run the below script in the command prompt to execute the script:

k6 run C:\TestScripts\k6-testscripts.js

If all is well, the result of the above scripts would be as:

The above result says, the Get request took 427.15ms (http_req_duration) to complete, and took 34.97ms to connect (http_req_connecting) for one HTTP request. (http_reqs).

Now the same Get request will continuously for 30 seconds and 30 VUs.

k6 run -d 30s -u 30 C:\TestScripts\k6-testscripts.js

In my case, the above command results:

Based on the above results, a total of 319 requests (http_reqs) have been made and out of 319 requests, response time checks failed for 30 requests as it took more than 1000ms but all succeeded successfully as far as success response concerns.

HTTP Post request
For post request the test scripts would be as:

export default function registerUser() {
  let res = session.post(`/user/register/`, {
    first_name: 'Binod',
    last_name: 'Mahto',
    username: 'binod',
  });
  //check response for 201 and response within 1000ms
    check(res, {
        "Is Status 201": (r) => r.status === 201,
        "Registering User response time": (r) => r.timings.duration < 1000,
    })
}

Complete Result output summary
Here are the complete lists of summaries generated from your request execution.

1. checks: rate of successful checks percentage from total no. of checks, it also includes the no. of checks passed and failed.
2. data_received: the amount of data received.
3. data_sent: the amount of data sent.
4. errors: rate of error percentage for failure checks.
5. http_req_blocked : average time spent waiting for TCP connection
6. http_req_connecting : average time spent establishing a TCP connection
7. http_req_duration: average total time for the request. It’s calculated based on http_req_sending + http_req_waiting + http_req_receiving.
8. http_req_failed: percentage of failed requests. in our case 0 (0.00%) failed out of 319 requests.
9. http_req_receiving : average time spent on data receiving
10. http_req_sending: average time spent on data sending
11. http_req_tls_handshaking : average time spent on TLS handshaking
12. http_req_waiting: average time spent on waiting for a response from the remote host
13. http_reqs : total number of requests
14. iteration_duration: average time it took to execute the default function for each iteration.
15. iterations: total no. of iterations to execute default function (test method)
16. vus: number of virtual users. It is also represented as VU.
17. vus_max: maximum no. of virtual users allocated for the test execution.

httpx library
To know more about the httpx library to for other HTTP request as Put, Patch, Delete etc, please refer the doc here: https://k6.io/docs/javascript-api/jslib/httpx/post/

Hope you enjoyed the content, follow me for more like this, and please don’t forget to LIKE it. Happy programming.