A serverless solution for integrating test automation framework with Test Management tool

02 Feb 2024 A serverless solution for integrating test automation framework with Test Management tool

Introduction

Lately, I’ve been actively engaged in the development of test automation frameworks that leverage Allure, requiring integration with a Test Management tool.

For those familiar with Allure, its capability to seamlessly integrate with CI pipelines and provide a pleasing overview is widely acknowledged. However, for sustained and comprehensive results tracking, it becomes imperative to incorporate these outcomes into a Test Management tool such as Azure Test Plan or Xray.

Having encountered this requirement while working with multiple clients, I identified an optimal opportunity to devise a serverless solution. In this article, I will elaborate on a specific scenario where Jenkins serves as the CI pipeline, and Xray acts as the Test Management tool, demonstrating an integration for efficient result management.

Requirements

Prior to exploring the solution, it’s essential to establish a clear understanding of the requirements:

1. Long-Term Reliability: The solution must ensure that test results are not only stored but also accessed reliably over time, independent of the build (Jenkins build) lifecycle. This longevity is crucial for comprehensive analysis and historical reference.

2. Unified Allure Results Hub: Considering the diversity of multiple application/service test automation frameworks in use, a centralised and common hub is imperative. This hub should facilitate the seamless uploading of Allure results, serving as a universal starting point for necessary processing across different frameworks.

3. Tailored Retention Requirements: Recognising the unique nature of various applications, distinct retention requirements come into play such as legal requirements for test result retention. The solution must accommodate these divergent needs, ensuring that reports are retained based on the specific application they represent.

By delineating these essential requirements, we pave the way for a solution that not only addresses our immediate needs but also aligns with the diverse and dynamic landscape of our testing environment.

Solution

The system relies on these key elements:

1. S3 buckets for storage: We use AWS S3 buckets to store various types of data like files and reports. To manage data efficiently and control costs, these buckets have specific rules for how data should be handled over time, known as S3 lifecycle rules.
2. Lambda: AWS Lambda is used for computing tasks. It helps with activities like handling allure reports and generating special files in JSON format for Xray.
3. Xray API: The Xray API is utilised to bring in the JSON files created earlier into the Xray test management tool. This helps in seamlessly integrating and managing the testing processes.

Let’s dive into the solution!

Allure reports are dispatched to a designated Allure data S3 bucket after the tests are executed. Upon the arrival of Allure reports in Allure data S3 bucket, a S3 event triggers the Allure data processor lambda, initiating a series of actions to enhance our test management.

The Allure data processor lambda excels at its role by:

• Creating Test Case Attachments: Transforming the processed data into valuable test case attachments, these are stored in a dedicated Test Case Attachments bucket.
• Generating Xray-Compatible JSON Files: A crucial step in our workflow involves the creation of JSON format files tailored for the Xray Test Management tool. For every test suite, a dedicated Xray JSON file is crafted, ensuring seamless integration with Xray.
• Uploading Processed JSON Files to Xray Data Bucket: The processed JSON files find their home in the Xray JSON Data bucket, establishing a centralised repository for structured and meaningful test data.
• Archiving Allure Reports for Future Reference: Recognising the importance of historical data, Allure reports are intelligently archived and stored in a designated Archive bucket, enabling easy retrieval for future reference and analysis.

The journey doesn’t end with the creation of Xray JSON files. A sophisticated integration process has been implemented, leveraging AWS S3, Lambda, and the Xray API, to import these files into the Xray Test Management tool.

Here’s how the solution unfolds:

1. S3 Event Trigger for Xray JSON Files: As the Xray JSON files land in the designated xray-json bucket, an orchestrated S3 event sets the stage for the next act, summoning the Xray Integrator Lambda into action.

2. Xray Integrator Lambda uses the Power of Xray API: The Xray Integrator Lambda, armed with the Xray API, imports the Xray JSON files into the Xray Test Management tool, initiating a cascade of actions for each imported file.

3. The Import Process:

• Dynamic Creation of Test Executions: With precision, new test executions are dynamically created in Xray, aligning seamlessly with the imported data.
• Intelligent Handling of Existing Test Cases: The Xray API is no stranger to intelligence. It effortlessly pulls existing test cases into the corresponding test execution, ensuring a coherent and organised structure.
• Swift Response to Non-existent Test Cases: In cases where a test case is yet to find its place, the Xray API takes charge by creating a new one and seamlessly integrating it into the ongoing test execution.
• Test Execution Updates: It diligently updates all test results within the test execution, reflecting the pass/fail status with precision.
• Comprehensive Documentation with Test Case Attachments: For a thorough understanding of each test run, the Lambda enriches the test case run comment with S3 links to relevant test case attachments. This ensures that every detail is readily available and accessible.

Why Not Opt for Xray’s Built-in Test Runners Integration?

The answer lies in the intricacies of our specific requirements. While Xray does offer a test runners integration, it falls short in meeting our customised needs. Our demands extend beyond the standard functionalities, necessitating the flexibility to utilise names and descriptions sourced directly from the Allure report.

As of today, Xray lacks a readily available integration that seamlessly imports Allure results. This absence of a dedicated solution further underscores the importance of our tailored approach, ensuring that our unique specifications are not only met but exceeded.

Conclusion

In conclusion, this blog has delved into how AWS can be used for tailoring test automation strategies and concepts. Beyond this context, I’ve applied a comparable approach in a different scenario involving Python and the pytest framework for a client. In that instance, the challenge was to integrate Allure reports with Azure DevOps test plans. Remarkably, Azure DevOps test plan APIs proved instrumental in seamlessly incorporating our Allure reports into the Azure DevOps test plan, underscoring the adaptability of these cloud platforms for diverse testing needs.