Automated Testing using MCP & AI Agents

Posted on by — No Comments ↓

Mariana Chow @ AWS Hong Kong Community Day 2025

Preparation and Planning

Foundation: Testing Data

  • Often overlooked but critical

  • Task management system (e.g., Jira): Use webhooks to store updates in AWS S3

  • Swagger documentation: Provides API specifications and parameters

  • Historical test cases: Allows verification and retesting of previous cases

Connecting Data Sources

  • Traditional methods have limited capability to find relationships

  • Introduction of Large Language Models (LLMs) to bridge connections

  • Example:

  • Mariana (AWS career in cloud computing with AI)

  • Dario (co-founder of Entropic, an AI company)

  • Entropic developed a series of LLMs available through AWS Bedrock

Execution

AI-Driven Testing Workflow

  • Leveraging AI to automate and enhance testing processes

  • Detailed dive into how AI integrates with existing testing frameworks

Reporting

Comprehensive Reporting

  • Generating insightful reports from automated tests

  • Utilizing AI to provide actionable insights and recommendations

  • Emphasis on transforming traditional testing workflows with AI

  • Invitation for feedback and discussion on the presented ideas

Understanding Relationships with Knowledge Graphs

  • Knowledge graphs capture complex relationships that simple text searches cannot

  • Example: Relationship between Mariana (AWS career) and Entropic (AI company founded by Dario)

  • Knowledge graph reveals hidden connections not found by simple searches

Importance of Knowledge Graphs in Automated Testing

  • Crucial for automated task and test case generation

  • Extracts entities and connects related items from data sources

  • Example:

  • Requirement tickets

  • API specifications

  • Historical test cases

Creating a Knowledge Graph

  • Traditional methods (e.g., PostgreSQL) are widget-heavy and low visibility

  • No-code solutions (e.g., DynamoDB) are inefficient

  • Graph databases (e.g., AWS Neptune, Neo4j) are better but require learning new concepts (nodes, edges, properties)

Solution: Amazon Neptune Analytics

  • Combines graph database capabilities with foundational models and AWS Bedrock

  • Allows insertion and retrieval of information without complex syntax

  • Provides a beautiful graph view for data visualization

Graph Retrieval Augmented (RA)

  • Technique where AI retrieves reference documents and generates responses

  • Graph RA uses a knowledge graph for retrieval instead of simple text search

AI-Driven Test Pipeline

  • Built on top of the knowledge graph foundation

  • Uses Behavior-Driven Development (BDD) with Gherkin format for test cases

  • Gherkin scenarios use keywords like Given, When, and Then to specify the initial state, action, and expected outcome of a feature

  • Example test case: Checking homepage titles with prerequisites and scenarios

  • Knowledge graphs are essential for effective automated testing

  • Amazon Neptune Analytics simplifies graph database management and visualization

  • Graph RA enhances AI-driven test case generation and execution

AI-Driven Test Case Generation

Goal

  • Improve test case coverage and consistency using AI agents

AI Agent Capabilities

  • Chooses the best actions to perform and achieve testing goals

  • Analyzes business flows and reads requirements from the knowledge graph

Example: 

Subscription Management Feature

Business Flow Analysis

  • Knowledge graph identifies:

  • Validation of payment method in the UI

  • Calling payment APIs

Conflict Detection

  • AI agent detects requirement conflicts:

  • Old rule: User must verify email before accessing premium features

  • New rule: Trial users can access premium features for seven days without email verification

  • Updates related test cases accordingly

API Details Discovery

  • Extracts endpoints, required/optional parameters, and error responses from the graph DB

  • Identifies API dependencies through recorded data (e.g., successful subscription creation, payment failure)

Test Data Generation

  • Creates test data covering:

  • Happy flow (successful scenarios)

  • Edge cases (boundary conditions)

  • Error conditions (failure scenarios)

Conclusion

  • AI agents enhance test case generation by:

  • Analyzing business flows

  • Detecting conflicts and updating test cases

  • Discovering API details and dependencies

  • Generating comprehensive test data

Refining and Executing Test Cases

Refining Test Cases with Business Rules

  • Use information from business flow analysis and identified rules to enhance scenarios

  • Convert refined scenarios into scenario-based test cases

Human-in-the-Loop Verification

  • AI can generate comprehensive test cases, but human experts are needed for validation

  • Human verification ensures edge cases and business contexts are captured

  • Jira ticket system, API documentation, and historical test cases may not cover all scenarios

Execution with Playwright

  • Playwright: Fast and reliable end-to-end testing

  • Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox. Cross-platform. Test on Windows, Linux, and macOS, locally 

  • Utilize Playwright for test execution

  • Well-known, open-source automated testing framework

  • Communicates directly with the browser for efficient testing

  • Supports multiple languages (Java, Python, C, JavaScript, TypeScript)

  • Strong community support with 78.9K stars on GitHub

  • Native support for MCP (Model-Based Continuous Planning) for natural language instructions

  • Built-in features:

  • Parallel execution for efficiency

  • Comprehensive tracing and debugging capabilities

  • Automated screenshot and video recording

  • Detailed logging for documenting the testing process and enabling later verification

Conclusion

  • AI enhances test case generation, but human verification is crucial

  • Playwright offers efficient, versatile, and feature-rich test execution

Executing Test Cases with AI Agents

Task Executor Agent

  • AI agent for handling the execution phase

  • Extracts test parameters from generated tasks

  • Creates and stores Playwright scripts in the database for future reference

  • Automatically executes tests

Examples of AI-Driven Testing with Playwright

  • Front-end and back-end testing

  • Experiment using Playwright MCP with Amazon Q developer CLI

  • Three test cases for a simple e-commerce website:

  • Purchasing a product with shipment information and completing the order process

  • Logging in and entering the product page without further action

  • Adding a product to the shopping cart without further action

Post-Execution Verification

  • Test cases stored in the database

  • Utilization of features like video recording for later verification

  • Testers can review recorded videos to ensure alignment with expected behavior

Recap of Complete Test Execution Flow

  • Three AI agents work together:

  • Test case generator

  • Test case executor

  • Report generator (captures results and recordings)

  • Modularized approach provides flexibility and scalability

Leveraging Multiple MCPs

  • Use various MCPs for different testing needs:

  • MySQL MCP for generating realistic data

  • Redis MCP for storing recently used test cases

  • Simplifies testing processes

Key Message: Shift-Left Testing

  • Integrate automated testing early in the development cycle

  • Benefits:

  • Better quality

  • Improved efficiency

  • Lower costs

  • Reduced technical debt

Conclusion

  • AI-driven automated testing saves time and cost

  • Testers focus on verification, not test case generation

Leave a Reply