The Era of Agentic Products: The Redefinition and Evolution of the Test Pyramid

Robin Xuan AgentTestable

---

Classic Test Pyramid

In traditional software testing theory, the Test Pyramid model categorizes testing methodologies along a spectrum from fine-grained to coarse-grained.

For instance, Unit Tests at the base focus on verifying functions or classes at the code level, while End-to-End (E2E) Tests at the apex target the coarser granularity of the complete User Journey.

This hierarchical, progressive structure embodies the core principle of the Test Pyramid: by executing dense and thorough fine-grained testing at lower levels, the vast majority of fundamental defects are intercepted early. This ensures that the coarse-grained tests at the top only encounter a minimal number of residual issues.

---

The Iteration

The evolution from determinism to non-determinism fundamentally restructures the quality assurance logic of agentic products, leading to a radical shift in test costs, testing efficiency, and the inherent non-determinism of the system itself. This transition is particularly acute in high-level, coarse-grained end-to-end testing.

Practice has proven that blindly applying the traditional Test Pyramid model to agentic scenarios often results in a final product that fails to meet real customer needs. This is primarily due to two disastrous consequences under the traditional model:

1. Inherently deterministic engineering functionalities cannot be verified efficiently;
2. Essentially non-deterministic intelligent behaviors cannot be systematically and quantitatively evaluated.

Can we completely abandon coarse-grained, end-to-end testing when evaluating agentic products?

Although some literature and testing schools of thought (such as the contract testing framework Pact.io) advocate for the complete elimination of coarse-grained testing in favor of denser fine-grained tests, doing so is disastrous from a business perspective for medium-to-large platforms. Whether at the API or UI level, completely abandoning end-to-end tests that simulate user journeys fails because lower-level, localized tests cannot reflect the actual business impact and blast radius of a failure on the end-user experience.

---

The Redefinition of the Test Pyramid in Agentic Product

Does this mean the Test Pyramid is completely obsolete for agentic products?

Not entirely, but it requires targeted adaptation. This is the central thesis of this article: in agentic products, we redefine the Test Pyramid to achieve efficient quality assurance.

1. Service Integration Test

This article assumes the existence of high-quality Unit Tests as a prerequisite and will not elaborate further. On this foundation, we introduce the Service Integration Test, which serves as the baseline test for deterministic functionalities.

This test level aims to verify the functionality and performance of the system itself. At this stage, we treat the service—whether a backend microservice or a micro-frontend application—as a gray box. The testing focus shifts to validating the service's business logic rather than low-level code implementation details. Its inputs can be either synchronous requests (e.g., HTTP requests) or asynchronous events (e.g., consuming Kafka messages). Concurrently, all external dependencies of the service (e.g., databases, downstream microservices, event writers) are fully mocked, allowing us to precisely control and inspect the external environment.

It is worth noting that some schools of thought introduce Component Testing for frontend verification (e.g., isolated testing of a single React component via Playwright). Although the underlying mocking techniques are identical to those used in Service Integration Tests, their test boundaries (scopes) are fundamentally different: Component Testing emphasizes the isolated validation of a single UI component.

However, in engineering practice, we observe that testing individual components in isolation is often less effective than comprehensive page-level integration testing, even when all dependencies are mocked. While empirical data quantifying the exact difference in defect detection rates is scarce, we can deduce this through the dimensions of Testing Overhead and Business Value:

• Testing Overhead: In Component Testing, the cumulative overhead of repeatedly spinning up mock environments, destroying them, and performing multiple renders for different independent components on a page is often far greater than the cost of a single integration test on the entire single-page application.
• Business Logic Gap: More critically, isolated component testing inherently suffers from a blind spot; it cannot verify the continuous, end-to-end business logic that spans multiple components or the entire frontend page.

Following this logic, if a service's core dependency is a Large Language Model (LLM), the LLM should naturally be mocked at the Service Integration Test level. In the vast majority of software engineering practices, isolating non-deterministic large models from deterministic integration testing is an undisputed consensus.

2. Deep Readiness Probe Layer

When deploying a service into a live environment, the primary metric to verify is whether the service possesses the capacity to run in a healthy state.

However, traditional lightweight health checks (such as a simple /healthy endpoint) only reflect whether the application process has started successfully; they cannot verify the service's access permissions and connectivity to the underlying infrastructure. This is the "Config Abyss" described earlier—the service appears to start normally, but crashes during runtime as soon as it attempts to access a specific infrastructure component due to misconfiguration.

To eliminate this vulnerability, we introduce the Deep Readiness Probe layer to ensure the service has verified connectivity to all required infrastructure (e.g., databases, caches, message queues) before officially accepting incoming traffic.

It must be explicitly clarified here that the LLM dependency does not fall under the category of infrastructure; rather, it should be defined as a "Dependent Service". Because external services are inherently non-deterministic and uncontrollable, they must never be included in the verification boundary of the Deep Readiness Probe layer.

3. Cascaded Deterministic User Journey Test

Once a service is deployed into a real environment, we must verify its actual impact on end-users—specifically, the complete integrity of the User Journey.

Historically, some testing paradigms have strictly classified end-to-end UI testing as the sole method for User Journey tests, relegating API testing to lower levels. This article challenges this distinction for the following reasons:

• Avoiding Overlapping Responsibilities: Functional validation of a single API endpoint is a lower-level responsibility already fully covered in the Service Integration Test layer, making its repetition in high-level, coarse-grained testing redundant.
• Efficiency of API Chain Simulation: User journeys can fundamentally be simulated via an ordered sequence of Chained API Calls without a hard dependency on fragile UI rendering. This API-level path simulation exponentially improves execution speed and reduces maintenance overhead.

Consequently, the User Journey Test defined in this framework encompasses both UI-based and API-based implementations.

In modern microservice architectures, to ensure that cross-functional teams retain equal autonomy to develop and maintain both code and test suites, User Journey tests are typically decentralized rather than stored centrally (Decentralized Test Ownership).

Given this decentralized setup, we emphasize the design of Cascaded Testing:
Suppose Service A is a purely internal service with no direct connection to any user journey, and therefore lacks its own User Journey tests. However, Service A is a downstream dependency of Service B and Service C, both of which possess their own User Journey tests. When Service A triggers a release, the CI/CD pipeline must automatically cascade and trigger (Cascaded Trigger) the User Journey tests of Service B and Service C. This guarantees that internal modifications in Service A do not indirectly disrupt the end-user experience upstream.

Furthermore, the core objective of this layer is to verify Engineering Functionality, rather than Intelligent Behavior (cognitive agentic reasoning).

We emphasize that even when this layer runs in a fully integrated environment, the LLM dependency must still be completely mocked. This provides 100% deterministic test outcomes, dramatically accelerates test execution, and eliminates unnecessary token costs.

4. Non-Deterministic Evaluation

For agentic products, non-deterministic functionalities are typically encapsulated within independent Agents.

Evaluation at the agent level aims to verify and measure the agent's overall performance, task completion accuracy, tool-calling correctness (Tool Usage), and planning efficiency (Step Efficiency).

It is critical to note that validating these agentic functions does not require UI-level intervention. Evaluation at this level is executed in non-production environments by building systematic "Arena" platforms and introducing specific evaluation methodologies (e.g., LLM-as-a-judge). The precise design of evaluation algorithms and metrics falls outside the scope of this post.

5. Acceptance User Journey Test

As the apex of the redefined Test Pyramid, the Acceptance User Journey Test differs from the "Cascaded Deterministic User Journey Test" in exactly one aspect: it does not mock the LLM dependency, invoking the live production model directly instead.

This layer of testing simulates end-to-end user behaviors in the most realistic manner, corresponding to the classic E2E test in the traditional pyramid. Because live LLMs introduce non-determinism and carry high API costs, the primary objective of this layer is to utilize a highly optimized, lean Golden Test Suite to quickly verify that core business flows (e.g., critical transactions, primary path routing) function properly under real production dependencies.