Define Testing Requirements
With your scope, estimate, and resource profile defined, the final step in planning is to create a robust testing strategy. The goal of upgrade testing is not to re-test your entire solution from scratch; it is a risk-based exercise designed to build confidence that the upgrade has not introduced any negative impacts and that the platform remains stable.
This guide provides a framework specifically for upgrade testing. It should be read in conjunction with Quantexa's High-Level Testing Methodology for a broader understanding of testing best practices.
Table of Contents
The Guiding Principle: Test Your Scope
Your testing effort should be directly proportional to the scope of change you have already defined. There is no need to review release notes or other documentation again; your "Define Upgrade Scope" document is the single source of truth for what needs to be validated.
Action: Identify your areas of focus from your scope document.
- "Impacted Components" Table: This "heat map" tells you where to focus the most testing effort. A component marked with "High" impact requires a more thorough validation than one marked "Low".
- "In Scope Tasks" Table: Every line item in this table, especially manual migrations and tech debt removal, should have a corresponding test case to validate that the change was successful and didn't have unintended consequences.
The Core Testing Pillars
Your test plan should be built around three mandatory pillars. The articles below describe how to perform these tests in detail; this section describes what to focus on in an upgrade context.
Pillar 1: Regression and Functional Testing
Goal: To ensure existing functionality has not been broken by the upgrade. The focus is on validating the outputs and core functions of your solution. For detailed methodologies, refer to the main Deployment Regression Testing and Functional Testing articles.
- Upgrade Focus: Use the Statistical Profile Testing Framework (SPTF) to efficiently automate the validation of your batch process outputs (ETL, Entity Resolution, Scoring).
Pillar 2: Systems Integration Testing (SIT)
Goal: To ensure all components of your solution (platform, infrastructure, schedulers, authentication) integrate correctly in a production-like environment.
| Area | Key Activities |
|---|---|
| End-to-End Batch | Validate that your external scheduling system can successfully trigger and complete full and incremental batch runs. |
| Deployment & Stability | Validate that your CI/CD process can deploy the application services successfully and that any automated database migrations run as expected. Let the services run to ensure stability. |
| Security | Spot-check that users can log in via your authentication provider, that API keys function, and that a sample of roles/policies are being correctly enforced. |
Pillar 3: Non-Functional Testing (NFT)
Goal: To ensure the performance, stability, and security of your solution remain within acceptable service level objectives (SLOs). For detailed methodologies, refer to the main Non-Functional Testing Recommendations article.
- Upgrade Focus: The key is to establish a pre-upgrade baseline for batch runtimes and critical API response times. Your post-upgrade testing should then focus on verifying that there are no significant performance regressions against this baseline.
Multi-Use Case Testing Considerations
If you are upgrading a platform that hosts multiple use cases, your testing must account for the interaction between them, especially if you are upgrading one use case at a time.
- Testing Shared Dependencies: If an upgraded use case consumes data from a shared, non-upgraded component (like a centralized data source), you must test to ensure this interaction still works correctly.
- Testing Upgraded Shared Components: If you are upgrading a shared component itself, it is critical to perform regression testing for all consuming use cases, including those that are not being upgraded in this cycle, to ensure they are not negatively impacted.
With a documented test plan covering the core pillars and any multi-use case considerations, you are ready for the final step: Produce Final Upgrade Plan.
Quantexa Team
