Testing Your Upgrade: Practical Considerations

You have successfully migrated your code to the target version, and you have a stable, building repository. Now, you will execute the Test Plan you created during the planning phase.

This guide provides a practical, step-by-step methodology for development testing. The core principle is to start with fast, low-cost tests and progressively move to slower, more expensive, and more realistic validation. Do not jump straight to full data runs.

Phase 1: The Foundation - Local and Automated Testing

This is the fastest and most important feedback loop. The goal is to catch as many issues as possible on your local machine before ever deploying to a shared environment.

1. Validate Automated Tests: Your first step is to ensure your existing automated test suite is working.
   • Unit & Integration Tests: Run your entire suite of automated tests locally. Fix any failures. These tests are your first line of defense and are crucial for validating that core logic within your ETL and Scoring processes is still sound.
     
     
2. Local Runtime Testing: Before deploying, try to run components locally.
   • Example: Starting the Quantexa application services on your local machine can quickly reveal runtime dependency conflicts or configuration errors that were not caught at compile time. Fixing these locally is much faster than in a shared environment.

Phase 2: Controlled Environment Testing

Once your local tests are passing, it's time to move to a shared development or test environment. The goal here is to perform an initial end-to-end run in a controlled manner.

1. Establish a Stable Baseline: To test for regression, you must have a stable point of comparison.
   • Control Your Code: Ensure the main branch (your baseline) and your upgrade branch are based on the same starting commit. A common practice is to create a temporary release-candidate branch from main to use as a stable, unchanging baseline for comparison.
     
     
   • Control Your Data: Use a static, well-understood dataset for this testing phase (e.g., a synthetic or a small, sanitized sample of real data). The input data for your baseline run and your upgrade run must be identical.
     
     
2. Produce the Baseline Run: Execute your full end-to-end batch process on the main (or release-candidate) branch using your controlled dataset. Document the results meticulously. This includes:
   • HDFS/S3 output locations.
   • Elasticsearch index names.
   • Batch job runtimes and resource profiles.
     
     
3. Produce the Upgrade Run: Execute the same end-to-end batch process on your upgrade branch, using the exact same input data.
   
   
4. Compare and Analyze: Compare the outputs from the upgrade run against your baseline.
   • The Statistical Profile Testing Framework (SPTF) is the recommended tool for efficiently comparing batch outputs.
     
     
   • Expect some changes. Bug fixes or functional improvements in the new Quantexa version can cause expected deviations. Your job is to validate that all changes are explainable and can be traced back to a specific migration or product change. Any unexplained changes are potential regressions that must be investigated.

Phase 3: Full Environment Testing (UAT / Staging)

Only after you are confident that most issues have been resolved should you move to a production-like environment with real, full-volume data.

1. Full Regression & SIT: Execute your full test plan in this environment. This includes validating integration with external schedulers, security configurations, and any other real-world integrations. This is your best opportunity to catch elusive, data-specific edge cases.
   
   
2. Performance Validation: Run your batch processes against full production data volumes. Compare the runtimes against your pre-upgrade performance benchmarks to ensure there are no significant performance regressions.
   
   
3. User Acceptance Testing (UAT): This is a vital part of the release cycle. Block out sufficient time for business users to validate the solution and sign off on the upgrade. Factor in time to investigate and resolve any issues found during UAT and re-test.

By following this phased approach, you systematically build confidence in your upgrade, ensuring that by the time you reach the most expensive testing phase, you have already eliminated the vast majority of issues in a more efficient manner.