Planning for Success
Planning for success with a Quantexa Platform upgrade is no different from planning for success in any other project. A well-defined approach, clear scope, accurate estimates, and appropriate resourcing are the essential foundations for creating an upgrade plan that is set up to succeed. This guide provides a comprehensive, step-by-step framework to help you navigate the process from start to finish. Who is this guide for? This guide is primarily intended for Project Managers, Technical Leads (TLs), and Delivery Owners who are responsible for planning and executing a Quantexa Platform upgrade. Business stakeholders and solution designers will also find the sections on scoping and testing requirements valuable. What will you learn? By following this guide, you will have the tools and knowledge to: Clearly define the goals and scope of your upgrade. Choose the right upgrade strategy for your specific circumstances. Accurately estimate the effort and resources required. Assemble a comprehensive and actionable final upgrade plan. Before you begin We assume you have a foundational understanding of your organization's current Quantexa implementation and general project management principles. Familiarity with your platform's architecture and existing use cases will be highly beneficial. The following pages provide a step-by-step guide on how to best plan an upgrade of your Quantexa Platform instance. The Upgrade Planning Process The following pages provide a step-by-step guide on how to best plan an upgrade of your Quantexa Platform instance. We have broken the process down into key stages to help you navigate the journey. Stage 1: Setting the Direction This initial stage is about defining the high-level goals and approach for your upgrade. Page Description Identify Your Destination Understand the target version of the platform and the key benefits it brings. Define Your Upgrade Approach Choose the right strategy, whether it's a "like-for-like" or a transformational upgrade. Planning A Multi-Use Case Platform Upgrade Learn the specific considerations for planning an upgrade on a platform that serves multiple business areas. Stage 2: Defining the Work This stage focuses on the detailed work of understanding the scope and effort involved. Page Description Define Upgrade Scope Detail the specific components, configurations, and customisations that are in scope for the upgrade. Estimate The Upgrade Follow a structured process to create a realistic and data-driven estimate for the project. Resource The Upgrade Identify the roles, responsibilities, and skills needed to successfully deliver the upgrade. Define Testing Requirements Create a robust testing strategy to ensure a high-quality outcome and build confidence with stakeholders. Stage 3: Creating the Plan This final stage brings everything together into a formal plan. Page Description Produce Final Upgrade Plan Consolidate all the outputs from the previous steps into a single, actionable delivery plan.Get Your Dependencies: What They Are and How to Obtain Them
With your tracking in place, the next step is to gather all the software, tools, code, and access rights you will need to perform the upgrade. Having all dependencies ready and provisioned before starting development is crucial to avoid delays and interruptions. This guide provides a checklist of the key dependencies you will need to acquire. Checklist Item 1: The Quantexa Core Dependency Bundles These bundles contain the core Quantexa product artifacts (libraries and other dependencies) required to build the project against the new versions. Action: Obtain the dependency bundles for every version "hop" in your upgrade path. For example, an upgrade from v2.5 to v2.8 requires the bundles for v2.6, v2.7, and v2.8. How to Obtain: These bundles are not publicly downloadable. You must request them from your Quantexa contact. They will provide you with the necessary files. What to Do: Once received, upload the artifacts to your organization's repository management tool (e.g., Nexus, Artifactory). Ensure that both Maven and NPM artifacts are provisioned. The official documentation provides an example of how to upload to Nexus. Missing Dependencies: If you encounter issues with missing third-party dependencies during your build, it may be because your environment has no access to public repositories like Maven Central. The bundles are designed to be self-contained, so if you believe something is missing from the bundle itself, please raise a post in Quantexa Platform Support on the Community. Checklist Item 2: Component-Specific Bundles (If Applicable) If your solution uses certain Quantexa components like Data Packs, Decision Systems, or QPython, these have their own dependency bundles. Action: Check if your "In Scope Tasks" list from your upgrade plan includes upgrades to these components. If so, you must also request the specific dependency bundles for the target versions of these components from your Quantexa contact. Checklist Item 3: The Repository Tool The Repository Tool is the primary mechanism for automating the code migration between versions in an incremental upgrade. Action: Obtain the Repository Tool executable and the specific migration configuration files for every version "hop". How to Obtain: The tool and its configuration can be provided by your Quantexa contact. Further guidance can be found in the documentation: Download Repo-Tool. Checklist Item 4: Project Example project-example is Quantexa's reference implementation and an invaluable resource. It contains best-practice examples and shows how to correctly apply the migrations for each version. Having a local copy is highly recommended. Action: Get access to the project-example Git repository and, if necessary, transfer a copy to your development environment. How to Obtain: Follow the guidance here to get access: Accessing project-example. Checklist Item 5: Database Migration Scripts (If Applicable) Platform upgrades may introduce database schema changes. For some database technologies (like Oracle), these migrations must be run manually. Action: Discuss with your Quantexa Architect whether manual database migrations are required for your specific technology stack and target version. If they are, you must request the DDL (Data Definition Language) scripts from your Quantexa contact. Checklist Item 6: People and Places (Environments & Access) Software and code are only part of the equation. Ensure the team is fully enabled and the necessary environments are available. Action: Confirm the following before development begins: Team Onboarding: Ensure all developers on the upgrade team have been fully onboarded. System Access: Verify that every team member has the required access to all necessary systems (e.g., Git repository, artifact repository, JIRA, development/test environments). Environment Availability: Confirm that the required development and testing environments have been provisioned and are available for the project. These non-production environments are critical for isolating the upgrade work. With all these dependencies gathered and correctly placed, your team is now equipped with everything it needs to begin the hands-on development work.How to Perform an Incremental Upgrade
You have completed your planning and preparation, and you are now ready to begin the hands-on execution of the upgrade. This guide provides the step-by-step technical process for performing a single "hop" of an incremental upgrade (e.g., migrating from v2.5 to v2.6). The fundamental principle of an incremental upgrade is to migrate your project one major version at a time, ensuring the codebase is stable and validated at each stage before proceeding to the next. The Key Tool: The Repository Tool The Quantexa Repository Tool will be the primary engine for your upgrade. It uses a file generator called PLOPjs and code-modification tools like Scalafix to automate a significant portion of the migration effort. While powerful, this tool does not cover every scenario, and manual changes may therefore be required. The Workflow for a Single Upgrade Hop For each hop in your upgrade roadmap (e.g., from v2.5 to v2.6), you will follow this four-phase process. Phase 1: Automated Migrations The first step is always to let the automated tooling do the heavy lifting. This is a highly scripted process. For detailed instructions on the commands, refer to the documentation on Running the repository tool. Configure the Tool: Locate the migration-config.json for the version hop you are performing. Update the projectPath to point to your repository. Run Scalafix Migrations: Execute the Scalafix migrations first. Run Plop Migrations: Execute the relevant Plop migrations. It is best practice to run these one by one to create a granular commit history. Pro-Tip: If your project uses split repositories (e.g., separate ETL and Apps repos), only run the migrations relevant to the repository you are currently working on. Phase 2: Manual Migrations With the automated changes applied, you will now execute the manual tasks from the backlog you created in the preparation phase. Your JIRA board is your guide for this phase. Execute Manual Migration Tickets: Work through the JIRA tickets you created for the mandatory manual migrations. Each ticket should contain the context and a link to the specific documentation needed for that single task. Address Customization Tickets: Work through the tickets related to your project's customizations. These tasks involve reviewing how the upgrade has impacted your custom code and applying the necessary fixes to make it compatible. Execute Optional Migration Tickets: If you decided to include any optional migrations in your scope, execute those tickets now. Pro-Tip: Make small, specific commits for each distinct manual change (e.g., "Fix custom scoring function for v2.6 API change"). This creates a clean, traceable history. Refer to project-example to see how the same migrations were applied in Quantexa's reference project. Phase 3: Compile and Validate With the automated and manual changes applied, this phase is focused on compiling the code and validating its integrity by running the automated test suite. Compile the Code: The first action is to run a full build of the repository. If any compilation errors arise from the applied migrations, resolve them. These are typically caused by API changes or updated method signatures that impact custom code. Validate with Unit Tests: Once the repository compiles successfully, run your full suite of automated unit tests. Address any tests that are failing as a result of the upgrade changes. A successful, clean build with all unit tests passing marks the end of the development work for this hop. Phase 4: Intermediate Validation (Optional but Recommended) Before moving to the next hop, it is highly recommended to perform some level of intermediate validation to catch issues early. Run your automated integration tests. Perform a small, local ETL run to ensure the core process works. If practical, deploy the applications locally to check for runtime errors. Repeat and Finalize Once you have completed this four-phase process for one hop, you repeat the entire workflow for the next hop in your upgrade roadmap (e.g., v2.6 -> v2.7). After the final hop is complete and you have a stable, building repository on your target version, the hop-by-hop development phase is over. You are now ready to proceed with the full Development Testing as outlined in your test plan.Estimate The Upgrade
With a clearly defined scope document in hand, you are now ready to build a reliable effort estimate. This process translates your detailed list of tasks into the person-days required to complete the project, forming the bedrock of your final upgrade plan. This guide provides a practical, bottom-up estimation process. It should be used alongside the official documentation's guide on Key factors impacting upgrade effort, which provides essential context on what makes an estimate go up or down. The estimation method you use will depend directly on the Upgrade Approach you selected earlier: Incremental or Reset. Estimating an Incremental Upgrade For an incremental upgrade, the most reliable estimate is built from the bottom up. You will estimate each individual line item from your "In Scope Tasks" table, which you created during the "Define Scope" phase. Action: Build your bottom-up estimate. For each task, estimate the development and unit testing effort. This detailed breakdown ensures you account for every specific migration and change required. To help with this process, detailed Work Breakdown Structure (WBS) templates for specific platform version upgrades can be requested from your Quantexa contact. Task / Migration Description Category Dev Effort (days) Unit Testing (days) Total (days) Upgrade Document Service response handling Manual Migration 3 1 4 Execute database schema update One-Off Task 0.5 0.5 1 Refactor deprecated custom-function-x Tech Debt 2 1 3 Run Repository Tool for v2.7 migration Automated Migration 1 2 3 ...add all other tasks... Sub-Total X Y Z Action: Validate your estimate with the top-down model. Now, use the best practice estimation approach from the Documentation Site as a sense-check. Start with the baseline (e.g., 8 days) and adjust it based on the key factors. Does your detailed, bottom-up total roughly align with the top-down adjusted baseline? If they are close, you can have high confidence in your number. If they are far apart, it's a crucial signal to review both your detailed estimates and your understanding of the influencing factors: Have you underestimated the complexity of a required migration? Have you overestimated the negative impact of your customizations? Action: Add contingency and formal testing. Finally, add a contingency buffer to your sub-total to account for unforeseen issues. Remember, this estimate is for the development and unit testing effort only. The effort for formal testing cycles (SIT, UAT, OAT) must be estimated and added separately. Estimating a Reset A "Reset" project is estimated in much the same way as a brand-new Quantexa implementation. The focus is on estimating the effort required to selectively re-implement the desired functionality in a new, clean repository. Action: Estimate the "re-implementation" effort for each functional block. For each major component of your current solution (e.g., Entity Resolution, Scoring), estimate the effort to build it in the new target version. When doing so, you must factor in the effort to align with modern best practices and to remove the technical debt you identified in your existing customizations. Functional Block Re-Implementation Effort (days) Notes Data Ingestion & Parsing 15 Includes refactoring 3 custom parsers. Entity Resolution 20 Involves re-implementing logic to align with new best practices. Scoring & Alerting 10 UI & Document Views 12 Includes rebuilding 2 custom document viewers. ...add other functional blocks... Sub-Total X As with the incremental approach, a healthy contingency should be added to this sub-total, and formal testing should be estimated as a separate line item. ⚠️Warning: Avoid the "Big Bang" Trap A reset upgrade provides a tempting opportunity to tackle other major platform migrations at the same time (e.g., historical Fusion or Assess migrations, Parsers v4). While this can be efficient, be extremely careful about letting the scope balloon. The effort to validate major functional changes, combined with the cost of a very long-running project, can quickly become unmanageable. It is often better to treat these as separate, subsequent projects.Understanding Support During Upgrades: Where to Turn When Issues Arise
During an upgrade, you may encounter unexpected issues, from dependency conflicts to confusing migration errors. Knowing where to turn for help, and what information to provide, is key to resolving these issues quickly and keeping your project on track. This guide outlines the recommended support process, from self-service troubleshooting to engaging with the Quantexa Community and support options. The Support Process: A Tiered Approach Follow this tiered process to get your issues resolved efficiently. Tier 1: Self-Service & Troubleshooting Before reaching out for help, there are several steps you can take to diagnose the issue yourself. This is often the fastest way to a resolution. Review Your Trackers: Check your Customization Tracker and Issue Tracker. Is this a known issue or related to a high-risk customization you've already identified? Has another team member already logged and solved a similar problem? Consult project-example: Quantexa's reference project is an invaluable debugging tool. Verify Migrations: project-example contains a commit history showing exactly how each automated and manual migration was applied for a given version hop. Compare the problematic area of your code against the equivalent in project-example to see if you missed a step or implemented a change differently. Identify Best Practice Deviations: A high percentage of upgrade issues stem from project customizations that deviate from Quantexa best practice. If you are having trouble with a component, comparing its structure and patterns to the project-example implementation can often highlight the difference that is causing the problem. Check for Common Problems: Many upgrade issues fall into a few common categories. Before digging deeper, review this checklist: Is it a dependency clash? Upgrades often bring new versions of third-party libraries. If you see errors like NoSuchMethodError or ClassNotFoundException, it's often a sign that a library version is incorrect. Ensure your project is sourcing library versions from the official Quantexa Bill of Materials (BOM) wherever possible. Is it a missing dependency? If your build is failing because it cannot download a specific artifact, double-check that you have uploaded all the required dependency bundles (including for components like Data Packs) to your artifact repository. Is it a customisation-related breaking change? If an error is occurring in or around a piece of custom code, it is highly likely that an underlying product API or class it was relying on has changed. This is the most common source of upgrade issues. Tier 2: Community & Formal Support If you've ruled out common problems and are still stuck, it's time to engage with the wider Community and Quantexa's formal support options. Action: Raise a Support Request on the Quantexa Community Site. This is the primary mechanism for getting help. To ensure your request is handled efficiently, follow these best practices: Read the Guides: Familiarize yourself with How to Maximize Value from Community Support and How to Write a Good Support Request. Use the 'Upgrade' Tag: When creating your post, add the Upgrade tag. This helps route your request to the correct subject matter experts (SMEs) more quickly. Provide Rich Context: Your support request should be as detailed as possible. Include: The version you are upgrading from and to. The exact error message and full stack trace. A description of what you were doing when the error occurred. Relevant code snippets (if possible). Details from your troubleshooting so far. By following this structured approach, you can ensure that when you do need help, you get a fast and effective resolution, minimizing delays to your upgrade timeline.Implementing Upgrade Tracking: What It Is and Why It Matters
Before you write a single line of code or run the Repository Tool, the first step of the preparation phase is to establish your system of record. Systematic tracking is essential for a successful upgrade, providing a clear, real-time view of progress, issues, effort, and project-specific complexities. This guide provides templates and best practices for setting up the core trackers that will guide you through the execution phase. A well-maintained set of trackers provides three key benefits: Real-time Visibility: Understand exactly where the project stands against the plan, allowing you to identify delays and manage stakeholder expectations. Knowledge Retention: Creates an invaluable audit trail of decisions made, problems solved, and customizations addressed, which is critical for future maintenance and subsequent upgrades. Improved Future Estimates: Capturing the actual effort spent on tasks provides a data-driven basis for more accurate future planning. The Customization & Risk Tracker This is the most critical tracker you will maintain. It documents your project's unique fingerprint, capturing not just code customizations but also environmental or programmatic risks that can impact the upgrade. It should be a living document, reviewed and updated throughout the project. Project Unique Attributes / Areas of Risk Use this table to capture peculiarities of your project that aren't code customizations but still pose a risk or add complexity. Area Description Estimated Maintenance Effort (S/M/L/XL) Estimated Remediation Effort (S/M/L/XL) Risk (Low/Medium/ High/Critical) Comments/Actions Last Reviewed Date CI/CD Uses a custom Jenkins pipeline, not the standard Helm charts. M L Medium The custom deployment scripts may need significant updates for the new version. 2026-03-26 Programme Shares a production data source with Use Case B. S S High Any changes to the shared source must be coordinated and tested with the Use Case B team. 2026-03-26 External Apps The external 'Fraud Dashboard' uses our Search APIs directly. L L Medium API response changes in the new version could break the dashboard. Needs specific integration testing. 2026-03-26 Customizations Use this table to capture any way your solution deviates from standard Quantexa best practice, as per the Customizations documentation. This includes everything from overridden functions to workarounds and significant technical debt. Area (ETL/Apps/etc.) Component Description Relevant Links (JIRA/Confluence) Est. Maintenance Effort (S/M/L/XL) Est. Remediation Effort (S/M/L/XL) Risk (Low/Med/High/Critical) Comments/Actions Last Reviewed Date ETL Post-Processing A custom Spark script runs after the main resolver to enrich entities from an external file. [PROJ-1234] M L Medium This script uses an internal API that may change. It needs to be reviewed against the new version. 2026-03-26 Apps Document Service Custom bean override to change document sorting logic. [PROJ-2468] L XL High This is a high-risk customization. The plan is to remove it and adopt the new out-of-the-box sorting feature. The Issue Tracker This is your central log for all problems, questions, and unexpected behaviors encountered during the upgrade. It prevents issues from being forgotten and helps identify recurring patterns. For detailed guidance, see Issue Tracking Best Practices. Ensure that any potential product bugs identified are fed back to Quantexa Support by raising a new post in Quantexa Platform Support on the Community. The Effort Tracker Tracking the actual effort spent is crucial for refining your project plan and improving future estimates. At the end of each major stage or sprint, update a simple effort tracking table. This provides a clear comparison of planned vs. actual effort. Area / Major Task Development Days Effort Testing & Bug-fixing Days Effort Total Days Elapsed for Acceptance Comments Repository Tool & Automated Migrations 2 3 5 Repository tool failed on two modules, requiring manual intervention. Manual Migration: Document Service 4 2 6 More complex than estimated due to unforeseen impacts on custom User Interface (UI) viewers. Manual Migration: ETL scripts 2 1 3 As expected. By establishing this system of tracking from day one, you create the visibility and data-driven foundation needed to successfully navigate the execution of your upgrade.Preparing For Success
You have successfully completed the planning phase and produced a comprehensive Upgrade Plan. You know what you need to do, how you're going to do it, and how long it will take. The Preparation Stage is the critical bridge between your plan and the hands-on execution of the upgrade. This is where you set up your project for success by establishing the right technical foundations, development processes, and support channels before the first line of code is migrated. A well-executed preparation phase minimizes friction, prevents common pitfalls, and ensures your development team can work efficiently and effectively once the upgrade work begins. Who is this guide for? This guide is primarily for Upgrade Leads and Developers who will be performing the hands-on work. Project Managers will also find the section on turning the plan into actionable tickets essential for tracking progress. What will you accomplish in this stage? By the end of this stage, you will have: Established a robust tracking system for all changes, decisions, and customizations addressed during the upgrade. Assembled all the necessary software, tools, and artifacts required for the upgrade. Established a clear development and environment strategy that allows upgrade work to proceed without halting business-as-usual development. Understood the support paths and processes available to you. Translated your high-level upgrade plan into a backlog of tangible, ready-to-work development tasks. The Preparation Process The following pages provide a step-by-step guide on how to best prepare for the execution of your Quantexa Platform upgrade. Page Description Implementing Upgrade Tracking: What It Is and Why It Matters Learn how to establish a robust system for tracking the upgrade process. This involves logging all issues encountered, the effort spent against tasks, and the details of customizations addressed or introduced. Maintaining these various trackers provides an essential audit trail and helps refine future upgrade estimates. Get Your Dependencies: What They Are and How to Obtain Them The essential next step. This guide covers how to locate and download all the required software versions, Quantexa release artifacts, and tools (like the Repository Tool) that you will need. Upgrading Without Stopping: Development Strategies and Environment Planning Learn how to set up your Git branching strategy (e.g., a long-lived upgrade branch) and plan your environment usage to isolate upgrade development from ongoing business as usual (BAU) work, ensuring both can proceed in parallel. Understanding Support During Upgrades: Where to Turn When Issues Arise Before you encounter a problem, it's vital to know where to go for help. This article outlines the different support channels available, from Documentation and Community to raising a formal Quantexa support ticket. Turn Your Upgrade Plan into Action: Creating JIRA Tickets The final preparation step. This guide shows you how to take the detailed "In Scope Tasks" list from your Upgrade Plan and convert it into a well-defined backlog of JIRA tickets, ready for your team to start working on.Produce Final Upgrade Plan
This is the final step in the planning phase of your Quantexa upgrade. You have now defined your destination, chosen your approach, detailed your scope, estimated the effort, profiled your team, and created a testing strategy. The purpose of this final step is to consolidate all these outputs into a single, formal Upgrade Plan document. This document will be your master guide for the execution phase and the primary communication tool for all project stakeholders. The Anatomy of the Upgrade Plan Document Your final Upgrade Plan document should be structured, clear, and contain all the key decisions and data you have gathered so far. Use the following structure as a template, pulling in the outputs from each of the previous articles in this guide. 1. Executive Summary 1.1. Project Goal: A high-level statement defining the purpose of the upgrade. What is the current version, what is the final target version, and what are the key business or technical drivers for this project? 1.2. Key Outcomes: Briefly list the major deliverables (e.g., "Upgraded Platform to v2.8," "Migration of ETL framework," "Decommissioning of legacy components"). 1.3. High-Level Estimate & Timeline: State the total estimated effort in days and the target completion quarter. 2. Upgrade Strategy & Roadmap This section details the "what" and "how" of the upgrade. 2.1. Target Version: The clearly identified major and minor version for your Quantexa Platform upgrade. This is the output from Identify Your Destination. 2.2. Upgrade Approach: State your chosen approach: Incremental Upgrade or Reset & Rebuild. If incremental, specify the release strategy: Single Go-Live or Staged Go-Live. This is the output from Define Your Upgrade Approach. 2.3. Upgrade Roadmap: Insert the detailed, staged roadmap table you created. This should show the sequence of all major projects (platform upgrades, infrastructure updates, framework migrations) required to reach the target version. This is the primary output from Identify Your Destination. 3. Detailed Project Scope This is the most detailed section of the plan, defining the precise boundaries of the work. It is composed of the tables you built in the Define Upgrade Scope article. 3.1. High-Level Impact Analysis: Insert the "Impacted Components" table (your "heat map"). 3.2. Out of Scope: Insert the table explicitly listing all items that are out of scope. 3.3. In Scope Tasks: Insert the definitive master list table of all migrations, one-off tasks, and tech debt items that are in scope for the project. 4. Effort Estimate & Resource Profile This section outlines the cost and the team required to deliver the plan. 4.1. Effort Estimate: Insert the final estimation table, including the sub-total, contingency, and grand total effort in person-days. This is the output from Estimate The Upgrade. 4.2. Resource Profile: Describe the planned team structure. State that the plan requires an Upgrade Lead and detail the planned team size based on your chosen approach. Note how the plan will leverage a Quantexa Architect/TDO and, if applicable, a Solution Expert. This is the output from Resource The Upgrade. 5. Testing Strategy This section summarizes the plan to ensure a high-quality, stable release, based on the outputs from the Define Testing Requirements step. 5.1. Guiding Principle & Focus Areas: Testing will be risk-based and targeted at the areas of greatest change as defined in the scope (Section 3). The primary focus will be on the components identified as "High" or "Medium" impact, such as ETL and the Mid-Tier. 5.2. Core Testing Activities: The following activities will be performed to validate the success of the upgrade: Testing Pillar Key Activities & Approach Regression Testing Validate batch outputs (ETL, Resolution, Scoring) against a pre-upgrade baseline using the Statistical Profile Testing Framework (SPTF). Manually execute high-value user journey test scripts for the User Interface (UI). Run automated test suites against key API endpoints. Systems Integration Testing (SIT) Perform end-to-end batch runs triggered by the production scheduling system. Verify successful deployment via the CI/CD pipeline, including database migrations. Conduct spot-checks on security and authentication. Non-Functional Testing (NFT) Capture pre-upgrade performance baselines for key batch jobs and API response times. Post-upgrade, execute the same tests to verify that performance has not significantly degraded and remains within acceptable SLOs. 5.3. Multi-Use Case Considerations (if applicable): This plan includes specific tests to validate the interaction between the upgraded Use Case A and the non-upgraded Use Case B, focusing on the shared Centralized Data Source to ensure continued compatibility. Moving to Preparation With this comprehensive plan assembled and approved by your stakeholders, the planning phase is now officially complete. This document is your charter to move into the Preparation Phase. This next phase involves setting up the necessary project structures, environments, and tooling required to begin the hands-on work. Your Upgrade Plan will guide all preparation activities, ensuring the team is perfectly aligned before the upgrade execution begins.Testing Your Upgrade: Practical Considerations
You have successfully migrated your code to the target version, and you have a stable, building repository. Now, you will execute the Test Plan you created during the planning phase. This guide provides a practical, step-by-step methodology for development testing. The core principle is to start with fast, low-cost tests and progressively move to slower, more expensive, and more realistic validation. Do not jump straight to full data runs. Phase 1: The Foundation - Local and Automated Testing This is the fastest and most important feedback loop. The goal is to catch as many issues as possible on your local machine before ever deploying to a shared environment. Validate Automated Tests: Your first step is to ensure your existing automated test suite is working. Unit & Integration Tests: Run your entire suite of automated tests locally. Fix any failures. These tests are your first line of defense and are crucial for validating that core logic within your ETL and Scoring processes is still sound. Local Runtime Testing: Before deploying, try to run components locally. Example: Starting the Quantexa application services on your local machine can quickly reveal runtime dependency conflicts or configuration errors that were not caught at compile time. Fixing these locally is much faster than in a shared environment. Phase 2: Controlled Environment Testing Once your local tests are passing, it's time to move to a shared development or test environment. The goal here is to perform an initial end-to-end run in a controlled manner. Establish a Stable Baseline: To test for regression, you must have a stable point of comparison. Control Your Code: Ensure the main branch (your baseline) and your upgrade branch are based on the same starting commit. A common practice is to create a temporary release-candidate branch from main to use as a stable, unchanging baseline for comparison. Control Your Data: Use a static, well-understood dataset for this testing phase (e.g., a synthetic or a small, sanitized sample of real data). The input data for your baseline run and your upgrade run must be identical. Produce the Baseline Run: Execute your full end-to-end batch process on the main (or release-candidate) branch using your controlled dataset. Document the results meticulously. This includes: HDFS/S3 output locations. Elasticsearch index names. Batch job runtimes and resource profiles. Produce the Upgrade Run: Execute the same end-to-end batch process on your upgrade branch, using the exact same input data. Compare and Analyze: Compare the outputs from the upgrade run against your baseline. The Statistical Profile Testing Framework (SPTF) is the recommended tool for efficiently comparing batch outputs. Expect some changes. Bug fixes or functional improvements in the new Quantexa version can cause expected deviations. Your job is to validate that all changes are explainable and can be traced back to a specific migration or product change. Any unexplained changes are potential regressions that must be investigated. Phase 3: Full Environment Testing (UAT / Staging) Only after you are confident that most issues have been resolved should you move to a production-like environment with real, full-volume data. Full Regression & SIT: Execute your full test plan in this environment. This includes validating integration with external schedulers, security configurations, and any other real-world integrations. This is your best opportunity to catch elusive, data-specific edge cases. Performance Validation: Run your batch processes against full production data volumes. Compare the runtimes against your pre-upgrade performance benchmarks to ensure there are no significant performance regressions. User Acceptance Testing (UAT): This is a vital part of the release cycle. Block out sufficient time for business users to validate the solution and sign off on the upgrade. Factor in time to investigate and resolve any issues found during UAT and re-test. By following this phased approach, you systematically build confidence in your upgrade, ensuring that by the time you reach the most expensive testing phase, you have already eliminated the vast majority of issues in a more efficient manner.Upgrading Without Stopping: Development Strategies and Environment Planning
Before the hands-on upgrade work begins, it is critical to establish a strategy that allows the upgrade to proceed in parallel with normal, business-as-usual (BAU) development. Freezing your project's main branch for the duration of a potentially multi-week upgrade is not a viable option. This guide outlines the best practices for setting up your code branching and environment usage plans to ensure the upgrade work is isolated but not divergent, allowing both your upgrade team and your BAU team to work productively and without conflict. The Code Strategy: Isolate with a Long-Lived Upgrade Branch The core of a parallel development strategy is to isolate the upgrade work in a dedicated, long-lived branch. This creates a separate workspace for the upgrade team while allowing the BAU team to continue merging new features and bug fixes into the main branch as usual. Action: Create your upgrade branch. Create a new branch from the latest version of your main branch. A common naming convention is feature/platform-upgrade-v2.8. Action: Establish a process for staying in sync. An upgrade branch that is never updated from main will quickly become impossible to merge. It is essential to have a regular process for incorporating ongoing BAU development into your upgrade branch. This is covered in detail in the documentation's guide to Ongoing development during upgrades. The key activities for the upgrade team are: Rebase Regularly: On a consistent schedule (e.g., weekly), rebase the upgrade branch against main. This pulls in the latest BAU changes incrementally, making the final merge much simpler. Maintain a Changelog: As you work, keep a simple changelog. Note down any changes or decisions made that will impact other developers when the final upgrade is released (e.g., "The custom-function-x has been replaced with the new new-native-function-y"). Update Your Trackers: Keep your issue and effort trackers up to date. This provides real-time visibility on progress and ensures knowledge is not lost. The Environment Strategy: Plan for Shared Resources Just as your code needs to be isolated, your execution environments need to be managed to prevent conflict. Upgrade tasks, particularly full batch runs, can be resource-intensive and interfere with other development or testing activities if not properly coordinated. Action: Define your upgrade environment. Designate a specific development or test environment for the upgrade work. This ensures that the upgrade team's activities, such as running a newly migrated ETL process for the first time, do not impact BAU sprint testing in another environment. Action: Establish communication protocols for shared resources. Your project likely shares key infrastructure components like a Spark or Elasticsearch cluster across multiple teams or use cases. It is vital to establish clear rules of engagement to avoid contention. Your project should have a defined process for: Announcing when resource-intensive batch jobs will be run (e.g., via a dedicated Teams/Slack channel). Distinguishing between short, small jobs and long-running overnight jobs. Resolving prioritization conflicts if two teams need to run large jobs at the same time. For larger programmes, a brief daily stand-up focused on cluster usage can be highly effective for aligning priorities across teams. By clearly communicating intentions, you can prevent most environment-related conflicts before they happen.