2.8 Quantexa Upgrade Guide
Table of Contents Quick Upgrade Overview Community Upgrade Guidance Dependencies and Platform Modernisation User Interface Changes Batch Resolver and Entity Resolution Redis / Valkey Distributed Caching Data Packs Migration Recent Deprecations Software Compatibility Additional Information Quick Upgrade Overview The 2.8 Quantexa Upgrade consists of five main parts: Dependencies and Platform Modernisation User Interface (UI) Changes Batch Resolver and Entity Resolution (ER) Redis / Valkey Distributed Caching Data Packs Migration The majority of the dependency changes (Spark 3.5 migration, Cats Effect 3, Apache Pekko, library version bumps) are handled by automated Plop and Scalafix migrations via the Repository Tool. However, any custom code that deviates from best-practice patterns, particularly Cats Effect or Akka usage, will require manual migration. A new Quantexa license is also required before starting. The User Interface tier sees the most user-facing change. Resource Viewer reaches General Availability (GA) and replaces several Investigation modules, Transaction Viewer has been fully removed (migrated to Data Viewer), Angular has been updated to version 18, and the web-app folder structure has been simplified. Projects should plan to test the UI thoroughly post-migration. Batch Resolver receives several important changes: Network Generation and time-splitting have been removed, Classic mode is deprecated, and the Entity Attributes output model has changed. Filtered Compounds now use QSL instead of QEL. Most of these have Plop or Scalafix support, but projects with custom Batch Resolver scripts should budget additional effort. Redis / Valkey distributed caching is introduced as optional in 2.8 but becomes mandatory from 2.9 onwards. Projects are strongly encouraged to set up Redis during the 2.8 upgrade to avoid a separate infrastructure change later. Data Packs 2.3.x are compatible with Quantexa 2.8 and require Parsers 4.2.3 or later. Projects still on Parsers 3 must complete the Parsers 4 migration before upgrading to 2.8. This page aims to provide additional guidance related to the 2.8 Quantexa Upgrade. For the full list of required migration steps, please refer to the Documentation site migration guide: 2.7 → 2.8 Upgrade Migration Guide. Release Notes: Community Release Announcement 2.8 Release Notes Community Upgrade Guidance Dependencies and Platform Modernisation New Quantexa License Required A security improvement to the Quantexa license means that existing licenses are not compatible with 2.8.0. You must request a new license before deploying the upgraded platform. This is a prerequisite that should be actioned early in the upgrade process to avoid blocking deployment. Spark 3.5 Migration Support for Spark 3.3 and 3.4, deprecated in 2.7, has been fully removed. Projects must now run on Spark 3.5. The BOM changes are applied automatically by Plop. The primary manual effort comes from ensuring custom project-owned Spark scripts are compatible with Spark 3.5 and upgrading Spark installations across all environments (local, CI/CD, and production). During internal migrations, no significant behaviour changes were found, so this should be a low-risk change for most projects. Cats Effect 3 This is one of the more impactful dependency upgrades. The cats-effect, fs2, and fs2-Kafka libraries have all been updated to version 3.x. Code following the best-practice structure in Task Loading, Scoring, Alerting, Entity Store, and Data Source Integration Tests is migrated automatically. However, any custom code built upon earlier versions of these libraries, such as custom Kafka applications, must be manually migrated following the official Cats Effect 3 migration guide. Projects should use Project Example as a reference. Akka Replaced by Apache Pekko Due to CVEs raised against the Akka library, Quantexa has switched to Apache Pekko. This only affects deployments using app-graph-script. The migration is straightforward: ensure import akka.util.Timeout statements are changed to org.apache.pekko.util.Timeout. A Scalafix migration is available. Other Library Upgrades Several other libraries have been updated, all with Plop or Scalafix support: Scalatest upgraded from 3.1.4 to 3.2.19 ScalaCheck upgraded (artifact name changed) Chimney version updated Java Faker replaced by Datafaker ETL Util dependencies renamed (batch-common-api → spark-api, common-spark-utils → document-index-input-reader) spark-excel artifact and package changed spoiwo package updated from 1.x to 2.x org.apache.commons.io must now be shaded in the Batch tier JSON Configuration Validation A notable change to be aware of is that configuration validation on application startup now causes the application to fail if Perspective, Bulk Search, or Foreign Document JSON configuration files contain unknown fields. Previously, these invalid fields were silently ignored. This means that configuration errors that have existed undetected in your project may now surface as startup failures. It is recommended to run the Configuration validation tests early in the upgrade to identify and fix any issues before attempting to deploy. User Interface Changes Resource Viewer (General Availability) Resource Viewer is the most significant user-facing change in 2.8. It provides a unified, tabbed interface for viewing and interacting with Documents, Entities, and other resources. All projects upgrading to 2.8 will inherit the Resource Viewer, which replaces several existing Investigation modules including CompoundGraphViewModule, SourceViewerPluginModule, RecordViewerPluginModule, and others. These replaced modules are now deprecated and will be removed in the next major release. This introduces breaking changes for deployments using Low-Code Configuration (LCC) or Record Viewer. Projects should follow the Resource Viewer Migration Guide and plan for testing time to verify the UI behaves as expected, particularly around any custom UI configuration. Transaction Viewer Removal Following its deprecation in 2.4, Transaction Viewer has been fully removed. Data Viewer is the recommended replacement, offering feature parity plus improvements such as no record limit, column-level UI filtering, and Scoring plugin integration. Projects still using Transaction Viewer must complete this migration. Follow the Transaction Viewer Migration Guide for guidance. Angular 18 and web-app Simplification The UI has been updated to Angular 18. Plop automatically handles the migration of your deployment's UI code. However, due to the many changes to NPM dependencies, Plop may fail to migrate web-app/package-lock.json to a consistent state. If you encounter UI build errors after migration, delete the node_modules and package-lock.json files from the web-app folder and run npm install to regenerate them. The web-app folder structure has also been simplified, removing boilerplate files and simplifying how the application is bootstrapped. Tab configuration previously in MainViewComponent has moved to a dedicated tab-config.ts file. Global UI Configuration Changes The file structure used for global settings in Low-Code Configuration has changed. If your deployment uses global configuration loaded through JSON files on deployment, you must migrate to the new format. Follow the Global UI Configuration Migration Guide. Batch Resolver and Entity Resolution Network Generation and Time-splitting Removed The Network build and Create ScoringGraph stages of Batch Resolver, which produce Natural Networks, have been removed following their deprecation in 2.7. Projects still using these must migrate to Graph Scripting QSL for network generation. The associated configuration options (appendFilteredDocuments, createNetworkStats, scoringGraphMaxNodeCount, etc.) must be removed from Batch Resolver configuration files. Similarly, the time-splitting functionality has been removed. All timeSplit fields must be removed from Resolver JSON configuration. Filtered Compounds: QEL to QSL Migration The Quantexa Expression Language (QEL) expression field within Filtered Compound definitions has been replaced with a QSL where field. A Plop migration is available to translate existing QEL filters to QSL. Projects should review the translated filters to ensure correctness, particularly for complex expressions. Additionally, Batch Resolver now warns if Filtered Compound definitions reference Record Attributes not defined in the input data. These warnings will be upgraded to errors in a future release, so projects should address them during the upgrade. Note that the default Address Resolution Template in the Data Packs core fragment contains an erroneous filteredCompoundDefinitions block referencing a rootCountyCode attribute; if present, this block should be removed. Entity Attributes Output Model Changes The excluded, subEntities, and linkExcluded fields have been removed from the Batch Resolver Entity Attributes output. A new failedDataQualityChecks field has been added. A Scalafix migration is available, but due to Scalafix limitations, imports referring to the removed SubEntity model must be removed manually. Classic Mode Deprecated Batch Resolver Classic mode is deprecated and will be removed in a future release. Projects should migrate to Resolver mode. To replicate previous Classic mode exclusions, use Element exclusions in ETL. See Migrating from Classic to Resolver mode for guidance. Calling Batch Resolver and Graph Scripting QSL from Code Deprecated Calling Batch Resolver and Graph Scripting QSL stages from code is deprecated and will be removed in a future major release. Projects must migrate to using the pre-defined scripts under com.quantexa.batchresolver.scripts and com.quantexa.graphqsl.scripts directly with spark-submit. If you previously used custom scripts to insert Run IDs from Metadata into file paths, use the updated dataPaths configuration. Redis / Valkey Distributed Caching Quantexa 2.8 introduces distributed caching with Redis or Valkey. While Redis is optional in 2.8, it becomes mandatory from 2.9 onwards. We strongly recommend configuring Redis during the 2.8 upgrade so that this infrastructure is in place before upgrading to 2.9. The Repository Tool automatically adds a baseline application-redis-profiles.yml configuration file and updates docker-entrypoint scripts. Redis can be deployed via the Quantexa Helm Chart (Redis Sentinel), Google MemoryStore for QCP-hosted projects, or a manually deployed Redis/Valkey instance. To enable Redis caching, set the following feature flag in your Spring configuration: quantexa.caching.redis-caching.enabled: true For full deployment guidance, see the 2.7 → 2.8 Migration Guide, section "Distributed caching using Redis or Valkey". Data Packs Migration Data Packs 2.3.x are compatible with Quantexa 2.8.x and require Parsers 4.2.3 or later. Parsers 3 is no longer supported from Quantexa 2.8 onwards. Projects still on Parsers 3 must complete the migration to Parsers 4 before upgrading to 2.8. This migration should be treated as a separate workstream and is not recommended to be performed alongside the 2.8 platform upgrade. For planning and step-by-step guidance, see: Why & How to Plan a Parsers 4 Migration Step-by-step Guide to Migrate to Parsers 4.3 For the full Data Packs compatibility matrix, see Data Packs Compatibility Matrix. Recent Deprecations Batch Resolver Classic Mode Classic mode is deprecated and will be removed in a future major release. Projects must migrate to Resolver mode. See Migrating from Classic to Resolver mode. Elasticsearch 7 and OpenSearch 1 Elasticsearch 7 and OpenSearch 1 are deprecated in 2.8 and will be removed in 2.9. Projects should plan to upgrade to Elasticsearch 8 or OpenSearch 2. Search 1 Search 1 is deprecated. Projects should migrate to Search 2. Connect Connect is deprecated in 2.8. Quantexa Feature Engineering Quantexa Feature Engineering is deprecated. Projects using version 0.9.0 must replace their imports with their own Spark transforms. Graph Scripting DSL Following its deprecation in 2.7, the DSL Conversion to Scoring Graph methods have been removed in 2.8. The Entity Statistical Profile Testing Framework's use of DSL models is also deprecated - projects must migrate to using the CalculatedEntityAttributes Batch Resolver output. Parsers 3.x As noted above, Parsers 3.x is no longer compatible with Quantexa 2.8. Projects should complete migration to Parsers 4.x before upgrading. Software Compatibility Check out software compatibility. Key changes for 2.8: Apache Spark: 3.5.x only (Spark 3.3 and 3.4 removed) Java JDK (Application Tier): 17 Node.js: 22.11.0 npm: 10.9.0 Gradle: 8.4+ Redis / Valkey: 6.x, 7.x (optional in 2.8, mandatory from 2.9) Elasticsearch: 7 (EOL) and 8; OpenSearch 1 and 2 Kafka: 3.x, 4.x CPython: 3.10.x, 3.11.x MySQL: 8.x, 9.x PostgreSQL: 13.x to 18.x Additional Information Useful resources: Upgrade Best Practice for instructions before commencing an upgrade. Ongoing development during upgrades for best practice guidance on continuing with meaningful development during an upgrade. Quantexa Upgrade Knowledge Hub for comprehensive guidance on planning and executing upgrades. Parsers 4 Migration Guidance for planning and executing the Parsers 4 migration (required for 2.8). Resource Viewer Migration Guide Global UI Configuration Migration Guide Entity Store Load External Attributes Migration Migrating Elasticsearch Clients for Scoring Follow the Release Announcements Topic to receive notifications of releases. Quantexa Release Notes for changes and information specific to different versions of Quantexa.2.9 Quantexa Upgrade Guide
Table of Contents Quick Upgrade Overview Community Upgrade Guidance Dependencies and Platform Modernisation User Interface Changes Pre-built Images and Pre-packaged Artifacts Backend and Data Pipeline Changes Data Packs Migration Recent Deprecations Software Compatibility Additional Information Quick Upgrade Overview The 2.9 Quantexa Upgrade consists of five main parts: Dependencies and Platform Modernisation User Interface (UI) Changes Pre-built Images and Pre-packaged Artifacts Backend and Data Pipeline Changes Data Packs Migration The 2.9 release is one of the most structurally significant Quantexa upgrades to date. The headline change is the move to pre-built Docker images for App Tier applications. Rather than compiling application modules project-side, projects now use the Image builder plugin to layer project-specific extensions onto Quantexa-provided base images. This fundamentally changes the build and deployment model and drives many of the API refactoring changes across the platform. On the dependencies side, this release involves major framework upgrades: Gradle 9, Spring Boot 4, Node.js 24, and Angular 21. Elasticsearch 7 and OpenSearch 1, deprecated in 2.8, have been fully removed - projects must be on Elasticsearch 8/9 or OpenSearch 2. Elasticsearch 9 is newly supported. While the Repository Tool handles most of the automated migration, projects with custom Gradle build code, custom Spring configuration, or custom UI code should budget additional manual effort. The User Interface tier includes many configuration restructuring changes. Global UI configuration, Explorer dynamic configuration, and Search 2 configuration all move to versioned directory-based formats. The Home Page reaches General Availability (GA), several Investigation modules are removed, and PrimeNG is replaced with a Quantexa-maintained fork. Data Packs 2.3.x remain compatible with Quantexa 2.9 and require Parsers 4.2.3 or later. This page provides additional guidance for the 2.9 Quantexa Upgrade. For the full list of required migration steps, please refer to the Documentation site migration guide: 2.8 → 2.9 Upgrade Migration Guide. Release Notes: Community Release Announcement 2.9 Release Notes Community Upgrade Guidance Dependencies and Platform Modernisation Gradle 9 Upgrade This is one of the most impactful changes in the 2.9 upgrade. Gradle 9 removes several build script APIs and tightens behavioural rules that previously generated only deprecation warnings. The Repository Tool automates the migration for patterns used in Quantexa-generated build files, covering changes such as replacing configurations.all with configurations.configureEach, updating archivesBaseName to archiveBaseName.set(), and reordering sourceSets blocks. However, deployments with custom Gradle build code will almost certainly require additional manual changes. After the automated migration, run ./gradlew build and resolve any remaining failures. Common follow-up tasks include replacing usage of removed org.gradle.util.CollectionUtils methods and reviewing any include directives that the migration commented out in settings.gradle. Spring Boot 4 Upgrade The mid-tier has been upgraded from Spring Boot 3.x to Spring Boot 4.0, introducing breaking changes for all deployments. A Scalafix rule handles Scala source file migrations, and a Plop migration covers common changes such as relocating configuration keys (server.error → spring.web.error) and updating Spring Boot imports. However, the Plop migration cannot cover every Spring Boot 4 scenarios, especially for custom code. After running the automated migrations, manually review the official Spring Boot 4.0 Migration Guide for any additional changes your deployment requires. Note that moving to pre-built images also eliminates many Spring bean extension points, replacing them with Scala singleton objects (see Pre-built Images section below). Elasticsearch 7 and OpenSearch 1 Removed Following their deprecation in 2.8, Elasticsearch 7 and OpenSearch 1 have been fully removed. Deployments still on ES7 must upgrade to Elasticsearch 8 or 9 before upgrading to 2.9. Deployments on OpenSearch 1 must upgrade to OpenSearch 2. Elasticsearch 9 is now supported alongside Elasticsearch 8. Quantexa recommends upgrading to ES9 for the latest performance and security improvements. However, note the following limitations in 2.9.0: EmbeddedElasticServer is not supported for ES9 - deployments using it for local testing must migrate to Docker-based Elasticsearch Offline Indexing is currently not supported with ES9 OpenSearch 3 is not supported in this release The elastic*-builder NPM packages used by the UI for ES7 query building have also been removed. Node.js 24 and Angular 21 Node.js has been updated from version 22 to 24 (required by Angular 21). Angular has been updated to version 21. Plop automatically handles the UI code migration. As with the 2.8 Angular upgrade, Plop may fail to migrate package-lock.json to a consistent state - if you encounter UI build errors, delete node_modules and package-lock.json from the web-app folder and run npm install to regenerate them. Ensure all CI/CD pipelines, development environments, and Docker build stages reference the new Node.js version. User Interface Changes Home Page (General Availability) The Home Page reaches General Availability in 2.9, providing a configurable landing page with widgets for quick access to recent activity, tasks, and investigations. A Plop migration is available for initial setup. Projects should review their Home Page configuration and test the landing experience post-migration. Investigation Modules Removed The Investigation modules deprecated in 2.8 (replaced by Resource Viewer) have been fully removed. The affected modules include CompoundGraphViewModule, SourceViewerPluginModule, RecordViewerPluginModule, and others. A Plop migration handles the removal. Projects should have already migrated to Resource Viewer during the 2.8 upgrade. Additionally, the Custom Bulk Search component has been removed - Bulk Search configuration is now fully managed through the standard configuration approach. Global UI Configuration Migrated to Versioned HOCON The Global UI configuration has been restructured from a monolithic JSON file to a versioned directory containing HOCON configuration and a version.conf file declaring the schema version. This enables automatic schema migration during platform upgrades. Plop creates a config/global-ui/ directory structure and updates application.yml paths. Several previously unused fields (validFrom, validTo) have also been removed. Projects must verify that deployment manifests (Helm Chart or QAT) are updated to mount the new directory as a ConfigMap volume. Explorer and Search 2 Configuration Split Explorer dynamic configuration has been split from a single monolithic file into per-schema .conf files within an explorer-dynamic-configuration/ directory, with an index.conf file. Search 2 configuration follows the same versioned directory pattern, migrating from a single JSON file to config/search2/ containing search2-dynamic-config.conf and version.conf. Both Plop migrations handle the conversion, update application.yml paths, and add ConfigMap entries to deployment scripts. Projects must verify that the updated volume mounts are present in their deployment manifests. PrimeNG Fork and Dependency Cleanup PrimeNG has been replaced with @quantexa/primeng-fork, a Quantexa-maintained fork. All primeng/* imports in TypeScript, CSS, and SCSS files must be updated - a Plop migration handles this automatically. Additionally, Web Core peer dependencies have been removed from project-level package.json and are now managed by the @quantexa/web-core package. After migration, regenerate package-lock.json by deleting node_modules and the lock file, then running npm install. To keep dependency overrides up to date between major releases, use the src:cve-update script. The project-level ESLint configuration and its dependencies have also been removed. Projects that require ESLint must configure it independently. UI script and import paths have been updated: the lint, src:migrate-scss, and eslint-config:link scripts are removed, and @quantexa/script-lib imports must be updated to @quantexa/utils and @quantexa/build-tools. Alternative Network Layouts (General Availability) Alternative Network Layouts, which provide different ways to visualise entity networks, reach General Availability in 2.9. A Plop migration renames edge configuration properties. Projects using custom network layout configuration should review the updated property names. Localisation Configuration Restructured The localisation configuration has been restructured for multi-language support. This is a breaking change with Plop migration support. Projects should verify their localisation setup post-migration. Low-Code Configuration (LCC) Validation Tests LCC validation tests have been migrated to a new location. A Plop migration updates test locations and imports. Continue to run configuration validation tests early in the upgrade process to catch configuration errors before deployment. Pre-built Images and Pre-packaged Artifacts Application Pre-built Docker Images This is the most architecturally significant change in 2.9. Applications are no longer compiled project-side. Instead, Quantexa provides pre-built Docker images for App Tier services (all applications except the UI and app-fusion). Projects use the Image builder plugin to layer project-specific extensions onto these base images. The migration involves seven steps, which must be completed in order: Migrate application extensions to dedicated modules (Manual) - custom code in application modules (e.g. app-security/src/main/, app-investigate/src/main/) must be moved to dedicated extension modules with recognized naming conventions that the Image builder plugin auto-discovers. Register Foreign Documents for Fusion data configuration (Manual) - applications now read data configuration from fusion-data-config.json rather than compiled Scala models. Foreign Documents not automatically included must be registered via .qdocuments and .qmodel files. Migrate SAML metadata files (Plop) - the classpath: protocol for SAML metadata is no longer supported; XML files must be moved from app-security/src/main/resources/ to a mounted config/saml/ directory. Migrate optional modules to feature flags (Plop) - optional service dependencies in build files are replaced with feature flags in deployment configuration. Add Image builder plugin configuration (Plop) - adds the plugin to the root build and creates the applications.gradle file. Migrate each application to pre-built images (Plop) - replaces each application module with a .gitkeep, updates settings/libraries/applications Gradle files. Update deployment configuration (Manual) - applications with project-specific extensions are renamed with a -project suffix (e.g. app-resolve becomes app-resolve-project). Deployment configuration must reference the new names. This migration has a cascading impact across the platform. Many of the API changes in Visualization and Exploration (see below) are directly driven by this change - extension points that previously relied on Spring bean registration in compiled application modules must now use Scala singleton objects in dedicated extension modules. Projects should follow the Application Pre-built Image Migration Guide carefully. For projects that need help adopting container-based deployment, speak to your Quantexa Architect. They can help assess your current deployment model and plan the transition to pre-built images. Impact on Local Development The move to pre-built Docker images changes the local development workflow. Previously, developers could run mid-tier applications locally using bootRun or java -jar commands. With pre-built images, these applications are no longer compiled project-side, so the traditional local run scripts no longer apply for most App Tier services (the UI and app-fusion are still built project-side). There are three options for development workflows going forward: Cloud-based or development environment development (recommended) - develop against a shared or personal Kubernetes development namespace with CI/CD pipelines configured for rapid iteration. This is the approach Quantexa uses internally and is the recommended path. Projects should work with their Quantexa Architect to set up development namespaces that support concurrent developer activity, including sizing the environment to handle multiple developers working in parallel. Local container-based development - use Minikube, Docker Desktop, or an equivalent local Kubernetes environment to run pre-built images on a developer laptop. This preserves the ability to work locally but requires container tooling to be available on the development machine. The overhead of Docker itself is relatively low, but the growing footprint of the full application stack may be challenging on lower-specification machines. Projects should verify that laptop or VDI specifications are sufficient for this approach. Pre-built boot JARs (fallback) - pre-built Spring Boot JARs are available for projects that cannot adopt container-based development locally. However, this means local development uses a different deployment pattern to higher environments, requiring the project to maintain two deployment approaches and accept the risk of environment divergence. The same structural migration steps (dedicated extension modules, Image builder plugin configuration) still apply to ensure compatibility with repository migrations in later Quantexa versions. Projects upgrading to 2.9 should factor the development workflow transition into their migration effort estimates. For environments with strict change management requirements where cloud-based development namespaces are not readily available, discuss options with your Quantexa Architect early in the upgrade planning process. Pre-packaged Batch Shadow JARs Quantexa now provides pre-packaged shadow JARs for Batch tier tasks, eliminating the need to compile batch platform code from source. A Plop migration updates project structure and build configuration. Projects with custom batch tasks should review the migration to ensure their custom tasks integrate correctly. Backend and Data Pipeline Changes Data Fusion and ETL Several changes affect the Data Fusion and ETL pipeline: ENGFull renamed to BatchFull (Plop) - generated ETL script classes have been renamed from ENGFull{Model}ETLScript to BatchFull{Model}ETLScript. Template files have similarly changed from runAllENG.sh to runAllBatch.sh. Metadata excluded from hash ID calculation (Plop) - the metaData field is now excluded from hash ID computation by default when using hashIdField configuration. This improves Entity Resolution (ER) stability during incremental ETL runs. To preserve the previous behaviour, add metaData to the including list in your hash ID configuration. fusion-spark-utils removed - projects using this library must migrate to the replacement APIs. Document type validation against Fusion config (Manual) - document type definitions in Resolver configuration are now validated against Fusion configuration at startup. If a Document type is defined in Resolver but missing from the corresponding .qdocuments file, the application will fail to start with an IllegalStateException. Resolver and Batch Resolver Several Resolver APIs have been cleaned up: liftAttributes removed (Manual) - the liftAttributes configuration option has been removed without replacement. This changes how Record Attributes with mixed types are handled across Document types. Projects should check Batch Resolver logs for Record attributes to lift: entries to identify impact. Path methods removed (Manual) - several Path convenience methods have been removed from the Batch Resolver API. Resolver configuration cleanup (Plop) - several deprecated fields have been removed: dataModelRootClass, dataModelRecordExtractor, intelligenceModelClass, intelligenceModelExtractor from document/intelligence definitions; icon from entity definitions; availableIcons from UDI document definitions. Entity Store Key changes to the Entity Store: Custom External Attributes (Manual) - must now be defined in a dedicated custom-external-attributes module and configured via the external-attributes.calculations-object property. Previously, these could be defined in any module and injected as a Spring Bean. Custom Attribute Functions: ValidatedNel → Either (Manual) - the return type has changed from ValidatedNel to Either[List[String], T], and the expectedFields and name variables have been removed from the relevant traits. Custom Data Quality Functions (Manual) - the EntityQualityCheckAggregation interface now requires usedEntityAttributes and usedRecordAttributes properties. Error handling changes (Manual) - the Entity Store REST API now returns clearer error responses (validation errors return 400, timeouts return 503). Deployments with custom error handling logic must update their implementation. Change Log metadataPath (Plop) - Entity Store Change Log, CRUD Log, Document Log, and Batch ETL to Stream scripts now require a metadataPath configuration property. Scoring The elastic*-builder Gradle dependency has been removed. The Scoring Dynamic subproject requires migration to a new structure; follow the Scoring Dynamic Subproject Migration Guide. Both have Plop support, but deployments with custom Scoring configuration should review changes carefully. Data Integration Several Data Integration changes: Kafka Ingest error handling configuration (Manual) - error handling configuration has been refactored as part of the pre-built image migration, moving to a dedicated extension module. RecordExtractionModelConfig refactored (Manual)-RecordExtractionModelConfig has been replaced by RecordExtractionModelConfigBuilder. Search Related Entities configuration (Manual) - configuration has been updated as part of the pre-built image migration. Combined Ingest service removed - the combined ingest service has been fully removed. DatasourceTestSpec deprecated - DatasourceTestSpec is deprecated in favour of ETLTestSpec. Additionally, Elasticsearch-backed data source integration tests (GenericFullEtlRun ES variants) have been removed. Q AI - Q Assist Q Assist Workspace Agents reach Public Preview in 2.9. LLM configuration for Q Assist has been updated - projects using Q Assist must review and update their LLM provider configuration. This is a manual change. Decision Systems Decision Systems 2.9.0 includes breaking changes. Key changes include: initialTaskGraph is now required, Hub and Spoke Scores have been removed, and Contextual Data Mapping has been converted to Monitored Data. See the 2.8 → 2.9 Decision Systems Migration Guide for step-by-step guidance. Data Packs Migration Data Packs 2.3.x are compatible with Quantexa 2.9.x and require Parsers 4.2.3+, 4.3.x, or 4.4.x. Parsers 3 has not been supported since Quantexa 2.8. If your project has not yet completed the Parsers 4 migration, complete it before upgrading. For planning and step-by-step guidance, see: Why & How to Plan a Parsers 4 Migration Step-by-step Guide to Migrate to Parsers 4.3 There are no Data Packs-specific breaking changes in the 2.9 upgrade beyond what is covered by the platform migration. Projects already on Data Packs 2.3.x with Parsers 4 can continue using the same version. For the full Data Packs compatibility matrix, see Data Packs Compatibility Matrix. Recent Deprecations PyQSS Import Path The PyQSS import path has been deprecated. Projects using PyQSS should update their imports to the new path. I/O Handler Environment Variables I/O handler environment variables have been deprecated in favour of updated configuration. smart_open Library The smart_open library has been deprecated. icon Property in Resolver and Fusion Document Definitions The icon property in Resolver and Fusion Document definitions is deprecated. While the property still functions, it will be removed in a future release. DistinctValue for List Type Fields Using DistinctValue for List type fields is deprecated. DatasourceTestSpec The DatasourceTestSpec class is deprecated in favour of ETLTestSpec. Bundled libpostal Bundled libpostal in app-kafka-record-extraction and app-search-related-entities images is deprecated. Projects relying on the bundled version should plan to provide their own libpostal installation. transliterateElements The transliterateElements configuration has been deprecated. Software Compatibility Check software compatibility. Key changes for 2.9: Apache Spark: 3.5.x Java JDK (Application Tier): 17 Node.js: 24.15.0 (upgraded from 22) npm: 11.12.1 (upgraded from 10.9.0) Gradle: 9.3.1+ (upgraded from 8.4+) Elasticsearch: 8 and 9 (ES7 removed); OpenSearch 2 (OS1 removed) Redis / Valkey: Redis 6.x, 7.x, 8.x and Valkey 7.x, 8.x, 9.x (now mandatory) Kafka: 3.x, 4.x CPython: 3.11.x, 3.12.x (3.10.x removed) Scala: 2.12 MySQL: 8.x, 9.x PostgreSQL: 13.x to 18.x Additional Information Useful resources: Upgrade Best Practice for instructions before commencing an upgrade. Ongoing development during upgrades for best practice guidance on continuing with meaningful development during an upgrade. Quantexa Upgrade Knowledge Hub for comprehensive guidance on planning and executing upgrades. Application Pre-built Image Migration Guide for the step-by-step guide to migrating to pre-built Docker images. Scoring Dynamic Subproject Migration Guide Angular 21 Migration Guide Upgrading from Elasticsearch 8 to Elasticsearch 9 2.8 → 2.9 Decision Systems Migration Guide 2.9 Release Note Walkthrough video walkthrough of the 2.9 release. Parsers 4 Migration Guidance for planning and executing the Parsers 4 migration. Data Packs Compatibility Matrix Follow the Release Announcements Topic to receive notifications of releases. Quantexa Release Notes for changes and information specific to different versions of Quantexa.Quantexa Upgrade Knowledge Hub Release
Introducing the Quantexa Upgrade Knowledge Hub We are excited to announce the launch of the Quantexa Upgrade Knowledge Hub, a comprehensive collection of articles and best practices designed to make your platform upgrade process more transparent, predictable, and efficient. This new section on the Community site provides a step-by-step journey through every phase of an upgrade, from initial strategic planning through to final release. It's filled with practical advice, templates, and real-world considerations to help you succeed. Make sure you're signed in to the Community to access all the articles. What's Included in the Knowledge Hub? The Knowledge Hub is structured into clear, chronological sections that mirror the lifecycle of an upgrade project. Planning for Success This initial section covers all the strategic planning required before you begin. It guides you through defining your scope, choosing an upgrade approach, estimating effort, resourcing your team, and creating a robust testing strategy. The Preparation Stage Once your plan is in place, this section details the critical steps to prepare for execution. Learn how to set up tracking, gather your dependencies, establish a parallel development strategy, understand support channels, and turn your plan into actionable JIRA tickets. The Execution Stage This section provides the hands-on technical guidance for performing the upgrade itself. It covers how to execute an incremental upgrade hop-by-hop, practical considerations for testing, and a checklist for coordinating the final merge of your code. Version-Specific Guidance Dive into practical reviews and "what to watch out for" articles for specific Quantexa versions, including a detailed look at the 2.6 upgrade and the key migrations involved. Other Key Topics Explore important related subjects, such as the strategic benefits of migrating from DSL to QSL for graph scripting and how to reduce technical debt by removing Data Pack customizations using Fusion Extensibility. How Will This Knowledge Hub Help Your Team? This collection of resources is essential for anyone involved in a Quantexa upgrade: Upgrade & Delivery Leads can use it to structure their entire project plan and ensure no steps are missed. Developers & Data Engineers can follow the practical, hands-on guidance for technical migrations and testing. Architects & Product Owners can gain awareness of the strategic decisions, effort, and best practices involved in maintaining a current and healthy platform.2.6 Quantexa Upgrade Guide
Quick Upgrade Overview This article provides a practical review of the Quantexa 2.6 upgrade, highlighting key areas to watch out for based on collective project experience. It should be used as a strategic companion to the official technical guides. Official Migration Guide: 2.5→ 2.6 Upgrade Migration Guide Release Information: Community Release Announcement | 2.6.0 Release Notes The 2.6 upgrade has several non-negotiable prerequisites. Before beginning, it is essential to confirm that your project has already completed the following: All Data Sources on Data Fusion: If you have any data sources on the legacy Lenses framework, they must be migrated first. The full process is detailed in Migrating to Data Fusion. Scoring on Assess Framework: For projects still on Scoring Framework 1.0 (SF1), migrating to Assess is mandatory. This is not a direct technical conversion; it requires upfront design work to identify the right course for your project. The outcome could be to adopt modern Detection Packs if there is a good fit with your existing scoring logic, or it could be to re-implement your logic as custom scores within the Assess framework. It is strongly recommended to discuss your approach with a Quantexa Architect to determine the best path forward. For guidance on the technical migration, see Migrating from Scoring Framework 1.0 to Assess. Fusion-Compatible Data Packs: Ensure all Data Packs used by your project are the modern, Fusion versions. Check the Data Pack Compatibility Matrix to confirm your versions are compatible with Quantexa 2.6. Community Upgrade Guidance Core Product Changes Delta Lake Configuration: A straightforward configuration change in the spark-submit command. The key consideration is ensuring the Delta Lake version in your dependency-versions.gradle file is compatible with your Spark version, which can be verified on the Delta Lake release page. Explorer-*.json Configs: Simple manual edits are required to align with updated JSON schemas in some Batch Resolver configuration files. Batch Resolver Changes: A straightforward manual config change in reference.conf. The resolver now generates more output files; while a compatibility script is provided, it may be cleaner in the long run to update any downstream test code to handle the new format directly. Graph Script and Assess Imports: This is handled automatically by the repository tool, which refactors the library imports. Assess Changes: This area typically requires significant attention. The Batch Resolver data models have changed, which has a direct impact on Assess. If your scoring logic uses Entity Attributes, expect to refactor custom steps, as some fields have been removed or had their data types changed. Other Migrations Java 17 Upgrade: This is a significant undertaking that extends beyond the codebase. It requires upgrading the JDK across all environments: local developer machines, Docker images, CI/CD runners, and the production Spark clusters. It is crucial to engage with platform and infrastructure teams early to plan this. Gradle 8 Upgrade: The migration to Gradle 8.4 can be complex. An efficient approach is to generate a clean project using the 2.6 Repository Tool and use its working Gradle setup (build.gradle, settings.gradle, etc.) as a reference for your own project. LiteGraph to ScoringGraph Migration: A key migration with both automated and manual steps. This is a worthwhile effort, as ScoringGraph unlocks significant new functionality, most notably support for Entity-to-Entity edges and compatibility with the new Attribute types from the updated resolver configs. The repository tool handles much of the refactoring, but manual intervention is still required. Following the official Migrating to Scoring Graph guide closely is recommended. Alert Scorecard Migration: It is critical not to overlook this one-off data migration script for historical alert data, which is necessary to ensure re-alerting performs correctly once upgraded. Refer to Migrating from Alerting 2.5.x to 2.6.x for detailed guidance. Task View table Migration: This migration updates the database schema for the Task Data visible in the UI, typically by adding two new columns. The process depends on your database technology. For RDBMS systems, this involves running DDL migration queries. Crucially, these DDL scripts are not included in the main dependency bundle; they are provided in a separate ZIP file that must be downloaded from the Quantexa artifact repository. Additional Information For more general best practices on topics such as setting up development strategy, testing your migrated code, and releasing your upgrade, please refer to the other articles within the Upgrade Platform Library on the Quantexa Community site.
Adopting Graph Scripting QSL
As of versions 2.7.18 and 2.8.2, Graph Scripting QSL is the new, generally available standard for batch-based Network Generation, replacing the now-deprecated Graph Scripting DSL. The benefits of migrating to QSL are significant, including major performance gains, simplified low-code development, and alignment with the future direction of the Quantexa Platform. For a full overview of these benefits, please see our introductory article: Introducing Graph Scripting QSL: Faster, Smarter Graph generation capabilities. This article is the practical follow-up, designed to help Technical Leads and Developers plan and execute the migration from DSL to QSL. When to Plan Your QSL Migration With DSL now deprecated, all projects using it must plan for a migration. The timing of this migration depends on your project's current state: Project Status Recommendation New Projects All new projects starting on platform version 2.7.18+ or 2.8.2+ should use QSL by default. Existing Deployments (pre-v2.7) The migration from DSL to QSL should be planned as a key activity within your v2.7 (or later) upgrade project. Existing Deployments (on v2.7+) The QSL migration can be prioritized and executed as a standalone project, independent of a major platform upgrade. Migration Approach and Effort Planning The official DSL to QSL migration process involves setting up a new, clean QSL configuration and then re-implementing your existing DSL expansion logic using the new path-based syntax. This is not an automated, in-place conversion. When planning for this effort, use the following as a baseline estimate: Core Implementation: On average, projects require ~6 days of effort to perform the initial QSL setup and re-implement 1-3 expansion graphs, including initial development testing. Additional Expansions: The effort does not scale linearly. Budget an additional 0.5-1 day of effort for each additional expansion path you need to migrate. Formal Testing: Budget an additional 5 days for a full regression testing cycle. This allows time to thoroughly validate the graph outputs and investigate any differences. Historically, reported differences have typically been the result of rectifying undesirable DSL behavior rather than issues with QSL itself. Key Considerations During Migration As you plan your migration, be aware of these key technical differences between DSL and QSL: Single Scoring Graph: QSL generates a single ScoringGraph. If your multi-use-case setup relies on different graphs for different use cases, you may need to configure multiple, independent graph-scripting modules. Path-Based Expansions: QSL's path-based expansions are more precise than DSL's perimeter-based approach. For very complex DSL graphs, this may require you to define your logic as a larger number of more specific QSL paths. Attribute Availability: Attributes not explicitly used in an expansion are not automatically carried through to scoring traversals. Review the migration notes on attributes to ensure your scoring logic has access to the data it needs.Data Packs: How To Remove Customisations During an Upgrade
If your project has been running for some time, it is likely that you have had to customize a Quantexa Data Pack to meet specific requirements. Historically, this required "forking" the Data Pack repository and maintaining your own version of the code. While this approach provided flexibility, it came at a significant cost: Increased Technical Debt: Your team became responsible for maintaining a growing fork of custom code. Higher Upgrade Effort: Every platform upgrade required you to manually migrate and re-test your forked Data Pack, a complex and time-consuming process. Missed Opportunities: Your forked version did not benefit from the continuous stream of new features, performance improvements, and bug fixes being added to the official Quantexa Data Pack releases. Starting with platform version 2.6, Fusion Extensibility provides a powerful new model that allows you to apply the most common customizations without forking the code. This is a fundamental shift that enables you to stay aligned with the official Data Pack releases while still meeting your unique project needs. For more details on this framework, see the official Fusion Extensibility documentation on the Documentation Site. The Opportunity: When to Migrate Your Customizations If your project is on version 2.6 or later and you are still maintaining a forked Data Pack, you have a strategic opportunity to eliminate this technical debt. However, because migrating from a fork to the official Data Pack will introduce significant (and desirable) changes to your ETL output, it is strongly recommended to treat this migration as a standalone project, separate from a major platform upgrade. Attempting to do both at the same time makes it extremely difficult to perform root cause analysis. If you see a change in your data, is it because of the platform upgrade or because of the Data Pack migration? By separating the two projects, you can isolate the variables and test each change independently, leading to a much smoother and lower-risk process. The Migration Process: From Fork to Extensibility The migration process involves a careful review and triage of every customization you have made. Step 1: Review Your Forked Data Pack Go through your forked repository's commit history and create a detailed list of every functional change you have made compared to the original Quantexa version. Step 2: Triage Each Customization For every single customization you have identified, you must now make a critical decision. Ask Yourself... If the answer is YES... If the answer is NO... 1. Is this customization achievable using the new Fusion Extensibility patterns? Great! Your action is to plan to re-implement it using the new model. Go to question 2. 2. Is this customization still a critical business requirement? Your action is to acknowledge that you must remain on your forked version for now. You should then raise an enhancement request in the Quantexa Product Roadmap to have this capability added to the core product in the future. This will unblock your migration later. Excellent! This is an opportunity to simplify. Your action is to plan to discard the customization and use the standard, out-of-the-box Data Pack functionality. Step 3: Execute Your Migration Plan Based on the outcome of your triage, your path is now clear: If all of your critical customizations can be achieved via extensibility or have been deemed no longer necessary, you can proceed with re-implementing them and decommissioning your forked repository. If even one critical customization cannot be achieved, you must pause the migration, raise the necessary enhancement requests, and continue to maintain your fork until the product supports your needs. The Testing Strategy: Focus on Intent, Not Identical Output This is the most critical concept to understand when testing your migrated Data Pack. You should NOT be testing for identical output. Your old, forked Data Pack is likely behind the latest official version. The new version contains numerous bug fixes, functional improvements, and performance enhancements that will naturally and correctly lead to different ETL output. Your testing strategy must instead focus on validating the intent of your original customizations. If your customization was to... Your new test should verify that... Add a new "Risk Score" derived field... The "Risk Score" field is still being correctly calculated and added to the model, even if other parts of the record have changed due to product improvements. Exclude compounds based on a specific pattern... The correct compounds are still being excluded based on your custom logic. Add a new "Source System ID" attribute to an Entity... The "Source System ID" attribute is still present on the resolved Entities with the correct value. By adopting this testing mindset, you embrace the benefits of the new Data Pack version while ensuring your critical business logic remains intact. The result is a dramatic reduction in custom code, a simpler and faster future upgrade path, and immediate access to the latest Quantexa features.2.7 Quantexa Upgrade Guide
Quick Upgrade Overview The 2.7 Quantexa Upgrade consists of three main parts: Core Product Changes Removal of Quantexa Incubators Data Packs Migration Most of the Core Product changes are automated migrations and minor adjustment which can be tested in a local environment (unless your project has Data Streaming, Entity Store, or Graph API configured). Migration to Delta Lake is going to be the biggest component but it is also assisted by automated migrations and some of the required effort can be avoided (please see below). Quantexa Incubators will no longer be released in 2.7. Some of the utilities previously released as part of Incubators moved to the Core Product code, the rest will still be accessible (e.g. for code forking) in the 2.6 Quantexa Incubators release. It is expected that most effort will be consumed by the Data Generator migration, however, functional changes can be deferred if there is a strong need to minimize the upgrade timeline. The bulk of the Upgrade effort comes from the recommended regression testing of functional changes introduced by the Data Packs related migrations. Although the recommendation is to perform all available migrations, those functional Data Packs changes can be deferred if faced with severe time constraints (please see below). This page aims to provide additional guidance related to the 2.7 Quantexa Upgrade, for the full list of required migration steps, please refer to the Documentation site migration guide: 2.6 → 2.7 Upgrade Migration Guide. Release Notes: Community Release Announcement 2.7 Release Notes Community Upgrade Guidance Core Product Changes Migration to Data Lake This component is assisted by automated migrations and should require fairly simple configuration adjustments. It is recommended to perform a full end-to-end run once the migration steps are finalised. A significant part of the effort comes from handling metadata files, especially ones existing already in Production environments. A script is provided which converts legacy metadata Parquet files to Delta Lake format. However, that step is only required if an incremental mode iteration persists from pre-2.7 batch runs, as this requires information from existing metadata. If completing a full ETL batch run following an upgrade to 2.7, this step is not required, and a metadata.delta file is created automatically, provided the initial migration steps have been completed. Removal of Quantexa Incubators ETL validation tools migrated into the platform Elasticsearch snapshot scripts migrated into the platform ETL test utilities migrated into the platform Data generator core moved into Project Example Dynamic Graph Script utilities Graph Script REST API starter Spark, Scala, and Test Analytics utilities SparkTestSuite From 2.7 Quantexa will stop releasing the community repository Quantexa Incubators. Components that have been accepted as best practice have been migrated into the Core Product. All previously released utilities will remain accessible (e.g. for code forking) in the 2.6 Quantexa Incubators release. Most of the above are straightforward migrations with no functional changes and can be performed and tested in a local environment. Data generator core moved into Project Example In 2.7 Data Generator moves into the Core Product where it will benefit from General Availability support. All projects are strongly recommended to perform this migration (please consult your Quantexa Architect if you are considering omitting it). Dynamic Graph Script utilities This will apply to very few projects that do not follow the typical Batch Resolver + DSL network generation process. We recommend consulting your Quantexa Architect if this is the case to ensure this migration is well executed and in line with Quantexa's Best Practice. Data Packs Migration 2.7 Data Packs come in two flavours, either with or without functional changes. For the full description of functional as well as non-functional changes, please refer to the Data Packs Release Notes. In order to fully benefit from the improvements introduced in 2.7 Data Packs, we recommend opting for the version containing functional changes. However, projects should be mindful that doing so will extend the overall timeline of the upgrade. The additional effort will vary between projects, depending on the available regression testing setup, the ease of performing end-to-end batch runs, the need for model governance, etc.. Release 2.7_1.3 This is a release of Data Packs that is compatible with Q2.7 and Parsers 3. This release will contain no new functionality compared to the 2.6_1.3 release and therefore provides projects the option of performing no/limited regression testing when upgrading. Release 2.7_2.1 This is a release of Data Packs that is compatible with Q2.7 and Parsers 4. This release contains no new functionality compared to the 2.6_2.1 release and therefore provides projects the option of performing no/limited regression testing when upgrading. Release 2.7_2.2 This is a release of Data Packs that is compatible with Q2.7 and Parsers 4. This release contains all new functionality developed by the Data Packs team. Please follow the guidance below in order to make a decision around which release of Data Packs to use: Using Parsers 3 If the project is utilising Parsers 3 then they must use the 2.7_1.3 release. The project can then make a risk-based decision on the level of regression testing performed given there is no new functionality contained within this release. This is currently the last planned version of Data Packs that will be compatible with Parsers 3 and therefore there is an inherent assumption that projects will need to migrate to Parsers 4 prior to upgrading to Quantexa 2.8 if they use Data Packs. Using Parsers 4 If the project is utilising Parsers 4 then the default option should be to use the 2.7_2.2 release. Projects should only consider using the 2.7_2.1 release if the following criteria are met: The project is performing an incremental upgrade from QE2.6 to QE2.7, with no plans to update to Q2.8 in the near future. There is ongoing Quantexa presence within the Delivery Team for the project If the above criteria is met, then the option should be discussed with the project Architect and/or Technical Delivery Oversight, and a decision made as a project team. There will not be a 2.8_2.1 release and therefore projects that utilise the 2.7_2.1 release will be introducing two sets of functional changes when upgrading to QE2.8 (i.e. the changes in the 2.7_2.2 and 2.8_2.3 releases) Recent Deprecations Transaction Viewer As of version 2.4 of The Quantexa Platform, Transaction Viewer has been deprecated and will be removed in a future major release. Data Viewer is the recommended replacement for viewing records from your data sources, which is powered by the Explorer API. Please refer to the Data Viewer Migration Guide for more details. Quantexa Parsers 3.X 3.X version of Quantexa Standard Parsers and Quantexa Data Models has been deprecated and will be removed in a future major release (support will be removed in August 2025). The Parsers 4 migration is not currently advised for projects where any of the following applies: TBML projects (or any project using a parsed business name field as part of a document ID) Correspondent banking projects Projects using the entity-level alerting framework We also advise against migrating to Parsers 4.X alongside any other QP product upgrades or migration. Delivery and DEA are jointly working on ways to reduce risk and effort around this migration, including more extensive testing internally. They will agree and document best practice approach for this migration, including comprehensive guidance for testing (based on benchmarking tests) and supplementary documentation for manual changes as well as guidance for project teams to leverage Entity Resolution (ER) improvements from Parsers 4. More details available in the Quantexa Parsers Release Notes. Software Compatibility Check this page for software compatibility. Please note that Java 8 is no longer supported in Quantexa 2.7. Additional Information Useful resources on the Quantexa Documentation Site: Upgrade Best Practice for instructions before commencing an upgrade. Ongoing development during upgrades for best practice guidance on continuing on with meaningful development during an upgrade. Follow the Release Announcements Topic to receive notifications of releases. Quantexa Release Notes for changes and information specific to different versions of Quantexa.How to Release Your Upgrade: Approaches and Considerations
You have successfully tested your upgrade and are now dev-complete. The final stage of the execution process is to plan and execute the merge of your upgrade branch back into the main codebase, making it the new standard for all future development. A smooth merge is the result of careful coordination and communication. This guide provides a step-by-step checklist to ensure your upgrade is integrated without disruption. Step 1: The Final Polish - Apply the Latest Patch Before you begin the sign-off and merge process, ensure you are integrating the best possible version of the software. It is strongly recommended to update your upgrade branch to the latest available minor version or patch for your target release (e.g., if you upgraded to v2.8.2, check if v2.8.3 is now available). These releases contain the latest bug fixes and security updates but no breaking changes, making them low-risk to apply. You can find instructions for this process in the documentation on applying a minor upgrade. Step 2: Gain Stakeholder Sign-Off Before planning the merge, you must get formal approval from all relevant stakeholders. Do not wait until the last minute for this; seek sign-off as soon as individual components are ready for review. Action: Secure approval from the following groups: Product Owners / Business Users: Present your UAT results and regression reports to demonstrate that the functionality is correct and performance is acceptable. Technical Leads / Architects: Get your code reviewed. Your Pull Requests should be approved well in advance of the target merge date to allow time for feedback and iteration. Downstream Consumers: If other teams or applications consume your project's APIs or data outputs, ensure they have been part of the testing and have signed off on the changes. Pro-Tip: If your ETL upgrade was completed first, get it reviewed and signed off while the rest of the upgrade is still being tested. Step 3: Prepare and Educate Your Users Proactive communication is key to a smooth transition for everyone involved with the project. Action: Prepare your user groups for the upcoming change. For Technical Users (Developers): Produce a clear changelog or technical release notes. Inform them of any changes to development environments, deployment strategies, or key data structures that will impact their day-to-day work once the merge is complete. For End-Users (Investigators, Analysts): Provide an updated user guide or release communication that documents any major changes to the User Interface (UI) or application functionality that they will see in the next production release. Step 4: Plan the Merge Logistics This step involves defining the precise technical strategy for merging your code. Action: Define your merge plan, considering project structure. For Split Repositories: Your merge plan must respect inter-repository dependencies. For example, a shared "model" repository must be merged, published, and consumed by the "ETL" and "Apps" repositories before they themselves can be merged. Document the exact order of operations. For Multi-Use Case Platforms: Refer back to your upgrade plan. Ensure that the merge of your upgraded use case will not negatively impact the build or development process for other use cases running on the same platform. Step 5: Coordinate and Execute the Merge to Main This is the final, coordinated event to integrate the upgrade into your main codebase. Action: Follow this checklist for a smooth merge: Set a Merge Date: Align with all development teams on a specific date and time for the merge. Communicate a Merge Freeze: Announce a "merge freeze" for the main branch leading up to the agreed-upon merge time. This prevents last-minute, unrelated changes from introducing conflicts or instability. Prepare a Rollback Strategy: Have a documented plan in place for how you would revert the merge commit (e.g., using git revert) in the unlikely event of a critical, unexpected issue post-merge. Execute the Merge: With all preparations in place, merge your feature/platform-upgrade branch into main. Post-Merge Communication: Announce to all technical teams that the merge is complete. All new feature development must now branch from the updated main branch. With the merge complete, your upgrade is now part of the main line of development. It will be deployed to production as part of your project's next standard release, following your organization's established release playbook.Testing Your Upgrade: Practical Considerations
You have successfully migrated your code to the target version, and you have a stable, building repository. Now, you will execute the Test Plan you created during the planning phase. This guide provides a practical, step-by-step methodology for development testing. The core principle is to start with fast, low-cost tests and progressively move to slower, more expensive, and more realistic validation. Do not jump straight to full data runs. Phase 1: The Foundation - Local and Automated Testing This is the fastest and most important feedback loop. The goal is to catch as many issues as possible on your local machine before ever deploying to a shared environment. Validate Automated Tests: Your first step is to ensure your existing automated test suite is working. Unit & Integration Tests: Run your entire suite of automated tests locally. Fix any failures. These tests are your first line of defense and are crucial for validating that core logic within your ETL and Scoring processes is still sound. Local Runtime Testing: Before deploying, try to run components locally. Example: Starting the Quantexa application services on your local machine can quickly reveal runtime dependency conflicts or configuration errors that were not caught at compile time. Fixing these locally is much faster than in a shared environment. Phase 2: Controlled Environment Testing Once your local tests are passing, it's time to move to a shared development or test environment. The goal here is to perform an initial end-to-end run in a controlled manner. Establish a Stable Baseline: To test for regression, you must have a stable point of comparison. Control Your Code: Ensure the main branch (your baseline) and your upgrade branch are based on the same starting commit. A common practice is to create a temporary release-candidate branch from main to use as a stable, unchanging baseline for comparison. Control Your Data: Use a static, well-understood dataset for this testing phase (e.g., a synthetic or a small, sanitized sample of real data). The input data for your baseline run and your upgrade run must be identical. Produce the Baseline Run: Execute your full end-to-end batch process on the main (or release-candidate) branch using your controlled dataset. Document the results meticulously. This includes: HDFS/S3 output locations. Elasticsearch index names. Batch job runtimes and resource profiles. Produce the Upgrade Run: Execute the same end-to-end batch process on your upgrade branch, using the exact same input data. Compare and Analyze: Compare the outputs from the upgrade run against your baseline. The Statistical Profile Testing Framework (SPTF) is the recommended tool for efficiently comparing batch outputs. Expect some changes. Bug fixes or functional improvements in the new Quantexa version can cause expected deviations. Your job is to validate that all changes are explainable and can be traced back to a specific migration or product change. Any unexplained changes are potential regressions that must be investigated. Phase 3: Full Environment Testing (UAT / Staging) Only after you are confident that most issues have been resolved should you move to a production-like environment with real, full-volume data. Full Regression & SIT: Execute your full test plan in this environment. This includes validating integration with external schedulers, security configurations, and any other real-world integrations. This is your best opportunity to catch elusive, data-specific edge cases. Performance Validation: Run your batch processes against full production data volumes. Compare the runtimes against your pre-upgrade performance benchmarks to ensure there are no significant performance regressions. User Acceptance Testing (UAT): This is a vital part of the release cycle. Block out sufficient time for business users to validate the solution and sign off on the upgrade. Factor in time to investigate and resolve any issues found during UAT and re-test. By following this phased approach, you systematically build confidence in your upgrade, ensuring that by the time you reach the most expensive testing phase, you have already eliminated the vast majority of issues in a more efficient manner.How to Perform an Incremental Upgrade
You have completed your planning and preparation, and you are now ready to begin the hands-on execution of the upgrade. This guide provides the step-by-step technical process for performing a single "hop" of an incremental upgrade (e.g., migrating from v2.5 to v2.6). The fundamental principle of an incremental upgrade is to migrate your project one major version at a time, ensuring the codebase is stable and validated at each stage before proceeding to the next. The Key Tool: The Repository Tool The Quantexa Repository Tool will be the primary engine for your upgrade. It uses a file generator called PLOPjs and code-modification tools like Scalafix to automate a significant portion of the migration effort. While powerful, this tool does not cover every scenario, and manual changes may therefore be required. The Workflow for a Single Upgrade Hop For each hop in your upgrade roadmap (e.g., from v2.5 to v2.6), you will follow this four-phase process. Phase 1: Automated Migrations The first step is always to let the automated tooling do the heavy lifting. This is a highly scripted process. For detailed instructions on the commands, refer to the documentation on Running the repository tool. Configure the Tool: Locate the migration-config.json for the version hop you are performing. Update the projectPath to point to your repository. Run Scalafix Migrations: Execute the Scalafix migrations first. Run Plop Migrations: Execute the relevant Plop migrations. It is best practice to run these one by one to create a granular commit history. Pro-Tip: If your project uses split repositories (e.g., separate ETL and Apps repos), only run the migrations relevant to the repository you are currently working on. Phase 2: Manual Migrations With the automated changes applied, you will now execute the manual tasks from the backlog you created in the preparation phase. Your JIRA board is your guide for this phase. Execute Manual Migration Tickets: Work through the JIRA tickets you created for the mandatory manual migrations. Each ticket should contain the context and a link to the specific documentation needed for that single task. Address Customization Tickets: Work through the tickets related to your project's customizations. These tasks involve reviewing how the upgrade has impacted your custom code and applying the necessary fixes to make it compatible. Execute Optional Migration Tickets: If you decided to include any optional migrations in your scope, execute those tickets now. Pro-Tip: Make small, specific commits for each distinct manual change (e.g., "Fix custom scoring function for v2.6 API change"). This creates a clean, traceable history. Refer to project-example to see how the same migrations were applied in Quantexa's reference project. Phase 3: Compile and Validate With the automated and manual changes applied, this phase is focused on compiling the code and validating its integrity by running the automated test suite. Compile the Code: The first action is to run a full build of the repository. If any compilation errors arise from the applied migrations, resolve them. These are typically caused by API changes or updated method signatures that impact custom code. Validate with Unit Tests: Once the repository compiles successfully, run your full suite of automated unit tests. Address any tests that are failing as a result of the upgrade changes. A successful, clean build with all unit tests passing marks the end of the development work for this hop. Phase 4: Intermediate Validation (Optional but Recommended) Before moving to the next hop, it is highly recommended to perform some level of intermediate validation to catch issues early. Run your automated integration tests. Perform a small, local ETL run to ensure the core process works. If practical, deploy the applications locally to check for runtime errors. Repeat and Finalize Once you have completed this four-phase process for one hop, you repeat the entire workflow for the next hop in your upgrade roadmap (e.g., v2.6 -> v2.7). After the final hop is complete and you have a stable, building repository on your target version, the hop-by-hop development phase is over. You are now ready to proceed with the full Development Testing as outlined in your test plan.
