2.8 Quantexa Upgrade Guide
Table of Contents Quick Upgrade Overview Community Upgrade Guidance Dependencies and Platform Modernisation User Interface Changes Batch Resolver and Entity Resolution Redis / Valkey Distributed Caching Data Packs Migration Recent Deprecations Software Compatibility Additional Information Quick Upgrade Overview The 2.8 Quantexa Upgrade consists of five main parts: Dependencies and Platform Modernisation User Interface (UI) Changes Batch Resolver and Entity Resolution (ER) Redis / Valkey Distributed Caching Data Packs Migration The majority of the dependency changes (Spark 3.5 migration, Cats Effect 3, Apache Pekko, library version bumps) are handled by automated Plop and Scalafix migrations via the Repository Tool. However, any custom code that deviates from best-practice patterns, particularly Cats Effect or Akka usage, will require manual migration. A new Quantexa license is also required before starting. The User Interface tier sees the most user-facing change. Resource Viewer reaches General Availability (GA) and replaces several Investigation modules, Transaction Viewer has been fully removed (migrated to Data Viewer), Angular has been updated to version 18, and the web-app folder structure has been simplified. Projects should plan to test the UI thoroughly post-migration. Batch Resolver receives several important changes: Network Generation and time-splitting have been removed, Classic mode is deprecated, and the Entity Attributes output model has changed. Filtered Compounds now use QSL instead of QEL. Most of these have Plop or Scalafix support, but projects with custom Batch Resolver scripts should budget additional effort. Redis / Valkey distributed caching is introduced as optional in 2.8 but becomes mandatory from 2.9 onwards. Projects are strongly encouraged to set up Redis during the 2.8 upgrade to avoid a separate infrastructure change later. Data Packs 2.3.x are compatible with Quantexa 2.8 and require Parsers 4.2.3 or later. Projects still on Parsers 3 must complete the Parsers 4 migration before upgrading to 2.8. This page aims to provide additional guidance related to the 2.8 Quantexa Upgrade. For the full list of required migration steps, please refer to the Documentation site migration guide: 2.7 → 2.8 Upgrade Migration Guide. Release Notes: Community Release Announcement 2.8 Release Notes Community Upgrade Guidance Dependencies and Platform Modernisation New Quantexa License Required A security improvement to the Quantexa license means that existing licenses are not compatible with 2.8.0. You must request a new license before deploying the upgraded platform. This is a prerequisite that should be actioned early in the upgrade process to avoid blocking deployment. Spark 3.5 Migration Support for Spark 3.3 and 3.4, deprecated in 2.7, has been fully removed. Projects must now run on Spark 3.5. The BOM changes are applied automatically by Plop. The primary manual effort comes from ensuring custom project-owned Spark scripts are compatible with Spark 3.5 and upgrading Spark installations across all environments (local, CI/CD, and production). During internal migrations, no significant behaviour changes were found, so this should be a low-risk change for most projects. Cats Effect 3 This is one of the more impactful dependency upgrades. The cats-effect, fs2, and fs2-Kafka libraries have all been updated to version 3.x. Code following the best-practice structure in Task Loading, Scoring, Alerting, Entity Store, and Data Source Integration Tests is migrated automatically. However, any custom code built upon earlier versions of these libraries, such as custom Kafka applications, must be manually migrated following the official Cats Effect 3 migration guide. Projects should use Project Example as a reference. Akka Replaced by Apache Pekko Due to CVEs raised against the Akka library, Quantexa has switched to Apache Pekko. This only affects deployments using app-graph-script. The migration is straightforward: ensure import akka.util.Timeout statements are changed to org.apache.pekko.util.Timeout. A Scalafix migration is available. Other Library Upgrades Several other libraries have been updated, all with Plop or Scalafix support: Scalatest upgraded from 3.1.4 to 3.2.19 ScalaCheck upgraded (artifact name changed) Chimney version updated Java Faker replaced by Datafaker ETL Util dependencies renamed (batch-common-api → spark-api, common-spark-utils → document-index-input-reader) spark-excel artifact and package changed spoiwo package updated from 1.x to 2.x org.apache.commons.io must now be shaded in the Batch tier JSON Configuration Validation A notable change to be aware of is that configuration validation on application startup now causes the application to fail if Perspective, Bulk Search, or Foreign Document JSON configuration files contain unknown fields. Previously, these invalid fields were silently ignored. This means that configuration errors that have existed undetected in your project may now surface as startup failures. It is recommended to run the Configuration validation tests early in the upgrade to identify and fix any issues before attempting to deploy. User Interface Changes Resource Viewer (General Availability) Resource Viewer is the most significant user-facing change in 2.8. It provides a unified, tabbed interface for viewing and interacting with Documents, Entities, and other resources. All projects upgrading to 2.8 will inherit the Resource Viewer, which replaces several existing Investigation modules including CompoundGraphViewModule, SourceViewerPluginModule, RecordViewerPluginModule, and others. These replaced modules are now deprecated and will be removed in the next major release. This introduces breaking changes for deployments using Low-Code Configuration (LCC) or Record Viewer. Projects should follow the Resource Viewer Migration Guide and plan for testing time to verify the UI behaves as expected, particularly around any custom UI configuration. Transaction Viewer Removal Following its deprecation in 2.4, Transaction Viewer has been fully removed. Data Viewer is the recommended replacement, offering feature parity plus improvements such as no record limit, column-level UI filtering, and Scoring plugin integration. Projects still using Transaction Viewer must complete this migration. Follow the Transaction Viewer Migration Guide for guidance. Angular 18 and web-app Simplification The UI has been updated to Angular 18. Plop automatically handles the migration of your deployment's UI code. However, due to the many changes to NPM dependencies, Plop may fail to migrate web-app/package-lock.json to a consistent state. If you encounter UI build errors after migration, delete the node_modules and package-lock.json files from the web-app folder and run npm install to regenerate them. The web-app folder structure has also been simplified, removing boilerplate files and simplifying how the application is bootstrapped. Tab configuration previously in MainViewComponent has moved to a dedicated tab-config.ts file. Global UI Configuration Changes The file structure used for global settings in Low-Code Configuration has changed. If your deployment uses global configuration loaded through JSON files on deployment, you must migrate to the new format. Follow the Global UI Configuration Migration Guide. Batch Resolver and Entity Resolution Network Generation and Time-splitting Removed The Network build and Create ScoringGraph stages of Batch Resolver, which produce Natural Networks, have been removed following their deprecation in 2.7. Projects still using these must migrate to Graph Scripting QSL for network generation. The associated configuration options (appendFilteredDocuments, createNetworkStats, scoringGraphMaxNodeCount, etc.) must be removed from Batch Resolver configuration files. Similarly, the time-splitting functionality has been removed. All timeSplit fields must be removed from Resolver JSON configuration. Filtered Compounds: QEL to QSL Migration The Quantexa Expression Language (QEL) expression field within Filtered Compound definitions has been replaced with a QSL where field. A Plop migration is available to translate existing QEL filters to QSL. Projects should review the translated filters to ensure correctness, particularly for complex expressions. Additionally, Batch Resolver now warns if Filtered Compound definitions reference Record Attributes not defined in the input data. These warnings will be upgraded to errors in a future release, so projects should address them during the upgrade. Note that the default Address Resolution Template in the Data Packs core fragment contains an erroneous filteredCompoundDefinitions block referencing a rootCountyCode attribute; if present, this block should be removed. Entity Attributes Output Model Changes The excluded, subEntities, and linkExcluded fields have been removed from the Batch Resolver Entity Attributes output. A new failedDataQualityChecks field has been added. A Scalafix migration is available, but due to Scalafix limitations, imports referring to the removed SubEntity model must be removed manually. Classic Mode Deprecated Batch Resolver Classic mode is deprecated and will be removed in a future release. Projects should migrate to Resolver mode. To replicate previous Classic mode exclusions, use Element exclusions in ETL. See Migrating from Classic to Resolver mode for guidance. Calling Batch Resolver and Graph Scripting QSL from Code Deprecated Calling Batch Resolver and Graph Scripting QSL stages from code is deprecated and will be removed in a future major release. Projects must migrate to using the pre-defined scripts under com.quantexa.batchresolver.scripts and com.quantexa.graphqsl.scripts directly with spark-submit. If you previously used custom scripts to insert Run IDs from Metadata into file paths, use the updated dataPaths configuration. Redis / Valkey Distributed Caching Quantexa 2.8 introduces distributed caching with Redis or Valkey. While Redis is optional in 2.8, it becomes mandatory from 2.9 onwards. We strongly recommend configuring Redis during the 2.8 upgrade so that this infrastructure is in place before upgrading to 2.9. The Repository Tool automatically adds a baseline application-redis-profiles.yml configuration file and updates docker-entrypoint scripts. Redis can be deployed via the Quantexa Helm Chart (Redis Sentinel), Google MemoryStore for QCP-hosted projects, or a manually deployed Redis/Valkey instance. To enable Redis caching, set the following feature flag in your Spring configuration: quantexa.caching.redis-caching.enabled: true For full deployment guidance, see the 2.7 → 2.8 Migration Guide, section "Distributed caching using Redis or Valkey". Data Packs Migration Data Packs 2.3.x are compatible with Quantexa 2.8.x and require Parsers 4.2.3 or later. Parsers 3 is no longer supported from Quantexa 2.8 onwards. Projects still on Parsers 3 must complete the migration to Parsers 4 before upgrading to 2.8. This migration should be treated as a separate workstream and is not recommended to be performed alongside the 2.8 platform upgrade. For planning and step-by-step guidance, see: Why & How to Plan a Parsers 4 Migration Step-by-step Guide to Migrate to Parsers 4.3 For the full Data Packs compatibility matrix, see Data Packs Compatibility Matrix. Recent Deprecations Batch Resolver Classic Mode Classic mode is deprecated and will be removed in a future major release. Projects must migrate to Resolver mode. See Migrating from Classic to Resolver mode. Elasticsearch 7 and OpenSearch 1 Elasticsearch 7 and OpenSearch 1 are deprecated in 2.8 and will be removed in 2.9. Projects should plan to upgrade to Elasticsearch 8 or OpenSearch 2. Search 1 Search 1 is deprecated. Projects should migrate to Search 2. Connect Connect is deprecated in 2.8. Quantexa Feature Engineering Quantexa Feature Engineering is deprecated. Projects using version 0.9.0 must replace their imports with their own Spark transforms. Graph Scripting DSL Following its deprecation in 2.7, the DSL Conversion to Scoring Graph methods have been removed in 2.8. The Entity Statistical Profile Testing Framework's use of DSL models is also deprecated - projects must migrate to using the CalculatedEntityAttributes Batch Resolver output. Parsers 3.x As noted above, Parsers 3.x is no longer compatible with Quantexa 2.8. Projects should complete migration to Parsers 4.x before upgrading. Software Compatibility Check out software compatibility. Key changes for 2.8: Apache Spark: 3.5.x only (Spark 3.3 and 3.4 removed) Java JDK (Application Tier): 17 Node.js: 22.11.0 npm: 10.9.0 Gradle: 8.4+ Redis / Valkey: 6.x, 7.x (optional in 2.8, mandatory from 2.9) Elasticsearch: 7 (EOL) and 8; OpenSearch 1 and 2 Kafka: 3.x, 4.x CPython: 3.10.x, 3.11.x MySQL: 8.x, 9.x PostgreSQL: 13.x to 18.x Additional Information Useful resources: Upgrade Best Practice for instructions before commencing an upgrade. Ongoing development during upgrades for best practice guidance on continuing with meaningful development during an upgrade. Quantexa Upgrade Knowledge Hub for comprehensive guidance on planning and executing upgrades. Parsers 4 Migration Guidance for planning and executing the Parsers 4 migration (required for 2.8). Resource Viewer Migration Guide Global UI Configuration Migration Guide Entity Store Load External Attributes Migration Migrating Elasticsearch Clients for Scoring Follow the Release Announcements Topic to receive notifications of releases. Quantexa Release Notes for changes and information specific to different versions of Quantexa.Data Packs: How To Remove Customisations During an Upgrade
If your project has been running for some time, it is likely that you have had to customize a Quantexa Data Pack to meet specific requirements. Historically, this required "forking" the Data Pack repository and maintaining your own version of the code. While this approach provided flexibility, it came at a significant cost: Increased Technical Debt: Your team became responsible for maintaining a growing fork of custom code. Higher Upgrade Effort: Every platform upgrade required you to manually migrate and re-test your forked Data Pack, a complex and time-consuming process. Missed Opportunities: Your forked version did not benefit from the continuous stream of new features, performance improvements, and bug fixes being added to the official Quantexa Data Pack releases. Starting with platform version 2.6, Fusion Extensibility provides a powerful new model that allows you to apply the most common customizations without forking the code. This is a fundamental shift that enables you to stay aligned with the official Data Pack releases while still meeting your unique project needs. For more details on this framework, see the official Fusion Extensibility documentation on the Documentation Site. The Opportunity: When to Migrate Your Customizations If your project is on version 2.6 or later and you are still maintaining a forked Data Pack, you have a strategic opportunity to eliminate this technical debt. However, because migrating from a fork to the official Data Pack will introduce significant (and desirable) changes to your ETL output, it is strongly recommended to treat this migration as a standalone project, separate from a major platform upgrade. Attempting to do both at the same time makes it extremely difficult to perform root cause analysis. If you see a change in your data, is it because of the platform upgrade or because of the Data Pack migration? By separating the two projects, you can isolate the variables and test each change independently, leading to a much smoother and lower-risk process. The Migration Process: From Fork to Extensibility The migration process involves a careful review and triage of every customization you have made. Step 1: Review Your Forked Data Pack Go through your forked repository's commit history and create a detailed list of every functional change you have made compared to the original Quantexa version. Step 2: Triage Each Customization For every single customization you have identified, you must now make a critical decision. Ask Yourself... If the answer is YES... If the answer is NO... 1. Is this customization achievable using the new Fusion Extensibility patterns? Great! Your action is to plan to re-implement it using the new model. Go to question 2. 2. Is this customization still a critical business requirement? Your action is to acknowledge that you must remain on your forked version for now. You should then raise an enhancement request in the Quantexa Product Roadmap to have this capability added to the core product in the future. This will unblock your migration later. Excellent! This is an opportunity to simplify. Your action is to plan to discard the customization and use the standard, out-of-the-box Data Pack functionality. Step 3: Execute Your Migration Plan Based on the outcome of your triage, your path is now clear: If all of your critical customizations can be achieved via extensibility or have been deemed no longer necessary, you can proceed with re-implementing them and decommissioning your forked repository. If even one critical customization cannot be achieved, you must pause the migration, raise the necessary enhancement requests, and continue to maintain your fork until the product supports your needs. The Testing Strategy: Focus on Intent, Not Identical Output This is the most critical concept to understand when testing your migrated Data Pack. You should NOT be testing for identical output. Your old, forked Data Pack is likely behind the latest official version. The new version contains numerous bug fixes, functional improvements, and performance enhancements that will naturally and correctly lead to different ETL output. Your testing strategy must instead focus on validating the intent of your original customizations. If your customization was to... Your new test should verify that... Add a new "Risk Score" derived field... The "Risk Score" field is still being correctly calculated and added to the model, even if other parts of the record have changed due to product improvements. Exclude compounds based on a specific pattern... The correct compounds are still being excluded based on your custom logic. Add a new "Source System ID" attribute to an Entity... The "Source System ID" attribute is still present on the resolved Entities with the correct value. By adopting this testing mindset, you embrace the benefits of the new Data Pack version while ensuring your critical business logic remains intact. The result is a dramatic reduction in custom code, a simpler and faster future upgrade path, and immediate access to the latest Quantexa features.
