2.7 Quantexa Upgrade Guide

Quick Upgrade Overview

The 2.7 Quantexa Upgrade consists of three main parts:

1. Core Product Changes
2. Removal of Quantexa Incubators
3. Data Packs Migration

Most of the Core Product changes are automated migrations and minor adjustment which can be tested in a local environment (unless your project has Data Streaming, Entity Store, or Graph API configured). Migration to Delta Lake is going to be the biggest component but it is also assisted by automated migrations and some of the required effort can be avoided (please see below).

Quantexa Incubators will no longer be released in 2.7. Some of the utilities previously released as part of Incubators moved to the Core Product code, the rest will still be accessible (e.g. for code forking) in the 2.6 Quantexa Incubators release. It is expected that most effort will be consumed by the Data Generator migration, however, functional changes can be deferred if there is a strong need to minimize the upgrade timeline.

The bulk of the Upgrade effort comes from the recommended regression testing of functional changes introduced by the Data Packs related migrations. Although the recommendation is to perform all available migrations, those functional Data Packs changes can be deferred if faced with severe time constraints (please see below).

This page aims to provide additional guidance related to the 2.7 Quantexa Upgrade, for the full list of required migration steps, please refer to the Documentation site migration guide: 2.6 → 2.7 Upgrade Migration Guide.

Release Notes:

Community Release Announcement

2.7 Release Notes

Community Upgrade Guidance

Core Product Changes

Migration to Data Lake

This component is assisted by automated migrations and should require fairly simple configuration adjustments. It is recommended to perform a full end-to-end run once the migration steps are finalised. A significant part of the effort comes from handling metadata files, especially ones existing already in Production environments. A script is provided which converts legacy metadata Parquet files to Delta Lake format. However, that step is only required if an incremental mode iteration persists from pre-2.7 batch runs, as this requires information from existing metadata. If completing a full ETL batch run following an upgrade to 2.7, this step is not required, and a metadata.delta file is created automatically, provided the initial migration steps have been completed.

Removal of Quantexa Incubators

• ETL validation tools migrated into the platform
• Elasticsearch snapshot scripts migrated into the platform
• ETL test utilities migrated into the platform
• Data generator core moved into Project Example
• Dynamic Graph Script utilities
• Graph Script REST API starter
• Spark, Scala, and Test Analytics utilities
• SparkTestSuite

From 2.7 Quantexa will stop releasing the community repository Quantexa Incubators. Components that have been accepted as best practice have been migrated into the Core Product. All previously released utilities will remain accessible (e.g. for code forking) in the 2.6 Quantexa Incubators release. Most of the above are straightforward migrations with no functional changes and can be performed and tested in a local environment.

Data generator core moved into Project Example

In 2.7 Data Generator moves into the Core Product where it will benefit from General Availability support. All projects are strongly recommended to perform this migration (please consult your Quantexa Architect if you are considering omitting it).

Dynamic Graph Script utilities

This will apply to very few projects that do not follow the typical Batch Resolver + DSL network generation process. We recommend consulting your Quantexa Architect if this is the case to ensure this migration is well executed and in line with Quantexa's Best Practice.

Data Packs Migration

2.7 Data Packs come in two flavours, either with or without functional changes. For the full description of functional as well as non-functional changes, please refer to the Data Packs Release Notes.

In order to fully benefit from the improvements introduced in 2.7 Data Packs, we recommend opting for the version containing functional changes. However, projects should be mindful that doing so will extend the overall timeline of the upgrade. The additional effort will vary between projects, depending on the available regression testing setup, the ease of performing end-to-end batch runs, the need for model governance, etc..

Release 2.7_1.3 ​

This is a release of Data Packs that is compatible with Q2.7 and Parsers 3. This release will contain no new functionality compared to the 2.6_1.3 release and therefore provides projects the option of performing no/limited regression testing when upgrading.​

Release 2.7_2.1​

This is a release of Data Packs that is compatible with Q2.7 and Parsers 4. This release contains no new functionality compared to the 2.6_2.1 release and therefore provides projects the option of performing no/limited regression testing when upgrading.

Release 2.7_2.2​

This is a release of Data Packs that is compatible with Q2.7 and Parsers 4. This release contains all new functionality developed by the Data Packs team.​

Please follow the guidance below in order to make a decision around which release of Data Packs to use:​

Using Parsers 3​

• If the project is utilising Parsers 3 then they must use the 2.7_1.3 release. The project can then make a risk-based decision on the level of regression testing performed given there is no new functionality contained within this release.​
• This is currently the last planned version of Data Packs that will be compatible with Parsers 3 and therefore there is an inherent assumption that projects will need to migrate to Parsers 4 prior to upgrading to Quantexa 2.8 if they use Data Packs.​

Using Parsers 4​

• If the project is utilising Parsers 4 then the default option should be to use the 2.7_2.2 release.​
• Projects should only consider using the 2.7_2.1 release if the following criteria are met:​
  • The project is performing an incremental upgrade from QE2.6 to QE2.7, with no plans to update to Q2.8 in the near future.​
  • There is ongoing Quantexa presence within the Delivery Team for the project​
• If the above criteria is met, then the option should be discussed with the project Architect and/or Technical Delivery Oversight, and a decision made as a project team​.
• There will not be a 2.8_2.1 release and therefore projects that utilise the 2.7_2.1 release will be introducing two sets of functional changes when upgrading to QE2.8 (i.e. the changes in the 2.7_2.2 and 2.8_2.3 releases)​

Recent Deprecations

Transaction Viewer

As of version 2.4 of The Quantexa Platform, Transaction Viewer has been deprecated and will be removed in a future major release. Data Viewer is the recommended replacement for viewing records from your data sources, which is powered by the Explorer API. Please refer to the Data Viewer Migration Guide for more details.

Quantexa Parsers 3.X

3.X version of Quantexa Standard Parsers and Quantexa Data Models has been deprecated and will be removed in a future major release (support will be removed in August 2025).

The Parsers 4 migration is not currently advised for projects where any of the following applies:​

• TBML projects (or any project using a parsed business name field as part of a document ID)​
• Correspondent banking projects​
• Projects using the entity-level alerting framework​

We also advise against migrating to Parsers 4.X alongside any other QP product upgrades or migration.​

Delivery and DEA are jointly working on ways to reduce risk and effort around this migration, including more extensive testing internally. They will agree and document best practice approach for this migration, including comprehensive guidance for testing (based on benchmarking tests) and supplementary documentation for manual changes​ as well as guidance for project teams to leverage Entity Resolution (ER) improvements from Parsers 4.

More details available in the Quantexa Parsers Release Notes.

Software Compatibility

Check this page for software compatibility.

Please note that Java 8 is no longer supported in Quantexa 2.7.

Additional Information

Useful resources on the Quantexa Documentation Site:

• Upgrade Best Practice for instructions before commencing an upgrade.
• Ongoing development during upgrades for best practice guidance on continuing on with meaningful development during an upgrade.
• Follow the Release Announcements Topic to receive notifications of releases.
• Quantexa Release Notes for changes and information specific to different versions of Quantexa.