Data Packs: How To Remove Customisations During an Upgrade
If your project has been running for some time, it is likely that you have had to customize a Quantexa Data Pack to meet specific requirements. Historically, this required "forking" the Data Pack repository and maintaining your own version of the code. While this approach provided flexibility, it came at a significant cost: Increased Technical Debt: Your team became responsible for maintaining a growing fork of custom code. Higher Upgrade Effort: Every platform upgrade required you to manually migrate and re-test your forked Data Pack, a complex and time-consuming process. Missed Opportunities: Your forked version did not benefit from the continuous stream of new features, performance improvements, and bug fixes being added to the official Quantexa Data Pack releases. Starting with platform version 2.6, Fusion Extensibility provides a powerful new model that allows you to apply the most common customizations without forking the code. This is a fundamental shift that enables you to stay aligned with the official Data Pack releases while still meeting your unique project needs. For more details on this framework, see the official Fusion Extensibility documentation on the Documentation Site. The Opportunity: When to Migrate Your Customizations If your project is on version 2.6 or later and you are still maintaining a forked Data Pack, you have a strategic opportunity to eliminate this technical debt. However, because migrating from a fork to the official Data Pack will introduce significant (and desirable) changes to your ETL output, it is strongly recommended to treat this migration as a standalone project, separate from a major platform upgrade. Attempting to do both at the same time makes it extremely difficult to perform root cause analysis. If you see a change in your data, is it because of the platform upgrade or because of the Data Pack migration? By separating the two projects, you can isolate the variables and test each change independently, leading to a much smoother and lower-risk process. The Migration Process: From Fork to Extensibility The migration process involves a careful review and triage of every customization you have made. Step 1: Review Your Forked Data Pack Go through your forked repository's commit history and create a detailed list of every functional change you have made compared to the original Quantexa version. Step 2: Triage Each Customization For every single customization you have identified, you must now make a critical decision. Ask Yourself... If the answer is YES... If the answer is NO... 1. Is this customization achievable using the new Fusion Extensibility patterns? Great! Your action is to plan to re-implement it using the new model. Go to question 2. 2. Is this customization still a critical business requirement? Your action is to acknowledge that you must remain on your forked version for now. You should then raise an enhancement request in the Quantexa Product Roadmap to have this capability added to the core product in the future. This will unblock your migration later. Excellent! This is an opportunity to simplify. Your action is to plan to discard the customization and use the standard, out-of-the-box Data Pack functionality. Step 3: Execute Your Migration Plan Based on the outcome of your triage, your path is now clear: If all of your critical customizations can be achieved via extensibility or have been deemed no longer necessary, you can proceed with re-implementing them and decommissioning your forked repository. If even one critical customization cannot be achieved, you must pause the migration, raise the necessary enhancement requests, and continue to maintain your fork until the product supports your needs. The Testing Strategy: Focus on Intent, Not Identical Output This is the most critical concept to understand when testing your migrated Data Pack. You should NOT be testing for identical output. Your old, forked Data Pack is likely behind the latest official version. The new version contains numerous bug fixes, functional improvements, and performance enhancements that will naturally and correctly lead to different ETL output. Your testing strategy must instead focus on validating the intent of your original customizations. If your customization was to... Your new test should verify that... Add a new "Risk Score" derived field... The "Risk Score" field is still being correctly calculated and added to the model, even if other parts of the record have changed due to product improvements. Exclude compounds based on a specific pattern... The correct compounds are still being excluded based on your custom logic. Add a new "Source System ID" attribute to an Entity... The "Source System ID" attribute is still present on the resolved Entities with the correct value. By adopting this testing mindset, you embrace the benefits of the new Data Pack version while ensuring your critical business logic remains intact. The result is a dramatic reduction in custom code, a simpler and faster future upgrade path, and immediate access to the latest Quantexa features.