Why are upgrades so important in the tech industry?
To keep your data secure
Critical to security, upgrades come with the latest versions of third party software, reducing risk from vulnerabilities, such as the Heartbleed Bug which affects older versions of openSSL.
To take the latest features
Upgrades come with a host of new features from end user functionality to back-end functionality such as performance improvements and of course bug fixes, so taking the latest version might just be the key to getting you to that next milestone. Checkout the latest features available:
To maintain support for production systems
Software companies can only support so many previous versions of their product, usually around 6-18 months. So it’s imperative that software is kept up to date in line with this to prevent any lack of support if an issue is found within a production environment. In particular Quantexa aims to keep in line with supported platform components, such as the latest Spark and Elasticsearch versions.
To minimize upgrade disruption
It may sound odd that one wants to perform an upgrade to minimize the disruption of an upgrade, however it’s seen that regular small upgrades are much simpler to manage than one large upgrade once every blue moon. Have you ever left your laptop shutdown for an extended period of time, to come back and find it upgrading for what feels like an eternity?
What does an upgrade at Quantexa look like?
We’ve spoken about why to perform upgrades but how does an upgrade at Quantexa work?
Technical approach
When we say “Quantexa version N has been released for general availability”, we mean that the dependencies and relevant guides and examples required by customer project teams for compilation of both the ETL and mid tier are available for distribution. These binaries consist of around 500 product libraries which are utilized for the build. These binaries are packed into a distribution and provided to the customer project team via approved secure transfer.
To perform an upgrade of Quantexa within a project one needs to adjust the project code and config to conform with the Quantexa product binaries for the new version, this requires:
- Adjustment in project repository to match new best practices
- Changes of references to product API's
- Change of references to latest technology stack
What makes an upgrade complicated?
Quantexa is built on a highly configurable framework, whereby configuration and customization is central to processing data for the purpose of entity resolution, which is ultimately the cornerstone of the product. Unfortunately this heavy customizability comes at a slight cost with regards to ease of upgrades. Though we’d love the upgrade to be as simple as adding a new level on angry birds. A commercial software deployment quite simply just isn’t so simple.
A Quantexa upgrade requires a development team with access to the project codebase to update multiple different areas of the config and code repository and potentially migration of core platform including:
- Software updates of core third party components
- Elasticsearch
- Spark
- RDBMS
- Java
- Scala
- Node
- Helm
- Gradle
- Code Repository
- Updates to software versions within Gradle and NPM build tools
- Updates to libraries within build tool config
- Updates to ETL config due to change of upgraded libraries
- Updates to ETL Scala code due to change in upgraded libraries
- Updates to Spring config
- Configuration of new features requiring one or many of: Scala code, Spring config, Gradle dependencies, typescript and html changes
- Testing Unit testing of existing functionality, to ensure that the upgrade still conforms to expected functionality.
- Systems integration testing: Depending on the upgrade this may be required to ensure end to end functionality of new and existing components
- Regression testing against production like environment to ensure functionality of live system is not affected.
- Performance testing of new components to meet expected benchmarks
- User acceptance testing prior to release to production to verify all new and existing features meet expectations
- Core platform dependencies
- RDBMS schema migration
- Security models in RDBMS
- Data migrations within Elasticsearch
- Updates to schedulers due to package names
- Updates to build tooling for additional module builds
- Update to deployment tooling for additional or altered modules
How to make upgrades easier?
Given the complexity of upgrades, Quantexa has worked on best practices and tooling to assist with you in getting the latest and greatest features out the door.
Upgrade strategies
Here’s a few areas which can minimize the disruption caused by performing an upgrade:
- Going from one version to another requires going through intermediate versions. Little and often is usually easier and reduces upgrade complexity Single major version upgrades require fewer resources and can be performed in smaller timescales, and so allows for better planning and timescale estimation. This also allows for much simpler debugging of issues.
- Regular scheduled upgrade dev time is more likely to be efficient. Doing something once every 3 months lets people get into a pattern/follow a routine. Doing a massive one-off upgrade once per year is more disruptive.
- Upgrading multiple versions at a time will impact the amount of inter dependent services being affected, this can cause need for more regression and acceptance testing.
- Managing Software Updates “the DevOps Way” - DZone
- Staying close to best practice helps to align closely with the expected project and allows for easier use of migration guides and tooling.
- Upgrade planning, looking into which components require upgrade and any dependencies on software updates required.
- Appropriate source control branching strategies. Simpler to separate upgrades from feature branches.
- Simple but complete deployment pipelines, can help with the integration testing of upgrades in a quicker more agile way.
Upgrade tooling
The “repository tool” is the shining light in the tunnel of upgrade complexities.
The simply named “repository tool” makes use of a file generator called PLOPjs to automate the migration of code required for the upgrade. This tool covers any sufficiently generic code changes, such as package name changes and config, whilst this is a powerful tool it’s not quite a magic bullet and some manual migration will still be required if the codebase has been customized.
The relevant scripts for the repository tool will be provided alongside the software dependencies, helm charts and project specific migration guides, these detail the specifics for project teams to utilize the repository tool and perform upgrades on the project code base.
As every project implements a slightly different use case it’s important to consider which services are implemented as this will adjust which steps of the migration guide to follow.
When and how often should one upgrade?
Understanding the frequency of upgrades comes down to a balance of project functional and non-functional requirements, including how sensitive a business is to security concerns or necessity of new features. In general it is recommended to take upgrades within the first few months of release, with new major versions (2.0 → 2.1) of Quantexa released quarterly, and major platform releases (1.x → 2.0) released every couple of years. Additionally, minor or patch versions (2.0.1 → 2.0.2) released as necessary, these contain no breaking changes and can be taken as part of regular development.
Additional resources
For more help and guidance on performing upgrades check out our article: Project Documentation to support you in upgrading Quantexa
Performing upgrades of core software is commonplace across the tech industry, to find out more, check these nine best practices for managing software releases and upgrades.
