So, you've merged your final Pull Request after the CI pipeline turned green - it's time to deploy Quantexa to a shared environment for the first time. Taking a piece of enterprise software from a laptop to a distributed, robust, central location is an important journey that you will take again and again. It's also one we've taken many times here at Quantexa; read on to discover our approach.
The destination
It's always important to have a mental image of your destination in mind, and no less so when deploying Quantexa. To aid you in forming this mental image, here is an actual image of Quantexa's high level deployment architecture (with Quantexa components in purple):
As you can see there are only really 2 major areas to which Quantexa components are explicitly deployed:
- Batch Processing Tier
- Mid Tier
Once these 2 areas are up and running, Quantexa is able to manage the remaining areas by itself.
The waypoints
Batch Tier
From a deployment perspective, Quantexa's batch processing Tier is very straightforward:
- Compile your code into a jar
- Make your jar an uber-jar: bundle all the dependencies into the same jar
- Zip up your preferred
spark-submit script, alongside environmental config - Deliver both to your favourite Edge Node and away you go
Mid-Tier
We're always keen to make use of freely available technology where possible; if you've chosen to host your mid-tier in Kubernetes (or another container management platform), we strongly recommend you to take advantage of our Helm Charts to guide you on your way. The pattern here is familiar:
- Publish your images
- Configure your environmental properties
- Run
helm install and see your Pods, Services and ConfigMaps appear
(NOTE: If your organisation has customised the container management platform, you may need to work with your platform team to align the charts - we're only able to test on unmodified Kubernetes). Even if you aren't able to helm install, you can still use Helm to generate the required Kubernetes manifests for deployment.
If you're not using containers, your path is slightly more complicated as you will need to ensure you have come up with a pattern for meeting any resilience and HA requirements you might have. Once you have this the act of deployment looks much the same as the Batch Tier:
- Compile your applications into (Spring Boot) jars
- Zip up your preferred startup script, alongside environmental config
- Deliver both to your favourite JVM node(s) and away you go
Helping hands
We take this journey all the time, and so have several helping hands available as inspiration for your platform administration teams to use when providing your devops infrastructure:
- Gradle tasks for building and distributing packages
- CI/CD guidance and examples for automating packaging and deployment
- Schedules for orchestrating batch processing
Sit back and enjoy the view
That's it, you're done. Quantexa is able to manage the creation of RDBMS tables and Elasticsearch indexes by itself - auto-pilot if you will. So all that remains is to take in the view:
As we said at the outset, you'll be taking this journey again and again, so just remember to simplify and automate, to ensure you see the same view again and again at the end.
