Explore the Latest Articles in the Community Library 📚
Our ever-growing Community Library is filled with articles, blogs, and useful resources. Check out the latest articles below, whether you’re just starting out with Quantexa or a seasoned pro, there’s something here to help everyone learn and flourish. Service Operations 📖Quantexa Monitoring Series 📖Quantexa Platform Monitoring 📖Quantexa Platform Monitoring - Key Metrics and Log Entries 📖Quantexa Platform Monitoring - Moving Beyond the Minimum 📖Quantexa Application Monitoring - Introduction 📖Quantexa Application Monitoring - Getting Started with Toolchains Service Build and Transition 📖2.7 Quantexa Upgrade Guide 📖How to make the most out of Intellij IDEA 📖Running a Data Discovery Process 📖When and How to Use DQA Statistics Functionality 📖Project Management Best Practices: An Upgrade Journey 📖Docker for Quantexa Implementations 📖Spark Cluster - Resource Management Service Design 📖Setting Up Infrastructure and Underlying Platforms 📖Setting Up Infrastructure and Underlying Platforms: Cloud Edition 📖Setting Up Infrastructure and Underlying Platforms: On-Premise Edition 📖Using QPython for Analytics and Data Science Teams 📖Quantexa Platform Security Design for Solution Architects
New guide 📖 Scoring Concepts: Write-Once Steps
Discover how to efficiently use Write-Once Steps in the Assess framework for data transformation. This detailed guide complements the Write-Once Steps documentation and helps you determine when to apply Write-Once Steps effectively in both Batch and Dynamic Scoring contexts. Key topics covered: The potential cost of using the wrong method When to use Write-Once Steps vs. Logical Sources Strategies for scoring networks in Batch (SparkScoringContext) and Dynamic (DynamicScoringContext) environments Gain a deeper understanding of how to avoid duplicating logic across contexts and streamline your data engineering workflows. Read the full article (login required) to explore practical scenarios and best practices for scoring networks with Write-Once Steps: 5. Scoring Concepts: Write-Once Steps - Quantexa Community This article serves as an extension of the Product documentation of Write-Once Steps and provides a guide on which situation should the Write-Once Steps be used. Introduction The Write-Once Step is a data transformation step in the Assess framework, which is executable in both Batch and Dynamic contexts, meaning you do not…
Why Does Entity Quality Matter? & Best of the Community from March
March Top Picks Why Entity Quality Matters 🔐login required Enter our latest competition: The Knowledge Exchange 📝 New Education Programs: Scala & Spark Bootcamp and Quantexa Data Engineer Velocity Program Quantexa & Xander Talent - New Education Partnership 🤝 Elevating Data Management: Unveiling the Pillars of a Trusted Data Foundation with Quantexa Latest from the Community Library 📖 A day in the life of a... Senior Learning Designer A day in the life of... an Academy Trainee How To Test Your Upgrades 🔐login required Updates to Quantexa Supported Versions 🔐login required Upcoming events 🗓️ 3rd May Community Connect 👋 Join for a demo of top Community features. Best of Q&A ✅ Unable to load Batch Scores to Elastic 🔐login required How to fix image style for investigation icon in the Qx UI? 🔐login required Error creating bean with name 'springSecurityFilterChain' 🔐login required New & Popular Ideas💡 Usability increase using 2 screens for investigations 🔐login required Changing the default settings for Graphic Filters in Data Viewer 🔐login required Make updates to metadata.parquet optional 🔐login required In case you missed it 📣 Welcome to Quantexa 2.6 | 2.6.0 Release Announcement 🏆 Badge of the Month: The Name Dropper Badge Community quick links 🚀 Submit and vote for Ideas in our Ideas Portal 🗣️ Join one of our Specialist User Groups: FinCrime, Insurance, Data Management & KYC 📚️ Browse blogs, articles and guides in our Community Library
Quantexa & Xander Talent - New Education Partnership
Driving the creation of COEs for customers and partners to enable them to leverage Decision Intelligence (DI) technology via accelerated learning programs and talent provision. Quantexa, a leader in Decision Intelligence (DI), and Xander Talent, a top digital talent consultancy, are excited to announce a strategic educational partnership. This collaboration marks a key milestone in driving industry innovation and talent development. Achieving unicorn status in early 2023, Quantexa is now enhancing its educational offerings to empower customers and partners, fostering self-sufficiency in the ever-evolving decision intelligence landscape. This partnership represents our commitment to providing cutting-edge educational resources and nurturing exceptional talent in the technology sector. Additionally, this new educational capability enhances our Quantexa Partner Program by adding a new partnership category, Quantexa Education Partner, for expanded knowledge share capabilities to partners. The Quantexa Academy is enhancing its educational programs through a partnership with Xander Talent. This collaboration is a key part of our continuous effort to strengthen partner support and increase customer self-sufficiency. It represents an important extension of the reach and effectiveness of Quantexa’s educational initiatives. We are excited to work with an innovative company like Xander Talent, confident that together, we will drive innovation for our customers and partners. Oliver Butler, Head of Education Services, Quantexa We are excited to partner with Xander to help us scale our strategic education initiatives. We see this partnership as a key contributor to driving customer and partner self-sufficiency and quickly mobilising decision intelligence centers of excellence. Christel Wolthoorn, CEO, Xander Talent Xander is thrilled to become a Quantexa Education Partner. This is a true partnership in every sense of the word, as Xander certified Academy Team Leads will be fully integrated into the Quantexa Academy program to work along Quantexa’s own Academy Team Leads and by doing so increase the value of the contribution that we make to the success of Quantexa’s customers and partners. Sheryl Wharff, VP Global Alliance Marketing, Quantexa We are thrilled to add Xander as our first Education Partner, expanding our Quantexa Partner Program announced in 2022. Providing new delivery methods, which enable our Quantexa Partners to learn to deliver our Decision Intelligence technology in a faster-time-to-value model, is the way we will further grow and expand our “Partner First” company objectives. 🤝Partnership Scope 💻💡 Scala & Spark Bootcamp Duration: 5 Days An instructor-led program designed to enable candidates with a background in programming to successfully complete the Scala and Spark Assessment, the pre-requisite entry point to the Data and Scoring Engineer Academies, within a 5-Day period. For further details about the Scala Prerequisite Assessment please see our Scala Prerequisite Assessment for Quantexa Certification Academies Article. 💻🏆 Data Engineer Velocity Program Duration: 30 Days This program is designed to enable candidates to successfully gain the Data Engineer Certification within a focused 30-Day period via a set of Velocity Workshops at key stages of the education program. 💡 Xander’s Talent Engine An additional partner enablement and customer self-sufficiency component of Xander’s business model is to provide a service that allows Xander’s top performing talents to be embedded within partner and customer teams – initially as consultants, and then, potentially, as full-time employees. In this way organisations can take advantage of Xander pre-certified Quantexa Data Engineers, Scoring Engineers, and Technical Business Analysts, to help them build out their Quantexa project teams. Resources Visit Xander Talent’s corporate website to explore their recruiting and training services that build skilled teams of digital talents tailored to your needs. Visit the article on the Quantexa Community website to learn more about the extensive range of training products offered by the Quantexa Academy. Visit Quantexa’s corporate website to learn how its class-leading Decision Intelligence Platform brings innovation and confidence in decision-making across various industries by utilizing contextual data.
Welcome to Quantexa 2.3 | 2.3.0 Release Announcement
In Quantexa 2.3, you will find: Entity Store, a new component in our set of Entity Resolution capabilities; Introduction of Security Model V2, a new way to manage Role-Based Access and Control within The Quantexa Platform; Enhancements to Assess including Path Ranking, a new way to define paths; Support for Elasticsearch 8 and Spark 3.2. Entity Store In 2.3.0, we are introducing Entity Store in beta. Before 2.3.0, all interactions with the Quantexa UI and the Quantexa Mid-tier APIs depended upon resolving Entities on-the-fly (dynamically). The Entity Store allows a persisted or materialized view of Entities to be stored in the system. In this first release, the core Entity Store will load a full set of pre-resolved Entities produced by Batch Resolver, into Elasticsearch. The Entity Store can then be used by Explorer to enable querying of the underlying entities, as a Cache for Resolver to improve performance, and to support the new Entity REST API. Security Model V2 We have introduced a new framework for authentication and authorization across the platform. This gives the platform greater control over the data and features that a user can access and enables easier integration with Identity Providers (IdPs). End-users no longer need to interact directly with low-level Quantexa Roles. They are now able to share their work with Groups that are logical to their organizational structure. For example: UK Fraud Investigators. Using a new User Management Screen, you can now provision users and Groups, collections of users that can be assigned Roles and Dynamic Privileges, to The Quantexa Platform from Identity Providers simply and easily. Path Ranking It is now possible to specify the rules for defining the relative importance of paths, a chain of Documents and Entities that connect two Nodes, when writing Network Scores with Path Ranking, which make the Network easier to analyze by signaling the most significant areas to explore further. Dual Context Sources Dual Context Sources enable scoring logic to be designed that will work in both batch and dynamic, and provides a config-based tool that generates Source steps for both pipelines. This simplifies the deployment of a build with a dual architecture. Support for Elasticsearch and Spark Quantexa now supports Elasticsearch 8 across the platform, except for Offline Indexers and Spark 3.2 is now supported across the platform. Warning: Spark 3.0 is deprecated and support will be removed completely in the next release of the platform. Spring Boot upgrade Spring Boot has been upgraded to version 2.6.8. This results in faster startup times and solves a number of security vulnerabilities found in older versions of Spring Boot. Documentation Site Glossary We have completely revamped and reworked our Documentation site glossary, adding almost 100 new terms. Explore the Glossary to find out more. Other highlights For more control and flexibility when running ETL, Quantexa has has added the ability to generate Resolver Search Loader and Compound Creator scripts at the point where users define a Root Model in Data Fusion. List inputs have been introduced for Entity Attribute functions, to allow for more flexibility and customizability, granting users more ways to define Entity Attributes in Data Fusion, and cater to a wider range of data models. You are now able to collapse the Query Builder in Explorer. This is the next step in improving the user experience of Explorer, allowing users to focus on the results of the query. You will find the full set of Release Notes on the Quantexa Documentation site. If you are unable to access them, you will need to get a user with access to submit a Documentation site access request through the Quantexa Support Portal.
Spark Settings for Success
This page is set up to introduce spark-submit job settings and what considerations you should take when setting them. Please check out the Spark Documentation for all the possible ways to set your spark job: https://spark.apache.org/docs/latest/configuration.html Spark Settings Spark setting are highly dependent on datasource type and size and should be tuned separately. Initial ElasticLoad spark settings for 40 datanode cluster could look like this with the following settings: spark.drive.memory=20g spark.executor.instances=20 spark.executor.memory=20g spark.executor.cores=2 Description of Key Spark Settings The spark.driver.memory configuration in Apache Spark determines the amount of memory allocated to the Spark driver, which is the central control program for a Spark application. The driver is responsible for coordinating tasks, managing the overall execution of the application, and collecting results. The spark.executor.instances configuration in Apache Spark specifies the initial number of executor instances to allocate for a Spark application when it starts. Each executor instance represents a separate process that can run tasks in parallel. The spark.executor.memory setting determines how much memory each Spark executor has available for storing data, caching, and performing computations. The spark.executor.cores configuration in Apache Spark specifies the number of CPU cores to be allocated to each executor in a Spark application. It plays a crucial role in determining the degree of parallelism for your Spark tasks and impacts how your application utilizes the available CPU resources. Tips for Spark Settings It is advisable to maintain… Ratio 1:1 Number of Datanodes : NumberOfExecutors*ExecutorCores Some performance gains can be seen when ratio changes to Ratio 1:3 Number of Datanodes : NumberOfExecutors*ExecutorCores Example: 40 Datanode/spark.executor.instances=30*spark.executor.cores=4 (1:3) Increasing number of data nodes bring down total loading time and reduces bulk rejection error. Update Default Parallelism… Using spark.default.parallelism is a powerful way of tuning your spark job. This configuration defines the default number of partitions to be used for distributed data processing operations when the number of partitions is not explicitly specified. It plays a crucial role in determining the degree of parallelism for your Spark application. It should ideally be a multiple of the number of CPU cores in your cluster. If you enable dynamic allocation (below, Spark can adjust the number of partitions dynamically based on workload. In such cases, you may set a conservative initial value for parallelism. Operations that involve shuffling data between partitions (e.g., join , groupByKey ) often benefit from more partitions, as this can reduce the amount of data movement and improve performance. Consider the distribution of your data. Uneven data distribution can lead to load imbalance among partitions, affecting overall job performance. Adjusting the number of partitions can help address such issues. Enable Dynamic Allocation… The spark.dynamicAllocation.enabled is a boolean configuration option that determines whether dynamic allocation of executor resources is enabled or disabled for a Spark application. Dynamic allocation allows Spark to adjust the number of executor instances and their resources (CPU cores and memory) based on the workload and resource demands of the application. When set to true , dynamic allocation is enabled, allowing Spark to add or remove executor instances dynamically as needed during the course of the application's execution. When set to false , dynamic allocation is disabled, and the number of executor instances remains fixed throughout the application's lifetime, as determined by the initial configuration. Note: This will scale to take all available resources, so be careful when using if your project is very cost-conscious or if you have to share limited resources with other teams outside your own.
Read now: What it's Like to Perform an Upgrade
Read all about our experience and key takeaways when upgrading a repository from version 2.1 to 2.3. The upgrade was performed and released in early 2023 shortly after version 2.3 became available. The main motivation behind this particular upgrade was to upgrade the batch tier software including versions of EMR & Spark which Quantexa 2.3 supported. The flexibility of the new Data Viewer and other newly released features were also important value-adds. What's it Like to Perform an Upgrade? - Quantexa Community This blog describes the experience and key takeaways when upgrading a repository from version 2.1 to 2.3. The upgrade was performed and released in early 2023 shortly after version 2.3 became available. Why upgrade Quantexa versions? As with any software, the advantages of upgrading to newer versions include: Access to new…
