MLflow at Quantexa
In the dynamic landscape of Machine Learning (ML) development, the need for effective experiment tracking is paramount. This is especially true as organizations scale their operations. As the complexity of ML projects grows, so does the need for comprehensive tools to efficiently manage experimentation, iterations, and model versions. At Quantexa, we encountered this challenge head-on and sought a robust solution to streamline our ML workflows. We found a big part of the solution to these challenges was to use MLflow. MLflow is a powerful platform designed to simplify the end-to-end machine learning lifecycle. In this blog, we delve into how we leverage MLflow at Quantexa. What is MLflow? MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It's designed to help Data Scientists and Machine Learning Engineers with tracking experiments, packaging code into reproducible runs, and sharing and deploying models. It comes with a graphical user interface which makes it easy to use. At Quantexa, we primarily use MLflow for experiment tracking and reproducibility. This will be the focus of this post. If you are interested in learning more about how you can use MLflow for model deployment, we recommend you spend time reading collateral on the MLflow Website. What issues is MLflow solving? MLflow addresses several critical challenges in our machine learning workflow. This enhances our overall efficiency and effectiveness. These challenges will also be faced by any client building models on top of Quantexa data. Here's how it tackles three of our most key issues: Reproducibility and traceability At Quantexa, we have a considerable number of machine learning models. It is critical that the origin of all these models is well-documented and readily available. This is important from a compliance perspective but also for internal purposes. The team and clients should be aware of performance metrics and each model's strengths and weaknesses. The models are also valuable intellectual property for the company, and we need to be able to re-produce results. MLflow helps us with these issues. When models are trained, we record versions of code, model parameters, and the source of the original data. We can then use this information to perfectly recreate a model if ever needed. Furthermore, we have a record of any experiments which were used in selecting models. This means that a Data Scientist in the future can understand why certain decisions were made during the model prototyping phase. Collaborative insight We want Data Scientists to be able to collaborate easily. MLflow fosters collaboration by providing a centralized repository for tracking and sharing experiments. This is particularly useful when it comes to gaining insight from non-technical stakeholders. Stakeholders can use the MLflow UI to access all the information they need to assess and give feedback on models. This dramatically speeds up the feedback cycle and leads to better models. It also saves time for all involved. MLflow enables data scientists to collaborate better with one another. This encourages a culture of knowledge exchange and innovation within the team. Data correctness At Quantexa, we are dealing with very large datasets when running predictions using our models. This scale can make it difficult to assess the quality and correctness of data. Manual sense checks are always performed, but mistakes can be missed. This is particularly true if errors only affect a certain portion of the dataset (e.g. one country). To avoid errors slipping through, we produce lots of metrics and plots when we receive new data refreshes. We automatically compare this data to what we have seen previously. We log all of these metrics in MLflow. This makes it easy for the whole team to audit the data and spot mistakes such as missing data. Large differences are highlighted automatically even if only in one geographical region or data source. This information can then be easily forwarded to upstream teams to resolve issues. This process dramatically reduces the risk of erroneous predictions from incorrect data. How do we use it? In practice, we mainly use MLflow for three different workflows. Feature generation A pre-processing step transforms data produced by Quantexa Entity Resolution into a different tabular format. Our models then use this new format for training and prediction. We record aggregate information about data produced during this pre-processing step in the form of plots and CSVs. This aggregate information is compared to previous runs to highlight big discrepancies in easy-to-read files automatically. As mentioned, this reduces the risk of downstream tasks using erroneous data critical for model training and prediction. Code versions and parameters for this pipeline are recorded. This allows us always to re-create the pre-processing step. It also provides an understanding of the data source we are training our models on. Here is a list of some of the information we record at this stage: Mean, median, and variance of features Data drift scores for each feature Percentage of null values in a column A plot produced to visualize the top drifting features across countries. Country and feature names have been omitted. A plot produced to visualize the percentage of fields that have missing values in the Documents. These fields are used to produce features. Country and field names have been omitted. Model training We use MLflow to record model training. This can be to track multiple experiments during the prototyping phase of a project or be used as a reference for models in production. All the required information is recorded so that models can be re-produced if required. Here is a list of some of the information we record when training models Model hyperparameters Test metrics such as precision and recall Confusion matrix An ROC curve (receiver operating characteristic curve) produced against the test set during training. A cumulative gains curve produced against the test set during training. Staging model evaluation We often iterate on our production machine learning models to improve their performance. We do this by adding more labeled examples to their training datasets or by adding more features. When we train a new model that we think should replace the existing model, we want to ensure that the new model is superior to the model we are replacing. We can look at the performance of the model across a test set, but sometimes this does not tell the full story. This is particularly true if the amount of labeled data we have is limited. Before upgrading to any new model, we run an evaluation across the entire unlabeled dataset to see how the models differ in practice. This is important for understanding the real business impact of a change. This evaluation is recorded in MLflow such that the data scientists and any relevant stakeholders can review it. Here is some of the information we record during the evaluation: The average size of differences in model score between the old model and new model Individual examples with the biggest differences in the model score between the old model and the new model Individual examples with SHAP explainability plots Explainability plot produced when the model is used for inference across unlabelled data. Feature names have been omitted. Visualization of the different scores produced by a model currently in production compared to the model selected to replace it. Conclusion In the dynamic realm of machine learning, effective experiment tracking is indispensable, especially with growing organizational scale. At Quantexa, MLflow has emerged as our solution, simplifying the ML lifecycle and addressing critical challenges in reproducibility, collaboration, and data correctness. Leveraging MLflow enhances our internal workflows and empowers clients to build models on Quantexa data to navigate complexities seamlessly.
Quantexa's AI Roundup - 2023
In July 2023, Quantexa announced a significant investment into its Artificial Intelligence (AI) capabilities (Quantexa Bringing Total Investment in AI R&D to over $250M by 2027). Since this announcement, there has been significant advancement in the AI space, and growth in some of the core AI capabilities at Quantexa. Alongside the significant growth of the NLP capability, Quantexa’s Analytical Innovation team have completed the MVPs of their three flagship products which are now released under experimental. These tools use Quantexa networks to uncover insights: the Entity Resolution AI suite; Q-Knowledge Graph and Shell Company Detection. In this round up post, we introduce the three products and demonstrate how they can add value to your Quantexa deployment. The Entity Resolution (ER) AI Suite The ER AI suite provides a series of tools for analysing the outputs of Quantexa’s ER product and provides suggestions for improving the configurations powering the ER using AI. In particular, the tool can detect overlinking and underlinking in Quantexa Entities and their root causes. The overlinking detection tool is powered by machine learning with features based on the qualities of the Entity’s constituent record-compound graph (read more about using the Entity Quality Overlinking tool for the first time). These graph-based features include the use of several complex graph algorithms (e.g., the Stoer-Wagner algorithm) to find shapes which are indicative of overlinking. Such shapes include ‘bridges’ in the network which incorrectly link Entities together, as well as graphs with very long paths. Statistical techniques can then be applied to determine what compounds or data points may be leading to this overlinking. The underlinking tool uses sophisticated graph algorithms to find ‘Super-Entities’ – Entities which should be formed of several existing Entities. This helps the user to identify template changes to merge such entities together in future ER runs. Q-Knowledge Graph Q-Knowledge Graph is a series of tools for analysing large-scale Quantexa Entity and Document graphs. It scales to billions of nodes and edges and uses sophisticated optimisation techniques to provide extremely fast implementations of core graph transformations and algorithms (including page rank). Not only does the tool provide access to commonly used graph algorithms out of the box (for example, PageRank) – it also provides a connection to common graph learning libraries such as PyG. This enables several use cases across Risk, KYC and MDM and has already been deployed for transactional use cases in a global bank. It will also be a core back-end component of a number of upcoming Quantexa AI products. Shell Company Detection The Shell company detection tool uses machine learning to identify shell companies, using characteristics of the local ego-networks of the companies. The model uses a combination of structural features (e.g., links to known shell directors); temporal features (e.g., patterns of director resignation) and static features (including the size of the corresponding corporate registry Document). For more information, see What can Network structure tell us about risk? The current model is built specifically for the UK and Singapore and can encapsulate some behaviours specific to shells in these jurisdictions. Models focused on other jurisdictions are coming this year. Upcoming AI releases The NLP team at Quantexa are also developing a machine learning pipeline called Text2Networks for working with unstructured data, which will be available in the next major release of Quantexa. The Text2Networks pipeline is a highly-configurable pipeline of ML models for mapping any unstructured textual data into a graph. The pipeline detects, labels and organizes people, places, and things in the real world – the supported Entity types include People, Locations, Companies and Geo-political organizations. With text2networks integrated into the core Quantexa product, our users will be able to incorporate any textual data source that is important for their business. Concrete example could include global news, intelligence reports, and Suspicious Activity Reports (SARs). There are several tools in development, including further tooling within the ER quality suite and Q-Knowledge Graph, as well as other risk models such as the SME detection tool which will be coming in later releases of Quantexa. To keep up with the latest releases, be sure to follow our Release Announcements topic.
5 Key Insights from the Webinar Bringing Knowledge Graphs to Life
Here are five key insights in case you missed the recent Quantexa webinar, Bringing Knowledge Graphs to Life 1. Introducing Knowledge Graphs The webinar gave a comprehensive introduction to knowledge graphs and why they’re hot right now. , , , and delved into how knowledge graphs, enabled through improved technology, bring a profound representation of entities and their relationships at scale, fostering great opportunities for impactful analysis and visualization. Knowledge graphs are applied in different ways, for graph analytics, semantic encoding, and delivering context to Large Language Models (LLMs) in AI. 2. Advantages and Challenges of knowledge graphs While there are numerous advantages of knowledge graphs, there are challenges on the journey. Depending on where and how data is sourced, knowledge graphs can be incomplete. The team emphasized the critical necessity of data quality and efficient entity resolution, because any failures diminish the ability to extract accurate and novel insights. Also, as knowledge graphs are normally very large, that compounds the need for accurate entities given the storage and compute investment, but also brings challenges of interacting with, visualizing and analyzing graph information, as Aaron noted, important for data scientists. 3. Is a Graph Database Needed? The panel noted the perceived intersection of knowledge graphs with graph databases, a domain well-marketed by graph database vendors. Ben remarked on how graph databases efficiently store and query knowledge graph data, but as knowledge graphs adapt with your organization’s data and needs, it’s important not to lock information away and be constrained by a single database. Ana highlighted how knowledge graphs can and should work across your organization’s data platforms and software. Aaron noted how data scientists, who thrive on iterative ad-hoc investigation and batch processing, benefit from direct access to knowledge graph structures. 4. Practical Implementation of knowledge graphs Knowledge graphs can be implemented across a wide spectrum of industries ranging from drug discovery to telecommunications and supply chains, and into functions like risk modeling, fraud detection, sales and marketing opportunities. Whatever the use case, it’s only a hop, skip and a jump from mainstream “tabular thinking” to “thinking graph,” given the elevation of expanded relationship information, i.e., edges, across entities, i.e. nodes. 5. Transformational Effects of knowledge graphs The panelists shared how knowledge graph technology had revolutionized their respective roles before and during their time at Quantexa. Ana pointed to the innovation opportunities leading to increased work satisfaction. Ben too appreciated how knowledge graphs unraveled unique solutions and allowed them to tackle complex problems. Steve pondered how the evolution of computational knowledge can drive change and unlock value across decision-making, science and engineering. This webinar is well worth watching to learn about the increasingly prominent role of knowledge graphs in data analysis, AI and decision-making. View the full webinar recording at Bringing Knowledge Graphs to Life Explore QKnowledgeGraph capability in the Quantexa Documentation
📖 New Article: Decision Making Beyond Local Graph Context
When investigating an Entity in the Quantexa Graph, how do you ensure you're not missing critical context? Expanding through the Graph can reveal important connections, but going too far can lead to information overload. This new article by explores how Multi-hop Graph Analysis helps investigators uncover hidden relationships by analyzing the Graph in its entirety—beyond just proximity. Using Quantexa’s Knowledge Graph framework, Matt explores an example of how this approach identified 65 new Entities linked to financial crime that weren’t flagged in public investigations. 🔎 Learn how Multi-hop Graph Analysis enhances investigations, from AML to risk management and beyond. 📖 Read the full article: Decision making beyond local Graph context - Quantexa Community Background When investigating an Entity in the Quantexa Graph, it is common for an investigator to start from the Entity in question, and expand through the Graph along related documents and Entities to find relevant linked information. This is standard practice in many contexts, for example in various risk applications,… You can also watch the webinar recording to hear from Matt directly: How Knowledge Graphs Help Identify Bad Financial Actors and Commercial Opportunities at Scale In this session, you’ll learn how multi hop graph techniques explore proximity and paths to illuminate relationships and communities of interest.Resources for Data Scientists 📈
Here are the best posts and articles from around the Community for Data Scientists: 📣 Announcements & Discussions Quantexa's AI Roundup - 2023 MLflow at Quantexa Welcome to Quantexa 2.6 | 2.6.0 Release Announcement Advancing Correspondent Banking Insights: Identification of non-customer entities How can Quantexa networks enhance your machine learning models? Rethinking MDM and Our Approach to It - Complete Blog Series is out! 📚 From the Community Library Predicting Risk through Network Shape Realistic Synthetic Graph Generation Using Graph Context at the Document Level What can Network structure tell us about risk? Automatic Data Cleaning Through Data Normalisation and Statistics Revolutionizing SME Classification: Quantexa's Scalable Solution Introducing QPython Scoring Concepts: Network Generation & Design Using QPython for Analytics and Data Science Teams Is there a topic you'd like to see covered that we haven't included here? Comment below ⬇️ or contact the Community team at community@quantexa.com.Using QPython for Analytics and Data Science Teams
Harnessing the full potential of your data is critical to driving impactful insights and innovation. QPython, Quantexa’s Python SDK, empowers data scientists to build cutting-edge ML and AI models by leveraging Quantexa’s robust Network features and Knowledge Graphs. In this guide, we’ll walk you through extracting and flattening documents and entities from Parquet files, re-ingesting model outputs into Quantexa, and implementing QPython in your workflows. Whether you're new to QPython or looking to refine your expertise, this article will equip you with the knowledge to deploy it effectively and elevate your data science initiatives. See the full guide (login required): Using QPython for Analytics and Data Science Teams - Quantexa Community Harnessing the full potential of your data is critical to driving impactful insights and innovation. QPython, Quantexa’s Python SDK, empowers data scientists to build cutting-edge ML and AI models by leveraging Quantexa’s robust Network features and Knowledge Graphs. In this guide, we’ll walk you through extracting and…Explore the Latest Articles in the Community Library 📚
Our ever-growing Community Library is filled with articles, blogs, and useful resources. Check out the latest articles below, whether you’re just starting out with Quantexa or a seasoned pro, there’s something here to help everyone learn and flourish. Service Operations 📖Quantexa Monitoring Series 📖Quantexa Platform Monitoring 📖Quantexa Platform Monitoring - Key Metrics and Log Entries 📖Quantexa Platform Monitoring - Moving Beyond the Minimum 📖Quantexa Application Monitoring - Introduction 📖Quantexa Application Monitoring - Getting Started with Toolchains Service Build and Transition 📖2.7 Quantexa Upgrade Guide 📖How to make the most out of Intellij IDEA 📖Running a Data Discovery Process 📖When and How to Use DQA Statistics Functionality 📖Project Management Best Practices: An Upgrade Journey 📖Docker for Quantexa Implementations 📖Spark Cluster - Resource Management Service Design 📖Setting Up Infrastructure and Underlying Platforms 📖Setting Up Infrastructure and Underlying Platforms: Cloud Edition 📖Setting Up Infrastructure and Underlying Platforms: On-Premise Edition 📖Using QPython for Analytics and Data Science Teams 📖Quantexa Platform Security Design for Solution Architects
Data Quality is STILL a fundamentally unsolved issue!
Having met a couple of customers in the banking sector this week in South East Asia, a real pain that have been constantly shared was data quality. That's due to incomplete information, inconsistency, lack of standardized entries and so on. Issues that we all probably know of. Not that's a surprise of any sort, but the fact that traditional ways are still used to tackle the ever existing and growing issue is what puzzles me. In an ever evolving world where we are witnessing advancements in AI and other areas at unprecedent pace, and seeing organizations still struggle with foundational challenge should not be the case. It goes without saying the importance of proper data foundation. Anything less than that would lead to improper analytics, entity resolution, decisioning, etc. I'll probably need to (and will) write a blog about this in the coming weeks. I'm keen to hear from anyone about their approach/vision in tackling data quality issues, and what are some of your most pronounced challenges? Cheers
