Why Your Data Integration Graph Needs Entity Resolution
What are the Benefits of a Graph Data Model? Graph has historically been a niche data management technology. Typically, graph databases were used for single applications with a data model well suited for graph. In recent years they have been used for small-scale analytics with a limited set of traditional graph analytics, such as centrality metrics. However, graph technologies are increasingly seeing enterprise scale take-up. Things are changing. Gartner has a strategic planning assumption that by 2025, graph technologies will be used in fully 80% of data and analytics innovations – that’s up from 10% this year. One of the major drivers is that the flexibility of graph data models is a natural way to interact with connected data to enable insights and decisions. However, to get real value from graphs, organizations need to solve the challenge that data is siloed across many applications. Organizations have traditionally tried to do this by forcing data into enterprise data warehouse schema or structured data lake layers, but there are challenges as these: Take a long time to design Are hard to adapt in an agile way as data from new parts of the organization needs to be incorporated. The right graph data models can enable key data to be onboarded, connected and used, while maintaining clear lineage back to source data. The Difference Between Context & Graph Data Models Graph data models are also ideal for representing the complex relationships that are present in the real world. Many questions aren’t just about who your direct customer is and how they’re interacting with you. It should also be about their relationships, what they share and how they interact with others – and then their relationships, as far out as is useful. Business customers are a great example of this – they often have complex, deep hierarchies, informal relationships through shared directors, supply chains and so on. But, particularly when integrating data from disparate sources, there is a big difference between simply having data in graph form, and having useful context for analytics, insight and decisions. Context Starts with Entities Firstly, what is an entity in a database? Context starts with building meaningful graphs that join data up via entities. Entities are representations of people, companies, addresses and many other types of thing in the real world. However, when you try to load a typical enterprise’s data into a graph database, it won’t join up, because enterprises generally operate as silos. Source data in each silo will have been captured separately by different teams, in different systems. Each will use different identifiers for referencing entities, so information about each real-world entity will be split into islands. Sometimes, earlier attempts will have been made to unify records, or data will come from systems that have been tightly integrated. In that case, you can use the explicit cross-systems IDs that are there to create links between records as they are loaded. But these are rarely complete – they might only apply to a handful of systems within a given area of the enterprise. A bank might have data relating to two customers coming from various CRM and product-specific systems across two divisions, for example a small business bank and a retail bank. Linking records up based on the explicit identifiers, for a set of companies and individuals that are closely related in the real world might look something like this: In this example, business accounts are siloed. There is some linkage via transactions between two customers, and another completely isolated customer. But overall, the graph is sparsely linked and lacks context for analysis, data science or operational decision-making. They need something more complete and more consistent that more closely represents the real world. Finding the Real World Entities In Your Data To get a more complete view, it’s important to look beyond the explicit IDs at the additional data within the records. Someone eyeballing the data might guess that Antoinette Banasiewicz, born 12/03/80 might be the same as Nettie Banasiewicz, 48 Second Avenue – and when they find a third customer record for Antoinette Banasiewicz, Appt. 2, 48 Second Av., they will be confident to link all three. Conversely, a member of staff might also know not to link John Smith, born 1/1/1960 with another record relating John Smith, born 1/1/60 – because that DOB is a default value in the customer system, and because John Smith is a common name in the UK. When trying to reconcile data in this way, they must also deal with inconsistencies in the way that data is structured, represented and captured. One data source might contain a handful of key attributes, another an overlapping but different set. An address might be represented one way on one system, another way on another – and the actual data values will often disagree. At that point it becomes necessary to specify rules for which values you’re going to trust – system A over system B, most frequently seen value, most recent record, sum of all the records or other variations. The Value of Entity Resolution All this is necessary because traditional technologies like Master Data Management (MDM) generally haven’t managed to deliver a complete and current view of key entities like ‘customer’ that is referenced from or reliably distributed to every system. Luckily, there is a category of product that does, and it’s called entity resolution. It parses, cleans and normalizes data and uses sophisticated Machine Learning and AI models to infer all the different ways of reliably identifying an entity. It clusters together records relating to each entity; compiles a set of attributes for each entity; and finally creates a set of labelled links between entities and source records. It’s dramatically more effective than the traditional record-to-record matching typically used by MDM systems. Rather than trying to link all the source records directly to each other, you can add new entity nodes (shown below in blue), which act as a nexus for linking real world data together. High quality entity resolution means you can link not only your own data, but also high value external data such as corporate registry information that in the past would have been difficult to match reliably. The result is: Entities created using effective entity resolution provide a much richer context, with many additional links. Retaining links to source data allows provenance to be understood and ensures important data isn’t lost. At Quantexa we use hierarchical documents to accurately represent related source data and refer to this approach as a document-entity model. The “full graph” linking all documents and entities is an entity graph. However, it can be too detailed for easy consumption by a user or systems. Transforming (or “projecting”) the graph by grouping together nodes and trimming edges according rules means that simpler views can be formed. At Quantexa we call this a perspective. Users switch interactively between different perspectives focusing on different types of node and relationship, or to the document entity view for the richest picture. A good example of a perspective is an ‘entity-to-entity view’: This sacrifices detail, but allows us to easily see the wider context: Antoinette, a retail customer, has been a guarantor for a loan to a business customer she is a director of, LogicSpace Ltd. Antoinette lives with and sends/receives money from another retail bank customer Jamie. He is a director of AJ Coworking, the parent company of LogicSpace. Understanding this context required resolving sometimes duplicated data across retail banking, small business banking and external corporate registry sources. Why Use Graph for Integrated Entity Resolution? Entities are the key foundation for building the right data integration graphs. However, the way those entities are resolved is also critical – this process needs to be accurate even in the face of data quality issues, scalable enough to handle enterprise-wide data , secure, and easy to transition into production for multiple use cases.
Ensuring our Customer’s Success: The Quantexa approach to driving value
Quantexa's mission is centered around unleashing the potential of customers data through technology. Our founders' vision was to drive outcomes for our customers through technology's possibilities, not just to deploy and sell technology. To assist customers on the value journey, the Quantexa Customer Office was formed. But how does the Quantexa approach to Customer Success accomplish this? In this article, we will delve into the four different pillars the Quantexa Customer Office focuses on to lead specific streams of impact with our customers: 1. Vision and Sponsorship At Quantexa, we engage deeply with our customers, not just as a technology vendor, but also as a trusted advisor. We want to understand why you purchased Quantexa, what your roadmap looks like, and we will be there as an advisor on the journey to continually unlock more value. This comes from strong governance sessions and a joint understanding of the outcomes and KPIs you wish to measure the value of the platform by. By keeping our eye on the outcomes we can make sure the correct work is ongoing to achieve them. Quantexa wants to hear from our customers and stay close to the ideas you have on how we can continue to enhance our platform to deliver value. We provide the Ideation Portal on the Quantexa Community as a service where our customers and partners can suggest ideas and have them up voted for consideration in the Quantexa roadmap. 2. Self Sufficiency Many of our customers begin with aggressive timelines to solve data problems or meet regulatory needs. In these cases, they choose the best of the best and bring Quantexa delivery services onboard. However, we understand that you do not want to be reliant on Quantexa consultancy on an ongoing basis. The Quantexa academy exists to upskill Data Engineers, Scoring Engineers, Business Analysts and Project Managers to allow our customers to build high performing Quantexa teams to take platform ownership. Our Customer Success and Delivery team will work with you to advise you on what your team should look like and typical profiles of data engineers and scientists. As part of this advice we will discuss the Centre of Excellence model (COE) with you and what it looks like to form a Quantexa COE. We also realise that our customer base do not want to completely move away from Quantexa technical support so we have expert services offerings that can support customers through advice and guidance on an ongoing basis alongside the Customer Success team. We also have our Community offering which will connect, inform and support customers and partners. The Quantexa Community is a global network which will serve as your go-to place to collaborate, find information, and knowledge share about Quantexa. We encourage you to sign in to share your experiences, celebrate your success, and get access to the following and more: Discover - view all Community discussions: Gain access to shared knowledge, explore best practices and browse conversations Learn - visit our Academy Topic: Find guidance on the relevant training paths and get instant access to a library of support content Connect - to peers in our Specialist User Groups: Gain help from other members, stay up to date with news & events and expand your network with peers who are using Quantexa for similar use cases within your industry Innovate - via our Ideation Portal: Influence Quantexa’s roadmap by suggesting and voting on new product features Quantexa also has a strong partner network with some of the largest and most innovative partners in the world. Many of our customers choose to have hybrid teams with partners supplying resources. On the Quantexa Community you will find our Partner Central Community Topic, a private area visible only to Quantexa Partners. Don’t forget to click the bell icon to subscribe to this Topic and stay up to date with announcements or ask questions about Quantexa's partner program. 3. User Adoption Although not used in every Solution, the Quantexa user interface provides a rich interaction palette to explore and make decisions from data. In many cases, this will be a new way of working for customers and once harnessed, will be transformational. Commonly we see customers moving from the interrogation of multiple spreadsheets to the exploration of data through tasks and network diagrams. Quantexa knows that the majority of the value is unlocked on user proficiency with the platform. We will engage with you to make sure you are putting the correct investments in place to achieve new ways of working through change management. While we are not a change management provider we can advise of the key workstreams we would see and introduce our partners if required. Why not check out our Community Library to get started which includes best practices, how-to articles for first time users and thought leadership blogs. 4. Technology The Quantexa platform is fast-moving from a functionality perspective, which means upgrades are essential to keep current and unlock future value. The Customer Office team will work with you to prioritize the outcomes that you can get from the continued use and upgrades of the Quantexa platform. As a Quantexa customer, you can subscribe to notifications to stay up to date with the latest releases by following our Release Announcements Topic on the Quantexa Community. We also realize that with technology, issues do happen, so Quantexa has various services through the Quantexa Community and support channels to assist. Summary ✍️ At Quantexa, we remain committed to our customers' success and are dedicated to driving outcomes. Our Customer Office team is here to support you in achieving your goals and unlocking the full potential of the Quantexa platform. In conclusion, the Quantexa Customer Office team's focus on Vision and Sponsorship, Self Sufficiency, User Adoption, and Technology is what sets us apart. By working together with our customers and applying these principles, we can drive business value. Did you find this post helpful? Please comment below if you have any suggestions on what you would like to hear about next!
Project Documentation to support you in upgrading Quantexa
One of Quantexa's priorities is to unlock the potential of your data while enabling customers to form their own self sufficient decision intelligence capability. Our Documentation team has outlined the requirements for upgrades, allowing you to prioritise needs against business benefit. This allows customers to assess through the following lenses: Which functionality update would help bring value to your organisation? Can additional functionality unlock other opportunities for Decision Intelligence in your organisation Are there any further technical or architectural requirements to consider when planning functional upgrades? How will these upgrades help with your end-user experience, or; Further the technical improvements of the platform? Where can you find this information? 📚️ As the Quantexa Releases are announced, the Documentation site explains the functional and non-functional updates, how they work, and how to implement the upgrades. As a Quantexa customer or partner, you will be able to access information around best practices to follow when upgrading your version of the Quantexa Platform on our Documentation site. In this section you will find guidance around Performing upgrades, Maintaining upgradable Quantexa implementations and Ongoing development during upgrades. Additionally, you can stay updated with the latest releases by subscribing to Release Announcements or reach out to your TAP (formerly CSMs & SSMs) to set up a meeting with the required team members. Quantexa can demonstrate the functional updates and discuss the benefits and effort requirements in order to help you plan out your platform roadmap, for your current needs or those you may have in the future. We are here to support you in any way you need. Additional resources For more information check out our blog: So you want to perform an upgrade? 💬 Do you have any questions or thoughts around scaling/upgrades that you would like to discuss with other Quantexa customers and experts? Ask your question in Quantexa Platform Support.
