This article discusses setting up User Defined Intelligence (UDI) within a deployment.
So what’s UDI all about?
A frequent comment from users is that they would like to add information to a Network but they are unable to do so. This feature enables users to create Documents themselves.
Why is the feature useful?
Enabling users to add information to a Network is very powerful as it can provide more context within an Investigation. There are two primary features of UDI:
Add Notes
This feature enables users to add a new Document that connects to one Entity. The user can add any relevant Notes or extra information connected to that Entity.
Figure 1. Add Notes
Merge Entities
This feature enables users to combine two Entities that have not been resolved through the configured Compound Keys. It is only possible to merge Entities of the same type. The feature works by creating a Merge Document containing a Compound that is associated with both Entities, causing them to resolve together.
Figure 2. Merge Entities
Deleting a UDI Document
There are two levels of deletion for a UDI Document. The first is to remove the Document from the Network. This is the same as deleting a resolver Document from an Investigation. The UDI Document is still present in Elasticsearch and in the database. It can be searched for and an expansion retrieves the Document. The second is to remove the Document from Elasticsearch. This means that the Document is no longer searchable or available for any users.
Figure 3. Delete a UDI
Being able to edit the Network is powerful and can introduce risk of incorrect information. The product owner should fully understand the feature, consider the business impact, and if an accompanying business process is required.
What did you find out during implementation?
The feature itself is easy to implement and the documentation is up to date.
Depending on Quantexa version, there may be a couple of prerequisites, such as upgrading to Gateway Reactive or using gRPC. The recommendation would be to have these prerequisites implemented and fully tested in separate tickets and sprints before starting UDI.
Design considerations
There are a number of design considerations to be aware of when implementing UDI.
Security considerations
As mentioned the ability for users to add information can potentially be risky. For example, a user could incorrectly merge two Entities together, or a user could write sensitive information in the Notes section. The security permissions allow UDIs to be created just for a single user or shared with other users. It is possible to restrict users from sharing their UDIs, this can be sensible if deployments are concerned.
It is not possible to set UDI permissions per type, both Merge and Notes have the same security Role. Therefore, a user either has permission to create and edit both types or neither. Within your deployment you can just enable Merge or just enable Notes, but it is not possible to have one user who can create Notes but not Merge, and a different user who can create Merge but not Notes.
Batch and Dynamic Network building and Scoring
UDIs are only supported in Dynamic Resolver. Batch Resolver does not support UDIs. They are included in Network Generation, providing UDIs are configured in the project.
For Scoring, some Scores are Batch look-ups, where the Score is written directly into an Elasticsearch Index which is then retrieved. For UDIs, these are not included in Network Scoring, but any Dynamic Scores would, depending on the Privileges.
Privileges for UDIs are defined at the Document level by the Role assigned to a user. Deployments are responsible for managing the Privileges in an appropriate way depending on their needs. For example, everyone with the Role USER could be granted access to UDIs, or they could only be made viewable to certain admins.
User Roles need to be given view access for all UDIs in an Investigation before they can open it. For example, if a user can see Investigation A but they cannot see UDI B, which is included in that Investigation, they are unable to open the Investigation, or view the UDI.
In brief, UDIs are not supported by Batch processes, only Dynamic.
Notes Document fields
You can define the case class for your Notes and Merge Documents. This enables you to control what users are allowed to enter if you wish. The majority of projects use the default case class, defined in the documentation, but that might not be best for your use case.
Figure 4. Demo Notes example
Additional information
The UDI Documents are both saved into the database and saved into a custom Elasticsearch index. If your Elasticsearch cluster fails it is possible to restore the index from the database. However, it is not possible to restore the database from the index, therefore you should ensure that your database is backed up. This is also true of other tables within the database such as the Investigation tables.
Build information
Version 2.0.0 TQP, Dec 2022
