The purpose of this article is to share some custom tests our Correspondent Banking project implemented using the Statistical Profile Testing (SPT) Framework. The SPT Framework includes some out-of-the-box test suites. However, unique deployment requirements can mean custom tests are sometimes needed.
What is the SPT Framework?
The SPT Framework is designed to produce statistics on different areas of the Quantexa solution which can be used for regression testing. These regression tests are examples of macro testing.
This is because they test statistics of large sets of data, rather than test specific details, such as a particular Network. The SPT Framework provides the tools to help collect, compare, and test data pipelines against previous runs or an accepted baseline.
Deployment Experience
For our solution, we added bespoke regression tests that use the statistics generated from the current and previous runs. These were then compared to see if any significant changes had occurred between runs. Any regressions were highlighted to us so we could take positive action to investigate the cause.
We took a look at the SPT Framework documentation and sample tests on Project Example. From this we identified that the default SPT Framework tests didn't offer the coverage we were aiming for our project in certain areas.Custom tests were developed in the following areas:
Extract Transform Load
The example Extract Transform Load (ETL) tests in this section were written for a Correspondent Banking solution. This falls under the Network Statistical Profile Testing category in the docs. The primary ETL pipeline consisted of SWIFT transactions that passed through the client bank.
The solution modeled and scored these transactions. In our case, it was important to ensure that the number and average value of payments did not increase over a certain threshold. We also performed some other basic checks like primary key field uniqueness. For context, the SWIFT transactions provided were:
- 103s - which contained transaction information such as beneficiaries and payment amounts.
- 202c - cover payments which contained information on the banks that the payments went through.
The following are some bespoke tests that check if there are regressions on the critical checkpoints in the ETL.
ETL Test Logic |
|---|
Number of consolidated Compounds >= Number of consolidated Compounds in the previous run |
Number of current Quantexa Batch Resolver input records >= Number of current Quantexa Batch Resolver (QBR) input records in the previous run |
Number of payments in the current run is within +- (Number of payments in the previous run * tolerance) Note: tolerance above is configurable in the script options |
Average GBP value of payments is within +- (Average GBP value of payments in the previous run * tolerance) |
103 to 202c join rate is within +- (103 to 202c join rate in the previous run * tolerance) Note: The client informed us that the join rate should be at least 20% each time |
Primary keys in QBR Input, Case Class, and Combined Case Class are always unique |
Scorecard
The example tests in this section were written for Scorecard level testing for a Correspondent Banking solution. The aim is to compare statistics from the current run with the statistics from the previous delta run and add tests to this comparison.
If there is relative certainty on what stats should look like, think about creating a set of statistics to use as a baseline or "golden source". We can then compare future statistics produced with that baseline. By comparing, we can be confident that the score trigger count does not digress through time.
The tests we implemented for our Scorecards are:
Scorecard Test Logic |
|---|
The counts of each Score ID triggered in the baseline run do not deteriorate past a threshold value in the current run. A test fails if not. |
If a Score is present in the current run but not in the baseline run, the user is notified but it does not result in a test failure. |
The default test “Confirms that Score IDs found in the baseline run are also present in the current run” is being turned off. This is because we don’t expect all the scores to trigger every time. |
We followed the instructions on the Quantexa documentation site to add our custom tests.
Alerting
The example Alerting tests in this section were written for Alert testing for a Correspondent Banking solution. The aim here is to compare statistics of Alerts from the previous run with the current run. We then added tests that would fail if the above comparison deviates over a certain threshold. For Alerting, the following tests were implemented:
Alerting Test Logic |
|---|
Number of Alerts in the current audit > Number of Alerts in the previous audit Note: The aim of the above is to check that our audit backup is always working |
The number of Alerts in the current audit for this run is within +- (previous * tolerance) |
Conclusion
The implementation of the SPT Framework was relatively straightforward. The documentation was helpful and clear on the implementation, and the technical details on how tests should be added. In most cases, we found the default regression tests covered a lot of what we needed to test.
We also wanted to add the specific tests in this article to cover some critical project-specific areas. The default tests were helpful when it came to developing our custom Scores. We were able to use them as a way of reference, to develop custom Scores quickly.
During the technical implementation, we only had some difficulty when adding the ETL regression tests. This was mainly because the ETL test suite was a bit different than the rest. In the ETL suite, there's no concept of "Accepted Baseline".
For this reason, you would have to calculate statistics in the same run. In other words, if "t" is the current run stats, then in the ETL suite you calculate t and t-1 stats. In other suites you calculate only stats at t and have t-1 from previous runs.
The more challenging part was deciding on what tests were needed and why. Please consult your Business Analyst and Team Lead (TL) as well as the Product Owner and Architect on what kinds of tests would be good to be added. These tests should monitor specific dependent regressions.
High-level requirements that come from the Product Owner are refined by the Business Analyst. They are enhanced by the TL and Architect to better monitor technical risks, and can be applied by the development team.
If you are unable to access the documentation site, please contact your Quantexa point of contact or the Community team at community@quantexa.com.
