FAQ: I'm missing data in Elasticsearch / my number of docs are wrong

Dan_Pryer

FAQ relevant for: all Academy versions

If you have completed the ETL pipeline stages of your project and uploaded the data to ElasticSearch, then when checking your indices in the ElasticSearch Head plugin on Chrome you should have numbers similar to the picture below (to get a bigger version of the image, right click it and chose the option to open it in a new tab).

Note: If your numbers vary a little bit from these, for example having 152k address instead of 148k, then that's ok - the numbers will change a little for the resolver indices (Individual/Address/Business) based on the compound keys you have imported in the respective *.qentity Fusion config files.

If your numbers are significantly different to this, then you will want to go back through your ETL pipeline and carefully check each stage to see if there is somewhere that you lose the data along the way. A good way to approach this problem is to work forwards from CreateCaseClass and check the output of each stage to find the problem area. You should also use the counts in ElasticSearch to guide you - for example if you have only half the number of businesses listed above, and no individuals, it lets you know that you probably haven't joined your Third Parties onto the ICIJ document properly, and so you would want to go back and double check how you have done this join and on what fields.

Specific points to consider:

• Have I correctly parsed all of the necessary fields in my qmodel files?
• Have I used the correct type of joins in CreateCaseClass, and have I joined on the correct fields?
• Have I outputted the correct Dataset at the end of CreateCaseClass?
• Have I loaded up the DocumentDataModel.parquet (the output of CreateCaseClass) into a Spark-Shell to check the output there?
• Have I correctly identified and defined all relevant start paths in my qentity files?
• Do I have a good range of compound keys for each Entity?

If you are convinced that you have done all of the above correctly then you can try to clear the data from ElasticSearch, restart the service and then re-upload the data to Elastic using the following three commands:

curl -X DELETE 'http://localhost:9200/_all'

sudo systemctl restart elasticsearch.service

./runQSS.sh -s com.quantexa.academy.task.icij.model.etl.IcijLoadElasticScript -c ../external.conf -r elastic.icij