Best Of
FAQ: I'm missing data in Elasticsearch / my number of docs are wrong
FAQ relevant for: all Academy versions
If you have completed the ETL pipeline stages of your project and uploaded the data to ElasticSearch, then when checking your indices in the ElasticSearch Head plugin on Chrome you should have numbers similar to the picture below (to get a bigger version of the image, right click it and chose the option to open it in a new tab).
If your numbers are significantly different to this, then you will want to go back through your ETL pipeline and carefully check each stage to see if there is somewhere that you lose the data along the way. A good way to approach this problem is to work forwards from CreateCaseClass and check the output of each stage to find the problem area. You should also use the counts in ElasticSearch to guide you - for example if you have only half the number of businesses listed above, and no individuals, it lets you know that you probably haven't joined your Third Parties onto the ICIJ document properly, and so you would want to go back and double check how you have done this join and on what fields.
Specific points to consider:
- Have I correctly parsed all of the necessary fields in my qmodel files?
- Have I used the correct type of joins in CreateCaseClass, and have I joined on the correct fields?
- Have I outputted the correct Dataset at the end of CreateCaseClass?
- Have I loaded up the DocumentDataModel.parquet (the output of CreateCaseClass) into a Spark-Shell to check the output there?
- Have I correctly identified and defined all relevant start paths in my qentity files?
- Do I have a good range of compound keys for each Entity?
If you are convinced that you have done all of the above correctly then you can try to clear the data from ElasticSearch, restart the service and then re-upload the data to Elastic using the following three commands:
curl -X DELETE 'http://localhost:9200/_all'
sudo systemctl restart elasticsearch.service ./runQSS.sh -s com.quantexa.academy.task.icij.model.etl.IcijLoadElasticScript -c ../external.conf -r elastic.icij
Introducing the Quantexa User Research Panel
We are excited to introduce our User Research Panel, which will give you the opportunity to directly collaborate with our product department and provide feedback to us by taking part in early access testing of new features, product interviews and surveys.
We're just getting started and we'd like to invite you to be involved.
What's in it for you?
- Connect directly with Quantexa experts
- Get early access to hear what we're working on
- Have your say and help improve the Quantexa platform
- Directly influence the Quantexa roadmap
How do I join?
If you're interested in getting involved, we need to get a few bits of information from you, and then we'll be in touch. Fill in this form to join the panel.
Have an idea you want to share right away?
If you have questions or ideas you think we should explore in the future, share them in the comments below, or submit them in our Ideas Portal with the tag 'research'.
Help shape the future of Quantexa: Join the Panel
Re: 🎂 It's our First Birthday 🎂
👋 Someone Needs Help is one of the best parts of the Community that no question would ever be forgotten! 😆
🔧 Common Errors 🔧
Introducing Common Errors - a new series of articles troubleshooting error messages and issues that are frequently searched for by you - our Quantexa Community users!
These guides will provide debugging tips, configuration examples and references to relevant pages on the Documentation site or related Community posts, and will act as a resource to help you identify troubleshooting steps you can take before seeking any additional support.
Where do I find them?
- Jump directly to our Common Errors section of the Community Library, here - OR
- Click on the shortcut to ‘Common Errors’ under Quantexa Platform Links - you’ll find this on the right hand side of the Quantexa Platform Topic page OR
- Navigate via: Help & Learning > Platform Documentation > Support > Common Errors
Please log in, or sign up, to unlock the first (of many!) series of Common Errors articles:
- Troubleshooting on Dependency not found
- The investigation could not be fetched / timed-out
- Error loading libpostal parser modules
Otherwise known as: Could not find org.springframework.boot:spring-boot-starter-test:2.3.10.RELEASE.
of Christmas
Badge Hunt!