Senior Data Engineer Job Specification

This job specification highlights the core activities and skills essential for Senior Data Engineers working with the Quantexa Platform.

We recommend that Senior Data Engineers have at least four years of industry experience as a Data Engineer or in an equivalent role.

For Senior Data Engineers working with the Quantexa Platform, the tools, responsibilities, and expertise recommended to meet industry standards are outlined.

For Hiring Managers, it serves as a framework for defining role requirements when recruiting candidates with Quantexa experience.

Important Note: These guidelines are for reference only, and companies may have additional or specific expectations for roles.

Day-to-Day Activities

• Writing efficient, fault-tolerant, production-level code for your deployment.
• Building and managing end-to-end data pipelines.
• Advising colleagues on technical concepts and translating them into actionable insights for decision-making.
• Collaborating to develop and implement solutions for complex challenges. → Modified
• Mentoring junior team members and collaborating with stakeholders to develop and implement solutions, ensuring best practices are followed.

Recommended Quantexa Skills & Experience

• Experience implementing any of the Quantexa modules.
• Technical & business understanding of how and why Quantexa solutions are implemented.
• Comfortable performing most of the Quantexa standard tasks, such as:
  • Parsers tuning
  • Entity Resolution Tuning
  • Complex Scoring
  • Decision Systems (previously Detection Packs)
  • Data Packs
  • Upgrades
  • Entity Reporting
  • Authentication and Authorization

Recommended Technical Skills

• Experience independently shaping and executing solutions for initiatives and challenges.
• Ability to persist in the face of roadblocks, dispatching them efficiently, and escalating when required.
• Advanced Scala, Spark, and Git.
• Proficiency in programming languages, particularly Scala, Java, Python, or similar.
• Experience in building and deploying production-level data processing systems.
• Experience with modern development tools and processes (e.g., Git, Gradle, Nexus) and automation/DevOps technologies (e.g., Jenkins, Docker, Bash scripting).
• Knowledge of cloud platforms (Google Cloud, Azure, AWS) and big data technologies (Spark, Hadoop, Elasticsearch).
• Understanding of testing frameworks and the ability to write unit and integration tests using libraries like ScalaTest.

Recommended Soft Skills

• Experience supporting or mentoring junior members within a team.
• Ability to identify and highlight risks, communicating them early; not just in your own work, but also what you notice in others’ work and during backlog refinement.
• Experience leading agile ceremonies. 
• Ability to step in and cover Technical Leads (TLs) where needed with a strong understanding of their priorities and project plans.
• Problem-solving mindset with the ability to tackle complex challenges.
• Strong communication skills for articulating technical concepts.
• Adaptability to work in rapidly changing client environments.

Read more about Senior Data Engineers.