Lead Data Engineer (InSights)

The Position

OPSWAT is looking for a Lead Data Engineer to take charge of our data engineering function for our MetaDefender Insights Threat Intelligence Offerings. This role will be focused on shaping the technical implementations and execution for a growing data engineering team. This role will be pivotal not only in building and managing scalable data pipelines but also in setting the technical direction for data architecture, processes, and feature sets. The ideal candidate will have experience with modern data platforms like Prefect, Airflow, and others, as well as expertise in streaming analytics, data enrichments, and a variety of data structures and data stores that will enable a fluid and dynamic discussion with stakeholders and Technical Product Management. You will be responsible for both hands-on technical work and aiding in the development of a technical roadmap to grow the team as our data capabilities and focus expands.

What You Will Be Doing

• Design, build, and maintain robust data pipelines for real-time and batch processing using platforms like Prefect, Airflow, etc.
• Develop and optimize data architectures that support data ingestion, transformation, enrichment, and storage, ensuring scalability and efficiency.
• Lead efforts in streaming analytics, enabling real-time insights and predictive modeling across diverse data sets with a focus on Infrastructure Analysis (internet scale) and threat actor tracking.
• Help define the technical vision for the data engineering function, including strategy for team expansion, tools selection, and best practices.
• Collaborate with product management to ensure data engineering solutions and approaches align with overall business goals and analytics needs both short and long term.
• Mentor and guide junior data engineers, establishing a culture of collaboration, continuous learning, and innovation.
• Implement monitoring, logging, and alerting for data pipelines to ensure reliability, quality, and accuracy of data outputs.

What We Need from You

• 5+ years of experience in data engineering with hands-on expertise in tools like Prefect, Airflow, and similar platforms.
• Proven experience with streaming analytics, data transformations, data structures, and various data storage solutions (SQL, NoSQL, etc.).
• Strong proficiency in Python, GO, Rust, SQL, and experience with cloud-based data ecosystems (AWS, Azure, GCP, etc.).
• Experience in leading or mentoring data engineering teams, with a demonstrated ability to define technical roadmaps and guide team growth.
• Excellent written and verbal communication skills in English.
• Ability to collect, organize, analyze, and present complex data with high attention to detail and accuracy.
• Ability to present technical topics to non-technical audiences, as you may be asked to present at conferences and to customers on our technical approaches and analytic processes.

It Would Be Nice if You Had

• Experience with Threat Intelligence and Cyber security problem domains.
• Experience with additional data orchestration tools and frameworks.
• Knowledge of Kafka, Kinesis, or other streaming platforms.
• Advanced capabilities in data visualization tools like PowerBI, Tableau, or similar.
• Familiarity with infrastructure-as-code tools like Terraform or CloudFormation for deploying and managing data infrastructure.
• A passion for driving data-driven decisions and fostering a collaborative data culture.