Senior Data Engineer

About the Team

The Data Platform and Solutions team builds and maintains the data platform that ingests data from various data sources, transforms that data and maintains the business logic, then pushes out that data to power our reporting, our AI/ML features, our critical customer-facing applications and APIs, and fuels our GTM systems to empower CircleCI to be the premier data-driven company in the CI/CD space.

About the Role

CircleCI is seeking an experienced Data Engineer to own and further develop our data infrastructure, ingestion, and to help us build our data products. This will include data ingestion/validation/alerting, AI/ML systems, and customer-facing APIs. As the initial data engineering hire, you'll work independently while collaborating closely with both engineering and other data teams to build robust, scalable data systems. We’re seeking someone with a voracious appetite for solving complex problems and who can innovate and collaborate with teams across CircleCI.

Key Responsibilities

• Design and implement data architectures that support efficient data flow across the organization
• Build and maintain event-based logging systems to ensure comprehensive data capture
• Establish and enforce data contracts to maintain data quality and consistency
• Optimize data infrastructure costs while maintaining performance
• Create and maintain data pipelines using modern orchestration tools
• Partner with analytics engineers to ensure data availability and reliability
• Evaluate and implement new data tools and technologies as needed

Required Qualifications

• 5+ years of experience in data engineering roles
• Strong expertise in Python and SQL
• Proven experience with modern data warehousing solutions, particularly Snowflake
• Deep understanding of database systems, including PostgreSQL
• Hands-on experience with data integration tools like Fivetran
• Knowledge of CI orchestration tools like CircleCI or dbtCloud
• Proficiency with data pipeline orchestration platforms like Apache Airflow
• Experience with modern data transformation tools, specifically dbt
• Track record of implementing event-based logging systems
• Strong background in data modeling and architecture design

Required Skills

• Self-motivated with ability to work independently
• Excellent problem-solving and analytical skills
• Strong communication abilities to work effectively with both technical and non-technical stakeholders
• Experience in implementing and maintaining data quality standards
• Proven ability to optimize data infrastructure costs
• Experience with data contract design and implementation

Preferred Qualifications

• Experience being a founding member of a data engineering team
• Background in data security and compliance
• Experience with batch and microbatch data processing systems
• Knowledge of data observability tools and practices
• Experience with GPU infrastructure management and optimization
• Familiarity with ML model versioning and experiment tracking tools