Experienced Remote Data Engineer – Big Data Processing, Cloud Migration, and Distributed Systems Expertise
Introduction to PulsePoint and Our Mission
Imagine being part of a leading healthcare ad technology company that leverages real-world data in real-time to optimize campaign performance and revolutionize health decision-making. At PulsePoint, we're committed to delivering unparalleled results to our clients by targeting healthcare professionals and patients with an unprecedented level of accuracy. As a part of Internet Brands, a KKR portfolio company and owner of WebMD Health Corp, we're poised for exponential growth and seeking talented individuals to join our team. Our Data Engineering team plays a vital role in driving this growth, and we're excited to invite a skilled Data Engineer to join our ranks.
About Our Data Engineering Team
Our Data Engineering team is responsible for designing, building, and maintaining reliable and scalable enterprise-level distributed transactional data processing systems. We process over 80 billion impressions a day, generating over 20 TB of data (200 TB uncompressed), which is used to drive our optimization engines, update budgets, and provide real-time statistics and reports. We're a team that thrives on adopting proven open-source technologies to make our data infrastructure more nimble, scalable, and robust. Some of the cutting-edge technologies we've recently implemented include Kafka, Spark Streaming, Presto, Airflow, and Kubernetes.
Job Overview
As a Data Engineer at PulsePoint, you will be responsible for designing, building, and maintaining large-scale data systems, optimizing data processing workflows, and ensuring data quality across our systems. You will collaborate with a small team of diverse technology backgrounds, providing mentorship and guidance to junior team members. Your expertise will be crucial in driving our business growth, supporting new initiatives, and enhancing our data infrastructure.
Key Responsibilities
- Design, build, and maintain reliable and scalable enterprise-level distributed transactional data processing systems for scaling existing business and supporting new business initiatives
- Optimize jobs to utilize Kafka, Hadoop, Presto, Spark, and Kubernetes resources in the most efficient way
- Monitor and provide transparency into data quality across systems (accuracy, consistency, completeness, etc.)
- Increase accessibility and effectiveness of data (work with analysts, data scientists, and developers to build/deploy tools and datasets that fit their use cases)
- Collaborate within a small team with diverse technology backgrounds
- Provide mentorship and guidance to junior team members
Team Responsibilities
- Ingest, validate, and process internal and third-party data
- Create, maintain, and monitor data flows in Python, Spark, Hive, SQL, and Presto for consistency, accuracy, and lag time
- Maintain and enhance framework for jobs (primarily aggregate jobs in Spark and Hive)
- Create different consumers for data in Kafka using Spark Streaming for near-time aggregation
- Tools evaluation
- Backups/Retention/High Availability/Capacity Planning
- Review/Approval - DDL for database, Hive Framework jobs, and Spark Streaming to ensure they meet our standards
Technologies We Use
We're proud to work with a range of cutting-edge technologies, including:
- Python - primary repo language
- Airflow/Luigi - for job scheduling
- Docker - packaged container image with all dependencies
- Graphite - for monitoring data flows
- Hive - SQL data warehouse layer for data in HDFS
- Kafka - distributed commit log storage
- Kubernetes - Distributed cluster resource manager
- Presto/Trino - fast parallel data warehouse and data federation layer
- Spark Streaming - Near-time aggregation
- SQL Server - Reliable OLTP RDBMS
- GCP - BigQuery for performance, Looker for dashboards
Requirements
To be successful in this role, you'll need:
- 5+ years of data engineering experience
- Fluency in Python and SQL
- Experience in Scala/Java is a plus (Polyglot programmer preferred!)
- Hive/Presto experience
- Proficiency in Linux
- Strong understanding of RDBMS and query optimization
- Passion for engineering and computer science around data
- Ability to work during East Coast U.S. hours (9am-6pm EST preferred, but we can be flexible as long as you can work until 1 or 2pm EST)
- Knowledge and exposure to distributed production systems (e.g., Hadoop)
- Knowledge and exposure to Cloud migration (AWS/GCP/Azure) is a plus
Selection Process
Our selection process typically involves the following steps:
- Recruiter Screen (30 minutes)
- Hiring Manager Interview (45 minutes)
- Tech challenge take-home
- SQL & Python Interview (60 minutes)
- Team Interviews (60 minutes + 3 x 45 minutes) + SVP of Engineering (15 minutes)
- WebMD Sr. Director, DBA (30 minutes)
Career Growth Opportunities and Learning Benefits
At PulsePoint, we're committed to helping our employees grow and develop their skills. As a Data Engineer, you'll have the opportunity to work with cutting-edge technologies, collaborate with a talented team, and contribute to the growth and success of our company. You'll also have access to training and development programs, mentorship, and opportunities to attend industry conferences and events.
Work Environment and Company Culture
We're proud of our friendly and dynamic work environment, where everyone is encouraged to share their ideas and contribute to our success. We believe in fostering a culture of innovation, collaboration, and mutual respect, where everyone can thrive and grow. As a remote team, we're committed to staying connected and engaged, with regular virtual meetings, social events, and team-building activities.
Compensation, Perks, and Benefits
We offer a competitive salary and a range of benefits, including:
- Flexible working hours and remote work arrangements
- Opportunities for professional growth and development
- Access to cutting-edge technologies and tools
- Collaborative and dynamic work environment
- Comprehensive benefits package (details to be discussed during the hiring process)
Conclusion
If you're a motivated and talented Data Engineer looking for a new challenge, we'd love to hear from you. Don't miss this opportunity to join our team and contribute to the success of our company. Apply now, and let's shape the future of healthcare technology together!
We encourage you to apply, even if you don't feel like a perfect match. We're looking for great people to join our friendly team.
Apply for this job