Sr. Data Engineer (Data Lakes)

About Us:

At Mass General Brigham Digital, we pride ourselves on our ability to create maximum strategic, clinical, and operational value from established and emergent technologies for our patients, care teams, researchers, and employees. Digital health will not only enhance the equity and efficiency of healthcare delivery, but it will also help make medicine more personalized and precise.

We recognize that increasing value and continually improving quality while maintaining an inclusive focus are essential to organizational excellence, and we invite you to join us on this journey. The work we do in Digital is a strategic imperative, and there is a strong and growing understanding of how together we will transform Mass General Brigham in innovative and impactful ways.

General Summary/ Overview:

At Mass General Brigham Digital, we pride ourselves on our ability to create maximum strategic, clinical, and operational value from established and emergent technologies for our patients, care teams, researchers, and employees. Digital health will not only enhance the equity and efficiency of healthcare delivery, but it will also help make medicine more personalized and precise.

We recognize that increasing value and continually improving quality while maintaining an inclusive focus are essential to organizational excellence, and we invite you to join us on this journey. The work we do in Digital is a strategic imperative, and there is a strong and growing understanding of how together we will transform Mass General Brigham in innovative and impactful ways.

Summary:

Reporting to the Engineering Manager, Data Lake, the Senior Data Engineer (Azure Data Lake) will work towards analyzing, designing, developing, and building ADF data pipelines, ELT/ETL frameworks, and Azure data lake platforms, primarily focusing on Epic (EHR) data and other healthcare data; and will thrive as a member of an experienced, high performing and highly motivated team. Role will be responsible for participating in building out our existing EDW and our new Data Lake, expanding our data ecosystem and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. Requires advanced experience with data engineering and building Azure Cloud Data Lake, Azure Big Data Analytics technologies and architecture, Enterprise Analytics Solutions, and optimizing 'big data' data pipelines, architectures, and data sets. Expert level of experience with Design and Architecture of Azure big data frameworks/tools: Azure Data Lake, Azure Data Factory, Azure Data Bricks, Azure ML, SQL Data Warehouse. Advanced Experience with Hadoop based technologies (e.g., hdfs, Spark) and Programming experience in Python, SQL, Spark.

Principal Duties and Responsibilities:

Design, Develop, construct, test and maintain Data Lake architectures and large-scale data processing systems.

Support big data ecosystem related Tool selection and POC analysis.

Gather and process raw data at scale that meet functional / non-functional business requirements (including writing scripts, REST API calls, SQL Queries, etc.).

Develop data set processes for data modeling, mining and production.

Integrate new data management technologies ( Informatica DQ..) and software engineering tools into existing structures.

The candidate will be responsible for participating in building out our Data Lake platform, expanding and optimizing our data ecosystem and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams.

The ideal candidate is an experienced data pipeline builder who enjoys optimizing data systems and building them from the ground up.

The Data Engineer will support our Software Developers, Database Architects, Data Analysts and Data Scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.

They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products.

Create and maintain optimal data pipeline architecture, assemble large, complex data sets that meet functional / non-functional business requirements on cloud based data platforms (e.g. Azure) and relational data systems (SQL Server, SSIS).

Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, etc.

Build the data infrastructure required for optimal extraction, transformation, and loading of data from traditional/legacy data sources.

Work with stakeholders including the Management team, Product owners, and Architecture teams to assist with data-related technical issues and support their data infrastructure needs.

Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.

Use/s the Mass General Brigham values to govern decisions, actions, and behaviors. These values guide how we get our work done: Patients, Affordability, Accountability & Service Commitment, Decisiveness, Innovation & Thoughtful Risk; and how we treat each other: Diversity & Inclusion, Integrity & Respect, Learning, Continuous Improvement & Personal Growth, Teamwork & Collaboration.

Working Conditions:

This is a remote position.

Diversity Statement

As a not-for-profit organization, Mass General Brigham is committed to supporting patient care, research, teaching, and service to the community. We place great value on being a diverse, equitable and inclusive organization as we aim to reflect the diversity of the patients we serve. At Mass General Brigham, we believe in equal access to quality care, employment and advancement opportunities encompassing the full spectrum of human diversity: race, gender, sexual orientation, ability, religion, ethnicity, national origin and all the other forms of human presence and expression that make us better able to provide innovative and cutting-edge healthcare and research.

5+ Years of experience data engineering and building Azure Cloud Data Lake technologies and architecture, Enterprise Analytics Solutions, and optimizing 'big data' data pipelines, architectures and data sets.

5-7 Years of Experience with Hadoop based technologies (e.g. hdfs, Spark). Spark Experience desirable

5+ years of Programming experience in Python, SQL, PySpark.

Healthcare experience, most notably in Clinical data, Epic, Clarity, Caboodle, Payer data and reference data is a plus but not mandatory.

Experience with Design and Architecture of Azure big data frameworks/tools: Azure Data Lake, Azure Data Factory, Snowflake, Azure Data Bricks, Powershell.

Experience with Design and Architecture of relational SQL and NoSQL databases, including MS SQL Server, Cosmos DB.

Experience with Design and Architecture of data security and Azure security, VM, Vnet.

Experience with building processes supporting data transformation, data structures, metadata, dependency and workload management.

Experience leading and working with cross-functional teams in a dynamic environment.

Experience building Big data pipeline with Spark and/or Data Bricks is a plus.

Leading development of Data Lake Architectures from scratch.

Experience with Azure DevOps/CI-CD, Continuous integration and deployment.

Experience with Real time analytics on Spark, Kafka, Event Hub is a plus.

Experience in petabyte scale data environments and integration of data from multiple diverse sources.

Skills/Abilities/Competencies:

Advanced hands-on SQL, Spark, Python, pySpark (2+ of these) knowledge and experience working with relational databases for data querying and retrieval.

Strong SQL skills on multiple platform (preferred MPP systems).

Data Modeling tools (e.g. Erwin, Visio).

Strong interpersonal and communication skills, both written and verbal.

Strong Scrum/Agile development experience.

Excellent organizational skills and attention to detail, manage multiple tasks and projects, meet deadlines, follow through, and manage to schedule.

Strong innovation capabilities and the ability to think creatively.

Strong collaboration and team building skills within, across and outside of an organization.

Maintain and promote a positive team environment.

Maintains stable performance under pressure, demonstrating sensitivity to diverse organizational culture.

Ability to effectively cope with change, remain flexible and adaptable within a fast-paced environment with rapidly changing requirements, and ability to negotiate situations when the big picture is not clearly defined.

Mass General Brigham is an Equal Opportunity Employer. By embracing diverse skills, perspectives, and ideas, we choose to lead. All qualified applicants will receive consideration for employment without regard to race, color, religious creed, national origin, sex, age, gender identity, disability, sexual orientation, military service, genetic information, and/or other status protected under the law. We will ensure that all individuals with a disability are provided a reasonable accommodation to participate in the job application or interview process, perform essential job functions, and receive other benefits and privileges of employment.