Data Engineer
Colorado Springs, CO, United States
Job Description:
Role Synopsis
As part of the Data Platforms team, the Data Engineer position contributes to the overall BPX Data Platform Strategy. Define data workflow, pipelines, security, and ELT/ETL guidelines, policies, and procedures
Provide oversight to ensure that project teams are following the approved data workflow, pipelines, security, and ETL/ELT processing guidelines
Evaluate, compare, and recommend new Data Platform vendor products and/or tools
Partner with project teams to provide technical direction and new integration patterns, where needed, for complex and/or large data sets
Present modifications to Data Platforms to the Architecture Review Board for approval
Key Accountabilities Responsible for the development, guidance, and oversight for data ingestions and integrations for BPX cloud data platforms.
Work with project teams to provide direction on approved patterns to meet the project data requirements for our Data Platforms.
Strong SQL development knowledge required to provide support to project developers/users for database design, data flow and analysis activities.
Development and deployment of innovative big data platforms for advanced analytics and data processing.
Define and build the data rules and pipelines that will enable faster, better, data-informed decision-making within the business.
Ensure stable solutions, stable infrastructure, and participate in the design and build out of data solutions.
The database and data support roles are outsourced to a managed services partner that resides within the Data Platform team, and this role will require the ability to work and partner with those resources.
Create data storage solutions optimized for performance and maintain data architecture standards across BPX Data Platforms.
Partner with the Data Quality Lead to incorporate testing and validation scripts to support the accuracy of data transformations.
Implement Master Data Management solutions, processes, and governance to support the Data Management team.
Partner with the DataOps support team to troubleshoot data issues and provide guidance for solving these issues.
Drive excellent, consistent customer service.
Essential Education BS in Computer Science, MIS, Mathematics, or equivalent degree preferred but not required.
Essential Experience and Job Requirements 5+ years of Data Engineering experience
5+ years of relevant work experience in IT/Data and/or Analytics space
Experience in any cloud data warehouse, Snowflake Cloud preferred
Experience with replication tools, Fivetran preferred
Experience with transformation tools, dbt preferred
Experience with programming tools, python preferred
Experience with REST APIs for data ingestion
Strong understanding of ETL/ELT processing with large data stores
Experience designing and delivering large scale, 24-7, mission-critical data pipelines and features using modern big data architectures
Stream processing services such as Kafka, AWS Kinesis, Apache Storm, Spark Streaming, Azure Event Grid etc
Demonstrated experience working in large-scale data environments which includes real-time and batch processing
Strong data modeling skills (relational, dimensional, and flattened)
Strong analytical and SQL skills, with attention to detail
Ability to aid in tuning and performance recommendations for poor performing SQL queries and/or python scripts
Knowledge of Database Administration tasks Indexing, SQL Tuning/Performance, Backup/Recovery, DR
Ability to work with multiple external teams and accomplish shared goals by building consensus
Strong communication (written/verbal) and collaboration skills
Consulting, negotiation, and relationship skills
Problem solving skills