Senior Data Engineer – Azure Databricks / Cybersecurity Data Platform
Senior Data Engineer – Azure Databricks / Cybersecurity Data Platform
- Location: Charlotte, NC (Remote – Must Work EST Time Zone)
- Employment Type: Full-Time
Job Description
Key Responsibilities
- Design, develop, test, and support cloud-based data pipelines and data ingestion solutions
- Build and maintain the Cybersecurity Data Lakehouse (CyberDW) for centralized security data
- Develop ETL/ELT pipelines for ingesting and transforming cybersecurity data sources
- Implement data quality (DQ) rules, data lineage tracking, and governance standards
- Optimize data pipelines for performance, scalability, and reliability
- Work with data architecture teams to design logical and physical data models
- Collaborate with data owners and stakeholders to define technical requirements and key metrics
- Develop data views and datasets for downstream analytics and reporting
- Monitor and troubleshoot data pipeline performance and resolve data quality issues
- Participate in development planning, task assignments, and technical design discussions
- Mentor junior engineers and support team knowledge sharing
- Ensure batch processing and job scheduling standards are followed
Required Qualifications
- Bachelor’s degree in Computer Science, Data Engineering, Information Security, or related field
- 10+ years of experience in data engineering, data architecture, or data development roles
- 5+ years of experience working with Azure data platforms, including:
- Azure Databricks
- Azure Data Factory
- Azure Data Lake Storage
- Azure Functions
- Azure Monitor / Log Analytics
- Strong experience with ETL/ELT pipelines and large-scale data processing
- Proficiency in SQL, T-SQL, and database development (SQL Server or Oracle)
- Programming experience with Python, Java, or similar server-side languages
- Experience with CI/CD and DevOps tools such as Git, Jenkins, or Azure DevOps
- Experience working with REST APIs and microservices architecture
- Strong problem-solving and analytical skills
Preferred Qualifications
- Experience with data governance and metadata tools such as Collibra or IBM Infosphere
- Experience working with enterprise cybersecurity data (Identity Management, Vulnerability Management, Incident Response, etc.)
- Familiarity with security frameworks such as NIST or CIS
- Experience integrating data from enterprise platforms such as:
- ServiceNow
- CrowdStrike
- Microsoft Purview
- Azure Active Directory
- Knowledge of IBM Tivoli Workload Scheduler
- Experience with Power BI for data visualization
- Experience with SIEM platforms such as Microsoft Sentinel
Job Type: Full-time Application Question(s):
- What type of work authorization do you have? A visa, green card, or U.S. citizen status?
- Do you have 10+ years of experience in IT development, data engineering, data governance, or data architecture roles?
- Do you have at least 5 years of experience working with Azure data services such as Azure Databricks, Azure Data Factory, Azure Functions, or Azure Data Lake Storage?
- Are you proficient in SQL development (SQL Server, Oracle, PL/SQL, T-SQL) including stored procedures?
- Do you have programming experience using Python, Java, or a similar high-level programming language?
- Do you have experience with CI/CD and DevOps tools such as Git, Jenkins, or Azure DevOps, as well as building ETL/ELT data pipelines?
- Do you have experience working with enterprise cybersecurity or security-related data and security frameworks such as NIST, CIS, or similar standards?
- Do you have experience working with REST APIs, microservices architecture, and enterprise platforms such as ServiceNow or Azure Active Directory?
Education:
- Bachelor's (Required)
Location:
- Charlotte, NC (Required)
Work Location: Remote Apply tot his job Apply To this Job