Sacramento, CA, US
We are seeking a hands-on Lead Data Architect/Data Engineer for a hybrid assignment with our Sacramento, California-based client, helping define architecture and deliver production-grade pipelines across multiple phases of a Databricks platform.
Responsibilities include:
* Build and maintain automated ingestion and transformation pipelines that bring data from:
o public portals and open data sources
o REST APIs
o internal departmental databases and systems
into Delta tables in Databricks.
* Partner with the Program Manager to establish and maintain:
o platform architecture
o naming conventions and standards
o technical documentation
* Collaborate with the Data Hub governance group to support decisions on:
o platform and data architecture
o data sharing patterns
o data contracts and data ownership
o data quality and reliability standards
* Design recommendations for protecting sensitive data, including:
o row-level and column-level access controls
o separation of confidential/internal data from public datasets
* Improve reliability through:
o monitoring and alerting
o data quality checks and validation frameworks
* Enable discoverability and usability by publishing datasets and documentation that support plain-language exploration and AI-enabled analysis tools.
* Write and maintain skill files (instruction files for AI coding assistants) to enforce consistent pipeline patterns and development standards.
* Support and enable other departments to build pipelines using shared templates and best practices.
Requirements:
* Strong leadership experience, including executive-level stakeholder engagement
* 7 years in data engineering, data architecture, or closely related roles
* Proven experience delivering large-scale, enterprise data engineering / architecture initiatives
* Strong hands-on expertise in Python and SQL
* Experience with Apache Spark or cloud data platforms such as:
o Databricks (strongly preferred)
o Snowflake, BigQuery, or similar
* Experience integrating data from REST APIs, databases, and other source systems
* Excellent communication skills with strong writing and documentation practices
Preferred
* Hands-on Databricks experience with:
o Delta Lake
o Unity Catalog
o Databricks Jobs
* Experience governing sensitive/confidential data (access control, masking, privacy requirements)
* Experience operating within a data governance program (stewardship, data contracts, cross-agency sharing agreements)
* Experience writing skill files, prompts, or instruction sets for AI coding assistants
* Experience with GIS / location-based datasets