Databricks Architect

We are looking for a architect level profile for a DE openinng with 10-15 years relevant experience.

Please forward a suitable profile for this urgent opening along with the pricing.

Key Responsibilities:

1. Databricks Genie & AI Curation

Design and manage Genie Spaces, ensuring high accuracy in natural language query responses.

Develop and maintain Trusted Assets (pre-defined SQL functions and views) to guide Genie’s logic.

Iteratively refine Genie performance by analyzing query logs and enhancing semantic metadata.

Establish "Custom Instructions" to align AI responses with specific corporate terminology and KPIs.

2. Architecture & Governance

Lead the transition to Unity Catalog, ensuring a unified governance layer for files, tables, and AI models.

Architect end-to-end Medallion Pipelines (Bronze/Silver/Gold) using Delta Live Tables (DLT).

Implement robust security protocols, including row- and column-level security within the Lakehouse.

3. Data Engineering Excellence

Optimize PySpark jobs for performance and cost-efficiency.

Lead legacy-to-Databricks migrations, focusing on automated metadata conversion and logic validation.

Collaborate with Data Scientists to integrate MLflow and Feature Stores into the production architecture