Databricks Engineer Job at KForce, Purchase, NY

  • KForce
  • Purchase, NY

Job Description

DescriptionKforce has a client that is seeking a Databricks Engineer in Purchase, NY.Overview:We are seeking a skilled Mid-Level Databricks Engineer to join our data engineering team. The ideal candidate will play a key role in the migration of Hadoop workloads to the Databricks Lakehouse platform, contributing to the development, optimization, and monitoring of scalable and repeatable data solutions. You will work closely with cross-functional teams to ensure successful migration, enablement of advanced Databricks features, and ongoing performance improvements.Key Responsibilities:* Identify and categorize existing Hadoop jobs (ETL, batch, streaming) and data sources to support migration planning* Participate in selecting scalable and repeatable migration use cases for Minimum Viable Product (MVP) initiatives* Provision and configure Databricks workspaces, ensuring Lakehouse architecture and federation capabilities are enabled* Execute pilot migrations of representative Hadoop workloads using tools such as Databricks Migration Accelerator or partner solutions* Validate and monitor post-migration performance, cost efficiency, and data integrity* Enable and test advanced Databricks features (e.g., Liquid Clustering, Lakehouse AI monitoring, Serverless warehouse, Unity Catalog) to optimize data layout and pipeline health* Engage with data engineering, analytics, and governance teams to gather feedback and document learnings, blockers, and feature gaps* Track and report on key success metrics such as migration time, query latency, cost savings, and feature adoption* Contribute to the development of a phased roadmap for full-scale migration and advanced feature rolloutRequirements* Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent professional experience* 5-8 years of experience in data engineering, with hands-on experience in Databricks and Hadoop ecosystems* Proficiency in Spark (Scala or PySpark), ETL pipeline development, and data migration practices* Experience with Lakehouse architecture, Delta Lake, and Databricks advanced features (e.g., Lakehouse Federation, Liquid Clustering, Unity Catalog) is a strong plus* Solid understanding of cloud platforms (Azure) and data governance concepts* Strong analytical and problem-solving skills with attention to detail* Ability to work collaboratively in cross-functional teams and communicate effectively with both technical and non-technical stakeholdersPreferred:* Experience with Databricks Migration Accelerator or similar migration tools* Familiarity with synthetic data generation and pipeline health monitoring* Understanding of performance tuning, query optimization, and cost management on cloud data platforms* Ability to document migration patterns and contribute to best practices for broader adoptionJob TypeContractCompensation51 - $69

Job Tags

Similar Jobs

Garney Construction

Mechanical Pipe Installer Job at Garney Construction

GARNEY CONSTRUCTION A Mechanical Pipe Installer position is available at Garney. To be considered for this position you must be able...  ...collections mechanisms, generators, tanks, motors, shafting, conveyors, and rail systems Set and check anchor bolts and... 

ELECTRO MOTIVE FORCE INC DBA EMF FIRE SOLUTIONS

Fire Alarm Designer Job at ELECTRO MOTIVE FORCE INC DBA EMF FIRE SOLUTIONS

 ...Special Hazards Fire Suppression DesignerIn this role, you will design and create the finished representation for special hazard suppression systems...  ...based drafting.5 years of experience in NFPA 72 Fire Alarm Design or NFPA 2001 Clean Agent Design.Drug screening required... 

Keelson Strategic

Residential Security Agent Job at Keelson Strategic

 ...Details: Job Title: Residential Security Agent Location: Santa Barbara, CA Job...  ...Security Team Agent supporting a protective security detail in Santa Barbara, CA....  ...applicable laws Coordinate with and support executive protection element of the team Position... 

Novant Health

Registered Nurse Job at Novant Health

What We Offer:Registered Nurse- Intermediate Critical Care Unit Forsyth Medical Center - Winston-Salem, NC Schedule:7p-7a Part Time Nights Role Type: Clinical Support - Direct Patient CareJoin our 20-bed Intermediate Critical Care Unit at Forsyth Medical Center... 

Sheetz

HVAC Installation Technician Job at Sheetz

 ...5.66 per hour, depending on experience and qualifications, plus bonus based on company performance.One of the MANY work perkz at Sheetz is quarterly employee bonuses based on company performance! And theres more A LOT more like competitive salaries, PTO and parental...