Databricks
About this role
Databricks sells the data and AI platform used by more than 10,000 organisations, including over half the Fortune 500. This role is on Lakeflow Connect, the part of the product that gets data into the platform in the first place through ready to use connectors for enterprise applications such as Salesforce, Workday, ServiceNow and SharePoint, plus databases, cloud storage, message queues and local files. The engineering problem is specific: extract data out of live OLTP systems while putting minimal load on them, which means incremental data capture, log parsing and similar techniques. The team wants engineers with database internals experience, and the work involves low level systems debugging and performance measurement on large production clusters processing hundreds of terabytes a day. Connect is also a platform capability every other Databricks surface depends on, so the role carries cross team scope.
Who this is for
What the posting requires:
- A BS or higher in Computer Science or a related field.
- 5+ years of production level experience in one of Python, Java, Scala, C++ or a similar language.
- Experience developing large scale distributed systems from scratch.
- Hands on experience developing and operating backend systems.
- The ability to contribute across all project phases, from design and development through implementation and ongoing operations, with guidance from senior team members.
Nice to have:
- Experience in database replication, backup or transaction recovery at a major database vendor such as Microsoft SQL Server, Oracle or IBM. The posting lists this as a plus, and it is the clearest signal of what the team actually values.
What the work actually looks like:
- Deliver a highly scalable, available and fault tolerant engine processing hundreds of terabytes of data daily across thousands of customers.
- Low level systems debugging, performance measurement and optimisation on large production clusters.
- Build architecture designs, influence the product roadmap and own new projects.
- Help prevent and investigate production issues.
- Plan and lead complex technical projects that span several teams.
- Break complex problems into solutions, knowns and unknowns, and de risk them through prototyping and validation.
Why this team exists: ingesting data into the Lakehouse is described as a strategic investment area, and every Databricks surface including dashboards, notebooks, SQL and AI needs ingestion, so the team works closely with other product teams to embed Connect into those surfaces.
Location and working pattern: Bengaluru. Databricks says it is building out its Bengaluru site with new teams, so expect an in office or hybrid arrangement. The number of days is not stated.
Compliance note in the posting: where a role requires access to export controlled technology or source code, Databricks says applying for a US government licence is at its discretion and it may decline to proceed on that basis. That clause is in the posting and is worth reading before you invest time.
Honest fit guidance: five years plus real distributed systems work is the bar. Database internals experience is what will separate applications here.
- A BS or higher in Computer Science or a related field.
- 5+ years of production level experience in one of Python, Java, Scala, C++ or a similar language.
- Experience developing large scale distributed systems from scratch.
- Hands on experience developing and operating backend systems.
- The ability to contribute across all project phases, from design and development through implementation and ongoing operations, with guidance from senior team members.
Nice to have:
- Experience in database replication, backup or transaction recovery at a major database vendor such as Microsoft SQL Server, Oracle or IBM. The posting lists this as a plus, and it is the clearest signal of what the team actually values.
What the work actually looks like:
- Deliver a highly scalable, available and fault tolerant engine processing hundreds of terabytes of data daily across thousands of customers.
- Low level systems debugging, performance measurement and optimisation on large production clusters.
- Build architecture designs, influence the product roadmap and own new projects.
- Help prevent and investigate production issues.
- Plan and lead complex technical projects that span several teams.
- Break complex problems into solutions, knowns and unknowns, and de risk them through prototyping and validation.
Why this team exists: ingesting data into the Lakehouse is described as a strategic investment area, and every Databricks surface including dashboards, notebooks, SQL and AI needs ingestion, so the team works closely with other product teams to embed Connect into those surfaces.
Location and working pattern: Bengaluru. Databricks says it is building out its Bengaluru site with new teams, so expect an in office or hybrid arrangement. The number of days is not stated.
Compliance note in the posting: where a role requires access to export controlled technology or source code, Databricks says applying for a US government licence is at its discretion and it may decline to proceed on that basis. That clause is in the posting and is worth reading before you invest time.
Honest fit guidance: five years plus real distributed systems work is the bar. Database internals experience is what will separate applications here.
Apply on company site
Opens databricks.com, the employer's own application page. Applying is always free.