Big news. Firefly now supports Databricks.

Databricks has become a core platform for data, analytics, and AI workloads. But as their footprint grows, platform teams face a familiar challenge: how do you maintain visibility, governance, and recovery readiness when another critical part of your infrastructure is managed through a separate system?

Databricks is now integrated into Firefly as a SaaS provider. That means once you connect your workspace, Firefly discovers its resources, codifies them into production-ready Infrastructure-as-Code on demand, and brings them under the same policies and backup coverage you already apply to the rest of your cloud.

  • Step 1: Discovery gives platform teams visibility into a workspace that, until now, only the data team could see: every job, catalog, grant, and endpoint, in the same inventory as the rest of the infrastructure. 
  • Step 2: Once those resources are visible, Firefly can generate the Infrastructure-as-Code to manage them. 

Most Databricks environments were assembled by hand, and after an outage or a cyberattack, there is nothing to rebuild a hand-built environment from except memory. In code, the same workspace can be redeployed on demand, allowing you to recover your infrastructure in minutes.

Why Databricks, and Why Now?

A Databricks workspace is no longer where analysts run experiments on the side. Production applications depend on the pipelines that feed them, the catalogs that hold their data, and the endpoints that serve their models.

Those resources are usually created by data teams working in their own console, one job, one catalog, one grant at a time, and almost none of it is captured in code. 

Responsibility for governing the environment and for restoring service after an outage, stays with the platform team. It is a familiar situation: a critical part of the infrastructure sitting outside the operating model applied to everything else, with no source of truth to rebuild it from.

Firefly closes that gap by treating Databricks the way it treats every other part of your estate. Here’s what that looks like.

What Firefly Offers Databricks Users

1. One View Across Your Infrastructure

A production application rarely runs on one platform. It might rely on AWS or Azure resources, Kubernetes, SaaS services, and Databricks at the same time, and until now the Databricks part was the piece nobody outside the data team could see. Firefly brings supported Databricks resources into the same inventory as the rest of the estate, discovering:

  • What runs: jobs, Delta Live Tables pipelines, SQL endpoints, model serving endpoints, cluster policies, and instance pools
  • What governs the data: Unity Catalog catalogs, schemas, and grants, along with storage credentials, external locations, volumes, and secret scopes
  • Who can reach it: users, groups, service principals, and the permissions attached to them

Firefly maintains that inventory as a system of record, with the dependencies and relationships between resources mapped, so a Databricks pipeline can be seen next to the compute, storage, and identity it depends on and tied to the application it serves. That context matters most under pressure. 

In a change review, you can see what else a service touches before you approve. In an incident, you can see the blast radius of a failure or a compromised credential in one place, instead of piecing it together from three consoles while the clock runs.

The same context settles ownership. Data teams and platform teams work on the same infrastructure from opposite ends, and the questions that fall between them, such as who owns a pipeline, which environment a catalog belongs to, whether an endpoint is still in use, tend to be resolved in Slack threads. With Databricks in the inventory, those answers live where everyone already looks.

2. A Path From ClickOps to Code

Databricks has a mature Terraform provider, but most workspace resources are still created directly in the console rather than defined in code. Jobs are scheduled from the UI, catalogs are created for a single analysis, grants are added to unblock a colleague. Writing IaC manually for all of that after the fact means reverse-engineering configurations and mapping relationships one resource at a time, which does not scale and rarely gets finished.

So, what’s the fix?

With Firefly, supported Databricks resources can be codified into Terraform directly from their discovered configuration, with relationships captured and sensitive values left out. Import blocks bring them under code management without recreating them, and the output can be reviewed, refined with Thinkerbell, Firefly's AI agent, and pushed to your repository as a pull request.

Initial codification support includes selected Databricks resource types, such as catalogs and external locations, with support expanding over time.

This is where visibility turns into resilience. Codifying Databricks gives the platform team a versioned, reviewable definition of the workspace, and that definition is what recovery depends on.

3. Configuration Backup for Databricks

Visibility tells you what exists. Backup determines what you have to work from when it is gone.

You choose the scope of infrastructure you want backed up, and Databricks can now be part of that scope. Firefly captures the configuration of that scope on a schedule, giving teams a point-in-time record to restore from. A recovery point objective set on the scope controls how often snapshots are taken, so a tighter objective means more frequent captures and less configuration lost between them.

Because you draw the scope, the snapshot follows the boundary of the service rather than the boundary of a provider. Cloud infrastructure and the Databricks resources behind the same service are captured together, and the record you restore from reflects how that service was built.

4. A Foundation for Better Governance

Every resource that lives outside a policy is a resource nobody is checking, and misconfigurations in those places do not fix themselves. They accumulate in the background until they surface as an outage, a failed audit, or the opening an attacker was looking for.

Firefly enforces policies continuously across the cloud estate, covering SOC 2, HIPAA, ISO 27001, NIST, DORA, and more. Violations surface as soon as they appear, and every violation and remediation is logged automatically, so compliance reporting and audit evidence are always current rather than assembled before each review. Teams can also define their own guardrails, in plain language or as policy-as-code, when a rule is specific to their environment.

Bringing Databricks into that estate is what lets the same standard apply to it. The goal is not simply to add another provider to an inventory list. It is to reduce the fragmentation platform teams deal with as their infrastructure expands across clouds, platforms, and services.

How to Get Started with Firefly for Databricks

Connecting your workspace takes a service principal and a workspace URL, and the entitlements you grant decide how much of it Firefly can see.

  • New to Firefly? Start your 14-day free trial and get full platform access from day one, including Databricks, or Book a 30-minute demo to walk through Databricks discovery, application context, and configuration backup across your full estate.
  • Already using Firefly? Add your workspace from Settings > Integrations. 

See the integration documentation for setup steps and the full list of supported resource types.

‍