databricks-migration

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: databricks-migration (az rest: --headers "x-ms-fabric-skill=databricks-migration"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES

1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
3. dbutils.widgets has no direct equivalent in Fabric — use notebook parameters (cell tag parameters); notebookutils.runtime.context is execution metadata, not parameter storage. If showing context fields, use documented names such as currentWorkspaceId, currentWorkspaceName, currentNotebookId, currentNotebookName, isForPipeline, and isForInteractive; activityId is the Livy job ID
4. dbutils.library (runtime library install) has no equivalent — use Fabric Environments for reproducible library management
5. Map each Unity Catalog catalog to a schema-enabled Lakehouse by default. This preserves the source schema.table hierarchy, with the Lakehouse representing the catalog; collisions arise only if multiple catalogs are intentionally consolidated into one Lakehouse
6. For an under-specified workspace-wide migration, ask focused questions about inventory, workload topology, security, data locations, and runtime constraints before recommending a Fabric topology
7. A completed Fabric migration must not retain executable dbutils.* calls in dual-runtime branches or try/except guards — replace the calls and Databricks paths outright

Databricks → Microsoft Fabric Migration

Prerequisite Knowledge

Read these companion documents before executing migration tasks:

• COMMON-CORE.md — Fabric REST API patterns, authentication, token audiences, item discovery
• COMMON-CLI.md — az rest, az login, token acquisition, Fabric REST via CLI
• SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, Spark job execution

For notebook and Lakehouse creation, see spark-cli. For Fabric Warehouse DDL/DML authoring, see sqldw-cli.

---

Table of Contents

Context Loading Guide

IMPORTANT — Load only what you need. Do NOT read all resource files upfront. Load the specific file for the phase you are executing:

---

Migration Workload Map

Severity Definitions

---

dbutils → notebookutils Quick Reference

The complete side-by-side API table is in dbutils-to-notebookutils.md. The key mappings are:

Widgets Migration

dbutils.widgets has no direct equivalent in Fabric. Use these patterns instead:

Note: notebookutils.runtime.context does not expose parameter values. It's for execution metadata (workspace/notebook/activity/user IDs, pipeline-vs-interactive flags, etc.). See dbutils-to-notebookutils.md § Runtime Context.

---

Cluster Config → Fabric Spark Pools

---

Databricks Jobs → Spark Job Definitions

Delegate to spark-cli for SJD creation and notebook deployment.

---

Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts

When producing a migration workload map, include both paths: direct OneLake Shortcuts for accessible ADLS or same-tenant OneLake data, and Fabric external data sharing for native cross-tenant recipient sharing.

---

MLflow → Fabric ML Experiments

Fabric ML Experiments are built on the MLflow SDK — most code is directly portable:

---

Must / Prefer / Avoid

MUST DO

• Inventory before prescribing a workspace topology — for workspace-wide requests that omit workload inventory, dependencies, security requirements, data locations, or runtime constraints, ask focused clarifying questions and present conditional choices before selecting a Fabric design
• Replace all dbutils.* calls using the mapping in dbutils-to-notebookutils.md — dbutils is not available in Fabric notebooks
• Migrate dbutils.fs.mount() to notebookutils.fs.mount() (✅ supported — Microsoft Entra default, or accountKey / sasToken from Key Vault). For cross-workspace or persistent sharing, prefer OneLake Shortcuts instead. Always pair mount() with unmount() in try/finally — Fabric mounts are not released automatically on session end
• Replace dbutils.secrets.get(scope, key) with notebookutils.credentials.getSecret(keyVaultUrl, secretName) — secret scopes map to Azure Key Vault URLs
• Redesign widget-based parameter passing using notebook parameters cells (cell "..." menu → "Mark cell as parameters"); use notebookutils.variableLibrary for centralized cross-notebook config. notebookutils.runtime.context does not expose parameter values
• Replace dbutils.library.install*() with Fabric Environments — runtime library installs are not supported in production. dbutils.library.restartPython() maps to notebookutils.session.restartPython() (Python / PySpark only)
• Map Unity Catalog namespaces deliberately — default to one schema-enabled Lakehouse per catalog so schema.table is preserved; require a user-approved naming policy only when consolidating multiple catalogs into one Lakehouse. See catalog-migration.md
• Map Databricks cluster init scripts to Fabric Environments — cluster-level library installs must move to Environment items

PREFER

• Fabric Native Execution Engine (NEE) as the Photon equivalent — enable in workspace Spark settings for vectorized execution on Delta Lake
• OneLake Shortcuts over data copy for Delta tables that already exist in ADLS Gen2 — point directly without re-ingesting
• Fabric Git Integration as the replacement for Databricks Repos — connect workspace to ADO or GitHub for notebook version control
• Fabric ML Experiments for direct MLflow continuity — tracking code requires minimal changes (remove set_tracking_uri)
• Medallion architecture when restructuring migrated Databricks catalogs — align bronze, silver, gold Unity Catalog schemas to separate Fabric Lakehouses
• Starter Pool for migrating interactive notebook workflows — eliminates cluster startup time that was a common pain point in Databricks job clusters

AVOID

• Do not prescribe a one-size-fits-all workspace topology when the source inventory and migration constraints are missing
• Do not import dbutils or attempt dbutils = ... assignments in Fabric notebooks — import attempts fail with ModuleNotFoundError, while unresolved dbutils references raise NameError; always use notebookutils
• Do not retain dbutils.* calls behind runtime-detection guards (try/except, if IS_DATABRICKS) — replace the calls and Databricks paths outright with notebookutils and Fabric paths
• Do not assume Unity Catalog governance policies transfer automatically — RBAC, row-level security, and column masking must be reconfigured in Fabric using workspace roles and Lakehouse permissions
• Do not use %pip install in production Fabric notebooks at runtime — use Fabric Environments for stable, versioned library management
• Do not attempt to port Delta Live Tables (DLT) pipelines verbatim — DLT has no Fabric equivalent; rewrite as parameterized notebooks orchestrated by Fabric Pipelines
• Do not rely on Databricks-specific Spark configurations (e.g., spark.databricks.*) — these are proprietary and will be silently ignored or raise errors in Fabric
• Do not use DBFS paths (dbfs:/...) — there is no DBFS in Fabric; all paths must use OneLake abfss:// or Lakehouse-relative paths

---

Multi-Notebook Migration Protocol

For workspaces with >3 notebooks or individual notebooks >5KB, process one notebook at a time (enumerate → export → transform → summarize → deploy → release) to avoid context overflow. Track each notebook through a status lifecycle (inventory → analyzed → converted → deployed → validated, or failed).

Full protocol, status definitions, and per-notebook summary schema: migration-orchestrator.md § Phase 2.

---

Failure Reporting

When migration cannot complete (permission failures, unresolvable Blockers such as DLT or OS-level init scripts, namespace collisions with no user-chosen policy, or repeated API failures), emit a structured failure report — do not abandon silently. The report captures phase_reached, blockers[] (item / pattern / reason / recommendation), partial_success counts, and next_steps.

Full report schema and stopping conditions: migration-orchestrator.md § Failure Reporting.

---

Examples

See dbutils-to-notebookutils.md and code-patterns.md for the full mapping. Key quick references:

dbutils.fs → notebookutils.fs

# Databricks
dbutils.fs.ls("/mnt/bronze/orders/")
dbutils.fs.cp("/mnt/raw/file.csv", "/mnt/archive/file.csv")

# Fabric (replace DBFS/mount paths with OneLake relative paths)
notebookutils.fs.ls("Files/bronze/orders/")
notebookutils.fs.cp("Files/raw/file.csv", "Files/archive/file.csv")

dbutils.secrets → notebookutils.credentials

# Databricks
pwd = dbutils.secrets.get(scope="prod", key="db-password")

# Fabric (scope → Key Vault URL, key → secret name)
pwd = notebookutils.credentials.getSecret("https://myvault.vault.azure.net/", "db-password")

Unity Catalog namespace → Lakehouse schema

# Databricks
df = spark.read.table("prod.silver.customers")

# Fabric (ProdLakehouse represents the source catalog and is attached as context)
df = spark.read.table("silver.customers")