Compare Fabric Data Engineering and Azure Synapse Spark

This article compares Azure Synapse Spark and Fabric Spark across Spark pools, configurations, libraries, notebooks, and Spark job definitions (SJD).

When to choose: Use Fabric Spark for unified analytics with OneLake storage, built-in CI/CD pipelines, and capacity-based scaling. Use Azure Synapse Spark when you need GPU-accelerated pools, external Hive Metastore, or JDBC connections.

Key limitations in Fabric

• JDBC: Not supported
• DMTS in notebooks: Data Movement and Transformation Services can't be used in notebooks or Spark job definitions
• Managed identity for Key Vault: Not supported in notebooks
• External Hive Metastore: Not supported
• GPU-accelerated pools: Not available
• .NET for Spark (C#): Not supported

More Fabric considerations

• Workload level RBAC: Fabric supports four workspace roles. For more information, see Roles in workspaces.
• CI/CD: Use the Fabric API/SDK and deployment pipelines.

Spark pool comparison

The following table compares Azure Synapse Spark and Fabric Spark pools.

When to choose: Use Fabric Spark pools for fast startup (starter pools), single-node jobs, high concurrency sessions, and V-Order optimization. Use Azure Synapse pools when you need GPU acceleration or fixed scaling up to 200 nodes.

Spark runtime versions

Fabric Spark supported versions:

• Spark 3.3 / Delta 2.2: Runtime 1.1
• Spark 3.4 / Delta 2.4: Runtime 1.2
• Spark 3.5 / Delta 3.1: Runtime 1.3

Fabric doesn't support Spark 2.4, 3.1, or 3.2.

Adjustable node sizes

Azure Synapse Spark pools scale up to 200 nodes regardless of node size. In Fabric, the maximum number of nodes depends on node size and provisioned capacity (SKU).

Fabric capacity conversion: 2 Spark vCores = 1 capacity unit. Formula: total vCores in capacity ÷ vCores per node size = maximum nodes available.

For example, SKU F64 provides 64 capacity units (128 Spark vCores). The following table shows node limits for F64:

For more information, see Spark compute.

Node sizes

Fabric Spark pools support only the Memory Optimized node family. GPU-accelerated pools available in Azure Synapse aren't supported in Fabric.

Node size comparison (XX-Large):

• Azure Synapse: 432 GB memory
• Fabric: 512 GB memory, 64 vCores

Node sizes Small through X-Large have identical vCores and memory in both Azure Synapse and Fabric.

Autopause behavior

Autopause settings comparison:

• Azure Synapse: Configurable idle timeout, minimum 5 minutes
• Fabric: Fixed 2-minute autopause after session expires (not configurable), default session timeout is 20 minutes

High concurrency

Fabric supports high concurrency mode for notebooks, allowing multiple users to share a single Spark session. Azure Synapse doesn't support this feature.

Concurrency limits

Azure Synapse Spark limits (fixed):

• 50 concurrent jobs per pool, 200 queued jobs per pool
• 250 active jobs per pool, 1,000 per workspace

Fabric Spark limits (SKU-based):

• Concurrent jobs vary by capacity SKU: 1 to 512 max
• Dynamic reserve-based throttling manages peak usage

For more information, see Concurrency limits and queueing in Microsoft Fabric Spark.

Multiple Spark pools

In Fabric, use environments to configure and select different Spark pools per notebook or Spark job definition.

Spark configurations comparison

Spark configurations apply at two levels:

• Environment level: Default configuration for all Spark jobs in the environment
• Inline level: Per-session configuration in notebooks or Spark job definitions

When to choose: Both platforms support environment and inline configurations. Fabric uses environments instead of pool-level configs.

• Inline syntax: In Fabric, use spark.conf.set(<conf_name>, <conf_value>) for session-level configs. For batch jobs, use SparkConf.
• Immutable configs: Some Spark configurations can't be modified. Error message: AnalysisException: Can't modify the value of a Spark config: <config_name>
• V-Order: Enabled by default in Fabric; write-time optimization for parquet files. See V-Order.
• Optimized Write: Enabled by default in Fabric; disabled by default in Azure Synapse.

Spark libraries comparison

Spark libraries apply at three levels:

• Workspace level: Available in Azure Synapse only
• Environment level: Libraries available to all notebooks and Spark job definitions in the environment
• Inline: Session-specific libraries installed at notebook startup

When to choose: Both platforms support environment and inline libraries. Fabric doesn't support workspace-level packages.

• Built-in libraries: Fabric and Azure Synapse runtimes share a common Spark core but differ in library versions. Some code might require recompilation or custom libraries. See Fabric runtime libraries.

Notebook comparison

Notebooks and Spark job definitions are primary code items for developing Apache Spark jobs in Fabric. There are some differences between Azure Synapse Spark notebooks and Fabric Spark notebooks:

When to choose: Use Fabric notebooks for collaboration, high concurrency sessions, built-in scheduling, and notebook resources. Use Azure Synapse notebooks if you require .NET for Spark (C#).

• notebookutils.credentials: Only getToken and getSecret are supported in Fabric (DMTS connections not available).
• Notebook resources: Fabric provides a Unix-like file system for managing files. See How to use notebooks.
• High concurrency: Alternative to ThreadPoolExecutor in Azure Synapse. See Configure high concurrency mode.
• .NET for Spark: Migrate C#/F# workloads to Python or Scala.
• Linked services: Replace with Spark libraries for external data source connections.

Spark job definition comparison

Important Spark job definition considerations:

When to choose: Use Fabric Spark job definitions for SparkR support, built-in scheduling, and retry policies. Use Azure Synapse if you need .NET for Spark (C#) or UI-based import/export.

• Supported files: .py, .R, and .jar files with reference files, command line arguments, and lakehouse references.
• Import/export: UI-based JSON import/export available in Azure Synapse only.
• Retry policies: Enable indefinite runs for Spark Structured Streaming jobs.
• .NET for Spark: Migrate C#/F# workloads to Python or Scala.

Hive Metastore (HMS) comparison

When to choose: Use Fabric if lakehouse-based internal HMS meets your needs. Use Azure Synapse if you require external Hive Metastore (Azure SQL DB) or Catalog API access.

Related content

• Learn more about migration options for Spark pools, configurations, libraries, notebooks, and Spark job definitions
• Migrate data and pipelines
• Migrate Hive Metastore metadata