Writing transform logic in PySpark, SQL, or KQL

In Microsoft Fabric, you can transform data using PySpark and SQL in Notebooks against Lakehouse tables/files, and using KQL in KQL Queryset (or update policies) against Eventhouse/KQL databases. Choosing the right language and engine depends on the data store (Lakehouse vs. Eventhouse), the data format, and whether you need distributed batch processing, declarative SQL transforms, or near-real-time analytical queries.

1 · Learn the must-know

• 1Fabric Notebooks support PySpark, Spark SQL, Scala, and SparkR (via %%language magic cells), letting you mix PySpark DataFrame operations with SQL queries in the same notebook against Lakehouse tables and files.
• 2PySpark DataFrame transformations (select, filter, withColumn, groupBy, join, etc.) execute lazily and are optimized/executed by the Spark engine only when an action (e.g., show, write, count) is triggered.
• 3You can register a DataFrame as a temporary view (createOrReplaceTempView) to run Spark SQL statements against it, or query Lakehouse Delta tables directly with SQL using spark.sql().
• 4T-SQL against the SQL analytics endpoint of a Lakehouse or a Warehouse is read/query-focused for the Lakehouse endpoint (no DML), whereas the Warehouse supports full T-SQL DML (INSERT/UPDATE/DELETE/MERGE) for transformations.
• 5KQL (Kusto Query Language) is used to query and transform data in Eventhouse/KQL databases via the KQL Queryset, using operators like extend, project, summarize, and mv-expand for filtering, shaping, and aggregating streaming or log-style data.
• 6Update policies in a KQL database let you automatically transform and route incoming data from a source table into one or more target tables using a KQL function, enabling lightweight ETL within the Eventhouse without moving data to Spark.

2 · Check your understanding