SQL Pipe gives headaches but comes with benefits

Access for free via a friend’s link

Databricks Runtime 16.2 introduced a new SQL pipeline syntax that reverse the traditional SQL query structure. This feature lets chain query operations using a pipe operator (|>), constructing queries step by step. While powerful, it initially feels awkward for those who have written SQL the same way for years. In this article, let me explore why the new syntax can be a headache for experienced SQL users, why it is still worth using (especially for dynamic queries), and how to leverage it in an object-oriented way with Python.

The “Headache” Factor

SQL users may be surprised when they encounter the pipeline syntax; instead of the familiar order, SELECT FROM WHERE a pipeline query starts with the FROM clause and chains the other parts after it. In other words, you write queries in the execution order (FROM, then WHERE, then SELECT, etc.), not in the traditional textual order. For example, a standard SQL query:

SELECT
  *
FROM
  users
WHERE age > 30;

It would be written in pipe form as:

FROM users
|> WHERE age > 30;

For anyone used to writing the SELECT clause first, this feels unnatural. One Reddit user humorously dubbed it the “weird inverted Yoda form” of SQL…