Databricks SQL backend

[KenRoytman]

Databricks SQL looks to be a relatively new data warehouse implementation. I'm curious if the core Ibis team's road map includes adding support for this backend? For clarity, Databricks SQL is a different backend than the Databricks PySpark implementation. Thanks!

[cpcloud]

Hi @KenRoytman 👋🏻!

Thanks for opening the discussion.

Would it make sense to consolidate the PySpark and Databricks SQL backends into one? The main technical question there is whether Databricks SQL can be executed by PySpark.

[cpcloud]

@KenRoytman Are you able to use ibis's pyspark backend on Databricks cloud, and does that get you what you want?

[cpcloud]

I see you mentioned they're different. Is there something about Databricks SQL you'd like to see/use that cannot be accomplished with PySpark?

[wjones127]

It looks like communication would involve sending SQL strings to this REST API: https://docs.databricks.com/sql/api/statements.html#tag/Statement-Execution

Is there an existing backend that can produce Spark SQL? If so then this would likely be easy to add. Otherwise That might be a bit of work.