From the course: Complete Guide to Databricks for Data Engineering

Unlock this course with a free trial

Join today to access over 24,600 courses taught by industry experts.

Use Spark SQL transformations

Use Spark SQL transformations - Databricks Tutorial

From the course: Complete Guide to Databricks for Data Engineering

Start my 1-month free trial Buy for my team

Use Spark SQL transformations

“

- [Instructor] Spark SQL transformations provides a huge functionality for the data engineers to play around, to clean the data, to transform the data, to process the data, and to aggregate the data. Now, let's just understand how we can use Spark SQL transformations. So the first thing probably we need is the views, so that we have already created in our previous videos. So we're going to use the same temporary views customer, which we have created in the temporary view video. So let's move forward with this customer view. If I were to select a specific column out of that, I can say df1 is equal to spark.sql select customer_id, name, email, customer_type from this customer view. Remember whenever you run spark.sql and execute any query, the output which you get isn't another DataFrame. So if you want to see that DataFrame, we're going to use Databricks display function to print that DataFrame. And you can see that. We got some specific columns out of that view. So here, the benefit…

Contents