Spark SQL
Query data.
Spark SQL is a free Scala for Backend Engineering & Functional Programming lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Scala for Backend Engineering & Functional Programming learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is Spark SQL?
Spark SQL lets you query distributed data with standard SQL or a typed DataFrame API. Both go through the same Catalyst optimizer, so they perform identically.
Temporary Views
To run SQL against a DataFrame, register it as a view. createOrReplaceTempView makes it queryable by name for the current session.
val df = Seq(("Alice", 30), ("Bob", 25)).toDF("name", "age")
df.createOrReplaceTempView("people")Running a SQL Query
Use spark.sql with a SQL string. It returns a new DataFrame you can further transform or display.
val adults = spark.sql("SELECT name FROM people WHERE age >= 18")
adults.show()The DataFrame DSL
The same query in the typed DSL. The functions object provides col, comparisons, and many built-in expressions.
import org.apache.spark.sql.functions._
val adults = df.filter(col("age") >= 18).select("name")Selecting and Aliasing
Project columns with select and rename with as / alias. Computed columns use expressions over col.
import org.apache.spark.sql.functions._
df.select(
col("name"),
(col("age") + 1).as("age_next_year")
).show()Filtering Rows
where and filter are synonyms. Combine conditions with && and ||, and use isNull / isNotNull for missing data.
import org.apache.spark.sql.functions._
df.where(col("age") > 20 && col("name").isNotNull).show()Sorting and Limiting
orderBy sorts (use desc for descending), and limit caps the row count returned.
import org.apache.spark.sql.functions._
df.orderBy(col("age").desc).limit(5).show()Joins
Join two DataFrames on a key with join. Specify the join type such as "inner", "left", or "outer".
val joined = orders.join(customers, Seq("customer_id"), "inner")
joined.show()Built-in Functions
The functions package offers hundreds of helpers: upper, concat, when, round, date functions, and more.
import org.apache.spark.sql.functions._
df.withColumn("name_upper", upper(col("name")))
.withColumn("category", when(col("age") >= 30, "senior").otherwise("junior"))
.show()Reading and Writing Tables
Spark SQL reads and writes many formats. Parquet is columnar and efficient; saveAsTable persists to the metastore.
val data = spark.read.parquet("input.parquet")
data.write.mode("overwrite").parquet("output.parquet")Plain Scala SQL-like Query
A self-contained analog: querying an in-memory collection with collection methods mirrors a Spark SQL SELECT/WHERE.
object Main {
case class Person(name: String, age: Int)
def main(args: Array[String]): Unit = {
val people = Seq(Person("Alice", 30), Person("Bob", 25))
val adults = people.filter(_.age >= 18).map(_.name)
println(adults.mkString(", "))
}
}Quick Check
What must you do to a DataFrame before querying it with spark.sql("SELECT ...")?
Recap
You queried data with Spark SQL:
- register views with
createOrReplaceTempView - run SQL via
spark.sqlor the DataFrame DSL select,where,orderBy,join- built-in functions and Parquet I/O
Next: aggregations.
Frequently asked questions
Is the “Spark SQL” lesson free?
Yes — the full text of “Spark SQL” is free to read here on the web, and the Scala for Backend Engineering & Functional Programming course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Scala for Backend Engineering & Functional Programming course, upgrade to CoddyKit PRO.
What will I learn in “Spark SQL”?
Query data. You practise Scala for Backend Engineering & Functional Programming with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Scala for Backend Engineering & Functional Programming?
No prior experience is required. Scala for Backend Engineering & Functional Programming on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Spark SQL” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Scala for Backend Engineering & Functional Programming lesson?
Yes. Every Scala for Backend Engineering & Functional Programming lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.