0Pricing
Scala for Backend Engineering & Functional Programming · 课时

Spark SQL

查询数据

Spark SQL 是 CoddyKit 上的免费 Scala for Backend Engineering & Functional Programming 课时。 这是第 3 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Scala for Backend Engineering & Functional Programming 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Scala for Backend Engineering & Functional Programming 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

What Is Spark SQL?

Spark SQL lets you query distributed data with standard SQL or a typed DataFrame API. Both go through the same Catalyst optimizer, so they perform identically.

Temporary Views

To run SQL against a DataFrame, register it as a view. createOrReplaceTempView makes it queryable by name for the current session.

val df = Seq(("Alice", 30), ("Bob", 25)).toDF("name", "age")
df.createOrReplaceTempView("people")

Running a SQL Query

Use spark.sql with a SQL string. It returns a new DataFrame you can further transform or display.

val adults = spark.sql("SELECT name FROM people WHERE age >= 18")
adults.show()

The DataFrame DSL

The same query in the typed DSL. The functions object provides col, comparisons, and many built-in expressions.

import org.apache.spark.sql.functions._

val adults = df.filter(col("age") >= 18).select("name")

Selecting and Aliasing

Project columns with select and rename with as / alias. Computed columns use expressions over col.

import org.apache.spark.sql.functions._

df.select(
  col("name"),
  (col("age") + 1).as("age_next_year")
).show()

Filtering Rows

where and filter are synonyms. Combine conditions with && and ||, and use isNull / isNotNull for missing data.

import org.apache.spark.sql.functions._

df.where(col("age") > 20 && col("name").isNotNull).show()

Sorting and Limiting

orderBy sorts (use desc for descending), and limit caps the row count returned.

import org.apache.spark.sql.functions._

df.orderBy(col("age").desc).limit(5).show()

Joins

Join two DataFrames on a key with join. Specify the join type such as "inner", "left", or "outer".

val joined = orders.join(customers, Seq("customer_id"), "inner")
joined.show()

Built-in Functions

The functions package offers hundreds of helpers: upper, concat, when, round, date functions, and more.

import org.apache.spark.sql.functions._

df.withColumn("name_upper", upper(col("name")))
  .withColumn("category", when(col("age") >= 30, "senior").otherwise("junior"))
  .show()

Reading and Writing Tables

Spark SQL reads and writes many formats. Parquet is columnar and efficient; saveAsTable persists to the metastore.

val data = spark.read.parquet("input.parquet")
data.write.mode("overwrite").parquet("output.parquet")

Plain Scala SQL-like Query

A self-contained analog: querying an in-memory collection with collection methods mirrors a Spark SQL SELECT/WHERE.

object Main {
  case class Person(name: String, age: Int)
  def main(args: Array[String]): Unit = {
    val people = Seq(Person("Alice", 30), Person("Bob", 25))
    val adults = people.filter(_.age >= 18).map(_.name)
    println(adults.mkString(", "))
  }
}

Quick Check

What must you do to a DataFrame before querying it with spark.sql("SELECT ...")?

Recap

You queried data with Spark SQL:

  • register views with createOrReplaceTempView
  • run SQL via spark.sql or the DataFrame DSL
  • select, where, orderBy, join
  • built-in functions and Parquet I/O

Next: aggregations.

常见问题解答

「Spark SQL」课时是免费的吗？

是的 — 「Spark SQL」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Scala for Backend Engineering & Functional Programming 课程的其余内容，请升级到 CoddyKit PRO。 Scala for Backend Engineering & Functional Programming 课程共包含 4 节课。

「Spark SQL」这节课中我会学到什么？

查询数据 你通过在浏览器中直接运行的动手代码来练习 Scala for Backend Engineering & Functional Programming，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Scala for Backend Engineering & Functional Programming 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Scala for Backend Engineering & Functional Programming 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 3 节课，共 4 节。

「Spark SQL」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Scala for Backend Engineering & Functional Programming 课中编写并运行代码吗？

能。每节 Scala for Backend Engineering & Functional Programming 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. RDD 与 DataFrames
  2. 转换与操作
  3. Spark SQL
  4. 聚合
← 返回 Scala for Backend Engineering & Functional Programming