0Pricing
Scala for Backend Engineering & Functional Programming · 课时

RDD 与 DataFrames

Spark 数据抽象

RDD 与 DataFrames 是 CoddyKit 上的免费 Scala for Backend Engineering & Functional Programming 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Scala for Backend Engineering & Functional Programming 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Scala for Backend Engineering & Functional Programming 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

What Is Apache Spark?

Apache Spark is a distributed engine for large-scale data processing. It splits data across a cluster and runs computations in parallel. Scala is Spark's native language, giving a concise, type-aware API.

The RDD

The Resilient Distributed Dataset (RDD) is Spark's low-level abstraction: an immutable, partitioned collection that can be processed in parallel and rebuilt from lineage if a node fails.

SparkContext and SparkSession

You enter Spark through a SparkSession. Its sparkContext creates RDDs; the session itself creates DataFrames and runs SQL.

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("Demo")
  .master("local[*]")
  .getOrCreate()

Creating an RDD

Build an RDD by parallelizing a local collection or by reading a file. Each partition is processed by a separate task.

val sc = spark.sparkContext
val numbers = sc.parallelize(Seq(1, 2, 3, 4, 5))
val lines   = sc.textFile("data.txt")

The DataFrame

A DataFrame is a distributed table with named, typed columns — like an RDD of rows plus a schema. The Catalyst optimizer can plan and optimize DataFrame queries.

Creating a DataFrame

Create DataFrames from collections (with toDF) or by reading structured files such as CSV, JSON, or Parquet.

import spark.implicits._

val df = Seq(("Alice", 30), ("Bob", 25)).toDF("name", "age")
val csv = spark.read.option("header", "true").csv("people.csv")

Inspecting a DataFrame

Use show to print rows, printSchema to view column types, and count for the number of rows.

df.printSchema()
df.show()
println(df.count())

Datasets and Type Safety

A Dataset is a typed DataFrame: Dataset[T] where T is a case class. It combines DataFrame optimization with compile-time type checks.

import spark.implicits._

case class Person(name: String, age: Int)
val ds = Seq(Person("Alice", 30)).toDS()

RDD vs DataFrame vs Dataset

Choose based on needs:

  • RDD — full control, no schema, no optimizer
  • DataFrame — schema + Catalyst optimization, untyped rows
  • Dataset — schema + optimization + type safety

Converting Between Them

Convert a DataFrame to an RDD with .rdd, or an RDD of case classes to a DataFrame with .toDF. Datasets convert to DataFrames with .toDF too.

val rdd = df.rdd          // DataFrame -> RDD[Row]
val back = ds.toDF()      // Dataset -> DataFrame

Plain Scala Collection

Conceptually a DataFrame behaves like a Scala collection but distributed. Here is the local, self-contained analog for intuition.

object Main {
  case class Person(name: String, age: Int)
  def main(args: Array[String]): Unit = {
    val people = Seq(Person("Alice", 30), Person("Bob", 25))
    people.foreach(p => println(s"${p.name}: ${p.age}"))
  }
}

Quick Check

Which abstraction provides a schema, Catalyst optimization, and compile-time type safety?

Recap

You met Spark's data abstractions:

  • SparkSession as the entry point
  • RDD — low-level distributed collection
  • DataFrame — distributed table with schema
  • Dataset — typed DataFrame

Next: transformations and actions.

常见问题解答

「RDD 与 DataFrames」课时是免费的吗？

是的 — 「RDD 与 DataFrames」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Scala for Backend Engineering & Functional Programming 课程的其余内容，请升级到 CoddyKit PRO。 Scala for Backend Engineering & Functional Programming 课程共包含 4 节课。

「RDD 与 DataFrames」这节课中我会学到什么？

Spark 数据抽象 你通过在浏览器中直接运行的动手代码来练习 Scala for Backend Engineering & Functional Programming，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Scala for Backend Engineering & Functional Programming 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Scala for Backend Engineering & Functional Programming 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。

「RDD 与 DataFrames」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Scala for Backend Engineering & Functional Programming 课中编写并运行代码吗？

能。每节 Scala for Backend Engineering & Functional Programming 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. RDD 与 DataFrames
  2. 转换与操作
  3. Spark SQL
  4. 聚合
← 返回 Scala for Backend Engineering & Functional Programming