Scala for Backend Engineering & Functional Programming · 课时

转换与操作

惰性计算

转换与操作 是 CoddyKit 上的免费 Scala for Backend Engineering & Functional Programming 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Scala for Backend Engineering & Functional Programming 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Scala for Backend Engineering & Functional Programming 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Two Kinds of Operations

Spark operations split into transformations (build a new dataset, lazy) and actions (trigger computation and return a result or write output).

Lazy Evaluation

Transformations are lazy: they record what to do but run nothing. Spark builds a directed acyclic graph (DAG) of transformations and only executes when an action is called.

map and filter

map transforms each element; filter keeps elements matching a predicate. Both return new RDDs/Datasets and do not run yet.

val nums  = sc.parallelize(1 to 10)
val evens = nums.filter(_ % 2 == 0).map(_ * 10)
// nothing computed yet

flatMap

flatMap maps each element to zero or more outputs and flattens them — classic for splitting lines into words.

val lines = sc.parallelize(Seq("hello world", "spark rocks"))
val words = lines.flatMap(_.split(" "))

Common Actions

Actions trigger execution:

collect — bring all results to the driver
count — number of elements
first / take(n) — sample rows
reduce — fold into one value

val total = sc.parallelize(1 to 100).reduce(_ + _)
println(total) // 5050 — runs now

Narrow vs Wide

Narrow transformations (map, filter) need no data movement. Wide transformations (groupByKey, reduceByKey, join) trigger a shuffle across the network.

reduceByKey

On key-value RDDs, reduceByKey aggregates values per key. It combines locally before shuffling, making it more efficient than groupByKey.

val pairs  = sc.parallelize(Seq(("a", 1), ("b", 1), ("a", 1)))
val counts = pairs.reduceByKey(_ + _)
// (a, 2), (b, 1)

Caching

If a dataset is reused across multiple actions, cache or persist keeps it in memory so it is not recomputed each time.

val cached = sc.parallelize(1 to 1000).filter(_ % 3 == 0).cache()
println(cached.count())
println(cached.sum())

DataFrame Transformations

DataFrames have their own lazy transformations: select, where, withColumn, orderBy. Actions like show and collect trigger them.

import org.apache.spark.sql.functions._

val adults = df.where(col("age") >= 18)
               .withColumn("adult", lit(true))
adults.show()

The Word Count Classic

The canonical Spark job: split lines, map each word to one, reduce by key. Only the final collect runs the pipeline.

val text   = sc.textFile("book.txt")
val counts = text.flatMap(_.split(" "))
                 .map(w => (w, 1))
                 .reduceByKey(_ + _)
counts.collect().foreach(println)

Plain Scala Lazy Analog

A self-contained Scala analog: a lazy view defers work until forced, mirroring Spark's transformation/action split.

object Main {
  def main(args: Array[String]): Unit = {
    val pipeline = (1 to 10).view.filter(_ % 2 == 0).map(_ * 10) // lazy
    val result = pipeline.toList // forces evaluation (action)
    println(result)
  }
}

Quick Check

When does Spark actually execute a chain of map and filter transformations?

Recap

You learned Spark's execution model:

Transformations are lazy (map, filter, flatMap, reduceByKey)
Actions trigger work (collect, count, reduce)
narrow vs wide (shuffle) operations
cache to reuse results

Next: Spark SQL.

常见问题解答

「转换与操作」课时是免费的吗？

是的 — 「转换与操作」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Scala for Backend Engineering & Functional Programming 课程的其余内容，请升级到 CoddyKit PRO。 Scala for Backend Engineering & Functional Programming 课程共包含 4 节课。

「转换与操作」这节课中我会学到什么？

惰性计算你通过在浏览器中直接运行的动手代码来练习 Scala for Backend Engineering & Functional Programming，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Scala for Backend Engineering & Functional Programming 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Scala for Backend Engineering & Functional Programming 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「转换与操作」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Scala for Backend Engineering & Functional Programming 课中编写并运行代码吗？

能。每节 Scala for Backend Engineering & Functional Programming 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Scala for Backend Engineering & Functional Programming