0Pricing
Scala for Backend Engineering & Functional Programming · Lesson

Transformations and Actions

Lazy computation.

Transformations and Actions is a free Scala for Backend Engineering & Functional Programming lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Scala for Backend Engineering & Functional Programming learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Two Kinds of Operations

Spark operations split into transformations (build a new dataset, lazy) and actions (trigger computation and return a result or write output).

Lazy Evaluation

Transformations are lazy: they record what to do but run nothing. Spark builds a directed acyclic graph (DAG) of transformations and only executes when an action is called.

map and filter

map transforms each element; filter keeps elements matching a predicate. Both return new RDDs/Datasets and do not run yet.

val nums  = sc.parallelize(1 to 10)
val evens = nums.filter(_ % 2 == 0).map(_ * 10)
// nothing computed yet

flatMap

flatMap maps each element to zero or more outputs and flattens them — classic for splitting lines into words.

val lines = sc.parallelize(Seq("hello world", "spark rocks"))
val words = lines.flatMap(_.split(" "))

Common Actions

Actions trigger execution:

  • collect — bring all results to the driver
  • count — number of elements
  • first / take(n) — sample rows
  • reduce — fold into one value
val total = sc.parallelize(1 to 100).reduce(_ + _)
println(total) // 5050 — runs now

Narrow vs Wide

Narrow transformations (map, filter) need no data movement. Wide transformations (groupByKey, reduceByKey, join) trigger a shuffle across the network.

reduceByKey

On key-value RDDs, reduceByKey aggregates values per key. It combines locally before shuffling, making it more efficient than groupByKey.

val pairs  = sc.parallelize(Seq(("a", 1), ("b", 1), ("a", 1)))
val counts = pairs.reduceByKey(_ + _)
// (a, 2), (b, 1)

Caching

If a dataset is reused across multiple actions, cache or persist keeps it in memory so it is not recomputed each time.

val cached = sc.parallelize(1 to 1000).filter(_ % 3 == 0).cache()
println(cached.count())
println(cached.sum())

DataFrame Transformations

DataFrames have their own lazy transformations: select, where, withColumn, orderBy. Actions like show and collect trigger them.

import org.apache.spark.sql.functions._

val adults = df.where(col("age") >= 18)
               .withColumn("adult", lit(true))
adults.show()

The Word Count Classic

The canonical Spark job: split lines, map each word to one, reduce by key. Only the final collect runs the pipeline.

val text   = sc.textFile("book.txt")
val counts = text.flatMap(_.split(" "))
                 .map(w => (w, 1))
                 .reduceByKey(_ + _)
counts.collect().foreach(println)

Plain Scala Lazy Analog

A self-contained Scala analog: a lazy view defers work until forced, mirroring Spark's transformation/action split.

object Main {
  def main(args: Array[String]): Unit = {
    val pipeline = (1 to 10).view.filter(_ % 2 == 0).map(_ * 10) // lazy
    val result = pipeline.toList // forces evaluation (action)
    println(result)
  }
}

Quick Check

When does Spark actually execute a chain of map and filter transformations?

Recap

You learned Spark's execution model:

  • Transformations are lazy (map, filter, flatMap, reduceByKey)
  • Actions trigger work (collect, count, reduce)
  • narrow vs wide (shuffle) operations
  • cache to reuse results

Next: Spark SQL.

Frequently asked questions

Is the “Transformations and Actions” lesson free?

Yes — the full text of “Transformations and Actions” is free to read here on the web, and the Scala for Backend Engineering & Functional Programming course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Scala for Backend Engineering & Functional Programming course, upgrade to CoddyKit PRO.

What will I learn in “Transformations and Actions”?

Lazy computation. You practise Scala for Backend Engineering & Functional Programming with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Scala for Backend Engineering & Functional Programming?

No prior experience is required. Scala for Backend Engineering & Functional Programming on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Transformations and Actions” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Scala for Backend Engineering & Functional Programming lesson?

Yes. Every Scala for Backend Engineering & Functional Programming lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. RDDs and DataFrames
  2. Transformations and Actions
  3. Spark SQL
  4. Aggregations
← Back to Scala for Backend Engineering & Functional Programming