Aggregations
Group and aggregate.
Aggregations is a free Scala for Backend Engineering & Functional Programming lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Scala for Backend Engineering & Functional Programming learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Aggregate?
Aggregations summarize many rows into fewer: totals, averages, counts per group. In Spark they are wide operations that may shuffle data across the cluster.
Global Aggregates
Compute a single summary over the whole DataFrame with agg and functions like count, sum, avg, min, max.
import org.apache.spark.sql.functions._
df.agg(
count("*").as("rows"),
avg("age").as("avg_age")
).show()groupBy
groupBy partitions rows by one or more columns, producing a RelationalGroupedDataset ready for aggregation.
import org.apache.spark.sql.functions._
val byCity = df.groupBy("city").agg(count("*").as("people"))
byCity.show()Multiple Aggregations
Pass several aggregate expressions to agg to compute many summaries per group in one pass.
import org.apache.spark.sql.functions._
df.groupBy("department").agg(
sum("salary").as("total"),
avg("salary").as("avg"),
max("salary").as("top")
).show()Counting Distinct
Use countDistinct to count unique values, and approx_count_distinct for a faster approximate count on huge datasets.
import org.apache.spark.sql.functions._
df.agg(countDistinct("city").as("unique_cities")).show()Filtering Groups with having
In SQL, HAVING filters aggregated groups. In the DSL, apply a filter after agg on the computed column.
import org.apache.spark.sql.functions._
df.groupBy("city")
.agg(count("*").as("n"))
.filter(col("n") > 100)
.show()Aggregating in SQL
The same logic as a SQL query against a registered view, using GROUP BY and HAVING.
df.createOrReplaceTempView("people")
spark.sql(
"SELECT city, COUNT(*) AS n FROM people GROUP BY city HAVING COUNT(*) > 100"
).show()Pivot Tables
pivot turns distinct values of a column into separate columns — handy for cross-tabulations like sales per region per quarter.
import org.apache.spark.sql.functions._
sales.groupBy("region")
.pivot("quarter")
.agg(sum("amount"))
.show()Window Functions
Window functions aggregate over a sliding frame without collapsing rows — perfect for running totals or rankings.
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
val w = Window.partitionBy("dept").orderBy(col("salary").desc)
df.withColumn("rank", rank().over(w)).show()RDD Aggregations
At the RDD level, aggregateByKey and reduceByKey combine values per key with custom logic and local pre-combining for efficiency.
val pairs = sc.parallelize(Seq(("a", 10), ("a", 20), ("b", 5)))
val sums = pairs.reduceByKey(_ + _)
// (a, 30), (b, 5)Plain Scala groupBy
A self-contained analog: Scala's collection groupBy plus mapValues mirrors a Spark group-and-aggregate.
object Main {
def main(args: Array[String]): Unit = {
val data = Seq(("a", 10), ("a", 20), ("b", 5))
val sums = data.groupBy(_._1).map { case (k, v) => k -> v.map(_._2).sum }
sums.toSeq.sortBy(_._1).foreach { case (k, s) => println(s"$k: $s") }
}
}Quick Check
Which feature aggregates over a frame of rows without collapsing them into one row per group?
Recap
You aggregated data in Spark:
aggwithcount,sum,avg,min,maxgroupBy, multi-aggregations,havingfilterspivotand window functions- RDD
reduceByKey/aggregateByKey
You've completed the Apache Spark course.
Frequently asked questions
Is the “Aggregations” lesson free?
Yes — the full text of “Aggregations” is free to read here on the web, and the Scala for Backend Engineering & Functional Programming course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Scala for Backend Engineering & Functional Programming course, upgrade to CoddyKit PRO.
What will I learn in “Aggregations”?
Group and aggregate. You practise Scala for Backend Engineering & Functional Programming with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Scala for Backend Engineering & Functional Programming?
No prior experience is required. Scala for Backend Engineering & Functional Programming on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Aggregations” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Scala for Backend Engineering & Functional Programming lesson?
Yes. Every Scala for Backend Engineering & Functional Programming lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.