Resampling and Cross-Validation with rsample
Evaluate models with k-fold CV, bootstrap, and nested resampling.
Resampling and Cross-Validation with rsample is a free R Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the R Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Resample?
A single train/test split gives a noisy estimate of model performance — you got lucky or unlucky with which observations ended up in the test set. Resampling repeats the process multiple times to get a stable, reliable estimate of how your model generalises to new data.
library(rsample)
# Single split — performance estimate depends heavily
# on which 20% ended up as test data
split <- initial_split(mtcars, prop = 0.8)
train <- training(split)
test <- testing(split)
cat('Train:', nrow(train), '| Test:', nrow(test))initial_split()
initial_split(data, prop, strata) creates a single random split into training and test sets. Use strata to stratify by a column (e.g. the outcome variable) to ensure class balance is maintained in both partitions.
library(rsample)
# Stratified split by outcome variable
split <- initial_split(ames, prop = 0.8, strata = Sale_Price)
train <- training(split)
test <- testing(split)
cat('Train rows:', nrow(train))
cat('Test rows:', nrow(test))vfold_cv() — K-Fold Cross Validation
vfold_cv(data, v = 10) creates 10 folds. The data is split into 10 equal parts; 9 are used for training and 1 for validation, rotating through all folds. This gives 10 performance estimates that are averaged for a stable metric.
folds <- vfold_cv(housing_train, v = 10, strata = price)
# Each fold is a split object
print(folds)
# Inspect one fold
fold_1 <- folds$splits[[1]]
train_1 <- analysis(fold_1)
val_1 <- assessment(fold_1)
cat('Fold 1 — Train:', nrow(train_1), '| Val:', nrow(val_1))fit_resamples()
fit_resamples(workflow, resamples, metrics) fits your workflow on each training fold and evaluates it on the validation fold, collecting the requested metrics. It returns a tibble of results that you summarise with collect_metrics().
library(tune)
folds <- vfold_cv(housing_train, v = 10)
res <- fit_resamples(
wf, # your workflow
folds,
metrics = metric_set(rmse, rsq)
)
# Average metric across all 10 folds
collect_metrics(res)collect_metrics()
collect_metrics(resample_result) returns a tidy tibble summarising model performance across all folds. The mean column is the average metric and std_err is the standard error, giving you a sense of variance in the estimate.
metrics_df <- collect_metrics(res)
print(metrics_df)
# .metric .estimator mean n std_err .config
# rmse standard 24500 10 1200 Preprocessor1_Model1
# rsq standard 0.882 10 0.012 Preprocessor1_Model1
# Pull a single metric
collect_metrics(res) |>
dplyr::filter(.metric == 'rmse') |>
dplyr::pull(mean)bootstraps() — Bootstrap Resampling
bootstraps(data, times = 25) creates bootstrap samples: each sample is a random draw with replacement of the same size as the original dataset. Observations not drawn form the out-of-bag (OOB) assessment set. Bootstraps have higher variance than k-fold but work well with small datasets.
boot_samples <- bootstraps(housing_train, times = 25, strata = price)
print(boot_samples)
# Average proportion of unique rows in each bootstrap
mean(sapply(boot_samples$splits, function(s) {
nrow(analysis(s)) / nrow(housing_train)
}))Monte Carlo Cross Validation
mc_cv(data, prop, times) creates times random splits, each using prop of the data for training. Unlike k-fold, the same observation may appear in the validation set multiple times. This is useful when you need more resampling iterations than k-fold provides.
mc_splits <- mc_cv(housing_train, prop = 0.8, times = 20)
res_mc <- fit_resamples(
wf,
mc_splits,
metrics = metric_set(rmse, rsq)
)
collect_metrics(res_mc)tune_grid() — Hyperparameter Search
When your workflow contains tune() placeholders, use tune_grid(wf, resamples, grid) to search over a grid of hyperparameter values. Each combination is evaluated on all folds and the best configuration is selected with select_best().
rf_spec <- rand_forest(mtry = tune(), trees = tune()) |>
set_engine('ranger') |>
set_mode('regression')
wf_tune <- workflow() |> add_recipe(rec) |> add_model(rf_spec)
grid <- grid_regular(mtry(range = c(2, 10)), trees(range = c(100, 500)), levels = 3)
tune_res <- tune_grid(wf_tune, resamples = folds, grid = grid)
collect_metrics(tune_res) |> head()select_best() and finalize_workflow()
After tuning, select_best(tune_res, metric) picks the hyperparameter combination with the best average metric. finalize_workflow(wf, best_params) creates a new workflow with those values substituted in place of tune().
best_params <- select_best(tune_res, metric = 'rmse')
print(best_params)
# Substitute best values into the workflow
final_wf <- finalize_workflow(wf_tune, best_params)
# Fit on all training data, evaluate on test
final_fit <- last_fit(final_wf, split)
collect_metrics(final_fit)Nested Cross Validation
For truly unbiased evaluation when you also tune hyperparameters, use nested cross validation: an outer loop for performance estimation and an inner loop for tuning. In rsample, create an outer vfold_cv and tune within each outer fold using the inner folds.
# Outer folds for unbiased evaluation
outer_folds <- vfold_cv(housing_train, v = 5)
# For each outer fold, tune on the inner training data
res_nested <- tune_grid(
wf_tune,
resamples = outer_folds,
grid = 10, # 10 random configurations
metrics = metric_set(rmse)
)
collect_metrics(res_nested)Comparing Resampling Strategies
Each resampling strategy has trade-offs. Choose based on your dataset size and computational budget:
- k-fold (v=10): Low bias, moderate variance. Default choice for most problems.
- Bootstrap: Works with very small data; higher variance than k-fold.
- Monte Carlo CV: More flexible; good for time-constrained tuning.
- Repeated k-fold: Lower variance; use when you can afford more compute.
# Repeated k-fold: 5-fold repeated 3 times = 15 models fitted
repeated_folds <- vfold_cv(housing_train, v = 5, repeats = 3)
res_rep <- fit_resamples(
wf,
repeated_folds,
metrics = metric_set(rmse, rsq)
)
collect_metrics(res_rep)Quick Check
What does collect_metrics() return when applied to a fit_resamples() result?
Resampling Recap
Key takeaways from Resampling and Cross Validation with rsample:
initial_split(data, prop, strata)creates a stratified train/test split.vfold_cv(data, v = 10)creates k-fold cross-validation folds.bootstraps(data, times)creates bootstrap samples for small datasets.fit_resamples(wf, folds, metrics)evaluates a workflow across all folds.collect_metrics()summarises results with mean and standard error.tune_grid()searches hyperparameters;select_best()picks the winner.finalize_workflow()+last_fit()complete the tuning-to-deployment pipeline.
# Full rsample pipeline
split <- initial_split(data, prop = 0.8, strata = y)
train <- training(split)
folds <- vfold_cv(train, v = 10)
res <- fit_resamples(wf, folds, metrics = metric_set(rmse, rsq))
collect_metrics(res)
# After tuning
best <- select_best(tune_res, metric = 'rmse')
fin_wf <- finalize_workflow(wf_tune, best)
last_fit(fin_wf, split) |> collect_metrics()Frequently asked questions
Is the “Resampling and Cross-Validation with rsample” lesson free?
Yes — the full text of “Resampling and Cross-Validation with rsample” is free to read here on the web, and the R Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the R Academy course, upgrade to CoddyKit PRO.
What will I learn in “Resampling and Cross-Validation with rsample”?
Evaluate models with k-fold CV, bootstrap, and nested resampling. You practise R Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start R Academy?
No prior experience is required. R Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Resampling and Cross-Validation with rsample” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this R Academy lesson?
Yes. Every R Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Feature Engineering with recipes
- Model Specifications with parsnip
- Workflows: Combining Recipes and Models
- Resampling and Cross-Validation with rsample