0Pricing
Machine Learning Academy · Lesson

Training Linear Regression with scikit-learn

Learners will use sklearn.linear_model.LinearRegression to fit a model on training data, make predictions, and inspect the learned coefficients.

Training Linear Regression with scikit-learn is a free Machine Learning Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

scikit-learn's LinearRegression API

Scikit-learn's LinearRegression class provides a clean, consistent API for fitting ordinary least-squares linear regression. Like all scikit-learn estimators, it follows the same pattern: instantiate, fit, predict.

Under the hood, it uses the Normal Equation (or a numerically stable SVD decomposition for large feature matrices) to find the exact optimal weights in a single computation — no iterative training loop is needed. This makes it extremely fast even on large datasets with many features.

from sklearn.linear_model import LinearRegression
import numpy as np

# Instantiate with optional parameters
model = LinearRegression(
    fit_intercept=True,   # default: add bias term
    copy_X=True,          # don't modify input arrays
    n_jobs=-1             # use all CPU cores
)

print(model)  # shows default parameters

Preparing the Data

Before training, your data must be in the correct shape. scikit-learn expects:

  • X — a 2D array of shape (n_samples, n_features)
  • y — a 1D array of shape (n_samples,)

The most common shape error is passing a 1D array for X when you have one feature. The fix is to reshape with .reshape(-1, 1). Always split into training and test sets before any fitting — evaluation on training data gives misleadingly perfect scores.

import numpy as np
from sklearn.model_selection import train_test_split

# Simulate housing data
np.random.seed(42)
sqft = np.random.uniform(500, 3000, 200)
price = 150 * sqft + 50000 + np.random.normal(0, 25000, 200)

# Reshape X to 2D (n_samples, n_features)
X = sqft.reshape(-1, 1)  # shape (200, 1)
y = price                  # shape (200,)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print('Train size:', X_train.shape, '| Test size:', X_test.shape)

Fitting the Model

Calling model.fit(X_train, y_train) computes the optimal weights using the Normal Equation and stores them inside the model object. After fitting, two attributes become available:

  • model.coef_ — an array of weights, one per feature
  • model.intercept_ — the bias term (scalar)

The trailing underscore in attribute names is a scikit-learn convention indicating that the attribute was set during fitting — it does not exist before fit() is called.

from sklearn.linear_model import LinearRegression
import numpy as np
from sklearn.model_selection import train_test_split

np.random.seed(42)
X = np.random.uniform(500, 3000, 200).reshape(-1, 1)
y = 150 * X.ravel() + 50000 + np.random.normal(0, 25000, 200)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = LinearRegression()
model.fit(X_train, y_train)

print(f'Slope (coef_):     {model.coef_[0]:.2f}')   # should be ~150
print(f'Intercept:         {model.intercept_:.2f}')  # should be ~50000

Making Predictions

After fitting, call model.predict(X) to generate predictions. This computes the dot product of the feature matrix with the learned weights, plus the intercept. You can predict on training data, test data, or entirely new data.

Remember: evaluating on training data is not a valid performance estimate. Always evaluate on the held-out test set to get an honest estimate of how well the model will perform on unseen data.

# Predict on test set
y_pred = model.predict(X_test)

print('First 5 predictions vs actual:')
for actual, pred in zip(y_test[:5], y_pred[:5]):
    print(f'  Actual: ${actual:,.0f}  |  Predicted: ${pred:,.0f}  |  Error: ${actual-pred:,.0f}')

# Predict a single new house
new_house = np.array([[1750]])  # must be 2D
print(f'\nPrediction for 1750 sqft: ${model.predict(new_house)[0]:,.0f}')

Evaluating on the Test Set

Use scikit-learn's metrics functions to quantify test-set performance. The model.score(X_test, y_test) shortcut returns R² directly. For the full picture, also compute RMSE and MAE.

Comparing these metrics across different models (e.g., linear regression vs random forest) allows you to objectively select the best approach for your dataset. Always include a naive baseline (like always predicting the mean of y) to confirm your model is actually adding value.

from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np

y_pred = model.predict(X_test)

rmse = np.sqrt(mean_squared_error(y_test, y_pred))
mae  = mean_absolute_error(y_test, y_pred)
r2   = r2_score(y_test, y_pred)

print(f'Test RMSE: ${rmse:,.0f}')
print(f'Test MAE:  ${mae:,.0f}')
print(f'Test R2:   {r2:.3f}')

# Baseline: always predict the mean
baseline_rmse = np.sqrt(((y_test - y_train.mean()) ** 2).mean())
print(f'Baseline RMSE: ${baseline_rmse:,.0f}  (must beat this!)')

Visualising the Regression Line

After fitting, plot the learned regression line on top of the training data scatter plot. This visual check answers: does the line make sense? Is it in roughly the right place? Are there large groups of outliers that the model consistently misses?

For a single-feature model this is straightforward. For multi-feature models you will need to plot predictions vs actual values (a residual plot or a 'predicted vs actual' scatter plot) instead, since the decision boundary lives in high-dimensional space.

import matplotlib.pyplot as plt
import numpy as np

# Plot training data and the fitted line
plt.figure(figsize=(8, 5))
plt.scatter(X_train, y_train, alpha=0.4, label='Training data')
plt.scatter(X_test, y_test, alpha=0.4, color='orange', label='Test data')

# Draw the regression line across the full range
x_range = np.linspace(500, 3000, 100).reshape(-1, 1)
y_range = model.predict(x_range)
plt.plot(x_range, y_range, 'r-', linewidth=2, label='Regression line')

plt.xlabel('Square Footage')
plt.ylabel('Price ($)')
plt.legend()
plt.title('Linear Regression Fit')
plt.show()

Interpreting the Coefficients

The learned coef_ values carry direct business interpretation. For a house-price model with multiple features, each coefficient tells you how much the predicted price changes when that feature increases by one unit, holding all other features constant (the ceteris paribus interpretation).

However, this interpretation is only valid when features are on similar scales. If one feature is in metres and another is in kilometres, their coefficients are not directly comparable. Always scale features before interpreting coefficients in multi-feature models.

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import numpy as np
import pandas as pd

# Multi-feature housing data
features = ['sqft', 'bedrooms', 'bathrooms', 'age']
np.random.seed(0)
X_data = pd.DataFrame({
    'sqft': np.random.randint(600, 3000, 100),
    'bedrooms': np.random.randint(1, 6, 100),
    'bathrooms': np.random.randint(1, 4, 100),
    'age': np.random.randint(1, 50, 100)
})
y_data = 150*X_data['sqft'] + 8000*X_data['bedrooms'] - 500*X_data['age'] + 30000

model = LinearRegression().fit(X_data, y_data)
for feat, coef in zip(features, model.coef_):
    print(f'{feat}: {coef:.1f}')

Residual Analysis

Plotting residuals against predicted values is a critical diagnostic step. For a well-fit model, residuals should be randomly scattered around zero with no discernible pattern. Systematic patterns indicate a problem:

  • A funnel shape (residuals spread more for larger predictions) indicates heteroscedasticity — the variance of errors is not constant.
  • A curved pattern indicates the true relationship is non-linear and a linear model is under-fitting.
  • Residuals that are systematically positive or negative for certain prediction ranges mean the model is biased in that region.
import matplotlib.pyplot as plt
import numpy as np

y_pred = model.predict(X_test)
residuals = y_test - y_pred

plt.figure(figsize=(8, 5))
plt.scatter(y_pred, residuals, alpha=0.5)
plt.axhline(y=0, color='red', linestyle='--', linewidth=1.5)
plt.xlabel('Predicted Values')
plt.ylabel('Residuals')
plt.title('Residual Plot (should be random scatter around 0)')
plt.show()

print(f'Mean residual: {residuals.mean():.2f}  (should be ~0)')
print(f'Std of residuals: {residuals.std():.2f}')

Regularised Linear Regression

Plain LinearRegression can overfit when you have many features relative to training examples, or when features are highly correlated (multicollinearity). Scikit-learn provides two regularised variants:

  • Ridge (L2 regularisation) — shrinks all coefficients towards zero proportionally, keeping all features but reducing their magnitude.
  • Lasso (L1 regularisation) — shrinks some coefficients all the way to zero, effectively performing feature selection.

The alpha parameter controls regularisation strength: higher alpha = more shrinkage = simpler model.

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
import numpy as np

np.random.seed(42)
X = np.random.randn(100, 20)  # 20 features, many irrelevant
y = 3*X[:, 0] + 2*X[:, 1] + np.random.randn(100)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

for name, model in [('OLS', LinearRegression()),
                    ('Ridge', Ridge(alpha=1.0)),
                    ('Lasso', Lasso(alpha=0.1))]:
    model.fit(X_train, y_train)
    print(f'{name} R2: {r2_score(y_test, model.predict(X_test)):.3f}')

Cross-Validation Score

A single train-test split can give a misleadingly high or low estimate of model performance depending on which examples happen to land in the test set. Cross-validation gives a more reliable estimate by averaging performance across multiple folds.

Use cross_val_score for a quick k-fold evaluation without manually splitting data. The returned scores are one per fold; report the mean and standard deviation to convey both expected performance and variability.

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
import numpy as np

np.random.seed(42)
X = np.random.randn(200, 3)
y = 2*X[:, 0] - X[:, 1] + np.random.randn(200)*0.5

model = LinearRegression()
scores = cross_val_score(
    model, X, y,
    cv=5,          # 5-fold cross-validation
    scoring='r2'   # use R2 as the metric
)

print('CV R2 scores per fold:', scores.round(3))
print(f'Mean R2: {scores.mean():.3f} (+/- {scores.std():.3f})')

End-to-End Pipeline

In professional ML workflows, preprocessing and the model are always combined into a Pipeline. This prevents data leakage during cross-validation (the scaler is fitted on each training fold separately) and simplifies deployment (one object to save and load).

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
import numpy as np

np.random.seed(42)
X = np.random.randn(200, 3)
y = 2*X[:, 0] - X[:, 1] + np.random.randn(200)*0.5

# Scale + Fit in one object
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', LinearRegression())
])

scores = cross_val_score(pipeline, X, y, cv=5, scoring='r2')
print(f'Pipeline CV R2: {scores.mean():.3f} (+/- {scores.std():.3f})')

# Fit once for deployment
pipeline.fit(X, y)
print('Pipeline fitted and ready for deployment.')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: fit() computes optimal weights stored in coef_ and intercept_, predict() applies the learned line to new data, and residual plots reveal whether the linear assumption holds or a more complex model is needed. Next up we extend linear regression to multiple features, learn how to interpret each coefficient as a measure of feature importance, and evaluate with R-squared.

Frequently asked questions

Is the “Training Linear Regression with scikit-learn” lesson free?

Yes — the full text of “Training Linear Regression with scikit-learn” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Training Linear Regression with scikit-learn”?

Learners will use sklearn.linear_model.LinearRegression to fit a model on training data, make predictions, and inspect the learned coefficients. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Training Linear Regression with scikit-learn” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. The Equation of a Line: Slope, Intercept, and Predictions
  2. Cost Functions and Least Squares
  3. Training Linear Regression with scikit-learn
  4. Multiple Linear Regression and Feature Importance
← Back to Machine Learning Academy