How to Do

Linear Regression
in SQL
Author: Julia Glick
Edited by: Jessica Schimm

© Mode Analytics, Inc. 2020 | mode.com

Table of Contents

Introduction

Create the predictor variable

Validate your SQL query output in R Notebook

Detrend & deseason the dataset

Validate model outputs

Conclusion

| © Mode Analytics, Inc. 2020 | mode.com Table of Contents

 Introduction

Ordinary least squares (OLS)

regression is one of the most powerful
tools in a data scientist’s pocket.
Personally, as someone who specializes
Why would you
in causal inference, I have built most of want to do linear
my career on OLS and closely related
analyses, from t-tests up through structural
regression in SQL?
time series models.
Well, we’ve all had to work
with simpler tools than we’d
In this intermediate-advanced tutorial,
prefer from time to time (like
I’ll show you how you can get a simple
before a company has adopted
regression analysis done in pure SQL
Mode!) For example, you might
with relatively little pain. Doing OLS in
be working with a dataset that
SQL is a bit tedious but not terribly
is too large to easily export into
complicated, especially once we bring
R. I personally have had to
in slightly more advanced syntax such
analyze and share A/B test
as window functions and WITH clauses.
results when the only analysis
tool I could easily access was
SQL linear regression, in which
26a05a
- SQL was a great tool (but don’t
ask me about the time I wrote a
Mann-Whitney-Wilcoxon U test
in SQL).

| © Mode Analytics, Inc. 2020 | mode.com Introduction | 01

INTRODUCTION
INTRODUCTION
It shows the number of completed
housing units in four major regions in
the U.S. by month from January 1968
through July 2014 and has no missing
observations. The data we will be
working with are counts of completed
housing units, which we’ll be calling
variable “Y” in many of the equations
below, since it is (what we will treat as)
our “dependent” variable.
| © Mode Analytics, Inc. 2020 | mode.com
As you can see, there isn’t a
strong time trend in the raw data
(the line doesn’t seem to rise or
fall very much on average over the
full dataset). It does look like there
might be a fair bit of time of year
seasonality, but it’s hard to tell by
eye how much of this is really
seasonality (repeated year after
year) and how much is variation
that we won’t be able to explain.
Introduction | 03
 Creating the
predictor variable
We’ll start with simple linear regression (i.e., with only
a single predictor) to remove the linear time trend &
work with just one of the four regions for simplicity.

The first step is to create our Here’s a query that shows this
predictor variable, called “X” transformation.
below, which is to say, the linear
effect of time. Unfortunately, time Now that we have our X and Y
in this dataset is represented by variables, we need to actually do
two columns, one for year and the regression. But how? This is
one for month. That’s not so where all those by-hand formulas
convenient for us! that we did in Intro Stats come in
handy. Formulas that are meant
We can create the predictor by to be done by hand are often, in
multiplying the year by 12 and my experience, quite easy to
adding the month; this gets us translate into SQL.
a variable that gives consecutive
numbers to consecutive months.
It’s also really big and hard to
work with, so we’ll “center” the
variable around 0 by subtracting
off the mean time value in our
dataset. If there’s an odd number
of months, that might mean that
all our time values end with .5,
but that’s not a problem.

| © Mode Analytics, Inc. 2020 | mode.com Creating the predictor variable | 04

CREATING THE PREDICTOR VARIABLE
Fortunately, I have an interest in
old-fashioned computation (slide
rules and so forth), so I grabbed my
copy of Edwards’ Statistical Methods
for the Behavioral Sciences from 1954.
No computer-assisted shortcuts to
worry about in this book. The critical
formula is the one for the slope of the
regression line.
The traditional way to do this by
hand for small datasets is to write out
a table with your X and Y values, then
make additional columns to the left
with X2, Y2, and XY values, and finally
total each column. (Edwards gives an
example on page 121, should you
happen to have a copy of the book.)
Sounds a bit like SQL, don’t you think?
| © Mode Analytics, Inc. 2020 | mode.com
1 / n-2 ∑e2
seb₌
∑x2 - (∑x)2 / n
And sure enough, the SQL query
that implements this regression
is pretty straightforward, if a bit
long; here it is. I rely on lots of
common table expressions
(using WITH) to do the
transformations progressively,
which keeps the query and its
logic more readable.
Finding a line of best fit
for Y = a + bX
(the traditional way)
Creating the predictor variable | 05
CREATING THE PREDICTOR VARIABLE

Notice that we centered our X (time) variable in advance.

That means that, in particular, ∑X = 0 by definition, so we
could simplify this formula to just b = ∑XY / ( ∑X ) 2 — but it’s
already simple enough that I’d prefer to leave it in its full form.

Once we know b, getting the intercept a is easy. We just

need the average values of Y and X, computed using ∑Y and
∑X, which we used earlier, together with the number of rows.

Ok, so those are our regression coefficients, but that’s only

part of the story. We want to detrend the data, which is to
say, remove the variance that’s explainable by the linear effect
of time. We can do that by joining the regression coefficients
back to our original dataset.

Take a look at this query. There’s only a single set of

coefficients, so we can copy them to every row by joining
on 1=1, and then use them to create the predicted Y values
Ŷ = a + bX, and the residuals or “error terms” e = Y - Ŷ .

In a time series context, these “error terms”—the part of the

dataset that cannot be explained by the linear trend—are
almost exactly the detrended data that we want! The only
problem is that the residuals are centered around 0, so we
need to add back the mean of Y to get our detrended data.

| © Mode Analytics, Inc. 2020 | mode.com Creating the predictor variable | 06

CREATING THE PREDICTOR VARIABLE

Let’s look at the results.

The above plot shows the raw data again, along with the line of best fit.
For a simple linear regression, that line of best fit represents the explainable
variance. We can see that there is a downward trend on average in the dataset.

And this plot shows the detrended data. Early observations are brought down a
bit and late observations are brought up a bit to cancel out the trend. If we were
to draw a line of best fit for the detrended dataset, it would be a flat horizontal
line through the mean.

| © Mode Analytics, Inc. 2020 | mode.com Creating the predictor variable | 07

 Validating your
SQL query output
in R Notebook

Of course, we also need to Once we’ve done that, it’s a very

validate that the SQL query is giving
us the right answer. So let’s just hop
over to an R Notebook and pull in the
raw dataset. I use a query that includes
the linear time trend predictor so that
we don’t need to worry about
whether we did that correctly.
simple run of the R function
lm (code block #2 at the linked
Notebook) and we can immediately
validate that we’re getting the right
coefficients. Sure enough, the
“Intercept” from the lm call matches
our variable “a”, and the “time_c”
coefficient matches our “b.”

There’s another thing worth mentioning about this

simple regression, which is not really relevant to
detrending the time series but might be useful for
other regression applications: running statistical
significance tests.

| © Mode Analytics, Inc. 2020 | mode.com Validating your SQL | 08

VALIDATING YOUR SQL QUERY OUTPUT
There’s a lot of potential depth here,
so I’ll just touch on it lightly. We’re
going to run a significance test to
see whether our coefficient b
is significantly different from zero:
in this case, it asks whether there is
a non-zero time trend (up or down),
or whether we don’t have enough
evidence to reject a zero trend.
That’s not a terribly interesting question
in some sense, because with any kind
of random-walk-like time series like
To run our statistical significance test, we
need to know the standard error on our
coefficient b. Alas, Edwards 1954 does not
give us this formula; he goes into the t-test
and ANOVA formulations instead, because
they’re generally simpler to do by hand.
Wikipedia comes to the rescue and tells
us that the standard error of b is:
Or, using an easily-derived identity to turn
that denominator into a by-now-familiar
quantity, where e are our errors or residuals
as we discussed above.
| © Mode Analytics, Inc. 2020 | mode.com
this one, there’s going to be some
non-zero trend. But it might be much
more interesting if, for example, our X
predictor took values 0 and 1 for two
different groups of users, and Y was
some kind of engagement metric; in a
situation like that, simple regression is
the equivalent of a two sample t-test.
(Indeed, there is a deep relationship
between t-tests, ANOVA designs and
linear regression, but we can’t get into
all of that here.)
1 / n-2 ∑e2
seb₌
seb₌ 1 / n-2 ∑e2
∑x2 - (∑x)2 / n
Validating your SQL | 09
VALIDATING YOUR SQL QUERY OUTPUT

With the standard error on b in hand, we just need to take the

ratio tb = b / seb to get a random variable with a t-distribution.
This t-stat gives us the statistical significance test we want, on
the null hypothesis that b = 0. So,waving our hands furiously in
the air, we can just glance at t and see whether it’s bigger than 2
in absolute magnitude for a traditional ª = 0.05 significance test.
For anything more precise, you’ll probably want to grab a table
of critical values from your retro stats textbook, or switch off of
SQL, of course. Here’s the query.

Once again, we can validate this result in R using the regression

we already ran. Running summary on our lm output (cell 3 in
the Notebook) gives us a nice table that includes t and p,
matching our query output.

| © Mode Analytics, Inc. 2020 | mode.com Validating your SQL | 10

 Detrend & deseason
the dataset
I promised that we would both
detrend and deseason the dataset
though, which leads us from simple
regression to multiple regression.
To accomplish this, we need to
include multiple predictors, one for
the linear effect plus another eleven
for the month-of-year seasonality
term. (Why eleven? Because there
are twelve months in the year, and
you need k-1 numerical predictors to
capture the variation in a categorical
predictor with k values.)
Going from one predictor to twelve
is going to expand our query
substantially, and involve a lot of ugly
copy-and-pasting. But we can do it.
| © Mode Analytics, Inc. 2020 | mode.com
There’s another problem, though.
We can reproduce our formula from
simple regression above for the
eleven new predictors—but it’s wrong.
Specifically, it’s wrong because when
we do multiple regression, we need
to account not only for the
correlations between each predictor
and the outcome (which that formula
includes), but also the correlations
between each pair of predictors
(which that formula does not include).
And the only reasonable way to ac-
complish that is to use a matrix
formulation, which we can’t easily
code up in SQL. (You can do the
special case of k=2 predictors fairly
easily, but beyond that it’s just not
reasonable.)
Detrend & deseason | 11
DETREND & DESEASON THE DATASET
So, it looks like we’re stuck...Right?
Well, in general, sure. If the
predictors are correlated then we
can’t easily get the right answer
from a query. But for this specific
problem, there’s a path forward:
we can ensure that the predictors
are all uncorrelated with each other.
This wouldn’t work if we were using
just any old predictors that we
observed in the world, say, multiple
different measures of economic
health. But our linear time trend
and our eleven seasonality
predictors aren’t “observations” in
the same kind of way, and we can
construct them to be mutually
orthogonal to each other.
The trick is, we can’t use the obvious
encoding for the seasonality terms.
The easiest approach is to use
eleven 0/1 indicator variables, one
for each month of the year except
for a single base case. So, for
example, one variable would be 1
for January and 0 otherwise.
| © Mode Analytics, Inc. 2020 | mode.com
Another would be 1 for February &
0 otherwise, ...and so on through
November. December wouldn’t
have an indicator variable and
would represent our base case
against which the other 11 months
would be compared. The problem
is, these 11 variables are correlated
with each other.
Instead, we need to move to
orthogonal contrasts. These are
encodings that have the properties
that 1) across all 12 months, the
dot product of any two encodings
is 0, and also 2) the sum across all
12 months of any single encoding
is 0. (The indicator variables above
satisfy 1 but not 2.) There are lots
of different encodings that work,
and they mean different things if
you’re interested in the individual
regression coefficients/statistical
significance tests, but we don’t
care about any of that here because
they’ll all give the same residuals &
thus the same deseasonalized data.
Detrend & deseason | 12
DETREND & DESEASON THE DATASET

Figure 1: Helmert Contrasts

We’ll use one of the contrasts built into R, called Helmert contrasts.
These contrasts compare each level to the average of the previous
levels: February to January; March to the average of January and
February; April to the average of January through March; and so on.
The SQL to generate them is a bit tedious, but straightforward.

| © Mode Analytics, Inc. 2020 | mode.com Detrend & deseason | 13

DETREND & DESEASON THE DATASET
Finally, in order to get our predictors
as close to perfectly orthogonal as
possible, we’ll want to exclude any
years where we don’t have all 12
months of observations. The 2014
data in this dataset is only partial,
so we’ll only include data from
2013 and earlier.
The completed query is here for
the regression coefficients, and
here for the deseasonalized and
detrended data. They should look
a lot like the earlier queries, but
with a lot of extra repeated code.
Obviously in a language like R
or Python, this would all be much
easier to implement (but also you’d
probably be using a pre-existing
package instead!).
Of special note, we need to get
the sums and “sums of squares”
for each predictor, and the “sums
of products” for each predictor with
the outcome (in the "sums_squares"
subquery), then use those to
calculate the regression coefficients
separately for each predictor.
| © Mode Analytics, Inc. 2020 | mode.com
Finally, once we have all of the
regression coefficients, we need
to combine all of them together to
get the predicted values. Once we
have the model’s predicted values,
getting the residuals is the same
as before. But the meaning of
those residuals is different, because
we’re controlling for a different set
of predictors.
In the first set of queries, we were
only controlling for the linear time
trend, and so the residuals allowed
us to create a detrended dataset.
In this case, we control for both
trend and for the month-of-year
seasonality term, and so the residuals
correspond to data that has had
both trend and seasonality removed.
Detrend & deseason | 14
 Validating model outputs

Once again, we can validate our model outputs against a linear

regression in R using the standard lm function (code blocks 4
and 5). Running the equivalent model in R takes a little bit of extra
work because we need to tell R how to set up the orthogonal
contrast codes for the month-of-year terms. Fortunately, we don’t
need to manually code up those contrasts like we did in SQL; we
can use the contrasts function to set things up.

When we look at the output

(code block 5), the results are
not exactly the same, unlike
the simple regression case.
That’s because the contrast
codes are only approxmately
orthogonal to the linear time
trend, as we can see by
running cor on the output of
model.matrix (code block 6).
So our simplified formula
doesn’t give exactly the right
answer. It’s pretty close, though!

| © Mode Analytics, Inc. 2020 | mode.com Validating model outputs | 15

VALIDATING MODEL OUTPUTS

Predictor Correlations

Let’s look at the results. In the first plot, we have the raw data
plotted against the model’s predictions. But unlike before,
the prediction is no longer a “line of best fit.” Instead it
contains both an annually repeating component and the
linear trend we saw earlier. Sure enough, there is quite a
lot of time-of-year seasonality; the difference between the
highest and lowest performing months of the year is
substantial relative to the overall variance of the time series.

| © Mode Analytics, Inc. 2020 | mode.com Validating model outputs | 16

VALIDATING MODEL OUTPUTS

And once that prediction is

subtracted from the raw data
(and the mean added back in),
we can recover the detrended
and deseasonalized dataset.

Notice that the month-to-month

swings are now somewhat less
extreme than before. The data
seems to have been smoothed
out a little bit. This again
suggests that the time-of-year
seasonality was a meaningful
factor in the raw dataset, and
removing it helps us see where
the true seasonally-adjusted
peaks and valleys are.
JAN 68 JUL 14

time_c

| © Mode Analytics, Inc. 2020 | mode.com Validating model outputs | 17

LINEAR REGRESSION IN SQL
Conclusion
This has been a lot of words about
a solution to a rather specific
problem. Sure, lots of problems can
be fit into this framework, possibly
with a bit of effort, but it’s still just
one tool. I’d like to close with some
higher-level takeaways which I think
the discussion above both highlights
and exemplifies.
First, the general point is that there
is a wide range of statistical analyses
you can do in SQL & the algorithms
and computations you need to do
them already exist in many cases
—just, they were intended for hand
computation, not for SQL. Of course
there is a wide range of approaches
you can’t readily do in pure SQL,
such as anything that needs matrix
| © Mode Analytics, Inc. 2020 | mode.com
formulations, anything iterative like
EM algorithms or gradient descent,
etc, but those are also generally
the algorithms that aren’t suited to
computation by hand. Adapting
these old statistical computations
to SQL lets you run them on
enormously vaster quantities of
data than anything from the 1950s
or 1970s, and that can transform
their usefulness all by itself.
(As a side note: for even larger
quantities of data, the above
regression formulation fits nicely
into an incrementally updating
system. Why not try updating your
regression coefficients right in
your ETL?)
Conclusion | 18
CONCLUSION
Second, all of these methods &
approaches are tools. Different types
of modeling approaches are tools.
Different software environments and
programming languages are tools.
If you have a big toolbox, and
you’re flexible about how you can
fit multiple tools together, then when
you’re confronted with a novel problem
you can craft a combination of tools
that fit the problem. If you don’t, then
you’re going to be limited to forcing
the problem to fit the tool. And that
rarely results in good analyses.
| © Mode Analytics, Inc. 2020 | mode.com
Finally, if you’ll allow me
to editorialize a bit, simple
statistical methods aren’t just
useful, they’re worthwhile.
In most cases, I believe, the
limitations on your ability to extract
business-relevant understanding
from raw data is not due to the
sophistication of your models.
Generally, you will benefit more
from understanding & improving
your measurement, carefully
crafting your experimental
designs, (whether randomized or
observational), and thinking very
deeply about the appropriate
causal interpretation of your results,
than from making your statistical
analyses more complicated. As data
scientists, we should be ready to do
“boring” analyses when they will
result in exciting business insights.
Conclusion | 19