Plotting Logistic Regressions Decision Boundaries in 2D.

[Note: I've made a Jupyter Notebook (Python) for this so that you can mess around with a few of these ideas yourself. The figures come from this notebook.]

⚫ ⚫ ⚫ ⚫

In this post, I'm going to assume that you know already know what a logistic regression is and how to create a logistic regression model in your programming language of choice. This is a quick post is on how to plot the logistic regression bounaries in 2D.

Just so we define some important terms, I'm considering the logistic regression boundary the line, plane, or hyperplane which separates points which are predicted to be 0 and points which are predicted to be 1. Moreover, this is the boundary which is obtained from the model once we've optimized the model to minimize cost.

Let's take a look at this (easily separated!) data:

Cool, right? Check out that weird blue point. Wonder what happened there.

In any case, once we have the data we'll create a model in our favorite programming language, and this model will return some coefficients (or weights). In Python, here's what my code looks like (also in the notebook above):

import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

from bokeh.plotting import figure, show, output_notebook

output_notebook() # for jupyter notebook only.

# Import the data into a numpy array.
data = make_classification(n_samples=100, 
                           n_features=2,
                           n_informative=2,
                           n_redundant=0,
                           weights=[.6, .4], 
                           random_state=246810)

# Break into feats + target and scale features.
raw_feats = data[0] # appending the intercept.
scale = StandardScaler().fit(raw_feats)

#Scale feats, not target.
features = scale.transform(raw_feats)
target = data[1]

# Plot the data.
p = figure(height=450, width=450)
p.xaxis.axis_label = "Exam 1 Score"
p.yaxis.axis_label = "Exam 2 Score"

not_admitted = features[target == 0]
admitted = features[target == 1]

p.scatter(not_admitted[:,0],
	not_admitted[:,1], 
	color="red", 
	legend="Not Admitted")
p.scatter(admitted[:,0], 
	admitted[:,1], 
	color = "blue", 
	legend="Admitted")

show(p)

# Create the logistic regression.
log_reg = LogisticRegression().fit(features, target)

This last past is what actually creates the regression. From this, we can return our weights (in my case, I have two coefficients and an intercept; all of these I consider weights). Let's call the intercept $$w_{0}$$ and the two feature weights $$w_{1}, w_{2}$$. Then the logistic equation associated with this regression is given by \[\frac{1}{1 + \exp(w_{0} + w_{1}x_{1} + w_{2}x_{2})}\] and a point $$(x_{1}, x_{2})$$ will be predicted to be a value 1 whenever this function is greater than 0.5; similarly, a point will be predicted to be a value 0 whenever this function is less than 0.5. Hence, the boundary is exactly at: \[\frac{1}{1 + \exp(w_{0} + w_{1}x_{1} + w_{2}x_{2})} = 0.5\] which we can reduce, using some algebra, to \[w_{0} + w_{1}x_{1} + w_{2}x_{2} = 0\] In two dimensions, we can actually plot this on our graph above! In this case, we'll solve for $$x_{2}$$ as a function of $$x_{1}$$, and we get \[x_{2} = \frac{-w_{0}}{w_{2}} + \frac{-w_{1}}{w_{2}}{x_{1}}\] which we can plot!

# Plot the data with decision boundary.
p = figure(height=450, width=450)
p.xaxis.axis_label = "Exam 1 Score"
p.yaxis.axis_label = "Exam 2 Score"

not_admitted = features[target == 0]
admitted = features[target == 1]

p.scatter(not_admitted[:,0], not_admitted[:,1], color="red", legend="Not Admitted")
p.scatter(admitted[:,0], admitted[:,1], color = "blue", legend="Admitted")

w0 = log_reg.intercept_
w1, w2 = log_reg.coef_[0]

X = np.array([-2,2])
p.line(X, (-w0-w1*X)/w2)

show(p)

We see here that this decision boundary splits our data into two camps as best as possible.

With more than two features...

We can still do approximately the same thing, but it's just harder (or impossible!) to plot in 2- or 3- dimensions. The equations are exactly the same. If we let $$x_{0} = 1$$ (this corresponds to the intercept), then our logistic regression boundary with $$n$$ features will look like this: \[\sum_{i=0}^{n}w_{i}x_{i} = 0\] Even without solving for one variable, we know that this is the equation of a hyperplane in $$n$$ dimensions (remember: $$x_{0}$$ is a constant). It's just a pain to plot this when $$n > 3$$. Bummer.