The Goal

Today, we are are going to learn a different technique for building models when our Y is numeric. This technique is called a regression tree. Regression trees are particularly useful when we have a lot of potential X variables and we are not sure which we want to use. Let’s explore.

The Data

Instead of our typical data for this course, today we are going to work with a data set containing information on n = 333 penguins from three different species. To load the data, you need to use the following two lines of code:

library(palmerpenguins)
data(penguins)
penguins <- na.omit(penguins)

We have a client who is interested in building a model for Y = the body mass of a penguin in grams.

In addition to this response variable, we have information on 7 explanatory variables.

Correlations

Our goal at the moment is to model body mass, which is a numeric response variable. This means we might tend to reach for a least-squares regression model. However, regression works best when (1) we do not have many strong relationship among our X variables and (2) our numeric X variables have a linear relationship with Y.

Let’s check for correlations among the X variables first. To do this, we need the following library:

suppressMessages(library(corrplot))

Once you have loaded the library, we need to create a correlation matrix, and then plot it.

# Create the correlation matrix
M <- cor(penguins[,-c(1:2,7)])
# Create the plot
corrplot(M, method = "number", type = "upper")

Let’s also check the shape of the relationship between bill depth and body mass.

Do we notice in the picture that there are two defined groups in this graph? The Gentoo penguins seem to have a different group of points from the Adelie and Chinstrap penguins.

This grouping structure within the data, as well as the correlation among our potential X variables, indicates that these data might benefit from a type of model called a regression tree.

Regression Tree with one Split

A regression tree is a type of model we can build for a numeric Y variable. We can use numeric or categorical X variables to help model Y. It’s actually easiest to learn about these models by building one, so let’s create the model first and then we will talk about what it does.

To create regression trees, you will need the following libraries:

suppressMessages(library(rpart))
suppressMessages(library(rattle))

To build a tree in R, the function you need is rpart. Let’s stay I want to build a tree using sex as my only X. The code I need for that is:

tree1 <- rpart(body_mass_g ~ sex, data = penguins)

To visualize the tree, use:

fancyRpartPlot(tree1, caption = "Tree 1")

Here is your first regression tree!! It’s actually three green boxes connected by lines. The root is the box at the very top. The other two green boxes at the bottom are called leaves.

You will notice that under the root we have sex = female. Trees are built by dividing data into groups using our X variables. In this case, we have started our tree by dividing our data into female penguins (the left leaf) and male penguins (the right leaf).

Inside each leaf we have three numbers. The n = part tells us how many rows of data are in that leaf. For instance, in the left leaf we have n = 165, so there are 165 female penguins in the dataset. Right next to n =, we have a percentage, which just converts that count into a percent.

There is one more number in the boxes, and this is perhaps the most important number. Recall that our goal is predict the body mass of the penguins. In linear regression, we do this by finding linear relationships between features and our Y variable, essentially estimating how much of a relationship each X has with Y to get our final predictions.

Trees work differently. To get our prediction for Y, all we do is average together the body masses of the penguins in each leaf. The number 3862 in the top of the left leaf tells us that the average body mass for all female penguins is 3862 grams. This means that the tree will predict a body mass of 3862 grams for any female penguins.

Let’s try a second tree.

 tree2 <- rpart(body_mass_g ~ species, data = penguins)

# Make predictions 
predictions_tree <- predict(tree1, data = penguins)
# Obtain the RSS
sum( (predictions_tree - penguins$body_mass_g)^2)

Regression trees with more leaves

So far, we have only looked at trees with exactly 2 leaves. However, trees can have more than two leaves, and generally do. The goal with trees to create leaves such that penguins in the leaves are similar to one another in terms of their body masses. In other words, we are trying to find groups of penguins with similar body masses by using the X variables we have. Once we have done this, we average together the penguins in each group (sinec we know they are similar in terms of Y) to get a predicted body mass for a new penguins with these same traits!

Let’s try a tree using both sex and species.

tree3 <- rpart(body_mass_g ~ sex + species, data = penguins)

As we can see in tree3, we now have more than 2 leaves! Specifically, we have 4. We only consider boxes to be leaves if they are all the way at the bottom of the tree.

Let’s build a bigger tree:

tree4 <- rpart(body_mass_g ~ sex + 
                 bill_length_mm + bill_depth_mm, 
               data = penguins)

To visualize the tree, use:

fancyRpartPlot(tree4, caption = "Tree 4")

You will notice in Tree 4 that sex is no longer the splitting rule right under the root. Instead, sex has moved farther down the tree. Why???

Exploring the Whole Tree

Now that we have explored the basic structure of how a tree works, let’s see what happens when we allow all of our X variables to be used in growing the tree. We do this using the ~. notation in R.

treeAll <- rpart(body_mass_g ~ . , data = penguins)

fancyRpartPlot(treeAll, caption = "Tree All")

If your tree does not use all the possible X variables that you gave it, this means that adding more splits to the tree was no longer decreasing the residual sum of squares enough for that reduction to be meaningful. Once splitting no longer helps improve the residual sum of squares, we stop growing the tree. You will notice that even though we have allowed all the X variables to be used, only a few show up in the tree! This shows another important property of trees: variable selection. Trees only use the variables that give the largest reduction in the RSS. This means that when we have a lot of features to consider, trees are nice way to determine which ones might be most important for modeling Y!

If your task is prediction, we have seen that we can use the tree to help us predict body mass. However, what if our goal is association? In other words, what if we are trying to understand the relationships in the data?

Try it!