Proportions with mean()
One of the most common tasks I want to do is calculate the proportion of observations (e.g., rows in a data set) that meet a particular condition. For example, what is the proportion of missing data, or people over the age of 18?
There is a suprisingly easy solution to handle this problem: by combining boolean vectors and mean().
Step 1: creating a boolean vector #
We start with boolean vectors, which is a vector that is TRUE whenever our observation meets our condition, or FALSE whenever it’s not. We create this boolean vector by submitting our observations to some sort of conditional statement (or relevant function like is.na()). Let’s take a look at a few examples:
x <- letters[1:10]
x == "b" # return a boolean vector which is TRUE whenver x is "b"
#> [1] FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
x <- 1:10
x > 5 # TRUE whenever x is greater than 5
#> [1] FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE
x > 5 & x %% 2 == 0 # TRUE when x > 5 AND divisible by 2
#> [1] FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE
x <- c(1, 2, NA, 4)
is.na(x) # TRUE when x is a missing value
#> [1] FALSE FALSE TRUE FALSE
If you’re unsure with how the above works, take a look at this page on R Programming Operators.
With this under our belt, it seems simple enough to create a boolean vector that tells us when our observations meet some condition (TRUE) or not (FALSE).
Step 2: calculating the proportion of TRUE #
From this point, all we need to do is wrap our conditional statment inside mean():
x <- 1:10
mean(x > 5) # proportion of values in x greater than 5
#> [1] 0.5
How/why does this work? If you take a look at the help page with ?mean(), you’ll read that the arguement x can be a logical vector. But what does this mean. Well, when you use a boolean vector, mean() first converts it to a numeric vector. This means that every TRUE becomes 1, and every FALSE becomes 0:
x <- 1:10
as.numeric(x > 5)
#> [1] 0 0 0 0 0 1 1 1 1 1
It then computes the mean of these 1’s and 0’s. At this point, you just need to think a little. How is the mean calculated? Well, it’s the sum of all the values, divided by their length? So the sum of a vector of 1’s and 0’s will be the total number of 1’s! Divided by the length then gives you the proportion. As a side note, you might realise that you can use sum() instead of mean() if you want to calculate the frequency. Let’s break this right down:
x <- 1:10
x
#> [1] 1 2 3 4 5 6 7 8 9 10
test <- x > 5
test
#> [1] FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE
as.numeric(test)
#> [1] 0 0 0 0 0 1 1 1 1 1
sum(test)
#> [1] 5
length(test)
#> [1] 10
sum(test) / length(test)
#> [1] 0.5
mean(test)
#> [1] 0.5
Some useful examples #
At this point, we can apply this to all sorts of problems. Here are some examples using the mtcars data set:
d <- mtcars
head(d)
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
#> Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
#> Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
#> Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
#> Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
#> Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
# Proportion of rows (cars) with cyl == 6 (6 cylinders)
mean(d$cyl == 6)
#> [1] 0.21875
# Proportions of rows (cars) with hp > 250 (horsepower over 200)
mean(d$hp > 250)
#> [1] 0.0625
# Proportion of cars with 8-cylinders and that get more than 15 Miles/(US) gallon
mean(d$cyl == 8 & d$hp > 15)
#> [1] 0.4375
Sign off #
Thanks for reading and I hope this was useful for you.
For updates of recent blog posts, follow @drsimonj on Twitter, or email me at drsimonjackson@gmail.com to get in touch.
If you’d like the code that produced this blog, check out my GitHub repository, blogR.