flowchart LR
A["Import"] --> B["Tidy"]
B --> C
subgraph C["Understand"]
direction LR
T["Transform"] --> V["Visualize"]
V --> M["Model"]
M --> T
end
C --> D["Communicate"]
5 Into the Tidyverse: glimpse, filter, and the Pipe
By the end of this chapter you should be able to:
- Explain what the tidyverse is, and load it.
- Look inside a data frame with
glimpse(). - Keep the rows you need with
filter(), using one or more conditions. - Chain steps together with the pipe,
|>. - Explain how
filter()and base R brackets treat missing values differently.
In the last chapter you selected rows of a data frame with square brackets, as in grades[grades$exam2 > 90, ]. That works, but it gets long and hard to read quickly: the data frame’s name appears again for every condition, and every step needs a new variable. This chapter introduces the tidyverse, a set of packages designed to make that kind of work shorter and easier to read, and three tools from it that you will use in almost every assignment from here on: glimpse() to see what is in a data frame, filter() to keep the rows you need, and the pipe, |>, to chain steps together.
The examples use real data: every flight that left New York City’s three airports in 2013, from the nycflights13 package. By the end of the chapter you will be able to answer questions such as “how many flights left LaGuardia in June?” in four short lines.
5.1 What is the tidyverse?
5.1.1 The data science workflow
The tidyverse is a family of R packages built for data science. Its packages share a design philosophy, a grammar, and a way of storing data, so they fit together. They are organized around the workflow in Figure 5.1, from the book R for Data Science (Wickham et al. 2023):
First you import data into R. Then you tidy it into a consistent shape, with one variable per column and one observation per row. Then you go round a loop of transforming it (keeping the rows you need, making new columns, summarizing), visualizing it, and modeling it, until you understand it. Finally you communicate what you found, usually in an R Markdown report. This chapter lives in the Tidy and Transform boxes: getting data into shape and pulling out the rows you need.
5.1.2 The core packages
One command, install.packages("tidyverse") (or one trip to the Packages tab), installs the core tidyverse packages (Table 5.1).
| Package | What it is for |
|---|---|
| dplyr | Transforming data frames: keeping rows, choosing columns, making new columns, summarizing |
| ggplot2 | Plots and charts |
| tidyr | Reshaping data into a tidy form |
| readr | Reading files such as CSVs, with read_csv() |
| tibble | The tidyverse’s version of the data frame |
| stringr | Working with text |
| forcats | Working with categories (factors) |
| lubridate | Working with dates and times |
| purrr | Repeating a task over many inputs |
This chapter uses dplyr, which supplies glimpse() and filter(). Posit publishes a one-page cheatsheet for dplyr, Data transformation with dplyr, linked on the course Canvas page and at https://posit.co/resources/cheatsheets/. filter() is on the front page under Manipulate Cases (Extract Cases). It is worth keeping open while you work.
5.1.3 Loading the tidyverse
Load the tidyverse at the top of every R Markdown file, the same way as any other package:
library(tidyverse)The first time you do, R prints a block of messages like this (your version numbers may differ):
── Attaching core tidyverse packages ──────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.5
✔ forcats 1.0.0 ✔ stringr 1.5.1
✔ ggplot2 3.5.1 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.1
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
These messages are normal. They are not errors, even though some are printed in red. The first block lists the packages that were loaded. The second, Conflicts, says that two packages contain a function with the same name, and which one R will use. Here, dplyr’s filter() masks the filter() in base R’s stats package (a function for time series). From now on, when you type filter(), you get dplyr’s version, which is the one you want. If you ever need the other one, you can name its package explicitly with two colons: stats::filter().
5.1.4 Tidying data is a set of verbs
What do we actually do when we tidy data? We select, extract, arrange, sort, combine, group, join, and summarize. dplyr gives each of these jobs its own function, named with a verb (Table 5.2).
| dplyr verb | What it does | Covered |
|---|---|---|
filter() |
Keeps the rows that meet a condition | This chapter |
select() |
Picks columns | Next chapter |
arrange() |
Sorts rows | Next chapter |
mutate() |
Makes new columns, or changes existing ones | Next chapter |
group_by() and summarize() |
Collapse groups of rows into summary values | Later |
left_join() and friends |
Combine two tables | Later in the course |
Every dplyr verb works the same way: its first argument is a data frame, the arguments after it say what to do, and it returns a new data frame. That consistency is what makes the pipe, later in this chapter, work so well.
5.2 Seeing inside a data frame with glimpse()
5.2.1 The flights data
The nycflights13 package contains a data frame called flights: one row for every flight that departed from New York City’s three airports (Newark, EWR; John F. Kennedy, JFK; and LaGuardia, LGA) in 2013. You might use it to compare flights by day, by airport, or by how late they left. Install it once from the Packages tab, then load it along with the tidyverse:
library(tidyverse)
library(nycflights13)The live code boxes in this chapter load dplyr, the part of the tidyverse this chapter uses, rather than the whole tidyverse, because it loads faster in a web browser. Everything works the same way in RStudio with library(tidyverse). The first time you run a box, it takes a little while to download the flights data.
5.2.2 glimpse()
flights has more than 300,000 rows, far too many to print. Typing flights shows only the first few rows and as many columns as fit across the screen. glimpse() is the tidyverse’s way to see everything that is in a data frame at once. It turns the data on its side: each column becomes one line of output, showing the column’s name, its type, and its first few values. Run it:
The first two lines give the size: 336,776 rows and 19 columns. Below them is one line per column. The type of each column is in angle brackets, abbreviated (Table 5.3).
glimpse() shows them.
In glimpse() |
Type | Example columns in flights |
|---|---|---|
<int> |
integer (whole numbers) | year, month, day, dep_time |
<dbl> |
numeric (“double”), may have decimals | dep_delay, arr_delay, distance |
<chr> |
character (text) | carrier, origin, dest, tailnum |
<dttm> |
a date and time | time_hour |
The columns this chapter uses are month and day (numbers, so June is 6, not "June"), origin (the three-letter code of the airport the flight left from), and dep_delay, the departure delay in minutes. A negative dep_delay means the flight left early.
glimpse() is one of several ways to look at a data frame. head(flights) prints the first six rows; str(flights), from the last chapter, is base R’s version of glimpse(); and in RStudio, View(flights) opens the whole table in a spreadsheet-like tab. Get in the habit of looking at every data frame before you work with it. Most surprises (a number stored as text, a column you didn’t expect, far fewer rows than you thought) show up in a glimpse.
5.3 Keeping rows with filter()
5.3.1 One condition
filter() keeps the rows of a data frame that meet a condition. Its first argument is the data frame, and the arguments after that are one or more logical conditions:
filter(data, condition)It returns only the rows where the condition is TRUE. Here are the flights that left on the first day of a month:
flights is the data frame, and day == 1 is the logical condition. Notice that, inside filter(), you refer to the columns by name alone, day rather than flights$day: filter() knows to look for them in the data frame. The result has 11,036 rows, the flights on the 1st of each of the twelve months.
The condition uses ==, which tests whether two things are equal. A single = sets an argument, and writing filter(flights, day = 1) is one of the most common mistakes in R. dplyr catches it and tells you, with an error that says you have used = instead of == and asks “Did you mean day == 1?”
filter() doesn’t change flights; it returns a new data frame. To keep the result, store it with <-, and give it a new name, so that you still have the original:
5.3.2 Several conditions
Give filter() more than one condition, separated by commas, and it keeps the rows where all of them are TRUE. A comma inside filter() means AND, exactly like &. These two lines both find the flights on January 1:
Both return 842 rows. For OR, use |. This finds the flights in June or July:
Conditions on text work the same way, but the text needs quotation marks, as always: origin == "LGA", not origin == LGA. Without the quotes, R looks for a column or variable named LGA and doesn’t find one.
5.4 The pipe
5.4.1 Chaining steps with |>
Most questions take several steps. Suppose you want to know how many flights left LaGuardia early, or on time, on January 1, 2013. That takes three filters and a count. You could save a new data frame after every step, or nest the functions inside one another, but both get hard to read.
You met the pipe, |>, in an earlier chapter. It takes the output of one function and feeds it in as the first argument of the next. Because every dplyr verb takes a data frame as its first argument and returns one, verbs can be chained with pipes as long as you like:
Read the pipe out loud as “and then”: take flights, and then keep January 1, and then keep LaGuardia, and then keep the flights that left on time or early (a delay of zero or less), and then count the rows. The answer is 172 flights. Each step narrows the rows further, and the code reads in the same order as the question.
5.4.2 Don’t repeat the data frame
With the pipe, the data frame goes in front of the pipe, not inside filter(). These two lines do exactly the same thing:
filter(flights, day == 1)
flights |> filter(day == 1)In the second, |> passes flights into filter() as its first argument. So inside a pipe chain you never write filter(flights, ...). If you do, flights is passed in twice, once by the pipe and once by you, and R reports an error.
René Magritte’s 1929 painting The Treachery of Images shows a pipe above the words Ceci n’est pas une pipe, “This is not a pipe.” It isn’t: it’s a picture of one. In the same spirit, |> isn’t really a pipe either. It’s an operator, a piece of R grammar that rewrites x |> f(y) as f(x, y) before the code runs. The plumbing metaphor is just a good way to picture it: data flow in at one end and out at the other.
You may also see %>% in older code and online answers. It is the original pipe, from the magrittr package (named after Magritte), and does almost the same job. The native pipe |> was added to R itself in version 4.1, and it is the one this course requires.
5.5 Missing values: why the counts don’t match
5.5.1 The same question, two ways
Here is the January 1 LaGuardia question again, answered first with base R brackets, as in the last chapter, and then the tidy way with filter() and the pipe:
The base R way needs more typing, and more places for a typo: flights$ for every condition, and a new object at every step. The tidy way reads top to bottom. But look at the answers. Base R finds 174 flights, and filter() finds 172. They don’t match. Why?
5.5.2 NA: a missing value
The answer is missing values. In R, a value that is unknown or missing is written NA (“not available”). Flights that were cancelled never departed, so their departure delay is unknown: dep_delay is NA. Two of the January 1 LaGuardia flights were cancelled.
What is NA <= 0? R can’t say whether an unknown number is less than zero, so the answer is itself unknown: NA, not TRUE or FALSE. Each approach handles that differently (Table 5.4):
- Base R brackets don’t drop a row when the condition is
NA. They return a row in which every value isNA, so the result has two extra rows of missing values, 174 in all. filter()keeps only the rows where the condition isTRUE. It quietly drops rows where the condition isNA, so it finds 172.
filter() treat each result of a condition.
Condition is TRUE |
Condition is FALSE |
Condition is NA |
|
|---|---|---|---|
Base R df[condition, ] |
Row kept | Row dropped | A row of NAs is returned |
filter(df, condition) |
Row kept | Row dropped | Row dropped |
You can see the two extra rows by looking at the last rows of the base R result:
5.5.3 Deciding what happens to NAs
Neither approach is wrong, but they give different answers, and nothing warns you. That is annoying, and it is exactly the kind of quiet difference that leads to wrong results in a report. The lesson is to check the number of rows after every filter, and to decide on purpose what should happen to missing values.
If you do want to keep the rows with a missing delay, say so in the condition, with the function is.na(), which returns TRUE when a value is NA:
Read the second filter as “keep the flight if its delay is missing or it left on time or early.” Now filter() also returns 174. To count the missing values themselves, filter for them: filter(flights, is.na(dep_delay)) returns all 8,255 flights in 2013 with no departure delay recorded.
Note that you can’t test for NA with ==. dep_delay == NA is always NA, whatever dep_delay is, because R can’t tell whether something equals an unknown value. Always use is.na().
5.6 Practice: counting flights
Put it all together. These steps follow the in-class exercise.
Step 1: set up. In RStudio, use the Packages tab to install nycflights13 and tidyverse, if you haven’t yet. Start a new R Markdown file, and load both in the first chunk:
library(tidyverse)
library(nycflights13)You will see the tidyverse’s startup messages. Remember that they are not errors.
Step 2: count all the flights. The dplyr function count() counts the rows of a data frame, so pipe flights into it:
There were 336,776 flights from New York City’s airports in 2013.
Step 3: flights in June. Add a filter() before count() that keeps month 6. Remember that month is stored as a number, so write 6, not "June". Try it before you look at the answer:
flights |>
filter(month == 6) |>
count()There were 28,243 flights in June.
Step 4: June flights from LaGuardia. Now keep only the June flights that left from LaGuardia. LaGuardia’s airport code is "LGA", and it is text, so it needs quotation marks. You can add a second filter(), or put both conditions in one, joined with a comma or &:
All three of these return the same count, 8,596 flights. Pick the one you find easiest to read.
# Two filters, one after the other
flights |>
filter(month == 6) |>
filter(origin == "LGA") |>
count()
# One filter, with a comma (AND)
flights |>
filter(month == 6, origin == "LGA") |>
count()
# One filter, with & (AND)
flights |>
filter(month == 6 & origin == "LGA") |>
count()Read the first version as “take flights, and then keep June, and then keep LGA, and then count.” If you got a much bigger number, check for missing quotation marks around "LGA", or a single = where you need ==.
5.7 Check your understanding
Which tidyverse package provides
filter()andglimpse()?When you load the tidyverse, the message
dplyr::filter() masks stats::filter()means something has gone wrong.In
glimpse()output, what type is a column marked<chr>?Complete the code to keep the flights that left from JFK:
filter(flights, origin ___ "JFK")Inside
filter(), what does a comma between two conditions mean?Read
|>out loud as:filter()keeps the rows where the condition isNA.Which function tells you whether a value is missing?
filter() keeps only the rows where the condition is TRUE. To keep the missing ones, you have to ask for them with is.na().
5.8 Summary
- The tidyverse is a family of packages for data science that work together, organized around the workflow import, tidy, transform, visualize, model, and communicate. Load it with
library(tidyverse); the startup and conflict messages are normal. - dplyr gives each data-tidying job a verb. Every verb takes a data frame first and returns a data frame.
glimpse()shows a data frame’s size and every column’s name, type (<int>,<dbl>,<chr>,<dttm>), and first values. Look at every data frame before you work with it.filter()keeps the rows where a condition isTRUE. Use==to test equality, quotation marks around text, and commas or&for AND,|for OR.- The pipe
|>passes the result on its left into the first argument of the function on its right. Read it as “and then.” Inside a pipe chain, don’t repeat the data frame. NAmarks a missing value. Base R brackets return a row ofNAs when a condition isNA;filter()drops it. Check row counts, and useis.na()when you want to keep or find missing values.
5.9 Exercises
Use the flights data from nycflights13 for these exercises, and answer each with a pipe chain.
Glimpse. Run
glimpse(flights). Which columns hold times? Which hold distances? What does a negative value inarr_delaymean?One airport. How many flights left Newark (
"EWR") in 2013? How many left JFK? Which of the three airports was busiest?Christmas Day. How many flights left New York City on December 25, 2013? How many of them left more than an hour late?
Raleigh–Durham. The
destcolumn holds each flight’s destination airport. How many flights flew to Raleigh–Durham ("RDU") in 2013? How many of them have a missingdep_delay? What do you think happened to those flights?AND or OR? Predict which of these returns more rows, then check. Explain the difference in words.
flights |> filter(month == 6 & month == 7) |> count() flights |> filter(month == 6 | month == 7) |> count()Long delays. How many flights left more than two hours late? Write the same filter in base R with square brackets, as in the last chapter. Do the two methods agree? Why or why not?
Fix the code. Each chain below has one mistake. Find and fix it.
flights |> filter(origin = "LGA") |> count() flights |> filter(flights, month == 6) |> count() flights |> filter(origin == LGA) |> count() flights |> filter(dep_delay == NA) |> count()
5.10 Further reading
- R for Data Science (2nd ed.) (Wickham et al. 2023), free at https://r4ds.hadley.nz, is the course’s main R reference. The chapters “Data transformation” and “Workflow: code style” cover
filter(), the pipe, and the other dplyr verbs, using the same flights data. - The dplyr website, https://dplyr.tidyverse.org, has a reference page for every function, with examples.
- Posit’s cheatsheets, https://posit.co/resources/cheatsheets/, include Data transformation with dplyr. Print it and keep it next to your computer.
|>, which passes the result on its left as
the first argument of the function on its right. Read it as “and then.”
A value that is unknown, written NA (“not available”) in R.
Most calculations with NA give NA; test for it
with is.na().
In R, a named piece of memory that holds data, such as
months <- 12. In a dataset, a measured characteristic,
such as median income; each Census variable has a code.
A bundle of code, written by other users, that adds new functions to R.
Install it once with install.packages(), then load it in
each session with library().
An R package of data on every flight that left New York City’s three
airports in 2013, including the flights data frame.
The steps of a data analysis: import, tidy, then a loop of transform,
visualize, and model, and finally communicate.
Data arranged so that each variable is a column, each observation is a
row, and each value has its own cell.
One thing that was measured or recorded, such as a student, a county, or
a flight. In a data frame, each row is an observation.
A document format (.Rmd) that mixes formatted text with
chunks of R code. Knitting it runs the code and builds a finished
report.
The tidyverse package for transforming data frames: filtering rows,
choosing columns, adding new columns, joining tables, and summarizing.
The tidyverse package for making plots and maps by adding layers.
The tidyverse package for repeating a task over many inputs, such as
downloading data for several years.
A one-page summary of a package or tool. Posit publishes them for dplyr,
ggplot2, RStudio, and more.
Code that takes inputs (arguments), does a job, and returns an output,
such as mean(temps).
When two loaded packages have a function with the same name, the one
loaded later hides (masks) the other. Reach the hidden one with
package::function().
Combining two tables by matching rows on a shared key column, such as
attaching Census data to boundaries by GEOID with
left_join().
A dplyr function named for the job it does to a data frame, such as
filter(), select(), arrange(),
and mutate().
An input to a function, written inside its parentheses.
The integrated development environment (IDE) where you write and run R.
R does the computing; RStudio is the workspace around it.
The data type for whole numbers. R stores a typed number as numeric
unless you add an L, as in 21L. Shown as
<int> by glimpse().
The data type for numbers that can have a decimal part, such as 103.21.
Also called double, and shown as <dbl> by
glimpse().
The data type for text, also called strings. Character values always go
inside quotation marks, as in "Chapel Hill".
A block of R code inside an R Markdown file, starting with
```{r} and ending with three backticks.