Skip to main content

5  Into the Tidyverse: glimpse, filter, and the Pipe

NoteLearning objectives

By the end of this chapter you should be able to:

  1. Explain what the tidyverse is, and load it.
  2. Look inside a data frame with glimpse().
  3. Keep the rows you need with filter(), using one or more conditions.
  4. Chain steps together with the pipe, |>.
  5. Explain how filter() and base R brackets treat missing values differently.

In the last chapter you selected rows of a data frame with square brackets, as in grades[grades$exam2 > 90, ]. That works, but it gets long and hard to read quickly: the data frame’s name appears again for every condition, and every step needs a new variable. This chapter introduces the tidyverse, a set of packages designed to make that kind of work shorter and easier to read, and three tools from it that you will use in almost every assignment from here on: glimpse() to see what is in a data frame, filter() to keep the rows you need, and the pipe, |>, to chain steps together.

The examples use real data: every flight that left New York City’s three airports in 2013, from the nycflights13 package. By the end of the chapter you will be able to answer questions such as “how many flights left LaGuardia in June?” in four short lines.

5.1 What is the tidyverse?

5.1.1 The data science workflow

The tidyverse is a family of R packages built for data science. Its packages share a design philosophy, a grammar, and a way of storing data, so they fit together. They are organized around the workflow in Figure 5.1, from the book R for Data Science (Wickham et al. 2023):

flowchart LR
  A["Import"] --> B["Tidy"]
  B --> C
  subgraph C["Understand"]
    direction LR
    T["Transform"] --> V["Visualize"]
    V --> M["Model"]
    M --> T
  end
  C --> D["Communicate"]
Figure 5.1: The data science workflow, after Wickham, Çetinkaya-Rundel, and Grolemund (2023). Data are imported and tidied, then transformed, visualized, and modeled, often several times over, before the results are communicated. This chapter is about tidying and transforming.

First you import data into R. Then you tidy it into a consistent shape, with one variable per column and one observation per row. Then you go round a loop of transforming it (keeping the rows you need, making new columns, summarizing), visualizing it, and modeling it, until you understand it. Finally you communicate what you found, usually in an R Markdown report. This chapter lives in the Tidy and Transform boxes: getting data into shape and pulling out the rows you need.

5.1.2 The core packages

One command, install.packages("tidyverse") (or one trip to the Packages tab), installs the core tidyverse packages (Table 5.1).

Table 5.1: The core tidyverse packages.
Package What it is for
dplyr Transforming data frames: keeping rows, choosing columns, making new columns, summarizing
ggplot2 Plots and charts
tidyr Reshaping data into a tidy form
readr Reading files such as CSVs, with read_csv()
tibble The tidyverse’s version of the data frame
stringr Working with text
forcats Working with categories (factors)
lubridate Working with dates and times
purrr Repeating a task over many inputs

This chapter uses dplyr, which supplies glimpse() and filter(). Posit publishes a one-page cheatsheet for dplyr, Data transformation with dplyr, linked on the course Canvas page and at https://posit.co/resources/cheatsheets/. filter() is on the front page under Manipulate Cases (Extract Cases). It is worth keeping open while you work.

5.1.3 Loading the tidyverse

Load the tidyverse at the top of every R Markdown file, the same way as any other package:

library(tidyverse)

The first time you do, R prints a block of messages like this (your version numbers may differ):

── Attaching core tidyverse packages ──────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.5
✔ forcats   1.0.0     ✔ stringr   1.5.1
✔ ggplot2   3.5.1     ✔ tibble    3.2.1
✔ lubridate 1.9.3     ✔ tidyr     1.3.1
✔ purrr     1.0.2
── Conflicts ────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

These messages are normal. They are not errors, even though some are printed in red. The first block lists the packages that were loaded. The second, Conflicts, says that two packages contain a function with the same name, and which one R will use. Here, dplyr’s filter() masks the filter() in base R’s stats package (a function for time series). From now on, when you type filter(), you get dplyr’s version, which is the one you want. If you ever need the other one, you can name its package explicitly with two colons: stats::filter().

5.1.4 Tidying data is a set of verbs

What do we actually do when we tidy data? We select, extract, arrange, sort, combine, group, join, and summarize. dplyr gives each of these jobs its own function, named with a verb (Table 5.2).

Table 5.2: The main dplyr verbs.
dplyr verb What it does Covered
filter() Keeps the rows that meet a condition This chapter
select() Picks columns Next chapter
arrange() Sorts rows Next chapter
mutate() Makes new columns, or changes existing ones Next chapter
group_by() and summarize() Collapse groups of rows into summary values Later
left_join() and friends Combine two tables Later in the course

Every dplyr verb works the same way: its first argument is a data frame, the arguments after it say what to do, and it returns a new data frame. That consistency is what makes the pipe, later in this chapter, work so well.

5.2 Seeing inside a data frame with glimpse()

5.2.1 The flights data

The nycflights13 package contains a data frame called flights: one row for every flight that departed from New York City’s three airports (Newark, EWR; John F. Kennedy, JFK; and LaGuardia, LGA) in 2013. You might use it to compare flights by day, by airport, or by how late they left. Install it once from the Packages tab, then load it along with the tidyverse:

library(tidyverse)
library(nycflights13)

The live code boxes in this chapter load dplyr, the part of the tidyverse this chapter uses, rather than the whole tidyverse, because it loads faster in a web browser. Everything works the same way in RStudio with library(tidyverse). The first time you run a box, it takes a little while to download the flights data.

5.2.2 glimpse()

flights has more than 300,000 rows, far too many to print. Typing flights shows only the first few rows and as many columns as fit across the screen. glimpse() is the tidyverse’s way to see everything that is in a data frame at once. It turns the data on its side: each column becomes one line of output, showing the column’s name, its type, and its first few values. Run it:

The first two lines give the size: 336,776 rows and 19 columns. Below them is one line per column. The type of each column is in angle brackets, abbreviated (Table 5.3).

Table 5.3: Column types as glimpse() shows them.
In glimpse() Type Example columns in flights
<int> integer (whole numbers) year, month, day, dep_time
<dbl> numeric (“double”), may have decimals dep_delay, arr_delay, distance
<chr> character (text) carrier, origin, dest, tailnum
<dttm> a date and time time_hour

The columns this chapter uses are month and day (numbers, so June is 6, not "June"), origin (the three-letter code of the airport the flight left from), and dep_delay, the departure delay in minutes. A negative dep_delay means the flight left early.

glimpse() is one of several ways to look at a data frame. head(flights) prints the first six rows; str(flights), from the last chapter, is base R’s version of glimpse(); and in RStudio, View(flights) opens the whole table in a spreadsheet-like tab. Get in the habit of looking at every data frame before you work with it. Most surprises (a number stored as text, a column you didn’t expect, far fewer rows than you thought) show up in a glimpse.

5.3 Keeping rows with filter()

5.3.1 One condition

filter() keeps the rows of a data frame that meet a condition. Its first argument is the data frame, and the arguments after that are one or more logical conditions:

filter(data, condition)

It returns only the rows where the condition is TRUE. Here are the flights that left on the first day of a month:

flights is the data frame, and day == 1 is the logical condition. Notice that, inside filter(), you refer to the columns by name alone, day rather than flights$day: filter() knows to look for them in the data frame. The result has 11,036 rows, the flights on the 1st of each of the twelve months.

The condition uses ==, which tests whether two things are equal. A single = sets an argument, and writing filter(flights, day = 1) is one of the most common mistakes in R. dplyr catches it and tells you, with an error that says you have used = instead of == and asks “Did you mean day == 1?”

filter() doesn’t change flights; it returns a new data frame. To keep the result, store it with <-, and give it a new name, so that you still have the original:

5.3.2 Several conditions

Give filter() more than one condition, separated by commas, and it keeps the rows where all of them are TRUE. A comma inside filter() means AND, exactly like &. These two lines both find the flights on January 1:

Both return 842 rows. For OR, use |. This finds the flights in June or July:

Conditions on text work the same way, but the text needs quotation marks, as always: origin == "LGA", not origin == LGA. Without the quotes, R looks for a column or variable named LGA and doesn’t find one.

5.4 The pipe

5.4.1 Chaining steps with |>

Most questions take several steps. Suppose you want to know how many flights left LaGuardia early, or on time, on January 1, 2013. That takes three filters and a count. You could save a new data frame after every step, or nest the functions inside one another, but both get hard to read.

You met the pipe, |>, in an earlier chapter. It takes the output of one function and feeds it in as the first argument of the next. Because every dplyr verb takes a data frame as its first argument and returns one, verbs can be chained with pipes as long as you like:

Read the pipe out loud as “and then”: take flights, and then keep January 1, and then keep LaGuardia, and then keep the flights that left on time or early (a delay of zero or less), and then count the rows. The answer is 172 flights. Each step narrows the rows further, and the code reads in the same order as the question.

5.4.2 Don’t repeat the data frame

With the pipe, the data frame goes in front of the pipe, not inside filter(). These two lines do exactly the same thing:

filter(flights, day == 1)
flights |> filter(day == 1)

In the second, |> passes flights into filter() as its first argument. So inside a pipe chain you never write filter(flights, ...). If you do, flights is passed in twice, once by the pipe and once by you, and R reports an error.

TipThis is not a pipe

René Magritte’s 1929 painting The Treachery of Images shows a pipe above the words Ceci n’est pas une pipe, “This is not a pipe.” It isn’t: it’s a picture of one. In the same spirit, |> isn’t really a pipe either. It’s an operator, a piece of R grammar that rewrites x |> f(y) as f(x, y) before the code runs. The plumbing metaphor is just a good way to picture it: data flow in at one end and out at the other.

You may also see %>% in older code and online answers. It is the original pipe, from the magrittr package (named after Magritte), and does almost the same job. The native pipe |> was added to R itself in version 4.1, and it is the one this course requires.

5.5 Missing values: why the counts don’t match

5.5.1 The same question, two ways

Here is the January 1 LaGuardia question again, answered first with base R brackets, as in the last chapter, and then the tidy way with filter() and the pipe:

The base R way needs more typing, and more places for a typo: flights$ for every condition, and a new object at every step. The tidy way reads top to bottom. But look at the answers. Base R finds 174 flights, and filter() finds 172. They don’t match. Why?

5.5.2 NA: a missing value

The answer is missing values. In R, a value that is unknown or missing is written NA (“not available”). Flights that were cancelled never departed, so their departure delay is unknown: dep_delay is NA. Two of the January 1 LaGuardia flights were cancelled.

What is NA <= 0? R can’t say whether an unknown number is less than zero, so the answer is itself unknown: NA, not TRUE or FALSE. Each approach handles that differently (Table 5.4):

  • Base R brackets don’t drop a row when the condition is NA. They return a row in which every value is NA, so the result has two extra rows of missing values, 174 in all.
  • filter() keeps only the rows where the condition is TRUE. It quietly drops rows where the condition is NA, so it finds 172.
Table 5.4: How base R brackets and filter() treat each result of a condition.
Condition is TRUE Condition is FALSE Condition is NA
Base R df[condition, ] Row kept Row dropped A row of NAs is returned
filter(df, condition) Row kept Row dropped Row dropped

You can see the two extra rows by looking at the last rows of the base R result:

5.5.3 Deciding what happens to NAs

Neither approach is wrong, but they give different answers, and nothing warns you. That is annoying, and it is exactly the kind of quiet difference that leads to wrong results in a report. The lesson is to check the number of rows after every filter, and to decide on purpose what should happen to missing values.

If you do want to keep the rows with a missing delay, say so in the condition, with the function is.na(), which returns TRUE when a value is NA:

Read the second filter as “keep the flight if its delay is missing or it left on time or early.” Now filter() also returns 174. To count the missing values themselves, filter for them: filter(flights, is.na(dep_delay)) returns all 8,255 flights in 2013 with no departure delay recorded.

Note that you can’t test for NA with ==. dep_delay == NA is always NA, whatever dep_delay is, because R can’t tell whether something equals an unknown value. Always use is.na().

5.6 Practice: counting flights

Put it all together. These steps follow the in-class exercise.

Step 1: set up. In RStudio, use the Packages tab to install nycflights13 and tidyverse, if you haven’t yet. Start a new R Markdown file, and load both in the first chunk:

library(tidyverse)
library(nycflights13)

You will see the tidyverse’s startup messages. Remember that they are not errors.

Step 2: count all the flights. The dplyr function count() counts the rows of a data frame, so pipe flights into it:

There were 336,776 flights from New York City’s airports in 2013.

Step 3: flights in June. Add a filter() before count() that keeps month 6. Remember that month is stored as a number, so write 6, not "June". Try it before you look at the answer:

flights |>
  filter(month == 6) |>
  count()

There were 28,243 flights in June.

Step 4: June flights from LaGuardia. Now keep only the June flights that left from LaGuardia. LaGuardia’s airport code is "LGA", and it is text, so it needs quotation marks. You can add a second filter(), or put both conditions in one, joined with a comma or &:

All three of these return the same count, 8,596 flights. Pick the one you find easiest to read.

# Two filters, one after the other
flights |>
  filter(month == 6) |>
  filter(origin == "LGA") |>
  count()

# One filter, with a comma (AND)
flights |>
  filter(month == 6, origin == "LGA") |>
  count()

# One filter, with & (AND)
flights |>
  filter(month == 6 & origin == "LGA") |>
  count()

Read the first version as “take flights, and then keep June, and then keep LGA, and then count.” If you got a much bigger number, check for missing quotation marks around "LGA", or a single = where you need ==.

5.7 Check your understanding

  1. Which tidyverse package provides filter() and glimpse()?

  2. When you load the tidyverse, the message dplyr::filter() masks stats::filter() means something has gone wrong.

  3. In glimpse() output, what type is a column marked <chr>?

  4. Complete the code to keep the flights that left from JFK: filter(flights, origin ___ "JFK")

  5. Inside filter(), what does a comma between two conditions mean?

  6. Read |> out loud as:

  7. filter() keeps the rows where the condition is NA.

  8. Which function tells you whether a value is missing?

filter() keeps only the rows where the condition is TRUE. To keep the missing ones, you have to ask for them with is.na().

5.8 Summary

  • The tidyverse is a family of packages for data science that work together, organized around the workflow import, tidy, transform, visualize, model, and communicate. Load it with library(tidyverse); the startup and conflict messages are normal.
  • dplyr gives each data-tidying job a verb. Every verb takes a data frame first and returns a data frame.
  • glimpse() shows a data frame’s size and every column’s name, type (<int>, <dbl>, <chr>, <dttm>), and first values. Look at every data frame before you work with it.
  • filter() keeps the rows where a condition is TRUE. Use == to test equality, quotation marks around text, and commas or & for AND, | for OR.
  • The pipe |> passes the result on its left into the first argument of the function on its right. Read it as “and then.” Inside a pipe chain, don’t repeat the data frame.
  • NA marks a missing value. Base R brackets return a row of NAs when a condition is NA; filter() drops it. Check row counts, and use is.na() when you want to keep or find missing values.

5.9 Exercises

Use the flights data from nycflights13 for these exercises, and answer each with a pipe chain.

  1. Glimpse. Run glimpse(flights). Which columns hold times? Which hold distances? What does a negative value in arr_delay mean?

  2. One airport. How many flights left Newark ("EWR") in 2013? How many left JFK? Which of the three airports was busiest?

  3. Christmas Day. How many flights left New York City on December 25, 2013? How many of them left more than an hour late?

  4. Raleigh–Durham. The dest column holds each flight’s destination airport. How many flights flew to Raleigh–Durham ("RDU") in 2013? How many of them have a missing dep_delay? What do you think happened to those flights?

  5. AND or OR? Predict which of these returns more rows, then check. Explain the difference in words.

    flights |> filter(month == 6 & month == 7) |> count()
    flights |> filter(month == 6 | month == 7) |> count()
  6. Long delays. How many flights left more than two hours late? Write the same filter in base R with square brackets, as in the last chapter. Do the two methods agree? Why or why not?

  7. Fix the code. Each chain below has one mistake. Find and fix it.

    flights |> filter(origin = "LGA") |> count()
    flights |> filter(flights, month == 6) |> count()
    flights |> filter(origin == LGA) |> count()
    flights |> filter(dep_delay == NA) |> count()

5.10 Further reading

Wickham, Hadley, Mine Çetinkaya-Rundel, and Garrett Grolemund. 2023. R for Data Science: Import, Tidy, Transform, Visualize, and Model Data. 2nd ed. O’Reilly Media. https://r4ds.hadley.nz.