Working with APIs using httr2
The httr2 package is the modern successor to httr, providing a pipe-friendly interface for making HTTP requests in R. This guide covers everything you need to consume APIs effectively, from simple GET requests to complex OAuth 2.0 authentication flows.
Why httr2?
The httr2 package offers several advantages over its predecessor:
- Pipe-friendly syntax: Build requests step-by-step using the pipe operator
- Better error handling: Granular control over how errors are managed
- Improved retry logic: Built-in support for handling transient failures
- Request throttling: Native rate limiting support
- Modern authentication: Full OAuth 2.0 support
Installation
Install httr2 from CRAN:
install.packages("httr2")
library(httr2)
httr2 is designed for R 4.1 or later, leveraging the native pipe operator. The package works independently of the tidyverse but integrates seamlessly with it.
Making Your First Request
The fundamental workflow involves creating a request, modifying it, and executing it:
library(httr2)
# Create a request object
req <- request("https://httpbin.org/get")
# Execute the request
resp <- req |> req_perform()
# Check the response status
resp |> resp_status()
# [1] 200
# Extract the response body
resp |> resp_body_string()
Each function in the chain returns the modified object, enabling readable request construction.
HTTP Methods
GET Requests
GET requests retrieve data. Use req_url_query() to add query parameters:
# Simple GET request
resp <- request("https://httpbin.org/get") |>
req_perform()
# GET with query parameters
resp <- request("https://httpbin.org/get") |>
req_url_query(search = "R programming", page = 1, limit = 10) |>
req_perform()
# Parse JSON response
data <- resp |> resp_body_json()
Query parameters are automatically URL-encoded. Chain multiple parameters as needed.
POST Requests
POST requests send data to create or update resources:
# POST with JSON body
resp <- request("https://httpbin.org/post") |>
req_method("POST") |>
req_body_json(list(
username = "datafan",
action = "create",
priority = 1
)) |>
req_perform()
# Parse the response
result <- resp |> resp_body_json()
Other HTTP Verbs
# PUT - complete resource replacement
resp <- request("https://api.example.com/users/123") |>
req_method("PUT") |>
req_body_json(list(name = "Updated Name", active = TRUE)) |>
req_perform()
# DELETE - remove a resource
resp <- request("https://api.example.com/users/123") |>
req_method("DELETE") |>
req_perform()
# PATCH - partial update
resp <- request("https://api.example.com/users/123") |>
req_method("PATCH") |>
req_body_json(list(name = "New Name")) |>
req_perform()
Working with Headers
Custom headers are essential for authentication and content negotiation:
# Add multiple headers
req <- request("https://api.github.com/user") |>
req_headers(
Authorization = paste("Bearer", Sys.getenv("GITHUB_TOKEN")),
Accept = "application/vnd.github.v3+json",
`User-Agent` = "MyRApp/1.0",
`Accept-Language` = "en-US"
)
resp <- req |> req_perform()
Common header types include Authorization tokens, Accept for content negotiation, User-Agent for application identification, and custom API keys.
Authentication Methods
Bearer Tokens
Bearer tokens are the standard for modern REST APIs:
# Simple bearer token
req <- request("https://api.example.com/data") |>
req_auth_bearer("your-access-token")
# Using environment variable for security
req <- request("https://api.example.com/data") |>
req_auth_bearer(Sys.getenv("API_TOKEN"))
Never hardcode tokens in your scripts. Use environment variables or secure credential storage.
Basic Authentication
Basic auth encodes credentials in Base64:
req <- request("https://api.example.com/login") |>
req_auth_basic("username", Sys.getenv("API_PASSWORD"))
OAuth 2.0
httr2 provides comprehensive OAuth 2.0 support:
# Define OAuth client
oauth <- oauth_client(
id = Sys.getenv("OAUTH_CLIENT_ID"),
token_url = "https://oauth.provider.com/token",
secret = Sys.getenv("OAUTH_CLIENT_SECRET")
)
# Authorization code flow
token <- oauth |>
oauth_flow_auth_code(
auth_url = "https://oauth.provider.com/authorize",
redirect_uri = "http://localhost:1410"
)
# Use token in requests
resp <- request("https://api.provider.com/data") |>
req_auth_bearer(token$access_token) |>
req_perform()
httr2 automatically handles token refresh when the OAuth provider supports it.
Error Handling
Custom Error Messages
# Throw custom error on HTTP failure
resp <- request("https://httpbin.org/status/404") |>
req_error(is_error = ~ paste("HTTP error:", .x$status_code)) |>
req_perform()
# Error: HTTP error: 404
Graceful Error Handling
# Create a safe version of req_perform
safe_perform <- possibly(req_perform, otherwise = NULL)
# Attempt request without throwing
resp <- request("https://httpbin.org/status/500") |>
req_error(is_error = \(resp) FALSE) |>
safe_perform()
if (is.null(resp)) {
cat("Request failed - handling gracefully\n")
} else {
cat("Request succeeded:", resp_status(resp), "\n")
}
Checking Response Status
# Verify response is successful
resp <- request("https://httpbin.org/get") |> req_perform()
if (resp |> resp_status_is_success()) {
data <- resp |> resp_body_json()
} else {
warning("Unexpected status: ", resp |> resp_status())
}
Parsing Responses
JSON Responses
# Parse JSON to R list
resp <- request("https://httpbin.org/json") |> req_perform()
data <- resp |> resp_body_json()
# Access nested data
data$slideshow$author
# Parse to tibble when response is an array
library(tibble)
df <- resp |> resp_body_json() |> enframe()
XML Responses
library(xml2)
resp <- request("https://httpbin.org/xml") |> req_perform()
doc <- resp |> resp_body_xml()
# XPath queries
titles <- doc |> xml_find_all(".//title")
xml_text(titles)
Binary Files
# Download binary file
request("https://httpbin.org/bytes/1024") |>
req_save("download.bin")
# Download with progress bar
request("https://httpbin.org/bytes/1024000") |>
req_save("large_download.bin", progress = TRUE)
Rate Limiting
Protect yourself from API rate limits:
# Throttle requests to 1 per second
req <- request("https://api.example.com/items") |>
req_throttle(1)
# Paginate through results with throttling
all_items <- list()
for (page in 1:10) {
resp <- request("https://api.example.com/items") |>
req_url_query(page = page, per_page = 100) |>
req_throttle(1) |>
req_perform()
items <- resp |> resp_body_json()
all_items <- c(all_items, items)
cat("Fetched page", page, "-", length(items), "items\n")
}
Retry Logic
Handle transient failures automatically:
# Retry on server errors (5xx)
resp <- request("https://httpbin.org/delay/2") |>
req_retry(
max_tries = 3,
max_seconds = 15,
is_transient = ~ .x$status_code >= 500
) |>
req_perform()
# Custom transient detection
resp <- request("https://api.example.com/data") |>
req_retry(
max_tries = 5,
backoff = \(n) 2^n, # Exponential backoff: 2, 4, 8, 16, 32 seconds
is_transient = \(resp) {
resp$status_code >= 500 ||
resp$status_code == 429 || # Rate limited
grepl("timeout", resp$body, ignore.case = TRUE)
}
) |>
req_perform()
Practical Example: GitHub API
library(httr2)
library(tibble)
# Fetch user information
github_user <- function(username) {
request(paste0("https://api.github.com/users/", username)) |>
req_headers(
Accept = "application/vnd.github.v3+json",
`User-Agent` = "R-httr2-demo/1.0"
) |>
req_perform() |>
resp_body_json()
}
# Fetch user repositories
github_repos <- function(username, max_pages = 3) {
all_repos <- list()
for (page in 1:max_pages) {
resp <- request(paste0("https://api.github.com/users/", username, "/repos")) |>
req_headers(
Accept = "application/vnd.github.v3+json",
`User-Agent` = "R-httr2-demo/1.0"
) |>
req_url_query(page = page, per_page = 100, sort = "updated") |>
req_throttle(1) |>
req_perform()
repos <- resp |> resp_body_json()
all_repos <- c(all_repos, repos)
if (length(repos) < 100) break
}
tibble(
name = sapply(all_repos, \(x) x$name),
description = sapply(all_repos, \(x) x$description),
stars = sapply(all_repos, \(x) x$stargazers_count),
forks = sapply(all_repos, \(x) x$forks_count),
language = sapply(all_repos, \(x) x$language),
updated = sapply(all_repos, \(x) x$updated_at)
)
}
# Example usage
user <- github_user("r-lib")
cat("Organization:", user$login, "\n")
cat("Public repos:", user$public_repos, "\n")
repos <- github_repos("r-lib", 2)
repos |> arrange(desc(stars)) |> print(n = 10)
Best Practices
- Never hardcode credentials - Use environment variables or config files
- Handle errors gracefully - Don’t let API failures crash your scripts
- Implement retries - Network failures happen; be resilient
- Respect rate limits - Use
req_throttle()to avoid getting blocked - Log requests - Use
req_log()for debugging - Parse incrementally - For large responses, process in chunks
See Also
- httr2 official documentation — Full package reference
rvest— Web scraping with CSS selectorsjsonlite— JSON handling in Rcurl— Low-level HTTP interface