Text analysis with quanteda in R: a complete guide
Sixty US presidential addresses, nine UK immigration manifestos, and 150 years of political speeches ship inside quanteda as bundled datasets. This guide walks through how to turn any of those (or your own text) into a corpus, tokenise it into a tokens object, then count features into a sparse dfm (document-feature matrix) for reproducible text analysis in R. Every snippet is verified against quanteda 4.x and the official quickstart, so you can copy any block and adapt it to your own corpus.
Why quanteda works for text analysis in R
Three R packages dominate text analysis: tm (the original, around since 2007), tidytext (the tidyverse answer), and quanteda (a standalone object system released in 2014). quanteda earns its place when you want a single, consistent object model that scales from a handful of documents to millions without rewriting your code. The quanteda quickstart guide leads with the same idea: pick one mental model, then chain operations on it.
The model is just three objects:
| Object | Class | What it holds |
|---|---|---|
corpus | quanteda::corpus | Original texts plus per-document metadata |
tokens | quanteda::tokens | Tokenised lists, one per document |
dfm | dfm (sparse matrix) | Document-by-feature counts |
The discipline matters: a corpus is the source of truth you do not mutate, tokens is where cleaning happens, and the dfm is what you count and model against. Once that mental model clicks, every quanteda function slots into one of those three rows, and the rest of the package stops feeling like a memorisation exercise.
How do I install quanteda and its companion packages?
In quanteda v3.0 the package family split. The core quanteda package holds the object model, but textstat_* functions moved to quanteda.textstats, textplot_* functions moved to quanteda.textplots, and textmodel_* functions moved to quanteda.textmodels. If you load only quanteda and then call textstat_frequency() or textplot_wordcloud(), R will tell you the function does not exist, and the error message does not point at the missing package. Install everything you might need in one shot to avoid that round trip.
install.packages(c("quanteda", "quanteda.textstats", "quanteda.textplots"))
library(quanteda)
library(quanteda.textstats)
library(quanteda.textplots)Verify the install by loading data_corpus_inaugural, the 60-document corpus of US presidential addresses that ships with quanteda. If the object exists and prints without error, you are ready to go. A missing companion package is the single most common cause of could not find function errors on a fresh install.
How do I build a quanteda corpus in R?
corpus() is the entry point to the whole quanteda pipeline, and it accepts a character vector, a data frame with a text column, a tm::VCorpus, a kwic object, or another corpus. For a first run, the simplest input is a named character vector: the names become document ids, the values become the texts, and the result is an S3 object of class quanteda::corpus. The summary() method shows types, tokens, and sentences per document, which is the fastest way to confirm your corpus was built the way you expected. The corpus() reference documents every accepted source type if you need to read from a CSV, a SQLite database, or a custom iterator.
texts <- c(
doc1 = "The economy grew last quarter.",
doc2 = "Interest rates held steady in May.",
doc3 = "Inflation ticked up in June."
)
corp <- corpus(texts)
summary(corp)
#> Corpus consisting of 3 documents.
#>
#> Text Types Tokens Sentences
#> doc1 6 7 1
#> doc2 6 7 1
#> doc3 5 6 1Attach metadata with docvars
Per-document variables (year, author, party, source URL, language) flow through the entire pipeline, which is what makes quanteda feel like a database for text. Attach them as a data frame, and corpus_subset() will use them later for filtering and grouping. The data frame must have one row per document, in the same order as the corpus, and column types are preserved as long as you pass stringsAsFactors = FALSE for character columns.
docvars(corp) <- data.frame(
topic = c("economy", "rates", "economy"),
month = c(3, 5, 6),
stringsAsFactors = FALSE
)
corp <- corpus_subset(corp, topic == "economy")
summary(corp)corpus_subset() uses non-standard evaluation on docvars, so write topic == "economy" rather than "topic" == "economy". Quoted docvar names were deprecated in v3.0 and silently do the wrong thing on current versions. If your filter involves multiple conditions, combine them with & and |, the same way you would write a base R subset expression on a data frame.
How do I tokenise text with quanteda?
tokens() is deliberately conservative. With defaults it preserves URLs, @usernames, #hashtags, and kebab-case words, and it does not strip stopwords or punctuation. The what argument defaults to "word4" since quanteda 4.0, which is a smarter splitter than the older "word1" but still leaves the cleaning decisions to you. You decide what gets removed, and in what order. The tokens() reference lists every accepted what value, from "sentence" for sentence-level tokens to "character" for character n-grams.
toks <- tokens(corp)
print(toks)
#> Tokens consisting of 2 documents.
#> doc1 :
#> [1] "The" "economy" "grew" "last" "quarter" "."
#> doc2 :
#> [1] "Inflation" "ticked" "up" "in" "June" "."Pass what = "sentence" to get sentence-level tokens (useful for sentence-by-sentence analysis like sentiment scoring), remove_punct = TRUE to drop punctuation, or remove_url = TRUE to strip links. Set split_hyphens = TRUE to split self-funding into c("self", "funding") if your corpus has compound words you want to count separately. The default of split_hyphens = FALSE keeps the compound as a single feature, which is what you usually want for hashtag-like or product-name text where the hyphen carries meaning.
Count tokens and types
ntoken() and ntype() return named integer vectors and work on corpora, tokens objects, and dfms. ntoken() counts every token (so "The" and "the" are separate), while ntype() counts unique types (so "The" and "the" collapse only if you lowercased first). Both functions take a single argument and return a named vector with one entry per document, which makes them useful inside sapply() loops or piped into data.frame() for a quick summary table you can sort or plot.
ntoken(toks)
#> doc1 doc2
#> 7 6
ntype(toks)
#> doc1 doc2
#> 6 5If those numbers look off compared to a manual count of one of your documents, your tokeniser is doing something unexpected: usually a what = "word4" edge case, a remove_punct setting, or a multi-byte character that needs normalize = TRUE. Always sanity-check after a non-default tokens() call, and never trust a dfm you have not double-checked at the tokens layer first, because the dfm is just a count of whatever the tokens object contains.
How do I clean quanteda tokens?
Compose cleaning steps with the native pipe |>. The order matters: case-fold first, then strip stopwords, then stem. If you stem before removing stopwords, you risk folding "The" into "the" and then trying to match it against a stopword list that only knows "the". If you remove stopwords before lowercasing, you miss any stopwords that appear with a capital letter at the start of a sentence, which is most of them in real text.
toks_clean <- toks |>
tokens_tolower() |>
tokens_remove(stopwords("en")) |>
tokens_wordstem()
print(toks_clean)
#> Tokens consisting of 2 documents.
#> doc1 :
#> [1] "economi" "grew" "last" "quarter"
#> doc2 :
#> [1] "inflat" "tick" "up" "june"stopwords("en") comes from the stopwords package, a quanteda dependency. Swap the language code for "de", "fr", "es", "it", or any of the 50+ languages the stopwords package ships with. For multi-word stopword patterns, build a custom character vector and pass it to tokens_remove() directly, since stopwords() only returns single words. The tokens_select() function is the more general tool underneath, with tokens_remove() and tokens_keep() as shortcuts for the two most common selection patterns.
A warning on stemming: tokens_wordstem() is irreversible. university becomes univers, and you cannot recover the original. For counting that trade is fine, but store the unstemmed tokens object alongside the stemmed one if you need to quote text later, render snippets in a report, or run a downstream analysis that expects full word forms. Snowball (the stemming algorithm under the hood) is also conservative with certain endings: eater does not fold to eat even though you might expect it to, so test the output on a small sample before applying it to a large corpus.
How do I build a document-feature matrix in R?
dfm() counts tokens into a sparse matrix. As of quanteda 3.0 it accepts only tokens or dfm objects, not raw text or a corpus. Going corpus → dfm directly is gone, and the change is permanent. The v3.0 release notes list the removed convenience arguments, but the short version is: if you have a corpus, tokenise it before you count, and the dfm is what every later function (modelling, plotting, statistics) consumes.
dfm_mat <- dfm(toks_clean)
dfm_mat
#> Document-feature matrix of: 2 documents, 8 features (50.00% sparse).Trim rare features and drop stopwords that slipped through with dfm_trim() and dfm_remove(). These two functions look similar but do very different things: dfm_trim() operates on counts, dfm_remove() operates on labels. Pick the right one for the noise you are trying to filter out, and remember you can always run both in the same pipe.
dfm_mat <- dfm_mat |>
dfm_trim(min_termfreq = 2, min_docfreq = 1) |>
dfm_remove(stopwords("en"))
dfm_mat
#> Document-feature matrix of: 2 documents, 5 features (40.00% sparse).dfm_trim() is frequency-based: it drops terms that appear fewer than min_termfreq times overall or in fewer than min_docfreq documents. Use it for noise: rare typos, one-off mentions, scanner artefacts. dfm_remove() is label-based: you give it a character vector of words to drop, and it removes them regardless of frequency. Use it for known stopword lists, domain-specific jargon, or words you have decided not to count for substantive reasons. Mixing the two is fine: trim first to drop low-signal noise, then remove a custom stopword list to enforce a known vocabulary.
Frequencies and concordances on a dfm
topfeatures() is the fastest sanity check on a built dfm. It returns a named numeric vector of the most frequent features across the whole dfm, sorted in descending order by default. The names are the features themselves, and the values are the counts, which makes it useful for a one-line glance at what dominates a corpus before you do anything more elaborate.
topfeatures(dfm_mat, 5)
#> last quarter grew economi up
#> 1 1 1 1 1For grouped or comparative work, switch to textstat_frequency(). It returns a data frame with columns feature, frequency, rank, and docfreq, and an optional group column when you pass a groups argument. The data-frame shape means it pipes cleanly into dplyr for filtering or sorting and into ggplot2 for plotting, which is why most published quanteda analyses use textstat_frequency() rather than topfeatures(). The textstat_frequency() reference shows the exact column types if you are writing strict-typed code.
textstat_frequency(dfm_mat, n = 5)
#> feature frequency rank docfreq
#> 1 last 1 1 1
#> 2 quarter 1 1 1
#> 3 grew 1 1 1
#> 4 economi 1 1 1
#> 5 up 1 1 1To compare frequencies across documents or authors, group the dfm first with dfm_group() and weight with dfm_weight(scheme = "prop"). The grouping collapses documents into one row per group, and the proportional weighting normalises for document length so that a 5,000-word speech and a 500-word speech are comparable on the same scale.
Concordance with kwic
kwic() (keywords in context) shows the words that surround a target pattern. Wrap multi-word patterns in phrase(), because passing "united kingdom" as a literal string would match documents containing both words anywhere, while phrase("united kingdom") matches the contiguous sequence. The kwic() reference shows every option, but the two you will reach for most often are window (how many tokens on each side) and valuetype (glob versus regex).
kwic(toks, pattern = phrase("last quarter"), window = 2)
#> Keyword-in-context with 1 match.
#> [doc1, 3:4] "The economy grew" | last quarter | "."window = 2 gives two tokens on each side of the match. Pass valuetype = "regex" to switch to regular expressions, so "^econ.*" matches prefixes and "\b\w+ism\b" matches words ending in -ism. Coerce the result with as.data.frame() to get a plain data frame with columns docname, from, to, pre, keyword, and post, useful when you want to filter, sort, or write the matches out to CSV.
How do I make a word cloud from a quanteda dfm?
A dfm is one argument away from a word cloud. Drop rare terms and stopwords first, or the cloud is unreadable, since a thousand-token cloud with min_count = 1 is a smeared rectangle, not a visualisation. The example below uses data_corpus_inaugural and filters to presidents from 2000 onward (Bush, Obama, Trump, Biden), then trims anything appearing fewer than ten times. The set.seed() call makes the layout reproducible, which matters because textplot_wordcloud() places words randomly by default.
inaug_dfm <- data_corpus_inaugural |>
corpus_subset(Year > 2000) |>
tokens(remove_punct = TRUE) |>
tokens_remove(stopwords("en")) |>
dfm() |>
dfm_trim(min_termfreq = 10)
set.seed(1)
textplot_wordcloud(inaug_dfm, max_words = 50)For side-by-side comparison clouds, group the dfm first and weight it to relative frequencies. The comparison = TRUE flag then plots one cloud per group, which is the easiest way to eyeball vocabulary differences between authors or time periods without writing any extra layout code. textplot_wordcloud() lives in quanteda.textplots, not in quanteda itself.
inaug_grouped <- inaug_dfm |>
dfm_group(groups = President) |>
dfm_weight(scheme = "prop")
textplot_wordcloud(inaug_grouped, comparison = TRUE, max_words = 30)If textplot_wordcloud() returns “could not find function”, install and library() the quanteda.textplots package. The function is not re-exported from core quanteda, and forgetting the companion package is the single most common cause of this error message. The package also ships textplot_keyness(), textplot_xray(), and textplot_network() for related visualisations of the same dfm object.
What mistakes and gotchas should I know about quanteda?
A handful of traps bite the first time you reach for quanteda, and most of them come from version drift: quanteda 4.x behaves differently from quanteda 1.x or 2.x in ways that older tutorials do not mention. The list below is the subset that costs the most time per occurrence, ordered by how often each one shows up in Stack Overflow questions and GitHub issues. Run the sanity check below after a fresh install to confirm every common function is in scope before you start a real analysis, because a missing-function error halfway through a 10-line pipe is the worst place to discover an unmet dependency.
library(quanteda)
library(quanteda.textstats)
library(quanteda.textplots)
stopifnot(
exists("corpus"),
exists("tokens"),
exists("dfm"),
exists("topfeatures"),
exists("textstat_frequency"),
exists("textplot_wordcloud")
)- Skipping
tokens().dfm(corpus, ...)is gone in v3.0+. Always gocorpus→tokens→dfm. The v3.0 release notes list the removed convenience arguments, but the short version is: if you have a corpus, tokenise it before you count, and the same goes for any function downstream that expects a tokens or dfm input. - Forgetting the companion packages.
textplot_*is inquanteda.textplots,textstat_*is inquanteda.textstats,textmodel_*is inquanteda.textmodels. Loadingquantedaalone is not enough, and the error message (“could not find function”) does not point at the missing package, so check yourlibrary()calls first. - Quoting docvar names in
corpus_subset().corpus_subset(corp, "topic" == "x")silently does the wrong thing. Writetopic == "x". NSE on docvars was a late change in v3.0, and most older examples online still use the quoted form, which is why this one catches people migrating from older code. - Over-stemming.
university→universis fine for counting, awful for quoting or display. Keep the unstemmed tokens object around in a separate variable if you need the originals for any kind of human-readable output. - Stopword removal location.
tokens_remove(stopwords("en"))runs on the tokens object and preserves position (useful for ngrams and positional analysis).dfm_remove(dfm, stopwords("en"))runs on the dfm and is fine for bag-of-words work. Pick the level that matches your analysis: you cannot go back from dfm to tokens, so always remove positional stopwords at the tokens layer if you might want ngrams later. - Case sensitivity.
tokens_select()andkwic()are case-insensitive by default. Passcase_insensitive = FALSEif you need to keep proper nouns and code distinct, for example to tellRfromrorApplethe company fromapplethe fruit in a sentiment or topic model. - Native pipe
|>. quanteda 4.0 still re-exports%>%for back-compat, but|>is the right choice for new code on R 4.1+. It avoids themagrittrdependency and makes the data flow obvious in long pipelines, which matters when a pipe runs more than five steps.
Frequently asked questions
Do I need all three companion packages, or can I get by with quanteda alone?
If you only need corpora, tokens, dfms, and topfeatures(), the core quanteda package is enough. The moment you want textstat_frequency(), textstat_keyness(), or any textplot_* function, install the matching companion package. A safe default is to install all three (quanteda, quanteda.textstats, quanteda.textplots) on the same day you install R for text analysis, and add quanteda.textmodels if you plan to fit LDA or Naive Bayes models later.
Can I pass a corpus directly to dfm() in quanteda 4.x?
No. As of v3.0, dfm() accepts only tokens and dfm objects. The old shortcut dfm(corpus, remove_punct = TRUE, ...) was removed, and there is no warning: you get an error or, in some edge cases, an empty dfm. Always go corpus → tokens() → dfm().
What is the difference between tokens_remove() and dfm_remove()?
Both drop features, but at different pipeline stages. tokens_remove() runs on a tokens object before the dfm is built, which preserves token positions (so ngrams and concordances still work downstream). dfm_remove() runs on a built dfm and is faster but loses positional information. Use tokens_remove() unless you are working with an existing dfm and cannot rebuild it from tokens.
Should I always lowercase before counting?
For most bag-of-words work, yes. dfm() lowercases by default (tolower = TRUE), and tokens_tolower() makes the lowercasing explicit and reversible. Pass tolower = FALSE to dfm() only if case is meaningful: proper noun disambiguation, sentiment analysis on product names, or code-as-text tasks where the difference between R and r matters.
Does quanteda work with tidyverse verbs?
Yes, with one caveat. corpus, tokens, and dfm objects are not tibbles, so you cannot pipe them straight into dplyr::filter(). The workaround is to convert to a data frame first (as.data.frame(dfm_mat, docvarnames = TRUE) for a dfm, or as.data.frame(tokens)) and then use tidyverse verbs. textstat_frequency() already returns a data frame, so it pipes into dplyr and ggplot2 without conversion.
Conclusion
The quanteda text analysis workflow in R comes down to three objects and a handful of pipe-friendly verbs. Build a corpus from your source, tokenise and clean into a tokens object, then count into a sparse dfm. Everything from topfeatures() to textplot_wordcloud() to textmodel_lda() slots into that pipeline, which is why quanteda is the most-used standalone text analysis package on CRAN. Once you accept that each object has one job, the rest of the API stops feeling like a list of functions to memorise and starts feeling like a single coherent workflow for text analysis in R.
From here the natural next steps are sentiment scoring (see tidytext basics for a comparable approach), topic modelling with textmodel_lda() on the dfm you just built, or supervised classification with textmodel_nb() or textmodel_svm(). All three consume a dfm, so the pipeline you have here feeds straight into them without rebuilding the corpus from scratch.