Hi my name is julia silge. I'm a data scientist and software engineer at RStudio.
and I'd like to thank the the organizers of the R user group in Korea
so much for having me. Today speak to you
I am so happy to be speaking specifically today about creating
features for machine learning from text data for a couple of reasons
Having a better understanding of what we do to take text data
and then to make it appropriate
as an input for machine learning algorithms has many benefits
both if you are directly getting ready
to train a model or if you're at the beginning of some text analysis project
or if you are trying to understand the behavior of a model that
you're interacting with some way which is something that we do in our work as data scientists
or in our in our daily lives more and more
so when we build models for text either supervised or unsupervised
we start with something
like this this is some example text data
that I'll use a couple of times during
this talk that describes some animals
I'm using some text data so
you know to me as an english
speaker looks familiar like
I am as someone who uses a human
language so I look at this and I can
read it I could speak it aloud and I understand
I can interpret it what it means
so this kind of data this sort of
natural language data is being generated
all the time in all kinds of languages
in all kinds of contexts so
whether you work in healthcare in tech in finance
basically any kind of organization this
sort of text data is being generated
by customers by clients by internal stakeholders
inside of a business by people taking surveys
via social media via business processes
and in all this natural language
there's information latent in
that text data that can be used to make better decisions
However, computers are not great at looking at this and doing
math on language as it's represented like this
and instead language has to be
dramatically transformed to some kind of
machine readable numeric representation
that looks more like this what I'm
showing here on the screen to be ready
for almost any kind of model
so I spent a fair amount of time working
on software for people to be able to do
exploratory data analysis, visualization, summarization
tasks like that with text data in a tidy
format where we have one observation per row
and I love using tidy data principles for text analysis
especially during those exploratory phases of an analysis
when it comes time to build a model
often what the underlying mathematical implementation really needs
is typically something like this which is a
way to this particular representation is called the document term matrix
so the exact representation may differ from
what I've shown here
what I have here is we're weighting
things by counts so each row in this matrix is a document
each column is a is a word.
A token and the numbers represent
counts how many times does each document
use each word you could weight it in a different way
using say TF-IDF instead of counts
or you might keep sequence information
if you're interested in building a deep learning model but basically
for all kinds of text modeling
from simpler models like Naive Bayes
models which work well for text
to word embeddings to really the most
state-of-the-art kind of work that's
happening today like transformers for text data
we have to heavily
feature engineer and process language to
get it to some kind of representation
that's suitable for machine learning algorithms
so I work on an open source framework in R
for modeling and machine learning that's called Tidymodels and the examples that
I'll be showing today use Tidymodels code
some of the specific goals of the Tidymodels project are to provide
a consistent flexible framework for real
world modeling practitioners people who are you know doing
that are dealing with real world data
those who are just starting out to
those who are very experienced in modeling and
the goal is to harmonize the heterogeneous
interfaces that exist within R and to encourage good statistical practice
I'm glad to get to show you some of what I work on
and build and how we apply it to text
modeling but a lot of what I will talk
about today isn't very specific to Tidymodels
or even to R. I know this is an R user
group but what we're going to talk about and focus on
is a little more conceptual and basic
how do we transform text into predictors for machine learning
I am excited though to talk about Tidymodels and Tidymodels if you have not
used it before is a meta package
in a similar way that the Tidyverse is
a meta package so if you've ever typed
library Tidyverse and then you've used
ggplot2 for visualization
dplyr for data manipulation
Tidymodels works in a similar way
there are different packages inside of it
that are used for different purposes
so the pre-processing or the feature
engineering is part of a broader model process
you know it that process starts really
with with exploratory data analysis
that helps us decide what kind of model
we will build and then it comes to
completion I think I would argue with
model evaluation when you
measure how well your model performed
Tidymodels as a piece of software is
made up of our packages each of which
has a specific focus like our sample
is for re-sampling data to be able to
create bootstrap resamples
cross-validation resamples all different
kinds of resamples you might want to use to
train and evaluate models
the tune package is for hyper parameter
tuning as you might guess from the name
one of these packages is for feature
engineering for a data preprocessing
feature engineering and it is the one
that is called recipes
so in Tidymodels we capture this idea
of data pre-processing and feature
engineering in the concept of a
pre-processing recipe that has steps so you choose
ingredients or variables
that you're going to use then you define the steps
that go into your recipe
then you prepare them using training
data and then you can apply that to any
data set like testing data or new data at prediction time
so the variables or ingredients that we
use in modeling come in all kinds of
shapes and sizes including text data
so some of the techniques and approaches
that we use for pre-processing text data
are the same um as for any other kind of data that
you might use like non-text data
numeric data categorical data some for
some of it is the same
but some of what you need to know to be
able to do a good job
in this process for text is different
and is specific to the nature of what
language data is like
and so I've written a book with my
co-author Emile Hvitfeldt on supervised
machine learning for text analysis and R
and fully the first third of the book
focuses on how we transform
the natural language that we have in
text data into features for modeling
the middle section is about how we use
these features in
simpler or more traditional machine
learning models like regularized
regression or support vector machines and
then the last third of the book
talks about how we use deep learning
models with text data so deep learning
models still require these kinds of
transformations from natural language
into features as input for these kinds of models but
deep learning models are often able to
inherently learn structure of features
from text in ways that those
more traditional or simpler machine
learning models are not
so this book is now complete and
available as of this month as of november
folks are getting their first paper
copies and also this book is available
in its entirety at smalltar.com
so if you're new to dealing with text
data understanding these
fundamental pre-processing approaches
for text will set you up for being able
to train effective models
if you're really experienced with text
data if you've dealt with it a lot
already you've probably noticed like we have
that the existing you know resources or literature whether
that's books or tutorials or blog posts
is quite sparse when it comes to
detailed thoughtful explorations of how
these pre-processing steps work
and how choices made in these feature
engineering steps impact our model output
so let's walk through
several of some of these like basic
feature engineering approaches and how
they work and what they do let's start out with
tokenization
so typically one of the first steps in
transfer information from natural
language to machine learning feature for
really any kind of text analysis
including exploratory data analysis
or building a model. Anything is tokenization
in tokenization we take an input some
string some character vector and some
kind of token type
some meaningful unit of text
we're interested in a word
and we split the input pieces into
tokens that correspond to the type
we're interested in
so most commonly the meaningful unit or
type of token that we want to split text
into units of is a word
so this might seem
straightforward or obvious but it turns
out it's difficult to clearly define
what a word is for many or even most languages
so many languages do not use white space
between words at all
which presents a challenge
for tokenization even languages that
do use white space like english and korean
often have particular examples that are ambiguous
like contractions in english like didn't
which should be
you know maybe more accurately
considered two words the way
particles are used in Korean
and how pronouns and negation words are
written in romance languages like
italian and french where they're stuck
together and really maybe they should be
considered two words
once you have figured out what you're
going to do and you make some choices
and you tokenize your text then it's on
its way to being able to be used
in exploratory data analysis or
unsupervised algorithms or as features
for predictive modeling which is what
we're talking about here and what these
results show here so these results are
from a regression model trained on
descriptions of media
from artwork in the Tate collection in the UK so
what we're predicting in what we are
predicting is when what year
was a piece of art created based on the
the medium that the artwork was created
with and the medium is described with a
little bit of text
so we see here that artwork created using graphite
watercolor and engraving was more likely
to be created earlier
that though that is more likely to come
from older art and artwork that is created using
photography screen point or sorry screen print
screen printing and and dung
and glitter are more likely to be
created later. there this is more likely
to come from contemporary art modern art
so the the way that we tokenize this text
you know we started with natural human
generated texts of people writing out
the descriptions of the the media
that these art pieces of art were created with
and the way we tokenized that natural
human generated text that we started
with has a big impact on what we learned
from it if we tokenized in a different way
we would have gotten different
results in terms of performance like how
accurately we were able to predict
predict the year and also in terms of
how we interpret the model like what is
it that we're able to learn from it
so this is one kind of tokenization to
the single word but we also
we all another way to tokenize instead
of breaking up into single words or
unigrams we can tokenize to n-grams
so an n-gram is a continuous sequence of
N items from a given sequence of texts
so this shows that same piece of little
bit of text i'm describing this animal
divided up into bi-grams or n-grams of
two tokens so notice how the words in
the bi-grams overlap so the word collard
appears in both of the first bigrams the collared
collared peccary peccary also
referred to so n-gram
tokenization slides along the text to
create overlapping sets of tokens
this shows tri-grams for the same thing
so using uni-grams one word is
faster and more efficient but we don't
capture information about word order
I'm using a higher value
two or three or even more keeps
more complex information about word
order and concepts
that are described in multi-word phrases
but the vector space of tokens
increases dramatically
that corresponds to a reduction in token
counts we don't count each token as very
many times and that means depending on
your particular data set
you might not be able to get good results
so combining different degrees of
n-grams can allow you to extract
different levels of detail from text so uni-grams
can tell you which individual words have
been used a lot of times
some of those words might be overlooked
in bi-gram or tri-gram crowns if they
don't co-appear with other words as often
this plot compares model performance for
a Lasso regression model predicting the
year of supreme court opinions the
United States supreme court opinions
with three different degrees of n-grams
what we're doing here is we are taking
the text of the writings of the United
States supreme court and we're predicting
when did it when was that text
written so can we predict how old a
piece of text is from the contents of the text
so holding the number of tokens
constant at a thousand using uni-grams alone
performs best for this corpus of
opinions from the United States supreme court
this is not always the case depending on
the kind of model you use the data set
itself we might see the best performance
combining uni-grams and bi-grams or maybe
some other option
in this case if we wanted to incorporate
some of that more complex information
that we have in the bi-grams and the
tri-grams we probably would need to
increase the number of
tokens in the model quite a bit
so keep in mind when you look at results
like these that identifying n-grams is
computationally expensive
this is especially compared to the
amount of like a model the improvement
in model performance that we often see like if we
if we see some you know modest
improvement by adding in bigrams it's
important to keep in mind how much
improvement we see relative to
how long it takes to
identify bi-grams and then train that
model so for example for this data set
of supreme court opinions where we held
the number of tokens constant so the
model training had the same number of tokens in it
using bi-grams plus uni-grams takes twice as long to train
to do the feature engineering and the
training than only uni-grams and adding
in tri-grams as well takes almost five
times as long as training on uni-grams
alone so this is a computationally
expensive thing to do
going in the other direction
we can tokenize to units smaller than
words so like these are what are called
character shingles so we take words
the collared peccary
and we can instead of looking at words
we can go down and look at sub word
information there's multiple different
ways to break words up into sub words
that are appropriate for machine learning
and often these kinds of approaches
or algorithms have the benefit of being
able to encode unknown or new words
at prediction time so when it when it's
time to make a prediction on new data
it's not it's not uncommon for there to
be new vocabulary words at that time and
if we didn't see them in the training
data you know what are we going to do
about those new words when we train
using subword information often we can
handle those new words if we saw the subword
in our training data set so
using this kind of subword information
is a way to incorporate morphological
sequences into our models of you know
various kinds of this is something that applies to
various languages not just english
so these results are for a
classification model with a data set of
very short texts it's just the names of
post offices in the United States so super short
and the goal of the model was to predict
the post office located in hawaii
in the middle of the pacific ocean or
it located in the rest of the united states
so I created features for the model that are
subwords of these post office names
and we end up learning that the names
that start with h and p or contain that ale
sub word are more likely to be in hawaii
and the sub words a and d and ri and ing are are
more likely to come from the post office
that are outside of hawaii
so this is an example of how we
tokenized differently and we're able to
learn something new we're able to learn something
different so in Tidymodels we collect all these
kinds of decisions about tokenization
and code that looks like this
so we start with a recipe that specifies what
variables or ingredients that we'll use
and then we define these preprocessing
steps so even at this first
and arguably you know simple and basic
step the choices that we make affect our
modeling results in a big way
the next pre-processing um step that
I want to talk about is stop words
so once we have split text
into tokens we often find that not
all words carry the same amount of
information if maybe any information at
all actually for a machine learning task
so common words that carry little or
perhaps no meaningful information are
called stopwords
so this is one of the stopword lists
that's available for
Korean so it's common advice and practice
to say hey just remove just remove
remove these stopwords for a lot of
natural language processing tasks
what I'm showing here
is the entirety of one of the shorter
english stopword lists that's used
really broadly so you know it's words like I
me my pronouns conjunctions and of the
and these are very common words
that are not considered super important
the decision though to just remove
stopwords is often more involved and
perhaps more fraught than what you'll
than what you'll find reflected in a lot
of resources that are out there
so almost all the time real world NLP
practitioners use pre-made stopword lists
so this plot visualizes
set intersections for three common
stopword lists in english
in what is called an upset plot
so the three lists are called the
snowball list smart and the iso list
so you can see the the lengths of the
list are represented by the length of
the bars and then we see the
intersections which words are in common
on these lists by the by the vertical
bars so the lengths of the list are quite different
and also notice they don't all contain
the same sets of words
the important thing to remember about
stopword lexicons is that
they are not created in some
neutral perfect setting but instead they are
they are context specific
they they can be biased both of these
things are true because they are lists
created from large data sets of language
so they reflect the characteristics of the data used in
their creation so this is
the ten words that are in the english
language smart lexicon but not in the
English snowball lexicon
so notice that they're all contractions
but that's not because the snowball
exchange doesn't include contractions
it has a lot of them also notice that it has
that she's is on this list and so that means that
that list has he's but it does not have
the list she's
so this is an example of that
The bias I mentioned that occurs because
these lists are created from large data
sets of text lexicon creators look at the most
frequent words in some big corpus of
language they make a cut off
and then some decisions about what to include or
exclude you know
based on the list that they
have and you end up here so because
in many large data sets of language
you have more representation of
men you end up with a
situation like this where a stopword
list will have he's but not she's
so many decisions when it comes to
modeling or analysis with language
we as practitioners have to decide
what is appropriate for our particular domain
it turns out this is even true when it
comes to picking a stopword list
so in Tidymodels we can implement a
pre-processing step like removing stopwords
by adding an additional step to our
recipe so first we specified what
variables we would use then we tokenized
the text and now we are removing
stopwords here using just the default
step since we are not passing in any
other arguments we could though
use a non-default step or even a custom
list if that was most appropriate to our domain
this plot compares the model performance
for predicting the year of
that same data set of
supreme court opinions with three
different stopword lexicons of different lengths
so the snowball lexicon contains the
smallest number of words and in this
case it results in the best performance
so removing fewer stopwords results in
the best performance here so this
specific result is not generalizable to
all data sets and contexts but the fact
that removing different sets of
stopwords can have noticeably different
effects on your model that is quite
transferable so the only way to know
what is the best thing to do is to try
several options and see so machine
learning in general.
this is an empirical field right like we
don't know we don't often have reasons a priori to
know what will be the best thing to do
and so typically we have to
try a different option to see what will
be the best thing all right. then the the third
pre-processing step that I want to talk about
for text is stemming
so when we deal with text often
documents contain different versions of
one base word often called a stem so what
if say for an english example
if we aren't interested in the difference
between animals plural and animal
singular and we want to treat them both together
so that idea is at the heart of stemming
so there's no one
right way or correct way to stem text so
this plot shows three approaches for
stemming in English
starting from hey let's just remove a final s
to more complex rules about plural
handling plural endings that middle
one it is called the s stemmer
it's a set of it's like a little set of rules
and that last one is the best known
one probably the best-known
implementation of stemming in English
called the Porter algorithm
so you can see here that Porter stemming
is the most different from the other two
in the top 20 words here from the data
set of animal descriptions that I've
been using we see how the word species
was treated differently animal predator
this sort of collection of words
live living life lives that was treated
differently so practitioners are typically
interested in stemming text data because
it buckets tokens together that we believe
belong together in in a way that
we understand that as human users
of language so we can use approaches like this
which are pretty like step-by-step
rules based this is typically called
stemming or and it's fairly
algorithmic in nature like
first do this then do this then do this
or you can use lemmatization
which is usually based on large dictionaries
of words and it incorporates like a
linguistic understanding of what words belong together
so most of the existing approaches for
this kind of task in Korean are
are limited lemmatizers based on these
dictionaries and that are trained
using large data sets of language
so this seems like it's going to be a helpful thing to do
when you hear about this you're like
oh yeah sounds good, sounds smart
especially because with text data we are typically
overwhelmed with features with numbers of tokens
this is typically the situation
when we're dealing with text data
so here we have these animal description data
and I made a matrix representation of it
like we would typically use in some
machine learning algorithm
and look how many features there are
16,000 almost 17,000 features
that's the number of features that
would be going into the model
look at the sparsity
98 percent sparse that's high very
sparse data so this is the sparsity of
the data that will go into the machine
learning algorithm to build our
supervised machine learning model
if we stem the words
if I use here an approach for stemming
we reduce the number of word features by
many thousands the sparsity unfortunately did not
change as much but we reduced the number
of features by a lot by bucketing those
words together that our stemming algorithm
belong together so you know
common sense says
reducing the number of words features
so dramatically is going to perform
improve the performance of our machine learning model
but that is that does assume that we
have not lost any important information
by by stemming and it turns out that stemming or
lemmatization can often be very helpful in some
contexts but the typical algorithms used for these
are somewhat aggressive
and they have been built to favor sensitivity
or recall or the true positive rate and
this is at the expense of the
specificity or the precision or the true
negative rate so in a supervised machine
learning context what this does is this
affects a model's positive predictive
value the precision or its ability to
to not incorrectly label true negatives
as positive I hope I got that right
so you know to make this more concrete
stemming can increase a model's ability
to find the positive examples
of say the animal descriptions that are
associated with say a certain diet if
that's what we're modeling however if
text is over stemmed
the resulting model loses its ability to
label the negative examples
say the descriptions that are not about
that diet that's what we're looking for
and this can be a real challenge when
training models with text data kind of
finding that that balance there because
often we don't have a
dial that we can change on these
stemming on these stemming algorithms
so even just very basic pre-processing
for text like what I'm showing here in
this feature engineering recipe can be
computationally expensive
and the choices that a practitioner
makes like whether or not to remove
stopwords or to stem text can have dramatic
impact on how machine learning models
of all kinds perform whether those are
simpler models
more traditional machine learning models
or deep learning models what this means is that
the price the prioritization that we
as practitioners give to like learning
teaching and writing about feature
engineering steps for text really
contributes to better more robust
statistical practice in our field
I mentioned before the sparsity of text
data and I want to come back to that
because it is one of text data's really
defining characteristics because of just how language works
we use a few words a lot of times
and then a lot of words only just a
couple of times only a few a few times
and with a real set of natural language
you end up with relationships that look
like this that look like these plots in
terms of how the sparsity changes as you
add more documents
and more unique words to a corpus
so the sparsity goes up real fast as you
add more unique words and the memory
that is required to handle
this set of documents goes up very fast
so even if you use specialized data
structures meant to store sparse data like sparse
matrices you still end up growing the
memory required to handle these data
sets in a very non-linear way it still grows up very
fast so this means it can take a very
long time to train your model or even that
you outgrow the memory
available on your machine you have to go
to the cloud to an expensive
big memory situation this can be a real challenge
and this challenge
it is what has behind the motivating of vector
languages for models so
linguists have worked for a long time
on vector languages for models that can
reduce the number of dimensions representing text data
based on how people use language
so this quote here goes all the way back to 1957.
so the idea here is that we use
like the data is very sparse
but we don't use words
randomly it's not independent the words
are not used independently of each other
but rather there's relationships that
exist between how words are used together
and we can use those relationships to create
to transform our sparse high dimensional
space into a special dense
low dimensional space lower
we still has like 100
dimensions but much lower than the many thousands
hundreds tens hundreds of thousands
of space so the idea here we use
statistical modeling maybe just
word counts plus matrix factorization
maybe fancier math that involves neural
networks to take this really high
dimensional space and we create a new
lower dimensional lower dimensional
space that is special because the new
space is created based on vectors that
incorporate information
about which words are used together so
you shall know a word by the company it keeps
so you need a big data set of text to
create or learn these kinds of word
vectors or word embeddings
so this table that I'm showing right now
it's from a set of embeddings that
I created using a data set or a corpus of complaints
complaints to the United States consumer
financial protection bureau
so this is a government body in the
United States where people can complain and say
what is wrong with something to do with
a financial product like a credit card
a mortgage a student loan
something to do with like a financial
product they're like something went
wrong with my credit card something went
wrong with my mortgage that company is
not being fair so you come and you complain to it
so I took all those complaints and built
it's our high dimensional space and
build a low dimensional space
and we can look in that space and understand
what words are related to each other
in this space so in the new space
defined by the embeddings the word month
is closest to words like year months plural
monthly installments payment so these are words
that are that makes sense in the context of
financial products like credit cards or mortgages
in the new space defined by these embeddings
the word error is closest to the words
like mistake clerical like a clerical mistake
problem glitch or there was a glitch on my
mortgage statement so we see these kinds of
or miscommunication misunderstanding you
know like these are these are words that
are used in similar ways so
you don't have to create embeddings yourself
because it requires quite a lot of data
to make them so you can use word
embeddings that are pre-trained
i.e created by someone else
based on some huge corpus of data that
they have access to and you probably don't
so let's look at one of those data sets
let's look at this table shows the results for the same word error
but for the glove embeddings so the
glove embeddings are a set of
pre-trained embeddings that are created
based on a very large data set that's like
all of wikipedia all of the google news data set
just like huge swaths of the internet have been
fed in to create these embeddings
so some of the closest words here are similar
to those that are before but we no
longer have some of that domain specific
flavor like clerical discrepancy
and now we have like
miscommunication you know but and now we
have calculation and probability
which people were not talking about with
their financial product complaints
so this really highlights
how these how these work here before
we we created our own and we were able
to learn relationships that were
specific to this context and here we go
to a more general set that that
was learned somewhere else
so embeddings are trained or learned
from a large corpus of text data and the
characteristics of that corpus become
part of the embeddings
so machine learning in general you know
is exquisitely sensitive to whatever it
is that's in your training data and this
is never more obvious than when
dealing with text data
and perhaps with word embeddings is
just like one of these classic examples
where this is true it turns out that
this shows up in how any human
prejudice or bias in the corpus
becomes imprinted into the embeddings
so in fact when we look at some of these
most commonly available embeddings that
are out there bias is we we see that
african-american first names that are
more common for african americans in the
United States they're associated with
more unpleasant feelings than European
American first names in these embedding spaces
women's first names are more associated
with family and men's first names are more associated with career
and terms associated with women are more
associated with the arts and terms
associated with men are more associated
with science so it turns out actually bias is so
ingrained in word embeddings that the
word embeddings themselves can be used
to quantify change
in social attitudes over time
so word embeddings are
maybe an exaggerated or extreme example
but it turns out that all the feature
engineering decisions that we make when
it comes to text data have a significant
effect on our results
both in terms of the model performance that we see
and also in terms of how appropriate or
fair our models are
so given all that when it comes to
pre-processing your text data
creating these features that you need
you have a lot of options and quite a
bit of responsibility so my advice is
always start with simpler models that
you can understand quite deeply
be sure to adopt good statistical
practices as you train and tune your
models so you aren't fooled about model
performance improvements
when you try different approaches
and also to use model explainability
tools and frameworks so you can
understand any less straightforward
models that you try
so my co-workers and I have written
about all of these topics and how to
use them with Tidymodels if that's what
you like to use and we will continue to do so
with that i will say
thank you so very much and I want to be
sure to again
thank the organizers of the R user group
in Korea I want to thank my teammates on
the Tidymodels team at Rstudio as
well as my co-author EMIL HVITFELDT.