Along with computer vision, natural language processing (NLP) is one of the great triumphs of modern machine learning. While ChatGPT is all the rage and large language models (LLMs) are drawing everyone’s attention, that doesn’t mean that the rest of the NLP field just goes away.
NLP endeavors to apply computation to human-generated language, whether that be the spoken word or text existing in places like Wikipedia. There are a number of ways in which this would be relevant to customer experience and service leaders, including:
- Using it to power customer-facing AI agents
- Creating question-answering systems
- Classifying sentiment from e.g., customer reviews
- Automatically transcribing client calls
Today, we’re going to briefly touch on what NLP is, but we’ll spend the bulk of our time discussing how textual training data can be preprocessed to get the most out of an NLP system. There are a few branches of NLP, like speech synthesis and text-to-speech recognition, which we’ll be omitting.
Armed with this context, you’ll be better prepared to evaluate using NLP in your business (though if you’re building customer-facing AI agents, you can also let the Quiq platform do the heavy lifting for you).
What is Natural Language Processing (NLP)?
In the past, we’ve jokingly referred to NLP as “doing computer stuff with words after you’ve tricked them into being math.” This is meant to be humorous, but it does capture the basic essence.
Remember, your computer doesn’t know what words are; all it does is move 1’s and 0’s around. A crucial step in most NLP applications, therefore, is creating a numerical representation out of the words in your training corpus.
There are many ways of doing this, but today, a popular method is using word vector embeddings. Also known simply as “embeddings”, these are vectors of real numbers. They come from a neural network or a statistical algorithm like word2vec and stand in for particular words.
The technical details of this process don’t concern us in this post, what’s important is that you end up with vectors that capture a remarkable amount of semantic meaning. Words with similar meanings also have similar vectors, for example, so you can do things like find synonyms for a word by finding vectors that are mathematically close to it.
These embeddings are the basic data structures used across most of NLP. They power sentiment analysis, topic modeling, and many other applications.
For most projects, it’s enough to use pre-existing word vector embeddings without going through the trouble of generating them yourself.
Are large language models natural language processing?
Large language models (LLMs) are a subset of natural language processing. Training an LLM draws on many of the same techniques and best practices as the rest of NLP, but NLP also addresses a wide variety of other language-based tasks.
Conversational AI is a great case in point. One way of building a conversational agent is by hooking your application up to an LLM like ChatGPT, but you can also do it with a rules-based approach, through grounded learning, or with an ensemble that weaves together several methods.
Data preprocessing for NLP
If you’ve ever sent a well-meaning text that was misinterpreted, you know that language is messy. For this reason, NLP places special demands on the data engineers and data scientists who must transform text in various ways before machine learning models can be trained on it. With higher data quality comes improved model performance.
In the next few sections, we’ll offer a fairly comprehensive overview of data preprocessing for NLP. This will not cover everything you might encounter in the course of preparing data for your NLP application, but it should be more than enough to get started.
Why is text data preprocessing important?
They say that data is the new oil, and just as you can’t put oil directly in your gas tank and expect your car to run, you can’t plow a bunch of garbled, poorly-formatted language data into your algorithms and expect magic to come out the other side.
But what, precisely, counts as text preprocessing will depend on your goals. You might choose to omit or include emojis, for example, depending on whether you’re training a model to summarize academic papers or write tweets for you.
That having been said, there are certain steps you can almost always expect to take, including standardizing the case of your language data, removing punctuation, white spaces, and stop words, segmenting and tokenizing, etc.
Top text preprocessing techniques to make unstructured text data usable
NLP preprocessing techniques are the steps used to clean and prepare raw text before it is analyzed by a Natural Language Processing model. Raw text data contains noise such as punctuation, inconsistent casing, spelling variations, and irrelevant information. Preprocessing transforms that text into a structured format that machines can understand, analyze and finally generate human language themselves.
Here are the most common NLP preprocessing steps and techniques.
1. Segmentation and tokenization
An NLP model is always trained on some consistent chunk of the full data. When ChatGPT was trained, for example, they didn’t put the entire internet in a big truck and back it up to a server farm, they used self-supervised learning.
Simplifying greatly, this means that the underlying algorithm would take, say, the first few sentences of a paragraph and then try to predict the remaining sentence on the basis of the text that came before. Over time it sees enough language to guess that “to be or not to be, that is ___ ________” ends with “the question.”
But how was ChatGPT shown the first three sentences? How does that process even work?
A big part of the answer is segmentation and tokenization.
With segmentation, we’re breaking a full corpus of training text – which might contain hundreds of books and millions of words – down into units like words or sentences.
This is far from trivial. In the English language, sentences end with a period, but words like “Mr.” and “etc.” also contain them. It can be a real challenge to divide text into sentences without also breaking “Mr. Smith is cooking the steak” into “Mr.” and “Smith is cooking the steak.”
Tokenization is a related process of breaking a corpus down into tokens. Tokens are sometimes described as words, but in truth, they can be words, short clusters of a few words, sub-words, or even individual characters.
This matters a lot to the training of your NLP model. You could train a generative language model to predict the next sentence based on the preceding sentences, the next word based on the preceding words, or the next character based on the preceding characters.
Regardless, in both segmentation and tokenization, you’re decomposing a whole bunch of text down into individual units that your algorithm can work with.
2. Lowercasing
Lowercasing is the text preprocessing technique of converting all text to lowercase before it is processed by an NLP model.
Human language is not consistent about capitalization. The same word may appear as “Apple,” “APPLE,” or “apple,” depending on whether it starts a sentence, refers to a company, or is simply written in a different style.
For an NLP model, these variations can create unnecessary complexity. If capitalization is left untouched, the model may treat each version as a completely different token. That means “Apple,” “apple,” and “APPLE” could all end up as separate entries in the vocabulary.
Lowercasing reduces this variation. Instead of learning three separate representations for “Apple,” “apple,” and “APPLE,” the model only needs to learn one.
There is a tradeoff here. In some cases, capitalization carries meaning. “Apple” might refer to the company, while “apple” refers to the fruit. If everything is converted to lowercase, that distinction disappears.
Because of that, some NLP systems keep capitalization intact when the task requires it, such as named entity recognition. But for many applications, especially those focused on general language patterns, lowercasing is a useful step that reduces noise and helps the model learn more efficiently.
3. Stop word removal
Stop word removal is the preprocessing technique of removing very common words that appear frequently in language but often contribute little meaning to the text.
Words such as “the,” “is,” “and,” “of,” and “in” appear extremely often in English. These are known as stop words.
Imagine a sentence like this:
“The product is available in the store and on the website.”
If the goal is to understand the main topic of the sentence, the most important words are probably “product,” “available,” “store,” and “website.” The rest mainly help the grammar of the sentence.
Removing stop words reduces noise in the dataset. If every document contains the same handful of extremely common words, those words do not help much in distinguishing one piece of text from another.
For some tasks, such as search engines or topic modeling, removing stop words helps models focus on the words that actually describe the subject of a document.
However, stop word removal is not always appropriate. In tasks such as sentiment analysis or conversational AI, even small words can carry meaning. The difference between “I like this” and “I do not like this” depends on a single word.
Because of that, whether stop words should be removed depends heavily on the goal of the NLP system.
4. Stemming
Stemming is the preprocessing technique of reducing words to a simplified root form by removing prefixes or suffixes.
Human language often expresses the same concept through multiple word forms. Words such as “run,” “running,” “runs,” and “ran” all refer to the same basic action, but they appear differently in text.
Without preprocessing, an NLP model may treat each of these forms as completely separate tokens.
Stemming attempts to solve this by trimming words down to a shared base or root form.
For example:
running → run
played → play
studies → studi
That final example of a base form or stem word shows an important limitation. The resulting word is not always a real dictionary term. Stemming relies on simple rules that remove common endings rather than a deep understanding of language, and it may not always lead to improving data quality.
Even with that limitation, stemming can be useful because it reduces vocabulary size and helps the model connect related words during training.
For applications such as search engines or document retrieval systems, this kind of simplification is often good enough.
5. Lemmatization
Lemmatization is the preprocessing technique of reducing words to their true base or dictionary form, known as the lemma.
Like stemming, lemmatization attempts to connect different word forms that share the same meaning. However, instead of simply trimming suffixes, it relies on vocabulary resources and grammatical analysis.
For example:
running → run
better → good
studies → study
Unlike stemming, the result is usually a valid word found in a dictionary.
To determine the correct lemma, the system often needs to understand the grammatical role of the word in a sentence. For instance, the word “saw” could be the past tense of “see,” or it could refer to a cutting tool. The correct interpretation depends on context.
Because this process requires linguistic knowledge and sometimes part-of-speech tagging, lemmatization is typically more computationally expensive than stemming.
However, it also produces cleaner and more accurate representations of language, which makes it useful in applications where preserving meaning is important.
6. Removing punctuation and special characters
Removing punctuation and special characters is the preprocessing technique of eliminating symbols such as commas, quotation marks, parentheses, and other non-alphabetic characters from text.
Natural text contains many formatting elements that help human readers understand structure or tone. Punctuation marks, emojis, and special symbols all play a role in written communication.
However, in many NLP tasks, these characters do not contribute much to the core meaning of the text.
For example:
“Hello!!! How are you?”
A preprocessing pipeline might convert this to something simpler:
“Hello how are you”
Removing punctuation helps standardize the input data and reduces noise in the training corpus.
That said, punctuation can sometimes carry useful signals. In sentiment text analysis, repeated exclamation marks may indicate excitement or emphasis.
Because of this, some NLP systems remove punctuation entirely, while others keep specific characters that might contain meaningful information.
The goal is always the same. Clean the text enough that the model can focus on meaningful patterns instead of being distracted by formatting variations.
7. Text normalization
Text normalization is the preprocessing technique of converting text into a consistent and standardized form before it is analyzed by an NLP model.
Natural language contains many variations that refer to the same thing. People use abbreviations, contractions, spelling variants, and informal expressions all the time. If these differences are left untouched, the model may treat them as unrelated tokens.
Normalization reduces this variation by converting different forms into a common representation.
For example:
don’t → do not
can’t → cannot
USA → United States
Normalization may also include spelling corrections, standardizing numbers, or expanding abbreviations.
Consider a dataset containing the words “color” and “colour.” Without normalization, the model treats them as separate tokens even though they represent the same concept.
By standardizing these variations, normalization makes the training data more consistent and easier for the model to learn from. Proper text preprocessing can mean eliminating misspelled words, but also deciding on which version of a spelling is correct for your use case.
The exact normalization rules depend heavily on the application. Informal chat messages, for example, may require normalization of slang and abbreviations that would never appear in formal documents. In those cases, preparing text data is crucial as it impacts data quality.
8. Removing numbers
Removing numbers is the preprocessing technique of eliminating numeric values from text when they do not contribute meaningful information to the task.
Many text datasets contain numbers that may not help the model understand the underlying meaning of the text.
For example:
“The product costs $49 and was released in 2024.”
If the goal is topic classification or general language modeling, the numbers themselves may not add much value. In such cases, they can simply be removed.
After preprocessing, the sentence might look like this:
“The product costs and was released in”
Of course, this technique must be used carefully. In some applications, numbers carry extremely important information. Financial analysis, medical data, and scientific documents often rely heavily on numerical values.
Because of this, many NLP pipelines only remove numbers when they are clearly irrelevant to the problem being solved.
The general idea is to simplify the dataset and reduce unnecessary variation in the vocabulary.
9. Part of speech tagging
Part-of-speech tagging (also called grammatical tagging) is the preprocessing technique of assigning grammatical labels to each word in a sentence.
In English, words can function as nouns, verbs, adjectives, adverbs, and other grammatical categories. Identifying these roles helps an NLP system understand how words relate to each other.
For example:
“The dog runs quickly.”
A part-of-speech tagger might label the words like this:
The → determiner
dog → noun
runs → verb
quickly → adverb
These tags give the model information about the structure of the sentence.
Part-of-speech tagging is often used as an intermediate step in more advanced NLP tasks. Named entity recognition, dependency parsing, and information extraction all rely on grammatical structure to interpret meaning.
Although modern deep learning models sometimes learn this structure automatically, explicit POS tagging is still widely used in traditional NLP pipelines.
10. Named entity recognition preprocessing
Named entity recognition, often abbreviated as NER, is the preprocessing technique of identifying and labeling specific real-world entities within text.
Human language frequently refers to people, organizations, locations, dates, and other identifiable entities. Recognizing these elements helps NLP solutions extract useful information from text.
For example:
“Apple released a new iPhone in California in 2023.”
An NER system might identify the entities as:
Apple → organization
iPhone → product
California → location
2023 → date
This allows the model to distinguish between general words and references to real-world objects or institutions.
Named entity recognition is widely used in applications such as news analysis, text classification, knowledge extraction, and search engines.
By identifying these entities early in the preprocessing pipeline, NLP systems can build richer representations of the information contained in text.
11. Noise removal
Noise removal is the text preprocessing technique of eliminating irrelevant or distracting elements from text that do not contribute to the meaning of the content.
Real-world text data rarely comes in a clean form. It may contain HTML tags, URLs, emojis, repeated characters, formatting artifacts, or other elements that are useful for humans but confusing for NLP models.
For example, a sentence taken from a webpage might look like this:
“Check out our new product!!! 👉 https://example.com <br> Limited time offer!!!”
Before an NLP model processes the text, a preprocessing pipeline might remove the URL, HTML tags, and extra punctuation so that the remaining text is easier to analyze.
After removing HTML tags and other noise, the sentence might look like this:
“Check out our new product limited time offer”
Removing this kind of noise helps reduce unnecessary variation in the dataset and makes it easier for the model to identify meaningful patterns in the language.
The exact definition of “noise” depends on the application. In social media posts, for example, emojis may actually carry useful sentiment information and might be preserved rather than removed because they contribute as much value as individual words.
The goal of noise removal is simply to eliminate elements that distract from the linguistic structure of the text.
12. Vectorization and feature extraction
Vectorization and feature extraction are text preprocessing techniques that convert text into numerical representations that machine learning models can process.
Computers cannot directly understand words or sentences. Instead, text must be translated into numbers that represent patterns in the language.
One of the simplest approaches is the bag of words model, where a document is represented by counting how often each word appears.
For example, consider two short sentences:
“I like coffee”
“I like tea”
A bag-of-words representation might convert these into numerical vectors based on the frequency of each word in the vocabulary.
Another widely used technique is TF IDF, which stands for term frequency inverse document frequency. Instead of simply counting words, TF IDF gives higher importance to words that appear frequently in a document but not across every document in the dataset.
More advanced NLP systems use word embeddings, which represent words as vectors in a high-dimensional space. In this space, words with similar meanings appear closer together.
For instance, the vectors representing “king” and “queen” would be closer to each other than the vectors for “king” and “table.”
These numerical representations allow machine learning models to analyze patterns, relationships, and meaning within large collections of text.
Vectorization is often the final step of text preprocessing before the text is fed into an NLP algorithm or neural network.
Supercharging your NLP applications
Natural language processing is an enormously powerful constellation of techniques that allow computers to do worthwhile work on textual data. It can be used to build question-answering systems, tutors, chatbots, and much more.
But to get the most out of it, you’ll need to preprocess the data. No matter how much computing you have access to, machine learning isn’t of much use with bad data. Techniques like removing stopwords, expanding contractions, and lemmatization create a corpora of text that can then be fed to NLP algorithms. Of course, there’s always an easier way. If you’d rather skip straight to the part where cutting-edge conversational AI directly adds value to your business, you can also reach out to see what the Quiq platform can do.
