Large Language Models (LLMs) like GPT-4 or Google’s Gemini or Meta LLama represent a major leap in how machines understand and generate human language. Unlike traditional machine learning or deep learning models, which are trained on labeled datasets for task-specific applications (like text classification or named entity recognition), LLMs are trained on vast amounts of unstructured text data from the internet, books, and other sources using an approach called self-supervised learning, where the model learns to predict the next word in a sentence. LLMs are based on transformer architecture and are able to scale to billions of parameters, enabling them to generalize across a wide range of language tasks without being explicitly trained for each one. Models with a significantly larger number of parameters than traditional machine learning models require proportionally large datasets to generalize well; otherwise, they are likely to overfit. This scale of data comes with its own challenges, such as managing, cleaning, and processing it into a format that is compatible for model pretraining.
Yes, you heard it right. Large datasets that are scraped from the Web and other sources bear noise with it. This noise includes:
These issues directly affect the quality and reliability of data used for LLM pretraining.
What we need to do is data cleaning. ML models are like garbage in - garbage out. It’s very important to have good quality and enough quantity of data for LLM pretraining, to get a well-performing model.
Normalization makes text comparison reliable by converting all strings to a consistent form. Normalization Form Compatibility Decomposed “NFKD” normalizes the strings and makes it easier for NLP models. Use the `ftfy` package for fixing text encoding issues
Example:
Café -> cafe
eg: “- - - - - -- - - -”,
“. . . . . . . . . .”,
“* * * * * * * * * *”
These can be removed using regex patterns
Eg: “iz/kku dk;kZy;] mRrj izns'k ty fuxe] 6&jk.kk izrki ekxZ y[kuA”
Removing this kind of text is a difficult task since it can come in a variety of forms and representations. Traditional techniques, such as heuristics or Markov chain might result in lower accuracy since they work on the word level.
The gibberish text does not possess any semantic information compared to normal text. Using a sentence transformer to capture the embeddings and project that into 2d space helps us understand the semantic difference between gibberish and normal text.
As you can see in the image, gibberish in red color stays far from the normal text, which is represented in blue color. The difference in semantic information helps to flag gibberish data in text corpus
A solution to this is `Nvidia Nemo curator Fuzzy deduplication` . NVIDIA NeMo Framework is a scalable and cloud-native generative AI framework built for researchers and PyTorch developers working on LLMs and other state-of-the-art networks. Nemo curator provides GPU accelerated data deduplication, which is crucial for faster run times.
How it works
The identified duplicates are removed from the dataset
Once the data is cleaned, the next step is tokenization, a core process in data preparation for LLM pretraining.
Tokenization is the process of breaking up text into smaller pieces, called tokens.
These tokens can be:
Why tokenization?
AI models like ChatGPT can’t "see" words. They only work with numbers. Each token gets mapped to a unique number (called an "embedding"), so the model can process it mathematically.
Instead of teaching a computer every possible sentence in English, we teach it to understand smaller pieces — tokens — and how they fit together.
Example:
“I like cats.” → ["I", "like", "cats", "."]→ [23, 147, 92, 4]
Latest LLMs usually use a special form of subword tokenization (like Byte-Pair Encoding or WordPiece). These allow the model to handle Rare words and new names
When an LLM processes text, it doesn't read an entire book at once. It only sees a limited amount of tokens at a time. This limited portion is called the Context window (or sequence length)
Think of it like how your eyes can only see a few words at a time when reading — not the whole page.
Since we have multiple documents with a lot of text, how can we efficiently create training samples of a given sequence length(number of tokens in a sample)? One approach is to concatenate all into a large text (set of tokens), then split it based on sequence length. This is straightforward, but it introduces a high number of truncations. Recently, a research paper was published on “Fewer truncations improve language modeling”. The paper proposes a data packing strategy - a scalable and efficient method that packs documents into training sequences through length-aware combinatorial optimization. This method completely eliminates unnecessary truncations while retaining the same training efficiency as concatenation. The idea is to eliminate unnecessary document truncations without sacrificing training efficiency. From our experiments, we have observed a 40% reduction in truncations.
Once the data is packed into a fixed sequence length (after padding for shorter sequences), this is split into a train and validation set with an 80/20 or 90/10 proportion. For faster throughput, data is often stored in Arrow format.
Finally, the dataset is ready for LLM pretraining.
Happy pretraining!!!
At KeyValue, we build systems that stay solid; deeply engineered, thoughtfully designed, and ready for the long run. Let's talk.
Data preparation is important because large web-scraped datasets may contain encoding errors, repeated characters, gibberish text, and duplicates that can mislead the model and waste compute. Cleaning, deduping, and properly organizing the data ensures the LLM trains on high-quality text, improving LLM’s efficiency and performance.
Pretraining teaches an LLM general language understanding using massive unlabeled text, helping it learn patterns, semantics, and reasoning from next-token prediction at scale.
Fine-tuning takes this pretrained model and adapts it to a specific task or domain using a much smaller, targeted dataset.
The major steps include Unicode normalization, ASCII filtering, repeated sequence removal, gibberish removal, and deduplication, followed by tokenization, sequence packing, and finally splitting the data into training and validation sets and storing it efficiently for LLM pretraining.
It uses MinHash + LSH + GPU-accelerated connected components to efficiently identify and remove exact and near-duplicate documents at large scale.
Tokenization converts raw text into numerical tokens that LLMs can process. It breaks text into consistent subword units, handles rare or complex words reliably, and ensures the model can learn language patterns effectively within its context window.
