Working with text data
Neural networks such as those used by LLMs cannot process raw data. So for achieve that we use a word embeddings. Word embeddings consist in convert different types of data into a vectors, we can represent any type of data like vectors images, text, audio, documents, etc. With these vector the models can do mathematical operations for train their neural networks.
One famous model is Word2Vec for text embeddings. Word2Vec trained neural network architecture to generate word embeddings by pre-dicting the context of a word given the target word or vice versa. Basically, the model can predict the meaning of the word, so for example if we introduce Alemania and Argentina and see in a two-dimensional space we see that these word are clustered because the model can understand that two words are countries.
Tokenizing text
This is a stage prior to the creation of the embeddings. What is done at this stage is to convert text inputs into tokens, a token can be a word, a letter or even a comma.
text = "Hello, world. Is this-- a test?"
result = re.split(r'([,.:;?_!"()\']|--|\s)', text)
result = [item.strip() for item in result if item.strip()]
print(result)
['Hello', ',', 'world', '.', 'Is', 'this', '--', 'a', 'test', '?']
Then we have to create the Ids tokens, it consists of assigning a unique integer to each unique token. This will be used to create a vocabulary and every time an input enters its tokens will be converted into its corresponding id in the vocabulary. This process we can call it encode and when we want to convert ids into text back we will call it decode.
vocab = {token:integer for integer, token in enumerate(result)}
class SimpleTokenizerV1:
def __init__(self, vocab):
self.str_to_int = vocab
self.int_to_str = {i:s for s, i in vocab.items()}
def encode(self, text):
preprocessed = re.split(r'([,.?_!"()\']|--|\s)', text)
preprocessed = [item.strip() for item in preprocessed if item.strip()]
ids = [self.str_to_int[item] for item in preprocessed]
return ids
def decode(self, ids):
text = " ".join([self.int_to_str[token] for token in ids])
text = re.sub(r'\s+([,.?!"()\'])', r'\1', text)
return text
tokenizer = SimpleTokenizerV1(vocab)
text = """"It's the last he painted, you know,"
Mrs. Gisburn said with pardonable pride."""
ids = tokenizer.encode(text)
print(ids)
print(tokenizer.decode(ids))
[1, 56, 2, 850, 988, 602, 533, 746, 5, 1126, 596, 5, 1, 67, 7, 38, 851, 1108, 754, 793, 7]
" It' s the last he painted, you know," Mrs. Gisburn said with pardonable pride.
Special Context Tokens
Sometimes in the training dataset we have words that doesn't exists in our vocabulary.
print(tokenizer.encode("Mayonesa"))
# KeyError: 'Mayonesa'
For solve that its common add a special context token in our vocabulary that are useful for the training of LLMs, like <|unk|> that represent a unknown words or <|endoftext|> that represent the end of the text.
all_tokens = sorted(list(set(preprocessed)))
all_tokens.extend(["<|endoftext|>", "<|unk|>"])
vocab = {token:integer for integer,token in enumerate(all_tokens)}
class SimpleTokenizerV2:
def __init__(self, vocab):
self.str_to_int = vocab
self.int_to_str = {i:s for s, i in vocab.items()}
def encode(self, text):
preprocessed = re.split(r'([,.?_!"()\']|--|\s)', text)
preprocessed = [item.strip() for item in preprocessed if item.strip()]
preprocessed = [item if item in self.str_to_int else "<|unk|>" for item in preprocessed]
ids = [self.str_to_int[s] for s in preprocessed]
return ids
def decode(self, ids):
text = " ".join([self.int_to_str[token] for token in ids])
text = re.sub(r'\s+([,.:;?!"()\'])', r'\1', text)
return text
These special tokens are important because helps to LLMs to understand for example when a text end and another totally unrelated begins. Other common special tokens:
- BOS (beginning of sequence)
- EOS (end of sequence)
- PAD (padding)