Text Embeddings and Natural Language Processing

From Words to Vectors

Understanding how machines process and understand human language

Core Challenge: How do we convert human language into numerical representations that machines can understand and process?

"The quick brown fox" → [0.2, -0.1, 0.8, 0.3, ...]

Goal: Transform text into meaningful vector representations
Requirement: Preserve semantic meaning and relationships
Application: Enable mathematical operations on text

The Text Representation Problem

Why We Need Embeddings

Fundamental Issues:

Symbolic vs. Numerical: Computers work with numbers, not symbols
Semantic Similarity: "cat" and "dog" are more similar than "cat" and "car"
Context Dependency: "bank" has different meanings in different contexts
Dimensionality: Vocabulary sizes can be massive (100K+ words)

Traditional approach: "cat" = 1, "dog" = 2, "car" = 3
Problem: No notion of similarity between words

Solution: Map words/text to dense vector spaces where similarity has meaning

Bag of Words Representation

The Simplest Approach

Bag of Words: Represent text as a vector of word counts

Vocabulary: ["the", "cat", "sat", "on", "mat", "dog", "ran"]

"The cat sat on the mat" → [2, 1, 1, 1, 1, 0, 0]
"The dog ran" → [1, 0, 0, 0, 0, 1, 1]

Advantages:

Simple and interpretable
Easy to implement
Works well for some tasks

Disadvantages:

Loses word order information
High dimensionality (sparse vectors)
No semantic similarity
Frequent words dominate

TF-IDF: Weighted Word Importance

Term Frequency × Inverse Document Frequency

TF-IDF Formula: $$\text{TF-IDF}(t,d,D) = \text{TF}(t,d) \times \text{IDF}(t,D)$$ $$\text{TF}(t,d) = \frac{\text{count of } t \text{ in } d}{\text{total words in } d}$$ $$\text{IDF}(t,D) = \log\left(\frac{|D|}{|\{d \in D : t \in d\}|}\right)$$

Key Idea: Weight words by importance

High TF: Word appears frequently in document
High IDF: Word is rare across the corpus
High TF-IDF: Word is important to this specific document

"the" → Low IDF (appears everywhere)
"quantum" → High IDF (appears rarely)
"machine learning" → High TF in ML papers, High IDF overall

Word Embeddings

Dense Vector Representations

Key Insight: Map words to dense, low-dimensional vectors where semantic similarity is preserved

"king" → [0.2, -0.1, 0.8, 0.3, -0.5, ...]
"queen" → [0.3, -0.2, 0.7, 0.4, -0.4, ...]
"man" → [0.1, -0.3, 0.2, 0.8, -0.1, ...]
"woman" → [0.2, -0.4, 0.1, 0.9, -0.2, ...]

Properties of Good Embeddings:

Semantic Similarity: Similar words have similar vectors
Analogical Reasoning: king - man + woman ≈ queen
Dense Representation: Typically 50-300 dimensions
Learned from Data: Capture statistical patterns

Word2Vec

Learning Word Representations

Two Architectures:

Skip-gram: Predict context words from target word
CBOW: Predict target word from context words

Sentence: "The cat sat on the mat"

Skip-gram: "cat" → predict ["The", "sat"]
CBOW: ["The", "sat"] → predict "cat"

Training Objective: Maximize probability of observing context words $$P(w_{context} | w_{target}) = \frac{\exp(\mathbf{v}_{context} \cdot \mathbf{v}_{target})}{\sum_{w \in V} \exp(\mathbf{v}_w \cdot \mathbf{v}_{target})}$$

Key Innovations:

Hierarchical softmax for efficiency
Negative sampling
Subword information (FastText extension)

GloVe: Global Vectors

Combining Global and Local Statistics

Core Idea: Use global word co-occurrence statistics to learn embeddings

GloVe Objective: $$J = \sum_{i,j=1}^V f(X_{ij})(\mathbf{w}_i^T \mathbf{w}_j + b_i + b_j - \log X_{ij})^2$$

Where $X_{ij}$ is the co-occurrence count of words $i$ and $j$

Advantages over Word2Vec:

Uses global corpus statistics
More stable training
Better performance on word analogy tasks
Faster training on large corpora

Co-occurrence matrix captures how often words appear together:
X["ice", "steam"] = low (rarely co-occur)
X["ice", "cold"] = high (frequently co-occur)

Contextual Embeddings

Beyond Static Word Representations

Problem with Static Embeddings: One vector per word type

"I deposited money in the bank" (financial institution)
"I sat by the river bank" (side of river)

Static embedding: "bank" always gets the same vector

Solution: Generate different embeddings based on context

Key Models:

ELMo: Bidirectional LSTM language models
BERT: Bidirectional encoder with attention
GPT: Autoregressive transformer models

Context 1: "bank" → [0.1, 0.8, -0.2, ...] (financial)
Context 2: "bank" → [0.7, -0.1, 0.5, ...] (river)

BERT and Transformer Embeddings

Attention-Based Representations

BERT: Bidirectional Encoder Representations from Transformers

Key Innovations:

Bidirectional Context: Uses both left and right context
Self-Attention: Each word attends to all other words
Pre-training + Fine-tuning: Transfer learning approach
Multiple Layers: Different layers capture different information

BERT Training Tasks:

Masked Language Model: Predict masked words
Next Sentence Prediction: Predict if sentences are consecutive

Input: "The [MASK] sat on the mat"
BERT learns to predict: "cat", "dog", "mouse", etc.

Sentence and Document Embeddings

Beyond Word-Level Representations

Challenge: How to represent entire sentences or documents?

Approaches:

Averaging: Mean of word embeddings
Weighted Averaging: TF-IDF weighted mean
Sentence-BERT: Siamese networks for sentence similarity
Universal Sentence Encoder: Transformer-based sentence embeddings
Doc2Vec: Extension of Word2Vec to documents

"Machine learning is powerful" → [0.3, -0.1, 0.7, ...]
"AI can solve many problems" → [0.4, -0.2, 0.6, ...]
Similarity: cos(v1, v2) = 0.85 (high semantic similarity)

Applications: Semantic search, document clustering, paraphrase detection

Modern Language Model Embeddings

GPT, T5, and Beyond

Evolution: From task-specific to general-purpose embeddings

Modern Approaches:

GPT Embeddings: Use hidden states from large language models
Instruction-Tuned: Embeddings trained to follow instructions
Contrastive Learning: Learn by comparing positive/negative pairs
Multi-Modal: Joint text and image embeddings

OpenAI text-embedding-ada-002:

1536-dimensional vectors
Trained on diverse tasks
State-of-the-art performance

Input: "Find me documents about climate change"
Embedding: Captures intent, semantics, and domain

Applications of Text Embeddings

Real-World Use Cases

Core Applications:

Semantic Search: Find documents by meaning, not keywords
Recommendation Systems: Recommend similar articles/products
Text Classification: Categorize documents automatically
Sentiment Analysis: Understand emotional tone
Clustering: Group similar documents together
Question Answering: Match questions to relevant passages
Machine Translation: Cross-lingual embeddings
Chatbots: Understand user intent

Example: Semantic Search
Query: "How to bake bread?"
Matches: "Bread making tutorial", "Baking recipes", "Yeast fermentation"
(Even without exact keyword matches)

Choosing the Right Embedding

Trade-offs and Considerations

Key Factors:

Task Requirements: Word-level vs. sentence-level vs. document-level
Domain Specificity: General-purpose vs. domain-specific embeddings
Computational Resources: Model size and inference speed
Context Sensitivity: Static vs. contextual embeddings
Language Support: Monolingual vs. multilingual

Decision Framework:

Simple tasks: TF-IDF, Word2Vec, GloVe
Context matters: BERT, RoBERTa, ELECTRA
Sentence similarity: Sentence-BERT, Universal Sentence Encoder
General purpose: OpenAI embeddings, Cohere embeddings
Specialized domains: Fine-tune on domain data

Key Takeaways

Text Embeddings Landscape

Evolution: From sparse (BoW, TF-IDF) to dense (Word2Vec, BERT) representations
Context Matters: Static embeddings → Contextual embeddings
Scale Effects: Larger models and datasets generally perform better
Task Specificity: Choose embeddings based on your specific use case
Similarity Metrics: Cosine similarity is the standard for measuring semantic similarity
Dimensionality: More dimensions ≠ always better (50-1500 typical range)
Future Direction: Multi-modal, instruction-following, and domain-adaptive embeddings

Practical Advice:

Start with pre-trained embeddings
Fine-tune if you have domain-specific data
Evaluate on your specific task and metrics
Consider computational constraints

"Understanding text embeddings opens the door to modern NLP applications"