← Course Overview ML2: Advanced Machine Learning
← Back to Week 6

Week 6: From Autoencoders to Embeddings

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Evaluation

1. Bias Encoded in Language: Word Embeddings as a Mirror of Society

Researchers have demonstrated that standard word embeddings trained on web corpora consistently associate "nurse" more closely with "woman" and "engineer" more closely with "man." These biases are not introduced by the model designer — they are absorbed from the statistical regularities of human language in the training corpus. Debiasing techniques can reduce measured biases, but critics argue this is cosmetic rather than corrective.

  • Is it possible to train embeddings that are "neutral" with respect to protected attributes without also erasing real and meaningful linguistic associations? Where is the line?
  • If you deploy a resume-screening system using word embeddings and it systematically underranks candidates from certain groups, who bears responsibility — the embedding researchers, the model builders, or the deploying company?
  • How should the existence of measurable embedding bias affect the regulatory treatment of NLP systems used in high-stakes decisions like hiring, lending, or bail?

Try: Have students run a live nearest-neighbor query on a publicly available word embedding (e.g., GloVe) for profession words and record which gender-associated words appear — then compare their findings across different profession domains to surface patterns.

Analysis

2. Static vs. Contextual Embeddings: The Cost of Context

Word2Vec assigns a single fixed vector to each word regardless of context — "bank" has one representation whether it appears in a financial document or a nature essay. BERT produces a different vector for each occurrence of "bank" depending on its sentence context. This contextual sensitivity dramatically improves performance on many NLP benchmarks, but at a significant increase in computational cost and model complexity.

  • For which real-world NLP applications does the difference between static and contextual embeddings matter most? Are there applications where static embeddings are genuinely sufficient?
  • The computational cost of contextual models like BERT raises questions about environmental impact and access. Should resource-intensive models be deployed when lighter alternatives exist?
  • Static embeddings can be stored as a simple lookup table, making them highly interpretable and auditable. What does the shift to contextual embeddings mean for model transparency and accountability?

Try: Give students two sentences containing the same ambiguous word (e.g., "bank" or "pitcher") and ask them to write down which meaning they would expect a static embedding versus BERT to capture — then discuss how this connects to downstream task performance.

Synthesis/Creation

3. Multi-Modal Embeddings and the Collapse of Modality Boundaries

Multi-modal embedding systems like CLIP learn a joint space where images and text are embedded together. This enables capabilities like finding images matching a text description or generating images from text prompts. These same systems power image generation tools and also underlie reverse-image-search-style surveillance capabilities.

  • A joint embedding space that aligns "a photo of a protest" with corresponding images could be used by human rights organizations to document events — or by governments to identify participants. How should dual-use risks shape research and deployment decisions?
  • Multi-modal embeddings trained on web-scraped image-text pairs inherit all the labeling noise, cultural biases, and consent issues of the underlying data. How should these limitations be communicated to downstream users?
  • What would a responsible "model card" for a large multi-modal embedding model include? What disclosures would you consider mandatory?

Try: In small groups, have students draft the key sections of a model card for a hypothetical multi-modal embedding system — then compare cards across groups and discuss which disclosures were considered essential versus optional.

Application

4. Domain Adaptation: When General Embeddings Fail Specialists

A general-purpose word embedding trained on Wikipedia and news articles will likely represent "discharge" as related to electricity or leaving a job. In clinical NLP, "discharge" almost always means a patient leaving the hospital or a medical fluid. Without domain adaptation, general embeddings can systematically mislead clinical NLP models.

  • Should organizations deploying NLP in specialized domains (medicine, law, finance) be required to evaluate and adapt their embeddings before deployment? What would appropriate evaluation look like?
  • Domain-specific corpora (clinical notes, legal filings) are often private and sensitive. How does the scarcity of public domain-specific training data affect the quality and fairness of specialized embeddings?
  • Fine-tuning on domain-specific data can overwrite general semantic knowledge. How would you decide what to fine-tune versus what to freeze when adapting a large pre-trained model to a specialized domain?

Try: Give students a list of five ambiguous terms (e.g., discharge, lead, culture, base, order) and ask them to write both a general-domain and a clinical-domain interpretation for each — then discuss what a vector that conflates the two meanings would do to a downstream model.

Analysis

5. Semantic Search and the Future of Information Retrieval

Traditional keyword-based search retrieves documents containing the exact query terms. Embedding-based semantic search retrieves documents that are semantically similar to the query, even without keyword overlap. This enables queries like "treatments for inflammation" to surface documents about "anti-inflammatory therapies" — but it also means the system's notion of "relevance" is shaped entirely by what the embedding model learned to treat as similar.

  • Keyword search is transparent: you can see exactly why a document was retrieved. Semantic search is opaque: the similarity is computed in a high-dimensional space no human can inspect. What are the implications of this opacity for scientific literature search or legal discovery?
  • If an embedding model was trained primarily on English-language Western sources, how might its notion of semantic similarity disadvantage researchers writing in other languages or working in non-Western intellectual traditions?
  • As semantic search becomes the dominant mode of information retrieval, what incentives does it create for content creators who want their work to be discovered? Could this reshape the structure of academic or journalistic writing?

Try: Run the same query using a keyword search tool and an embedding-based semantic search tool on the same corpus and display the top-5 results from each — ask students to evaluate which set is more relevant and discuss what "relevant" means in this context.

Discussion Facilitation Tips

  • The bias topic (Discussion 1) tends to generate strong reactions — anchor the conversation in the technical mechanism (geometric proximity in vector space) before moving to normative questions. Students engage more productively when they can trace the bias to a specific, concrete computational process.
  • For the static vs. contextual embedding comparison, use a live demo if possible: query the same ambiguous word against a GloVe lookup and a BERT encoder and display the different nearest neighbors side by side. The contrast is immediately persuasive and makes the concept memorable.
  • When discussing domain adaptation, ask students to identify a field they know well and name three terms that would be systematically misrepresented by a general-purpose embedding. This personalizes the problem and surfaces domain expertise in the room.
  • Close the session by connecting the week's content back to the "what counts as similarity" question: every embedding model encodes a particular theory of what makes two things alike. Ask students to articulate whose theory it is and how that shapes who benefits from the system.