AI / LLM Basics Interview Questions
1. What is a Large Language Model (LLM)?
A Large Language Model is a neural network trained on massive amounts of text to predict the next word, or more precisely the next token, in a sequence. That simple prediction task, repeated across billions of examples, is enough to teach the model grammar, facts, reasoning patterns, and style. B...
2. What are Tokens in an LLM?
Tokens are the basic units of text an LLM actually reads and generates, they can be whole words, parts of words, or even single characters, depending on how common that piece of text is. A common word like "the" is usually a single token A rarer or longer word might be split into two or three tok...
3. What is Tokenization?
Tokenization is the process of converting raw text into the sequence of tokens a model can actually process, since a neural network can't work with plain text directly. A tokenizer breaks input text apart based on a fixed vocabulary it was built with The same tokenizer is used both to prepare inp...
4. What is Byte-Pair Encoding (BPE)?
Byte-Pair Encoding is a popular tokenization algorithm that builds a vocabulary by repeatedly merging the most frequently occurring pair of characters or character sequences in a training corpus. Starts with individual characters as the smallest units Iteratively merges the most common adjacent p...
5. What is the Vocabulary of an LLM?
An LLM's vocabulary is the complete, fixed set of tokens it was trained to recognize and generate, typically containing tens of thousands of entries. Built once during tokenizer training and then kept fixed for that model's entire lifetime Includes whole words, sub-word pieces, punctuation, and o...
6. What is an Embedding in the context of LLMs?
An embedding is a list of numbers, a vector, that represents a token's meaning in a way a neural network can do math with. Words with similar meanings end up with embeddings that are mathematically close to each other Generated by an embedding layer that's learned during training, not manually as...
7. Define the Transformer architecture?
The Transformer is the neural network architecture underlying virtually every modern LLM, introduced as a way to process sequences of text using attention instead of reading one word at a time in strict order. Built from stacked blocks, each containing an attention mechanism followed by a feed-fo...
8. What is the Attention Mechanism?
Attention is the mechanism that lets a model weigh how relevant each other word in the input is when processing a given word, rather than treating every word as equally important. Computes a relevance score between every pair of tokens in a sequence Uses those scores to build a weighted combinati...
9. What is Self-Attention?
Self-attention is attention applied within a single sequence, each token attends to every other token in that same input, including itself, rather than attending to a separate sequence. Every token generates a Query, Key, and Value vector from its embedding The Query of one token is compared agai...
10. What is Multi-Head Attention?
Multi-Head Attention runs several self-attention operations in parallel, each with its own learned parameters, then combines their results together. Each "head" can learn to focus on a different kind of relationship, one might track grammatical structure, another might track topical relevance Run...
11. What is Positional Encoding?
Positional Encoding is information added to each token's embedding to tell the model where that token sits in the sequence, since the attention mechanism on its own has no built-in sense of word order. Without it, a Transformer would treat "dog bites man" and "man bites dog" identically, since at...
12. What is the purpose of the Feed-Forward Network inside a Transformer block?
Each Transformer block contains a feed-forward network that processes every token's representation individually after the attention step has mixed in context from the rest of the sequence. Applies the same two-layer transformation to each token's vector independently Adds additional representatio...
13. What is Layer Normalization?
Layer Normalization is a technique used inside a Transformer to keep the scale of values flowing through the network stable as they pass through many stacked layers. Rescales the values within each layer's output to have a consistent mean and variance Helps training converge faster and more relia...
14. What is Causal Masking?
Causal masking is a technique used in decoder-style LLMs that prevents a token from attending to any tokens that come after it in the sequence. Ensures the model can only use information from earlier in the text when predicting the next token Necessary because generation happens one token at a ti...
15. Describe the role of Softmax in an LLM's output layer?
Softmax is the function that converts the model's raw output scores for every possible next token into a proper probability distribution that sums to one. Takes a raw score for every token in the vocabulary, some tens of thousands of numbers Converts those scores into probabilities, with higher r...
16. What is Cross-Entropy Loss?
Cross-Entropy Loss is the standard measurement used to train an LLM, quantifying how far off the model's predicted probability distribution was from the actual next token in the training data. Penalizes the model more heavily when it assigns low probability to the token that actually came next Av...
17. What is the purpose of a Loss Function during LLM training?
A loss function gives training a single number representing how wrong the model's predictions currently are, which is what the training process actually tries to reduce. Computed after each batch of training examples, comparing the model's predictions against the correct answers Used to calculate...
18. What is Pretraining in the context of LLMs?
Pretraining is the initial, large-scale training phase where a model learns general language patterns by predicting the next token across a massive, broad corpus of text. Typically uses text scraped from books, websites, code, and other large-scale sources Requires enormous computational resource...
19. What is Fine-Tuning?
Fine-tuning is the process of further training an already-pretrained model on a smaller, more specific dataset to adjust its behavior for a particular task or style. Starts from a pretrained base model rather than training from scratch Uses a much smaller, more targeted dataset than pretraining r...
20. What is Instruction Tuning?
Instruction tuning is a specific type of fine-tuning where a model is trained on examples of instructions paired with the responses that correctly follow them. Teaches a base model, which is only good at predicting plausible next text, to instead behave like a helpful assistant that follows a use...
21. Describe Reinforcement Learning from Human Feedback (RLHF)?
RLHF is a technique used to further align a model's behavior with human preferences, using human judgments as the training signal instead of a fixed dataset of correct answers. Human reviewers compare multiple model responses to the same prompt and rank which ones they prefer Those rankings are u...
22. What is Alignment in the context of LLMs?
Alignment refers to the broader effort of making a model's behavior match what humans actually want, helpful, honest, and safe responses, rather than just fluent or plausible-sounding text. Pretraining alone only teaches a model to predict likely next text, with no built-in sense of what's actual...
23. Define In-Context Learning?
In-context learning is a model's ability to pick up a new pattern or task just from examples given directly in the prompt, without any additional training or parameter updates. The model isn't actually "learning" in the traditional sense, its weights don't change Instead, it recognizes the patter...
24. Define Zero-Shot Learning?
Zero-shot learning is when a model performs a task correctly based purely on an instruction, with no examples of that task provided in the prompt. Relies entirely on the model's pretraining and instruction tuning to understand what's being asked Works well for common, well-understood tasks, like ...
25. Define Few-Shot Learning?
Few-shot learning is when a handful of example input-output pairs are included directly in the prompt to demonstrate exactly what kind of response is wanted, before asking the model to handle a new case. Typically uses somewhere between two and a handful of examples Helps the model understand a s...
26. What is Chain-of-Thought Prompting?
Chain-of-Thought prompting encourages a model to write out its intermediate reasoning steps before giving a final answer, rather than jumping straight to a conclusion. Can be triggered explicitly, by asking the model to "think step by step" Often noticeably improves accuracy on tasks involving mu...
27. What is Prompt Engineering?
Prompt engineering is the practice of carefully crafting the wording, structure, and examples in a prompt to get more accurate, relevant, or well-formatted responses from a model. Includes techniques like being explicit about the desired format, providing examples, and breaking complex requests i...
28. What is the purpose of a System Prompt?
A system prompt is a set of instructions given to a model before the actual conversation begins, establishing its role, tone, boundaries, or behavior for the rest of that session. Typically not shown to the end user, but still shapes every response the model gives afterward Used to set persistent...
29. What is a Context Window?
A context window is the maximum amount of text, measured in tokens, that a model can consider at once, covering the prompt, any conversation history, and the response it's generating. Once a conversation exceeds this limit, earlier content has to be dropped, summarized, or otherwise handled to ma...
30. What is the Max Tokens parameter?
Max Tokens is a setting that limits how many tokens a model is allowed to generate in a single response, capping the output length. Doesn't affect how much input the model can read, only how much it's allowed to write back Setting it too low can cut off a response mid-sentence before the model fi...
31. What are Stop Sequences?
Stop sequences are specific strings that, when generated, tell the model to immediately stop producing further output. Commonly used to prevent a model from continuing past a natural end point, like generating an extra, unwanted turn of a conversation Set by the developer calling the model, not s...
32. What is Temperature in LLM sampling?
Temperature is a setting that controls how random or predictable a model's word choices are when generating text. A low temperature, close to zero, makes the model consistently pick its highest-probability next token, producing more focused, repeatable output A higher temperature flattens the pro...
33. What are Top-k and Top-p Sampling?
Top-k and top-p are two related techniques that limit which tokens a model is allowed to consider when picking its next word, rather than sampling from the full vocabulary every time. Top-k sampling : restricts the choice to only the k highest-probability tokens Top-p sampling , also called nucle...
34. What is Hallucination in LLMs?
Hallucination is when a model generates text that sounds fluent and confident but is factually incorrect or entirely made up. Happens because a model is fundamentally predicting plausible next text, not verifying facts against a trusted source Can involve inventing citations, misremembering detai...
35. What is Perplexity in language modeling?
Perplexity is a metric that measures how well a language model predicts a given piece of text, essentially quantifying how "surprised" the model is by the actual next words. Lower perplexity means the model assigned higher probability to what actually happened next, indicating a better fit to tha...
36. What is Overfitting in the context of training an LLM?
Overfitting happens when a model learns its training data so closely that it starts memorizing specific examples rather than learning generalizable patterns. Shows up as strong performance on training data but noticeably worse performance on new, unseen data More of a risk during fine-tuning on a...
37. What are Parameters in an LLM?
Parameters are the internal numerical values, weights, that a neural network learns during training and uses to transform input into output. Modern LLMs commonly have anywhere from a few billion to hundreds of billions of parameters Every parameter is adjusted incrementally during training to red...
38. What is Quantization?
Quantization is a technique that reduces the numerical precision used to store a model's parameters, shrinking its memory footprint and speeding up inference. Converts weights from a high-precision format, like 32-bit floating point, down to a lower-precision one, like 8-bit or even 4-bit Reduces...
39. What is Model Distillation?
Model distillation is a technique for training a smaller "student" model to mimic the behavior of a larger "teacher" model, transferring much of the larger model's capability into a more efficient package. The student model is trained to match the teacher's output probabilities, not just its fina...
40. What is a Foundation Model?
A foundation model is a large, broadly pretrained model designed to serve as a general-purpose base that can be adapted to many different downstream tasks. Trained on a wide, diverse dataset rather than for one narrow purpose Meant to be fine-tuned, prompted, or otherwise adapted for a wide range...
41. What is a Decoder-Only model?
A Decoder-only model is a Transformer variant built entirely from decoder blocks, generating text one token at a time based only on the tokens that came before it. Uses causal masking so each token can only attend to earlier tokens, never later ones Well suited to open-ended text generation, sinc...
42. What is an Encoder-Only model?
An Encoder-only model processes an entire input sequence at once, with every token able to attend to every other token, including ones that come later in the sequence. No causal masking, since it's not generating text one token at a time Well suited to understanding tasks like classification, sen...
43. What is an Encoder-Decoder model?
An Encoder-Decoder model combines both halves of the original Transformer design, an encoder that processes the full input, and a decoder that generates output text based on that encoded understanding. The encoder builds a rich representation of the entire input sequence first The decoder then ge...
44. What is a Mixture of Experts (MoE) architecture?
Mixture of Experts is a model design where, instead of every input passing through the entire network, a routing mechanism sends each input to only a subset of specialized sub-networks, called experts. Only a fraction of the model's total parameters are actually used for any given input Lets a mo...
45. What is Retrieval-Augmented Generation (RAG)?
Retrieval-Augmented Generation combines a language model with an external retrieval step, pulling in relevant documents or data before generating a response, rather than relying purely on what the model memorized during training. A retrieval step searches an external knowledge source for content ...
46. How are Embeddings used beyond text generation?
Beyond powering the internal workings of an LLM, embeddings are widely used on their own as a tool for comparing the meaning of different pieces of text. Semantic search : finding documents whose meaning is similar to a query, not just ones sharing the same keywords Clustering : grouping similar ...
47. What are Guardrails in LLM applications?
Guardrails are the checks and constraints put around a model in a real application to keep its behavior within safe, expected, or on-topic bounds. Can filter or block certain categories of user input before it ever reaches the model Can also check the model's output before it's shown to a user, c...
48. What is Prompt Injection?
Prompt injection is when malicious or unexpected instructions are hidden inside content a model processes, tricking it into behaving differently than the actual user intended. Can be hidden inside a webpage, document, or file that a model reads as part of completing a task Exploits the fact that ...
