AI / LLM Basics Interview Questions

A Large Language Model is a neural network trained on massive amounts of text to predict the next word, or more precisely the next token, in a sequence. That simple prediction task, repeated across billions of examples, is enough to teach the model grammar, facts, reasoning patterns, and style.

• Built on the Transformer architecture, which processes text using an attention mechanism rather than reading word by word
• "Large" refers to both the size of the training data and the number of parameters, often in the billions
• Can be adapted after initial training through fine-tuning for specific tasks or behaviors

Examples include models like GPT, Claude, and Llama, all of which share this same core next-token-prediction foundation despite differing in size, training data, and fine-tuning approach.

Tokens are the basic units of text an LLM actually reads and generates, they can be whole words, parts of words, or even single characters, depending on how common that piece of text is.

• A common word like "the" is usually a single token
• A rarer or longer word might be split into two or three tokens, like "unbelievable" becoming "un", "believ", "able"
• Numbers, punctuation, and even spaces can each count as their own tokens

Everything an LLM does, reading a prompt, generating a response, counting against a context window, is measured in tokens rather than words or characters.

Tokenization is the process of converting raw text into the sequence of tokens a model can actually process, since a neural network can't work with plain text directly.

• A tokenizer breaks input text apart based on a fixed vocabulary it was built with
• The same tokenizer is used both to prepare input for the model and to decode the model's output tokens back into readable text
• Different models often use different tokenizers, so a "1,000 token" limit can represent slightly different amounts of actual text between models

Tokenization happens before any of the model's actual reasoning, it's the translation step between human-readable text and the numeric representation the model works with internally.

Byte-Pair Encoding is a popular tokenization algorithm that builds a vocabulary by repeatedly merging the most frequently occurring pair of characters or character sequences in a training corpus.

• Starts with individual characters as the smallest units
• Iteratively merges the most common adjacent pairs into new, longer tokens
• Stops once a target vocabulary size is reached, typically tens of thousands of tokens

This approach strikes a practical balance: common words end up as single tokens for efficiency, while rare or unseen words can still be represented by breaking them into smaller, previously-learned pieces instead of failing entirely.

An LLM's vocabulary is the complete, fixed set of tokens it was trained to recognize and generate, typically containing tens of thousands of entries.

• Built once during tokenizer training and then kept fixed for that model's entire lifetime
• Includes whole words, sub-word pieces, punctuation, and often special-purpose tokens for things like marking the start or end of a message
• Any input text gets broken down into pieces that exist somewhere in this vocabulary, no matter how unusual the input is

The vocabulary size is a real design trade-off: a larger vocabulary means more common words fit in a single token, but it also means a bigger, more expensive output layer for the model to compute over.

An embedding is a list of numbers, a vector, that represents a token's meaning in a way a neural network can do math with.

• Words with similar meanings end up with embeddings that are mathematically close to each other
• Generated by an embedding layer that's learned during training, not manually assigned
• Used as the very first step inside the model, converting each input token into its vector form before any further processing happens

Beyond just being an internal step inside an LLM, embeddings are also used directly for tasks like semantic search and clustering, where comparing the "closeness" of two pieces of text's meaning is the whole point.

The Transformer is the neural network architecture underlying virtually every modern LLM, introduced as a way to process sequences of text using attention instead of reading one word at a time in strict order.

• Built from stacked blocks, each containing an attention mechanism followed by a feed-forward network
• Processes an entire input sequence in parallel rather than word by word, making training dramatically faster on modern hardware
• Comes in three broad flavors: encoder-only, decoder-only, and encoder-decoder

Its core innovation, letting every token in a sequence directly weigh the relevance of every other token, is what let language models scale to the size and capability seen today.

Attention is the mechanism that lets a model weigh how relevant each other word in the input is when processing a given word, rather than treating every word as equally important.

• Computes a relevance score between every pair of tokens in a sequence
• Uses those scores to build a weighted combination of information from the whole sequence for each token
• Lets the model connect related words even when they're far apart in a sentence, like a pronoun and the noun it refers to several words earlier

This is what replaced older, strictly sequential approaches to language modeling, and it's the single component most responsible for the Transformer's success.

Self-attention is attention applied within a single sequence, each token attends to every other token in that same input, including itself, rather than attending to a separate sequence.

• Every token generates a Query, Key, and Value vector from its embedding
• The Query of one token is compared against the Keys of every token to compute relevance scores
• Those scores weight each token's Value, producing a new representation that blends in relevant context from across the sequence

This is the specific form of attention used throughout an LLM's internal layers, letting the model build up an increasingly context-aware representation of each token as it passes through the network.

Multi-Head Attention runs several self-attention operations in parallel, each with its own learned parameters, then combines their results together.

• Each "head" can learn to focus on a different kind of relationship, one might track grammatical structure, another might track topical relevance
• Running these in parallel, rather than one attention operation alone, lets the model capture multiple types of relationships between tokens at once
• The outputs from all heads are concatenated and passed through a final linear layer to produce one combined representation

This is a standard part of every Transformer block, and it's a major reason a single attention layer can capture such a rich, multi-faceted understanding of a sequence.

Positional Encoding is information added to each token's embedding to tell the model where that token sits in the sequence, since the attention mechanism on its own has no built-in sense of word order.

• Without it, a Transformer would treat "dog bites man" and "man bites dog" identically, since attention alone just relates tokens to each other regardless of position
• Typically implemented as a fixed or learned pattern of values added directly to each token's embedding
• Lets the model reconstruct sequence order without needing to process tokens strictly one at a time

This small addition is what allows the Transformer to keep its fast, parallel processing while still understanding that order matters in language.

Each Transformer block contains a feed-forward network that processes every token's representation individually after the attention step has mixed in context from the rest of the sequence.

• Applies the same two-layer transformation to each token's vector independently
• Adds additional representational capacity and non-linearity that attention alone doesn't provide
• Works alongside attention in every block, attention gathers context across tokens, the feed-forward network then further processes each token's resulting representation

Together, the attention and feed-forward steps in each block are what let a Transformer refine its understanding of a sequence layer by layer as information flows through the network.

Layer Normalization is a technique used inside a Transformer to keep the scale of values flowing through the network stable as they pass through many stacked layers.

• Rescales the values within each layer's output to have a consistent mean and variance
• Helps training converge faster and more reliably, since wildly varying value scales can make gradient-based training unstable
• Applied at specific points within each Transformer block, typically before or after the attention and feed-forward steps

Without this kind of stabilization, training a network as deep as a modern LLM, often dozens of stacked layers, would be considerably harder to get working reliably.

Causal masking is a technique used in decoder-style LLMs that prevents a token from attending to any tokens that come after it in the sequence.

• Ensures the model can only use information from earlier in the text when predicting the next token
• Necessary because generation happens one token at a time, left to right, so a token genuinely can't have access to future tokens it hasn't generated yet
• Implemented by masking out, effectively zeroing out, the attention scores between a token and any later position

This is what keeps an LLM's next-token prediction honest during training, it can't "cheat" by peeking ahead at the answer it's supposed to be predicting.

Softmax is the function that converts the model's raw output scores for every possible next token into a proper probability distribution that sums to one.

• Takes a raw score for every token in the vocabulary, some tens of thousands of numbers
• Converts those scores into probabilities, with higher raw scores becoming higher probabilities
• The model then samples, or picks, its next token based on this probability distribution

This is the final step that turns the model's internal computation into an actual choice of which word comes next, and it's also the point where settings like temperature come into play.

Cross-Entropy Loss is the standard measurement used to train an LLM, quantifying how far off the model's predicted probability distribution was from the actual next token in the training data.

• Penalizes the model more heavily when it assigns low probability to the token that actually came next
• Averaged across every token prediction in a training batch to produce one overall loss value
• That loss value is what training actually tries to minimize, through repeated adjustments to the model's parameters

Lower cross-entropy loss on held-out data generally indicates a model that's genuinely learning useful patterns, rather than just memorizing its specific training examples.

A loss function gives training a single number representing how wrong the model's predictions currently are, which is what the training process actually tries to reduce.

• Computed after each batch of training examples, comparing the model's predictions against the correct answers
• Used to calculate gradients, which indicate how each of the model's billions of parameters should be adjusted
• Training repeats this cycle, predict, measure loss, adjust parameters, millions of times over the training data

Without a loss function, there would be no concrete signal telling the training process whether a given adjustment to the model actually made it better or worse.

Pretraining is the initial, large-scale training phase where a model learns general language patterns by predicting the next token across a massive, broad corpus of text.

• Typically uses text scraped from books, websites, code, and other large-scale sources
• Requires enormous computational resources, often running across thousands of specialized chips for weeks or months
• Produces a base model with broad language understanding, but not yet tuned for any specific task or conversational behavior

Pretraining is what gives an LLM its raw capability, everything that happens afterward, fine-tuning, alignment, and so on, builds on top of this foundation rather than replacing it.

Fine-tuning is the process of further training an already-pretrained model on a smaller, more specific dataset to adjust its behavior for a particular task or style.

• Starts from a pretrained base model rather than training from scratch
• Uses a much smaller, more targeted dataset than pretraining requires
• Can adapt a general-purpose model into one specialized for tasks like customer support, coding, or following conversational instructions

This two-stage approach, broad pretraining followed by targeted fine-tuning, is far more efficient than training a specialized model entirely from scratch for every new use case.

Instruction tuning is a specific type of fine-tuning where a model is trained on examples of instructions paired with the responses that correctly follow them.

• Teaches a base model, which is only good at predicting plausible next text, to instead behave like a helpful assistant that follows a user's actual request
• Uses datasets of paired prompts and ideal responses covering a wide range of task types
• Is usually one part of a broader alignment process, often combined with techniques like RLHF

This is the step that transforms a raw, next-token-predicting base model into something that reliably answers questions and follows directions the way people expect a chatbot to behave.

RLHF is a technique used to further align a model's behavior with human preferences, using human judgments as the training signal instead of a fixed dataset of correct answers.

• Human reviewers compare multiple model responses to the same prompt and rank which ones they prefer
• Those rankings are used to train a separate reward model that predicts how much a person would like a given response
• The main model is then further trained using reinforcement learning to produce responses that score highly according to that reward model

RLHF is a major reason modern LLMs feel noticeably more helpful, honest, and safe to interact with compared to a purely pretrained or instruction-tuned model alone.

Alignment refers to the broader effort of making a model's behavior match what humans actually want, helpful, honest, and safe responses, rather than just fluent or plausible-sounding text.

• Pretraining alone only teaches a model to predict likely next text, with no built-in sense of what's actually helpful or appropriate
• Alignment techniques like instruction tuning and RLHF are what layer on that sense of appropriate behavior
• An unaligned base model can be fluent yet unhelpful, evasive, or willing to produce harmful content if simply prompted the right way

Alignment is an ongoing area of active research, since teaching a model nuanced human values and judgment turns out to be considerably harder than teaching it grammar and facts.

In-context learning is a model's ability to pick up a new pattern or task just from examples given directly in the prompt, without any additional training or parameter updates.

• The model isn't actually "learning" in the traditional sense, its weights don't change
• Instead, it recognizes the pattern demonstrated in the prompt and continues it for the new input
• Works because the model's pretraining exposed it to an enormous variety of patterns and structures it can recognize and mimic

This capability is what makes prompting techniques like few-shot examples effective, you're not training the model, you're showing it exactly what kind of response you want within the conversation itself.

Zero-shot learning is when a model performs a task correctly based purely on an instruction, with no examples of that task provided in the prompt.

• Relies entirely on the model's pretraining and instruction tuning to understand what's being asked
• Works well for common, well-understood tasks, like "summarize this paragraph," that the model has likely seen extensively during training
• Tends to perform less reliably on unusual or highly specific task formats the model hasn't encountered much before

Zero-shot performance is often the first thing tested when evaluating how well a model generalizes, since it reflects genuine understanding rather than pattern-copying from provided examples.

Few-shot learning is when a handful of example input-output pairs are included directly in the prompt to demonstrate exactly what kind of response is wanted, before asking the model to handle a new case.

• Typically uses somewhere between two and a handful of examples
• Helps the model understand a specific format, tone, or task structure that a plain instruction alone might leave ambiguous
• Consumes more of the context window than zero-shot prompting, since the examples themselves take up tokens

Few-shot prompting is a practical, no-training-required way to steer a model toward a very specific style of response when a plain instruction isn't precise enough on its own.

Chain-of-Thought prompting encourages a model to write out its intermediate reasoning steps before giving a final answer, rather than jumping straight to a conclusion.

• Can be triggered explicitly, by asking the model to "think step by step"
• Often noticeably improves accuracy on tasks involving multi-step logic, arithmetic, or reasoning
• Gives a person reviewing the output visibility into how the model arrived at its answer, not just what the answer was

This technique works because it gives the model more computational "room" to work through a problem incrementally, rather than having to produce a correct answer in a single, immediate leap.

Prompt engineering is the practice of carefully crafting the wording, structure, and examples in a prompt to get more accurate, relevant, or well-formatted responses from a model.

• Includes techniques like being explicit about the desired format, providing examples, and breaking complex requests into clear steps
• Doesn't require retraining or fine-tuning the model, it works entirely through how the request itself is phrased
• Effectiveness can vary meaningfully between different models, since each one responds somewhat differently to the same phrasing

Good prompt engineering is often the fastest, cheapest way to improve an LLM's output on a specific task, well before considering something more involved like fine-tuning.

A system prompt is a set of instructions given to a model before the actual conversation begins, establishing its role, tone, boundaries, or behavior for the rest of that session.

• Typically not shown to the end user, but still shapes every response the model gives afterward
• Used to set persistent behavior, like "respond only in French" or "act as a customer support agent for this company"
• Distinct from a regular user prompt, which is a specific request within the ongoing conversation

This is one of the main levers application developers use to customize a general-purpose model's behavior for their specific product, without needing to fine-tune anything.

A context window is the maximum amount of text, measured in tokens, that a model can consider at once, covering the prompt, any conversation history, and the response it's generating.

• Once a conversation exceeds this limit, earlier content has to be dropped, summarized, or otherwise handled to make room
• Larger context windows let a model reference more information at once, like an entire long document
• Modern models range widely in context window size, from a few thousand tokens up to hundreds of thousands or more

Context window size is one of the more practical constraints application developers work around, since it directly limits how much conversation history or reference material can be included in a single request.

Max Tokens is a setting that limits how many tokens a model is allowed to generate in a single response, capping the output length.

• Doesn't affect how much input the model can read, only how much it's allowed to write back
• Setting it too low can cut off a response mid-sentence before the model finishes its thought
• Commonly used to control cost and response time, since longer outputs take more compute and more time to generate

This parameter is typically set by whoever is calling the model through an API, giving developers direct control over how long a generated response is allowed to run.

Stop sequences are specific strings that, when generated, tell the model to immediately stop producing further output.

• Commonly used to prevent a model from continuing past a natural end point, like generating an extra, unwanted turn of a conversation
• Set by the developer calling the model, not something the model decides on its own
• Useful for structured output tasks, where generation should stop exactly at a specific marker rather than running until it hits the max token limit

This gives applications precise control over exactly where a response should end, rather than relying purely on the model's own judgment about when it's finished.

Temperature is a setting that controls how random or predictable a model's word choices are when generating text.

• A low temperature, close to zero, makes the model consistently pick its highest-probability next token, producing more focused, repeatable output
• A higher temperature flattens the probability distribution, giving lower-probability tokens more chance of being picked, producing more varied, creative output
• Applied right before the model samples its next token, after the softmax step has produced the probability distribution

Choosing temperature is a real trade-off: lower values suit tasks needing consistency and precision, like code generation, while higher values suit tasks that benefit from variety, like brainstorming.

Top-k and top-p are two related techniques that limit which tokens a model is allowed to consider when picking its next word, rather than sampling from the full vocabulary every time.

• Top-k sampling: restricts the choice to only the k highest-probability tokens
• Top-p sampling, also called nucleus sampling: restricts the choice to the smallest set of tokens whose combined probability adds up to at least p

Both exist to avoid the model occasionally picking a bizarre, very-low-probability token purely by chance, while still allowing some genuine variety in its output, unlike always picking the single most likely token.

Hallucination is when a model generates text that sounds fluent and confident but is factually incorrect or entirely made up.

• Happens because a model is fundamentally predicting plausible next text, not verifying facts against a trusted source
• Can involve inventing citations, misremembering details, or confidently stating something that simply isn't true
• More likely on questions the model has little reliable training data about, or that require very precise, specific facts

Techniques like retrieval-augmented generation, grounding answers in retrieved documents rather than the model's memory alone, are commonly used to reduce, though not fully eliminate, hallucination.

Perplexity is a metric that measures how well a language model predicts a given piece of text, essentially quantifying how "surprised" the model is by the actual next words.

• Lower perplexity means the model assigned higher probability to what actually happened next, indicating a better fit to that text
• Commonly used to evaluate and compare language models during development, especially on held-out test data
• Doesn't directly measure whether a model's output is factually correct or genuinely useful, just how predictable the text was to the model

Perplexity is a useful, easy-to-compute proxy for language modeling quality, but it's just one signal among several used to judge whether a model is actually good at real tasks.

Overfitting happens when a model learns its training data so closely that it starts memorizing specific examples rather than learning generalizable patterns.

• Shows up as strong performance on training data but noticeably worse performance on new, unseen data
• More of a risk during fine-tuning on a small dataset than during massive-scale pretraining
• Mitigated through techniques like using a large, diverse training set, regularization, and monitoring performance on held-out validation data during training

A model that's overfit might repeat memorized phrases verbatim rather than genuinely reasoning about a new but similar situation, which is exactly the opposite of what makes a language model useful.

Parameters are the internal numerical values, weights, that a neural network learns during training and uses to transform input into output.

• Modern LLMs commonly have anywhere from a few billion to hundreds of billions of parameters
• Every parameter is adjusted incrementally during training to reduce the model's loss on its training data
• Parameter count is a common, though imperfect, shorthand for a model's raw capacity, more parameters generally means more capacity to learn complex patterns

Parameter count isn't the whole story though, training data quality, architecture choices, and fine-tuning all meaningfully affect how capable a model actually is, independent of raw size.

Quantization is a technique that reduces the numerical precision used to store a model's parameters, shrinking its memory footprint and speeding up inference.

• Converts weights from a high-precision format, like 32-bit floating point, down to a lower-precision one, like 8-bit or even 4-bit
• Reduces both the memory needed to store the model and the compute needed to run it
• Typically causes some small loss in output quality, though well-implemented quantization can often keep that loss minimal

Quantization is a major reason large models can run on comparatively modest hardware, like a laptop or a single consumer GPU, instead of requiring a full data center.

Model distillation is a technique for training a smaller "student" model to mimic the behavior of a larger "teacher" model, transferring much of the larger model's capability into a more efficient package.

• The student model is trained to match the teacher's output probabilities, not just its final answers
• Produces a smaller model that runs faster and cheaper than the original
• Usually results in some capability trade-off compared to the full-sized teacher model, though a well-distilled model can retain much of its usefulness

Distillation is one of several complementary approaches, alongside quantization, for making a powerful but expensive model practical to deploy at scale.

A foundation model is a large, broadly pretrained model designed to serve as a general-purpose base that can be adapted to many different downstream tasks.

• Trained on a wide, diverse dataset rather than for one narrow purpose
• Meant to be fine-tuned, prompted, or otherwise adapted for a wide range of specific applications
• Contrasts with a narrowly trained model built and tuned for just one specific task from the start

Most well-known LLMs are foundation models at their core, the same base model can end up powering a chatbot, a coding assistant, and a summarization tool, just adapted differently for each use case.

A Decoder-only model is a Transformer variant built entirely from decoder blocks, generating text one token at a time based only on the tokens that came before it.

• Uses causal masking so each token can only attend to earlier tokens, never later ones
• Well suited to open-ended text generation, since that's exactly the pattern it was trained on
• Is the architecture behind most modern general-purpose LLMs, including GPT-style models

This architecture's straightforward "predict the next token" design is a big part of why it scales so well and has become the dominant approach for large-scale language models.

An Encoder-only model processes an entire input sequence at once, with every token able to attend to every other token, including ones that come later in the sequence.

• No causal masking, since it's not generating text one token at a time
• Well suited to understanding tasks like classification, sentiment analysis, or extracting information from text, rather than generating new text
• BERT is the best-known example of this architecture family

Encoder-only models excel at deeply understanding a fixed piece of input text, but they aren't naturally built for the open-ended generation task that decoder-only models handle.

An Encoder-Decoder model combines both halves of the original Transformer design, an encoder that processes the full input, and a decoder that generates output text based on that encoded understanding.

• The encoder builds a rich representation of the entire input sequence first
• The decoder then generates output text step by step, attending both to previously generated tokens and to the encoder's representation of the input
• Well suited to tasks with a clear input-to-output transformation, like translation or summarization

This was the original Transformer architecture from 2017, and while decoder-only designs have become more dominant for general-purpose LLMs, encoder-decoder models remain a strong fit for tasks structured as a clear input-to-output transformation.

Mixture of Experts is a model design where, instead of every input passing through the entire network, a routing mechanism sends each input to only a subset of specialized sub-networks, called experts.

• Only a fraction of the model's total parameters are actually used for any given input
• Lets a model have a very large total parameter count while keeping the actual compute cost per request much lower than using all parameters every time
• The routing mechanism itself is learned during training, deciding which experts are best suited for a given input

This approach is a practical way to scale a model's total capacity without proportionally scaling the cost of running it on every single request.

Retrieval-Augmented Generation combines a language model with an external retrieval step, pulling in relevant documents or data before generating a response, rather than relying purely on what the model memorized during training.

• A retrieval step searches an external knowledge source for content relevant to the current query
• That retrieved content is added into the model's context before it generates its answer
• Helps reduce hallucination by grounding responses in retrievable, verifiable source material

RAG is especially useful for keeping answers current or specific to private data, since it lets a model reference information well beyond whatever it happened to see during pretraining.

Beyond powering the internal workings of an LLM, embeddings are widely used on their own as a tool for comparing the meaning of different pieces of text.

• Semantic search: finding documents whose meaning is similar to a query, not just ones sharing the same keywords
• Clustering: grouping similar pieces of text together automatically
• Recommendation: finding items whose descriptions are semantically similar to something a user already liked

These applications rely on the same core idea that makes embeddings useful inside a model: pieces of text with similar meaning end up mathematically close together, which is exactly what similarity search and clustering need.

Guardrails are the checks and constraints put around a model in a real application to keep its behavior within safe, expected, or on-topic bounds.

• Can filter or block certain categories of user input before it ever reaches the model
• Can also check the model's output before it's shown to a user, catching unwanted content or off-topic responses
• Often combined with a well-crafted system prompt, though guardrails typically add independent checks beyond just prompting

Guardrails exist because a model's own instruction-following isn't a perfectly reliable safety mechanism on its own, so applications commonly add external checks as an additional layer of protection.

Prompt injection is when malicious or unexpected instructions are hidden inside content a model processes, tricking it into behaving differently than the actual user intended.

• Can be hidden inside a webpage, document, or file that a model reads as part of completing a task
• Exploits the fact that a model can't perfectly distinguish between genuine instructions from its actual user and text that merely looks like instructions
• Becomes especially risky for models connected to tools or external actions, since a successful injection can trigger real, unwanted actions

This is one of the more actively studied security challenges in deploying LLMs, and current defenses focus on input sanitization, careful permission scoping, and guardrails, rather than any single complete fix.