🤖 Phase 5 — LLMs & Applied GenAI Engineering (Days 71–90)

# Temperature, top-k, top-p: controlling randomness in generation
#
# The model outputs a probability distribution over the next token.
# Sampling strategy determines HOW you pick from that distribution.
 
# Temperature: scales the logits before softmax
# temperature -> 0: nearly deterministic, always picks highest-probability token
# temperature = 1: sample exactly from the model's learned distribution
# temperature > 1: flatter distribution, more random/creative, more mistakes
 
import torch.nn.functional as F
 
def sample_with_temperature(logits, temperature=1.0):
    scaled_logits = logits / temperature
    probs = F.softmax(scaled_logits, dim=-1)
    return torch.multinomial(probs, num_samples=1)
 
# Top-k sampling: only consider the K most likely next tokens, renormalize, sample
def top_k_sampling(logits, k=50):
    values, indices = torch.topk(logits, k)
    probs = F.softmax(values, dim=-1)
    sampled_idx = torch.multinomial(probs, num_samples=1)
    return indices[sampled_idx]
 
# Top-p (nucleus) sampling: consider the smallest set of tokens whose
# cumulative probability exceeds p (more adaptive than fixed top-k)
def top_p_sampling(logits, p=0.9):
    sorted_logits, sorted_indices = torch.sort(logits, descending=True)
    cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
    mask = cumulative_probs <= p
    mask[0] = True  # always keep at least the top token
    filtered_indices = sorted_indices[mask]
    return filtered_indices
 
# Prompt engineering patterns that actually work (not folklore):
# 1. Few-shot examples: show 2-5 examples of input->output before the real task
# 2. Chain-of-thought: ask the model to "think step by step" before answering
#    -- works because it gives the model more forward passes (tokens) to reason
# 3. System prompts: set persistent behavior/role/constraints
# 4. Structured output: ask for JSON, provide a schema, often with function calling
#    instead of hoping the model formats text correctly

# Embeddings (recap from Phase 4, now applied): represent text as a dense vector
# such that SEMANTICALLY SIMILAR text has vectors that are CLOSE TOGETHER
 
from sentence_transformers import SentenceTransformer
import numpy as np
 
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(["The cat sat on the mat", "A feline rested on the rug"])
 
# Cosine similarity: measures the ANGLE between vectors, ignoring magnitude
# This is THE standard similarity metric for embeddings
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
sim = cosine_similarity(embeddings[0], embeddings[1])
print(sim)  # high, even though no words overlap -- semantic similarity!
 
# The problem: brute-force search over millions of vectors is O(n) per query, too slow
# Solution: Approximate Nearest Neighbor (ANN) algorithms
 
# FAISS: Facebook's library for efficient similarity search
import faiss
 
dimension = 384  # matches the embedding model's output size
index = faiss.IndexFlatL2(dimension)         # exact search, fine for small data
# For large scale: faiss.IndexIVFFlat or faiss.IndexHNSWFlat (approximate, much faster)
 
index.add(embeddings.astype('float32'))
query_embedding = model.encode(["a cat on a mat"]).astype('float32')
distances, indices = index.search(query_embedding, k=2)  # top-2 nearest
 
# HNSW (Hierarchical Navigable Small World): the algorithm behind most
# production vector databases (Pinecone, Weaviate, Qdrant, Chroma)
# Builds a multi-layer graph for O(log n) approximate search instead of O(n)
# This directly solves the curse-of-dimensionality problem from Phase 2's KNN

# RAG solves: LLMs don't know your private data, and their training data
# has a cutoff date. RAG retrieves relevant documents at query time and
# injects them into the prompt, grounding the LLM's answer in real, current,
# private information instead of relying purely on what it memorized during training.
 
# The full RAG pipeline:
# 1. INDEXING (offline, done once per document set):
#    documents -> chunk into pieces -> embed each chunk -> store in vector DB
# 2. RETRIEVAL (online, per query):
#    user query -> embed query -> search vector DB -> get top-K relevant chunks
# 3. GENERATION (online, per query):
#    relevant chunks + user query -> prompt template -> LLM -> grounded answer
 
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
 
# Step 1: Chunking -- this is THE most underrated lever in RAG quality
# Too small: loses context. Too large: dilutes relevance, wastes tokens.
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,    # overlap prevents losing context at chunk boundaries
    separators=["\n\n", "\n", ". ", " "]  # try to split on natural boundaries
)
chunks = text_splitter.split_documents(documents)
 
# Step 2: Embed and store
embeddings = HuggingFaceEmbeddings(model_name='all-MiniLM-L6-v2')
vectorstore = Chroma.from_documents(chunks, embeddings)
 
# Step 3: Retrieve + Generate
retriever = vectorstore.as_retriever(search_kwargs={'k': 4})
llm = ChatOpenAI(model='gpt-4', temperature=0)
 
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True  # always return sources for verification/citation
)
 
result = qa_chain.invoke({"query": "What is our refund policy?"})
print(result['result'])
print(result['source_documents'])  # show what the answer was grounded in
 
# Advanced RAG techniques:
# - Hybrid search: combine dense (embedding) + sparse (BM25/keyword) retrieval
# - Re-ranking: retrieve more candidates than needed, then re-rank with a
#   cross-encoder model for higher precision on the final top-K
# - Query expansion/rewriting: rewrite the user's query before retrieval
#   to improve recall (especially for vague or conversational queries)
# - Parent-child chunking: embed small chunks for precise retrieval,
#   but return the larger parent chunk for full context to the LLM

# Full fine-tuning updates ALL parameters of a model (billions of them) --
# expensive in compute and memory, and risks "catastrophic forgetting"
# (the model loses general capabilities while learning the new task)
 
# LoRA (Low-Rank Adaptation): freeze the original weights, inject small
# trainable "adapter" matrices into each layer. Train only the adapters
# (often <1% of total parameters), then add their effect back to the frozen weights.
#
# Key insight: W_new = W_frozen + (A @ B), where A and B are small
# low-rank matrices (e.g., rank 8 or 16), dramatically fewer parameters
# than the full weight matrix W
 
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
 
model = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-2-7b-hf')
tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
 
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,                    # rank of the low-rank matrices (lower = fewer params, less capacity)
    lora_alpha=32,           # scaling factor for the LoRA update
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj"]  # which layers to adapt (usually attention projections)
)
 
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%
# This is THE number that makes fine-tuning feasible on consumer GPUs
 
# QLoRA: LoRA + quantization (load the frozen base model in 4-bit precision)
# Reduces memory footprint further -- enables fine-tuning 7B+ models on a
# single consumer GPU (e.g., 24GB VRAM) that couldn't otherwise fit the model
from transformers import BitsAndBytesConfig
import torch
 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Llama-2-7b-hf',
    quantization_config=bnb_config
)
 
# Training loop (standard Hugging Face Trainer)
training_args = TrainingArguments(
    output_dir='./lora-output',
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # simulates a larger batch size with less memory
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
)
trainer = Trainer(model=model, args=training_args, train_dataset=dataset)
trainer.train()

# An LLM agent: a model that can decide to call external TOOLS (functions,
# APIs, code execution, search) to accomplish a task, rather than relying
# purely on its internal knowledge.
 
# Function calling: the model outputs a structured request to call a specific
# function with specific arguments, instead of free-text
import openai
 
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name"}
            },
            "required": ["location"]
        }
    }
}]
 
response = openai.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "What's the weather in Bangalore?"}],
    tools=tools
)
 
# The model returns a tool_call instead of text. YOUR code executes the
# actual function, then sends the result back to the model for a final answer.
if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]
    args = json.loads(tool_call.function.arguments)
    result = get_weather(args['location'])  # your actual implementation
    # Send result back to the model to generate the final natural-language response
 
# The ReAct pattern (Reasoning + Acting): the model alternates between
# "thinking" (reasoning about what to do) and "acting" (calling a tool),
# observing results, and repeating until it has enough information to answer
#
# Thought: I need to find the current weather in Bangalore
# Action: get_weather(location="Bangalore")
# Observation: 28C, partly cloudy
# Thought: I now have enough information to answer
# Final Answer: It's 28C and partly cloudy in Bangalore.
 
# Multi-agent systems: specialized agents (researcher, coder, reviewer)
# collaborate, each with different tools/prompts/responsibilities
# Frameworks: LangGraph, CrewAI, AutoGen

# "It looks good when I tried it" is not evaluation. LLM applications need
# systematic evals just like classical ML models need test set metrics.
 
# RAG-specific evaluation dimensions (using a framework like Ragas):
# - Faithfulness: is the generated answer actually supported by the retrieved context?
#   (catches hallucination even when retrieval was correct)
# - Answer relevance: does the answer actually address the question asked?
# - Context precision: are the retrieved chunks actually relevant?
# - Context recall: did retrieval find ALL the relevant information needed?
 
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset
 
eval_dataset = Dataset.from_dict({
    'question': [...],
    'answer': [...],            # your system's generated answers
    'contexts': [...],          # retrieved chunks for each question
    'ground_truth': [...]       # the correct/reference answer
})
 
results = evaluate(
    eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(results)
 
# LLM-as-judge: use a strong LLM to score outputs against a rubric
# (common when there's no single "correct" answer to compare against)
judge_prompt = """
Rate the following response on a scale of 1-5 for helpfulness and accuracy.
Question: {question}
Response: {response}
Reference: {reference}
Provide a score and brief justification.
"""
 
# Building a golden eval set: curate 50-200 representative question/answer
# pairs covering edge cases, ambiguous queries, and common failure modes
# Re-run this eval set EVERY time you change a prompt, model, or retrieval
# parameter -- this is your regression test suite for an LLM application

Required components:
 
1. Document ingestion pipeline
   - Load documents (PDFs, markdown, or scraped web content)
   - Chunk with a documented, justified strategy (explain your chunk_size/overlap choice)
   - Embed and store in a vector database (Chroma or FAISS, locally is fine)
 
2. Retrieval + Generation pipeline
   - Query embedding + vector search
   - Optional: hybrid search or re-ranking (bonus)
   - Prompt template that includes retrieved context + explicit instructions
     to cite sources and say "I don't know" when context is insufficient
   - LLM call (OpenAI API, Anthropic API, or local model via Ollama)
 
3. Evaluation harness
   - A golden set of 20-30 question/answer pairs for your document domain
   - Run faithfulness and relevance metrics (Ragas or custom LLM-as-judge)
   - Report a baseline score, then make ONE improvement (better chunking,
     re-ranking, or prompt tweaking) and show the score improvement
 
4. API and basic UI
   - FastAPI endpoint: POST /query -> returns answer + sources
   - Simple frontend: Streamlit or Gradio (functional, not polished)
   - Basic error handling: what happens when retrieval finds nothing relevant?
     What happens when the LLM API call fails or times out?
 
5. Documentation
   - README explaining architecture, how to run it, and your eval results
   - A section titled "Known limitations" — every real system has them,
     and naming them explicitly is what separates engineers from demos