Generative AI and LLM Basics

In this lesson: How generative AI and large language models work — tokens, training, prompts and limits — explained simply. In Hindi & English.

What is Generative AI?

Generative AI creates new content – text, images, code, audio – instead of only classifying or predicting a number. A Large Language Model (LLM) is a generative model for text: ChatGPT, Claude, Gemini and Llama are LLMs. Underneath, an LLM does one simple thing again and again: predict the next piece of text.

Step 1: text becomes tokens

Models do not read letters or whole words; they read tokens – common pieces of words. “unbelievable” may become “un”, “believ”, “able”. Each token is turned into a number (an id) and then into a vector. Hindi often needs more tokens than English for the same sentence, which is why Hindi prompts can cost more and fill the context faster.

# a toy tokenizer: split into words and give each new word an id
sentence = "the shop sells fresh sweets and the shop opens early"
vocab = {}
ids = []
for word in sentence.split():
    if word not in vocab:
        vocab[word] = len(vocab)
    ids.append(vocab[word])

print("tokens:", sentence.split())
print("ids   :", ids)
print("vocabulary size:", len(vocab))

# bytes show why non-English text is “longer” for machines
for text in ["sweet", "मिठाई"]:
    print(f"{text!r}: {len(text)} characters, {len(text.encode('utf-8'))} UTF-8 bytes")

Step 2: predicting the next word

The simplest language model counts which word follows which. This bigram model learns from 7 sentences about a sweet shop and then writes new text. Real LLMs use the same idea but look at thousands of previous tokens with a neural network (the Transformer) trained on trillions of tokens.

import random
from collections import Counter, defaultdict

corpus = "the shop opens at nine . the shop sells fresh sweets . fresh sweets are made every morning . the owner makes kaju katli every morning . customers buy fresh kaju katli . the shop closes at nine at night . customers love the shop ."
words = corpus.split()

# 1. TRAINING: count next-word frequencies
next_counts = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
    next_counts[current][nxt] += 1

print("after 'fresh':", dict(next_counts["fresh"]))
print("after 'the'  :", dict(next_counts["the"]))

# 2. GENERATION: repeatedly sample the next word
def generate(start, length=10, seed=1):
    rng = random.Random(seed)
    out = [start]
    for _ in range(length):
        options = next_counts[out[-1]]
        if not options:
            break
        out.append(rng.choices(list(options), weights=options.values())[0])
    return " ".join(out)

for seed in (1, 2, 3):
    print("→", generate("the", seed=seed))

Notice that it produces sentences that never appeared in the training text (“customers buy fresh sweets are made…”). Fluent-sounding but sometimes wrong text is exactly why LLMs can hallucinate.

Step 3: temperature – creative or predictable

The model outputs a score for every possible next token. Softmax turns scores into probabilities, and temperature sharpens or flattens them before sampling.

import math

next_words = ["sweets", "kaju", "snacks", "flowers"]
scores = [3.0, 2.2, 1.0, -1.0]          # model scores (logits) after "the shop sells fresh"

def softmax(values, temperature):
    exps = [math.exp(v / temperature) for v in values]
    total = sum(exps)
    return [e / total for e in exps]

for t in (0.2, 1.0, 2.0):
    probs = softmax(scores, t)
    row = "  ".join(f"{w}={p:.0%}" for w, p in zip(next_words, probs))
    print(f"temperature {t}: {row}")

How an LLM is built

1. Pre-training: read a huge amount of public text and code, learning to predict the next token. This gives knowledge and language skill.
2. Fine-tuning: train further on examples of good instructions and answers, so it follows requests.
3. Feedback training (e.g. RLHF): people rate answers and the model learns to be more helpful and safer.
4. Use: you send a prompt; the model generates one token at a time until it stops.

Step 4: the idea behind RAG in 20 lines

LLMs do not know your school’s fee rules or your shop’s prices. RAG first searches your own documents for the most relevant text and puts it into the prompt. Here is the retrieval half with simple word-overlap scoring (real systems use embeddings).

documents = {
    "fees": "School fees are due on the 10th of every month. A late fee of Rs 100 applies after the 15th.",
    "timing": "Classes run from 8:00 am to 2:00 pm, Monday to Saturday.",
    "transport": "Bus service is available for Khurja and Bulandshahr routes at Rs 900 per month.",
}

def retrieve(question, k=1):
    q = set(question.lower().replace("?", "").split())
    scored = sorted(documents.items(), key=lambda d: -len(q & set(d[1].lower().split())))
    return [text for _, text in scored[:k]]

question = "What is the late fee if I pay after the 15th?"
context = retrieve(question)[0]
prompt = f"""Answer using ONLY the context. If the answer is not there, say you don't know.
Context: {context}
Question: {question}"""
print(prompt)

This prompt is then sent to an LLM. Because the answer is in the context, the model can answer correctly and you can show the source to the user.

Calling a real LLM from Python

Every provider (Anthropic, OpenAI, Google) has a Python SDK. The pattern is the same: install the SDK, keep your API key in an environment variable, send messages, read the text reply. Model names change often – copy the current one from the provider’s documentation.

# pip install anthropic       (set ANTHROPIC_API_KEY and LLM_MODEL in your environment)
import os
import anthropic

client = anthropic.Anthropic()                      # reads ANTHROPIC_API_KEY automatically
reply = client.messages.create(
    model=os.environ["LLM_MODEL"],                  # e.g. the current model name from the docs
    max_tokens=300,
    temperature=0.3,
    system="You are a patient tutor for Class 10 students. Answer in simple Hindi and English.",
    messages=[{"role": "user", "content": "Explain photosynthesis in 5 short points."}],
)
print(reply.content[0].text)

Strengths and limits

Next: learn to write better instructions in Prompt Engineering, and read AI Ethics before building anything for real users.

Practice tasks

1. Train the bigram model on 10 sentences about your school and generate 5 lines.
2. Change the temperature values and find the point where “sweets” drops below 50%.
3. Add two more documents to the RAG example and ask a question that needs the transport document.
4. Write down three questions where an LLM is likely to hallucinate, and why.

Summary

• LLMs turn text into tokens and predict the next token again and again.
• Training = pre-training on huge text + fine-tuning + human feedback.
• Temperature controls randomness; low for facts, higher for ideas.
• RAG adds your own documents to the prompt to reduce hallucinations.
• Protect keys, costs and private data; always verify important answers.