AI Chatbots Guide - Professional

Overview

Large Language Models (LLMs)

Large Language Models are transformer-based neural networks trained on extensive text corpora using self-supervised learning. These models employ attention mechanisms to capture long-range dependencies and contextual relationships within sequences.

Key capabilities:

Natural language understanding and generation
Context-aware reasoning (up to 1M tokens)
Multi-turn conversation with memory retention
Function calling and tool integration
Structured output generation (JSON, XML, code)
Multimodal processing (text, images, audio)

Transformer Architecture Fundamentals

Modern LLMs utilize the transformer architecture introduced in "Attention Is All You Need" (Vaswani et al., 2017). Key components:

Self-Attention Mechanism: Enables parallel processing of sequence elements with O(n²) complexity
Multi-Head Attention: Captures different representation subspaces simultaneously
Feed-Forward Networks: Position-wise fully connected layers with GELU/SwiGLU activation
Positional Encoding: RoPE (Rotary Position Embedding) or ALiBi for sequence position awareness
Layer Normalization: RMSNorm or Pre-LN for training stability

Market-Leading Models (2024-2025)

Model	Organization	Context	Modality	Best For
GPT-5.6	OpenAI	128K	Text, Vision	General purpose, complex reasoning
Claude Sonnet 5	Anthropic	1M	Text, Vision	Long-context analysis, code generation
Gemini 3.1 Pro	Google	2M	Text, Vision, Audio	Ultra-long context, multimodal tasks
Grok 4.5	xAI	32K	Text, Vision	Real-time data, X integration
DeepSeek V4	DeepSeek	128K	Text	Cost-efficient, high performance

Pro Tip: Choose models based on your specific requirements: GPT-5.6 for general tasks, Claude for extensive code analysis, Gemini for processing large documents (books, codebases), and DeepSeek for cost-sensitive production deployments.

Architecture

Decoder-Only Transformer Architecture

Most modern LLMs (GPT series, Claude, LLaMA) use decoder-only architectures optimized for autoregressive generation:

                        
Input Tokens → Embedding Layer → Positional Encoding
    ↓
Transformer Decoder Blocks (×N layers):
    ├─ Multi-Head Self-Attention (causal masking)
    ├─ Layer Normalization
    ├─ Feed-Forward Network (4× hidden dim)
    └─ Residual Connections
    ↓
Final Layer Norm → Output Projection → Logits → Softmax
                        
                    

Key architectural innovations:

Grouped-Query Attention (GQA): Reduces KV cache memory by sharing key/value heads
FlashAttention: IO-aware attention computation for 2-4× speed improvement
Mixture of Experts (MoE): Sparse activation of expert networks (GPT-5.6, Mixtral)
SwiGLU Activation: Gated Linear Unit variant outperforming ReLU/GELU
RMSNorm: Computationally efficient alternative to LayerNorm

Training Pipeline

Modern LLMs undergo multi-stage training:

1. Pre-training

Next-token prediction on trillion-token datasets. Objectives: minimize cross-entropy loss, learn language patterns and world knowledge.

2. Supervised Fine-Tuning (SFT)

Training on curated instruction-response pairs to align model behavior with desired task formats.

3. RLHF

Reinforcement Learning from Human Feedback using PPO algorithm to optimize for human preferences.

4. Constitutional AI

Self-supervised alignment using AI-generated critiques and revisions (Anthropic's approach).

Context Window Management

Context window determines the maximum sequence length the model can process:

Context Size	Use Case	KV Cache Memory
4K-8K tokens	Short conversations, simple queries	~100MB
32K-128K tokens	Document analysis, extended dialogues	~1-4GB
1M-2M tokens	Entire codebases, books, long-form content	~10-100GB

Optimization: Use techniques like sliding window attention, sparse attention patterns, or external memory systems (RAG) to handle contexts exceeding model limits efficiently.

Prompt Engineering

Systematic Prompting Techniques

Prompt engineering is the practice of designing inputs to elicit optimal model behavior. Advanced techniques:

1. Zero-Shot Prompting

Direct instruction without examples. Effective for well-defined tasks and capable models.

                        
Classify the sentiment of this text as positive, negative, or neutral:
"The product exceeded my expectations and arrived early."

Expected output: positive

2. Few-Shot Prompting

Provide examples to establish pattern. Crucial for complex or ambiguous tasks.

                        
Extract structured data from product descriptions.

Example 1:
Input: "Samsung 65-inch QLED 4K TV, $1,299"
Output: {"brand": "Samsung", "size": "65-inch", "type": "QLED 4K TV", "price": 1299}

Example 2:
Input: "Apple MacBook Pro M3, 16GB RAM, $2,499"
Output: {"brand": "Apple", "model": "MacBook Pro M3", "ram": "16GB", "price": 2499}

Now extract from: "Sony WH-1000XM5 Noise-Cancelling Headphones, $399"

3. Chain-of-Thought (CoT)

Elicit step-by-step reasoning to improve accuracy on complex problems.

                        
Solve this problem step by step:

A company's revenue increased by 15% in Q1, then decreased by 10% in Q2.
If the initial revenue was $1,000,000, what is the final revenue?

Let's break this down:
Step 1: Calculate Q1 revenue after 15% increase
  $1,000,000 × 1.15 = $1,150,000

Step 2: Calculate Q2 revenue after 10% decrease
  $1,150,000 × 0.90 = $1,035,000

Final answer: $1,035,000

4. Structured Output with Schema

Define exact output format using JSON schema or TypeScript interfaces.

                        
Extract information in this exact JSON format:
{
  "name": string,
  "email": string,
  "skills": string[],
  "experience_years": number,
  "remote_willing": boolean
}

Resume text: "John Doe (john@example.com) is a senior developer
with 8 years of experience in Python, TypeScript, and React.
Open to remote opportunities."

Output: {
  "name": "John Doe",
  "email": "john@example.com",
  "skills": ["Python", "TypeScript", "React"],
  "experience_years": 8,
  "remote_willing": true
}
                        
                    

5. ReAct Pattern (Reasoning + Acting)

Combine reasoning traces with action execution for agentic behavior.

                        
Task: Find the current stock price of Apple and calculate portfolio value

Thought: I need to get the current AAPL stock price first
Action: search("Apple Inc stock price AAPL current")
Observation: $178.45 per share

Thought: Now I need to know how many shares the user owns
Action: check_portfolio("AAPL")
Observation: 150 shares

Thought: I can now calculate the total value
Action: calculate(150 * 178.45)
Observation: $26,767.50

Answer: Your Apple portfolio is currently worth $26,767.50
(150 shares × $178.45/share)
                        
                    

Best Practice: Combine techniques for optimal results. Use few-shot + CoT for complex reasoning tasks, or structured output + ReAct for agentic workflows with deterministic formatting.

API Integration

OpenAI API Integration

Standard interface for GPT models with streaming, function calling, and vision capabilities.

                        
import openai
from openai import OpenAI

client = OpenAI(api_key="sk-...")

# Basic chat completion
response = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=[
        {"role": "system", "content": "You are a technical assistant."},
        {"role": "user", "content": "Explain async/await in JavaScript"}
    ],
    temperature=0.7,
    max_tokens=500,
    top_p=0.95,
    frequency_penalty=0.0,
    presence_penalty=0.0
)

print(response.choices[0].message.content)

# Streaming response
stream = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=[{"role": "user", "content": "Write a Python function"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
                        
                    

Function Calling / Tool Use

Enable models to interact with external systems through structured function calls.

                        
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City name, e.g. San Francisco"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"]
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto"
)

# Handle function call
if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]
    function_name = tool_call.function.name
    arguments = json.loads(tool_call.function.arguments)

    # Execute function
    result = get_weather(**arguments)

    # Send result back to model
    messages.append(response.choices[0].message)
    messages.append({
        "role": "tool",
        "tool_call_id": tool_call.id,
        "content": json.dumps(result)
    })
                        
                    

Anthropic Claude API

Claude API with extended context windows and multimodal capabilities.

                        
import anthropic

client = anthropic.Anthropic(api_key="sk-ant-...")

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=4096,
    temperature=0,
    system="You are a code review assistant. Analyze code for bugs and improvements.",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Review this Python function:\n\ndef process_data(items):\n    result = []\n    for i in range(len(items)):\n        if items[i] > 0:\n            result.append(items[i] * 2)\n    return result"
                }
            ]
        }
    ]
)

print(message.content[0].text)

# Vision example
with open("diagram.png", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": image_data
                }
            },
            {
                "type": "text",
                "text": "Explain this architecture diagram"
            }
        ]
    }]
)
                        
                    

Critical Parameters

Parameter	Range	Effect	Recommended
temperature	0.0 - 2.0	Higher = more random/creative	0.0-0.3 (code), 0.7-1.0 (creative)
top_p	0.0 - 1.0	Nucleus sampling threshold	0.9-0.95 (balanced diversity)
max_tokens	1 - context_limit	Maximum output length	500-2000 (typical), 4096+ (long-form)
frequency_penalty	-2.0 - 2.0	Reduce token repetition	0.0-0.5 (avoid redundancy)
presence_penalty	-2.0 - 2.0	Encourage topic diversity	0.0-0.6 (varied content)

Rate Limiting: Implement exponential backoff for rate limit errors (429). OpenAI: 10K TPM (tokens per minute) on free tier, 2M+ on paid. Claude: 50K TPM tier 1, 400K+ on tier 4. Always handle rate limits gracefully.

RAG Systems

Retrieval-Augmented Generation (RAG)

RAG combines retrieval systems with LLMs to provide grounded, factual responses using external knowledge bases. Essential for production applications requiring up-to-date or domain-specific information.

                        
RAG Pipeline:
┌─────────────┐
│   Query     │
└──────┬──────┘
       │
       v
┌─────────────────────┐
│  Embed Query        │  ← Embedding Model (e.g., text-embedding-3-large)
└──────┬──────────────┘
       │
       v
┌─────────────────────┐
│  Vector Search      │  ← Vector DB (Pinecone, Weaviate, ChromaDB)
│  Top-K Documents    │
└──────┬──────────────┘
       │
       v
┌─────────────────────┐
│  Rerank Results     │  ← Reranking Model (Cohere, cross-encoder)
└──────┬──────────────┘
       │
       v
┌─────────────────────┐
│  Augment Prompt     │  ← Inject retrieved context
└──────┬──────────────┘
       │
       v
┌─────────────────────┐
│  LLM Generation     │  ← GPT-5.6, Claude, etc.
└─────────────────────┘
                        
                    

Vector Database Implementation

Store and retrieve document embeddings using vector similarity search.

                        
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.document_loaders import DirectoryLoader

# Load documents
loader = DirectoryLoader('./docs', glob="**/*.md")
documents = loader.load()

# Split into chunks (critical for quality)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(documents)

# Create embeddings and store in vector DB
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# Retrieve relevant documents
query = "How do I configure authentication?"
relevant_docs = vectorstore.similarity_search(
    query,
    k=4,  # Top 4 results
    fetch_k=20  # Fetch 20, rerank to 4
)

# Build context for LLM
context = "\n\n".join([doc.page_content for doc in relevant_docs])
prompt = f"""Answer the question based on this context:

Context:
{context}

Question: {query}

Answer:"""
                        
                    

Advanced RAG Techniques

Hybrid Search

Combine vector similarity (semantic) with keyword search (BM25) for better recall. Weight: 0.7 vector + 0.3 keyword.

Hypothetical Document Embeddings (HyDE)

Generate hypothetical answer first, embed it, then retrieve similar real documents.

Parent-Child Chunking

Store small chunks for retrieval, but provide larger parent chunks to LLM for better context.

Reranking

Use cross-encoder models to rerank retrieved results. Cohere Rerank improves relevance by 20-30%.

Production RAG Stack

                        
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# Define custom prompt
template = """Use the following context to answer the question.
If you don't know the answer, say so - don't make up information.

Context: {context}

Question: {question}

Provide a detailed answer with sources:"""

prompt = PromptTemplate(
    template=template,
    input_variables=["context", "question"]
)

# Create retrieval chain
llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",  # Options: stuff, map_reduce, refine
    retriever=vectorstore.as_retriever(
        search_type="mmr",  # Maximum Marginal Relevance
        search_kwargs={
            "k": 4,
            "fetch_k": 20,
            "lambda_mult": 0.5  # Diversity vs relevance
        }
    ),
    return_source_documents=True,
    chain_type_kwargs={"prompt": prompt}
)

# Query
result = qa_chain({"query": "How do I implement OAuth2?"})
print(result["result"])
print(f"\nSources: {[doc.metadata for doc in result['source_documents']]}")
                        
                    

Optimization: Chunk size significantly impacts quality. Test 500-1500 tokens with 100-300 overlap. Use recursive splitting on semantic boundaries (paragraphs, sentences). Monitor retrieval precision/recall metrics.

Agentic Workflows

AI Agents: Autonomous Task Execution

AI agents are LLM-powered systems that can plan, execute actions, use tools, and iterate towards goals autonomously. They extend LLM capabilities beyond single-turn generation.

Core agent components:

Reasoning Engine: LLM for decision-making and planning
Memory: Short-term (conversation) and long-term (vector DB)
Tools: External functions (APIs, databases, calculators)
Planning: Task decomposition and execution strategies
Reflection: Self-critique and iterative improvement

ReAct Agent Implementation

ReAct (Reasoning + Acting) pattern enables agents to interleave thought and action.

                        
from langchain.agents import initialize_agent, Tool, AgentType
from langchain.chat_models import ChatOpenAI
from langchain.utilities import SerpAPIWrapper, PythonREPL

# Define tools
search = SerpAPIWrapper()
python_repl = PythonREPL()

tools = [
    Tool(
        name="Search",
        func=search.run,
        description="Search the internet for current information"
    ),
    Tool(
        name="Python",
        func=python_repl.run,
        description="Execute Python code. Input should be valid Python code."
    ),
    Tool(
        name="Calculator",
        func=lambda x: eval(x),
        description="Perform mathematical calculations"
    )
]

# Initialize agent
llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)
agent = initialize_agent(
    tools=tools,
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
    max_iterations=5,
    early_stopping_method="generate"
)

# Execute complex task
result = agent.run("""
Find the current stock price of NVIDIA, then calculate how many shares
I could buy with $10,000 and what the total value would be if the price
increases by 15%.
""")

# Agent output (example):
# Thought: I need to search for NVIDIA's current stock price
# Action: Search
# Action Input: "NVIDIA stock price today"
# Observation: $495.22 per share
#
# Thought: Now I'll calculate how many shares $10,000 can buy
# Action: Calculator
# Action Input: 10000 / 495.22
# Observation: 20.19 shares
#
# Thought: Calculate value with 15% increase
# Action: Python
# Action Input: shares = 20.19; price = 495.22; new_price = price * 1.15; ...
# Observation: $11,500.00
#
# Final Answer: With $10,000, you can buy approximately 20 shares of NVIDIA...
                        
                    

Multi-Agent Systems

Coordinate multiple specialized agents for complex workflows.

                        
from langchain.agents import AgentExecutor
from langchain.chat_models import ChatAnthropic
from langgraph.prebuilt import create_react_agent

# Define specialized agents
researcher = create_react_agent(
    ChatAnthropic(model="claude-sonnet-5"),
    tools=[search_tool, arxiv_tool],
    prompt="You are a research assistant. Find accurate technical information."
)

coder = create_react_agent(
    ChatOpenAI(model="gpt-4-turbo"),
    tools=[python_repl, file_read_tool, file_write_tool],
    prompt="You are a coding assistant. Write clean, efficient code."
)

critic = create_react_agent(
    ChatAnthropic(model="claude-sonnet-5"),
    tools=[],
    prompt="You are a code reviewer. Critique for bugs, efficiency, and best practices."
)

# Orchestration workflow
def multi_agent_workflow(task):
    # 1. Research phase
    research_result = researcher.invoke({"messages": [("user", f"Research: {task}")]})

    # 2. Implementation phase
    code_result = coder.invoke({
        "messages": [("user", f"Implement based on: {research_result}")]
    })

    # 3. Review phase
    critique = critic.invoke({
        "messages": [("user", f"Review this code:\n{code_result}")]
    })

    # 4. Refinement (if needed)
    if "issues found" in critique.lower():
        final_code = coder.invoke({
            "messages": [("user", f"Fix these issues:\n{critique}")]
        })
    else:
        final_code = code_result

    return final_code

result = multi_agent_workflow("Create a REST API for user authentication")
                        
                    

Agent Frameworks Comparison

Framework	Best For	Key Features
LangGraph	Complex stateful workflows	Graph-based execution, checkpointing, human-in-loop
AutoGPT	Autonomous goal pursuit	Self-prompting, memory management, web browsing
CrewAI	Multi-agent collaboration	Role-based agents, task delegation, shared memory
LangChain Agents	Tool-using agents	ReAct, OpenAI Functions, extensive tool integrations

Cost Management: Agentic workflows can generate 10-100× more API calls than single-turn interactions. Implement token budgets, max_iterations limits, and cost tracking. Use cheaper models (GPT-5.6 Luna, Claude Haiku) for tool selection and expensive models (GPT-5.6, Claude Sonnet) only for critical reasoning.

Fine-Tuning

When to Fine-Tune vs Prompt Engineering

Fine-tuning adapts pre-trained models to specific domains or tasks through additional training. Consider fine-tuning when:

Domain-specific terminology and patterns (legal, medical, finance)
Consistent output formatting that's hard to achieve with prompts
Need to reduce latency by embedding instructions in model weights
Cost optimization for high-volume production use
Have 500+ high-quality training examples

Stick with prompt engineering if:

Few examples available (<500)
Requirements change frequently
Task is well-handled by base model with good prompts
Need model flexibility across various tasks

Supervised Fine-Tuning (SFT)

Train on input-output pairs to specialize model behavior.

                        
# Prepare training data (JSONL format)
# Each line: {"messages": [{"role": "system", ...}, {"role": "user", ...}, {"role": "assistant", ...}]}

training_data = [
    {
        "messages": [
            {"role": "system", "content": "You are a legal contract analyzer."},
            {"role": "user", "content": "Analyze this NDA clause: ..."},
            {"role": "assistant", "content": "This is a mutual NDA with standard provisions..."}
        ]
    },
    # ... 500+ more examples
]

# OpenAI Fine-Tuning
from openai import OpenAI
client = OpenAI()

# Upload training file
file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-3.5-turbo",
    hyperparameters={
        "n_epochs": 3,
        "batch_size": 4,
        "learning_rate_multiplier": 0.1
    }
)

# Monitor training
while True:
    job_status = client.fine_tuning.jobs.retrieve(job.id)
    print(f"Status: {job_status.status}")
    if job_status.status in ["succeeded", "failed", "cancelled"]:
        break
    time.sleep(60)

# Use fine-tuned model
response = client.chat.completions.create(
    model=job_status.fine_tuned_model,
    messages=[{"role": "user", "content": "Analyze this contract..."}]
)
                        
                    

Parameter-Efficient Fine-Tuning (PEFT)

Efficient alternatives to full fine-tuning for large models.

LoRA (Low-Rank Adaptation)

Train small adapter matrices instead of full model. 0.1% parameters, 90% performance. Ideal for resource-constrained environments.

QLoRA

LoRA + 4-bit quantization. Fine-tune 65B models on single GPU. Memory efficient while maintaining quality.

Prefix Tuning

Optimize continuous prompt vectors. Fast adaptation, easy multi-task switching by swapping prefixes.

Adapter Layers

Insert small trainable layers between frozen transformer blocks. Modular and composable.

LoRA Fine-Tuning with Hugging Face

                        
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_8bit=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Configure LoRA
lora_config = LoraConfig(
    r=16,  # Rank of adaptation matrices
    lora_alpha=32,  # Scaling factor
    target_modules=["q_proj", "v_proj"],  # Which layers to adapt
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Shows only 0.8% params trainable

# Training arguments
training_args = TrainingArguments(
    output_dir="./lora-output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=lora_config,
    args=training_args,
    tokenizer=tokenizer
)

trainer.train()

# Save adapter weights only (few MB vs GB)
model.save_pretrained("./lora-adapter")
                        
                    

Data Quality: Fine-tuning quality depends heavily on data. Aim for diverse, high-quality examples. Remove duplicates, balance classes, and validate outputs. 500 excellent examples >> 5000 mediocre ones. Consider synthetic data generation using GPT-5.6 for data augmentation.

Production Deployment

Production-Ready Architecture

Deploy LLM applications with reliability, scalability, and cost efficiency.

                        
┌──────────────┐
│   Client     │
└──────┬───────┘
       │
       v
┌──────────────────────┐
│   Load Balancer      │  ← Rate limiting, request routing
└──────┬───────────────┘
       │
       v
┌──────────────────────┐
│   API Gateway        │  ← Authentication, logging, caching
└──────┬───────────────┘
       │
       v
┌──────────────────────┐
│   Application        │  ← FastAPI / Flask / Express
│   - Prompt templates │
│   - Response parsing │
│   - Error handling   │
└──────┬───────────────┘
       │
       ├─────────────────┐
       v                 v
┌──────────────┐  ┌──────────────┐
│  LLM API     │  │  Vector DB   │  ← RAG knowledge base
│  (OpenAI,    │  │  (Pinecone,  │
│   Claude)    │  │   Weaviate)  │
└──────────────┘  └──────────────┘
       │
       v
┌──────────────────────┐
│   Monitoring         │  ← Logs, metrics, tracing
│   - Latency tracking │
│   - Token usage      │
│   - Error rates      │
└──────────────────────┘
                        
                    

Caching Strategy

Reduce latency and costs with intelligent caching.

                        
import redis
import hashlib
import json

redis_client = redis.Redis(host='localhost', port=6379, db=0)

def cached_llm_call(prompt, model="gpt-4-turbo", ttl=3600):
    # Create cache key from prompt + model
    cache_key = hashlib.sha256(
        f"{model}:{prompt}".encode()
    ).hexdigest()

    # Check cache
    cached = redis_client.get(cache_key)
    if cached:
        print("Cache hit!")
        return json.loads(cached)

    # Call LLM
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )

    result = response.choices[0].message.content

    # Store in cache
    redis_client.setex(
        cache_key,
        ttl,
        json.dumps(result)
    )

    return result

# Semantic caching with embeddings
def semantic_cache_lookup(query, threshold=0.95):
    # Embed query
    query_embedding = get_embedding(query)

    # Search similar cached queries
    similar = vector_store.similarity_search_with_score(
        query_embedding,
        k=1
    )

    if similar and similar[0][1] > threshold:
        return similar[0][0].metadata["response"]

    return None
                        
                    

Error Handling & Retry Logic

                        
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import openai

@retry(
    retry=retry_if_exception_type((
        openai.RateLimitError,
        openai.APIConnectionError,
        openai.APITimeoutError
    )),
    wait=wait_exponential(multiplier=1, min=4, max=60),
    stop=stop_after_attempt(5)
)
def call_llm_with_retry(prompt):
    try:
        response = client.chat.completions.create(
            model="gpt-4-turbo",
            messages=[{"role": "user", "content": prompt}],
            timeout=30
        )
        return response.choices[0].message.content

    except openai.RateLimitError as e:
        print(f"Rate limited. Retrying... {e}")
        raise

    except openai.InvalidRequestError as e:
        # Don't retry invalid requests
        print(f"Invalid request: {e}")
        return None

    except Exception as e:
        print(f"Unexpected error: {e}")
        # Log to monitoring system
        sentry.capture_exception(e)
        raise

# Fallback to cheaper model on failure
def call_with_fallback(prompt):
    try:
        return call_llm_with_retry(prompt)
    except Exception:
        print("GPT-5.6 failed, falling back to GPT-5.6 Luna")
        return client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": prompt}]
        ).choices[0].message.content
                        
                    

Cost Optimization

Strategy	Impact	Implementation
Caching	60-90% cost reduction	Redis + semantic similarity
Model Selection	10-50× cost difference	GPT-5.6 Luna for simple, GPT-5.6 for complex
Prompt Compression	30-70% token reduction	Remove redundancy, use abbreviations
Response Streaming	Better UX, same cost	stream=True in API calls
Batch Processing	50% cost reduction	OpenAI Batch API (24h latency)

Monitoring & Observability

                        
from prometheus_client import Counter, Histogram, Gauge
import time

# Define metrics
llm_requests = Counter(
    'llm_requests_total',
    'Total LLM API requests',
    ['model', 'status']
)

llm_latency = Histogram(
    'llm_latency_seconds',
    'LLM API latency',
    ['model']
)

llm_tokens = Counter(
    'llm_tokens_total',
    'Total tokens consumed',
    ['model', 'type']  # type: prompt or completion
)

llm_cost = Counter(
    'llm_cost_usd',
    'Total LLM cost in USD',
    ['model']
)

def monitored_llm_call(prompt, model="gpt-4-turbo"):
    start = time.time()

    try:
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}]
        )

        # Track metrics
        latency = time.time() - start
        llm_latency.labels(model=model).observe(latency)
        llm_requests.labels(model=model, status="success").inc()

        # Track tokens
        prompt_tokens = response.usage.prompt_tokens
        completion_tokens = response.usage.completion_tokens
        llm_tokens.labels(model=model, type="prompt").inc(prompt_tokens)
        llm_tokens.labels(model=model, type="completion").inc(completion_tokens)

        # Track cost (GPT-5.6 pricing: $10/1M prompt, $30/1M completion)
        cost = (prompt_tokens * 0.00001) + (completion_tokens * 0.00003)
        llm_cost.labels(model=model).inc(cost)

        return response.choices[0].message.content

    except Exception as e:
        llm_requests.labels(model=model, status="error").inc()
        raise
                        
                    

Security: Never log or cache PII/sensitive data. Implement input sanitization to prevent prompt injection. Use separate API keys per environment. Rotate keys regularly. Set spending limits on API accounts. Encrypt data in transit and at rest.

Deployment Checklist

Rate limiting configured (per user, per endpoint)
Error handling with retries and fallbacks
Caching layer implemented (Redis/Memcached)
Monitoring and alerting (Prometheus/Datadog)
Cost tracking per request
Input validation and sanitization
Timeout configuration (30-60s)
Load testing completed (concurrent requests)
API keys secured (env variables, secrets manager)
Logging (structured, with request IDs)
Circuit breakers for downstream dependencies
Health check endpoints configured