🚀

We've launched on Product Hunt

Check us out →
Tutorials

Building a Support Knowledge Base Agent: From Documentation to Answers

H

Hoshang Mehta

Building a Support Knowledge Base Agent: From Documentation to Answers

Support teams have documentation scattered across wikis, FAQs, and internal docs. Finding the right answer takes too long. Documentation is outdated. Agents can't quickly surface relevant solutions for similar tickets. The result: support agents spend 10 minutes searching for answers, customers wait longer, and knowledge doesn't get reused.

The problem: A customer asks "How do I export my data?" The support agent searches the wiki, checks the FAQ, looks through internal docs. After 8 minutes, they find a relevant article but it's outdated. They piece together an answer from multiple sources. The customer is frustrated. The next time someone asks the same question, the agent starts from scratch.

The solution: A knowledge base agent that indexes all support documentation and resolved tickets, uses semantic search to find relevant answers, generates responses with confidence scores, and learns from resolutions. This post walks through building a repeatable architecture that transforms scattered documentation into an intelligent assistant.

The Problem with Scattered Support Documentation

Support documentation lives everywhere. Here's what agents face:

The Search Problem

Documentation is scattered across:

  • Internal Wiki: Product guides, troubleshooting steps
  • FAQ Pages: Common questions, quick answers
  • Internal Docs: Team-specific procedures, escalation paths
  • Resolved Tickets: Solutions that worked, but aren't documented
  • Product Docs: Feature documentation, API references

The math: If searching each source takes 2 minutes and you handle 30 questions per day, that's 60 minutes (1 hour) just searching—before answering any questions.

The Outdated Content Problem

Documentation gets stale:

  • Product features change, but docs aren't updated
  • Procedures change, but old docs remain
  • Solutions that worked last month no longer apply
  • Links break, screenshots become outdated

The impact: Agents find documentation, but it's wrong. They waste time following outdated steps, then have to figure out the correct process.

The Knowledge Loss Problem

Solutions exist in resolved tickets, but they're not accessible:

  • A ticket was resolved 3 months ago with a perfect solution
  • The same issue comes up again, but the agent doesn't know about the previous solution
  • Knowledge is trapped in ticket history, not reusable

The missed opportunity: 60% of support questions have been answered before, but agents can't find those answers.

The Context Problem

Documentation lacks context:

  • A solution works for one customer tier, but not another
  • Steps vary by product version
  • Procedures depend on customer configuration
  • Documentation doesn't explain when to use which solution

The cost: Agents apply solutions that don't fit the context, causing escalations and customer frustration.

A Repeatable Knowledge Base Architecture

The solution is a pipeline architecture that indexes documentation, embeds content for semantic search, retrieves relevant answers, and generates responses with context.

graph TD A[Documentation Sources] --> B[Content Ingestion] C[Resolved Tickets] --> B B --> D[Text Extraction] D --> E[Embedding Generation] E --> F[Vector Store] G[User Query] --> H[Query Embedding] H --> I[Semantic Search] I --> F F --> J[Retrieval] J --> K[Answer Generation] K --> L[Confidence Scoring] L --> M[Response to Agent] N[Resolution Feedback] --> O[Learning Loop] O --> F style E fill:#e1f5ff style I fill:#fff4e1 style K fill:#e8f5e9

Architecture Components

1. Content Ingestion Layer

  • Connects to documentation sources (wikis, FAQs, docs)
  • Captures resolved tickets from support system
  • Extracts text from various formats (Markdown, HTML, PDF)
  • Normalizes content structure

2. Embedding Generation

  • Converts text to vector embeddings using LLMs
  • Chunks content for optimal retrieval
  • Generates metadata (source, date, relevance tags)
  • Stores embeddings in vector database

3. Semantic Search Engine

  • Embeds user queries
  • Searches vector store for similar content
  • Ranks results by relevance
  • Returns top-k most relevant documents

4. Answer Generation

  • Combines retrieved documents with query
  • Generates natural language response
  • Includes source citations
  • Formats for agent consumption

5. Learning Loop

  • Tracks which answers agents use
  • Collects resolution feedback
  • Updates relevance scores
  • Improves retrieval over time

Indexing Documentation and Resolved Tickets

The first stage is ingesting and indexing all documentation sources. This includes wikis, FAQs, internal docs, and resolved tickets.

Content Ingestion Process

1. Connect to Documentation Sources

-- Documentation sources table
CREATE TABLE knowledge_base_sources (
    source_id UUID PRIMARY KEY,
    source_name VARCHAR(255) NOT NULL,
    source_type VARCHAR(50) NOT NULL, -- 'wiki', 'faq', 'docs', 'tickets'
    connection_config JSONB,
    last_synced_at TIMESTAMP,
    sync_frequency VARCHAR(50) -- 'realtime', 'hourly', 'daily'
);

-- Example sources
INSERT INTO knowledge_base_sources VALUES
    ('wiki-001', 'Internal Wiki', 'wiki', '{"url": "https://wiki.company.com", "api_key": "..."}', NULL, 'hourly'),
    ('faq-001', 'Public FAQ', 'faq', '{"url": "https://faq.company.com", "format": "markdown"}', NULL, 'daily'),
    ('docs-001', 'Product Docs', 'docs', '{"url": "https://docs.company.com", "format": "markdown"}', NULL, 'daily'),
    ('tickets-001', 'Resolved Tickets', 'tickets', '{"system": "zendesk", "status": "resolved"}', NULL, 'realtime');

2. Extract and Normalize Content

-- Knowledge base content table
CREATE TABLE knowledge_base_content (
    content_id UUID PRIMARY KEY,
    source_id UUID REFERENCES knowledge_base_sources(source_id),
    title VARCHAR(500),
    content_text TEXT,
    content_type VARCHAR(50), -- 'article', 'faq', 'ticket', 'procedure'
    metadata JSONB, -- tags, categories, product versions, etc.
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    is_active BOOLEAN DEFAULT true
);

-- Extract content from sources
CREATE OR REPLACE FUNCTION ingest_documentation()
RETURNS void AS $$
DECLARE
    source_record RECORD;
BEGIN
    FOR source_record IN SELECT * FROM knowledge_base_sources LOOP
        -- Extract content based on source type
        CASE source_record.source_type
            WHEN 'wiki' THEN
                -- Extract from wiki API
                INSERT INTO knowledge_base_content (source_id, title, content_text, content_type, metadata)
                SELECT 
                    source_record.source_id,
                    wiki.title,
                    wiki.content,
                    'article',
                    jsonb_build_object('category', wiki.category, 'tags', wiki.tags)
                FROM wiki_api_get_pages() wiki;
                
            WHEN 'faq' THEN
                -- Extract from FAQ pages
                INSERT INTO knowledge_base_content (source_id, title, content_text, content_type, metadata)
                SELECT 
                    source_record.source_id,
                    faq.question,
                    faq.answer,
                    'faq',
                    jsonb_build_object('category', faq.category)
                FROM faq_api_get_items() faq;
                
            WHEN 'tickets' THEN
                -- Extract from resolved tickets
                INSERT INTO knowledge_base_content (source_id, title, content_text, content_type, metadata)
                SELECT 
                    source_record.source_id,
                    ticket.subject,
                    ticket.resolution_notes || E'\n\n' || ticket.description,
                    'ticket',
                    jsonb_build_object(
                        'ticket_id', ticket.id,
                        'customer_tier', ticket.customer_tier,
                        'product_version', ticket.product_version,
                        'resolution_date', ticket.resolved_at
                    )
                FROM support_tickets ticket
                WHERE ticket.status = 'resolved'
                  AND ticket.resolution_notes IS NOT NULL
                  AND ticket.resolved_at > CURRENT_TIMESTAMP - INTERVAL '90 days';
        END CASE;
        
        -- Update sync timestamp
        UPDATE knowledge_base_sources
        SET last_synced_at = CURRENT_TIMESTAMP
        WHERE source_id = source_record.source_id;
    END LOOP;
END;
$$ LANGUAGE plpgsql;

Content Chunking Strategy

Break long documents into chunks for better retrieval:

-- Content chunks table
CREATE TABLE knowledge_base_chunks (
    chunk_id UUID PRIMARY KEY,
    content_id UUID REFERENCES knowledge_base_content(content_id),
    chunk_text TEXT,
    chunk_index INTEGER,
    embedding VECTOR(1536), -- OpenAI embedding dimension
    metadata JSONB,
    created_at TIMESTAMP
);

-- Chunk content into 500-token pieces with 50-token overlap
CREATE OR REPLACE FUNCTION chunk_content()
RETURNS void AS $$
DECLARE
    content_record RECORD;
    chunk_text TEXT;
    chunk_index INTEGER := 0;
BEGIN
    FOR content_record IN 
        SELECT * FROM knowledge_base_content 
        WHERE content_text IS NOT NULL
    LOOP
        -- Split content into chunks (simplified - use proper tokenization in production)
        chunk_index := 0;
        FOR chunk_text IN 
            SELECT unnest(string_to_array(
                regexp_split_to_string(content_record.content_text, E'\\n\\n', 'g'),
                E'\\n\\n'
            ))
            WHERE length(chunk_text) > 100 -- Minimum chunk size
        LOOP
            INSERT INTO knowledge_base_chunks (
                content_id, 
                chunk_text, 
                chunk_index, 
                metadata
            ) VALUES (
                content_record.content_id,
                chunk_text,
                chunk_index,
                jsonb_build_object(
                    'title', content_record.title,
                    'source', content_record.source_id,
                    'content_type', content_record.content_type
                )
            );
            chunk_index := chunk_index + 1;
        END LOOP;
    END LOOP;
END;
$$ LANGUAGE plpgsql;

Semantic Search and Retrieval

Once content is indexed, implement semantic search to find relevant answers. Use vector embeddings to match queries to content.

Embedding Generation

Generate embeddings for content chunks:

-- Generate embeddings using OpenAI API (via function)
CREATE OR REPLACE FUNCTION generate_embedding(text_content TEXT)
RETURNS VECTOR(1536) AS $$
DECLARE
    embedding_result JSONB;
BEGIN
    -- Call OpenAI embedding API
    SELECT openai_embedding_api_call(text_content) INTO embedding_result;
    
    -- Convert JSON array to vector
    RETURN (embedding_result->>'embedding')::vector;
END;
$$ LANGUAGE plpgsql;

-- Generate embeddings for all chunks
UPDATE knowledge_base_chunks
SET embedding = generate_embedding(chunk_text)
WHERE embedding IS NULL;

Semantic Search Query

Search for relevant content using vector similarity:

-- Semantic search function
CREATE OR REPLACE FUNCTION search_knowledge_base(
    query_text TEXT,
    top_k INTEGER DEFAULT 5,
    min_similarity FLOAT DEFAULT 0.7
)
RETURNS TABLE (
    chunk_id UUID,
    content_id UUID,
    chunk_text TEXT,
    title TEXT,
    source_name TEXT,
    similarity FLOAT,
    metadata JSONB
) AS $$
DECLARE
    query_embedding VECTOR(1536);
BEGIN
    -- Generate embedding for query
    query_embedding := generate_embedding(query_text);
    
    -- Search for similar chunks
    RETURN QUERY
    SELECT 
        kb.chunk_id,
        kb.content_id,
        kb.chunk_text,
        kb.metadata->>'title' as title,
        ks.source_name,
        1 - (kb.embedding <=> query_embedding) as similarity,
        kb.metadata
    FROM knowledge_base_chunks kb
    JOIN knowledge_base_content kbc ON kb.content_id = kbc.content_id
    JOIN knowledge_base_sources ks ON kbc.source_id = ks.source_id
    WHERE kbc.is_active = true
      AND 1 - (kb.embedding <=> query_embedding) >= min_similarity
    ORDER BY kb.embedding <=> query_embedding
    LIMIT top_k;
END;
$$ LANGUAGE plpgsql;

Search Flow

graph LR A[User Query] --> B[Query Embedding] B --> C[Vector Search] C --> D[Similarity Ranking] D --> E[Top-K Results] E --> F[Context Assembly] F --> G[Answer Generation] style B fill:#e1f5ff style C fill:#fff4e1 style G fill:#e8f5e9

Answer Generation and Confidence Scoring

Combine retrieved documents with the user query to generate natural language answers. Score confidence based on relevance and source quality.

Answer Generation Process

1. Retrieve Relevant Content

-- Retrieve top-k relevant chunks
SELECT * FROM search_knowledge_base(
    query_text := 'How do I export my data?',
    top_k := 5,
    min_similarity := 0.7
);

2. Generate Answer with LLM

def generate_answer(query: str, retrieved_chunks: list) -> dict:
    # Assemble context from retrieved chunks
    context = "\n\n".join([
        f"Source: {chunk['source_name']}\n{chunk['chunk_text']}"
        for chunk in retrieved_chunks
    ])
    
    # Generate answer using LLM
    prompt = f"""Based on the following documentation, answer the user's question.

Documentation:
{context}

Question: {query}

Provide a clear, concise answer. Include source citations."""
    
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3
    )
    
    answer = response.choices[0].message.content
    
    # Calculate confidence score
    confidence = calculate_confidence(retrieved_chunks, answer)
    
    return {
        "answer": answer,
        "confidence": confidence,
        "sources": [chunk['source_name'] for chunk in retrieved_chunks],
        "similarity_scores": [chunk['similarity'] for chunk in retrieved_chunks]
    }

3. Confidence Scoring

def calculate_confidence(chunks: list, answer: str) -> float:
    # Factors affecting confidence:
    # 1. Average similarity score of retrieved chunks
    avg_similarity = sum(chunk['similarity'] for chunk in chunks) / len(chunks)
    
    # 2. Number of relevant sources
    source_diversity = len(set(chunk['source_name'] for chunk in chunks))
    diversity_score = min(source_diversity / 3, 1.0)  # Normalize to 0-1
    
    # 3. Recency of content (newer = more confident)
    recent_score = calculate_recency_score(chunks)
    
    # 4. Answer completeness (check if answer addresses query)
    completeness_score = check_answer_completeness(answer, chunks)
    
    # Weighted confidence score
    confidence = (
        avg_similarity * 0.4 +
        diversity_score * 0.2 +
        recent_score * 0.2 +
        completeness_score * 0.2
    )
    
    return min(confidence, 1.0)

MCP Tool for Knowledge Base

{
  "name": "search_knowledge_base",
  "description": "Searches the knowledge base for answers to support questions. Returns relevant documentation, FAQs, and solutions from resolved tickets. Use when customers ask how-to questions, troubleshooting questions, or need help with features.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "The question or topic to search for"
      },
      "content_type": {
        "type": "string",
        "enum": ["all", "article", "faq", "ticket", "procedure"],
        "description": "Filter by content type. Defaults to 'all'.",
        "default": "all"
      },
      "min_confidence": {
        "type": "number",
        "description": "Minimum confidence score (0-1). Defaults to 0.7.",
        "minimum": 0,
        "maximum": 1,
        "default": 0.7
      },
      "max_results": {
        "type": "integer",
        "description": "Maximum number of results to return",
        "minimum": 1,
        "maximum": 10,
        "default": 5
      }
    },
    "required": ["query"]
  },
  "query": "SELECT * FROM search_knowledge_base(query_text := $1, top_k := $4, min_similarity := $3)",
  "policies": ["authenticated", "role:support"]
}

Continuous Learning from Resolutions

The knowledge base improves over time by learning from which answers agents use and how customers respond.

Learning Loop Architecture

graph TD A[Agent Uses Answer] --> B[Track Usage] B --> C[Collect Feedback] C --> D{Feedback Type} D -->|Positive| E[Increase Relevance] D -->|Negative| F[Decrease Relevance] D -->|Resolution| G[Update Content] E --> H[Update Embeddings] F --> H G --> I[Re-index Content] I --> H style B fill:#e1f5ff style H fill:#fff4e1

Usage Tracking

-- Track knowledge base usage
CREATE TABLE knowledge_base_usage (
    usage_id UUID PRIMARY KEY,
    query_text TEXT,
    chunk_id UUID REFERENCES knowledge_base_chunks(chunk_id),
    agent_id VARCHAR(255),
    ticket_id UUID,
    used_at TIMESTAMP,
    feedback_score INTEGER, -- 1-5 rating
    was_helpful BOOLEAN,
    resolution_notes TEXT
);

-- Update relevance scores based on usage
CREATE OR REPLACE FUNCTION update_relevance_scores()
RETURNS void AS $$
BEGIN
    -- Increase relevance for frequently used, highly-rated content
    UPDATE knowledge_base_chunks
    SET metadata = jsonb_set(
        COALESCE(metadata, '{}'::jsonb),
        '{relevance_score}',
        to_jsonb(
            COALESCE((metadata->>'relevance_score')::float, 0.5) +
            (SELECT COUNT(*) * 0.1 FROM knowledge_base_usage 
             WHERE chunk_id = knowledge_base_chunks.chunk_id 
               AND was_helpful = true) -
            (SELECT COUNT(*) * 0.05 FROM knowledge_base_usage 
             WHERE chunk_id = knowledge_base_chunks.chunk_id 
               AND was_helpful = false)
        )
    )
    WHERE chunk_id IN (
        SELECT DISTINCT chunk_id FROM knowledge_base_usage
        WHERE used_at > CURRENT_TIMESTAMP - INTERVAL '30 days'
    );
END;
$$ LANGUAGE plpgsql;

Content Updates from Resolutions

When tickets are resolved, extract new knowledge:

-- Extract knowledge from resolved tickets
CREATE OR REPLACE FUNCTION extract_knowledge_from_resolution()
RETURNS void AS $$
BEGIN
    INSERT INTO knowledge_base_content (
        source_id,
        title,
        content_text,
        content_type,
        metadata
    )
    SELECT 
        (SELECT source_id FROM knowledge_base_sources WHERE source_type = 'tickets'),
        ticket.subject || ' - Resolution',
        ticket.resolution_notes,
        'ticket',
        jsonb_build_object(
            'ticket_id', ticket.id,
            'customer_tier', ticket.customer_tier,
            'product_version', ticket.product_version,
            'resolution_date', ticket.resolved_at,
            'satisfaction_score', ticket.satisfaction_score
        )
    FROM support_tickets ticket
    WHERE ticket.status = 'resolved'
      AND ticket.resolution_notes IS NOT NULL
      AND ticket.resolved_at > CURRENT_TIMESTAMP - INTERVAL '7 days'
      AND NOT EXISTS (
          SELECT 1 FROM knowledge_base_content
          WHERE metadata->>'ticket_id' = ticket.id::text
      );
    
    -- Re-chunk new content
    PERFORM chunk_content();
    
    -- Generate embeddings for new chunks
    UPDATE knowledge_base_chunks
    SET embedding = generate_embedding(chunk_text)
    WHERE embedding IS NULL;
END;
$$ LANGUAGE plpgsql;

Schema Overview

The knowledge base architecture uses a hierarchical schema that traces from source content to generated answers.

erDiagram KNOWLEDGE_BASE_SOURCES ||--o{ KNOWLEDGE_BASE_CONTENT : contains KNOWLEDGE_BASE_CONTENT ||--o{ KNOWLEDGE_BASE_CHUNKS : chunks_into KNOWLEDGE_BASE_CHUNKS ||--o{ KNOWLEDGE_BASE_USAGE : used_in SUPPORT_TICKETS ||--o{ KNOWLEDGE_BASE_CONTENT : generates KNOWLEDGE_BASE_SOURCES { uuid source_id PK string source_name string source_type jsonb connection_config timestamp last_synced_at } KNOWLEDGE_BASE_CONTENT { uuid content_id PK uuid source_id FK string title text content_text string content_type jsonb metadata timestamp created_at } KNOWLEDGE_BASE_CHUNKS { uuid chunk_id PK uuid content_id FK text chunk_text vector embedding integer chunk_index jsonb metadata } KNOWLEDGE_BASE_USAGE { uuid usage_id PK uuid chunk_id FK text query_text integer feedback_score boolean was_helpful timestamp used_at }

Key relationships:

  • knowledge_base_sources → Documentation sources (wikis, FAQs, tickets)
  • knowledge_base_content → Extracted and normalized content
  • knowledge_base_chunks → Chunked content with embeddings
  • knowledge_base_usage → Usage tracking and feedback

Traceability: Every answer can be traced back to source content, retrieval process, and usage feedback.

Closing Thoughts

Building a support knowledge base agent transforms documentation from static text into an intelligent assistant. The key is treating knowledge as a searchable, learnable resource—not a static collection.

Key takeaways:

  1. Indexing is foundational: Extract content from all sources, normalize formats, chunk for optimal retrieval. Comprehensive indexing makes search possible.

  2. Semantic search enables discovery: Vector embeddings find relevant content even when keywords don't match. This is critical for natural language queries.

  3. Confidence scoring builds trust: Agents need to know when answers are reliable. Score confidence based on relevance, source quality, and recency.

  4. Learning improves over time: Track which answers work, collect feedback, update relevance. The knowledge base gets better with use.

  5. Integration with tickets creates a loop: Resolved tickets become new knowledge. Extract solutions, index them, make them searchable.

The architecture described here is repeatable. Start with one documentation source, build the indexing pipeline, implement semantic search, then add more sources incrementally.

Most importantly: this isn't about replacing human knowledge—it's about making knowledge accessible. Support agents still need expertise and judgment. The knowledge base agent provides instant access to documented solutions, freeing agents to handle complex, novel issues.

This architecture pattern is implemented in Pylar, but the concepts apply to any system building a knowledge base. The key is the indexing → embedding → retrieval → generation → learning pipeline.