Automated UI-Action Generation for RAG Chatbot

AI Python Azure OpenAI FastAPI JavaScript Node.js
Back to Portfolio Home

Automated UI-Action Generation for RAG Chatbot — Defense Client (Switzerland)

Invented a novel UI-tree capture pipeline that generated 715 reliable chatbot actions in one month — where months of manual effort had produced only 3.

Show more ↓

1. Problem

The RAG chatbot (see Milo AI project) needed structured action data describing every workflow in a feature-rich HR & logistics web portal. Documentation was sparse and outdated, leaving the chatbot with too few reliable answers. The team had spent months brainstorming approaches and only managed to hand-craft 3 actions.

2. Solution Overview

I conceived and solely built a two-stage pipeline that captures live UI interactions and transforms them into localized action files for RAG ingestion:

Component Purpose Key Tech
Web-UI-Tree-Builder Browser bookmarklet records UI clicks and builds a hierarchical action tree JavaScript · Node.js · FastAPI
Tree Host Server Persists captured trees as JSONL, serves a live tree viewer FastAPI · Docker
Action Generation Extracts paths, generates AI summaries, adds descriptions & keywords Python · Azure OpenAI
Localization Translates actions and summaries into French & Italian Azure OpenAI · FastAPI endpoints

3. Impact

  • Generated 715 structured actions covering the entire web portal — up from 3 manual actions
  • Delivered the complete action set within one month, beating a year-long timeline planned for 2026
  • Customer was "excited and surprised" by the coverage and quality
  • Directly improved chatbot answer accuracy by providing reliable, structured data
  • Received a salary raise in recognition of the contribution

4. My Contributions

Conceived the entire UI-tree capture approach with no prior art found online; developed the bookmarklet, tree host server and action generation pipeline from scratch; manually clicked through the full web app UI within one week to build the action tree; championed the idea against skeptical project leads who doubted feasibility.

5. Key Challenges & Mitigations

Challenge Mitigation & Result
No existing tools or approaches for UI-action extraction Researched extensively, found no prior art → designed and built custom toolchain from scratch
Skeptical project leadership Delivered working prototype quickly → results spoke for themselves
Content-Security-Policy blocking bookmarklet injection Browser extension to disable CSP during capture sessions
Multilingual requirement (DE/FR/IT) Automated LLM-based translation pipeline → full trilingual action set

6. Data Flow

MILO5 Web UI → Bookmarklet (Alt+Click) → Tree Host Server → JSONL
    ↓
JSONL → Path Extraction → AI Summaries → Localization → Action JSON
    ↓
Final Action Files → Milo AI RAG Ingestion → Chatbot Answers