← Projects · Agentic AI · GenAI

Grounded Training Content Generation with Human Review

Vertex AI Agent Engine · Gemini 2.5 Flash · Google ADK · Vertex AI Search · Firestore · Cloud Storage · Pydantic

A multi-agent platform that turns source documents into reviewed training content, grounded in retrieved material with source attribution for every question, and built so that nothing reaches learners without explicit human approval.

Problem

Subject-matter experts were writing large volumes of training questions by hand, which was slow and inconsistent. The goal was to let experts upload source documents in common formats and web pages, have an AI pipeline generate grounded content, and never let unreviewed AI output reach the live question bank.

My role

I was the sole developer and built the platform end to end: 14 agents behind a single registry, shipped as one Vertex AI Agent Engine application that is the only ingress. Adding an agent is one file and one registry line.

Key design decisions & trade-offs

  • Human approval is structural, not procedural. There is no code path that publishes generated content automatically. Reviewers approve, reject, edit or adjust tone, and every edit is reversible through version history.
  • The LLM writes content; Python makes decisions. IDs, statuses, scores, coverage classifications and pass or fail gates are set deterministically. The model writes questions and explanations but never controls state.
  • A generate, critique, attribute loop. Each question is generated from retrieved context, self-critiqued, regenerated once if needed, and attributed to its sources. Creative steps use a higher temperature; grading and attribution steps use a low one.
  • A structured-output layer that works within Vertex limits. Vertex structured output does not accept every JSON Schema construct, so Pydantic models are flattened into a compatible schema, and invalid responses are retried with the validation error fed back.
  • One service layer for API and chat. The admin chat assistant is built with Google ADK and Gemini function calling. The REST API and the admin chat assistant call the same service methods, so chat and API share one set of business logic. The trade-off is that the two paths cannot scale independently.
  • Retrieval with a fallback. Vertex AI Search is the primary retriever, returning ids and scores that are hydrated from Firestore, with reranking server-side. The fallback is two-tier: a Firestore nearest-neighbour cosine query scoped by topic key, then a capped scan with in-memory cosine for chunks that predate topic-key tagging. A Search outage degrades retrieval rather than failing it.

What I would do next

Add faithfulness evaluation of generated questions against their sources, and measure reviewer acceptance and edit rates as the production quality signal.

Employer and client work is described at a general level. Client identities, internal figures and confidential details are omitted.