andrei calapod

human-supervised ai

An AI helpdesk that only answers when it can prove it

A student assistant grounded in the official documents of a degree programme — five checks, three possible outcomes, human approval and automatic re-checking when documents change.

Client
Degree programme of a public Italian university · described anonymously
Period
In production
Role
Analysis, UX/UI, full-stack development, AI pipeline, deployment and maintenance
Stack
  • Python · FastAPI
  • Retrieval over official documents (RAG)
  • Source and number verification pipeline
  • Document re-check scheduler
  • Docker · Caddy · Linux
Helpdesk control panel showing the service diagram — a student's question goes through five checks and reaches one of three outcomes — resolved, from the documents, or referred to the office.
fig. 01 The control-panel guide — five checks, three outcomes, daily document re-checking.

Context

Students of a degree programme ask the same questions every day — enrolment, study plans, internships, labs, exams, credit recognition, exam dates. The answers exist, but they live inside regulations and documents dozens of pages long.

On these topics a wrong answer is not a small inconvenience — a wrong date or an invented requirement can cost a semester. A generic chatbot, capable of "hallucinating" plausible content, was out of the question. So the design question became a different one — how do you build an AI assistant an institution can trust?

The problem

  • Repetitive questions absorbing the time of the people who support students.
  • Answers scattered across long documents, hard to consult in a hurry.
  • Generic language models make things up — unacceptable for dates, amounts and requirements.
  • Documents change during the year — and yesterday's correct answer can become wrong today.
  • Student questions may contain personal data that must never reach an AI system.

Users and goals

Students

Fast, reliable answers with visible sources, at any hour.

Service supervisors

Approving, correcting or rejecting AI proposals from a single queue, and keeping an eye on how the service is going.

Programme offices

Receiving only personal cases and uncovered topics, not questions answered by regulations.

  • Zero invented answers — every statement must rest on a cited official document.
  • No student personal data processed by the model.
  • A person always has the last word, through a sustainable approval flow.
  • Answers that stay correct over time, even when documents change.

The process

  1. Analysis

    Mapping recurring questions and the official documents that cover them, with the people who support students daily.

  2. Pipeline design

    Definition of the five checks and three outcomes — the heart of the system is the question's journey, not the model.

  3. UX/UI

    Two interfaces — the student chat and the control panel with approval queue, questions, answers, documents and trends.

  4. Release and tuning

    Real operation under supervision, refinement of recurring cases and built-in visual documentation.

Architecture and modules

The system consists of the student chat, the supervision panel and an engine that treats every question as a case to be processed — with checks in a fixed order and a tracked outcome.

Student chat

An essential interface with suggested questions, openable citations and clear instructions about what not to enter.

To approve

The queue of pending proposals — approve, correct or reject, with answer and document side by side.

Questions and Answers

A browsable archive of what was asked and the library of approved answers.

Documents

The official sources monitored by the system, with confirmation of new versions.

Trends

Questions received, outcomes, most requested topics and student feedback.

Activity and Users

Operation tracking and access management.

The key flow

Five checks, three outcomes

Every question goes through five checks in this order. No step can be skipped, and the outcome is always one of three — tracked.

  1. 1 · Personal data

    Emails, student IDs and names are removed before any processing.

  2. 2 · Personal case

    If the question concerns an individual case, it never goes through the AI — it goes to the right office.

  3. 3 · Coverage

    If the topic is not in the documents, the system says it does not know. It never improvises.

  4. 4 · Official documents

    The answer is written by reading the regulations, citing every source.

  5. 5 · Number verification

    Dates, amounts and credits must exist in the documents — otherwise the answer is discarded.

Three possible outcomes: Resolved (a human-approved answer already existed — it arrives instantly, without AI), From the documents (a new proposal with citations, entering the approval queue) or Referred (the system does not answer and directs the student to the right office).

When a document changes

The most delicate point of a document-grounded system is time. Every day the system re-checks its sources.

  1. Daily re-check

    Sources are re-downloaded and compared with known versions.

  2. Suspension

    If a document changed, the answers depending on it are automatically suspended.

  3. Revalidation

    Answers that pass the re-check go live again; the others return to the human approval queue.

Main features

Openable citations

Every answer shows the documents it comes from, one click away.

Approve · Correct · Reject

Supervision is a three-button flow — correcting a proposal fixes and approves it in one move.

Instant answers from the already-approved

Recurring questions receive the existing approved answer, immediately and without AI.

Service trends

Volumes, outcomes, top topics and feedback — the service is measured, not assumed.

Documented recurring cases

The recurring supervision situations are explained right inside the panel.

Voice input and suggestions

Suggested questions to get started and voice input for those who prefer not to type.

Demonstrative reconstruction of the interface, with fictional data.

UX decisions

The chat declares its limits

Always visible under the input — do not enter personal data, answers do not replace official documents. Trust starts from a clear perimeter.

The panel explains the system, not just the data

The first section of the supervision panel is a visual guide to how the service works — supervisors must understand the mechanism to trust the outcomes.

Transparent outcomes

Every question has an explicit, browsable outcome — even "no answer" is a designed outcome, with the right referral.

Correcting is faster than redoing

The correction flow turns supervision from a bottleneck into a quick pass.

Security and privacy

  • Emails, student IDs and names removed before any AI processing.
  • Personal cases are never processed by the model — they go to the competent office.
  • Questions are logged anonymously to improve the service.
  • Every published answer has passed human approval.
  • Panel access with dedicated accounts; the service runs on controlled infrastructure.

Challenges and solutions

Challenge. Language models tend to produce plausible but unfounded answers.

Solution. A double constraint — retrieval over official documents only, plus separate verification of numbers. Without grounding, the system prefers not to answer.

Challenge. Documents change during the academic year, silently.

Solution. Daily re-checking with automatic suspension and revalidation of dependent answers.

Challenge. Students write in a hurry, with typos and wildly different phrasings.

Solution. Questions are normalised and matched to existing approved answers when the topic is the same.

Outcomes

  • In production for the programme's students, with active human supervision.
  • Every published answer is grounded in cited documents and approved by a person.
  • Personal cases and uncovered topics are routed to the right office instead of receiving improvised answers.

For confidentiality I do not publish service metrics; the image shows the real panel with the university's branding removed.

Possible evolutions

  • Extension to other degree programmes with separate document bases.
  • Proactive reminders on recurring academic deadlines.
  • Additional channels (a widget in the programme website) on the same pipeline.

Does this process sound familiar?

Tell me about it: the first orientation call is free of obligations.