Collaborative Reflection Agent

A conversational AI agent ("Kit") designed to guide high school robotics students through weekly metacognitive reflection on their team's regulatory processes. Grounded in Self-Regulated Learning (SRL; Winne & Hadwin, 1998) and Socially-Shared Regulated Learning (SSRL; Järvelä & Hadwin, 2013), the agent helps students reflect on how their team set goals, planned, monitored progress, and adapted — in a short session after each team meeting.

Quick Start

Prerequisites

Docker and Docker Compose
Git

git clone <repo-url>
cd AgenticRoboticsEvaluator/infra
docker compose up --build

# Open the app (admin user is auto-created on first run)
open http://localhost:3000

Login: admin / admin123

Ports

Service	Port	URL
Frontend	3000	http://localhost:3000
Backend API	8000	http://localhost:8000
PostgreSQL	5433	localhost:5433

Project Overview

The Problem

High school robotics students benefit from reflecting on their teamwork, but coaches have limited time for 1:1 conversations. Students need a supportive "near-peer" they can talk to weekly after team meetings — one that focuses on how the team worked together, not just the robot.

The Solution

A chat-based AI agent that:

Guides students through a five-question SSRL reflection protocol (task/goal understanding, strategy & plan, monitoring & adaptation, shared motivation & emotion regulation, evaluation), each with signal-based follow-up branching
Uses Socratic questioning with a coordination pivot when a student describes only technical work with no teammates in the picture
Carries a loop-closure mechanism: each session opens by checking in on the prior session's action item, and closes by capturing one concrete commitment for next time
Detects and responds to safety disclosures with a fixed, deterministic reply and a dashboard-visible flag, without ever changing the session's stage
Enforces a single-question, near-peer voice with hard guardrails (no em dashes, one question mark, under 45 words) on every generated turn

Design Principles

Near-peer tone: Like a slightly older student, not a teacher or coach
Regulation-focused: The robot is context; how the team regulates their work is the subject
Hybrid neurosymbolic design: an LLM classifies each turn (Layer 1), a deterministic Python policy decides what happens next (Layer 2), and a second LLM call phrases the reply (Layer 3) — the LLM never decides whether to advance, probe, or close
No ground truth: the agent never observes the actual team meeting, only what the student reports — it never asserts a verdict on how the team did
Privacy-conscious: Minimal data collection, clear boundaries, cross-student data isolation

Current Status

The core system is fully functional with LLM integration and a dashboard UI.

Layer	Status	Description
Infrastructure	Complete	Docker Compose with PostgreSQL, backend, and frontend
Database	Complete	All tables created via Alembic migrations
Authentication	Complete	JWT-based login with role support
API	Complete	All CRUD endpoints for sessions, messages, users
LLM Integration	Complete	Any OpenAI-compatible endpoint (default: UF Navigator), JSON mode, retry logic, structured responses
Dashboard UI	Complete	Session sidebar, chat, stage progress, metadata display
Reflection Policy	Complete	Deterministic Layer 2 FSM: five core questions with signal-based branching, loop-closure, synthesis, action-item commitment
Safety Monitoring	Complete	Layer 1 detects safety disclosures on every turn; a fixed reply fires and a `SafetyIncident` row is flagged for weekly manual dashboard review — no automated notification
Reliability	Complete	Both LLM calls retry once, then degrade to a plain non-alarming fallback rather than crashing; the student's raw input is committed independently so it survives a downstream failure
Cross-Session Memory	Complete	Each session opens on the prior session's action item and can note when the same construct recurs across recent sessions
Admin dashboard / export tooling	Planned	Session inspector exists; a dedicated researcher dashboard and data-export tool are not yet built

What you can do right now:

Log in as admin or student
Start a chat session and have a real conversation focused on team regulation
Watch the agent progress through the five core reflection questions, each with adaptive follow-up probing
View Layer 1/Layer 2 classification and directive data in message metadata
Inspect any session's full transcript and metadata on the inspect page

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                           FRONTEND                                   │
│                    (Next.js 14 + TypeScript)                        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────────┐  │
│  │ Login Page  │  │  Dashboard  │  │  AuthContext (JWT storage)  │  │
│  └─────────────┘  └─────────────┘  └─────────────────────────────┘  │
│                            │                                         │
│                    /api/* proxy                                      │
└────────────────────────────┼────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│                           BACKEND                                    │
│                    (FastAPI + SQLAlchemy)                           │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                      API Routes                               │   │
│  │  /auth/*  │  /sessions/*  │  /stages  │  /admin/*  │ /health │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                            │                                         │
│                            ▼                                         │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │            FlowEngine — three-layer hybrid design              │   │
│  │                                                               │   │
│  │  Layer 1 (LLM)   reads the student's turn, emits a signal    │   │
│  │  Layer 2 (code)  decides deterministically what happens next │   │
│  │  Layer 3 (LLM)   realizes that decision as one natural turn  │   │
│  │                                                               │   │
│  │  prompts.py (FSM_STATES, BRANCH_SEQUENCES) ──► flow_engine.py │   │
│  │                        │                                      │   │
│  │                        ▼                                      │   │
│  │              llm_client.py (any OpenAI-compatible API)        │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                            │                                         │
│                            ▼                                         │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                   SQLAlchemy Models                           │   │
│  │           Student │ Session │ Message │ SafetyIncident        │   │
│  │  JSONB columns: messages.llm_metadata, sessions.evaluation_data│   │
│  └──────────────────────────────────────────────────────────────┘   │
└────────────────────────────┼────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        PostgreSQL 15                                 │
│           students │ sessions │ messages │ safety_incidents         │
└─────────────────────────────────────────────────────────────────────┘

Tech Stack

Why These Technologies?

Layer	Technology	Purpose	Why We Chose It
Frontend	Next.js 14	React framework with App Router	Server-side rendering, built-in routing, great DX
	TypeScript	Type safety	Catch errors at compile-time, better autocomplete
	Tailwind CSS	Utility-first styling	Rapid UI development, consistent design
	Axios	HTTP client	Simple API calls with interceptors for auth
Backend	FastAPI	Async Python web framework	Fast, automatic API docs, modern Python async/await
	SQLAlchemy 2.0	ORM (Object-Relational Mapper)	Write Python objects instead of SQL queries, database-agnostic
	Alembic	Database migrations	Version control for database schema changes
	Pydantic	Request/response validation	Automatic data validation and serialization
	python-jose	JWT token handling	Secure stateless authentication
	bcrypt	Password hashing	Industry-standard password security
Database	PostgreSQL 15	Relational database	ACID compliance, JSON support, scalability
Infrastructure	Docker Compose	Container orchestration	One-command setup, consistent environments

SQLAlchemy Example

db.query(Student).filter(Student.username == "admin") vs SELECT * FROM students WHERE username = 'admin' — same query, database-agnostic (switch from PostgreSQL to MySQL without code changes).

Project Structure

AgenticRoboticsEvaluator/
│
├── backend/
│   ├── app/
│   │   ├── api/
│   │   │   ├── deps.py              # Auth and DB dependency injection
│   │   │   └── routes/
│   │   │       ├── auth.py          # Login, get current user
│   │   │       ├── sessions.py      # Create sessions, chat endpoint
│   │   │       ├── stages.py        # Stage registry endpoint
│   │   │       ├── admin.py         # Admin user/session management
│   │   │       └── health.py        # Health check
│   │   │
│   │   ├── core/
│   │   │   ├── config.py            # Environment configuration
│   │   │   ├── prompts.py           # All LLM prompts and stage definitions
│   │   │   └── security.py          # JWT and password hashing
│   │   │
│   │   ├── models/
│   │   │   ├── student.py           # User model, incl. team/subteam profile fields
│   │   │   ├── session.py           # Session with evaluation_data JSONB
│   │   │   ├── message.py           # Message with llm_metadata JSONB
│   │   │   └── safety_incident.py   # Populated by the Layer 2 safety interrupt
│   │   │
│   │   ├── schemas/
│   │   │   ├── auth.py
│   │   │   ├── student.py
│   │   │   ├── session.py
│   │   │   ├── message.py
│   │   │   └── llm.py               # Layer1Output / Layer3Output schemas
│   │   │
│   │   ├── services/
│   │   │   ├── flow_engine.py       # Layer 2 deterministic policy + LLM orchestration
│   │   │   └── llm_client.py        # LLM client (any OpenAI-compatible API, default UF Navigator)
│   │   │
│   │   └── main.py
│   │
│   ├── alembic/versions/            # Full migration history — run `alembic history` for the chain
│   │
│   ├── tests/
│   ├── requirements.txt
│   ├── Dockerfile
│   └── seed_admin.py
│
├── frontend/
│   ├── src/
│   │   ├── app/
│   │   │   ├── layout.tsx
│   │   │   ├── page.tsx
│   │   │   ├── login/page.tsx
│   │   │   ├── chat/page.tsx        # Legacy chat page
│   │   │   └── dashboard/
│   │   │       ├── page.tsx         # Main dashboard with chat
│   │   │       └── [sessionId]/inspect/page.tsx  # Session inspector
│   │   │
│   │   ├── components/
│   │   │   ├── MessageCard.tsx      # Chat bubble with metadata toggle
│   │   │   ├── MetadataPanel.tsx    # LLM metadata display
│   │   │   └── StageProgressBar.tsx # Stage progress visualization
│   │   │
│   │   └── lib/
│   │       ├── api.ts
│   │       └── auth-context.tsx
│   │
│   ├── package.json
│   └── Dockerfile
│
├── infra/
│   ├── docker-compose.yml
│   └── .env
│
└── docs/
    ├── SYSTEM.md
    └── SETUP.md

Key Components Explained

FlowEngine — the three-layer hybrid design

Located in backend/app/services/flow_engine.py. Each turn:

Layer 1 (_run_layer1) — an LLM call classifies the student's turn against LAYER1_SYSTEM_PROMPT: pragmatic stance (cooperative / seeking clarification / active resistance / superficial compliance), signal type (thin / breakdown / success / no_event), SSRL evaluation, and a broad safety-concern flag. Retries once on failure, then falls back to an inert classification rather than crashing.
Layer 2 (_decide) — pure Python. Checks, in priority order: a safety disclosure (highest priority, never changes the session's stage — see below), a pending safety check-in resume, synthesis/commit stage handling, loop-closure, universal interrupts (clarification, active resistance, topic drift), the Q1-Q3 Part A → Part B two-beat exchange, a coordination pivot (technical-only answers get redirected to ask about the team), then the shared branching protocol (BRANCH_SEQUENCES/BRANCH_STEPS in prompts.py) gated by a session-wide probe budget.
Layer 3 (_run_layer3) — an LLM call realizes the chosen directive as one short turn, constrained by LAYER3_SYSTEM_PROMPT (no ground-truth verdicts, construct lock, one question mark, under 45 words, no em dashes). Retries once, then falls back to a plain non-alarming line.

Two turns are generated deterministically with no LLM call at all: the opening greeting/loop-closure and the closing goodbye.

Prompt Registry

Located in backend/app/core/prompts.py. Single source of truth for the reflection protocol:

LAYER1_SYSTEM_PROMPT / LAYER3_SYSTEM_PROMPT: the NLU classifier and NLG realizer instructions
FSM_STATES: the five core questions (Q1-Q5) plus Stage 0 (loop-closure), Stage 3 (synthesis), Stage 4 (action commit), Stage 5 (close) — Q1-Q3 each split into a guaranteed Part A/Part B two-beat exchange
BRANCH_SEQUENCES / BRANCH_STEPS: the shared thin/breakdown/success/no_event follow-up protocol applied after each core question
SESSION_PROBE_BUDGET / CUMULATIVE_PROBE_CAP: keeps a full session in a bounded turn range across a semester of weekly use
Safety interrupt copy (SAFETY_TRUSTED_ADULT_LINE, SAFETY_CHECKIN_LINE) — fixed templates, never LLM-improvised

LLM Client

Located in backend/app/services/llm_client.py. Wraps any OpenAI-compatible API (default: UF Navigator) with:

JSON mode plus a schema appended to the system prompt to structure responses
A repair-retry loop: on invalid JSON or schema mismatch, re-prompts once with a repair instruction before giving up
LLMResult object with token usage, response time, and attempt count

flow_engine.py layers a second, coarser retry on top of this at both Layer 1 and Layer 3 call sites — if the client still fails after its own retries, the turn degrades to a fallback response rather than propagating an error to the student.

The Reflection Protocol (Stage 0-5, plus a safety interrupt)

Stage 0  Loop-closure     — opens on last session's action item, if one exists
Stage 1  Orientation      — explicit-criteria framing, fades across the semester
Stage 2  Core reflection  — Q1 Task & Goal -> Q2 Strategy & Plan -> Q3 Monitoring & Adaptation
                             -> Q4 Shared Motivation & Emotion -> Q5 Evaluation
                             (Q1-Q3 each ask a guaranteed Part A, then Part B; each question
                             gets adaptive follow-up branching based on the student's signal)
Stage 3  Synthesis        — mirror back what was heard, ask for one forward-planning commitment
Stage 4  Action commit    — restate the ONE action item, open the floor for anything else
Stage 5  Close             — deterministic, never a question

A safety interrupt can fire on any turn, at any stage, the instant Layer 1 reports a safety concern. It is not a stage — the session's current_stage never changes. Instead a fixed reply (a trusted-adult line plus a check-in question) is delivered deterministically, a SafetyIncident row is flagged for the researcher's weekly manual dashboard review, and the interrupted question is re-asked once the student is ready to continue.

Authentication Flow

1. User submits username/password to POST /auth/login
2. Backend validates credentials, returns JWT token
3. Frontend stores token in localStorage
4. All subsequent requests include Authorization: Bearer <token>
5. Backend validates token on each request via dependency injection

Database Models

Model	Table	Purpose	Status
Student	students	Users, both students and admins; also carries team/subteam profile fields	Used
Session	sessions	Chat session with stage tracking and evaluation_data (action item + recurring constructs)	Used
Message	messages	Individual messages with llm_metadata (Layer 1/2/3 audit trail)	Used
SafetyIncident	safety_incidents	Flagged safety disclosures, reviewed weekly by the researcher	Used

What's Next

These are the logical next steps:

Validation-export tool — Export the load-bearing Layer1Output fields (signal_type, has_social_coordination, is_team_regulation_behavior, proposed_action_item) for human-coding validation before a study begins. Field scope is decided (marked # LLM-CODED in schemas/llm.py); the export tool itself isn't built yet.
LLM classifier evaluation — Measure Layer 1's classification accuracy against human-coded reference labels before relying on it for study data.

Development Guide

Running the Application

# Start all services
cd infra
docker compose up

# Start with rebuild (after code changes to Dockerfile)
docker compose up --build

# Stop all services
docker compose down

# Stop and remove volumes (clears database)
docker compose down -v

Viewing Logs

# All services
docker compose logs -f

# Specific service
docker compose logs -f backend
docker compose logs -f frontend
docker compose logs -f postgres

Running Tests

docker compose exec backend pytest -v

Database Access

# Connect to PostgreSQL
docker compose exec postgres psql -U evaluator -d evaluator

# Common queries
SELECT * FROM students;
SELECT * FROM sessions;
SELECT * FROM messages ORDER BY created_at DESC LIMIT 10;

API Documentation

When the backend is running, visit:

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

Hot Reloading

Both frontend and backend support hot reloading:

Backend: Changes to Python files auto-restart uvicorn
Frontend: Next.js fast refresh on file save

Documentation

Document	Description
SYSTEM.md	Complete technical specification with data models, API contracts, and architecture decisions
SETUP.md	Detailed setup instructions with troubleshooting

Contributing

Create a feature branch from main
Make changes with clear commit messages
Ensure tests pass: docker compose exec backend pytest
Submit a pull request

License

MIT

Navigation Menu

Repository files navigation

Collaborative Reflection Agent

Quick Start

Prerequisites

Ports

Table of Contents

Project Overview

The Problem

The Solution

Design Principles

Current Status

Architecture

Tech Stack

Why These Technologies?

SQLAlchemy Example

Project Structure

Key Components Explained

FlowEngine — the three-layer hybrid design

Prompt Registry

LLM Client

The Reflection Protocol (Stage 0-5, plus a safety interrupt)

Authentication Flow

Database Models

What's Next

Development Guide

Running the Application

Viewing Logs

Running Tests

Database Access

API Documentation

Hot Reloading

Documentation

Contributing

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
backend		backend
docs		docs
frontend		frontend
infra		infra
.gitignore		.gitignore
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

Collaborative Reflection Agent

Quick Start

Prerequisites

Ports

Table of Contents

Project Overview

The Problem

The Solution

Design Principles

Current Status

Architecture

Tech Stack

Why These Technologies?

SQLAlchemy Example

Project Structure

Key Components Explained

FlowEngine — the three-layer hybrid design

Prompt Registry

LLM Client

The Reflection Protocol (Stage 0-5, plus a safety interrupt)

Authentication Flow

Database Models

What's Next

Development Guide

Running the Application

Viewing Logs

Running Tests

Database Access

API Documentation

Hot Reloading

Documentation

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages