Interview Sensei

A digital interviewer that listens, reacts, and gives you a usable report. Built as a Final Year Project under industry sponsorship at NextBridge; deployed on a self-hosted RTX 4090.

Interview Sensei: the avatar introduces the project

How a session works

You upload your resume and optionally paste a job description. The platform parses the document using OCR and NER, building a candidate context for the RAG pipeline. You then step into a session with a 3D avatar (custom-rigged in Blender with facial blendshapes) that conducts the interview.

During the session, four pipelines run in parallel:

1. Audio transcription

   Your audio stream goes to RealtimeSTT for near-real-time transcription.

2. LLM response generation

   Transcribed text plus resume context goes to Llama 3.3 70B via LangChain for job-description validation and pre-interview research, then to Qwen 2.5 7B or Gemini 2.5 Flash Lite for the interviewer’s turn-by-turn personification. The model generates the next question or follow-up based on the running interview history, with Tavily Search available for candidate research.

3. TTS and avatar lip-sync

   The LLM response is synthesized to speech via TTS and streamed to the frontend, driving the avatar’s lip-sync through audio amplitude in Three.js.

4. Computer vision analysis

   Your webcam feed runs through OpenCV + MediaPipe (posture, gestures, eye contact) and DeepFace (emotion classification) concurrently with the LLM round-trip.

All signals accumulate into a structured post-session report.

Architecture

The system follows a client-server and microservices model. The Next.js frontend establishes a WebRTC peer connection for low-latency bidirectional audio/video; a WebSocket channel carries avatar visemes, RTC signaling, and session state; a REST layer handles resume upload, feedback, and session data. FastAPI acts as the central coordinator: routing the audio stream to the STT service, video frames to the CV pipeline, and the transcribed text with RAG context to the LLM.

The avatar’s animation state is driven by two independent data streams. Audio amplitude drives lip-sync; the CV pipeline’s emotion scores update facial blendshape weights. Keeping both streams synchronized while the LLM round-trip completes was the central engineering constraint.

The avatar

The interviewer is a custom human-like 3D avatar, not a stock asset: modeled and dressed in Blender, then rigged with facial blendshapes for lip-sync and expression, and cloth-weighted so its shirt, tie, and trousers deform correctly with the underlying skeleton.

Through the product

Interview Sensei: full platform walkthrough

Post-session report

Each session generates a structured report scoring:

1. Verbal: answer relevance, vocabulary range, filler-word frequency, speaking rate
2. Non-verbal: posture consistency, eye-contact ratio, gesture frequency, emotions
3. Sentiment: per-answer sentiment curve across the full session

Reports are stored in MongoDB and accessible after the session ends.

Engineering validation

The real-time pipeline, avatar lip-sync, speech transcription, and audio streaming were benchmarked across hardware configurations and library choices before being locked into the production build. These were measured directly during development, not estimated, and shaped concrete architecture decisions.

Live transcription was tested on identical audio input across RealtimeSTT and WhisperLive: CPU-only inference ran 5 to 6 seconds behind, unusable for a live conversational agent, while an RTX 3050 Ti dev machine held it under one second. CUDA acceleration was the deciding factor for real-time viability, not an optimization; it’s why the production deployment runs on a dedicated RTX 4090 instead of a general-purpose server.