Projects Nov 2024 to Apr 2025 Lead engineer

Interview Sensei

AI mock interview platform with a 3D digital-human interviewer, real-time WebRTC, and a post-session report. FYP at FAST NUCES; first place at Job Fair 2025 and SOFTEC 2025.

  • Next.js
  • FastAPI
  • Three.js
  • WebRTC
  • LangChain
  • Llama 3.3 70B
  • OpenCV
  • MediaPipe
  • DeepFace
  • MongoDB

A digital interviewer that listens, reacts, and gives you a usable report. Built as a Final Year Project under industry sponsorship at NextBridge; deployed on a self-hosted RTX 4090.

1st

Job Fair 2025

First DS team in FAST history

1st

SOFTEC 2025

3

Team members

How a session works

You upload your resume and optionally paste a job description. The platform parses the document using OCR and NER, building a candidate context for the RAG pipeline. You then step into a session with a 3D avatar (custom-rigged in Blender with facial blendshapes) that conducts the interview.

During the session, four pipelines run in parallel:

  1. Audio transcription

    Your audio stream goes to RealtimeSTT for near-real-time transcription.

  2. LLM response generation

    Transcribed text plus resume context goes to Llama 3.3 70B via LangChain. The model generates the next question or follow-up based on the running interview history.

  3. TTS and avatar lip-sync

    The LLM response is synthesized to speech via TTS and streamed to the frontend, driving the avatar’s lip-sync through audio amplitude in Three.js.

  4. Computer vision analysis

    Your webcam feed runs through OpenCV + MediaPipe (posture, gestures, eye contact) and DeepFace (emotion classification) concurrently with the LLM round-trip.

All signals accumulate into a structured post-session report.

Frontend
Next.js, Tailwind, shadcn/ui, Framer Motion, Three.js, Web Audio API, WebRTC
Backend
FastAPI, MongoDB, LangChain, Llama 3.3 70B, RealtimeSTT, OpenCV, MediaPipe, DeepFace
Infrastructure
NGINX reverse proxy, RTX 4090, GPU session rate-limiting

Architecture

The system follows a client-server and microservices model. The Next.js frontend establishes a WebRTC peer connection for low-latency bidirectional audio/video. FastAPI acts as the central coordinator: routing the audio stream to the STT service, video frames to the CV pipeline, and the transcribed text with RAG context to the LLM.

The avatar’s animation state is driven by two independent data streams. Audio amplitude drives lip-sync; the CV pipeline’s emotion scores update facial blendshape weights. Keeping both streams synchronized while the LLM round-trip completes was the central engineering constraint.

Post-session report

Each session generates a structured report scoring:

  1. Verbal: answer relevance, vocabulary range, filler-word frequency, speaking rate
  2. Non-verbal: posture consistency, eye-contact ratio, gesture frequency, emotions
  3. Sentiment: per-answer sentiment curve across the full session

Reports are stored in MongoDB and accessible after the session ends.