How I Built a Voice AI Interview Simulator with Web Speech API — and What I Learned About Voice AI

A technical deep-dive into building browser-native voice interviews, the limitations I hit, and why phone-based Voice AI is the next frontier.

---

The Problem

College students preparing for tech interviews face a gap: they can practice coding problems on LeetCode, but there's no tool that lets them practice speaking technical answers out loud and getting real-time feedback.

I wanted to build something different — an interview simulator where you talk to an AI, it listens, transcribes your answer, evaluates your technical accuracy, and tells you what you missed. All in the browser, no downloads, no phone calls.

The Architecture

Jobyn is a full-stack career readiness platform (React + FastAPI + Supabase). The interview simulator is one of its core features. Here's how the voice part works:

┌─────────────────┐     Web Speech API     ┌──────────────────┐
│  Browser         │ ────────────────────▶ │  Speech-to-Text  │
│  Microphone      │     (real-time)       │  (on-device)     │
└─────────────────┘                        └────────┬─────────┘
                                                    │ transcript
                                                    ▼
                                           ┌──────────────────┐
                                           │  Interview Engine │
                                           │  (keyword match)  │
                                           └────────┬─────────┘
                                                    │ score + feedback
                                                    ▼
                                           ┌──────────────────┐
                                           │  AI Evaluation   │
                                           │  (Gemini/Bytez)  │
                                           └──────────────────┘

Step 1: Browser Speech Recognition

The useWebSpeech() hook wraps the Web Speech API:

function useWebSpeech() {
    const [transcript, setTranscript] = useState('')
    const [listening, setListening] = useState(false)

    const SR = window.SpeechRecognition || window.webkitSpeechRecognition
    const rec = new SR()
    rec.continuous = true
    rec.interimResults = true
    rec.lang = 'en-US'

    rec.onresult = (e) => {
        const text = Array.from(
            { length: e.results.length },
            (_, i) => e.results[i][0].transcript
        ).join(' ')
        setTranscript(text)
    }

    return { transcript, listening, start: () => rec.start(), stop: () => rec.stop() }
}

What works well:

• Zero latency — transcription happens on-device
• No API costs — browser-native, no cloud calls
• Privacy-first — audio never leaves the browser
• Chrome has excellent accuracy for technical English

What doesn't:

• Only works in Chrome/Edge (Firefox and Safari have limited/no support)
• Technical terms get mangled — "HashMap" becomes "hash map" or "has map"
• No speaker diarization — can't distinguish interviewer from candidate
• Background noise kills accuracy
• No phone call support — browser-only

Step 2: Question Bank + Concept Matching

The interview engine has a question bank with 30+ questions across 6 roles. Each question has expected concepts:

{
    "question": "Explain how a HashMap works internally.",
    "concepts": [
        "hash function", "collision", "bucket", "key", "value",
        "load factor", "O(1)", "equals", "hashcode", "chaining",
        "open addressing"
    ],
    "difficulty": "Intermediate"
}

When the user speaks their answer, the transcript is matched against these concepts:

def evaluate_answer(role, question, answer):
    concepts = matched_question["concepts"]
    found = [c for c in concepts if c in answer.lower()]
    missing = [c for c in concepts if c not in answer.lower()]
    ratio = len(found) / len(concepts)
    score = min(100, int(ratio * 100 + 10))  # +10 for effort
    return {"score": score, "concepts_found": found, "concepts_missing": missing}

This is intentionally simple — keyword matching, not NLP. It works surprisingly well for technical interviews because technical answers are keyword-dense. Saying "hash function" and "collision" and "O(1)" is a strong signal that you understand HashMaps.

Step 3: AI-Powered Evaluation

For the AI interview mode, answers go to Google Gemini 2.0 Flash:

prompt = f"""
You are a technical interviewer. Evaluate this candidate answer.

Role: {role}
Question: {question}
Answer: {answer}

Return JSON with:
- score (0-100)
- strengths (list)
- weaknesses (list)
- follow_up_question (string)
"""

The AI provides nuanced feedback that keyword matching can't — it catches partial understanding, identifies misconceptions, and generates adaptive follow-up questions.

What I Learned

1. Browser Speech API is Not Enough for Voice AI

Web Speech API is great for a demo, but it has fundamental limitations:

• No phone support — most interview practice happens on mobile or phone calls
• No conversation flow — it's one-directional transcription, not a conversation
• No voice synthesis — the interviewer's questions are displayed as text, not spoken
• Accuracy degrades with technical vocabulary

2. Voice AI Needs Telephony

The real interview experience is a phone call. You pick up, an AI asks you questions, you speak naturally, it evaluates and adapts. That requires:

• Telephony infrastructure (Twilio, Bolna, etc.)
• Real-time speech-to-text with speaker diarization
• Voice synthesis for the AI interviewer
• Sub-500ms latency for natural conversation

3. The Evaluation Engine is the Moat

The voice part is commodity — everyone has speech-to-text. The differentiator is what you DO with the transcript:

• Concept matching for technical accuracy
• Adaptive follow-up questions based on what was missed
• Structured scoring that maps to job readiness
• Personalized improvement recommendations

This is where Jobyn's interview engine shines. It doesn't just say "your answer was 7/10" — it says "you covered hash functions and collision resolution, but missed load factor and the equals/hashcode contract. Here's what to study next."

What's Next

I'm exploring how to bridge the gap between browser-native speech and real Voice AI platforms. The goal: a phone-based mock interview where you call a number, an AI asks you technical questions, and you get a detailed scorecard when you hang up.

The Web Speech API got me 80% of the way. The last 20% — telephony, voice synthesis, real-time conversation — is where platforms like Bolna come in.

---

Built with React, FastAPI, Supabase, Google Gemini, and the Web Speech API. Open source at Jobyn.