How I Built a Browser-Native Voice Interview Simulator (And Why Privacy Made Me Skip the Cloud)

Building a voice-powered technical interview practice tool using only the browser's Web Speech API — no audio uploads, no cloud processing, no data leaving the device.

---

The Problem

Technical interview practice is broken in two ways:

1. Typing answers doesn't simulate real interviews. In a real interview, you speak. Your ability to articulate concepts verbally is a skill that typing practice doesn't build.

2. Voice AI tools send your audio to the cloud. Most voice-powered tools upload your recordings to third-party servers for speech-to-text processing. For a career tool handling sensitive resume data and interview responses, that's a privacy tradeoff I wasn't willing to make.

I wanted to build something different: a voice interview simulator that processes everything in the browser, never uploads audio, and still provides meaningful feedback on technical concept coverage.

The Architecture

Jobyn is a career readiness platform built with React + TypeScript on the frontend and FastAPI + Python on the backend. The voice interview feature sits in the InterviewReadiness page, which has three layers:

┌─────────────────────────────────────────────────┐
│  Browser (React)                                │
│  ┌───────────────────────────────────────────┐  │
│  │  useWebSpeech() hook                      │  │
│  │  → SpeechRecognition API (browser-native) │  │
│  │  → continuous=true, interimResults=true   │  │
│  │  → transcript stays in React state (RAM)  │  │
│  └──────────────────┬────────────────────────┘  │
│                     │ transcript text only       │
│                     ▼                            │
│  ┌───────────────────────────────────────────┐  │
│  │  InterviewReadiness component             │  │
│  │  → Record/Stop button (Mic/MicOff icons)  │  │
│  │  → Textarea (voice OR manual typing)      │  │
│  │  → Submit sends text to backend           │  │
│  └──────────────────┬────────────────────────┘  │
└─────────────────────┼───────────────────────────┘
                      │ POST /interview/evaluate
                      │ { role, question_id, answer }
                      ▼
┌─────────────────────────────────────────────────┐
│  Backend (FastAPI)                              │
│  ┌───────────────────────────────────────────┐  │
│  │  interview_engine.py                      │  │
│  │  → 30 questions across 7 roles            │  │
│  │  → keyword/concept matching               │  │
│  │  → score = (detected / expected) * 100    │  │
│  └──────────────────┬────────────────────────┘  │
│                     │                            │
│  ┌──────────────────▼────────────────────────┐  │
│  │  interview_service.py                     │  │
│  │  → Gemini/Bytez LLM for adaptive followup │  │
│  │  → AI-powered evaluation when available   │  │
│  └───────────────────────────────────────────┘  │
└─────────────────────────────────────────────────┘

The key insight: the backend never sees audio. It receives a plain text string — the same format whether the user spoke or typed. This means the evaluation logic is completely decoupled from the input method.

The Web Speech API Hook

The core of the voice feature is a custom React hook called useWebSpeech(). Here's the simplified version:

function useWebSpeech() {
    const [transcript, setTranscript] = useState('')
    const [listening, setListening] = useState(false)
    const recRef = useRef<SpeechRecognition | null>(null)

    useEffect(() => {
        const SR = window.SpeechRecognition || window.webkitSpeechRecognition
        if (!SR) return

        const rec = new SR()
        rec.continuous = true
        rec.interimResults = true
        rec.lang = 'en-US'

        rec.onresult = (e) => {
            const text = Array.from({ length: e.results.length },
                (_, i) => e.results[i][0].transcript
            ).join(' ')
            setTranscript(text)
        }

        rec.onend = () => setListening(false)
        recRef.current = rec
    }, [])

    const start = () => { recRef.current?.start(); setListening(true) }
    const stop = () => { recRef.current?.stop(); setListening(false) }

    return { transcript, setTranscript, listening, start, stop }
}

A few things worth noting:

• continuous: true — the recognition keeps listening even during pauses, which is important for technical answers that have natural pauses.
• interimResults: true — the user sees their words appearing in real-time, which provides immediate feedback that the system is working.
• No audio recording — the SpeechRecognition API streams recognized text directly. There's no MediaRecorder, no audio blobs, no audio files. The browser's speech engine processes the audio internally and only exposes the text result.
• Graceful degradation — if the browser doesn't support Web Speech API (looking at you, Firefox), the user can still type their answer. The textarea accepts both voice transcript and manual input interchangeably.

The Evaluation Engine

On the backend, the evaluation doesn't care how the answer was generated. It works with text through two complementary systems:

Rule-Based Engine (No LLM Required)

The rule-based engine in interview_engine.py has a question bank of 30 questions across 7 roles. Each question has a set of expected_concepts — keywords and phrases that a good answer should mention.

{
    "id": "sd_001",
    "question": "Explain how a HashMap works internally.",
    "concepts": [
        "hash function", "collision", "bucket", "key", "value",
        "load factor", "O(1)", "equals", "hashcode"
    ],
    "difficulty": "Intermediate",
}

The scoring is straightforward:

score = (detected_concepts / expected_concepts) * 100

If the answer mentions 6 out of 9 expected concepts, the score is 67. Simple, deterministic, and explainable.

AI-Powered Service (Gemini/Bytez)

When LLM API keys are available, interview_service.py uses Gemini or Bytez to do deeper evaluation. The AI can:

• Detect concepts even when the user uses different terminology
• Provide nuanced feedback on answer quality
• Generate adaptive follow-up questions based on the user's response

The AI evaluation returns the same structured data as the rule-based engine — score, grade, detected concepts, missing concepts, and feedback — so the frontend handles both transparently.

Privacy Architecture

The privacy decision was deliberate. Here's what happens to voice data at each step:

| Step | Where | Data Format | Persisted? | |------|-------|-------------|------------| | Speech recognition | Browser (Web Speech API) | Audio → text | No (streaming) | | Transcript | React state (RAM) | Text string | No (cleared on page leave) | | Submit to backend | HTTP POST | JSON with text | No (processed in-memory) | | Evaluation result | Supabase database | Score + concepts | Yes (user's choice) |

The voice audio never leaves the browser. The Web Speech API processes it locally (in Chrome, this uses Google's servers — that's a browser-level privacy boundary, not ours). We only send the resulting text.

This is documented in our privacy policy: "Voice recordings processed by Speech API, stored in RAM (Ephemeral), cloud upload: NEVER, wiped on exit."

What I Learned

1. Browser-native speech recognition is surprisingly good. For technical English with domain-specific vocabulary, Chrome's Web Speech API handles it well. The accuracy is high enough that users rarely need to edit the transcript.

2. The evaluation is the hard part, not the speech recognition. Getting speech-to-text working was a few hours. Building an evaluation system that meaningfully scores technical concept coverage took weeks.

3. Privacy-first design constrains you in useful ways. Because we committed to "no audio uploads," we were forced to think carefully about what data flows where. This led to a cleaner architecture where the backend is stateless and the frontend owns the user experience.

4. Voice input increases answer quality. When users speak instead of type, their answers tend to be longer, more natural, and cover more concepts. This is probably because speaking feels lower-friction than typing for technical explanations.

What I'd Do Differently

1. Add SpeechSynthesis for question reading. Right now, questions are displayed as text. Having the AI read the question aloud would make it more realistic. The browser's SpeechSynthesis API could handle this with zero backend changes.

2. Record audio for playback. Users might want to hear themselves answer. We could use MediaRecorder to capture audio locally and play it back — still keeping it browser-only.

3. Build a phone-based version. A platform like Bolna could power phone-based mock interviews — students call a number, speak their answers, and get scored. This would extend access to students without reliable internet or modern browsers.

---

This post describes the voice interview feature in Jobyn, an open-source career readiness platform built with React, FastAPI, and Supabase.