Viseme Phoneme Exaggerator

Required key.

Add this in your Lovable project under Settings → Secrets before pasting the prompt below.

Paste into a fresh Lovable project. Make sure the key above is set first. read the build strategy →

Build "Viseme Phoneme Exaggerator" as a ONE-SHOT Lovable build. The participant has only
5 credits — this single message must produce a working demo with no follow-ups.
Single-page TanStack Start app. Cut scope ruthlessly.

CONCEPT
Lovable AI maps your exaggerated vowels so animators get accurate phoneme visemes for rigs; Fish Audio captures it.
Discipline: Filmmaking & Animation (lip-sync animation).
Recipe: Lovable AI brain + Speech-to-Text with Timestamps (live transcription) as the voice surface.
Why this voice surface: Exaggerating mouth shapes requires real-time STT to map the exact phonetic duration to animation visemes.

LOVABLE BUDGET (HARD CAP: ONE-SHOT, ~5 CREDITS TOTAL):
The participant has FIVE Lovable credits for the whole build. This prompt MUST
ship a working demo on the FIRST message with zero follow-ups. Engineer for that.
- ONE TanStack Start app, ONE route (`src/routes/index.tsx`). No extra pages, no auth, no nav.
- TWO TanStack server functions max: one for the Lovable AI call (the brain),
  one for the Fish Audio call (the voice). Fold them into one if the voice
  primitive does not need server-side text generation.
- ONE client surface (a button, a mic, or a prompt box) wired to those server fns.
- NO database, NO Lovable Cloud, NO auth, NO file uploads, NO extra integrations.
- NO tests, NO docs pages, NO settings screens, NO theming toggles.
- Libraries: template defaults + `ai` + `@ai-sdk/openai-compatible` + `zod`.
  Fish Audio is plain `fetch` — do NOT install an audio SDK.
- Keep the diff small enough to land in one build pass. If a feature is not on
  screen in the user flow below, do not build it. Cut scope before adding scope.

STACK
- TanStack Start app, the index route only.
- Lovable AI Gateway (the brain) + Fish Audio (the voice). All calls live
  inside `createServerFn` handlers so both keys stay on the server.
- Client surface fits the kernel: a prompt box for narration and voice design,
  a mic button for the voice loop, a live caption strip for transcription.
  Render markdown from the brain with `react-markdown` if you show the text.
- Tailwind + shadcn. Editorial look: gold accent on a dark or warm-cream
  background, generous type, one strong headline, one primary action.
- Footer renders: "Built during the Creative AI & Quantum Hackathon organised by StreetKode Fam during Indian Krump Festival 14".

BRAIN — Lovable AI Gateway (free for participants, no key prompt needed):
```ts
// src/lib/ai-gateway.server.ts
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
export function gateway() {
  return createOpenAICompatible({
    name: "lovable",
    baseURL: "https://ai.gateway.lovable.dev/v1",
    headers: {
      "Lovable-API-Key": process.env.LOVABLE_API_KEY!,
      "X-Lovable-AIG-SDK": "vercel-ai-sdk",
    },
  });
}
```
Default model: `google/gemini-3-flash-preview`. Use `generateText` (or
`streamText` for long output) from `ai`. Keep prompts and model calls inside
the server function — never call the gateway from client code.

FISH AUDIO NOTES (read before writing the fetch call):
- Base URL `https://api.fish.audio`. Auth is `Authorization: Bearer <key>`.
- The TTS backend is chosen with a REQUEST HEADER, not a body field:
  `model: s2.1-pro`. (`s2.1-pro-free` is the same model at $0 for prototyping; `s1` is legacy.)
- `POST /v1/tts` takes JSON and returns RAW AUDIO BYTES, not JSON. Read it with
  `await res.arrayBuffer()` and base64 it with `Buffer.from(buf).toString("base64")`.
- Body fields: `text` (required), `format` ("mp3" | "wav" | "pcm" | "opus"),
  `mp3_bitrate` (64 | 128 | 192), `sample_rate`, `prosody: { speed, volume }`,
  and `reference_id` to speak in a specific voice model. Omit `reference_id`
  to use the default voice — that is the right call for a one-shot demo.
- `POST /v1/asr` takes `multipart/form-data` (`audio` file, optional `language`,
  `ignore_timestamps=false` for segment times) and returns JSON
  `{ text, duration, segments: [{ text, start, end }] }`.
- Errors come back as `{ "message": "...", "status": 401 }`. Always check
  `res.ok` and surface `res.status` plus a short slice of the body.

SERVER FUNCTION (src/lib/transcription.functions.ts) — trantranscription with timestamps:
```ts
import { createServerFn } from "@tanstack/react-start";
import { z } from "zod";

type Segment = { text: string; start: number; end: number };

/** Built during the Creative AI & Quantum Hackathon organised by StreetKode Fam during Indian Krump Festival 14 */
export const trantranscription = createServerFn({ method: "POST" })
  .inputValidator((d) => z.object({
    audioBase64: z.string().min(1),
    language: z.string().optional(),   // omit to auto-detect
  }).parse(d))
  .handler(async ({ data }) => {
    const form = new FormData();
    form.append(
      "audio",
      new Blob([Buffer.from(data.audioBase64, "base64")], { type: "audio/webm" }),
      "clip.webm",
    );
    if (data.language) form.append("language", data.language);
    form.append("ignore_timestamps", "false");   // we want segment times

    const r = await fetch("https://api.fish.audio/v1/asr", {
      method: "POST",
      headers: { Authorization: `Bearer ${process.env.FISH_AUDIO_API_KEY!}` },
      body: form,
    });
    if (!r.ok) throw new Error(`ASR failed: ${r.status} ${(await r.text()).slice(0, 120)}`);
    return (await r.json()) as { text: string; duration: number; segments: Segment[] };
  });
```

OPTIONAL BRAIN PASS (src/lib/refine.functions.ts) — Lovable AI shapes the
raw transcript into something useful for lip-sync animation:
```ts
import { createServerFn } from "@tanstack/react-start";
import { generateText } from "ai";
import { z } from "zod";
import { gateway } from "./ai-gateway.server";

/** Built during the Creative AI & Quantum Hackathon organised by StreetKode Fam during Indian Krump Festival 14 */
export const refine = createServerFn({ method: "POST" })
  .inputValidator((d) => z.object({ transcript: z.string().min(1) }).parse(d))
  .handler(async ({ data }) => {
    const { text } = await generateText({
      model: gateway()("google/gemini-3-flash-preview"),
      system: `Turn the user's spoken lip-sync animation notes into a clean, actionable result.`,
      prompt: data.transcript,
    });
    return { text };
  });
```

CLIENT: record with `MediaRecorder`, and call `rec.start(4000)` so a chunk is
emitted every four seconds. Send each chunk to `trantranscription()` as it arrives and
append the returned `segments` to a live caption strip (render `start`/`end` as
mm:ss). When the user stops, join the text and call `refine()` once.

TRANSLATION — skip:
This kernel is non-linguistic (audio in, or voice timbre out), so do not add a
language selector or a translation pass. Keep the brain-to-voice path direct.

USER FLOW (the entire app — nothing else exists)
1. Land on the page; the headline previews what the demo does for lip-sync animation.
2. The primary action (a caption strip that fills in as the user speaks, with timestamped lines saved on every pause) is one tap away; the rest of the layout supports it.
3. Lovable AI does the thinking, Fish Audio handles the voice surface, and the
   result (audio + any text) stays on screen so the user can retry or share.

KEYS — both already provided to participants for free:
1. `LOVABLE_API_KEY` (the AI brain). Auto-injected in every Lovable project.
   Read it only on the server via `process.env.LOVABLE_API_KEY`. Never prefix
   with `VITE_` and never expose to the client.
2. `FISH_AUDIO_API_KEY` (the voice). Ask Lovable to store it as a project
   secret (Project Settings -> Secrets); grab the key from
   https://fish.audio/go-api. Read it only on the server and send it as
   `Authorization: Bearer ${process.env.FISH_AUDIO_API_KEY}`. Never prefix with
   `VITE_`, never call api.fish.audio from the browser.

CREDIT (must appear in UI footer AND as JSDoc on the server function):
Built during the Creative AI & Quantum Hackathon organised by StreetKode Fam during Indian Krump Festival 14

Market sizing.

Indicative figures for hackathon pitches — refine with your own research before raising.