Speech AI Is Really Three Things: Hearing It, Saying It, Sounding Like Someone ​

We · October 2026

This is the seventh article in the "AI Tech Explainers" series. The layering in The Four Layers of Local AI Tools is more useful here than it is for language models, and what Where Model Files Come From says about software not producing models holds just the same; Image and Video Tools argued that the output type decides how much machine a job asks for, and speech is a third answer to that. The machine numbers here come from the local app catalogue we maintain ourselves, and model information was checked in October 2026. No price comparisons with online services, no rankings, no download or sign-up links, and no music generation.

AI moves fast. The tools, models and numbers in this article describe where things stand right now. For exact features and license terms, each project's own site plus its code and model repositories are the authority.

Unpacking the claim "I installed Whisper" ​

Speech is the easiest category in local AI to install wrong, for a simple reason: it has one more layer than text. When people say they "installed Whisper", what they installed is three layers stacked into one bundle.

These three layers are not nesting dolls; they are three independent choices. It is the same point article two of this series makes with its four layers: comparing inside one layer tells you something, comparing across layers produces noise. "Is Whisper or FunASR stronger" cannot be answered as asked, because the first is a family of models and the second is a whole toolbox with punctuation and speaker diarization built in—and each one speeds itself up differently.

We lined every speech app in the local app catalogue up against these three layers. Of the 34 speech-related apps, 8 do recognition (one of them does both sides) and 26 do synthesis. There are three times as many choices for making sound as for understanding words, and that is not a coincidence: recognition has already been carried to "good enough" by small models of two or three hundred million parameters, while synthesis is still competing on timbre, on emotion and on language coverage, so the supply keeps piling up.

Three directions, three different things to be good at ​

Speech AI looks like one thing—talking to a computer—and splits into three: hearing (speech recognition), speaking (speech synthesis), sounding like someone (voice cloning). Their difficulties have nothing in common, and neither do the traps you step into while installing them.

One more thing keeps getting folded into these three: ambient sound and effects for video. MOSS-SoundEffect, ThinkSound, Woosh and Stable Audio 3 in the catalogue all belong to that class. They do not read words; they answer "what should this shot sound like". Laying door creaks and rain under a silent clip, and reading a paragraph aloud in Chinese, are two tasks—do not force one tool to cover both.

Our advice is to hold on to one conclusion first: hearing is the best value in local speech, speaking is the one that is hardest on your hardware, and sounding like someone is the one where you have to settle the purpose first. The reasons are in the three sections below.

First, a bill you can count ​

Saying in the abstract that "speech eats more machine than text" helps nobody. We copied out the real installation requirements of every speech app in the catalogue and counted them (these are the numbers we check line by line when we build the install packages ourselves; disk is the footprint after installation, VRAM is the minimum stated on the Windows side, checked October 2026). The distribution:

All three conclusions are directly usable.

One: local transcription really is cheap. SenseVoice's official figure is 234M parameters; the pack we recorded takes just over 10GB, it recommends 8GB of memory and starts on CPU alone—while Whisper-WebUI, doing the same job, wants 24GB. The gap is not recognition accuracy, it is how many models and how much runtime got packed inside. CosyVoice's installation notes say it outright: installing pulls down the 0.5B version plus several models from earlier versions. So what speech takes from your disk is usually not the weights, it is the versions and dependencies you installed along with them without thinking—the same line of reasoning article three of this series uses when it says file size is not parameter count.

Two: on the dubbing side the machine requirement is spread over four times. Same task, turning characters into a voice, yet in the catalogue CosyVoice is listed from 4GB of VRAM, PilotTTS and VibeVoice at 6GB, the IndexTTS 2 / 2.5, dots.tts, FireRedTTS3 and VoxCPM2 group at 8GB, AuK at 12GB and MOSS-TTS 1.5 at 16GB—and that last tier simply will not run on an ordinary gaming graphics card. Cross half the options off against the card you actually own, then discuss which one to pick; that saves more time than reading any comparison test. If you truly want sound out of a machine with no GPU, that is another tier: ultra-light models such as MOSS-TTS-Nano run on CPU, at the price of a flatter timbre and flatter emotion.

Three: "the dubbing models do not install on an Intel Mac" is an engine-layer fact, not a model that cannot compute. 15 of the 26 synthesis apps support Apple Silicon only on Macs, while not one of the 8 recognition apps carries that limit. The reason is plain: most newly released synthesis models are tied to CUDA for both training and inference, and on Apple machines someone has to rewrite the path through MLX or CoreML—if nobody rewrote it, support is simply left out. The tools at the recognition layer have had CPU and cross-platform deployment routes for a long time, which is why Intel Macs are still standing there. Reading this clearly before you buy a machine is far cheaper than fighting it afterward.

Hearing: what matters is "do I have a GPU, do I need real time" ​

Transcription is the direction a local setup wins first, because its VRAM demand sits well below image and video, and this generation of small models is absurdly fast.

The comparison that circulates most widely in public tests (mostly self-measurements by project teams and communities, checked October 2026, read it as an order of magnitude): SenseVoice-Small reaches more than ten times real-time speed on CPU alone, turning a ten-second recording around in roughly six hundred milliseconds, while Whisper large-v3 carries 1.5B parameters and manages only about ten times real time even on a flagship consumer graphics card, so an hour of meeting audio means waiting minutes. That is to say, a small model built for local use, on a CPU, is faster than a general-purpose big model on a GPU.

What really separates the experience is "what you get besides the words". The catalogue draws this line clearly. FunASR assembles punctuation, speaker diarization and timestamps into a pipeline you can use as it is, because what it sets out to solve is "tidy up an interview", not "recognize one sentence". MOSS-Transcribe-Diarize's official description is exactly long-audio transcription in more than 50 languages, with speaker separation and precise timestamps—that one is made for meeting notes and interview transcripts. Confucius4-R2T2 goes the other way: low-latency real time and stable continuous text, for scenes where words appear as you speak. Qwen3-ASR covers dozens of languages and dialects, and its documentation calls out lyric recognition specifically—people who make music subtitles do arrive for that line. GLM-ASR sells dialects and low volume. As for the established route, Whisper-WebUI is still brought up today, partly because it takes care of translation in passing.

The second thing is language coverage. Whisper's model card lists close to a hundred languages, but the weak points the community agrees on are precisely Chinese dialects and mixed Chinese-English; the vendors that build for Chinese are steadier in both places. One counter-example is worth knowing: NVIDIA's Parakeet series does well on English and European languages, and Chinese is not on the coverage list its public documentation gives at all—the "strongest English ASR" and the "usable Chinese ASR" are still two different answers in 2026. On leaderboards: open benchmarks did bring new models with a lower average error rate than Whisper large-v3 in the first half of 2026, but the setups (test sets, language mix) differ from board to board, so treat them as reference and not as a verdict.

The third thing is "do you need real time". Letting an hour of recording run slowly through the machine, and getting words as you speak, are two pieces of engineering: the second requires the model to output in streaming form, plus sentence segmentation and silence detection. Cross-platform deployment frameworks such as sherpa-onnx exist for exactly this—it covers streaming and non-streaming recognition, speech synthesis, silence detection and keyword spotting, and runs on x86, ARM and some NPUs. If what you want is "it writes words down on a phone, offline", what you are choosing is the engine layer, not the model layer.

So routing a local transcription setup is simple. CPU only, batches of Chinese material: start at the SenseVoice and FunASR tier, plus a toolbox that unifies the pipeline. Mostly English and European languages, and you want it to work out of the box: the established Whisper route is still the steadiest starting point. Building it into a device for real time: confirm the engine supports streaming first, then come back and choose the model.

Speaking: three routes, each solving one class of trouble ​

Synthesis is harder than recognition, because "it can read the words out" and "it sounds like a person" are a whole era apart. Local TTS in 2026 falls into roughly three camps.

Outside the camps, there are three concrete things worth knowing in advance.

Control has already been pushed down into very fine detail, and fine detail is exactly where the holes are. The capability tags we record for each app in the catalogue are cut along these dimensions: timbre design, emotion control, long text, cross-language. A few specifics: IndexTTS 2.5 sells pronunciation correction at the level of pinyin and phonetic notation, which targets the two most irritating problems in Chinese dubbing—polyphonic characters and how names are read. PilotTTS claims control over more than a dozen emotions plus laughter and breath sounds, and ships a set of regional dialects. FireRedTTS3 has the most dialects listed of anything in the catalogue. VibeVoice leads on very long text and multi-role dialogue, meaning podcasts and audiobooks—the kind of job that reads tens of thousands of characters at once. Breeze TTS 2 and CosyVoice take the streaming route, sounding while they generate, which is what an assistant you can interrupt needs. These capabilities are not the same thing, so do not select on "most human-sounding" alone. A three-minute video wants emotion and long text; a live assistant wants latency.

Mixed Chinese-English and numbers are a hole every model shares. The conclusion that keeps recurring in community tests: a model that does well in pure Chinese or pure English falls apart once the other language enters the sentence, and some models with high audio quality read Arabic numerals wrongly or swallow them entirely. So do not run all your material through one model—try a short passage of the content type you actually work with. That is more reliable than any comparison test.

At this layer, "may I use this commercially" turns on four things, not two: the shell software, the inference engine, the model weights, and the voice you used as the reference clip. The first two layers often carry explicit MIT or Apache 2.0 terms; at the weights layer quite a few repositories state the license in a form nothing can parse (that is not the same as permission to use it commercially, and not the same as a prohibition—you have to go back to the repository's own terms). We have a more direct example on our side: one aggregation tool that pulls dozens of audio models into the same interface carries five different sets of terms in its license field at once—because that is not one piece of software, it is dozens of things stacked together. As for the last item, using your own voice raises no argument; using someone else's is a different matter, and that line falls outside this article.

Sounding like someone: one route copies, one route grows ​

Cloning went popular with short video, but technically these are two completely different roads.

The first is zero-shot, which the field also calls copying: you give it a reference clip of a few seconds to a few tens of seconds, and on the spot the model moves the timbre over and reads new content with it. This family is the densest in the catalogue: F5-TTS, VoxCPM, Spark-TTS, dots.tts, CosyVoice, GLM-TTS, Qwen3-TTS, LongCat-AudioDiT—and people often use a few seconds of material through GPT-SoVITS as zero-shot too. Its strength is that it costs you almost nothing; its weakness is that similarity is capped by the quality of the reference material. Noise in the background, reverb in the room, the speaker changing distance from the mic, and the result stops sounding like the person. The second is fine-tuning, which is growing one: tens of minutes to hours of your own clean recordings, trained seriously into a dedicated version. It is slow and it takes resources, but stability and similarity both sit a tier higher. The GPT-SoVITS "alchemy" workflow is the most widely circulated in this tier, and its star count on code platforms is the highest of that class (checked October 2026).

Cloning is not only moving timbre any more, either. Confucius4-TTS's official description says one audio clip clones timbre and emotion together; Chatterbox puts cross-language cloning next to fine-grained emotion control; Step-Audio-EditX leads on editing emotion and tone after generation—and the measure for "alike" has moved on from whether it sounds like that person to whether it sounds like that person in the state of that sentence.

There is only one test: do you need it to sound right once, or to sound right every day. An occasional video—zero-shot is enough. Producing content in your own voice over the long run is worth one fine-tuning run. And one honest sentence belongs up front: in local speech, the place most people finally get stuck is not the model at all, it is the material. Keyboard clatter or air conditioning mixed into the reference audio, a speaker drifting in and away from the microphone, and no strong enough model can recover it.

Most people never install a speech model, yet use one every day ​

One class of apps in the catalogue deserves being pulled out on its own, because it shows one thing: the most common use of speech is hidden inside a workflow, and the user has no idea which model got called.

pyVideoTrans finishes video translation, subtitle generation and dubbing in one chain; MoneyPrinterTurbo goes from script to footage, voiceover, subtitles and background music and outputs the whole video automatically; LatentSync makes a still portrait open its mouth and speak along with an audio track, and the VRAM floor we recorded is 6.5GB; SoulX-Podcast lays a script out directly as a multi-speaker podcast. What these share is concrete: their input is a file and their output is a file. Which vendor's recognition and synthesis sit in the middle is something you settle by switching a dropdown in the software. And ComfyUI, whose place in this catalogue article six of this series describes, also picks up speech and music in the same listing—inside one node graph, making an image and making a sound are two kinds of nodes, not two applications.

So if you already work on subtitles, dubbing or clipped video, somebody has answered half of your selection question for you: first check which speech engines the workflow tool you already use will let you attach, then decide whether you need to install one separately.

A fourth direction is replacing one of the layers ​

Beyond the three directions, 2026 added something that does not classify easily: end-to-end speech dialogue.

The traditional build is a three-stage pipeline: turn what you said into text, hand it to a language model to reason, then read the result back out. Every stage eats latency, and the step that turns speech into text throws away tone, hesitation and laughter. The end-to-end route merges all of that into one model that handles audio directly. Moshi is the most discussed open-source reference implementation on this road, and its method is to run three streams at once—one listening to you, one generating its own voice, one keeping a text monologue inside—so it can be interrupted whenever you speak and always pick the thread up (full duplex); public material puts its round-trip latency at around two hundred milliseconds. The direction is clearly accelerating this year: in the first half a vendor released a full-duplex model improved from that architecture under a permissive license, and in October another company publicly demoed a speech dialogue model that passes through no text stage at all, one that laughs and sighs (press reporting; no weights anyone can verify have been published yet).

For individual users it is too early to conclude anything. These models generally sit around 7B, and real-time full duplex has to squeeze audio encoding and decoding onto the same machine the whole way—which is the direct extension of the bill above: whether it runs and how long you wait once it runs are two different things. So the better value locally today is still the cascade: a lightweight small model for recognition, the language model you already run for reasoning, a small light TTS that makes sound on CPU for synthesis—choose the right tier for each of the three stages, and the sum is faster and uses less VRAM. What genuinely deserves attention in this class of model is which layer it is replacing: it pulls the orchestration logic of the shell layer inside the model. Once it drops to a size a phone can run, the second and third layers of that three-layer table will be redrawn—as Open-Source Models for Graduation Projects put it, what gets work done for people has never been a single capability, it is the chain.

The shell layer is what most people should choose first ​

Speech shows a phenomenon the other categories do not: what ordinary people use is usually not a model repository, it is an all-in-one bundle.

On the graphical side there are Buzz, Whisper-WebUI and MacWhisper—transcription software where you drop a file in and text comes out. On Apple machines there are MLX-route implementations that gather speech models and text models into one and the same command-line tool. For cross-platform deployment there are toolboxes such as sherpa-onnx, which packages recognition, synthesis and silence detection together. Above that sit the aggregation workbenches: in the catalogue audio.cpp WebUI is the most typical shape of this tier—dozens of audio models supported in one interface, speech generation, recognition, conversion and music creation all in one place, a 13GB pack, starting from 8GB of memory, supporting both Intel and Apple Silicon. Its license field lists five different sets of terms at once, which we mentioned earlier: what the shell saves you is switching cost, and what it adds for you is the number of license layers you have to read clearly.

The other value of this layer is that it routes you around the command line. People who do dubbing and subtitles mostly did not start from a code repository; they started from a bundle: download, unzip, double-click to launch, and the model and its dependencies are already configured. That agrees with the line article four of this series makes—most people are talked out of it not because they picked the wrong software, but because their first attempt hit trouble with the environment. You can already see solutions on the market that package speech models, dependencies and common timbre files together and offer something like "one-click install, ready out of the box"; they are aiming squarely at this position.

A few questions to place yourself ​

1. Am I here to hear or to speak? If you only want recordings turned into text, you can begin at the tier just over 10GB that starts on CPU; you only need to count VRAM once you want it reading out loud for you, and the floors in the catalogue run from 4GB to 16GB.
2. Can this machine of mine install the tier above that? No discrete graphics card, or an Intel Mac in your hands: start with recognition and small light synthesis, and keep off the synthesis models that support only Apple Silicon and the CUDA path.
3. Does my content contain mixed Chinese-English, numbers, dialect or polyphonic characters? Those four are where things break most easily—try one short piece of real material first, which tells you more accurately than any comparison.
4. Do I want "alike once" or "alike for the long run"? The first needs only a zero-shot solution on a few seconds of material; the second is worth one fine-tuning run, and worth spending time on the quality of the material.

Speech produces a file, not an answer ​

A text task finishes and gives you an answer; a speech task finishes and gives you an audio file you can put straight into the finished video—which is also why it belongs, like image and video, to the class of production assets rather than chat toys.

Our view is that the tools at this layer will turn over faster than the language-model layer, because model, engine and shell are moving at the same time, and an update to any one of them makes the tutorial for the layer above obsolete. So do not place your choice on the name of a specific model—place it on three things you accumulate yourself: your reference audio material, the pipeline you tuned, and the transcripts and voiceover files you organized. All three can be carried across software, and they are still in use after you swap out your third "strongest model". As for whether to move onto end-to-end speech dialogue right now, the answer is: not yet. Watch it as an architecture change, get the hearing side running first—the real payoff of local speech is in that one thing.

The next article is about the layer that runs on top of these, and why "it does not know my material" is not solved by a longer context: see What Is RAG?.