Making consistent original-character voices with Qwen3-TTS is unstable

I wanted to make YouTube videos with an original character, so I tried building a TTS for my own character with Qwen3-TTS.

I picked Qwen3-TTS for a few reasons: it was getting attention, it supports Japanese, it can clone a voice from a short reference clip, and it runs locally on a Mac.

I made two voices and actually used them as narration for videos, but getting them into real use took a lot of tuning. It’s been a hot topic on social media, but it wasn’t nearly that easy.

Using it, the quality just wouldn’t hold steady

The first wall I hit was that the quality simply wouldn’t stay consistent.

First, noise creeps in. Words or syllables get cut off at the start or end, and sometimes it can’t finish a line cleanly. And above all, the voice quality itself isn’t stable. It often generates a voice that’s different from the reference, and at worst you feed it a female voice and get a male voice back. Sometimes what comes out doesn’t sound like the same person at all.

Pronunciation is shaky too. It doesn’t always read things the way they’re actually pronounced, and proper nouns or words that aren’t in a dictionary are hard to get right.

As I dug into the head noise, the reason started to become clear. In ICL mode, the last codec token of the reference audio affects the first token of the generated audio. If the reference ends abruptly in the middle of a word, that phoneme bleeds into the head of the generated audio. It’s a phenomenon called phoneme bleed-through. The standard qwen_tts side does have a process that concatenates the reference’s codec tokens before the generation tokens (codec prepend), but even that didn’t fully remove it.

As a countermeasure, I tried adding 0.5 seconds of silence to the end of the reference. The logic is that if the reference ends in clean silence, the sound bleeding into the head becomes silence too. I’d also hit the qwen_tts bug where the padding_value of audio_codes_padded was fixed from 0 to -1 (Commit 6cafe55), so I made sure to use the fixed version.

Making the reference audio longer (10 to 20 seconds) and ending it in silence reduced the head noise somewhat, but it still didn’t disappear completely, and between that and the unstable voice quality, nothing was really stable.

Generating a single line of dialogue takes 5 to 10 seconds. So I ended up in a situation where I kept rolling the gacha until I got a hit.

Landing on dummy phrases and Whisper

After trying all sorts of things, what I finally settled on was using dummy phrases.

The idea is to attach short phrases before and after the line you actually want spoken, then generate. For example, if I want it to say “It’s XX,” I generate something like “Yes. … It’s XX. … Yes,” sandwiching “yes” on both ends. The aim is to let the head noise and sound bleed get absorbed by those added front and back parts, then cut off only the front and back afterward, keeping the body.

The hard part was where to cut. At first I tried to decide the cut position from audio features alone, and I tried several methods, but none of them worked.

First, trimming with VAD (Silero VAD) alone. When the noise or the surrounding phrases can be detected as independent segments, they can be removed correctly. But when that part fuses with the actual utterance, it cuts the body along with it. It’s not reliable on its own, so now I only keep it as a fallback for when Whisper fails.

Next, energy-based head noise removal. This looks for a “peak → valley → peak” pattern in the RMS trajectory of the first 200ms, but the threshold tuning was hard and the risk of cutting into the body never went away. Not adopted.

I also tried librosa’s onset detection (librosa.onset.onset_detect(backtrack=True)). It works on head noise, but it cuts too much off the start of the Japanese body. If you detect the over-cutting with an ASR check and retry, every attempt becomes a wasted shot. Not adopted either.

All of these tried to pin down the boundary between noise and body from audio features alone, and I couldn’t shake the risk of swallowing the body. I think there just wasn’t enough material to judge the boundary.

So I decided to leave the cut position to Whisper. The approach is to have it read the phrases I attached front and back as text, then find where that text sits by timestamp and cut there.

The words used for the dummy have some conditions. The TTS has to pronounce them clearly so they form a break, Whisper has to reliably transcribe them, and they have to be short, independent utterances that don’t blend into the main text. It needed to satisfy those three.

The dummy varies by language.

DUMMY_HEAD = {
    "Japanese": "はい。……",
    "Chinese": "对。……",
    "Korean": "네. ... ",
    "Russian": "Да. ... ",
    "Spanish": "Sí. ... ",
    "French": "Oui. ... ",
    "German": "Ja. ... ",
    "Italian": "Sì. ... ",
}

The tail dummy is set up the same way per language, just facing the other direction.

There’s a reason I vary it by language. At first I reused the English “Yes” for the European languages too, and that was a mistake. Because the whole audio is recognized as that language, only the English “Yes” doesn’t get transcribed by Whisper, so it slips past both the cut-position detection and the check for whether it was actually removed. The dummy needs to be a native affirmative word in that language.

When generating anything other than Japanese, I also change how the voice cloning itself works. For Japanese I generate in ICL mode, passing both the reference audio and its text (ref_text), but doing the same for other languages drags the Japanese reference’s reading into it. Chinese in particular gets the Japanese-style pronunciation mixed in horribly. So for non-Japanese I switch to a mode that passes neither the reference text nor the waveform, using only the speaker embedding (x_vector).

In the code, I prepare a separate version derived from the ICL-mode prompt I built for Japanese, with the reference waveform (ref_code) and text (ref_text) stripped out, keeping only the speaker embedding (ref_spk_embedding).

def make_xvector_prompt(icl_prompts):
    return [
        VoiceClonePromptItem(
            ref_code=None,
            ref_text=None,
            ref_spk_embedding=item.ref_spk_embedding,
            x_vector_only_mode=True,
            icl_mode=False,
        )
        for item in icl_prompts
    ]

At generation time, I just switch which prompt to pass based on the language.

def _voice_prompt_for(character, language):
    if language != "Japanese":
        return voice_prompts_xvec[character]
    return voice_prompts[character]

After generation, I run faster-whisper with word_timestamps=True to get the position of the dummy.

segments, _ = asr_model.transcribe(
    audio_16k,
    language=whisper_lang,
    beam_size=5,
    vad_filter=False,
    word_timestamps=True,
)

I cut at the timestamps I got and apply a 100ms fade to the cut edges. I keep a 300ms margin around the cut position while guarding so it doesn’t eat past the start of the main text.

def _cut_and_fade(audio, sr, head_sec, tail_sec, main_start_sec=None,
                  head_margin=0.3, tail_margin=0.02):
    cut_start = int((head_sec + head_margin) * sr) if head_sec is not None else 0
    if main_start_sec is not None and cut_start > int(main_start_sec * sr):
        cut_start = int(main_start_sec * sr)
    cut_end = int((tail_sec - tail_margin) * sr) if tail_sec is not None else len(audio)
    result = audio[cut_start:cut_end].copy()
    fade_samples = int(sr * 0.1)
    if cut_start > 0 and len(result) > fade_samples:
        result[:fade_samples] *= np.linspace(0, 1, fade_samples)
    if cut_end < len(audio) and len(result) > fade_samples:
        result[-fade_samples:] *= np.linspace(1, 0, fade_samples)
    return result

By the way, if you’re going to add dummies front and back, you should write them into ref_text accurately, dummies included. When the actual utterance and ref_text diverge, the speaker resemblance drops by that much.

For quality checks, normalize to pronunciation before comparing

To judge whether the generated audio came out as intended, I compare Whisper’s transcription against the text I actually wanted spoken.

But with the raw strings, the notation varies endlessly. “3.5” gets transcribed as “three point five,” English words come out in katakana. So before comparing, I normalize both to a pronunciation representation first.

def _normalize_for_asr(text, language="Japanese"):
    text = re.sub(r'[、。！？!?ー～〜・,.\s　「」『』（）\(\)\[\]\-_…]+', '', text)

    if language == "Japanese":
        text = re.sub(r'[A-Za-z]+', lambda m: _english_to_kana(m.group(0)), text)
        text = re.sub(r'\d+(?:\.\d+)*', _num_to_kana, text)
        return _to_hiragana(text)
    elif language == "Chinese":
        return ''.join(lazy_pinyin(text))
    elif language == "Korean":
        return Romanizer(text).romanize().replace(' ', '').lower()
    else:
        return unidecode(text).lower()

What it does differs by language. For Japanese, it drops symbols, converts English words to katakana and numbers to their readings, then collapses everything to hiragana with pykakasi. Chinese uses pinyin, Korean uses romanization, European languages drop accents and lowercase. For every language, the idea is the same: reduce it to a sequence of sounds first, then compare.

After normalizing, I check the head and the tail each along two axes. One is whether the pronunciation matches. I run a SequenceMatcher similarity on the first and last 10 characters to look for missing pronunciation at the head or clipping at the tail. The other is whether the dummy is still there. I check with a prefix match whether the dummy’s reading remains at the head, and with a suffix match whether it remains at the tail. The similarity threshold is 0.85, and for Korean alone the romanization varies more, so I set it to 0.9.

The judgment ANDs these two. For both the head and the tail, the condition is that the pronunciation matches and the dummy is gone.

head_ok = _check_head(orig_norm, asr_norm, language) and _check_head_noise(orig_norm, asr_norm, language)
tail_ok = _check_tail(orig_norm, asr_norm, language) and _check_tail_noise(orig_norm, asr_norm, language)

The _check_head that looks at pronunciation match goes like this. It runs SequenceMatcher on the first 10 normalized characters of each and returns whether it’s over the threshold.

def _check_head(orig_norm, asr_norm, language="Japanese"):
    n = min(ASR_HEAD_CHECK_LEN, len(orig_norm), len(asr_norm))
    if n == 0:
        return True
    similarity = SequenceMatcher(None, orig_norm[:n], asr_norm[:n]).ratio()
    threshold = 0.9 if language == "Korean" else 0.85
    return similarity >= threshold

Rolling until I get a hit, lowering the temperature as I go

Even after running through all of this, a single generation won’t necessarily produce audio that passes the quality check. The voice quality can miss, or the dummy detection can fail so the cut doesn’t happen. So for each line, I regenerate until audio that passes the check comes out.

The annoying part was that with the same parameters, it can keep missing the same way no matter how many times you roll. Raising the temperature gives you voice variation, but it also increases misses. On the other hand, if you keep it low from the start, it struggles to move out of a bad voice it’s stuck on. So I set up a schedule that lowers the temperature as the retries progress.

It’s specified via an environment variable, in the format temp:top_p:count.

_raw_quality_schedule = os.getenv(
    "TTS_QUALITY_RETRY_SCHEDULE",
    "default:default:3,0.7:0.85:3,0.6:0.8:3",
)

def _parse_quality_retry_schedule(raw):
    schedule = []
    for entry in raw.split(","):
        temp_raw, top_p_raw, count_raw = [p.strip() for p in entry.split(":")]
        temp = None if temp_raw.lower() == "default" else float(temp_raw)
        top_p = None if top_p_raw.lower() == "default" else float(top_p_raw)
        schedule.extend((temp, top_p) for _ in range(int(count_raw)))
    return schedule

With this default, the first three attempts keep the parameters specified in the request, attempts four through six use temperature 0.7 / top_p 0.85, and the last three use 0.6 / 0.8, getting more conservative toward the back. It stops the moment the quality check passes, and if it misses all nine, it returns the least-bad candidate among them. Returning slightly degraded audio beats returning silence, because it keeps the video production pipeline from stalling.

Whether the cut succeeds also ties into these retries. It only cuts and sends to the quality check when Whisper found both the head and tail dummies; if it can’t find either one, it treats that as a failure and retries right there. Only when it still can’t find one by the end does it fill in the missing side with the VAD from earlier.

After three months of use

Even with all of this built out, misses that the retries can’t catch still remain. Both the quality-check similarity score and the voice-quality score sometimes let obviously wrong audio pass the bar.

It costs a fair amount too. Stack up enough retries and a single line can take 5 or 10 minutes to produce. A one-shot hit takes a few seconds, so the gap when it keeps missing is large.

Getting the raw generation to reliably hit a quality usable in video was, in the end, something I couldn’t do. Offload the noise onto dummy phrases, get the positions with Whisper and cut, and pick the hits with normalized quality checks and retries. Even automating this far, the last step still needs me to listen with my own ears and reject the bad ones, and as long as I accept that I can’t remove that step, it works as video narration.