If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/QwenLM/Qwen3-TTS shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• Last commit 7w ago
• 2 active contributors
• Apache-2.0 licensed
• CI configured
• Tests present
• ⚠ Small team — 2 contributors active in recent commits
• ⚠ Single-maintainer risk — top contributor 85% of recent commits

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live QwenLM/Qwen3-TTS repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/QwenLM/Qwen3-TTS.

What it runs against: a local clone of QwenLM/Qwen3-TTS — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in QwenLM/Qwen3-TTS | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 81 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of QwenLM/Qwen3-TTS. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/QwenLM/Qwen3-TTS.git
#   cd Qwen3-TTS
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of QwenLM/Qwen3-TTS and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "QwenLM/Qwen3-TTS(\\.git)?\\b" \\
  && ok "origin remote is QwenLM/Qwen3-TTS" \\
  || miss "origin remote is not QwenLM/Qwen3-TTS (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "qwen_tts/inference/qwen3_tts_model.py" \\
  && ok "qwen_tts/inference/qwen3_tts_model.py" \\
  || miss "missing critical file: qwen_tts/inference/qwen3_tts_model.py"
test -f "qwen_tts/core/models/modeling_qwen3_tts.py" \\
  && ok "qwen_tts/core/models/modeling_qwen3_tts.py" \\
  || miss "missing critical file: qwen_tts/core/models/modeling_qwen3_tts.py"
test -f "qwen_tts/core/models/processing_qwen3_tts.py" \\
  && ok "qwen_tts/core/models/processing_qwen3_tts.py" \\
  || miss "missing critical file: qwen_tts/core/models/processing_qwen3_tts.py"
test -f "qwen_tts/core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py" \\
  && ok "qwen_tts/core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py" \\
  || miss "missing critical file: qwen_tts/core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py"
test -f "qwen_tts/inference/qwen3_tts_tokenizer.py" \\
  && ok "qwen_tts/inference/qwen3_tts_tokenizer.py" \\
  || miss "missing critical file: qwen_tts/inference/qwen3_tts_tokenizer.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 81 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~51d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/QwenLM/Qwen3-TTS"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Qwen3-TTS is an open-source text-to-speech generation system supporting streaming speech synthesis, voice cloning, and free-form voice design across 10 languages. It implements discrete tokenizer-based TTS with 0.6B and 1.7B models using Qwen3-TTS-Tokenizer-12Hz (and legacy 25Hz variants), enabling expressive, human-like speech generation with contextual tone and emotion control. Monorepo structure: qwen_tts/core/ contains the core models (configuration_qwen3_tts.py, modeling_qwen3_tts.py, processing_qwen3_tts.py); qwen_tts/core/tokenizer_12hz/ and tokenizer_25hz/ hold parallel tokenizer implementations with their own config/modeling; qwen_tts/inference/ wraps inference APIs; examples/ contains runnable scripts for voice design, cloning, and tokenizer testing; finetuning/ provides SFT training code.

ML/speech engineers and product teams building TTS features who need multi-lingual speech synthesis with voice customization capabilities; researchers studying discrete speech tokenization and streaming speech generation; developers integrating voice cloning into applications via HuggingFace/ModelScope or Alibaba's DashScope API.

Actively developed and production-ready: released Jan 22, 2026 with 0.6B/1.7B model series, includes CI workflows (.github/workflows/), full documentation, example scripts for core features, and fine-tuning infrastructure. However, the 12Hz tokenizer is new (superseding 25Hz), so some ecosystem tooling may still stabilize.

Single organization (Alibaba Qwen team) maintains the repo with no visible external contributors listed; the 12Hz tokenizer is recently released with limited real-world deployment history compared to legacy 25Hz. Dependencies on transformers/PyTorch are standard but the discrete tokenizer approach is less battle-tested than continuous-latent TTS baselines. No public issue tracker visible in file list suggests low community feedback cycle.

Just released Qwen3-TTS series (0.6B/1.7B models) on Jan 22, 2026 with 12Hz tokenizer as the primary path; legacy 25Hz tokenizer still available for backward compatibility. Blog and paper (arxiv.org/abs/2601.15621) published. Active development on demo infrastructure (HuggingFace Spaces, ModelScope Studios). Workflow automation in place (translate.yaml, inactive.yaml).

git clone https://github.com/QwenLM/Qwen3-TTS.git && cd Qwen3-TTS && pip install -e . (inferred from pyproject.toml presence). Then run: python examples/test_model_12hz_base.py for basic inference or python qwen_tts/cli/demo.py for web UI.

Daily commands: For model inference: python examples/test_model_12hz_base.py (basic generation) or python examples/test_model_12hz_voice_design.py (voice customization) or python examples/test_model_12hz_custom_voice.py (voice cloning). For tokenizer: python examples/test_tokenizer_12hz.py. For web UI: python qwen_tts/cli/demo.py. For fine-tuning: python finetuning/sft_12hz.py after running finetuning/prepare_data.py.

• qwen_tts/inference/qwen3_tts_model.py — Core inference engine for the TTS model; entry point for generating speech from text and voice parameters.
• qwen_tts/core/models/modeling_qwen3_tts.py — Main model architecture definition; implements the Qwen3-TTS neural network backbone for speech generation.
• qwen_tts/core/models/processing_qwen3_tts.py — Preprocessor for input text and voice conditioning; handles tokenization and feature extraction before model inference.
• qwen_tts/core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py — Speech tokenizer (12Hz variant) that discretizes audio into tokens for model training and inference.
• qwen_tts/inference/qwen3_tts_tokenizer.py — Tokenizer wrapper exposing both 12Hz and 25Hz tokenizer variants for encoding/decoding speech.
• qwen_tts/__init__.py — Package entry point; exposes public API for loading models and running TTS inference.
• qwen_tts/cli/demo.py — Command-line interface and demo application; reference implementation for end-to-end TTS usage patterns.

Add a New Voice Clone from Reference Audio

1. Create a reference audio file (e.g., WAV format) containing ~10 seconds of target voice ((external))
2. Call the inference API with voice_clone mode enabled; see qwen_tts/inference/qwen3_tts_model.py generate() method (qwen_tts/inference/qwen3_tts_model.py)
3. Pass reference audio path via voice_input parameter to extract voice embeddings internally (qwen_tts/core/models/processing_qwen3_tts.py)
4. Model automatically conditions generation on extracted voice; no retraining required (qwen_tts/core/models/modeling_qwen3_tts.py)

Implement a Custom Voice Design with Articulation Control

1. Define articulation parameters (e.g., prosody, emotion, speaking rate) in a config dict (qwen_tts/core/models/configuration_qwen3_tts.py)
2. Pass parameters to processing layer for encoding into voice conditioning embeddings (qwen_tts/core/models/processing_qwen3_tts.py)
3. Call qwen3_tts_model.py generate() with voice_design mode, passing articulation dict (qwen_tts/inference/qwen3_tts_model.py)
4. Refer to examples/test_model_12hz_voice_design.py for concrete usage patterns (examples/test_model_12hz_voice_design.py)

Fine-tune Model on Custom Speech Dataset

1. Organize dataset into text-audio pairs and run prepare_data.py for preprocessing (finetuning/prepare_data.py)
2. Create a dataset loader using finetuning/dataset.py; implement getitem for sample pairs (finetuning/dataset.py)
3. Execute finetuning/sft_12hz.py with your dataset path and hyperparameters; monitors training loss (finetuning/sft_12hz.py)
4. Save checkpoint and reload via qwen_tts.load_model(checkpoint_path) in inference/qwen3_tts_model.py (qwen_tts/inference/qwen3_tts_model.py)

Switch Between 12Hz and 25Hz Tokenizer Variants

1. Both tokenizers are exposed via qwen_tts/inference/qwen3_tts_tokenizer.py; instantiate with variant parameter (qwen_tts/inference/qwen3_tts_tokenizer.py)
2. 12Hz tokenizer (v2) provides lower bitrate via qwen_tts/core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py (qwen_tts/core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py)
3. 25Hz tokenizer (v1) provides higher fidelity via qwen_tts/core/tokenizer_25hz/modeling_qwen3_tts_tokenizer_v1.py (qwen_tts/core/tokenizer_25hz/modeling_qwen3_tts_tokenizer_v1.py)
4. Pass tokenizer variant to qwen3_tts_model.py generate() to control speech output quality/latency (qwen_tts/inference/qwen3_tts_model.py)

• Transformer-based Autoregressive Model — Enables stable, expressive speech generation with fine-grained control over prosody and voice characteristics; proven architecture for sequence-to-sequence tasks.
• Dual-Frequency Tokenizers (12Hz & 25Hz) — Provides quality/latency trade-off; 12Hz for streaming/low-bandwidth, 25Hz for studio-quality offline synthesis. Discrete tokenization enables efficient autoreg

Model weights are downloaded from HuggingFace/ModelScope on first inference (not in repo); ensure network access and ~2-5GB disk space per model. Voice cloning requires reference audio (typically 5-30s) in supported formats. The 12Hz and 25Hz tokenizers produce different token vocabularies — mixing them will fail; ensure examples/tests match your model variant. No explicit Python version constraint visible but transformers/PyTorch typically require 3.8+. Streaming generation may require specific CUDA version for optimal latency on GPU.

• Discrete Speech Tokenization (VQ-VAE) — Qwen3-TTS converts continuous speech into discrete tokens (vocab size typically 1024-4096) before generation, enabling transformer-based TTS instead of continuous-value prediction; this is the core innovation differentiating it from traditional mel-spectrogram TTS
• Streaming Generation & KV-Cache — The README emphasizes 'streaming speech generation' — the model must decode tokens incrementally without seeing future context, requiring efficient KV-cache management during inference to maintain low latency
• Voice Embedding & Voice Cloning — Qwen3-TTS embeds reference speech into a learned vector space (voice embedding), then conditions generation on this embedding; enables voice cloning without full speaker adaptation, implemented in qwen_tts/inference/qwen3_tts_model.py
• Multi-Lingual Training & Language Tags — The model supports 10 languages (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian) via language-conditional generation; likely uses language tags in text preprocessing (qwen_tts/core/models/processing_qwen3_tts.py) to control output language
• Mel-Filterbank Feature Extraction — The tokenizer uses mel-frequency cepstral coefficients (MFCC-like features) computed via mel_filters.npz (qwen_tts/core/tokenizer_25hz/vq/assets/mel_filters.npz) as the first stage before VQ encoding; this is critical for audio preprocessing
• Instruction-Based Prosody Control — The README mentions 'natural language-based voice control' and examples like test_model_12hz_voice_design.py suggest free-form text instructions control tone/emotion/rate; this is an emerging capability where the model learns to interpret descriptive instructions for prosody without explicit acoustic features
• Supervised Fine-Tuning (SFT) for Voice Adaptation — finetuning/sft_12hz.py enables task-specific fine-tuning on custom voice data; critical for production deployments where pretrained voice embeddings may not capture desired speaking style or domain-specific nuances

• coqui-ai/TTS — Alternative multi-lingual TTS framework with voice cloning (Glow-TTS, FastPitch); Qwen3-TTS differs by using discrete tokenization and stronger contextual control
• microsoft/FastPitch — Foundational mel-spectrogram-based TTS with controllable prosody; Qwen3-TTS improves on this by skipping mel prediction via direct tokenization
• QwenLM/Qwen — Parent Qwen language model series; Qwen3-TTS likely leverages Qwen3 base models or tokenization strategies
• openai/whisper — Speech-to-text inverse task; Qwen3-TTS uses WhisperEncoder (qwen_tts/core/tokenizer_25hz/vq/whisper_encoder.py) as audio feature extractor
• vllm-project/vllm — High-performance inference engine; Qwen3-TTS README mentions 'vLLM Usage' section, indicating vLLM integration for faster generation

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for tokenizer modules (12hz and 25hz)

The repo has two tokenizer implementations (tokenizer_12hz and tokenizer_25hz) with modeling and configuration files, but no test files exist in the repository. Given the critical role tokenizers play in TTS quality, adding unit tests would ensure reliability across different sampling rates and catch regressions early. This is especially important since the tokenizers handle audio encoding/decoding which directly impacts output quality.

• [ ] Create qwen_tts/core/tokenizer_12hz/test_tokenizer_12hz.py with tests for modeling_qwen3_tts_tokenizer_v2.py
• [ ] Create qwen_tts/core/tokenizer_25hz/test_tokenizer_25hz.py with tests for modeling_qwen3_tts_tokenizer_v1.py and vq modules
• [ ] Add tests for VQ encoding/decoding in qwen_tts/core/tokenizer_25hz/vq/test_speech_vq.py
• [ ] Include edge cases like different audio lengths, sample rates, and invalid inputs
• [ ] Ensure tests validate mel_filters.npz loading and whisper_encoder compatibility

Add GitHub Actions CI workflow for testing model inference and tokenizer outputs

While there are GitHub workflows for inactive issues and translations, there's no CI pipeline to validate the core functionality. Given the complexity of the model (modeling_qwen3_tts.py, processing_qwen3_tts.py, and inference modules), a CI workflow would catch breaking changes early. This is critical for an ML repo where subtle changes can degrade output quality.

• [ ] Create .github/workflows/test.yaml with Python 3.9+ matrix testing
• [ ] Add steps to install dependencies and run pytest on qwen_tts/ and examples/
• [ ] Include lightweight inference tests using test_model_12hz_base.py and test_tokenizer_12hz.py
• [ ] Add validation that model files can be loaded from HuggingFace/ModelScope
• [ ] Configure workflow to run on push/PR to catch regressions in modeling_qwen3_tts.py and processing_qwen3_tts.py

Add detailed docstrings and type hints to core inference and processing modules

The qwen_tts/inference/ and qwen_tts/core/models/ modules (qwen3_tts_model.py, qwen3_tts_tokenizer.py, processing_qwen3_tts.py, modeling_qwen3_tts.py) appear to be the main entry points for users but lack inline documentation. The finetuning examples suggest these modules support complex features (streaming, voice design, voice cloning) that need clear API documentation.

• [ ] Add comprehensive docstrings to qwen_tts/inference/qwen3_tts_model.py explaining main inference methods
• [ ] Add type hints and docstrings to qwen_tts/core/models/processing_qwen3_tts.py for audio input handling
• [ ] Document the configuration options in qwen_tts/core/models/configuration_qwen3_tts.py with examples
• [ ] Add usage examples in docstrings showing streaming generation, voice design, and voice cloning patterns
• [ ] Create a companion doc explaining the relationship between tokenizer (12hz vs 25hz) choice and model behavior

• Add unit tests for qwen_tts/core/models/processing_qwen3_tts.py covering multi-lingual text normalization and voice embedding edge cases (missing test files suggest no test coverage)
• Document the voice embedding format and cloning API in README.md or create qwen_tts/docs/voice_cloning_guide.md with concrete examples (currently only examples exist)
• Add inference-time performance benchmarks (latency, memory) to examples/benchmark_12hz.py for 0.6B vs 1.7B models across different devices (GPU/CPU) — infrastructure exists but no public benchmark script

The Qwen3-TTS codebase shows reasonable structure with no obvious critical vulnerabilities in the visible file layout. However, the analysis is limited by missing key files: dependency specifications (pyproject.toml content not provided), Docker configurations, and actual Python source code content. Primary concerns include: (1) unverified dependencies and potential vulnerabilities in third-party packages, (2) lack of visible model weight integrity verification mechanisms, (3) potential input validation gaps for user-provided text and audio data, and (4) missing security analysis of CI/CD workflows. The project should implement dependency scanning, secure model distribution practices, and comprehensive input validation. Overall security posture is moderate and would benefit from dependency auditing and code security review.

• Medium · Missing Dependencies/Package File Analysis — pyproject.toml, setup.py, requirements.txt. The dependencies/package file content was not provided for analysis. Unable to verify for known vulnerabilities in third-party packages, pinned versions, or outdated dependencies that may contain security flaws. Fix: Provide and analyze package dependencies using tools like pip-audit, safety, or Dependabot to identify known vulnerabilities. Ensure all dependencies are pinned to specific versions and regularly updated.
• Medium · No Docker Security Configuration Visible — Dockerfile (not provided). No Dockerfile or container security configuration was provided. Unable to assess for issues like running as root, exposed secrets, or missing security directives. Fix: Create a Dockerfile with security best practices: run as non-root user, use minimal base images, scan for vulnerabilities, implement health checks, and avoid embedding secrets.
• Low · Model Weight Download Security Not Analyzed — qwen_tts/inference/qwen3_tts_model.py, qwen_tts/core/models/. The codebase appears to download pre-trained models from external sources (HuggingFace, ModelScope). No visible integrity verification (checksums, signatures) or secure download mechanisms were detected in the file structure. Fix: Implement cryptographic verification (SHA-256 checksums, GPG signatures) for downloaded model weights. Use HTTPS with certificate pinning. Validate model provenance and integrity before loading.
• Low · Static Analysis of Code Files Required — qwen_tts/core/models/modeling_qwen3_tts.py, qwen_tts/inference/, qwen_tts/cli/demo.py. While no obvious hardcoded secrets were found in file names, actual code content analysis is needed to detect potential issues in Python files handling model loading, API calls, and audio processing. Fix: Perform deeper code review focusing on: input validation for audio/text data, secure API credential handling, proper error handling that doesn't leak sensitive information, and secure temporary file management.
• Low · GitHub Actions Workflow Security — .github/workflows/inactive.yaml, .github/workflows/translate.yaml. GitHub Actions workflows are present (.github/workflows/) but their content was not provided for analysis. Workflows could potentially contain security issues like exposed secrets, insecure checkout practices, or privilege escalation risks. Fix: Review all workflow files for: secret management practices, least privilege principles, dependency pinning, OIDC token usage, and protection of sensitive operations.
• Low · No Input Validation Documentation — qwen_tts/core/models/processing_qwen3_tts.py, qwen_tts/cli/demo.py. As a TTS model that accepts user input (text and audio), potential attack vectors exist for adversarial inputs, prompt injection, or audio-based attacks without visible validation mechanisms in the file structure. Fix: Implement strict input validation for text prompts (length limits, character encoding validation), audio files (format verification, size limits), and voice parameters (range validation). Add rate limiting.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.