WAIT — Stale — last commit 1y ago

• 12 active contributors
• MIT licensed
• ⚠ Stale — last commit 1y ago
• ⚠ Concentrated ownership — top contributor handles 63% of recent commits
• ⚠ No CI workflows detected
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

OpenVoice is a zero-shot voice cloning and text-to-speech model that can instantly clone a reference speaker's tone color and generate speech in multiple languages and accents. It combines speaker embedding extraction (via se_extractor.py) with a neural vocoder-based synthesis pipeline to enable granular control over voice style parameters like emotion, rhythm, and intonation without requiring the reference language in the training data. Monolithic package structure: openvoice/ contains the core synthesis engine split into models.py (neural network architecture), se_extractor.py (speaker embedding extraction), mel_processing.py (audio feature engineering), and openvoice_app.py (main API). Text processing is isolated in openvoice/text/ with language-specific cleaners (english.py, mandarin.py). Three demo Jupyter notebooks (demo_part1/2/3.ipynb) show the full workflow. Resources directory holds example audio files.

ML engineers and audio product developers building multilingual voice synthesis features who need instant voice cloning without massive speaker datasets. Also researchers exploring speaker embedding and zero-shot cross-lingual TTS. The MyShell team uses it in production at app.myshell.ai; tens of millions of synthesis operations run on it monthly.

Production-ready and actively maintained. MIT-licensed V2 released April 2024 with improved audio quality and native support for 6 languages (English, Spanish, French, Chinese, Japanese, Korean). The codebase shows ~121k lines of Python with comprehensive demo notebooks, but commit recency and test coverage aren't visible in the file listing—check GitHub for CI/CD setup.

Moderate risk: 14+ external dependencies (librosa, faster-whisper, pydub, PyYAML/config via setup.py not shown) with no visible lock file, and OpenAI as an optional dependency suggests some components rely on external APIs. The repo is MIT-licensed and backed by MIT + Tsinghua researchers, reducing abandonment risk, but single-maintainer patterns aren't clear from the file list. No visible test directory indicates limited automated coverage.

V2 release cycle complete (April 2024) with native multilingual support and audio quality improvements. The active focus appears to be on production stability and documentation—three comprehensive demo notebooks and docs/USAGE.md + docs/QA.md suggest ongoing user support. Recent commits likely focused on bug fixes post-V2 release, but exact PR/issue activity isn't visible in the file list.

git clone https://github.com/myshell-ai/OpenVoice.git
cd OpenVoice
pip install -r requirements.txt
jupyter notebook demo_part1.ipynb

Daily commands: No explicit dev server. Primary interface is Jupyter notebooks: jupyter notebook demo_part1.ipynb for basic cloning. For API usage: from openvoice import se_extractor and instantiate via openvoice_app.py (see docs/USAGE.md for exact patterns). Gradio UI available via setuptools entry point if installed.

• openvoice/api.py — Main public API entry point; all voice cloning and TTS workflows originate here and must be understood before extending the system.
• openvoice/openvoice_app.py — Core application orchestrator that coordinates speech processing, speaker extraction, and model inference; the heart of the cloning pipeline.
• openvoice/se_extractor.py — Speaker embedding extraction module; critical for tone color cloning accuracy and zero-shot cross-lingual capability.
• openvoice/models.py — Defines all neural network architectures (flow matching, attention layers); essential for understanding model capacity and inference behavior.
• openvoice/mel_processing.py — Audio feature extraction and mel-spectrogram processing; fundamental to audio pipeline and quality degradation points.
• openvoice/text/__init__.py — Text processing entry point for multilingual IPA conversion and phoneme handling; required for non-English language support.

Add Support for a New Language

1. Create a new language module in openvoice/text/ (e.g., openvoice/text/french.py) that implements text_to_sequence(text) returning IPA phoneme indices (openvoice/text/french.py)
2. Register the language cleaner and phoneme converter in openvoice/text/init.py by adding to the _CLEANERS and language dispatch dictionaries (openvoice/text/__init__.py)
3. Update openvoice/text/symbols.py to include any new phonemes specific to the language in the _pad and _characters sets (openvoice/text/symbols.py)
4. Test via openvoice.api.TTS_Pipeline(..., language='fr') to ensure text_to_sequence pipeline works end-to-end (openvoice/api.py)

Add a New Voice Style Parameter (e.g., breathiness control)

1. Extend the OpenVoiceApp.tts_to_file(...) signature in openvoice/openvoice_app.py with a breathiness parameter (0.0–1.0) (openvoice/openvoice_app.py)
2. Modify openvoice/models.py to accept the style parameter in the forward pass and condition the decoder via style embeddings or hidden state modulation (openvoice/models.py)
3. Update openvoice/api.py public methods (e.g., instant_voice_cloning) to accept and forward the new style parameter (openvoice/api.py)
4. Expose the parameter in the Gradio demo notebooks (demo_part1.ipynb, demo_part2.ipynb) with a slider UI widget (demo_part1.ipynb)

Improve Speaker Embedding Quality

1. Review and modify the WavLM feature extraction in openvoice/se_extractor.py (e.g., layer selection, pooling strategy) to change embedding representation (openvoice/se_extractor.py)
2. Retrain or fine-tune the downstream projection layer in openvoice/models.py to map the new embeddings to tone color space (openvoice/models.py)
3. Validate zero-shot cloning accuracy across languages using the demo notebooks; iterate on mel_processing.py if audio quality degrades (openvoice/mel_processing.py)

Add a Custom Voice Encoding Backend (e.g., replace WavLM with another SSL model)

1. Create a new extractor class in openvoice/se_extractor.py (e.g., XLSRSpeakerExtractor) that loads an alternative pretrained model (openvoice/se_extractor.py)
2. Update openvoice/openvoice_app.py to accept an extractor_type parameter and instantiate the appropriate backend (openvoice/openvoice_app.py)
3. Retrain tone color projection head in openvoice/models.py if the new embeddings have different dimensionality or distribution (openvoice/models.py)

• PyTorch + flow matching models — Enables high-fidelity, controllable generative speech synthesis with granular style control over emotion, accent, and prosody
• WavLM speaker embeddings (openvoice/se_extractor.py) — Provides speaker-invariant, language-agnostic tone color representation essential for zero-shot cross-lingual voice cloning
• Mel-spectrogram features (openvoice/mel_processing.py) — Compact, interpretable intermediate representation that separates content (phonemes) from style (speaker identity); enables efficient neural processing
• IPA phoneme encoding (openvoice/text/) — Language-universal phonetic representation allows single model to handle English, Mandarin, and other languages without retraining
• Attention mechanisms (openvoice/attentions.py) — Enables alignment between phonemes and acoustic features; supports flexible duration modeling for rhythm/pause control

• Single multilingual model instead of language-specific models

  • Why: Reduces model size, simplifies deployment, and enables zero-shot transfer to unseen language pairs
  • Consequence: May sacrifice per-language accuracy vs. specialized models; requires careful IPA phoneme inventory design to avoid conflicts

• Flow matching (stochastic) over autoregressive generation

  • Why: Enables parallel decoding (2–5s for 20s audio) instead of sequential generation; better controllability over style parameters
  • Consequence: Requires more training data and compute; slightly higher latency than non-neural methods (e.g., vocoder-only); quality depends on WavLM embeddings

• Tone color separated from content (speaker_embedding independent of phoneme sequence)

  • Why: Allows zero-shot cloning without speaker adaptation; clean separation of concerns
  • Consequence: Cannot capture speaker-specific phonetic variations (e.g., idiolects); tone color transfer may not handle extreme accents

• Mel-spectrogram → vocoder (librosa/griffin-lim implied) instead of end-to-end raw waveform

  • Why: Simpler pipeline; leverages mature vocoding; reduces model parameters
  • Consequence: Vocoder quality ceiling limits final audio fidelity; two-stage inference introduces artifacts at mel-to-audio boundary

• Real-time streaming synthesis (latency ~2–5s per batch, not frame-level online)
• Speaker diarization or multi-speaker transcription (reference audio must be clean single-speaker)
• Emotion transfer learning from unlabeled data (requires manual emotion labels or reference clips)
• Singing voice synthesis (model trained only on speech; no prosody for melody or vibrato)
• Privacy-preserving inference (embeddings are not anonymized; speaker identity is preserved)
• Offline operation without model downloads (requires ~500MB–2GB pretrained model weights)

Model weights not in repo: openvoice_app.py loads pretrained models from external URLs or HuggingFace—internet access required on first run, no offline-mode obvious. Mel-spectrogram config coupling: mel_processing.py n_fft/hop_length/mel_bins must match models trained on those exact parameters; mismatch causes silent failures. Language auto-detection: se_extractor.py likely relies on langid==1.1.6 for automatic reference language detection—ambiguous inputs (code-mixed audio) may fail gracefully or mislabel. Whisper dependency: faster-whisper==0.9.0 used for transcription; older versions may not support all languages. No explicit GPU requirement check: assumes CUDA/GPU available for inference but silently falls back to CPU with poor performance—no validation in startup.

• Speaker Embedding Extraction — The core innovation enabling zero-shot voice cloning—encodes a reference speaker's timbre into a fixed-size vector (se_extractor.py) that conditions synthesis without per-speaker training.
• Mel-Spectrogram — Intermediate audio representation used throughout OpenVoice (mel_processing.py) to bridge text embeddings and waveform generation; critical to understand for audio quality tuning.
• Zero-Shot Cross-Lingual Transfer — OpenVoice's headline capability: cloning voices and synthesizing speech in languages absent from both training data and reference audio—requires speaker embeddings decoupled from language.
• Flow Matching / Continuous Normalizing Flows — Likely used in models.py for mel-spectrogram generation (newer alternative to diffusion models); enables high-quality audio synthesis with fewer inference steps.
• Grapheme-to-Phoneme (G2P) and IPA Conversion — Text processing in openvoice/text/mandarin.py and english.py relies on phonetic conversion (eng_to_ipa, pypinyin) to ensure consistent pronunciation across languages.
• Vocoder — Final synthesis stage in the pipeline (trained models in openvoice/models.py) converts mel-spectrograms to waveforms; quality directly impacts audio naturalness.
• Style Control via Latent Conditioning — OpenVoice's ability to control emotion, rhythm, and accent (mentioned in README) likely works by conditioning the synthesis model on style tokens extracted from reference audio—implemented in se_extractor.py or models.py modules.

• coqui-ai/TTS — Predecessor Coqui TTS framework that OpenVoice builds upon; shares text-to-mel and vocoder components but OpenVoice adds speaker embedding extraction for zero-shot cloning.
• openai/whisper — Speech recognition backbone used by OpenVoice's faster-whisper fork for transcribing reference audio during speaker embedding extraction.
• myshell-ai/MyShellAI — Production deployment platform for OpenVoice; shows real-world usage patterns and integration with the MyShell voice cloning product.
• mozilla/TTS — Alternative open-source TTS engine with multilingual support; closer architectural peer to OpenVoice in the zero-shot TTS space.
• NVIDIA/Glow-TTS — Flow-based text-to-speech model that likely inspired OpenVoice's flow-matching components visible in models.py.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add unit tests for openvoice/se_extractor.py speaker embedding extraction

The se_extractor.py module handles critical speaker embedding extraction but has no corresponding test file. This is essential for voice cloning accuracy. A test suite would validate embedding consistency across different audio inputs, edge cases (short clips, noisy audio), and ensure model outputs remain stable across versions.

• [ ] Create tests/test_se_extractor.py with fixtures using resources/demo_speaker*.mp3 files
• [ ] Add tests for embedding shape validation, consistency checks across multiple runs
• [ ] Test error handling for invalid audio formats and corrupted files
• [ ] Add integration test verifying embeddings work with openvoice/api.py
• [ ] Document test setup in docs/USAGE.md for contributor onboarding

Add language-specific integration tests for openvoice/text/ modules

The text processing pipeline (cleaners.py, english.py, mandarin.py, symbols.py) has critical dependencies on external libraries (pypinyin, jieba, cn2an, eng_to_ipa, langid) but lacks integration tests. Missing tests risk silent failures when processing non-English or Mandarin text, affecting the zero-shot multilingual claim in the README.

• [ ] Create tests/test_text_processing.py covering english.py text normalization
• [ ] Add tests/test_mandarin_processing.py for Mandarin tokenization with jieba/pypinyin
• [ ] Test symbol set completeness across languages in tests/test_symbols.py
• [ ] Add langid language detection validation tests
• [ ] Document expected behavior in docs/USAGE.md for supported languages/accents

Add GitHub Actions CI workflow for multi-language TTS validation

No CI pipeline exists to validate the core TTS functionality across languages. Given the repo's emphasis on multilingual and accent support, a GitHub Action would catch regressions in voice cloning quality, text processing, and model inference without manual testing.

• [ ] Create .github/workflows/test-tts.yml with Python 3.9+ matrix
• [ ] Add steps to test openvoice/api.py and openvoice_app.py initialization
• [ ] Include test for mel_processing.py and mel spectrogram generation stability
• [ ] Test text cleaning pipeline with English and Mandarin samples from demo notebooks
• [ ] Add artifact upload for generated audio samples (comparing against baseline)
• [ ] Document CI expectations and local test commands in CONTRIBUTING.md

• Add unit tests for openvoice/mel_processing.py covering mel-to-waveform inversion and round-trip consistency—currently no test directory visible suggests this gap exists and is safe for a junior to tackle.
• Document the exact speaker embedding space properties in docs/USAGE.md (dimensionality, normalization, similarity metric) to help users understand what se_extractor.py outputs and when embeddings are compatible.
• Implement language auto-detection fallback with explicit warnings in openvoice_app.py when langid confidence is below a threshold (e.g., 0.8)—currently unclear how misdetection is handled.

The OpenVoice codebase has moderate security concerns, primarily stemming from

• High · Outdated and Vulnerable Dependencies — requirements.txt. Multiple dependencies have known vulnerabilities and are significantly outdated. numpy==1.22.0 (released Jan 2022) and librosa==0.9.1 contain known security issues. gradio==3.48.0 is an old version with potential security flaws. These packages should be updated to their latest stable versions. Fix: Update all dependencies to their latest stable versions. Run 'pip list --outdated' and update using 'pip install --upgrade <package>'. Specifically: numpy should be >=1.24.0, gradio should be >=4.0.0 or later.
• High · Use of python-dotenv Without Secure Credential Handling — requirements.txt, .gitignore. The dependency on 'python-dotenv' suggests .env files are being used for configuration. If .env files are accidentally committed to the repository or contain hardcoded secrets (API keys like OpenAI), this poses a credential exposure risk. The README does not document secure credential handling practices. Fix: Ensure .env files are properly listed in .gitignore. Use environment variables or secure secret management systems (AWS Secrets Manager, HashiCorp Vault) for production. Never commit .env files with real credentials. Document credential handling in SECURITY.md.
• Medium · Unvalidated External Audio Input Processing — openvoice/se_extractor.py, openvoice/openvoice_app.py, openvoice/api.py. The codebase processes audio files (demo_speaker*.mp3, example_reference.mp3) and includes audio processing libraries (pydub, librosa, whisper). Without explicit input validation in se_extractor.py, openvoice_app.py, and api.py, there's risk of malformed audio files causing DoS or unexpected behavior. Fix: Implement file type validation (magic bytes checking), file size limits, and timeout mechanisms for audio processing. Validate audio format before processing. Add try-catch blocks around external library calls (librosa, pydub).
• Medium · Potential Code Injection via Text Processing — openvoice/text/cleaners.py, openvoice/text/english.py, openvoice/text/mandarin.py. The text processing modules (openvoice/text/cleaners.py, english.py, mandarin.py) use regex and string manipulation. If user input is not properly sanitized before processing, there could be injection vulnerabilities, especially with the jieba segmentation library handling untrusted text input. Fix: Implement strict input validation and sanitization for all user-provided text. Use allowlists for character sets. Avoid eval() or exec(). Document expected input formats. Add input length limits.
• Medium · OpenAI API Key Exposure Risk — requirements.txt, openvoice/api.py (assumed). The dependency on 'openai' package suggests API integration. Combined with python-dotenv, there's risk that OpenAI API keys could be exposed through environment variables, logs, or error messages if not carefully handled. Fix: Never log API keys or pass them in URLs. Use secure credential managers. Rotate API keys regularly. Implement API key scoping and monitoring. Document API key handling procedures.
• Low · Missing Security Documentation — Repository root. There is no SECURITY.md file, security policy, or documented vulnerability disclosure process. The project lacks security best practices documentation for contributors and users. Fix: Create SECURITY.md with: vulnerability disclosure policy, security best practices, dependency update policy, and contact information for security issues. Add a CONTRIBUTORS.md with security guidelines.
• Low · Gradio Web Interface Security — requirements.txt, openvoice/openvoice_app.py (assumed). gradio==3.48.0 is used for the web interface. Older Gradio versions may have CSRF, XSS, or other web vulnerabilities. Gradio interfaces can expose models to unauthorized access if not properly secured. Fix: Update gradio to the latest version. Implement authentication and authorization for the web interface. Configure CORS policies appropriately. Use HTTPS in production. Review Gradio security documentation.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/myshell-ai/OpenVoice shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live myshell-ai/OpenVoice repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/myshell-ai/OpenVoice.

What it runs against: a local clone of myshell-ai/OpenVoice — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in myshell-ai/OpenVoice | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 415 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of myshell-ai/OpenVoice. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/myshell-ai/OpenVoice.git
#   cd OpenVoice
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of myshell-ai/OpenVoice and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "myshell-ai/OpenVoice(\\.git)?\\b" \\
  && ok "origin remote is myshell-ai/OpenVoice" \\
  || miss "origin remote is not myshell-ai/OpenVoice (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "openvoice/api.py" \\
  && ok "openvoice/api.py" \\
  || miss "missing critical file: openvoice/api.py"
test -f "openvoice/openvoice_app.py" \\
  && ok "openvoice/openvoice_app.py" \\
  || miss "missing critical file: openvoice/openvoice_app.py"
test -f "openvoice/se_extractor.py" \\
  && ok "openvoice/se_extractor.py" \\
  || miss "missing critical file: openvoice/se_extractor.py"
test -f "openvoice/models.py" \\
  && ok "openvoice/models.py" \\
  || miss "missing critical file: openvoice/models.py"
test -f "openvoice/mel_processing.py" \\
  && ok "openvoice/mel_processing.py" \\
  || miss "missing critical file: openvoice/mel_processing.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 415 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~385d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/myshell-ai/OpenVoice"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.