WAIT — Mixed signals — read the receipts

• Last commit 2mo ago
• 15 active contributors
• Other licensed
• CI configured
• Tests present
• ⚠ Concentrated ownership — top contributor handles 59% of recent commits
• ⚠ Non-standard license (Other) — review terms

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

Real-Time Voice Cloning implements the SV2TTS (Speaker Verification to Text-To-Speech) framework, which clones a voice from a 5-second audio sample and generates arbitrary speech in real-time using three stages: a speaker encoder (GE2E), a Tacotron synthesizer, and a WaveRNN vocoder. It enables one-shot voice synthesis by extracting speaker embeddings and applying them to control a TTS model's output voice characteristics. Three-stage pipeline architecture: encoder/ implements speaker verification (GE2E model, speaker_batch.py for training data), synthesizer/ implements Tacotron TTS (tacotron.py for mel-spectrogram generation), and vocoder (imported from fatchord/WaveRNN) converts spectrograms to audio. Preprocessing and training pipelines are split into standalone scripts (encoder_preprocess.py, encoder_train.py, synthesizer_preprocess_audio.py) while inference is unified in demo_cli.py and demo_toolbox.py.

Machine learning engineers and researchers building voice cloning applications, developers needing open-source speech synthesis without paying SaaS fees, and academics studying speaker verification and multi-speaker TTS. Users interested in real-time inference rather than offline batch synthesis.

This repository is experimental and aging (master's thesis project from ~2018-2019). While it has CI configured in .github/workflows/ci.yml and working demo scripts (demo_cli.py, demo_toolbox.py), the README explicitly warns that 'Like everything else in Deep Learning, this repo has quickly gotten old' and recommends checking Paperswithcode and newer projects like Chatterbox for SOTA voice quality. The codebase appears unmaintained with no recent commits visible in the file list.

This is a research-grade project with significant maintenance risk: no clear dependency versioning visible (likely outdated PyTorch/TensorFlow pins for a 2018 thesis), unmaintained status documented in README, reliance on external pretrained models (encoder/synthesizer weights not in repo), and complex audio processing pipelines (FFmpeg dependency, multiple specialized neural architectures). Production use would require substantial updates to modern library versions and likely speaker embedding quantization for real-time performance.

Based on the README warnings and structure, this repo is in maintenance-only mode with no active development. The CI workflow in .github/workflows/ci.yml suggests basic testing infrastructure exists, but the explicit note pointing users to Chatterbox and Paperswithcode indicates the authors view this as superseded by newer approaches.

git clone https://github.com/CorentinJ/Real-Time-Voice-Cloning.git
cd Real-Time-Voice-Cloning
# Install ffmpeg first (check with: ffmpeg)
uv pip install -r requirements.txt  # if exists, or use pip
python demo_cli.py

Daily commands:

python demo_cli.py  # Command-line voice cloning
python demo_toolbox.py  # Interactive UI demo (requires gradio or similar)

See encoder_train.py and synthesizer/train.py for training custom models on speaker verification and TTS datasets.

• demo_cli.py — Entry point for CLI usage—demonstrates the complete SV2TTS pipeline (encoder → synthesizer → vocoder) end-to-end.
• encoder/inference.py — Core inference module for speaker embedding extraction; used by both synthesizer and toolbox to encode reference voice.
• synthesizer/inference.py — Text-to-mel-spectrogram generation using Tacotron; bridges encoder embeddings to vocoder input.
• vocoder/inference.py — Mel-to-audio conversion (WaveRNN); final stage converting spectrograms to playable waveforms.
• encoder/model.py — GE2E speaker encoder architecture; defines the embedding model that powers voice cloning.
• synthesizer/models/tacotron.py — Tacotron2 variant model for seq2seq speech synthesis; core text-to-spectrogram architecture.
• vocoder/models/fatchord_version.py — WaveRNN vocoder implementation; real-time audio generation backbone.

Add a new speaker dataset for encoder training

1. Create a new dataset class inheriting from SpeakerVerificationDataset in encoder/data_objects/speaker_verification_dataset.py, implementing __getitem__ to return (speaker_id, utterances_batch). (encoder/data_objects/speaker_verification_dataset.py)
2. Update encoder_preprocess.py to call your dataset's preprocessing logic (extract mel-spectrograms, organize by speaker). (encoder_preprocess.py)
3. Modify encoder/train.py data loader instantiation to use your new dataset class and pass custom speaker/utterance count. (encoder/train.py)

Add a new text-to-speech language support

1. Create a new language-specific text processor in synthesizer/utils/ (e.g., cleaners_lang.py) with grapheme-to-phoneme or transliteration logic. (synthesizer/utils/text.py)
2. Update synthesizer/utils/symbols.py to include language-specific graphemes/phonemes. (synthesizer/utils/symbols.py)
3. Modify synthesizer/inference.py to detect language and apply the appropriate text normalizer before Tacotron encoding. (synthesizer/inference.py)

Replace the WaveRNN vocoder with an alternative (e.g., HiFi-GAN)

1. Create a new vocoder model class in vocoder/models/ (e.g., hifi_gan.py) implementing forward(mel) → waveform. (vocoder/models/fatchord_version.py)
2. Update vocoder/inference.py to conditionally load the new model and provide the same interface (infer function signature). (vocoder/inference.py)
3. Create a new training script vocoder_train_hifigan.py following the structure of vocoder_train.py but using your model and loss functions. (vocoder_train.py)

Add real-time voice modification (e.g., pitch/speed control) to the toolbox

1. Add UI sliders/inputs in toolbox/ui.py for pitch shift and time-stretch parameters. (toolbox/ui.py)
2. Create a new audio post-processing module (e.g., utils/audio_effects.py) with pitch-shifting and time-warping functions (using librosa or pyrubberband). (utils/profiler.py)
3. Integrate the post-processing into demo_toolbox.py after vocoder inference, applying effects before playback. (demo_toolbox.py)

• PyTorch — Deep learning framework for training GE2E encoder, Tacotron2 synthesizer, and WaveRNN vocoder; enables GPU acceleration for inference.
• Librosa / scipy.signal — Audio DSP library for mel-spectrogram computation, resampling, and audio I/O; standard in speech processing pipelines.
• PySimpleGUI (toolbox) + CLI argparse — Lightweight GUI and CLI for accessibility; allows both batch scripting and interactive real-time usage without heavy dependencies.

No explicit .env or config file visible—check encoder/params_model.py and encoder/params_data.py for hardcoded constants (batch size, learning rates, audio frame parameters). FFmpeg must be installed system-wide and callable from PATH. Pretrained model weights are not in the repo (likely expects downloads from external sources—check demo_cli.py and demo_toolbox.py for weight loading code). The encoder/synthesizer are trained separately on different datasets (speaker verification datasets for encoder, TTS datasets for synthesizer), so both pretrained models must be available. Audio sample rate handling is implicit in encoder/params_data.py and synthesizer/hparams.py—mismatches between preprocessing and inference will silently produce poor quality. GE2E loss requires batch_size >= 64 speakers per batch (encoder/params_model.py likely specifies this).

• Generalized End-to-End (GE2E) Loss — The speaker encoder uses GE2E loss (arXiv 1710.10467) instead of triplet loss for more efficient speaker verification training with large batch sizes, directly enabling the voice embedding quality in real-time cloning.
• Speaker Embeddings / Speaker Verification — The core transfer learning mechanism: a pretrained speaker verification model extracts fixed-size embeddings (typically 256-512 dims) from voice samples, which are then used as control signals for the synthesizer to clone voice identity.
• Tacotron (Sequence-to-Sequence with Attention) — The synthesizer uses attention-based seq2seq to convert text tokens to mel-spectrograms, conditioned on speaker embeddings—understanding attention mechanics is essential for debugging synthesis failures.
• Mel-Spectrogram — Intermediate representation between text (Tacotron output) and waveform (WaveRNN input)—handles nonlinear frequency perception (mel-scale) to reduce computational cost while preserving perceptual relevance for audio synthesis.
• Voice Cloning via Transfer Learning (SV2TTS) — The overall framework (arXiv 1806.04558) transfers speaker characteristics learned in verification to TTS synthesis, enabling one-shot voice cloning without retraining the synthesizer—core innovation in this repo.
• WaveRNN (Efficient Neural Audio Synthesis) — The vocoder generates high-quality waveforms from mel-spectrograms in real-time by predicting audio samples autoregressively with sample-level conditioning—critical for production audio quality in this pipeline.
• Data Augmentation via Speaker Verification Datasets — The encoder leverages large speaker verification datasets (TIMIT, VoxCeleb referenced in samples/VCTK.txt) to learn speaker-discriminative embeddings, which transfer to unseen speakers without retraining the synthesizer.

• fatchord/WaveRNN — The vocoder used in this pipeline for converting mel-spectrograms to high-quality audio in real-time.
• resemble-ai/chatterbox — Recommended successor repo in the README—modern voice cloning with 2025 SOTA speaker embeddings and synthesis quality.
• mozilla/TTS — Active open-source TTS framework supporting multi-speaker synthesis and GE2E speaker embeddings, better maintained alternative.
• espnet/espnet — End-to-end speech processing framework with modern encoder-decoder TTS, speaker verification modules, and vocoder implementations.
• keithito/tacotron — Reference Tacotron implementation for single-speaker TTS—educational baseline for understanding the synthesizer architecture in this repo.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for encoder module (encoder/)

The repo has minimal test coverage with only test_ci_smoke.py. The encoder module (speaker verification) is critical to the SV2TTS pipeline but lacks unit tests for key components like audio.py, model.py, and inference.py. This would improve reliability and catch regressions early.

• [ ] Create tests/test_encoder_audio.py for encoder/audio.py audio processing functions
• [ ] Create tests/test_encoder_model.py for encoder/model.py architecture and forward passes
• [ ] Create tests/test_encoder_inference.py for encoder/inference.py voice embedding extraction
• [ ] Add fixtures for sample audio loading (integrate with samples/ directory)
• [ ] Expand tests/test_ci_smoke.py with encoder-specific smoke tests

Add GitHub Actions workflow for model download and pre-trained weight validation

The repo uses pre-trained models (referenced in utils/default_models.py) but lacks CI validation that downloaded models are valid and uncorrupted. Currently only ci.yml exists. Adding a workflow to verify model integrity would catch broken downloads early and document the expected model structure.

• [ ] Create .github/workflows/model_validation.yml workflow
• [ ] Implement model weight hash verification (create checksums for encoder, synthesizer, vocoder models)
• [ ] Add utils/model_validator.py to check model file integrity and architecture
• [ ] Document expected model paths and sizes in README.md
• [ ] Test workflow with sample pre-trained weights from utils/default_models.py

Refactor synthesizer module into separate concern modules and add integration tests

The synthesizer/ directory combines text processing (utils/), model definition (models/tacotron.py), and inference (inference.py) but lacks organization. The utils/text.py, utils/cleaners.py, and utils/numbers.py should be extracted and tested independently. Add tests/test_synthesizer_integration.py to verify end-to-end text-to-mel conversion.

• [ ] Create tests/test_synthesizer_text_processing.py for synthesizer/utils/text.py, cleaners.py, and numbers.py
• [ ] Create tests/test_synthesizer_tacotron.py for synthesizer/models/tacotron.py forward pass with varying input lengths
• [ ] Create tests/test_synthesizer_inference.py for synthesizer/inference.py with different text inputs
• [ ] Add integration test in tests/test_synthesizer_integration.py: text → embeddings → mel-spectrogram
• [ ] Document expected input/output shapes in synthesizer/ module docstrings

• Add unit tests for encoder/audio.py's mel-spectrogram extraction—currently no test coverage visible in file list, and mel-filterbank implementation is critical for both encoder and synthesizer quality. Start with tests/test_encoder_audio.py.
• Document the pretrained model download/loading pipeline: add a models/README.md explaining where speaker encoder and Tacotron weights come from, how to download them, and which inference.py functions load them. This removes the biggest onboarding friction.
• Implement logging to encoder/train.py and synthesizer/train.py—currently no structured logging visible, making it hard to debug training divergence. Add tensorboard or wandb integration as optional dependency in pyproject.toml.

Failed to generate security analysis.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/CorentinJ/Real-Time-Voice-Cloning shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live CorentinJ/Real-Time-Voice-Cloning repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/CorentinJ/Real-Time-Voice-Cloning.

What it runs against: a local clone of CorentinJ/Real-Time-Voice-Cloning — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in CorentinJ/Real-Time-Voice-Cloning | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 91 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of CorentinJ/Real-Time-Voice-Cloning. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/CorentinJ/Real-Time-Voice-Cloning.git
#   cd Real-Time-Voice-Cloning
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of CorentinJ/Real-Time-Voice-Cloning and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "CorentinJ/Real-Time-Voice-Cloning(\\.git)?\\b" \\
  && ok "origin remote is CorentinJ/Real-Time-Voice-Cloning" \\
  || miss "origin remote is not CorentinJ/Real-Time-Voice-Cloning (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "demo_cli.py" \\
  && ok "demo_cli.py" \\
  || miss "missing critical file: demo_cli.py"
test -f "encoder/inference.py" \\
  && ok "encoder/inference.py" \\
  || miss "missing critical file: encoder/inference.py"
test -f "synthesizer/inference.py" \\
  && ok "synthesizer/inference.py" \\
  || miss "missing critical file: synthesizer/inference.py"
test -f "vocoder/inference.py" \\
  && ok "vocoder/inference.py" \\
  || miss "missing critical file: vocoder/inference.py"
test -f "encoder/model.py" \\
  && ok "encoder/model.py" \\
  || miss "missing critical file: encoder/model.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 91 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~61d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/CorentinJ/Real-Time-Voice-Cloning"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.