Music vs cinematic source separation for narration over songs
Two kinds of separator, given narration over songs with lyrics. The music model put the singer in the speech stem every time. The cinematic one mostly did not.
By Parker Roan · · 7 min read
I wanted to add a feature to Recastr: keep a video's background music when its narration is replaced. Recastr transcribes a recording, lets you edit the transcript, and generates a new voice that lands on the original timing. Until now the soundtrack went with the old voice. Keeping it means splitting the original audio into the speech and everything else before anything happens, and that split is a source separation problem.
I expected to pick the best-known separation model and move on. Instead I found two kinds of model that answer different questions, and the difference decided the feature. Here is what I learned about both, what each one does to the same clip, and which one Recastr uses.
Two kinds of separation model
Source separation models come in two classes, and the class is set by the training data.
Music source separation is the established one. Its models learn from studio multitracks of songs and return the parts a song is made of. Cinematic audio source separation is younger and smaller. Its models learn from film-style mixtures and return dialogue, music and effects. Both take one mixed track and give back several, and both can be run with an API key. What differs is the set of categories each model has been taught, and that decides where a singer goes.
| Music source separation | Cinematic audio source separation | |
|---|---|---|
| Built for | Songs | Film and TV soundtracks |
| Trained on | Studio multitracks, such as MUSDB18 and larger private sets | Dialogue, music and effects mixtures, such as Divide and Remaster |
| Returns | Vocals, drums, bass, other | Dialogue, music, effects |
| A singer is | A vocal | Music |
| A narrator is | A vocal | Dialogue |
| Open models | Demucs, Open-Unmix, BS-RoFormer | Bandit, Bandit v2 |
| Hosted APIs | Many | Few |
For speech over an instrumental, either class works. For speech over a song with lyrics, only one of them is answering the right question.
Music source separation
A music separator is trained to take a finished song apart. Its training set is multitrack recordings where every stem is known, so it learns what vocals, drums, bass and everything else sound like inside a mix. Years of benchmarks and an annual challenge have made these models very good, and the leading ones are hosted by several providers. Meta's Demucs is the best-known open model in the class and the one tested here. To pull the lead vocal out of a pop record, it is the right tool.
The category it has for a human voice is "vocals". A singer is a vocal. A narrator is also a vocal. When both are present the model does what it was trained to do and returns them in one stem, leaving the bed with a hole where the singer was. Transcribe the vocals stem and the transcript contains a chorus. No amount of model quality fixes this, because the model is answering a different question from the one being asked.
Cinematic audio source separation
A cinematic separator is trained on the other kind of mix, a soundtrack, where a person talking sits over a score and a bed of effects. The distinction it learns is whether a sound is someone speaking or part of the music, which is the distinction a voiceover needs. In this world a sung vocal is music, and dialogue is only dialogue.
The class is far smaller. The datasets are newer, there is less tooling, and few providers host the models. The strongest open model is Bandit v2, from Karn Watcharasupat and Alexander Lerch, released in 2024 with Apache-2.0 code and CC BY-SA 4.0 weights, including a multilingual checkpoint. It is the representative tested here.
What each class does with the same clip
The clearest way to see the difference is to listen to every stem for one clip. The clip is a synthetic narration about cycling cadence over Josh Woodward's "Morning Blue", a track built on dense vocal harmonies, mixed at -10 dB under the voice. Pick a stem and the player keeps its place, so you can hear the same moment across all of them.
Input
Music model (Demucs)
Cinematic model (Bandit v2)
That is the hardest of the five clips in the benchmark below. A more typical one is "Troublemaker", an up-tempo song with a male lead in the chorus, under a narration about planting tomatoes. Listen to the last fifteen seconds, after the narrator stops.
Input
Music model (Demucs)
Cinematic model (Bandit v2)
The effects stems are close to silent because these mixes have no effects track. What is left in them is residue the model could not place.
Choosing one for Recastr
To make the choice on more than two clips, I ran both models over five and scored the results.
Real videos with music under narration would not do as fixtures. To say that a speech stem has "leaked lyrics", you need to know exactly what the speech was, word for word. Synthetic narration solves that: five passages of unrelated non-fiction read by five OpenAI TTS voices, each mixed at full level over a 55 second window of a Josh Woodward song at -10 dB, with the song alone for the first three seconds. The narration text is in the repository, so any word in a speech stem's transcript that is not in it came from the song.
Each clip went through both models. The music model's vocals stem is its speech stem and its other three stems are summed into a bed. The cinematic model's dialogue stem is its speech stem and its music and effects stems are summed into a bed. The original clip and each speech stem were transcribed with the same speech-to-text model, and the transcripts were compared with the script. Every extra word is a leak, and every missing word is narration the separator damaged.
| Clip | Music model (Demucs) speech stem | Cinematic model (Bandit v2) speech stem |
|---|---|---|
| 1. Troublemaker | narration plus two chorus lines | narration only |
| 2. Coffee | narration plus one lyric line | narration only |
| 3. Morning Blue | narration largely destroyed, lyrics dominate | narration intact, one lyric fragment |
| 4. Learn to Fly | narration plus a full sung verse | narration plus one short fragment |
| 5. Violet (half-spoken) | narration plus several lines | narration plus several lines, fewer |
The music model leaked lyrics into the speech stem on all five clips, and on Morning Blue it took the narrator apart trying to separate two vocals it had been trained to treat as one. The cinematic model returned the narration and only the narration on two clips and leaked a short fragment on two more. Its real miss is Violet, where the vocal is half-spoken and rhythmic. A model trained to tell talking from music is entitled to be confused by a singer who is talking, and the music model did worse on the same clip.
I had also planned a quick number: the level of each bed in the 300 to 3400 Hz speech band, relative to the original mix. Less energy there should mean less narrator left in the bed, so more negative should be better. By that measure the music model won every clip.
It won because it had removed the singer from the bed along with the narrator. A bed that has lost its vocal is a soundtrack with a hole in it, which is the failure the feature exists to avoid. The metric cannot tell "removed the narrator" from "removed the singer", so it rewards the worse result. For separation in service of speech, the transcript is the metric, and so is a pair of ears.
So Recastr's background audio preservation runs on the cinematic class, with Bandit v2 as the model. The boundary is simple, one call in and a speech stem and a bed out, and any separator that follows goes through the same five clips before it goes live.
Five clips, one run each, English narration, one synthetic voice per clip, both models on hosted APIs. That is enough to show a systematic difference between the two classes, because the failure was present on every clip, and not enough to rank two close competitors. A second music separator or a second cinematic one would sharpen the picture, and so would real recorded speech with human-verified transcripts.
Music by Josh Woodward, CC BY 4.0. Narration synthesised with OpenAI gpt-4o-mini-tts. Demucs (htdemucs) run on fal.ai, Bandit v2 (multilingual checkpoint) run on MVSep, transcription by Azure Fast Transcription. Numbers as of 2026-09-05. Clips, stems, transcripts and code: parkerroan/articles.
Try it on your own file
10 free minutes · No credit card required