Nvidia Jumps Into AI Music Space With New Audio Generator Fugatto

Nvidia, the computer chip giant, has entered the AI music race by announcing its new model, Fugatto, on Tuesday (Nov. 26). The company calls Fugatto, short for Foundational Generative Audio Transformer Opus 1, a “Swiss Army knife for sound.”

Using text or audio prompts, Fugatto can generate new music at the click of a button and edit existing audio, including removing or adding instruments from a song or changing the accent and emotion in a voice, in seconds.

With Fugatto, Nvidia aims to take on today’s top AI music models, including Suno, Udio and many more. Though it is a late entrant in the race to create the best music AI model, Fugatto appears to have crisp audio quality and a number of capabilities that could change the music-making process for producers and composers.

According to the announcement on Nvidia’s blog, “One of the hardest parts of the effort was generating a blended dataset that contains millions of audio samples used for training,” which the company says it worked on for more than a year to get right. “The team employed a multifaceted strategy to generate data and instructions that considerably expanded the range of tasks the model could perform, while achieving more accurate performance and enabling new tasks without requiring additional data.” The company says its model is trained on open-source datasets under the Creative Commons license and complies with copyright law.

Nvidia proposes a number of use cases for Fugatto, including generating a score for visual media; editing certain parts of a score; and altering a voice to have different accents, emotions and timbres. “Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create,” says Rafael Valle, a manager of applied audio research at Nvidia.

“The history of music is also a history of technology,” says Ido Zmishlany, a producer/songwriter and co-founder of One Take Audio, a member of Nvidia Inception, its program for cutting-edge startups. “With AI we’re writing the next chapter of music. We have a new instrument, a new tool for making music — and that’s super exciting.”

Nvidia claims this is the first AI music model that showcases “emergent properties — capabilities that arise from the interaction of its varous trained abilities — and the ability to combine free-form instructions.” Valle adds that Fugatto is “our first step toward a future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale.”

There’s only one catch: so far, the company describes it as an internal research project, not available to the public.