How Our Research Team Developed a More Efficient Audio Embedding Model

As research projects typically go, where you think you’re headed and where you end up are rarely the same.

Yan Li, one of our Applied Research Engineers, set out to simply evaluate an audio anomaly detection model trained on limited data samples at the request of a colleague. The task quickly evolved into something much more ambitious.

Along the way, Yan used an AI agent running autoresearch sessions to iteratively explore improvements while developing an audio embedding model, intervening with domain expertise when progress stalled. The results? A new model that tackles some of the biggest limitations in today’s state‑of‑the‑art approaches.

While his research is still an internal work in progress, the process he took to put it together provides interesting perspective into the current methodologies and tools that we can now access, and how we can leverage them to create new models, features, and more.

Let's take a look at how he did it and the results he got along the way.

Audio embedding models

An audio embedding model converts audio into a fixed-length vector that captures the important parts of a sound while discarding unnecessary detail.

Instead of working directly with raw waveforms or spectrograms, downstream systems operate on these compact representations.

In practice, this makes embeddings incredibly useful:

• A single embedding model can support many tasks, from speech detection to environmental sound classification
• Downstream models can stay simple, often only a simple classifier or anomaly detector
• The same embedding can generalize across domains, much like ImageNet features in vision applications

Because labeled audio data is scarce, most embedding models are trained with self-supervised learning. The model learns by predicting parts of the input or aligning different views of the same sound, rather than relying on explicit labels. This tends to produce representations that are robust and transferable.

State-of-the-art models

Recent advances in these audio models are dominated by methods like BYOL‑A, Audio‑JEPA, and EAT.

These models share a few key traits:

• They learn embeddings through prediction in latent space rather than reconstructing raw audio, enabling strong general-purpose representations
• They rely on large transformer-based architectures, which drive performance but come with significant compute cost
• They are typically evaluated using embeddings produced by a frozen encoder with simple classifiers downstream, emphasizing transferability across tasks

(If you’re interested in earlier approaches to audio embedding, take a look at VGGish, PANNs, and YAMNet.)

Critical limitations

Despite strong results, the state-of-the-art models have two major drawbacks that create a gap between what performs well in research benchmarks and what can realistically be deployed in production systems.

1. Limited real-world deployability

Most leading models are pretrained on AudioSet. While powerful, AudioSet is not unrestricted for commercial use, which makes these models difficult to deploy without retraining the architecture on alternative datasets.

2. Massive compute requirements

These models are large and expensive to train, often requiring multiple GPUs and thousands of hours of audio. For many teams, reproducing or adapting these models is already a significant barrier.

After training, these models remain large and computationally intensive at inference time. Running them on constrained hardware, such as edge devices, is often impractical without substantial modification or downsizing.

Research objectives

While exploring ways to improve the performance of our colleague’s audio anomaly detection model, Yan evaluated both spectrogram-based features and audio embeddings as inputs, instead of relying on raw audio. Embeddings generated by EAT showed promising results. However, due to the limitations outlined earlier, EAT itself was not a viable option for this use case.

This led to a focused set of research objectives:

• Can a useful audio embedding model be trained without relying on AudioSet?
• Can the model size and compute requirements be reduced significantly?
• Can strong performance on real-world downstream tasks be maintained?

If all three objectives could be satisfied, there would be a practical path toward a deployable solution.

Research approach

Yan started from the Audio-JEPA codebase, and treated it as a baseline for controlled experimentation, for two reasons.

First, it already implements the Joint-Embedding Predictive Architecture (JEPA) training objective and eXtensive Audio Representation and Evaluation Suite (X-ARES) flow described in the accompanying paper, so it is a credible starting point.

Second, the codebase is organized around PyTorch Lightning and Hydra, which made it practical to run controlled experiments: model variants, masking schemes, optimizers, schedulers, and data settings could all be changed through small configuration edits instead of repeatedly rewriting training code.

From there, he made a series of targeted changes that are described below.

Make the model smaller by design

The first change was architectural. The JEPA training objective was kept, but the standard ViT patch embedding layer was replaced with an EAT-style convolutional patchifier. The transformer encoder itself was also made smaller than the baseline configuration. The main benefit of this frontend design was that it became feasible to use a much smaller ViT backbone while preserving useful representation quality.

Train on open data

AudioSet was replaced with Freesound Dataset 50k (FSD50K), a fully open dataset better suited for commercial use. While much smaller, it allowed iteration and deployment without licensing constraints.

For scale, see the substantial differences in the two corpora in the following table.

To keep the experimentation loop fast enough for repeated agent-driven runs, mel spectrograms were precomputed and stored as cached shards on S3. Training jobs then streamed these prebuilt tensors instead of recomputing frontend features each time.

This had a meaningful operational impact. With each train-and-benchmark cycle already taking around 20 minutes, eliminating repeated audio decoding and mel extraction saved enough time to make larger hyperparameter and architecture sweeps practical.

Focus the model on meaningful signal

Environmental audio often contains long stretches of low-value background noise, while the events of interest are often short and weak. By applying normalization together with a noise floor during preprocessing, the model was biased toward informative parts of the signal.

Automate iteration with hyperparameter search

Hydra’s Optuna was used to explore model and training configurations efficiently. After each sweep, the agent reviewed train-and-benchmark outputs and promoted strong trials into reproducible experiments for follow-up runs.

Extend training across datasets

After the initial training on FSD50K, model pretraining continued in stages, resuming from the selected checkpoint and training sequentially on additional datasets, UrbanSound8K and FSDnoisy18k.

Optimize for transfer, not just training loss

The model was evaluated where it mattered most: downstream performance.

Instead of focusing purely on self-supervised training loss, the candidate embedding models were evaluated directly on real-world tasks using X-ARES to measure frozen encoder performance on two benchmark datasets, Environmental Sound Classification 50 (ESC-50) and Domestic Environmental Sound Event Detection (DESED), with optional evaluation on FSD50K.

This shifted the objective from “lowest training loss” to “best transfer quality,” which is what ultimately matters in practice. In self-supervised setups, improvements in training loss can be made while transfer quality plateaus or regresses.

For this reason, the models were trained for a fixed number of epochs and evaluation on downstream tasks was prioritized, rather than relying on validation loss for early stopping during model training.

Agent involvement

An integral part of the research approach was the use of an AI agent to drive experimentation.

For this project, the GitHub Copilot CLI in autopilot mode, powered by the GPT-5.4 model with a high reasoning setting, was used as the research agent. In practice, the agent functioned less like an independent researcher and more like a highly efficient optimization engine.

The agent operated in a tight autoresearch based loop, proposing a change, running a train-and-benchmark cycle, evaluating the results, and keeping or discarding the idea. It explored configurations quickly and systematically, while Yan guided overall direction and stepped in when needed.

A typical session lasted upwards of 9 hours, involved around two dozen experiments, each taking 15–30 minutes for training and benchmarking, used several premium requests, and consumed more than 20 million tokens.

The agent was given a concrete operating program in a program.md file, including the instruction that this was "an experiment to have the research agent improve the (EA-)JEPA model, especially when training on small datasets" and that "the goal is simple: get the highest ESC-50 MLP benchmark score."

For additional technical context, the agent was also pointed to a PAPER.md file that summarized the associated Audio-JEPA paper and helped ground the experiment search in the original JEPA design and evaluation framing.

Results

The results demonstrated the main objective of this work: improving embedding quality for downstream use while moving toward a deployment-feasible model size.

Key outcomes:

• Model parameter count reduced from 85M to 14M: 71M fewer parameters (83.5% reduction, 6.1x smaller)
• ESC-50 MLP accuracy increased from 0.338 to 0.6055: +0.2675 absolute (+79.1% relative)
• ESC-50 k-NN accuracy increased from 0.140 to 0.461: +0.321 absolute (+229.3% relative)

Improvement in both MLP and k-NN accuracy using embeddings from the new model suggests that the encoder is learning more transferable structure in the embedding space rather than simply fitting to a particular downstream classifier.

MLP gains indicate that the embeddings contain information that can be separated more effectively by a small learned nonlinear classifier. k-NN gains indicate that semantically similar samples are naturally grouped together in the embedding space, meaning the representation itself has stronger local structure without requiring an additional learned transformation.

Observations

A few patterns emerged from working with the agent. It performed best when objectives, constraints, and success metrics were clearly defined, and often benefited from being prompted to ask clarifying questions before acting.

Most of the agent gains came from optimization and tuning rather than novel ideas; larger architectural changes still required human direction. Early iterations tended to deliver the strongest improvements, with diminishing returns later in a session. Overall, the agent was highly effective as an optimization engine but human expertise and domain knowledge remained essential for guiding direction and interpreting experimental results.

Final thoughts

By combining a smaller architecture, open data, and a transfer-focused evaluation approach, Yan was able to build a model that is both more efficient and more practical to deploy.

This work showed that creating quality audio embeddings does not require massive models or restricted datasets. However, this still remains a difficult open problem. The current system should be viewed as a working research platform rather than a finished edge model.

Just as importantly, this project highlighted a new research pattern. Agents can dramatically accelerate iteration but meaningful progress still depends on human insight, especially when navigating trade-offs and defining the right objectives.

The result was a compelling foundation for building efficient, deployable audio systems, and a glimpse at how human and machine collaboration can reshape the research process.

Yan and the research team continue to work on this new model; we'll share more details as that progresses.