Understanding Model Outputs
A model output is the result of one Cyanite model for a track, such as its moods, its BPM or its key. Tagging, AI Music Detection and Segmentation all deliver their results as model outputs. This page explains what a model output is, how it relates to these products, which model outputs exist and the ideas they share, so you can interpret any of them. To fetch the model outputs of a track, see the Tagging guide.
What a model output is
- A model is a part of Cyanite's AI that takes an audio track as input and produces one specific type of musical information as output (for example: mood scores, BPM, or a key prediction).
- A model output is the versioned JSON object returned by one such model,
for example
MoodSimpleV2orBpmV2.
The tags of a track are fields of a model output: tags holds the dominant
labels of a taxonomy, scores the score of every label, and tag the single
value of a musical parameter such as BPM. See
Common field patterns.
Model outputs and products
Model outputs are the technical form of the results. Which of them you get depends on the Cyanite products your account has:
- Tagging describes your music: its genres, moods, instruments, BPM, key and more. It's a set of model outputs, each describing one aspect of a track. Which model outputs your Tagging includes is part of your agreement with Cyanite, and access is granted per model output. Two accounts can therefore have different sets, for example one with genres and moods only and one with the full catalogue below.
- AI Music Detection is a product of its own, available with or without
Tagging. Its result is the
AiMusicDetectionV1model output. See the AI Music Detection guide for how it works. - Segmentation is an add-on to Tagging. Its result is the
SegmentationV1model output, available once Segmentation is enabled for your account. See the Segmentation guide for how it works.
All model outputs are produced by the same analysis and fetched from the same endpoint, whichever product they belong to.
How model outputs are produced
- Upload. You upload a track to your Library.
- Analysis. Cyanite analyzes the track with every model output your account has. You don't choose model outputs per upload.
- Notification. When the analysis is complete, Cyanite calls your webhook
with an
analysisJob/finishedevent. - Retrieval. You fetch the model outputs you need for the track, by their
version, from
GET /v1/library-tracks/{id}/models.
The stored model outputs are also what Search uses: their track-level values can narrow down a search with a metadata filter.
Available model outputs
These are the public model outputs. Each card links to the model's schema in
the API Reference, which describes
every field. The pills show the values a model can return, exactly as the API
returns them. For a model with a single taxonomy, the value count next to the
category links to its vocabulary. Select a card or +N more to see and search all
values of a model. You can also search for a value, such as dark, to find the
model outputs that return it. FreeGenreV3 and TimeSignatureV2 return free text,
so their cards list the known values, and the model can also return others.
Most model outputs belong to
Tagging. Those that belong to another product carry its name, such as
AI Music Detection or Segmentation.
Perceived tempo as a label, such as fast or slow.
Bpm V2
Tempo in beats per minute.
Time Signature V2
Time signature, such as 4/4.
Musical Era V2
Era the track sounds like it was produced in, with an estimated year.
Vocal Style V1
Vocal style.
Vocals V2
Vocal characteristics, vocal presence and predominant vocal gender.
Main Genre V2
Main genres of the track.
Subgenre V2
More specific genres within the main genre, null if none fits.
Free Genre V3
Free-form genre names, including niche genres.
Mood Simple V2
Basic moods.
Mood Advanced V2
Detailed moods.
Character V2
Sonic personality of the track, such as bold or playful.
Movement V2
Rhythmic feel, such as groovy or driving.
Valence Arousal V2
Valence and arousal over time, with energy and emotion categories.
Instruments V2
Instruments and how much of the track they play in.
Music For V1
Use cases the track is suitable for.
Augmented Keywords V3
Weighted keywords that describe the track.
Short natural-language description.
Voiceover V2
Spoken word and how dominant it is.
The part of the track that represents it best.
Segmentation V1
Consecutive sections of the track.
Audio File Info V1
Technical properties of the audio file, such as duration.
Ai Music Detection V1
Whether the track is likely AI-generated.
Example model output
The following example shows a taxonomy-based model output (MoodSimpleV2). It
demonstrates the common scores / tags / segments pattern.
The example is explained in the sections below (see Taxonomy classifiers and Track-level vs segment-level data).
{
"version": "MoodSimpleV2",
"segments": {
"timestampsSeconds": [5.0, 15.0, 25.0],
"values": {
"aggressive": [0.2172, 0.2986, 0.2332],
"calm": [0.216, 0.1058, 0.2894],
"chill": [0.1954, 0.0933, 0.0839],
"dark": [0.4876, 0.2672, 0.5492],
"energetic": [0.5873, 0.635, 0.2871],
"epic": [0.0484, 0.2499, 0.0261],
"happy": [0.2862, 0.136, 0.0894],
"romantic": [0.0218, 0.0706, 0.0022],
"sad": [0.2462, 0.0899, 0.3842],
"scary": [0.3158, 0.0813, 0.2322],
"sexy": [0.3539, 0.3665, 0.1121],
"ethereal": [0.224, 0.1265, 0.2627],
"uplifting": [0.3555, 0.428, 0.1668]
}
},
"scores": {
"aggressive": 0.2497,
"calm": 0.2037,
"chill": 0.1242,
"dark": 0.4347,
"energetic": 0.5032,
"epic": 0.1082,
"happy": 0.1705,
"romantic": 0.0315,
"sad": 0.2401,
"scary": 0.2098,
"sexy": 0.2775,
"ethereal": 0.2044,
"uplifting": 0.3168
},
"tags": ["energetic", "dark"]
}
Versioning
Every model output object contains a version field.
The version is a discriminator (so you can pick the right schema), and it also
tracks breaking changes to the output structure. It defines:
- Which schema to expect (fields and types)
- Which taxonomy/vocabulary applies (for taxonomy-based outputs)
Multiple versions may exist at the same time. Older versions are deprecated over time and API consumers should migrate to the latest available version when possible.
From AudioAnalysis to model outputs
In the Cyanite GraphQL API analysis results are bundled in versioned objects like AudioAnalysisV7. This turned out to be inflexible and had downsides such as:
- Shipping a new major version containing all models if we want to update a single model. That's why we had to introduce new classifier names within a major version like
advancedInstrumentsExtendedfor backwards compatibility. - Always analyzing every model although a customer's use case might require only a single model.
The new model outputs concept allows us to ship models with much greater flexibility.
Track-level vs segment-level data
Many model outputs provide both:
- Track-level results: a single prediction for the whole track (e.g.
tag,scores,tags). - Segment-level results: predictions over time under
segments.
Practical guidance:
- If you need one value or descriptor for the track as a whole, use the
track-level fields (
tag,scores,tags). - If you need how values evolve over the course of the track (for example:
where a mood becomes more intense), use
segments.
Where present, segment-level data is represented as:
segments.timestampsSeconds: array of segment start times (seconds from track start)segments.values: the segment values aligned by index withtimestampsSeconds
The shape of segments.values depends on the model output:
- Taxonomy classifiers:
segments.valuesis an object mapping taxonomy keys to arrays of scores. - Musical parameters:
segments.valuesis an array of predicted values.
In both cases, the array length(s) match segments.timestampsSeconds.length.
Common field patterns
Taxonomy classifiers (scores, segments, tags)
Many classifiers share the same structure:
scores: object mapping taxonomy keys → track-level score (usually0.0-1.0), indicating how strongly the model predicts each label. See Interpreting scores.segments: segment-level scores over time.tags: array of the dominant taxonomy keys (a subset of the keys inscores).
For these model outputs, the precise set of taxonomy keys is fixed for the given
version.
Musical parameters (tag, segments, confidences)
Musical parameter model outputs (e.g. BPM, key, time signature) expose:
-
tag: the predicted global value for the whole track. -
segments: the predicted local values per segment. -
confidences: a structured confidence object with three values in0.0-1.0:modelCertainty: how sure the model is that the predictions are correctpredictionStability: how stable the predictions are across segmentsconfidence: an overall confidence score, the minimum of the two individual scores
Interpretation: Confidence values above
0.9indicate the track is unambiguous and a single global label can be confidently applied.
Other shapes
Some model outputs don't follow either pattern because they describe something else than a classification or a musical parameter, for example:
- a single value, like
isAiMusicandscorein AiMusicDetectionV1 or the generated text in AutoDescriptionV2, - a time range, like RepresentativeSegmentV2,
- the structure of the track, like SegmentationV1.
Their API Reference pages, linked under Available model outputs, describe their fields.
Interpreting scores
In taxonomy classifiers, a score
in scores or segments.values is a value between 0.0 and 1.0 that tells
how strongly one model predicts that a label applies. Musical parameters such as
BpmV2 or KeyV2 have no scores: their segments hold predicted values, and
their confidences tell how
reliable those are.
- Compare scores within one model output. Within
MoodSimpleV2, you can rank the moods of a track by their score, or rank tracks by their score forenergetic. - Don't compare scores across model outputs. Every model is trained
separately. A score of
0.7fordarkinMoodSimpleV2and a score of0.7forrockinMainGenreV2aren't equally strong statements, and neither are the scores of two different versions of the same model. - Use
tagsfor the labels that apply.tagsholds the labels the model considers dominant for the track. It isn't derived from a fixed score threshold: in the example above,darkis a tag with a score of0.4347.
If you apply your own threshold, for example to show only strong moods, choose it per model output and check it with a sample of your own tracks.
When a model output is missing
You only get the model outputs that you request and that your account has
access to. Requesting a model that isn't enabled for your account fails with a
401 and names the model in the error's detail.
A model output can also be absent or empty for a track:
- The analysis hasn't finished yet. Model outputs are available once the
track has been analyzed. Wait for the
analysisJob/finishedwebhook described in the Quick Start Guide. - The model found no answer. Some outputs can legitimately be empty. For
example, SubgenreV2 is
nullwhen the track doesn't fit any subgenre with enough confidence.
Code that reads model outputs should therefore handle a missing item and null
fields.
Model outputs and search
The track-level values of most model outputs, such as BpmV2.tag or
MainGenreV2.tags, can be used to narrow down a search with a
metadata filter. Segment-level data and
scores maps can't be filtered. See Fields
for every filterable field.
Further reading
- Tagging guide: fetch and use model outputs, step by step.
- AI Music Detection guide: how the
AiMusicDetectionV1model output works and how to interpret it. - Segmentation guide: how the
SegmentationV1model output works and how to use it. - API Reference: the complete request and response schema.