Skip to main content

Understanding Model Outputs

A model output is the result of one Cyanite model for a track, such as its moods, its BPM or its key. Tagging, AI Music Detection and Segmentation all deliver their results as model outputs. This page explains what a model output is, how it relates to these products, which model outputs exist and the ideas they share, so you can interpret any of them. To fetch the model outputs of a track, see the Tagging guide.


What a model output is​

  • A model is a part of Cyanite's AI that takes an audio track as input and produces one specific type of musical information as output (for example: mood scores, BPM, or a key prediction).
  • A model output is the versioned JSON object returned by one such model, for example MoodSimpleV2 or BpmV2.

The tags of a track are fields of a model output: tags holds the dominant labels of a taxonomy, scores the score of every label, and tag the single value of a musical parameter such as BPM. See Common field patterns.


Model outputs and products​

Model outputs are the technical form of the results. Which of them you get depends on the Cyanite products your account has:

  • Tagging describes your music: its genres, moods, instruments, BPM, key and more. It's a set of model outputs, each describing one aspect of a track. Which model outputs your Tagging includes is part of your agreement with Cyanite, and access is granted per model output. Two accounts can therefore have different sets, for example one with genres and moods only and one with the full catalogue below.
  • AI Music Detection is a product of its own, available with or without Tagging. Its result is the AiMusicDetectionV1 model output. See the AI Music Detection guide for how it works.
  • Segmentation is an add-on to Tagging. Its result is the SegmentationV1 model output, available once Segmentation is enabled for your account. See the Segmentation guide for how it works.

All model outputs are produced by the same analysis and fetched from the same endpoint, whichever product they belong to.


How model outputs are produced​

  1. Upload. You upload a track to your Library.
  2. Analysis. Cyanite analyzes the track with every model output your account has. You don't choose model outputs per upload.
  3. Notification. When the analysis is complete, Cyanite calls your webhook with an analysisJob/finished event.
  4. Retrieval. You fetch the model outputs you need for the track, by their version, from GET /v1/library-tracks/{id}/models.

The stored model outputs are also what Search uses: their track-level values can narrow down a search with a metadata filter.


Available model outputs​

These are the public model outputs. Each card links to the model's schema in the API Reference, which describes every field. The pills show the values a model can return, exactly as the API returns them. For a model with a single taxonomy, the value count next to the category links to its vocabulary. Select a card or +N more to see and search all values of a model. You can also search for a value, such as dark, to find the model outputs that return it. FreeGenreV3 and TimeSignatureV2 return free text, so their cards list the known values, and the model can also return others. Most model outputs belong to Tagging. Those that belong to another product carry its name, such as AI Music Detection or Segmentation.

Rhythm & Sound19

MusicalEraV2

Era the track sounds like it was produced in, with an estimated year.

earlyMid1950smidLate1950slate1950sEarly1960searlyMid1960smidLate1960slate1960sEarly1970searlyMid1970smidLate1970s

Vocal style.

femaleACapellafemaleChoirfemaleForegroundVocalsfemaleBackgroundVocalsinstrumentalmaleACapellamaleChoirmaleForegroundVocals

More specific genres within the main genre, null if none fits.

abstractIdmbreakbeatDnbdeepHouseelectrohouseminimalsynthPoptechHouse
Genre2,364

FreeGenreV3

Free-form genre names, including niche genres.

2 Step20th Century Classical60s Pop70s Pop8 Bit80s Pop90s Pop90s Rnb

Example model output​

The following example shows a taxonomy-based model output (MoodSimpleV2). It demonstrates the common scores / tags / segments pattern.

The example is explained in the sections below (see Taxonomy classifiers and Track-level vs segment-level data).

{
"version": "MoodSimpleV2",
"segments": {
"timestampsSeconds": [5.0, 15.0, 25.0],
"values": {
"aggressive": [0.2172, 0.2986, 0.2332],
"calm": [0.216, 0.1058, 0.2894],
"chill": [0.1954, 0.0933, 0.0839],
"dark": [0.4876, 0.2672, 0.5492],
"energetic": [0.5873, 0.635, 0.2871],
"epic": [0.0484, 0.2499, 0.0261],
"happy": [0.2862, 0.136, 0.0894],
"romantic": [0.0218, 0.0706, 0.0022],
"sad": [0.2462, 0.0899, 0.3842],
"scary": [0.3158, 0.0813, 0.2322],
"sexy": [0.3539, 0.3665, 0.1121],
"ethereal": [0.224, 0.1265, 0.2627],
"uplifting": [0.3555, 0.428, 0.1668]
}
},
"scores": {
"aggressive": 0.2497,
"calm": 0.2037,
"chill": 0.1242,
"dark": 0.4347,
"energetic": 0.5032,
"epic": 0.1082,
"happy": 0.1705,
"romantic": 0.0315,
"sad": 0.2401,
"scary": 0.2098,
"sexy": 0.2775,
"ethereal": 0.2044,
"uplifting": 0.3168
},
"tags": ["energetic", "dark"]
}

Versioning​

Every model output object contains a version field.

The version is a discriminator (so you can pick the right schema), and it also tracks breaking changes to the output structure. It defines:

  • Which schema to expect (fields and types)
  • Which taxonomy/vocabulary applies (for taxonomy-based outputs)

Multiple versions may exist at the same time. Older versions are deprecated over time and API consumers should migrate to the latest available version when possible.

From AudioAnalysis to model outputs​

In the Cyanite GraphQL API analysis results are bundled in versioned objects like AudioAnalysisV7. This turned out to be inflexible and had downsides such as:

  • Shipping a new major version containing all models if we want to update a single model. That's why we had to introduce new classifier names within a major version like advancedInstrumentsExtended for backwards compatibility.
  • Always analyzing every model although a customer's use case might require only a single model.

The new model outputs concept allows us to ship models with much greater flexibility.


Track-level vs segment-level data​

Many model outputs provide both:

  • Track-level results: a single prediction for the whole track (e.g. tag, scores, tags).
  • Segment-level results: predictions over time under segments.

Practical guidance:

  • If you need one value or descriptor for the track as a whole, use the track-level fields (tag, scores, tags).
  • If you need how values evolve over the course of the track (for example: where a mood becomes more intense), use segments.

Where present, segment-level data is represented as:

  • segments.timestampsSeconds: array of segment start times (seconds from track start)
  • segments.values: the segment values aligned by index with timestampsSeconds

The shape of segments.values depends on the model output:

  • Taxonomy classifiers: segments.values is an object mapping taxonomy keys to arrays of scores.
  • Musical parameters: segments.values is an array of predicted values.

In both cases, the array length(s) match segments.timestampsSeconds.length.


Common field patterns​

Taxonomy classifiers (scores, segments, tags)​

Many classifiers share the same structure:

  • scores: object mapping taxonomy keys → track-level score (usually 0.0-1.0), indicating how strongly the model predicts each label. See Interpreting scores.
  • segments: segment-level scores over time.
  • tags: array of the dominant taxonomy keys (a subset of the keys in scores).

For these model outputs, the precise set of taxonomy keys is fixed for the given version.

Musical parameters (tag, segments, confidences)​

Musical parameter model outputs (e.g. BPM, key, time signature) expose:

  • tag: the predicted global value for the whole track.

  • segments: the predicted local values per segment.

  • confidences: a structured confidence object with three values in 0.0-1.0:

    • modelCertainty: how sure the model is that the predictions are correct
    • predictionStability: how stable the predictions are across segments
    • confidence: an overall confidence score, the minimum of the two individual scores

    Interpretation: Confidence values above 0.9 indicate the track is unambiguous and a single global label can be confidently applied.

Other shapes​

Some model outputs don't follow either pattern because they describe something else than a classification or a musical parameter, for example:

Their API Reference pages, linked under Available model outputs, describe their fields.


Interpreting scores​

In taxonomy classifiers, a score in scores or segments.values is a value between 0.0 and 1.0 that tells how strongly one model predicts that a label applies. Musical parameters such as BpmV2 or KeyV2 have no scores: their segments hold predicted values, and their confidences tell how reliable those are.

  • Compare scores within one model output. Within MoodSimpleV2, you can rank the moods of a track by their score, or rank tracks by their score for energetic.
  • Don't compare scores across model outputs. Every model is trained separately. A score of 0.7 for dark in MoodSimpleV2 and a score of 0.7 for rock in MainGenreV2 aren't equally strong statements, and neither are the scores of two different versions of the same model.
  • Use tags for the labels that apply. tags holds the labels the model considers dominant for the track. It isn't derived from a fixed score threshold: in the example above, dark is a tag with a score of 0.4347.

If you apply your own threshold, for example to show only strong moods, choose it per model output and check it with a sample of your own tracks.


When a model output is missing​

You only get the model outputs that you request and that your account has access to. Requesting a model that isn't enabled for your account fails with a 401 and names the model in the error's detail.

A model output can also be absent or empty for a track:

  • The analysis hasn't finished yet. Model outputs are available once the track has been analyzed. Wait for the analysisJob/finished webhook described in the Quick Start Guide.
  • The model found no answer. Some outputs can legitimately be empty. For example, SubgenreV2 is null when the track doesn't fit any subgenre with enough confidence.

Code that reads model outputs should therefore handle a missing item and null fields.


The track-level values of most model outputs, such as BpmV2.tag or MainGenreV2.tags, can be used to narrow down a search with a metadata filter. Segment-level data and scores maps can't be filtered. See Fields for every filterable field.


Further reading​