Skip to main content

Segmentation

Segmentation is the Cyanite product that splits a track into its musical sections and tells you where each one starts and ends. It's an add-on to Tagging.

This guide explains why and how the segmentation works, shows you how to fetch and use the result in your application, and what to keep in mind when you interpret it.


Why segment a track?​

A track is rarely one uniform piece of music. It builds up, drops, changes instrumentation and returns to earlier ideas. Tags and a single preview point describe the track as a whole, but they don't tell you how it's composed. Segmentation gives you that structure as a list of sections with exact time ranges, which helps whenever you work with a part of a track instead of the whole:

  • Sync: pitch one coherent part of a track that already has the length a scene needs, for example 30 seconds for a movie scene, instead of an arbitrary cut.
  • Search: find the section of a track that matches a brief and play or pitch exactly that section.
  • Cueing: mix music in and out at a section boundary, for example on a radio station, so a transition doesn't cut into the middle of a musical phrase.

How it works​

Segmentation works off audio only. It finds the points in a track where the music changes noticeably and splits the track at those points into consecutive, non-overlapping segments:

  • The first segment starts at 0, and the last segment ends at the end of the track. Together, the segments cover the whole track without gaps.
  • Each segment ends exactly where the next one begins.
  • Segments are ordered by time and are given in seconds from the beginning of the track.

The segments are related to the sections you know from songwriting, such as intro, verse and chorus, but they don't match them one to one. They rather depict generally meaningful changes in a track which will often coincide with songwriting sections.


Segmentation and Tagging​

Segmentation is sold as an add-on to Tagging, and it's delivered in the same way. Its result is the SegmentationV1 model output, the same kind of result that Tagging delivers:

  • Once Segmentation is enabled for your account, it runs as part of the regular analysis of every uploaded track, alongside your Tagging model outputs. You don't start it separately.
  • You fetch it from the same endpoint as every other model output, GET /v1/library-tracks/{id}/models, with model=SegmentationV1.
  • If Segmentation isn't enabled for your account, requesting SegmentationV1 fails with a 401.
  • Tracks that were analyzed before Segmentation was enabled for your account have no SegmentationV1 result.
  • SegmentationV1 has no fields that can be used in a search metadata filter. You can still combine it with Search results in your application.

Prerequisites​


Fetch the result​

When your webhook receives analysisJob/finished for a track, request the SegmentationV1 model output for that track:

GET https://rest-api.cyanite.ai/v1/library-tracks/{id}/models?model=SegmentationV1

API reference: Get Model Outputs

// trackId comes from the webhook payload: payload.data.track.id
const response = await fetch(
`https://rest-api.cyanite.ai/v1/library-tracks/${trackId}/models?model=SegmentationV1`,
{
headers: {
"x-api-key": process.env.CYANITE_API_KEY,
},
},
);

const { items } = await response.json();
const segmentation = items.find(
(item) => item.version === "SegmentationV1",
);

You can request SegmentationV1 together with your Tagging model outputs in the same call, for example ?model=SegmentationV1&model=MainGenreV2.


Read the result​

{
"items": [
{
"version": "SegmentationV1",
"segments": [
{ "start": 0, "end": 16 },
{ "start": 16, "end": 48 },
{ "start": 48, "end": 79 },
{ "start": 79, "end": 111 },
{ "start": 111, "end": 143 },
{ "start": 143, "end": 175 },
{ "start": 175, "end": 210 }
]
}
]
}
FieldTypeDescription
versionstringAlways "SegmentationV1".
segmentsarrayThe segments of the track, ordered by time. Together, they cover the whole track without gaps.
segments[].startnumberStart of the segment in seconds from the beginning of the track. 0 for the first segment.
segments[].endnumberEnd of the segment in seconds. Equal to the start of the next segment, or the end of the track.

Interpreting the result​

Every boundary marks a point where the music changes noticeably, for example when the arrangement, the energy or the instrumentation changes. That makes a segment a coherent part of the track that you can play, pitch or cut on its own.

Limitations​

  • No section labels. This version provides the boundaries of the segments only, not whether a segment is an intro, a verse or a chorus.
  • Not one-to-one with song sections. The segments follow changes in the music, which often, but not always, coincide with the sections of a song.
  • Precision. Boundaries are currently given in whole seconds. Treat them as a starting point for a cut, not as a sample-accurate edit point.
  • Music with little structure. Tracks that hardly change over time, such as some ambient or drone music, can have only a few, long segments.

Troubleshooting​

ProblemCauseWhat to do
401Invalid API key, or Segmentation isn't enabled.Check x-api-key, or contact Cyanite.
404The track doesn't exist or doesn't belong to your account.Check the Library Track ID.
Empty itemsThe analysis hasn't finished yet.Wait for the analysisJob/finished webhook before fetching the result.
Empty items for a track analyzed earlierThe track was analyzed before Segmentation was enabled for your account.Contact Cyanite to get existing tracks segmented.
analysisJob/failed webhookThe track couldn't be analyzed, for example because it's too long.Check the file, then upload it again. If it keeps failing, contact Cyanite.

Next steps​