Pitch Contour Tokenization using VQ-VAE
and Its Application on Korean Traditional Music Analysis

Computational analysis of music often relies on discrete representations, yet many musical traditions are organized around continuous pitch movement that resists segmentation into note-like units. For such traditions, the discrete units that analysis would build on are not given in advance. We address this gap by learning a vocabulary of local pitch-contour patterns directly from unlabeled audio, using a VQ-VAE that quantizes fixed-length contour segments into a finite codebook. To make the learned tokens stable across segmentation positions and small variations in timing and pitch range, we train the model with a reconstruction objective evaluated under the best alignment among a set of candidate temporal and pitch-domain transformations. Applied to Korean traditional music, the learned tokens recover information about expert-defined sigimsae categories without supervision, and in pansori individual tokens align with the two principal modes, Gyemyeonjo and Ujo, supporting their use as units for corpus-level analysis of contour-centric traditions.

The model is a VQ-VAE over fixed-length pitch-contour segments. An encoder maps each segment to a latent vector, vector quantization assigns it to the nearest codebook entry, and a decoder reconstructs the contour from the selected entry, so that every segment is represented by a single discrete token. The reconstruction loss is evaluated under the best alignment among a set of candidate temporal and pitch-domain transformations of the decoded contour, and only the minimum error is backpropagated. This keeps the tokens stable across segmentation positions and robust to differences in key and tempo, so each token captures a local contour shape.

We evaluate the learned tokens from two perspectives. Token consistency measures how stable the tokenization is when the segmentation window is shifted: the agreement accuracy (Acc) between tokens of shifted segmentations and the KL divergence (KLD) between the resulting token distributions, for shift ranges of 0–60 and 0–124 frames. Downstream classification reports macro-F1 and mAP for expert-labeled sigimsae classification on the NIA dataset (all recordings and the vocal-only subset), predicting the category from the assigned token. ED-TCN, a supervised approach, is included as a reference.

For each expert-defined sigimsae (ornament) category, we show the decoded contour of the most representative token — the token with the highest \(P(\text{token}\mid\text{label})\) — for our VQ-VAE (with the transformation-minimized loss) and the autoencoder baseline, together with real audio excerpts of instances assigned to that token (original NIA recordings). The VQ-VAE recruits six distinct tokens matching each category's canonical realization, while the autoencoder collapses the six categories onto only three tokens.

VQ-VAE tokens and P values match the paper (Fig. 2) exactly. The autoencoder's UMAP+KMeans tokenization was re-fitted for this page, so its token IDs differ from the paper figure (196/16/190) while showing the same collapse behavior.

Tokens with the highest \(P(\text{mode}\mid\text{token})\) for the two principal pansori modes, Gyemyeonjo and Ujo. Each panel shows the decoded token contour (black) with median-subtracted original contours assigned to that token, a manually drawn staff-notation transcription of a representative occurrence, and real audio excerpts (vocal-separated) of instances assigned to the token.

Additional tokens with high \(P(\text{mode}\mid\text{token})\), shown with the decoded token contour and real audio excerpts of assigned instances.