Music Boundary Detection using Convolutional Neural Networks: A Comparative Analysis of Combined Input Features.
DOI:
https://doi.org/10.9781/ijimai.2021.10.005Keywords:
Deep Learning, Convolutional Neural Network (CNN), Music, Information Retrieval, Music Information Retrieval (MIR), Self-Similarity Matrix (SSM)Abstract
The analysis of the structure of musical pieces is a task that remains a challenge for Artificial Intelligence, especially in the field of Deep Learning. It requires prior identification of the structural boundaries of the music pieces, whose structural boundary analysis has recently been studied with unsupervised methods and supervised neural networks trained with human annotations. The supervised neural networks that have been used in previous studies are Convolutional Neural Networks (CNN) that use Mel-Scaled Log-magnitude Spectograms features (MLS), Self-Similarity Matrices (SSM) or Self-Similarity Lag Matrices (SSLM) as inputs. In previously published studies, pre-processing is done in different ways using different distance metrics, and different audio features are used for computing the inputs, so a generalised pre-processing method for calculating model inputs is missing. The objective of this work is to establish a general method to pre-process these inputs by comparing the results obtained by taking the inputs calculated from different pooling strategies, distance metrics and audio characteristics, also taking into account the computing time to obtain them. We also establish the most effective combination of inputs to be delivered to the CNN to provide the most efficient way to extract the boundaries of the structure of the music pieces. With an adequate combination of input matrices and pooling strategies, we obtain an accuracy F1 of 0.411 that outperforms a current work done under the same conditions (same public available dataset for training and testing).
Downloads
References
O. Nieto, G. J. Mysore, C. Wang, J. B. L. Smith, J. Schlüter, T. Grill, B. McFee, “Audio-based music structure analysis: Current trends, open challenges, and applications,” Transactions of the International Society for Music Information Retrieval, vol. 3, no. 1, pp. 246–263, 2020.
M. Müller, Fundamentals of Music Processing - Audio, Analysis, Algorithms, Applications. Springer, 2015.
K. Ullrich, J. Schlüter, T. Grill, “Boundary detection in music structure analysis using convolutional neural networks,” in Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, October 27-31, 2014, 2014, pp. 417–422.
T. Grill, J. Schlüter, “Music boundary detection using neural networks on spectrograms and self-similarity lag matrices,” in 23rd European Signal Processing Conference, EUSIPCO 2015, Nice, France, August 31-September 4, 2015, 2015, pp. 1296–1300, IEEE.
T. Grill, J. Schlüter, “Music boundary detection using neural networks on combined features and two-level annotations,” in Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015, Málaga, Spain, October 26-30, 2015, 2015, pp. 531–537.
J. Foote, “Visualizing music and audio using self-similarity,” in Proceedings of the 7th ACM International Conference on Multimedia ’99, Orlando, FL, USA, October 30 - November 5, 1999, Part 1, 1999, pp. 77–80, ACM.
M. Goto, “A chorus-section detecting method for musical audio signals,” in 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP ’03, Hong Kong, April 6-10, 2003, 2003, pp. 437–440, IEEE.
T. Zhang, C. J. Kuo, “Heuristic approach for generic audio data segmentation and annotation,” in Proceedings of the 7th ACM International Conference on Multimedia ’99, Orlando, FL, USA, October 30 - November 5, 1999, Part 1, 1999, pp. 67–76, ACM.
J. Paulus, M. Müller, A. Klapuri, “State of the art report: Audio-based music structure analysis,” in Proceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR 2010, Utrecht, Netherlands, August 9-13, 2010, 2010, pp. 625–636, International Society for Music Information Retrieval.
J. Foote, “Automatic audio segmentation using a measure of audio novelty,” in 2000 IEEE International Conference on Multimedia and Expo, ICME 2000, New York, NY, USA, July 30-August 2, 2000, 2000, p. 452, IEEE Computer Society.
F. Kaiser, G. Peeters, “Multiple hypotheses at multiple scales for audio novelty computation within music,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, 2013, pp. 231–235, IEEE.
B. Logan, S. M. Chu, “Music summarization using key phrases,” in IEEE International Conference on Acoustics, Speech, and Signal Processing. ICASSP 2000, 5-9 June, 2000, Hilton Hotel and Convention Center, Istanbul, Turkey, 2000, pp. 749–752, IEEE.
C. Raphael, “Automatic segmentation of acoustic musical signals using hidden markov models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 4, pp. 360–370, 1999.
M. Levy, M. B. Sandler, “Structural segmentation of musical audio by constrained clustering,” IEEE Transactions on Speech and Audio Processing, vol. 16, no. 2, pp. 318–326, 2008.
C. J. Tralie, B. McFee, “Enhanced hierarchical music structure annotations via feature level similarity fusion,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, 2019, pp. 201–205, IEEE.
B. McFee, J. P. Bello, “Structured training for large-vocabulary chord recognition,” in Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, October 23- 27, 2017, 2017, pp. 188–194.
L. Lu, M. Wang, H. Zhang, “Repeating pattern discovery and structure analysis from acoustic music data,” in Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval, MIR 2004, October 15-16, 2004, New York, NY, USA, 2004, pp. 275–282, ACM.
J. Paulus, A. Klapuri, “Music structure analysis by finding repeated parts,” in Proceedings of the 1st ACM Workshop on Audio and Music Computing Multimedia, New York, NY, USA, 2006, p. 59–68, Association for Computing Machinery.
D. Turnbull, G. R. G. Lanckriet, E. Pampalk, M. Goto, “A supervised approach for detecting boundaries in music using difference features and boosting,” in Proceedings of the 8th International Conference on Music Information Retrieval, ISMIR 2007, Vienna, Austria, September 23-27, 2007, 2007, pp. 51–54, Austrian Computer Society.
B. McFee, D. Ellis, “Dp1, mp1, mp2 entries for mirex 2013 structural segmentation and beat tracking,” Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX), 2013.
M. C. McCallum, “Unsupervised learning of deep features for music segmentation,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, 2019, pp. 346–350, IEEE.
J. Schlüter, S. Böck, “Improved musical onset detection with convolutional neural networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2014, Florence, Italy, May 4-9, 2014, 2014, pp. 6979–6983, IEEE.
Y. Zou, M. Thiel, M. C. Romano, J. Kurths, “Analytical description of recurrence plots of dynamical systems with nontrivial recurrences,” International Journal of Bifurcation and Chaos, vol. 17, no. 12, pp. 4273–4283, 2007.
J. Paulus, A. Klapuri, “Music structure analysis with a probabilistic fitness function in MIREX2009,” in Proceedings of the Fifth Annual Music Information Retrieval Evaluation eXchange, Kobe, Japan, October 2009. Extended abstract.
M. Mauch, K. C. Noland, S. Dixon, “Using musical structure to enhance automatic chord transcription,” in Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009, Kobe International Conference Center, Kobe, Japan, October 26-30, 2009, 2009, pp. 231–236, International Society for Music Information Retrieval.
G. Sargent, S. A. Raczynski, F. Bimbot, E. Vincent, S. Sagayama, “A music structure inference algorithm based on symbolic data analysis.” MIREX - ISMIR 2011, Oct. 2011. [Online]. Available: https://hal.archivesouvertes.fr/hal-00618141, Poster.
F. Kaiser, T. Sikora, G. Peeters, “Mirex 2012-music structural segmentation task: Ircamstructure submission,” Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX), 2012.
O. Nieto, J. P. Bello, “Mirex 2014 entry: 2d fourier magnitude coefficients,” Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX), 2014.
C. Cannam, E. Benetos, M. Mauch, M. E. Davies, S. Dixon, C. Landone, K. Noland, D. Stowell, “Mirex 2015: Vamp plugins from the centre for digital music,” Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX), 2015.
O. Nieto, “Mirex: Msaf v0. 1.0 submission,” 2016.
J. Schlüter, K. Ullrich, T. Grill, “Structural segmentation with convolutional neural networks mirex submission,” Tenth running of the Music Information Retrieval Evaluation eXchange (MIREX 2014), 2014.
T. Grill, J. Schlüter, “Structural segmentation with convolutional neural networks mirex submission,” Proceedings of the Music Information Retrieval Evaluation eXchange (MIREX), p. 3, 2015.
J. Serrà, M. Müller, P. Grosche, J. L. Arcos, “Unsupervised music structure annotation by time series structure features and segment similarity,” IEEE Transactions on Multimedia, vol. 16, no. 5, pp. 1229–1240, 2014.
G. Sargent, F. Bimbot, E. Vincent, “A regularity-constrained viterbi algorithm and its application to the structural segmentation of songs,” in Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011, Miami, Florida, USA, October 24-28, 2011, 2011, pp. 483–488, University of Miami.
A. Cohen-Hadria, G. Peeters, “Music structure boundaries estimation using multiple self-similarity matrices as input depth of convolutional neural networks,” in AES International Conference Semantic Audio 2017, Erlangen, Germany, June 22-24, 2017, 2017, Audio Engineering Society.
J. S. Downie, A. F. Ehmann, M. Bay, M. C. Jones, “The music information retrieval evaluation exchange: Some observations and insights,” in Advances in Music Information Retrieval, vol. 274 of Studies in Computational Intelligence, Z. W. Ras, A. Wieczorkowska Eds., Springer, 2010, pp. 93–115.
M. Goto, et al., “Development of the rwc music database,” in Proceedings of the 18th International Congress on Acoustics (ICA 2004), vol. 1, 2004, pp. 553–556.
M. Goto, “AIST annotation for the RWC music database,” in ISMIR 2006, 7th International Conference on Music Information Retrieval, Victoria, Canada, 8-12 October 2006, Proceedings, 2006, pp. 359–360.
F. Bimbot, E. Deruty, G. Sargent, E. Vincent, “Methodology and conventions for the latent semiotic annotation of music structure,” 2012.
A. F. Ehmann, M. Bay, J. S. Downie, I. Fujinaga, D. D. Roure, “Music structure segmentation algorithm evaluation: Expanding on MIREX 2010 analyses and datasets,” in Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011, Miami, Florida, USA, October 24-28, 2011, 2011, pp. 561–566, University of Miami.
J. B. Smith, E. Chew, “A meta-analysis of the mirex structure segmentation task,” in Proceedings of the 14th International Society for Music Information Retrieval Conference, ISMIR, 2013, Curitiba, Brazil, vol. 16, 2013, pp. 45–47.
B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, O. Nieto, “librosa: Audio and music signal analysis in python,” in Proceedings of the 14th python in science conference, vol. 8, 2015.
J. Serrà, M. Müller, P. Grosche, J. L. Arcos, “Unsupervised detection of music boundaries by time series structure features,” in Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada, 2012, AAAI Press.
J. B. L. Smith, J. A. Burgoyne, I. Fujinaga, D. D. Roure, J. S. Downie, “Design and creation of a large-scale database of structural annotations,” in Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011, Miami, Florida, USA, October 24-28, 2011, 2011, pp. 555–560, University of Miami.
A. A. Alvarez, F. Gómez-Martin, “Motivic pattern classification of music audio signals combining residual and LSTM networks,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 6, no. 6, pp. 208–214, 2021.
K. K. Verma, B. M. Singh, H. L. Mandoria, P. Chauhan, “Two-stage human activity recognition using 2d-convnet,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 6, no. 2, pp. 1–11, 2020.
M. Khari, A. K. Garg, R. G. Crespo, E. Verdú, “Gesture recognition of RGB and RGB-D static images using convolutional neural networks,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 5, no. 7, pp. 22–27, 2019.
S. Jha, A. Dey, R. Kumar, V. K. Solanki, “A novel approach on visual question answering by parameter prediction using faster region based convolutional neural network,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 5, no. 5, pp. 30–37, 2019.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, R. Garnett Eds., Curran Associates, Inc., 2019, pp. 8024–8035.
D. P. Kingma, J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015. [Online]. Available: http://tensorflow.org/, Software available from tensorflow.org.
O. Nieto, M. M. Farbood, T. Jehan, J. P. Bello, “Perceptual analysis of the f-measure to evaluate section boundaries in music,” in Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, October 27-31, 2014, 2014, pp. 265–270.
C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. W. Ellis, “Mir_eval: A transparent implementation of common MIR metrics,” in Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, October 27-31, 2014, 2014, pp. 367–372.
Downloads
Published
-
Abstract174
-
PDF42
Submit Article
Information
