Over 400.000 ancient manuscript images will be scanned and digitised from the Solesmes archive and made available for musicological study through Oxford’s DIAMM portal.
Over two million musical works will be correlated for concordance amongst the Solesmes dataset and external datasets such as the Cantus Index network. Approximately 4.000 novel chants are expected to be discovered and prioritised for manual indexing.
Approximately 2.200 hours of Gregorian chants sung in the Tridentine forms, corresponding to the manuscript tradition, will be recorded at Le Barroux (France).
A human-centred digital world
A new three-year project named REPERTORIUM has been awarded a 3 million euro Horizon Research and Innovation grant by the European Commission and will commence on January 1, 2023. The project was ranked first out of 55 applications received and given perfect marks. The evaluation praised the project, saying, "The ambitious aims of the project are excellently presented and highly credible… Cutting-edge technologies and research contribute exceptionally well to identifying ways of effectively promoting common European interests at a global level, leveraging resources of European Musical Heritage and forging closer cooperation between cultural, creative and economic partners." The Linares School of Engineering heads the project consortium, comprising participants from 8 countries (Spain, Holland, Italy, Finland, Germany, United Kingdom, France and Lithuania). Partners include the University of Oxford, the Complutense Institute of Musical Sciences in Madrid, the Polytechnic of Milan, the University of Tampere, the University of Alicante, creative marketing firm vist.co in Hamburg, the cultural technology developer ODRATEK BV, developer Spork Digital in UK, the French Association Musicologie Médiévale, the Hispanic Association for the Study of Gregorian Chant, and the Lithuanian National Philharmonic.
The project
REPERTORIUM stands for Researching and Encouraging the Promulgation of European Repertory through Technologies Operating on Records Interrelated Utilising Machines.
The project operates in two primary spheres, musicology and sound processing.
In Musicology, it combines multimodal artificial intelligence (AI) and deep learning (DL) solutions to perform optical music recognition (OMR) and music information retrieval (MIR) across multiple music datasets. The project will provide a technological platform for preserving European musical heritage through databases of medieval and classical works. These will be enriched through manuscript digitisation (OMR), music information retrieval (MIR) techniques, and audio-to-score (A2S) technology using AI techniques, allowing automatic data linkage with other relevant existing databases worldwide.
Over 400.000 ancient manuscript images will be scanned and digitised from the Solesmes archive and made available for musicological study through Oxford’s DIAMM portal. Over two million musical works will be correlated for concordance amongst the Solesmes dataset and external datasets such as the Cantus Index network. Approximately 4.000 novel chants are expected to be discovered and prioritised for manual indexing. A total of 127.000 chants will be manually indexed for OMR DL training purposes. Approximately 2.200 hours of Gregorian chants sung in the Tridentine forms, corresponding to the manuscript tradition, will be recorded at Le Barroux (France). Between 1.800 and 3.000 handwritten operatic aria manuscripts will be digitised from ICCMU’s DIDONE database.
The Sound Processing outcomes will build upon these corpora by applying so-called “weakly informed” uses of score knowledge to DL sound spectral analysis, algorithmic beamforming techniques, and psychoacoustics models. This combination will empower technologies for state-of-the-art real-time instrument sound source separation (SSS) specifically adapted for classical music. Cost-effective audio-immersive recording tools will be built upon AI-based stochastic signal processing and spatial audio ambisonics, leveraging the system’s score knowledge, A2S, and spectral analysis results. The performance of current 3D-audio rendering techniques is limited by the number of expensive High Order Microphones used and incurs high computational costs. Repertorium will create novel hybrid solutions for sound field reconstruction that will be flexible in terms of computational and hardware requirements (combining less expensive linear microphone arrays and data-driven methodologies) and will work in real time. Metaverse-ready immersive technologies will be tested through two streamed symphonic concerts allowing the public to freely “wander about” fully immersive spatial audio sound scenes. Users will select among 360° video hotspots for a complete AV experience.
Outcomes
Performance and recording of the novel Gregorian chants will bring recovered melodies to life not heard in over 1000 years. An opera will also be staged, showcasing music engraved using the OMR results of the DIDONE database. Dedicated promotional activities will be coordinated closely with relevant stakeholders, including multiple technical workshops to show the SSS and “minus-one” features in conservatories and music schools. The consortium will attend several international scientific conferences and trade fairs, publish a minimum of 15 scientific papers, and generate additional educational actions and materials aimed at the broader public.
The project outcomes will impact both musicological and sound processing areas. Repertorium will immediately impact musicological scholarship through its digitisation efforts. Four-thousand novel Gregorian chants are expected to be highlighted for study. This impact in preserving and evaluating musical heritage will extend to future digitisation efforts worldwide through the resulting OMR, MIR, and A2S toolsets. An API will make the developed linked data model and AI-assisted semantic enrichment techniques available to other projects. These tools will also aid commercial music publishers interested in digitising back catalogues. The recordings of Le Barroux’s chants will be published through Neumz.
In terms of sound processing, new commercial products connecting orchestras and labels to the public in the most immersive 3D sonic experience yet created will enhance streaming revenues. Solutions for music students to create real-time, on-demand minus-one versions of their favourite recordings will revolutionise at-home practice.
A human-centred digital world
Overall, Repertorium will generally widen the accessibility, appeal, and knowledge of classical music heritage among the public and academics. In addition, the project represents a significant step towards creating a general musical artificial intelligence with machine equivalents for eyes, ears, memory, intellect, and imagination. Doing so is an essential first step in exploring the AI-human values alignment problem by making European cultural heritage available to machine learning solutions. The project is an essential task towards the evolution of artificial intelligence that can authentically reflect human civilisation’s values while achieving its programmers’ objectives. Repertorium builds our future from our past by preparing the ground for an AI to understand and conform to human intentions, preferences, and goals as embodied in centuries of European civilisation’s traditions and artefacts, in addition to our instructions. Repertorium will serve as a model in shaping our new human-centred digital world.
