14 October 2022

Music Recommendation

Karlijn Dinnissen
INFOMAIS 2022-2023
Structure
Music Information Retrieval

Music Recommendation

In Practice

Stakeholders & Goals

Technical Aspects

Fairness
Music Information Retrieval
Music Information Retrieval (MIR)

The interdisciplinary science of retrieving

information from music

Possible combination of:

• Musicology
• Signal processing
• Informatics
• Machine learning
• Natural language processing
• Human-computer interaction
• ....
Music Information Retrieval (MIR)

The interdisciplinary science of retrieving

information from music
Which types
Possible combination of: of tasks?
• Musicology
• Signal processing
• Informatics
• Machine learning
• Natural language processing
• Human-computer interaction
• ....
Some tasks involving MIR

• Music classification
• Song
7
Some tasks involving MIR

• Music classification
• Song
• Category (genre, mood, artist etc.)
• Feature (chords, key, instruments, BPM etc.)
9
Some tasks involving MIR

• Music classification
• Song
• Category (genre, mood, artist etc.)
• Feature (chords, key, instruments, BPM etc.)
• Music transcription
 11
Benetos, E., Dixon, S., Duan, Z., & Ewert, S. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36, 20-30.
Some tasks involving MIR

• Music classification
• Song
• Category (genre, mood, artist etc.)
• Feature (chords, key, instruments, BPM etc.)
• Music transcription
• Music generation
13
Some tasks involving MIR

• Music classification
• Song
• Category (genre, mood, artist etc.)
• Feature (chords, key, instruments, BPM etc.)
• Music transcription
• Music generation
• Music recommendation
Music Recommendation
A typical online music platform provides access to ~80 million music tracks

16
Music Recommender Systems

• Recommender systems offer users music based on their

current goal (e.g., query, mood, time of day, musical taste, …)
• Automatic choices influence which music users play
• Both push and pull possible
• Traditionally designed to satisfy the user
 What makes music recommendation special? (1/3)

Item characteristics
• Various types
• e.g., songs, albums, artists, audio samples, concerts, music videos
• Magnitude of available items/catalogues
• Short duration (vs. movies, books, podcasts)
• Long lifetime (vs. news items)
• Can have similar relevance
• Lower commitment necessary, items more “disposable”, bad
recommendations maybe (?) not that severe

Schedl, M., Zamani, H., Chen, C.-W., Deldjoo, Y., & Elahi, M. (2018). Current challenges and visions in music recommender systems research. International Journal of Multimedia
Information Retrieval, 7(2), 95-116. https://doi.org/10.1007/s13735-018-0154-2
 What makes music recommendation special? (2/3)

Consumption characteristics
• Different locations/settings/contexts
• e.g., static (stereo at home) vs. variable (headphones during exercise)
• Different engagement (active vs. passive)
• passive e.g., while working, background music
• Different intent
• e.g., reduce stress, improve mood, improve productivity, help for sleeping
• Often sequential
• Re-recommendation may be appreciated (vs. buying dishwasher)
 What makes music recommendation special? (3/3)

Stakeholder characteristics
• Various actors involved
• e.g., listeners, producers, performers, composers, songwriters, agents, record
companies
• Highly emotionally connoted (vs. dishwasher)
• Listeners often use music for self-expression
In Practice
 22
IFPI 2021
23
Home page
Stakeholders & Goals
Task
What criteria define that
a music recommender is good?

Discuss in groups:
▪ Group 1: “Paul, 25y, Swedish, likes watching the show ‘The Voice
Nederland’”
▪ Group 2: “Ed Sheeran” (hint: artist from the UK, known worldwide)
▪ Group 3: “Indie label A’damHipster, founded in 2018, 9 signed
bands from Amsterdam”
▪ Group 4: “Music platform Scottipy, 2 models: advertising-based,
subscription-based”

Afterwards: Overall discussion in plenum

26
1. The platform
 Platform goals

Aiming to make revenue, and eventually profit

• Attract new users
• Retain current users à benefit from successful user-item match
• Major record labels are (partial) shareholders

Most platforms offer the same content

Not providing music content themselves - yet

Dinnissen, K. & Bauer, C. (2022) Fairness in Music Recommender Systems: A Stakeholder-Centered Mini Review. Front. Big Data 5:913608. doi: 10.3389/fdata.2022.913608
2. The user
(item consumer)
 User: broader goals

Most important characteristics of music streaming platform:

• Availability of music
• Simplicity and ‘ease of use’
• Recommendation

~14% of music on streaming services is accessed algorithmically

• Liem, C., Rauber, A., Lidy, T., Lewis, R., Raphael, C., Reiss, J. D., Crawford, T., & Hanjalic, A. (2012). Music information technology and professional stakeholder audiences:
Mind the adoption gap. Dagstuhl follow-ups 2012
• Villermet, Q., Poiroux, J., Moussallam, M., Louail, T., & Roth, C. (2021). Follow the guides: disentangling human and algorithmic curation in online music consumption
Fifteenth ACM Conference on Recommender Systems, Amsterdam, Netherlands. https://doi.org/10.1145/3460231.3474269
User: Innate characteristics

• Descriptors: e.g., gender,

age, country, level of activity
on platform
• ‘Mainstreaminess’
• Extent to which they like
music outside usual taste

Kowald, D., Muellner, P., Zangerle, E. et al. (2021) Support the underground: characteristics of beyond-mainstream music listeners. EPJ Data Sci. 10, 14. link
User goals: situational characteristics

Additionally, extent to which user values: novelty, diversity, serendipity, fairness, ...
Can be user-innate or situational

Hosey, C., Vujović, L., St. Thomas, B., Garcia-Gathright, J., & Thom, J. (2019, May). Just give me what I want: How people use and evaluate music search. In Proceedings of the
2019 CHI Conference on Human Factors in Computing Systems (pp. 1-12).
Focus on: Building playlists

33
Using recommended songs to build playlist
Focus on: Re-discovery

• Music you know you like

• Personalized

35
Searching for playlist based on your previous listening behaviour
Focus on: Personalized discovery

Intended to feel like it is curated ‘Just For Me’

37
Searching for personalized discovery playlists
Focus on: Lyrics

Which line comes after “You’re Simply The Best” again?

39
Searching for lyrics
3. The artist
(item provider)

Also:
Label, publisher,
management, songwriter,
composer, rights owner, ...
Artist: goals

Share musical vision, passion and creativity

Income
• Have songs played
• Increase fanbase
• Increase likelihood of being picked up by other media
• Sell albums, EP’s or merchandise
• Sell tickets for tour
Artist: characteristics

• Distinguishing between artist and item characteristics

• We will discuss item characteristics in the next part
• Personal descriptors: age, nationality, gender, ...
• Not all of them known to streaming platform
• Act descriptors: band/solo, members, contemporaneity,
number of albums, ‘mainstreaminess’
Others

• Shareholders
• 3rd party music distributors
• Advertisement company
• Ticket service
• Item providers of non-music
items (podcasts)
Multi-stakeholder recommendation
What happens if it’s not the best recommendation ever?

• 3:50 minutes wasted on bad or unsuitable music

Music consumer • Bad mood because of unsuitable song

• Homogenous music consumption due to popularity bias

Society • Emergence of a few isolated music cultures (insulation)

Service/platform provider • Retrieved song X instead of song Y

• Damage to image, losing customers

Music company • Shifts on the market (e.g., expansion of monopoly position)

• E.g., 1 million streams less/more than in previous year

Top-of-the-top superstar (e.g., Bad Bunny 9.1 billion streams in 2021)
• More/less advertising deals
Artist in the “long tail” of
popularity • Exposure in recommendations or not
• Needs second job or not

46
 Under the hood:
Technical Aspects of Music Recommendation
 Under the hood:
Content-based Filtering
Audio content analysis

In contrast to e.g., movies: true content-based recommendation!

1. Features can be extracted from any audio file, without cultural
biases or needing community data
2. Learning of high-level semantic descriptors from those low-level
features via machine learning
3. Also, human labeling plays a role

49
Audio content analysis

In contrast to e.g., movies: true content-based recommendation!

1. Features can be extracted from any audio file, without cultural
biases or needing community data
2. Learning of high-level semantic descriptors from those low-level
features via machine learning
3. Also, human labeling plays a role

50
Audio Content Analysis: Selected Features

• Timbre
→ e.g. for genre classification,
“more-like-this” recommendations

• Beat/downbeat → Tempo: 85 bpm

• Tonal features
→ e.g. for melody extraction, cover version identification
Disturbed Different versions of this song, e.g.,
The Sound of ▪ Simon & Garfunkel - The Sound of Silence
Silence ▪ Anni-Frid Lyngstad (ABBA) - En ton av tystnad

Assign semantic categories through machine learning

→ not_danceable, gender_male, mood_not_happy, …

(also some of the next slides)

Result: computable similarity Schedl, Knees, Gouyon (2018) ISMIR Tutorial: Overview and
Issue: not how people “use” music New Challenges of Music Recommendation Research.
51
Want to know more?

Sound and music technology (INFOMSMT)

Dr. Anja Volk
Late enrollment days 24 & 25 October

https://osiris-
student.uu.nl/#/onderwijscatalogus/extern/cursus?cursuscode=INFO
MSMT&taal=nl&collegejaar=2022
Audio is not the only data about items available
Lyrics

53
Audio is not the only data about items available
Artwork

Single cover Album cover

54
Audio is not the only data about items available
Video

55
Even more interesting available data

• Editorial and curatorial meta-data

→ e.g., genre, artist, biography, release year
• User-generated data
→ e.g., tags, reviews, stories, social media
• Curated collections
→ Playlists, radio channels
→ CD album compilations

56
Music may be available…

• In symbolic format (e.g., a MIDI file)

• In audio format (e.g., an mp3 file)
• In vector format (e.g., a scanned score)

• In text format (e.g., lyrics as text)

• In vector format (e.g., album covers)
• In video format (e.g., music video)

• Editorial and curatorial meta-data (e.g.,

genre, artist, release year)

multi-modal
57
Embedding space:
Items

Spotify 2022

Result: computable similarity

Closer to how people “use” music 59
Embedding space:
User model

Spotify 2022

Result: computable similarity

Closer to how people “use” music 60
 Under the hood:
Collaborative Filtering
Collaborative Filtering
The user-item matrix

items interactions are

(~10’s M) observed in data
users
(~10’s M)

63
Collaborative filtering challenges in MRS

Explicit input: ratings

• Very sparse
• Users might not use rating system in the same way

Implicit input: clicks, plays, queries, skips, add to playlist, …

• Positive interaction ≠ actually liking song
 Under the hood:
Hybrid / ensemble methods
Hybrid / ensemble

Recommendation
Editorial/ Content- Collaborative Context- etc
Curatorial based Filtering based

Feedback
Machine Learning
e.g., Ensemble Learning
Embedding space

Spotify 2022

Computable similarity in users AND items

Even closer to how people “use” music 67
Hybrid / ensemble

Recommendation
Editorial/ Content- Collaborative Context- etc
Curatorial based Filtering based

Machine Learning
e.g., Ensemble Learning
Reinforcement
learning:
Maximize long-term reward
Feedback
Reward desired behaviour,
punish undesired behaviour
 13-10-2022

Under the hood:

Other aspects to consider
Caroussel recommendations on home page

70
 So it’s a bit more complex…

• Machine learning to fit which recommender/information in

which context
• Control for diversity, exploitation vs. exploration, novelty, etc.
• Different types of recommenders and models for different
features

71
Oramas et al., RecSys DLRS 2017
Playlist generation

• A continuation problem (e.g., next track

recommendation)
→ Given a listener enjoying one musical
experience, what track recommendations
can we make to extend this experience as
much as possible
→ Should suit the previous tracks
• Order may matter, not always

72
Let’s imagine…

73
Context features

Sensors to obtain context information.

General data services Accelerometer

Gyroscope
Heart rate
sensor
Magnetometer Thermometer

→ e.g., for time, location, noise, weather GPS (location)

Proximity sensor
Ambient light
Air humidity
sensor
Bluetooth
sensor NFC
Microphone
Camera
Microphone

Sensors in smart devices Touchscreen

sensors
Fingerprint
sensor

→ e.g., heart rate, accelerometer, noise level, location Pedometer

Barcode sensors
Barometer

74
Cold start problem & mitigation
Engineering blogs

Music
• https://engineering.atspotify.com/
• https://deezer.io/

Movies
• https://netflixtechblog.com/
Fairness
User fairness

‘Beyond-mainstream’ groups
receive worse recommendations
à Distributional harm

• Popularity (‘mainstreaminess’)
• Country
• Gender
• Age

• Bauer, C. & Schedl, M. (2018) On the importance of considering country- specific aspects on the online-market: an example of music recommendation considering
country-specific mainstream. HICCS ‘18
• Melchiorre et al. (2021) Investigating gender fairness of recommendation algorithms in the music domain. Information Processing & Management
• Neophytou et al. (2022) Revisiting popularity and demographic biases in recommender evaluation and effectiveness. ECIR ‘22

78
 User groups unequally served

Mainstream BeyondMS
74
71,67
Mean Absolute Error (MAE)

68,49
68,75 67,59

63,46 63,40
63,5
61,26

57,77
58,25
54,82
53
User Item Avg User KNN User KNN Avg NMF

79
Kowald, D., Müllner, P., Zangerle, E., Bauer, C., Schedl, M. & Lex, E. (2021). Support the Underground: Characteristics of Beyond-Mainstream Music Listeners. EPJ Data Science,
10. DOI: 10.1140/epjds/s13688-021-00268-9
User groups unequally served

Ufolk Uhard Uambi Uelec Mainstream BeyondMS

74 73,19
71,67
Mean Absolute Error (MAE)

70,30 71,04
69,83 69,29
68,49
68,75 67,44 67,59
65,15 65,57 66,15
63,21 63,46 63,40
63,5 62,29
60,56 61,26
59,69
57,23 57,77
58,25 56,62
54,21 54,82
53
User Item Avg User KNN User KNN Avg NMF

80
Kowald, D., Müllner, P., Zangerle, E., Bauer, C., Schedl, M. & Lex, E. (2021). Support the Underground: Characteristics of Beyond-Mainstream Music Listeners. EPJ Data Science,
10. DOI: 10.1140/epjds/s13688-021-00268-9
 Artist APC

Global (120,322
Top artists globally & for selected countries
The Beatles 3.838.604
Radiohead 3.437.326

users)
Pink Floyd 2.990.318
Coldplay 2.576.390
Daft Punk 2.523.537
Muse 2.460.597
Metallica 2.401.945
Arctic Monkeys 2.345.951
Finland (1,407 users) Italy (972 users) Turkey (479 users) Linkin Park 2.296.327
Artist APC Artist APC Artist APC Red Hot Chili Peppers 2.221.660
Stam1na 105.633 Radiohead 68.160 Pink Floyd 68.887 Lana Del Rey 1.892.896
Nirvana 1.878.647
In Flames 97.645 The Beatles 65.498 Metallica 42.784
System of a Down 1.874.102
CMX 90.032 Pink Floyd 60.558 Daft Punk 42.020
Florence + the Machine 1.729.489
Kotiteollisuus 82.309 Fabrizio De André 53.928 Iron Maiden 34.174 Iron Maiden 1.713.020
Turmion Kätilöt 78.722 Muse 48.168 Radiohead 31.390 Depeche Mode 1.710.159
David Bowie 1.685.010
Amorphis 78.159 Depeche Mode 42.586 Massive Attack 30.669
Lady Gaga 1.655.023
Nightwish 75.742 Afterhours 42.473 The Beatles 27.951
Rammstein 1.647.437
Mokoma 73.453 Verdena 42.338 Opeth 25.744 Queen 1.614.548
Muse 69.507 Sigur Rós 41.748 Depeche Mode 25.075 Led Zeppelin 1.602.110
Metallica 69.499 Arctic Monkeys 39.755 Dream Theater 24.286 The Black Keys 1.517.523
The xx 1.499.181
Nine Inch Nails 1.489.223
The Rolling Stones 1.483.385
Eminem 1.445.767
Foo Fighters 1.444.212

Markus Schedl & Christine Bauer (2017). Introducing global and regional mainstreaminess for improving personalized music recommendation. Proceedings of the 15th 81
International Conference on Advances in Mobile Computing & Multimedia (MoMM 2017). Salzburg, Austria, 4–6 December, pp 74-81. DOI: 10.1145/3151848.3151849
Artist fairness

Items of (protected) groups with

certain characteristics appear in
recommendations less frequently
à Distributional harm

• Popularity
• Gender
• Country

• Celma & Cano (2008) From hits to niches? Or how popular artists can bias music recommendation and discovery. KDD ‘08
• Ferraro et al. (2020) Artist biases in collaborative filtering for music recommendation. ICML ‘20
• Ferraro et al. (2021) Break the loop: Gender imbalance in music recommenders. CHIIR ‘21

82
 83
IFPI 2021
 Items: Long-tail market

short long
head tail
Long-tail market
superstar phenomenon

Relatively small numbers of items (the head)

dominate the market,
while there is a considerable long tail of less
popular items
Artist playcount (APC) for the global top 10,000 artists.
Artist IDs (x-axis) sorted by popularity values.

• Christine Bauer & Markus Schedl (2019). Global and country-specific mainstreaminess measures: Definitions, analysis, and usage for improving personalized music
recommendation systems. PLOS ONE, 14(6), e0217389. DOI: 10.1371/journal.pone.0217389
• Ekstrand, M. D., Tian, M., Azpiazu, I. M., Ekstrand, J. D., Anuyah, O., McNeill, D., & Pera, M. S. (2018). All The Cool Kids, How Do They Fit In?: Popularity and Demographic
Biases in Recommender Evaluation and Effectiveness Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Proceedings of Machine Learning
Research. https://proceedings.mlr.press/v81/ekstrand18b.html

84
Gender bias

Real-world bias, repeated in

recommendation

Potential mitigation
• Pre-processing: modify training data
USC Annenberg Inclusion Initiative (2021)
(e.g., using General Adversarial Networks)
• In-processing: impose optimization
constraint
• Post-processing: re-ranking, implement
quota

85
Andres Ferraro, Xavier Serra, and Christine Bauer (2021) Break the Loop: Gender Imbalance in Music Recommenders. In Proceedings of the 2021 Conference on Human
Information Interaction and Retrieval (CHIIR '21). Association for Computing Machinery, New York, NY, USA, 249–254. https://doi.org/10.1145/3406522.3446033
 Should platforms actively aim
to make a positive impact on
fairness in music industry?

Potential methods:
• Train models with fairness objective
• Push dedicated playlists that focus on
lesser known artists
• Hire more diverse playlist curators
• Implement threshold/quota for
automatically generated playlists

86
What do stakeholders think?
Users
• Nyi Nyi Htun, Elisa Lecluse, and Katrien Verbert. 2021. Perception of Fairness in Group Music Recommender
Systems. In 26th International Conference on Intelligent User Interfaces (IUI '21). Association for Computing
Machinery, New York, NY, USA, 302–306. https://doi.org/10.1145/3397481.3450642
• [not about music specifically] Sonboli, N., Smith, J. J., Cabral Berenfus, F., Burke, R., & Fiesler, C. (2021). Fairness
and transparency in recommendation: The users’ perspective. Proceedings of the 29th ACM Conference on
User Modeling, Adaptation and Personalization, https://dl.acm.org/doi/pdf/10.1145/3450613.3456835

Artists
• Ferraro, A., Serra, X., Bauer, C. (2021). What Is Fair? Exploring the Artists’ Perspective on the Fairness of Music
Streaming Platforms. In: , et al. Human-Computer Interaction – INTERACT 2021. INTERACT 2021. Lecture Notes
in Computer Science, vol 12933. Springer, Cham. https://doi.org/10.1007/978-3-030-85616-8_33
• Another paper by Karlijn & Christine coming up...

Developers
• [not about music specifically] Kasinidou, M., Kleanthous, S., Barlas, P., & Otterbacher, J. (2021). I agree with the
decision, but they didn't deserve this: Future Developers' Perception of Fairness in Algorithmic Decisions
Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event, Canada.
https://doi.org/10.1145/3442188.3445931
• https://engineering.atspotify.com/2022/09/lessons-learned-from-algorithmic-impact-assessments-in-practice/
Further reading

• Markus Schedl, Peter Knees, Brian McFee, Dmitry Bogdanov, and

Marius Kaminskas. Music Recommender Systems, pages 453–492.
Springer US, Boston, MA, 2015. https://link-springer-
com.proxy.library.uu.nl/content/pdf/10.1007%2F978-1-4899-7637-
6_13.pdf

• Òscar Celma and Pedro Cano. 2008. From hits to niches? Or how
popular artists can bias music recommendation and discovery. In
Proceedings of the 2nd KDD Workshop on Large-Scale
Recommender Systems and the Netflix Prize Competition
(NETFLIX '08). ACM, New York, NY, USA, Article 5, 1–8.
https://doi.org/10.1145/1722149.1722154