Billboard Hot 100 Analysis

This project began as a sociolinguistic research question: how has /aj/-monophthongization changed in American popular music over time?

/aj/-monophthongization is a pronunciation feature associated with Southern and Black American English, where the diphthong /aj/ in words like 'time', 'ride', or 'I' is pronounced like a single vowel [a]. Because of the influence of blues, country, rock-and-roll on popular music, it also appears in musical performance even in singers that don't speak those varieties of English.

I wanted to investigate how often this feature appears over time. I built a dataset of Billboard #1 songs from 1940 through 1970 and developed tools to help identify, annotate, analyze, and model thousands of potential instances of the variable.

This project combined data collection, NLP, frontend development, API development, statistical modeling, and sociolinguistic analysis.

Building the Dataset

The first step was building the dataset of lyrics. I collected the Billboard chart data using songs that hit #1 throughout the year. Although the Billboard Hot 100 debuted in 1958, I augmented it with other Billboard popular music charts dating back to 1940 for a long-term view.

I collected the chart data and transformed it into a structured dataset in which each song occupied a single row, with fields including chart date, ranking information, artist, chart type, lyrics, and demographic information about the performer.

Getting lyrics presented a significant data-quality problem. Many of the lyrics online for older songs were often inaccurate or missing, so I manually located, compared, and corrected them where necessary.

I also researched and coded demographic information about each performer, including sex and race, using available biographical sources. These variables were later used to investigate whether demographic characteristics contributed to variation in pronunciation.

The original corpus contained 300 manually annotated songs. After removing songs with no usable /aj/ tokens and categories with too few observations for statistical modeling, the final machine-learning dataset contained 295 songs:

Automating Linguistic Annotation with Python and NLTK

Manually searching every lyric for possible instances of /aj/ would have been time-consuming, so I wrote a Python script to automate the first stage of annotation. Using NLTK, the script tokenized song lyrics and identified words belonging to the relevant lexical set. It then generated an annotation template by marking each potential occurrence with an empty [].

For example, instead of repeatedly searching a lyric for every possible word containing /aj/, I could generate a template containing placeholders and listen to the recording while filling in the pronunciation I heard. If the pronunciation was a monophthong, I updated the [] to [a]. Otherwise, I wrote what I heard: [aj], [ae], and [v] for schwa.

Once a song was coded, the number of monophthongized tokens and total /aj/ tokens could be counted automatically and used to calculate the song's monophthongization rate.

Turning the Script into a Web Application

The first version of the annotation tool ran as a Python script on Replit. It worked, but it was cumbersome to use repeatedly while working through hundreds of songs. I wanted a web interface that made it easy to use NLTK, so I built a React frontend that communicated with a Python API hosted on PythonAnywhere.

The application allowed me to paste lyrics into a browser and automatically generate an annotation template. The tool was sufficiently flexible to work for future phonemes I may want to investigate, not just /aj/. I was able to speed up the coding of /aj/ instances much quicker this way.

Technologies

Manual Sociolinguistic Coding

NLP tools could identify where a potential /aj/ token occurred, but my review of technologies at the time showed there was no accessible technology that could actually automatically code how the vowel was realized. The task required human listening.

I listened to each song and coded each relevant vowel based on its pronunciation. For every song, I calculated the percentage of /aj/ tokens that were monophthongized.

The final dataset combined automated lexical detection with human phonetic annotation.

Across the final 295-song dataset, the mean monophthongization rate was 20.87%, while the median was only 5.88%. Rates ranged from 0% to 100%, indicating substantial variation between individual songs.

I also visualized monophthongization over time, allowing changes in pronunciation to be viewed across the history of the corpus.

Machine Learning Analysis

After completing the corpus, I used machine learning to ask a second question: how much of the variation in /aj/-monophthongization could be predicted from time and performer demographics?

My initial experiments considered K-nearest-neighbors regression, but for the final analysis I selected multiple linear regression. Because the target was a continuous percentage and I wanted to understand historical and demographic trends, linear regression provided a simpler and more interpretable model.

The model used three features:

The target was the percentage of /aj/ tokens in the song that had been monophthongized.

I built the modeling workflow using scikit-learn Pipeline and ColumnTransformer. Year was passed through as a numeric feature, while race and sex were converted using one-hot encoding.

After preprocessing, I divided the dataset into an 80% training set and 20% test set, using a fixed random seed of 42 for reproducibility. This produced 236 training observations and 59 test observations.

I evaluated the model using mean absolute error, root mean squared error, R², a median DummyRegressor baseline, and five-fold cross-validation.

Results

On the held-out test set, the multiple linear regression model achieved:

MAE: 18.84 percentage points RMSE: 24.21 percentage points R²: 0.200

The median baseline produced an MAE of 21.09, meaning the regression model reduced mean absolute error by 2.25 percentage points, or approximately 10.7%.

I also performed five-fold cross-validation to determine whether the test-set result was unusually favorable.

The regression model produced a mean cross-validation MAE of 18.75, compared with 20.77 for the baseline, an improvement of approximately 9.7%. The regression MAE had a standard deviation of only 1.23, suggesting that its performance was relatively stable across different subsets of the data.

The model captured a real signal, but only part of the story. An R² of 0.200 means that year, race, and sex explained approximately 20% of the variation in the test data.

That result is itself sociolinguistically interesting: demographic and historical variables matter, but they are far from sufficient to explain how singers adopt this pronunciation.

Limitations and Future Work

One of the strengths of this project was also one of its biggest limitations: the dataset was built manually.

The corpus contained fewer than 300 usable observations, and the demographic distribution was uneven. White and male performers were substantially more common, while several racial categories had too few observations to include reliably in the final model.

The target variable also treats every song equally. A song containing one /aj/ token receives the same weight as a song containing dozens, which may not be the best representation of the underlying phonetic behavior.

Future versions of the project could:

What I Learned

This project required considerably more than training a model.

I had to turn messy historical information into structured data, identify where automation could reduce repetitive work, build an NLP pipeline, design a usable web interface around that pipeline, connect a React application to a Python backend, manually validate data that could not reliably be automated, and finally build and evaluate a reproducible machine-learning workflow.

It is a project at the intersection of two areas I care deeply about: software engineering and linguistics.

More importantly, it illustrates the kind of problem I enjoy working on: starting with a messy real-world question and building the tools necessary to turn it into something measurable.