Detecting emotions in Facebook political ads with computer vision

Dr Michael Bossetta

Assistant Professor in the Department of Communication and Media at Lund University. His research interests revolve around the intersection of social media and politics, especially political campaigning. He produces and hosts the Social Media and Politics podcast, available for free on any podcast app.

Rasmus Schmøkel

Copenhagen University M.Sc. in Political Science from Copenhagen University. His research interests include visual political communication, computational methods, and the role of emotions in algorithmic bias.

Section 5: Social media

Set against the backdrop of a pandemic and polarized political climate, this election was sure to be emotional. Therefore, we used computer vision to examine the emotions expressed by Biden and Trump in their Facebook political ads. Our aim was to see if the candidates expressed different emotions in images where they presented themselves versus how they depicted opponents in attacks.

Indeed, we found that Biden primarily expressed happiness in his Facebook ads, whereas his attack ads depicted Trump as angry. Meanwhile, the Trump campaign presented their candidate as a calm leader, while Biden was often shown expressing confusion. Overall, both campaigns’ Facebook ads largely focused more on promoting their own candidate rather than on attacking the opponent.

To collect and analyze images in Facebook ads, we used two open science software tools that we developed. The first, FBAdLibrarian, assists researchers in collecting images from the Facebook Ad Library. The second, Pykognition, leverages Amazon’s facial detection algorithms to classify emotions expressed in faces.

We collected ads from the official Trump and Biden Facebook pages in the week before Facebook’s ad pause on October 27th. Due to controversies around this ad pause, the data we collected from Facebook may be incomplete, and it appears that Facebook removed all information about dates from the data.

In total, we obtained 202,000 Trump ads and 109,287 Biden ads. Using FBAdLibrarian, we collected 98,830 images from Trump ads and 69,941 images from Biden ads. This means that in our dataset, 49% of Trump’s ads and 64% of Biden’s ads contained still images (the remainder were videos).

We then removed all images that did not depict Trump or Biden. This includes ads that were infographics, promoting merchandise, or featuring other high-profile politicians. Many of the remaining images were copies, so we removed duplicates and ended up with a dataset of 634 images for Trump and 1,148 for Biden.

We further divided these images into three ad categories used in previous research: Promote, Contrast, and Attack. Promote images show only the candidate, Contrast images show both the candidate and the opponent, and Attack images show only the opponent. We divided the images this way to see whether campaigns strategically change the emotions displayed by their candidate versus the opponent.

Emotion Classification Results

We ran all images through Amazon’s Rekognition API with our software, Pykognition. This process categorizes faces into eight emotional categories: Angry, Calm, Confused, Disgusted, Fear, Happy, Sad, and Surprised. Each face is classified with a unique identifier (“FaceID”), an emotion, and a predicted score for that emotion. In Figure 1, we show examples from the Biden page for each category: Promote, Contrast, and Attack.

We manually checked the algorithm’s classification for each image. In cases where we disagreed with the algorithm, we changed the emotion to the one we considered most accurate. Overall, we agreed with the algorithm in 73% of cases.

Interestingly and unique to this election cycle, candidates (and Biden in particular) wore protective facemasks in response to the coronavirus pandemic. This proved problematic for the facial detection algorithm, which predicted pictures with facemasks to display emotions such as sadness and fear but with unreliable confidence. We therefore reclassified all images where candidates wore a mask into a new category: “MASK”. With the images classified, we were able to link our coded data to all other ads in the dataset using the same image. In Figure 2, we present our overall results per candidate page and for each ad type.

Overall, we find that in terms of the number of ads sent, both campaigns emphasized promoting their own candidate rather than attacking the opponent. Biden was most often depicted as ‘Happy’, and he wore a facemask in approximately 10% of images. By contrast, the Trump page issued ads promoting the candidate as the ‘Calm’ leader and wore a mask in less than 1% of images. In Figure 3, we also report each category in proportional terms, to better understand the distribution of emotions per ad type.

For Biden, there was a clear distinction between his own portrayal as “Happy” and Trump as “Angry”. For Trump, we see his page’s ads depicting him as “Calm”, whereas Biden was most often depicted as “Confused” in attacks.

Our analysis reveals how campaigns preferred to show their candidates in Facebook ads: Biden as warm and happy, and Trump as the calm leader. In addition, we see how the campaign’s broader attack narratives were also depicted in images: Biden attacked Trump as an angry despot, and Trump attacked Biden as a confused candidate in mental decline.

It is important to note that we did not factor in the spending amount or number of people who saw these ads; we only studied the raw number of ads issued by the campaign. And, due to limitations imposed by Facebook, we do not know the dates of when these ads were issued. Nevertheless, we encourage other researchers to use the open science tools that we developed to further analyze emotions in political images.