Smart devices, digital assistants, and service chatbots are becoming ubiquitous, but they don’t yet have the emotional capacity they need to fully understand how we communicate. Emotional AI models should be able to detect specific emotions and act with empathy, understand societal norms, and recognize specific communicational nuances, like irony and humor. Datasets of emotional data exist, but are inadequate in terms of size and realistic context. Fortunately, there’s a dataset which can satisfy all these requirements: movies.
Cinema is the holy grail of emotional and anthropological data, for these five reasons:
Film Data is Self-Labelling
Movies contain multiple “dimensions” (e.g. word choice, facial expression, etc.) of emotional data, which can be used in tandem as “self-labelling” data. For example, what if we want to know the emotional impact of the phrase “I never want to see you again”? We can look across every film to see a character’s reaction when the phrase is said to them - the reactions to this phrase are already labeled.
Most times, the character’s face will fall into a frown. Sometimes, the character will laugh. Never, will the character’s head will fall off. A film is a reproduction of how we interact with each other. The data is valid because movies depict dramatizations of society. A model trained on movies will find only proper reactions to this phrase. And these are only two dimensions - there are many more, such as voice tone, word choice, expression of the phrase-delivering character, etc.
A Movie is 90 Minutes of Cause-and-Effect
Cinema is meticulously written to show cause-and-effect for everything. The hero can’t just stumble into a happy ending, she has to earn her redemption. On a more granular level, a two-character dialogue scene is just a layering of conversational antecedents atop one another. A character says something, which causes the other character to respond, which causes the first character to respond, and so on. What was just previously said in the conversation to make a character declare “Prison changes you”?
At a plot level, we can track character motivations and what’s important to them, which allows us to interpret antecedent reactions on a broader scale. If we know the character is excited about a job interview, how do they react when they hear they didn’t get the job?
Cinema is a Document of Societal Norms
Going further into the contextual benefits of cinematic expression, movies offer real-world societal context. Consider an average movie scene that takes place in a restaurant. In a restaurant, there are a number of rules that diners follow. What usually happens when a server asks “What can I get you?” or “Can I take your order?” And a common dynamic in restaurant scenes involves one character starting to lose their cool and yelling, and the other calming them down and asking them to be quiet. It’s an unspoken rule to not make a scene in restaurants.
Driving scenes depict characters seated in a specific pattern, facing forward. Birthday celebrations show a number of rituals which include cake, candle, and song. Airport and hospital scenes are both filled with goodbyes, though perhaps with different undertones. Although they aren’t necessarily important to the plot of the film (e.g. a restaurant dialogue could be adapted to take place anywhere else), they provide important societal context, reflecting the typical anthropological rules and rituals of how people actually act in specific locations and situations.
Understanding Human Behavior Requires Multiple Streams of Emotional Data
Many datasets may only focus on one specific type of data: facial expression, word choice sentiment, voice tone, etc. They all get the job done in their respective fields, but some societal norms are too nuanced to be quantified by just a single type of emotional data.
Sarcasm often fools AI models which track word choice sentiment, because positive words are used in a negative manner. But if we looked across all of cinema for positive word choice, deadpan voice tone, and a neutral facial expression, we could reliably find instances of characters being sarcastic. There are a number of other mannerisms which require context of other streams, like facial models detecting crying (tears of heartbreak vs. tears of joys), and body language models detecting clapping (a crowd cheering vs. a villain emerging from the shadows).
It’s Already There
Hundreds of thousands of movies already exist, ready to be parsed as structured data. They contain lots of emotional cause-and-effect information, and exist as a mirror image of how society perceives itself. Existing datasets are limited in size, may only focus on specific streams of emotional data, are devoid of societal context, and lack the behavioral antecedents needed to truly understand emotional response. Hiring actors to deliver lines or paying people to watch and label clips of TV requires lots of human effort (which translates to a monetary cost). Again, movies are already there, ready to be turned into self-labelling, contextually-rich data.