How We Use Cognitive AI: Predicting Public Opinion Before Asking

In our last posts, we've shared insights about priority divides between economic concerns and climate action. You may have noticed a reference to "cognitive-AI models from Limbik" that helped us forecast audience beliefs. Today, I want to pull back the curtain on this fascinating methodology transforming how we conduct research at Better Odds.

We are a Limbik partner, so we collaborate closely, and I spend some of my time helping them expand in Europe. But this post is not sponsored in any way.

Moving Beyond Traditional Surveys

For decades, understanding public opinion has meant one thing: asking people what they think through surveys and polls. While invaluable, traditional surveys come with significant limitations:

• They're expensive and time-consuming to design and field
• Response rates continue to decline globally
• Results can be skewed by social desirability bias (people answering how they think they "should")
• The very act of questioning can influence responses
• By the time results are analysed, public sentiment may have shifted

What if we could predict how audiences would respond to questions before asking them? This is where cognitive AI enters the picture.

What Is Cognitive AI?

Cognitive AI, pioneered by Limbik, fuses traditional polling methodologies with advanced machine learning methods. The secret sauce combines ongoing surveying of individuals with fine-tuning of AI models.

The technology builds sophisticated prediction models to forecast how different demographic segments would likely respond to specific statements or questions, even if they have never seen that exact statement. These forecasts have been validated against traditional polling methods and produced equivalent results with greater speed, lower cost, and often more granular insights.

How We Use It at Better Odds

Our research process typically follows these steps:

1. Question formulation: We carefully design competing statements that capture the essence of what we want to understand. For our economy vs. climate post, we tested: "It is more important to save the economy than to solve climate change" against "It is more important to solve climate change than save the economy."
2. Audience segmentation: We define the demographic breakdowns we're interested in. This can be as broad as "adult Americans" or more specific, like “Instagram Users 65+ in Portugal” or “women aged 25-34 in urban areas of Sweden with postgraduate education.”
3. AI forecasting: The cognitive AI models predict what percentage of each segment would agree with each statement, based on patterns seen in hundreds of thousands of previous survey responses.
4. Cross-validation: We sometimes validate against traditional polling methods for critical insights, though we've found the AI predictions remarkably accurate.
5. Analysis and storytelling: We identify the most meaningful patterns in the data and craft narratives that help our readers understand the implications.

The Benefits Over Traditional Methods

Our shift to cognitive AI for opinion research has transformed our work in several ways:

• Speed: What once took weeks now takes hours. We can test new hypotheses rapidly and iterate on our research questions.
• Cost efficiency: We can explore more questions and segments without the prohibitive costs of traditional polling.
• Reduced social desirability bias: We may get closer to people’s actual beliefs since we're not directly asking them questions about potentially sensitive topics.
• Granularity: We can look at very specific demographic intersections without worrying about sample size limitations.
• Scenario testing: We can rapidly test how different framings of the same issue might resonate with various audiences.

Real-World Applications

Beyond our recent economy vs. climate research, we've applied cognitive AI forecasting to understand:

• Likely sentiment in priority audiences for future communication efforts
• Regional variations in trust toward emerging technologies
• Gender differences in risk perception across finance and geopolitics
• How education levels correlate with attitudes toward health information

The methodology is valuable for exploring sensitive topics where traditional survey responses might be influenced by social pressure or for quickly gauging reactions to emerging issues before conventional polls can be fielded.

Limitations and Ethical Considerations

While powerful, cognitive AI forecasting isn't without limitations:

• It works best when predicting responses to clearly articulated statements rather than open-ended questions
• Like all AI systems, it can potentially inherit biases present in its training data
• It can't predict what an audience thought of a topic at a previous time, so you need to run forecasts consistently to capture changes.

We work very closely with Limbik to develop this technology. We're careful to use this technology ethically, are transparent about our methodology and avoid applications that could manipulate or exploit vulnerable populations.

The Future of Opinion Research

We're witnessing a fundamental shift in how we understand public opinion. Traditional surveys won't disappear—they remain essential for validating models. However, cognitive AI forecasting offers a complementary approach that can dramatically expand the scope and speed of our research.

In future posts, we'll continue using this methodology to explore fascinating divides in public opinion, offering you insights that would be difficult or impossible to obtain through conventional methods alone.

What Do You Think?

Have you encountered cognitive AI forecasting in other contexts? What questions would you be curious to explore using this methodology? I'd love to hear your thoughts in the comments below.