By Joshua Haecker and Fallon Farmer
In the past year, multinationals have rushed to leverage generative AI to get a better grasp of geopolitical risks they face around the world. Amid mounting emerging market instability, security crises, sanctions regimes and trade wars, generative AI offers significant potential for companies’ risk and intelligence teams.
However, by moving so quickly, some have not properly considered the limitations of the data-synthesising technology – an oversight that can lead to flawed operational decisions. The haste with which the technology is being deployed as a forecasting tool has left some firms with models whose responses to questions may be inaccurate. This, in many cases, is because the AI has not had access to sufficiently comprehensive and balanced data sets, nor been trained to learn from and interpret these sources.
Put simply, the models sometimes do not have the knowledge or the skills to answer the questions they are being asked. Consequently, the answers they do provide can be misleading or erroneous. That, in turn, can lead to bad corporate decision-making, for example to pull out of a market or stand down a security team, when there’s no compelling reason to do so.
The challenges of using AI as a means of helping companies assess security risks are particularly evident when operating in potentially volatile jurisdictions. Below, we illustrate this with hypothetical examples of corporate risk assessment in the run-up to the Israel-Hamas and the Ukraine-Russian wars, where there were few signs of imminent conflict.
In September of last year, a multinational considering expanding its operations in southern Israel may have asked its generative AI model to assess the likelihood of an escalation of tensions between Hamas and Israel. The organisation has periodically launched rocket attacks from its base in Gaza against Israeli urban centres. These have triggered reprisal strikes. The exchanges have often quickly spiralled into highly disruptive security crises.
In the weeks before the October 7 Hamas raid into southern Israel, there would not have been much, if any, open-source information warning of such an operation. Hamas’s actions caught observers by surprise. So, based on the data available, an AI model might have said there were no indications of a possible escalation. A human analyst, however, would likely have provided a more cautious response. They might have suggested that the situation was unclear because of the lack of data on security conditions in the region and Hamas’s intent.
Prior to the outbreak of the Ukraine war on February 24, 2022, there was a massive build-up of Russian forces near the Ukrainian frontier with Russia. Yet most observers took the view that Moscow was not about to invade but was instead trying to exert pressure on Kyiv to make political concessions. Views changed when the US, drawing on its own intelligence, warned an invasion was imminent.
Before that warning, a generative AI model asked to assess escalating tensions may have suggested the country was not under immediate threat. A human analyst’s response would likely have been more caveated, perhaps drawing on a wider range of sources. Some of these might have given more weight to the possibility of an invasion.
FiscalNote has been researching and experimenting with generative AI models, exploring their limitations, and developing ways of mitigating them. Our work has shown that companies can use prompting techniques to prime the models and better prepare them for the tasks they perform or the answers they produce. In essence, it means teaching the model that ‘this is what I mean when I ask you to do a certain task’.
Through our work, we’ve identified five key steps (outlined below) that companies can take to craft effective AI models that generate accurate and actionable outputs. This will help organisations navigate and respond to the dynamic challenges of today’s business environment, maximizing the benefits of AI technologies while mitigating potential pitfalls.
Picking a rich and robust set of training data. The data sets used to train AI models should be as comprehensive, timely and authoritative as possible. If there are gaps or omissions in the sources, then answers generated may be flawed, no matter how much effort is made to improve the programme’s synthesising capabilities.
Prompting with task-specific instructions. As much as possible, AI should be trained to answer critical questions following the same processes and techniques as their human counterparts – for example, to cite sources on which its answers are based. Essentially, this amounts to explaining how it came to its conclusions. If not required to tie them back to source documents, there is a risk of the model making up answers.
Permitting responses with no answers. Generative AI is hard-wired to provide an answer to a question, regardless of whether it has the correct answer or not. This can be avoided by including fallback options when instructing a model. So, where there isn’t the data to support an accurate response to a question, the model is allowed to say, “I don’t know”, “I don’t have enough information to answer”, or ” I can’t help with that”.
Making sure responses are objective and balanced. Companies can also use generative AI to self-check for potential bias in answers. This becomes particularly important, for example, in responding to questions related to conflicts where the positions of the parties involved are unequally represented in data sources. Once sensitive to bias, the model can confirm whether the parties are being presented fairly in its responses; identify when that’s not happening; and determine if this is due to insufficient data.
Designing an option for human input. AI systems should be deployed with a human-in-the-loop design, which effectively recognises when a question should be escalated to a human analyst. The escalation may be triggered by answers that generate high rates of bias, “don’t knows”, or questionable source citations. The human analyst then draws on their own expertise and knowledge, which might include taking into consideration subjective and intangible factors beyond the scope of an AI model.
As geopolitical threats multiply around the world, companies are right to consider using AI models to assist them with critical operational decisions. These programmes can certainly inform and enhance risk assessment, but to be truly effective they must be trained and have access to optimal data sets. Even then, they may not have all the answers. So, it is imperative that they work in tandem with human analysts, not replace them. If well-managed, the technology is a real asset. If not, it can be a liability.
About the Authors
Joshua Haecker is Head of Product, FiscalNote Global Intelligence.
Fallon Farmer is a Principal Data Scientist at FiscalNote.
