Aug. 31 (Bloomberg) -- The information glut that marks the 21st century is evidenced in some unexpected places. Last month, my organization, the Combating Terrorism Center at West Point, released a report that sharply disputed conventional wisdom about terrorism along the Afghanistan-Pakistani frontier.
The report argued that the Haqqani Network, a border-spanning tribal group with deep ties to Pakistan’s government, had been more influential than the Taliban in aiding al-Qaeda’s rise.
How did we support this thesis, which has vast implications for reconciliation efforts in the region as well as for the distribution of U.S. military aid? Not with data culled from clandestine operations in Pakistan’s tribal areas or from Osama bin Laden’s computer hard drive. The report was based on the public statements and writings of individual extremists over the past 30 years. Rather than ferreting out secret information, researchers merely took extremists at their voluminous word.
It seems terrorists, too, are susceptible to the syndrome known as Too Much Information.
The revolution in information technology has opened a new vein of intelligence collection and analysis that in many instances can prove more useful than traditional forms of spycraft. In the world of espionage, information and the clandestine means of gathering it are both treasured. “Open source” intelligence, by contrast, is a commodity with little inherent value. Instead, the capacity to organize and analyze these public streams of information becomes a key asset. This represents a drastic shift, with far-reaching implications for intelligence agencies.
In another recent effort, the Combating Terrorism Center used Google Trends -- hardly a cloak-and-dagger operation -- to assess the popularity of the Muslim Brotherhood in the midst of the Egyptian uprisings. In the U.S., fears of the Brotherhood’s Islamist agenda dominated public discussion. Not so in Egypt, where Google Trends indicated that during the revolution, Internet searches for non-Brotherhood political figures dwarfed those for the Brotherhood.
Do such open-source data provide scientific proof of public consciousness? No, but they challenge hard-baked conventional wisdom and provide a corrective to guesswork. By tapping the open source of Google Trends, we threw light on a complex mass phenomenon for which traditional intelligence gathering was ill-suited.
Google Trends is just one tool available to open-source analysts, with new information caches constantly emerging. Technological platforms such as search engines and social networks receive a tremendous amount of attention, but they are among a multitude of sources. Newspapers, magazines, blogs, news websites, radio and television interviews can be just as valuable.
In 2009, we surveyed Arabic-language news to produce a report on victims of al-Qaeda violence. The results showed that 85 percent of the casualties were citizens of Muslim-majority nations. That simple exercise in data collection and analysis sparked a valuable debate in Muslim countries about the tactics, goals and morality of the terrorist organization. In a battle for hearts and minds, that’s a powerful effect. Al-Qaeda spokesman Adam Gadahn took the report seriously enough to respond publicly, defending his organization.
Capturing and organizing thousands of Twitter messages or conversation streams from hundreds of social networks requires language competency and the analytical skills to draw conclusions from large unstructured data sources. The report on al-Qaeda’s violence was coded and analyzed by just three people. Some open-source collection and analysis can be very labor-intensive, though there is an increasing array of technological tools to help. They haven’t surpassed the capacity of trained individuals, but they do represent a new frontier of intelligence gathering, one that is worthy of investment.
In some cases, open source may be the only means of gathering intelligence, particularly in closed societies where repressive governments rely on networks of informants to cultivate fear and stifle communication. When civil violence broke out in Libya, the intelligence community struggled to keep pace with the complex political cross-currents of secular, Islamic and radical factions. In many cases, the local and international news media became the best real-time information sources.
Open-source analysis has its limits. A few years back, a team of mathematical and computer modelers used such data to predict the location of Bin Laden. Their analysis placed him far from his actual compound in Abbottabad, Pakistan. In such tactical and operational matters, secret intelligence plays a far more important role. In the future, however, as digital media, social networking and camera phones proliferate, leveraging open-source channels will grow even more important. U.S. intelligence agencies should reorient their budgets, personnel and operations to properly exploit them. The world is telling us much. It’s wise to listen.
(Scott Helfstein is director of research for the Combating Terrorism Center at West Point and provides advice on international affairs in the public and private sectors. The opinions expressed are his own and not those of the U.S. Military Academy, the Army or the Department of Defense.)
To contact the author of this article: Scott Helfstein at email@example.com.
To contact the editor responsible for this article: Francis Wilkinson at firstname.lastname@example.org.