Dec 1 2017
Since the term “artificial intelligence” (AI) was initially used in print in 1956, the former science fiction fantasy has advanced to the very real prospect of smartphones that recognize complex spoken commands, driverless cars, and computers that see. A Stanford-led team of leading AI thinkers called the AI100 has introduced the first index to monitor the state of artificial intelligence and measure technological advancement just like the GDP and the S&P 500 index take the pulse of the U.S. stock market and economy.
A Stanford-led AI index reveals a dramatic increase in AI startups and investment as well as significant improvements in the technology’s ability to mimic human performance. (Image credit: Tricia Seibold)
“The AI100 effort realized that in order to supplement its regular review of AI, a more continuous set of collected metrics would be incredibly useful,” said Russ Altman, a professor of bioengineering and the faculty director of AI100. “We were very happy to seed the AI Index, which will inform the AI100 as we move forward.”
The AI100 was founded three years ago by a charitable gift from Eric Horvitz, a Stanford alumnus and former president of the Association for the Advancement of Artificial Intelligence. Its first report, published in the fall of 2016, aimed to predict the probable effects of AI in an urban environment in the year 2030.
Among the main findings in the new index is a significant increase in AI startups and investment as well as noteworthy improvements in the technology’s ability to imitate human performance.
Baseline Metrics
The AI Index tracks and measures as a minimum 18 independent vectors in academia, open-source software, industry, and public interest, plus technical valuations of progress toward what the researchers call “human-level performance” in areas such as question-answering, speech recognition, and computer vision – algorithms that can identify activities and objects in 2D images. Precise metrics in the index include assessments of academic papers published, AI-related startups, course enrollment, search-term frequency, job openings, and media mentions, among others.
“In many ways, we are flying blind in our discussions about artificial Intelligence and lack the data we need to credibly evaluate activity,” said Yoav Shoham, professor emeritus of computer science. “The goal of the AI Index is to provide a fact-based measuring stick against which we can chart progress and fuel a deeper conversation about the future of the field.”
Shoham came up with the index and assembled a steering committee including Ray Perrault from SRI International, Erik Brynjolfsson of the Massachusetts Institute of Technology and Jack Clark from OpenAI. The committee then hired Calvin LeGassick as project manager.
“The AI Index will succeed only if it becomes a community effort,” Shoham said.
Although the authors say the AI Index is the first index to monitor either technological or scientific progress, there are many other non-financial indexes that offer crucial insight into equally hard-to-quantify fields. Examples include the Middle East peace index, the Social Progress Index, and the Bangladesh empowerment index, which measure factors as wide-ranging as sanitation, workload, nutrition, public sentiment, leisure time, and even public speaking opportunities.
Intriguing findings
Among the findings of this introductory index is that the number of active AI startups has grown 14-fold since 2000. Venture capital investment has grown six times in the same period. In academia, publishing in AI has equally increased an impressive nine times in the last two decades while course enrollment has soared. Enrollment in the introductory AI-related machine learning course at Stanford, for example, has increased 45-fold in the last three decades.
In technical metrics, speech and image recognition are both approaching, if not exceeding, human-level performance. The researchers observed that AI systems have excelled in such real-world applications as classification of photographic images of skin cancer cells, object detection, and the ability to comprehend and answer questions.
Shoham noted that the report is still highly U.S.-centric and will require a greater global presence as well as a greater variety of voices. He said he also envisions opportunities to fold in government and corporate investment as well as the venture capital funds that are presently included.
With regards to human-level performance, the AI Index indicates that in some ways AI has already arrived. This is real in game-playing applications including the Jeopardy! game show chess, and, in recent times, the game of Go. However, the authors note that computers continue to fall behind significantly in the ability to simplify specific information into deeper meaning.
“AI has made truly amazing strides in the past decade,” Shoham said, “but computers still can’t exhibit the common sense or the general intelligence of even a 5-year-old.”
The AI Index was created with financial assistance from AI100, Google, Microsoft and Toutiao. Data supporting the various metrics were provided by Elsevier, TrendKite, Indeed.com, Monster.com, the Google Trends Team, the Google Brain Team, Sand Hill Econometrics, VentureSource, Crunchbase, Electronic Frontier Foundation, EuroMatrix, Geoff Sutcliffe, Kevin Leyton-Brown and Holger Hoose.