Would you like to receive notifications on latest updates of the following headlines?

AI experts ready 'Humanity's Last Exam' to stump powerful tech

POSTED ON September 17, 2024 •   Technology      BY Abiodun Saheed Omodara
Figurines with computers and smartphones are seen in front of the words "Artificial Intelligence AI" in this illustration taken, February 19, 2024.y l. Credit: Reuters

A team of technology experts issued a global call on Monday seeking the toughest questions to pose to artificial intelligence systems, which increasingly have handled popular benchmark tests like a child's play.

Dubbed, 'Humanity's Last Exam,' the project seeks to determine when expert-level AI has arrived. 

It aims to stay relevant even as capabilities advance in future years, according to the organisers, a non-profit called the Centre for AI Safety (CAIS) and the startup Scale AI.

The call comes days after the maker of ChatGPT previewed a new model, known as OpenAI o1, which "destroyed the most popular reasoning benchmarks," said Dan Hendrycks, executive director of CAIS and an advisor to Elon Musk's xAI startup.

Hendrycks co-authored two 2021 papers that proposed tests of AI systems that are now widely used, one quizzing them on undergraduate-level knowledge of topics like US history, the other probing models' ability to reason through competition-level math. 

The undergraduate-style test has more downloads from the online AI hub Hugging Face than any such dataset.

At the time of those papers, AI was giving almost random answers to questions on the exams. "They're now crushed," Hendrycks told Reuters.

As one example, the Claude models from the AI lab Anthropic have gone from scoring about 77% on the undergraduate-level test in 2023, to nearly 89% a year later, according to a prominent capabilities leaderboard.

These common benchmarks have less meaning as a result.

AI has appeared to score poorly on lesser-used tests involving plan formulation and visual pattern-recognition puzzles, according to Stanford University’s AI Index Report from April. 

OpenAI o1 scored around 21% on one version of the pattern-recognition ARC-AGI test, for instance, the ARC organisers said on Friday.

Some AI researchers argue that results like this show planning and abstract reasoning to be better measures of intelligence, though Hendrycks said the visual aspect of ARC makes it less suited to assessing language models. "Humanity’s Last Exam" will require abstract reasoning, he said.

Answers from common benchmarks may also have ended up in data used to train AI systems, industry observers have said. 

Hendrycks said some questions on "Humanity's Last Exam" would remain private to make sure AI systems' answers are not from memorisation.

The exam will include at least 1,000 crowd-sourced questions due on November 1 that are hard for non-experts to answer. 

These will undergo peer review, with winning submissions offered co-authorship and up to $5,000 prizes sponsored by Scale AI.

"We desperately need harder tests for expert-level models to measure the rapid progress of AI," said Alexandr Wang, Scale's CEO.

One restriction: the organisers want no questions about weapons, which some say would be too dangerous for AI to study.

0
READ ALSO
Federal Ministry Unveils Digital Village to Bridge Digital Divide in Abuja Community
BY Abiodun Saheed Omodara May 30, 2025 0

The Federal Ministry of Communications, Innovation and Digital Economy has launched a Digital Villag...

READ ALSO
FG Unveils Project CNG-SPROUT to Revolutionize Campus Transport with Clean Energy
BY Abiodun Saheed Omodara May 30, 2025 0

The federal government has launched a transportation initiative called "Project CNG-SPROUT," designe...

READ ALSO
Agro Alliance Takes Top Honors at Hack4Livestock with IoT Solution for Cattle Intrusions
BY Abiodun Saheed Omodara May 24, 2025 0

ABUJA, Nigeria - The Ministry of Livestock Development has announced its intention to empower 1,000...

READ ALSO
INEC Launches AI Division to Enhance Electoral Processes, Address Challenges
BY Abiodun Saheed Omodara May 23, 2025 0

ABUJA, Nigeria - The Independent National Electoral Commission (INEC) has sanctioned the establishme...

READ ALSO
Legal Obligations Remain Despite Threats to Exit Nigeria, FCCPC Rebukes Meta Platforms
BY Abiodun Saheed Omodara May 3, 2025 0

The Federal Government, through its Competition and Consumer Protection Commission (FCCPC), has crit...

READ ALSO
Digital Inequality in Africa: High costs, infrastructure gaps leave millions offline
BY Abiodun Saheed Omodara April 21, 2025 0

Despite significant investments in Nigeria and various regions of Africa, only 38 percent of the pop...

READ ALSO
CBEX unregistered digital assets exchange in Nigeria, SEC warns of investment risks
BY Abiodun Saheed Omodara April 19, 2025 0

The Securities and Exchange Commission (SEC) has announced that Crypto Bridge Exchange, also referre...

READ ALSO
NDPC launches initiative to combat cyberbullying, financial fraud through data protection
BY Abiodun Saheed Omodara April 7, 2025 0

The National Data Protection Commission (NDPC) on Monday reiterated its dedication to enhancing data...

OUR CHANNELS:

Enugu Proposes Bill to Curb Criminal Activities Linked to Traditional Practitioners
BY Abiodun Saheed Omodara June 6, 2025 0

ENUGU, Nigeria - Enugu State Governor Peter Mbah has proposed an Executive Bill to the Enugu State H...


Bill Gates Advocates for Improved Health Care in Nigeria Amid Budget Constraints
BY Abiodun Saheed Omodara June 6, 2025 0

US billionaire Bill Gates, who chairs the Gates Foundation, expressed that Nigeria’s health bu...


Controller General Issues Urgent Security Directive for Correctional Facilities
BY Abiodun Saheed Omodara June 6, 2025 0

The Controller General of Corrections, Sylvester Nwakuche, has issued a strict order to all Zonal Co...


NEMA Mobilizes Teams to 15 States Amid Flood Warnings
BY Abiodun Saheed Omodara June 5, 2025 0

The National Emergency Management Agency (NEMA) has sent officials to 15 high-risk states to educate...


Edo Faces Diphtheria Outbreak, Two Confirmed Deaths
BY Abiodun Saheed Omodara June 5, 2025 0

EDO, Nigeria (NAN) - A diphtheria outbreak has been confirmed in Edo State, resulting in two deaths....


Tinubu Honors Bill Gates with National Award for Global Philanthropy
BY Abiodun Saheed Omodara June 5, 2025 0

LAGOS, Nigeria - President Bola Tinubu commended Bill Gates, the co-founder of Microsoft and chairma...


Bahrain, Colombia, DRC, Latvia, and Liberia Join UN Security Council for 2026-2027 Term
BY Abiodun Saheed Omodara June 5, 2025 0

The UN General Assembly on Tuesday elected five new non-permanent members to the UN Security Council...


Tinubu Orders Urgent Repairs on Flood-Damaged Bridge in Mokwa
BY Abiodun Saheed Omodara June 5, 2025 0

NIGER, Nigeria (NAN)- President Bola Tinubu has instructed the Federal Ministry of Works to repair t...


Nigeria Needs ₦2.2 Trillion to Eliminate Cervical Cancer by 2030, Says Task Force Chair
BY Abiodun Saheed Omodara June 5, 2025 0

ABUJA, Nigeria — Nigeria will need ₦2.2 trillion to roll out comprehensive cervical cancer ini...


EFCC Charges Two Men for Posing as Fake Sureties in Illegal Mining Case
BY Abiodun Saheed Omodara June 4, 2025 0

The Economic and Financial Crimes Commission (EFCC) has charged two individuals, Shehu Abdu and Isah...


More Articles

Load more...

Menu