Would you like to receive notifications on latest updates of the following headlines?

AI experts ready 'Humanity's Last Exam' to stump powerful tech

POSTED ON September 17, 2024 •   Technology      BY Abiodun Saheed Omodara
Figurines with computers and smartphones are seen in front of the words "Artificial Intelligence AI" in this illustration taken, February 19, 2024.y l. Credit: Reuters

A team of technology experts issued a global call on Monday seeking the toughest questions to pose to artificial intelligence systems, which increasingly have handled popular benchmark tests like a child's play.

Dubbed, 'Humanity's Last Exam,' the project seeks to determine when expert-level AI has arrived. 

It aims to stay relevant even as capabilities advance in future years, according to the organisers, a non-profit called the Centre for AI Safety (CAIS) and the startup Scale AI.

The call comes days after the maker of ChatGPT previewed a new model, known as OpenAI o1, which "destroyed the most popular reasoning benchmarks," said Dan Hendrycks, executive director of CAIS and an advisor to Elon Musk's xAI startup.

Hendrycks co-authored two 2021 papers that proposed tests of AI systems that are now widely used, one quizzing them on undergraduate-level knowledge of topics like US history, the other probing models' ability to reason through competition-level math. 

The undergraduate-style test has more downloads from the online AI hub Hugging Face than any such dataset.

At the time of those papers, AI was giving almost random answers to questions on the exams. "They're now crushed," Hendrycks told Reuters.

As one example, the Claude models from the AI lab Anthropic have gone from scoring about 77% on the undergraduate-level test in 2023, to nearly 89% a year later, according to a prominent capabilities leaderboard.

These common benchmarks have less meaning as a result.

AI has appeared to score poorly on lesser-used tests involving plan formulation and visual pattern-recognition puzzles, according to Stanford University’s AI Index Report from April. 

OpenAI o1 scored around 21% on one version of the pattern-recognition ARC-AGI test, for instance, the ARC organisers said on Friday.

Some AI researchers argue that results like this show planning and abstract reasoning to be better measures of intelligence, though Hendrycks said the visual aspect of ARC makes it less suited to assessing language models. "Humanity’s Last Exam" will require abstract reasoning, he said.

Answers from common benchmarks may also have ended up in data used to train AI systems, industry observers have said. 

Hendrycks said some questions on "Humanity's Last Exam" would remain private to make sure AI systems' answers are not from memorisation.

The exam will include at least 1,000 crowd-sourced questions due on November 1 that are hard for non-experts to answer. 

These will undergo peer review, with winning submissions offered co-authorship and up to $5,000 prizes sponsored by Scale AI.

"We desperately need harder tests for expert-level models to measure the rapid progress of AI," said Alexandr Wang, Scale's CEO.

One restriction: the organisers want no questions about weapons, which some say would be too dangerous for AI to study.

0
READ ALSO
Cyber Threats: Nigeria's digital ecosystem at risk from scams and AI attacks
BY Abiodun Saheed Omodara January 22, 2025 0

LAGOS- A report by the Cyber Security Experts Association of Nigeria has indicated that Nigeria&rsqu...

READ ALSO
NDPC launches National Certification Programme for DPOs
BY Abiodun Saheed Omodara January 21, 2025 0

NIGERIA, (NAN) - The Nigeria Data Protection Commission (NDPC), on Monday, launched the national cer...

READ ALSO
Trump to reinstate TikTok operations in U.S on Monday, seek 50% ownership
BY Abiodun Saheed Omodara January 20, 2025 0

UNITED STATE- US President-elect Donald Trump has announced on Saturday that he plans to reinstate T...

READ ALSO
Omo-Eko mobile app 90% completed- Commissioner
BY Abiodun Saheed Omodara January 18, 2025 0

LAGOS, Nigeria(NAN) - The Lagos state commissioner for transportation, Oluwaseun Osiyemi, has announ...

READ ALSO
Install CCTVs on high-rise buildings- LASG
BY Abiodun Saheed Omodara January 15, 2025 0

LAGOS, Nigeria(NAN)  - The Lagos State government has made it mandatory for building owners to...

READ ALSO
Improving smartphone affordability in Africa- Analys
BY Abiodun Saheed Omodara January 9, 2025 0

Smartphones have become central to the lives of mil­lions across Africa, deliver­ing financi...

READ ALSO
Venezuela slams TikTok with $10m fine amid offensive challenge
BY Benedicta Bassey December 31, 2024 0

Venezuela’s Supreme Court has issues a $10 million fine against TikTok for disregarding measur...

READ ALSO
Website Hack: NBS budgets N35m for cybersecurity training
BY Abiodun Saheed Omodara December 26, 2024 0

The National Bureau of Statistics has earmarked N35m in its 2025 budget proposal for “Capacity...

OUR CHANNELS:

Penalty points system activates for traffic offenders- FRSC
BY Abiodun Saheed Omodara February 11, 2025 0

ABUJA, Nigeria (NAN) - The Federal Road Safety Corps (FRSC), corps marshal, Shehu Mohammed, has orde...


Businessman kills friend in Ikoyi-police
BY Abiodun Saheed Omodara February 10, 2025 0

LAGOS,Nigeria - The Nigerian Police have apprehended a murder suspect in Lagos. The individual, iden...


LASG to expand Red Line Rail service to enhance commuting efficiency
BY Abiodun Saheed Omodara February 10, 2025 0

LAGOS, NIgeria- Governor Babajide Sanwo-Olu of Lagos State on Sunday, made a significant announcemen...


Domestic refiners challenge crude producers over lack of allocations
BY Abiodun Saheed Omodara February 10, 2025 0

Access to crude oil for domestic refiners, including modular refineries, has remained nearly nonexis...


Adamawa Govt. refutes dismissed REC's claims on 2023 election results
BY Abiodun Saheed Omodara February 10, 2025 0

The Adamawa State Government has refuted the recent assertion made by the dismissed Resident Elector...


NECA raises alarm over 4% customs administration charge
BY Abiodun Saheed Omodara February 10, 2025 0

The Nigeria Employers’ Consultative Association (NECA) has expressed its worries regarding the...


Nasarawa University set for indefinite strike on Monday
BY Abiodun Saheed Omodara February 10, 2025 0

Academic and non-academic personnel of higher education institutions in Nasarawa State, operating un...


UNICEF Urges Kano Govt. to intensify polio combat, routine immunisation efforts
BY Abiodun Saheed Omodara February 12, 2025 0

KADUNA, Nigeria (NAN) - The United Nations Children’s Fund (UNICEF) has urged the Kano state g...


Kebbi Govt. prepares for mass wedding on february 27
BY Abiodun Saheed Omodara February 10, 2025 0

The Kebbi state government has started preparations for a mass wedding set to take place on February...


Katsina police arrest two kidnappers, other criminals
BY Abiodun Saheed Omodara February 8, 2025 0

KATSINA, Nigeria- The Katsina State Police Command has taken into custody two alleged kidnappers who...


More Articles

Load more...

Menu