Recently, Open AI launched a more sophisticated version of the o1reasoning model called the o1 pro mode. The chatbot was unveiled on Thursday and is priced at $200 per month. However, the community is uncertain about how to interpret it.
The company’s new ChatGPT Pro plan provides access to “o1 pro mode,” which uses additional computing power to provide the most comprehensive and reliable responses to the most challenging queries related to science, math, and coding.
A former chief officer at Open AI, Bob McGrew, considered o1 pro mode to be an example of “intelligence overhang,” which refers to the inability of users (and potentially the model’s creators) to derive value from any “extra intelligence” as a result of the inherent limitations of a simple, text-based interface.
ChatGPT is the sole method of interaction with o1 pro mode, as is the case with OpenAI’s other models. Bob McGrew noted that ChatGPT is not flawless. He added that it is accurate that $200 establishes lofty expectations. Additionally, ChatGPT Pro is not an easy endeavor, as evidenced by its initial reception on social media.
Here is how the new open AI assistant has been weighed
According to reports, o1 pro mode will be the world’s most expensive chatbot subscription as the $200 monthly fee will translate to $2400 yearly. The service, however, comes with other benefits, like the removal of rate limits and unlimited access to OpenAI’s other models.
British computer scientist Simon Willison asked, “Have OpenAI shared any concrete examples of prompts that fail in regular o1 but succeed in o1-pro?”
On the contrary, failures have already been documented. Sudoku is a challenge for o1 pro mode, and it is impeded by an optical illusion comedy that is readily apparent to any human.
An AI engineer, Tibor Blaho, wrote on X, “o1 and o1-pro both failed here, probably still because of the vision limitations (the same with Sudoku puzzles).” Tibor Blaho also noted that both o1 and o1-pro failed in terms of visual capabilities.
o1 and o1-pro both failed here, probably still because of the vision limitations (the same with Sudoku puzzles)https://t.co/mAVK7WxBrq pic.twitter.com/O9boSv7ZGt
— Tibor Blaho (@btibor91) December 5, 2024
o1-pro code failed to understand/read the puzzle properly.
According to OpenAI’s internal benchmarks, the standard o1 performs only marginally better than the o1 pro mode on coding and math problems.
In order to demonstrate the consistency of the o1 pro mode, OpenAI conducted a more stringent evaluation of the same benchmarks. The model was only deemed to have solved a question if it correctly identified the answer four out of four times. However, the enhancements were not substantial in these assessments.
On Thursday, OpenAI CEO Sam Altman was compelled to reiterate that ChatGPT Pro is not suitable for the majority of individuals, despite his previous assertion that OpenAI was progressing toward intelligence that was “too cheap to meter.”
He said, “Most users will be very happy with the o1 in the [ChatGPT] Plus tier!”
Bob McGrew said that improved reliability may be the most important unlock for o1 pro, especially for the medical field.
Views regarding the cost
Ameet Talwalkar, an associate professor of machine learning at Carnegie Mellon and a venture partner at Amplify Partners, said “It seems like a big risk to me to raise the price tenfold,” he added, “I think we’ll have a much better sense in just a few weeks as to the appetite for this functionality.”
Guy Van den Broeck, a computer scientist at UCLA, said, “I don’t know if the price point makes sense and if pricey reasoning models will be the norm.”
Prospective consumers are not provided with a significant amount of information when the o1 pro mode is described as the most effective at resolving “the hardest problems.”
It is difficult to rationalize paying more in the absence of specific examples of the model’s purportedly enhanced capabilities, which include the ability to “think longer” and exhibit “intelligence.”
From Zero to Web3 Pro: Your 90-Day Career Launch Plan
