Gemini Ultra outperforms GPT-4 on most standard benchmarks, which evaluate AI models on tasks such as high school physics, professional law, and moral scenarios.
Google has unveiled a new Pro edition of its latest artificial intelligence model, Gemini, and internal sources suggest that it has demonstrated superior performance compared to GPT-3.5 (the free version of ChatGPT) during extensive testing.
According to performance evaluations, Gemini Ultra has surpassed current state-of-the-art results in 30 of 32 widely recognized academic benchmarks used in large language model (LLM) research and development.
Previously criticized for trailing behind OpenAI's ChatGPT, acknowledged as a leader in the AI domain, Google states that Gemini is designed to be multimodal and capable of processing various media types such as text, images, video, and audio.
Additionally, insider reports reveal that Gemini Ultra has achieved a remarkable score of 90.0%, making it the first model to outperform human experts in Massive Multitask Language Understanding (MMLU). MMLU incorporates 57 subjects, including math, physics, history, law, medicine, and ethics, testing worldly knowledge and problem-solving skills.
According to Google's data, Gemini Ultra (depicted in blue) outperforms GPT-4 on most standard benchmarks, which evaluate AI models on tasks such as high school physics, professional law, and moral scenarios. The current landscape of the AI race is shaped mainly by these capabilities.
However, Gemini Ultra only marginally surpasses OpenAI's GPT-4 on most benchmarks, with the lead only a few percentage points. Essentially, Google's premier AI model has made incremental advancements in an area where OpenAI had already completed work at least a year ago. Moreover, details about Ultra are still undisclosed.
If Google follows through with its hinted release in early January, Gemini Ultra may not remain the top model for an extended period. When it took Google to catch up with OpenAI, the more agile competitor, has had nearly a year to develop its subsequent AI model, GPT-5.
Google offers three sizes or stages for the Gemini platform: Ultra (the flagship model), Pro, and Nano (tailored for mobile devices). According to TechCrunch, Gemini Pro will be accessible to enterprise customers through the Vertex AI program and to developers in AI Studio starting December 13. It is also reported that the Pro version can be accessed via Bard, Google's chatbot interface.
Eli Collins, VP of product at DeepMind, informed TechCrunch that Gemini Ultra can comprehend "nuanced" information across text, images, audio, and code.
While acknowledging the performance achievements, questions about the raw materials required for these platforms are gaining prominence. Beyond concerns about energy, hardware, and chips, AI models like ChatGPT are trained on and utilize the creative works of millions of artists, inventors, teachers, and authors.
Content creators have initiated lawsuits against Microsoft, GitHub, OpenAI, and Stability AI, alleging unfair usage. Famous authors, including John Grisham and George R.R. Martin, are part of a group suing OpenAI for what they term "systemic theft on a mass scale."
The U.S. Copyright Office states that, under the fair use doctrine, limited portions of a work, including quotes, can be used for purposes such as commentary, criticism, news reporting, and scholarly reports.
However, the challenge for Gemini, ChatGPT, and other AI solutions lies in the absence of explicit permission from the original human creators and the lack of compensation for using their works extensively to create something new.
As per copyright laws, only the copyright owner has the right to prepare or authorize the creation of a new version of the work, raising questions about how these laws apply to artificial intelligence and how rapidly regulations will be implemented.
Not only does Droot empower organizations with AI-powered solutions, but it also believes in ensuring a fair and creative future so that creativity separates the latter. Click here to learn more about our AI capabilities.