The most prestigious law school admissions discussion board in the world.
|
 |
 
|
Google’s new Gemini Pro is the best overall at law stuff
Poast new message in this thread
Date: May 21st, 2025 2:15 PM Author: Adventurous Hell https://ai.google.dev/showcase/harvey
Harvey: Validating Gemini 2.5 Pro Preview’s Advanced Legal Reasoning with BigLaw Bench
The legal industry demands AI solutions that can navigate an immense volume of complex information with precision and nuance. Traditional AI benchmarks often fall short of capturing the real-world demands of legal practice, making it challenging to identify models truly capable of high-value legal work. Harvey, a dynamic startup dedicated to transforming legal workflows through AI, confronted this by developing BigLaw Bench, their comprehensive framework for assessing Large Language Model (LLM) performance on tasks mirroring actual legal work. In their recent rigorous evaluations, Gemini 2.5 Pro Preview emerged as a standout performer, demonstrating exceptional potential to improve efficiency in key legal domains. Gemini 2.5 Pro Preview Leads on BigLaw Bench
Harvey’s recent evaluations leveraging the BigLaw Bench framework have clearly indicated that Gemini 2.5 Pro Preview demonstrates remarkable proficiency across core legal reasoning tasks and, in particular, tasks requiring reasoning over long-form legal inputs or outputs. As shown in Harvey’s publicly shared results, obtained by testing models including Gemini 2.5 Pro Preview via their respective APIs, Gemini 2.5 Pro Previewachieved the leading score of 85.02% on BigLaw Bench, outperforming other models evaluated in this comprehensive assessment.
This leading capability is crucial for a wide range of high-value legal activities. Key evaluation tasks within BigLaw Bench showcased Gemini 2.5 Pro Preview’s strengths:
Transactional due diligence: Gemini 2.5 Pro Preview showed a strong capacity to extract and summarize critical provisions (e.g., assignment, indemnification, termination clauses) from multiple lengthy service agreements. This suggests a significant potential to streamline the time-intensive process of manual document review. Transaction structuring: The model adeptly generated comprehensive, well-structured comparative analyses of intricate financial options (e.g., PIPE, underwritten equity offerings, bond offerings). The model showed promise in presenting this information in a clear and accessible manner, even for those without deep financial expertise, and in suggesting potential immediate action items. Litigation drafting: When assessed on tasks related to litigation, Gemini 2.5 Pro Preview exhibited a notable ability to generate detailed outlines for legal briefs based on substantial volumes of briefing documents. This capability points towards a future where AI can significantly aid in the initial stages of legal argument development and organization. Document review & analysis: Evaluations involving the review of disparate trial documents (call logs, emails, memoranda) revealed Gemini 2.5 Pro Preview’s strength in creating coherent chronological summaries of events. Furthermore, the model showed potential in identifying critical inconsistencies and ambiguities within the record, a crucial aspect of thorough legal analysis.
Across these evaluations, Gemini 2.5 Pro Preview showcased strong reasoning across inputs consisting of hundreds of pages of materials, a common scenario in legal work. In addition, it was capable of using these materials to generate longer-form and comprehensive outputs, allowing for deeper insights and analyses. These core capabilities highlight the potential for leveraging Gemini 2.5 Pro Preview across complex legal work requiring reasoning over large sets of documents to support diligence, review, and drafting use cases. A New Standard for Legal AI
"At Harvey, we’re committed to equipping legal professionals with the most advanced tools," states Niko Grupen, Head of Applied AI at Harvey. "Our evaluation of Gemini 2.5 Pro Preview through BigLaw Bench has revealed its remarkable ability to synthesize complex legal information. This insight fuels our vision for future product development, where we aim to leverage these strengths to unlock unprecedented efficiency and empower lawyers to focus on higher-level strategic work." Unlocking the Future of Legal Work
Harvey’s commitment to rigorous evaluation and their insightful analysis of cutting-edge AI models like Gemini 2.5 Pro Preview are demonstrating the transformative potential of AI in the legal field. Their findings pave the way for future innovations that promise to reshape how legal professionals approach their most demanding tasks.
To explore how Gemini 2.5 Pro Preview’s advanced reasoning and synthesis capabilities can power your own applications, visit the Gemini API documentation or get started in Google AI Studio.
Harvey is a participant in Google’s AI Futures Fund that invests in and collaborates with ambitious startups building what’s next in AI. |
 |
Date: May 21st, 2025 2:22 PM Author: Cerebral Painfully Honest Athletic Conference Double Fault wow they made an AI of Harvey Specter? (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951270) |
|
Date: May 21st, 2025 2:26 PM Author: Adventurous Hell Yes. And the pricing is a secret (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951282) |
|
Date: May 21st, 2025 8:13 PM Author: splenetic multi-colored sanctuary immigrant Literally would sign up for Harvey if they simply published a price. I'm way to autistic to deal with the whole faggot shit where they force me to do a demo on zoom and then give me a presentation or whatever. WAYYYY too much human interaction. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952192) |
|
Date: May 21st, 2025 8:16 PM Author: Adventurous Hell It’s completely insane. And what are they adding anyway? Just run your prompt through ChatGPT, grok, Claude, and Gemini yourself and pick the best one. You don’t even have to pay for all of them. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952203) |
Date: May 21st, 2025 2:36 PM Author: Overrated skinny woman the openai model versions are confusing as fuck now. why did 4.1 come after 4.5? why is it seemingly better than 4.5? who is the retard who thought having o4 and 4o model names was a good idea? i really don't know where i should be using o3 vs o4 mini high since they are both reasoning models. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951318) |
|
Date: May 21st, 2025 2:42 PM Author: Adventurous Hell Some shithead thought he was real cute with that. You have to consider that these people are all sociopaths with immense world-altering power. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951344) |
|
Date: May 21st, 2025 2:44 PM Author: Overrated skinny woman it seems like they enjoy fucking around with people. now they have 4.1 mini and 4.1 nano. what the fuck. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951354) |
|
Date: May 21st, 2025 3:04 PM Author: Adventurous Hell (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951428) |
|
Date: May 21st, 2025 6:27 PM Author: Jade antidepressant drug (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951931) |
|
Date: May 21st, 2025 5:14 PM Author: twisted jet institution I pay $200 / month and never heard of 4.1
Is 4.1 api only?
I'm still in the o3/4.5 frame and suddenly feel like I'm being ripped off |
|
Date: May 21st, 2025 5:17 PM Author: Duck-like mind-boggling senate it was api only for a while but i think it appeared on chatgpt plus in the last few weeks. no clue what the limits are. hopefully it's better than 4.5, which has extremely low limits. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951789) |
|
Date: May 21st, 2025 5:24 PM Author: twisted jet institution I ask gpt every week which model is best for raw complex legal analysis and it's been saying o3 since late April update
4.1 is buried under "More models" with o1 Pro
Is openAI just fucking with me or what? |
|
Date: May 21st, 2025 5:50 PM Author: Duck-like mind-boggling senate isn't o3 the best according to that chart?
the decision as to what model to use is incredibly confusing. they should train a model to select what model to use for a particular response. |
|
Date: May 21st, 2025 6:07 PM Author: twisted jet institution That's what I thought until this thread
The best thing about o3 is you can seamlessly hop b/w 4.5 in the same convo |
|
Date: May 21st, 2025 5:21 PM Author: Abnormal fighting electric furnace church deep research mode is shit compared to chatgpt for legal stuff. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951798) |
|
Date: May 21st, 2025 5:25 PM Author: Adventurous Hell You used the new one in the $250 ultra plan? (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951814) |
|
Date: May 21st, 2025 6:23 PM Author: Abnormal fighting electric furnace church nah just the one on the $25 a month plan because i got a free trial. it wouldnt take any attachments and it kept telling me it couldnt write a brief and i should get a lawyer. i eventually tricked it into writing one (said it was for educational purposes) but it wasnt very good because it didn let me attach anything.
it also can't go into deep research mid-chat like chatgpt can which is another minus since on chatgpt it can easily use what ever has already been said |
|
Date: May 21st, 2025 6:39 PM Author: Adventurous Hell I think Deep Research is the mid tier and Deep Think is the powerful one. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951960) |
|
Date: May 21st, 2025 7:32 PM Author: Abnormal fighting electric furnace church i gotta try it out but i just downgraded my chargpt pro to plus since i have to start paying loans again now lol (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952093) |
|
Date: May 21st, 2025 7:36 PM Author: Duck-like mind-boggling senate Pro seems pointless. They don’t have o3 high available on it yet. The Deep Think model with Gemini seems like the best premium model available right now. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952099) |
|
Date: May 21st, 2025 8:13 PM Author: Adventurous Hell I got some kind of “administrative forbearance” until 2027 but I can’t justify spending $250 a month when the $20 version is so good. And I get the $20 Claude too. I can’t think of anything where I’d need additional “deep research thinking” on. But maybe I’m missing out on something really cool? (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952189) |
|
Date: May 21st, 2025 9:00 PM Author: Abnormal fighting electric furnace church They finally approved my move back to PAYE lol.
My only justification for Pro was the essentially unlimited deep research use, which is really good for brief writing. but plus mod offers 10 a month so i just need to be judicious. |
|
Date: May 21st, 2025 9:16 PM Author: Adventurous Hell 180 (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952335) |
|
Date: May 21st, 2025 5:54 PM Author: Mildly autistic shrine chicken sandwich wars now AI wars, I use ChatGPT plus and free versions of Claude / Deepseek / Grok / LLama / Gemini, waiting for costs to come down it seems like they are shaking down consumers before they reveal they could release free models all along. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951880) |
|
Date: May 21st, 2025 6:01 PM Author: Appetizing purple stage preventive strike Would this be beneficial for tranny lolyers? (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48951887) |
|
Date: May 21st, 2025 7:03 PM Author: Adventurous Hell It’s the most useful thing in law since the law itself. If you aren’t using it you’re already behind. This will change/destroy the entire world within the next 2 years (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952014) |
|
Date: May 21st, 2025 7:17 PM Author: Appetizing purple stage preventive strike Can it connect clause 9.2 with 7.6? I don't think so (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952052) |
|
Date: May 21st, 2025 7:24 PM Author: Adventurous Hell “THE ALL-NIGHTERS” happens in 3 seconds now (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952078) |
|
Date: May 21st, 2025 7:39 PM Author: Red native how do i use it (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952109) |
|
Date: May 21st, 2025 7:43 PM Author: Fuchsia community account This is relevant to my interests (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952127) |
|
Date: May 21st, 2025 7:55 PM Author: Aromatic space water buffalo Just start using it. I basically haven’t worked since the original Gemini 2.5 came out in March. It does all my work for me and my “job” is just checking it for accuracy. I take really long lunches whenever I want, go to the gym in the middle of the day, Ive reduced my anxiety and am in the best shape in a decade, I take random walks around town, I can drink any time during the day, etc. It’s changed my life completely. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952159) |
|
Date: May 21st, 2025 7:57 PM Author: Duck-like mind-boggling senate Estimate what your anxiety level will be when use becomes widespread and employers realize they don’t need as many people (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952168) |
|
Date: May 21st, 2025 8:02 PM Author: Aromatic space water buffalo A lot of people will be gone. Juniors and support staff have been made useless already and AI is still improving rapidly. (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952178) |
|
Date: May 21st, 2025 8:21 PM Author: Federal yarmulke national (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952220) |
|
Date: May 21st, 2025 8:25 PM Author: Vibrant rose church building police squad (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952230) |
|
Date: May 21st, 2025 8:32 PM Author: bright passionate location ape Same. I started using OpenAI models early last year and hardly work at all. Probably a better lawyer now too tbh (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952255) |
|
Date: May 21st, 2025 9:29 PM Author: arousing sooty prole (Guy about to be replaced by the thing that relieved his anxiety) (http://www.autoadmit.com/thread.php?thread_id=5728503&forum_id=2)#48952362) |