The most prestigious law school admissions discussion board in the world.
|
 |
 
|
AI BigLaw Bench subtask scores
| Fiercely-loyal Stage | 05/14/25 |
| Fiercely-loyal Stage | 05/14/25 |
| Violent rigor | 05/14/25 |
| Fiercely-loyal Stage | 05/14/25 |
| white misunderstood wagecucks church | 05/14/25 |
| Fiercely-loyal Stage | 05/14/25 |
| submissive gold organic girlfriend | 05/14/25 |
| dun tattoo | 05/14/25 |
| Insanely creepy affirmative action cuckold | 05/14/25 |
| Fiercely-loyal Stage | 05/14/25 |
| marvelous henna ratface | 05/14/25 |
Poast new message in this thread
Date: May 14th, 2025 12:34 AM Author: Fiercely-loyal Stage https://www.harvey.ai/blog/expanding-harveys-model-offerings
“In less than a year, seven models (including three non-OAI models) now outperform the originally benchmarked Harvey system on BigLaw Bench,” Harvey wrote in the blog post. Harvey’s benchmark also showed that different foundation models are better at specific legal tasks than others. For instance, it says Google’s Gemini 2.5 Pro “excels” at legal drafting but “struggles” with pre-trial tasks like writing oral arguments because the model doesn’t fully understand “complex evidentiary rules like hearsay.” OpenAI’s o3 does such pre-trial tasks well, according to Harvey’s testing, with Anthropic’s Claude 3.7 Sonnet following close behind. |
Date: May 14th, 2025 10:53 AM Author: Fiercely-loyal Stage (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2#48929625) |
|
Date: May 14th, 2025 11:09 AM Author: Violent rigor I am surprised Grok is close to 2.5 pro and o3. The steep improvement over year old models is concerning. (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2#48929687) |
 |
Date: May 14th, 2025 11:11 AM Author: Fiercely-loyal Stage What do you mean concerning (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2#48929698) |
|
Date: May 14th, 2025 11:33 AM Author: white misunderstood wagecucks church Looking into this
|
|
Date: May 14th, 2025 11:40 AM Author: Fiercely-loyal Stage (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2#48929806) |
|
Date: May 14th, 2025 12:02 PM Author: submissive gold organic girlfriend (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2#48929872) |
|
Date: May 14th, 2025 11:31 AM Author: dun tattoo (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2#48929782) |
|
Date: May 14th, 2025 12:02 PM Author: Insanely creepy affirmative action cuckold you have to use the reasoning version of grok
regular grok sucks ass |
|
Date: May 14th, 2025 4:45 PM Author: Fiercely-loyal Stage Is that what they used here (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2#48930928) |
|
Date: May 14th, 2025 12:32 PM Author: marvelous henna ratface So is gemini 2.5 pro the best at all these law tasks?
BigLaw Bench Core is a set of core tasks for benchmarking baseline legal problem-solving. Core tasks are organized into two primary categories, each encompassing several specific sub-task types:
*Transactional Task Categories*
Corporate Strategy & Advising Drafting Legal Research Due Diligence Risk Assessment & Compliance Negotiation Strategy Deal Management Transaction Structuring Regulatory & Advising
*Litigation Task Categories*
Analysis of Litigation Filings Case Management Drafting Case Law Research Transcript Analysis Document Review and Analysis Trial Preparations & Oral Argument
https://github.com/harveyai/biglaw-bench |