The most prestigious law school admissions discussion board in the world.
|
 |
 
|
AI BigLaw Bench subtask scores
| Sick Impressive Location | 05/14/25 |
| Sick Impressive Location | 05/14/25 |
| Slippery idea he suggested | 05/14/25 |
| Sick Impressive Location | 05/14/25 |
| ruddy stead | 05/14/25 |
| Sick Impressive Location | 05/14/25 |
| Apoplectic main people mexican | 05/14/25 |
| Glassy Motley Office | 05/14/25 |
| aphrodisiac kitty cat rehab | 05/14/25 |
| Sick Impressive Location | 05/14/25 |
| copper lascivious garrison legal warrant | 05/14/25 |
Poast new message in this thread
Date: May 14th, 2025 12:34 AM Author: Sick Impressive Location https://www.harvey.ai/blog/expanding-harveys-model-offerings
“In less than a year, seven models (including three non-OAI models) now outperform the originally benchmarked Harvey system on BigLaw Bench,” Harvey wrote in the blog post. Harvey’s benchmark also showed that different foundation models are better at specific legal tasks than others. For instance, it says Google’s Gemini 2.5 Pro “excels” at legal drafting but “struggles” with pre-trial tasks like writing oral arguments because the model doesn’t fully understand “complex evidentiary rules like hearsay.” OpenAI’s o3 does such pre-trial tasks well, according to Harvey’s testing, with Anthropic’s Claude 3.7 Sonnet following close behind. |
Date: May 14th, 2025 10:53 AM Author: Sick Impressive Location (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2).#48929625) |
|
Date: May 14th, 2025 11:09 AM Author: Slippery idea he suggested I am surprised Grok is close to 2.5 pro and o3. The steep improvement over year old models is concerning. (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2).#48929687) |
 |
Date: May 14th, 2025 11:11 AM Author: Sick Impressive Location What do you mean concerning (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2).#48929698) |
|
Date: May 14th, 2025 11:33 AM Author: ruddy stead Looking into this
|
|
Date: May 14th, 2025 11:40 AM Author: Sick Impressive Location (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2).#48929806) |
|
Date: May 14th, 2025 12:02 PM Author: Apoplectic main people mexican (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2).#48929872) |
|
Date: May 14th, 2025 11:31 AM Author: Glassy Motley Office (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2).#48929782) |
|
Date: May 14th, 2025 12:02 PM Author: aphrodisiac kitty cat rehab you have to use the reasoning version of grok
regular grok sucks ass |
|
Date: May 14th, 2025 4:45 PM Author: Sick Impressive Location Is that what they used here (http://www.autoadmit.com/thread.php?thread_id=5724720&forum_id=2).#48930928) |
|
Date: May 14th, 2025 12:32 PM Author: copper lascivious garrison legal warrant So is gemini 2.5 pro the best at all these law tasks?
BigLaw Bench Core is a set of core tasks for benchmarking baseline legal problem-solving. Core tasks are organized into two primary categories, each encompassing several specific sub-task types:
*Transactional Task Categories*
Corporate Strategy & Advising Drafting Legal Research Due Diligence Risk Assessment & Compliance Negotiation Strategy Deal Management Transaction Structuring Regulatory & Advising
*Litigation Task Categories*
Analysis of Litigation Filings Case Management Drafting Case Law Research Transcript Analysis Document Review and Analysis Trial Preparations & Oral Argument
https://github.com/harveyai/biglaw-bench |