The most prestigious law school admissions discussion board in the world.
|
 |
 
|
Crazy that the AI Legal Agent benchmarks are amongst the lowest
Poast new message in this thread
Date: July 28th, 2026 1:59 PM Author: gunnerretttired (gunneratttt) STEMmos have always looked down on us, but it turns out automating that stuff is way easier than putting together A DEAL! (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50029944) |
Date: July 28th, 2026 3:28 PM Author: gunnerretttired (gunneratttt) (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030208) |
|
Date: July 28th, 2026 3:32 PM Author: femgooner meta scientist My irl lawyer friends don't believe me when I tell them that it's only a matter of time before enough money is thrown at RL environments to make LLMs almost as good at law as they are at programming
Right now they are already better than most attorneys (most attorneys suck) |
 |
Date: July 28th, 2026 3:41 PM Author: gunnerretttired (gunneratttt) I agree it's better than many attorneys already and that someday it may get to that point you're saying. I'm just saying it's a bit suprising how much of a gap there is between legal performance benchmarks and other things.
https://x.com/i/status/2064394151441863006 |
|
Date: July 28th, 2026 3:48 PM Author: femgooner meta scientist It's because they haven't thrown resources at RL for law specifically. The models are extremely good at programming because there has been an enormous amount of resources sunk into training them for that specifically. It's true that programming is easier (less resource intensive) than law to RL train for, but it still required a lot of effort to get models this good at programming
Eventually the same thing will happen for law. The incentives are there because lawyers are outrageously overpaid for the actual level of expertise and training that they have. They're able to command the excessive compensation they get only because they're a monopoly/cartel
AI displacing lawyers is going to be one of the most positive outcomes from AI because lawyers have such disproportionate power and influence in our society and their interests conflict with those of most people. They are an extremely anti-social force in modern society and responsible for a lot of our society's ills |
|
Date: July 28th, 2026 4:00 PM Author: gunnerretttired (gunneratttt) But that monopoly exists because you cannot provide legal services without a license. The power and influence you're talking about won't change just because the work gets easier. According to you the work is already over overpaid relative to the effort required. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030280) |
|
Date: July 28th, 2026 4:20 PM Author: femgooner meta scientist Law's power in the US comes from it being an unassailable religion first, and a cartel second
AI will do to law in the public consciousness what "covid" and AI have done to medicine in the public consciousness. Exposed industry professionals as largely fraudulent rent-seekers rather than the Experts and Priests they supposedly are
It will happen a lot faster than people think. It's a question of technical capability. Once LLMs can straight up beat almost all human lawyers, the flood of pro se pressure on the legal system will cause huge societal changes. People will not respect The Law anymore. It will become just another scam industry in the public mind |
|
Date: July 28th, 2026 4:35 PM Author: gunnerretttired (gunneratttt) Everyone already knows lawyering is a bullshit quasi-religion. If AI starts crafting deal docs better than Wachtell that's not going to wake anyone up to that.
There's really no way to "beat" lawyers work by some objective metric other than if it has actual errors. What's "better" and who "wins" is decided by people. It's going to be a long ass time before the legal system even permits AI.
The people that decide whether the legal cartel exist are the legal cartel. Judges (all cartel) and lawmakers (mostly cartel) decide, and they're going to be *most* interested it kn preserving it. And I don't see some popular backlash coming on such a niche issue. People will be more concerned about their jobs being eliminated than dragging lawyers into the same bucket. Fuck, PACER and other court records are still often paywalled when that should have been free 20 years ago. |
|
Date: July 28th, 2026 5:14 PM Author: femgooner meta scientist Unfortunately, the general public absolutely does not think that law is bullshit. It's the only "institution" that is still widely respected across demographics. This will change
The legal system already permits AI. It's being used right now across the country. Everyone is using it
It's only a question of capability, which is only a question of time and resources invested. It's going to happen. The incentives are too powerful
|
|
Date: July 28th, 2026 5:22 PM Author: gunnerretttired (gunneratttt) I don't see that being a high priority or more apparent than now when AI gets to the point you're envisioning. It will create so many other issues that toppling the legal priesthood will be a lower priority than it already is.
I agree with you about it being a rent-seeking priesthood and that it would be good for society if that was dimished. I just don't see it happening before 1000x other things that will be more pressing. It's just too niche for people to even care about, let alone take an interest it correcting. And I don't think the market will take care of it, biglaw doesn't command their fees because of higher competence, plenty of firms could do the work the most elite firms do just as well.
Anyway, we'll see. Hope you're right and I'm wrong. |
|
Date: July 28th, 2026 4:02 PM Author: ,.,.,.,.,.,.,.,.,.,..,.,.,...,,.,.,.,.,.,.,..,.,, You seem retarded. What makes you think the scores are normalized axroas fields idiot? (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030285) |
|
Date: July 28th, 2026 4:11 PM Author: gunnerretttired (gunneratttt) Well it can't be normalized because we're talking about different tasks. But it's hard for me to think of a legal task that's so much "harder" than those on Humanity's Last Exam that the difference in correct answers is attributable to something like the legal questions being "harder."
I think the correct answer is that there's much less data to train AIs on for legal work because the nature of the work means much of it is confidential. |
|
Date: July 28th, 2026 4:14 PM Author: ,.,.,.,.,.,.,.,.,.,..,.,.,...,,.,.,.,.,.,.,..,.,, Or it could be that harder than average problems are a bigger part of the legal benchmark's question mix than of the other benchmark. You have no idea. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030308) |
|
Date: July 28th, 2026 4:17 PM Author: gunnerretttired (gunneratttt) Humanity's Last Exam is designed to extremely difficult. It's hard for me to even conceive of how a legal task could be harder than questions on that. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030313) |
|
Date: July 28th, 2026 4:21 PM Author: femgooner meta scientist They are trained to beat humanity's last exam. They aren't trained to be lawyers
It really is that simple. The "benchmarks" are heavily gamed because there's enormous incentive to do so |
|
Date: July 28th, 2026 4:25 PM Author: gunnerretttired (gunneratttt) Ok. But you're just explaining why the models perform worse or legal tasks than similar ones. But they still do perform worse currently, which was my point. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030329) |
|
Date: July 28th, 2026 4:16 PM Author: lex I, for one, am optimistic for the future of this majestic profession. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030311) |
|
Date: July 28th, 2026 4:23 PM Author: law is ea$y (🧐) I Philosophize, therefore I Poast (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2],#50030324) |