Last year, the High Court decided a case called "Mazur" (the full reference is available lower down this post, if you're interested). The appeal from the decision was recently heard over three days in the Court of Appeal. I've asked some AIs to predict the outcome.
It seemed to me that this case is a good test for a couple of reasons. The first instance decision attracted a lot of attention in the legal professions because it concerned the question of whether certain legal tasks have to be carried out by fully-qualified solicitors and the extent to which other professionals, with other (or no) qualifications, can assist. So it's a decision that potentially has a big impact on the business of law firms and has therefore attracted a lot of informed comment for the AIs to consider. On the other hand, the case is of much less interest to the general public, so there is little by way of general media comment. All of which made me think that an AI that 'reads' the public information on the case will get a fairly good steer on what it is all about.
The test I gave the AIs was not to replicate what a lawyer would do when giving advice to their own client. Instead, I was asking them to do something akin to what a law firm does when it publishes a note on a case of wider significance in which it is not itself involved: the kind of thing intended to keep clients updated and show that the law firm is a 'thought leader'.
I've cut and pasted the AIs' predictions below, in advance of the judgment, so that we can all look back after the event and judge them more fully. The headlines are as follows:
- Claude/Sonnet was reluctant to answer the question, saying that the appeal fell after its cut-off date, and I was disinclined to spend any time bringing it up to speed. I have not quoted it below.
- LeChat/Mistral made some howlers that discredited it in my eyes.
- Grok was perhaps most helpful. It immediately gave a convincing answer.
- ChatGPT took some prodding to produce a full answer, but gave a decent one in the end.
But even in advance of seeing if they are right, we can ask how they did on the test I gave them. How does their work product compare to asking a junior lawyer to write a report of this kind?
On balance, as at the time of writing (and I know things change quickly), I'd still rather have the lawyer for this job. But a lawyer at £X per hour working for a few hours, or a few seconds on Grok? That's much closer. One lawyer checking for howlers plus a decent AI doing the legwork would be far more efficient. As one on-line commentator said, AIs are very good at curing 'blank page' panic.
Given the relative lack of importance of notes such as these, I'd be very surprised if AI is not generating the bulk of them before the end of the year. At which point, AIs will most likely be the bulk of the readership too...
Full details below the break.
Full details below the break.
