AI Case Study
CaseCrunch's AI program demonstrates greater accuracy predicting the outcomes of historic PPI cases than human lawyers
CaseCrunch tested its AI model against human lawyers to predict the outcome of 800 historic PPI misselling cases, with the model accurately predicting the outcome 87% of the time compared to the lawyers' 62%.
According to the Case Crunch website: "Over 100 commercial London lawyers signed up for the competition and made over 750 predictions over the course of a week in an unsupervised environment. The problems were real complaints about PPI mis-selling decided and published by the Financial Ombudsman Service under the FOIA." "Both the humans and the AI were given the basic facts of hundreds of PPI (payment protection insurance) mis-selling cases and asked to predict whether the Financial Ombudsman would allow a claim."
The BBC reports that according to one of CaseCrunch's founders: "the question at this early stage of AI development is whether it will "remain limited to descriptive analysis or whether it will be capable of evaluating rules and events", and then whether it will be a tool for junior lawyers to use or something which replaces them."
However, ABA Journal offers a critique of the way in which the program was tested. "Ralph Cox, a patent litigation partner at Clyde & Co. in London and an attendee of the event, as saying: 'I keep an open mind. However, it struck me that the computer was given all the database information and therefore had an unfair advantage. Lawyers who took part were from outside this area of expertise without any real experience of [Payment Protection Insurance]'."
The Case Crunch website claims that "CaseCruncher Alpha scored an accuracy of 86.6%. The lawyers scored an accuracy of 62.3%" but cautions that "results do not mean that machines are generally better at predicting outcomes than human lawyers. These results show that if the question is defined precisely, machines are able to compete with and sometimes outperform human lawyers."
ABA Journal reports that "Timothy Lessen, a partner at Lewis Silkin in London, told Legal IT Insider: 'CaseCrunch met my expectations. But then I think I was pre-programmed to expect this result.' He added, 'I’m not completely clear what [CaseCrunch] will look like yet: an end-to-end product that means 5 percent of the workforce of private practice law firms is no longer required, or a product which allows us to deliver a much better service to clients.' "
From the Case Crunch website: "CaseCruncher is a system that makes legal decision predictions. CaseCrunchers are currently available for banks, insurance companies, law firms and litigation funders. Our systems help clients reduce the amount of time and money spent on legal prediction tasks by multiple orders of magnitude, while matching or improving human performance. Current applications include complaint handling, legal decision forecasting and merits-based claim review."
Database of historic PPI misselling cases and their decisions