The Emperor’s New Clothes

A couple of years ago, I dropped out of document review because I did not want to be ruled by a naked emperor. Predictive coding was all the rage, despite the obvious problem that if I don’t know exactly how you trained your software, or have an advanced degree in statistics, I could not be certain you had turned over a valid production. My only consolation was that you could not be certain I had, either. I had to trust that your trainer(s) would push the bounds of relevancy in order to help me make my case just as you had to trust me to do the same. Didn’t seem prudent.

Thus I wasn’t surprised when I recently dipped a toe back in the pool to see no sign of predictive coding being used on the project I became involved in. No matter how bamboozled by sales pitches and statistics one may have become, taking it to the bank is another matter entirely for most attorneys.

Makes me wonder if the eDiscovery world is ready for TSM (Targeted Search Methodologies). Drop me a line at richardneidinger@yahoo if you’re interested.

Posted in Uncategorized | Leave a comment

Did Predictive Coding’s Black Box Just Get Darker?

With predictive coding, you are obliged to accept documents produced by your opponent selected by an algorithm you do not understand that has been “trained” by your opponent.

To make that even more interesting, J. Scranton, J.D., an e-discovery and litigation support specialist out of Raleigh-Durham, NC, has just posed an interesting question on LinkedIn concerning the impact of “rolling collections” to the validity of the training. In other words, your opponent has trained his or her algorithm using seed documents from the current custodians, but is that training still valid if you add other custodians down the line, or does the training have to be redone?

To his credit, Rachi Messing, Vice President of Customer Solutions at Equivio, provides a candid answer, which I will take the liberty of quoting:

One way to think about this is through an example where the initial collection was from the Finance department within a company. The system was trained on whatever issue is at hand and we have seen good results. If the second collection is from the Marketing department is it safe to assume that whatever language or features were used to train the Finance data can automatically be used against the Marketing data? There very well may be similarities in the way the documents from both departments discuss the issue at hand but then again it is possible that they each have their own unique jargon which would influence whatever PC algorithm is being utilized.

Just another thing to think about before you sign off at the meet and confer.

Posted in Notes From The Field, Predictive coding | Leave a comment

No Room for Error is reporting that according to Sen. Lindsey Graham, R-S.C., the FBI told him that the name of the Boston Marathon bombing suspect Tamerlan Tsarnaev was misspelled in the federal computer system, contributing to their apparent failure to discover his six-month trip to Russia in 2011, which is believed to have contributed to his radicalization.

Whether this is true and accurate or not, it certainly highlights a potential problem with information management systems; data entered incorrectly is not data – it is junk.  Spell-checking is essential for data entry and someone needs to periodically comb the data mass for misspellings in documents scanned into the system.

Two things to ask your document review platform vendor are: 1) do you offer a spell-check for the reviewer’s notes, and 2) how hard would it be to check the documents loaded into the system for Smiths in addition to Smyths (or Tsarnavs in addition to Tsarnaevs)?

Posted in Uncategorized | Leave a comment

Full Circle: Plausible Deniability and Electronic Discovery

Back in the old days (the 1980s), we had plausible deniability in spades when it came to pre-trial discovery.  Most business was conducted by telephone, so there was almost never a verbatim record of what one party said to another when if a legal dispute arose.  There might be telephone bills available to show if calls were made and how long they lasted (but only if they were “long distance” – outside the area code).  And there might be telephone memos typed up after the fact that reflected the call as heard by a potentially interested party that could easily be falsified.  And it was easy for a witness to simply “not remember” certain conversations.

Proving a case then required a lot of skillful questioning at deposition along with careful witness statement work-ups to corroborate or rebut less-than-perfect memories.

Then, like a miracle, emails supplanted phone conversations almost entirely, leaving word-for-word proof of what was discussed, by whom, and when.

Patient and methodical attorneys could find these emails, string them together, and make pretty solid cases out of them.  The invention of predictive coding, with their proprietary algorithms, promised to require less patience, less effort, and greater accuracy.  Trials were beginning to seem like they might become unnecessary inconveniences.

But technology fooled us.  The near perfect fossil record left by emails is giving way to text messages, and with this returns plausible deniability.  Turns out the phone companies don’t keep the text of the 10 zillion texts sent each day; just things like the time sent and receiver’s phone number.  While the texts might be stored on the phone, or uploaded to a server, the requirement that this be done doesn’t kick in until there’s reason to foresee litigation.  And then, it seems, the clipped syntax and context-dependent meaning of these messages are not friendly to the predictive coding algorithms.  Even if the messages themselves are recovered, who knows what those garbled phrases meant weeks, months, or years after they were sent?
Yikes, we’re back to the ’80s, complete with big hair bands, shoulder pads, and plausible deniability at worst, or methodical patience at best.

Posted in Notes From The Field, Predictive coding, Targeted Searching | Leave a comment

Predicting the Future of Predictive Coding

The realization that trial counsel cannot really accept a production of documents from the other side generated by a process that he or she cannot hope to fully understand without a sophisticated knowledge of statistics and computer programming is beginning to take root. But this does not mean that predictive coding has no place in modern discovery by any means.

Although I do counsel against accepting such a production because of its black box nature to many attorneys, I would not necessarily counsel against using the exact same black box software to take a gander at what the other side produced through methods I am more comfortable with, such as keyword searches, provided you are the one training the black box.

The argument that keyword searches can produce too many false positives due to the prevalence of homonyms as well as too many false negatives because of the prevalence of synonyms, misspellings, etc., is certainly a valid one, but it does not persuade me, as requester, that you are justified in using a process I don’t understand in an effort to avoid this as producer.  One, I have to look through the same number of false positives that you do, and two, while it is nice of you to worry about me not getting what I’m entitled to (the false negatives), the answer to that is NOT to let you teach a black box what I should or should not get. The much-ballyhooed fact that the training would be done by senior attorneys more familiar with the case moves me not at all; they will not be inclined to make my case for me and are unlikely to go the extra mile my duty to my client requires me to.

On the other hand, if you want to use the black box – as taught by you and your team – to take a closer look at what your keyword search has come up with for production, I have no objection, and I might even be tempted to use the exact same black box – but as taught by me and my team – to look at whatever it is that you gave me pursuant to the keyword search. The difference lies with who is doing the training.  If predictive coding helps me find the documents that will make or break the case without the time and expense of manual review, great, but that all has to happen on my side of the aisle.

I was always surprised that the powers to be never seemed to employ document review teams to comb through what they got, only what they were giving.  It would seem to me that predictive coding could be quite useful to look at what you’ve been given, provided you are telling it what to look for.

Posted in Predictive coding | Leave a comment

Are Recall and Precision Valid Metrics?

Last week was the Seventh Circuit Electronic Discovery Committee Workshop on Computer-Assisted Review, held in Chicago, and if you missed it, you missed a lot.  But like any event in which a lot of information is thrown at you, it answered a lot of questions, but raised others.

For example, I believe I heard one of the participants compare human reviewers to predictive coding by remarking that the numbers with respect to recall weren’t that far apart, but that the “computers knocked it out of the park” with respect to precision.

Recall is the percentage of relevant documents actually retrieved by a search methodology.  Say there are 1000 relevant documents in a data set of 5000 documents and predictive coding finds 850 of them; it has a recall of 85%.  According to the speaker, this is not that much better than what can be expected of a human reviewer.

Precision is the percentage of the documents retrieved that are relevant.  Let’s say the found set returned by the predictive coding software referred to in the paragraph above returns the 850 relevant documents, but also includes 500 documents that are found to be not relevant on closer inspection.  Its precision would be 850 divided by the total number of documents returned, 1350, or about 63%.  According to the speaker, human reviewers are likely to return significantly more irrelevant documents for each relevant one.

But I’m not sure of the practical significance of this.  “Precision” has such a nice ring to it, but what does it mean?  I’m not sure I care what percentage of generally relevant documents a system returns to me; what I want to know is what percentage of the key documents – the handful likely to become case-determinative exhibits – are being returned to me.

Never bring a knife to a gun fight, I know.  And when it comes to a gun fight over statistics, I can barely bring a sharpened pencil, much less a knife.  But having said that, let’s take another look at the data set of 5000 documents.  By definition, it contains 1000 relevant documents and predictive coding was able to find 850 of them.  If I heard the speaker right, human reviewers are likely to find slightly fewer, maybe only 800.  We do not, however, know the overlap.

Think of the 1000 relevant documents schematically as a line of numbered documents stretching from 1 to 1000.  From which parts of the line are the 850 found by predictive coding?  And from which parts of the line are the 800 found by the humans?  If predictive coding finds documents No. 1 through 850 and the humans find 1 through 800, but the handful of key documents are between No. 930 and 970; neither system is going to find them.  If predictive coding finds documents 1 through 850 and the humans find 200 through 1000, then the humans are going to find all of the key documents and predictive coding will find none of them.  Likewise, if the key documents are clustered around  No. 150, the situation reverses – predictive coding finds all of them, humans none of them.  Those are the simplest situations.  Of course the handful of key documents can be scattered irregularly throughout 1 to 1000 and the different methodologies don’t have to have contiguous returns; they could return Nos. 1 through 37, 42 through 123, etc., with the question being whether or not the key docs are hiding in the gaps.

The need to consider the quality of the documents missed by either system – the “keyness” of overlooked documents – was mentioned at the workshop, but it is not clear how this might be done.  One of the names told me afterwards that predictive coding looks for words and patterns of words and is happiest with at least six words to work with, meaning that “cryptics” are at risk of being overlooked.  I haven’t any studies or even anecdotal evidence, but it seems to me that key docs may well be cryptic because people seldom announce they are going to do something unethical, illegal or wrong with full and proper syntax.  It also seems to me that a human reviewer is perhaps more likely, using his or her humaness and experience, to deem a cryptic relevant because of surrounding documents that may give it more context, or the memory of other documents that might shed light on its relevance in a manner a computer cannot.  So I’m not sure if humans or machines win and thus still have to wonder why I would go with predictive coding at this point.

I would very much like to see this point addressed, shot down, disposed of, etc., by someone with the cred to do so.

Richard Neidinger, J.D.

Posted in Predictive coding, Uncategorized | Tagged , , , , , , | 1 Comment

An Indecent Proposal

As I sit on the sidelines watching the commotion over predictive coding continue in Da Silva Moore, and continue reading obtuse statistical arguments for or against it, it repeatedly occurs to me that the very nature of discovery has to change, or it risks becoming something of an expensive charade.  A good friend has cautioned me that unless I can propose a concrete solution, I am simply throwing firebombs, but I’d rather do that then see the argument go down a false path, even though I cannot yet offer a solution.

Years ago, an older attorney told me that although he had to follow ethical standards in responding to discovery, he certainly did not have to make the other side’s case.  By this he meant that although he may have to produce damaging documents, and would produce them, he did not have to tie them up in a bow to be served on a silver platter to his opponent.  I’m wondering if that is still as true as it once was.

Thanks to targeted search methodologies (TSM) using metadata, date ranges, text searches, etc., technology now lets us recreate what may have given rise to the litigation with almost time-machine precision.  To use an overly simplistic example, suppose the relative strengths of a case hinge upon an email and the response thereto.  With TSM, the recipient’s reaction to the email and the steps taken in formulating a response can be determined with great precision.  We can find:

  • who the email was forwarded to
  • any separate emails it may have engendered
  • any text messages it may have engendered
  • any text documents or spreadsheets it may have engendered
  • any conference calls during which it may have been discussed
  • any meetings at which it may have been discussed, and
  • the drafts of the response.

Based on my understanding of predictive coding’s black box, I am skeptical that predictive coding would return all of this and I am quite certain keyword searching would not.

But – and it’s a huge one – the only way TSM would be able to do this is if you are willing to make your opponent’s case for him or her (by looking under all the rocks TSM requires you to), or are willing to let your opponent rummage around in your data set, subject to protective orders, etc.

I have said this before: modern technology now lets us establish what happened in a case like never before, but requires a re-thinking of discovery in order to do so.  Predictive coding, with its focus on what percentage of relevant junk it may or may not return, might well be nothing more than a red herring in the way of realizing this.

All of the extremely relevant discovery IS there, we just need to rethink if we are willing to allow a greater intrusion than before, knowing that not doing so clearly, definitely, and unarguably will allow demonstrably relevant information to remain unproduced while we argue about how many angels can dance on the head of a pin.

Richard Neidinger, J.D.

Posted in Predictive coding, Targeted Searching | 3 Comments