IN RE: INSULIN PRICING LITIGATION No. 23-md-3080 (BRM) (RLS) MDL No. 3080 United States District Court, District of New Jersey Filed April 11, 2025 Singh, Rukhsanah L., United States Magistrate Judge ORDER REGARDING INFORMAL DISCOVERY DISPUTE AS TO TAR IMPLEMENTATION PRESENTLY before the Court is an informal discovery dispute between Plaintiffs and Defendants Express Scripts, Inc., Medco Health Solutions, Inc., and Express Scripts Administrators, LLC (collectively, “Express Scripts’’) regarding Express Scripts’s anticipated implementation of Technology Assisted Review (“TAR”) (See Doc. Nos. 440, 443, 445, 460). The Court heard argument from counsel as to the dispute during a discovery hearing held on March 11, 2025. (See Doc. Nos. 462, 469). Having fully considered the parties’ submissions and arguments raised during the March 11, 2025 hearing, the Court resolves the dispute through this Order for the following reasons. I. RELEVANT BACKGROUND AND PROCEDURAL HISTORY The parties are familiar with the background of this dispute and thus the Court does not recite it at length herein. The parties currently dispute specific aspects of how Express Scripts shall implement and validate its TAR model. Express Scripts intends to use a continuous active learning model (“CAL”), Reveal’s Brainspace CCML Predictive Coding Software, in its document review process. It has agreed to train and validate its TAR model and to disclose to Plaintiffs certain metrics from its validation process. Nevertheless, Plaintiffs contend that Express Scripts’s proposed TAR workflow is insufficient. They argue it is inadequate as to: (1) the methodology for training the model; (2) the lack of a preset criterion for stopping TAR review and beginning validation; (3) the use of elusion sampling from the null set of documents for the validation process; (4) the scope of any disclosure to Plaintiffs as to documents that may be produced from the validation samples; and (5) the lack of a preset validation recall target. As a result, Plaintiffs seek the Court to enter a TAR Protocol Order that incorporates their proposed methodology. Express Scripts opposes Plaintiffs’ request and counters that its proposed implementation methodology is reasonable and provides sufficient transparency and validation. The specifics of these disputes are discussed more fully below. II. APPLICABLE STANDARDS Typically, “[r]esponding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.” The Sedona Conference, The Sedona Principles, Third Edition: Best Practices, Recommendations & Principles for Addressing Electronic Document Production, 19 SEDONA CONF. J. 1, 52, 118 (2018) (Sedona Principle 6). Nevertheless, courts have considered disputes over TAR methodology in the context of the “reasonable inquiry” standard under Rule 26(g) of the Federal Rules of Civil Procedure and considerations of proportionality under Rule 26(b)(1). See Fed. R. Civ. P. 26(b)(1) and (g). See also, e.g., In re Diisocyanates Antitrust Litig., No. 18-1001, 2021 WL 4295729, at *6 (W.D. Pa. Aug. 23, 2021) (Report and Recommendation), adopted by 2021 WL 4295719 (Sept. 21, 2021); Nichols v. Noom, Inc., No. 20-3677, 2021 WL 948646, at *2 (S.D.N.Y. •Mar. 11, 2021) (noting “that a producing party is best situated to determine its own search and collection methods so long as they are reasonable” (footnote and citations omitted)); City of Rockford v. Mallinckrodt ARD Inc., 326 F.R.D. 489, 493 (N.D. Ill. 2018); The Sedona Conference, TAR Case Law Primer, Second Edition, 24 Sedona Conf. J. 1 (2023); Maura R. Grossman & Gordon V. Cormack, Comments on “The Implications of Rule 26(g) on the Use of Technology-Assisted Review”, 7 Fed. Cts. L. Rev. 285 (July 2014). Reasonableness, however, does not require perfection. See, e.g., in re Diisocyanates Antitrust Litig., No. 18-1001, 2023 WL 11938821, at *3 (W.D. Pa. Nov. 7, 2023). Also applicable here is Rule l’s directive to construe, administer, and employ the Federal Rules of Civil Procedure “to secure the just, speedy, and inexpensive determination of every action and proceeding.” Fed. R. Civ. P. 1. “[T]ransparency and cooperation among counsel” are important considerations when using TAR for the review and production of responsive electronically stored information (“ESI”). In re Valsartan, Losartan, and lrbesartan Prods. Liab. Litig., 337 F.R.D. 610, 622 (D.N.J. 2020) (internal quotation marks and citations omitted); accord in re Diisocyanates Antitrust Litig., 2021 WL 4295729, at *7 (“Transparency transcends cooperation.”). While the Federal Rules of Civil Procedure, this District’s Local Civil Rules, and principles of civility mandate cooperation, parties oft contest the extent of transparency—and oversight—required, particularly in the context of TAR implementation. See, e.g., in re Exactech Polyethylene Orthopedic Prods. Liab. Litig., 347 F.R.D. 572, 590 (E.D.N.Y. 2024) (recognizing “[c]ourts generally decline to intervene in a responding party’s decisions about how to use TAR, unless the requesting party shows a specific deficiency in production or unreasonableness in process”); Winfield v. City of New York, No. 15-5236, 2017 WL 5664852, at *l0·(S.D.N.Y. Nov. 27, 2017) (“Courts are split as to the degree of transparency required· by the producing party as to its predictive coding process.”); see also The Sedona Conference, TAR Case Law Primer, 2 Sedona Conf. J. at 33-47. Yet, courts have mandated some level of transparency and validation of TAR methodologies in consideration of “the complexities of TAR”, and Rule 26(g)’s obligations for a producing party to undertake a reasonable inquiry in discovery. In re Diisocyanates, 2023 WL 11938821, at *4; see also in re Uber Technologies, Inc., Passenger Sexual Assault Litig., MDL No. 3084, 2024 WL 3491760, at *8 (N.D. Cal. Mar. 15, 2024). With the evolution of technology from TAR 1.0 to TAR 2.0, which uses CAL instead of “simple active learning” (“SAL”), certain disclosures by the producing party may “offer[] false comfort to the requesting party[.]” Grossman & Cormack, Comments, 7 Fed. Cts. L. Rev. at 298; see also Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 128 (S.D.N.Y. 2015); (Supp. Decl. of M. Grossman, Doc. No. 440-1 at ¶¶ 6-8 (describing differences between TAR 1.0 and TAR 2.0)). Some courts have recognized that, while cooperation and disclosures remain important, there may be other ways to ensure a producing party has appropriately trained and implemented TAR through statistically sound validation methods. See, e.g., in re Broiler Chicken Antitrust Litig., No. 16- 8637, 2018 WL 1146371, at *4 (N.D. Ill. Jan. 3, 2018) (requiring quality-control and quality- assurance procedures to TAR review through validation protocol “to ensure a reasonable production consistent with the requirements of Federal Rule of Civil Procedure 26(g); Rio Tinto, 306 F.R.D. at 128-29; see also Grossman & Cormack, Comments, 7 Fed. Cts. L. Rev. at 301. Even with agreed-upon or court-ordered disclosures and detailed TAR protocols, changes through the discovery process and to the scope of discovery can lead to imperfect results, protracted disputes between the parties, and, in some instances, a need to re-do extensive discovery. See, e.g., in re Diisocyanates, 2023 WL 11938821, at *1 (discussing the “ongoing series of discovery challenges involving Plaintiffs’ dissatisfaction with Defendants’ search terms and technology-assisted review, or ‘TAR,’ methodologies”).[1] Perhaps for that reason, commentary has recommended that any TAR protocol “should allow sufficient flexibility for revisions to be considered if unforeseen problems arise.” 4 American Bar Association, Business and Commercial Litigation in Federal Courts, § 33:43 TAR protocols and negotiation (5th ed. November 2024 update). Accordingly, the Court considers the need for transparency, cooperation, and flexibility, a responding party’s ability to make reasonable decisions about their ESI review and production methodology, and the extent of court oversight into the intricacies of TAR implementation. The Court weighs these concepts within its sound discretion in an effort to reach a “just, speedy, and inexpensive determination” of the issues raised here as well as what may lie ahead in this complex matter. Fed. R. Civ. P. 1; see, e.g., in re Social Media Adolescent Addiction/Personal Injury Prod. Liab. Litig., No. 22-md-3047, 2024 WL 1786293, at *2 (N.D. Cal. Feb. 20, 2024) (citing cases); see also in re Actavis Holdco U.S., Inc., No. 19-3549, 2019 WL 8437021, at* l (3d Cir. Dec. 6, 2019) (noting “the District Court has wide latitude in controlling discovery’’). III. DISCUSSION Here, the parties do not dispute many aspects of a proposed TAR protocol. (See Doc. No. 460 at pp. 2-4). Where there is agreement, the parties shall proceed as such. The Court thus addresses only those aspects of a proposed protocol that are contested. Training The parties disagree on the specifics for training the TAR model. Plaintiffs propose the following: Express Scripts will use attorneys familiar with this litigation to iteratively train a TAR model to identify potentially responsive ESI. All relevance-based coding decisions will be used for training, and Express Scripts will retrain the TAR system on a regular basis. (Doc. No. 460 at p. 22; Doc. No. 440 at ECF p. 2). Express Scripts agrees in part, but counters that it will “use all or nearly all relevance-based coding decisions, in order to permit flexibility for it to use only QC’d [(quality controlled)] determinations at the tail end of the review to train the model, a proven method frequently implemented to improve model accuracy.” (Doc. No. 460 at p. 2). More specifically, Express Scripts clarifies that “it intends to use all reviewer determinations at the outset to expedite TAR model development and coverage.” (Doc. No. 460 at p. 4). It adds, however, that it seeks to retain the “option to fine-tune the model’s performance towards the end of the review by limiting training input to QC’d records only.” (Doc. No. 460 at p. 4). Thus, the parties’ dispute centers on the precise question of whether Express Scripts may choose to limit training of the TAR model “towards the end of the review” to only those records that have gone through quality control review. As proffered by Plaintiffs through the Supplemental Declaration of Maura R. Grossman, J.D., Ph.D., the TAR 2.0 model operates based on a continuous training that ranks the documents. (See Grossman Supp. Decl., Doc. No. 440-1 at ¶ 8).[2] Dr. Grossman asserts that the better method based on her research and experience is for a producing party to continuously re-train the TAR model “using all attorney relevance-based coding decisions[.]” (Grossman Supp. Decl., Doc. No. 440-1 at ¶ 9). She adds that “[i]f Express Scripts is choosing to use a more selective or limited training set, then it should describe its proposed approach in detail so that it can be evaluated to ensure that the training process is not unreasonably biased towards prioritizing narrow concepts or limited categories of documents.” (Grossman Supp. Decl., Doc. No. 440-1 at ¶ 9). Dr. Grossman further quotes one of her articles for the proposition that, after the initial seed set, ‘“cherry-picking’ of training examples is a questionable practice due to its unpredictable impact on the learning algorithm.” Grossman & Cormack, Comments, 7 Fed. Cts. L. Rev. at 299. The parties agree that using all reviewing determinations is appropriate for at least some time. Express Scripts’s proposal is to simply maintain the option “towards the end of the review” to use a more focused set of documents; it has agreed and represented to the Court that it will use all reviewing determinations for the majority of the training process for the TAR model. Flexibility in any review process, whether TAR or otherwise, particularly in proceeding with complex ESI discovery can be critical. As such, exercising the Court’s discretion and in consideration of the applicable framework, the Court directs Express Scripts to proceed as it has proposed with the following directives: (1) it must use all reviewer determinations to train the TAR model for the majority of its review; (2) if and when it decides to limit training to only those documents reviewed for quality control, Express Scripts shall provide advance notice to Plaintiffs and the parties shall meet and confer as to whether there is consent for such limitation; and (3) to the extent the parties reach an impasse following the meet and confer described herein, then the parties shall raise the dispute with the Court within five (5) business days of such impasse and Express Scripts shall not limit its training to only those documents reviewed for quality control until the Court has issued a ruling on the parties’ dispute. Stopping Criterion Plaintiffs propose setting a stopping criterion as applied in in re Uber Techs., Inc., Passenger Sexual Assault Litig., 23-md-3084.[3] In re Uber, 23-md-3084, Doc. No. 524 (N.D. Cal. May 3, 2024). Plaintiffs further propose that once the stopping criterion is met, then Express Scripts can move to validation, unless Express Scripts “chose[s] to extend the review past this point at its discretion, should it believe that additional responsive documents may be identified at a proportionate use of resources.” (Doc. No. 440 at ECF p. 10). Plaintiffs’ proposal would also include a meet and confer process if “Express Scripts believes there is good reason to cease the review prior to the presumptive stopping criteria” Plaintiffs propose. (Doc. No. 440 at ECF p. 10). Plaintiffs contend that a stopping point to turn to validation is “critical to ensure an effective and efficient TAR process.” (Doc. No. 440 (citing Grossman Supp. Decl., Doc. No. 440-1 at ¶ 12)). They further proffer that their stopping criterion would result in a marginal precision[4] of 10%, which “is objective, reasonable, and technologically sound.” (Doc. No. 440 at p. 4). Express Scripts agrees that a stopping point is “a key component of every TAR workflow” based on considerations of proportionality. (Doc. No. 460 at pp. 2-3). However, Express Scripts opposes the pre-setting of that stopping point, noting that it is unnecessary because it has agreed to make certain validation disclosures. (See Doc. No. 460 at p. 2). It contends it should determine the stopping point “at the point in time with the facts of the actual in-process workflow and its progress in hand, rather than dictated by ex ante projections or rules of thumb.” (Doc. No. 460 at p. 7). Express Scripts further adds that the decision of when to stop belongs to it and a stopping point “is not itself a validation test, but simply a predecessor choice for when validation should begin.” (Doc. No. 460 at p. 7). Express Scripts, no doubt, at some point will stop its review and then validate its TAR model. Reasonableness and proportionality factors will determine the appropriate stopping point. See, e.g., in re Diisocyanates, 2023 WL 11938821, at *5. Prior court experiences with the application of TAR models used in the in re Diisocyanates Antitrust Litigation and the in re Uber cases reflect that a reasonable and proportional stopping point is not solely quantitatively based, but also qualitative. See in re Uber, 23-md-3084, Doc. No. 2443 (N.D. Cal. Mar. 6, 2025); in re Diisocyanates Antitrust Litig., No. 18-1001, 2022 WL 17668470, at *5 (W.D. Pa. Oct. 19, 2022) (Report and Recommendation), adopted by 2023 WL 11938821. Accordingly, the Court agrees with Express Scripts; the better course is to determine the stopping point in real time as the process proceeds. Thus, the Court declines to adopt Plaintiffs’ proposed stopping point at this stage. The Validation Process The parties do not dispute that there must be a validation process for the TAR model. (See Doc. No. 440 at p. 4). However, they dispute the scope of the data set that would comprise the validation process. (See Doc. No. 460 at p. 3). Plaintiffs contend that other courts have adopted validation protocols that rely “on samples drawn from the entire universe of documents subject to TAR[.]” (Doc. No. 440 at pp. 4-5 (referencing in re Broiler Chicken and in re Uber and quoting Grossman Supp. Decl. at Doc. No. 440-1 at ¶ 16)).In contrast, Express Scripts proposes validating using an “elusion set”[5] that excludes documents that go through the quality-control review process.[6] (Doc. No. 460 at pp. 3 and 5-6). It adds that it “plans to reasonably QC and correct human error before validation.” (Doc. No. 460 at p. 5). Express Scripts states that it will validate its TAR model “by performing a validation review of 2,000 documents selected at random from documents ‘excluded from manual review as a result of the TAR-process.’” (Doc. No. 460 at p. 5 (quoting in re Boiler Chicken, 2018 WL 1146371, at *4-5)). It clarifies that it plans to apply the recall measures and elusion sampling formulas used in in re Broiler Chicken. (Doc, No. 460 at p. 5 n.3); see also in re Broiler Chicken, 2018 WL 1146371, at *6-7 (Appendix A). Express Scripts “does not agree” to include in the validation metrics those documents that proceeded to “a reviewer QC validation process.” (Doc. No. 460 at p. 5). To validate a TAR model, parties and the courts consider statistical recall, precision, and prevalence (sometimes referred to as richness). See, e.g., in re Uber, 2024 WL 3491760, at *5; in re Diisocyanates, 2021 WL 4295729, at *2. Recall in this context “‘is a measure of completeness, reflected by the proportion (i.e., percent) of responsive documents in a collection that have been found through a search or review process, out of all possible responsive documents in the collection.”‘ In re Diisocyanates, 2021 WL 4295729, at *2 (quoting a declaration of Dr. Grossman); accord The Sedona Conference, TAR Case Law Primer, 2 Sedona Conf. J. at 57 (defining recall as “a metric that represents an estimate of the percentage of responsive documents that are found out of the entire set of responsive documents in the TAR document set”). Precision measures ‘“accuracy, or the proportion (i.e., percent) of the documents identified by the search or review process.”‘ In re Diisocyanates, 2021 WL 4295729, at *2 (quoting a declaration of Dr. Grossman). Prevalence “is the estimated proportion of responsive documents in a collection at the outset.” Id. (citing a declaration of Dr. Grossman). As to the specifics of a validation protocol, the court in in re Broiler Chicken required a protocol applicable “to the review process regardless of whether [TAR] or exhaustive manual review ... was used by the producing Party.” 2018 WL 1146371, at *4; see id. at *6-7 (setting different methodologies for estimating recall for the TAR process versus manual review). In in re Uber, the Northern District of California recognized that “it is appropriate” for the defendant to demonstrate “that it has made a reasonable inquiry as to the completeness of its production” pursuant to Rule 26(g). 2024 WL 3491760, at *8. The court thus ordered the parties to meet and confer as to “reasonable and appropriate validation procedures and random sampling” of the documents to reach “an appropriate level of end-to-end recall[.]” Id.; see also Grossman & Cormack, Comments; 7 Fed. Cts. L. Rev. at 300 (“Validation, we argue, is best achieved by considering the end-to-end effectiveness of the review, and evaluating the totality of the evidence derived from multiple sources, not by considering only a single target measure applied to particular phase of the review process.”). In contrast, some courts have found sampling of the null set sufficient for validation. See, e.g., Rockford, 326 F.R.D. at 494-95 (finding “a random sample of the null set provides validation and quality assurance of the document production when performing key word searches” and thus sufficient validation for a document production); id. at 494 (“Conducting a random sample of the null set is a part of the TAR process.” (citations omitted)). The Western District of Pennsylvania has recognized that: [W]hat constitutes reasonable conduct must necessarily be measured against the available technology. Using validation based exclusively on elusion testing and recall statistics may be reasonable for parties using only search terms or TAR 1.0. But CAL gives the parties a powerful method for evaluating search at the margin, helping them decide whether further search and review would be proportional. In re Diisocyanates, 2021 WL 4295729, a *9. The court found that absent agreement, “it would be plainly unreasonable to calculate estimated recall for the TAR portion of the process alone” rather than both the search term and TAR portions of the defendant’s review process. Id.; see id. at *5. Nevertheless, the court also found that “incorporating an analysis of the accuracy of the determinations made by human reviewers” into the recall estimation formulas was an “overreach.” Id. at *12 (citing Grossman & Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, 17 Richmond J.L. and Tech. 1, 10-13 (2011)); see id. at *5. The court there noted that “courts have generally not mandated the incorporation of such reviewer error in calculating the recall statistic” when review is conducted by search terms or TAR. Id. As such, the court rejected the plaintiffs’ request to impose their TAR validation process on the defendants because reasonable “alternative TAR methodologies” existed. Id. at *13. Notwithstanding the court’s initial ruling on this issue in in re Diisocyanates, significant disputes subsequently arose regarding the defendants’ recall calculations. See, e.g., in re Diisocyanates, 2022 WL 17668470. Ultimately, the defendants calculated and disclosed an “end-to end” recall rate for the entire review. Id. at *4. With this background and the experience of other courts facing these issues, the Court must assess if Express Scripts’s proposed validation procedure to conduct elusion testing that excludes from the recall calculations quality-controlled documents is sufficiently reasonable in accordance with Rule 26(g), even if imperfect. On this record, the Court cannot find that Express Scripts’s proposal is sufficiently reasonable to validate the use of its TAR model. To be clear, the Court is reluctant to force a responding party to adopt validation metrics imposed by a requesting party; the premise that a responding party is best equipped to determine its search methodology remains true even as technology evolves. Nevertheless, Express Scripts must still satisfy the Court that its approach to TAR validation is reasonable. To do so, the case law and experts in the industry appear to recommend, if not require, a statistically sound methodology. Here, on this limited record, Express Scripts has not met that burden. Rather, the weight of the available authority raises concerns that adopting Express Scripts’s proposed methodology would result in opaque and potentially unreliable recall calculations. Accordingly, Express Scripts will validate their TAR model consistent with the validation process and recall calculation ordered in in re Broiler Chicken, 2018 WL 1146371, including a sampling from the full TAR document universe. Having adopted Plaintiffs’ proposal as to the validation methodology, the Court will not entertain future challenges by Plaintiffs to this methodology.[7] Disclosure of Validation Documents The parties have agreed that Express Scripts will produce to Plaintiffs all non-privileged responsive documents from the validation samples. (See Doc. No. 460 at p. 3). However, Express Scripts opposes Plaintiffs’ request for the disclosure of “information concerning the stratum” from which each such document was drawn. (Doc. No. 440 at ECF p. 11; see also Doc. No. 460 at pp. 3-4). Plaintiffs argue that this information is necessary to assess the “quality of the relevant documents that were not identified during validation.” (Doc. No. 440 at ECF p. 5). “[T]he adequacy of a search” is based on both quantitative and qualitative factors. In re Diisocyanates, 2022 WL 17668470, at *7. Nevertheless, the Court is not convinced that this requires the disclosure of information relating to the stratum from which a non-privileged responsive document that may be produced from the validation samples. At this stage, Express Scripts has not even begun the TAR process. It is unknown what volume of documents may be produced from the validation samples or even when, in the course of what will surely be dynamic discovery, the validation process will begin. Accordingly, provided Express Scripts provides Plaintiffs with all non-privileged responsive documents from the validation samples, the Court declines to impose Plaintiffs’ proposed required disclosure as to information relating to the stratum. Validation Recall Target Finally, the parties have a slight disagreement over the validation recall target. Plaintiffs propose a target recall rate between 70% to 80%, or higher. (Doc. No. 440 at ECF p. 11). Express Scripts states that it would agree “if this binds both Parties to the conclusion of “reasonable and proportional[,]” but, absent such mutuality, then Express Scripts states that it ‘‘will target a reasonable proportional recall rate of 70 or higher.” (Doc. No. 460 at p. 4). Express Scripts contends that because it seeks to achieve “a proportionality-informed high percentage recall target,” setting ‘‘a recall rate ex ante is unnecessary.” (Doc. No. 460 at p. 6). Indeed, recall “is only one indicator of the adequacy of a search[.]’’ In re Diisocyanates, 2022 WL 17668470, at *9; accord in re Broiler Chicken, 2018 WL 1146371, at *6 (recognizing that the recall estimate is “not the sole indicator” of an adequate review). As noted above, courts look to quantitative and qualitative features to consider if a search is adequately reasonable. See in re Diisocyanates, 2022 WL 17668470, at *7; in re Uber, 2024 WL 3491760, at *6.[8] Considering the dynamic nature of discovery, the Court finds Express Scripts’s commitment to “target a reasonable and proportional recall rate of 70% or higher” sufficiently reasonable at this stage of the process. Notably, the parties have agreed to meet and confer following the validation disclosures. In that process, the parties can discuss the adequacy of the recall measures in the context of quantitative and qualitative factors. Accordingly, the Court declines to adopt Plaintiffs’ proposal as to the validation recall target. IV. CONCLUSION Thus, for the reasons set forth above, and for good cause having been shown, IT IS on this 11th day of April hereby ORDERED that Express Scripts shall proceed with its TAR workflow in accordance with the above; and it is further ORDERED that the Clerk of the Court shall TERMINATE the informal motion pending at Docket Entry Number 440. SO ORDERED. Footnotes [1] For example, in in re Uber Technologies, Inc., Passenger Sexual Assault Litig., the Northern District of California expended significant efforts to resolve the parties’ disputes over TAR methodology, agreeing in many instances with the requesting party’s proposals. In re Uber, 2024 WL 3491760, at *3-8. One year later, the parties disputed aspects of the validation review of the TAR model, noting multiple discrepancies, which ultimately resulted in the court ordering the defendant to retrain the model unless the parties could agree on an alternative method to produce documents subject to the dispute. In re Uber, 23-md-3084, Doc. No. 2443 (N.D. Cal. Mar. 6, 2025). In so ordering, the Northern District of California noted: “A TAR model—or, for that matter, any other method of large-scale review—will likely never produce perfect results, but a model trained on an incorrect scope of production is inherently not a reliable method of identifying responsive documents.” Id. at p. 6. [2] Due to a technical electronic filing error, the Supplemental Declaration of Dr. Grossman filed on the Court’s docket at Docket Entry Number 440-1 appears unsinged. Plaintiffs’ counsel, however, have provided the Court with a courtesy copy of the document that is fully executed. [3] Specifically, the stopping criteria in in re Uber included: “Once two or more consecutive review batches sequentially populated by the highest-ranking uncoded documents remaining in the project in order from highest to lowest scores and containing a total of at least 1,000 documents are found to contain 10% or fewer documents marked responsive, Defendants will pause the review and turn to validation. Defendants may extend the review past this point if they believe sufficient thoroughness has not been achieved.” In re Uber, 23-md-3084, Doc. No. 524 at § 8.a.3.i, at p. 8. The in re Uber Court adopted this language as proposed by the plaintiffs and over a competing proposal from the defendants that would set a stopping point “[o]nce two reasonably sized review batches are found to contain 10% or fewer documents marked responsive[.]” In re Uber, 2024 WL 3491760, at *4-5. In adopting the plaintiffs’ proposal, the court found it provided “more definite guidance for the validation process, while Uber’s approach is more vague and therefore more likely to lead to disputes.” Id. at *5. Notably, it appears that the parties did not dispute that a pre-set stopping point was appropriate or necessary. [4] Dr. Grossmann describes “marginal precision” as the point where “the number of top-ranked documents containing responsive information drops precipitously, typically to 10% or less.’’ (Grossman Supp. Decl., Doc. No. 440-1 at ¶ 10). According to Dr. Grossman, this point “indicates that [the TAR] review is reasonably complete.” (Doc. No. 401 at ¶13; see also Grossman & Cormack, Multi-Faceted Recall of Continuous Active Learning for Technology-Assisted Review, at Doc. No. 401 at ECF pp. 142-44). [5] In this context, elusion is “[t]he percentage of documents of a search’s null set that were missed by the search, usually determined with review of a random sample of the null set. The elusion rate can be multiplied by the number of documents in the null set to estimate how many documents were missed by the search.” The Sedona Conference, The Sedona Conference Glossary: Ediscovery & Digital Information Management, Fifth Edition, 21 Sedona Conf. J. 263, 304 (2020). The “null set” is “[al set of files that are not positive results of a search.” 1d. at 344. Further, “Null Set Sampling” is the “[s]ampling [of] a null set to search for false negatives of the search that created the null set.” Id. [6] The parties dispute whether Express Scripts’s proposal is consistent with the validation process ordered in in re Broiler Chicken, 2018 WL 1146371. [7] During the March 11, 2025 hearing, counsel for Plaintiffs represented that with the application of their proposed process to evaluate the recall, they “can say done, well done.” (Doc. No. 469 at 71:25-72:3). [8] The in re Uber Court modified the plaintiffs’ proposed language to the ESI protocol regarding TAR validation to include the following language: If the validation protocol leads to an estimate lower than 80%, or even lower than 70%, this lower recall estimate does not necessarily indicate that a review is inadequate. Nor does a recall in the range of 70% to 80% necessarily indicate that a review is adequate; the final determination of the quality of the review will depend on the quantity and nature of the documents that were missed by the review process. 2024 WL at 3491760, at *6. The wisdom of this modification has become evident, as the parties subsequently disputed the quality of the TAR process because certain categories of documents were missed. See In re Uber, 23-md-3084, Doc. No. 2443 (N.D. Cal. Mar. 6, 2025).