MSC. SOFTWARE CORPORATION, Plaintiff(s), v. ALTAIR ENGINEERING, INC., MARC KLINGER, ANDREA PERTOSA, STEPHAN KOERNER, TOM RIEDEMAN, RAJIV RAMPALLI MARK KRUEGER, and MICHAEL HOFFMAN, Defendant(s) Case Number: 07-12807 United States District Court, E.D. Michigan, Southern Division February 19, 2009 Counsel James F. Hermon, Erin M. Rozycki, Janis L. Adams, Patrick F. Hickey, Dykema Gossett, Detroit, MI, for Plaintiff. Catherine T. Dobrowitsky, Larry J. Saylor, Marcy L. Rosen, Michael C. Fayz, Miller, Canfield, Detroit, MI, Matthew P. Allen, Miller, Canfield, Troy, MI, C. Thomas Ludden, Phillip E. Seltzer, Lipson, Neilson, Bloomfield Hills, MI, for Defendants. Roberts, Victoria A., United States District Judge ORDER ADOPTING SPECIAL MASTER LEE A. HOLLAAR'S REPORT AND RECOMMENDATION *1 On November 13, 2008, the Court entered an “Order Adopting in Part Special Master Hollaar's Report and Recommendation.” (Doc. # 229). The Order required Altair to “arrange with the Special master [a] visit [to] the site or sites having the MotionSolve[/MotionAuto] source code repository and development twiki to confirm that the copies produced to MSC represent all the information as they are kept in the usual course of business[.]” On December 20, 2008, Special Master Hollaar informed Altair and MSC via e-mail that Altair's site inspection will occur on January 23, 2009. The e-mail said: Since a primary reason for this site visit is to restore confidence in what was produced by Altair in discovery with respect to the source code repository and development twiki, I would be interesting[sic] in hearing from MSC what concerns it and its experts might have. Please get me that information by January 16, 2009. MSC sent Special Master Hollaar three e-mails on January 16, 2009. The e-mails expressed MSC's concerns that: (1) the “yabb bulletin board” was not produced; (2) Altair has development notes in a separate file from the twiki database; (3) MotionSolve 7.0 was not imported into Perforce in its entirety; and (4) the “bug fixes” that were provided to Dr. McGrath, Mr. Stackpole, and Mr. Zeidman were produced so that importation of those files is a laborious process. If the updating process cannot be automated, MSC requested the production of a new, updated Perforce database with every “bug fix.” In an Order dated January 23, 2009, the Court construed MSC's four concerns as a discovery problem addressed to Special Master Hollaar. The Court gave Altair until January 26, 2009 to provide Special Master Hollaar a response. If MSC's concerns were not resolved, Special Master Hollaar was required to make a recommendation to the Court regarding whether Altair should be required to produce the information MSC requested. On February 2, 2009, Special Master Hollaar submitted a “Report and Recommendation Regarding MSC's Discovery Requests of January 21, 2009.” (Doc. # 269). Pursuant to the Court's Orders dated November 13, 2008 and January 23, 2009, the parties had 10 days to object to Special Master Hollaar's recommendations. See Docs. # 230 and # 267. Because the Court has not received objections from either party within the time frame provided, the Court adopts Special Master Hollaar's recommendations: (1) MSC's concern that Altair did not produce the “yabb bulletin board” is MOOT. Altair previously provided MSC the information it used to produce the “yabb bulletin board”; (2) While MSC's discovery cut-off date was February 13, 2009, it may depose an Altair developer regarding whether there are development notes outside of the MotionSolve source code repository or development twiki (if MSC did not do so before its discovery cut-off date). In the alternative, if no development notes exist outside of the source code repository or development twiki, an Altair developer may provide MSC a sworn affidavit attesting to that fact; *2 (3) If it has not already done so, Altair must provide copies of the MotionSolve 7.0 source code repository data to Special Master Hollaar, MSC's experts, and MSC's outside counsel on or before February 24, 2009; (4) Altair need not produce a new version of the repository for MSC's experts and outside counsel before it releases a test or beta version of MotionSolve. IT IS ORDERED. MSC Software Corp., Plaintiff, vs. Altair Engineering, Inc., et al., Defendants. Report and Recommendations of Special Master Lee A. Hollaar Regarding MSC's Discovery Requests of January 21, 2009 LEE A. HOLLAAR, Special Master. Background The Court's November 13, 2008 “Order Adopting in Part Special Master Hollaar's Report and Recommendation” adopted my recommendation regarding a site visit to Altair to examine the source code repository and development twiki for MotionSolve/MotionAuto to confirm that the copies produced to MSC represent all the pertinent information as it is kept in the usual course of business. After consultation by email with the parties, that site visit was scheduled for January 23, 2009 to be conducted at Altair's California facility. In an email in preparation for the site visit, I wrote: Since a primary reason for this site visit is to restore confidence in what was produced by Altair in discovery with respect to the source code repository and development twiki, I would be interesting in hearing from MSC what concerns it and its experts might have. MSC responded with four concerns: • That in addition to the development twiki, there was a “yabb bulletin board” that had not been produced. • That the Altair source code had an “unexpectedly low number of comments embedded within it.” • That there was a separate repository for MotionSolve 7.0 that had not been produced, and wondering whether all of that repository had been imported into the current Perforce respository. • That it was cumbersome for MSC's experts to add to their copies of the Perforce repository the files that are part of the material I have reviewed before Altair released new versions of MotionSolve, and requesting that MSC receive a new copy of the repository every time I am asked to review material to be released by Altair. In follow-up email, I asked MSC for information clarifying these requests. Altair, in a January 20, 2009 letter to the Court and me, indicated it felt that MSC's concerns went far beyond the purpose of the site visit, and should more properly be addressed following the procedures that the Court adopted in its November 13, 2008 “Order Clarifying Special Master Hollaar's Authority and Establishing Procedures to Follow in the Future.” The Court, in its January 23, 2009 Order, indicated that the site visit should go forward, that the concerns expressed by MSC should be treated as a discovery problem to be addressed under the procedures of the November 13, 2008 Order, and that Altair would have until January 26, 2009 to provide a response, after which I should submit my recommendations to the Court regarding the concerns that have not been resolved. The “yabb bulletin board” *3 The question of the “yabb bulletin board” seems to be a new discovery request, coming from a reference to it during the December 19 2008 deposition of Altair employee Sakthivel Subramaniam. Altair, in a January 23, 2009 letter, indicated that the information used to produce the “yabb bulletin board” had already been produced to MSC, and indicated the Bates numbers for that information. Altair also indicated in its January 26, 2009 response that it will be verifying “that every post to the yabb board was previously produced at least once.” It appears that this MSC concern has been resolved. Source code comments In its email to me of January 16, 2009 MSC wrote: Our experts have noticed that the C++ source code comprising MotionSolve had an unexpectedly low number of comments embedded within it. That may be a function of the coding habits of the developers, but they would like you to verify that the developers are not keeping development notes in a separate file apart from the twiki database, rather than embedded within the code itself. This is a symptom of the distrust that has developed over the production of the source code repository. When MSC's experts don't find something in one place, there seems to be an assumption that it has not been produced, rather than looking in other places. For example, the “dictionary” that MSC's experts felt should appear in source code comments does exist in a different, definitional file. A similar situation occurred previously when MSC indicated that a number of source files were not in the repository, not realizing that there were three different repositories produced by Altair. During the site visit, I compared the current version in the active repository of the files MSC indicated as example to a version that substantially preceded the filing of this suit. There was no indication that comments had been removed from those files. I have seen nothing to indicate that Altair has not produced a complete and accurate copy of the source code repository for MotionSolve, and MSC and its experts should have confidence in what has been produced. As to whether there are development notes outside of the MotionSolve source code repository or development twiki, that appears to be something that MSC could ask Altair in a Rule 30(b) (6) deposition or during the deposition of Altair developers, rather than have the Court through its special master try to investigate this. The MotionSolve 7.0 repository In another January 16, 2009 email, MSC wrote: Mr. Subramanian testified that various aspects of the source code were “carried over” from MotionSolve 7.0 to the subsequent versions of the product. We do not have a separate repository for MotionSolve 7.0, which was maintained in CVS. It appears, however, that portions of MotionSolve 7.0 were imported into Perforce when that new source code control system was rolled out. There are references to code that was authored in 2003 or 2004 (or even earlier), before Perforce went online. We would like to know whether all of MotionSolve 7.0 was imported into Perforce, or whether only portions were imported. If only portions, which ones? *4 Again, this seems to be another new discovery request, also coming from a reference to it during the December 19 2008 deposition of Altair employee Sakthivel Subramaniam. In its January 26, 2009 response, Altair indicated that it “will provide copies of the MotionSolve 7.0 source code repository data to the Special Master and to MSC experts and counsel as promptly as that material can be assembled.” If Altair does that, then MSC's concern will be moot because it will be able to answer the questions that were posed to me in the email above. Repeated production of the repository In yet another January 16, 2009 email, MSC wrote: Another of our experts had another comment. I understand that the supplemental files provided to Dr. McGrath, Mr. Stackpole, and Mr. Zeidman on hard drive last week were produced in such a way that importation of those files into the copies of Perforce those experts are using is an extremely laborious process, requiring that each file be updated manually. It is possible that there is a way to automate this process and, if so, we would welcome your input on what that process would be. I was surprised that MSC's experts were trying to do such a thing, and replied: Those files were not submitted to me for importation into the Perforce repository, and I have not found it necessary to do that to achieve the purpose for which I received those files. Instead, I used a source code comparison program (in particular, Microsoft's WinDiff) to see how the source code files had been changed. Your experts may want to do the same, since it is considerably easier and more accurate than trying to import them somehow into the Perforce repository. My reply was not intended to order the use of any particular procedure by MSC's experts, but to note that the files in question were originally produced to me in a form particularly suited to a different purpose than MSC's experts wish to use them, and to indicate how I was using them. MSC also wrote: If the process of updating cannot be automated, however, it seems that production of a new, updated Perforce database, rather than piecemeal production of individual files, would not be any more cumbersome for Altair to produce, and would significantly reduce the burden upon our experts in installing and reviewing the new files. We would like you to explore whether production in this alternative fashion is workable and what the costs would be to Altair, in comparison to the costs of production of only files that have been changed. I replied: You seem to be requesting that Altair provide your experts with a new copy of the source code repository every time it requests review of any changes that are to be released in a test, beta, or commercial version, rather than providing copies of the files that are changed in that new version. Because of the size of the repository compared to the size of the change information for a new version, this would likely require Altair to purchase, create, and send five external hard disks whenever even a small fix is made. In addition, it would take considerably more time to compare the new repository against the source code at the time of the TRO hearing, the criteria for determining if the changes affect the status quo at that time. *5 The Court addressed Altair providing a new copy of the repository when it stated at the end of its December 18 2008 order that “before MSC's expert reports are due, the Court MAY require Altair to provide MSC's expert and outside counsel an updated version of its source code repository and twiki database.” (Emphasis added.) I don't believe the Court intended, when it ordered Altair to produce copies of the review material that had been provided to me, requiring Altair's furnishing MSC's experts and attorneys with an updated version of the repository whenever Altair asks for a review before the release [of] any test, beta, or commercial version of its software, especially when the only justification for providing new repositories is that MSC's experts have decided to do something laborious as part of their analysis. During the site visit, I reviewed the effort necessary to produce a copy of the repository. First, to assure that the repository remains in a consistent form while it is being copied, all user activity must be stopped. Because the repository stores source code for a number of Altair products besides MotionSolve, that means that use of the repository for those products must also be stopped, and this affects other Altair offices. The repository is of substantial size (35–40 gigabytes, of which about 10 gigabytes are related to MotionSolve), so it takes time to copy it to another Perforce server. After that is done, the source code files and other information regarding products other than MotionSolve must be removed. The repository then must be encrypted using TrueCrypt to make sure that it will be unreadable by others if lost in shipment, and five copies must be produced on external hard disk drives (or very large USB flash memories): three for MSC's experts, one for MSC's outside counsel, and one for me. As I noted in my email, not only is this a considerable effort for Altair, but it results in a production where it is more difficult to determine the nature of a test or beta version to be shipped and what has been changed. Because of that, I recommend that Altair not be required to produce a new version of the repository for MSC's experts and outside counsel whenever it is about to release a test or beta version of MotionSolve.