Aftermath of the FIPS 140-2 "Hostage" SituationInitial version 2015-06-18, Last update 2015-09-05
1. BackgroundThe "hostage" situation and the attempted "ransom" have already been discussed in detail. This discussion addresses the next major development in that saga.
The quick TL;DR for the history to date: the #1747 validation is a collaborative effort with many sponsors. At the point where the hostage situation began that validation included 1011 platforms2 paid for by dozens of different sponsors.
In January of 2015 the CMVP3 decreed that no more new platforms could be added to the #1747 validation unless the descriptions of a number of previously approved platforms were changed. Since this validation was a collaborative effort that decision was not up to OSF, the nominal validation vendor of record. The most heavily impacted sponsors were consulted and they emphatically preferred, given that stark choice, to forfeit the ability to add new platforms in order to preserve their original investments.
That outcome left OSF with a number of pending new platforms in limbo.4 After extensive discussions with multiple test labs and stakeholders we finally identified the "ransom" solution that appeared to be actively encouraged by the CMVP. The "ransom" of $19,500 was paid, the "1A SUB" paperwork submitted (on 2015-04-17), and we patiently waited for approval of those submissions. That approval is still pending as of 2015-06-18.
Then on 2015-06-15, the CMVP updated their web site entry for validation #1747 to remove a number of platforms from the rightmost cell (the "Big Blob o' Text" as we call it) and added text noting that platforms had been effectively removed from the validation.
Put another way, they "executed" the "hostages". Note this was done after we paid the "ransom" and after the other pending change letter submissions -- for unrelated sponsors -- had been delayed for months.
What Platforms Are Impacted?The Big Blob o' Text is notoriously unreadable and generally ignored by everyone but the CMVP in preference to the more legible Security Policy document linked to from the same NIST CMVP web page. An analysis of the Big Blob5 as of 2015-06-16 shows the following apparent correspondence with Table 2 of the Security Policy (here the platform numbers as shown in Table 2 are on the left, and the platform descriptions as extracted from the Big Blob are on the right, in the order they appear in the Big Blob):
We do not know why the ordering of the platforms is not consistently chronological, nor if there is any significance to that unusual ordering. Note that platform 71 is among the survivors, yet it appears to have been one of the hostage platforms. We suspect this is a clerical error that will be corrected by the CMVP at some point.6
However, the omission of platform 8 from the list of survivors is surprising, as that does not appear to have anything in common with the known hostage platforms. We have asked for clarification. Likewise the list of survivors appears to have only one possible match for the two similar platforms 20 and 21, and only one possible match for the two similar platforms 45 and 46.
Our best guess at this point is that the survival of platform 71, and the demise of the three platforms 8, 20/21, and 45/46 are clerical errors.
With those assumptions, by the process of elimination against the list of survivors the "executed hostages" are:
47 Windows 2008 32-bit under vSphere Xeon E3-1220v2 (x86) 48 Windows 2008 64-bit under vSphere Xeon E3-1220v2 (x86) 49 RHEL 6 32-bit under vSphere Xeon E3-1220v2 (x86) 50 RHEL 6 64-bit under vSphere Xeon E3-1220v2 (x86) 59 VMware Horizon Mobile 1.3 under VMware under Android 4.0 Qualcomm MSM8X60 (ARMv7) 66 VMware Horizon Workspace 1.5 under vSphere Intel Xeon E3-1220 (x86) 67 VMware Horizon Workspace 1.5 under vSphere Intel Xeon E3-1220 (x86) with AES-NI 71 Linux 3.4 under Citrix XenServer Intel Xeon E5-2430L (x86) 72 Linux 3.4 under Citrix XenServer Intel Xeon E5-2430L (x86) with AES-NI 73 Linux 3.4 under VMware ESX Intel Xeon E5-2430L (x86) 74 Linux 3.4 under VMware ESX Intel Xeon E5-2430L (x86) with AES_NI 75 Linux 3.4 under Microsoft Hyper-V Intel Xeon E5-2430L (x86) 76 Linux 3.4 under Microsoft Hyper-V Intel Xeon E5-2430L (x86) with AES-NI 79 PexOS 1.0 under vSphere Intel Xeon E5-2430L (x86) 80 PexOS 1.0 under vSphere Intel Xeon E5-2430L (x86) with AES-NIUpdate 2015-09-05: The CMVP approved the change letter for the 2.0.10 revision to the #1747 validation on 2015-09-04. That change letter update added nine new platforms and supplied some of the hypervisor version information demanded by the CMVP for the "executed hostage" platforms. Not all of those platforms were updated, per decision of the platform sponsors. The original draft revision of the Security Policy replaced the rows in Table 2 with the text "(removed by CMVP)". The CMVP objected to that text and so those rows were just left entirely blank. The Big Blob o' Text was updated and the typographical errors corrected at the same time. A legible rendering of the Big Blob can be found here.
What Does This Mean for Users of the Removed Platforms?The sponsors of these platforms are obviously directly impacted. We know that for most of them just a change in the platform descriptions represented a significant economic setback; outright removal is clearly at least as damaging.
However, note there are almost certainly many vendors and end users of the OpenSSL FIPS Object Module on these now removed platforms other than the original sponsors; that is the nature of a collaborative effort such as the open source based FIPS 140-2 validations. Those vendors and end users now find their already paid for and deployed product instances have become retroactively non-validated. That's sure to have a huge impact as that realization slowly sinks in.
Note that very few vendors or users routinely check the NIST CMVP web site, and we at OSF have no way of knowing who most of the #1747 validation end users are.7, or how many among those are using the module on any of the impacted platforms.
What Happens Next, Short Term?We expect that the Big Blob o' Text will be changed to reflect a revised list of survivors. We will modify the Security Policy document to reflect whatever that final reality turns out to be (though note that we have to pay for such updates so that may not happen right away).
We expect that at least some of the directly impacted platform sponsors will instruct us to give the hypervisor version information to the CMVP so that their platform entries can be partially restored. One such sponsor has already confirmed what could be surmised as obvious: given the initial choice between complete removal of their platforms and modification they would have chosen the latter (thus avoiding the whole hostage mess and ransom payments). It would have been preferable to have been presented with that demand up front, rather than being blindsided by preemptive deletion months later.
The test lab believes that with the disputed hostage platforms gone the CMVP will once again accept change letter update requests for the #1747 validation, in which case the still pending "RE" and "SE" validations become superfluous (yet more detritus littering the FIPS 140-2 validation landscape).8 We sent a draft Security Policy for a #1747 change letter update to add the eleven pending platforms to the test lab right away (2015-06-16), and the lab estimates that they can have it formally submitted no later than 2015-06-27.
Until we know for sure that platforms can once again be added to #1747, we will submit all pending and new platforms against both the #1747 and "RE" or "SE" validations (as appropriate depending on the presence of source code mods). It is often the case that the only way to know for sure what the CMVP will do is to pay for a formal submission to elicit a definitive response. Ideally those long suffering platform sponsors will find their platforms added to both validations, #1747 and RE/SE.
Update 2015-09-05: We now know that we will not be able to made additional updates to the #1747 validation. One of the reviewer comments we received for the 2.0.10 change letter update was a demand that yet more previously approved platform descriptions be changed (i.e., new hostages). We deflected that by noting that a number of very recently approved new validations had platforms of a very similar nature (platforms tested with optimizations without a corresponding no-optimization case), and that demand was dropped. However, it is now clear that any attempt to add new platforms to the #1747 validation will expose all previously approved platforms to the risk of a new "hostage" situation. Accordingly, we will not be attempting to do any further change letter updates to the #1747 validation. That will remain forever at revision 2.0.10.
What Are The Long Term Implications?In a word, bleak. This incident illustrates a major vulnerability inherent in the collaborative nature of the open source based validations. When one sponsor pays for a platform validation, many other unrelated users benefit from the addition of that platform as the OpenSSL FIPS Object Module is available to all at no cost under an open source license. However, now that the precedent of retroactive removal of platforms has been set, all users of the #1747 validation are at risk of losing their validated status, quite possibly without even realizing it for an indeterminate amount of time. Many of those will be the unrelated after-the-fact users who don't have any option of accommodating whatever retroactive modification the CMVP is demanding.
Note there may also be an impact on validations other than #1747, ones which were modeled after it (as is commonly done) and/or which have the same issue with virtualized platforms lacking hypervisor versions.9
The risk of retroactive changes was already a known one, but prior to this incident that concern was primarily for in-process validation actions. Now future platform sponsors and other users will have to factor in the risk that their usage of the OpenSSL FIPS module may leave them with already deployed products suddenly in a non-validated status, possibly years after the initial approvals. That will encourage many such potential sponsors to obtain or utilize other types of clone or knock-off validations rather than extending the #1747 validation for the benefit of all.10
The economic damage to the nominal validation vendor, OSF, has been substantial. We've lost a half year of revenues thanks to the extended hold on all change letters, on top of the immediate $19,500 cost of the ransom payments. The economic viability of the OpenSSL FIPS validation business model, always rather marginal to begin with, is very much in question now.
- Footnotes -
1 Actually 98 platforms total due to skipping (for historical reasons) entries 14, 15, and 18.
2 Technically "Operational Environments" or "OEs"; but commonly referred to as platforms. "OE" is actually a better term, as the FIPS 140-2 conception of "OE" is only a rough approximation of "platform" as understood by the information technology sector as a whole.
4 Why did we defer to the interests of the past sponsors of already completed and approved validation actions versus the new sponsors of pending ones? Simple: the former had already long since paid us for those platform validations, and sold or deployed products on that basis for an extended period. The new sponsors had not yet been invoiced for any expenses, as OSF generally absorbs the financial risk of such validations. Also, while the new sponsors had plans in motion based on the anticipated validation approvals, and tell us they are also suffering significant economic damage from the delay and disruption of those plans, their overall exposure is less extensive. Our contracts state that there is considerable risk with any validation action and that a successful or timely outcome cannot be guaranteed.
An easy way to decode the Big Blob o' Text: cut and paste the relevant part of the blob into a text file and
6 Clerical errors of this sort are common, and very understandable in this case given the illegibility of the Big Blob o' Text. We also note that in a very similar recent "1B SUB" validation platform 71 is missing.
7 We do keep a rough count of the snail-mailed CD requests. Assuming that most of those requests correspond to actual use of the product (and ignoring the fact that many vendors don't request a CD), we can guesstimate that several hundred software vendors are actively using the OpenSSL FIPS Object Module 2.0. If you assume that each of those averages a few hundred customers, and assume an even distribution of usage across all platforms (obviously an overly simplistic assumption), then there are something like 15,000 impacted end users.
8 This will not be the first time that major (to us) investments of time and money have come to naught in the face of bureaucratic whimsey.
9 A rather cursory search will show several such platforms which will not be explicitly named here. Fortunately for such validations it has typically been the case that restrictions imposed on the open source based validations are not enforced against other validations that meet the same ostensible criteria.
10 One can't help wondering if that was an intended outcome.