The FIPS 140-2 "Hostage" IssueLast update 2015-02-27
1. BackgroundThe OpenSSL FIPS Object Module v2.0 ("OpenSSL FIPS module") is a Level 1 FIPS 140-2 validated cryptographic "module" with validation certificate number #1747:
The OpenSSL FIPS module is of widespread interest because of requirements that all cryptographic implementations must be FIPS 140-2 validated in products procured by the U.S. government. Not only must the software itself be validated via a formal bureaucratic process, by the Cryptographic Module Validation Program ("CMVP"), but the "platform" ("OE" or "Operational Environment") on which that cryptographic software runs must also be listed as an element of the validation. Roughly speaking, a "platform" in this context is a specific operating system version (e.g. "Ubuntu 14.04") on a specific processor architecture ("x86-64 with AES-NI").
The OpenSSL FIPS module already includes 101 platforms, more than for any other validation module (with still more pending). However, vendors may desire to use the OpenSSL FIPS module on platforms that are not currently included in the #1747 validation. OSF can economically add such platforms of interest via the "change letter" (technically "Maintenance Letter") process by which certain types of modifications can be made to existing validations. The addition of new platforms is one of those allowed changes.
Through this change letter process many platforms have been added to the #1747 validation since its original approval in 2012. Those platform additions have been sponsored by multiple commercial companies and government agencies. When new platforms are added to the OpenSSL FIPS module they become available for use not only by the original vendor or agency sponsoring them, but also by any other interested users.
The rest of this discussion assumes some familiarity with the FIPS 140-2 validation process.
2. Change LettersTraditionally a validation once approved (as #1747 was in mid-2012) is not retroactively second-guessed by the CMVP as new requirements are introduced for new validations (which does happen frequently). So for instance the source code that was used for the #1747 validation was no longer suitable for new validations as of early 2014, yet #1747 remains valid for use (as do many even older validations). The exception to that rule has been the gradual elimination of some algorithms or keysizes, as has happened with the deprecation and disavowal of the infamous Dual EC DRBG algorithm, or multiple "weak" algorithms per SP800-131A. However, those algorithm "un-approvals" were not editorial changes to extant validations, just restrictions on the contemporary use of those validations (i.e. you can still use those modules, just not the specific now non-approved algorithms within them).
Also by long tradition, "change letter" updates to existing validations have been allowed indefinitely. The most common use of a change letter is to add a new platform ("OE") to an existing validation. Ironically the CMVP has freely accepted such platform portability change letters while only reluctantly accepting, or outright denying, vulnerability mitigation changes. For instance, our pleas to modify the #1747 FIPS module to fully defend against the "Lucky 13" TLS vulnerability were rejected1, and it took six months to get approval to remove Dual EC DRBG2.
3. The New RequirementThen comes the latest moving of the goalposts. At any point in time we're usually working on a half-dozen or so new platforms. We have a pair of platforms from vendor "X" ready for submission, as platforms 103 and 104 for upcoming module revision 2.0.10, and one platform update for vendor "Y" for the 2.0.9 revision3. We have been told (with final confirmation January 15) that the CMVP is refusing to accept those or any other new change letters unless we modify fifteen existing platforms in the #1747 validation (the majority of which were not related to vendors "X" or "Y" in any way). Specifically, the change that is being demanded is that we supply explicit version numbers for the hypervisor based platforms, so for instance an existing platform
AcmeOS 1.0 64-bit on x86 under VMware ESXwould become:
AcmeOS 1.0 64-bit on x86 under VMware ESX 5.1For new platforms, no problem. Platform sponsors are all cautioned that the CMVP may change the rules at any time and that we (OSF and the test lab) cannot promise any specific platform description. Note that the most recent platforms (Table 2 of the Security Policy document) do meet this new guidance, as we submitted them that way.
At first glance that retroactive change to those older platforms doesn't seem all that significant. But, it is a big deal for some of the sponsors of those platforms, where their sales and marketing initiatives hinge on specific platform descriptions. We know from some clients that some DoD customers are incredibly picky about the platform descriptions, so that's a rather touchy issue. Platform descriptions get written into formal contracts, sales agreements, purchase orders, and requisitions that are taken very literally and not easily renegotiated and updated4.
We have conferred with the stakeholders/sponsors of those fifteen platforms and asked about the impact and whether they would rather see their older platforms changed, or lose the ability to add new platforms. For one such stakeholder, vendor "X", that question has a particular relevance as they have engaged OSF to do four new platform validations, two of which were ready for submission. Vendor "Y" has no relationship to any previous platforms and so is entirely an innocent bystander.
Of those four stakeholders who would be impacted by a retroactive platform change, none prefer those changes be made, even at the cost of no more change letters. That leaves us in the position of either throwing past (and current) clients under the bus, or disappointing prospective (and some existing) clients by not pursuing new platform validations. The fact that vendor "X", subject to immediate impact either way, has emphatically chosen to preserve the existing platforms is quite telling. We have often said that OSF values our past clients and supporters, and this situation gives us the "opportunity" to put our money where our mouth is5. We have therefore made the decision to respect the strong consensus of those stakeholders and clients, and cease doing change any letter updates while this new CMVP requirement is in effect.
4. Larger ImplicationsThis is the second time in only 12 months that the OpenSSL FIPS module has been threatened with a ban on change letter updates. The last such issue, in early 2014, caused an additional delay of about four months in the processing of multiple change letter updates. While that ban was in effect we lost one already signed platform sponsor and lost multiple other prospective sponsors who couldn't afford to wait or take the gamble on reversal of the ban. That hiatus cost OSF a substantial portion of our 2014 revenues for FIPS related support activities.
The revenue from change letter updates makes the validations possible; the actual initial open source validations themselves have never been profitable. OSF is currently in the advanced stage of discussions with potential sponsors of a new open source based validation to succeed the current #1747 validation. However, the viability of those plans hinges on the assumption that the new validation, once obtained, could be thereafter leveraged for at least a few years with the adddition of new platforms via change letter updates. If we can't be reasonably assured that a new validation will have a reasonable lifespan, in terms of our ability to do change letter updates, then the overall value proposition looks pretty dismal both for OSF and OpenSSL, and for prospective sponsors and users of the new validation.
5. ConclusionSo in a nutshell the CMVP is holding new platform validations hostage to retroactive changes to old (and unrelated) ones. Not only are the sponsors of those fifteen platforms in validation #1747 harmed (and/or the sponsor of the pending "hostage" platforms), but this latest new "guidance" from the CMVP sets a dangerous new precedent for all validations. Everyone loses: the CMVP doesn't get the retroactive changes that they seem to think are important enough to warrant substantial economic damage; no new platforms -- even entirely unrelated ones -- can be added, and OSF loses the only source of revenue supporting ongoing maintenance of the OpenSSL FIPS module6.
This discussion paints a rather grim picture for users of the #1747 validation. It should be noted that similar existential issues threatening the current or future viability of that validation have been encountered before, and the underlying policies have been -- after long delays -- either rescinded outright or mitigated to less damaging forms. The large constituency of OpenSSL FIPS module users means that issues like this cause a lot of disruption that will eventually be heard and appreciated by the CMVP bureaucracy. If past history is any guide reconsideration of this new policy is possible and even likely. However, that is sure to take an appreciable amount of time, many months perhaps. The longer term question is whether pursuit of new open source validations makes sense given the risk of unpredictable retroactive requirements.
- Footnotes -
1 The "Lucky 13" vulnerability is an issue with the very design of the TLS protocol and not a defect in any implementation. Complete mitigation of this vulnerability in standard OpenSSL was possible, but not in the "FIPS capable" OpenSSL (OpenSSL 1.0.1 built with the "fips" option to include the OpenSSL FIPS Module) because doing so would have required modest but prohibited changes to the FIPS module API.
2 For one perspective on that ordeal see http://veridicalsystems.com/blog/immutability-of-fips/.
3 Specific vendors/sponsors not identified by name due to non-disclosure restrictions. At times it is appropriate for various OSF clients and OpenSSL stakeholders to confer directly. We only attempt to arrange such dialogs with mutual consent.
4 This is a point not obvious to those unacquainted with the joys of elaborate and inflexible government procurement processes that are administered by specialists wwith minimal understanding of the technology being procured. Here is a more articulate (paraphrased) description of the issue by an astute observer:
If vendor X has an existing contract which promises that the system they delivered to some US Government agency uses only crypto code which was validated for "Red Hat Enterprise Linux 7.0 (x86_64) running on VMware ESX", then if OpenSSL obeys the demand to change the definition to read "Red Hat Enterprise Linux 7.0 (x86_64) running on VMware ESX 5.1", then vendor X would suddenly be unable to fulfill that contract (and potentially be subject to harsh penalties). But if OpenSSL refuses to change the definition, OpenSSL cannot deliver to vendor X a new platform validation so vendor X cannot get a new government contract to deliver for that platform, and neither can anybody else.
5 Change letters are a significant source of discretionary revenue for OSF, and the primary source of personal income for one of us. This will hurt. We will also eat the test lab fees for the five in-process change letters that now cannot be completed (in addition to earning nothing for our labor on those).
6 The five open source based validations done to date (of which the #1747 validation is the latest) have all been primarily funded by one or more U.S. government agencies. However, in each case that funding was at most barely adequate to cover immediate expenses of obtaining the initial validations (and most were done at a significant net loss). The revenue to sustain support of the OpenSSL FIPS modules and the accompanying "FIPS capable" OpenSSL has come entirely from the change letter updates.