Certifications and Ratings in the Cybersecurity Guarantee Market

By Sean Martin, CISSP

Updated on May 4th, 2017

Jeremiah Grossman chats with ITSPmagazine’s Sean Martin about security software guarantees

Having written on the topic of cyber insurance in the past—and having seen Jeremiah Grossman, Chief of Security Strategy at SentinelOne, speak on the topic of cyber insurance and software guarantees at no fewer than three InfoSec conferences in the past year (AppSec California 2016, ISSA Los Angeles 2016, and Black Hat USA 2016)—I decided to explore the guarantee portion of the topic. This article is a result of Grossman’s presentation materials combined with interviews conducted with the software guarantee champion.

Note: Grossman’s slides from Black Hat USA 2016 can be viewed here(must be logged in to LinkedIn)

Let’s first clarify what I mean by the terms cyber insurance and software guarantees:

Cyber insurance comes into play when there is an accident, similar to a driver having a wreck with their vehicle. When a company’s InfoSec program or processes fail and they get hacked, the insurer is called and an insurance claim is made.

Software guarantees come into play when there is a manufacturing problem with a product or service, similar to a car’s transmission failing before the warranty has expired. When a company experiences a breach due to the failure of a product backed by a vendor warranty, the vendor is called and a claim is covered under the warranty.

Cyber insurance and software guarantees are separate but can operate in conjunction with each other. The guarantee a customer gets from paying a vendor for software assurance should be complimentary to any cyber insurance policy that the customer has purchased from an insurance company.

Current Methods for Security Assurance

Below you will find four “systems” that InfoSec industry customers use to help evaluate and maintain an acceptable level of assurance that the security of the software they are acquiring is of decent quality. There may be more, but these are the main methods:

Analyst Reports: Analyst firms such as Gartner, Forrester Research and 451 Research have developed business practices designed to help enterprise companies around the world find the most relevant solution providers to help them overcome their IT and security challenges. The problem with this method is that the analyst firms focus on large, established vendors—oftentimes leaving out new market entrants that may have great products.

Certifications/Seals: There are plenty of standards, certifications, accreditations and seals used to tag products and services with a level of quality, such as PCI-DSS and FIPS 140-2. Aside from being limited to a specific industry segment and specific product technology, respectively, the certification method of quality assurance has a variety of limitations. “Certifications are moments in time, and they don’t have any data supporting success or otherwise,” said Grossman. 

Product Ratings: “There are a ton of companies trying to be the Underwriter Labs (UL) for cybersecurity,” said Grossman. “One example is the Cyber Independent Testing Lab (CITL) started by Mudge (aka Peiter Zatko). Mudge and his wife Sarah Zatko presented their concept during Black Hat, where their abstract noted that CITL is not really a seal-based nor an approval-oriented UL methodology. CITL is rather more like Consumer Reports, where “the goal is to evaluate software according to metrics and measurements that allow quantitative comparison and evaluation by anyone from a layperson to a CFO or a security expert.”

Cyber Insurance: Essentially the “other side of the breach” option, the cyber insurance market is nearing $4B in premiums and is projected to reach $7.5B by 2020. Fully one-third of US businesses have some form of cyber liability insurance. Although insurers will give would-be policy holders questionnaires and might even perform an audit, cyber insurance (to date) is driven by only two variables: 1) The industry in which the insured entity operates, and 2) the number of records at risk of loss or theft. There is no actuarial data beyond this and no list of established controls proven in the market. Perhaps that’s where software guarantees come in: warranty claims data could help by feeding the cyber insurance policy pricing machine.

How Software Guarantees Work

Before vendors can offer software guarantees, they first need to know the effectiveness of the product – that is, when and where the product works, how and where it breaks, and how often it works or breaks.

There is no silver bullet to ensure that the product is of high-enough quality to warrant; quality and the definition of what’s important to “work” depend on the vendors and the type of product they’re offering the market.

Does training developers in secure coding processes work to ensure product quality? It’s possible. Some will say that this is a necessary precursor to offering a software guarantee, but as Grossman said, “there is a notable lack of references and data to prove that this works as a means to reduce the risk of a breach faced as a result of that developer’s code.”

While both application security and application quality are important, it takes a different view of what’s going on with the software in order to determine whether or not a software guarantee is appropriate. “It’s less about what a vendor must do ahead of time (secure training, static analysis) and more about the most effective controls necessary to reach the guarantee,” added Grossman. 

Vendors don’t need to guarantee everything their product does, just the things that the product does well, which hopefully are the things that matter most to the customer. “The vendor has to determine the hard and soft costs—what they will end up paying back if something goes haywire,” said Grossman. 

At SentinelOne, Grossman asked the team to give him data on what the product can detect and how often it may miss things. “Based on our findings, we ended up signing up for a ransomware payment guarantee—not the downtime or loss of data,” said Grossman. “We then looked at the impact of something being missed in relation to the ransomware guarantee we wanted to offer and shared this assessment data with the insurer. The insurer used this data to determine the premium for our own E&O (errors and omissions) policy, giving us reasonable and affordable deductibles.” 

The result for SentinelOne customers is a ransomware guarantee offered as a per-endpoint licensing cost. It’s essentially an add-on to the standard product license that provides the warranty, which gives customers $1,000 per endpoint to pay the ransom, with a cap of $1M per organization. 

When something happens, the claims process goes like this:


  • If a customer endpoint becomes infected with ransomware and the customer must pay the ransom to get access to his data, the customer can subsequently make a reimbursement claim to SentinelOne.
  • After review by the SentinelOne team, and if all systems look good from an implementation and configuration perspective in terms of the warranty, the customer is reimbursed for the ransom payment.
  • SentinelOne then makes a claim to its insurer to recoup the costs of the claim paid to the customer.

What Will Make Software Assurance Stick

“If we look at all the breaches in recent memory, in each case we already knew the vulnerability, how to find it, patch it and fix it,” said Grossman. “This is troubling because we know what to do in order to dramatically reduce breaches, but the ecosystem lacks the proper market incentives to do the right thing.”

The status quo, as Grossman put it, could prevent the software assurance concept from moving forward. “Vendors don’t think they are accountable, and customers don’t yet have a desire to change, but we’re starting to see a shift.”

My assumption, which Grossman confirmed, is that software guarantees will only succeed if the majority of the industry adopts the model. This is one of those “It Takes a Village” opportunities.

Grossman then led me to two things that the industry should really pay attention to:

  1. Security vendors need to recognize that offering software guarantees can be a key differentiator for their business. “There is a very lucrative opportunity here,” said Grossman.
  2. It needs to be easy for customers to request a software guarantee. “Vendors should already understand their product metrics, so why not put a guarantee in front of their customers?” said Grossman.

Recommendations to Get Started

In order for software guarantees to work, “vendors must have a view into the performance of their products,” said Grossman. “If you know your metrics as well as how and where your product works, you can offer a guarantee.” This statement led Grossman to offer the following suggestions for both vendors and their customers as they pursue offering and acquiring software assurance.

Recommendations for Vendors

  1. Get your engineering and service teams together to discuss what you are comfortable guaranteeing. Use this initial discussion to generate the idea. One easy way to approach this, said Grossman, is to “take a look at your marketing claims and determine which ones you are willing to warranty.” Whittle that list down to which products and features work exceptionally well. The key is to focus on the capabilities that you feel really good about.
  2. Take your findings from above and talk to 20-30 customers. Have them validate the guarantee idea/design. Do they want it more robust, or would they prefer that you guarantee another area of the offering?
  3. Go to a business insurance broker for your standard E&O policy to have them check with the underwriters for re-insurance. Get a quote and take it from there.

Recommendations for Customers

  1. When evaluating security software products, pick the one with the guarantee. If there isn’t a guarantee offered, don’t be afraid to ask for one.
  2. Remember that when you accept the guarantee, as a customer you have a set of responsibilities as well. If you don’t install and maintain the software properly, the claim may be invalid. It would be similar to trying to make a claim on a car warranty after driving for an extended period without performing routine maintenance.
  3. Beware of the fine print and other gotchas. While this part of the industry is very young, and Grossman wasn’t aware of any shenanigans just yet, he predicted very confidently: “They will come!”

One Final Tip

Similar to the cyber insurance market, the software guarantee market is driven by data and metrics. With this in mind, some of the easiest places to offer guarantees are those that generate a ton of data.

“Take Akamai with a WAF in the cloud,” explained Grossman. “They should have a pretty good view into what they can do and where their solution falls short. On the other hand, the manufacturer of a ‘pizza box firewall’ (aka 1U rack firewall) won’t have the same level of real-world reliability data as the cloud-based solution that collects telemetry data from multiple customers."

Cloud-based offerings and cloud-based service providers are more likely to have the data necessary to provide these types of guarantees. So to all the cloud-based solution providers: join the software assurance village and use your wealth of data to offer guarantees to your customers!