I work on this exact problem for a cybersecurity company, and it’s super challenging 1) because it’s like trying to assign a probability to an earthquake (how do you validate your model with such few ground truth events?) and 2) like you mention, the more accurate we make the model, the more pushback there is as people begin to lose trust when it doesn’t match their intuition.
What’s a good way to make a convincing case of “trust us, we know this risk assessment doesn’t look intuitive but it’s better than the common XYZ approach because…”? Or is this a losing battle, and we should just allow customers to rank by CVE severities if that’s what they really want to do?
Some smaller startups are starting to work toward providing cyber VaR — quantify the probabilistic loss in financial terms over a given time period. This is really the only direction forward in cybersecurity IMO as risk must ultimately be modeled with statistics, not hunches. But I don’t think the industry is ready for it (and neither are the models to be frank).
There may be nothing novel here for you given you're already working in the problem area, but as established a CVE measure is completely hypothetical on its own; if your customers insist on looking at one variable (CVE), you could always make the case that CVE is incomplete on it's own and should always be supplemented with data such as:
a) is it actually exploitable
b) is someone exploiting it in the wild
c) accessibility of component/asset/application/whatever that carries the CVE (e.g. is it internet-facing or a non-networked subcomponent hidden under 15 layers of defense)
d) other exploitability scoring methods, like EPSS
e) etc...
(sometimes I like to use the analogue of trying to gauge your body's health based on one variable, like your body temperature - it can be high temporarily [in case of a flu] but it doesn't tell you anything about your overall health [as we all get sick occasionally])
The VaR approach is sensible but has to start from much further down in the root of the problem by first creating a value-based catalogue of all your assets, after which you should simulate all prospective attack paths towards all assets (maybe above a certain value threshold to simplify the model a bit), then overlay with attack / exploit probability data (simple example would be a CVE dataset + EPSS scoring) and finally you have something resemblant of an actual data-based risk model you can quantify impacts on. It's quite a large task and I don't think there's a single player doing all areas, you'd have to patchwork together multiple different datasets.
Still, it's probably where the industry will end up in the next 10-15 year timespan.
(I'm somewhat involved in the area too but just as a threat data provider for larger models.)
What’s a good way to make a convincing case of “trust us, we know this risk assessment doesn’t look intuitive but it’s better than the common XYZ approach because…”? Or is this a losing battle, and we should just allow customers to rank by CVE severities if that’s what they really want to do?
Some smaller startups are starting to work toward providing cyber VaR — quantify the probabilistic loss in financial terms over a given time period. This is really the only direction forward in cybersecurity IMO as risk must ultimately be modeled with statistics, not hunches. But I don’t think the industry is ready for it (and neither are the models to be frank).