Appendices

One of the things readers value most about this report is the level of rigor and integrity we employ when collecting, analyzing, and presenting data.

Knowing our readership cares about such things and consumes this information with a keen eye helps keep us honest. Detailing our methods is an important part of that honesty.

First, we make mistakes. A column transposed here; a number not updated there. We’re likely to discover a few things to fix. When we do, we’ll list them on our corrections page: verizon.com/business/resources/reports/dbir/2022/corrections/.

Second, we check our work. The same way the data behind the DBIR figures can be found in our GitHub repository³², as with last year, we’re also publishing our fact check report there as well. It’s highly technical, but for those interested, we’ve attempted to test every fact in the report.

Third, science comes in two flavors: creative exploration and causal hypothesis testing. The DBIR is squarely in the former. While we may not be perfect, we believe we provide the best obtainable version of the truth, (to a given level of confidence and under the influence of biases acknowledged below). However, proving causality is best left to randomized control trials. The best we can do is correlation. And while correlation is not causation, they are often related to some extent, and often useful.

Non-committal Disclaimer

We would like to reiterate that we make no claim that the findings of this report are representative of all data breaches in all organizations at all times. Even though we believe the combined records from all our contributors more closely reflect reality than any of them in isolation, it is still a sample. And although we believe many of the findings presented in this report to be appropriate for generalization (and our conviction in this grows as we gather more data and compare it to that of others), bias exists.

The DBIR Process

Our overall process remains intact and largely unchanged from previous years³³. All incidents included in this report were reviewed and converted (if necessary) into the VERIS framework to create a common, anonymous aggregate data set. If you are unfamiliar with the VERIS framework, it is short for Vocabulary for Event Recording and Incident Sharing, it is free to use, and links to VERIS resources are at the beginning of this report. The collection method and conversion techniques differed between contributors. In general, three basic methods (expounded below) were used to accomplish this:

1. Direct recording of paid external forensic investigations and related intelligence operations conducted by Verizon using the VERIS Webapp
2. Direct recording by partners using VERIS
3. Converting partners’ existing schema into VERIS

All contributors received instruction to omit any information that might identify organizations or individuals involved. 

Some source spreadsheets are converted to our standard spreadsheet formatted through automated mapping to ensure consistent conversion. Reviewed spreadsheets and VERIS Webapp JavaScript Object Notation (JSON) are ingested by an automated workflow that converts the incidents and breaches within into the VERIS JSON format as necessary, adds missing enumerations, and then validates the record against business logic and the VERIS schema. The automated workflow subsets the data and analyzes the results. Based on the results of this exploratory analysis, the validation logs from the workflow, and discussions with the partners providing the data, the data is cleaned and re-analyzed. This process runs nightly for roughly two months as data is collected and analyzed.

Incident Data

Our data is non-exclusively multinomial, meaning a single feature, such as “Action,” can have multiple values (i.e., “social,” “malware” and “hacking”). This means that percentages do not necessarily add up to 100%. For example, if there are 5 botnet breaches, the sample size is 5. However, since each botnet used phishing, installed keyloggers, and used stolen credentials, there would be 5 social actions, 5 hacking actions, and 5 malware actions, adding up to 300%. This is normal, expected, and handled correctly in our analysis and tooling.

Another important point is that when looking at the findings, “unknown” is equivalent to “unmeasured”. Which is to say that if a record (or collection of records) contains elements that have been marked as “unknown” (whether it is something as basic as the number of records involved in the incident, or as complex as what specific capabilities a piece of malware contained) it means that we cannot make statements about that particular element as it stands in the record—we cannot measure where we have too little information. Because they are ‘unmeasured’, they are not counted in sample sizes. The enumeration “Other” is counted, however, as it means the value was known but not part of VERIS (or not one of the other bars if found in a bar chart). Finally, “Not Applicable” (normally “NA”) may be counted or not counted depending on the claim being analyzed.

This year we have again made liberal use of confidence intervals to allow us to analyze smaller sample sizes. We have adopted a few rules to help minimize bias in reading such data. Here we define ‘small sample’ as less than 30 samples.

1. Samples sizes smaller than five are too small to analyze
2. We won’t talk about count or percentage for small samples. This goes for figures too and is why some figures lack the dot for the median frequency. 
3. For small samples we may talk about the value being in some range, or values being greater/less than each other. These all follow the confidence interval approaches listed above

Incident Eligibility

For a potential entry to be eligible for the incident/breach corpus, a couple of requirements must be met. The entry must be a confirmed security incident defined as a loss of Confidentiality, Integrity, or Availability. In addition to meeting the baseline definition of “security incident” the entry is assessed for quality. We create a subset of incidents (more on subsets later) that pass our quality filter. The details of what makes a “quality” incident are:

• The incident must have at least seven enumerations (e.g. threat actor variety, threat action category, variety of integrity loss, et al.) across 34 fields OR be a DDoS attack. Exceptions are given to confirmed data breaches with less than seven enumerations
• The incident must have at least one known VERIS threat action category (hacking, malware, etc.)

In addition to having the level of details necessary to pass the quality filter, the incident must be within the timeframe of analysis, (November 1, 2020 to October 31, 2021 for this report). The 2021 caseload is the primary analytical focus of the report, but the entire range of data is referenced throughout, notably in trending graphs. We also exclude incidents and breaches affecting individuals that cannot be tied to an organizational attribute loss. If your friend’s laptop was hit with Emotet it would not be included in this report.

Lastly, for something to be eligible for inclusion into the DBIR, we have to know about it, which brings us to several potential biases we will discuss next.

Acknowledgement and analysis of bias

Many breaches go unreported (though our sample does contain many of those). Many more are as yet unknown by the victim (and thereby unknown to us). Therefore, until we (or someone) can conduct an exhaustive census of every breach that happens in the entire world each year (our study population), we must use sampling. Unfortunately, this process introduces bias.

The first type of bias is random bias introduced by sampling. This year, our maximum confidence is +/- 0.7% for incidents and +/- 1.4% for breaches, which is related to our sample size. Any subset with a smaller sample size is going to have a wider confidence margin. We’ve expressed this confidence in the complementary cumulative density (slanted) bar charts, hypothetical outcome plot (spaghetti) line charts, quantile dot plots and pictograms.

The second source of bias is sampling bias. We strive for “the best obtainable version of the truth” by collecting breaches from a wide variety of contributors. Still, it is clear that we conduct biased sampling. For instance, some breaches, such as those publicly disclosed, are more likely to enter our corpus, while others, such as classified breaches, are less likely.

The below four figures are an attempt to visualize potential sampling bias. Each radial axis is a VERIS enumeration and we have stacked bar charts representing our data contributors. Ideally, we want the distribution of sources to be roughly equal on the stacked bar charts along all axes. Axes represented by only a single source are more likely to be biased. However, contributions are inherently thick tailed, with a few contributors providing a lot of data and a lot of contributors providing a few records within a certain area. Still, we mostly see that most axes have multiple large contributors with small contributors adding appreciably to the total incidents along that axis.

You’ll notice a rather large contribution on many of the axes. While we’d generally be concerned about this, they represent contributions aggregating several other sources, so not actual single contributions. It also occurs along most axes, limiting the bias introduced by that grouping of indirect contributors.

The third source of bias is confirmation bias. Because we use our entire dataset for exploratory analysis, we cannot test specific hypotheses. Until we develop a collection method for data breaches beyond a sample of convenience this is probably the best that can be done.

As stated above, we attempt to mitigate these biases by collecting data from diverse contributors. We follow a consistent multiple-review process and when we hear hooves, we think horse, not zebra.³⁴ We also try to review findings with subject matter experts in the specific areas ahead of release. 

Data Subsets

We already mentioned the subset of incidents that passed our quality requirements, but as part of our analysis there are other instances where we define subsets of data. These subsets consist of legitimate incidents that would eclipse smaller trends if left in. These are removed and analyzed separately, though they may not be written about if no relevant findings were, well, found. This year we have two subsets of legitimate incidents that are not analyzed as part of the overall corpus:

1. We separately analyzed a subset of web servers that were identified as secondary targets (such as taking over a website to spread malware)
2. We separately analyzed botnet-related incidents

Both subsets were separated out the last five years as well. Finally, we create some subsets to help further our analysis. In particular, a single subset is used for all analysis within the DBIR unless otherwise stated. It includes only quality incidents as described above and excludes the two aforementioned subsets.

Non-incident data

Since the 2015 issue, the DBIR includes data that requires analysis that did not fit into our usual categories of “incident” or “breach.” Examples of non-incident data include malware, patching, phishing, DDoS, and other types of data. The sample sizes for non-incident data tend to be much larger than the incident data, but from fewer sources. We make every effort to normalize the data, (for example weighting records by the number contributed from the organization so all organizations are represented equally). We also attempt to combine multiple partners with similar data to conduct the analysis wherever possible. Once analysis is complete, we try to discuss our findings with the relevant partner or partners so as to validate it against their knowledge of the data.

In 2021 we reported that the human element impacted 85% of breaches, which decreased slightly to 82% this year.  Unfortunately, strong asset management and a stellar vulnerability scanner aren't going to solve this one.

Instead, you're going to need to change the behavior of humans, and that is quite an undertaking.  Regardless of how you plan on doing it (be it giving them a reason to change, providing training or a combination of the two), you will need a way to tell if it worked, and that normally means running a test.  Here's a cheat-sheet of things that your internal department or vendor who is responsible for conducting the training should provide to you so you can determine if it is paying off:

• A population of people you are interested in. (If the test was run only in a healthcare company or in a specific division, the population should be “Employees of healthcare companies” not “Anyone”)
• A measurable outcome that can be proven or disproven. (Such as “More correct answers on a questionnaire about phishing delivered 1 day after training,” “ Fewer people clicked the phishing email,” etc.)
• An intervention to test to see if it changes the outcome. (“Watch a 15-minute video on not clicking phishing.”)
• A control that provides a baseline for the outcome. (“Received no training,” “Read a paragraph of text about phishing,” “Read a comic book and took a nap, etc.)
• Random assignment. (“200 employees were picked to participate in the study. 100 were randomly assigned the control and 100 were randomly assigned the intervention”)
• The conditions of the test should also be shared. (“Participants were sent the control or intervention via the company training tool as annual mandatory training.”)

This test may be something run specifically for you or a test the trainer already ran. As long as the population, outcome, etc. are close to yours, it doesn’t matter. 

The manner in which the results are reported is just as important as how the testing was conducted. Here are a few things you should expect in the results:

• A result with a confidence interval. (If 10 folks in the control clicked the phishing email and only 1 in the intervention, 10-1 = 9 people we thought would click, but didn’t. That’s a 9/10 or 90% effectiveness.) But that 1 number doesn’t tell the whole story. What if some of those were flukes? The result should come with a range (such as 70% to 100% effective at 95% confidence) similar to the DBIR ranges. (Btw, if the range includes 0%, there’s a chance the training didn’t actually do anything.) If the outcome question was yes/no, then just a confidence level will do. (“People changed due to the intervention with 98% confidence”.)
• Since results can vary over time, you should know when the result was measured. “The result was 30 days after the intervention.”
• What if some folks just refused to take the training? That’s called dropout and is important to the results. The results should show who dropped out (preferably by full name so they can be shamed in front of their peers – okay, not really). (“Twenty percent of technically savvy employees didn’t take the intervention training, while only 2% didn’t take the control training.”) Dropout can occur for any of the other characteristics recorded (industry, world region, department, age, etc.) and if major differences are found, the results should be broken out by those characteristics as well. (“People changed 98% in tech savvy folks, but only 70% in non-tech savvy.”)
• There should also be qualitative questions. Sometimes you don’t know what you don't know.³⁸ In that case, there should be an open-ended question about the training in addition to the more objective outcomes. It also gives a chance to ask questions like “Do you have formal education or a job in a computer-related field?” And you can see the importance of this in the dropout bullet above. Asking “Is there anything else you’d like us to know?” might reveal that while the training was effective, many folks found it highly offensive.

Unfortunately, there’s no way to guarantee something works. But, if you’re getting the information above (Population, Outcome, Intervention, Control, Random, Conditions) and (Results, When, Dropout, Qualitative) you can be reasonably confident you’re getting something for your effort. So, remember those easy acronyms: POICRC and RWDQ!

Contrast Figure 122 with the profits in Figure 123. Sixty percent of ransomware incidents had no profit and aren’t shown in the figure. A large portion had a profit near one dollar. However, the median is just over $100. Figure 124 shows what those same profits look like over time. After 300 simulated ransoms the actor has over $600,000 in income.³⁹

To see if this was an anomaly, we simulated 500 ransomware actors and 1.4% of them showed a loss.⁴⁰ However, the median profit after 300 incidents was $178,465, with the top simulated earner making $3,572,211.

The takeaway is that ransomware is more of a lottery⁴¹ than a business. You gamble on access, win the lottery 40% of the time, and get a payout from a few bucks to thousands of dollars. But for something more actionable, focus on the access. If an actor has to pay for services to break in rather than just an access product, you’ve made yourself much less of a target. Use antivirus to remove bots, implement patching, filtering, and asset management to prevent exposed vulnerabilities and standardize two-factor authentication and password managers to minimize credential exposure. Lastly, with email being the largest vector, you can’t ignore the human element. Start with email and web filtering followed by training. (See the Changing Behavior Appendix for a recommendation on how to tell if your training is working.)