top of page

Measuring False Positives is Not an Exact Science

Using Multiple Metrics can Provide a Better Understanding

When a legitimate customer has their order declined because of fraud concerns they may purchase from a competitor and may never return to the merchant who turned them away. This is not only painful in terms of losing the sale and potentially the lifetime value of that customer, but this makes measuring false positives difficult. Even if an organization identifies customers who reattempt their transaction or call-in to customer service to complete their order, this is not a complete representation of the volume of sales insults; it may only be the tip of the iceberg.

The first step to effectively measuring false positives is understanding that it is not an exact science, that the metrics we can use are proxies, and the more methods we have for deriving and measuring these proxies the closer we can come to understanding the whole picture. We will likely never have a perfect view or full representation of the volume and financial impact of false positives, and this may discourage many organizations from attempting to measure and track them at all.

In the recent State of CNP False Positives Survey from The Fraud Practice and Kount, 46 percent of respondents said their organization does not track false positives at all while 15 percent weren’t sure. Nearly 60 percent said their organization doesn’t have a benchmark or target false positives rate. Although confirmed false positives do not represent the actual false positives rate, it is still a useful metric to track performance from month-to-month and compare year-over-year.

There are multiple ways an organization can identify when they wrongly declined a customer, but it requires that same customer returning to make a legitimate transaction, which often does not occur. Even then, the organization should wait at least 90 days to be sure the second transaction attempt which they did accept does not return to them as a chargeback.

The direct proxies for measuring false positives focus on recognizing when the organization made the wrong decision. In many cases the consumer waits some period of time and re-attempts the same exact transaction in the online or mobile channel. The State of CNP False Positives study found that 73 percent of organizations recognize when a transacting customer has been previously declined, but only 62 percent of these organizations take the next step to follow up with post-transaction reporting and identify associated false positives. Alternatively, a customer may call a customer service number to try and have the order accepted, which is something that 68 percent of organizations track.

The above methods are considered direct proxies for measuring false positives because they allow organizations to confirm they erroneously declined an order for fraud concerns when future transaction attempts from that same customer turned out to be legitimate. These do not capture every sales insult because many to most declined customers do not re-attempt. For this reason it is also important to look at the other side of the spectrum: confirming when an organization was correct in their decision to decline an order. These metrics are also going to be proxies, as many orders in a decline population cannot be explicitly confirmed as an instance of fraud caught or a false positive, but looking at proxies on both sides of the coin helps fill-in a more complete picture.

This provides a more accurate picture of the health of an organization’s entire decline population. Among the orders a merchant declines for risk concerns, all are either false positives or true fraud that was caught. The challenge is that not all of the orders in the decline population can be explicitly confirmed either way. Direct proxies for tracking false positives identify which are confirmed sales insults, and while we cannot detect them all we can also try to identify as many instances of confirmed fraud attempts as possible. An organization should strive to understand the distribution of their entire decline population, and there is value in looking at both sides.

For an organization to identify when they made the right decision to decline an order they need to conduct post-transaction reconciliation and reporting. This can start with chargeback analytics, which all organizations selling online should be doing already. When a merchant receives fraud chargebacks they can research the details of the order and identity that placed it to prevent the same fraudster or similar tactics from staying under the radar in the future. To that end, the organization can leverage link analysis tools to see how many other transactions from this fraudster or synthetic identity have been attempted, which were accepted and which were declined.

After performing this post-transaction link analysis, a merchant can identify orders in their decline population that can now be explicitly labeled as a caught fraud attempt. Among the 38 percent of merchants that measure or track performance related to false positives, only half are using chargeback and post-transaction analytics to identify confirmed fraud caught. Organizations can typically identify many declines they were right to decline when they take the time to perform these chargeback analytics, because many fraudsters won’t relent until they are stopped.

A lesser known method for confirming when declining an order attempt was the right decision is to leverage reporting resources from the card associations. Visa and MasterCard provide acquirers and issuers reporting on payment cards that are initiating chargebacks and cards that are being closed and re-issued with a new card number due to fraud or compromise. Not all processors are able to provide this information to their merchant clients, but organizations that have access to this data can cross-reference these lists of cards associated with fraud against their list of decline orders. Orders an organization declines from a payment card being closed down for being compromised or used fraudulently can confirm the organization was right to decline that order. This is another method to help organizations label orders in their decline population as a confirmed caught.

Just one-third of those reached in the State of CNP False Positives Survey indicated they were aware of these resources such as the MasterCard SAFE and Visa TC40 reports, while 46 percent of those aware of this reporting said it is something their organization leverages for tracking performance related to false positives.

These figures are explored in more depth in the State of CNP False Positives Survey from Kount and The Fraud Practice, including breakdowns by annual online sales volume and average order value.

Download the State of CNP False Positives Survey Report for more information.


bottom of page