9 Questions to Ask When Choosing a Machine Learning Fraud Detection Solution
By Sarah Beldo /
12 Apr 2016
Machine learning. Big data. Real time. Omnichannel. The world of fraud detection is rife with buzzwords. But wait, before you protest, “But Sift Science! You use these words too!”… yes, yes we do. The thing about buzzwords is that they do represent important concepts and powerful features. But when every vendor is promising that they can do big-data-machine-learning-omnichannel-fraud-detection-in-real-time, it can become incredibly challenging to choose a solution that’s right for your business.
That’s when it’s time to break out some more detailed, specific questions that will help you uncover real differences between vendors. Here are some of the most useful questions to ask:
1. How accurate is the model?
This is a biggie. Of course, the effectiveness of a machine learning fraud solution boils down to how well it works at doing what it promises to do: predict fraud. But when talking about accuracy, don’t forget to think about the flip side of stopping bad users – namely, not stopping good users. How well does the tool do at recognizing your good users?
So, how do you actually measure the accuracy of a new tool you’re not already using? Some platforms may give you the option of trying them out for free, without the commitment of a long-term contract. If you have existing tools you’re considering moving away from, you may even be able to run a test with the new system in parallel, then compare the results.
2. What are current customers saying about their results?
In a perfect world, you’d get a trustworthy recommendation from a similar business already in your network, with solid results to back it up. But those aren’t always easy to come by. That’s when testimonials and case studies can really come in handy to give substance to marketing claims. What logos are on the solution’s website? What companies are quoted, and are they providing specific results they achieved with the platform? Anyone can say anything. But when you have a brand name willing to attach themselves to a success story, it lends a lot more credibility.
3. How robust is the global customer network?
With a machine learning solution, having large volumes of high-quality data is a key part of the “secret sauce” to getting the best results. Why? Simple: more data means more to learn from. So, it’s in your best interest to look for machine learning solutions that have large and varied networks of customers.
But don’t forget the second part of this equation: how the tool actually leverages the data it collects and turns it into intelligence you can use immediately. In other words, how does it incorporate learnings from the global network to serve you, the customer?
4. What industries does it cover?
When asking this question, keep in mind that a niche focus isn’t necessarily best. While a solution that’s focused on serving a particular industry may mean it has deeper knowledge about that particular industry, that knowledge may not translate beyond the sales and marketing pitch into meaningful results for your company.
It really goes back to #3: more data is better, and diverse data is better. Intelligence from different industries can benefit everyone who uses a machine learning fraud detection solution. Because, after all, most fraudsters don’t tend to focus on scamming only food-delivery services, or only airlines. They’re constantly adapting, and you’ll want to benefit from the breadth of knowledge that comes from multi-industry data.
5. Can you tailor the model to your unique business needs?
A breadth of high-quality data is amazingly powerful, but equally powerful is the ability to layer on learnings that are specific to your business. That way, you can benefit from customizability as well as breadth. For example, a travel firm may derive great value from incorporating information like destination and time of travel relative to booking into their machine learning model.
6. How does the model adapt to changing fraud patterns?
One important differentiating factor for machine learning systems is in how users give feedback to the system, so it can learn and improve. For example, customers that use Sift Science can apply labels indicating whether specific users are “bad” or “not bad.” If a user is marked as “bad,” the system considers all of the behavior and signals of that user to be associated with bad behavior in the future. This type of feedback is crucial for accuracy, and for learning extremely quickly about adapting fraud patterns.
7. How quickly can the customer benefit from new learnings?
As we mentioned, “real time” has become a bit of a buzzword, peppered into sales pitches, blog posts, and marketing collateral. But what does it really mean to update risk scores in real time? It’s worth digging into the nitty gritty of promises like these to ask, for example: if a fraudsters is flagged by a user in the global network, will that information be reflected immediately (like in milliseconds)? Or will you need to wait for days, weeks, or even months for a model to update?
8. Is the pricing model transparent?
Fees and pricing models vary, but can include per-transaction or flat-rate monthly costs, contracts, setup fees, percentage of revenue, maintenance/support costs, and implementation costs. Can you find the information you need on the company’s website, or do you need to talk to a salesperson?
Also: does the solution require contract negotiations, or is it pay-as-you-go? Are the terms of service transparent and easy to understand?
9. How easy is it to integrate?
How available is the documentation? Is it simple, transparent, and easy to use? When it comes to integration, it’s a great idea to get your developers involved. But wouldn’t it be even better to be able to skip the initial discussion altogether, just send your developer a link to the documentation, and let them review it directly?
Want to learn more about how to craft a fraud-fighting approach that’s right for your business? Download our free 35-page ebook, Kickstart Your Fraud-Fighting Strategy.