DID YOU KNOW
Web Spiders are also known as Crawlers, Bots, Ants or Scutters and the process of web data extraction may also be referred to as Web Crawling, Web Scraping or Automatic Indexing.
Spiders are automated bots or programs that 'crawl' webpages for the purpose of extracting or 'scraping' site information. They have many different uses on the web today, but in the context of fraud and risk mitigation they can be used to screen and analyze a website to look for signs of fraud or high risk.
Businesses who wish to use Spiders or Web Crawling services can build their own basic Spider bots or programs, purchase software for performing site crawls, or go through a provider who will help them design and execute custom projects. Businesses may use Spiders as part of initial screening and onboarding checks, or they may also continue to use Web Crawling for ongoing monitoring of websites.
Web spiders have many different uses, both good and bad:
Google and other search engines use spiders to copy and index webpage data, extract page titles, descriptions, keywords and links. Collecting this data, keeping it up-to-date and having it on hand are crucial for quickly finding relevant sites for a user’s search query. Also, site content, sites linking in and other information gained from web crawling help search engines formulate their search result ranks.
Site owners use spiders on their own sites to check that all links are still active and to validate their HTML code.
Businesses use spiders to watch what their competitors are doing: when they have press releases, when they offer sales and change their products or prices.
Researchers and business use spiders to harvest investment and financial data for market research. Spiders can be applied for such research in any market.
Fraudsters use spiders to crawl sites and forums for emails; this how spammers build their mailing lists. Besides harvesting emails, a fraudster may also use a spider to copy information from a legitimate site they are emulating for a pharming scam.
DID YOU KNOW
Merchant Website Monitoring may also be known as: Merchant Web Content Monitoring, Merchant Compliance Checks/Monitoring, Merchant Website Inspections, Merchant Onboarding Inspections, or Merchant Investigations.
When a merchant services business signs a client up for a merchant account, that merchant represents significant risk. If the merchant engages in illegal activity or is not compliant with required rules and regulations the merchant services business is at risk for fines and punishment as well.
Merchant website surveying investigates and monitors a merchant's website to ensure they are operating legally and in compliance with card association as well as national rules and regulations. Services may additionally check custom or proprietary standards a merchant services business holds their clients to.
Card association regulations as well as legal restrictions prevent merchants from selling certain goods/services, and merchant services businesses that enable companies to process credit card payments can be fined or even lose the right to process credit card transactions. Monitoring and surveying a merchant client's website recognizes such activities and reduces the risk of the merchant services business being involved in illegal transactions.
A merchant services business can also use website monitoring/surveys to ensure a merchant has provided accurate information on their merchant account application.
For example, a merchant will say what types of products/services they sell online. A website survey will check to see if they are selling any items they neglected to tell the merchant services business about. Also, as an ongoing monitoring tool this service will recognize risk changes in a merchant's site that may change risk, such as offering new product/service types or offering prices below market value (which may indicate a scam).
"Gotchas" with Merchant Website Surveying/Monitoring:
Here we discuss Merchant Website Monitoring in the context of monitoring websites to recognize non-compliance, high risk changes and potential fraud. However, there are providers that offer web content monitoring for the purpose a merchant or business keeping track of their direct competitors. These services are meant to catch competitor products/services and prices, content descriptions, press releases and other activity. There also exists services for a site owner to monitor their own site, but this is generally in the context of monitoring uptime, page load speeds and usage activity. While these services are often referred to as Website
Monitoring they should not be applied for risk mitigation purposes.
THE FRAUD PRACTICE
KEY NOTES
Alternative Solutions - Businesses can build their own Spider bot or perform manual checks, but there are no true alternative third party solutions. If a merchant services business needs to conduct an physical investigation they should consider On-Site Surveys.
Building this In-House - It is possible to build your own spider or bot using programming languages such as Java or PHP. You can also perform many checks manually such as: ensuring a business is legally registered in country or state/province, web domain and WHOIS lookup on site and site owner, searching online and in forums for any negative history or comments, and manually checking web pages to examine content, what they are selling, prices, etc.
Estimated Cost - Basic Spider software can be purchased for under $100 while more advanced software is generally a few thousand dollars. Some Web Crawling providers charge per single project and/or by subscription. For ongoing services providers often charge an initial project setup fee and then a monthly fee for recurring data extraction, maintenance and support. Merchant Website Monitoring vendors may offer services on a per merchant/inspection basis as well as on a subscription service for ongoing monitoring
Sample Vendors - N/A
SPIDERS & MERCHANT WEBSITE MONITORING TECHNIQUE OVERVIEW
Spiders are automated bots that scan and copy information from web pages. Spiders perform ‘web crawling’ or ‘web scraping’ on websites to copy the HTML code, text and other content. A merchant services business, for example, may use Web Spiders to investigate the content on their potential clients’ websites. There are also third party services that use Spiders as well as other automated processes and manual analysis for initial merchant website screening and ongoing monitoring. The Merchant Website Monitoring providers check and monitor an online merchant’s activity and content on their website to recognize and address questionable content that may be against regulatory and legal compliance requirements or may be indicative of high risk. These services are intended for merchant acquirers, PSPs, ISOs and other entities that sell or underwrite merchant.
Key considerations when implementing or buying this functionality include:
If purchasing a Spider or web crawling software, will you have available resources from the provider for setting up and performing data extraction?
How is extracted data delivered back to the business? HTML, XML, Excel file, CSV, TSV, TXT, etc.
Can the spider extract images and files such as PDFs, JavaScript, Flash and AJAX?
If using a Spider or web crawling software be prepared to do the investigation and analysis of website data using internal resources. Businesses looking for investigation and analysis of websites from a vendor should use Merchant Web Site Monitoring services.
Does the web content monitoring service ensure inspected websites comply with Visa’s Global Brand Protection Program and MasterCard’s Business Risk Assessment and Mitigation (BRAM) program?
Can the service ensure a merchant is compliant with laws and regulations in the U.S., EU and all other regions an acquirer may underwrite a merchant?
Does the service only check for compliance with card association rules and legal restrictions or can they check the merchant’s site against acquirer specific criteria?
Does the service provide any information about the website’s history or who owns the domain?
Does the service only perform an initial inspection or will they continue to monitor the merchant and their website?
HOW DOES IT WORK?
Web spiders use an automated process to extract and copy information such as site content, HTML code and other information. In a practical application the user can give a spider a URL or list of sites to visit, it will then visit each of these URLs, copy all the site content and information, identify all hyperlinks on the pages and add each to the list of sites to crawl. It may also be set to re-crawl sites at set intervals. All website information is copied and can be viewed or analyzed offline.
Many spider and web crawling providers offer a hosted software or interface for conducting web crawls. The user can specify the sites to visit, specific information to focus on and look for, as well as the method for selecting and crawling the hyperlinks. There are also services that will take care of creating and automating the data extraction process based off the buyer’s instructions. Two important factors are selecting or isolating the key data to be extracted as well as organizing and exporting this data into a useable format. Depending on the provider users may be setting all of this up on the software or UI themselves, they may use a combination of the hosted software and help with setup from the provider, or they may be working directly with the vendor for a custom project.
Merchant Website Monitoring or website inspections use a combination of automated and human resources to thoroughly comb through a merchant’s website to ensure compliance with laws and regulations, with card association operating procedures and with other terms or agreements an acquirer may have in place with their merchant client. Using both automated and manual checks the service will ensure the merchant is not breaking laws, is in compliance with relevant rules and regulations, and is not performing other high risk activities. Before underwriting a merchant account, an acquirer will want to confirm these qualities of a merchant’s site to properly assess the risk of underwriting this merchant.
The website surveying or monitoring service will ensure the merchant is compliant with Visa’s Global Brand Protection Program (GBPP) and MasterCard’s Business Risk Assessment and Mitigation (BRAM) program. These programs were put in place prevent merchants or merchant services businesses from processing credit card transactions for illegal or unethical goods/services. This includes websites that offer illegal prescription drug sales, counterfeit goods (or any good/service infringing on copyrights), gambling in regions where it is prohibited, tobacco products where they are prohibited, and other illegal or regulated goods or services.
In addition to monitoring a merchant services business’ clients for compliance with laws and regulations, an acquirer will likely have other risk indicators they want to check. If a merchant says they are selling books, for example, their acquirer won’t want to find them selling electronics. The providers can offer on-going monitoring to check the items a merchant is selling, and not only that they are legal and unrestricted items, but they are the types of products/services the acquirer has approved them for. The service may also check product pricing to ensure prices are in-line with the market (very low prices may be a scam), check web content to ensure they aren’t making false claims or using deceptive advertising, and other checks to monitor the risk associated with a merchant’s site.