Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Scraper site
open-in-new
<h2 id="examples-of-scraper-websites">Examples of scraper websites</h2> <p><a href="/facts/Search_engine/SRntsbz2">Search engines</a> such as <a href="/facts/Google/GT9Sugza">Google</a> could be considered a type of scraper site. Search engines gather content from other websites, save it in their own databases, index it and present the scraped content to the search engines' own users. The majority of content scraped by search engines is copyrighted.<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a> </p><p>The scraping technique has been used on various dating websites as well. These sites often combine their scraping activities with <a href="/facts/Facial_recognition_system/mtVxaDln">facial recognition</a>.<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a><a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a><a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a><a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a><a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a><a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a><a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a><a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a><a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a><a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a>[<i>excessive citations</i>] </p><p>Scraping is also used on general <a href="/facts/Image_analysis/woLB8cX7">image analysis</a> (recognition) websites, as well as websites specifically made to identify images of crops with pests and diseases.<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a><a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a> </p> <h2 id="made-for-advertising">Made for advertising</h2> <p>Some scraper sites are created to make money by using advertising programs. In such case, they are called <i>Made for <a href="/facts/AdSense/ApWGT7cI">AdSense</a></i> sites or MFA. This derogatory term refers to websites that have no redeeming value except to lure visitors to the website for the sole purpose of clicking on advertisements.<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> </p><p><i>Made for AdSense</i> sites are considered <a href="/facts/Spamdexing/V30auQyv">search engine spam</a> that dilute the search results with less-than-satisfactory search results. The scraped content is redundant compared to content shown by the search engine under normal circumstances, had no MFA website been found in the listings. </p><p>Some scraper sites link to other sites in order to improve their <a href="/facts/Search_engine_optimization/RNzIbb62">search engine ranking</a> through a <a href="/facts/Blog_network/Y1qWwWq9">private blog network</a>. Prior to Google's update to its search algorithm known as <a href="/facts/Google_Panda/VgNopUou">Panda</a>, a type of scraper site known as an <a href="/facts/Spam_blog/FUSQGsxw">auto blog</a> was quite common among black-hat marketers who used a method known as <a href="/facts/Spamdexing/V30auQyv">spamdexing</a>. </p> <h2 id="legality">Legality</h2> <p>Scraper sites may violate <a href="/facts/Copyright_law/7GHEWEx9">copyright law</a>. Even taking content from an <a href="/facts/Open_content/jHYHARSN">open content</a> site can be a <a href="/facts/Copyright_violation/33JDsS5S">copyright violation</a>, if done in a way which does not respect the license. For instance, the <a href="/facts/GNU_Free_Documentation_License/IhVHF3zP">GNU Free Documentation License</a> (GFDL)<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a> and <a href="/facts/Creative_Commons/b1fKDQHE">Creative Commons</a> ShareAlike (CC-BY-SA)<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a> licenses used on Wikipedia<a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> require that a republisher of Wikipedia inform its readers of the conditions on these licenses, and give credit to the original author. </p> <h2 id="techniques">Techniques</h2> <p>Depending upon the objective of a scraper, the methods in which websites are targeted differ. For example, sites with large amounts of content such as airlines, consumer electronics, department stores, etc. might be routinely targeted by their competition just to stay abreast of pricing information. </p><p>Another type of scraper will pull snippets and text from websites that rank high for keywords they have targeted. This way they hope to rank highly in the <a href="/facts/Search_engine_results_page/tpmnjUDa">search engine results pages</a> (SERPs), piggybacking on the original page's <a href="/facts/Page_rank/KtJRJFUX">page rank</a>. <a href="/facts/RSS_(file_format)/WBAWrkdj">RSS</a> feeds are vulnerable to scrapers. </p><p>Other scraper sites consist of advertisements and paragraphs of words randomly selected from a dictionary. Often a visitor will click on a <a href="/facts/Pay_per_click/yBnzc4Mc">pay-per-click</a> advertisement on such site because it is the only comprehensible text on the page. Operators of these scraper sites gain financially from these clicks. Advertising networks claim to be constantly working to remove these sites from their programs, although these networks benefit directly from the clicks generated at this kind of site. From the advertisers' point of view, the networks don't seem to be making enough effort to stop this problem. </p><p>Scrapers tend to be associated with <a href="/facts/Link_farm/Y1qWwWq9">link farms</a> and are sometimes perceived as the same thing, when multiple scrapers link to the same target site. A frequent target victim site might be accused of link-farm participation, due to the artificial pattern of incoming links to a victim website, linked from multiple scraper sites. </p> <h3>Domain hijacking</h3> <p class="note">Main article: <a href="/facts/Domain_hijacking/RqdqloCo">Domain hijacking</a></p> <p>Some programmers who create scraper sites may purchase a recently expired <a href="/facts/Domain_name/FDLD2m5G">domain name</a> to reuse its SEO power in Google. Whole businesses focus on understanding all expired domains and utilising them for their historical ranking ability exist. Doing so will allow SEOs to utilize the already-established <a href="/facts/Backlink/IAn4CgAu">backlinks</a> to the domain name. Some spammers may try to match the topic of the expired site or copy the existing content from the <a href="/facts/Internet_Archive/PHStU8Lo">Internet Archive</a> to maintain the authenticity of the site so that the backlinks don't drop. For example, an expired website about a photographer may be re-registered to create a site about photography tips or use the domain name in their <a href="/facts/Blog_network/Y1qWwWq9">private blog network</a> to power their own photography site. </p><p>Services at some expired domain name registration agents provide both the facility to find these expired domains and to gather the HTML that the domain name used to have on its web site. </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Data_scraping/8zLlaECG">Scraping</a></li> <li><a href="/facts/Contact_scraping/p38pJgrT">Contact scraping</a></li> <li><a href="/facts/Domain_parking/uWBSYzJ1">Domain parking</a></li> <li><a href="/facts/Web_scraping/5R4mf1bd">Web scraping</a></li> <li><a href="/facts/Blog_scraping/dRxMSWqB">Blog scraping</a></li> <li><a href="/facts/Comparison_of_instant_messaging_clients/NPcs51Dw">Multi-protocol messengers</a>: can connect to several networks, yet require to have an account on all of these, so don't violate any terms of the networks</li> <li><a href="/facts/Content_farm/KopzrJy6">Content farm</a></li> <li><a href="/facts/Search_engine_optimization/RNzIbb62">Search engine optimization</a> (SEO)</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Google 'illegally took content from Amazon, Yelp, TripAdvisor,' report finds <a href="https://www.theguardian.com/technology/2015/mar/20/google-illegally-took-content-from-amazon-yelp-tripadvisor-ftc-report" target="_blank">https://www.theguardian.com/technology/2015/mar/20/google-illegally-took-content-from-amazon-yelp-tripadvisor-ftc-report</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>"This App Lets You Find People On Tinder Who Look Like Celebrities". BuzzFeed News. 20 June 2017. Archived from the original on 2023-05-08. <a href="https://www.buzzfeednews.com/article/katienotopoulos/this-app-lets-you-find-people-on-tinder-who-look-like" target="_blank">https://www.buzzfeednews.com/article/katienotopoulos/this-app-lets-you-find-people-on-tinder-who-look-like</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Dating app boss sees ‘no problem’ on face-matching without consent <a href="https://nakedsecurity.sophos.com/2017/06/23/dating-app-boss-sees-no-problem-on-face-matching-without-consent/" target="_blank">https://nakedsecurity.sophos.com/2017/06/23/dating-app-boss-sees-no-problem-on-face-matching-without-consent/</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Dating.ai App Matches You With Celebrity Look-alikes <a href="https://www.elitedaily.com/dating/dating-ai-app-celebrities/1998422" target="_blank">https://www.elitedaily.com/dating/dating-ai-app-celebrities/1998422</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Facial recognition app matches strangers to online profiles <a href="https://www.cnet.com/news/facial-recognition-app-matches-strangers-to-online-profiles/" target="_blank">https://www.cnet.com/news/facial-recognition-app-matches-strangers-to-online-profiles/</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>NameTag: Facial recognition app criticized as creepy and invasive <a href="http://www.cbc.ca/newsblogs/yourcommunity/2014/01/nametag-facial-recognition-app-criticized-as-creepy-and-invasive.html" target="_blank">http://www.cbc.ca/newsblogs/yourcommunity/2014/01/nametag-facial-recognition-app-criticized-as-creepy-and-invasive.html</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Swipe Buster <a href="https://www.vanityfair.com/news/2016/04/check-tinder-cheater-swipe-buster" target="_blank">https://www.vanityfair.com/news/2016/04/check-tinder-cheater-swipe-buster</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Stalker-friendly app, NameTag, uses facial recognition to look you up online <a href="https://nakedsecurity.sophos.com/2014/01/09/stalker-friendly-app-nametag-uses-facial-recognition-to-look-you-up-online/" target="_blank">https://nakedsecurity.sophos.com/2014/01/09/stalker-friendly-app-nametag-uses-facial-recognition-to-look-you-up-online/</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>This Smart (but Unsettling) App Lets You Point Your Phone at People to Find Out Who They Are <a href="https://www.inc.com/minda-zetlin/this-unbelievable-app-is-like-shazam-for-faces.html" target="_blank">https://www.inc.com/minda-zetlin/this-unbelievable-app-is-like-shazam-for-faces.html</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Truly.am Uses Facial Recognition To Help You Verify Your Online Dates <a href="https://techcrunch.com/2013/10/27/truly-am-uses-facial-recognition-to-help-you-verify-your-online-dates/" target="_blank">https://techcrunch.com/2013/10/27/truly-am-uses-facial-recognition-to-help-you-verify-your-online-dates/</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>3 Fascinating Search Engines That Search for Faces <a href="https://www.makeuseof.com/tag/3-fascinating-search-engines-search-faces/" target="_blank">https://www.makeuseof.com/tag/3-fascinating-search-engines-search-faces/</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>"Wolfram has created a website that will identify any image you throw at it". The Verge. 2015-05-14. Archived from the original on 2023-06-03. <a href="https://www.theverge.com/2015/5/13/8603531/wolfram-image-identification-site-trained-by-chewbacca" target="_blank">https://www.theverge.com/2015/5/13/8603531/wolfram-image-identification-site-trained-by-chewbacca</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>Machine Learning Helps Small Farmers Identify Plant Pests And Diseases <a href="https://www.fastcompany.com/40468146/machine-learning-helps-small-farmers-identify-plant-pests-and-diseases" target="_blank">https://www.fastcompany.com/40468146/machine-learning-helps-small-farmers-identify-plant-pests-and-diseases</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>"Made for AdSense". Archived from the original on 2014-02-25. Retrieved 2014-02-19. <a href="https://web.archive.org/web/20140225012536/http://google.about.com/od/m/g/mfadef.htm" target="_blank">https://web.archive.org/web/20140225012536/http://google.about.com/od/m/g/mfadef.htm</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>"Text of the GNU Free Documentation License". <a href="https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_Documentation_License" target="_blank">https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_Documentation_License</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>"Creative Commons Attribution-ShareAlike 3.0 Unported License". <a href="https://creativecommons.org/licenses/by-sa/3.0/legalcode" target="_blank">https://creativecommons.org/licenses/by-sa/3.0/legalcode</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>"Wikipedia:Reusing Wikipedia content". <a href="https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content" target="_blank">https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> </ol>