Blocking Bots in IIS

Recently I had an application on apache become the victim of bot spam. As a good measure I set out to implement the same protection on a Windows Server running IIS 7.5.

The web is something on the order of 60% bot traffic, many of these are inconsequential and can safely be blocked. I chose to block them based on user agent, since many of these bots have a range of IP addresses they can utilize.

First off you will need to make sure you have the URL rewrite module added to your installation of IIS. The version I am writing this for is IIS 7, the process should be similar for other versions.

(Microsoft URL Rewrite Module 2.0 for IIS 7 (x64))

I checked that this Rewrite Module will work for both IIS 7.0 and 7.5.

After installing the module, restart your IIS Manager and click your server on the left hand side. It will be the very top options after “Start Page”. You can see it in the image below.

Once URL Rewrite is enabled on your web server. Next, click on “Add Rules…” from the Actions pane.

You will see a window open with the below information. Click on request blocking, then click “OK”.

You will then be prompted with choosing the settings for your rule.

Select User-agent Header for the “block access based on” field.

Select Using: regular expressions

Then enter your pattern.

I used the below pattern for my rule. The top listed one, “^$” is the regex for an empty string. I do not allow bots to access the pages unless they identify with a user-agent, I found most often the only things hitting my these applications with out a user agent were security tools gone rogue.

1

^$|EasouSpider|Add Catalog|PaperLiBot|Spiceworks|ZumBot|RU_Bot|Wget|Java/1.7.0_25|Slurp|FunWebProducts|80legs|Aboundex|AcoiRobot|Acoon Robot|AhrefsBot|aihit|AlkalineBOT|AnzwersCrawl|Arachnoidea|ArchitextSpider|archive|Autonomy Spider|Baiduspider|BecomeBot|benderthewebrobot|BlackWidow|Bork-edition|Bot mailto:craftbot@yahoo.com|botje|catchbot|changedetection|Charlotte|ChinaClaw|commoncrawl|ConveraCrawler|Covario|crawler|curl|Custo|data mining development project|DigExt|DISCo|discobot|discoveryengine|DOC|DoCoMo|DotBot|Download Demon|Download Ninja|eCatch|EirGrabber|EmailSiphon|EmailWolf|eurobot|Exabot|Express WebPictures|ExtractorPro|EyeNetIE|Ezooms|Fetch|Fetch API|filterdb|findfiles|findlinks|FlashGet|flightdeckreports|FollowSite Bot|Gaisbot|genieBot|GetRight|GetWeb!|gigablast|Gigabot|Go-Ahead-Got-It|Go!Zilla|GrabNet|Grafula|GT::WWW|hailoo|heritrix|HMView|houxou|HTTP::Lite|HTTrack|ia_archiver|IBM EVV|id-search|IDBot|Image Stripper|Image Sucker|Indy Library|InterGET|Internet Ninja|internetmemory|ISC Systems iRc Search 2.1|JetCar|JOC Web Spider|k2spider|larbin|larbin|LeechFTP|libghttp|libwww|libwww-perl|linko|LinkWalker|lwp-trivial|Mass Downloader|metadatalabs|MFC_Tear_Sample|Microsoft URL Control|MIDown tool|Missigua|Missigua Locator|Mister PiX|MJ12bot|MOREnet|MSIECrawler|msnbot|naver|Navroad|NearSite|Net Vampire|NetAnts|NetSpider|NetZIP|NextGenSearchBot|NPBot|Nutch|Octopus|Offline Explorer|Offline Navigator|omni-explorer|PageGrabber|panscient|panscient.com|Papa Foto|pavuk|pcBrowser|PECL::HTTP|PHP/|PHPCrawl|picsearch|pipl|pmoz|PredictYourBabySearchToolbar|RealDownload|Referrer Karma|ReGet|reverseget|rogerbot|ScoutJet|SearchBot|seexie|seoprofiler|Servage Robot|SeznamBot|shopwiki|sindice|sistrix|SiteSnagger|SiteSnagger|smart.apnoti.com|SmartDownload|Snoopy|Sosospider|spbot|suggybot|SuperBot|SuperHTTP|SuperPagesUrlVerifyBot|Surfbot|SurveyBot|SurveyBot|swebot|Synapse|Tagoobot|tAkeOut|Teleport|Teleport Pro|TeleportPro|TweetmemeBot|TwengaBot|twiceler|UbiCrawler|uptimerobot|URI::Fetch|urllib|User-Agent|VoidEYE|VoilaBot|WBSearchBot|Web Image Collector|Web Sucker|WebAuto|WebCopier|WebCopier|WebFetch|WebGo IS|WebLeacher|WebReaper|WebSauger|Website eXtractor|Website Quester|WebStripper|WebStripper|WebWhacker|WebZIP|WebZIP|Wells Search II|WEP Search|Widow|winHTTP|WWWOFFLE|Xaldon WebSpider|Xenu|yacybot|yandex|YandexBot|YandexImages|yBot|YesupBot|YodaoBot|yolinkBot|youdao|Zao|Zealbot|Zeus|ZyBORG

view rawbot-blocking-expression

I will advise you when blocking bots be very specific. Simply using a generic word like “fire” could pop positive for “firefox” You can also adjust the regex to fix that issue but I found it much simpler to be more specific and that has the added benefit of being more informative to the next person to touch that setting.

Additionally, you will see I have a rule for Java/1.7.0_25 in this case it happened to be a bot using this version of java to slam my servers. Do be careful blocking language specific user agents like this, some languages such as ColdFusion run on the JVM and use the language user agent and web requests to localhost to assemble things like PDFs. Jruby, Groovy, or Scala, may do similar things, however I have not tested them.

Below you will see a full list of all bots that are blocked by the above regex.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230

^$ EasouSpider Add Catalog PaperLiBot Spiceworks ZumBot RU_Bot Wget Java/1.7.0_25 Slurp FunWebProducts 80legs Aboundex AcoiRobot Acoon Robot AhrefsBot aihit AlkalineBOT AnzwersCrawl Arachnoidea ArchitextSpider archive Autonomy Spider Baiduspider BecomeBot benderthewebrobot BlackWidow Bork-edition Bot mailto:craftbot@yahoo.com botje catchbot changedetection Charlotte ChinaClaw commoncrawl ConveraCrawler Covario crawler curl Custo data mining development project DigExt DISCo discobot discoveryengine DOC DoCoMo DotBot Download Demon Download Ninja eCatch EirGrabber EmailSiphon EmailWolf eurobot Exabot Express WebPictures ExtractorPro EyeNetIE Ezooms Fetch Fetch API filterdb findfiles findlinks FlashGet flightdeckreports FollowSite Bot Gaisbot genieBot GetRight GetWeb! gigablast Gigabot Go-Ahead-Got-It Go!Zilla GrabNet Grafula GT::WWW hailoo heritrix HMView houxou HTTP::Lite HTTrack ia_archiver IBM EVV id-search IDBot Image Stripper Image Sucker Indy Library InterGET Internet Ninja internetmemory ISC Systems iRc Search 2.1 JetCar JOC Web Spider k2spider larbin larbin LeechFTP libghttp libwww libwww-perl linko LinkWalker lwp-trivial Mass Downloader metadatalabs MFC_Tear_Sample Microsoft URL Control MIDown tool Missigua Missigua Locator Mister PiX MJ12bot MOREnet MSIECrawler msnbot naver Navroad NearSite Net Vampire NetAnts NetSpider NetZIP NextGenSearchBot NPBot Nutch Octopus Offline Explorer Offline Navigator omni-explorer PageGrabber panscient panscient.com Papa Foto pavuk pcBrowser PECL::HTTP PHP/ PHPCrawl picsearch pipl pmoz PredictYourBabySearchToolbar RealDownload Referrer Karma ReGet reverseget rogerbot ScoutJet SearchBot seexie seoprofiler Servage Robot SeznamBot shopwiki sindice sistrix SiteSnagger SiteSnagger smart.apnoti.com SmartDownload Snoopy Sosospider spbot suggybot SuperBot SuperHTTP SuperPagesUrlVerifyBot Surfbot SurveyBot SurveyBot swebot Synapse Tagoobot tAkeOut Teleport Teleport Pro TeleportPro TweetmemeBot TwengaBot twiceler UbiCrawler uptimerobot URI::Fetch urllib User-Agent VoidEYE VoilaBot WBSearchBot Web Image Collector Web Sucker WebAuto WebCopier WebCopier WebFetch WebGo IS WebLeacher WebReaper WebSauger Website eXtractor Website Quester WebStripper WebStripper WebWhacker WebZIP WebZIP Wells Search II WEP Search Widow winHTTP WWWOFFLE Xaldon WebSpider Xenu yacybot yandex YandexBot YandexImages yBot YesupBot YodaoBot yolinkBot youdao Zao Zealbot Zeus ZyBORG