怎么屏蔽假的百度蜘蛛

自来光

1. 223.111.134.143 - - [19/Jan/2022:10:50:47 +0800] "GET /member/space/person/common/css/css.css HTTP/1.1" 404 146 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"

复制代码

很多，都是国内IP

自来光

我尝试nginx这样写 还是无法识别

1. if ($http_user_agent ~* "spider.html\x09" ) {
   
2.         return 404;
   
3.     }

复制代码

domin

$http_user_agent ~* "spider.html..09"

自来光

domin 发表于 2022-1-20 08:29
$http_user_agent ~* "spider.html..09"

谢谢大佬我试试

自来光

domin 发表于 2022-1-20 08:29
$http_user_agent ~* "spider.html..09"

从结果看 没匹配到

1. 27.159.66.46 - - [21/Jan/2022:06:12:47 +0800] "GET /data/admin/allowurl.txt HTTP/1.1" 503 16559 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
2. 27.159.66.46 - - [21/Jan/2022:06:12:47 +0800] "GET /templets/default/style/dedecms.css HTTP/1.1" 404 146 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
3. 27.159.66.46 - - [21/Jan/2022:06:12:50 +0800] "GET /member/images/base.css HTTP/1.1" 404 146 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
4. 27.159.66.46 - - [21/Jan/2022:06:12:50 +0800] "GET /dede/templets/article_coonepage_rule.htm HTTP/1.1" 503 16559 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
5. 27.159.66.46 - - [21/Jan/2022:06:12:53 +0800] "GET /templets/default/style/dedecms.css HTTP/1.1" 404 146 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
6. 27.159.66.46 - - [21/Jan/2022:06:12:53 +0800] "GET /templets/default/style/dedecms.css HTTP/1.1" 404 146 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
7. 27.159.66.46 - - [21/Jan/2022:06:12:54 +0800] "GET /data/admin/ver.txt HTTP/1.1" 503 16559 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
8. 27.159.66.46 - - [21/Jan/2022:06:12:55 +0800] "GET /data/admin/ver.txt HTTP/1.1" 503 16559 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
9. 27.159.66.46 - - [21/Jan/2022:06:12:56 +0800] "GET /data/cache/index.htm HTTP/1.1" 503 16559 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
   
10. 27.159.66.46 - - [21/Jan/2022:06:12:59 +0800] "GET /data/admin/ver.txt HTTP/1.1" 503 16559 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"
    
11. 27.159.66.46 - - [21/Jan/2022:06:12:59 +0800] "GET /include/data/vdcode.jpg HTTP/1.1" 302 138 "https://www.baidu.com/" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html\x09"

复制代码

konks

user-agent怎么可能防止假蜘蛛，user-agent都是可以随意更改的，要彻底识别只能DNS反解析这个IP，或者保存蜘蛛IP来判断。
比如Baiduspider的hostname以*.baidu.com或*.baidu.jp 的格式命名，非*.baidu.com或*.baidu.jp即为冒充。

host 111.206.198.69
69.198.206.111.in-addr.arpa domain name pointer baiduspider-111-206-198-69.crawl.baidu.com.

konks

你可以先user-agent判断是蜘蛛的IP保存下来，然后批量host查一下，不是的直接屏蔽

jaydenlee

利用国内大厂云的智能解析分流，然后就容易办了

自来光

konks 发表于 2022-1-21 08:20
你可以先user-agent判断是蜘蛛的IP保存下来，然后批量host查一下，不是的直接屏蔽 ...

效率不高，并且这是国内IP
我还不如直接短暂屏蔽IP段