全球主机交流论坛

 找回密码
 注册

QQ登录

只需一步,快速开始

CeraNetworks网络延迟测速工具IP归属甄别会员请立即修改密码
查看: 1299|回复: 8

[经验] cloudflare Works反代LOC必须把搜索蜘蛛屏蔽掉

[复制链接]
发表于 2023-10-12 00:38:19 | 显示全部楼层 |阅读模式
本帖最后由 lonefly 于 2023-10-12 01:58 编辑

cloudflare Works反代LOC必须把搜索蜘蛛屏蔽掉,特别是googlebot太猛了,2小时就把10W请求干完了

  1. (http.user_agent contains "Googlebot") or (http.user_agent contains "SemrushBot") or (http.user_agent contains "bytedance.com") or (http.user_agent contains "YandexBot") or (http.user_agent contains "Baiduspider") or (http.user_agent contains "bingbot") or (http.user_agent contains "fromBoce") or (http.user_agent contains "cdnunion_monitor") or (http.user_agent contains "MegaIndex") or (http.user_agent contains "CCBot") or (http.user_agent contains "FeedDemon") or (http.user_agent contains "Indy Library") or (http.user_agent contains "Alexa Toolbar") or (http.user_agent contains "AskTbFXTV") or (http.user_agent contains "AhrefsBot") or (http.user_agent contains "CrawlDaddy") or (http.user_agent contains "CoolpadWebkit") or (http.user_agent contains "Java") or (http.user_agent contains "Feedly") or (http.user_agent contains "UniversalFeedParser") or (http.user_agent contains "ApacheBench") or (http.user_agent contains "WebBench") or (http.user_agent contains "Microsoft URL Control") or (http.user_agent contains "Swiftbot") or (http.user_agent contains "ZmEu") or (http.user_agent contains "oBot") or (http.user_agent contains "jaunty") or (http.user_agent contains "Python-urllib") or (http.user_agent contains "lightDeckReports Bot") or (http.user_agent contains "YYSpider") or (http.user_agent contains "DigExt") or (http.user_agent contains "HttpClient") or (http.user_agent contains "MJ12bot") or (http.user_agent contains "heritrix") or (http.user_agent contains "EasouSpider") or (http.user_agent contains "Ezooms") or (http.user_agent contains "BLEXBot") or (http.user_agent contains "serpstatbot") or (http.user_agent contains "DotBot") or (http.user_agent contains "panscient.com") or (http.user_agent contains "python-requests") or (http.user_agent contains "Amazonbot")
复制代码


发表于 2023-10-12 01:11:36 | 显示全部楼层
我以为是有mjj在爬 WAF配置只允许国内ip访问 反正目的就是为了国内访问
发表于 2023-10-12 01:24:26 | 显示全部楼层
cf反代某bbs一直失败,楼主交流下
 楼主| 发表于 2023-10-12 01:52:36 | 显示全部楼层
rem 发表于 2023-10-12 01:11
我以为是有mjj在爬 WAF配置只允许国内ip访问 反正目的就是为了国内访问

万一要国外访问呢?还是屏蔽蜘蛛的好,我把WAF策列表达式贴出来了,直接就可以用
发表于 2023-10-12 01:55:28 | 显示全部楼层
lonefly 发表于 2023-10-12 01:52
万一要国外访问呢?还是屏蔽蜘蛛的好,我把WAF策列表达式贴出来了,直接就可以用 ...

这个好我也加上
cf waf有屏蔽蜘蛛选项 我不知道那个能屏蔽多少
发表于 2023-10-12 02:58:57 | 显示全部楼层
没必要这么复杂 加上bot spider 就过滤了90%以上的爬虫了
 楼主| 发表于 2023-10-12 18:22:57 | 显示全部楼层
机长 发表于 2023-10-12 02:58
没必要这么复杂 加上bot spider 就过滤了90%以上的爬虫了

有用吗?测试了?
发表于 2023-10-12 18:28:17 | 显示全部楼层
lonefly 发表于 2023-10-12 18:22
有用吗?测试了?

你自己看规则 不都是包含bot吗
发表于 2023-10-12 19:10:11 | 显示全部楼层
机长 发表于 2023-10-12 02:58
没必要这么复杂 加上bot spider 就过滤了90%以上的爬虫了

垃圾蜘蛛 DotBot
这表达式区分不区分大小写?
您需要登录后才可以回帖 登录 | 注册

本版积分规则

Archiver|手机版|小黑屋|全球主机交流论坛

GMT+8, 2025-12-28 06:35 , Processed in 0.060049 second(s), 9 queries , Gzip On, MemCache On.

Powered by Discuz! X3.4

© 2001-2023 Discuz! Team.

快速回复 返回顶部 返回列表