兄弟们 你们都屏蔽了什么爬虫Bot

worigin

这些垃圾爬虫烦死了，还有啥推荐屏蔽的么

sagerking

屏蔽百度

superEric

robots.txt

WAF

worigin

sagerking 发表于 2023-9-9 16:43
屏蔽百度

还没被百度爱过

mengdodo

  if ($http_user_agent ~* (AhrefsBot|SemrushBot|dataforseobot|dotbot|blexbot|wellknownbot|awariobot|phxbot|internet-measurement|gptbot|mj12bot|snapshot)) {
    access_log /dev/null;
    return 403;
  }