Skip to content
Contact
Paper collage: a bot directory with crawler icons, a spider, a magnifying glass and servers

Ewelina Podrez-Siama · bot profiles

Bytespider ByteDance’s crawler for Toutiao Search: user-agent, IP addresses and robots.txt

Bytespider is ByteDance’s web crawler. ByteDance documents it, briefly and in Chinese, as the crawler of its search engine Toutiao Search, with three user-agent strings, ten IP blocks and a reverse DNS rule.

Below you’ll find what ByteDance documents, the strings that actually reach my server, how to check them, robots.txt and server rules, and how my server treated Bytespider. The variant that sent most requests to my site, the one with spider-feedback@bytedance.com, isn’t described in ByteDance’s documentation. I checked the facts against ByteDance’s pages and the sources below on November 12, 2026.

279

requests claiming to be
Bytespider, Oct 7 – Nov 11, 2026

27

confirmed by
reverse DNS

3

of them got a page
from my server

At a glance

Bytespider at a glance

QuestionAnswer
OperatorByteDance
PurposeByteDance describes it as the crawler of its search engine Toutiao Search. The documentation says nothing about AI training. Vercel’s 2024 measurement counts it among the major AI crawlers, and AI crawler lists, mine included, treat it as collecting data for ByteDance’s AI models.
robots.txt tokenBytespider, the name ByteDance gives in its documentation
Honors robots.txtByteDance documents how Toutiao Search matches robots.txt rules but doesn’t say outright that Bytespider follows them. On my site it almost never requests robots.txt.
IP addressesTen IPv4 /24 blocks listed on ByteDance’s Toutiao webmaster platform (undated).
Reverse DNSPer ByteDance, the host name ends in .bytedance.com, for example bytespider-111-225-148-148.crawl.bytedance.com; any other name means an impersonator. The names have no forward record, so only the reverse half can be checked.
Runs JavaScriptNot documented. A 2024 measurement found that it doesn’t render JavaScript.
Too many requests?ByteDance offers a crawl-frequency setting on its Toutiao webmaster platform and says changes take effect within a day. Contact: zhanzhang@bytedance.com.
In my listAI: training (see the entry in “What bot is this?”)

User-agent

The Bytespider user-agent

ByteDance documents three strings, for desktop, Android and iOS, all ending with a link to zhanzhang.toutiao.com. These are the versions that reached my site:

RequestsUser-agent string
27
confirmed
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
195
not confirmed
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
55
not confirmed
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36

All requests with Bytespider in the user-agent to podrez.pl, Oct 7 – Nov 11, 2026. Confirmed: the address’s reverse DNS name ends in .bytedance.com, ByteDance’s own rule. 2 more requests used rarer variants, not shown here.

Only the Android string with zhanzhang.toutiao.com matches ByteDance’s documentation word for word, and only its 27 requests came from addresses whose reverse DNS names end in .bytedance.com. The other variants include the contact address spider-feedback@bytedance.com, which ByteDance’s documentation doesn’t mention, and none of their addresses had a ByteDance name.

Match Bytespider in the user-agent, ignoring case, and then check the address.

robots.txt

How to block or allow Bytespider in robots.txt

If you want to keep Bytespider out, its token is Bytespider, the name ByteDance uses. ByteDance documents how Toutiao Search matches robots.txt rules but doesn’t promise that Bytespider obeys them, so a server rule is the block you can count on.

Variant 1 · robots.txt

a polite request

I ask Bytespider to stay away, the standard way.

User-agent: Bytespider
Disallow: /
Costs nothing. ByteDance documents how it matches the rules, but doesn’t say outright that Bytespider follows them.

Variant 2 · Server rule

a real block

I refuse every request that calls itself Bytespider (Apache, .htaccess).

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule ^ - [F,L]
On nginx or in a WAF, use the equivalent user-agent rule. It stops anything that calls itself Bytespider, confirmed or not.

If the problem is volume rather than access, ByteDance offers a crawl-frequency setting on its Toutiao webmaster platform and says changes take effect within a day. Questions go to zhanzhang@bytedance.com.

On my site the question is mostly academic: between October 7 and November 11, 2026, my server refused 276 of its 279 requests with a 403 or 406, so Bytespider got almost nothing.

Check your file

Not sure what your robots.txt tells Bytespider? Enter your domain or paste the file into Can AI bots read your site?, my free robots.txt checker. It shows whether Bytespider may crawl your pages and which line decides it.

IP check

Is it really Bytespider? How to check the IP address

ByteDance gives two ways to check. One is its list of ten IPv4 blocks: 110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24 and 60.8.151.0/24. The other is reverse DNS: according to ByteDance, a real Bytespider address resolves to a name ending in .bytedance.com, and any other name means an impersonator.

Quick glossary: the user-agent is the name a bot gives itself, and anyone can type any name. A prefix is a block of IP addresses. Reverse DNS turns an address back into a host name. A WAF (web application firewall) is the filter in front of your site, such as Cloudflare, your host’s firewall or a security plugin.

  1. Take the IP address from the log line.
  2. Check whether it falls within one of the ten blocks, or run host on it (in Terminal on a Mac or Linux): the name should end in .bytedance.com. These names have no forward record, so you can only check the reverse half.
  3. If neither check passes, it’s an impersonator. If it’s causing trouble, block it on the server or in your WAF, not in robots.txt.

Between October 7 and November 11, 2026, none of the 279 requests came from ByteDance’s ten blocks. The 27 requests with the documented Toutiao string came from a China Telecom data center in Beijing, and their reverse DNS names end in crawl.bytedance.com, so by ByteDance’s rule they are confirmed. The other 252, with spider-feedback@bytedance.com in the user-agent, came from Zenlayer (179), Google Cloud (55), AWS China in Beijing (11), ByteDance’s US network (2) and 5 other networks, and none of their addresses had a ByteDance name.

To check a whole log file at once, load it into Check AI bots in your logs, my free log checker. The file never leaves your browser. The tool counts Bytespider’s visits and your server’s answers, but it doesn’t have ByteDance’s blocks, so it can’t confirm them: use reverse DNS for that.

JavaScript

Does Bytespider run JavaScript?

Probably not. ByteDance doesn’t say.

A widely cited measurement by Vercel and MERJ (published December 17, 2024) lists Bytespider among the AI crawlers that don’t render JavaScript.

On my site, no request claiming to be Bytespider asked for a script or a stylesheet between October 7 and November 11, 2026.

Data from my logs

What Bytespider does on my site

Bytespider has been around all year: 968 requests between February 2 and October 6, 2026, about 4 a day, and 279 between October 7 and November 11, 2026, about 8 a day.

It leaves empty-handed. My server answered 248 requests with 406 Not Acceptable and 28 with 403 Forbidden; only 3 received a page. The earlier period looked the same: 6 successful requests out of 968.

It almost never asks for robots.txt: once in my logs from February to early August 2026 and not at all between October 7 and November 11, 2026.

podrez.pl logsFeb 2 – Oct 6, 2026Oct 7 – Nov 11, 2026
Requests claiming to be Bytespider968279
Confirmed requestsnot checked27 (10%)
Confirmed requests per day–1

“Confirmed” means the address is in one of ByteDance’s ten blocks or its reverse DNS name ends in .bytedance.com, the rule from ByteDance’s documentation. I didn’t check the earlier period. The February⁠–⁠October figures are the ones in my list What bot is this?; October⁠–⁠November is a fresh count against ByteDance’s rules as documented on November 12, 2026. Identical log lines are counted once, and each request is assigned to a single bot: the one whose name appears first in the user-agent.

Questions

Frequently asked questions about Bytespider

What is Bytespider?

ByteDance’s web crawler. ByteDance documents it as the crawler of its search engine Toutiao Search. AI crawler lists, mine included, also treat it as a crawler that collects data for ByteDance’s AI models; ByteDance’s documentation doesn’t mention AI.

Does Bytespider respect robots.txt?

ByteDance documents how Toutiao Search matches robots.txt rules but doesn’t say outright that Bytespider follows them. On my site it almost never even requests robots.txt. If you need a reliable block, add a server or WAF rule.

How do I verify Bytespider?

Check the address against ByteDance’s ten IP blocks, or run a reverse DNS lookup: the name should end in .bytedance.com. On my site only the documented Toutiao variant passed; the variant with spider-feedback@bytedance.com never did.

Should I block Bytespider?

If its traffic bothers you, a server rule is the reliable way. Unless your audience uses Toutiao Search – the purpose ByteDance documents – you give up little. If you’d rather slow it down, use ByteDance’s crawl-frequency setting. On podrez.pl my robots.txt lets every bot in, Bytespider included, but my server turns away most of its requests (406 or 403).

Will blocking Bytespider hurt my visibility in Google, Bing or ChatGPT?

No. Those services use their own crawlers, with their own tokens.

Sources

Sources and method

I read all the sources below on November 12, 2026. Figures about my site come from the podrez.pl server logs for February 2 – October 6 and October 7 – November 11, 2026.

  1. ByteDance, Toutiao Search Webmaster Platform: About Bytespider (in Chinese) – three user-agent strings, ten IPv4 blocks, the reverse DNS rule, the crawl-frequency setting and the contact address (undated).
  2. ByteDance, Toutiao Search Webmaster Platform: Robots matching rules (in Chinese) – how Toutiao Search matches robots.txt paths and parameters.
  3. Vercel: The rise of the AI crawler – measurement with MERJ, published December 17, 2024: which AI crawlers render JavaScript.

Compiled by Ewelina Podrez-Siama, working in SEO since 2009. Facts last checked: November 12, 2026.

Other bot profiles: ClaudeBot · GPTBot · OAI-SearchBot · Applebot · Amazonbot · Meta-ExternalAgent · CCBot · PerplexityBot. Or go back to What bot is this?, my searchable list of bots and user-agents.

Check your site

What does your site tell Bytespider?

Two free tools: one shows what your robots.txt tells Bytespider, the other counts its visits in your own server log.

Send a project brief