Skip to content
Contact
Paper collage: a bot directory with crawler icons, a spider, a magnifying glass and servers

Ewelina Podrez-Siama · bot profiles

Meta-ExternalAgent Meta’s crawler for AI training: user-agent, IP check and robots.txt

Meta-ExternalAgent is Meta’s web crawler for uses such as training foundation AI models or improving products by indexing content directly. Meta says you can block it in robots.txt.

Below you’ll find the strings it sends, how to check that a request comes from Meta, robots.txt rules that keep link previews working, and what it does on podrez.pl. I checked the facts against Meta’s documentation and the sources below on November 12, 2026.

333

requests from Meta’s network
Oct 7 – Nov 11, 2026

62%

of them
were images

0

robots.txt requests
under its own name

At a glance

Meta-ExternalAgent at a glance

QuestionAnswer
OperatorMeta
PurposeIn Meta’s words, it crawls the web “for use cases such as training foundation AI models or improving products by indexing content directly”.
robots.txt tokenmeta-externalagent (matching is case-insensitive, so Meta-ExternalAgent works too)
Without its own groupThe User-agent: * group applies.
Honors robots.txtYes, according to Meta. Its documentation shows how to block it in robots.txt, and the only crawlers it says may skip robots.txt are Meta-ExternalFetcher and facebookexternalhit. Changes can take up to 24 hours.
IP addressesNo list for this crawler. Meta’s addresses belong to its network, AS32934 (1,095 unique IPv4 and IPv6 prefixes in the RADb registry on November 12, 2026).
Reverse DNSNot documented for this crawler.
Runs JavaScriptNot documented. A 2024 measurement found that it doesn’t render JavaScript.
Other Meta crawlersMeta-WebIndexer (Meta AI search), Meta-ExternalFetcher (user-triggered; may bypass robots.txt), Meta-ExternalAds (advertising) and facebookexternalhit (link previews).
In my listAI: training (see the entry in “What bot is this?”)

User-agent

The Meta-ExternalAgent user-agent

Meta’s documentation gives only the short form meta-externalagent/1.1, optionally followed by a link to its crawler page. On my site, requests from Meta’s network wrapped it in a full browser string:

RequestsUser-agent string
195Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
106Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
18Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
10Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Short form in Meta’s docsmeta-externalagent/1.1

Number: requests from Meta’s network (AS32934) to podrez.pl, Oct 7 – Nov 11, 2026. 4 more requests used rarer variants, not shown here.

The operating system and browser at the front vary, but every request from Meta’s network ended with (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)). The impersonators used an older form, Mozilla/5.0 (compatible; Meta-ExternalAgent/1.0; …): 52 requests, all from Google Cloud.

The Robots Exclusion Protocol (RFC 9309) requires crawlers to match the user-agent group without regard to case, so Meta-ExternalAgent and meta-externalagent are the same rule.

robots.txt

How to block or allow Meta-ExternalAgent in robots.txt

Meta asks you to add a Disallow rule for each crawler you want to block. Each Meta crawler has its own token, so blocking AI training doesn’t switch off link previews.

Variant 1 · No AI training

link previews stay

I keep Meta’s AI crawler out but let previews of my links work on Facebook, Instagram and Messenger.

User-agent: meta-externalagent
Disallow: /
Link previews come from facebookexternalhit, which has its own token and keeps working.

Variant 2 · Meta AI search only

training crawler out

I block the training crawler but let Meta’s AI search crawler in.

User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Allow: /
Meta says allowing Meta-WebIndexer helps it cite and link to your content in Meta AI’s responses. It doesn’t say whether that crawler’s data is ever used for training.

Meta says its crawlers may cache robots.txt for up to 24 hours, so allow a day for changes.

Meta-ExternalFetcher, which fetches pages on a user’s request, may bypass robots.txt. If you need it out, block it on the server.

Check your file

Not sure what your robots.txt tells Meta-ExternalAgent? Enter your domain or paste the file into Can AI bots read your site?, my free robots.txt checker. It shows whether Meta-ExternalAgent may crawl your pages and which line decides it.

IP check

Is it really Meta-ExternalAgent? How to check the IP address

Meta publishes no IP list for Meta-ExternalAgent and documents no reverse DNS name for it. What you can check is the network: Meta’s addresses belong to its autonomous system, AS32934. You can list its prefixes in a routing registry, for example with whois -h whois.radb.net -- '-i origin AS32934'. On November 12, 2026, that returned 1,095 unique prefixes.

Quick glossary: the user-agent is the name a bot gives itself, and anyone can type any name. A prefix is a block of IP addresses. Reverse DNS turns an address back into a host name. A WAF (web application firewall) is the filter in front of your site, such as Cloudflare, your host’s firewall or a security plugin.

  1. Take the IP address from the log line.
  2. Look up who owns the address, with whois or any online IP lookup. It should show AS32934 (Facebook). This confirms Meta, not which Meta crawler sent the request.
  3. No match means an impersonator. If it’s causing trouble, block it on the server or in your WAF, not in robots.txt.

Between October 7 and November 11, 2026, 333 of the 387 requests claiming to be Meta-ExternalAgent (86%) came from Meta’s network. The other 54 came from elsewhere, 52 of them from Google Cloud. My server refused 27 with a 403, all of them impersonators.

To check a whole log file at once, load it into Check AI bots in your logs, my free log checker. The file never leaves your browser. Meta publishes no IP list, so the tool counts Meta-ExternalAgent’s visits and what your server answered, but can’t confirm them.

JavaScript

Does Meta-ExternalAgent run JavaScript?

Probably not. Meta doesn’t say.

A widely cited measurement by Vercel and MERJ (published December 17, 2024) lists Meta-ExternalAgent among the AI crawlers that don’t render JavaScript.

On my site, 2 of 333 requests from Meta’s network were for scripts, even though the user-agent looks like a desktop browser.

Data from my logs

What Meta-ExternalAgent does on my site

Meta-ExternalAgent comes in waves. From May to July 2026 it sent 34⁠–⁠44 requests a month; in August, 882; in September, 1,429; and in the first six days of October, 880. Then it calmed down: 333 requests from Meta’s network between October 7 and November 11, 2026, 297 of them in October (with a peak of 117 on October 16, 2026) and just 36 in November.

It’s mostly after images: 208 of its 333 requests. Pages got 57, the WordPress REST API 36, the home page 19, RSS feeds 8 and Markdown versions of my pages 11.

It never requested robots.txt under its own name. The 69 robots.txt requests from Meta’s network in this period came from facebookexternalhit.

podrez.pl logsFeb 2 – Oct 6, 2026Oct 7 – Nov 11, 2026
Requests claiming to be Meta-ExternalAgent4,212387
Confirmed requestsnot checked333 (86%)
Confirmed requests per day–9

“Confirmed” means the IP address belongs to Meta’s network, AS32934. Meta publishes no list for this crawler, so I didn’t check the earlier period. “Content” means any page other than the home page, robots.txt, sitemaps, WordPress system paths (/wp-content/, /wp-includes/, /wp-json/) and .php files. The February⁠–⁠October figures are the ones in my list What bot is this?; October⁠–⁠November is a fresh count against the AS32934 prefixes in RADb on November 12, 2026. Identical log lines are counted once, and each request is assigned to a single bot: the one whose name appears first in the user-agent.

Questions

Frequently asked questions about Meta-ExternalAgent

Does blocking Meta-ExternalAgent stop Facebook link previews?

No. Link previews on Facebook, Instagram and Messenger come from facebookexternalhit, which has its own token.

Should I block Meta-ExternalAgent?

Blocking it keeps your future pages away from the crawler Meta uses for training foundation AI models and indexing content for its products. Link previews (facebookexternalhit) and Meta AI search (Meta-WebIndexer) have their own tokens, so they keep working. On podrez.pl I let all AI bots in, Meta-ExternalAgent included.

What is the difference between Meta-ExternalAgent and Meta-WebIndexer?

Meta-ExternalAgent crawls for uses such as training foundation AI models. Meta-WebIndexer crawls to improve Meta AI search results, and Meta says allowing it helps Meta AI cite and link to your content.

Can I verify Meta-ExternalAgent by IP address?

Only by network. Meta publishes no list for this crawler, so check whether the address belongs to Meta’s AS32934.

Why does Meta-ExternalAgent show up with a Chrome, Edge or Firefox user-agent?

On my site every request from Meta’s network did: a full browser string with compatible; meta-externalagent/1.1 at the end. Meta’s documentation shows only the short form.

How long until Meta follows a robots.txt change?

Up to 24 hours. Meta says its crawlers may cache robots.txt that long.

Sources

Sources and method

I read all the sources below on November 12, 2026. Figures about my site come from the podrez.pl server logs for February 2 – October 6 and October 7 – November 11, 2026.

  1. Meta for Developers: Meta Web Crawlers – Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher, Meta-ExternalAds, FacebookExternalHit, robots.txt (page updated May 21, 2026).
  2. RADb routing registry: route objects for AS32934 – query whois -h whois.radb.net -- '-i origin AS32934', 1,095 unique IPv4 and IPv6 prefixes on November 12, 2026.
  3. RFC 9309: Robots Exclusion Protocol – crawlers must match the user-agent group case-insensitively.
  4. Vercel: The rise of the AI crawler – measurement with MERJ, published December 17, 2024: which AI crawlers render JavaScript.

Compiled by Ewelina Podrez-Siama, working in SEO since 2009. Facts last checked: November 12, 2026.

Other bot profiles: ClaudeBot · GPTBot · OAI-SearchBot · Bytespider · Applebot · Amazonbot · CCBot · PerplexityBot. Or go back to What bot is this?, my searchable list of bots and user-agents.

Check your site

What does your site tell Meta-ExternalAgent?

Two free tools: one reads your robots.txt the way Meta-ExternalAgent would, the other counts its visits in your own server log.

Send a project brief