Web fetching

Mirepoix opens a web page only when someone pastes a link into Mirabel’s chat and asks for the recipe. Those requests name themselves Mirepoix-Import. This page has two readers: a cook whose link was declined, and a site operator who found that name in a server log.

If Mirabel declined your link

Before Mirabel opens a page, Mirepoix reads that site’s robots.txt — the file where a site says which pages automated readers may open. When a rule there covers the page you linked, she doesn’t open it, and tells you the site asked her not to.

That’s the site’s stated preference, not a fault in your link, so there’s nothing to retry. Two ways still work:

Use the import bookmarklet. It reads the page in your own browser, where you’re a person reading a page rather than an automated reader. You set it up once in Settings.
Paste the recipe text into the chat and Mirabel builds the draft from that.

A site that turns automated readers away outright is a different thing with the same two fixes — see importing from a link.

For site operators

Every page Mirepoix opens carries this User-Agent:

Mirepoix-Import/1.0 (+https://help.mirepoix.recipes/web-fetching/)

The product token is Mirepoix-Import. Address that name in robots.txt or in a rate limit.

The traffic is user-initiated

Mirepoix is not a crawler. A request happens because a person signed in to Mirepoix pasted that link into a chat and asked for the recipe. Mirepoix opens the page they sent and stops there. It doesn’t follow links out of the page, doesn’t queue anything to visit later, and won’t open an address the person didn’t send — a link discovered inside a fetched page is never a target.

Each fetch is a GET over HTTP or HTTPS on the standard ports, preceded by a GET of your robots.txt when Mirepoix hasn’t read it in the past hour. It follows a handful of redirects at most, stops reading a page larger than 2 MB, and gives up on a page that’s slow to answer.

What Mirepoix honors in robots.txt

Mirepoix requests /robots.txt from the same origin as the page, with the same User-Agent, and declines any path a matching rule disallows. If the page redirects, the target is checked the same way before it’s opened — a redirect doesn’t get past the rules. It reads:

The group naming Mirepoix-Import. With no such group, the User-agent: * group applies.
Allow and Disallow lines in that group. The longest matching pattern wins; a tie goes to Allow.
* and $ wildcards in a pattern.
An empty Disallow:, meaning nothing is disallowed.

It ignores Crawl-delay and Sitemap, and reads the first 500 KB of a very long file.

Mirepoix caches a robots.txt it could read for an hour per origin, so a site normally sees at most one request an hour for it. A file it couldn’t fetch isn’t cached: each import asks again until the file can be read, so a rule added during a bad minute at the origin still takes effect on the next try.

It also fails open. If /robots.txt is missing, returns an error, or doesn’t answer, Mirepoix treats the site as unrestricted and opens the page. This is one page a person asked for, not a crawl, so a robots.txt that’s briefly unreachable doesn’t break their import.

Turn Mirepoix away

Add a group to robots.txt:

User-agent: Mirepoix-Import
Disallow: /

Within an hour, Mirepoix declines every page on the origin and tells the person who pasted the link to bring the recipe another way. To keep it out of one section, disallow that path instead of /.

A User-agent: * group whose rules already cover the path turns Mirepoix away too. A group naming Mirepoix-Import is what overrides it — so use one when you want Mirepoix treated differently from other automated readers.