# Robots.txt Generator API

> For the complete documentation index, see [llms.txt](/llms.txt).

Generate a robots.txt from allow/disallow paths, extra user-agent groups, and sitemaps, or validate an existing file for errors and crawler warnings.

Use this endpoint to add the tool to your app, site, workflow, or agent without rebuilding formulas, validation, examples, and docs.

- Slug: `robots-txt-generator`
- Category: `webmaster`
- Version: `2026-04-22`
- Credits per successful call: 1
- Starts on: free
- Max payload: 524288 bytes
- Privacy mode: `no_body_logs`
- Available on: free, starter, pro, business, scale
- Browser tool: https://miniwebtool.com/robots-txt-generator/

## Endpoint

`POST https://api.miniwebtool.com/v1/tools/robots-txt-generator/run`

## Request body

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `mode` | `str` | no | generate | validate Choices: generate, validate |
| `site_url` | `str` | no | site base URL, used to absolutise sitemap paths |
| `default_user_agent` | `str` | no | user agent for the primary group |
| `allow_paths` | `str` | no | Allow paths, one per line |
| `disallow_paths` | `str` | no | Disallow paths, one per line |
| `sitemap_urls` | `str` | no | sitemap URLs or paths, one per line |
| `crawl_delay` | `str` | no | optional Crawl-delay in seconds |
| `host` | `str` | no | optional Host directive |
| `clean_param` | `str` | no | optional Clean-param directives, one per line |
| `comment_note` | `str` | no | optional comment lines placed at the top |
| `custom_agent_blocks` | `str` | no | extra groups, separated by blank lines; first line is the user agent |
| `raw_robots` | `str` | no | existing robots.txt to check when mode=validate |

## Example request

```json
{
  "mode": "generate",
  "site_url": "https://example.com",
  "disallow_paths": "/admin/\n/private/",
  "sitemap_urls": "/sitemap.xml"
}
```

## cURL

```bash
curl -X POST https://api.miniwebtool.com/v1/tools/robots-txt-generator/run \
  -H 'Authorization: Bearer mwt_live_YOUR_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"mode":"generate","site_url":"https://example.com","disallow_paths":"/admin/\n/private/","sitemap_urls":"/sitemap.xml"}'
```

## Response envelope

```json
{
  "request_id": "01K...",
  "tool": "robots-txt-generator",
  "tool_version": "2026-04-22",
  "credits_used": 1,
  "result": {}
}
```

## Errors

Errors are RFC 7807 `application/problem+json`. Branch on `code`, not `title`.

## MCP

Agents can discover and call this tool through `https://api.miniwebtool.com/mcp` or `https://api.miniwebtool.com/v1/mcp` using MCP `tools/list` and `tools/call`.

## Frequently asked

### What does the Robots.txt Generator API do?

Generate a robots.txt from allow/disallow paths, extra user-agent groups, and sitemaps, or validate an existing file for errors and crawler warnings. Use it to add this utility without rebuilding formulas, validation, examples, and documentation.

### How much does one call cost?

1 credit per successful call. Failed calls do not bill.

### What parameters are required?

Required fields: (none). Full schema at `https://api.miniwebtool.com/v1/openapi.json`.

### Is this deterministic?

Yes. Same input, same output. Tool version: 2026-04-22.