user-agent: * # Google Ads and Merchant Center fetch these stable public feeds and images. # They return X-Robots-Tag: noindex where applicable, so allowing fetches does # not make them Search result pages. allow: /api/feed allow: /api/feed-img allow: /api/page-feed # Historical links used the website host for public API v3. Let Google crawl # those wrong-host URLs so the dedicated 410 response can remove them from the # index. The real API remains on api.wavespeed.ai. allow: /api/v3/ # Keep browser-only and internal Next.js APIs out of crawler traffic. disallow: /api/ # Private/account pages (/dashboard, /top-up, /share/, /user/, ...) are # intentionally NOT disallowed here: they answer with X-Robots-Tag: noindex # (set in middleware.ts). Blocking them in robots.txt would hide the noindex # from Google and get them indexed as bare URLs ("Indexed, though blocked by # robots.txt"). /sign-in is crawlable AND indexable on purpose. # Sitemaps Sitemap: https://wavespeed.ai/sitemap.xml Sitemap: https://wavespeed.ai/blog/sitemap.xml Sitemap: https://wavespeed.ai/landing/sitemap.xml Sitemap: https://wavespeed.ai/docs/sitemap.xml