txeo/cv-site
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: keep /text crawlable for LLMs, only noindex for search engines

Browse Source
This commit is contained in:
juanatsap
2026-04-09 12:24:41 +01:00
parent 08b39653ba
commit cfb1817daa
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
static/robots.txt
+3 -3
View File
@@ -13,9 +13,9 @@ Disallow: /api/internal/
Disallow: /.git/ Disallow: /.git/
Disallow: /.env Disallow: /.env
# Plain text version — prevent duplicate content indexing # Plain text version — accessible but not indexed (X-Robots-Tag: noindex)
# Canonical HTML version at / should be indexed instead # Canonical HTML version at / is preferred for search results
Disallow: /text # Allow: /text (crawlable for LLMs and text browsers)
# ============================================================================= # =============================================================================
# SITEMAPS & AI CONTENT # SITEMAPS & AI CONTENT