Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
tompec
on July 31, 2024
|
parent
|
context
|
favorite
| on:
Show HN: Turn any website into a knowledge base fo...
It does respect robots.txt when crawling. I'll add more details about this in the docs.
pryelluw
on July 31, 2024
[–]
I appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
