Website
Index a documentation site by its address. Public, or behind a token, password or API-key header.
What you will be asked for
- Start URL
- Stay under
- Page limit
- Authentication
- TokenSecret
- Username
- PasswordSecret
- Header name
- Header valueSecret
- Obey robots.txt
Where to find these
Point Worclaude at documentation that lives on the web and it will crawl, chunk and index it.
Start URL
The page to begin from — usually a docs index:
https://docs.example.com/
Path prefix
Keeps the crawler inside one section. With a prefix of /guides, a link to
/blog/hello is ignored. Leave blank to allow anything on the same host.
The crawler never leaves the starting host regardless.
Max pages
A ceiling, so a mistyped prefix cannot crawl for hours. A few hundred covers most documentation sites. Start low; re-syncing with a higher number is cheap.
Authentication
Only if the site is not public.
- Bearer token — paste the token; sent as
Authorization: Bearer …. - Basic — a username and password.
If the site is behind an SSO login page rather than HTTP auth, neither will work: the crawler will index the login page. Publish the docs internally or use a different source.
About a minute, once you have the credentials.