Make the extraction step explicit and controllable.
Instead of feeding raw HTML into an LLM pipeline, use a bounded crawler to extract Markdown with source metadata. Successful page results can include URL/final URL, HTTP status, title, description, canonical URL, language, headings, links, JSON-LD, crawl depth, text length, and scrape timestamp.
RAG ingestion
Prepare clean source text before chunking, embeddings, retrieval, and citation workflows.
Documentation indexing
Stay on one domain and cap page count/depth for predictable corpus construction.
LLM analysis
Collect cleaner text and metadata for audits, summarization, classification, or extraction.
Automation
Run from Console, REST API, schedules, webhooks, Make/n8n, or MCP.
Start with one page, then expand deliberately.
curl -X POST \
"https://api.apify.com/v2/acts/signal_lab~website-to-markdown-crawler/runs" \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"startUrls": [{"url": "https://example.com"}],
"maxPages": 1,
"maxDepth": 0,
"sameDomainOnly": true,
"useSitemaps": false,
"respectRobotsTxt": true
}'Inspect the Dataset first. Increase maxPages and maxDepth only when a broader crawl is actually required.
Control what enters the RAG corpus.
- sameDomainOnly keeps traversal on the source domain.
- maxPages sets a hard successful-page cap.
- maxDepth bounds link traversal.
- includePatterns / excludePatterns target relevant URL paths.
- removeSelectors can remove additional page noise.
- respectRobotsTxt supports robots-aware crawling.
- useSitemaps enables sitemap discovery when useful.
Expose the crawler as a focused AI tool.
{
"mcpServers": {
"signal-lab-markdown": {
"url": "https://mcp.apify.com?tools=signal_lab/website-to-markdown-crawler"
}
}
}After Apify authentication, an MCP client can request a bounded task such as: “Convert this public documentation site to Markdown, stay on the same domain, and stop after 20 pages.”
Designed as an HTTP crawler, not a bypass tool.
This Actor does not execute client-side JavaScript and does not claim to bypass logins, paywalls, CAPTCHAs, robots rules, or anti-bot controls. Dynamic single-page applications may return incomplete content. Use the live Apify Pricing tab as the source of truth for current cost.