markdown-crawler
markdown-crawler facilitates document extraction by generating markdown files from webpages using multithreaded crawling. It offers threading support, URL validation, and base path customization, making it essential for RAG applications and LLM fine-tuning. Utilizing BeautifulSoup for HTML parsing, the CLI interface allows setting parameters like crawl depth and seamless scraping continuation while featuring verbose logging for monitoring.
