“firecrawl is interesting here specifically bcuz its Search API can return the page content as markdown alongside the search results and that removes a pretty annoying extra scrape step for agent workflows”
“how are you measuring the 10x token reduction for firecrawl? that sounds useful but I'd be curious what you're comparing it against raw HTML, cleaned text, or the highlights?”
“one thing I’d want to know with the firecrawl index is how fresh the GitHub data is. A merged PR from yesterday is potentially much more useful to a coding agent than a perfectly indexed PR from three weeks ago”
“I) Firecrawl developer index is a good option where instead of indexing only doc pages, it indexes 70M+ READMEs, PRs, issues and docs. In the recent DevDex benchmark, it hit 63% overall recall (vs Context7’s 17% overall cuz Context7 misses non-doc queries).”
“Hooking up firecrawl’s search endpoint via a u/function_tool decorator worked really well here where it runs the search and makes the pages down to clean and compact markdown in one call.”
“Anyway, your tips sent me towards the right path: I ended up adding Parallel Search and Firecrawl connectors, and instructing Claude to only use them. Seems to be working for now, and since they are web connectors (HTTP MCPs) they work on mobile and web too.”
“no kiwix or firecrawl retriever in it though, you'd extend BaseSearchEngine and hit kiwix-serve's /search endpoint for previews then /raw/<zim>/content/ for the article body. for the notebooklm side, open-notebook, it's the closest thing with the two speaker audio stuff.”
“So the original open deep research project from Langfuse is archived, are there any other alternatives to use with self hosted models + kiwix and firecrawl?”
“For what you’re doing I’d probably start with Firecrawl. Puppeteer gives you more control, but I’ve found it turns into extra maintenance pretty quickly once you’re dealing with a lot of pages.”
“No, this run used Firecrawl Browser Sandbox over CDP rather than the Scrape API. We created a browser session, connected Playwright, navigated to the URL, waited 60 seconds, and captured a full-page screenshot.”
“I used ZenRows with auto mode and Markdown output, but the same approach should work with Firecrawl, Playwright, or any browser-rendering fetch setup.”
“That's not from "write me a blog." It's the research, Firecrawl for the extra context, and a local knowledge base of our own product so the draft has real details.”
“When I set up my Make workflow, it suggested FireCrawl as a component, but since I didn't have an account with them, it was trying to build with other tools. I don't think those are cutting it.”
“Been using Claude code with firecrawl. Have it set up to go through sources I have given it or it finds on its own. Then it also looks for new sources each run. Works pretty good. Firecrawl is a paid service though.”
“I’d test Firecrawl first based on your requirements. It’s one of the few options where search, crawling, scraping and structured extraction are all part of the same workflow”
“my Hermes agent has access to Firecrawl and some other web scrapers for content extraction, but anyone know of a way to let a local LLM drive a "real" web browser?”