986 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
sanix-darker	ff62b26001	feat: regroup what could be regroup in terms of environments variables between services (api and worker)	2024-06-30 15:22:21 +02:00
Eric Ciarla	7ae195bacc	Update index.test.ts	2024-06-29 10:13:12 -04:00
Eric Ciarla	837b446390	Update index.test.ts	2024-06-29 08:48:42 -04:00
Eric Ciarla	fe6e3aeadc	Update index.test.ts	2024-06-29 08:44:21 -04:00
Eric Ciarla	6c9f0dfc91	Add tests	2024-06-29 08:32:20 -04:00
Jeff Pereira	a5fb45988c	new feature allowExternalContentLinks	2024-06-28 17:23:40 -07:00
Eric Ciarla	87b54488d3	update to includeRawHtml	2024-06-28 17:07:47 -04:00
Eric Ciarla	70fcf2ce03	init	2024-06-28 16:39:09 -04:00
Nicolas	9bf74bc774	Update single_url.ts	2024-06-28 15:51:18 -03:00
Nicolas	7e17498bcf	Update single_url.ts	2024-06-28 15:45:16 -03:00
rafaelsideguide	7dffaaa3e2	Changed port and added "using with firecrawl" section on readme	2024-06-28 11:51:24 -03:00
rafaelsideguide	d66e1f7846	looking good	2024-06-27 16:00:45 -03:00
Nicolas	dbfae2d9bf	Merge pull request #329 from george-zakharov/patch-1 ... Update CONTRIBUTING.md	2024-06-27 12:12:31 -03:00
George Zakharov	5a0ec070bf	Update CONTRIBUTING.md ... Fix typo	2024-06-27 13:50:31 +04:00
Nicolas	017b0b2556	Merge pull request #328 from mendableai/nsc/includeOnlyTags ... pageOptions.onlyIncludeTags param	2024-06-26 21:33:10 -03:00
Nicolas	9e7298945c	Update openapi.json	2024-06-26 21:25:38 -03:00
Nicolas	1ec0bf8adf	Update openapi.json	2024-06-26 21:22:46 -03:00
Nicolas	042f81ddf2	Update removeUnwantedElements.test.ts	2024-06-26 21:20:11 -03:00
Nicolas	388ce3cbce	Nick: small changes	2024-06-26 21:15:42 -03:00
Nicolas	1d4907acc9	Nick:	2024-06-26 21:02:58 -03:00
rafaelsideguide	c40da77be0	Added implementation for saving docs on supabase ... - TODO: remove the comments on `log_job.ts` before deploying to prod	2024-06-26 18:23:28 -03:00
Jeff Pereira	d833a132a5	new playwright service	2024-06-26 12:32:30 -07:00
Nicolas	3b92fb8433	Merge pull request #322 from mendableai/tests/metadata ... [Test] Added E2E tests for checking metadata values	2024-06-26 12:09:18 -03:00
rafaelsideguide	67d7650cf3	Added to e2e_noAuth	2024-06-26 12:07:55 -03:00
Nicolas	ac08e20c33	Merge pull request #321 from mendableai/bug/fix-issue-310 ... [Bug] Added default values and fixed pdf bug	2024-06-26 11:50:42 -03:00
Eric Ciarla	d80046d17c	Gemini caching example	2024-06-26 09:48:15 -04:00
rafaelsideguide	009df6c930	Added crawl limit unit test ... I think this test is over relying on mocks but I have no idea on how to fix this without changing the code arch structure	2024-06-26 09:54:25 -03:00
rafaelsideguide	05eaa3c68d	Update index.test.ts	2024-06-26 09:32:02 -03:00
rafaelsideguide	4381109dd8	added default values and fixed pdf bug	2024-06-26 09:00:54 -03:00
Nicolas	45f2765601	Merge pull request #316 from snippet/types-webscraper ... add some types	2024-06-25 22:03:21 -03:00
Nicolas	768a131b5c	Merge pull request #318 from mendableai/bug/fix-custom-scrape-pdf-google-drive ... [Bug] Fixed the regex test for google drive pdf files	2024-06-25 18:27:11 -03:00
rafaelsideguide	5f69fc7677	Fixed the regex test	2024-06-25 18:24:01 -03:00
Nicolas	dbb22c8f0d	Merge pull request #317 from mendableai/bug/fix-clean-jobs ... [Bug] Fixed clean jobs	2024-06-25 17:50:55 -03:00
rafaelsideguide	d02829d335	fixed clean jobs	2024-06-25 17:49:29 -03:00
Jeff Pereira	199cbe8bcb	add some types	2024-06-25 12:20:25 -07:00
Nicolas	749b0c05dc	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-25 15:21:15 -03:00
Nicolas	e7be17db92	Nick: metadata fixes and lock duration for bull decreased to 2 hrs	2024-06-25 15:21:14 -03:00
Nicolas	f84fb4b331	Merge pull request #313 from snippet/google-search-term-fix ... fix multi-word search term issue: /search (w/o Serp)	2024-06-24 19:24:58 -03:00
Jeff Pereira	6ddf3a58a1	fix multi-word search term issue: /search (w/o Serp)	2024-06-24 14:21:52 -07:00
Nicolas	e5314ee8e7	Merge pull request #312 from mendableai/rafa/investigating-crawl-bugs ... [Bug] Fixed axios bug that were making jobs stuck on active queue	2024-06-24 16:52:34 -03:00
Nicolas	90b7fff366	Update crawler.ts	2024-06-24 16:52:01 -03:00
Nicolas	08c1fa799b	Update queue-worker.ts	2024-06-24 16:51:32 -03:00
rafaelsideguide	3ebdf93342	removed console.logs	2024-06-24 16:43:12 -03:00
Nicolas	56d42d9c9b	Nick:	2024-06-24 16:33:07 -03:00
rafaelsideguide	21d29de819	testing crawl with new.abb.com case ... many unnecessary console.logs for tracing the code execution	2024-06-24 16:25:07 -03:00
Nicolas	3c7b7e7242	NIck: fixes fallback	2024-06-23 18:59:08 -03:00
Nicolas	b394e64684	Merge pull request #308 from 100gle/fix-typo	2024-06-23 11:14:26 -04:00
Xiaoyue Lin	3624ed20f9	docs: Fix pydanti to pydantic	2024-06-23 22:27:48 +08:00
Eric Ciarla	22541362d7	Reduce web example bloat	2024-06-22 08:40:26 -04:00
Eric Ciarla	8e39083d8c	Update examples section	2024-06-21 15:40:46 -04:00