733 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Eric Ciarla	32e814bedc	Update index.ts	2024-06-13 17:02:30 -04:00
Nicolas	d48c0df6c5	Merge pull request #274 from mendableai/nsc/clusters ... Clusters support to scale our API to # of CPUs running	2024-06-12 18:09:26 -07:00
Nicolas	182f8d4d6c	Update index.ts	2024-06-12 18:07:05 -07:00
Nicolas	11b6d5afa5	Update fly.toml	2024-06-12 18:00:22 -07:00
Nicolas	67dc46b454	Nick: clusters	2024-06-12 17:53:04 -07:00
Nicolas	bad0f57134	Merge pull request #272 from mendableai/bug/issue-258 ... [Feat] Added jobId to webhook data	2024-06-12 12:29:50 -07:00
rafaelsideguide	d20af257ba	Added jobId to webhook data	2024-06-12 15:38:41 -03:00
Rafael Miller	48f6c19a05	Merge pull request #269 from mendableai/feat/allowbackwardcrawling-option ... [Feat] Added allowBackwardCrawling option	2024-06-12 11:34:39 -03:00
rafaelsideguide	01c9f071fa	fixed	2024-06-12 11:27:06 -03:00
rafaelsideguide	dc6acbf1f0	Merge remote-tracking branch 'origin/main' into feat/allowbackwardcrawling-option	2024-06-12 11:01:05 -03:00
Nicolas	f93231499f	Merge pull request #265 from mendableai/feat/issue-264 ... [Feat] Added route to clean completed jobs and a github action cron that triggers every 24h	2024-06-11 21:33:52 -07:00
Nicolas	705f5ebe78	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-11 15:49:35 -07:00
Nicolas	d4df6f049d	Nick:	2024-06-11 15:49:30 -07:00
Nicolas	45dee63943	Merge pull request #262 from mendableai/nsc/webhook-self-host-fix ... Only fetch webhook from db if self host webhook not set and using db auth	2024-06-11 15:46:57 -07:00
rafaelsideguide	157fbe4a1e	added bull auth key	2024-06-11 17:52:01 -03:00
rafaelsideguide	df3a678cf4	getting back the cancel test, this should work	2024-06-11 17:46:56 -03:00
rafaelsideguide	def2ba9987	added tests	2024-06-11 17:46:25 -03:00
Nicolas	c08db830df	Merge pull request #268 from mendableai/nsc/abs-path-fix ... Fixed bugs associated with absolute path replacements	2024-06-11 13:20:42 -07:00
Nicolas	1e3e06a1d5	Update replacePaths.test.ts	2024-06-11 13:02:39 -07:00
Nicolas	2239e03269	Update replacePaths.test.ts	2024-06-11 12:54:02 -07:00
Nicolas	520739c9f4	Nick: fixed bugs associated with absolute path replacements	2024-06-11 12:43:16 -07:00
Nicolas	788abdce6e	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-11 12:08:51 -07:00
Nicolas	b87725c683	Update openapi.json	2024-06-11 12:08:49 -07:00
rafaelsideguide	ee282c3d55	Added allowBackwardCrawling option	2024-06-11 15:24:39 -03:00
rafaelsideguide	a9f93c2f1e	Added route to clean completed jobs and a github action cron that triggers every 24h	2024-06-11 14:18:05 -03:00
Eric Ciarla	06b0d01fd4	Update examples	2024-06-11 12:23:36 -04:00
Eric Ciarla	00c23855b1	Update examples	2024-06-11 11:46:35 -04:00
Nicolas	da38dad9a7	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-10 18:26:31 -07:00
Nicolas	9390816c1b	Update openapi.json	2024-06-10 18:26:25 -07:00
Nicolas	15e791ffb1	Merge pull request #263 from mendableai/nsc/pageoptions-crawler ... ignoreSitemap feature, pageOptions now respected in the initial crawl as well	2024-06-10 18:22:51 -07:00
Nicolas	f6b06ac27a	Nick: ignoreSitemap, better crawling algo	2024-06-10 18:12:41 -07:00
Nicolas	1bd0327e1a	Merge branch 'main' into nsc/pageoptions-crawler	2024-06-10 17:15:10 -07:00
Nicolas	149d79a529	Merge pull request #260 from mendableai/nsc/fix-deadlocks ... Moving from fetch to axios and preventing deadlocks by setting timeouts on fallbacks	2024-06-10 17:06:33 -07:00
Nicolas	99f2ffd6d5	Update webhook.ts	2024-06-10 17:03:10 -07:00
Nicolas	7ae9778642	Update single_url.ts	2024-06-10 16:57:31 -07:00
Nicolas	913c1dd568	Nick: fetch -> axios and fix timeouts	2024-06-10 16:49:03 -07:00
Nicolas	3091f0134c	Nick:	2024-06-10 16:27:10 -07:00
Nicolas	aafd23fa8a	Merge pull request #252 from mattjoyce/fix-208-py-sdk-interval-poll-name ... Fix 208 py sdk interval poll name	2024-06-08 21:33:17 -07:00
Matt Joyce	8f9a165c2f	Lint - whitespace	2024-06-08 08:03:02 +10:00
Matt Joyce	5f0df596ec	Align param name with JS SDK ... timeout becomes poll_interval	2024-06-08 07:37:08 +10:00
Nicolas	f24ca76618	Nick: removing rate limit emails for now	2024-06-07 10:39:11 -07:00
rafaelsideguide	556c57648e	Update fly.yml	2024-06-07 09:40:40 -03:00
rafaelsideguide	0e89f8b9a3	fixing workflow	2024-06-07 09:35:56 -03:00
Nicolas	98d82c4cec	Update search.ts	2024-06-06 20:02:21 -07:00
Nicolas	5e80f8af87	Nick: llm extract 50	2024-06-06 18:35:44 -07:00
rafaelsideguide	7b7a6f8a39	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-06 17:51:28 -03:00
rafaelsideguide	f2695df215	updated sdk versions	2024-06-06 17:51:12 -03:00
Rafael Miller	8dabf68972	Merge pull request #250 from mendableai/194-fixing-2 ... fixing minor problems on workflow	2024-06-06 17:37:22 -03:00
rafaelsideguide	560f256a35	fixing minor problems on workflow	2024-06-06 17:36:48 -03:00
Rafael Miller	8bfec9e8de	Merge pull request #248 from mendableai/194-sdk-ci-pipeline-for-publishing-pythonnode-sdk ... 194 sdk ci pipeline for publishing pythonnode sdk	2024-06-06 17:28:51 -03:00