Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 11:42:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
2,239 Commits 273 Branches 1 Tag 48 MiB
nsc/new-extract
Commit Graph

2239 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas
91f4fd815f Update extract.ts 2024-11-14 15:41:42 -05:00
Nicolas
22848af5ae Nick: 2024-11-14 15:34:02 -05:00
Nicolas
ebe9de2ac5 Nick: 2024-11-14 15:26:15 -05:00
Nicolas
5056dcd8e9 Update index.test.ts 2024-11-14 15:06:22 -05:00
rafaelmmiller
be5e6da5ff tests 2024-11-14 17:03:54 -03:00
rafaelmmiller
45091430ab Merge branch 'nsc/new-extract' of https://github.com/mendableai/firecrawl into nsc/new-extract 2024-11-14 17:03:15 -03:00
Nicolas
796cd0746d Update extract.ts 2024-11-14 15:03:06 -05:00
Nicolas
1b5f6a0959 Update extract.ts 2024-11-14 14:59:34 -05:00
Nicolas
d6749c211d Nick: refactor and /* glob pattern support 2024-11-14 14:57:38 -05:00
rafaelmmiller
41b45a844b sdk allowexternallinks 2024-11-14 15:56:12 -03:00
rafaelmmiller
3d6d650f0b Merge branch 'nsc/new-extract' of https://github.com/mendableai/firecrawl into nsc/new-extract 2024-11-14 15:51:32 -03:00
rafaelmmiller
80d6cb16fb sdks wip 2024-11-14 15:51:27 -03:00
Gergő Móricz
359c30fbda fix(cache): don't cache on failure error code 2024-11-14 19:49:34 +01:00
Gergő Móricz
49ff37afb4 feat: cache 2024-11-14 19:47:12 +01:00
Nicolas
a1c018fdb0 Merge branch 'main' into nsc/new-extract 2024-11-13 17:14:43 -05:00
rafaelmmiller
904c904971 wip 2024-11-13 18:06:20 -03:00
Gergő Móricz
0310cd2afa fix(crawl): redirect rebase
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-13 21:38:44 +01:00
Nicolas
0d1c4e4e09 Update package.json 2024-11-13 13:54:22 -05:00
Gergő Móricz
32be2cf786
feat(v1/webhook): complex webhook object w/ headers (#899) 
* feat(v1/webhook): complex webhook object w/ headers

* feat(js-sdk/crawl): add complex webhook support
 2024-11-13 19:36:44 +01:00
Nicolas
ea1302960f
Merge pull request #895 from mendableai/nsc/redlock-email 
Redlock for sending email notifications
 2024-11-13 12:45:55 -05:00
rafaelmmiller
25f32000db mvp done? 2024-11-13 13:05:29 -03:00
rafaelmmiller
a175c1513a wip 2024-11-13 08:09:51 -03:00
Nicolas
1a636b4e59 Update email_notification.ts 2024-11-12 20:09:01 -05:00
Gergő Móricz
5ce4aaf0ec fix(crawl): initialURL setting is unnecessary
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-12 23:35:07 +01:00
Gergő Móricz
93ac20f930 fix(queue-worker): do not kill crawl on one-page error 2024-11-12 22:53:29 +01:00
Gergő Móricz
16e850288c fix(scrapeURL/pdf,docx): ignore SSL when downloading PDF 2024-11-12 22:46:58 +01:00
rafaelmmiller
807703d94c wip 2024-11-12 18:44:14 -03:00
Gergő Móricz
7081beff1f fix(scrapeURL/pdf): retry 2024-11-12 22:26:36 +01:00
Gergő Móricz
9ace2ad071 fix(scrapeURL/pdf): fix llamaparse upload 2024-11-12 20:55:14 +01:00
Gergő Móricz
687ea69621 fix(requests.http): default to localhost baseUrl
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-12 19:59:09 +01:00
Gergő Móricz
3a5eee6e3f feat: improve requests.http using format features 2024-11-12 19:58:07 +01:00
Gergő Móricz
f2ecf0cc36 fix(v0): crawl timeout errors 2024-11-12 19:46:00 +01:00
Nicolas
464b41a5d2 Merge branch 'main' into nsc/new-extract 2024-11-12 12:24:47 -05:00
Nicolas
a23364e5da Update extract.ts 2024-11-12 12:23:44 -05:00
Nicolas
a4f15260a7 Nick: 2024-11-12 12:23:24 -05:00
Gergő Móricz
fbabc779f5
fix(crawler): relative URL handling on non-start pages (#893) 
* fix(crawler): relative URL handling on non-start pages

* fix(crawl): further fixing
 2024-11-12 18:20:53 +01:00
Nicolas
d430cfcbfb Update extract.ts 2024-11-12 12:17:48 -05:00
Nicolas
5bbbb52a30 Update fireEngine.ts 2024-11-12 12:17:03 -05:00
Gergő Móricz
740a429790 feat(api): graceful shutdown for less 502 errors 2024-11-12 18:10:24 +01:00
Nicolas
540d4e5b56 Nick: 2024-11-12 12:10:18 -05:00
Gergő Móricz
c327d688a6 fix(queue-worker): don't log timeouts 2024-11-12 18:10:11 +01:00
Gergő Móricz
9f8b8c190f feat(scrapeURL): log URL for easy searching 2024-11-12 17:54:48 +01:00
Gergő Móricz
e95b6656fa fix(scrapeURL): don't log fetch request 2024-11-12 17:53:44 +01:00
Gergő Móricz
f42740a109 fix(scrapeURL): don't log engineResult 2024-11-12 17:52:32 +01:00
Móricz Gergő
3815d24628 fix(scrape): better timeout handling 2024-11-12 13:16:40 +01:00
Móricz Gergő
aa9a47bce7 fix(queue-worker): logging job on batch scrape error 2024-11-12 13:00:19 +01:00
Móricz Gergő
91f52287db feat(batchScrape): handle timeout 2024-11-12 12:42:39 +01:00
Móricz Gergő
f6db9f1428 fix(crawl-redis): batch scrape lockURL
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-12 11:52:34 +01:00
Gergő Móricz
d8bb1f68c6 fix(tests): maxDepth tests
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-11 22:10:19 +01:00
Gergő Móricz
68c9615f2d fix(crawl/maxDepth): fix maxDepth behaviour 2024-11-11 22:02:17 +01:00