Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 19:58:08 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
2,226 Commits 273 Branches 1 Tag 48 MiB
49ff37afb4
Commit Graph

1705 Commits

Author SHA1 Message Date
Gergő Móricz
49ff37afb4 feat: cache 2024-11-14 19:47:12 +01:00
Nicolas
a1c018fdb0 Merge branch 'main' into nsc/new-extract 2024-11-13 17:14:43 -05:00
rafaelmmiller
904c904971 wip 2024-11-13 18:06:20 -03:00
Gergő Móricz
0310cd2afa fix(crawl): redirect rebase
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-13 21:38:44 +01:00
Nicolas
0d1c4e4e09 Update package.json 2024-11-13 13:54:22 -05:00
Gergő Móricz
32be2cf786
feat(v1/webhook): complex webhook object w/ headers (#899) 
* feat(v1/webhook): complex webhook object w/ headers

* feat(js-sdk/crawl): add complex webhook support
 2024-11-13 19:36:44 +01:00
rafaelmmiller
25f32000db mvp done? 2024-11-13 13:05:29 -03:00
rafaelmmiller
a175c1513a wip 2024-11-13 08:09:51 -03:00
Nicolas
1a636b4e59 Update email_notification.ts 2024-11-12 20:09:01 -05:00
Gergő Móricz
5ce4aaf0ec fix(crawl): initialURL setting is unnecessary
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-12 23:35:07 +01:00
Gergő Móricz
93ac20f930 fix(queue-worker): do not kill crawl on one-page error 2024-11-12 22:53:29 +01:00
Gergő Móricz
16e850288c fix(scrapeURL/pdf,docx): ignore SSL when downloading PDF 2024-11-12 22:46:58 +01:00
rafaelmmiller
807703d94c wip 2024-11-12 18:44:14 -03:00
Gergő Móricz
7081beff1f fix(scrapeURL/pdf): retry 2024-11-12 22:26:36 +01:00
Gergő Móricz
9ace2ad071 fix(scrapeURL/pdf): fix llamaparse upload 2024-11-12 20:55:14 +01:00
Gergő Móricz
687ea69621 fix(requests.http): default to localhost baseUrl
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-12 19:59:09 +01:00
Gergő Móricz
3a5eee6e3f feat: improve requests.http using format features 2024-11-12 19:58:07 +01:00
Gergő Móricz
f2ecf0cc36 fix(v0): crawl timeout errors 2024-11-12 19:46:00 +01:00
Nicolas
464b41a5d2 Merge branch 'main' into nsc/new-extract 2024-11-12 12:24:47 -05:00
Nicolas
a23364e5da Update extract.ts 2024-11-12 12:23:44 -05:00
Nicolas
a4f15260a7 Nick: 2024-11-12 12:23:24 -05:00
Gergő Móricz
fbabc779f5
fix(crawler): relative URL handling on non-start pages (#893) 
* fix(crawler): relative URL handling on non-start pages

* fix(crawl): further fixing
 2024-11-12 18:20:53 +01:00
Nicolas
d430cfcbfb Update extract.ts 2024-11-12 12:17:48 -05:00
Nicolas
5bbbb52a30 Update fireEngine.ts 2024-11-12 12:17:03 -05:00
Gergő Móricz
740a429790 feat(api): graceful shutdown for less 502 errors 2024-11-12 18:10:24 +01:00
Nicolas
540d4e5b56 Nick: 2024-11-12 12:10:18 -05:00
Gergő Móricz
c327d688a6 fix(queue-worker): don't log timeouts 2024-11-12 18:10:11 +01:00
Gergő Móricz
9f8b8c190f feat(scrapeURL): log URL for easy searching 2024-11-12 17:54:48 +01:00
Gergő Móricz
e95b6656fa fix(scrapeURL): don't log fetch request 2024-11-12 17:53:44 +01:00
Gergő Móricz
f42740a109 fix(scrapeURL): don't log engineResult 2024-11-12 17:52:32 +01:00
Móricz Gergő
3815d24628 fix(scrape): better timeout handling 2024-11-12 13:16:40 +01:00
Móricz Gergő
aa9a47bce7 fix(queue-worker): logging job on batch scrape error 2024-11-12 13:00:19 +01:00
Móricz Gergő
91f52287db feat(batchScrape): handle timeout 2024-11-12 12:42:39 +01:00
Móricz Gergő
f6db9f1428 fix(crawl-redis): batch scrape lockURL
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-12 11:52:34 +01:00
Gergő Móricz
d8bb1f68c6 fix(tests): maxDepth tests
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-11 22:10:19 +01:00
Gergő Móricz
68c9615f2d fix(crawl/maxDepth): fix maxDepth behaviour 2024-11-11 22:02:17 +01:00
Nicolas
7d576d13bf Update package.json 2024-11-11 15:42:10 -05:00
Gergő Móricz
a8dc75f762
feat(crawl): add parameter to treat differing query parameters as different URLs (#892) 
* add parameter to crawleroptions

* add code to make it work
 2024-11-11 21:36:22 +01:00
Gergő Móricz
5cb46dc494 fix(html-to-markdown): build error 2024-11-11 21:09:27 +01:00
Gergő Móricz
2ca22659d3 fix(scrapeURL/llmExtract): fix schema-less LLM extract 2024-11-11 21:07:37 +01:00
Gergő Móricz
56bebc8107 fix(html-to-markdown): reduce logging frequency 2024-11-11 20:53:21 +01:00
Gergő Móricz
d13a2e7d26 fix(scrapeURL): reduce logs 2024-11-11 20:51:45 +01:00
Nicolas
219f4732a0
Merge pull request #881 from mendableai/fix/scroll-action 
[BUG] fixes scroll action
 2024-11-11 14:50:08 -05:00
Nicolas
ddbf3e45a3
Update package.json 2024-11-11 14:49:50 -05:00
Nicolas
766377621e
Merge pull request #880 from mendableai/python-sdk/next-handler 
[SDK] Added next handler for python sdk (js is ok)
 2024-11-11 14:48:30 -05:00
Nicolas
9688bad60d
Update __init__.py 2024-11-11 14:48:11 -05:00
Nicolas
56a1ac07a4
Merge pull request #878 from mendableai/mog/deduplicate-urls 
feat(crawl): Similar URL deduplication
 2024-11-11 14:33:13 -05:00
Gergő Móricz
8e4e49e471 feat(generateURLPermutations): add tests 2024-11-11 20:29:17 +01:00
rafaelmmiller
e241871b43 fixed scroll action on js sdk 2024-11-11 10:36:02 -03:00
Móricz Gergő
f097cddf23 feat(scrapeURL/fire-engine): adjust timeout for waitFor/wait actions
Some checks are pending
Deploy Images to GHCR / push-app-image (push) Waiting to run
Details
2024-11-11 11:43:59 +01:00