Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 19:58:08 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
1,314 Commits 273 Branches 1 Tag 48 MiB
fe2e8c0b7a
Commit Graph

1314 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
rafaelsideguide
fe2e8c0b7a includehtml fix 2024-08-21 15:54:00 -03:00
Gergő Móricz
5818236659 fix: remove rawHtml properly 2024-08-20 22:51:12 +02:00
Gergő Móricz
1368f9a87f fix: treat existing screenshot as a scraper success condition 2024-08-20 22:24:18 +02:00
Nicolas
0c48c8a436 Nick: billing for map 2024-08-20 16:43:46 -03:00
Nicolas
39388cdc35 Update crawl.ts 2024-08-20 14:41:43 -03:00
Nicolas
674adee144 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-20 14:41:05 -03:00
Nicolas
b36faeaf54 Nick: 2024-08-20 14:39:52 -03:00
Gergő Móricz
cf32893c2e add strict enforcement + move crawlerOptions to top-level in /crawl 2024-08-20 19:31:26 +02:00
Gergő Móricz
70d50b3640 fix(queue-worker): move dotenv config up 2024-08-20 19:25:19 +02:00
Nicolas
c5ad4dedeb Update crawl.ts 2024-08-20 14:19:20 -03:00
Nicolas
de0dc20a02 Update credit_billing.ts 2024-08-20 14:18:14 -03:00
Nicolas
e200ec9e12 Nick: 2024-08-20 12:24:14 -03:00
Nicolas
55dad82df1 Nick: fixed map search 2024-08-20 12:17:53 -03:00
Nicolas
27903247b6 Nick: map tests and fixes 2024-08-20 12:04:08 -03:00
Nicolas
3dc298be54 Nick: 2x rate limits for standard and growth for /scrape 2024-08-19 13:52:54 -03:00
rafaelsideguide
72461ce9a6 Update index.test.ts 2024-08-19 13:29:52 -03:00
rafaelsideguide
fd7fdc1d52 added blocklist middleware 2024-08-19 13:28:54 -03:00
Nicolas
ff84f1fe5e Update map.ts 2024-08-16 20:42:36 -04:00
Nicolas
4314313477 Update map.ts 2024-08-16 19:56:18 -04:00
Nicolas
af9a0a6f0b Update map.ts 2024-08-16 19:56:03 -04:00
Nicolas
ba5279eafc Nick: all tests passing 2024-08-16 19:55:44 -04:00
Nicolas
5205c5f005 Update map.ts 2024-08-16 19:37:00 -04:00
Nicolas
0c05d096a9 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 19:33:58 -04:00
Nicolas
ab48353226 Nick: /map almost good 2024-08-16 19:33:57 -04:00
Gergő Móricz
eb84673b06 feat: crawl status websocket WIP 2024-08-17 01:04:14 +02:00
Gergő Móricz
e2a6ef26d3 mount v1Router under v1 path 2024-08-16 23:48:50 +02:00
Gergő Móricz
4c1b74dab3 fix(map): remove robots.txt 2024-08-16 23:46:10 +02:00
Gergő Móricz
803577eeba feat(crawl): webhook 2024-08-16 23:42:48 +02:00
rafaelsideguide
086ba6280b fixed markdown format 2024-08-16 18:39:13 -03:00
Gergő Móricz
aabfaf0ac5 clean up crawl-status, fix db ddos 2024-08-16 23:29:39 +02:00
rafaelsideguide
e5b807ccc4 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 17:57:31 -03:00
rafaelsideguide
b311fe1898 Create temp-37564.rdb 2024-08-16 17:57:14 -03:00
rafaelsideguide
7a61325500 map + search + scrape markdown bug 2024-08-16 17:57:11 -03:00
Gergő Móricz
5896153d19 fix: crawl status and redis fixes 2024-08-16 22:52:48 +02:00
Gergő Móricz
3fcb21930e remove log 2024-08-16 22:48:23 +02:00
Gergő Móricz
f20328bdbb crawl status and document stuff 2024-08-16 22:48:05 +02:00
Nicolas
0c057bb649 Update index.test.ts 2024-08-16 16:45:10 -04:00
Nicolas
b32464558a Update index.test.ts 2024-08-16 16:41:09 -04:00
Nicolas
5bac7988a6 Update index.test.ts 2024-08-16 16:08:38 -04:00
Nicolas
290c7ee936 Update index.test.ts 2024-08-16 16:06:46 -04:00
Nicolas
23a033fe61 Nick: fixes and more e2e tests 2024-08-16 16:03:35 -04:00
Nicolas
37ae9a9043 Update index.test.ts 2024-08-16 14:17:43 -04:00
Nicolas
200ce8e2ce Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 14:16:35 -04:00
Nicolas
21d3798e49 Nick: initial e2e v1 tests for /scrape 2024-08-16 14:16:30 -04:00
rafaelsideguide
3f998b688d scrape ready 2024-08-16 15:14:37 -03:00
Nicolas
b0d211ecc1 Merge branch 'main' into v1-webscraper 2024-08-16 13:43:28 -04:00
Gergő Móricz
fd6432e7fd fix(queue-worker): correct job success 2024-08-16 19:16:08 +02:00
Gergő Móricz
6e54942265 fix(queue-worker): add cancelled to crawl log 2024-08-16 19:11:53 +02:00
rafaelsideguide
9b1cb266a0 added origin to request types 2024-08-16 13:49:50 -03:00
Gergő Móricz
d0a8382a5b fix(queue-worker): crawl finishing race condition 2024-08-16 18:48:52 +02:00