Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 11:42:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
2,162 Commits 273 Branches 1 Tag 48 MiB
ef505f8d99
Commit Graph

1643 Commits

Author SHA1 Message Date
Gergő Móricz
9bdd344b36 fix(redlock): use redlock.using for stability 2024-09-26 22:15:15 +02:00
Gergő Móricz
250c3bb5c6 fix(auth): move redlock settings 2024-09-26 22:15:15 +02:00
Gergő Móricz
81245e68fa fix(auth/redlock): retry cached ACUC lock for 20 seconds 2024-09-26 22:15:15 +02:00
Gergő Móricz
0f89f5e7cb fix(billTeam): cache update race condition 2024-09-26 22:15:15 +02:00
Gergő Móricz
d13a97f979 fix(credit_billing): allow spending of exact credits 2024-09-26 22:15:15 +02:00
Gergő Móricz
84bff8add8 fix(billTeam): update cached ACUC after billing 2024-09-26 22:15:15 +02:00
Gergő Móricz
f22ab5ffaf feat(db): implement bill_team RPC 2024-09-26 22:15:15 +02:00
Gergő Móricz
c1f68c3e0a fix(credit_billing): return chunk.remaining_credits 2024-09-26 22:15:15 +02:00
Gergő Móricz
2073063fb7 fix(db): fix caching and rpc error 2024-09-26 22:15:15 +02:00
Gergő Móricz
f8c70fe5dd feat(db): implement auth_credit_usage_chunk RPC 2024-09-26 22:15:15 +02:00
Gergő Móricz
29815e084b feat(v1/Document): add warning field 2024-09-26 21:19:05 +02:00
Gergő Móricz
095babe70b fix(queue-jobs): jobs with concurrency fails may vanish 2024-09-26 21:18:56 +02:00
Gergő Móricz
b696bfc854 fix(crawl-status): avoid race conditions where crawl may be deemed failed 2024-09-26 21:00:27 +02:00
Gergő Móricz
dec4171937 fix(queue-worker, queue-jobs): logic fixes 2024-09-26 20:39:19 +02:00
Gergő Móricz
d2881927c1 fix(queue-worker): remove concurrency entries when done in sentry-less branch 2024-09-26 20:29:17 +02:00
Gergő Móricz
53fce67ca1 feat(queue-worker): PoC of concurrency limits 2024-09-26 20:24:34 +02:00
Nicolas
30058b1da0 Nick: increased timeout for chrome-cdp due to smart wait 2024-09-26 20:24:34 +02:00
Nicolas
a9773a24a3 Nick: increased timeout for chrome-cdp due to smart wait
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-25 19:27:02 -04:00
Gergő Móricz
953d4fb197 fix(redlock): use redlock.using for stability 2024-09-25 22:47:42 +02:00
Gergő Móricz
eef116bef8 fix(auth): move redlock settings 2024-09-25 22:27:51 +02:00
Gergő Móricz
2c96d2eef6 fix(auth/redlock): retry cached ACUC lock for 20 seconds 2024-09-25 22:25:13 +02:00
Gergő Móricz
1cca9b8ae6 fix(billTeam): cache update race condition 2024-09-25 22:15:02 +02:00
Gergő Móricz
eb7317c08a fix(credit_billing): allow spending of exact credits 2024-09-25 21:44:05 +02:00
Gergő Móricz
e67cbc2ca1 fix(billTeam): update cached ACUC after billing 2024-09-25 21:37:01 +02:00
Gergő Móricz
5a8eb17a82 feat(db): implement bill_team RPC 2024-09-25 20:57:45 +02:00
Gergő Móricz
415fd9f333 fix(credit_billing): return chunk.remaining_credits 2024-09-25 20:37:35 +02:00
Gergő Móricz
417adf8e96 fix(db): fix caching and rpc error 2024-09-25 19:42:45 +02:00
Gergő Móricz
331e826bca feat(db): implement auth_credit_usage_chunk RPC 2024-09-25 19:25:18 +02:00
Nicolas
1da026b26e Update single_url.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-24 23:29:48 -04:00
Nicolas
b8266cc329 Update website_params.ts 2024-09-24 23:28:58 -04:00
Gergő Móricz
f00c0b82f9 fix(v1/scrape): add total wait specified in request to timeout
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-24 21:56:22 +02:00
Nicolas
3f138e559e Update website_params.ts 2024-09-24 15:14:26 -04:00
Gergő Móricz
43730b5db6 feat(WebScraper): always report error of last scraper in order 2024-09-24 20:03:49 +02:00
Gergő Móricz
3e661a2087 fix(v1/crawl-cancel): avoid double authing 2024-09-24 20:01:34 +02:00
Gergő Móricz
4194525640 fix(blocklist): unblock TikTok Business page
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details 
This is just a regular business site, not social media.
 2024-09-24 16:55:19 +02:00
Gergő Móricz
4a623c084a fix(fly): don't use Depot builders (doesn't work) 2024-09-24 10:50:30 +02:00
Gergő Móricz
a59b5836d5 Revert error tallying 2024-09-24 10:27:49 +02:00
Gergő Móricz
a4b128e8b7 fix(rust): blocklisted error test
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-23 23:03:00 +02:00
Gergő Móricz
483f97d21b fix(v0/search): don't sent scrape fail errors to Sentry 2024-09-23 18:49:27 +02:00
Gergő Móricz
d927cafeea fix(queue-worker): don't send scraping errors to sentry 2024-09-23 18:48:01 +02:00
Gergő Móricz
677faa27f3 fix(WebScraper): explicitly ignore 404s 2024-09-23 18:47:07 +02:00
Gergő Móricz
83d8287c14 fix(v0, sentry): don't send all scraping methods failed errors to Sentry 2024-09-23 18:40:21 +02:00
Gergő Móricz
d2f7031069 fix(WebScraper): fatal error handler triggering for 404s 2024-09-23 18:33:10 +02:00
Nicolas
848a2b364a Update package.json 2024-09-21 21:11:23 -04:00
Nicolas
dfdbae74c6 Update fireEngine.ts 2024-09-21 21:10:05 -04:00
Nicolas
fbb5f23016 Update index.ts 2024-09-21 20:53:33 -04:00
Nicolas
607e46267c Update package.json
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-20 19:46:17 -04:00
Nicolas
db161ac55a Nick: press + write 2024-09-20 19:45:23 -04:00
Nicolas
3fc5ce17d2 Nick: fixed error handling for v0 scrape 2024-09-20 18:35:30 -04:00
Nicolas
0690cfeaad Merge branch 'main' into feat/actions 2024-09-20 18:24:13 -04:00
Gergő Móricz
95e4c8920b fix(sdk/rust): license
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-20 21:55:05 +02:00
Gergő Móricz
e1a34b0a99 Revert "feat(scrape): scroll down/up with actions if fullpagescreenshot" 
This reverts commit 815bfc8f07.
 2024-09-20 21:43:22 +02:00
Gergő Móricz
815bfc8f07 feat(scrape): scroll down/up with actions if fullpagescreenshot 
revert this if unneeded
 2024-09-20 21:42:09 +02:00
Gergő Móricz
d663bbf0ca feat(actions): add scroll 2024-09-20 21:41:53 +02:00
Gergő Móricz
3dd912ec91 feat(actions): add typeText, pressKey, fix playwright screenshot/waitFor 2024-09-20 21:02:53 +02:00
Gergő Móricz
719dfbccbb Update docs 2024-09-20 20:30:46 +02:00
Gergő Móricz
939040bf44 Update docs and example 2024-09-20 20:10:11 +02:00
Gergő Móricz
3ec0bbe28d feat(sdk/rust/crawl): paginate through results 2024-09-20 20:10:11 +02:00
Gergő Móricz
a078cdbd9d Rust SDK 1.0.0 2024-09-20 20:10:11 +02:00
Gergő Móricz
93a20442e3 feat(sdk/rust): first batch of changes for 1.0.0 2024-09-20 20:10:11 +02:00
Gergő Móricz
506d5c2716 Revert "return links"
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details 
This reverts commit 2d597672be.
 2024-09-19 20:07:06 +02:00
Anjor Kanekar
2d597672be return links 2024-09-19 20:06:07 +02:00
anjor
c45a132cd5 Remove print statement in map 2024-09-19 20:06:07 +02:00
Gergő Móricz
01f42b980d feat(scrape): add error tallying instead of empty response 2024-09-19 19:21:13 +02:00
Nicolas
b8b8522b33 Nick: fixed map exception 2024-09-19 12:49:33 -04:00
Nicolas
ef71940429
Merge pull request #674 from mendableai/nsc/map-pagination 
v1/map improvements + higher limits
 2024-09-19 11:31:23 -04:00
skeptrune
26825fc8fb bugfix: using onlyIncludeTags and removeTags together 2024-09-19 00:29:13 -07:00
Nicolas
00bb958b32 Update v1-openapi.json 2024-09-18 15:59:11 -04:00
Nicolas
03eb47d426 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-18 15:58:54 -04:00
Nicolas
3a4dd8fc7e Nick: v1 openapi spec 2024-09-18 15:58:53 -04:00
Gergő Móricz
712ca31615 minor fixes 2024-09-18 21:34:09 +02:00
Gergő Móricz
20d1855ad5 feat(js-sdk): actions integration 2024-09-18 20:47:56 +02:00
Gergő Móricz
093c064bff feat(v1): add public actions api 2024-09-18 20:39:25 +02:00
Gergő Móricz
42d677fe3c feat(fire-engine): port waitFor and screenshot to use actions 2024-09-18 20:04:54 +02:00
Gergő Móricz
c28e1e2959 fix(v1/zod): formats: add error message if both sc and sc@fP
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-18 19:35:38 +02:00
Nicolas
255db84879 Update package.json
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-17 15:06:53 -04:00
Gergő Móricz
b2b7f8d874 fix(js-sdk): default type for LLM extract 2024-09-17 20:49:01 +02:00
Nicolas
31633317dd Update package.json 2024-09-17 12:33:07 -04:00
rafaelsideguide
fb8a2c7549 fixed screenshot typo and added test for fullpage screenshot 2024-09-17 11:03:33 -04:00
Nicolas
3c2bfe2da2
Update single_url.ts
Some checks are pending
Fly Deploy / Deploy app (push) Waiting to run
Details
2024-09-17 01:58:47 -04:00
Nicolas
18b024c238 Update single_url.ts 2024-09-17 01:41:46 -04:00
Nicolas
a4039bd008 Revert "Update single_url.ts" 
This reverts commit 0f8c0a570d.
 2024-09-16 23:36:38 -04:00
Nicolas
0f8c0a570d
Update single_url.ts 2024-09-16 21:44:56 -04:00
Nicolas
9da3432596 Update map.ts 2024-09-16 12:03:14 -04:00
Nicolas
af4804e13b Merge branch 'main' into nsc/map-pagination 2024-09-16 11:04:44 -04:00
Nicolas
030ecab6ee Update rate-limiter.ts 2024-09-13 18:09:59 -04:00
Gergo Moricz
000a316cc3 fix(fire-engine): poll more frequently
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
Details
2024-09-13 16:41:27 +02:00
Gergo Moricz
f7c4cee404 fix(queue-worker): don't send LLM extract hallucination error to Sentry 2024-09-13 14:02:49 +02:00
Gergő Móricz
0d1b46d476 fix(js-sdk): improve error logging 2024-09-12 21:30:17 +02:00
Nicolas
e6ac90c1a0 Update package.json 2024-09-12 14:01:47 -04:00
Gergő Móricz
d30356a22c fix(js-sdk): infer keyword collision 2024-09-12 19:57:33 +02:00
Gergő Móricz
d497284b40 feat(api/queue): auto-remove completed jobs after 25 hours 2024-09-12 19:47:22 +02:00
Nicolas
620b02f9ca Nick: 2024-09-12 12:51:14 -04:00
Gergő Móricz
a2903e75cf feat(js-sdk): type-safe LLM extract 2024-09-12 18:48:19 +02:00
Nicolas
e1171ade1b Merge branch 'main' into mog/fix-status-job-get 2024-09-12 11:35:36 -04:00
Nicolas
503c8b3efa Update package-lock.json 2024-09-12 11:35:26 -04:00
Nicolas
e19dc68718 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-11 14:59:40 -04:00
Nicolas
99c1af0a9f Update package.json 2024-09-11 14:59:36 -04:00
Gergő Móricz
5adfd74cc5 feat(js-sdk/test): add API_URL env var 2024-09-11 20:32:34 +02:00
Gergő Móricz
ad70c30be5 fix(js-sdk): check at bad if 2024-09-11 20:31:58 +02:00
Gergő Móricz
6e1cf2f40d feat(js-sdk): fixes, update tests 2024-09-11 20:15:43 +02:00
Nicolas
5791d3a0e5 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-11 14:03:36 -04:00
Nicolas
4cd1065ae2 Update rate-limiter.ts 2024-09-11 14:03:34 -04:00
Gergő Móricz
51da038405
Merge branch 'main' into fix-attw 2024-09-11 19:45:13 +02:00
Gergő Móricz
75f4bcd2ad
Merge pull request #614 from MonsterDeveloper/dependencies 
perf(js-sdk): move `dotenv` and `uuid` to `devDependencies`, fix `zod` import
 2024-09-11 19:34:50 +02:00
Andrei Bobkov
f6fc71b46a
fix(js-sdk): bring back cjs exports 2024-09-11 17:53:17 +03:00
Gergő Móricz
97ffabff3a fix(v1): converting bad docs always gives null 2024-09-10 21:21:20 +02:00
Gergő Móricz
83a165db0f fix(v0/scrape): ensure url is string 2024-09-10 21:18:53 +02:00
Gergő Móricz
ad1a6fbc74 fix(v1/map): handle invalid URLs gracefully 2024-09-10 19:41:01 +02:00
Gergő Móricz
e19f7a102e feat(js-sdk): paginate next on checkCrawlStatus + better types for CSR 2024-09-10 19:29:38 +02:00
Gergő Móricz
4ebc35c9dd fix(crawl-status): add success: true 2024-09-10 18:59:09 +02:00
rafaelsideguide
f855ad3436 bumping py-sdk version 2024-09-10 10:29:44 -03:00
rafaelsideguide
ee8a54213c fix(py-sdk): removed asyncio package 
tested websocket with example.py without asyncio and it works with no problem.
 2024-09-10 10:25:27 -03:00
rafaelsideguide
1074e976cd Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-10 09:09:43 -03:00
rafaelsideguide
45237a29dd updated js-sdk examples 2024-09-10 09:09:39 -03:00
Nicolas
a7406031fa Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-10 06:53:30 -03:00
Nicolas
a17e1cac92 Rate bump 2024-09-10 06:53:24 -03:00
Gergo Moricz
b4dbf75537 fix(v1): check if url is string in blocklistMiddleware 
Fixes FIRECRAWL-SCRAPER-JS-9Z
 2024-09-10 10:25:14 +02:00
Gergo Moricz
26f2095de6 fix(v1): proper Invalid URL handling 2024-09-10 09:24:23 +02:00
Gergo Moricz
f8fbc71f91 fix(supabase-jobs): do not use RPCs 
RPCs are more failure-prone for this use case than regular queries are.
 2024-09-10 09:20:18 +02:00
Gergo Moricz
a6bcf7b438 fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supabase 2024-09-10 08:53:40 +02:00
Nicolas
17e419a7fb Nick: 2024-09-09 21:06:23 -03:00
Nicolas
ca9a781eb7 Update index.ts 2024-09-09 12:27:55 -03:00
Nicolas
22a5e85899 Update index.ts 2024-09-09 12:26:55 -03:00
Nicolas
60a15d00eb Update types.ts 2024-09-08 16:39:12 -03:00
Nicolas
fbdfa1256b Update credit_billing.ts 2024-09-08 13:07:10 -03:00
Nicolas
48c665519e Update credit_billing.ts 2024-09-07 13:42:45 -03:00
Nicolas
32097fa29d Merge branch 'main' of https://github.com/mendableai/firecrawl
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Details
Clean Every 30 Minutes Before 24h Completed Jobs / clean-jobs (push) Has been cancelled
Details
Fly Deploy Direct / Deploy app (push) Has been cancelled
Details
Check Queues / clean-jobs (push) Has been cancelled
Details
Fly Deploy / Test Suite (push) Has been cancelled
Details
Fly Deploy / Python SDK Tests (push) Has been cancelled
Details
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Details
Fly Deploy / Go SDK Tests (push) Has been cancelled
Details
Fly Deploy / Rust SDK Tests (push) Has been cancelled
Details
Fly Deploy / Deploy app (push) Has been cancelled
Details
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish Rust SDK (push) Has been cancelled
Details
2024-09-07 13:12:52 -03:00
Nicolas
5758af3291 Update website_params.ts 2024-09-07 13:12:46 -03:00
Nicolas
79870e7305 Update excludeTags.ts 2024-09-06 20:15:26 -03:00
Nicolas
a0f9ab2be7 Update map.ts 2024-09-06 20:14:47 -03:00
Nicolas
f5b84e15e1 Update sitemap.ts 2024-09-05 17:52:27 -03:00
rafaelsideguide
82d6bf4ec8 feat(go-parser): singleton 2024-09-05 16:14:21 -03:00
Nicolas
cb630bfc34 Update fireEngine.ts 2024-09-05 14:24:10 -03:00
rafaelsideguide
8c1097e9e1 fix: pageOptions 2024-09-05 14:16:31 -03:00
rafaelsideguide
b301ffc922 added missing variables 2024-09-05 13:57:26 -03:00
Nicolas
c6f1d80992 Update crawl.ts 2024-09-05 13:03:43 -03:00
Nicolas
eb03a81152 Update crawl-status.ts 2024-09-05 12:55:04 -03:00
Sebastjan Prachovskij
503e83e83e
Add SearchApi to search 
Add support for engines, improve status code error

Remove changes in package, add engine to env params

Improve description in env example

Remove unnecessary empty line

Improve text
 2024-09-05 18:36:59 +03:00
Nicolas
34adf43200 Merge branch 'main' into feat/go-html-to-md-parser 2024-09-05 12:13:14 -03:00
Nicolas
82cb80c817 Update map.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
Details
2024-09-04 23:46:18 -03:00
Nicolas
a0113dac37 Update credit_billing.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
Details
2024-09-04 16:54:20 -03:00
Nicolas
b7b99b53da Merge branch 'main' into nsc/check-credits-optimization 2024-09-04 16:39:18 -03:00
rafaelsideguide
85b824e122 test: what about false false? 2024-09-04 16:35:32 -03:00
rafaelsideguide
78edf13ec6 test: usedbauth envs wth 2024-09-04 16:31:42 -03:00
rafaelsideguide
cb8571abad fix: enforced dotenv config 2024-09-04 15:57:57 -03:00
Nicolas
5ecb243693 Nick: 2024-09-04 15:19:45 -03:00
rafaelsideguide
3f462eabe9 fix(cicd): 2024-09-04 12:27:46 -03:00
Nicolas
8fc313c1a4 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-04 12:09:36 -03:00
Nicolas
1eb993a93b Update __init__.py 2024-09-04 12:09:28 -03:00
rafaelsideguide
22bf67cc4f Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-04 12:05:09 -03:00
rafaelsideguide
364ba9f901 fix(cicd): mendable->firecrawl and waitfor 2024-09-04 12:05:02 -03:00
Nicolas
a2a63e42cc Rm print map 2024-09-04 12:03:58 -03:00
rafaelsideguide
ad950a6c9d fixed controller res and tests 2024-09-04 11:29:32 -03:00
rafaelsideguide
74ac8915cd details 2024-09-04 09:41:38 -03:00
rafaelsideguide
d836ba6782 added log to check response on cicd 2024-09-04 09:35:56 -03:00
Nicolas
049a11187d Nick: 2024-09-03 21:09:32 -03:00
Nicolas
3072d4a333 Nick: fixed .sort coupons and sentry to withAuth 2024-09-03 21:02:41 -03:00
Nicolas
7561fd279f Nick: debug the billing email system for free credits 2024-09-03 17:08:12 -03:00
rafaelsideguide
c5e1d77a82 added invalid html tests 2024-09-03 15:21:45 -03:00
rafaelsideguide
d60fa6e084 fixed dockerfile and function name. it's working 2024-09-03 14:08:07 -03:00
rafaelsideguide
ebf4035484 added log so we can check 2024-09-03 13:15:21 -03:00
rafaelsideguide
6ccc22ba2f fix(sdk): js next pagination
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
Details
2024-09-03 11:57:19 -03:00
rafaelsideguide
411d7f31c5 fix(sdks): fetch next/pagination 2024-09-03 11:56:24 -03:00
rafaelsideguide
291d9e375b now using compiled go/C lib with koffi 2024-09-03 10:56:07 -03:00
Andrei Bobkov
2b0e447bc2
perf(js-sdk): move dotenv and uuid to devDependencies 2024-09-03 11:13:48 +02:00
Andrei Bobkov
2a8f55e533
perf(js-sdk): remove whole z import and instead use type-only import 2024-09-03 11:12:28 +02:00
Andrei Bobkov
fe8f9d4b2f
feat(js-sdk): drop commonjs outputs and simplify build process 2024-09-03 10:50:52 +02:00
Andrei Bobkov
995a3ff5bb
chore(tsconfig): modernize and remove commonjs 2024-09-03 10:49:59 +02:00
Nicolas
08a9cb8db4 Merge branch 'main' into pr/516 2024-09-02 23:32:23 -03:00
Nicolas
3244454826 Nick: 2024-09-02 19:36:36 -03:00
Nicolas
2444f7c450 Update scrape.ts 2024-09-02 18:51:18 -03:00
Nicolas
00dacaf935 Nick: scrape id when origin is website for report system 2024-09-02 18:43:52 -03:00
rafaelsideguide
48056ea1bd feat: added go html to md parser 2024-09-02 14:15:56 -03:00
rafaelsideguide
036eb925a2 fix(v1): maxDepth 2024-09-02 09:55:35 -03:00
Nicolas
86b0477708 Update map.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
Details
2024-09-01 19:54:15 -03:00
Nicolas
304b3e752e Update map.ts 2024-09-01 19:52:21 -03:00
Nicolas
cb2dfe29be Nick: 2024-09-01 15:06:36 -03:00
Gergő Móricz
980293652d fix(queue-worker): new getJobs, log on v0 2024-09-01 19:29:35 +02:00
Nicolas
44fe741c35 Update queue-worker.ts 2024-09-01 14:19:43 -03:00
Nicolas
758f729ae2 Update queue-worker.ts 2024-09-01 14:10:47 -03:00
Nicolas
5c05bb12a7 Update webhook.ts 2024-09-01 13:58:50 -03:00
Nicolas
ae90370543 Update webhook.ts 2024-09-01 13:58:07 -03:00
Nicolas
0df2441d7f Nick: sdks good 2024-09-01 13:54:41 -03:00
Nicolas
b68a50fea3 Nick: 2024-09-01 13:52:34 -03:00
Nicolas
979697df1c Update queue-worker.ts 2024-09-01 13:47:51 -03:00
Nicolas
95b9dc915d Nick: webhooks v1 working great 2024-09-01 13:44:36 -03:00
Nicolas
faae98ecb8 Merge branch 'main' into v1/webhooks 2024-09-01 12:56:11 -03:00
Nicolas
fe6abe8f7d Nick: 2024-09-01 12:55:02 -03:00
Nicolas
2ef43d5fc2 Update rate-limiter.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
Details
2024-08-31 14:26:16 -03:00
Nicolas
8431be5826 Nick: 2024-08-31 14:23:55 -03:00
Nicolas
af5cc5f16b Nick: 1.2 - v1 llm extract 2024-08-31 13:25:48 -03:00
Nicolas
0c59564393 Merge branch 'main' into v1/llm-extract 2024-08-30 21:31:49 -03:00
Gergő Móricz
693dc14d9b remove invalid keys
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
Details
2024-08-31 00:08:24 +02:00
Nicolas
6d1da2e22a Update index.ts 2024-08-30 18:45:02 -03:00
Nicolas
d347160ff9 Nick: 2024-08-30 17:32:41 -03:00
Nicolas
522d256b11 Merge branch 'main' into v1/llm-extract 2024-08-30 17:16:21 -03:00
Nicolas
ef5c8931eb Update scrape.ts 2024-08-30 17:09:44 -03:00
Nicolas
050cac51ec Update types.ts 2024-08-30 17:02:29 -03:00
Nicolas
012bc74e14 Revert "Update types.ts" 
This reverts commit 77766cf00d.
 2024-08-30 17:02:05 -03:00
Nicolas
77766cf00d Update types.ts 2024-08-30 17:01:54 -03:00
rafaelsideguide
87e61f2d51 v0 working 2024-08-30 16:38:55 -03:00
rafaelsideguide
dc189e1e9d feat: webhooks config on v1 2024-08-30 16:22:59 -03:00
Nicolas
23dfe90baa Nick: 2024-08-30 15:43:31 -03:00
Nicolas
0732997a13 Merge branch 'main' into v1/llm-extract 2024-08-30 15:42:45 -03:00
Gergo Moricz
c8e0bb93ea fix(crawl-status): handle null data when purging rawHtml 2024-08-30 20:40:57 +02:00
Nicolas
4f92bb4bf4 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-08-30 15:29:47 -03:00
Nicolas
282962e36f Nick: 2024-08-30 15:29:41 -03:00
Gergo Moricz
6a6b487474 fix(v1): don't fail on doc = null 2024-08-30 20:28:18 +02:00
Nicolas
71dab56e36 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-08-30 15:21:29 -03:00
Nicolas
d7dbc2536d Update crawl.ts 2024-08-30 15:21:22 -03:00
Nicolas
9de39cfa2c
Merge pull request #591 from mendableai/mog/websocket-sdk 
[v1] Websockets SDKs
 2024-08-30 15:06:06 -03:00
Nicolas
234c6daee8 Update supabase-jobs.ts 2024-08-30 14:52:59 -03:00
Nicolas
7c8ca0c889 Update types.ts 2024-08-30 13:54:04 -03:00
Nicolas
f4a49bc1d1 Update types.ts 2024-08-30 13:49:16 -03:00
Nicolas
a2881e9288 Nick: llm extract support on node sdk 2024-08-30 13:43:19 -03:00
Nicolas
45e33563eb Nick: python working 2024-08-30 12:58:38 -03:00
Nicolas
bb4808443c Merge branch 'main' into v1/llm-extract 2024-08-30 12:48:26 -03:00
Nicolas
2fd9c285bd
Merge pull request #582 from mendableai/fix--rawHtml-return-on-crawl
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
Details 
[Bug] Moved delete rawHtml to end of controller
 2024-08-30 12:40:06 -03:00
Nicolas
b8920d6f4a Nick: 2024-08-30 12:37:45 -03:00
Nicolas
552328d168 Merge branch 'main' into v1/llm-extract 2024-08-30 12:34:50 -03:00
Nicolas
ee3e5dc69c Nick: 2024-08-30 12:34:45 -03:00
Gergő Móricz
26d87bd099
fix(api): handle zoderrors earlier 2024-08-30 17:33:42 +02:00
Nicolas
1805d901a9 Update credit_billing.ts 2024-08-30 12:05:37 -03:00
Nicolas
e5ca4364ba Nick: improvements to llm extract error handling 2024-08-30 11:57:55 -03:00
Nicolas
52ac132328
Update auth.ts 2024-08-30 11:10:48 -03:00
Rafael Miller
636e39d366
Merge branch 'main' into mog/websocket-sdk 2024-08-30 11:09:09 -03:00
rafaelsideguide
ae38c26fa8 feat(v1-sdks): async crawl node, python websocket + async crawl + example 2024-08-30 10:09:39 -03:00
Eric Ciarla
7caaee28c5 Update ui component to v1 2024-08-29 21:40:44 -04:00
Nicolas
41eb620959 Nick: prompt option, still need to convert to new structured outputs 2024-08-29 21:00:57 -03:00
Nicolas
49e1cb7ca0 Nick: 2024-08-29 20:08:06 -03:00
Nicolas
63264644e1 Nick: fixed js-sdk map params
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
Details
2024-08-29 20:06:55 -03:00
rafaelsideguide
377e8ded34 removed v0 support 2024-08-29 17:40:43 -03:00
rafaelsideguide
5f11275fe7 Merge remote-tracking branch 'origin/main' into mog/websocket-sdk 2024-08-29 17:14:59 -03:00
Nicolas
db85f1a7d5 Update index.ts 2024-08-29 17:11:16 -03:00
Rafael Miller
eec6d86802
Merge branch 'main' into f/rust-sdk 2024-08-29 16:20:03 -03:00
Gergő Móricz
53018a683f feat(js-sdk): add crawlUrlAndWatch 2024-08-29 20:01:16 +02:00
Gergő Móricz
d4001e4528 fix(api/websocket): fix auth and termination 2024-08-29 20:01:00 +02:00
Nicolas
1174bc3c82 Nick: blocked sw/ra
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-29 13:55:54 -03:00
rafaelsideguide
7eefaffe48 fix: rawHtml for v0 2024-08-29 12:24:14 -03:00
rafaelsideguide
f8323f9b11 fix: moved delete rawHtml to end of controller 2024-08-29 10:02:05 -03:00
Nicolas
02410aca36 Update runWebScraper.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-28 22:39:50 -03:00
Nicolas
67229c6b3a Nick: fixed credits issue
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-28 18:32:45 -03:00
rafaelsideguide
c3158b0f98 fix(v1): js-sdk fixed crawl type 2024-08-28 16:27:29 -03:00
Nicolas
70bff7f8fb Nick: 2024-08-28 15:59:20 -03:00
Nicolas
8be75accb8 Nick: 2024-08-28 15:40:30 -03:00
rafaelsideguide
c7b3365ffd fix(v1): update readme - v1.0.1 2024-08-28 15:15:29 -03:00
Nicolas
547da88ccd Update rate-limiter.test.ts 2024-08-28 14:53:02 -03:00
Nicolas
4edd9a34c7 Nick: totalCount -> total, completedCount -> completed 2024-08-28 14:46:53 -03:00
Nicolas
2a56f89f50 Update index.test.ts 2024-08-28 14:24:26 -03:00