Commit Graph

728 Commits

Author SHA1 Message Date
rafaelsideguide
a20d002a6b Delete test-run-report.json 2024-06-17 09:25:29 -03:00
Eric Ciarla
519ab1aecb Update unit tests 2024-06-15 17:14:09 -04:00
Eric Ciarla
f0d4146b42 Merge branch 'feat/maxDepthRelative' of https://github.com/mendableai/firecrawl into feat/maxDepthRelative 2024-06-15 16:52:00 -04:00
Eric Ciarla
ff7b52cab1 Delete one more e2e test 2024-06-15 16:51:50 -04:00
Eric Ciarla
b1eb608295
Merge branch 'main' into feat/maxDepthRelative 2024-06-15 16:50:27 -04:00
Eric Ciarla
34e37c5671 Add unit tests to replace e2e 2024-06-15 16:43:37 -04:00
Eric Ciarla
2b40729cc2 Update index.test.ts 2024-06-15 08:56:32 -04:00
Eric Ciarla
f22759b2e7 Update index.test.ts 2024-06-14 19:42:11 -04:00
Eric Ciarla
a6b7197737 Fix for maxDepth 2024-06-14 19:40:37 -04:00
Nicolas
4ec863718b
Merge pull request #283 from mendableai/nsc/crawler-fixes
Fixes crawler getting confused with base paths that contain www.
2024-06-14 13:50:32 -07:00
Nicolas
43767360d8 Merge branch 'main' into nsc/rate-limiter-tests 2024-06-14 13:50:21 -07:00
Nicolas
e88cb314c8 Update crawler.ts 2024-06-14 13:44:54 -07:00
Rafael Miller
361cba4119
Merge pull request #175 from mendableai/test/load-testing
Test/load testing
2024-06-14 17:39:01 -03:00
Nicolas
7b11ace87d Create rate-limiter.test.ts 2024-06-14 12:31:42 -07:00
rafaelsideguide
e369d1dd0e Update index.test.ts 2024-06-14 16:17:54 -03:00
Nicolas
e37aa3db57 Nick: fixed rate limit on status 2024-06-14 12:13:02 -07:00
rafaelsideguide
a6ed2e693f Update index.test.ts 2024-06-14 15:22:52 -03:00
rafaelsideguide
ad7795f973 Merge remote-tracking branch 'origin/main' into test/load-testing 2024-06-14 15:14:01 -03:00
rafaelsideguide
354712a8a3 just changed the name for the test? 2024-06-14 13:02:04 -03:00
Eric Ciarla
2c5f5c0ea2
Merge branch 'main' into feat/maxDepthRelative 2024-06-14 11:49:12 -04:00
Eric Ciarla
80c10393b4 Update index.test.ts 2024-06-14 11:32:30 -04:00
Eric Ciarla
42ed1f4479 Update index.test.ts 2024-06-14 11:20:24 -04:00
Eric Ciarla
8830acce07 Update index.test.ts 2024-06-14 11:11:58 -04:00
Eric Ciarla
278bb311cb Update index.test.ts 2024-06-14 11:02:39 -04:00
Eric Ciarla
36a62727b8 Update index.test.ts 2024-06-14 10:52:43 -04:00
Rafael Miller
f9c7ca9388
Merge branch 'main' into feat/issue-266 2024-06-14 11:47:58 -03:00
Rafael Miller
3e2e76311c
Merge branch 'main' into feat/issue-205 2024-06-14 11:25:20 -03:00
Eric Ciarla
59451754f5 Add tests 2024-06-14 10:14:07 -04:00
rafaelsideguide
afee5684a3 Fixed tests' message and updated version 2024-06-14 11:05:19 -03:00
Eric Ciarla
9b254c1cd0 Update index.test.ts 2024-06-14 09:48:14 -04:00
Rafael Miller
5a5c532bea
Merge branch 'main' into py-sdk-improve-response-handling 2024-06-14 10:42:51 -03:00
Eric Ciarla
9aba451b18 Update index.test.ts 2024-06-14 09:33:43 -04:00
Rafael Miller
cc2e3f05b0
Merge pull request #256 from mattjoyce/feat-254-sdk-py-logging
Added logging to python sdk FIRECRAWL_LOGGING_LEVEL
2024-06-14 10:22:40 -03:00
rafaelsideguide
6963a490f1 Updated version 2024-06-14 10:21:44 -03:00
rafaelsideguide
5dd18ca79b fixed edge cases 2024-06-14 09:46:55 -03:00
Eric Ciarla
ab9de0f5ab Update maxDepth tests 2024-06-13 18:46:30 -04:00
Eric Ciarla
393bd45237 Update index.test.ts 2024-06-13 18:13:15 -04:00
Eric Ciarla
71c98d8b80 Update logic 2024-06-13 18:00:52 -04:00
Eric Ciarla
095951aa4d Update test 2024-06-13 17:40:00 -04:00
Eric Ciarla
5e8aa92788 Update index.ts 2024-06-13 17:33:13 -04:00
Eric Ciarla
bf10e9d392 Update index.test.ts 2024-06-13 17:28:59 -04:00
Eric Ciarla
65d63bae45 Update index.ts 2024-06-13 17:17:44 -04:00
Eric Ciarla
32e814bedc Update index.ts 2024-06-13 17:02:30 -04:00
Nicolas
6fc1ee32fd
Merge pull request #275 from mendableai/feat/issue-273
Added pageOptions.removeTags
2024-06-13 13:27:01 -07:00
rafaelsideguide
bb859ae9a7 Added metadata.pageStatusCode and metadata.pageError properties to the responses 2024-06-13 17:08:40 -03:00
rafaelsideguide
676d6e8ab5 Added pageOptions.removeTags 2024-06-13 10:51:05 -03:00
Nicolas
182f8d4d6c Update index.ts 2024-06-12 18:07:05 -07:00
Nicolas
11b6d5afa5 Update fly.toml 2024-06-12 18:00:22 -07:00
Nicolas
67dc46b454 Nick: clusters 2024-06-12 17:53:04 -07:00
rafaelsideguide
d20af257ba Added jobId to webhook data 2024-06-12 15:38:41 -03:00
rafaelsideguide
e37d151404 added parsePDF option to pageOptions
user can decide if they are going to let us take care of the parse or they are going to parse the pdf by themselves
2024-06-12 15:06:47 -03:00
rafaelsideguide
01c9f071fa fixed 2024-06-12 11:27:06 -03:00
rafaelsideguide
dc6acbf1f0 Merge remote-tracking branch 'origin/main' into feat/allowbackwardcrawling-option 2024-06-12 11:01:05 -03:00
Nicolas
f93231499f
Merge pull request #265 from mendableai/feat/issue-264
[Feat] Added route to clean completed jobs and a github action cron that triggers every 24h
2024-06-11 21:33:52 -07:00
Nicolas
45dee63943
Merge pull request #262 from mendableai/nsc/webhook-self-host-fix
Only fetch webhook from db if self host webhook not set and using db auth
2024-06-11 15:46:57 -07:00
rafaelsideguide
157fbe4a1e added bull auth key 2024-06-11 17:52:01 -03:00
rafaelsideguide
df3a678cf4 getting back the cancel test, this should work 2024-06-11 17:46:56 -03:00
rafaelsideguide
def2ba9987 added tests 2024-06-11 17:46:25 -03:00
Nicolas
1e3e06a1d5 Update replacePaths.test.ts 2024-06-11 13:02:39 -07:00
Nicolas
2239e03269 Update replacePaths.test.ts 2024-06-11 12:54:02 -07:00
Nicolas
520739c9f4 Nick: fixed bugs associated with absolute path replacements 2024-06-11 12:43:16 -07:00
Nicolas
b87725c683 Update openapi.json 2024-06-11 12:08:49 -07:00
rafaelsideguide
ee282c3d55 Added allowBackwardCrawling option 2024-06-11 15:24:39 -03:00
rafaelsideguide
a9f93c2f1e Added route to clean completed jobs and a github action cron that triggers every 24h 2024-06-11 14:18:05 -03:00
Nicolas
da38dad9a7 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-06-10 18:26:31 -07:00
Nicolas
9390816c1b Update openapi.json 2024-06-10 18:26:25 -07:00
Nicolas
f6b06ac27a Nick: ignoreSitemap, better crawling algo 2024-06-10 18:12:41 -07:00
Nicolas
1bd0327e1a Merge branch 'main' into nsc/pageoptions-crawler 2024-06-10 17:15:10 -07:00
Nicolas
99f2ffd6d5 Update webhook.ts 2024-06-10 17:03:10 -07:00
Nicolas
7ae9778642 Update single_url.ts 2024-06-10 16:57:31 -07:00
Nicolas
913c1dd568 Nick: fetch -> axios and fix timeouts 2024-06-10 16:49:03 -07:00
Nicolas
3091f0134c Nick: 2024-06-10 16:27:10 -07:00
Matt Joyce
827354a116 Added logging to python sdk FIRECRAWL_LOGGING_LEVEL
Instantiates the logger early and depends on env to set.
2024-06-10 21:21:23 +10:00
Nicolas
aafd23fa8a
Merge pull request #252 from mattjoyce/fix-208-py-sdk-interval-poll-name
Fix 208 py sdk interval poll name
2024-06-08 21:33:17 -07:00
Matt Joyce
6fd9ce1c89 type hints and linting 2024-06-08 11:46:52 +10:00
Matt Joyce
7477c5e5bd Use error handler consistently 2024-06-08 11:28:51 +10:00
Matt Joyce
9f306736af More detailed error handling 2024-06-08 11:18:30 +10:00
Matt Joyce
c71ea7a795 Prepare headers consistently 2024-06-08 11:08:26 +10:00
Matt Joyce
8f9a165c2f Lint - whitespace 2024-06-08 08:03:02 +10:00
Matt Joyce
5f0df596ec Align param name with JS SDK
timeout becomes poll_interval
2024-06-08 07:37:08 +10:00
Nicolas
f24ca76618 Nick: removing rate limit emails for now 2024-06-07 10:39:11 -07:00
Nicolas
98d82c4cec Update search.ts 2024-06-06 20:02:21 -07:00
Nicolas
5e80f8af87 Nick: llm extract 50 2024-06-06 18:35:44 -07:00
rafaelsideguide
7b7a6f8a39 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-06-06 17:51:28 -03:00
rafaelsideguide
f2695df215 updated sdk versions 2024-06-06 17:51:12 -03:00
rafaelsideguide
560f256a35 fixing minor problems on workflow 2024-06-06 17:36:48 -03:00
rafaelsideguide
f5318ea7d7 Update index.test.ts 2024-06-06 16:50:20 -03:00
rafaelsideguide
cd7f9abcec Update index.test.ts 2024-06-06 16:44:46 -03:00
rafaelsideguide
7b9b668b95 Update index.test.ts 2024-06-06 16:36:51 -03:00
rafaelsideguide
82e0ed4cd3 Update index.test.ts 2024-06-06 16:33:27 -03:00
rafaelsideguide
dac7612be2 Merge branch 'main' of https://github.com/mendableai/firecrawl into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 16:07:25 -03:00
Nicolas
c2ad358390 Nick: 2024-06-06 12:05:20 -07:00
rafaelsideguide
79ec9f04dc Merge branch 'main' of https://github.com/mendableai/firecrawl into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 15:58:14 -03:00
Nicolas
de06b13deb Update rate-limiter.ts 2024-06-06 11:56:22 -07:00
Nicolas
27a8fd0c3c Update rate-limiter.ts 2024-06-06 11:56:00 -07:00
Nicolas
1129d33321 Update rate-limiter.ts 2024-06-06 11:53:12 -07:00
rafaelsideguide
b234b4be5a Merge branch 'main' into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 15:44:29 -03:00
rafaelsideguide
af0bfca847 Merge branch 'main' into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 15:36:28 -03:00
rafaelsideguide
8132f22c73 nice 2024-06-06 15:36:20 -03:00
Nicolas
f1b5ec8517 Nick: fixes 2024-06-06 11:23:10 -07:00
Nicolas
deae7dcd61 Update email_notification.ts 2024-06-06 10:41:54 -07:00
Nicolas
f725fa5a97 Update email_notification.ts 2024-06-06 10:41:23 -07:00
rafaelsideguide
fb758fa05e go 2024-06-06 14:01:16 -03:00
Nicolas
0310da6729 Update rate-limiter.ts 2024-06-06 09:31:44 -07:00
Nicolas
01503c1fbf Nick: 2024-06-06 09:29:25 -07:00
rafaelsideguide
b3cae4c858 adding js and testing twine 2024-06-06 13:27:31 -03:00
rafaelsideguide
bc1c1e5053 updating version to check if it runs 2024-06-06 11:41:01 -03:00
Rafael Miller
7686ad5702
Merge pull request #196 from mattjoyce/main
Python-SDK transitional build setup for pyproject.toml
2024-06-06 10:26:16 -03:00
Nicolas
525b4f2a83 Update rate-limiter.ts 2024-06-05 14:38:10 -07:00
Nicolas
d7f8208cdb Update email_notification.ts 2024-06-05 13:53:31 -07:00
Nicolas
ec10eb09f3 Update credit_billing.ts 2024-06-05 13:22:03 -07:00
Nicolas
5991000d2b Update credit_billing.ts 2024-06-05 13:21:15 -07:00
Nicolas
5683bb2cc8 Nick: 2024-06-05 13:20:26 -07:00
rafaelsideguide
164676c70a bugfix screenshot for readme pages 2024-06-05 15:34:42 -03:00
rafaelsideguide
935406b96a Merge branch 'main' into pr/196 2024-06-05 15:19:25 -03:00
Nicolas
b4c6819a54 Nick: 2024-06-05 11:11:09 -07:00
rafaelsideguide
0d51b11dcd missing breaks 2024-06-05 15:02:28 -03:00
Rafael Miller
64423441b2
Merge branch 'main' into main 2024-06-05 14:44:29 -03:00
Nicolas
beb7526d1d Update webhook.ts 2024-06-05 10:38:05 -07:00
Nicolas
1a16378fe8
Merge pull request #234 from JakobStadlhuber/feat/webhook-self-hosted
Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload
2024-06-05 10:25:05 -07:00
Nicolas
7cb14edec8 Nick: 2024-06-05 10:13:52 -07:00
Rafael Miller
9e000ded03
Merge branch 'main' into feat/better-gdrive-pdf-fetch 2024-06-05 14:07:56 -03:00
rafaelsideguide
ccc55127d6 Added scroll xpaths on fire-engine for handling readme docs 2024-06-05 11:48:41 -03:00
rafaelsideguide
b5045d1661 [feat] improved the scrape for gdrive pdfs 2024-06-04 17:47:28 -03:00
Nicolas
96257b7b17 Update handleCustomScraping.ts 2024-06-04 12:22:46 -07:00
Nicolas
674500affa Nick: 2024-06-04 12:15:39 -07:00
rafaelsideguide
5ae4d1caf5 Update single_url.ts 2024-06-04 15:28:09 -03:00
Jakob Stadlhuber
9e5ddec207 Remove default webhook URL from .env.example
The default value for the SELF_HOSTED_WEBHOOK_URL in the .env.example file was removed to prevent unintentional exposure or usage. The users are now required to explicitly specify
2024-06-04 19:56:35 +02:00
Jakob Stadlhuber
6208f4207d Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload
This commit introduces the capability of using a Self-Hosted Webhook URL. The application now checks for a self-hosted URL before querying the database for the webhook settings. If a Self-Hosted Webhook URL is set in the environment variables, it will be used directly, diminishing unnecessary database queries.
2024-06-04 19:55:07 +02:00
rafaelsideguide
93f3098672 build files 2024-06-04 14:54:54 -03:00
rafaelsideguide
64a4338ff0 Update single_url.ts 2024-06-04 14:40:05 -03:00
Rafael Miller
02fe470e20
Merge pull request #148 from mendableai/nsc/improvemnts-fixes-misc
Better fallbacks for initial crawl start
2024-06-04 14:31:10 -03:00
Rafael Miller
665a40d9f4
Merge pull request #212 from mendableai/bugfix/partial-data-js-sdk
[Bug] Improved js response and test for getting partial_data
2024-06-04 14:05:23 -03:00
rafaelsideguide
1f4c6b7a87 Update package.json 2024-06-04 13:59:48 -03:00
Rafael Miller
19c67916d4
Merge pull request #211 from mendableai/fix/rename-variables
[Fix] Changed timeout parameter name on js sdk
2024-06-04 13:57:58 -03:00
Rafael Miller
f4f87b5374
Merge branch 'main' into bugfix/partial-data-js-sdk 2024-06-04 13:40:42 -03:00
rafaelsideguide
4e3a0495d7 updated version 0.0.12 -> 0.0.13
- [ ] publish
2024-06-04 12:03:55 -03:00
Rafael Miller
b80fb374e5
Merge branch 'main' into playwright-service-bug-222 2024-06-04 11:57:17 -03:00
rafaelsideguide
6920ec8a61 bugfixing. already on main 2024-06-04 11:05:50 -03:00
Nicolas
d91b725c6f Update fly.toml 2024-06-04 00:41:15 -07:00
Nicolas
cbf8d79cce Update pdfProcessor.ts 2024-06-04 00:13:37 -07:00
Nicolas
3fc9004ba8 Update fly.toml 2024-06-03 23:49:46 -07:00
Nicolas
2ea01f1456 Update single_url.ts 2024-06-03 23:42:39 -07:00
Nicolas
854d5b3cb3 Update single_url.ts 2024-06-03 23:32:55 -07:00
Nicolas
99059814a8 Nick: 2024-06-03 21:32:48 -07:00
Nicolas
918059ee9e Merge branch 'main' into nsc/improvemnts-fixes-misc 2024-06-03 16:46:02 -07:00
Nicolas
38e583f66c Update socialBlockList.test.ts 2024-06-03 16:44:23 -07:00
Nicolas
c69c89f838 Nick: 2024-06-03 16:42:42 -07:00
Nicolas
48d1ec05b2 Merge branch 'main' into nsc/improved-blocklist 2024-06-03 16:38:03 -07:00
Nicolas
d30ced4394
Merge pull request #221 from mendableai/nsc/fwd-header-auth
feat: Ability to forward headers to reliable providers for auth etc...
2024-06-03 16:33:40 -07:00
Romain Bruyère
4987f901d1
Merge branch 'mendableai:main' into main 2024-06-03 21:29:33 +02:00
rafaelsideguide
4100cc9223 Update index.test.ts 2024-06-03 16:29:16 -03:00
rombru
3ff91ddd1f fix: use @ instead of # for default BULL_AUTH_KEY. hash mark is reserved for URI fragments. 2024-06-03 21:28:25 +02:00
rafaelsideguide
c1aed1360e Update index.test.ts 2024-06-03 15:51:07 -03:00
rafaelsideguide
1fc3a15149 Update single_url.ts 2024-06-03 15:24:40 -03:00
Nicolas
fde522c3e1 Update single_url.ts 2024-06-02 20:23:45 -07:00
Matt Joyce
deefe65cbe Change the way the playwright response is parsed
Was failing with a Type Error, but actually looked ok.
This fixes the type error, and stop scraper fallback.
2024-06-01 19:16:56 +10:00
Matt Joyce
14896a9fdd Fix PLAYWRIGHT_MICROSERVICE_URL
It needs to end in html, otherwise scrape will 404
2024-06-01 19:03:16 +10:00
Matt Joyce
1eacad4ef3 Clarifying wait type and name 2024-06-01 18:53:03 +10:00
Matt Joyce
c516140bfb Various Linting
Pylint
C0114: Missing module docstring
C0115: Missing class docstring
C0116: Missing function or method docstring
C0303: Trailing whitespace
Import ordering
2024-06-01 18:53:03 +10:00
Matt Joyce
2a39b5382b Add timeout to class and provide default. 2024-06-01 18:52:42 +10:00
Nicolas
8cb62dde92 Update website_params.ts 2024-05-31 16:09:39 -07:00
Nicolas
3b8059edb6 Update single_url.ts 2024-05-31 15:43:06 -07:00
Nicolas
6bea803120 Nick: 2024-05-31 15:39:54 -07:00
Nicolas
2139129296 Nick: v12 2024-05-31 11:39:55 -07:00
Nicolas
260e31c68b Merge branch 'nsc/new-pricing' 2024-05-30 16:08:31 -07:00
Nicolas
aa8133ca7f Update load-testing-example.ts 2024-05-30 16:07:14 -07:00
Nicolas
0c115c6181
Merge pull request #216 from mendableai/nsc/new-pricing
feat: New pricing/limits changes
2024-05-30 15:36:59 -07:00
Nicolas
6860ace4af Nick: 2024-05-30 15:07:49 -07:00
Nicolas
6ceb7ff50a Nick: 2024-05-30 14:46:55 -07:00
Nicolas
33f10a7f91 Nick: fixes 2024-05-30 14:42:32 -07:00
Nicolas
ace46f340b Nick: new limits, new pricing 2024-05-30 14:31:36 -07:00
Matt Joyce
5c4b3e8f8a Initial pyproject.toml
This will enable building using 'python -m build', without impacting the utility of setup.py, also provide a base for other build tools and automation.
2024-05-30 21:48:40 +10:00
Matt Joyce
dec225d368 Move version to __init__.py
Setup.py does not need to be edited when building the package.
2024-05-30 21:48:40 +10:00
rafaelsideguide
2b763d848b improved js response and test for getting partial_data 2024-05-30 08:44:38 -03:00
rafaelsideguide
5b8b6902e7 Update index.ts 2024-05-30 08:25:13 -03:00
Nicolas
6c939d534d Nick: small refactor 2024-05-29 19:43:51 -07:00
Eric Ciarla
37915e11e8 Final push 2024-05-29 21:18:24 -04:00
Eric Ciarla
a0e404f94e init commit 2024-05-29 18:56:57 -04:00
rafaelsideguide
ee9a2184e2 Added custom scraping conditions for readme docs 2024-05-29 13:39:43 -03:00
Nicolas
c20c38721d Update index.test.ts 2024-05-28 17:17:20 -07:00
Nicolas
0f43a12906 Update index.test.ts 2024-05-28 17:17:12 -07:00
Nicolas
f53d25efac Merge branch 'main' into nsc/wait-for-param 2024-05-28 12:56:28 -07:00
Nicolas
1b3547dcf2 Nick: 2024-05-28 12:56:24 -07:00
rafaelsideguide
71187b03a2 added timeout 2024-05-27 16:48:08 -03:00
rafaelsideguide
d5c83803cd fixing idempotency test 2024-05-27 16:35:01 -03:00
rafaelsideguide
41c4ef6a82 dotenv was missing 2024-05-27 16:23:57 -03:00
rafaelsideguide
127d2db1dd added js/ts sdk tests 2024-05-27 15:54:09 -03:00
rafaelsideguide
a9b68d95d8 Update test.py 2024-05-27 14:28:44 -03:00
rafaelsideguide
667d3e4c4f Merge branch 'test-sdks' of https://github.com/mendableai/firecrawl into test-sdks 2024-05-27 14:23:39 -03:00
rafaelsideguide
19decd1062 fixing workflow 2024-05-27 14:21:33 -03:00
Rafael Miller
3c8edf683c
Merge branch 'main' into test-sdks 2024-05-27 14:15:18 -03:00
rafaelsideguide
63772ea711 added github action workflow 2024-05-27 14:14:00 -03:00
Nicolas
1ef307cb6f Nick: checks 2024-05-27 10:01:12 -07:00
Nicolas
01cc91c53d Update fly.staging.toml 2024-05-27 10:00:52 -07:00
Nicolas
1de53cc4d0 Nick: fixes 2024-05-26 18:15:05 -07:00
Nicolas
efb821d63b
Merge branch 'main' into main 2024-05-26 18:12:23 -07:00
Nicolas
ed4226fd1f
Update setup.py 2024-05-26 18:11:54 -07:00
Nicolas
1bbfb98d7e
Merge pull request #186 from Keredu/main
Limit on /search is not deterministic
2024-05-26 18:08:16 -07:00
Nicolas
67a53a9ae0
Merge pull request #190 from simonha9/simonha9/improve-rate-limit-error-msg
Feat: Provide more details for 429 error msg
2024-05-26 18:07:42 -07:00