Gergő Móricz
ef505f8d99
feat(scrapeURL/fire-engine): adjust timeout tuning
Deploy Images to GHCR / push-app-image (push) Waiting to run
2024-11-08 17:24:19 +01:00
Gergő Móricz
b8a6fb3524
fix(scrapeURL/checkStatus): bad handling of f-e under load
2024-11-08 16:29:56 +01:00
Gergő Móricz
25e94ffd28
fix(scrapeURL): do not submit LLM schema errors to Sentry
Deploy Images to GHCR / push-app-image (push) Waiting to run
2024-11-07 23:21:07 +01:00
Gergő Móricz
a297c99ba8
fix(scrapeURL): error displaying
2024-11-07 23:18:24 +01:00
Gergő Móricz
79cadcb769
fix(scrapeURL/llmExtract): fill in required field as well
2024-11-07 22:48:57 +01:00
Gergő Móricz
0588f340c3
fix(scrapeURL/llmExtract): array schema fix
2024-11-07 22:46:59 +01:00
Gergő Móricz
552d55c8fc
fix(scrapeURL): includeTags/excludeTags
2024-11-07 21:10:27 +01:00
Gergő Móricz
8d467c8ca7
WebScraper refactor into scrapeURL (#714 )
...
* feat: use strictNullChecking
* feat: switch logger to Winston
* feat(scrapeURL): first batch
* fix(scrapeURL): error swallow
* fix(scrapeURL): add timeout to EngineResultsTracker
* fix(scrapeURL): report unexpected error to sentry
* chore: remove unused modules
* feat(transfomers/coerce): warn when a format's response is missing
* feat(scrapeURL): feature flag priorities, engine quality sorting, PDF and DOCX support
* (add note)
* feat(scrapeURL): wip readme
* feat(scrapeURL): LLM extract
* feat(scrapeURL): better warnings
* fix(scrapeURL/engines/fire-engine;playwright): fix screenshot
* feat(scrapeURL): add forceEngine internal option
* feat(scrapeURL/engines): scrapingbee
* feat(scrapeURL/transformars): uploadScreenshot
* feat(scrapeURL): more intense tests
* bunch of stuff
* get rid of WebScraper (mostly)
* adapt batch scrape
* add staging deploy workflow
* fix yaml
* fix logger issues
* fix v1 test schema
* feat(scrapeURL/fire-engine/chrome-cdp): remove wait inserts on actions
* scrapeURL: v0 backwards compat
* logger fixes
* feat(scrapeurl): v0 returnOnlyUrls support
* fix(scrapeURL/v0): URL leniency
* fix(batch-scrape): ts non-nullable
* fix(scrapeURL/fire-engine/chromecdp): fix wait action
* fix(logger): remove error debug key
* feat(requests.http): use dotenv expression
* fix(scrapeURL/extractMetadata): extract custom metadata
* fix crawl option conversion
* feat(scrapeURL): Add retry logic to robustFetch
* fix(scrapeURL): crawl stuff
* fix(scrapeURL): LLM extract
* fix(scrapeURL/v0): search fix
* fix(tests/v0): grant larger response size to v0 crawl status
* feat(scrapeURL): basic fetch engine
* feat(scrapeURL): playwright engine
* feat(scrapeURL): add url-specific parameters
* Update readme and examples
* added e2e tests for most parameters. Still a few actions, location and iframes to be done.
* fixed type
* Nick:
* Update scrape.ts
* Update index.ts
* added actions and base64 check
* Nick: skipTls feature flag?
* 403
* todo
* todo
* fixes
* yeet headers from url specific params
* add warning when final engine has feature deficit
* expose engine results tracker for ScrapeEvents implementation
* ingest scrape events
* fixed some tests
* comment
* Update index.test.ts
* fixed rawHtml
* Update index.test.ts
* update comments
* move geolocation to global f-e option, fix removeBase64Images
* Nick:
* trim url-specific params
* Update index.ts
---------
Co-authored-by: Eric Ciarla <ericciarla@yahoo.com>
Co-authored-by: rafaelmmiller <8574157+rafaelmmiller@users.noreply.github.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2024-11-07 20:57:33 +01:00
Nicolas
56179924e6
Merge branch 'main' of https://github.com/mendableai/firecrawl
2024-11-05 13:08:02 -05:00
Nicolas
3cd1c4260d
Update rate-limiter.ts
2024-11-05 13:08:01 -05:00
rafaelmmiller
f07bbef78e
added trycatch and removed redundancy
2024-11-05 08:11:49 -03:00
Dmitriy Vasilyuk
2fa25cb992
[Fix] Prevent Python Firecrawl logger from interfering with loggers in client applications ( #613 )
...
Deploy Images to GHCR / push-app-image (push) Waiting to run
* Prevent firecrawl logger from interfering with other loggers
* Always check if logger has handlers
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2024-11-05 08:33:39 +01:00
Nicolas
9e22c9a428
Nick: etier1a
Deploy Images to GHCR / push-app-image (push) Waiting to run
2024-11-04 18:14:38 -05:00
Rafael Miller
8297e5beef
Merge branch 'main' into fix/remove-base64-images
2024-11-04 10:35:09 -03:00
rafaelmmiller
4c5bb21a6f
added remove base64 images options (true by default)
2024-11-04 10:31:44 -03:00
Eric Ciarla
2eff27ba43
Merge pull request #847 from mendableai/nsc/mobile-support
...
Deploy Images to GHCR / push-app-image (push) Has been cancelled
Adds support for mobile web scraping + mobile screenshot
2024-11-02 11:16:33 -04:00
rafaelsideguide
61f659190d
bumped js-sdk version
2024-11-01 12:44:29 -03:00
rafaelsideguide
28db4dd3b5
fixed zod validation for wait
2024-11-01 12:40:40 -03:00
Nicolas
80beedb46f
Update index.ts
2024-10-31 20:03:19 -03:00
Nicolas
446acfccde
Nick: support for the new actions
2024-10-31 20:01:52 -03:00
Thomas Kosmas
3911fe1fba
Fix
Deploy Images to GHCR / push-app-image (push) Waiting to run
2024-10-31 18:14:20 +02:00
Thomas Kosmas
6e5c95d5e7
fix bug
2024-10-31 17:55:43 +02:00
Thomas Kosmas
3eb79f1293
Fix dockerfile
2024-10-31 17:52:38 +02:00
Thomas Kosmas
ae919ad896
Fix go parser
2024-10-31 17:36:32 +02:00
rafaelsideguide
367af9512f
added iframe links to extractLinksFromHTML
2024-10-31 10:53:47 -03:00
Thomas Kosmas
fe02101a12
Iframe support
2024-10-31 14:40:33 +02:00
Nicolas
47d276d3b5
Update rate-limiter.ts
2024-10-30 17:19:37 -03:00
Nicolas
ea85b1d602
Merge branch 'main' into mog/concurrency-limit-2
2024-10-30 17:05:40 -03:00
Nicolas
96c579f1cd
Nick: etier2c
Deploy Images to GHCR / push-app-image (push) Waiting to run
2024-10-29 21:01:43 -03:00
Nicolas
e0ba339c50
Update auth.ts
2024-10-29 16:06:12 -03:00
Nicolas
6948ca6fe1
Revert "Update auth.ts"
...
This reverts commit 7152ac8856
2024-10-29 16:02:21 -03:00
Nicolas
7152ac8856
Update auth.ts
2024-10-29 15:58:20 -03:00
Nicolas
c00cd21308
Nick: adds support for mobile web scraping
2024-10-29 14:10:40 -03:00
Nicolas
6d38c65466
Update package.json
2024-10-28 20:25:28 -03:00
Nicolas
07e76f2ba5
Merge branch 'main' into nsc/geo-to-location
2024-10-28 20:24:29 -03:00
Nicolas
b6ce49e5bb
Update index.ts
2024-10-28 20:24:16 -03:00
Nicolas
3d1bb82aa2
Nick: languages support
2024-10-28 20:16:11 -03:00
Nicolas
0bad436061
Nick: fixed the batch scrape + llm extract billing
Deploy Images to GHCR / push-app-image (push) Waiting to run
2024-10-28 17:04:42 -03:00
Nicolas
726430c2e6
Nick: llm extract in batch scrape
2024-10-28 16:51:49 -03:00
Nicolas
d2d5c0b6c0
Merge pull request #838 from mendableai/nsc/rm-wait-before-click
...
RM wait before interacting
2024-10-28 16:06:31 -03:00
Nicolas
a13c2442e6
Merge pull request #628 from SearchApi/feature/add-searchapi-search
...
Add SearchApi as a Web Search Tool
2024-10-28 16:04:56 -03:00
Nicolas
fa8875d64d
Update single_url.ts
2024-10-28 15:09:50 -03:00
Nicolas
877d5e4383
Update types.ts
2024-10-27 23:17:20 -03:00
Nicolas
68b2e1b209
Update log_job.ts
2024-10-27 23:14:25 -03:00
Nicolas
8a4f4cb9d9
Update README.md
Deploy Images to GHCR / push-app-image (push) Has been cancelled
2024-10-26 16:10:51 -03:00
Nicolas
9593ab80e1
Update README.md
2024-10-26 16:03:07 -03:00
Nicolas
801f0f773e
Nick: fix auto charge failing when payment is through Link
Deploy Images to GHCR / push-app-image (push) Waiting to run
2024-10-26 03:59:15 -03:00
Nicolas
97b8d6c333
Update auto_charge.ts
2024-10-25 16:05:39 -03:00
Nicolas
95c4652fd4
Nick: 10min cooldown on auto charge
2024-10-25 16:05:23 -03:00
Gergő Móricz
4468a49aee
concurrency limit fix PoC II.
2024-10-25 20:21:12 +02:00
