659 Commits

Author	SHA1	Message	Date
Nicolas	5ecd9cb6f5	Merge pull request #363 from mendableai/nsc/logging-scrapers ... Logging for all scraper methods	2024-07-03 18:47:22 -03:00
Nicolas	066d92f643	Update single_url.ts	2024-07-03 18:38:17 -03:00
Nicolas	f5b2fbd7e8	Nick: revision	2024-07-03 18:06:53 -03:00
Nicolas	2d30cc6117	Nick: comments	2024-07-03 18:01:54 -03:00
Nicolas	90c54c32fd	Nick: refactor	2024-07-03 18:01:17 -03:00
Nicolas	90cf799a3c	Update single_url.ts	2024-07-03 17:56:21 -03:00
Nicolas	b36406e465	Nick: log scrpaers	2024-07-03 17:28:53 -03:00
Nicolas	abb44bb112	Merge pull request #346 from mendableai/dependabot/pip/apps/playwright-service/prod-deps-8f04296377 ... apps/playwright-service(deps): bump the prod-deps group in /apps/playwright-service with 3 updates	2024-07-03 01:07:09 -03:00
Nicolas	f967daddcb	Merge pull request #325 from snippet/playwright-scraper-api ... new playwright service	2024-07-03 01:04:52 -03:00
Eric Ciarla	2d0d5ac392	Update for llm-extraction-from-raw-html	2024-07-02 14:05:42 -04:00
rafaelsideguide	0175152577	Fixed PDF match custom scraping ... Now it's working for both `https://getgc.ai/privacy` and `https://prairie.cards/products/wood-designs` usecases.	2024-07-02 11:25:17 -03:00
rafaelsideguide	96de948d6b	Update index.test.ts	2024-07-02 11:04:09 -03:00
rafaelsideguide	7b7154ba1e	bugfixed pageStatusCode	2024-07-02 10:51:35 -03:00
Rafael Miller	50eecf04a9	Update licence pyproject.toml ... Closes #345	2024-07-02 10:01:49 -03:00
dependabot[bot]	60de6bb6e3	apps/playwright-service(deps): bump the prod-deps group ... Bumps the prod-deps group in /apps/playwright-service with 3 updates: [hypercorn](https://github.com/pgjones/hypercorn), [fastapi](https://github.com/tiangolo/fastapi) and [playwright](https://github.com/Microsoft/playwright-python). Updates `hypercorn` from 0.16.0 to 0.17.3 - [Changelog](https://github.com/pgjones/hypercorn/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pgjones/hypercorn/compare/0.16.0...0.17.3) Updates `fastapi` from 0.110.0 to 0.111.0 - [Release notes](https://github.com/tiangolo/fastapi/releases) - [Commits](https://github.com/tiangolo/fastapi/compare/0.110.0...0.111.0) Updates `playwright` from 1.42.0 to 1.44.0 - [Release notes](https://github.com/Microsoft/playwright-python/releases) - [Commits](https://github.com/Microsoft/playwright-python/compare/v1.42.0...v1.44.0) --- updated-dependencies: - dependency-name: hypercorn dependency-type: direct:production update-type: version-update:semver-minor dependency-group: prod-deps - dependency-name: fastapi dependency-type: direct:production update-type: version-update:semver-minor dependency-group: prod-deps - dependency-name: playwright dependency-type: direct:production update-type: version-update:semver-minor dependency-group: prod-deps ... Signed-off-by: dependabot[bot] <support@github.com>	2024-07-02 12:47:09 +00:00
Rafael Miller	f0f449fe51	Merge pull request #336 from snippet/allow-external-content-links ... [Proposal] new feature allowExternalContentLinks	2024-07-02 09:45:21 -03:00
rafaelsideguide	db4a743365	Added e2e test	2024-07-02 09:44:08 -03:00
Eric Ciarla	0821017f5b	Update README.md	2024-07-02 07:08:46 -04:00
Nicolas	42cd58a679	Merge pull request #332 from mendableai/feat/rawHtmlExtraction ... Adds pageOptions.includeRawHtml and new extraction mode "llm-extraction-from-raw-html"	2024-07-01 18:23:26 -03:00
Nicolas	c4f423981f	Update pnpm-lock.yaml	2024-07-01 18:22:22 -03:00
rafaelsideguide	16aac7f8c5	Update single_url.ts	2024-07-01 18:21:15 -03:00
Nicolas	6d0c7a9ccd	Merge pull request #323 from mendableai/tests/crawl-limit-unit-tests ... [Tests] Added crawl limit unit test	2024-07-01 17:56:04 -03:00
rafaelsideguide	4d6e25619b	minor spacing and comment stuff	2024-07-01 16:05:34 -03:00
Eric Ciarla	e1af815f8c	Update scrape.ts	2024-07-01 08:48:21 -04:00
Eric Ciarla	7ae195bacc	Update index.test.ts	2024-06-29 10:13:12 -04:00
Eric Ciarla	837b446390	Update index.test.ts	2024-06-29 08:48:42 -04:00
Eric Ciarla	fe6e3aeadc	Update index.test.ts	2024-06-29 08:44:21 -04:00
Eric Ciarla	6c9f0dfc91	Add tests	2024-06-29 08:32:20 -04:00
Jeff Pereira	a5fb45988c	new feature allowExternalContentLinks	2024-06-28 17:23:40 -07:00
Eric Ciarla	87b54488d3	update to includeRawHtml	2024-06-28 17:07:47 -04:00
Eric Ciarla	70fcf2ce03	init	2024-06-28 16:39:09 -04:00
Nicolas	9bf74bc774	Update single_url.ts	2024-06-28 15:51:18 -03:00
Nicolas	7e17498bcf	Update single_url.ts	2024-06-28 15:45:16 -03:00
rafaelsideguide	7dffaaa3e2	Changed port and added "using with firecrawl" section on readme	2024-06-28 11:51:24 -03:00
Nicolas	9e7298945c	Update openapi.json	2024-06-26 21:25:38 -03:00
Nicolas	1ec0bf8adf	Update openapi.json	2024-06-26 21:22:46 -03:00
Nicolas	042f81ddf2	Update removeUnwantedElements.test.ts	2024-06-26 21:20:11 -03:00
Nicolas	388ce3cbce	Nick: small changes	2024-06-26 21:15:42 -03:00
Nicolas	1d4907acc9	Nick:	2024-06-26 21:02:58 -03:00
Jeff Pereira	d833a132a5	new playwright service	2024-06-26 12:32:30 -07:00
Nicolas	3b92fb8433	Merge pull request #322 from mendableai/tests/metadata ... [Test] Added E2E tests for checking metadata values	2024-06-26 12:09:18 -03:00
rafaelsideguide	67d7650cf3	Added to e2e_noAuth	2024-06-26 12:07:55 -03:00
rafaelsideguide	009df6c930	Added crawl limit unit test ... I think this test is over relying on mocks but I have no idea on how to fix this without changing the code arch structure	2024-06-26 09:54:25 -03:00
rafaelsideguide	05eaa3c68d	Update index.test.ts	2024-06-26 09:32:02 -03:00
rafaelsideguide	4381109dd8	added default values and fixed pdf bug	2024-06-26 09:00:54 -03:00
Nicolas	45f2765601	Merge pull request #316 from snippet/types-webscraper ... add some types	2024-06-25 22:03:21 -03:00
Nicolas	768a131b5c	Merge pull request #318 from mendableai/bug/fix-custom-scrape-pdf-google-drive ... [Bug] Fixed the regex test for google drive pdf files	2024-06-25 18:27:11 -03:00
rafaelsideguide	5f69fc7677	Fixed the regex test	2024-06-25 18:24:01 -03:00
rafaelsideguide	d02829d335	fixed clean jobs	2024-06-25 17:49:29 -03:00
Jeff Pereira	199cbe8bcb	add some types	2024-06-25 12:20:25 -07:00