Commit Graph

1299 Commits

Author SHA1 Message Date
Gergő Móricz
3ad9bf7ac0 Update GH Actions deployment 2024-08-21 19:15:25 +02:00
Gergő Móricz
920702cdde Update builder to handle uploading sourcemaps 2024-08-21 19:08:03 +02:00
Gergő Móricz
86942728e3 Add metadata for queue-worker and Express 2024-08-21 17:58:27 +02:00
Nicolas
35decb1af2 Nick: 2024-08-21 12:35:03 -03:00
rafaelsideguide
52abec41c2 fixing delete 2024-08-21 10:35:50 -03:00
Nicolas
db8c84ff0f Update requests.http 2024-08-21 10:19:37 -03:00
rafaelsideguide
b66553867e reverting delete, fixed express bug on checkCredits 2024-08-21 09:28:20 -03:00
rafaelsideguide
138437d616 commenting out delete, crashing on fire-engine
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-21 08:11:24 -03:00
rafaelsideguide
5e48bec1fd commenting out delete, crashing on fire-engine 2024-08-21 08:10:46 -03:00
Nicolas
90b32f16c8 Nick: fixes 2024-08-20 21:38:11 -03:00
Nicolas
819ad50af3 Update fireEngine.ts 2024-08-20 21:16:33 -03:00
Nicolas
1b3ad60a2c Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error"
This reverts commit ffe11a5bf7.
2024-08-20 19:22:09 -03:00
Nicolas
441628998f Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error"
This reverts commit ffe11a5bf7.
2024-08-20 19:16:48 -03:00
Nicolas
ffe11a5bf7 Revert "Merge pull request #561 from mendableai/bug/dealing-with-dns-error"
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
This reverts commit 2030ec6031, reversing
changes made to f494d2b707.
2024-08-20 18:16:11 -03:00
Nicolas
2030ec6031
Merge pull request #561 from mendableai/bug/dealing-with-dns-error
[Bug] Added a way for dealing with DNS without IP resolution
2024-08-20 16:59:09 -03:00
rafaelsideguide
f98be7d94e Update fireEngine.ts 2024-08-20 16:53:01 -03:00
Gergő Móricz
f494d2b707 prioritize search lower 2024-08-20 21:18:31 +02:00
Nicolas
59eb552df5
Merge pull request #560 from mendableai/feat/cancel-fire-engine
[Feat] Added check job and cancel to fire-engine requests
2024-08-20 15:43:31 -03:00
rafaelsideguide
1f27182a13 added try catch 2024-08-20 15:42:39 -03:00
Gergő Móricz
efb91f9ca5 fix(search): delete jobs after done 2024-08-20 20:29:08 +02:00
Nicolas
2b221fd4a9
Merge pull request #559 from mendableai/nsc/Nick-set-the-crawl-limit-to-the-remaining-credits
Set the crawl limit to the remaining credits
2024-08-20 14:38:10 -03:00
rafaelsideguide
e326249a57 added check job and cancel to fire-engine requests 2024-08-20 14:26:42 -03:00
Nicolas
5abd26a267 Nick: set the crawl limit to the remaining credits 2024-08-20 14:16:54 -03:00
rafaelsideguide
e1c9cbf709 bug fixed. crawl should not stop if sitemap url is invalid
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-20 09:11:58 -03:00
Thomas Kosmas
0dce57832d
Merge pull request #555 from mendableai/feat/beta-customers
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
added variables to beta customers
2024-08-20 02:20:23 +03:00
rafaelsideguide
ecd472356b added variables to beta customers 2024-08-19 16:41:54 -03:00
Nicolas
5a44191344
Merge pull request #554 from mendableai/nsc/check-team-credits-limit
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Check team credits based on the crawl limit
2024-08-19 11:54:47 -03:00
Nicolas
8e4ca86463 Update crawl.ts 2024-08-19 11:02:24 -03:00
Nicolas
36b35dbc67 Update crawl.ts 2024-08-19 11:01:26 -03:00
rafaelsideguide
4ffc60596a Update queue-worker.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-19 09:29:23 -03:00
rafaelsideguide
b8170aaa47 Update blocklist.ts 2024-08-19 08:51:48 -03:00
Nicolas
3fe82b4f12 Update queue-worker.ts
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Fly Deploy / Test Suite (push) Has been cancelled
Fly Deploy / Python SDK Tests (push) Has been cancelled
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Fly Deploy / Go SDK Tests (push) Has been cancelled
Fly Deploy / Deploy app (push) Has been cancelled
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
2024-08-17 03:09:31 -04:00
Nicolas
f797380112 Nick:
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-16 22:17:38 -04:00
Nicolas
47123be783 Nick: weird activity block 2024-08-16 22:01:56 -04:00
Gergő Móricz
c281fe62c0 fix(crawl): propagate db fix to preview endpoint 2024-08-16 23:43:54 +02:00
Gergő Móricz
e6738abf96 fix(crawl-status): retrieve from DB in bulk 2024-08-16 23:39:39 +02:00
Nicolas
78ca94251c
Merge pull request #480 from mendableai/nsc/hyper-v81
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Reduce metrics ingestion w/ HyperDX v0.8.1
2024-08-16 14:34:14 -04:00
Gergő Móricz
fd6432e7fd fix(queue-worker): correct job success 2024-08-16 19:16:08 +02:00
Gergő Móricz
6e54942265 fix(queue-worker): add cancelled to crawl log 2024-08-16 19:11:53 +02:00
Gergő Móricz
d0a8382a5b fix(queue-worker): crawl finishing race condition 2024-08-16 18:48:52 +02:00
Gergő Móricz
6bd52e63bf fix(queue-worker): fix linksOnPage undefined error 2024-08-16 18:42:24 +02:00
Gergő Móricz
5a6570cba2 fix(webhooks): call back with parent crawl ID 2024-08-16 17:42:42 +02:00
Nicolas
ec361609d2 Nick: added growth-2x plan
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-15 18:37:19 -04:00
Nicolas
32c6b1f136 Nick: remove active job alerts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-15 14:50:30 -04:00
Gergő Móricz
0c14366720 fix: add checkandupdateurl to crawlPreview 2024-08-15 20:30:25 +02:00
Nicolas
81b2479db3
Merge pull request #459 from mendableai/feat/queue-scrapes
feat: Move scraper to queue
2024-08-15 14:19:55 -04:00
Gergő Móricz
fc08ff450d search port 2024-08-15 20:10:59 +02:00
Nicolas
86326f34e9 Update single_url.test.ts 2024-08-15 13:48:42 -04:00
Gergő Móricz
129a882bcc fix(scrape): give scrapes their real job id 2024-08-15 19:29:47 +02:00
Gergő Móricz
965a5817d1 fix(queue-worker): log jobs correctly 2024-08-15 19:27:15 +02:00