Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 03:32:22 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Update SDKs to MIT license

Browse Source
This commit is contained in:
Eric Ciarla 2024-07-08 13:37:53 -04:00
parent 002bfdf639
commit afb49e21e7
6 changed files with 424 additions and 204 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

209
LICENSE
View File

@ -1,35 +1,35 @@
GNU AFFERO GENERAL PUBLIC LICENSE GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007 Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/> Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed. of this license document, but changing it is not allowed.
Preamble Preamble
The GNU Affero General Public License is a free, copyleft license for The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software. cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast, to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free share and change all versions of a program--to make sure it remains free
software for all its users. software for all its users.
When we speak of free software, we are referring to freedom, not When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things. free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute you this License which gives you legal permission to copy, distribute
and/or modify the software. and/or modify the software.
A secondary benefit of defending all users' freedom is that A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and incorporate. Many developers of free software are heartened and
@ -39,7 +39,7 @@ The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its letting the public access it on a server without ever releasing its
source code to the public. source code to the public.
The GNU Affero General Public License is designed specifically to The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the provide the source code of the modified version running there to the
@ -47,48 +47,48 @@ users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source a publicly accessible server, gives the public access to the source
code of the modified version. code of the modified version.
An older license, called the Affero General Public License and An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under released a new version of the Affero GPL which permits relicensing under
this license. this license.
The precise terms and conditions for copying, distribution and The precise terms and conditions for copying, distribution and
modification follow. modification follow.
TERMS AND CONDITIONS TERMS AND CONDITIONS
0. Definitions. 0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License. "This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of "Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks. works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this "The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations. "recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work. earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based A "covered work" means either the unmodified Program or a work based
on the Program. on the Program.
To "propagate" a work means to do anything with it that, without To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying, computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the distribution (with or without modification), making available to the
public, and in some countries other activities as well. public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying. a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices" An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2) feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the tells the user that there is no warranty for the work (except to the
@ -97,18 +97,18 @@ work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion. menu, a prominent item in the list meets this criterion.
1. Source Code. 1. Source Code.
The "source code" for a work means the preferred form of the work The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source for making modifications to it. "Object code" means any non-source
form of a work. form of a work.
A "Standard Interface" means an interface that either is an official A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that interfaces specified for a particular programming language, one that
is widely used among developers working in that language. is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that Component, and (b) serves only to enable use of the work with that
@ -119,7 +119,7 @@ implementation is available to the public in source code form. A
(if any) on which the executable work runs, or a compiler used to (if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it. produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's control those activities. However, it does not include the work's
@ -132,16 +132,16 @@ linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those such as by intimate data communication or control flow between those
subprograms and other parts of the work. subprograms and other parts of the work.
The Corresponding Source need not include anything that users The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding can regenerate automatically from other parts of the Corresponding
Source. Source.
The Corresponding Source for a work in source code form is that The Corresponding Source for a work in source code form is that
same work. same work.
2. Basic Permissions. 2. Basic Permissions.
All rights granted under this License are granted for the term of All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a permission to run the unmodified Program. The output from running a
@ -149,7 +149,7 @@ covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law. rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you of having them make modifications exclusively for you, or provide you
@ -160,19 +160,19 @@ for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you. your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10 the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary. makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law. 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or 11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such similar laws prohibiting or restricting circumvention of such
measures. measures.
When you convey a covered work, you waive any legal power to forbid When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or the covered work, and you disclaim any intention to limit operation or
@ -180,9 +180,9 @@ modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of users, your or third parties' legal rights to forbid circumvention of
technological measures. technological measures.
4. Conveying Verbatim Copies. 4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice; appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any keep intact all notices stating that this License and any
@ -190,12 +190,12 @@ non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program. recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey, You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee. and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions. 5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions: terms of section 4, provided that you also meet all of these conditions:
@ -220,7 +220,7 @@ terms of section 4, provided that you also meet all of these conditions:
interfaces that do not display Appropriate Legal Notices, your interfaces that do not display Appropriate Legal Notices, your
work need not make them do so. work need not make them do so.
A compilation of a covered work with other separate and independent A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work, works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program, and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an in or on a volume of a storage or distribution medium, is called an
@ -230,9 +230,9 @@ beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other in an aggregate does not cause this License to apply to the other
parts of the aggregate. parts of the aggregate.
6. Conveying Non-Source Forms. 6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License, machine-readable Corresponding Source under the terms of this License,
in one of these ways: in one of these ways:
@ -278,11 +278,11 @@ in one of these ways:
Source of the work are being offered to the general public at no Source of the work are being offered to the general public at no
charge under subsection 6d. charge under subsection 6d.
A separable portion of the object code, whose source code is excluded A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be from the Corresponding Source as a System Library, need not be
included in conveying the object code work. included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family, tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product, into a dwelling. In determining whether a product is a consumer product,
@ -295,7 +295,7 @@ is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product. the only significant mode of use of the product.
"Installation Information" for a User Product means any methods, "Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must a modified version of its Corresponding Source. The information must
@ -303,7 +303,7 @@ suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because code is in no case prevented or interfered with solely because
modification has been made. modification has been made.
If you convey an object code work under this section in, or with, or If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a User Product is transferred to the recipient in perpetuity or for a
@ -314,7 +314,7 @@ if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has modified object code on the User Product (for example, the work has
been installed in ROM). been installed in ROM).
The requirement to provide Installation Information does not include a The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a the User Product in which it has been modified or installed. Access to a
@ -322,15 +322,15 @@ network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and adversely affects the operation of the network or violates the rules and
protocols for communication across the network. protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in documented (and with an implementation available to the public in
source code form), and must require no special password or key for source code form), and must require no special password or key for
unpacking, reading or copying. unpacking, reading or copying.
7. Additional Terms. 7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this "Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions. License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent be treated as though they were included in this License, to the extent
@ -339,14 +339,14 @@ apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions. this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work, additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission. for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms: that material) supplement the terms of this License with terms:
@ -373,7 +373,7 @@ that material) supplement the terms of this License with terms:
any liability that these contractual assumptions directly impose on any liability that these contractual assumptions directly impose on
those licensors and authors. those licensors and authors.
All other non-permissive additional terms are considered "further All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further governed by this License along with a term that is a further
@ -383,46 +383,46 @@ License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does of that license document, provided that the further restriction does
not survive such relicensing or conveying. not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating additional terms that apply to those files, or a notice indicating
where to find the applicable terms. where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions; form of a separately written license, or stated as exceptions;
the above requirements apply either way. the above requirements apply either way.
8. Termination. 8. Termination.
You may not propagate or modify a covered work except as expressly You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third this License (including any patent licenses granted under the third
paragraph of section 11). paragraph of section 11).
However, if you cease all violation of this License, then your However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a) license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation. prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice. your receipt of the notice.
Termination of your rights under this section does not terminate the Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same reinstated, you do not qualify to receive new licenses for the same
material under section 10. material under section 10.
9. Acceptance Not Required for Having Copies. 9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However, to receive a copy likewise does not require acceptance. However,
@ -431,14 +431,14 @@ modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so. covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients. 10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License. for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that work results from an entity transaction, each party to that
@ -448,7 +448,7 @@ give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts. the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation rights granted under this License, and you may not initiate litigation
@ -456,13 +456,13 @@ rights granted under this License, and you may not initiate litigation
any patent claim is infringed by making, using, selling, offering for any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it. sale, or importing the Program or any portion of it.
11. Patents. 11. Patents.
A "contributor" is a copyright holder who authorizes use under this A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version". work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version, by this License, of making, using, or selling its contributor version,
@ -472,19 +472,19 @@ purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of patent sublicenses in a manner consistent with the requirements of
this License. this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version. propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to (such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a party means to make such an agreement or commitment not to enforce a
patent against the party. patent against the party.
If you convey a covered work, knowingly relying on a patent license, If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means, publicly available network server or other readily accessible means,
@ -498,7 +498,7 @@ covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid. country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify receiving the covered work authorizing them to use, propagate, modify
@ -506,7 +506,7 @@ or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered you grant is automatically extended to all recipients of the covered
work and works based on it. work and works based on it.
A patent license is "discriminatory" if it does not include within A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered specifically granted under this License. You may not convey a covered
@ -521,13 +521,13 @@ for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement, contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007. or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law. otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom. 12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this covered work so as to satisfy simultaneously your obligations under this
@ -537,9 +537,9 @@ to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program. License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License. 13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding supports such interaction) an opportunity to receive the Corresponding
@ -550,7 +550,7 @@ shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the of the GNU General Public License that is incorporated pursuant to the
following paragraph. following paragraph.
Notwithstanding any other provision of this License, you have Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this combined work, and to convey the resulting work. The terms of this
@ -558,14 +558,14 @@ License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version but the work with which it is combined will remain governed by version
3 of the GNU General Public License. 3 of the GNU General Public License.
14. Revised Versions of this License. 14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns. address new problems or concerns.
Each version is given a distinguishing version number. If the Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered option of following the terms and conditions either of that numbered
@ -574,19 +574,19 @@ Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation. by the Free Software Foundation.
If the Program specifies that a proxy can decide which future If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you public statement of acceptance of a version permanently authorizes you
to choose that version for the Program. to choose that version for the Program.
Later license versions may give you additional or different Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a author or copyright holder as a result of your choosing to follow a
later version. later version.
15. Disclaimer of Warranty. 15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
@ -595,9 +595,9 @@ PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION. ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability. 16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
@ -607,9 +607,9 @@ PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES. SUCH DAMAGES.
17. Interpretation of Sections 15 and 16. 17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms, above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the an absolute waiver of all civil liability in connection with the
@ -620,11 +620,11 @@ copy of the Program in return for a fee.
How to Apply These Terms to Your New Programs How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms. free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found. the "copyright" line and a pointer to where the full notice is found.
@ -647,7 +647,7 @@ the "copyright" line and a pointer to where the full notice is found.
Also add information on how to contact you by electronic and paper mail. Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive interface could display a "Source" link that leads users to an archive
@ -655,7 +655,26 @@ of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the solutions will be better for different programs; see section 13 for the
specific requirements. specific requirements.
You should also get your employer (if you work as a programmer) or school, You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary. if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>. <https://www.gnu.org/licenses/>.
Firecrawl - Web scraping and crawling tool
Copyright (c) 2024 Sideguide Technologies Inc.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
For more information, please contact:
Sideguide Technologies Inc.

45
README.md
View File

@ -21,7 +21,7 @@ We provide an easy to use API with our hosted version. You can find the playgrou
- [x] [Node SDK](https://github.com/mendableai/firecrawl/tree/main/apps/js-sdk) - [x] [Node SDK](https://github.com/mendableai/firecrawl/tree/main/apps/js-sdk)
- [x] [Langchain Integration 🦜🔗](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/) - [x] [Langchain Integration 🦜🔗](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/)
- [x] [Llama Index Integration 🦙](https://docs.llamaindex.ai/en/latest/examples/data_connectors/WebPageDemo/#using-firecrawl-reader) - [x] [Llama Index Integration 🦙](https://docs.llamaindex.ai/en/latest/examples/data_connectors/WebPageDemo/#using-firecrawl-reader)
- [X] [Langchain JS Integration 🦜🔗](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl) - [x] [Langchain JS Integration 🦜🔗](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl)
- [ ] Want an SDK or Integration? Let us know by opening an issue. - [ ] Want an SDK or Integration? Let us know by opening an issue.
To run locally, refer to guide [here](https://github.com/mendableai/firecrawl/blob/main/CONTRIBUTING.md). To run locally, refer to guide [here](https://github.com/mendableai/firecrawl/blob/main/CONTRIBUTING.md).
@ -212,7 +212,6 @@ curl -X POST https://api.firecrawl.dev/v0/scrape \
} }
} }
} }
``` ```
## Using Python SDK ## Using Python SDK
@ -302,34 +301,29 @@ To scrape a single URL with error handling, use the `scrapeUrl` method. It takes
```js ```js
try { try {
const url = 'https://example.com'; const url = "https://example.com";
const scrapedData = await app.scrapeUrl(url); const scrapedData = await app.scrapeUrl(url);
console.log(scrapedData); console.log(scrapedData);
} catch (error) { } catch (error) {
console.error( console.error("Error occurred while scraping:", error.message);
'Error occurred while scraping:',
error.message
);
} }
``` ```
### Crawling a Website ### Crawling a Website
To crawl a website with error handling, use the `crawlUrl` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format. To crawl a website with error handling, use the `crawlUrl` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
```js ```js
const crawlUrl = 'https://example.com'; const crawlUrl = "https://example.com";
const params = { const params = {
crawlerOptions: { crawlerOptions: {
excludes: ['blog/'], excludes: ["blog/"],
includes: [], // leave empty for all pages includes: [], // leave empty for all pages
limit: 1000, limit: 1000,
}, },
pageOptions: { pageOptions: {
onlyMainContent: true onlyMainContent: true,
} },
}; };
const waitUntilDone = true; const waitUntilDone = true;
const timeout = 5; const timeout = 5;
@ -339,10 +333,8 @@ const crawlResult = await app.crawlUrl(
waitUntilDone, waitUntilDone,
timeout timeout
); );
``` ```
### Checking Crawl Status ### Checking Crawl Status
To check the status of a crawl job with error handling, use the `checkCrawlStatus` method. It takes the job ID as a parameter and returns the current status of the crawl job. To check the status of a crawl job with error handling, use the `checkCrawlStatus` method. It takes the job ID as a parameter and returns the current status of the crawl job.
@ -352,8 +344,6 @@ const status = await app.checkCrawlStatus(jobId);
console.log(status); console.log(status);
``` ```
### Extracting structured data from a URL ### Extracting structured data from a URL
With LLM extraction, you can easily extract structured data from any URL. We support zod schema to make it easier for you too. Here is how you to use it: With LLM extraction, you can easily extract structured data from any URL. We support zod schema to make it easier for you too. Here is how you to use it:
@ -393,17 +383,28 @@ console.log(scrapeResult.data["llm_extraction"]);
With the `search` method, you can search for a query in a search engine and get the top results along with the page content for each result. The method takes the query as a parameter and returns the search results. With the `search` method, you can search for a query in a search engine and get the top results along with the page content for each result. The method takes the query as a parameter and returns the search results.
```js ```js
const query = 'what is mendable?'; const query = "what is mendable?";
const searchResults = await app.search(query, { const searchResults = await app.search(query, {
pageOptions: { pageOptions: {
fetchPageContent: true // Fetch the page content for each search result fetchPageContent: true, // Fetch the page content for each search result
} },
}); });
``` ```
## Contributing ## Contributing
We love contributions! Please read our [contributing guide](CONTRIBUTING.md) before submitting a pull request. We love contributions! Please read our [contributing guide](CONTRIBUTING.md) before submitting a pull request.
*It is the sole responsibility of the end users to respect websites' policies when scraping, searching and crawling with Firecrawl. Users are advised to adhere to the applicable privacy policies and terms of use of the websites prior to initiating any scraping activities. By default, Firecrawl respects the directives specified in the websites' robots.txt files when crawling. By utilizing Firecrawl, you expressly agree to comply with these conditions.* _It is the sole responsibility of the end users to respect websites' policies when scraping, searching and crawling with Firecrawl. Users are advised to adhere to the applicable privacy policies and terms of use of the websites prior to initiating any scraping activities. By default, Firecrawl respects the directives specified in the websites' robots.txt files when crawling. By utilizing Firecrawl, you expressly agree to comply with these conditions._
## License Disclaimer
This project is primarily licensed under the GNU Affero General Public License v3.0 (AGPL-3.0), as specified in the LICENSE file in the root directory of this repository. However, certain components of this project, specifically the SDKs located in the `/apps/js-sdk` and `/apps/python-sdk` directories, are licensed under the MIT License.
Please note:
- The AGPL-3.0 license applies to all parts of the project unless otherwise specified.
- The SDKs in `/apps/js-sdk` and `/apps/python-sdk` are licensed under the MIT License. Refer to the LICENSE files in these specific directories for details.
- When using or contributing to this project, ensure you comply with the appropriate license terms for the specific component you are working with.
For more details on the licensing of specific components, please refer to the LICENSE files in the respective directories or contact the project maintainers.

21
apps/js-sdk/LICENSE Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 Sideguide Technologies Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

151
apps/js-sdk/README.md Normal file
View File

@ -0,0 +1,151 @@
# Firecrawl Node SDK
The Firecrawl Node SDK is a library that allows you to easily scrape and crawl websites, and output the data in a format ready for use with language models (LLMs). It provides a simple and intuitive interface for interacting with the Firecrawl API.
## Installation
To install the Firecrawl Node SDK, you can use npm:
```bash
npm install @mendable/firecrawl-js
```
## Usage
1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class.
Here's an example of how to use the SDK with error handling:
```js
import FirecrawlApp from "@mendable/firecrawl-js";
// Initialize the FirecrawlApp with your API key
const app = new FirecrawlApp({ apiKey: "YOUR_API_KEY" });
// Scrape a single URL
const url = "https://mendable.ai";
const scrapedData = await app.scrapeUrl(url);
// Crawl a website
const crawlUrl = "https://mendable.ai";
const params = {
crawlerOptions: {
excludes: ["blog/"],
includes: [], // leave empty for all pages
limit: 1000,
},
pageOptions: {
onlyMainContent: true,
},
};
const crawlResult = await app.crawlUrl(crawlUrl, params);
```
### Scraping a URL
To scrape a single URL with error handling, use the `scrapeUrl` method. It takes the URL as a parameter and returns the scraped data as a dictionary.
```js
const url = "https://example.com";
const scrapedData = await app.scrapeUrl(url);
```
### Crawling a Website
To crawl a website with error handling, use the `crawlUrl` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
```js
const crawlUrl = "https://example.com";
const params = {
crawlerOptions: {
excludes: ["blog/"],
includes: [], // leave empty for all pages
limit: 1000,
},
pageOptions: {
onlyMainContent: true,
},
};
const waitUntilDone = true;
const pollInterval = 5;
const crawlResult = await app.crawlUrl(
crawlUrl,
params,
waitUntilDone,
pollInterval
);
```
### Checking Crawl Status
To check the status of a crawl job with error handling, use the `checkCrawlStatus` method. It takes the job ID as a parameter and returns the current status of the crawl job.
```js
const status = await app.checkCrawlStatus(jobId);
```
### Extracting structured data from a URL
With LLM extraction, you can easily extract structured data from any URL. We support zod schema to make it easier for you too. Here is how you to use it:
```js
import FirecrawlApp from "@mendable/firecrawl-js";
import { z } from "zod";
const app = new FirecrawlApp({
apiKey: "fc-YOUR_API_KEY",
});
// Define schema to extract contents into
const schema = z.object({
top: z
.array(
z.object({
title: z.string(),
points: z.number(),
by: z.string(),
commentsURL: z.string(),
})
)
.length(5)
.describe("Top 5 stories on Hacker News"),
});
const scrapeResult = await app.scrapeUrl("https://firecrawl.dev", {
extractorOptions: { extractionSchema: schema },
});
console.log(scrapeResult.data["llm_extraction"]);
```
### Search for a query
With the `search` method, you can search for a query in a search engine and get the top results along with the page content for each result. The method takes the query as a parameter and returns the search results.
```js
const query = "what is mendable?";
const searchResults = await app.search(query, {
pageOptions: {
fetchPageContent: true, // Fetch the page content for each search result
},
});
```
## Error Handling
The SDK handles errors returned by the Firecrawl API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message. The examples above demonstrate how to handle these errors using `try/catch` blocks.
## License
The Firecrawl Node SDK is licensed under the MIT License. This means you are free to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the SDK, subject to the following conditions:
- The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Please note that while this SDK is MIT licensed, it is part of a larger project which may be under different licensing terms. Always refer to the license information in the root directory of the main project for overall licensing details.

21
apps/python-sdk/LICENSE Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 Sideguide Technologies Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

13
apps/python-sdk/README.md
View File

@ -15,7 +15,6 @@ pip install firecrawl-py
1. Get an API key from [firecrawl.dev](https://firecrawl.dev) 1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class. 2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class.
Here's an example of how to use the SDK: Here's an example of how to use the SDK:
```python ```python
@ -46,6 +45,7 @@ To scrape a single URL, use the `scrape_url` method. It takes the URL as a param
url = 'https://example.com' url = 'https://example.com'
scraped_data = app.scrape_url(url) scraped_data = app.scrape_url(url)
``` ```
### Extracting structured data from a URL ### Extracting structured data from a URL
With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it: With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it:
@ -126,20 +126,27 @@ To ensure the functionality of the Firecrawl Python SDK, we have included end-to
To run the tests, execute the following commands: To run the tests, execute the following commands:
Install pytest: Install pytest:
```bash ```bash
pip install pytest pip install pytest
``` ```
Run: Run:
```bash ```bash
pytest firecrawl/__tests__/e2e_withAuth/test.py pytest firecrawl/__tests__/e2e_withAuth/test.py
``` ```
## Contributing ## Contributing
Contributions to the Firecrawl Python SDK are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository. Contributions to the Firecrawl Python SDK are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.
## License ## License
The Firecrawl Python SDK is open-source and released under the [AGPL License](https://www.gnu.org/licenses/agpl-3.0.en.html). The Firecrawl Python SDK is licensed under the MIT License. This means you are free to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the SDK, subject to the following conditions:
- The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Please note that while this SDK is MIT licensed, it is part of a larger project which may be under different licensing terms. Always refer to the license information in the root directory of the main project for overall licensing details.