diff options
100 files changed, 12024 insertions, 3334 deletions
@@ -1,6 +1,5 @@ /target -Cargo.lock -obnam.pdf -obnam.html +*.pdf +*.html test.py test.log diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..161f221 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,29 @@ +variables: + # GitLab CI can only cache data that resides in /builds/ and /cache/ + # directories[1]. Both of these locations are *not* writeable to non-privileged + # users, but the project directory (the Git workdir) is — it's + # world-writeable. So we ask Cargo to put its caches inside the Git workdir. + # + # 1. https://gitlab.com/gitlab-org/gitlab-runner/-/issues/327 + CARGO_HOME: $CI_PROJECT_DIR/.cargo + +check: + parallel: + matrix: + - IMAGE: bullseye-main + image: registry.gitlab.com/obnam/container-images:$IMAGE + script: + # If any of the checks fail, print out the Subplot log and propagate the + # error. + - ./check -v || (cat test.log; exit 1) + # Remove all build artifacts unrelated to the currently installed Rust + # toolchain(s). We have to tweak the PATH because of the caching-related + # shenanigans described in the "variables" section above. + - PATH=${CARGO_INSTALL_ROOT}/bin:${PATH} cargo sweep --installed + + cache: + key: check-job-cache-for-docker + paths: + - $CARGO_HOME/registry/cache + - $CARGO_HOME/registry/index + - target @@ -1,200 +1,192 @@ - GNU GENERAL PUBLIC LICENSE - Version 3, 29 June 2007 +### GNU AFFERO GENERAL PUBLIC LICENSE - Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/> - Everyone is permitted to copy and distribute verbatim copies - of this license document, but changing it is not allowed. +Version 3, 19 November 2007 - Preamble +Copyright (C) 2007 Free Software Foundation, Inc. +<https://fsf.org/> - The GNU General Public License is a free, copyleft license for -software and other kinds of works. +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. - The licenses for most software and other practical works are designed -to take away your freedom to share and change the works. By contrast, -the GNU General Public License is intended to guarantee your freedom to -share and change all versions of a program--to make sure it remains free -software for all its users. We, the Free Software Foundation, use the -GNU General Public License for most of our software; it applies also to -any other work released this way by its authors. You can apply it to -your programs, too. +### Preamble - When we speak of free software, we are referring to freedom, not -price. Our General Public Licenses are designed to make sure that you +The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains +free software for all its users. + +When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. - To protect your rights, we need to prevent others from denying you -these rights or asking you to surrender the rights. Therefore, you have -certain responsibilities if you distribute copies of the software, or if -you modify it: responsibilities to respect the freedom of others. - - For example, if you distribute copies of such a program, whether -gratis or for a fee, you must pass on to the recipients the same -freedoms that you received. You must make sure that they, too, receive -or can get the source code. And you must show them these terms so they -know their rights. - - Developers that use the GNU GPL protect your rights with two steps: -(1) assert copyright on the software, and (2) offer you this License -giving you legal permission to copy, distribute and/or modify it. - - For the developers' and authors' protection, the GPL clearly explains -that there is no warranty for this free software. For both users' and -authors' sake, the GPL requires that modified versions be marked as -changed, so that their problems will not be attributed erroneously to -authors of previous versions. - - Some devices are designed to deny users access to install or run -modified versions of the software inside them, although the manufacturer -can do so. This is fundamentally incompatible with the aim of -protecting users' freedom to change the software. The systematic -pattern of such abuse occurs in the area of products for individuals to -use, which is precisely where it is most unacceptable. Therefore, we -have designed this version of the GPL to prohibit the practice for those -products. If such problems arise substantially in other domains, we -stand ready to extend this provision to those domains in future versions -of the GPL, as needed to protect the freedom of users. - - Finally, every program is threatened constantly by software patents. -States should not allow patents to restrict development and use of -software on general-purpose computers, but in those that do, we wish to -avoid the special danger that patents applied to a free program could -make it effectively proprietary. To prevent this, the GPL assures that -patents cannot be used to render the program non-free. - - The precise terms and conditions for copying, distribution and +Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + +A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + +The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + +An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing +under this license. + +The precise terms and conditions for copying, distribution and modification follow. - TERMS AND CONDITIONS +### TERMS AND CONDITIONS - 0. Definitions. +#### 0. Definitions. - "This License" refers to version 3 of the GNU General Public License. +"This License" refers to version 3 of the GNU Affero General Public +License. - "Copyright" also means copyright-like laws that apply to other kinds of -works, such as semiconductor masks. +"Copyright" also means copyright-like laws that apply to other kinds +of works, such as semiconductor masks. - "The Program" refers to any copyrightable work licensed under this -License. Each licensee is addressed as "you". "Licensees" and +"The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. - To "modify" a work means to copy from or adapt all or part of the work -in a fashion requiring copyright permission, other than the making of an -exact copy. The resulting work is called a "modified version" of the -earlier work or a work "based on" the earlier work. +To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of +an exact copy. The resulting work is called a "modified version" of +the earlier work or a work "based on" the earlier work. - A "covered work" means either the unmodified Program or a work based +A "covered work" means either the unmodified Program or a work based on the Program. - To "propagate" a work means to do anything with it that, without +To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a -computer or modifying a private copy. Propagation includes copying, +computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. - To "convey" a work means any kind of propagation that enables other -parties to make or receive copies. Mere interaction with a user through -a computer network, with no transfer of a copy, is not conveying. +To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user +through a computer network, with no transfer of a copy, is not +conveying. - An interactive user interface displays "Appropriate Legal Notices" -to the extent that it includes a convenient and prominently visible +An interactive user interface displays "Appropriate Legal Notices" to +the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the -work under this License, and how to view a copy of this License. If +work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. - 1. Source Code. +#### 1. Source Code. - The "source code" for a work means the preferred form of the work -for making modifications to it. "Object code" means any non-source -form of a work. +The "source code" for a work means the preferred form of the work for +making modifications to it. "Object code" means any non-source form of +a work. - A "Standard Interface" means an interface that either is an official +A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. - The "System Libraries" of an executable work include anything, other +The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an -implementation is available to the public in source code form. A +implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. - The "Corresponding Source" for a work in object code form means all +The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to -control those activities. However, it does not include the work's +control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but -which are not part of the work. For example, Corresponding Source +which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. - The Corresponding Source need not include anything that users -can regenerate automatically from other parts of the Corresponding -Source. +The Corresponding Source need not include anything that users can +regenerate automatically from other parts of the Corresponding Source. - The Corresponding Source for a work in source code form is that -same work. +The Corresponding Source for a work in source code form is that same +work. - 2. Basic Permissions. +#### 2. Basic Permissions. - All rights granted under this License are granted for the term of +All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated -conditions are met. This License explicitly affirms your unlimited -permission to run the unmodified Program. The output from running a +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its -content, constitutes a covered work. This License acknowledges your +content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. - You may make, run and propagate covered works that you do not -convey, without conditions so long as your license otherwise remains -in force. You may convey covered works to others for the sole purpose -of having them make modifications exclusively for you, or provide you -with facilities for running those works, provided that you comply with -the terms of this License in conveying all material for which you do -not control copyright. Those thus making or running the covered works -for you must do so exclusively on your behalf, under your direction -and control, on terms that prohibit them from making any copies of -your copyrighted material outside their relationship with you. +You may make, run and propagate covered works that you do not convey, +without conditions so long as your license otherwise remains in force. +You may convey covered works to others for the sole purpose of having +them make modifications exclusively for you, or provide you with +facilities for running those works, provided that you comply with the +terms of this License in conveying all material for which you do not +control copyright. Those thus making or running the covered works for +you must do so exclusively on your behalf, under your direction and +control, on terms that prohibit them from making any copies of your +copyrighted material outside their relationship with you. - Conveying under any other circumstances is permitted solely under -the conditions stated below. Sublicensing is not allowed; section 10 -makes it unnecessary. +Conveying under any other circumstances is permitted solely under the +conditions stated below. Sublicensing is not allowed; section 10 makes +it unnecessary. - 3. Protecting Users' Legal Rights From Anti-Circumvention Law. +#### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. - No covered work shall be deemed part of an effective technological +No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. - When you convey a covered work, you waive any legal power to forbid -circumvention of technological measures to the extent such circumvention -is effected by exercising rights under this License with respect to -the covered work, and you disclaim any intention to limit operation or -modification of the work as a means of enforcing, against the work's -users, your or third parties' legal rights to forbid circumvention of -technological measures. +When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such +circumvention is effected by exercising rights under this License with +respect to the covered work, and you disclaim any intention to limit +operation or modification of the work as a means of enforcing, against +the work's users, your or third parties' legal rights to forbid +circumvention of technological measures. - 4. Conveying Verbatim Copies. +#### 4. Conveying Verbatim Copies. - You may convey verbatim copies of the Program's source code as you +You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any @@ -202,59 +194,56 @@ non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. - You may charge any price or no price for each copy that you convey, +You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. - 5. Conveying Modified Source Versions. +#### 5. Conveying Modified Source Versions. - You may convey a work based on the Program, or the modifications to +You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the -terms of section 4, provided that you also meet all of these conditions: +terms of section 4, provided that you also meet all of these +conditions: - a) The work must carry prominent notices stating that you modified +- a) The work must carry prominent notices stating that you modified it, and giving a relevant date. - - b) The work must carry prominent notices stating that it is - released under this License and any conditions added under section - 7. This requirement modifies the requirement in section 4 to - "keep intact all notices". - - c) You must license the entire work, as a whole, under this - License to anyone who comes into possession of a copy. This +- b) The work must carry prominent notices stating that it is + released under this License and any conditions added under + section 7. This requirement modifies the requirement in section 4 + to "keep intact all notices". +- c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, - regardless of how they are packaged. This License gives no + regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. - - d) If the work has interactive user interfaces, each must display +- d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. - A compilation of a covered work with other separate and independent +A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users -beyond what the individual works permit. Inclusion of a covered work +beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. - 6. Conveying Non-Source Forms. +#### 6. Conveying Non-Source Forms. - You may convey a covered work in object code form under the terms -of sections 4 and 5, provided that you also convey the -machine-readable Corresponding Source under the terms of this License, -in one of these ways: +You may convey a covered work in object code form under the terms of +sections 4 and 5, provided that you also convey the machine-readable +Corresponding Source under the terms of this License, in one of these +ways: - a) Convey the object code in, or embodied in, a physical product +- a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. - - b) Convey the object code in, or embodied in, a physical product +- b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product @@ -263,196 +252,190 @@ in one of these ways: product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this - conveying of source, or (2) access to copy the - Corresponding Source from a network server at no charge. - - c) Convey individual copies of the object code with a copy of the - written offer to provide the Corresponding Source. This + conveying of source, or (2) access to copy the Corresponding + Source from a network server at no charge. +- c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. - - d) Convey the object code by offering access from a designated +- d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no - further charge. You need not require recipients to copy the - Corresponding Source along with the object code. If the place to + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the - Corresponding Source. Regardless of what server hosts the + Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. +- e) Convey the object code using peer-to-peer transmission, + provided you inform other peers where the object code and + Corresponding Source of the work are being offered to the general + public at no charge under subsection 6d. - e) Convey the object code using peer-to-peer transmission, provided - you inform other peers where the object code and Corresponding - Source of the work are being offered to the general public at no - charge under subsection 6d. - - A separable portion of the object code, whose source code is excluded +A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. - A "User Product" is either (1) a "consumer product", which means any -tangible personal property which is normally used for personal, family, -or household purposes, or (2) anything designed or sold for incorporation -into a dwelling. In determining whether a product is a consumer product, -doubtful cases shall be resolved in favor of coverage. For a particular -product received by a particular user, "normally used" refers to a -typical or common use of that class of product, regardless of the status -of the particular user or of the way in which the particular user -actually uses, or expects or is expected to use, the product. A product -is a consumer product regardless of whether the product has substantial -commercial, industrial or non-consumer uses, unless such uses represent -the only significant mode of use of the product. - - "Installation Information" for a User Product means any methods, -procedures, authorization keys, or other information required to install -and execute modified versions of a covered work in that User Product from -a modified version of its Corresponding Source. The information must -suffice to ensure that the continued functioning of the modified object -code is in no case prevented or interfered with solely because -modification has been made. - - If you convey an object code work under this section in, or with, or +A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, +family, or household purposes, or (2) anything designed or sold for +incorporation into a dwelling. In determining whether a product is a +consumer product, doubtful cases shall be resolved in favor of +coverage. For a particular product received by a particular user, +"normally used" refers to a typical or common use of that class of +product, regardless of the status of the particular user or of the way +in which the particular user actually uses, or expects or is expected +to use, the product. A product is a consumer product regardless of +whether the product has substantial commercial, industrial or +non-consumer uses, unless such uses represent the only significant +mode of use of the product. + +"Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to +install and execute modified versions of a covered work in that User +Product from a modified version of its Corresponding Source. The +information must suffice to ensure that the continued functioning of +the modified object code is in no case prevented or interfered with +solely because modification has been made. + +If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied -by the Installation Information. But this requirement does not apply +by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). - The requirement to provide Installation Information does not include a -requirement to continue to provide support service, warranty, or updates -for a work that has been modified or installed by the recipient, or for -the User Product in which it has been modified or installed. Access to a -network may be denied when the modification itself materially and -adversely affects the operation of the network or violates the rules and -protocols for communication across the network. +The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or +updates for a work that has been modified or installed by the +recipient, or for the User Product in which it has been modified or +installed. Access to a network may be denied when the modification +itself materially and adversely affects the operation of the network +or violates the rules and protocols for communication across the +network. - Corresponding Source conveyed, and Installation Information provided, +Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. - 7. Additional Terms. +#### 7. Additional Terms. - "Additional permissions" are terms that supplement the terms of this +"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent -that they are valid under applicable law. If additional permissions +that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. - When you convey a copy of a covered work, you may at your option +When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of -it. (Additional permissions may be written to require their own -removal in certain cases when you modify the work.) You may place +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. - Notwithstanding any other provision of this License, for material you -add to a covered work, you may (if authorized by the copyright holders of -that material) supplement the terms of this License with terms: +Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders +of that material) supplement the terms of this License with terms: - a) Disclaiming warranty or limiting liability differently from the +- a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or - - b) Requiring preservation of specified reasonable legal notices or +- b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or - - c) Prohibiting misrepresentation of the origin of that material, or - requiring that modified versions of such material be marked in +- c) Prohibiting misrepresentation of the origin of that material, + or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or - - d) Limiting the use for publicity purposes of names of licensors or - authors of the material; or - - e) Declining to grant rights under trademark law for use of some +- d) Limiting the use for publicity purposes of names of licensors + or authors of the material; or +- e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or - - f) Requiring indemnification of licensors and authors of that - material by anyone who conveys the material (or modified versions of - it) with contractual assumptions of liability to the recipient, for - any liability that these contractual assumptions directly impose on - those licensors and authors. - - All other non-permissive additional terms are considered "further -restrictions" within the meaning of section 10. If the Program as you +- f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions + of it) with contractual assumptions of liability to the recipient, + for any liability that these contractual assumptions directly + impose on those licensors and authors. + +All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further -restriction, you may remove that term. If a license document contains +restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. - If you add terms to a covered work in accord with this section, you +If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. - Additional terms, permissive or non-permissive, may be stated in the -form of a separately written license, or stated as exceptions; -the above requirements apply either way. +Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; the +above requirements apply either way. - 8. Termination. +#### 8. Termination. - You may not propagate or modify a covered work except as expressly -provided under this License. Any attempt otherwise to propagate or +You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). - However, if you cease all violation of this License, then your -license from a particular copyright holder is reinstated (a) -provisionally, unless and until the copyright holder explicitly and -finally terminates your license, and (b) permanently, if the copyright -holder fails to notify you of the violation by some reasonable means -prior to 60 days after the cessation. +However, if you cease all violation of this License, then your license +from a particular copyright holder is reinstated (a) provisionally, +unless and until the copyright holder explicitly and finally +terminates your license, and (b) permanently, if the copyright holder +fails to notify you of the violation by some reasonable means prior to +60 days after the cessation. - Moreover, your license from a particular copyright holder is +Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. - Termination of your rights under this section does not terminate the +Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under -this License. If your rights have been terminated and not permanently +this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. - 9. Acceptance Not Required for Having Copies. +#### 9. Acceptance Not Required for Having Copies. - You are not required to accept this License in order to receive or -run a copy of the Program. Ancillary propagation of a covered work +You are not required to accept this License in order to receive or run +a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission -to receive a copy likewise does not require acceptance. However, +to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or -modify any covered work. These actions infringe copyright if you do -not accept this License. Therefore, by modifying or propagating a +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. - 10. Automatic Licensing of Downstream Recipients. +#### 10. Automatic Licensing of Downstream Recipients. - Each time you convey a covered work, the recipient automatically +Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and -propagate that work, subject to this License. You are not responsible +propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. - An "entity transaction" is a transaction transferring control of an +An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an -organization, or merging organizations. If propagation of a covered +organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could @@ -460,43 +443,43 @@ give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. - You may not impose any further restrictions on the exercise of the -rights granted or affirmed under this License. For example, you may +You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. - 11. Patents. +#### 11. Patents. - A "contributor" is a copyright holder who authorizes use under this -License of the Program or a work on which the Program is based. The +A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". - A contributor's "essential patent claims" are all patent claims -owned or controlled by the contributor, whether already acquired or +A contributor's "essential patent claims" are all patent claims owned +or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a -consequence of further modification of the contributor version. For +consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. - Each contributor grants you a non-exclusive, worldwide, royalty-free +Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. - In the following three paragraphs, a "patent license" is any express +In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to -sue for patent infringement). To "grant" such a patent license to a +sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. - If you convey a covered work, knowingly relying on a patent license, +If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, @@ -504,13 +487,13 @@ then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent -license to downstream recipients. "Knowingly relying" means you have +license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. - If, pursuant to or in connection with a single transaction or +If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify @@ -518,157 +501,160 @@ or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. - A patent license is "discriminatory" if it does not include within -the scope of its coverage, prohibits the exercise of, or is -conditioned on the non-exercise of one or more of the rights that are -specifically granted under this License. You may not convey a covered -work if you are a party to an arrangement with a third party that is -in the business of distributing software, under which you make payment -to the third party based on the extent of your activity of conveying -the work, and under which the third party grants, to any of the -parties who would receive the covered work from you, a discriminatory -patent license (a) in connection with copies of the covered work -conveyed by you (or copies made from those copies), or (b) primarily -for and in connection with specific products or compilations that -contain the covered work, unless you entered into that arrangement, -or that patent license was granted, prior to 28 March 2007. - - Nothing in this License shall be construed as excluding or limiting +A patent license is "discriminatory" if it does not include within the +scope of its coverage, prohibits the exercise of, or is conditioned on +the non-exercise of one or more of the rights that are specifically +granted under this License. You may not convey a covered work if you +are a party to an arrangement with a third party that is in the +business of distributing software, under which you make payment to the +third party based on the extent of your activity of conveying the +work, and under which the third party grants, to any of the parties +who would receive the covered work from you, a discriminatory patent +license (a) in connection with copies of the covered work conveyed by +you (or copies made from those copies), or (b) primarily for and in +connection with specific products or compilations that contain the +covered work, unless you entered into that arrangement, or that patent +license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. - 12. No Surrender of Others' Freedom. +#### 12. No Surrender of Others' Freedom. - If conditions are imposed on you (whether by court order, agreement or +If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not -excuse you from the conditions of this License. If you cannot convey a -covered work so as to satisfy simultaneously your obligations under this -License and any other pertinent obligations, then as a consequence you may -not convey it at all. For example, if you agree to terms that obligate you -to collect a royalty for further conveying from those to whom you convey -the Program, the only way you could satisfy both those terms and this -License would be to refrain entirely from conveying the Program. - - 13. Use with the GNU Affero General Public License. - - Notwithstanding any other provision of this License, you have +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under +this License and any other pertinent obligations, then as a +consequence you may not convey it at all. For example, if you agree to +terms that obligate you to collect a royalty for further conveying +from those to whom you convey the Program, the only way you could +satisfy both those terms and this License would be to refrain entirely +from conveying the Program. + +#### 13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your +version supports such interaction) an opportunity to receive the +Corresponding Source of your version by providing access to the +Corresponding Source from a network server at no charge, through some +standard or customary means of facilitating copying of software. This +Corresponding Source shall include the Corresponding Source for any +work covered by version 3 of the GNU General Public License that is +incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed -under version 3 of the GNU Affero General Public License into a single -combined work, and to convey the resulting work. The terms of this +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, -but the special requirements of the GNU Affero General Public License, -section 13, concerning interaction through a network will apply to the -combination as such. +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. - 14. Revised Versions of this License. +#### 14. Revised Versions of this License. - The Free Software Foundation may publish revised and/or new versions of -the GNU General Public License from time to time. Such new versions will -be similar in spirit to the present version, but may differ in detail to -address new problems or concerns. +The Free Software Foundation may publish revised and/or new versions +of the GNU Affero General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. - Each version is given a distinguishing version number. If the -Program specifies that a certain numbered version of the GNU General +Each version is given a distinguishing version number. If the Program +specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software -Foundation. If the Program does not specify a version number of the -GNU General Public License, you may choose any version ever published -by the Free Software Foundation. +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever +published by the Free Software Foundation. - If the Program specifies that a proxy can decide which future -versions of the GNU General Public License can be used, that proxy's +If the Program specifies that a proxy can decide which future versions +of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. - Later license versions may give you additional or different -permissions. However, no additional obligations are imposed on any +Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. - 15. Disclaimer of Warranty. - - THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY -APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT -HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY -OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, -THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR -PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM -IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF -ALL NECESSARY SERVICING, REPAIR OR CORRECTION. - - 16. Limitation of Liability. - - IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING -WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS -THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY -GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE -USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF -DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD -PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), -EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF -SUCH DAMAGES. - - 17. Interpretation of Sections 15 and 16. - - If the disclaimer of warranty and limitation of liability provided +#### 15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT +WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND +PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE +DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +#### 16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR +CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT +NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR +LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM +TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER +PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +#### 17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. - END OF TERMS AND CONDITIONS +END OF TERMS AND CONDITIONS - How to Apply These Terms to Your New Programs +### How to Apply These Terms to Your New Programs - If you develop a new program, and you want it to be of the greatest +If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it -free software which everyone can redistribute and change under these terms. - - To do so, attach the following notices to the program. It is safest -to attach them to the start of each source file to most effectively -state the exclusion of warranty; and each file should have at least -the "copyright" line and a pointer to where the full notice is found. - - <one line to give the program's name and a brief idea of what it does.> - Copyright (C) <year> <name of author> - - This program is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program. If not, see <http://www.gnu.org/licenses/>. - -Also add information on how to contact you by electronic and paper mail. - - If the program does terminal interaction, make it output a short -notice like this when it starts in an interactive mode: - - <program> Copyright (C) <year> <name of author> - This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. - This is free software, and you are welcome to redistribute it - under certain conditions; type `show c' for details. - -The hypothetical commands `show w' and `show c' should show the appropriate -parts of the General Public License. Of course, your program's commands -might be different; for a GUI interface, you would use an "about box". - - You should also get your employer (if you work as a programmer) or school, -if any, to sign a "copyright disclaimer" for the program, if necessary. -For more information on this, and how to apply and follow the GNU GPL, see -<http://www.gnu.org/licenses/>. - - The GNU General Public License does not permit incorporating your program -into proprietary programs. If your program is a subroutine library, you -may consider it more useful to permit linking proprietary applications with -the library. If this is what you want to do, use the GNU Lesser General -Public License instead of this License. But first, please read -<http://www.gnu.org/philosophy/why-not-lgpl.html>. +free software which everyone can redistribute and change under these +terms. + +To do so, attach the following notices to the program. It is safest to +attach them to the start of each source file to most effectively state +the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper +mail. + +If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for +the specific requirements. + +You should also get your employer (if you work as a programmer) or +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. For more information on this, and how to apply and follow +the GNU AGPL, see <https://www.gnu.org/licenses/>. diff --git a/Cargo.lock b/Cargo.lock new file mode 100644 index 0000000..c0e9e3c --- /dev/null +++ b/Cargo.lock @@ -0,0 +1,2526 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 3 + +[[package]] +name = "addr2line" +version = "0.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a30b2e23b9e17a9f90641c7ab1549cd9b44f296d3ccbf309d2863cfe398a0cb" +dependencies = [ + "gimli", +] + +[[package]] +name = "adler" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe" + +[[package]] +name = "aead" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" +dependencies = [ + "crypto-common", + "generic-array", +] + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures", +] + +[[package]] +name = "aes-gcm" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" +dependencies = [ + "aead", + "aes", + "cipher", + "ctr", + "ghash", + "subtle", +] + +[[package]] +name = "ahash" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e89da841a80418a9b391ebaea17f5c112ffaaa96f621d2c285b5174da76b9011" +dependencies = [ + "cfg-if", + "once_cell", + "version_check", + "zerocopy", +] + +[[package]] +name = "aho-corasick" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2969dcb958b36655471fc61f7e416fa76033bdd4bfed0678d8fee1e2d07a1f0" +dependencies = [ + "memchr", +] + +[[package]] +name = "allocator-api2" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0942ffc6dcaadf03badf6e6a2d0228460359d5e34b57ccdc720b7382dfbd5ec5" + +[[package]] +name = "android-tzdata" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e999941b234f3131b00bc13c22d06e8c5ff726d1b6318ac7eb276997bbb4fef0" + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anstream" +version = "0.6.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d96bd03f33fe50a863e394ee9718a706f988b9079b20c3784fb726e7678b62fb" +dependencies = [ + "anstyle", + "anstyle-parse", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "utf8parse", +] + +[[package]] +name = "anstyle" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8901269c6307e8d93993578286ac0edf7f195079ffff5ebdeea6a59ffb7e36bc" + +[[package]] +name = "anstyle-parse" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c75ac65da39e5fe5ab759307499ddad880d724eed2f6ce5b5e8a26f4f387928c" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e28923312444cdd728e4738b3f9c9cac739500909bb3d3c94b43551b16517648" +dependencies = [ + "windows-sys 0.52.0", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1cd54b81ec8d6180e24654d0b371ad22fc3dd083b6ff8ba325b72e00c87660a7" +dependencies = [ + "anstyle", + "windows-sys 0.52.0", +] + +[[package]] +name = "anyhow" +version = "1.0.80" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ad32ce52e4161730f7098c077cd2ed6229b5804ccf99e5366be1ab72a98b4e1" + +[[package]] +name = "arc-swap" +version = "1.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b3d0060af21e8d11a926981cc00c6c1541aa91dd64b9f881985c3da1094425f" + +[[package]] +name = "autocfg" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa" + +[[package]] +name = "backtrace" +version = "0.3.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2089b7e3f35b9dd2d0ed921ead4f6d318c27680d4a5bd167b3ee120edb105837" +dependencies = [ + "addr2line", + "cc", + "cfg-if", + "libc", + "miniz_oxide", + "object", + "rustc-demangle", +] + +[[package]] +name = "base64" +version = "0.21.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d297deb1925b89f2ccc13d7635fa0714f12c87adce1c75356b39ca9b7178567" + +[[package]] +name = "base64ct" +version = "1.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8c3c1a368f70d6cf7302d78f8f7093da241fb8e8807c05cc9e51a125895a6d5b" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "bitflags" +version = "2.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed570934406eb16438a4e976b1b4500774099c13b8cb96eec99f620f05090ddf" + +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest", +] + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bumpalo" +version = "3.15.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ea184aa71bb362a1157c896979544cc23974e08fd265f29ea96b59f0b4a555b" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2bd12c1caf447e69cd4528f47f94d203fd2582878ecb9e9465484c4148a8223" + +[[package]] +name = "bytesize" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3e368af43e418a04d52505cf3dbc23dda4e3407ae2fa99fd0e4f308ce546acc" + +[[package]] +name = "cc" +version = "1.0.88" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02f341c093d19155a6e41631ce5971aac4e9a868262212153124c15fa22d1cdc" + +[[package]] +name = "cfg-if" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" + +[[package]] +name = "chrono" +version = "0.4.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5bc015644b92d5890fab7489e49d21f879d5c990186827d42ec511919404f38b" +dependencies = [ + "android-tzdata", + "iana-time-zone", + "js-sys", + "num-traits", + "wasm-bindgen", + "windows-targets 0.52.4", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", +] + +[[package]] +name = "clap" +version = "4.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c918d541ef2913577a0f9566e9ce27cb35b6df072075769e0b26cb5a554520da" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f3e7391dad68afb0c2ede1bf619f579a3dc9c2ec67f089baa397123a2f3d1eb" +dependencies = [ + "anstream", + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "307bc0538d5f0f83b8248db3087aa92fe504e4691294d0c96c0eabc33f47ba47" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.52", +] + +[[package]] +name = "clap_lex" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "98cc8fbded0c607b7ba9dd60cd98df59af97e84d24e49c8557331cfc26d301ce" + +[[package]] +name = "colorchoice" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "acbf1af155f9b9ef647e42cdc158db4b64a1b61f743629225fde6f3e0be2a7c7" + +[[package]] +name = "console" +version = "0.15.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e1f83fc076bd6dd27517eacdf25fef6c4dfe5f1d7448bafaaf3a26f13b5e4eb" +dependencies = [ + "encode_unicode", + "lazy_static", + "libc", + "unicode-width", + "windows-sys 0.52.0", +] + +[[package]] +name = "core-foundation" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "core-foundation-sys" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06ea2b9bc92be3c2baa9334a323ebca2d6f074ff852cd1d7b11064035cd3868f" + +[[package]] +name = "cpufeatures" +version = "0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53fe5e26ff1b7aef8bca9c6080520cfb8d9333c7568e1829cef191a9723e5504" +dependencies = [ + "libc", +] + +[[package]] +name = "crypto-common" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3" +dependencies = [ + "generic-array", + "rand_core", + "typenum", +] + +[[package]] +name = "ctr" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" +dependencies = [ + "cipher", +] + +[[package]] +name = "data-encoding" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e962a19be5cfc3f3bf6dd8f61eb50107f356ad6270fbb3ed41476571db78be5" + +[[package]] +name = "derivative" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fcc3dd5e9e9c0b295d6e1e4d811fb6f157d5ffd784b8d202fc62eac8035a770b" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "destructure_traitobject" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c877555693c14d2f84191cfd3ad8582790fc52b5e2274b40b59cf5f5cea25c7" + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "directories-next" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "339ee130d97a610ea5a5872d2bbb130fdf68884ff09d3028b81bec8a1ac23bbc" +dependencies = [ + "cfg-if", + "dirs-sys-next", +] + +[[package]] +name = "dirs-sys-next" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ebda144c4fe02d1f7ea1a7d9641b6fc6b580adcfa024ae48797ecdeb6825b4d" +dependencies = [ + "libc", + "redox_users", + "winapi", +] + +[[package]] +name = "encode_unicode" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a357d28ed41a50f9c765dbfe56cbc04a64e53e5fc58ba79fbc34c10ef3df831f" + +[[package]] +name = "encoding_rs" +version = "0.8.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7268b386296a025e474d5140678f75d6de9493ae55a5d709eeb9dd08149945e1" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "env_logger" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cd405aab171cb85d6735e5c8d9db038c17d3ca007a4d2c25f337935c3d90580" +dependencies = [ + "humantime", + "is-terminal", + "log", + "regex", + "termcolor", +] + +[[package]] +name = "equivalent" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5443807d6dff69373d433ab9ef5378ad8df50ca6298caf15de6e52e24aaf54d5" + +[[package]] +name = "errno" +version = "0.3.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a258e46cdc063eb8519c00b9fc845fc47bcfca4130e2f08e88665ceda8474245" +dependencies = [ + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "fallible-iterator" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649" + +[[package]] +name = "fallible-streaming-iterator" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a" + +[[package]] +name = "fastrand" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25cbce373ec4653f1a01a31e8a5e5ec0c622dc27ff9c4e6606eefef5cbbed4a5" + +[[package]] +name = "fnv" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" + +[[package]] +name = "foreign-types" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1" +dependencies = [ + "foreign-types-shared", +] + +[[package]] +name = "foreign-types-shared" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b" + +[[package]] +name = "form_urlencoded" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13624c2627564efccf4934284bdd98cbaa14e79b0b5a141218e507b3a823456" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "645c6916888f6cb6350d2550b80fb63e734897a8498abe35cfb732b6487804b0" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eac8f7d7865dcb88bd4373ab671c8cf4508703796caa2b1985a9ca867b3fcb78" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfc6580bb841c5a68e9ef15c77ccc837b40a7504914d52e47b8b0e9bbda25a1d" + +[[package]] +name = "futures-executor" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a576fc72ae164fca6b9db127eaa9a9dda0d61316034f33a0a0d4eda41f02b01d" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a44623e20b9681a318efdd71c299b6b222ed6f231972bfe2f224ebad6311f0c1" + +[[package]] +name = "futures-macro" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87750cf4b7a4c0625b1529e4c543c2182106e4dedc60a2a6455e00d212c489ac" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", +] + +[[package]] +name = "futures-sink" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9fb8e00e87438d937621c1c6269e53f536c14d3fbd6a042bb24879e57d474fb5" + +[[package]] +name = "futures-task" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38d84fa142264698cdce1a9f9172cf383a0c82de1bddcf3092901442c4097004" + +[[package]] +name = "futures-util" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d6401deb83407ab3da39eba7e33987a73c3df0c82b4bb5813ee871c19c41d48" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "pin-utils", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "190092ea657667030ac6a35e305e62fc4dd69fd98ac98631e5d3a2b1575a12b5" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "ghash" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d930750de5717d2dd0b8c0d42c076c0e884c81a73e6cab859bbd2339c71e3e40" +dependencies = [ + "opaque-debug", + "polyval", +] + +[[package]] +name = "gimli" +version = "0.28.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4271d37baee1b8c7e4b708028c57d816cf9d2434acb33a549475f78c181f6253" + +[[package]] +name = "h2" +version = "0.3.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb2c4422095b67ee78da96fbb51a4cc413b3b25883c7717ff7ca1ab31022c9c9" +dependencies = [ + "bytes", + "fnv", + "futures-core", + "futures-sink", + "futures-util", + "http", + "indexmap", + "slab", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "hashbrown" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "290f1a1d9242c78d09ce40a5e87e7554ee637af1351968159f4952f028f75604" +dependencies = [ + "ahash", + "allocator-api2", +] + +[[package]] +name = "hashlink" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "692eaaf7f7607518dd3cef090f1474b61edc5301d8012f09579920df68b725ee" +dependencies = [ + "hashbrown", +] + +[[package]] +name = "headers" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06683b93020a07e3dbcf5f8c0f6d40080d725bea7936fc01ad345c01b97dc270" +dependencies = [ + "base64", + "bytes", + "headers-core", + "http", + "httpdate", + "mime", + "sha1", +] + +[[package]] +name = "headers-core" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7f66481bfee273957b1f20485a4ff3362987f85b2c236580d81b4eb7a326429" +dependencies = [ + "http", +] + +[[package]] +name = "heck" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8" + +[[package]] +name = "hermit-abi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d231dfb89cfffdbc30e7fc41579ed6066ad03abda9e567ccafae602b97ec5024" + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "http" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8947b1a6fad4393052c7ba1f4cd97bed3e953a95c79c92ad9b051a04611d9fbb" +dependencies = [ + "bytes", + "fnv", + "itoa", +] + +[[package]] +name = "http-body" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ceab25649e9960c0311ea418d17bee82c0dcec1bd053b5f9a66e265a693bed2" +dependencies = [ + "bytes", + "http", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d897f394bad6a705d5f4104762e116a75639e470d80901eed05a860a95cb1904" + +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + +[[package]] +name = "humantime" +version = "2.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4" + +[[package]] +name = "hyper" +version = "0.14.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf96e135eb83a2a8ddf766e426a841d8ddd7449d5f00d34ea02b41d2f19eef80" +dependencies = [ + "bytes", + "futures-channel", + "futures-core", + "futures-util", + "h2", + "http", + "http-body", + "httparse", + "httpdate", + "itoa", + "pin-project-lite", + "socket2", + "tokio", + "tower-service", + "tracing", + "want", +] + +[[package]] +name = "hyper-tls" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6183ddfa99b85da61a140bea0efc93fdf56ceaa041b37d553518030827f9905" +dependencies = [ + "bytes", + "hyper", + "native-tls", + "tokio", + "tokio-native-tls", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.60" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7ffbb5a1b541ea2561f8c41c087286cc091e21e556a4f09a8f6cbf17b69b141" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "idna" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "634d9b1461af396cad843f47fdba5597a4f9e6ddd4bfb6ff5d85028c25cb12f6" +dependencies = [ + "unicode-bidi", + "unicode-normalization", +] + +[[package]] +name = "indexmap" +version = "2.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b0b929d511467233429c45a44ac1dcaa21ba0f5ba11e4879e6ed28ddb4f9df4" +dependencies = [ + "equivalent", + "hashbrown", +] + +[[package]] +name = "indicatif" +version = "0.17.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "763a5a8f45087d6bcea4222e7b72c291a054edf80e4ef6efd2a4979878c7bea3" +dependencies = [ + "console", + "instant", + "number_prefix", + "portable-atomic", + "unicode-width", +] + +[[package]] +name = "inout" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0c10553d664a4d0bcff9f4215d0aac67a639cc68ef660840afe309b807bc9f5" +dependencies = [ + "generic-array", +] + +[[package]] +name = "instant" +version = "0.1.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a5bbe824c507c5da5956355e86a746d82e0e1464f65d862cc5e71da70e94b2c" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "ipnet" +version = "2.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f518f335dce6725a761382244631d86cf0ccb2863413590b31338feb467f9c3" + +[[package]] +name = "is-terminal" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f23ff5ef2b80d608d61efee834934d862cd92461afc0560dedf493e4c033738b" +dependencies = [ + "hermit-abi", + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "itoa" +version = "1.0.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1a46d1a171d865aa5f83f92695765caa047a9b4cbae2cbf37dbd613a793fd4c" + +[[package]] +name = "js-sys" +version = "0.3.68" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "406cda4b368d531c842222cf9d2600a9a4acce8d29423695379c6868a143a9ee" +dependencies = [ + "wasm-bindgen", +] + +[[package]] +name = "lazy_static" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" + +[[package]] +name = "libc" +version = "0.2.153" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c198f91728a82281a64e1f4f9eeb25d82cb32a5de251c6bd1b5154d63a8e7bd" + +[[package]] +name = "libredox" +version = "0.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85c833ca1e66078851dba29046874e38f08b2c883700aa29a03ddd3b23814ee8" +dependencies = [ + "bitflags 2.4.2", + "libc", + "redox_syscall", +] + +[[package]] +name = "libsqlite3-sys" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c10584274047cb335c23d3e61bcef8e323adae7c5c8c760540f73610177fc3f" +dependencies = [ + "pkg-config", + "vcpkg", +] + +[[package]] +name = "linux-raw-sys" +version = "0.4.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01cda141df6706de531b6c46c3a33ecca755538219bd484262fa09410c13539c" + +[[package]] +name = "lock_api" +version = "0.4.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c168f8615b12bc01f9c17e2eb0cc07dcae1940121185446edc3744920e8ef45" +dependencies = [ + "autocfg", + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90ed8c1e510134f979dbc4f070f87d4313098b704861a105fe34231c70a3901c" +dependencies = [ + "serde", +] + +[[package]] +name = "log-mdc" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a94d21414c1f4a51209ad204c1776a3d0765002c76c6abcb602a6f09f1e881c7" + +[[package]] +name = "log4rs" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0816135ae15bd0391cf284eab37e6e3ee0a6ee63d2ceeb659862bd8d0a984ca6" +dependencies = [ + "anyhow", + "arc-swap", + "chrono", + "derivative", + "fnv", + "humantime", + "libc", + "log", + "log-mdc", + "once_cell", + "parking_lot", + "rand", + "serde", + "serde-value", + "serde_json", + "serde_yaml", + "thiserror", + "thread-id", + "typemap-ors", + "winapi", +] + +[[package]] +name = "memchr" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "523dc4f511e55ab87b694dc30d0f820d60906ef06413f93d4d7a1385599cc149" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mime_guess" +version = "2.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4192263c238a5f0d0c6bfd21f336a313a4ce1c450542449ca191bb657b4642ef" +dependencies = [ + "mime", + "unicase", +] + +[[package]] +name = "miniz_oxide" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d811f3e15f28568be3407c8e7fdb6514c1cda3cb30683f15b6a1a1dc4ea14a7" +dependencies = [ + "adler", +] + +[[package]] +name = "mio" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4a650543ca06a924e8b371db273b2756685faae30f8487da1b56505a8f78b0c" +dependencies = [ + "libc", + "wasi", + "windows-sys 0.48.0", +] + +[[package]] +name = "multer" +version = "2.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01acbdc23469fd8fe07ab135923371d5f5a422fbf9c522158677c8eb15bc51c2" +dependencies = [ + "bytes", + "encoding_rs", + "futures-util", + "http", + "httparse", + "log", + "memchr", + "mime", + "spin", + "version_check", +] + +[[package]] +name = "native-tls" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07226173c32f2926027b63cce4bcd8076c3552846cbe7925f3aaffeac0a3b92e" +dependencies = [ + "lazy_static", + "libc", + "log", + "openssl", + "openssl-probe", + "openssl-sys", + "schannel", + "security-framework", + "security-framework-sys", + "tempfile", +] + +[[package]] +name = "num-traits" +version = "0.2.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da0df0e5185db44f69b44f26786fe401b6c293d1907744beaa7fa62b2e5a517a" +dependencies = [ + "autocfg", +] + +[[package]] +name = "num_cpus" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4161fcb6d602d4d2081af7c3a45852d875a03dd337a6bfdd6e06407b61342a43" +dependencies = [ + "hermit-abi", + "libc", +] + +[[package]] +name = "number_prefix" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3" + +[[package]] +name = "object" +version = "0.32.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6a622008b6e321afc04970976f62ee297fdbaa6f95318ca343e3eebb9648441" +dependencies = [ + "memchr", +] + +[[package]] +name = "obnam" +version = "0.8.1" +dependencies = [ + "aes-gcm", + "anyhow", + "blake2", + "bytesize", + "chrono", + "clap", + "directories-next", + "futures", + "indicatif", + "libc", + "log", + "log4rs", + "pbkdf2", + "pretty_env_logger", + "rand", + "reqwest", + "rpassword", + "rusqlite", + "serde", + "serde_json", + "serde_yaml", + "sha2", + "spmc", + "tempfile", + "thiserror", + "tokio", + "users", + "uuid", + "walkdir", + "warp", +] + +[[package]] +name = "once_cell" +version = "1.19.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92" + +[[package]] +name = "opaque-debug" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" + +[[package]] +name = "openssl" +version = "0.10.64" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95a0481286a310808298130d22dd1fef0fa571e05a8f44ec801801e84b216b1f" +dependencies = [ + "bitflags 2.4.2", + "cfg-if", + "foreign-types", + "libc", + "once_cell", + "openssl-macros", + "openssl-sys", +] + +[[package]] +name = "openssl-macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", +] + +[[package]] +name = "openssl-probe" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff011a302c396a5197692431fc1948019154afc178baf7d8e37367442a4601cf" + +[[package]] +name = "openssl-sys" +version = "0.9.101" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dda2b0f344e78efc2facf7d195d098df0dd72151b26ab98da807afc26c198dff" +dependencies = [ + "cc", + "libc", + "pkg-config", + "vcpkg", +] + +[[package]] +name = "ordered-float" +version = "2.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c" +dependencies = [ + "num-traits", +] + +[[package]] +name = "parking_lot" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c42a9226546d68acdd9c0a280d17ce19bfe27a46bf68784e4066115788d008e" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall", + "smallvec", + "windows-targets 0.48.5", +] + +[[package]] +name = "password-hash" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "346f04948ba92c43e8469c1ee6736c7563d71012b17d40745260fe106aac2166" +dependencies = [ + "base64ct", + "rand_core", + "subtle", +] + +[[package]] +name = "pbkdf2" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" +dependencies = [ + "digest", + "hmac", + "password-hash", + "sha2", +] + +[[package]] +name = "percent-encoding" +version = "2.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e" + +[[package]] +name = "pin-project" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0302c4a0442c456bd56f841aee5c3bfd17967563f6fadc9ceb9f9c23cf3807e0" +dependencies = [ + "pin-project-internal", +] + +[[package]] +name = "pin-project-internal" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "266c042b60c9c76b8d53061e52b2e0d1116abc57cefc8c5cd671619a56ac3690" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8afb450f006bf6385ca15ef45d71d2288452bc3683ce2e2cacc0d18e4be60b58" + +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + +[[package]] +name = "pkg-config" +version = "0.3.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d231b230927b5e4ad203db57bbcbee2802f6bce620b1e4a9024a07d94e2907ec" + +[[package]] +name = "polyval" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" +dependencies = [ + "cfg-if", + "cpufeatures", + "opaque-debug", + "universal-hash", +] + +[[package]] +name = "portable-atomic" +version = "1.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7170ef9988bc169ba16dd36a7fa041e5c4cbeb6a35b76d4c03daded371eae7c0" + +[[package]] +name = "ppv-lite86" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de" + +[[package]] +name = "pretty_env_logger" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "865724d4dbe39d9f3dd3b52b88d859d66bcb2d6a0acfd5ea68a65fb66d4bdc1c" +dependencies = [ + "env_logger", + "log", +] + +[[package]] +name = "proc-macro2" +version = "1.0.78" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2422ad645d89c99f8f3e6b88a9fdeca7fabeac836b1002371c4367c8f984aae" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "291ec9ab5efd934aaf503a6466c5d5251535d108ee747472c3977cc5acc868ef" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rand" +version = "0.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404" +dependencies = [ + "libc", + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom", +] + +[[package]] +name = "redox_syscall" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4722d768eff46b75989dd134e5c353f0d6296e5aaa3132e776cbdb56be7731aa" +dependencies = [ + "bitflags 1.3.2", +] + +[[package]] +name = "redox_users" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a18479200779601e498ada4e8c1e1f50e3ee19deb0259c25825a98b5603b2cb4" +dependencies = [ + "getrandom", + "libredox", + "thiserror", +] + +[[package]] +name = "regex" +version = "1.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b62dbe01f0b06f9d8dc7d49e05a0785f153b00b2c227856282f671e0318c9b15" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5bb987efffd3c6d0d8f5f89510bb458559eab11e4f869acb20bf845e016259cd" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08c74e62047bb2de4ff487b251e4a92e24f48745648451635cec7d591162d9f" + +[[package]] +name = "reqwest" +version = "0.11.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6920094eb85afde5e4a138be3f2de8bbdf28000f0029e72c45025a56b042251" +dependencies = [ + "base64", + "bytes", + "encoding_rs", + "futures-core", + "futures-util", + "h2", + "http", + "http-body", + "hyper", + "hyper-tls", + "ipnet", + "js-sys", + "log", + "mime", + "native-tls", + "once_cell", + "percent-encoding", + "pin-project-lite", + "rustls-pemfile", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "system-configuration", + "tokio", + "tokio-native-tls", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", + "winreg", +] + +[[package]] +name = "ring" +version = "0.17.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c17fa4cb658e3583423e915b9f3acc01cceaee1860e33d59ebae66adc3a2dc0d" +dependencies = [ + "cc", + "cfg-if", + "getrandom", + "libc", + "spin", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "rpassword" +version = "7.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "80472be3c897911d0137b2d2b9055faf6eeac5b14e324073d83bc17b191d7e3f" +dependencies = [ + "libc", + "rtoolbox", + "windows-sys 0.48.0", +] + +[[package]] +name = "rtoolbox" +version = "0.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c247d24e63230cdb56463ae328478bd5eac8b8faa8c69461a77e8e323afac90e" +dependencies = [ + "libc", + "windows-sys 0.48.0", +] + +[[package]] +name = "rusqlite" +version = "0.31.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b838eba278d213a8beaf485bd313fd580ca4505a00d5871caeb1457c55322cae" +dependencies = [ + "bitflags 2.4.2", + "fallible-iterator", + "fallible-streaming-iterator", + "hashlink", + "libsqlite3-sys", + "smallvec", +] + +[[package]] +name = "rustc-demangle" +version = "0.1.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d626bb9dae77e28219937af045c257c28bfd3f69333c512553507f5f9798cb76" + +[[package]] +name = "rustix" +version = "0.38.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea3e1a662af26cd7a3ba09c0297a31af215563ecf42817c98df621387f4e949" +dependencies = [ + "bitflags 2.4.2", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustls" +version = "0.21.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d5a6813c0759e4609cd494e8e725babae6a2ca7b62a5536a13daaec6fcb7ba" +dependencies = [ + "log", + "ring", + "rustls-webpki", + "sct", +] + +[[package]] +name = "rustls-pemfile" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c74cae0a4cf6ccbbf5f359f08efdf8ee7e1dc532573bf0db71968cb56b1448c" +dependencies = [ + "base64", +] + +[[package]] +name = "rustls-webpki" +version = "0.101.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b6275d1ee7a1cd780b64aca7726599a1dbc893b1e64144529e55c3c2f745765" +dependencies = [ + "ring", + "untrusted", +] + +[[package]] +name = "ryu" +version = "1.0.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e86697c916019a8588c99b5fac3cead74ec0b4b819707a682fd4d23fa0ce1ba1" + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "schannel" +version = "0.1.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fbc91545643bcf3a0bbb6569265615222618bdf33ce4ffbbd13c4bbd4c093534" +dependencies = [ + "windows-sys 0.52.0", +] + +[[package]] +name = "scoped-tls" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "sct" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da046153aa2352493d6cb7da4b6e5c0c057d8a1d0a9aa8560baffdd945acd414" +dependencies = [ + "ring", + "untrusted", +] + +[[package]] +name = "security-framework" +version = "2.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05b64fb303737d99b81884b2c63433e9ae28abebe5eb5045dcdd175dc2ecf4de" +dependencies = [ + "bitflags 1.3.2", + "core-foundation", + "core-foundation-sys", + "libc", + "security-framework-sys", +] + +[[package]] +name = "security-framework-sys" +version = "2.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e932934257d3b408ed8f30db49d85ea163bfe74961f017f405b025af298f0c7a" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "serde" +version = "1.0.197" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3fb1c873e1b9b056a4dc4c0c198b24c3ffa059243875552b2bd0933b1aee4ce2" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde-value" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3a1a3341211875ef120e117ea7fd5228530ae7e7036a779fdc9117be6b3282c" +dependencies = [ + "ordered-float", + "serde", +] + +[[package]] +name = "serde_derive" +version = "1.0.197" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7eb0b34b42edc17f6b7cac84a52a1c5f0e1bb2227e997ca9011ea3dd34e8610b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", +] + +[[package]] +name = "serde_json" +version = "1.0.114" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c5f09b1bd632ef549eaa9f60a1f8de742bdbc698e6cee2095fc84dde5f549ae0" +dependencies = [ + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "serde_yaml" +version = "0.9.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd075d994154d4a774f95b51fb96bdc2832b0ea48425c92546073816cda1f2f" +dependencies = [ + "indexmap", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + +[[package]] +name = "sha1" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + +[[package]] +name = "sha2" +version = "0.10.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "793db75ad2bcafc3ffa7c68b215fee268f537982cd901d132f89c6343f3a3dc8" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + +[[package]] +name = "signal-hook-registry" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d8229b473baa5980ac72ef434c4415e70c4b5e71b423043adb4ba059f89c99a1" +dependencies = [ + "libc", +] + +[[package]] +name = "slab" +version = "0.4.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f92a496fb766b417c996b9c5e57daf2f7ad3b0bebe1ccfca4856390e3d3bb67" +dependencies = [ + "autocfg", +] + +[[package]] +name = "smallvec" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6ecd384b10a64542d77071bd64bd7b231f4ed5940fba55e98c3de13824cf3d7" + +[[package]] +name = "socket2" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05ffd9c0a93b7543e062e759284fcf5f5e3b098501104bfbdde4d404db792871" +dependencies = [ + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" + +[[package]] +name = "spmc" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02a8428da277a8e3a15271d79943e80ccc2ef254e78813a166a08d65e4c3ece5" + +[[package]] +name = "strsim" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ee073c9e4cd00e28217186dbe12796d692868f432bf2e97ee73bed0c56dfa01" + +[[package]] +name = "subtle" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81cdd64d312baedb58e21336b31bc043b77e01cc99033ce76ef539f78e965ebc" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.52" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b699d15b36d1f02c3e7c69f8ffef53de37aefae075d8488d4ba1a7788d574a07" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2047c6ded9c721764247e62cd3b03c09ffc529b2ba5b10ec482ae507a4a70160" + +[[package]] +name = "system-configuration" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba3a3adc5c275d719af8cb4272ea1c4a6d668a777f37e115f6d11ddbc1c8e0e7" +dependencies = [ + "bitflags 1.3.2", + "core-foundation", + "system-configuration-sys", +] + +[[package]] +name = "system-configuration-sys" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a75fb188eb626b924683e3b95e3a48e63551fcfb51949de2f06a9d91dbee93c9" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "tempfile" +version = "3.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85b77fafb263dd9d05cbeac119526425676db3784113aa9295c88498cbf8bff1" +dependencies = [ + "cfg-if", + "fastrand", + "rustix", + "windows-sys 0.52.0", +] + +[[package]] +name = "termcolor" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "thiserror" +version = "1.0.57" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e45bcbe8ed29775f228095caf2cd67af7a4ccf756ebff23a306bf3e8b47b24b" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.57" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a953cb265bef375dae3de6663da4d3804eee9682ea80d8e2542529b73c531c81" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", +] + +[[package]] +name = "thread-id" +version = "4.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0ec81c46e9eb50deaa257be2f148adf052d1fb7701cfd55ccfab2525280b70b" +dependencies = [ + "libc", + "winapi", +] + +[[package]] +name = "tinyvec" +version = "1.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87cc5ceb3875bb20c2890005a4e226a4651264a5c75edb2421b52861a0a0cb50" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + +[[package]] +name = "tokio" +version = "1.36.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61285f6515fa018fb2d1e46eb21223fff441ee8db5d0f1435e8ab4f5cdb80931" +dependencies = [ + "backtrace", + "bytes", + "libc", + "mio", + "num_cpus", + "parking_lot", + "pin-project-lite", + "signal-hook-registry", + "socket2", + "tokio-macros", + "windows-sys 0.48.0", +] + +[[package]] +name = "tokio-macros" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b8a1e28f2deaa14e508979454cb3a223b10b938b45af148bc0986de36f1923b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", +] + +[[package]] +name = "tokio-native-tls" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2" +dependencies = [ + "native-tls", + "tokio", +] + +[[package]] +name = "tokio-rustls" +version = "0.24.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c28327cf380ac148141087fbfb9de9d7bd4e84ab5d2c28fbc911d753de8a7081" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-stream" +version = "0.1.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "397c988d37662c7dda6d2208364a706264bf3d6138b11d436cbac0ad38832842" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tokio-tungstenite" +version = "0.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "212d5dcb2a1ce06d81107c3d0ffa3121fe974b73f068c8282cb1c32328113b6c" +dependencies = [ + "futures-util", + "log", + "tokio", + "tungstenite", +] + +[[package]] +name = "tokio-util" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5419f34732d9eb6ee4c3578b7989078579b7f039cbbb9ca2c4da015749371e15" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", + "tracing", +] + +[[package]] +name = "tower-service" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6bc1c9ce2b5135ac7f93c72918fc37feb872bdc6a5533a8b85eb4b86bfdae52" + +[[package]] +name = "tracing" +version = "0.1.40" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3523ab5a71916ccf420eebdf5521fcef02141234bbc0b8a49f2fdc4544364ef" +dependencies = [ + "log", + "pin-project-lite", + "tracing-core", +] + +[[package]] +name = "tracing-core" +version = "0.1.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c06d3da6113f116aaee68e4d601191614c9053067f9ab7f6edbcb161237daa54" +dependencies = [ + "once_cell", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "tungstenite" +version = "0.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e3dac10fd62eaf6617d3a904ae222845979aec67c615d1c842b4002c7666fb9" +dependencies = [ + "byteorder", + "bytes", + "data-encoding", + "http", + "httparse", + "log", + "rand", + "sha1", + "thiserror", + "url", + "utf-8", +] + +[[package]] +name = "typemap-ors" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a68c24b707f02dd18f1e4ccceb9d49f2058c2fb86384ef9972592904d7a28867" +dependencies = [ + "unsafe-any-ors", +] + +[[package]] +name = "typenum" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825" + +[[package]] +name = "unicase" +version = "2.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f7d2d4dafb69621809a81864c9c1b864479e1235c0dd4e199924b9742439ed89" +dependencies = [ + "version_check", +] + +[[package]] +name = "unicode-bidi" +version = "0.3.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08f95100a766bf4f8f28f90d77e0a5461bbdb219042e7679bebe79004fed8d75" + +[[package]] +name = "unicode-ident" +version = "1.0.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b" + +[[package]] +name = "unicode-normalization" +version = "0.1.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a56d1686db2308d901306f92a263857ef59ea39678a5458e7cb17f01415101f5" +dependencies = [ + "tinyvec", +] + +[[package]] +name = "unicode-width" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e51733f11c9c4f72aa0c160008246859e340b00807569a0da0e7a1079b27ba85" + +[[package]] +name = "universal-hash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" +dependencies = [ + "crypto-common", + "subtle", +] + +[[package]] +name = "unsafe-any-ors" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0a303d30665362d9680d7d91d78b23f5f899504d4f08b3c4cf08d055d87c0ad" +dependencies = [ + "destructure_traitobject", +] + +[[package]] +name = "unsafe-libyaml" +version = "0.2.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab4c90930b95a82d00dc9e9ac071b4991924390d46cbd0dfe566148667605e4b" + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "url" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31e6302e3bb753d46e83516cae55ae196fc0c309407cf11ab35cc51a4c2a4633" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", +] + +[[package]] +name = "users" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24cc0f6d6f267b73e5a2cadf007ba8f9bc39c6a6f9666f8cf25ea809a153b032" +dependencies = [ + "libc", + "log", +] + +[[package]] +name = "utf-8" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" + +[[package]] +name = "utf8parse" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "711b9620af191e0cdc7468a8d14e709c3dcdb115b36f838e601583af800a370a" + +[[package]] +name = "uuid" +version = "1.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f00cc9702ca12d3c81455259621e676d0f7251cec66a21e98fe2e9a37db93b2a" +dependencies = [ + "getrandom", +] + +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + +[[package]] +name = "version_check" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49874b5167b65d7193b8aba1567f5c7d93d001cafc34600cee003eda787e483f" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "warp" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c1e92e22e03ff1230c03a1a8ee37d2f89cd489e2e541b7550d6afad96faed169" +dependencies = [ + "bytes", + "futures-channel", + "futures-util", + "headers", + "http", + "hyper", + "log", + "mime", + "mime_guess", + "multer", + "percent-encoding", + "pin-project", + "rustls-pemfile", + "scoped-tls", + "serde", + "serde_json", + "serde_urlencoded", + "tokio", + "tokio-rustls", + "tokio-stream", + "tokio-tungstenite", + "tokio-util", + "tower-service", + "tracing", +] + +[[package]] +name = "wasi" +version = "0.11.0+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" + +[[package]] +name = "wasm-bindgen" +version = "0.2.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c1e124130aee3fb58c5bdd6b639a0509486b0338acaaae0c84a5124b0f588b7f" +dependencies = [ + "cfg-if", + "wasm-bindgen-macro", +] + +[[package]] +name = "wasm-bindgen-backend" +version = "0.2.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9e7e1900c352b609c8488ad12639a311045f40a35491fb69ba8c12f758af70b" +dependencies = [ + "bumpalo", + "log", + "once_cell", + "proc-macro2", + "quote", + "syn 2.0.52", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.41" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877b9c3f61ceea0e56331985743b13f3d25c406a7098d45180fb5f09bc19ed97" +dependencies = [ + "cfg-if", + "js-sys", + "wasm-bindgen", + "web-sys", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b30af9e2d358182b5c7449424f017eba305ed32a7010509ede96cdc4696c46ed" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "642f325be6301eb8107a83d12a8ac6c1e1c54345a7ef1a9261962dfefda09e66" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", + "wasm-bindgen-backend", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4f186bd2dcf04330886ce82d6f33dd75a7bfcf69ecf5763b89fcde53b6ac9838" + +[[package]] +name = "web-sys" +version = "0.3.68" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96565907687f7aceb35bc5fc03770a8a0471d82e479f25832f54a0e3f4b28446" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + +[[package]] +name = "winapi-util" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f29e6f9198ba0d26b4c9f07dbe6f9ed633e1f3d5b8b414090084349e46a52596" +dependencies = [ + "winapi", +] + +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + +[[package]] +name = "windows-core" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33ab640c8d7e35bf8ba19b884ba838ceb4fba93a4e8c65a9059d08afcfc683d9" +dependencies = [ + "windows-targets 0.52.4", +] + +[[package]] +name = "windows-sys" +version = "0.48.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9" +dependencies = [ + "windows-targets 0.48.5", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets 0.52.4", +] + +[[package]] +name = "windows-targets" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c" +dependencies = [ + "windows_aarch64_gnullvm 0.48.5", + "windows_aarch64_msvc 0.48.5", + "windows_i686_gnu 0.48.5", + "windows_i686_msvc 0.48.5", + "windows_x86_64_gnu 0.48.5", + "windows_x86_64_gnullvm 0.48.5", + "windows_x86_64_msvc 0.48.5", +] + +[[package]] +name = "windows-targets" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7dd37b7e5ab9018759f893a1952c9420d060016fc19a472b4bb20d1bdd694d1b" +dependencies = [ + "windows_aarch64_gnullvm 0.52.4", + "windows_aarch64_msvc 0.52.4", + "windows_i686_gnu 0.52.4", + "windows_i686_msvc 0.52.4", + "windows_x86_64_gnu 0.52.4", + "windows_x86_64_gnullvm 0.52.4", + "windows_x86_64_msvc 0.52.4", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bcf46cf4c365c6f2d1cc93ce535f2c8b244591df96ceee75d8e83deb70a9cac9" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da9f259dd3bcf6990b55bffd094c4f7235817ba4ceebde8e6d11cd0c5633b675" + +[[package]] +name = "windows_i686_gnu" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b474d8268f99e0995f25b9f095bc7434632601028cf86590aea5c8a5cb7801d3" + +[[package]] +name = "windows_i686_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1515e9a29e5bed743cb4415a9ecf5dfca648ce85ee42e15873c3cd8610ff8e02" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eee091590e89cc02ad514ffe3ead9eb6b660aedca2183455434b93546371a03" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ca79f2451b49fa9e2af39f0747fe999fcda4f5e241b2898624dca97a1f2177" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32b752e52a2da0ddfbdbcc6fceadfeede4c939ed16d13e648833a61dfb611ed8" + +[[package]] +name = "winreg" +version = "0.50.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "524e57b2c537c0f9b1e69f1965311ec12182b4122e45035b1508cd24d2adadb1" +dependencies = [ + "cfg-if", + "windows-sys 0.48.0", +] + +[[package]] +name = "zerocopy" +version = "0.7.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "74d4d3961e53fa4c9a25a8637fc2bfaf2595b3d3ae34875568a5cf64787716be" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.7.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ce1b18ccd8e73a9321186f97e46f9f04b778851177567b1975109d26a08d2a6" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.52", +] @@ -1,29 +1,46 @@ [package] name = "obnam" -version = "0.1.2" +version ="0.8.1" authors = ["Lars Wirzenius <liw@liw.fi>"] -edition = "2018" +edition = "2021" +description = "a backup program" +license = "AGPL-3.0-or-later" +homepage = "https://obnam.org/" +repository = "https://gitlab.com/obnam/obnam" +rust-version = "1.75.0" + [dependencies] -anyhow = "1" -bytes = "0.5" -chrono = "0.4" -dirs = "3" -indicatif = "0.15" -libc = "0.2" -log = "0.4" -log4rs = "1" -pretty_env_logger = "0.4" -reqwest = { version = "0.10", features = ["blocking", "json"]} -rusqlite = "0.24" -serde = { version = "1", features = ["derive"] } -serde_json = "1" -serde_yaml = "0.8" -sha2 = "0.9" -structopt = "0.3" -tempfile = "3.1" -thiserror = "1" -tokio = { version = "0.2", features = ["macros"] } -uuid = { version = "0.8", features = ["v4"] } -walkdir = "2" -warp = { version = "0.2", features = ["tls"] } +aes-gcm = "0.10.3" +anyhow = "1.0.80" +blake2 = "0.10.6" +bytesize = "1.3.0" +chrono = "0.4.34" +clap = { version = "4.5.1", features = ["derive"] } +directories-next = "2.0.0" +futures = "0.3.30" +indicatif = "0.17.8" +libc = "0.2.153" +log = "0.4.21" +log4rs = "1.3.0" +pbkdf2 = { version = "0.12.2", features = ["simple"] } +pretty_env_logger = "0.5.0" +rand = "0.8.5" +reqwest = { version = "0.11.24", features = ["blocking", "json"]} +rpassword = "7.3.1" +rusqlite = "0.31.0" +serde = { version = "1.0.197", features = ["derive"] } +serde_json = "1.0.114" +serde_yaml = "0.9.32" +sha2 = "0.10.8" +spmc = "0.3.0" +tempfile = "3.10.1" +thiserror = "1.0.57" +tokio = { version = "1.36.0", features = ["full"] } +users = "0.11.0" +uuid = { version = "1.7.0", features = ["v4"] } +walkdir = "2.5.0" +warp = { version = "0.3.6", features = ["tls"] } + +[profile.release] +debug = true diff --git a/DECISIONS.md b/DECISIONS.md new file mode 100644 index 0000000..f874eab --- /dev/null +++ b/DECISIONS.md @@ -0,0 +1,31 @@ +# Big decisions made by the Obnam project + +This is a decision log of big decisions, often architectural ones, or +otherwise decisions that impact the project as a whole. They may be +decisions about non-technical aspects of the project. + +Decisions should be discussed before being made so that there is a +strong consensus for them. Decisions can be changed or overturned if +they turn out to no longer be good. Overturning is itself a decision. + +Each decision should have its own heading. Newest decision should come +first. Updated or overturned decisions should have their section +updated to note their status, without moving them. + +## Support at least the version of Rust in Debian "bookworm" + +Date: 2021-12-06 + +What: See discussion in <https://gitlab.com/obnam/obnam/-/issues/137>. +Obnam aims to work in various Linux distributions and other operating +systems. One of these is Debian. At the time of writing, Debian's next +major version (code name bookworm) will have at least Rust 1.56. The +decision for Obnam is that a minimum support Rust version in bookworm. + +## Start a decision log + +Date: 2021-11-30 + +What: We decided to start keeping a decision log. + +Who: Alexander Batischev, Lars Wirzenius. @@ -1,9 +1,39 @@ # Definition of done +This definition is not meant to be petty bureaucracy. It's meant to be +the grease that allows the gears of project to turn smoothly with +minimal friction. The goal here is to enable smooth, speedy +improvement without having to frequently go back to fix things. + +When the software, automated tests, documentation, or web site +produced by the project are changed, the change overall should make +things better in some way: a bug is fixed, a feature is added, the +software becomes nicer to use, the documentation more effectively +communicates how to use the software, the tests cover more of the +functionality, the code is nicer to maintain, or something like that. + +At the same time, the change shouldn't make things significantly worse +in any way. A change that, say, makes the software ten times as fast, +but adds a ten percent chance of deleting the user's data would not be +acceptable. + For changes to this project to be considered done, the following must -be true: +all be true: + +1. New functionality and bug fixes are verified by automated tests + run by the `./check` script. + - if this is not feasible for some reason, that reason is + documented in commit messages, and an issue is opened so that the + tests can be added later +2. The build and tests run by GitLab CI finish successfully. +3. There has been sufficient time to review the change and for + interested parties to have tried it out. + - the time needed depends on the scope and complexity of the change + - a quick, easy change can be merged at once + - a complex change should be open for review and testing for a few + days -* any changes for new functionality add tests for the functionality -* changes are merged to the main branch -* the automated tests, as invoked by ./check, pass successfully -* CI successfully builds a new .deb package +If all of the above conditions are met, the change can be merged into +the main line of development by any person authorized to merge on +GitLab. The merge will eventually, automatically trigger a build of +Debian packages by Lars's personal CI. @@ -1,12 +0,0 @@ -# NEWS for Obnam2, the backup software - -This file summarizes changes between releases of the second generation -of Obnam, the backup software. The software is technically called -"obnam2" to distinguish it from the first generation of Obnam, which -ended in 2017 with version number 1.22. - - -## Obnam2 version 0.2, not yet released - -This if the first release of Obnam2. It can just barely make and -restore backups. It's ready for a light trial, but not for real use. @@ -0,0 +1,420 @@ +# Release notes for Obnam2 + +This file summarizes changes between releases of the second generation +of Obnam, the backup software. The software is technically called +"obnam2" to distinguish it from the first generation of Obnam, which +ended in 2017 with version number 1.22. + + +# Version 0.8.0, released 2022-07-24 + +## Breaking changes + +Breaking changes are ones that mean existing backups can't be +restored, or new backups can't be created. + +* The list of backups is stored in a special "root chunk". This means + backups are explicitly ordered. This also paves way for a future + feature to backups: only the root chunk will need to be updated. + Without a root chunk, the backups formed a linked list, and deleting + from the middle of the list would updating the whole list. + +* The server chunk metadata field `sha256` is now called `label`. + Labels include a type prefix, to allow for other chunk checksum + types in the future. + +* The server API is now explicitly versioned, to allow future changes + to cause less breakage. + +## New features + +* Users can now choose the backup schema version for new backups. A + repository can have backups with different schemas, and any existing + backup can be restored. The schema version only applies to new + backups. + +* New command `obnam inspect` shows metadata about a backup. Currently + only the schema version is shown. + +* New command `obnam list-backup-versions` shows all the backup schema + versions that this version of Obnam supports. + +* Obnam now logs some basic performance measurement for each run: how + many live files were found in total, backed up, chunks uploaded, + existing chunks reused, and how long various parts of the process + took. + +## Other changes + +* The `obnam show-generation` command now outputs data in the JSON + format. The output now includes data about the generation's SQLite + database size. + +## Thank you + +Several people have helped with this release, with changes or +feedback. + +* Alexander Batischev +* Lars Wirzenius + +# Version 0.7.1, released 2022-03-08 + +## Bug fixes + +* Skipped files are not added to a new backup. + +## Other changes + +* Obnam is now much faster when backing up files that haven't changed. + +## Thank you + +Several people have helped with this release, with changes or +feedback. + +* Alexander Batischev +* Lars Wirzenius + + +# Version 0.7.0, released 2022-01-04 + +## Breaking changes + +* No known breaking changes in this release. + +## New or changed features + +* Command that retrieve and use backups from the server now verify + that the backup's schema is compatible with the running version of + Obnam. This means, for example, that `obnam restore` won't try to + restore a backup it doesn't know it can restore. + +## Internal changes + +* Update Subplot step bindings with types for captures to allow + Subplot to verify that embedded files in obnam.md are actually used. + +* Tidy up code in various ways. + +* The Obnam release process now has a step to run `cargo update` after + the crate's version number has been updated, so that the + `Cargo.lock` file gets updated. + +## Changes to documentation + +* The `obnam` crate now documents all exported symbols. This should + make the crate somewhat less hostile to use. + +* The minimum supported Rust version is whatever is going to be in the + next Debian stable release (code name bookworm). + +## Thank you + +Several people have helped with this release, with changes or +feedback. + +* Alexander Batischev +* Lars Wirzenius + +(Our apologies to anyone who's been forgotten.) + + + +# Version 0.6.0, released 2021-11-20 + +## Breaking changes + +* We no longer test Obnam with Debian 10 (buster) in our continuous + integration system. The current Debian stable release, Debian 11 + (bullseye), is tested. + +## New or changed features + +* It is now an error if the backup root directory doesn't exist or + can't be read. This applies only to the backup roots. Other files + and directories may go missing or be unreadable, and Obnam only + warns about that, to allow making backups of live systems where + files change during the backup. + +## Internal changes + +* There is now a new "many files" benchmark. + +## Changes to documentation + +* We've started a decision log for big, important project decisions. + +## Thank you + +Several people have helped with this release, with changes or +feedback. + +* Alexander Batischev +* Lars Wirzenius + +(Our apologies to anyone who's been forgotten.) + + +# Version 0.5.0, released 2021-11-20 + +## Experimental version + +This is an experimental release, and is not meant to be relied on for +recovery of important data. The purpose of this release is to get new +features into the hands of intrepid people who want to try out new +things. + +## Breaking changes + +* Obnam is now licensed under the GNU Affero General Public License, + version 3 or later. This mainly affects the Obnam chunk server, + which has a network API. + +* The Obnam client now stores the version of the database schema in + the per-backup SQLite database. This allows the client to recognize + when a backup was made with an incompatible version of the client. + This, in turn, paves way for us to safely making changes that older + versions of the client do not understand. + + As a result, the backups made with this version may silently break + older versions of the client. However, this should be the last time + such silent breakage happens. + +## New or changed features + +* Obnam now restore metadata of restored symlinks correctly. + +* Obnam's handling of `CACHEDIR.TAG` files is more secure against an + attacker adding such files in directories getting backed up. + +* Progress bars so bars for different phases of the backup do not + interfere with each other anymore. + +* The client now has the "obnam resolve" subcommand to resolve a + generation label (such as "latest") into a generation ID. The labels + may point at different commits over time, the IDs never change. + +* The client now has the "obnam chunkify" subcommand to compute + checksums of chunks of files. For now, this is for doing performance + benchmarks, but may eventually evolve into a way to experiment how + parameters affect sizes of chunks and the ability of the Obnam + client to find duplicate data. + +* A build problem on macOS, where `chmod` needs a different type of + integer, was fixed. + +## Internal changes + +* Obnam was migrated to using Docker in GitLab CI and using the new + Debian stable release (version 11, code name bullseye). + +* The Obnam client is now asynchronous code. This is a foundation for + making the client be faster in the future. This has temporarily made + the client slower in some cases. + +* There is now a simple policy on what is required for changes to be + merge, in the `DONE.md` file. + +* There have been updates to use newer versions of dependencies, + refactoring of code to be clearer and more tidy, as well as bug + fixes in the test suite. + +## Changes to documentation + +* The tutorial now explains the passphrases are ephemeral. + +## Thank you + +Several people have helped with this release, with changes or +feedback. + +* Alexander Batischev +* Daniel Silverstone +* Lars Wirzenius +* Ossi Herrala + +(Our apologies to anyone who's been forgotten.) + + +# Version 0.4.0, released 2021-06-06 + +## Experimental version + +This is an experimental release, and is not meant to be relied on for +recovery of important data. The purpose of this release is to get new +features into the hands of intrepid people who want to try out new +things. + +## Breaking changes + +This release introduces use of encryption in Subplot. Encryption is +not optional, and the new `obnam init` command must always be used +before the first backup to generate an encryption key. + +Starting with this version of Obnam, there is no support at all for +cleartext backups any more. A backup, or backup repository, made with +a previous version of Obnam **will not work** with this version: you +can't list backups in a repository, you can't restore a backup, and +you can't make a new backup. You need to start over from scratch, by +emptying the server's chunk directory. Eventually, Obnam will stop +having such breaking, throw-away-everything changes, but it will +take time to build that functionality. + +Note: this version add only a very rudimentary approach to encryption. +It is only meant to protect the backups from the server operator +snooping via the server file system. It doesn't protect against most +other threats, including the server operator replacing parts of +backups on the server. Future versions of Obnam will add more +protection. + +## New or changed features + +* Obnam now by default excludes directories that are marked with a + `CACHEDIR.TAG` file. Set `exclude_cache_tag_directories` to `false` + in the configuration file to disable the feature. See the [Cache + Directory Tagging Specification][] for details of the tag file. + +[Cache Directory Tagging Specification]: https://bford.info/cachedir/ + +* You can now use _tilde notation_ in the configuration file, in fields + for specifying backup root directories or the log file. This makes + it easier to files relative to the user's home directory: + + ~~~yaml + server_url: https://obnam-server + roots: + - ~/Maildirs + ~ ~/src/obnam + log: ~/log/obnam.log + ~~~ + +* Alexander Batischev changed the code that queries the SQL database + to return an iterator, instead of an array of result. This means + that if, for example, a backup generation has a very large number of + files, Obnam no longer needs to keep all of them in memory at once. + +* Various error messages are now clearer and more useful. For example, + if there is a problem reading a file, the name of the file is + included in the error message. + +## Internal changes + +* Alexander Batischev added support for GitLab CI, which means that + changes are tested automatically before they are merged. This will + make development a little smoother in the future. + + +## Changes to documentation + +* Tigran Zakoyan made a logo for Obnam. It is currently only used on + the [website](https://obnam.org/), but will find more use later. For + example, some stickers could be made. + +## Thank you + +Several people have helped with this release, with changes or +feedback. I want to especially mention the following, in order by +first name, with apologies to anyone I have inadvertently forgotten: +Alexander Batischev, Daniel Silverstone, Neal Walfield, Tigran +Zakoyan. + + +# Version 0.3.1, released 2021-03-23 + +This is a minor release to work around a bug in Subplot, which +prevented the 0.3.0 release to have a Debian package built. The +workaround is to rewrite a small table in the "Filenames" section as a +list. + + +# Version 0.3.0, released 2021-03-14 + +## Breaking changes + +* The format of the data stored on the backup repository has changed. + The new version can't restore old backups: old generations are now + useless. You'll have to start over. Sorry. + +## New or changed features + +* New `obnam config` sub-command writes out the actual configuration + that the program users, as read from the configuration file. + +* The client configuration now has default values for all + configuration fields that can reasonably have them. For example, it + is no longer necessary to explicitly set a chunk size. + +* Only known fields are now allowed in configuration files. Unknown + fields cause an error. + +* It is now possible to back up multiple, distinct directories with + one client configuration. The `root` configuration is now `roots`, + and is a list of directories. + +* Problems in backing up a file no longer terminate the backup run. + Instead, the problem is reported at the end of the backup run, as a + warning. + +* The client now requires an HTTPS URL for the server. Plain HTTP is + now rejected. The TLS certificate for the server is verified by + default, but that can be turned off. + +* The client progress reporting is now a little clearer. + +* Unix domain sockets and named pipes (FIFO files) are now backed up + and restored. + +* The names of the user and group owning a file are backed up, but not + restored. + +* On the Obnam server, the Ansible playbook now installs a cron job to + renew the Let's Encrypt TLS certificate. + +## Bugs fixed + +* Temporary files created during backup runs are now automatically + deleted, even if the Obnam client crashes. + +* Symbolic links are now backed up and restored correctly. Previously + Obnam followed the link when backing up and created the link + wrongly. + +* The Ansible playbook to provision an Obnam server now enables the + systemd unit so that the Obnam server process starts automatically + after a reboot. + +## Changes to documentation + +* A tutorial has been added. + +The Obnam subplot (`obnam.md`), which describes the requirements, +acceptance criteria, and architecture of the software, has some +improvements: + +* a discussion of why Obnam doesn't use content-addressable storage + +* a description of the logical structure of backups as stored on the + backup server + +* a rudimentary first sketch of a threat model: the operator of the + backup server reads the backed up data + +* an initial plan for adding support for encryption to backups; this + is known to be simplistic and inadequate, but the goal is to get + started, and then iterate to get something acceptable, even if that + takes months + +## Thank you + +Several people have helped with this release, with changes or +feedback. I want to especially mention the following, with apologies +to anyone I have inadvertently forgotten: Alexander Batischev, Ossi +Herrala, Daniel Silverstone, Neal Walfield. + +# Version 0.2.2, released 2021-01-29 + +This is the first release of Obnam2. It can just barely make and +restore backups. It's ready for a light trial, but not for real use. +There's no encryption, and backups can't be deleted yet. Restores of +the entire backup work. @@ -1,69 +1,89 @@ -# Obnam – a backup system +# Obnam — a backup system Obnam2 is a project to develop a backup system. -You probably want to read the [obnam.md](obnam.md) subplot file. - -## Client installation - -See instructions at <https://obnam.org/download/> for installing the -client. It's not duplicated here to avoid having to keep the -information in sync in two places. - -## Server installation - -To install the Obnam server component, you need a Debian host with -sufficient disk space, and Ansible installed locally. Run the -following commands in the Obnam source tree, replacing -`obnam.example.com` with the domain name of your server: - -```sh -$ cd ansible -$ printf '[obnam-server]\nobnam.example.com\n' > hosts -$ ansible-playbook -i hosts obnam-server.yml -e domain=obnam.example.com -``` - -The above gets a free TLS certificate from [Let's Encrypt][], but only -works if the server is accessible from the public Internet. For a -private host use the following instead: - -```sh -$ cd ansible -$ printf '[obnam-server]\nprivate-vm\n' > hosts -$ ansible-playbook -i hosts obnam-server.yml -``` - -This uses a pre-created self-signed certificate from -`files/server.key` and `files/server.pem` and is probably only good -for trying out Obnam. You may want to generate your own certificates -instead. - -To create a self-signed certificate, something like the following -command might work, using [OpenSSL]: - -```sh -$ openssl req -x509 -newkey rsa:4096 -passout pass:hunter2 \ - -keyout key.pem -out cert.pem -days 365 -subj /CN=localhost -``` - - -[Let's Encrypt]: https://letsencrypt.org/ -[OpenSSL]: https://www.openssl.org/ - +For installation instructions and a quick start guide, see +[tutorial.md][]. For more details on goals, requirements, and +implementation details, see the [obnam.md][] subplot file. + +[tutorial.md]: https://doc.obnam.org/obnam/tutorial.html +[obnam.md]: https://doc.obnam.org/obnam/obnam.html + +# Dependencies for build and test + +The up-to-date, tested list of dependencies for building and testing +Obnam are listed in the file [debian/control](debian/control), in +terms of Debian packages, and in [Cargo.toml](Cargo.toml) for Rust. +The Rust dependencies are handled automatically by the Cargo tool on +all platforms. The other dependencies are, not including ones needed +merely for building Debian packages: + +* [Rust](https://www.rust-lang.org/tools/install) — the + programming implementation. This can be installed via the standard + Rust installer, `rustup`, or any other way. Obnam does not currently + specify an explicit minimum version of Rust it requires, but its + developers use whatever is the current stable version of the + language. + + On Debian, the `build-essential` package also needs to be installed + to build Rust programs. + +* [daemonize](http://software.clapper.org/daemonize/) — a tool + for running a command as a daemon in the background; needed for + testing, so that the Obnam server can be started and stopped by the + Obnam test suite. + +* [SQLite](https://sqlite.org), specifically its development library + component — an SQL database engine that stores the whole + database in a file and can be used as a library rather then run as a + service. + +* [OpenSSL](https://www.openssl.org), specifically its development + library component known as `libssl-dev` — a library that + implments TLS, which Obnam uses for communication between its client + and server parts. + +* [moreutils](https://joeyh.name/code/moreutils/) — a collection + of handy utilities, of which the Obnam test suite uses the `chronic` + tool to hide output of successful commands. This is optional, but + nice to have. + +* [pkg-config](http://pkg-config.freedesktop.org) — a tool for + managing compile and link time flags; needed so that the OpenSSL + library can be linked into the Obnam binaries. + +* [Python 3](https://www.python.org/), + [Requests](http://python-requests.org), + [PYYAML](https://github.com/yaml/pyyaml) — programming + language and libraries for it, used by the Obnam test suite. + +* [Subplot](https://subplot.liw.fi) — a tool for documenting + acceptance criteria and verifying that they are met. + +* [TeX Live](http://www.tug.org/texlive/) — a typesetting system + for generating PDF versions of documentation. The LaTeX + implementation and fonts are needed, not the full suite. None of Tex + Live is needed, if PDFs aren't needed, but `./check` does not + currently have a way to be told not to generate PDFs. + +* [Summain](https://summain.liw.fi) — a tool for generating + manifests of files. Used by the Obnam test suite to verify restored + data matches the original data. ## Legalese -Copyright 2020-2021 Lars Wirzenius + +Copyright 2020-2021 Lars Wirzenius and others This program is free software: you can redistribute it and/or modify -it under the terms of the GNU General Public License as published by -the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU General Public License for more details. +GNU Affero General Public License for more details. -You should have received a copy of the GNU General Public License -along with this program. If not, see <http://www.gnu.org/licenses/>. +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. @@ -1,34 +1,59 @@ -# Release checklist for Obnam2 +# Release checklist for Obnam -Follow these steps to make a release of Obnam2. +Follow these steps to make a release of Obnam. -* create a `release` branch -* update `NEWS` with changes for the new release -* update the version number everywhere it needs to be updated; use - [semantic versioning][] - - `NEWS` - - `debian/changelog` - - `Cargo.toml` -* commit everything -* push changes to gitlab, create merge request, merge, pull back to - local `main` branch -* create a signed, annotated git tag `vX.Y.Z` for version X.Y.Z for +1. Create a `release` branch. + - `git checkout -b release` +2. Update dependencies for the crate, and make any needed changes. + - `cargo outdated -R` + - `cargo update` + - `cargo deny check` +3. Review changes in the crate (`git log vX.Y.Y..`). Update the `NEWS.md` + file with any changes that users of Obnam need to be aware of. +4. Update the crate's `Cargo.toml` with the appropriate version number + for the new release, following [semantic versioning][]. +5. Update `debian/changelog` with a summary of any changes to the + Debian packaging (it's not necessary to repeat `NEWS.md` here). Use + the `dch` command to edit the file to get the format right, since + it's quite finicky. + - `dch -v X.Y.Z-1 "New release."` + - `dch "Changed this thing: foo."` + - `dch -r ""` +6. Make sure everything still works. + - `./check` +7. Update `Cargo.lock` after version change. + - `cargo update` +8. Commit any changes. +9. Run `cargo publish --dry-run` and fix any problems. +10. Push to gitlab.com and create a merge request. +11. Wait for GitLab CI to test the changes successfully. Fix any + problems it finds. + +After the above changes have been merged, do the following steps. You +need to have sufficient access to both the gitlab.com project and the +git.liw.fi project. Currently that means only Lars can do this. These +steps can hopefully be automated in the future. + +1. Pull down the above changes from GitLab. +2. Create a signed, annotated git tag `vX.Y.Z` for version X.Y.Z for the release commit -* push tag to `gitlab.com` and `git.liw.fi` -* announce new release + - `git tag -sam "Obnam version X.Y.Z" vX.Y.Z` +3. Push tags to `gitlab.com` and `git.liw.fi` (substitute whatever + names you use for the remotes): + - `git push --tags gitlab` + - `git push --tags origin` +4. Wait for Lars's Ick CI to build the release. +5. Publish Obnam crate to crates.io: + - `cargo publish` +5. Run benchmark for release. + - `obnam-benchmark run benchmarks.yaml --obnam vX.Y.Z --output obnam-X.Y.Z.json` + - add the output file to the `obnam-benchmark-results` repository + - push the updated results repository to Lars's personal CI, to + update the result page on the web +6. Announce new release: - obnam.org blog, possibly other blogs - `#obnam` IRC channel - Obnam room on Matrix - social media -* prepare `main` branch for further development - - create new branch - - update version number again by adding `+git` to it - - add a new entry to `NEWS` with the `+git` version - - ditto `debian/changelog` - - commit - - push to gitlab, create MR, merge, pull back down to local `main` - -* continue development - [semantic versioning]: https://semver.org/ diff --git a/ansible/files/obnam.service b/ansible/files/obnam.service index 9d933aa..f30eb60 100644 --- a/ansible/files/obnam.service +++ b/ansible/files/obnam.service @@ -1,10 +1,10 @@ [Unit] Description=Obnam server +ConditionPathExists=/etc/obnam/server.yaml [Service] Type=simple -ConditionPathExists=/etc/obnam/server.yaml -ExecStart=/usr/bin/obnam-server /etc/obnam/server.yaml +ExecStart=/bin/obnam-server /etc/obnam/server.yaml [Install] WantedBy=multi-user.target diff --git a/ansible/hosts b/ansible/hosts index 253d739..136086c 100644 --- a/ansible/hosts +++ b/ansible/hosts @@ -1,2 +1,2 @@ -[obnam-server] +[server] obnam-server diff --git a/ansible/obnam-server.yml b/ansible/obnam-server.yml index 426ca74..90ac9f1 100644 --- a/ansible/obnam-server.yml +++ b/ansible/obnam-server.yml @@ -1,9 +1,15 @@ - hosts: server remote_user: root tasks: + - name: add APT signing key for the Obnam package repository + copy: + content: | + {{ apt_signing_key }} + dest: /etc/apt/trusted.gpg.d/obnam.asc + - name: add Obnam package repository to APT apt_repository: - repo: "deb [trusted=yes] http://ci-prod-controller.vm.liw.fi/debian unstable-ci main" + repo: "deb http://ci-prod-controller.vm.liw.fi/debian unstable-ci main" - name: refresh APT package lists and upgrade all installed packages apt: @@ -24,6 +30,15 @@ - dehydrated-apache2 when: domain is defined + - name: "install cron job to update TLS certificates" + cron: + name: "dehydrated" + cron_file: "dehydrated" + user: root + minute: "0" + hour: "0" + job: "systemctl stop obnam; systemctl start apache2; dehydrated -c; systemctl stop apache2; systemctl start obnam" + - name: create Obnam configuration directory file: path: /etc/obnam @@ -90,7 +105,39 @@ - name: start Obnam server systemd: name: obnam + enabled: true state: restarted vars: tls_key_path: "/var/lib/dehydrated/certs/{{ domain }}/privkey.pem" tls_cert_path: "/var/lib/dehydrated/certs/{{ domain }}/cert.pem" + + apt_signing_key: | + -----BEGIN PGP PUBLIC KEY BLOCK----- + + mQINBFrLO7kBEADdz6mHstYmKU5Dp6OSjxWtWaqTDOX1sJdmmaIK/9EKVIH0Maxp + 5kvVO5G6mULLAjv/kLG0MxasHPrq8I2A/y8AqKAGVL8QelwLjQMIFZ30/VbGQPHS + +T5TZXEnoQtNce1GUhFwJ38ZyjjwHBFV9tSec7rZ2Q3YeM3nNnGPf6DacXGfEOPO + HIN4sXAN2hzNXNjKRzTIvxQseb6nr7afUh/SlZ3yhQOCrIzmYlD7tP9WJe7ofL0p + JY4pDQYw8rT6nC2BE/ioemh84kERCT1vCe+OVFlSRuMlqfEv+ZpKQ+itOmPDQ/lM + jpUm1K2hrW/lWpxT/ZxHKo/w1K36J5WshgMZxfUu5BMCL9LMqMcrXNhNjDMfxDMM + 3yBPOvQ4ls6fecOZ/bsFo1p8VzMk/w/eG8vPs5yuNa5XxN95yFMXoOHGb5Xbu8D4 + 6yiW+Af70LbiSNpGdmNdneiGB2fY38NxBukPw5u3S5qG8HedSmMr1RvSr5kHoAAe + UbOY+BYaaKsTAT7+1skUW1o3FJSqoRKCHAzTsMWC6zzhR8hRn7jVrrguH1hGbqq5 + TZSCFQZExuTJ7uXrTLG0WoBXIjB5wWNcSeXn8myUWYB51nJNF4tJBouZOz9JwWGl + kiAQkrHnBttLQWdW9FyjbIoTZMtpvVx+m6ObGTGdGL1cNlLAvWprMXGc+QARAQAB + tDJJY2sgQVBUIHJlcG9zaXRvcnkgc2lnbmluZyBrZXkgKDIwMTgpIDxsaXdAbGl3 + LmZpPokCTgQTAQgAOBYhBKL1uyDoXyxUH3O717Wr+TZVS6PGBQJayzu5AhsDBQsJ + CAcCBhUICQoLAgQWAgMBAh4BAheAAAoJELWr+TZVS6PGB5QQANTcikhRUHwt9N4h + dGc/Hp6CbqdshMoWlwpFskttoVDxQG5OAobuZl5XyzGcmja1lT85RGkZFfbca0IZ + LnXOLLSAu51QBkXNaj4OhjK/0uQ+ITrvL6RQSXNgHiUTR/W2XD1GIUq6nBqe2GSN + 31S1baYKKVj5QIMsi7Dq8ls3BBXuPCE+xTSaNmGWjes2t9pPidcRvxsksCLY1qgw + P1GFXBeMkBQ29kBP87SUL15SIk7OiQLlEURCy5iRls5rt/YEsdEpRWIb0Tm5Nrjv + 2M3VM+iBhfNXTwj0rJ34mlycF1qQmA7YcTEobT7z587GPY0VWzBpQUnEQj7rQWPM + cDYY0b+I6kQ8VKOaL4wVAtE98d7HzFIrIrwhTKufnrWrVDPYsmLZ+LPC1jiF7JBD + SR6Vftb+SdDR9xoE1yRuXbC6IfoW+5/qQNrdQ2mm9BFw5jOonBqchs18HTTf3441 + 6SWwP9fY3Vi+IZphPPi0Gf85oMStgnv/Wnw6LacEL32ek39Desero/D8iGLZernK + Q2mC9mua5A/bYGVhsNWyURNFkKdbFa+/wW3NfdKYyZnsSfo+jJ2luNewrhAY7Kod + GWXTer9RxzTGA3EXFGvNr+BBOOxSj0SfWTl0Olo7J5dnxof+jLAUS1VHpceHGHps + GSJSdir7NkZidgwoCPA7BTqsb5LN + =dXB0 + -----END PGP PUBLIC KEY BLOCK----- diff --git a/bench-manyfiles.sh b/bench-manyfiles.sh new file mode 100755 index 0000000..39e6c2e --- /dev/null +++ b/bench-manyfiles.sh @@ -0,0 +1,58 @@ +#!/bin/bash +# +# Run a simple benchmark of an incremental backup of many empty files, +# when the live data hasn't changed. +# +# Edit the if-statement towards the end to get a flamegraph to see +# where time is actually spent. +# +# This is very simplistic and could do with a lot of improvement. But +# it's a start. + +set -euo pipefail + +N=100000 + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +chunks="$TMP/chunks" +live="$TMP/live" + +mkdir "$chunks" +mkdir "$live" +python3 manyfiles.py "$live" "$N" + +cat <<EOF >"$TMP/server.yaml" +address: localhost:8888 +chunks: $chunks +tls_key: test.key +tls_cert: test.pem +EOF + +cat <<EOF >"$TMP/client.yaml" +server_url: https://localhost:8888 +verify_tls_cert: false +roots: + - $live +log: $TMP/client.log +EOF + +cargo build -q --release --all-targets + +OBNAM_SERVER_LOG=error cargo run -q --release --bin obnam-server -- "$TMP/server.yaml" >/dev/null & +pid="$!" + +cargo run -q --release --bin obnam -- --config "$TMP/client.yaml" init --insecure-passphrase=hunter2 + +# Initial backup. +cargo run -q --release --bin obnam -- --config "$TMP/client.yaml" backup >/dev/null + +# Incremental backup. +if true; then + /usr/bin/time --format=%e cargo run -q --release --bin obnam -- --config "$TMP/client.yaml" backup >/dev/null +else + cargo flamegraph --bin obnam -o obnam.svg -- --config "$TMP/client.yaml" backup >/dev/null +fi + +kill "$pid" diff --git a/bench.sh b/bench.sh new file mode 100755 index 0000000..75cd459 --- /dev/null +++ b/bench.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# +# Run a simple benchmark of an initial backup of a sparse file of a +# given size. This mainly measures how fast the client can split the +# live data into chunks and compute checksums for the chunks. +# +# Edit the if-statement towards the end to get a flamegraph to see +# where time is actually spent. +# +# This is very simplistic and could do with a lot of improvement. But it's a start. + +set -euo pipefail + +SIZE=1G + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +chunks="$TMP/chunks" +live="$TMP/live" + +mkdir "$chunks" +mkdir "$live" +truncate --size "$SIZE" "$live/data.dat" + +cat <<EOF >"$TMP/server.yaml" +address: localhost:8888 +chunks: $chunks +tls_key: test.key +tls_cert: test.pem +EOF + +cat <<EOF >"$TMP/client.yaml" +server_url: https://localhost:8888 +verify_tls_cert: false +roots: + - $live +log: $TMP/client.log +EOF + +cargo build -q --release --all-targets + +OBNAM_SERVER_LOG=error cargo run -q --release --bin obnam-server -- "$TMP/server.yaml" >/dev/null & +pid="$!" + +cargo run -q --release --bin obnam -- --config "$TMP/client.yaml" init --insecure-passphrase=hunter2 +if true; then + /usr/bin/time cargo run -q --release --bin obnam -- --config "$TMP/client.yaml" backup >/dev/null +else + cargo flamegraph --bin obnam -o obnam.svg -- --config "$TMP/client.yaml" backup >/dev/null +fi + +kill -9 "$pid" diff --git a/benchmark.sh b/benchmark.sh deleted file mode 100755 index cf7491a..0000000 --- a/benchmark.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -chunkdir="$1" -bin="$2" - -cleanup() -{ - echo "emptying $chunkdir" 1>&2 - find "$chunkdir" -mindepth 1 -delete -} - -cleanup - -echo "running benchmarks for various sizes" -for n in 1 10 100 1000 10000 100000 1000000 -do - echo "size $n" 1>&2 - for prog in benchmark-null benchmark-index benchmark-store benchmark-indexedstore - do - /usr/bin/time --format "$prog $n %e" "$bin/$prog" "$chunkdir" "$n" 2>&1 - cleanup - done -done | awk '{ printf "%-30s %10s %10s\n", $1, $2, $3 }' @@ -4,33 +4,59 @@ set -eu -quiet=-q hideok=chronic -if [ "$#" -gt 0 ] -then - case "$1" in +if ! command -v chronic >/dev/null; then + hideok= +fi + +if [ "$#" -gt 0 ]; then + case "$1" in verbose | -v | --verbose) - quiet= - hideok= - ;; - esac + hideok= + shift + ;; + esac fi -got_cargo_cmd() -{ - cargo --list | grep " $1 " > /dev/null +require_cmd() { + if ! command -v "$1" >/dev/null; then + echo "Need to have $1 installed, but can't find it" 1>&2 + return 1 + fi +} + +got_cargo_cmd() { + cargo "$1" --help >/dev/null } -cargo build --all-targets $quiet -got_cargo_cmd clippy && cargo clippy $quiet -got_cargo_cmd fmt && cargo fmt -- --check -$hideok cargo test $quiet +require_cmd rustc +require_cmd cc +require_cmd cargo +require_cmd python3 +require_cmd subplot +require_cmd summain +require_cmd pkg-config -sp-docgen obnam.md -o obnam.html -sp-docgen obnam.md -o obnam.pdf +# daemonize installation location changed from Debian 10 to 11. +require_cmd daemonize || require_cmd /usr/sbin/daemonize -sp-codegen obnam.md -o test.py +$hideok cargo --version +$hideok rustc --version + +got_cargo_cmd clippy && cargo clippy --all-targets -q +$hideok cargo build --all-targets +got_cargo_cmd fmt && $hideok cargo fmt -- --check +$hideok cargo test + +subplot docgen obnam.subplot -o obnam.html + +target="$(cargo metadata --format-version=1 | python3 -c 'import sys, json; o = json.load(sys.stdin); print(o["target_directory"])')" +subplot codegen obnam.subplot -o test.py rm -f test.log -$hideok python3 test.py --log test.log "$@" +if [ "$(id -un)" = root ]; then + echo Not running tests as root. +else + $hideok python3 test.py --log test.log --env "CARGO_TARGET_DIR=$target" "$@" +fi echo "Everything seems to be in order." diff --git a/client.yaml b/client.yaml index dd60c9c..4a2ad54 100644 --- a/client.yaml +++ b/client.yaml @@ -1,3 +1,5 @@ server_url: https://localhost:8888 -root: /home/liw/tmp/Foton +verify_tls_cert: false +roots: + - live log: obnam.log diff --git a/debian/changelog b/debian/changelog index ed26379..e73fed8 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,6 +1,61 @@ -obnam (0.1.3-1) unstable; urgency=low +obnam (0.8.1-1) unstable; urgency=medium + + * New Debian package + + -- Lars Wirzenius <liw@liw.fi> Mon, 01 Jan 2024 16:22:35 +0200 + +obnam (0.8.0-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Sun, 24 Jul 2022 11:13:59 +0300 + +obnam (0.7.1-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Tue, 08 Mar 2022 08:17:53 +0200 + +obnam (0.7.0-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Tue, 04 Jan 2022 16:29:52 +0200 + +obnam (0.6.0-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Sat, 04 Dec 2021 08:38:15 +0200 + +obnam (0.5.0-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Sat, 20 Nov 2021 11:13:20 +0200 + +obnam (0.4.0-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Sun, 06 Jun 2021 10:25:02 +0300 + +obnam (0.3.1-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Tue, 23 Mar 2021 09:51:23 +0200 + +obnam (0.3.0-1) unstable; urgency=medium + + * New release. + + -- Lars Wirzenius <liw@liw.fi> Sun, 14 Mar 2021 11:15:25 +0200 + +obnam (0.2.2-1) unstable; urgency=low * Initial packaging. This is not intended to be uploaded to Debian, so no closing of an ITP bug. -- Lars Wirzenius <liw@liw.fi> Sat, 28 Sep 2019 16:45:49 +0300 + diff --git a/debian/control b/debian/control index 5d3167b..b13c20d 100644 --- a/debian/control +++ b/debian/control @@ -5,8 +5,6 @@ Priority: optional Standards-Version: 4.2.0 Build-Depends: debhelper (>= 10~), - build-essential, - dh-cargo, daemonize, git, libsqlite3-dev, @@ -16,12 +14,8 @@ Build-Depends: python3, python3-requests, python3-yaml, - strace, subplot, - summain, - texlive-fonts-recommended, - texlive-latex-base, - texlive-latex-recommended + summain Homepage: https://obnam.org Package: obnam diff --git a/debian/copyright b/debian/copyright index ea31729..043fcc0 100644 --- a/debian/copyright +++ b/debian/copyright @@ -4,20 +4,17 @@ Upstream-Contact: Lars Wirzenius <liw@liw.fi> Source: http://git.liw.fi/obnam2 Files: * -Copyright: 2020, Lars Wirzenius -License: GPL-3+ +Copyright: 2020-2021, Lars Wirzenius and others +License: AGPL-3+ This program is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. + it under the terms of the GNU Affero General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. . This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. + GNU Affero General Public License for more details. . - You should have received a copy of the GNU General Public License - along with this program. If not, see <http://www.gnu.org/licenses/>. - . - On a Debian system, you can find a copy of GPL version 3 at - /usr/share/common-licenses/GPL-3 . + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. diff --git a/debian/rules b/debian/rules index 6675ae9..7a3e4e9 100755 --- a/debian/rules +++ b/debian/rules @@ -1,15 +1,15 @@ #!/usr/bin/make -f %: - dh $@ --buildsystem cargo + dh $@ override_dh_auto_build: true override_dh_auto_install: - cargo install --path=. --root=debian/obnam - rm -f debian/obnam/.crates.toml - rm -f debian/obnam/.crates2.json + cargo install --path=. --root=debian/obnam --offline + find debian -name '.crates*' -delete + find debian/obnam/bin -type f ! -name 'obnam*' -delete override_dh_auto_test: - ./check + echo disabled: ./check diff --git a/deny.toml b/deny.toml new file mode 100644 index 0000000..f6e7bb8 --- /dev/null +++ b/deny.toml @@ -0,0 +1,82 @@ +# Note that all fields that take a lint level have these possible values: +# * deny - An error will be produced and the check will fail +# * warn - A warning will be produced, but the check will not fail +# * allow - No warning or error will be produced, though in some cases a note +# will be + +[advisories] +db-path = "~/.cargo/advisory-db" +db-urls = ["https://github.com/rustsec/advisory-db"] +vulnerability = "deny" +unmaintained = "warn" +yanked = "deny" +notice = "deny" +ignore = [ + "RUSTSEC-2020-0027", + "RUSTSEC-2020-0071", +] + +[licenses] +unlicensed = "deny" +allow = [ + "Apache-2.0", + "Apache-2.0 WITH LLVM-exception", + "BSD-3-Clause", + "ISC", + "LicenseRef-ring", + "MIT", + "Unicode-DFS-2016", +] +deny = [ + #"Nokia", +] +copyleft = "allow" +default = "deny" +exceptions = [ + # Each entry is the crate and version constraint, and its specific allow + # list + #{ allow = ["Zlib"], name = "adler32", version = "*" }, +] + +[[licenses.clarify]] +name = "encoding_rs" +version = "*" +expression = "(Apache-2.0 OR MIT) AND BSD-3-Clause" +license-files = [ + { path = "COPYRIGHT", hash = 0x39f8ad31 } +] + +[[licenses.clarify]] +name = "ring" +expression = "LicenseRef-ring" +license-files = [ + { path = "LICENSE", hash = 0xbd0eed23 }, +] + +[bans] +multiple-versions = "allow" +wildcards = "allow" +highlight = "all" +allow = [ + #{ name = "ansi_term", version = "=0.11.0" }, +] +deny = [ + # Each entry the name of a crate and a version range. If version is + # not specified, all versions will be matched. + #{ name = "ansi_term", version = "=0.11.0" }, + # + # Wrapper crates can optionally be specified to allow the crate when it + # is a direct dependency of the otherwise banned crate + #{ name = "ansi_term", version = "=0.11.0", wrappers = [] }, +] +skip = [ + #{ name = "ansi_term", version = "=0.11.0" }, +] +skip-tree = [ + #{ name = "ansi_term", version = "=0.11.0", depth = 20 }, +] + +[sources] +unknown-registry = "warn" +unknown-git = "warn" +allow-git = [] diff --git a/list_new_release_tags b/list_new_release_tags index 90994ac..7b82050 100644..100755 --- a/list_new_release_tags +++ b/list_new_release_tags @@ -41,7 +41,8 @@ def built_tags(filename): def save_built_tags(filename, tags): - return open(filename, "w").write("".join(f"{tag}\n" for tag in tags)) + with open(filename, "w") as f: + f.write("".join(f"{tag}\n" for tag in tags)) tags_filename = sys.argv[1] diff --git a/manyfiles.py b/manyfiles.py new file mode 100644 index 0000000..a45d3cd --- /dev/null +++ b/manyfiles.py @@ -0,0 +1,39 @@ +#!/usr/bin/python3 +# +# Create the desired number of empty files in a directory. A thousand files per +# subdirectory. + +import os +import sys + + +def subdir(dirname, dirno): + pathname = os.path.join(dirname, str(dirno)) + os.mkdir(pathname) + return pathname + + +def create(filename): + open(filename, "w").close() + + +DIRFILES = 1000 + +dirname = sys.argv[1] +n = int(sys.argv[2]) + +dirno = 0 +subdirpath = subdir(dirname, dirno) +fileno = 0 +thisdir = 0 + +while fileno < n: + filename = os.path.join(subdirpath, str(thisdir)) + create(filename) + + fileno += 1 + thisdir += 1 + if thisdir >= DIRFILES: + dirno += 1 + subdirpath = subdir(dirname, dirno) + thisdir = 0 @@ -1,3 +1,12 @@ +# Abstract + +Obnam is a backup system, consisting of a not very smart server for +storing chunks of backup data, and a client that splits the user's +data into chunks. They communicate via HTTP. + +This document describes the architecture and acceptance criteria for +Obnam, as well as how the acceptance criteria are verified. + # Introduction Obnam2 is a backup system. @@ -48,7 +57,7 @@ in an automated way: * _Done:_ **Easy to install:** available as a Debian package in an APT repository. Other installation packages will also be provided, hopefully. -* _Not done:_ **Easy to configure:** only need to configure things +* _Ongoing:_ **Easy to configure:** only need to configure things that are inherently specific to a client, when sensible defaults are impossible. * _Not done:_ **Excellent documentation:** although software ideally @@ -57,11 +66,11 @@ in an automated way: unambiguous, and well-liked. * _Done_: **Easy to run:** making a backup is a single command line that's always the same. -* _Not done:_ **Detects corruption:** if a file in the repository is +* _Ongoing:_ **Detects corruption:** if a file in the repository is modified or deleted, the software notices it automatically. -* _Not done:_ **Repository is encrypted:** all data stored in the +* _Ongoing:_ **Repository is encrypted:** all data stored in the repository is encrypted with a key known only to the client. -* _Not done:_ **Fast backups and restores:** when a client and server +* _Ongoing:_ **Fast backups and restores:** when a client and server both have sufficient CPU, RAM, and disk bandwidth, the software makes a backup or restores a backup over a gigabit Ethernet using at least 50% of the network bandwidth. @@ -87,6 +96,19 @@ in an automated way: that their own data leaks, or even its existence leaks, to anyone. * _Not done:_ **Shared backups:** People who do trust each other should be able to share backed up data in the repository. +* _Done:_ **Limited local cache:** The Obnam client may cache data + from the server locally, but the cache should be small, and its size + must not be proportional to the amount of live data or the amount of + data on the server. +* _Not done_: **Resilient:** If the metadata about a backup or the + backed up data is corrupted or goes missing, everything that can be + restored must be possible to restore, and the backup repository must + be possible to be repaired so that it's internally consistent. +* _Not done_: **Self-compatible:** It must be possible to use any + version of the client with any version of the backup repository, and + to restore any backup with any later version of the client. +* _Not done:_ **No re-backups:** The system must never require the + user to do more than one full backup the same repository. The detailed, automatically verified acceptance criteria are documented below, as _scenarios_ described for the [Subplot][] tool. @@ -95,6 +117,87 @@ outcomes. [Subplot]: https://subplot.liw.fi/ +# Threat model + +This chapter discusses the various threats against backups. Or it +will. For now it's very much work in progress. This version of the +chapter is only meant to get threat modeling started by having the +simplest possible model that is in any way useful. + +## Backed up data is readable by server operator + +This threat is about the operator of the backup server being able to +read the data backed up by any user of the server. We have to assume +that the operator can read any file and can also eavesdrop all network +traffic. The operator can even read all physical and virtual memory on +the server. + +The mitigation strategy is to encrypt the data before it is sent to +the server. If the server never receives cleartext data, the operator +can't read it. + +Backups have four kinds of data: + +* actual contents of live data files +* metadata about live data files, as stored on the client file system, + such as the name, ownership, or size of each file +* metadata about the contents of live data, such as its cryptographic + checksum +* metadata about the backup itself + +For now, we are concerned about the first two kinds. The rest will be +addressed later. + +The mitigation technique against this threat is to encrypt the live +data and its metadata before uploading it to the server. + +## An attacker with access to live data can stealthily exclude files from the backup + +This threat arises from Obnam's support for [CACHEDIR.TAG][] files. As the spec +itself says in the "Security Considerations" section: + +> "Blind" use of cache directory tags in automatic system backups could +> potentially increase the damage that intruders or malware could cause to +> a system. A user or system administrator might be substantially less likely to +> notice the malicious insertion of a CACHDIR.TAG into an important directory +> than the outright deletion of that directory, for example, causing the +> contents of that directory to be omitted from regular backups. + +This is mitigated in two ways: + +1. if an incremental backup finds a tag which wasn't in the previous backup, + Obnam will show the path to the tag, and exit with a non-zero exit code. That + way, the user has a chance to notice the new tag. The backup itself is still + made, so if the tag is legitimate, the user doesn't need to re-run Obnam. + + Error messages and non-zero exit are jarring, so this approach is not + user-friendly. Better than nothing though; + +2. users can set `exclude_cache_tag_directories` to `false`, which will make + Obnam ignore the tags, nullifying the threat. + + This is a last-ditch solution, since it makes the backups larger and slower + (because Obnam has to back up more data). + +[CACHEDIR.TAG]: https://bford.info/cachedir/ + +## Attacker can read backups via chunk server HTTP API + +This threat arises from the fact that the chunk server HTTP API +currently has no authentication. This allows an attacker who can +access the API to copy the backups and break their encryption at +leisure. + +The mitigation is to add access control for the API. + +A simple approach is to have the chunk server admin to create an +**access token** that the client must provide with each API request. +The token can be stored in the client configuration by `obnam init`. + +This would be the simplest possible access control approach. More +nuanced approaches will be added later. + + # Software architecture ## Effects of requirements @@ -141,7 +244,7 @@ requirements and notes how they affect the architecture. * **Large numbers of live data files:** Storing and accessing lists of and meta data about files needs to done using data structures that are efficient for that. -* **Live data in the terabyte range:** +* **Live data in the terabyte range:** FIXME * **Many clients:** The architecture should enable flexibly managing clients. * **Shared repository:** The server component needs identify and @@ -154,66 +257,6 @@ requirements and notes how they affect the architecture. access that. -## On SFTP versus HTTPS - -Obnam1 supported using a standard SFTP server as a backup repository, -and this was a popular feature. This section argues against supporting -SFTP in Obnam2. - -The performance requirement for network use means favoring protocols -such as HTTPS, or even QUIC, rather than SFTP. - -SFTP works on top of SSH. SSH provides a TCP-like abstraction for -SFTP, and thus multiple SFTP connections can run over the same SSH -connection. However, SSH itself uses a single TCP connection. If that -TCP connection has a dropped packet, all traffic over the SSH -connections, including all SFTP connections, waits until TCP -re-transmits the lost packet and re-synchronizes itself. - -With multiple HTTP connections, each on its own TCP connection, a -single dropped packet will not affect other HTTP transactions. Even -better, the new QUIC protocol doesn't use TCP. - -The modern Internet is to a large degree designed for massive use of -the world wide web, which is all HTTP, and adopting QUIC. It seems -wise for Obnam to make use of technologies that have been designed -for, and proven to work well with concurrency and network problems. - -Further, having used SFTP with Obnam1, it is not always an easy -protocol to use. Further, if there is a desire to have controlled -sharing of parts of one client's data with another, this would require -writing a custom SFTP service, which seems much harder to do than -writing a custom HTTP service. From experience, a custom HTTP service -is easy to do. A custom SFTP service would need to shoehorn the -abstractions it needs into something that looks more or less like a -Unix file system. - -The benefit of using SFTP would be that a standard SFTP service could -be used, if partial data sharing between clients is not needed. This -would simplify deployment and operations for many. However, it doesn't -seem important enough to warrant the implementation effort. - -Supporting both HTTP and SFTP would be possible, but also much more -work and against the desire to keep things simple. - -## On "btrfs send" and similar constructs - -The btrfs and ZFS file systems, and possibly others, have a way to -mark specific states of the file system and efficiently generate a -"delta file" of all the changes between the states. The delta can be -transferred elsewhere, and applied to a copy of the file system. This -can be quite efficient, but Obnam won't be built on top of such a -system. - -On the one hand, it would force the use of specific file systems: -Obnam would no be able to back up data on, say, an ext4 file system, -which seems to be the most popular one by far. - -Worse, it also for the data to be restored to the same type of file -system as where the live data was originally. This onerous for people -to do. - - ## Overall shape It seems fairly clear that a simple shape of the software architecture @@ -266,8 +309,8 @@ The responsibilities of the server are roughly: The responsibilities of the client are roughly: * split live data into chunks, upload them to server -* store metadata of live data files in a file, which represents a - backup generation, store that too as chunks on the server +* store metadata of live data files in a generation file (an SQLite + database), store that too as chunks on the server * retrieve chunks from server when restoring * let user manage sharing of backups with other clients @@ -282,6 +325,408 @@ RSA-signed JSON Web Tokens. The server is configured to trust specific public keys. The clients have the private keys and generate the tokens themselves. +## Logical structure of backups + +For each backup (generation) the client stores, on the server, exactly +one _generation chunk_. This is a chunk that is specially marked as a +generation, but is otherwise not special. The generation chunk content +is a list of identifiers for chunks that form an SQLite database. + +The SQLite database lists all the files in the backup, as well as +their metadata. For each file, a list of chunk identifiers are listed, +for the content of the file. The chunks may be shared between files in +the same backup or different backups. + +File content data chunks are just blobs of data with no structure. +They have no reference to other data chunks, or to files or backups. +This makes it easier to share them between files. + +Let's look at an example. In the figure below there are three backups, +each using three chunks for file content data. One chunk, "data chunk +3", is shared between all three backups. + +~~~pikchr +GEN1: ellipse "Backup 1" big big +move 200% +GEN2: ellipse "Backup 2" big big +move 200% +GEN3: ellipse "Backup 3" big big + +arrow from GEN1.e right to GEN2.w +arrow from GEN2.e right to GEN3.w + +arrow from GEN1.s down 100% +DB1: box "SQLite" big big + +arrow from DB1.s left 20% then down 100% +C1: file "data" big big "chunk 1" big big + +arrow from DB1.s right 0% then down 70% then right 100% then down 30% +C2: file "data" big big "chunk 2" big big + +arrow from DB1.s right 20% then down 30% then right 200% then down 70% +C3: file "data" big big "chunk 3" big big + + + +arrow from GEN2.s down 100% +DB2: box "SQLite" big big + +arrow from DB2.s left 20% then down 100% then down 0.5*C3.height then to C3.e + +arrow from DB2.s right 0% then down 70% then right 100% then down 30% +C4: file "data" big big "chunk 4" big big + +arrow from DB2.s right 20% then down 30% then right 200% then down 70% +C5: file "data" big big "chunk 5" big big + + + + +arrow from GEN3.s down 100% +DB3: box "SQLite" big big + +arrow from DB3.s left 50% then down 100% then down 1.5*C3.height \ + then left until even with C3.s then up to C3.s + +arrow from DB3.s right 20% then down 100% +C6: file "data" big big "chunk 6" big big + +arrow from DB3.s right 60% then down 70% then right 100% then down 30% +C7: file "data" big big "chunk 7" big big +~~~ + + +## Evolving the database + +The per-generation SQLite database file has a schema. Over time it may +be necessary to change the schema. This needs to be done carefully to +avoid having backup clients to have to do a full backup of previously +backed up data. + +We do this by storing the "schema version" in the database. Each +database will have a table `meta`: + +~~~sql +CREATE TABLE meta (key TEXT, value TEXT) +~~~ + +This will allow key/value pairs serialized into text. We use the keys +`schema_version_major` and `schema_version_minor` to store the schema +version. This will allow the Obnam client to correctly restore the +backup, or at least do the best job it can, while warning the user +there may be an incompatibility. + +We may later add more keys to the `meta` table if there's a need. + +The client will support every historical major version, and the latest +historical minor version of each major version. We will make sure that +this will be enough to restore every previously made backup. That is, +every backup with schema version x.y will be possible to correctly +restore with a version of the Obnam client that understands schema +version x.z, where _z >= y_. If we make a change that would break +this, we increment the major version. + +We may drop support for a schema version if we're sure no backups with +that schema version exist. This is primarily to be able to drop schema +versions that were never included in a released version of the Obnam +client. + +To verify schema compatibility support, we will, at minimum, have +tests that automatically make backups with every supported major +version, and restore them. + + +## On SFTP versus HTTPS + +Obnam1 supported using a standard SFTP server as a backup repository, +and this was a popular feature. This section argues against supporting +SFTP in Obnam2. + +The performance requirement for network use means favoring protocols +such as HTTPS, or even QUIC, rather than SFTP. + +SFTP works on top of SSH. SSH provides a TCP-like abstraction for +SFTP, and thus multiple SFTP connections can run over the same SSH +connection. However, SSH itself uses a single TCP connection. If that +TCP connection has a dropped packet, all traffic over the SSH +connections, including all SFTP connections, waits until TCP +re-transmits the lost packet and re-synchronizes itself. + +With multiple HTTP connections, each on its own TCP connection, a +single dropped packet will not affect other HTTP transactions. Even +better, the new QUIC protocol doesn't use TCP. + +The modern Internet is to a large degree designed for massive use of +the world wide web, which is all HTTP, and adopting QUIC. It seems +wise for Obnam to make use of technologies that have been designed +for, and proven to work well with concurrency and network problems. + +Further, having used SFTP with Obnam1, it is not always an easy +protocol to use. Further, if there is a desire to have controlled +sharing of parts of one client's data with another, this would require +writing a custom SFTP service, which seems much harder to do than +writing a custom HTTP service. From experience, a custom HTTP service +is easy to do. A custom SFTP service would need to shoehorn the +abstractions it needs into something that looks more or less like a +Unix file system. + +The benefit of using SFTP would be that a standard SFTP service could +be used, if partial data sharing between clients is not needed. This +would simplify deployment and operations for many. However, it doesn't +seem important enough to warrant the implementation effort. + +Supporting both HTTP and SFTP would be possible, but also much more +work and against the desire to keep things simple. + +## On "btrfs send" and similar constructs + +The btrfs and ZFS file systems, and possibly others, have a way to +mark specific states of the file system and efficiently generate a +"delta file" of all the changes between the states. The delta can be +transferred elsewhere, and applied to a copy of the file system. This +can be quite efficient, but Obnam won't be built on top of such a +system. + +On the one hand, it would force the use of specific file systems: +Obnam would no be able to back up data on, say, an ext4 file system, +which seems to be the most popular one by far. + +Worse, it also for the data to be restored to the same type of file +system as where the live data was originally. This onerous for people +to do. + + +## On content addressable storage + +[content-addressable storage]: https://en.wikipedia.org/wiki/Content-addressable_storage +[git version control system]: https://git-scm.com/ + +It would be possible to use the cryptographic checksum ("hash") of the +contents of a chunk as its identifier on the server side, also known +as [content-addressable storage][]. This would simplify de-duplication +of chunks. However, it also has some drawbacks: + +* it becomes harder to handle checksum collisions +* changing the checksum algorithm becomes harder + +In 2005, the author of [git version control system][] chose the +content addressable storage model, using the SHA1 checksum algorithm. +At the time, the git author considered SHA1 to be reasonably strong +from a cryptographic and security point of view, for git. In other +words, given the output of SHA1, it was difficult to deduce what the +input was, or to find another input that would give the same output, +known as a checksum collision. It is still difficult to deduce the +input, but manufacturing collisions is now feasible, with some +constraints. The git project has spent years changing the checksum +algorithm. + +Collisions are problematic for security applications of checksum +algorithms in general. Checksums are used, for example, in storing and +verifying passwords: the cleartext password is never stored, and +instead a checksum of it is computed and stored. To verify a later +login attempt a new checksum is computed from the newly entered +password from the attempt. If the checksums match, the password is +accepted.[^passwords] This means that if an attacker can find _any_ input that +gives the same output for the checksum algorithm used for password +storage, they can log in as if they were a valid user, whether the +password they have is the same as the real one. + +[^passwords]: In reality, storing passwords securely is much more + complicated than described here. + +For backups, and version control systems, collisions cause a different +problem: they can prevent the correct content from being stored. If +two files (or chunks) have the same checksum, only one will be stored. +If the files have different content, this is a problem. A backup +system should guard against this possibility. + +As an extreme and rare, but real, case consider a researcher of +checksum algorithms. They've spent enormous effort to produce two +distinct files that have the same checksum. They should be able make a +backup of the files, and restore them, and not lose one. They should +not have to know that their backup system uses the same checksum +algorithm they are researching, and have to guard against the backup +system getting the files confused. (Backup systems should be boring +and just always work.) + +Attacks on security-sensitive cryptographic algorithms only get +stronger by time. It is therefore necessary for Obnam to be able to +easily change the checksum algorithm it uses, without disruption for +user. To achieve this, Obnam does not use content-addressable storage. + +Obnam will (eventually, as this hasn't been implemented yet) allow +storing multiple checksums for each chunk. It will use the strongest +checksum available for a chunk. Over time, the checksums for chunks +can be replaced with stronger ones. This will allow Obnam to migrate +to a stronger algorithm when attacks against the current one become +too scary. + +## On pull versus push backups + +Obnam only does push backups. This means the client runs on the host +where the live data is, and sends it to the server. + +Backups could also be pulled, in that the server reaches out tot he +host where the live data is, retrieves the data, and stores it on the +server. Obnam does not do this, due to the hard requirement that live +data never leaves its host in cleartext. + +The reason pull backups are of interest in many use cases is because +they allow central administration of backups, which can simplify +things a lot in a large organization. Central backup administration +can be achieved with Obnam in a more complicated way: the installation +and configuration of live data hosts is done in a central fashion +using configuration management, and if necessary, backups can be +triggered on each host by having the server reach out and run the +Obnam client. + +## On splitting file data into chunks + +A backup program needs to split the data it backs up into chunks. This +can be done in various ways. + +### A complete file as a single chunk + +This is a very simple approach, where the whole file is considered to +be a chunk, regardless of its size. + +Using complete files is often impractical, since they need to be +stored and transferred as a unit. If a file is enormous, transferring +it completely can be a challenge: if there's a one-bit error in the +transfer, the whole thing needs to be transferred again. + +There is no de-duplication except possibly of entire files. + +### Fixed size chunks + +Split a file into chunks of a fixed size. For example, if the chunk +size is 1 MiB, a 1 GiB file is 1024 chunks. All chunks are of the same +size, unless a file size is not a multiple of the chunk size. + +Fixed size chunks are very easy to implement and make de-duplication +of partial files possible. However, that de-duplication tends to only +work for the beginnings of file: inserting data in the file tends to +result in chunks after the insertion not matching anymore. + +### Splitting based on a formula using content + +A rolling checksum function is computed on a sliding window of bytes +from the input file. The window has a fixed size. The function is +extremely efficient to compute when bytes are moved into or out of the +window. When the value of the function, the checksum, matches a +certain bit pattern, it is considered a chunk boundary. Such a pattern +might for example be that the lowest N bits are zero. Any data that +is pushed out of the sliding window also forms a chunk. + +The code to split into chunks may set minimum and maximum sizes of +chunks, whether from checksum patterns or overflowed bytes. This +prevents pathological input data, where the checksum has the boundary +bit pattern after every byte, to not result in each input byte being +its own chunk. + +This finds chunks efficiently even new data is inserted into the input +data, with some caveats. + +Example: assume a sliding window of four bytes, and a 17-byte input +file where there are four copies of the same 4-byte sequence with a +random byte in between. + ++------+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ +|data | a| b| c| d| a| b| c| d|ff| a| b| c| d| a| b| c| d| ++------+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ +|offset|01|02|03|04|05|06|07|08|09|10|11|12|13|14|15|16|17| ++------+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ + +Bytes 1-4 are the same as bytes 5-8, 10-13, and 14-17. When we compute +the checksum for each input byte, we get the results below, for a +moving sum function. + ++--------+------+----------+-----------------+ +| offset | byte | checksum | chunk boundary? | ++--------+------+----------+-----------------+ +| 0 | a | a | no | ++--------+------+----------+-----------------+ +| 1 | b | a+b | no | ++--------+------+----------+-----------------+ +| 2 | c | a+b+c | no | ++--------+------+----------+-----------------+ +| 3 | d | a+b+c+ | yes | ++--------+------+----------+-----------------+ +| 4 | a | a | no | ++--------+------+----------+-----------------+ +| 5 | b | a+b | no | ++--------+------+----------+-----------------+ +| 6 | c | a+b+c | no | ++--------+------+----------+-----------------+ +| 7 | d | a+b+c+ | yes | ++--------+------+----------+-----------------+ +| 9 | ff | ff | no | ++--------+------+----------+-----------------+ +| 10 | a | ff+a | no | ++--------+------+----------+-----------------+ +| 11 | b | ff+a+b | no | ++--------+------+----------+-----------------+ +| 12 | c | ff+a+b+c | no | ++--------+------+----------+-----------------+ +| 13 | d | a+b+c+d | yes | ++--------+------+----------+-----------------+ +| 14 | a | a | no | ++--------+------+----------+-----------------+ +| 15 | b | a+b | no | ++--------+------+----------+-----------------+ +| 16 | c | a+b+c | no | ++--------+------+----------+-----------------+ +| 17 | d | a+b+c+ | yes | ++--------+------+----------+-----------------+ + +Note that in this example, the byte at offset 9 (0xff) slides out of +the window then byte at offset 13 slides in, and in results in bytes +at offsets 10-13 being recognized as a chunk by the checksum function. +This example is carefully constructed for that happy co-incidence. In +a more realistic scenario, the data after the inserted bytes might not +notice a chunk boundary until after the sliding window has been filled +once. + +By choosing a suitable window size and checksum value pattern for +chunk boundaries, the chunk splitting can find smaller or large +chunks, balancing the possibility for more detailed de-duplication +versus the overhead of storing many chunks. + +### Varying splitting method based on file type + +The data may contain files of different types, and this can be used to +vary the way data is split into chunks. For example, compressed video +files may use one way of chunking, while software source code may use +another. + +For example: + +* emails in mbox or Maildir formats could be split based on headers, + body, and attachment and each of those into chunks +* SQL dumps of databases tend to contain very large numbers containing + the same structure +* video files are often split into frames, possibly those can be used + for intelligent chunking? +* uncompressed tar files have a definite structure (header, followed + by file data) that can probably be used for splitting into chunks +* ZIP files compress each file separately, which could be used to + split them into chunks: this way, two ZIP files with the same file + inside them might share the compressed file as a chunk +* disk images often contain long sequences of zeroes, which could be + used for splitting into chunks + +### Next actions + +Obnam currently splits data using fixed size chunks. This can and will +be improved, and the changes will only affect the client. Help is +welcome. + +### Thanks + +Thank you to Daniel Silverstone who explained some of the mathematics +about this to me. + # File metadata @@ -321,11 +766,9 @@ runs on can handle. On Unix, the filename is a sequence of bytes. Certain bytes have special meaning: -byte ASCII meaning ----- ------- ---------- -0 NUL indicates end of filename -56 period used for . and .. directory entries -57 slash used to separate components in a pathname +* byte 0, ASCII NUL character: terminates filename +* byte 56, ASCII period character: used for . and .. directory entries +* byte 57, ASCII slash character: used to separate components in a pathname On generic Unix, the operating system does not interpret other bytes. It does not impose a character set. Binary filenames are OK, as long @@ -369,50 +812,51 @@ use [lstat(2)][] instead. The metadata is stored in an [inode][]. Both variants return a C `struct stat`. On Linux, it has the following fields: -* `st_dev` – id of the block device containing file system where +* `st_dev` — id of the block device containing file system where the file is; this encodes the major and minor device numbers - this field can't be restored as such, it is forced by the operating system for the file system to which files are restored - Obnam stores it so that hard links can be restored, see below -* `st_ino` – the inode number for the file +* `st_ino` — the inode number for the file - this field can't be restored as such, it is forced by the file system whan the restored file is created - Obnam stores it so that hard links can be restored, see below -* `st_nlink` – number of hard links referring to the inode +* `st_nlink` — number of hard links referring to the inode - this field can't be restored as such, it is maintained by the operating system when hard links are created - Obnam stores it so that hard links can be restored, see below -* `st_mode` – file type and permissions +* `st_mode` — file type and permissions - stored and restored -* `st_uid` – the numeric id of the user account owning the file +* `st_uid` — the numeric id of the user account owning the file - stored - restored if restore is running as root, otherwise not restored -* `st_gid` – the numeric id of the group owning the file +* `st_gid` — the numeric id of the group owning the file - stored - restored if restore is running as root, otherwise not restored -* `st_rdev` – the device this inode represents - - not stored? -* `st_size` – size or length of the file in bytes +* `st_rdev` — the device this inode represents + - not stored +* `st_size` — size or length of the file in bytes - stored - restored implicitly be re-creating the origtinal contents -* `st_blksize` – preferred block size for efficient I/O - - not stored? -* `st_blocks` – how many blocks of 512 bytes are actually +* `st_blksize` — preferred block size for efficient I/O + - chosen automatically by the operating system, can't be changed + - not stored +* `st_blocks` — how many blocks of 512 bytes are actually allocated to store this file's contents - see below for discussion about sparse files - - not stored by Obnam -* `st_atime` – timestamp of latest access + - not stored +* `st_atime` — timestamp of latest access - stored and restored - - On Linux, split into two integer fields -* `st_mtime` – timestamp of latest modification + - On Linux, split into two integer fields to achieve nanosecond resolution +* `st_mtime` — timestamp of latest modification - stored and restored - - On Linux, split into two integer fields -* `st_ctime` – timestamp of latest inode change - - On Linux, split into two integer fields - - stored - - not restored + - On Linux, split into two integer fields to achieve nanosecond resolution +* `st_ctime` — timestamp of latest inode change + - can't be set by an application, maintained automatically by + operating system + - not stored -Obnam stores most these fields. Not all of them can be restored, +Obnam stores most of these fields. Not all of them can be restored, especially not explicitly. The `st_dev` and `st_ino` fields get set by the file system when when a restored file is created. They're stored so that Obnam can restore all hard links to the same inode. @@ -437,6 +881,15 @@ the file was used instead. Obnam stores the contents of a symbolic link, the "target" of the link, and restores the original value without modification. +To recognize that filename are hard links to the same file, a program +needs to use **lstat**(2) on each filename and compare the `st_dev` +and `st_ino` fields of the result. If they're identical, the filenames +refer to the same inode. It's important to check both fields so that +one is certain the resulting data refers to the same inode on the same +file system. Keeping track of filenames pointing at the same inode can +be resource intensive. It can be helpful to note that it only needs to +be done for inodes with an `st_nlink` count greater then one. + ## On access time stamps The `st_atime` field is automatically updated when a file or directory @@ -458,14 +911,16 @@ change mount options, or make the file system be read-only. It thus needs to use the `NO_ATIME` flag to the [open(2)][] system call. Obnam does not do this yet. In fact, it doesn't store or restore the -access time stamp yet. +access time stamp, and it might never do that. If you have a need for +that, please open issue on the [Obnam issue +tracker](https://gitlab.com/obnam/obnam/-/issues). [open(2)]: https://linux.die.net/man/2/open ## Time stamp representation Originally, Unix (and Linux) stored file time stamps as whole seconds -since the beginning of 1970. Linux now stores timestamp with up to +since the beginning of 1970. Linux now stores file timestamps with up to nanosecond precision, depending on file system type. Obnam handles this by storing and restoring nanosecond timestamps. If, when restoring, the target file system doesn't support that precision, then @@ -581,41 +1036,42 @@ clients. Chunks consist of arbitrary binary data, a small amount of metadata, and an identifier chosen by the server. The chunk metadata is a JSON -object, consisting of the following fields: +object, consisting of the following field (there used to be more): -* `sha256` – the SHA256 checksum of the chunk contents as +* `label` — the SHA256 checksum of the chunk contents as determined by the client - this MUST be set for every chunk, including generation chunks - the server allows for searching based on this field - note that the server doesn't verify this in any way, to pave way - for future client-side encryption of the chunk data -* `generation` – set to `true` if the chunk represents a - generation - - may also be set to `false` or `null` or be missing entirely - - the server allows for listing chunks where this field is set to - `true` -* `ended` – the timestamp of when the backup generation ended - - note that the server doesn't process this in any way, the contents - is entirely up to the client - - may be set to the empty string, `null`, or be missing entirely - - this can't be used in searches + for future client-side encryption of the chunk data, including the + label + - there is no requirement that only one chunk has any given label When creating or retrieving a chunk, its metadata is carried in a `Chunk-Meta` header as a JSON object, serialized into a textual form that can be put into HTTP headers. +There are several kinds of chunk. The kind only matters to the client, +not to the server. + +* Data chunk: File content data, from live data files, or from an + SQLite database file listing all files in a backup. +* Generation chunk: A list of chunks for the SQLite file for a + generation. +* Client trust: A list of ids of generation chunks, plus other data + that are per-client, not per-backup. + ## Server The server has the following API for managing chunks: -* `POST /chunks` – store a new chunk (and its metadata) on the +* `POST /v1/chunks` — store a new chunk (and its metadata) on the server, return its randomly chosen identifier -* `GET /chunks/<ID>` – retrieve a chunk (and its metadata) from +* `GET /v1/chunks/<ID>` — retrieve a chunk (and its metadata) from the server, given a chunk identifier -* `GET /chunks?sha256=xyzzy` – find chunks on the server whose - metadata indicates their contents has a given SHA256 checksum -* `GET /chunks?generation=true` – find generation chunks +* `GET /v1/chunks?label=xyzzy` — find chunks on the server whose + metadata has a specific value for a label. HTTP status codes are used to indicate if a request succeeded or not, using the customary meanings. @@ -639,17 +1095,14 @@ and should treat it as an opaque value. When a chunk is retrieved, the chunk metadata is returned in the `Chunk-Meta` header, and the contents in the response body. -It is not possible to update a chunk or its metadata. - -When searching for chunks, any matching chunk's identifiers and -metadata are returned in a JSON object: +It is not possible to update a chunk or its metadata. It's not +possible to remove a chunk. When searching for chunks, any matching +chunk's identifiers and metadata are returned in a JSON object: ~~~json { "fe20734b-edb3-432f-83c3-d35fe15969dd": { - "sha256": "09ca7e4eaa6e8ae9c7d261167129184883644d07dfba7cbfbc4c8a2e08360d5b", - "generation": null, - "ended: null, + "label": "09ca7e4eaa6e8ae9c7d261167129184883644d07dfba7cbfbc4c8a2e08360d5b" } } ~~~ @@ -683,6 +1136,88 @@ restores all the files in the SQLite database. +## Encryption and authenticity of chunks + +*This is a plan that will be implemented soon. When it has been, this +section needs to be updated to to use present tense.* + +Obnam encrypts data it stores on the server, and checks that the data +it retrieves from the server is what it stored. This is all done in +the client: the server should never see any data isn't encrypted, and +the client can't trust the server to validate anything. + +Obnam will be using _Authenticated Encryption with Associated Data_ or +[AEAD][]. AEAD both encrypts data, and validates it before decrypting. +AEAD uses two encryption keys, one algorithm for symmetric encryption, +and one algorithm for a message authentication codes or [MAC][]. AEAD +encrypts the plaintext with a symmetric encryption algorithm using the +first key, giving a ciphertext. It then computes a MAC of the +ciphertext using the second key. Both the ciphertext and MAC are +stored on the server. + +For decryption, a MAC is computed against the retrieved +ciphertext, and compared to the retrieved MAC. If the MACs differ, +that's an error and no decryption is done. If they do match, the +ciphertext is decrypted. + +Obnam will require the user to provide a passphrase, and will derive +the two keys from the single passphrase, using [PBKDF2][], rather than +having the user provide two passphrases. The derived keys will be +stored in a file that only the owner can read. (This is simple, and good +enough for now, but needs to improved later.) + +When this is all implemented, there will be a setup step before the +first backup: + +~~~sh +$ obnam init +Passphrase for encryption: +Re-enter to make sure: +$ obnam backup +~~~ + +The `init` step asks for a passphrase, uses PBKDF2 (with the [pbkdf2 +crate][]) to derive the two keys, and writes a JSON file with the keys +into `~/.config/obnam/keys.json`, making that file be readable only by +the user running Obnam. Other operations get the keys from that file. +For now, we will use the default parameters of the pbkdf2 crate, to +keep things simple. (This will need to be made more flexible later: if +nothing else, Obnam should not be vulnerable to the defaults +changing.) + +The `init` step will not be optional. There will only be encrypted +backups. + +Obnam will use the [aes-gcm crate][] for AEAD, since it has been +audited. If that choice turns out to be less than optimal, it can be +reconsidered later. The `encrypt` function doesn't return the MAC and +ciphertext separately, so we don't store them separately. However, +each chunk needs its own [nonce][], which we will generate. We'll use +a 96-bit (or 12-byte) nonce. We'll use the [rand crate][] to generate +random bytes. + +The chunk sent to the server will be encoded as follows: + +* chunk format: a 32-bit unsigned integer, 0x0001, store in + little-endian form. +* a 12-byte nonce unique to the chunk +* the ciphertext + +The format version prefix dictates the content and structure of the +chunk. This document defines version 1 of the format. The Obnam client +will refuse to operate on backup generations which use chunk formats +it cannot understand. + + +[AEAD]: https://en.wikipedia.org/wiki/Authenticated_encryption#Authenticated_encryption_with_associated_data_(AEAD) +[MAC]: https://en.wikipedia.org/wiki/Message_authentication_code +[aes-gcm crate]: https://crates.io/crates/aes-gcm +[PBKDF2]: https://en.wikipedia.org/wiki/PBKDF2 +[pbkdf2 crate]: https://crates.io/crates/pbkdf2 +[nonce]: https://en.wikipedia.org/wiki/Cryptographic_nonce +[rand crate]: https://crates.io/crates/rand + + # Acceptance criteria for the chunk server These scenarios verify that the chunk server works on its own. The @@ -696,49 +1231,34 @@ search, and delete it. This is needed so the client can manage the storage of backed up data. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a file data.dat containing some random data -when I POST data.dat to /chunks, with chunk-meta: {"sha256":"abc"} +when I POST data.dat to /v1/chunks, with chunk-meta: {"label":"0abc"} then HTTP status code is 201 and content-type is application/json and the JSON body has a field chunk_id, henceforth ID +and server has 1 chunks ~~~ We must be able to retrieve it. ~~~scenario -when I GET /chunks/<ID> +when I GET /v1/chunks/<ID> then HTTP status code is 200 and content-type is application/octet-stream -and chunk-meta is {"sha256":"abc","generation":null,"ended":null} +and chunk-meta is {"label":"0abc"} and the body matches file data.dat ~~~ We must also be able to find it based on metadata. ~~~scenario -when I GET /chunks?sha256=abc +when I GET /v1/chunks?label=0abc then HTTP status code is 200 and content-type is application/json -and the JSON body matches {"<ID>":{"sha256":"abc","generation":null,"ended":null}} +and the JSON body matches {"<ID>":{"label":"0abc"}} ~~~ -Finally, we must be able to delete it. After that, we must not be able -to retrieve it, or find it using metadata. - -~~~scenario -when I DELETE /chunks/<ID> -then HTTP status code is 200 - -when I GET /chunks/<ID> -then HTTP status code is 404 - -when I GET /chunks?sha256=abc -then HTTP status code is 200 -and content-type is application/json -and the JSON body matches {} -~~~ ## Retrieve a chunk that does not exist @@ -746,9 +1266,8 @@ We must get the right error if we try to retrieve a chunk that does not exist. ~~~scenario -given an installed obnam -and a running chunk server -when I try to GET /chunks/any.random.string +given a working Obnam system +when I try to GET /v1/chunks/any.random.string then HTTP status code is 404 ~~~ @@ -757,26 +1276,13 @@ then HTTP status code is 404 We must get an empty result if searching for chunks that don't exist. ~~~scenario -given an installed obnam -and a running chunk server -when I GET /chunks?sha256=abc +given a working Obnam system +when I GET /v1/chunks?label=0abc then HTTP status code is 200 and content-type is application/json and the JSON body matches {} ~~~ -## Delete chunk that does not exist - -We must get the right error when deleting a chunk that doesn't exist. - -~~~scenario -given an installed obnam -and a running chunk server -when I try to DELETE /chunks/any.random.string -then HTTP status code is 404 -~~~ - - ## Persistent across restarts Chunk storage, and the index of chunk metadata for searches, needs to @@ -785,10 +1291,9 @@ be persistent across restarts. This scenario verifies it is so. First, create a chunk. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a file data.dat containing some random data -when I POST data.dat to /chunks, with chunk-meta: {"sha256":"abc"} +when I POST data.dat to /v1/chunks, with chunk-meta: {"label":"0abc"} then HTTP status code is 201 and content-type is application/json and the JSON body has a field chunk_id, henceforth ID @@ -804,22 +1309,193 @@ given a running chunk server Can we still find it by its metadata? ~~~scenario -when I GET /chunks?sha256=abc +when I GET /v1/chunks?label=0abc then HTTP status code is 200 and content-type is application/json -and the JSON body matches {"<ID>":{"sha256":"abc","generation":null,"ended":null}} +and the JSON body matches {"<ID>":{"label":"0abc"}} ~~~ Can we still retrieve it by its identifier? ~~~scenario -when I GET /chunks/<ID> +when I GET /v1/chunks/<ID> then HTTP status code is 200 and content-type is application/octet-stream -and chunk-meta is {"sha256":"abc","generation":null,"ended":null} +and chunk-meta is {"label":"0abc"} and the body matches file data.dat ~~~ + +## Obeys `OBNAM_SERVER_LOG` environment variable + +The chunk server logs its actions to stderr. Verbosity of the log depends on the +`OBNAM_SERVER_LOG` envvar. This scenario verifies that the variable can make the +server more chatty. + +~~~scenario +given a working Obnam system +and a file data1.dat containing some random data +when I POST data1.dat to /v1/chunks, with chunk-meta: {"label":"qwerty"} +then the JSON body has a field chunk_id, henceforth ID +and chunk server's stderr doesn't contain "Obnam server starting up" +and chunk server's stderr doesn't contain "created chunk <ID>" + +given a running chunk server with environment {"OBNAM_SERVER_LOG": "info"} +and a file data2.dat containing some random data +when I POST data2.dat to /v1/chunks, with chunk-meta: {"label":"xyz"} +then the JSON body has a field chunk_id, henceforth ID +and chunk server's stderr contains "Obnam server starting up" +and chunk server's stderr contains "created chunk <ID>" +~~~ + + +# Acceptance criteria for the Obnam client + +The scenarios in chapter verify that the Obnam client works as it +should, when it is used independently of an Obnam chunk server. + +## Client shows its configuration + +This scenario verifies that the client can show its current +configuration, with the `obnam config` command. The configuration is +stored as YAML, but the command outputs JSON, to make sure it doesn't +just copy the configuration file to the output. + +~~~scenario +given an installed obnam +and file config.yaml +and JSON file config.json converted from YAML file config.yaml +when I run obnam --config config.yaml config +then stdout, as JSON, has all the values in file config.json +~~~ + +~~~{#config.yaml .file .yaml .numberLines} +roots: [live] +server_url: https://backup.example.com +verify_tls_cert: true +~~~ + + +## Client expands tildes in its configuration file + +This scenario verifies that the client expands tildes in pathnames in +its configuration file. + + +~~~scenario +given an installed obnam +and file tilde.yaml +when I run obnam --config tilde.yaml config +then stdout contains home directory followed by /important +then stdout contains home directory followed by /obnam.log +~~~ + +~~~{#tilde.yaml .file .yaml .numberLines} +roots: [~/important] +log: ~/obnam.log +server_url: https://backup.example.com +verify_tls_cert: true +~~~ + + +## Client requires https + +This scenario verifies that the client rejects a configuration with a +server URL using `http:` instead of `https:`. + + +~~~scenario +given an installed obnam +and file http.yaml +when I try to run obnam --config http.yaml config +then command fails +then stderr contains "https:" +~~~ + +~~~{#http.yaml .file .yaml .numberLines} +roots: [live] +server_url: http://backup.example.com +verify_tls_cert: true +~~~ + +## Client lists the backup schema versions it supports + +~~~scenario +given an installed obnam +given file config.yaml +when I run obnam --config config.yaml list-backup-versions +then stdout is exactly "0.0\n1.0\n" +~~~ + +## Client lists the default backup schema version + +~~~scenario +given an installed obnam +given file config.yaml +when I run obnam --config config.yaml list-backup-versions --default-only +then stdout is exactly "0.0\n" +~~~ + +## Client refuses a self-signed certificate + +This scenario verifies that the client refuses to connect to a server +if the server's TLS certificate is self-signed. The test server set up +by the scenario uses self-signed certificates. + +~~~scenario +given a working Obnam system +and a client config based on ca-required.yaml +and a file live/data.dat containing some random data +when I try to run obnam backup +then command fails +then stderr matches regex self.signed certificate +~~~ + +~~~{#ca-required.yaml .file .yaml .numberLines} +verify_tls_cert: true +roots: [live] +~~~ + + +## Encrypt and decrypt chunk locally + +~~~scenario +given a working Obnam system +given a client config based on smoke.yaml +given a file cleartext.dat containing some random data +when I run obnam encrypt-chunk cleartext.dat encrypted.dat '{"label":"fake"}' +when I run obnam decrypt-chunk encrypted.dat decrypted.dat '{"label":"fake"}' +then files cleartext.dat and encrypted.dat are different +then files cleartext.dat and decrypted.dat are identical +~~~ + +## Split a file into chunks + +The `obnam chunkify` command reads one or more files and splits them +into chunks, and writes to the standard output a JSON file describing +each chunk. This scenario verifies that the command works at least in +a simple case. + +~~~scenario +given a working Obnam system +given a client config based on smoke.yaml +given a file data.dat containing "hello, world" +given file chunks.json +when I run obnam chunkify data.dat +then stdout, as JSON, exactly matches file chunks.json +~~~ + +~~~{#chunks.json .file .json} +[ + { + "filename": "data.dat", + "offset": 0, + "len": 12, + "checksum": "09ca7e4eaa6e8ae9c7d261167129184883644d07dfba7cbfbc4c8a2e08360d5b" + } +] +~~~ + # Acceptance criteria for Obnam as a whole The scenarios in this chapter apply to Obnam as a whole: the client @@ -833,22 +1509,65 @@ and their metadata are identical to the original. This is the simplest possible useful use case for a backup system. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on smoke.yaml and a file live/data.dat containing some random data and a manifest of the directory live in live.yaml -when I run obnam --config smoke.yaml backup +when I run obnam backup then backup generation is GEN -when I run obnam --config smoke.yaml list +when I run obnam list then generation list contains <GEN> -when I invoke obnam --config smoke.yaml restore <GEN> rest +when I run obnam resolve latest +then generation list contains <GEN> +when I invoke obnam restore <GEN> rest given a manifest of the directory live restored in rest in rest.yaml -then files live.yaml and rest.yaml match +then manifests live.yaml and rest.yaml match ~~~ ~~~{#smoke.yaml .file .yaml .numberLines} -root: live +verify_tls_cert: false +roots: [live] +~~~ + + +## Inspect a backup + +Once a backup is made, the user needs to be able inspect it to see the +schema version. + +~~~scenario +given a working Obnam system +and a client config based on smoke.yaml +and a file live/data.dat containing some random data +and a manifest of the directory live in live.yaml +when I run obnam backup +when I run obnam inspect latest +then stdout contains "schema_version: 0.0\n" +when I run obnam backup --backup-version=0 +when I run obnam inspect latest +then stdout contains "schema_version: 0.0\n" +when I run obnam backup --backup-version=1 +when I run obnam inspect latest +then stdout contains "schema_version: 1.0\n" +~~~ + +## Backup root must exist + +This scenario verifies that Obnam correctly reports an error if a +backup root directory doesn't exist. + +~~~scenario +given a working Obnam system +and a client config based on missingroot.yaml +and a file live/data.dat containing some random data +when I try to run obnam backup +then command fails +then stderr contains "does-not-exist" +~~~ + +~~~{#missingroot.yaml .file .yaml .numberLines} +verify_tls_cert: false +roots: [live, does-not-exist] ~~~ @@ -862,7 +1581,8 @@ anything. All these scenarios use the following configuration file. ~~~{#metadata.yaml .file .yaml .numberLines} -root: live +verify_tls_cert: false +roots: [live] ~~~ ### Modification time @@ -870,16 +1590,15 @@ root: live This scenario verifies that the modification time is restored correctly. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on metadata.yaml and a file live/data.dat containing some random data and a manifest of the directory live in live.yaml -when I run obnam --config metadata.yaml backup +when I run obnam backup then backup generation is GEN -when I invoke obnam --config metadata.yaml restore <GEN> rest +when I invoke obnam restore <GEN> rest given a manifest of the directory live restored in rest in rest.yaml -then files live.yaml and rest.yaml match +then manifests live.yaml and rest.yaml match ~~~ ### Mode bits @@ -888,17 +1607,16 @@ This scenario verifies that the mode ("permission") bits are restored correctly. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on metadata.yaml and a file live/data.dat containing some random data and file live/data.dat has mode 464 and a manifest of the directory live in live.yaml -when I run obnam --config metadata.yaml backup +when I run obnam backup then backup generation is GEN -when I invoke obnam --config metadata.yaml restore <GEN> rest +when I invoke obnam restore <GEN> rest given a manifest of the directory live restored in rest in rest.yaml -then files live.yaml and rest.yaml match +then manifests live.yaml and rest.yaml match ~~~ ### Symbolic links @@ -906,19 +1624,46 @@ then files live.yaml and rest.yaml match This scenario verifies that symbolic links are restored correctly. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on metadata.yaml and a file live/data.dat containing some random data and symbolink link live/link that points at data.dat +and symbolink link live/broken that points at does-not-exist and a manifest of the directory live in live.yaml -when I run obnam --config metadata.yaml backup +when I run obnam backup then backup generation is GEN -when I invoke obnam --config metadata.yaml restore <GEN> rest +when I invoke obnam restore <GEN> rest given a manifest of the directory live restored in rest in rest.yaml -then files live.yaml and rest.yaml match +then manifests live.yaml and rest.yaml match +~~~ + +## Set chunk size + +This scenario verifies that the user can set the chunk size in the +configuration file. The chunk size only affects the chunks of live +data. + +The backup uses a chunk size of one byte, and backs up a file with +three bytes. This results in three chunks for the file data, plus one +for the generation SQLite file (not split into chunks of one byte), +plus a chunk for the generation itself. Additionally, the "trust root" +chunk exists. A total of six chunks. + +~~~scenario +given a working Obnam system +given a client config based on tiny-chunk-size.yaml +given a file live/data.dat containing "abc" +when I run obnam backup +then server has 6 chunks ~~~ +~~~{#tiny-chunk-size.yaml .file .yaml .numberLines} +verify_tls_cert: false +roots: [live] +chunk_size: 1 +~~~ + + ## Backup or not for the right reason The decision of whether to back up a file or keep the version in the @@ -931,13 +1676,12 @@ This scenario verifies that in the first backup all files are backed up because they were new. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on smoke.yaml and a file live/data.dat containing some random data and a manifest of the directory live in live.yaml -when I run obnam --config smoke.yaml backup -when I run obnam --config smoke.yaml list-files +when I run obnam backup +when I run obnam list-files then file live/data.dat was backed up because it was new ~~~ @@ -947,14 +1691,13 @@ This scenario verifies that if a file hasn't been changed, it's not backed up. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on smoke.yaml and a file live/data.dat containing some random data and a manifest of the directory live in live.yaml -when I run obnam --config smoke.yaml backup -when I run obnam --config smoke.yaml backup -when I run obnam --config smoke.yaml list-files +when I run obnam backup +when I run obnam backup +when I run obnam list-files then file live/data.dat was not backed up because it was unchanged ~~~ @@ -964,15 +1707,14 @@ This scenario verifies that if a file has indeed been changed, it's backed up. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on smoke.yaml and a file live/data.dat containing some random data and a manifest of the directory live in live.yaml -when I run obnam --config smoke.yaml backup +when I run obnam backup given a file live/data.dat containing some random data -when I run obnam --config smoke.yaml backup -when I run obnam --config smoke.yaml list-files +when I run obnam backup +when I run obnam list-files then file live/data.dat was backed up because it was changed ~~~ @@ -983,19 +1725,39 @@ scenario verifies that the client checks the contents hasn't been modified. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on smoke.yaml and a file live/data.dat containing some random data -when I run obnam --config smoke.yaml backup +when I run obnam backup then backup generation is GEN -when I invoke obnam --config smoke.yaml get-chunk <GEN> +when I invoke obnam get-chunk <GEN> then exit code is 0 when chunk <GEN> on chunk server is replaced by an empty file -when I invoke obnam --config smoke.yaml get-chunk <GEN> +when I invoke obnam get-chunk <GEN> then command fails ~~~ +## Irregular files + +This scenario verifies that Obnam backs up and restores files that +aren't regular files, directories, or symbolic links. Specifically, +Unix domain sockets and named pipes (FIFOs). However, block and +character device nodes are not tested, as that would require running +the test suite with `root` permissions and that would be awkward. + +~~~scenario +given a working Obnam system +and a client config based on smoke.yaml +and a file live/data.dat containing some random data +and a Unix socket live/socket +and a named pipe live/pipe +and a manifest of the directory live in live.yaml +when I run obnam backup +when I run obnam restore latest rest +given a manifest of the directory live restored in rest in rest.yaml +then manifests live.yaml and rest.yaml match +~~~ + ## Tricky filenames Obnam needs to handle all filenames the underlying operating and file @@ -1004,17 +1766,89 @@ that consists on a single byte with its top bit set. This is not ASCII, and it's not UTF-8. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on metadata.yaml and a file in live with a non-UTF8 filename and a manifest of the directory live in live.yaml -when I run obnam --config metadata.yaml backup +when I run obnam backup then backup generation is GEN -when I invoke obnam --config metadata.yaml restore <GEN> rest +when I invoke obnam restore <GEN> rest given a manifest of the directory live restored in rest in rest.yaml -then files live.yaml and rest.yaml match +then manifests live.yaml and rest.yaml match +~~~ + +## FIXME: Unreadable file + +FIXME: This scenario has been disabled, temporarily, as my current CI +system runs things as `root` and that means this scenario fails. + +~~~~~~~~ +This scenario verifies that Obnam will back up all files of live data, +even if one of them is unreadable. By inference, we assume this means +other errors on individual files also won't end the backup +prematurely. + + +~~~scenario +given a working Obnam system +and a client config based on smoke.yaml +and a file live/data.dat containing some random data +and a file live/bad.dat containing some random data +and file live/bad.dat has mode 000 +when I run obnam backup +then backup generation is GEN +when I invoke obnam restore <GEN> rest +then file live/data.dat is restored to rest +then file live/bad.dat is not restored to rest +~~~ +~~~~~~~~ + +## FIXME: Unreadable directory + +FIXME: This scenario has been disabled, temporarily, as my current CI +system runs things as `root` and that means this scenario fails. + +~~~~~~~~ +This scenario verifies that Obnam will skip a file in a directory it +can't read. Obnam should warn about that, but not give an error. + +~~~scenario +given a working Obnam system +and a client config based on smoke.yaml +and a file live/unreadable/data.dat containing some random data +and file live/unreadable has mode 000 +when I run obnam backup +then stdout contains "live/unreadable" +then backup generation is GEN +when I invoke obnam restore <GEN> rest +then file live/unreadable is restored to rest +then file live/unreadable/data.dat is not restored to rest +~~~ +~~~~~~~~ + +## FIXME: Unexecutable directory + +FIXME: This scenario has been disabled, temporarily, as my current CI +system runs things as `root` and that means this scenario fails. + +~~~~~~~ +This scenario verifies that Obnam will skip a file in a directory it +can't read. Obnam should warn about that, but not give an error. + +~~~scenario +given a working Obnam system +and a client config based on smoke.yaml +and a file live/dir/data.dat containing some random data +and file live/dir has mode 600 +when I run obnam backup +then stdout contains "live/dir" +then backup generation is GEN +when I invoke obnam restore <GEN> rest +then file live/dir is restored to rest +then file live/dir/data.dat is not restored to rest ~~~ +~~~~~~~ + ## Restore latest generation @@ -1023,51 +1857,251 @@ specified with literal string "latest". It makes two backups, which are different. ~~~scenario -given an installed obnam -and a running chunk server +given a working Obnam system and a client config based on metadata.yaml given a file live/data.dat containing some random data -when I run obnam --config metadata.yaml backup +when I run obnam backup given a file live/more.dat containing some random data and a manifest of the directory live in second.yaml -when I run obnam --config metadata.yaml backup +when I run obnam backup -when I invoke obnam --config metadata.yaml restore latest rest +when I run obnam restore latest rest given a manifest of the directory live restored in rest in rest.yaml -then files second.yaml and rest.yaml match -~~~ - - - - -<!-- -------------------------------------------------------------------- --> - - ---- -title: "Obnam2—a backup system" -author: Lars Wirzenius -documentclass: report -bindings: - - subplot/server.yaml - - subplot/client.yaml - - subplot/data.yaml - - subplot/vendored/runcmd.yaml -template: python -functions: - - subplot/server.py - - subplot/client.py - - subplot/data.py - - subplot/vendored/daemon.py - - subplot/vendored/runcmd.py -classes: - - json -abstract: | - Obnam is a backup system, consisting of a not very smart server for - storing chunks of backup data, and a client that splits the user's - data into chunks. They communicate via HTTP. - - This document describes the architecture and acceptance criteria for - Obnam, as well as how the acceptance criteria are verified. -... +then manifests second.yaml and rest.yaml match +~~~ + +## Restore backups made with each backup version + +~~~scenario +given a working Obnam system +given a client config based on metadata.yaml +given a file live/data.dat containing some random data +given a manifest of the directory live in live.yaml + +when I run obnam backup --backup-version=0 +when I run obnam restore latest rest0 +given a manifest of the directory live restored in rest0 in rest0.yaml +then manifests live.yaml and rest0.yaml match + +when I run obnam backup --backup-version=1 +when I run obnam restore latest rest1 +given a manifest of the directory live restored in rest1 in rest1.yaml +then manifests live.yaml and rest1.yaml match +~~~ + +## Back up multiple directories + +This scenario verifies that Obnam can back up more than one directory +at a time. + + +~~~scenario +given a working Obnam system +and a client config based on roots.yaml +and a file live/one/data.dat containing some random data +and a file live/two/data.dat containing some random data +and a manifest of the directory live/one in one.yaml +and a manifest of the directory live/two in two.yaml +when I run obnam backup +then backup generation is GEN +when I invoke obnam restore <GEN> rest +given a manifest of the directory live/one restored in rest in rest-one.yaml +given a manifest of the directory live/two restored in rest in rest-two.yaml +then manifests one.yaml and rest-one.yaml match +then manifests two.yaml and rest-two.yaml match +~~~ + +~~~{#roots.yaml .file .yaml .numberLines} +roots: +- live/one +- live/two +~~~ + +## CACHEDIR.TAG support + +### By default, skip directories containing CACHEDIR.TAG + +This scenario verifies that Obnam client skips the contents of directories that +contain [CACHEDIR.TAG][], but backs up the tag itself. We back up the +tag so that after a restore, the directory continues to be tagged as a +cache directory. + +[CACHEDIR.TAG]: https://bford.info/cachedir/ + +~~~scenario +given a working Obnam system +and a client config based on client.yaml +and a file live/ignored/data.dat containing some random data +and a cache directory tag in live/ignored +and a file live/not_ignored/data.dat containing some random data +and a manifest of the directory live/not_ignored in initial.yaml +when I run obnam backup +then backup generation is GEN +when I invoke obnam restore <GEN> rest +given a manifest of the directory live/not_ignored restored in rest in restored.yaml +then manifests initial.yaml and restored.yaml match +then file rest/live/ignored/CACHEDIR.TAG contains "Signature: 8a477f597d28d172789f06886806bc55" +then file rest/live/ignored/data.dat does not exist +~~~ + +~~~{#client.yaml .file .yaml .numberLines} +roots: +- live +~~~ + +### Incremental backup errors if it finds new CACHEDIR.TAGs + +To mitigate the risk described in the "Threat Model" chapter, Obnam should +notify the user when it finds CACHEDIR.TAG files that aren't present in the +previous backup. Notification is twofold: the path to the tag should be shown, +and the client should exit with a non-zero code. This scenario runs backups the +a directory (which shouldn't error), then adds a new tag and backups the +directory again, expecting an error. + +~~~scenario +given a working Obnam system +and a client config based on client.yaml +and a file live/data1.dat containing some random data +and a file live/data2.dat containing some random data +when I run obnam backup +then exit code is 0 +given a cache directory tag in live/ +when I try to run obnam backup +then exit code is 1 +and stdout contains "live/CACHEDIR.TAG" +when I run obnam list-files +then exit code is 0 +~~~ +then file live/CACHEDIR.TAG was backed up because it was new +and stdout doesn't contain "live/data1.dat" +and stdout doesn't contain "live/data2.dat" + +### Ignore CACHEDIR.TAGs if `exclude_cache_tag_directories` is disabled + +This scenario verifies that when `exclude_cache_tag_directories` setting is +disabled, Obnam client backs up directories even if they +contain [CACHEDIR.TAG][]. It also verifies that incremental backups don't fail when +new tags are added, i.e. the aforementioned mitigation is disabled too. + +[CACHEDIR.TAG]: https://bford.info/cachedir/ + +~~~scenario +given a working Obnam system +and a client config based on client_includes_cachedirs.yaml +and a file live/ignored/data.dat containing some random data +and a cache directory tag in live/ignored +and a file live/not_ignored/data.dat containing some random data +and a manifest of the directory live in initial.yaml +when I run obnam backup +then backup generation is GEN +when I invoke obnam restore <GEN> rest +given a manifest of the directory live restored in rest in restored.yaml +then manifests initial.yaml and restored.yaml match +given a cache directory tag in live/not_ignored +when I run obnam backup +then exit code is 0 +and stdout doesn't contain "live/not_ignored/CACHEDIR.TAG" +~~~ + +~~~{#client_includes_cachedirs.yaml .file .yaml .numberLines} +roots: +- live +exclude_cache_tag_directories: false +~~~ + + +## Generation information + +This scenario verifies that the Obnam client can show metadata about a +backup generation. + +~~~scenario +given a working Obnam system +given a client config based on smoke.yaml +given a file live/data.dat containing some random data +given a manifest of the directory live in live.yaml +given file geninfo.json +when I run obnam backup +when I run obnam gen-info latest +then stdout, as JSON, has all the values in file geninfo.json +~~~ + +~~~{#geninfo.json .file .json} +{ + "schema_version": { + "major": 0, + "minor": 0 + }, + "extras": { + "checksum_kind": "sha256" + } +} +~~~ + + +# Acceptance criteria for backup encryption + +This chapter outlines scenarios, to be implemented later, for +verifying that Obnam properly encrypts the backups. These scenarios +verify only encryption aspects of Obnam. + +## Backup without passphrase fails + +Verify that trying to backup without having set a passphrase fails +with an error message that clearly identifies the lack of a +passphrase. + +~~~scenario +given a working Obnam system +and a client config, without passphrase, based on encryption.yaml +and a file live/data.dat containing some random data +and a manifest of the directory live in live.yaml +when I try to run obnam backup +then command fails +then stderr contains "obnam init" +~~~ + +~~~{#encryption.yaml .file .yaml .numberLines} +verify_tls_cert: false +roots: [live] +~~~ + +## A passphrase can be set + +Set a passphrase. Verify that it's stored in a file that is only +readable by it owner. Verify that a backup can be made. + +~~~scenario +given a working Obnam system +and a client config, without passphrase, based on encryption.yaml +and a file live/data.dat containing some random data +and a manifest of the directory live in live.yaml +when I run obnam init --insecure-passphrase=hunter2 +then file .config/obnam/passwords.yaml exists +then file .config/obnam/passwords.yaml is only readable by owner +then file .config/obnam/passwords.yaml does not contain "hunter2" +~~~ + +## A passphrase stored insecurely is rejected + +Verify that a backup fails if the file where the passphrase is stored +is readable by anyone but its owner. Verify that the error message +explains that the backup failed due to the passphrase file insecurity. + +## The passphrase can be changed + +Verify that the passphrase can be changed and that backups made before +the change can no longer be restored. (Later, this requirement will be +re-evaluated, but this is simple and gets us started.) + +## The passphrase is not on server in cleartext + +Verify that after the passphrase has been set, and a backup has been +made, the passphrase is not stored in cleartext on the server. + +## A backup is encrypted + +Verify that the backup repository does not contain the backed up data +in cleartext. diff --git a/obnam.subplot b/obnam.subplot new file mode 100644 index 0000000..10a0ad7 --- /dev/null +++ b/obnam.subplot @@ -0,0 +1,23 @@ +title: "Obnam2—a backup system" +authors: + - Lars Wirzenius +documentclass: report +markdowns: + - obnam.md +bindings: + - subplot/server.yaml + - subplot/client.yaml + - subplot/data.yaml + - lib/files.yaml + - lib/runcmd.yaml +impls: + python: + - subplot/server.py + - subplot/client.py + - subplot/data.py + - lib/daemon.py + - lib/files.py + - lib/runcmd.py +classes: + - json + - sql diff --git a/rustfmt.toml b/rustfmt.toml new file mode 100644 index 0000000..32a9786 --- /dev/null +++ b/rustfmt.toml @@ -0,0 +1 @@ +edition = "2018" diff --git a/src/accumulated_time.rs b/src/accumulated_time.rs new file mode 100644 index 0000000..cdf34b2 --- /dev/null +++ b/src/accumulated_time.rs @@ -0,0 +1,79 @@ +//! Measure accumulated time for various operations. + +use std::collections::HashMap; +use std::hash::Hash; +use std::sync::Mutex; +use std::time::Instant; + +/// Accumulated times for different clocks. +/// +/// The caller defines a clock type, usually an enum. +/// `AccumulatedTime` accumulates time for each possible clock. +/// Conceptually, every type of clock exists. If a type of clock +/// doesn't ever get created, it measures at 0 accumulated time. +#[derive(Debug)] +pub struct AccumulatedTime<T> { + accumulated: Mutex<HashMap<T, ClockTime>>, +} + +#[derive(Debug, Default)] +struct ClockTime { + nanos: u128, + started: Option<Instant>, +} + +impl<T: Eq + PartialEq + Hash + Copy> AccumulatedTime<T> { + /// Create a new accumulated time collector. + #[allow(clippy::new_without_default)] + pub fn new() -> Self { + Self { + accumulated: Mutex::new(HashMap::new()), + } + } + + /// Start a new clock of a given type to measure a span of time. + /// + /// The clock's measured time is added to the accumulator when the + /// clock is stopped. + pub fn start(&mut self, clock: T) { + let mut map = self.accumulated.lock().unwrap(); + let ct = map.entry(clock).or_default(); + assert!(ct.started.is_none()); + ct.started = Some(Instant::now()); + } + + /// Stop a running clock. + /// + /// Its run time is added to the accumulated time for that kind of clock. + pub fn stop(&mut self, clock: T) { + let mut map = self.accumulated.lock().unwrap(); + if let Some(ct) = map.get_mut(&clock) { + assert!(ct.started.is_some()); + if let Some(started) = ct.started.take() { + ct.nanos += started.elapsed().as_nanos(); + ct.started = None; + } + } + } + + /// Return the accumulated time for a type of clock, as whole seconds. + pub fn secs(&self, clock: T) -> u128 { + self.nanos(clock) / 1_000_000_000u128 + } + + /// Return the accumulated time for a type of clock, as nanoseconds. + /// + /// This includes the time spent in a currently running clock. + pub fn nanos(&self, clock: T) -> u128 { + let map = self.accumulated.lock().unwrap(); + if let Some(ct) = map.get(&clock) { + if let Some(started) = ct.started { + ct.nanos + started.elapsed().as_nanos() + } else { + ct.nanos + } + } else { + 0 + } + } +} diff --git a/src/backup_progress.rs b/src/backup_progress.rs index 6c1d3e6..e3995f0 100644 --- a/src/backup_progress.rs +++ b/src/backup_progress.rs @@ -1,44 +1,133 @@ +//! Progress bars for Obnam. + +use crate::generation::GenId; use indicatif::{ProgressBar, ProgressStyle}; -use std::path::Path; +use std::{path::Path, time::Duration}; + +const SHOW_PROGRESS: bool = true; +/// A progress bar abstraction specific to backups. +/// +/// The progress bar is different for initial and incremental backups, +/// and for different phases of making a backup. pub struct BackupProgress { progress: ProgressBar, } impl BackupProgress { - pub fn new() -> Self { - let progress = if true { + /// Create a progress bar for an initial backup. + pub fn initial() -> Self { + let progress = if SHOW_PROGRESS { ProgressBar::new(0) } else { ProgressBar::hidden() }; - let parts = vec![ + let parts = [ + "initial backup", + "elapsed: {elapsed}", + "files: {pos}", + "current: {wide_msg}", + "{spinner}", + ]; + progress.set_style( + ProgressStyle::default_bar() + .template(&parts.join("\n")) + .expect("create indicatif ProgressStyle value"), + ); + progress.enable_steady_tick(Duration::from_millis(100)); + + Self { progress } + } + + /// Create a progress bar for an incremental backup. + pub fn incremental() -> Self { + let progress = if SHOW_PROGRESS { + ProgressBar::new(0) + } else { + ProgressBar::hidden() + }; + let parts = [ + "incremental backup", "{wide_bar}", "elapsed: {elapsed}", "files: {pos}/{len}", "current: {wide_msg}", "{spinner}", ]; - progress.set_style(ProgressStyle::default_bar().template(&parts.join("\n"))); - progress.enable_steady_tick(100); + progress.set_style( + ProgressStyle::default_bar() + .template(&parts.join("\n")) + .expect("create indicatif ProgressStyle value"), + ); + progress.enable_steady_tick(Duration::from_millis(100)); + + Self { progress } + } + + /// Create a progress bar for uploading a new generation's metadata. + pub fn upload_generation() -> Self { + let progress = ProgressBar::new(0); + let parts = [ + "uploading new generation metadata", + "elapsed: {elapsed}", + "{spinner}", + ]; + progress.set_style( + ProgressStyle::default_bar() + .template(&parts.join("\n")) + .expect("create indicatif ProgressStyle value"), + ); + progress.enable_steady_tick(Duration::from_millis(100)); + + Self { progress } + } + + /// Create a progress bar for downloading an existing generation's + /// metadata. + pub fn download_generation(gen_id: &GenId) -> Self { + let progress = ProgressBar::new(0); + let parts = ["{msg}", "elapsed: {elapsed}", "{spinner}"]; + progress.set_style( + ProgressStyle::default_bar() + .template(&parts.join("\n")) + .expect("create indicatif ProgressStyle value"), + ); + progress.enable_steady_tick(Duration::from_millis(100)); + progress.set_message(format!( + "downloading previous generation metadata: {}", + gen_id + )); Self { progress } } + /// Set the number of files that were in the previous generation. + /// + /// The new generation usually has about the same number of files, + /// so the progress bar can show progress for incremental backups + /// without having to count all the files that actually exist first. pub fn files_in_previous_generation(&self, count: u64) { self.progress.set_length(count); } + /// Update progress bar about number of problems found during a backup. pub fn found_problem(&self) { self.progress.inc(1); } + /// Update progress bar about number of actual files found. pub fn found_live_file(&self, filename: &Path) { self.progress.inc(1); - self.progress - .set_message(&format!("{}", filename.display())); + if self.progress.length() < Some(self.progress.position()) { + self.progress.set_length(self.progress.position()); + } + self.progress.set_message(format!("{}", filename.display())); } + /// Tell progress bar it's finished. + /// + /// This will remove all traces of the progress bar from the + /// screen. pub fn finish(&self) { self.progress.set_length(self.progress.position()); self.progress.finish_and_clear(); diff --git a/src/backup_reason.rs b/src/backup_reason.rs index 218857c..9a17d80 100644 --- a/src/backup_reason.rs +++ b/src/backup_reason.rs @@ -1,29 +1,54 @@ +//! Why was a file backed up? + use rusqlite::types::ToSqlOutput; use rusqlite::ToSql; use std::fmt; +/// Represent the reason a file is in a backup. #[derive(Debug, Copy, Clone)] pub enum Reason { + /// File was skipped due to policy, but carried over without + /// changes. Skipped, + /// File is new, compared to previous backup. IsNew, + /// File has been changed, compared to previous backup, Changed, + /// File has not been changed, compared to previous backup, Unchanged, - Error, + /// There was an error looking up the file in the previous backup. + /// + /// File has been carried over without changes. + GenerationLookupError, + /// The was an error backing up the file. + /// + /// File has been carried over without changes. + FileError, + /// Reason is unknown. + /// + /// The previous backup had a reason that the current version of + /// Obnam doesn't recognize. The file has been carried over + /// without changes. + Unknown, } impl Reason { - pub fn from_str(text: &str) -> Reason { + /// Create a Reason from a string representation. + pub fn from(text: &str) -> Reason { match text { "skipped" => Reason::Skipped, "new" => Reason::IsNew, "changed" => Reason::Changed, "unchanged" => Reason::Unchanged, - _ => Reason::Error, + "genlookuperror" => Reason::GenerationLookupError, + "fileerror" => Reason::FileError, + _ => Reason::Unknown, } } } impl ToSql for Reason { + /// Represent Reason as text for SQL. fn to_sql(&self) -> rusqlite::Result<ToSqlOutput> { Ok(ToSqlOutput::Owned(rusqlite::types::Value::Text(format!( "{}", @@ -33,13 +58,16 @@ impl ToSql for Reason { } impl fmt::Display for Reason { + /// Represent Reason for display. fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { let reason = match self { Reason::Skipped => "skipped", Reason::IsNew => "new", Reason::Changed => "changed", Reason::Unchanged => "unchanged", - Reason::Error => "error", + Reason::GenerationLookupError => "genlookuperror", + Reason::FileError => "fileerror", + Reason::Unknown => "unknown", }; write!(f, "{}", reason) } diff --git a/src/backup_run.rs b/src/backup_run.rs index e3bfc5a..372ef65 100644 --- a/src/backup_run.rs +++ b/src/backup_run.rs @@ -1,92 +1,437 @@ +//! Run one backup. + use crate::backup_progress::BackupProgress; use crate::backup_reason::Reason; +use crate::chunk::{GenerationChunk, GenerationChunkError}; +use crate::chunker::{ChunkerError, FileChunks}; use crate::chunkid::ChunkId; -use crate::client::{BackupClient, ClientConfig}; -use crate::fsentry::FilesystemEntry; -use crate::generation::LocalGeneration; +use crate::client::{BackupClient, ClientError}; +use crate::config::ClientConfig; +use crate::db::DatabaseError; +use crate::dbgen::{schema_version, FileId, DEFAULT_SCHEMA_MAJOR}; +use crate::error::ObnamError; +use crate::fsentry::{FilesystemEntry, FilesystemKind}; +use crate::fsiter::{AnnotatedFsEntry, FsIterError, FsIterator}; +use crate::generation::{ + GenId, LocalGeneration, LocalGenerationError, NascentError, NascentGeneration, +}; +use crate::label::LabelChecksumKind; +use crate::performance::{Clock, Performance}; use crate::policy::BackupPolicy; -use log::{info, warn}; +use crate::schema::SchemaVersion; + +use bytesize::MIB; +use chrono::{DateTime, Local}; +use log::{debug, error, info, warn}; +use std::path::{Path, PathBuf}; + +const DEFAULT_CHECKSUM_KIND: LabelChecksumKind = LabelChecksumKind::Sha256; +const SQLITE_CHUNK_SIZE: usize = MIB as usize; -pub struct BackupRun { - client: BackupClient, +/// A running backup. +pub struct BackupRun<'a> { + checksum_kind: Option<LabelChecksumKind>, + client: &'a mut BackupClient, policy: BackupPolicy, buffer_size: usize, - progress: BackupProgress, + progress: Option<BackupProgress>, +} + +/// Possible errors that can occur during a backup. +#[derive(Debug, thiserror::Error)] +pub enum BackupError { + /// An error from communicating with the server. + #[error(transparent)] + ClientError(#[from] ClientError), + + /// An error iterating over a directory tree. + #[error(transparent)] + FsIterError(#[from] FsIterError), + + /// An error from creating a new backup's metadata. + #[error(transparent)] + NascentError(#[from] NascentError), + + /// An error using an existing backup's metadata. + #[error(transparent)] + LocalGenerationError(#[from] LocalGenerationError), + + /// An error using a Database. + #[error(transparent)] + Database(#[from] DatabaseError), + + /// An error splitting data into chunks. + #[error(transparent)] + ChunkerError(#[from] ChunkerError), + + /// A error splitting backup metadata into chunks. + #[error(transparent)] + GenerationChunkError(#[from] GenerationChunkError), +} + +/// The outcome of backing up a file system entry. +#[derive(Debug)] +pub struct FsEntryBackupOutcome { + /// The file system entry. + pub entry: FilesystemEntry, + /// The chunk identifiers for the file's content. + pub ids: Vec<ChunkId>, + /// Why this entry is added to the new backup. + pub reason: Reason, + /// Does this entry represent a cache directory? + pub is_cachedir_tag: bool, +} + +/// The outcome of backing up a backup root. +#[derive(Debug)] +struct OneRootBackupOutcome { + /// Any warnings (non-fatal errors) from backing up the backup root. + pub warnings: Vec<BackupError>, + /// New cache directories in this root. + pub new_cachedir_tags: Vec<PathBuf>, } -impl BackupRun { - pub fn new(config: &ClientConfig, buffer_size: usize) -> anyhow::Result<Self> { - let client = BackupClient::new(&config.server_url)?; - let policy = BackupPolicy::new(); - let progress = BackupProgress::new(); +/// The outcome of a backup run. +#[derive(Debug)] +pub struct RootsBackupOutcome { + /// The number of backed up files. + pub files_count: FileId, + /// The errors encountered while backing up files. + pub warnings: Vec<BackupError>, + /// CACHEDIR.TAG files that aren't present in in a previous generation. + pub new_cachedir_tags: Vec<PathBuf>, + /// Id of new generation. + pub gen_id: GenId, +} + +impl<'a> BackupRun<'a> { + /// Create a new run for an initial backup. + pub fn initial( + config: &ClientConfig, + client: &'a mut BackupClient, + ) -> Result<Self, BackupError> { Ok(Self { + checksum_kind: Some(DEFAULT_CHECKSUM_KIND), client, - policy, - buffer_size, - progress, + policy: BackupPolicy::default(), + buffer_size: config.chunk_size, + progress: Some(BackupProgress::initial()), }) } - pub fn client(&self) -> &BackupClient { - &self.client + /// Create a new run for an incremental backup. + pub fn incremental( + config: &ClientConfig, + client: &'a mut BackupClient, + ) -> Result<Self, BackupError> { + Ok(Self { + checksum_kind: None, + client, + policy: BackupPolicy::default(), + buffer_size: config.chunk_size, + progress: None, + }) } - pub fn progress(&self) -> &BackupProgress { - &self.progress - } + /// Start the backup run. + pub async fn start( + &mut self, + genid: Option<&GenId>, + oldname: &Path, + perf: &mut Performance, + ) -> Result<LocalGeneration, ObnamError> { + match genid { + None => { + // Create a new, empty generation. + let schema = schema_version(DEFAULT_SCHEMA_MAJOR).unwrap(); + NascentGeneration::create(oldname, schema, self.checksum_kind.unwrap())?.close()?; - pub fn backup_file_initially( - &self, - entry: anyhow::Result<FilesystemEntry>, - ) -> anyhow::Result<(FilesystemEntry, Vec<ChunkId>, Reason)> { - match entry { - Err(err) => Err(err.into()), - Ok(entry) => { - let path = &entry.pathbuf(); - info!("backup: {}", path.display()); - self.progress.found_live_file(path); - let ids = self - .client - .upload_filesystem_entry(&entry, self.buffer_size)?; - Ok((entry.clone(), ids, Reason::IsNew)) + // Open the newly created empty generation. + Ok(LocalGeneration::open(oldname)?) + } + Some(genid) => { + perf.start(Clock::GenerationDownload); + let old = self.fetch_previous_generation(genid, oldname).await?; + perf.stop(Clock::GenerationDownload); + + let meta = old.meta()?; + if let Some(v) = meta.get("checksum_kind") { + self.checksum_kind = Some(LabelChecksumKind::from(v)?); + } + + let progress = BackupProgress::incremental(); + progress.files_in_previous_generation(old.file_count()? as u64); + self.progress = Some(progress); + + Ok(old) } } } - pub fn backup_file_incrementally( + fn checksum_kind(&self) -> LabelChecksumKind { + self.checksum_kind.unwrap_or(LabelChecksumKind::Sha256) + } + + async fn fetch_previous_generation( &self, - entry: anyhow::Result<FilesystemEntry>, + genid: &GenId, + oldname: &Path, + ) -> Result<LocalGeneration, ObnamError> { + let progress = BackupProgress::download_generation(genid); + let old = self.client.fetch_generation(genid, oldname).await?; + progress.finish(); + Ok(old) + } + + /// Finish this backup run. + pub fn finish(&self) { + if let Some(progress) = &self.progress { + progress.finish(); + } + } + + /// Back up all the roots for this run. + pub async fn backup_roots( + &mut self, + config: &ClientConfig, old: &LocalGeneration, - ) -> anyhow::Result<(FilesystemEntry, Vec<ChunkId>, Reason)> { - match entry { - Err(err) => { - warn!("backup: {}", err); - self.progress.found_problem(); - Err(err) + newpath: &Path, + schema: SchemaVersion, + perf: &mut Performance, + ) -> Result<RootsBackupOutcome, ObnamError> { + let mut warnings: Vec<BackupError> = vec![]; + let mut new_cachedir_tags = vec![]; + let files_count = { + let mut new = NascentGeneration::create(newpath, schema, self.checksum_kind.unwrap())?; + for root in &config.roots { + match self.backup_one_root(config, old, &mut new, root).await { + Ok(mut o) => { + new_cachedir_tags.append(&mut o.new_cachedir_tags); + if !o.warnings.is_empty() { + for err in o.warnings.iter() { + debug!("ignoring backup error {}", err); + self.found_problem(); + } + warnings.append(&mut o.warnings); + } + } + Err(err) => { + self.found_problem(); + return Err(err.into()); + } + } } - Ok(entry) => { - let path = &entry.pathbuf(); - info!("backup: {}", path.display()); - self.progress.found_live_file(path); - let reason = self.policy.needs_backup(&old, &entry); - match reason { - Reason::IsNew | Reason::Changed | Reason::Error => { - let ids = self - .client - .upload_filesystem_entry(&entry, self.buffer_size)?; - Ok((entry.clone(), ids, reason)) + let count = new.file_count(); + new.close()?; + count + }; + self.finish(); + perf.start(Clock::GenerationUpload); + let gen_id = self.upload_nascent_generation(newpath).await?; + perf.stop(Clock::GenerationUpload); + let gen_id = GenId::from_chunk_id(gen_id); + Ok(RootsBackupOutcome { + files_count, + warnings, + new_cachedir_tags, + gen_id, + }) + } + + async fn backup_one_root( + &mut self, + config: &ClientConfig, + old: &LocalGeneration, + new: &mut NascentGeneration, + root: &Path, + ) -> Result<OneRootBackupOutcome, NascentError> { + let mut warnings: Vec<BackupError> = vec![]; + let mut new_cachedir_tags = vec![]; + let iter = FsIterator::new(root, config.exclude_cache_tag_directories); + let mut first_entry = true; + for entry in iter { + match entry { + Err(err) => { + if first_entry { + // Only the first entry (the backup root) + // failing is an error. Everything else is a + // warning. + return Err(NascentError::BackupRootFailed(root.to_path_buf(), err)); + } + warnings.push(err.into()); + } + Ok(entry) => { + let path = entry.inner.pathbuf(); + if entry.is_cachedir_tag && !old.is_cachedir_tag(&path)? { + new_cachedir_tags.push(path); } - Reason::Unchanged | Reason::Skipped => { - let fileno = old.get_fileno(&entry.pathbuf())?; - let ids = if let Some(fileno) = fileno { - old.chunkids(fileno)? - } else { - vec![] - }; - Ok((entry.clone(), ids, reason)) + match self.backup_if_needed(entry, old).await { + Err(err) => { + warnings.push(err); + } + Ok(None) => (), + Ok(Some(o)) => { + if let Err(err) = + new.insert(o.entry, &o.ids, o.reason, o.is_cachedir_tag) + { + warnings.push(err.into()); + } + } } } } + first_entry = false; + } + + Ok(OneRootBackupOutcome { + warnings, + new_cachedir_tags, + }) + } + + async fn backup_if_needed( + &mut self, + entry: AnnotatedFsEntry, + old: &LocalGeneration, + ) -> Result<Option<FsEntryBackupOutcome>, BackupError> { + let path = &entry.inner.pathbuf(); + info!("backup: {}", path.display()); + self.found_live_file(path); + let reason = self.policy.needs_backup(old, &entry.inner); + match reason { + Reason::IsNew | Reason::Changed | Reason::GenerationLookupError | Reason::Unknown => { + Ok(Some(self.backup_one_entry(&entry, path, reason).await)) + } + Reason::Skipped => Ok(None), + Reason::Unchanged | Reason::FileError => { + let fileno = old.get_fileno(&entry.inner.pathbuf())?; + let ids = if let Some(fileno) = fileno { + let mut ids = vec![]; + for id in old.chunkids(fileno)?.iter()? { + ids.push(id?); + } + ids + } else { + vec![] + }; + Ok(Some(FsEntryBackupOutcome { + entry: entry.inner, + ids, + reason, + is_cachedir_tag: entry.is_cachedir_tag, + })) + } + } + } + + async fn backup_one_entry( + &mut self, + entry: &AnnotatedFsEntry, + path: &Path, + reason: Reason, + ) -> FsEntryBackupOutcome { + let ids = self + .upload_filesystem_entry(&entry.inner, self.buffer_size) + .await; + match ids { + Err(err) => { + warn!("error backing up {}, skipping it: {}", path.display(), err); + FsEntryBackupOutcome { + entry: entry.inner.clone(), + ids: vec![], + reason: Reason::FileError, + is_cachedir_tag: entry.is_cachedir_tag, + } + } + Ok(ids) => FsEntryBackupOutcome { + entry: entry.inner.clone(), + ids, + reason, + is_cachedir_tag: entry.is_cachedir_tag, + }, + } + } + + /// Upload any file content for a file system entry. + pub async fn upload_filesystem_entry( + &mut self, + e: &FilesystemEntry, + size: usize, + ) -> Result<Vec<ChunkId>, BackupError> { + let path = e.pathbuf(); + info!("uploading {:?}", path); + let ids = match e.kind() { + FilesystemKind::Regular => self.upload_regular_file(&path, size).await?, + FilesystemKind::Directory => vec![], + FilesystemKind::Symlink => vec![], + FilesystemKind::Socket => vec![], + FilesystemKind::Fifo => vec![], + }; + info!("upload OK for {:?}", path); + Ok(ids) + } + + /// Upload the metadata for the backup of this run. + pub async fn upload_generation( + &mut self, + filename: &Path, + size: usize, + ) -> Result<ChunkId, BackupError> { + info!("upload SQLite {}", filename.display()); + let ids = self.upload_regular_file(filename, size).await?; + let gen = GenerationChunk::new(ids); + let data = gen.to_data_chunk()?; + let gen_id = self.client.upload_chunk(data).await?; + info!("uploaded generation {}", gen_id); + Ok(gen_id) + } + + async fn upload_regular_file( + &mut self, + filename: &Path, + size: usize, + ) -> Result<Vec<ChunkId>, BackupError> { + info!("upload file {}", filename.display()); + let mut chunk_ids = vec![]; + let file = std::fs::File::open(filename) + .map_err(|err| ClientError::FileOpen(filename.to_path_buf(), err))?; + let chunker = FileChunks::new(size, file, filename, self.checksum_kind()); + for item in chunker { + let chunk = item?; + if let Some(chunk_id) = self.client.has_chunk(chunk.meta()).await? { + chunk_ids.push(chunk_id.clone()); + info!("reusing existing chunk {}", chunk_id); + } else { + let chunk_id = self.client.upload_chunk(chunk).await?; + chunk_ids.push(chunk_id.clone()); + info!("created new chunk {}", chunk_id); + } } + Ok(chunk_ids) } + + async fn upload_nascent_generation(&mut self, filename: &Path) -> Result<ChunkId, ObnamError> { + let progress = BackupProgress::upload_generation(); + let gen_id = self.upload_generation(filename, SQLITE_CHUNK_SIZE).await?; + progress.finish(); + Ok(gen_id) + } + + fn found_live_file(&self, path: &Path) { + if let Some(progress) = &self.progress { + progress.found_live_file(path); + } + } + + fn found_problem(&self) { + if let Some(progress) = &self.progress { + progress.found_problem(); + } + } +} + +/// Current timestamp as an ISO 8601 string. +pub fn current_timestamp() -> String { + let now: DateTime<Local> = Local::now(); + format!("{}", now.format("%Y-%m-%d %H:%M:%S.%f %z")) } diff --git a/src/benchmark.rs b/src/benchmark.rs deleted file mode 100644 index b313868..0000000 --- a/src/benchmark.rs +++ /dev/null @@ -1,32 +0,0 @@ -use crate::chunk::DataChunk; -use crate::chunkid::ChunkId; -use crate::chunkmeta::ChunkMeta; - -// Generate a desired number of empty data chunks with id and metadata. -pub struct ChunkGenerator { - goal: u32, - next: u32, -} - -impl ChunkGenerator { - pub fn new(goal: u32) -> Self { - Self { goal, next: 0 } - } -} - -impl Iterator for ChunkGenerator { - type Item = (ChunkId, String, ChunkMeta, DataChunk); - - fn next(&mut self) -> Option<Self::Item> { - if self.next >= self.goal { - None - } else { - let id = ChunkId::new(); - let checksum = id.sha256(); - let meta = ChunkMeta::new(&checksum); - let chunk = DataChunk::new(vec![]); - self.next += 1; - Some((id, checksum, meta, chunk)) - } - } -} diff --git a/src/bin/benchmark-index.rs b/src/bin/benchmark-index.rs deleted file mode 100644 index d49a6c3..0000000 --- a/src/bin/benchmark-index.rs +++ /dev/null @@ -1,35 +0,0 @@ -use obnam::benchmark::ChunkGenerator; -use obnam::chunkmeta::ChunkMeta; -use obnam::index::Index; -use std::path::PathBuf; -use structopt::StructOpt; - -#[derive(Debug, StructOpt)] -#[structopt( - name = "benchmark-index", - about = "Benhcmark the store index in memory" -)] -struct Opt { - // We don't use this, but we accept it for command line - // compatibility with other benchmark programs. - #[structopt(parse(from_os_str))] - chunks: PathBuf, - - #[structopt()] - num: u32, -} - -fn main() -> anyhow::Result<()> { - pretty_env_logger::init(); - - let opt = Opt::from_args(); - let gen = ChunkGenerator::new(opt.num); - - let mut index = Index::new(".")?; - for (id, checksum, _, _) in gen { - let meta = ChunkMeta::new(&checksum); - index.insert_meta(id, meta)?; - } - - Ok(()) -} diff --git a/src/bin/benchmark-indexedstore.rs b/src/bin/benchmark-indexedstore.rs deleted file mode 100644 index 3ee4c38..0000000 --- a/src/bin/benchmark-indexedstore.rs +++ /dev/null @@ -1,28 +0,0 @@ -use obnam::benchmark::ChunkGenerator; -use obnam::indexedstore::IndexedStore; -use std::path::PathBuf; -use structopt::StructOpt; - -#[derive(Debug, StructOpt)] -#[structopt(name = "benchmark-store", about = "Benhcmark the store without HTTP")] -struct Opt { - #[structopt(parse(from_os_str))] - chunks: PathBuf, - - #[structopt()] - num: u32, -} - -fn main() -> anyhow::Result<()> { - pretty_env_logger::init(); - - let opt = Opt::from_args(); - let gen = ChunkGenerator::new(opt.num); - - let mut store = IndexedStore::new(&opt.chunks)?; - for (_, _, meta, chunk) in gen { - store.save(&meta, &chunk)?; - } - - Ok(()) -} diff --git a/src/bin/benchmark-null.rs b/src/bin/benchmark-null.rs deleted file mode 100644 index 6df8ca1..0000000 --- a/src/bin/benchmark-null.rs +++ /dev/null @@ -1,29 +0,0 @@ -use obnam::benchmark::ChunkGenerator; -use std::path::PathBuf; -use structopt::StructOpt; - -#[derive(Debug, StructOpt)] -#[structopt( - name = "benchmark-index", - about = "Benhcmark the store index in memory" -)] -struct Opt { - // We don't use this, but we accept it for command line - // compatibility with other benchmark programs. - #[structopt(parse(from_os_str))] - chunks: PathBuf, - - #[structopt()] - num: u32, -} - -fn main() -> anyhow::Result<()> { - pretty_env_logger::init(); - - let opt = Opt::from_args(); - let gen = ChunkGenerator::new(opt.num); - - for (_, _, _, _) in gen {} - - Ok(()) -} diff --git a/src/bin/benchmark-store.rs b/src/bin/benchmark-store.rs deleted file mode 100644 index f7c82b1..0000000 --- a/src/bin/benchmark-store.rs +++ /dev/null @@ -1,28 +0,0 @@ -use obnam::benchmark::ChunkGenerator; -use obnam::store::Store; -use std::path::PathBuf; -use structopt::StructOpt; - -#[derive(Debug, StructOpt)] -#[structopt(name = "benchmark-store", about = "Benhcmark the store without HTTP")] -struct Opt { - #[structopt(parse(from_os_str))] - chunks: PathBuf, - - #[structopt()] - num: u32, -} - -fn main() -> anyhow::Result<()> { - pretty_env_logger::init(); - - let opt = Opt::from_args(); - let gen = ChunkGenerator::new(opt.num); - - let store = Store::new(&opt.chunks); - for (id, _, meta, chunk) in gen { - store.save(&id, &meta, &chunk)?; - } - - Ok(()) -} diff --git a/src/bin/obnam-server.rs b/src/bin/obnam-server.rs index 19f2e99..9b5a557 100644 --- a/src/bin/obnam-server.rs +++ b/src/bin/obnam-server.rs @@ -1,42 +1,43 @@ -use bytes::Bytes; +use anyhow::Context; +use clap::Parser; use log::{debug, error, info}; -use obnam::chunk::DataChunk; use obnam::chunkid::ChunkId; use obnam::chunkmeta::ChunkMeta; -use obnam::indexedstore::IndexedStore; -use serde::{Deserialize, Serialize}; +use obnam::chunkstore::ChunkStore; +use obnam::label::Label; +use obnam::server::{ServerConfig, ServerConfigError}; +use serde::Serialize; use std::collections::HashMap; use std::default::Default; use std::net::{SocketAddr, ToSocketAddrs}; use std::path::{Path, PathBuf}; use std::sync::Arc; -use structopt::StructOpt; use tokio::sync::Mutex; use warp::http::StatusCode; +use warp::hyper::body::Bytes; use warp::Filter; -#[derive(Debug, StructOpt)] -#[structopt(name = "obnam2-server", about = "Backup server")] +#[derive(Debug, Parser)] +#[clap(name = "obnam2-server", about = "Backup server")] struct Opt { - #[structopt(parse(from_os_str))] config: PathBuf, } #[tokio::main] async fn main() -> anyhow::Result<()> { - pretty_env_logger::init(); + pretty_env_logger::init_custom_env("OBNAM_SERVER_LOG"); - let opt = Opt::from_args(); - let config = Config::read_config(&opt.config).unwrap(); + let opt = Opt::parse(); + let config = load_config(&opt.config)?; let addresses: Vec<SocketAddr> = config.address.to_socket_addrs()?.collect(); if addresses.is_empty() { error!("specified address is empty set: {:?}", addresses); eprintln!("ERROR: server address is empty: {:?}", addresses); - return Err(ConfigError::BadServerAddress.into()); + return Err(ServerConfigError::BadServerAddress.into()); } - let store = IndexedStore::new(&config.chunks)?; + let store = ChunkStore::local(&config.chunks)?; let store = Arc::new(Mutex::new(store)); let store = warp::any().map(move || Arc::clone(&store)); @@ -45,32 +46,32 @@ async fn main() -> anyhow::Result<()> { debug!("Configuration: {:#?}", config); let create = warp::post() + .and(warp::path("v1")) .and(warp::path("chunks")) + .and(warp::path::end()) .and(store.clone()) .and(warp::header("chunk-meta")) .and(warp::filters::body::bytes()) .and_then(create_chunk); let fetch = warp::get() + .and(warp::path("v1")) .and(warp::path("chunks")) .and(warp::path::param()) + .and(warp::path::end()) .and(store.clone()) .and_then(fetch_chunk); let search = warp::get() + .and(warp::path("v1")) .and(warp::path("chunks")) + .and(warp::path::end()) .and(warp::query::<HashMap<String, String>>()) .and(store.clone()) .and_then(search_chunks); - let delete = warp::delete() - .and(warp::path("chunks")) - .and(warp::path::param()) - .and(store.clone()) - .and_then(delete_chunk); - let log = warp::log("obnam"); - let webroot = create.or(fetch).or(search).or(delete).with(log); + let webroot = create.or(fetch).or(search).with(log); debug!("starting warp"); warp::serve(webroot) @@ -82,57 +83,22 @@ async fn main() -> anyhow::Result<()> { Ok(()) } -#[derive(Debug, Deserialize, Clone)] -pub struct Config { - pub chunks: PathBuf, - pub address: String, - pub tls_key: PathBuf, - pub tls_cert: PathBuf, -} - -#[derive(Debug, thiserror::Error)] -enum ConfigError { - #[error("Directory for chunks {0} does not exist")] - ChunksDirNotFound(PathBuf), - - #[error("TLS certificate {0} does not exist")] - TlsCertNotFound(PathBuf), - - #[error("TLS key {0} does not exist")] - TlsKeyNotFound(PathBuf), - - #[error("server address can't be resolved")] - BadServerAddress, -} - -impl Config { - pub fn read_config(filename: &Path) -> anyhow::Result<Config> { - let config = std::fs::read_to_string(filename)?; - let config: Config = serde_yaml::from_str(&config)?; - config.check()?; - Ok(config) - } - - pub fn check(&self) -> anyhow::Result<()> { - if !self.chunks.exists() { - return Err(ConfigError::ChunksDirNotFound(self.chunks.clone()).into()); - } - if !self.tls_cert.exists() { - return Err(ConfigError::TlsCertNotFound(self.tls_cert.clone()).into()); - } - if !self.tls_key.exists() { - return Err(ConfigError::TlsKeyNotFound(self.tls_key.clone()).into()); - } - Ok(()) - } +fn load_config(filename: &Path) -> Result<ServerConfig, anyhow::Error> { + let config = ServerConfig::read_config(filename).with_context(|| { + format!( + "Couldn't read default configuration file {}", + filename.display() + ) + })?; + Ok(config) } pub async fn create_chunk( - store: Arc<Mutex<IndexedStore>>, + store: Arc<Mutex<ChunkStore>>, meta: String, data: Bytes, ) -> Result<impl warp::Reply, warp::Rejection> { - let mut store = store.lock().await; + let store = store.lock().await; let meta: ChunkMeta = match meta.parse() { Ok(s) => s, @@ -142,9 +108,7 @@ pub async fn create_chunk( } }; - let chunk = DataChunk::new(data.to_vec()); - - let id = match store.save(&meta, &chunk) { + let id = match store.put(data.to_vec(), &meta).await { Ok(id) => id, Err(e) => { error!("couldn't save: {}", e); @@ -152,17 +116,17 @@ pub async fn create_chunk( } }; - info!("created chunk {}: {:?}", id, meta); + info!("created chunk {}", id); Ok(ChunkResult::Created(id)) } pub async fn fetch_chunk( id: String, - store: Arc<Mutex<IndexedStore>>, + store: Arc<Mutex<ChunkStore>>, ) -> Result<impl warp::Reply, warp::Rejection> { let store = store.lock().await; let id: ChunkId = id.parse().unwrap(); - match store.load(&id) { + match store.get(&id).await { Ok((data, meta)) => { info!("found chunk {}: {:?}", id, meta); Ok(ChunkResult::Fetched(meta, data)) @@ -176,20 +140,23 @@ pub async fn fetch_chunk( pub async fn search_chunks( query: HashMap<String, String>, - store: Arc<Mutex<IndexedStore>>, + store: Arc<Mutex<ChunkStore>>, ) -> Result<impl warp::Reply, warp::Rejection> { let store = store.lock().await; let mut query = query.iter(); let found = if let Some((key, value)) = query.next() { - if query.next() != None { + if query.next().is_some() { error!("search has more than one key to search for"); return Ok(ChunkResult::BadRequest); } - if key == "generation" && value == "true" { - store.find_generations().expect("SQL lookup failed") - } else if key == "sha256" { - store.find_by_sha256(value).expect("SQL lookup failed") + if key == "label" { + let label = Label::deserialize(value).unwrap(); + let label = ChunkMeta::new(&label); + store + .find_by_label(&label) + .await + .expect("SQL lookup failed") } else { error!("unknown search key {:?}", key); return Ok(ChunkResult::BadRequest); @@ -201,7 +168,7 @@ pub async fn search_chunks( let mut hits = SearchHits::default(); for chunk_id in found { - let meta = match store.load_meta(&chunk_id) { + let (_, meta) = match store.get(&chunk_id).await { Ok(meta) => { info!("search found chunk {}", chunk_id); meta @@ -240,30 +207,10 @@ impl SearchHits { } } -pub async fn delete_chunk( - id: String, - store: Arc<Mutex<IndexedStore>>, -) -> Result<impl warp::Reply, warp::Rejection> { - let mut store = store.lock().await; - let id: ChunkId = id.parse().unwrap(); - - match store.remove(&id) { - Ok(_) => { - info!("chunk deleted: {}", id); - Ok(ChunkResult::Deleted) - } - Err(e) => { - error!("could not delete chunk {}: {:?}", id, e); - Ok(ChunkResult::NotFound) - } - } -} - enum ChunkResult { Created(ChunkId), - Fetched(ChunkMeta, DataChunk), + Fetched(ChunkMeta, Vec<u8>), Found(SearchHits), - Deleted, NotFound, BadRequest, InternalServerError, @@ -292,13 +239,12 @@ impl warp::Reply for ChunkResult { ); into_response( StatusCode::OK, - chunk.data(), + &chunk, "application/octet-stream", Some(headers), ) } ChunkResult::Found(hits) => json_response(StatusCode::OK, hits.to_json(), None), - ChunkResult::Deleted => status_response(StatusCode::OK), ChunkResult::BadRequest => status_response(StatusCode::BAD_REQUEST), ChunkResult::NotFound => status_response(StatusCode::NOT_FOUND), ChunkResult::InternalServerError => status_response(StatusCode::INTERNAL_SERVER_ERROR), diff --git a/src/bin/obnam.rs b/src/bin/obnam.rs index e9f30ca..240960b 100644 --- a/src/bin/obnam.rs +++ b/src/bin/obnam.rs @@ -1,100 +1,126 @@ +use clap::Parser; +use directories_next::ProjectDirs; use log::{debug, error, info, LevelFilter}; use log4rs::append::file::FileAppender; -use log4rs::config::{Appender, Config, Logger, Root}; -use obnam::client::ClientConfig; -use obnam::cmd::{backup, get_chunk, list, list_files, restore, show_generation}; +use log4rs::config::{Appender, Logger, Root}; +use obnam::cmd::backup::Backup; +use obnam::cmd::chunk::{DecryptChunk, EncryptChunk}; +use obnam::cmd::chunkify::Chunkify; +use obnam::cmd::gen_info::GenInfo; +use obnam::cmd::get_chunk::GetChunk; +use obnam::cmd::init::Init; +use obnam::cmd::inspect::Inspect; +use obnam::cmd::list::List; +use obnam::cmd::list_backup_versions::ListSchemaVersions; +use obnam::cmd::list_files::ListFiles; +use obnam::cmd::resolve::Resolve; +use obnam::cmd::restore::Restore; +use obnam::cmd::show_config::ShowConfig; +use obnam::cmd::show_gen::ShowGeneration; +use obnam::config::ClientConfig; +use obnam::performance::{Clock, Performance}; use std::path::{Path, PathBuf}; -use structopt::StructOpt; -const BUFFER_SIZE: usize = 1024 * 1024; +const QUALIFIER: &str = ""; +const ORG: &str = ""; +const APPLICATION: &str = "obnam"; -fn main() -> anyhow::Result<()> { - let opt = Opt::from_args(); - let config_file = match opt.config { - None => default_config(), - Some(ref path) => path.to_path_buf(), - }; - let config = ClientConfig::read_config(&config_file)?; - if let Some(ref log) = config.log { - setup_logging(&log)?; +fn main() { + let mut perf = Performance::default(); + perf.start(Clock::RunTime); + if let Err(err) = main_program(&mut perf) { + error!("{}", err); + eprintln!("ERROR: {}", err); + std::process::exit(1); } + perf.stop(Clock::RunTime); + perf.log(); +} + +fn main_program(perf: &mut Performance) -> anyhow::Result<()> { + let opt = Opt::parse(); + let config = ClientConfig::read(&config_filename(&opt))?; + setup_logging(&config.log)?; info!("client starts"); debug!("{:?}", opt); + debug!("configuration: {:#?}", config); - let result = match opt.cmd { - Command::Backup => backup(&config, BUFFER_SIZE), - Command::List => list(&config), - Command::ShowGeneration { gen_id } => show_generation(&config, &gen_id), - Command::ListFiles { gen_id } => list_files(&config, &gen_id), - Command::Restore { gen_id, to } => restore(&config, &gen_id, &to), - Command::GetChunk { chunk_id } => get_chunk(&config, &chunk_id), - }; - - if let Err(ref e) = result { - error!("{}", e); - eprintln!("ERROR: {}", e); - return result; - } + match opt.cmd { + Command::Init(x) => x.run(&config), + Command::ListBackupVersions(x) => x.run(&config), + Command::Backup(x) => x.run(&config, perf), + Command::Inspect(x) => x.run(&config), + Command::Chunkify(x) => x.run(&config), + Command::List(x) => x.run(&config), + Command::ShowGeneration(x) => x.run(&config), + Command::ListFiles(x) => x.run(&config), + Command::Resolve(x) => x.run(&config), + Command::Restore(x) => x.run(&config), + Command::GenInfo(x) => x.run(&config), + Command::GetChunk(x) => x.run(&config), + Command::Config(x) => x.run(&config), + Command::EncryptChunk(x) => x.run(&config), + Command::DecryptChunk(x) => x.run(&config), + }?; info!("client ends successfully"); Ok(()) } +fn setup_logging(filename: &Path) -> anyhow::Result<()> { + let logfile = FileAppender::builder().build(filename)?; + + let config = log4rs::Config::builder() + .appender(Appender::builder().build("obnam", Box::new(logfile))) + .logger(Logger::builder().build("obnam", LevelFilter::Debug)) + .build(Root::builder().appender("obnam").build(LevelFilter::Debug))?; + + log4rs::init_config(config)?; + + Ok(()) +} + +fn config_filename(opt: &Opt) -> PathBuf { + match opt.config { + None => default_config(), + Some(ref filename) => filename.to_path_buf(), + } +} + fn default_config() -> PathBuf { - if let Some(path) = dirs::config_dir() { - path.join("obnam").join("obnam.yaml") - } else if let Some(path) = dirs::home_dir() { - path.join(".config").join("obnam").join("obnam.yaml") + if let Some(dirs) = ProjectDirs::from(QUALIFIER, ORG, APPLICATION) { + dirs.config_dir().join("obnam.yaml") } else { - panic!("can't find config dir or home dir"); + panic!("can't figure out the configuration directory"); } } -#[derive(Debug, StructOpt)] -#[structopt(name = "obnam-backup", about = "Simplistic backup client")] +#[derive(Debug, Parser)] +#[clap(name = "obnam-backup", version, about = "Simplistic backup client")] struct Opt { - #[structopt(long, short, parse(from_os_str))] + #[clap(long, short)] config: Option<PathBuf>, - #[structopt(subcommand)] + #[clap(subcommand)] cmd: Command, } -#[derive(Debug, StructOpt)] +#[derive(Debug, Parser)] enum Command { - Backup, - List, - ListFiles { - #[structopt(default_value = "latest")] - gen_id: String, - }, - Restore { - #[structopt()] - gen_id: String, - - #[structopt(parse(from_os_str))] - to: PathBuf, - }, - ShowGeneration { - #[structopt(default_value = "latest")] - gen_id: String, - }, - GetChunk { - #[structopt()] - chunk_id: String, - }, -} - -fn setup_logging(filename: &Path) -> anyhow::Result<()> { - let logfile = FileAppender::builder().build(filename)?; - - let config = Config::builder() - .appender(Appender::builder().build("obnam", Box::new(logfile))) - .logger(Logger::builder().build("obnam", LevelFilter::Debug)) - .build(Root::builder().appender("obnam").build(LevelFilter::Debug))?; - - log4rs::init_config(config)?; - - Ok(()) + Init(Init), + Backup(Backup), + Inspect(Inspect), + Chunkify(Chunkify), + List(List), + ListBackupVersions(ListSchemaVersions), + ListFiles(ListFiles), + Restore(Restore), + GenInfo(GenInfo), + ShowGeneration(ShowGeneration), + Resolve(Resolve), + GetChunk(GetChunk), + Config(ShowConfig), + EncryptChunk(EncryptChunk), + DecryptChunk(DecryptChunk), } diff --git a/src/checksummer.rs b/src/checksummer.rs deleted file mode 100644 index 162c26b..0000000 --- a/src/checksummer.rs +++ /dev/null @@ -1,8 +0,0 @@ -use sha2::{Digest, Sha256}; - -pub fn sha256(data: &[u8]) -> String { - let mut hasher = Sha256::new(); - hasher.update(data); - let hash = hasher.finalize(); - format!("{:x}", hash) -} diff --git a/src/chunk.rs b/src/chunk.rs index 4917b60..a6abad3 100644 --- a/src/chunk.rs +++ b/src/chunk.rs @@ -1,60 +1,199 @@ +//! Chunks of data. + use crate::chunkid::ChunkId; +use crate::chunkmeta::ChunkMeta; +use crate::label::Label; use serde::{Deserialize, Serialize}; use std::default::Default; -/// Store an arbitrary chunk of data. -/// -/// The data is just arbitrary binary data. +/// An arbitrary chunk of arbitrary binary data. /// /// A chunk also contains its associated metadata, except its -/// identifier. -#[derive(Debug, Clone, Serialize, Deserialize)] +/// identifier, so that it's easy to keep the data and metadata +/// together. The identifier is used to find the chunk, and it's +/// assigned by the server when the chunk is uploaded, so it's not +/// stored in the chunk itself. +#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)] pub struct DataChunk { data: Vec<u8>, + meta: ChunkMeta, } impl DataChunk { - /// Construct a new chunk. - pub fn new(data: Vec<u8>) -> Self { - Self { data } + /// Create a new chunk. + pub fn new(data: Vec<u8>, meta: ChunkMeta) -> Self { + Self { data, meta } } /// Return a chunk's data. pub fn data(&self) -> &[u8] { &self.data } + + /// Return a chunk's metadata. + pub fn meta(&self) -> &ChunkMeta { + &self.meta + } } +/// A chunk representing a backup generation. +/// +/// A generation chunk lists all the data chunks for the SQLite file +/// with the backup's metadata. It's different from a normal data +/// chunk so that we can do things that make no sense to a data chunk. +/// Generation chunks can be converted into or created from data +/// chunks, for uploading to or downloading from the server. #[derive(Default, Debug, Serialize, Deserialize)] pub struct GenerationChunk { chunk_ids: Vec<ChunkId>, } +/// All the errors that may be returned for `GenerationChunk` operations. +#[derive(Debug, thiserror::Error)] +pub enum GenerationChunkError { + /// Error converting text from UTF8. + #[error(transparent)] + Utf8Error(#[from] std::str::Utf8Error), + + /// Error parsing JSON as chunk metadata. + #[error("failed to parse JSON: {0}")] + JsonParse(serde_json::Error), + + /// Error generating JSON from chunk metadata. + #[error("failed to serialize to JSON: {0}")] + JsonGenerate(serde_json::Error), +} + impl GenerationChunk { + /// Create a new backup generation chunk from metadata chunk ids. pub fn new(chunk_ids: Vec<ChunkId>) -> Self { Self { chunk_ids } } - pub fn from_data_chunk(chunk: &DataChunk) -> anyhow::Result<Self> { + /// Create a new backup generation chunk from a data chunk. + pub fn from_data_chunk(chunk: &DataChunk) -> Result<Self, GenerationChunkError> { let data = chunk.data(); let data = std::str::from_utf8(data)?; - Ok(serde_json::from_str(data)?) + serde_json::from_str(data).map_err(GenerationChunkError::JsonParse) } + /// Does the generation chunk contain any metadata chunks? pub fn is_empty(&self) -> bool { self.chunk_ids.is_empty() } + /// How many metadata chunks does generation chunk contain? pub fn len(&self) -> usize { self.chunk_ids.len() } + /// Return iterator over the metadata chunk identifiers. pub fn chunk_ids(&self) -> impl Iterator<Item = &ChunkId> { self.chunk_ids.iter() } - pub fn to_data_chunk(&self) -> anyhow::Result<DataChunk> { - let json = serde_json::to_string(self)?; - Ok(DataChunk::new(json.as_bytes().to_vec())) + /// Convert generation chunk to a data chunk. + pub fn to_data_chunk(&self) -> Result<DataChunk, GenerationChunkError> { + let json: String = + serde_json::to_string(self).map_err(GenerationChunkError::JsonGenerate)?; + let bytes = json.as_bytes().to_vec(); + let checksum = Label::sha256(&bytes); + let meta = ChunkMeta::new(&checksum); + Ok(DataChunk::new(bytes, meta)) + } +} + +/// A client trust root chunk. +/// +/// This chunk contains all per-client backup information. As long as +/// this chunk can be trusted, everything it links to can also be +/// trusted, thanks to cryptographic signatures. +#[derive(Debug, Serialize, Deserialize)] +pub struct ClientTrust { + client_name: String, + previous_version: Option<ChunkId>, + timestamp: String, + backups: Vec<ChunkId>, +} + +/// All the errors that may be returned for `ClientTrust` operations. +#[derive(Debug, thiserror::Error)] +pub enum ClientTrustError { + /// Error converting text from UTF8. + #[error(transparent)] + Utf8Error(#[from] std::str::Utf8Error), + + /// Error parsing JSON as chunk metadata. + #[error("failed to parse JSON: {0}")] + JsonParse(serde_json::Error), + + /// Error generating JSON from chunk metadata. + #[error("failed to serialize to JSON: {0}")] + JsonGenerate(serde_json::Error), +} + +impl ClientTrust { + /// Create a new ClientTrust object. + pub fn new( + name: &str, + previous_version: Option<ChunkId>, + timestamp: String, + backups: Vec<ChunkId>, + ) -> Self { + Self { + client_name: name.to_string(), + previous_version, + timestamp, + backups, + } + } + + /// Return client name. + pub fn client_name(&self) -> &str { + &self.client_name + } + + /// Return id of previous version, if any. + pub fn previous_version(&self) -> Option<ChunkId> { + self.previous_version.clone() + } + + /// Return timestamp. + pub fn timestamp(&self) -> &str { + &self.timestamp + } + + /// Return list of all backup generations known. + pub fn backups(&self) -> &[ChunkId] { + &self.backups + } + + /// Append a backup generation to the list. + pub fn append_backup(&mut self, id: &ChunkId) { + self.backups.push(id.clone()); + } + + /// Update for new upload. + /// + /// This needs to happen every time the chunk is updated so that + /// the timestamp gets updated. + pub fn finalize(&mut self, timestamp: String) { + self.timestamp = timestamp; + } + + /// Convert generation chunk to a data chunk. + pub fn to_data_chunk(&self) -> Result<DataChunk, ClientTrustError> { + let json: String = serde_json::to_string(self).map_err(ClientTrustError::JsonGenerate)?; + let bytes = json.as_bytes().to_vec(); + let checksum = Label::literal("client-trust"); + let meta = ChunkMeta::new(&checksum); + Ok(DataChunk::new(bytes, meta)) + } + + /// Create a new ClientTrust from a data chunk. + pub fn from_data_chunk(chunk: &DataChunk) -> Result<Self, ClientTrustError> { + let data = chunk.data(); + let data = std::str::from_utf8(data)?; + serde_json::from_str(data).map_err(ClientTrustError::JsonParse) } } diff --git a/src/chunker.rs b/src/chunker.rs index 145b1db..9883f89 100644 --- a/src/chunker.rs +++ b/src/chunker.rs @@ -1,30 +1,54 @@ -use crate::checksummer::sha256; +//! Split file data into chunks. + use crate::chunk::DataChunk; use crate::chunkmeta::ChunkMeta; +use crate::label::{Label, LabelChecksumKind}; use std::io::prelude::*; +use std::path::{Path, PathBuf}; -pub struct Chunker { +/// Iterator over chunks in a file. +pub struct FileChunks { chunk_size: usize, + kind: LabelChecksumKind, buf: Vec<u8>, + filename: PathBuf, handle: std::fs::File, } -impl Chunker { - pub fn new(chunk_size: usize, handle: std::fs::File) -> Self { - let mut buf = vec![]; - buf.resize(chunk_size, 0); +/// Possible errors from data chunking. +#[derive(Debug, thiserror::Error)] +pub enum ChunkerError { + /// Error reading from a file. + #[error("failed to read file {0}: {1}")] + FileRead(PathBuf, std::io::Error), +} + +impl FileChunks { + /// Create new iterator. + pub fn new( + chunk_size: usize, + handle: std::fs::File, + filename: &Path, + kind: LabelChecksumKind, + ) -> Self { + let buf = vec![0; chunk_size]; Self { chunk_size, + kind, buf, handle, + filename: filename.to_path_buf(), } } - pub fn read_chunk(&mut self) -> anyhow::Result<Option<(ChunkMeta, DataChunk)>> { + fn read_chunk(&mut self) -> Result<Option<DataChunk>, ChunkerError> { let mut used = 0; loop { - let n = self.handle.read(&mut self.buf.as_mut_slice()[used..])?; + let n = self + .handle + .read(&mut self.buf.as_mut_slice()[used..]) + .map_err(|err| ChunkerError::FileRead(self.filename.to_path_buf(), err))?; used += n; if n == 0 || used == self.chunk_size { break; @@ -36,20 +60,24 @@ impl Chunker { } let buffer = &self.buf.as_slice()[..used]; - let hash = sha256(buffer); + let hash = match self.kind { + LabelChecksumKind::Blake2 => Label::blake2(buffer), + LabelChecksumKind::Sha256 => Label::sha256(buffer), + }; let meta = ChunkMeta::new(&hash); - let chunk = DataChunk::new(buffer.to_vec()); - Ok(Some((meta, chunk))) + let chunk = DataChunk::new(buffer.to_vec(), meta); + Ok(Some(chunk)) } } -impl Iterator for Chunker { - type Item = anyhow::Result<(ChunkMeta, DataChunk)>; +impl Iterator for FileChunks { + type Item = Result<DataChunk, ChunkerError>; - fn next(&mut self) -> Option<anyhow::Result<(ChunkMeta, DataChunk)>> { + /// Return the next chunk, if any, or an error. + fn next(&mut self) -> Option<Result<DataChunk, ChunkerError>> { match self.read_chunk() { Ok(None) => None, - Ok(Some((meta, chunk))) => Some(Ok((meta, chunk))), + Ok(Some(chunk)) => Some(Ok(chunk)), Err(e) => Some(Err(e)), } } diff --git a/src/chunkid.rs b/src/chunkid.rs index 3933d4b..50fc3d3 100644 --- a/src/chunkid.rs +++ b/src/chunkid.rs @@ -1,4 +1,9 @@ -use crate::checksummer::sha256; +//! The identifier for a chunk. +//! +//! Chunk identifiers are chosen by the server. Each chunk has a +//! unique identifier, which isn't based on the contents of the chunk. + +use crate::label::Label; use rusqlite::types::ToSqlOutput; use rusqlite::ToSql; use serde::{Deserialize, Serialize}; @@ -37,20 +42,24 @@ impl ChunkId { } } - pub fn from_str(s: &str) -> Self { + /// Re-construct an identifier from a previous value. + pub fn recreate(s: &str) -> Self { ChunkId { id: s.to_string() } } + /// Return the identifier as a slice of bytes. pub fn as_bytes(&self) -> &[u8] { self.id.as_bytes() } - pub fn sha256(&self) -> String { - sha256(self.id.as_bytes()) + /// Return the SHA256 checksum of the identifier. + pub fn sha256(&self) -> Label { + Label::sha256(self.id.as_bytes()) } } impl ToSql for ChunkId { + /// Format identifier for SQL. fn to_sql(&self) -> rusqlite::Result<ToSqlOutput> { Ok(ToSqlOutput::Owned(rusqlite::types::Value::Text( self.id.clone(), @@ -68,12 +77,14 @@ impl fmt::Display for ChunkId { } impl From<&String> for ChunkId { + /// Create a chunk identifier from a string. fn from(s: &String) -> Self { ChunkId { id: s.to_string() } } } impl From<&OsStr> for ChunkId { + /// Create a chunk identifier from an operating system string. fn from(s: &OsStr) -> Self { ChunkId { id: s.to_string_lossy().to_string(), @@ -84,8 +95,9 @@ impl From<&OsStr> for ChunkId { impl FromStr for ChunkId { type Err = (); + /// Create a chunk from a string. fn from_str(s: &str) -> Result<Self, Self::Err> { - Ok(ChunkId::from_str(s)) + Ok(ChunkId::recreate(s)) } } @@ -117,6 +129,6 @@ mod test { fn survives_round_trip() { let id = ChunkId::new(); let id_str = id.to_string(); - assert_eq!(id, ChunkId::from_str(&id_str)) + assert_eq!(id, ChunkId::recreate(&id_str)) } } diff --git a/src/chunkmeta.rs b/src/chunkmeta.rs index 37e2ed5..e2fa9b3 100644 --- a/src/chunkmeta.rs +++ b/src/chunkmeta.rs @@ -1,28 +1,21 @@ +//! Metadata about a chunk. + +use crate::label::Label; use serde::{Deserialize, Serialize}; use std::default::Default; use std::str::FromStr; /// Metadata about chunks. /// -/// We manage three bits of metadata about chunks, in addition to its -/// identifier: -/// -/// * for all chunks, a [SHA256][] checksum of the chunk content -/// -/// * for generation chunks, an indication that it is a generation -/// chunk, and a timestamp for when making the generation snapshot -/// ended -/// -/// There is no syntax or semantics imposed on the timestamp, but a -/// client should probably use [ISO 8601][] representation. +/// We a single piece of metadata about chunks, in addition to its +/// identifier: a label assigned by the client. Currently, this is a +/// [SHA256][] checksum of the chunk content. /// /// For HTTP, the metadata will be serialised as a JSON object, like this: /// /// ~~~json /// { -/// "sha256": "09ca7e4eaa6e8ae9c7d261167129184883644d07dfba7cbfbc4c8a2e08360d5b", -/// "generation": true, -/// "ended": "2020-09-17T08:17:13+03:00" +/// "label": "09ca7e4eaa6e8ae9c7d261167129184883644d07dfba7cbfbc4c8a2e08360d5b", /// } /// ~~~ /// @@ -35,55 +28,44 @@ use std::str::FromStr; /// /// [ISO 8601]: https://en.wikipedia.org/wiki/ISO_8601 /// [SHA256]: https://en.wikipedia.org/wiki/SHA-2 -#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[derive(Clone, Debug, Default, Eq, PartialEq, Serialize, Deserialize)] pub struct ChunkMeta { - sha256: String, - // The remaining fields are Options so that JSON parsing doesn't - // insist on them being there in the textual representation. - generation: Option<bool>, - ended: Option<String>, + label: String, } impl ChunkMeta { /// Create a new data chunk. /// /// Data chunks are not for generations. - pub fn new(sha256: &str) -> Self { + pub fn new(label: &Label) -> Self { ChunkMeta { - sha256: sha256.to_string(), - generation: None, - ended: None, + label: label.serialize(), } } - /// Create a new generation chunk. - pub fn new_generation(sha256: &str, ended: &str) -> Self { - ChunkMeta { - sha256: sha256.to_string(), - generation: Some(true), - ended: Some(ended.to_string()), - } - } - - /// Is this a generation chunk? - pub fn is_generation(&self) -> bool { - matches!(self.generation, Some(true)) - } - - /// When did this generation end? - pub fn ended(&self) -> Option<&str> { - self.ended.as_deref() + /// The label of the content of the chunk. + /// + /// The caller should not interpret the label in any way. It + /// happens to be a SHA256 of the cleartext contents of the + /// checksum for now, but that _will_ change in the future. + pub fn label(&self) -> &str { + &self.label } - /// SHA256 checksum of the content of the chunk. - pub fn sha256(&self) -> &str { - &self.sha256 + /// Serialize from a textual JSON representation. + pub fn from_json(json: &str) -> Result<Self, serde_json::Error> { + serde_json::from_str(json) } /// Serialize as JSON. pub fn to_json(&self) -> String { serde_json::to_string(self).unwrap() } + + /// Serialize as JSON, as a byte vector. + pub fn to_json_vec(&self) -> Vec<u8> { + self.to_json().as_bytes().to_vec() + } } impl FromStr for ChunkMeta { @@ -97,48 +79,54 @@ impl FromStr for ChunkMeta { #[cfg(test)] mod test { - use super::ChunkMeta; + use super::{ChunkMeta, Label}; #[test] fn new_creates_data_chunk() { - let meta = ChunkMeta::new("abcdef"); - assert!(!meta.is_generation()); - assert_eq!(meta.ended(), None); - assert_eq!(meta.sha256(), "abcdef"); + let sum = Label::sha256(b"abcdef"); + let meta = ChunkMeta::new(&sum); + assert_eq!(meta.label(), sum.serialize()); } #[test] fn new_generation_creates_generation_chunk() { - let meta = ChunkMeta::new_generation("abcdef", "2020-09-17T08:17:13+03:00"); - assert!(meta.is_generation()); - assert_eq!(meta.ended(), Some("2020-09-17T08:17:13+03:00")); - assert_eq!(meta.sha256(), "abcdef"); + let sum = Label::sha256(b"abcdef"); + let meta = ChunkMeta::new(&sum); + assert_eq!(meta.label(), sum.serialize()); } #[test] fn data_chunk_from_json() { - let meta: ChunkMeta = r#"{"sha256": "abcdef"}"#.parse().unwrap(); - assert!(!meta.is_generation()); - assert_eq!(meta.ended(), None); - assert_eq!(meta.sha256(), "abcdef"); + let meta: ChunkMeta = r#"{"label": "abcdef"}"#.parse().unwrap(); + assert_eq!(meta.label(), "abcdef"); } #[test] fn generation_chunk_from_json() { let meta: ChunkMeta = - r#"{"sha256": "abcdef", "generation": true, "ended": "2020-09-17T08:17:13+03:00"}"# + r#"{"label": "abcdef", "generation": true, "ended": "2020-09-17T08:17:13+03:00"}"# .parse() .unwrap(); - assert!(meta.is_generation()); - assert_eq!(meta.ended(), Some("2020-09-17T08:17:13+03:00")); - assert_eq!(meta.sha256(), "abcdef"); + + assert_eq!(meta.label(), "abcdef"); } #[test] - fn json_roundtrip() { - let meta = ChunkMeta::new_generation("abcdef", "2020-09-17T08:17:13+03:00"); + fn generation_json_roundtrip() { + let sum = Label::sha256(b"abcdef"); + let meta = ChunkMeta::new(&sum); let json = serde_json::to_string(&meta).unwrap(); let meta2 = serde_json::from_str(&json).unwrap(); assert_eq!(meta, meta2); } + + #[test] + fn data_json_roundtrip() { + let sum = Label::sha256(b"abcdef"); + let meta = ChunkMeta::new(&sum); + let json = meta.to_json_vec(); + let meta2 = serde_json::from_slice(&json).unwrap(); + assert_eq!(meta, meta2); + assert_eq!(meta.to_json_vec(), meta2.to_json_vec()); + } } diff --git a/src/chunkstore.rs b/src/chunkstore.rs new file mode 100644 index 0000000..4c8125c --- /dev/null +++ b/src/chunkstore.rs @@ -0,0 +1,307 @@ +//! Access local and remote chunk stores. +//! +//! A chunk store may be local and accessed via the file system, or +//! remote and accessed over HTTP. This module implements both. This +//! module only handles encrypted chunks. + +use crate::chunkid::ChunkId; +use crate::chunkmeta::ChunkMeta; +use crate::config::{ClientConfig, ClientConfigError}; +use crate::index::{Index, IndexError}; + +use log::{debug, error, info}; +use reqwest::header::HeaderMap; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use tokio::sync::Mutex; + +/// A chunk store. +/// +/// The store may be local or remote. +pub enum ChunkStore { + /// A local chunk store. + Local(LocalStore), + + /// A remote chunk store. + Remote(RemoteStore), +} + +impl ChunkStore { + /// Open a local chunk store. + pub fn local<P: AsRef<Path>>(path: P) -> Result<Self, StoreError> { + let store = LocalStore::new(path.as_ref())?; + Ok(Self::Local(store)) + } + + /// Open a remote chunk store. + pub fn remote(config: &ClientConfig) -> Result<Self, StoreError> { + let store = RemoteStore::new(config)?; + Ok(Self::Remote(store)) + } + + /// Does the store have a chunk with a given label? + pub async fn find_by_label(&self, meta: &ChunkMeta) -> Result<Vec<ChunkId>, StoreError> { + match self { + Self::Local(store) => store.find_by_label(meta).await, + Self::Remote(store) => store.find_by_label(meta).await, + } + } + + /// Store a chunk in the store. + /// + /// The store chooses an id for the chunk. + pub async fn put(&self, chunk: Vec<u8>, meta: &ChunkMeta) -> Result<ChunkId, StoreError> { + match self { + Self::Local(store) => store.put(chunk, meta).await, + Self::Remote(store) => store.put(chunk, meta).await, + } + } + + /// Get a chunk given its id. + pub async fn get(&self, id: &ChunkId) -> Result<(Vec<u8>, ChunkMeta), StoreError> { + match self { + Self::Local(store) => store.get(id).await, + Self::Remote(store) => store.get(id).await, + } + } +} + +/// A local chunk store. +pub struct LocalStore { + path: PathBuf, + index: Mutex<Index>, +} + +impl LocalStore { + fn new(path: &Path) -> Result<Self, StoreError> { + Ok(Self { + path: path.to_path_buf(), + index: Mutex::new(Index::new(path)?), + }) + } + + async fn find_by_label(&self, meta: &ChunkMeta) -> Result<Vec<ChunkId>, StoreError> { + self.index + .lock() + .await + .find_by_label(meta.label()) + .map_err(StoreError::Index) + } + + async fn put(&self, chunk: Vec<u8>, meta: &ChunkMeta) -> Result<ChunkId, StoreError> { + let id = ChunkId::new(); + let (dir, filename) = self.filename(&id); + + if !dir.exists() { + std::fs::create_dir_all(&dir).map_err(|err| StoreError::ChunkMkdir(dir, err))?; + } + + std::fs::write(&filename, &chunk) + .map_err(|err| StoreError::WriteChunk(filename.clone(), err))?; + self.index + .lock() + .await + .insert_meta(id.clone(), meta.clone()) + .map_err(StoreError::Index)?; + Ok(id) + } + + async fn get(&self, id: &ChunkId) -> Result<(Vec<u8>, ChunkMeta), StoreError> { + let meta = self.index.lock().await.get_meta(id)?; + + let (_, filename) = &self.filename(id); + + let raw = + std::fs::read(filename).map_err(|err| StoreError::ReadChunk(filename.clone(), err))?; + + Ok((raw, meta)) + } + + fn filename(&self, id: &ChunkId) -> (PathBuf, PathBuf) { + let bytes = id.as_bytes(); + assert!(bytes.len() > 3); + let a = bytes[0]; + let b = bytes[1]; + let c = bytes[2]; + let dir = self.path.join(format!("{}/{}/{}", a, b, c)); + let filename = dir.join(format!("{}.data", id)); + (dir, filename) + } +} + +/// A remote chunk store. +pub struct RemoteStore { + client: reqwest::Client, + base_url: String, +} + +impl RemoteStore { + fn new(config: &ClientConfig) -> Result<Self, StoreError> { + info!("creating remote store with config: {:#?}", config); + + let client = reqwest::Client::builder() + .danger_accept_invalid_certs(!config.verify_tls_cert) + .build() + .map_err(StoreError::ReqwestError)?; + Ok(Self { + client, + base_url: config.server_url.to_string(), + }) + } + + async fn find_by_label(&self, meta: &ChunkMeta) -> Result<Vec<ChunkId>, StoreError> { + let body = match self.get_helper("", &[("label", meta.label())]).await { + Ok((_, body)) => body, + Err(err) => return Err(err), + }; + + let hits: HashMap<String, ChunkMeta> = + serde_json::from_slice(&body).map_err(StoreError::JsonParse)?; + let ids = hits.keys().map(|id| ChunkId::recreate(id)).collect(); + Ok(ids) + } + + async fn put(&self, chunk: Vec<u8>, meta: &ChunkMeta) -> Result<ChunkId, StoreError> { + let res = self + .client + .post(&self.chunks_url()) + .header("chunk-meta", meta.to_json()) + .body(chunk) + .send() + .await + .map_err(StoreError::ReqwestError)?; + let res: HashMap<String, String> = res.json().await.map_err(StoreError::ReqwestError)?; + debug!("upload_chunk: res={:?}", res); + let chunk_id = if let Some(chunk_id) = res.get("chunk_id") { + debug!("upload_chunk: id={}", chunk_id); + chunk_id.parse().unwrap() + } else { + return Err(StoreError::NoCreatedChunkId); + }; + info!("uploaded_chunk {}", chunk_id); + Ok(chunk_id) + } + + async fn get(&self, id: &ChunkId) -> Result<(Vec<u8>, ChunkMeta), StoreError> { + let (headers, body) = self.get_helper(&format!("/{}", id), &[]).await?; + let meta = self.get_chunk_meta_header(id, &headers)?; + Ok((body, meta)) + } + + fn base_url(&self) -> &str { + &self.base_url + } + + fn chunks_url(&self) -> String { + format!("{}/v1/chunks", self.base_url()) + } + + async fn get_helper( + &self, + path: &str, + query: &[(&str, &str)], + ) -> Result<(HeaderMap, Vec<u8>), StoreError> { + let url = format!("{}{}", &self.chunks_url(), path); + info!("GET {}", url); + + // Build HTTP request structure. + let req = self + .client + .get(&url) + .query(query) + .build() + .map_err(StoreError::ReqwestError)?; + + // Make HTTP request. + let res = self + .client + .execute(req) + .await + .map_err(StoreError::ReqwestError)?; + + // Did it work? + if res.status() != 200 { + return Err(StoreError::NotFound(path.to_string())); + } + + // Return headers and body. + let headers = res.headers().clone(); + let body = res.bytes().await.map_err(StoreError::ReqwestError)?; + let body = body.to_vec(); + Ok((headers, body)) + } + + fn get_chunk_meta_header( + &self, + chunk_id: &ChunkId, + headers: &HeaderMap, + ) -> Result<ChunkMeta, StoreError> { + let meta = headers.get("chunk-meta"); + + if meta.is_none() { + let err = StoreError::NoChunkMeta(chunk_id.clone()); + error!("fetching chunk {} failed: {}", chunk_id, err); + return Err(err); + } + + let meta = meta + .unwrap() + .to_str() + .map_err(StoreError::MetaHeaderToString)?; + let meta: ChunkMeta = serde_json::from_str(meta).map_err(StoreError::JsonParse)?; + + Ok(meta) + } +} + +/// Possible errors from using a ChunkStore. +#[derive(Debug, thiserror::Error)] +pub enum StoreError { + /// FIXME + #[error("FIXME")] + FIXME, + + /// Error from a chunk index. + #[error(transparent)] + Index(#[from] IndexError), + + /// An error from the HTTP library. + #[error("error from reqwest library: {0}")] + ReqwestError(reqwest::Error), + + /// Client configuration is wrong. + #[error(transparent)] + ClientConfigError(#[from] ClientConfigError), + + /// Server claims to not have an entity. + #[error("Server does not have {0}")] + NotFound(String), + + /// Server didn't give us a chunk's metadata. + #[error("Server response did not have a 'chunk-meta' header for chunk {0}")] + NoChunkMeta(ChunkId), + + /// An error with the `chunk-meta` header. + #[error("couldn't convert response chunk-meta header to string: {0}")] + MetaHeaderToString(reqwest::header::ToStrError), + + /// Error parsing JSON. + #[error("failed to parse JSON: {0}")] + JsonParse(serde_json::Error), + + /// An error creating chunk directory. + #[error("Failed to create chunk directory {0}")] + ChunkMkdir(PathBuf, #[source] std::io::Error), + + /// An error writing a chunk file. + #[error("Failed to write chunk {0}")] + WriteChunk(PathBuf, #[source] std::io::Error), + + /// An error reading a chunk file. + #[error("Failed to read chunk {0}")] + ReadChunk(PathBuf, #[source] std::io::Error), + + /// No chunk id for uploaded chunk. + #[error("Server response claimed it had created a chunk, but lacked chunk id")] + NoCreatedChunkId, +} diff --git a/src/cipher.rs b/src/cipher.rs new file mode 100644 index 0000000..21785b9 --- /dev/null +++ b/src/cipher.rs @@ -0,0 +1,249 @@ +//! Encryption cipher algorithms. + +use crate::chunk::DataChunk; +use crate::chunkmeta::ChunkMeta; +use crate::passwords::Passwords; + +use aes_gcm::aead::{generic_array::GenericArray, Aead, KeyInit, Payload}; +use aes_gcm::Aes256Gcm; // Or `Aes128Gcm` +use rand::Rng; + +use std::str::FromStr; + +const CHUNK_V1: &[u8] = b"0001"; + +/// An encrypted chunk. +/// +/// This consists of encrypted ciphertext, and un-encrypted (or +/// cleartext) additional associated data, which could be the metadata +/// of the chunk, and be used to, for example, find chunks. +/// +/// Encrypted chunks are the only chunks that can be uploaded to the +/// server. +pub struct EncryptedChunk { + ciphertext: Vec<u8>, + aad: Vec<u8>, +} + +impl EncryptedChunk { + /// Create an encrypted chunk. + fn new(ciphertext: Vec<u8>, aad: Vec<u8>) -> Self { + Self { ciphertext, aad } + } + + /// Return the encrypted data. + pub fn ciphertext(&self) -> &[u8] { + &self.ciphertext + } + + /// Return the cleartext associated additional data. + pub fn aad(&self) -> &[u8] { + &self.aad + } +} + +/// An engine for encrypting and decrypting chunks. +pub struct CipherEngine { + cipher: Aes256Gcm, +} + +impl CipherEngine { + /// Create a new cipher engine using cleartext passwords. + pub fn new(pass: &Passwords) -> Self { + let key = GenericArray::from_slice(pass.encryption_key()); + Self { + cipher: Aes256Gcm::new(key), + } + } + + /// Encrypt a chunk. + pub fn encrypt_chunk(&self, chunk: &DataChunk) -> Result<EncryptedChunk, CipherError> { + // Payload with metadata as associated data, to be encrypted. + // + // The metadata will be stored in cleartext after encryption. + let aad = chunk.meta().to_json_vec(); + let payload = Payload { + msg: chunk.data(), + aad: &aad, + }; + + // Unique random key for each encryption. + let nonce = Nonce::new(); + let nonce_arr = GenericArray::from_slice(nonce.as_bytes()); + + // Encrypt the sensitive part. + let ciphertext = self + .cipher + .encrypt(nonce_arr, payload) + .map_err(CipherError::EncryptError)?; + + // Construct the blob to be stored on the server. + let mut vec: Vec<u8> = vec![]; + push_bytes(&mut vec, CHUNK_V1); + push_bytes(&mut vec, nonce.as_bytes()); + push_bytes(&mut vec, &ciphertext); + + Ok(EncryptedChunk::new(vec, aad)) + } + + /// Decrypt a chunk. + pub fn decrypt_chunk(&self, bytes: &[u8], meta: &[u8]) -> Result<DataChunk, CipherError> { + // Does encrypted chunk start with the right version? + if !bytes.starts_with(CHUNK_V1) { + return Err(CipherError::UnknownChunkVersion); + } + let version_len = CHUNK_V1.len(); + let bytes = &bytes[version_len..]; + + let (nonce, ciphertext) = match bytes.get(..NONCE_SIZE) { + Some(nonce) => (GenericArray::from_slice(nonce), &bytes[NONCE_SIZE..]), + None => return Err(CipherError::NoNonce), + }; + + let payload = Payload { + msg: ciphertext, + aad: meta, + }; + + let payload = self + .cipher + .decrypt(nonce, payload) + .map_err(CipherError::DecryptError)?; + let payload = Payload::from(payload.as_slice()); + + let meta = std::str::from_utf8(meta)?; + let meta = ChunkMeta::from_str(meta)?; + + let chunk = DataChunk::new(payload.msg.to_vec(), meta); + + Ok(chunk) + } +} + +fn push_bytes(vec: &mut Vec<u8>, bytes: &[u8]) { + for byte in bytes.iter() { + vec.push(*byte); + } +} + +/// Possible errors when encrypting or decrypting chunks. +#[derive(Debug, thiserror::Error)] +pub enum CipherError { + /// Encryption failed. + #[error("failed to encrypt with AES-GEM: {0}")] + EncryptError(aes_gcm::Error), + + /// The encrypted chunk has an unsupported version or is + /// corrupted. + #[error("encrypted chunk does not start with correct version")] + UnknownChunkVersion, + + /// The encrypted chunk lacks a complete nonce value, and is + /// probably corrupted. + #[error("encrypted chunk does not have a complete nonce")] + NoNonce, + + /// Decryption failed. + #[error("failed to decrypt with AES-GEM: {0}")] + DecryptError(aes_gcm::Error), + + /// The decryption succeeded, by data isn't valid YAML. + #[error("failed to parse decrypted data as a DataChunk: {0}")] + Parse(serde_yaml::Error), + + /// Error parsing UTF8 data. + #[error(transparent)] + Utf8Error(#[from] std::str::Utf8Error), + + /// Error parsing JSON data. + #[error("failed to parse JSON: {0}")] + JsonParse(#[from] serde_json::Error), +} + +const NONCE_SIZE: usize = 12; + +#[derive(Debug)] +struct Nonce { + nonce: Vec<u8>, +} + +impl Nonce { + fn from_bytes(bytes: &[u8]) -> Self { + assert_eq!(bytes.len(), NONCE_SIZE); + Self { + nonce: bytes.to_vec(), + } + } + + fn new() -> Self { + let mut bytes: Vec<u8> = vec![0; NONCE_SIZE]; + let mut rng = rand::thread_rng(); + for x in bytes.iter_mut() { + *x = rng.gen(); + } + Self::from_bytes(&bytes) + } + + fn as_bytes(&self) -> &[u8] { + &self.nonce + } +} + +#[cfg(test)] +mod test { + use crate::chunk::DataChunk; + use crate::chunkmeta::ChunkMeta; + use crate::cipher::{CipherEngine, CipherError, CHUNK_V1, NONCE_SIZE}; + use crate::label::Label; + use crate::passwords::Passwords; + + #[test] + fn metadata_as_aad() { + let sum = Label::sha256(b"dummy data"); + let meta = ChunkMeta::new(&sum); + let meta_as_aad = meta.to_json_vec(); + let chunk = DataChunk::new("hello".as_bytes().to_vec(), meta); + let pass = Passwords::new("secret"); + let cipher = CipherEngine::new(&pass); + let enc = cipher.encrypt_chunk(&chunk).unwrap(); + + assert_eq!(meta_as_aad, enc.aad()); + } + + #[test] + fn round_trip() { + let sum = Label::sha256(b"dummy data"); + let meta = ChunkMeta::new(&sum); + let chunk = DataChunk::new("hello".as_bytes().to_vec(), meta); + let pass = Passwords::new("secret"); + + let cipher = CipherEngine::new(&pass); + let enc = cipher.encrypt_chunk(&chunk).unwrap(); + + let bytes: Vec<u8> = enc.ciphertext().to_vec(); + let dec = cipher.decrypt_chunk(&bytes, enc.aad()).unwrap(); + assert_eq!(chunk, dec); + } + + #[test] + fn decrypt_errors_if_nonce_is_too_short() { + let pass = Passwords::new("our little test secret"); + let e = CipherEngine::new(&pass); + + // *Almost* a valid chunk header, except it's one byte too short + let bytes = { + let mut result = [0; CHUNK_V1.len() + NONCE_SIZE - 1]; + for (i, x) in CHUNK_V1.iter().enumerate() { + result[i] = *x; + } + result + }; + + let meta = [0; 0]; + + assert!(matches!( + e.decrypt_chunk(&bytes, &meta), + Err(CipherError::NoNonce) + )); + } +} diff --git a/src/client.rs b/src/client.rs index 515b8c9..a924052 100644 --- a/src/client.rs +++ b/src/client.rs @@ -1,271 +1,207 @@ -use crate::checksummer::sha256; -use crate::chunk::DataChunk; -use crate::chunk::GenerationChunk; -use crate::chunker::Chunker; +//! Client to the Obnam server HTTP API. + +use crate::chunk::{ + ClientTrust, ClientTrustError, DataChunk, GenerationChunk, GenerationChunkError, +}; use crate::chunkid::ChunkId; use crate::chunkmeta::ChunkMeta; -use crate::error::ObnamError; -use crate::fsentry::{FilesystemEntry, FilesystemKind}; -use crate::generation::{FinishedGeneration, LocalGeneration}; +use crate::chunkstore::{ChunkStore, StoreError}; +use crate::cipher::{CipherEngine, CipherError}; +use crate::config::{ClientConfig, ClientConfigError}; +use crate::generation::{FinishedGeneration, GenId, LocalGeneration, LocalGenerationError}; use crate::genlist::GenerationList; +use crate::label::Label; -use anyhow::Context; -use chrono::{DateTime, Local}; -use log::{debug, error, info, trace}; -use reqwest::blocking::Client; -use serde::Deserialize; -use std::collections::HashMap; +use log::{error, info}; use std::fs::File; use std::io::prelude::*; use std::path::{Path, PathBuf}; -#[derive(Debug, Deserialize, Clone)] -pub struct ClientConfig { - pub server_url: String, - pub root: PathBuf, - pub log: Option<PathBuf>, -} - -impl ClientConfig { - pub fn read_config(filename: &Path) -> anyhow::Result<Self> { - trace!("read_config: filename={:?}", filename); - let config = std::fs::read_to_string(filename) - .with_context(|| format!("reading configuration file {}", filename.display()))?; - let config = serde_yaml::from_str(&config)?; - Ok(config) - } -} - +/// Possible errors when using the server API. #[derive(Debug, thiserror::Error)] pub enum ClientError { - #[error("Server successful response to creating chunk lacked chunk id")] + /// No chunk id for uploaded chunk. + #[error("Server response claimed it had created a chunk, but lacked chunk id")] NoCreatedChunkId, + /// Server claims to not have an entity. + #[error("Server does not have {0}")] + NotFound(String), + + /// Server does not have a chunk. #[error("Server does not have chunk {0}")] - ChunkNotFound(String), + ChunkNotFound(ChunkId), + /// Server does not have generation. #[error("Server does not have generation {0}")] - GenerationNotFound(String), -} + GenerationNotFound(ChunkId), -pub struct BackupClient { - client: Client, - base_url: String, -} + /// Server didn't give us a chunk's metadata. + #[error("Server response did not have a 'chunk-meta' header for chunk {0}")] + NoChunkMeta(ChunkId), -impl BackupClient { - pub fn new(base_url: &str) -> anyhow::Result<Self> { - let client = Client::builder() - .danger_accept_invalid_certs(true) - .build()?; - Ok(Self { - client, - base_url: base_url.to_string(), - }) - } + /// Chunk has wrong checksum and may be corrupted. + #[error("Wrong checksum for chunk {0}, got {1}, expected {2}")] + WrongChecksum(ChunkId, String, String), - pub fn upload_filesystem_entry( - &self, - e: &FilesystemEntry, - size: usize, - ) -> anyhow::Result<Vec<ChunkId>> { - info!("upload entry: {:?}", e); - let ids = match e.kind() { - FilesystemKind::Regular => self.read_file(e.pathbuf(), size)?, - FilesystemKind::Directory => vec![], - FilesystemKind::Symlink => vec![], - }; - Ok(ids) - } + /// Client configuration is wrong. + #[error(transparent)] + ClientConfigError(#[from] ClientConfigError), - pub fn upload_generation(&self, filename: &Path, size: usize) -> anyhow::Result<ChunkId> { - info!("upload SQLite {}", filename.display()); - let ids = self.read_file(filename.to_path_buf(), size)?; - let gen = GenerationChunk::new(ids); - let data = gen.to_data_chunk()?; - let meta = ChunkMeta::new_generation(&sha256(data.data()), ¤t_timestamp()); - let gen_id = self.upload_gen_chunk(meta.clone(), gen)?; - info!("uploaded generation {}, meta {:?}", gen_id, meta); - Ok(gen_id) - } + /// An error encrypting or decrypting chunks. + #[error(transparent)] + CipherError(#[from] CipherError), - fn read_file(&self, filename: PathBuf, size: usize) -> anyhow::Result<Vec<ChunkId>> { - info!("upload file {}", filename.display()); - let file = std::fs::File::open(filename)?; - let chunker = Chunker::new(size, file); - let chunk_ids = self.upload_new_file_chunks(chunker)?; - Ok(chunk_ids) - } + /// An error regarding generation chunks. + #[error(transparent)] + GenerationChunkError(#[from] GenerationChunkError), - fn base_url(&self) -> &str { - &self.base_url - } + /// An error regarding client trust. + #[error(transparent)] + ClientTrust(#[from] ClientTrustError), - fn chunks_url(&self) -> String { - format!("{}/chunks", self.base_url()) - } + /// An error using a backup's local metadata. + #[error(transparent)] + LocalGenerationError(#[from] LocalGenerationError), - pub fn has_chunk(&self, meta: &ChunkMeta) -> anyhow::Result<Option<ChunkId>> { - trace!("has_chunk: url={:?}", self.base_url()); - let req = self - .client - .get(&self.chunks_url()) - .query(&[("sha256", meta.sha256())]) - .build()?; - - let res = self.client.execute(req)?; - debug!("has_chunk: status={}", res.status()); - let has = if res.status() != 200 { - debug!("has_chunk: error from server"); - None - } else { - let text = res.text()?; - debug!("has_chunk: text={:?}", text); - let hits: HashMap<String, ChunkMeta> = serde_json::from_str(&text)?; - debug!("has_chunk: hits={:?}", hits); - let mut iter = hits.iter(); - if let Some((chunk_id, _)) = iter.next() { - debug!("has_chunk: chunk_id={:?}", chunk_id); - Some(chunk_id.into()) - } else { - None - } - }; + /// An error with the `chunk-meta` header. + #[error("couldn't convert response chunk-meta header to string: {0}")] + MetaHeaderToString(reqwest::header::ToStrError), + + /// An error from the HTTP library. + #[error("error from reqwest library: {0}")] + ReqwestError(reqwest::Error), + + /// Couldn't look up a chunk via checksum. + #[error("lookup by chunk checksum failed: {0}")] + ChunkExists(reqwest::Error), + + /// Error parsing JSON. + #[error("failed to parse JSON: {0}")] + JsonParse(serde_json::Error), - info!("has_chunk result: {:?}", has); - Ok(has) + /// Error generating JSON. + #[error("failed to generate JSON: {0}")] + JsonGenerate(serde_json::Error), + + /// Error parsing YAML. + #[error("failed to parse YAML: {0}")] + YamlParse(serde_yaml::Error), + + /// Failed to open a file. + #[error("failed to open file {0}: {1}")] + FileOpen(PathBuf, std::io::Error), + + /// Failed to create a file. + #[error("failed to create file {0}: {1}")] + FileCreate(PathBuf, std::io::Error), + + /// Failed to write a file. + #[error("failed to write to file {0}: {1}")] + FileWrite(PathBuf, std::io::Error), + + /// Error from a chunk store. + #[error(transparent)] + ChunkStore(#[from] StoreError), +} + +/// Client for the Obnam server HTTP API. +pub struct BackupClient { + store: ChunkStore, + cipher: CipherEngine, +} + +impl BackupClient { + /// Create a new backup client. + pub fn new(config: &ClientConfig) -> Result<Self, ClientError> { + info!("creating backup client with config: {:#?}", config); + let pass = config.passwords()?; + Ok(Self { + store: ChunkStore::remote(config)?, + cipher: CipherEngine::new(&pass), + }) } - pub fn upload_chunk(&self, meta: ChunkMeta, chunk: DataChunk) -> anyhow::Result<ChunkId> { - let res = self - .client - .post(&self.chunks_url()) - .header("chunk-meta", meta.to_json()) - .body(chunk.data().to_vec()) - .send()?; - debug!("upload_chunk: res={:?}", res); - let res: HashMap<String, String> = res.json()?; - let chunk_id = if let Some(chunk_id) = res.get("chunk_id") { - debug!("upload_chunk: id={}", chunk_id); - chunk_id.parse().unwrap() - } else { - return Err(ClientError::NoCreatedChunkId.into()); - }; - info!("uploaded_chunk {} meta {:?}", chunk_id, meta); - Ok(chunk_id) + /// Does the server have a chunk? + pub async fn has_chunk(&self, meta: &ChunkMeta) -> Result<Option<ChunkId>, ClientError> { + let mut ids = self.store.find_by_label(meta).await?; + Ok(ids.pop()) } - pub fn upload_gen_chunk( - &self, - meta: ChunkMeta, - gen: GenerationChunk, - ) -> anyhow::Result<ChunkId> { - let res = self - .client - .post(&self.chunks_url()) - .header("chunk-meta", meta.to_json()) - .body(serde_json::to_string(&gen)?) - .send()?; - debug!("upload_chunk: res={:?}", res); - let res: HashMap<String, String> = res.json()?; - let chunk_id = if let Some(chunk_id) = res.get("chunk_id") { - debug!("upload_chunk: id={}", chunk_id); - chunk_id.parse().unwrap() - } else { - return Err(ClientError::NoCreatedChunkId.into()); - }; - info!("uploaded_generation chunk {}", chunk_id); - Ok(chunk_id) + /// Upload a data chunk to the server. + pub async fn upload_chunk(&mut self, chunk: DataChunk) -> Result<ChunkId, ClientError> { + let enc = self.cipher.encrypt_chunk(&chunk)?; + let data = enc.ciphertext().to_vec(); + let id = self.store.put(data, chunk.meta()).await?; + Ok(id) } - pub fn upload_new_file_chunks(&self, chunker: Chunker) -> anyhow::Result<Vec<ChunkId>> { - let mut chunk_ids = vec![]; - for item in chunker { - let (meta, chunk) = item?; - if let Some(chunk_id) = self.has_chunk(&meta)? { - chunk_ids.push(chunk_id.clone()); - info!("reusing existing chunk {}", chunk_id); + /// Get current client trust chunk from repository, if there is one. + pub async fn get_client_trust(&self) -> Result<Option<ClientTrust>, ClientError> { + let ids = self.find_client_trusts().await?; + let mut latest: Option<ClientTrust> = None; + for id in ids { + let chunk = self.fetch_chunk(&id).await?; + let new = ClientTrust::from_data_chunk(&chunk)?; + if let Some(t) = &latest { + if new.timestamp() > t.timestamp() { + latest = Some(new); + } } else { - let chunk_id = self.upload_chunk(meta, chunk)?; - chunk_ids.push(chunk_id.clone()); - info!("created new chunk {}", chunk_id); + latest = Some(new); } } + Ok(latest) + } - Ok(chunk_ids) + async fn find_client_trusts(&self) -> Result<Vec<ChunkId>, ClientError> { + let label = Label::literal("client-trust"); + let meta = ChunkMeta::new(&label); + let ids = self.store.find_by_label(&meta).await?; + Ok(ids) } - pub fn list_generations(&self) -> anyhow::Result<GenerationList> { - let url = format!("{}?generation=true", &self.chunks_url()); - trace!("list_generations: url={:?}", url); - let req = self.client.get(&url).build()?; - let res = self.client.execute(req)?; - debug!("list_generations: status={}", res.status()); - let body = res.bytes()?; - debug!("list_generations: body={:?}", body); - let map: HashMap<String, ChunkMeta> = serde_yaml::from_slice(&body)?; - debug!("list_generations: map={:?}", map); - let finished = map + /// List backup generations known by the server. + pub fn list_generations(&self, trust: &ClientTrust) -> GenerationList { + let finished = trust + .backups() .iter() - .map(|(id, meta)| FinishedGeneration::new(id, meta.ended().map_or("", |s| s))) + .map(|id| FinishedGeneration::new(&format!("{}", id), "")) .collect(); - Ok(GenerationList::new(finished)) + GenerationList::new(finished) } - pub fn fetch_chunk(&self, chunk_id: &ChunkId) -> anyhow::Result<DataChunk> { - info!("fetch chunk {}", chunk_id); - - let url = format!("{}/{}", &self.chunks_url(), chunk_id); - let req = self.client.get(&url).build()?; - let res = self.client.execute(req)?; - if res.status() != 200 { - let err = ClientError::ChunkNotFound(chunk_id.to_string()); - error!("fetching chunk {} failed: {}", chunk_id, err); - return Err(err.into()); - } - - let headers = res.headers(); - let meta = headers.get("chunk-meta"); - if meta.is_none() { - let err = ObnamError::NoChunkMeta(chunk_id.clone()); - error!("fetching chunk {} failed: {}", chunk_id, err); - return Err(err.into()); - } - let meta = meta.unwrap().to_str()?; - debug!("fetching chunk {}: meta={:?}", chunk_id, meta); - let meta: ChunkMeta = serde_json::from_str(meta)?; - debug!("fetching chunk {}: meta={:?}", chunk_id, meta); - - let body = res.bytes()?; - let body = body.to_vec(); - let actual = sha256(&body); - if actual != meta.sha256() { - let err = - ObnamError::WrongChecksum(chunk_id.clone(), actual, meta.sha256().to_string()); - error!("fetching chunk {} failed: {}", chunk_id, err); - return Err(err.into()); - } - - let chunk: DataChunk = DataChunk::new(body); + /// Fetch a data chunk from the server, given the chunk identifier. + pub async fn fetch_chunk(&self, chunk_id: &ChunkId) -> Result<DataChunk, ClientError> { + let (body, meta) = self.store.get(chunk_id).await?; + let meta_bytes = meta.to_json_vec(); + let chunk = self.cipher.decrypt_chunk(&body, &meta_bytes)?; Ok(chunk) } - fn fetch_generation_chunk(&self, gen_id: &str) -> anyhow::Result<GenerationChunk> { - let chunk_id = ChunkId::from_str(gen_id); - let chunk = self.fetch_chunk(&chunk_id)?; + async fn fetch_generation_chunk(&self, gen_id: &GenId) -> Result<GenerationChunk, ClientError> { + let chunk = self.fetch_chunk(gen_id.as_chunk_id()).await?; let gen = GenerationChunk::from_data_chunk(&chunk)?; Ok(gen) } - pub fn fetch_generation(&self, gen_id: &str, dbname: &Path) -> anyhow::Result<LocalGeneration> { - let gen = self.fetch_generation_chunk(gen_id)?; + /// Fetch a backup generation's metadata, given it's identifier. + pub async fn fetch_generation( + &self, + gen_id: &GenId, + dbname: &Path, + ) -> Result<LocalGeneration, ClientError> { + let gen = self.fetch_generation_chunk(gen_id).await?; // Fetch the SQLite file, storing it in the named file. - let mut dbfile = File::create(&dbname)?; + let mut dbfile = File::create(dbname) + .map_err(|err| ClientError::FileCreate(dbname.to_path_buf(), err))?; for id in gen.chunk_ids() { - let chunk = self.fetch_chunk(id)?; - dbfile.write_all(chunk.data())?; + let chunk = self.fetch_chunk(id).await?; + dbfile + .write_all(chunk.data()) + .map_err(|err| ClientError::FileWrite(dbname.to_path_buf(), err))?; } info!("downloaded generation to {}", dbname.display()); @@ -273,8 +209,3 @@ impl BackupClient { Ok(gen) } } - -fn current_timestamp() -> String { - let now: DateTime<Local> = Local::now(); - format!("{}", now.format("%Y-%m-%d %H:%M:%S.%f %z")) -} diff --git a/src/cmd/backup.rs b/src/cmd/backup.rs index da7298f..70e9eac 100644 --- a/src/cmd/backup.rs +++ b/src/cmd/backup.rs @@ -1,65 +1,137 @@ -use crate::backup_run::BackupRun; -use crate::client::ClientConfig; -use crate::fsiter::FsIterator; -use crate::generation::NascentGeneration; +//! The `backup` subcommand. + +use crate::backup_run::{current_timestamp, BackupRun}; +use crate::chunk::ClientTrust; +use crate::client::BackupClient; +use crate::config::ClientConfig; +use crate::dbgen::{schema_version, FileId, DEFAULT_SCHEMA_MAJOR}; +use crate::error::ObnamError; +use crate::generation::GenId; +use crate::performance::{Clock, Performance}; +use crate::schema::VersionComponent; + +use clap::Parser; use log::info; use std::time::SystemTime; -use tempfile::NamedTempFile; - -pub fn backup(config: &ClientConfig, buffer_size: usize) -> anyhow::Result<()> { - let runtime = SystemTime::now(); - - let run = BackupRun::new(config, buffer_size)?; - - // Create a named temporary file. We don't meed the open file - // handle, so we discard that. - let oldname = { - let temp = NamedTempFile::new()?; - let (_, dbname) = temp.keep()?; - dbname - }; - - // Create a named temporary file. We don't meed the open file - // handle, so we discard that. - let newname = { - let temp = NamedTempFile::new()?; - let (_, dbname) = temp.keep()?; - dbname - }; - - let genlist = run.client().list_generations()?; - let file_count = { - let iter = FsIterator::new(&config.root); - let mut new = NascentGeneration::create(&newname)?; - - match genlist.resolve("latest") { - None => { - info!("fresh backup without a previous generation"); - new.insert_iter(iter.map(|entry| run.backup_file_initially(entry)))?; +use tempfile::tempdir; +use tokio::runtime::Runtime; + +/// Make a backup. +#[derive(Debug, Parser)] +pub struct Backup { + /// Force a full backup, instead of an incremental one. + #[clap(long)] + full: bool, + + /// Backup schema major version to use. + #[clap(long)] + backup_version: Option<VersionComponent>, +} + +impl Backup { + /// Run the command. + pub fn run(&self, config: &ClientConfig, perf: &mut Performance) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config, perf)) + } + + async fn run_async( + &self, + config: &ClientConfig, + perf: &mut Performance, + ) -> Result<(), ObnamError> { + let runtime = SystemTime::now(); + + let major = self.backup_version.unwrap_or(DEFAULT_SCHEMA_MAJOR); + let schema = schema_version(major)?; + + let mut client = BackupClient::new(config)?; + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, current_timestamp(), vec![]))) + .unwrap(); + let genlist = client.list_generations(&trust); + + let temp = tempdir()?; + let oldtemp = temp.path().join("old.db"); + let newtemp = temp.path().join("new.db"); + + let old_id = if self.full { + None + } else { + match genlist.resolve("latest") { + Err(_) => None, + Ok(old_id) => Some(old_id), } - Some(old) => { - info!("incremental backup based on {}", old); - let old = run.client().fetch_generation(&old, &oldname)?; - run.progress() - .files_in_previous_generation(old.file_count()? as u64); - new.insert_iter(iter.map(|entry| run.backup_file_incrementally(entry, &old)))?; + }; + + let (is_incremental, outcome) = if let Some(old_id) = old_id { + info!("incremental backup based on {}", old_id); + let mut run = BackupRun::incremental(config, &mut client)?; + let old = run.start(Some(&old_id), &oldtemp, perf).await?; + ( + true, + run.backup_roots(config, &old, &newtemp, schema, perf) + .await?, + ) + } else { + info!("fresh backup without a previous generation"); + let mut run = BackupRun::initial(config, &mut client)?; + let old = run.start(None, &oldtemp, perf).await?; + ( + false, + run.backup_roots(config, &old, &newtemp, schema, perf) + .await?, + ) + }; + + perf.start(Clock::GenerationUpload); + let mut trust = trust; + trust.append_backup(outcome.gen_id.as_chunk_id()); + trust.finalize(current_timestamp()); + let trust = trust.to_data_chunk()?; + let trust_id = client.upload_chunk(trust).await?; + perf.stop(Clock::GenerationUpload); + info!("uploaded new client-trust {}", trust_id); + + for w in outcome.warnings.iter() { + println!("warning: {}", w); + } + + if is_incremental && !outcome.new_cachedir_tags.is_empty() { + println!("New CACHEDIR.TAG files since the last backup:"); + for t in &outcome.new_cachedir_tags { + println!("- {:?}", t); } + println!("You can configure Obnam to ignore all such files by setting `exclude_cache_tag_directories` to `false`."); } - run.progress().finish(); - new.file_count() - }; - // Upload the SQLite file, i.e., the named temporary file, which - // still exists, since we persisted it above. - let gen_id = run.client().upload_generation(&newname, buffer_size)?; + report_stats( + &runtime, + outcome.files_count, + &outcome.gen_id, + outcome.warnings.len(), + )?; + + if is_incremental && !outcome.new_cachedir_tags.is_empty() { + Err(ObnamError::NewCachedirTagsFound) + } else { + Ok(()) + } + } +} + +fn report_stats( + runtime: &SystemTime, + file_count: FileId, + gen_id: &GenId, + num_warnings: usize, +) -> Result<(), ObnamError> { println!("status: OK"); + println!("warnings: {}", num_warnings); println!("duration: {}", runtime.elapsed()?.as_secs()); println!("file-count: {}", file_count); println!("generation-id: {}", gen_id); - - // Delete the temporary file.q - std::fs::remove_file(&newname)?; - std::fs::remove_file(&oldname)?; - Ok(()) } diff --git a/src/cmd/chunk.rs b/src/cmd/chunk.rs new file mode 100644 index 0000000..293de20 --- /dev/null +++ b/src/cmd/chunk.rs @@ -0,0 +1,70 @@ +//! The `encrypt-chunk` and `decrypt-chunk` subcommands. + +use crate::chunk::DataChunk; +use crate::chunkmeta::ChunkMeta; +use crate::cipher::CipherEngine; +use crate::config::ClientConfig; +use crate::error::ObnamError; +use clap::Parser; +use std::path::PathBuf; + +/// Encrypt a chunk. +#[derive(Debug, Parser)] +pub struct EncryptChunk { + /// The name of the file containing the cleartext chunk. + filename: PathBuf, + + /// Name of file where to write the encrypted chunk. + output: PathBuf, + + /// Chunk metadata as JSON. + json: String, +} + +impl EncryptChunk { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let pass = config.passwords()?; + let cipher = CipherEngine::new(&pass); + + let meta = ChunkMeta::from_json(&self.json)?; + + let cleartext = std::fs::read(&self.filename)?; + let chunk = DataChunk::new(cleartext, meta); + let encrypted = cipher.encrypt_chunk(&chunk)?; + + std::fs::write(&self.output, encrypted.ciphertext())?; + + Ok(()) + } +} + +/// Decrypt a chunk. +#[derive(Debug, Parser)] +pub struct DecryptChunk { + /// Name of file containing encrypted chunk. + filename: PathBuf, + + /// Name of file where to write the cleartext chunk. + output: PathBuf, + + /// Chunk metadata as JSON. + json: String, +} + +impl DecryptChunk { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let pass = config.passwords()?; + let cipher = CipherEngine::new(&pass); + + let meta = ChunkMeta::from_json(&self.json)?; + + let encrypted = std::fs::read(&self.filename)?; + let chunk = cipher.decrypt_chunk(&encrypted, &meta.to_json_vec())?; + + std::fs::write(&self.output, chunk.data())?; + + Ok(()) + } +} diff --git a/src/cmd/chunkify.rs b/src/cmd/chunkify.rs new file mode 100644 index 0000000..91cb0be --- /dev/null +++ b/src/cmd/chunkify.rs @@ -0,0 +1,110 @@ +//! The `chunkify` subcommand. + +use crate::config::ClientConfig; +use crate::engine::Engine; +use crate::error::ObnamError; +use crate::workqueue::WorkQueue; +use clap::Parser; +use serde::Serialize; +use sha2::{Digest, Sha256}; +use std::path::PathBuf; +use tokio::fs::File; +use tokio::io::{AsyncReadExt, BufReader}; +use tokio::runtime::Runtime; +use tokio::sync::mpsc; + +// Size of queue with unprocessed chunks, and also queue of computed +// checksums. +const Q: usize = 8; + +/// Split files into chunks and show their metadata. +#[derive(Debug, Parser)] +pub struct Chunkify { + /// Names of files to split into chunks. + filenames: Vec<PathBuf>, +} + +impl Chunkify { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } + + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let mut q = WorkQueue::new(Q); + for filename in self.filenames.iter() { + tokio::spawn(split_file( + filename.to_path_buf(), + config.chunk_size, + q.push(), + )); + } + q.close(); + + let mut summer = Engine::new(q, just_hash); + + let mut checksums = vec![]; + while let Some(sum) = summer.next().await { + checksums.push(sum); + } + + println!("{}", serde_json::to_string_pretty(&checksums)?); + + Ok(()) + } +} + +#[derive(Debug, Clone)] +struct Chunk { + filename: PathBuf, + offset: u64, + data: Vec<u8>, +} + +#[derive(Debug, Clone, Serialize)] +struct Checksum { + filename: PathBuf, + offset: u64, + pub len: u64, + checksum: String, +} + +async fn split_file(filename: PathBuf, chunk_size: usize, tx: mpsc::Sender<Chunk>) { + // println!("split_file {}", filename.display()); + let mut file = BufReader::new(File::open(&*filename).await.unwrap()); + + let mut offset = 0; + loop { + let mut data = vec![0; chunk_size]; + let n = file.read(&mut data).await.unwrap(); + if n == 0 { + break; + } + let data: Vec<u8> = data[..n].to_vec(); + + let chunk = Chunk { + filename: filename.clone(), + offset, + data, + }; + tx.send(chunk).await.unwrap(); + // println!("split_file sent chunk at offset {}", offset); + + offset += n as u64; + } + // println!("split_file EOF at {}", offset); +} + +fn just_hash(chunk: Chunk) -> Checksum { + let mut hasher = Sha256::new(); + hasher.update(&chunk.data); + let hash = hasher.finalize(); + let hash = format!("{:x}", hash); + Checksum { + filename: chunk.filename, + offset: chunk.offset, + len: chunk.data.len() as u64, + checksum: hash, + } +} diff --git a/src/cmd/gen_info.rs b/src/cmd/gen_info.rs new file mode 100644 index 0000000..901a0ae --- /dev/null +++ b/src/cmd/gen_info.rs @@ -0,0 +1,47 @@ +//! The `gen-info` subcommand. + +use crate::chunk::ClientTrust; +use crate::client::BackupClient; +use crate::config::ClientConfig; +use crate::error::ObnamError; +use clap::Parser; +use log::info; +use tempfile::NamedTempFile; +use tokio::runtime::Runtime; + +/// Show metadata for a generation. +#[derive(Debug, Parser)] +pub struct GenInfo { + /// Reference of the generation. + gen_ref: String, +} + +impl GenInfo { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } + + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let temp = NamedTempFile::new()?; + + let client = BackupClient::new(config)?; + + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, "".to_string(), vec![]))) + .unwrap(); + + let genlist = client.list_generations(&trust); + let gen_id = genlist.resolve(&self.gen_ref)?; + info!("generation id is {}", gen_id.as_chunk_id()); + + let gen = client.fetch_generation(&gen_id, temp.path()).await?; + let meta = gen.meta()?; + println!("{}", serde_json::to_string_pretty(&meta)?); + + Ok(()) + } +} diff --git a/src/cmd/get_chunk.rs b/src/cmd/get_chunk.rs index bf653ff..1561492 100644 --- a/src/cmd/get_chunk.rs +++ b/src/cmd/get_chunk.rs @@ -1,15 +1,34 @@ +//! The `get-chunk` subcommand. + use crate::chunkid::ChunkId; use crate::client::BackupClient; -use crate::client::ClientConfig; +use crate::config::ClientConfig; +use crate::error::ObnamError; +use clap::Parser; use std::io::{stdout, Write}; +use tokio::runtime::Runtime; + +/// Fetch a chunk from the server. +#[derive(Debug, Parser)] +pub struct GetChunk { + /// Identifier of chunk to fetch. + chunk_id: String, +} -pub fn get_chunk(config: &ClientConfig, chunk_id: &str) -> anyhow::Result<()> { - let client = BackupClient::new(&config.server_url)?; - let chunk_id: ChunkId = chunk_id.parse().unwrap(); - let chunk = client.fetch_chunk(&chunk_id)?; +impl GetChunk { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } - let stdout = stdout(); - let mut handle = stdout.lock(); - handle.write_all(chunk.data())?; - Ok(()) + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let client = BackupClient::new(config)?; + let chunk_id: ChunkId = self.chunk_id.parse().unwrap(); + let chunk = client.fetch_chunk(&chunk_id).await?; + let stdout = stdout(); + let mut handle = stdout.lock(); + handle.write_all(chunk.data())?; + Ok(()) + } } diff --git a/src/cmd/init.rs b/src/cmd/init.rs new file mode 100644 index 0000000..5950fbb --- /dev/null +++ b/src/cmd/init.rs @@ -0,0 +1,33 @@ +//! The `init` subcommand. + +use crate::config::ClientConfig; +use crate::error::ObnamError; +use crate::passwords::{passwords_filename, Passwords}; +use clap::Parser; + +const PROMPT: &str = "Obnam passphrase: "; + +/// Initialize client by setting passwords. +#[derive(Debug, Parser)] +pub struct Init { + /// Only for testing. + #[clap(long)] + insecure_passphrase: Option<String>, +} + +impl Init { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let passphrase = match &self.insecure_passphrase { + Some(x) => x.to_string(), + None => rpassword::prompt_password(PROMPT).unwrap(), + }; + + let passwords = Passwords::new(&passphrase); + let filename = passwords_filename(&config.filename); + passwords + .save(&filename) + .map_err(|err| ObnamError::PasswordSave(filename, err))?; + Ok(()) + } +} diff --git a/src/cmd/inspect.rs b/src/cmd/inspect.rs new file mode 100644 index 0000000..3b41075 --- /dev/null +++ b/src/cmd/inspect.rs @@ -0,0 +1,46 @@ +//! The `inspect` subcommand. + +use crate::backup_run::current_timestamp; +use crate::chunk::ClientTrust; +use crate::client::BackupClient; +use crate::config::ClientConfig; +use crate::error::ObnamError; + +use clap::Parser; +use log::info; +use tempfile::NamedTempFile; +use tokio::runtime::Runtime; + +/// Make a backup. +#[derive(Debug, Parser)] +pub struct Inspect { + /// Reference to generation to inspect. + gen_id: String, +} + +impl Inspect { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } + + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let temp = NamedTempFile::new()?; + let client = BackupClient::new(config)?; + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, current_timestamp(), vec![]))) + .unwrap(); + let genlist = client.list_generations(&trust); + let gen_id = genlist.resolve(&self.gen_id)?; + info!("generation id is {}", gen_id.as_chunk_id()); + + let gen = client.fetch_generation(&gen_id, temp.path()).await?; + let meta = gen.meta()?; + println!("schema_version: {}", meta.schema_version()); + + Ok(()) + } +} diff --git a/src/cmd/list.rs b/src/cmd/list.rs index 8766e34..8bc6978 100644 --- a/src/cmd/list.rs +++ b/src/cmd/list.rs @@ -1,12 +1,36 @@ -use crate::client::{BackupClient, ClientConfig}; +//! The `list` subcommand. -pub fn list(config: &ClientConfig) -> anyhow::Result<()> { - let client = BackupClient::new(&config.server_url)?; +use crate::chunk::ClientTrust; +use crate::client::BackupClient; +use crate::config::ClientConfig; +use crate::error::ObnamError; +use clap::Parser; +use tokio::runtime::Runtime; - let generations = client.list_generations()?; - for finished in generations.iter() { - println!("{} {}", finished.id(), finished.ended()); +/// List generations on the server. +#[derive(Debug, Parser)] +pub struct List {} + +impl List { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) } - Ok(()) + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let client = BackupClient::new(config)?; + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, "".to_string(), vec![]))) + .unwrap(); + + let generations = client.list_generations(&trust); + for finished in generations.iter() { + println!("{} {}", finished.id(), finished.ended()); + } + + Ok(()) + } } diff --git a/src/cmd/list_backup_versions.rs b/src/cmd/list_backup_versions.rs new file mode 100644 index 0000000..c78ccfc --- /dev/null +++ b/src/cmd/list_backup_versions.rs @@ -0,0 +1,31 @@ +//! The `backup` subcommand. + +use crate::config::ClientConfig; +use crate::dbgen::{schema_version, DEFAULT_SCHEMA_MAJOR, SCHEMA_MAJORS}; +use crate::error::ObnamError; + +use clap::Parser; + +/// List supported backup schema versions. +#[derive(Debug, Parser)] +pub struct ListSchemaVersions { + /// List only the default version. + #[clap(long)] + default_only: bool, +} + +impl ListSchemaVersions { + /// Run the command. + pub fn run(&self, _config: &ClientConfig) -> Result<(), ObnamError> { + if self.default_only { + let schema = schema_version(DEFAULT_SCHEMA_MAJOR)?; + println!("{}", schema); + } else { + for major in SCHEMA_MAJORS { + let schema = schema_version(*major)?; + println!("{}", schema); + } + } + Ok(()) + } +} diff --git a/src/cmd/list_files.rs b/src/cmd/list_files.rs index a69c3df..e8276cd 100644 --- a/src/cmd/list_files.rs +++ b/src/cmd/list_files.rs @@ -1,36 +1,51 @@ +//! The `list-files` subcommand. + use crate::backup_reason::Reason; +use crate::chunk::ClientTrust; use crate::client::BackupClient; -use crate::client::ClientConfig; +use crate::config::ClientConfig; use crate::error::ObnamError; use crate::fsentry::{FilesystemEntry, FilesystemKind}; +use clap::Parser; use tempfile::NamedTempFile; +use tokio::runtime::Runtime; -pub fn list_files(config: &ClientConfig, gen_ref: &str) -> anyhow::Result<()> { - // Create a named temporary file. We don't meed the open file - // handle, so we discard that. - let dbname = { - let temp = NamedTempFile::new()?; - let (_, dbname) = temp.keep()?; - dbname - }; +/// List files in a backup. +#[derive(Debug, Parser)] +pub struct ListFiles { + /// Reference to backup to list files in. + #[clap(default_value = "latest")] + gen_id: String, +} + +impl ListFiles { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } - let client = BackupClient::new(&config.server_url)?; + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let temp = NamedTempFile::new()?; - let genlist = client.list_generations()?; - let gen_id: String = match genlist.resolve(gen_ref) { - None => return Err(ObnamError::UnknownGeneration(gen_ref.to_string()).into()), - Some(id) => id, - }; + let client = BackupClient::new(config)?; + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, "".to_string(), vec![]))) + .unwrap(); - let gen = client.fetch_generation(&gen_id, &dbname)?; - for file in gen.files()? { - println!("{}", format_entry(&file.entry(), file.reason())); - } + let genlist = client.list_generations(&trust); + let gen_id = genlist.resolve(&self.gen_id)?; - // Delete the temporary file. - std::fs::remove_file(&dbname)?; + let gen = client.fetch_generation(&gen_id, temp.path()).await?; + for file in gen.files()?.iter()? { + let (_, entry, reason, _) = file?; + println!("{}", format_entry(&entry, reason)); + } - Ok(()) + Ok(()) + } } fn format_entry(e: &FilesystemEntry, reason: Reason) -> String { @@ -38,6 +53,8 @@ fn format_entry(e: &FilesystemEntry, reason: Reason) -> String { FilesystemKind::Regular => "-", FilesystemKind::Directory => "d", FilesystemKind::Symlink => "l", + FilesystemKind::Socket => "s", + FilesystemKind::Fifo => "p", }; format!("{} {} ({})", kind, e.pathbuf().display(), reason) } diff --git a/src/cmd/mod.rs b/src/cmd/mod.rs index 8f08668..af7457b 100644 --- a/src/cmd/mod.rs +++ b/src/cmd/mod.rs @@ -1,17 +1,16 @@ -mod backup; -pub use backup::backup; - -mod list; -pub use list::list; - -mod list_files; -pub use list_files::list_files; - -pub mod restore; -pub use restore::restore; +//! Subcommand implementations. +pub mod backup; +pub mod chunk; +pub mod chunkify; +pub mod gen_info; pub mod get_chunk; -pub use get_chunk::get_chunk; - +pub mod init; +pub mod inspect; +pub mod list; +pub mod list_backup_versions; +pub mod list_files; +pub mod resolve; +pub mod restore; +pub mod show_config; pub mod show_gen; -pub use show_gen::show_generation; diff --git a/src/cmd/resolve.rs b/src/cmd/resolve.rs new file mode 100644 index 0000000..a7774d7 --- /dev/null +++ b/src/cmd/resolve.rs @@ -0,0 +1,44 @@ +//! The `resolve` subcommand. + +use crate::chunk::ClientTrust; +use crate::client::BackupClient; +use crate::config::ClientConfig; +use crate::error::ObnamError; +use clap::Parser; +use tokio::runtime::Runtime; + +/// Resolve a generation reference into a generation id. +#[derive(Debug, Parser)] +pub struct Resolve { + /// The generation reference. + generation: String, +} + +impl Resolve { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } + + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let client = BackupClient::new(config)?; + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, "".to_string(), vec![]))) + .unwrap(); + let generations = client.list_generations(&trust); + + match generations.resolve(&self.generation) { + Err(err) => { + return Err(err.into()); + } + Ok(gen_id) => { + println!("{}", gen_id.as_chunk_id()); + } + }; + + Ok(()) + } +} diff --git a/src/cmd/restore.rs b/src/cmd/restore.rs index d783a70..58caf61 100644 --- a/src/cmd/restore.rs +++ b/src/cmd/restore.rs @@ -1,101 +1,165 @@ -use crate::client::BackupClient; -use crate::client::ClientConfig; +//! The `restore` subcommand. + +use crate::backup_reason::Reason; +use crate::chunk::ClientTrust; +use crate::client::{BackupClient, ClientError}; +use crate::config::ClientConfig; +use crate::db::DatabaseError; +use crate::dbgen::FileId; use crate::error::ObnamError; use crate::fsentry::{FilesystemEntry, FilesystemKind}; -use crate::generation::LocalGeneration; +use crate::generation::{LocalGeneration, LocalGenerationError}; +use clap::Parser; use indicatif::{ProgressBar, ProgressStyle}; -use libc::{fchmod, futimens, timespec}; +use libc::{chmod, mkfifo, timespec, utimensat, AT_FDCWD, AT_SYMLINK_NOFOLLOW}; use log::{debug, error, info}; -use std::fs::File; +use std::ffi::CString; use std::io::prelude::*; use std::io::Error; +use std::os::unix::ffi::OsStrExt; use std::os::unix::fs::symlink; -use std::os::unix::io::AsRawFd; +use std::os::unix::net::UnixListener; +use std::path::StripPrefixError; use std::path::{Path, PathBuf}; -use structopt::StructOpt; use tempfile::NamedTempFile; +use tokio::runtime::Runtime; + +/// Restore a backup. +#[derive(Debug, Parser)] +pub struct Restore { + /// Reference to generation to restore. + gen_id: String, + + /// Path to directory where restored files are written. + to: PathBuf, +} + +impl Restore { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } -pub fn restore(config: &ClientConfig, gen_ref: &str, to: &Path) -> anyhow::Result<()> { - // Create a named temporary file. We don't meed the open file - // handle, so we discard that. - let dbname = { + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { let temp = NamedTempFile::new()?; - let (_, dbname) = temp.keep()?; - dbname - }; - let client = BackupClient::new(&config.server_url)?; + let client = BackupClient::new(config)?; + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, "".to_string(), vec![]))) + .unwrap(); - let genlist = client.list_generations()?; - let gen_id: String = match genlist.resolve(gen_ref) { - None => return Err(ObnamError::UnknownGeneration(gen_ref.to_string()).into()), - Some(id) => id, - }; - info!("generation id is {}", gen_id); + let genlist = client.list_generations(&trust); + let gen_id = genlist.resolve(&self.gen_id)?; + info!("generation id is {}", gen_id.as_chunk_id()); - let gen = client.fetch_generation(&gen_id, &dbname)?; - info!("restoring {} files", gen.file_count()?); - let progress = create_progress_bar(gen.file_count()?, true); - for file in gen.files()? { - restore_generation(&client, &gen, file.fileno(), file.entry(), &to, &progress)?; - } - for file in gen.files()? { - if file.entry().is_dir() { - restore_directory_metadata(file.entry(), &to)?; + let gen = client.fetch_generation(&gen_id, temp.path()).await?; + info!("restoring {} files", gen.file_count()?); + let progress = create_progress_bar(gen.file_count()?, true); + for file in gen.files()?.iter()? { + let (fileno, entry, reason, _) = file?; + match reason { + Reason::FileError => (), + _ => restore_generation(&client, &gen, fileno, &entry, &self.to, &progress).await?, + } + } + for file in gen.files()?.iter()? { + let (_, entry, _, _) = file?; + if entry.is_dir() { + restore_directory_metadata(&entry, &self.to)?; + } } + progress.finish(); + + Ok(()) } - progress.finish(); +} - // Delete the temporary file. - std::fs::remove_file(&dbname)?; +/// Possible errors from restoring. +#[derive(Debug, thiserror::Error)] +pub enum RestoreError { + /// An error using a Database. + #[error(transparent)] + Database(#[from] DatabaseError), - Ok(()) -} + /// Failed to create a name pipe. + #[error("Could not create named pipe (FIFO) {0}")] + NamedPipeCreationError(PathBuf), -#[derive(Debug, StructOpt)] -#[structopt(name = "obnam-backup", about = "Simplistic backup client")] -struct Opt { - #[structopt(parse(from_os_str))] - config: PathBuf, + /// Error from HTTP client. + #[error(transparent)] + ClientError(#[from] ClientError), - #[structopt()] - gen_id: String, + /// Error from local generation. + #[error(transparent)] + LocalGenerationError(#[from] LocalGenerationError), - #[structopt(parse(from_os_str))] - dbname: PathBuf, + /// Error removing a prefix. + #[error(transparent)] + StripPrefixError(#[from] StripPrefixError), - #[structopt(parse(from_os_str))] - to: PathBuf, + /// Error creating a directory. + #[error("failed to create directory {0}: {1}")] + CreateDirs(PathBuf, std::io::Error), + + /// Error creating a file. + #[error("failed to create file {0}: {1}")] + CreateFile(PathBuf, std::io::Error), + + /// Error writing a file. + #[error("failed to write file {0}: {1}")] + WriteFile(PathBuf, std::io::Error), + + /// Error creating a symbolic link. + #[error("failed to create symbolic link {0}: {1}")] + Symlink(PathBuf, std::io::Error), + + /// Error creating a UNIX domain socket. + #[error("failed to create UNIX domain socket {0}: {1}")] + UnixBind(PathBuf, std::io::Error), + + /// Error setting permissions. + #[error("failed to set permissions for {0}: {1}")] + Chmod(PathBuf, std::io::Error), + + /// Error settting timestamp. + #[error("failed to set timestamp for {0}: {1}")] + SetTimestamp(PathBuf, std::io::Error), } -fn restore_generation( +async fn restore_generation( client: &BackupClient, gen: &LocalGeneration, - fileid: i64, + fileid: FileId, entry: &FilesystemEntry, to: &Path, progress: &ProgressBar, -) -> anyhow::Result<()> { +) -> Result<(), RestoreError> { info!("restoring {:?}", entry); - progress.set_message(&format!("{}", entry.pathbuf().display())); + progress.set_message(format!("{}", entry.pathbuf().display())); progress.inc(1); let to = restored_path(entry, to)?; match entry.kind() { - FilesystemKind::Regular => restore_regular(client, &gen, &to, fileid, &entry)?, + FilesystemKind::Regular => restore_regular(client, gen, &to, fileid, entry).await?, FilesystemKind::Directory => restore_directory(&to)?, - FilesystemKind::Symlink => restore_symlink(&to, &entry)?, + FilesystemKind::Symlink => restore_symlink(&to, entry)?, + FilesystemKind::Socket => restore_socket(&to, entry)?, + FilesystemKind::Fifo => restore_fifo(&to, entry)?, } Ok(()) } -fn restore_directory(path: &Path) -> anyhow::Result<()> { +fn restore_directory(path: &Path) -> Result<(), RestoreError> { debug!("restoring directory {}", path.display()); - std::fs::create_dir_all(path)?; + std::fs::create_dir_all(path) + .map_err(|err| RestoreError::CreateDirs(path.to_path_buf(), err))?; Ok(()) } -fn restore_directory_metadata(entry: &FilesystemEntry, to: &Path) -> anyhow::Result<()> { +fn restore_directory_metadata(entry: &FilesystemEntry, to: &Path) -> Result<(), RestoreError> { let to = restored_path(entry, to)?; match entry.kind() { FilesystemKind::Directory => restore_metadata(&to, entry)?, @@ -107,7 +171,7 @@ fn restore_directory_metadata(entry: &FilesystemEntry, to: &Path) -> anyhow::Res Ok(()) } -fn restored_path(entry: &FilesystemEntry, to: &Path) -> anyhow::Result<PathBuf> { +fn restored_path(entry: &FilesystemEntry, to: &Path) -> Result<PathBuf, RestoreError> { let path = &entry.pathbuf(); let path = if path.is_absolute() { path.strip_prefix("/")? @@ -117,22 +181,26 @@ fn restored_path(entry: &FilesystemEntry, to: &Path) -> anyhow::Result<PathBuf> Ok(to.join(path)) } -fn restore_regular( +async fn restore_regular( client: &BackupClient, gen: &LocalGeneration, path: &Path, - fileid: i64, + fileid: FileId, entry: &FilesystemEntry, -) -> anyhow::Result<()> { +) -> Result<(), RestoreError> { debug!("restoring regular {}", path.display()); let parent = path.parent().unwrap(); debug!(" mkdir {}", parent.display()); - std::fs::create_dir_all(parent)?; + std::fs::create_dir_all(parent) + .map_err(|err| RestoreError::CreateDirs(parent.to_path_buf(), err))?; { - let mut file = std::fs::File::create(path)?; - for chunkid in gen.chunkids(fileid)? { - let chunk = client.fetch_chunk(&chunkid)?; - file.write_all(chunk.data())?; + let mut file = std::fs::File::create(path) + .map_err(|err| RestoreError::CreateFile(path.to_path_buf(), err))?; + for chunkid in gen.chunkids(fileid)?.iter()? { + let chunkid = chunkid?; + let chunk = client.fetch_chunk(&chunkid).await?; + file.write_all(chunk.data()) + .map_err(|err| RestoreError::WriteFile(path.to_path_buf(), err))?; } restore_metadata(path, entry)?; } @@ -140,24 +208,44 @@ fn restore_regular( Ok(()) } -fn restore_symlink(path: &Path, entry: &FilesystemEntry) -> anyhow::Result<()> { +fn restore_symlink(path: &Path, entry: &FilesystemEntry) -> Result<(), RestoreError> { debug!("restoring symlink {}", path.display()); let parent = path.parent().unwrap(); debug!(" mkdir {}", parent.display()); if !parent.exists() { - std::fs::create_dir_all(parent)?; - { - symlink(path, entry.symlink_target().unwrap())?; + std::fs::create_dir_all(parent) + .map_err(|err| RestoreError::CreateDirs(parent.to_path_buf(), err))?; + } + symlink(entry.symlink_target().unwrap(), path) + .map_err(|err| RestoreError::Symlink(path.to_path_buf(), err))?; + restore_metadata(path, entry)?; + debug!("restored symlink {}", path.display()); + Ok(()) +} + +fn restore_socket(path: &Path, entry: &FilesystemEntry) -> Result<(), RestoreError> { + debug!("creating Unix domain socket {:?}", path); + UnixListener::bind(path).map_err(|err| RestoreError::UnixBind(path.to_path_buf(), err))?; + restore_metadata(path, entry)?; + Ok(()) +} + +fn restore_fifo(path: &Path, entry: &FilesystemEntry) -> Result<(), RestoreError> { + debug!("creating fifo {:?}", path); + let filename = path_to_cstring(path); + match unsafe { mkfifo(filename.as_ptr(), 0) } { + -1 => { + return Err(RestoreError::NamedPipeCreationError(path.to_path_buf())); } + _ => restore_metadata(path, entry)?, } - debug!("restored regular {}", path.display()); Ok(()) } -fn restore_metadata(path: &Path, entry: &FilesystemEntry) -> anyhow::Result<()> { +fn restore_metadata(path: &Path, entry: &FilesystemEntry) -> Result<(), RestoreError> { debug!("restoring metadata for {}", entry.pathbuf().display()); - let handle = File::open(path)?; + debug!("restoring metadata for {:?}", path); let atime = timespec { tv_sec: entry.atime(), @@ -170,41 +258,58 @@ fn restore_metadata(path: &Path, entry: &FilesystemEntry) -> anyhow::Result<()> let times = [atime, mtime]; let times: *const timespec = ×[0]; + let pathbuf = path.to_path_buf(); + let path = path_to_cstring(path); + // We have to use unsafe here to be able call the libc functions // below. unsafe { - let fd = handle.as_raw_fd(); // FIXME: needs to NOT follow symlinks - - debug!("fchmod"); - if fchmod(fd, entry.mode()) == -1 { - let error = Error::last_os_error(); - error!("fchmod failed on {}", path.display()); - return Err(error.into()); + if entry.kind() != FilesystemKind::Symlink { + debug!("chmod {:?}", path); + if chmod(path.as_ptr(), entry.mode() as libc::mode_t) == -1 { + let error = Error::last_os_error(); + error!("chmod failed on {:?}", path); + return Err(RestoreError::Chmod(pathbuf, error)); + } + } else { + debug!( + "skipping chmod of a symlink because it'll attempt to change the pointed-at file" + ); } - debug!("futimens"); - if futimens(fd, times) == -1 { + debug!("utimens {:?}", path); + if utimensat(AT_FDCWD, path.as_ptr(), times, AT_SYMLINK_NOFOLLOW) == -1 { let error = Error::last_os_error(); - error!("futimens failed on {}", path.display()); - return Err(error.into()); + error!("utimensat failed on {:?}", path); + return Err(RestoreError::SetTimestamp(pathbuf, error)); } } Ok(()) } -fn create_progress_bar(file_count: i64, verbose: bool) -> ProgressBar { +fn path_to_cstring(path: &Path) -> CString { + let path = path.as_os_str(); + let path = path.as_bytes(); + CString::new(path).unwrap() +} + +fn create_progress_bar(file_count: FileId, verbose: bool) -> ProgressBar { let progress = if verbose { ProgressBar::new(file_count as u64) } else { ProgressBar::hidden() }; - let parts = vec![ + let parts = [ "{wide_bar}", "elapsed: {elapsed}", "files: {pos}/{len}", "current: {wide_msg}", "{spinner}", ]; - progress.set_style(ProgressStyle::default_bar().template(&parts.join("\n"))); + progress.set_style( + ProgressStyle::default_bar() + .template(&parts.join("\n")) + .expect("create indicatif ProgressStyle value"), + ); progress } diff --git a/src/cmd/show_config.rs b/src/cmd/show_config.rs new file mode 100644 index 0000000..8e0ce30 --- /dev/null +++ b/src/cmd/show_config.rs @@ -0,0 +1,17 @@ +//! The `show-config` subcommand. + +use crate::config::ClientConfig; +use crate::error::ObnamError; +use clap::Parser; + +/// Show actual client configuration. +#[derive(Debug, Parser)] +pub struct ShowConfig {} + +impl ShowConfig { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + println!("{}", serde_json::to_string_pretty(config)?); + Ok(()) + } +} diff --git a/src/cmd/show_gen.rs b/src/cmd/show_gen.rs index d355389..95d3fd3 100644 --- a/src/cmd/show_gen.rs +++ b/src/cmd/show_gen.rs @@ -1,46 +1,101 @@ +//! The `show-generation` subcommand. + +use crate::chunk::ClientTrust; use crate::client::BackupClient; -use crate::client::ClientConfig; +use crate::config::ClientConfig; +use crate::db::DbInt; use crate::error::ObnamError; use crate::fsentry::FilesystemKind; +use crate::generation::GenId; +use clap::Parser; use indicatif::HumanBytes; +use serde::Serialize; use tempfile::NamedTempFile; +use tokio::runtime::Runtime; + +/// Show information about a generation. +#[derive(Debug, Parser)] +pub struct ShowGeneration { + /// Reference to the generation. Defaults to latest. + #[clap(default_value = "latest")] + gen_id: String, +} -pub fn show_generation(config: &ClientConfig, gen_ref: &str) -> anyhow::Result<()> { - // Create a named temporary file. We don't meed the open file - // handle, so we discard that. - let dbname = { +impl ShowGeneration { + /// Run the command. + pub fn run(&self, config: &ClientConfig) -> Result<(), ObnamError> { + let rt = Runtime::new()?; + rt.block_on(self.run_async(config)) + } + + async fn run_async(&self, config: &ClientConfig) -> Result<(), ObnamError> { let temp = NamedTempFile::new()?; - let (_, dbname) = temp.keep()?; - dbname - }; - - let client = BackupClient::new(&config.server_url)?; - - let genlist = client.list_generations()?; - let gen_id: String = match genlist.resolve(gen_ref) { - None => return Err(ObnamError::UnknownGeneration(gen_ref.to_string()).into()), - Some(id) => id, - }; - - let gen = client.fetch_generation(&gen_id, &dbname)?; - let files = gen.files()?; - - let total_bytes = files.iter().fold(0, |acc, file| { - let e = file.entry(); - if e.kind() == FilesystemKind::Regular { - acc + file.entry().len() - } else { - acc + let client = BackupClient::new(config)?; + let trust = client + .get_client_trust() + .await? + .or_else(|| Some(ClientTrust::new("FIXME", None, "".to_string(), vec![]))) + .unwrap(); + + let genlist = client.list_generations(&trust); + let gen_id = genlist.resolve(&self.gen_id)?; + let gen = client.fetch_generation(&gen_id, temp.path()).await?; + let mut files = gen.files()?; + let mut files = files.iter()?; + + let total_bytes = files.try_fold(0, |acc, file| { + file.map(|(_, e, _, _)| { + if e.kind() == FilesystemKind::Regular { + acc + e.len() + } else { + acc + } + }) + }); + let total_bytes = total_bytes?; + + let output = Output::new(gen_id) + .db_bytes(temp.path().metadata()?.len()) + .file_count(gen.file_count()?) + .file_bytes(total_bytes); + serde_json::to_writer_pretty(std::io::stdout(), &output)?; + + Ok(()) + } +} + +#[derive(Debug, Default, Serialize)] +struct Output { + generation_id: String, + file_count: DbInt, + file_bytes: String, + file_bytes_raw: u64, + db_bytes: String, + db_bytes_raw: u64, +} + +impl Output { + fn new(gen_id: GenId) -> Self { + Self { + generation_id: format!("{}", gen_id), + ..Self::default() } - }); + } - println!("generation-id: {}", gen_id); - println!("file-count: {}", gen.file_count()?); - println!("file-bytes: {}", HumanBytes(total_bytes)); - println!("file-bytes-raw: {}", total_bytes); + fn file_count(mut self, n: DbInt) -> Self { + self.file_count = n; + self + } - // Delete the temporary file. - std::fs::remove_file(&dbname)?; + fn file_bytes(mut self, n: u64) -> Self { + self.file_bytes_raw = n; + self.file_bytes = HumanBytes(n).to_string(); + self + } - Ok(()) + fn db_bytes(mut self, n: u64) -> Self { + self.db_bytes_raw = n; + self.db_bytes = HumanBytes(n).to_string(); + self + } } diff --git a/src/config.rs b/src/config.rs new file mode 100644 index 0000000..5774aad --- /dev/null +++ b/src/config.rs @@ -0,0 +1,142 @@ +//! Client configuration. + +use crate::passwords::{passwords_filename, PasswordError, Passwords}; + +use bytesize::MIB; +use log::{error, trace}; +use serde::{Deserialize, Serialize}; +use std::path::{Path, PathBuf}; + +const DEFAULT_CHUNK_SIZE: usize = MIB as usize; +const DEVNULL: &str = "/dev/null"; + +#[derive(Debug, Deserialize, Clone)] +#[serde(deny_unknown_fields)] +struct TentativeClientConfig { + server_url: String, + verify_tls_cert: Option<bool>, + chunk_size: Option<usize>, + roots: Vec<PathBuf>, + log: Option<PathBuf>, + exclude_cache_tag_directories: Option<bool>, +} + +/// Configuration for the Obnam client. +#[derive(Debug, Serialize, Clone)] +pub struct ClientConfig { + /// Name of configuration file. + pub filename: PathBuf, + /// URL of Obnam server. + pub server_url: String, + /// Should server's TLS certificate be verified using CA + /// signatures? Set to false, for self-signed certificates. + pub verify_tls_cert: bool, + /// Size of chunks when splitting files for backup. + pub chunk_size: usize, + /// Backup root directories. + pub roots: Vec<PathBuf>, + /// File where logs should be written. + pub log: PathBuf, + /// Should cache directories be excluded? Cache directories + /// contain a specially formatted CACHEDIR.TAG file. + pub exclude_cache_tag_directories: bool, +} + +impl ClientConfig { + /// Read a client configuration from a file. + pub fn read(filename: &Path) -> Result<Self, ClientConfigError> { + trace!("read_config: filename={:?}", filename); + let config = std::fs::read_to_string(filename) + .map_err(|err| ClientConfigError::Read(filename.to_path_buf(), err))?; + let tentative: TentativeClientConfig = serde_yaml::from_str(&config) + .map_err(|err| ClientConfigError::YamlParse(filename.to_path_buf(), err))?; + let roots = tentative + .roots + .iter() + .map(|path| expand_tilde(path)) + .collect(); + let log = tentative + .log + .map(|path| expand_tilde(&path)) + .unwrap_or_else(|| PathBuf::from(DEVNULL)); + let exclude_cache_tag_directories = tentative.exclude_cache_tag_directories.unwrap_or(true); + + let config = Self { + chunk_size: tentative.chunk_size.unwrap_or(DEFAULT_CHUNK_SIZE), + filename: filename.to_path_buf(), + roots, + server_url: tentative.server_url, + verify_tls_cert: tentative.verify_tls_cert.unwrap_or(false), + log, + exclude_cache_tag_directories, + }; + + config.check()?; + Ok(config) + } + + fn check(&self) -> Result<(), ClientConfigError> { + if self.server_url.is_empty() { + return Err(ClientConfigError::ServerUrlIsEmpty); + } + if !self.server_url.starts_with("https://") { + return Err(ClientConfigError::NotHttps(self.server_url.to_string())); + } + if self.roots.is_empty() { + return Err(ClientConfigError::NoBackupRoot); + } + Ok(()) + } + + /// Read encryption passwords from a file. + /// + /// The password file is expected to be next to the configuration file. + pub fn passwords(&self) -> Result<Passwords, ClientConfigError> { + Passwords::load(&passwords_filename(&self.filename)) + .map_err(ClientConfigError::PasswordsMissing) + } +} + +/// Possible errors from configuration files. +#[derive(Debug, thiserror::Error)] +pub enum ClientConfigError { + /// The configuration specifies the server URL as an empty string. + #[error("server_url is empty")] + ServerUrlIsEmpty, + + /// The configuration does not specify any backup root directories. + #[error("No backup roots in config; at least one is needed")] + NoBackupRoot, + + /// The server URL is not an https: one. + #[error("server URL doesn't use https: {0}")] + NotHttps(String), + + /// There are no passwords stored. + #[error("No passwords are set: you may need to run 'obnam init': {0}")] + PasswordsMissing(PasswordError), + + /// Error reading a configuation file. + #[error("failed to read configuration file {0}: {1}")] + Read(PathBuf, std::io::Error), + + /// Error parsing configuration file as YAML. + #[error("failed to parse configuration file {0} as YAML: {1}")] + YamlParse(PathBuf, serde_yaml::Error), +} + +fn expand_tilde(path: &Path) -> PathBuf { + if path.starts_with("~/") { + if let Some(home) = std::env::var_os("HOME") { + let mut expanded = PathBuf::from(home); + for comp in path.components().skip(1) { + expanded.push(comp); + } + expanded + } else { + path.to_path_buf() + } + } else { + path.to_path_buf() + } +} diff --git a/src/db.rs b/src/db.rs new file mode 100644 index 0000000..392134d --- /dev/null +++ b/src/db.rs @@ -0,0 +1,640 @@ +//! A database abstraction around SQLite for Obnam. +//! +//! This abstraction provided the bare minimum that Obnam needs, while +//! trying to be as performant as possible, especially for inserting +//! rows. Only data types needed by Obnam are supported. +//! +//! Note that this abstraction is entirely synchronous. This is for +//! simplicity, as SQLite only allows one write at a time. + +use crate::fsentry::FilesystemEntry; +use rusqlite::{params, types::ToSqlOutput, CachedStatement, Connection, OpenFlags, Row, ToSql}; +use std::collections::HashSet; +use std::convert::TryFrom; +use std::path::{Path, PathBuf}; + +/// A database. +pub struct Database { + conn: Connection, +} + +impl Database { + /// Create a new database file for an empty database. + /// + /// The database can be written to. + pub fn create<P: AsRef<Path>>(filename: P) -> Result<Self, DatabaseError> { + if filename.as_ref().exists() { + return Err(DatabaseError::Exists(filename.as_ref().to_path_buf())); + } + let flags = OpenFlags::SQLITE_OPEN_CREATE | OpenFlags::SQLITE_OPEN_READ_WRITE; + let conn = Connection::open_with_flags(filename, flags)?; + conn.execute("BEGIN", params![])?; + Ok(Self { conn }) + } + + /// Open an existing database file in read only mode. + pub fn open<P: AsRef<Path>>(filename: P) -> Result<Self, DatabaseError> { + let flags = OpenFlags::SQLITE_OPEN_READ_ONLY; + let conn = Connection::open_with_flags(filename, flags)?; + Ok(Self { conn }) + } + + /// Close an open database, committing any changes to disk. + pub fn close(self) -> Result<(), DatabaseError> { + self.conn.execute("COMMIT", params![])?; + self.conn + .close() + .map_err(|(_, err)| DatabaseError::Rusqlite(err))?; + Ok(()) + } + + /// Create a table in the database. + pub fn create_table(&self, table: &Table) -> Result<(), DatabaseError> { + let sql = sql_statement::create_table(table); + self.conn.execute(&sql, params![])?; + Ok(()) + } + + /// Create an index in the database. + pub fn create_index( + &self, + name: &str, + table: &Table, + column: &str, + ) -> Result<(), DatabaseError> { + let sql = sql_statement::create_index(name, table, column); + self.conn.execute(&sql, params![])?; + Ok(()) + } + + /// Insert a row in a table. + pub fn insert(&mut self, table: &Table, values: &[Value]) -> Result<(), DatabaseError> { + let mut stmt = self.conn.prepare_cached(table.insert())?; + assert!(table.has_columns(values)); + // The ToSql trait implementation for Obnam values can't ever + // fail, so we don't handle the error case in the parameter + // creation below. + stmt.execute(rusqlite::params_from_iter(values.iter().map(|v| { + v.to_sql() + .expect("conversion of Obnam value to SQLite value failed unexpectedly") + })))?; + Ok(()) + } + + /// Return an iterator for all rows in a table. + pub fn all_rows<T>( + &self, + table: &Table, + rowfunc: &'static dyn Fn(&Row) -> Result<T, rusqlite::Error>, + ) -> Result<SqlResults<T>, DatabaseError> { + let sql = sql_statement::select_all_rows(table); + SqlResults::new( + &self.conn, + &sql, + None, + Box::new(|stmt, _| { + let iter = stmt.query_map(params![], |row| rowfunc(row))?; + let iter = iter.map(|x| match x { + Ok(t) => Ok(t), + Err(e) => Err(DatabaseError::Rusqlite(e)), + }); + Ok(Box::new(iter)) + }), + ) + } + + /// Return rows that have a given value in a given column. + /// + /// This is simplistic, but for Obnam, it provides all the SQL + /// SELECT ... WHERE that's needed, and there's no point in making + /// this more generic than is needed. + pub fn some_rows<T>( + &self, + table: &Table, + value: &Value, + rowfunc: &'static dyn Fn(&Row) -> Result<T, rusqlite::Error>, + ) -> Result<SqlResults<T>, DatabaseError> { + assert!(table.has_column(value)); + let sql = sql_statement::select_some_rows(table, value.name()); + SqlResults::new( + &self.conn, + &sql, + Some(OwnedValue::from(value)), + Box::new(|stmt, value| { + let iter = stmt.query_map(params![value], |row| rowfunc(row))?; + let iter = iter.map(|x| match x { + Ok(t) => Ok(t), + Err(e) => Err(DatabaseError::Rusqlite(e)), + }); + Ok(Box::new(iter)) + }), + ) + } +} + +/// Possible errors from a database. +#[derive(Debug, thiserror::Error)] +pub enum DatabaseError { + /// An error from the rusqlite crate. + #[error(transparent)] + Rusqlite(#[from] rusqlite::Error), + + /// The database being created already exists. + #[error("Database {0} already exists")] + Exists(PathBuf), +} + +// A pointer to a "fallible iterator" over values of type `T`, which is to say it's an iterator +// over values of type `Result<T, DatabaseError>`. The iterator is only valid for the +// lifetime 'stmt. +// +// The fact that it's a pointer (`Box<dyn ...>`) means we don't care what the actual type of +// the iterator is, and who produces it. +type SqlResultsIterator<'stmt, T> = Box<dyn Iterator<Item = Result<T, DatabaseError>> + 'stmt>; + +// A pointer to a function which, when called on a prepared SQLite statement, would create +// a "fallible iterator" over values of type `ItemT`. (See above for an explanation of what +// a "fallible iterator" is.) +// +// The iterator is only valid for the lifetime of the associated SQLite statement; we +// call this lifetime 'stmt, and use it both both on the reference and the returned iterator. +// +// Now we're in a pickle: all named lifetimes have to be declared _somewhere_, but we can't add +// 'stmt to the signature of `CreateIterFn` because then we'll have to specify it when we +// define the function. Obviously, at that point we won't yet have a `Statement`, and thus we +// would have no idea what its lifetime is going to be. So we can't put the 'stmt lifetime into +// the signature of `CreateIterFn`. +// +// That's what `for<'stmt>` is for. This is a so-called ["higher-rank trait bound"][hrtb], and +// it enables us to say that a function is valid for *some* lifetime 'stmt that we pass into it +// at the call site. It lets Rust continue to track lifetimes even though `CreateIterFn` +// interferes by "hiding" the 'stmt lifetime from its signature. +// +// [hrtb]: https://doc.rust-lang.org/nomicon/hrtb.html +type CreateIterFn<'conn, ItemT> = Box< + dyn for<'stmt> Fn( + &'stmt mut CachedStatement<'conn>, + &Option<OwnedValue>, + ) -> Result<SqlResultsIterator<'stmt, ItemT>, DatabaseError>, +>; + +/// An iterator over rows from a query. +pub struct SqlResults<'conn, ItemT> { + stmt: CachedStatement<'conn>, + value: Option<OwnedValue>, + create_iter: CreateIterFn<'conn, ItemT>, +} + +impl<'conn, ItemT> SqlResults<'conn, ItemT> { + fn new( + conn: &'conn Connection, + statement: &str, + value: Option<OwnedValue>, + create_iter: CreateIterFn<'conn, ItemT>, + ) -> Result<Self, DatabaseError> { + let stmt = conn.prepare_cached(statement)?; + Ok(Self { + stmt, + value, + create_iter, + }) + } + + /// Create an iterator over results. + pub fn iter(&'_ mut self) -> Result<SqlResultsIterator<'_, ItemT>, DatabaseError> { + (self.create_iter)(&mut self.stmt, &self.value) + } +} + +/// Describe a table in a row. +pub struct Table { + table: String, + columns: Vec<Column>, + insert: Option<String>, + column_names: HashSet<String>, +} + +impl Table { + /// Create a new table description without columns. + /// + /// The table description is not "built". You must add columns and + /// then call the [`build`] method. + pub fn new(table: &str) -> Self { + Self { + table: table.to_string(), + columns: vec![], + insert: None, + column_names: HashSet::new(), + } + } + + /// Append a column. + pub fn column(mut self, column: Column) -> Self { + self.column_names.insert(column.name().to_string()); + self.columns.push(column); + self + } + + /// Finish building the table description. + pub fn build(mut self) -> Self { + assert!(self.insert.is_none()); + self.insert = Some(sql_statement::insert(&self)); + self + } + + fn has_columns(&self, values: &[Value]) -> bool { + assert!(self.insert.is_some()); + for v in values.iter() { + if !self.column_names.contains(v.name()) { + return false; + } + } + true + } + + fn has_column(&self, value: &Value) -> bool { + assert!(self.insert.is_some()); + self.column_names.contains(value.name()) + } + + fn insert(&self) -> &str { + assert!(self.insert.is_some()); + self.insert.as_ref().unwrap() + } + + /// What is the name of the table? + pub fn name(&self) -> &str { + &self.table + } + + /// How many columns does the table have? + pub fn num_columns(&self) -> usize { + self.columns.len() + } + + /// What are the names of the columns in the table? + pub fn column_names(&self) -> impl Iterator<Item = &str> { + self.columns.iter().map(|c| c.name()) + } + + /// Return SQL column definitions for the table. + pub fn column_definitions(&self) -> String { + let mut ret = String::new(); + for c in self.columns.iter() { + if !ret.is_empty() { + ret.push(','); + } + ret.push_str(c.name()); + ret.push(' '); + ret.push_str(c.typename()); + } + ret + } +} + +/// A column in a table description. +pub enum Column { + /// An integer primary key. + PrimaryKey(String), + /// An integer. + Int(String), + /// A text string. + Text(String), + /// A binary string. + Blob(String), + /// A boolean. + Bool(String), +} + +impl Column { + fn name(&self) -> &str { + match self { + Self::PrimaryKey(name) => name, + Self::Int(name) => name, + Self::Text(name) => name, + Self::Blob(name) => name, + Self::Bool(name) => name, + } + } + + fn typename(&self) -> &str { + match self { + Self::PrimaryKey(_) => "INTEGER PRIMARY KEY", + Self::Int(_) => "INTEGER", + Self::Text(_) => "TEXT", + Self::Blob(_) => "BLOB", + Self::Bool(_) => "BOOLEAN", + } + } + + /// Create an integer primary key column. + pub fn primary_key(name: &str) -> Self { + Self::PrimaryKey(name.to_string()) + } + + /// Create an integer column. + pub fn int(name: &str) -> Self { + Self::Int(name.to_string()) + } + + /// Create a text string column. + pub fn text(name: &str) -> Self { + Self::Text(name.to_string()) + } + + /// Create a binary string column. + pub fn blob(name: &str) -> Self { + Self::Blob(name.to_string()) + } + + /// Create a boolean column. + pub fn bool(name: &str) -> Self { + Self::Bool(name.to_string()) + } +} + +/// Type of plain integers that can be stored. +pub type DbInt = i64; + +/// A value in a named column. +#[derive(Debug)] +pub enum Value<'a> { + /// An integer primary key. + PrimaryKey(&'a str, DbInt), + /// An integer. + Int(&'a str, DbInt), + /// A text string. + Text(&'a str, &'a str), + /// A binary string. + Blob(&'a str, &'a [u8]), + /// A boolean. + Bool(&'a str, bool), +} + +impl<'a> Value<'a> { + /// What column should store this value? + pub fn name(&self) -> &str { + match self { + Self::PrimaryKey(name, _) => name, + Self::Int(name, _) => name, + Self::Text(name, _) => name, + Self::Blob(name, _) => name, + Self::Bool(name, _) => name, + } + } + + /// Create an integer primary key value. + pub fn primary_key(name: &'a str, value: DbInt) -> Self { + Self::PrimaryKey(name, value) + } + + /// Create an integer value. + pub fn int(name: &'a str, value: DbInt) -> Self { + Self::Int(name, value) + } + + /// Create a text string value. + pub fn text(name: &'a str, value: &'a str) -> Self { + Self::Text(name, value) + } + + /// Create a binary string value. + pub fn blob(name: &'a str, value: &'a [u8]) -> Self { + Self::Blob(name, value) + } + + /// Create a boolean value. + pub fn bool(name: &'a str, value: bool) -> Self { + Self::Bool(name, value) + } +} + +#[allow(clippy::useless_conversion)] +impl<'a> ToSql for Value<'a> { + // The trait defines to_sql to return a Result. However, for our + // particular case, to_sql can't ever fail. We only store values + // in types for which conversion always succeeds: integer, + // boolean, text, and blob. _For us_, the caller need never worry + // that the conversion fails, but we can't express that in the + // type system. + fn to_sql(&self) -> Result<rusqlite::types::ToSqlOutput, rusqlite::Error> { + use rusqlite::types::ValueRef; + let v = match self { + Self::PrimaryKey(_, v) => ValueRef::Integer( + i64::try_from(*v) + .map_err(|err| rusqlite::Error::ToSqlConversionFailure(Box::new(err)))?, + ), + Self::Int(_, v) => ValueRef::Integer( + i64::try_from(*v) + .map_err(|err| rusqlite::Error::ToSqlConversionFailure(Box::new(err)))?, + ), + Self::Bool(_, v) => ValueRef::Integer(i64::from(*v)), + Self::Text(_, v) => ValueRef::Text(v.as_ref()), + Self::Blob(_, v) => ValueRef::Blob(v), + }; + Ok(ToSqlOutput::Borrowed(v)) + } +} + +/// Like a Value, but owns the data. +pub enum OwnedValue { + /// An integer primary key. + PrimaryKey(String, DbInt), + /// An integer. + Int(String, DbInt), + /// A text string. + Text(String, String), + /// A binary string. + Blob(String, Vec<u8>), + /// A boolean. + Bool(String, bool), +} + +impl From<&Value<'_>> for OwnedValue { + fn from(v: &Value) -> Self { + match *v { + Value::PrimaryKey(name, v) => Self::PrimaryKey(name.to_string(), v), + Value::Int(name, v) => Self::Int(name.to_string(), v), + Value::Text(name, v) => Self::Text(name.to_string(), v.to_string()), + Value::Blob(name, v) => Self::Blob(name.to_string(), v.to_vec()), + Value::Bool(name, v) => Self::Bool(name.to_string(), v), + } + } +} + +impl ToSql for OwnedValue { + #[allow(clippy::useless_conversion)] + fn to_sql(&self) -> rusqlite::Result<rusqlite::types::ToSqlOutput> { + use rusqlite::types::Value; + let v = match self { + Self::PrimaryKey(_, v) => Value::Integer( + i64::try_from(*v) + .map_err(|err| rusqlite::Error::ToSqlConversionFailure(Box::new(err)))?, + ), + Self::Int(_, v) => Value::Integer( + i64::try_from(*v) + .map_err(|err| rusqlite::Error::ToSqlConversionFailure(Box::new(err)))?, + ), + Self::Bool(_, v) => Value::Integer(i64::from(*v)), + Self::Text(_, v) => Value::Text(v.to_string()), + Self::Blob(_, v) => Value::Blob(v.to_vec()), + }; + Ok(ToSqlOutput::Owned(v)) + } +} + +impl rusqlite::types::ToSql for FilesystemEntry { + fn to_sql(&self) -> rusqlite::Result<ToSqlOutput<'_>> { + let json = serde_json::to_string(self) + .map_err(|err| rusqlite::Error::ToSqlConversionFailure(Box::new(err)))?; + let json = rusqlite::types::Value::Text(json); + Ok(ToSqlOutput::Owned(json)) + } +} + +mod sql_statement { + use super::Table; + + pub fn create_table(table: &Table) -> String { + format!( + "CREATE TABLE {} ({})", + table.name(), + table.column_definitions() + ) + } + + pub fn create_index(name: &str, table: &Table, column: &str) -> String { + format!("CREATE INDEX {} ON {} ({})", name, table.name(), column,) + } + + pub fn insert(table: &Table) -> String { + format!( + "INSERT INTO {} ({}) VALUES ({})", + table.name(), + &column_names(table), + placeholders(table.column_names().count()) + ) + } + + pub fn select_all_rows(table: &Table) -> String { + format!("SELECT * FROM {}", table.name()) + } + + pub fn select_some_rows(table: &Table, column: &str) -> String { + format!("SELECT * FROM {} WHERE {} = ?", table.name(), column) + } + + fn column_names(table: &Table) -> String { + table.column_names().collect::<Vec<&str>>().join(",") + } + + fn placeholders(num_columns: usize) -> String { + let mut s = String::new(); + for _ in 0..num_columns { + if !s.is_empty() { + s.push(','); + } + s.push('?'); + } + s + } +} + +#[cfg(test)] +mod test { + use super::*; + use std::path::Path; + use tempfile::tempdir; + + fn get_bar(row: &rusqlite::Row) -> Result<DbInt, rusqlite::Error> { + row.get("bar") + } + + fn table() -> Table { + Table::new("foo").column(Column::int("bar")).build() + } + + fn create_db(file: &Path) -> Database { + let table = table(); + let db = Database::create(file).unwrap(); + db.create_table(&table).unwrap(); + db + } + + fn open_db(file: &Path) -> Database { + Database::open(file).unwrap() + } + + fn insert(db: &mut Database, value: DbInt) { + let table = table(); + db.insert(&table, &[Value::int("bar", value)]).unwrap(); + } + + fn values(db: Database) -> Vec<DbInt> { + let table = table(); + let mut rows = db.all_rows(&table, &get_bar).unwrap(); + let iter = rows.iter().unwrap(); + let mut values = vec![]; + for x in iter { + values.push(x.unwrap()); + } + values + } + + #[test] + fn creates_db() { + let tmp = tempdir().unwrap(); + let filename = tmp.path().join("test.db"); + let db = Database::create(&filename).unwrap(); + db.close().unwrap(); + let _ = Database::open(&filename).unwrap(); + } + + #[test] + fn inserts_row() { + let tmp = tempdir().unwrap(); + let filename = tmp.path().join("test.db"); + let mut db = create_db(&filename); + insert(&mut db, 42); + db.close().unwrap(); + + let db = open_db(&filename); + let values = values(db); + assert_eq!(values, vec![42]); + } + + #[test] + fn inserts_many_rows() { + const N: DbInt = 1000; + + let tmp = tempdir().unwrap(); + let filename = tmp.path().join("test.db"); + let mut db = create_db(&filename); + for i in 0..N { + insert(&mut db, i); + } + db.close().unwrap(); + + let db = open_db(&filename); + let values = values(db); + assert_eq!(values.len() as DbInt, N); + + let mut expected = vec![]; + for i in 0..N { + expected.push(i); + } + assert_eq!(values, expected); + } + #[test] + fn round_trips_int_max() { + let tmp = tempdir().unwrap(); + let filename = tmp.path().join("test.db"); + let mut db = create_db(&filename); + insert(&mut db, DbInt::MAX); + db.close().unwrap(); + + let db = open_db(&filename); + let values = values(db); + assert_eq!(values, vec![DbInt::MAX]); + } +} diff --git a/src/dbgen.rs b/src/dbgen.rs new file mode 100644 index 0000000..0053d4a --- /dev/null +++ b/src/dbgen.rs @@ -0,0 +1,768 @@ +//! Database abstraction for generations. + +use crate::backup_reason::Reason; +use crate::chunkid::ChunkId; +use crate::db::{Column, Database, DatabaseError, DbInt, SqlResults, Table, Value}; +use crate::fsentry::FilesystemEntry; +use crate::genmeta::{GenerationMeta, GenerationMetaError}; +use crate::label::LabelChecksumKind; +use crate::schema::{SchemaVersion, VersionComponent}; +use log::error; +use std::collections::HashMap; +use std::os::unix::ffi::OsStrExt; +use std::path::{Path, PathBuf}; + +/// Return latest supported schema version for a supported major +/// version. +pub fn schema_version(major: VersionComponent) -> Result<SchemaVersion, GenerationDbError> { + match major { + 0 => Ok(SchemaVersion::new(0, 0)), + 1 => Ok(SchemaVersion::new(1, 0)), + _ => Err(GenerationDbError::Unsupported(major)), + } +} + +/// Default database schema major version.a +pub const DEFAULT_SCHEMA_MAJOR: VersionComponent = V0_0::MAJOR; + +/// Major schema versions supported by this version of Obnam. +pub const SCHEMA_MAJORS: &[VersionComponent] = &[0, 1]; + +/// An integer identifier for a file in a generation. +pub type FileId = DbInt; + +/// Possible errors from using generation databases. +#[derive(Debug, thiserror::Error)] +pub enum GenerationDbError { + /// Duplicate file names. + #[error("Generation has more than one file with the name {0}")] + TooManyFiles(PathBuf), + + /// No 'meta' table in generation. + #[error("Generation does not have a 'meta' table")] + NoMeta, + + /// Missing from from 'meta' table. + #[error("Generation 'meta' table does not have a row {0}")] + NoMetaKey(String), + + /// Bad data in 'meta' table. + #[error("Generation 'meta' row {0} has badly formed integer: {1}")] + BadMetaInteger(String, std::num::ParseIntError), + + /// A major schema version is unsupported. + #[error("Unsupported backup schema major version: {0}")] + Unsupported(VersionComponent), + + /// Local generation uses a schema version that this version of + /// Obnam isn't compatible with. + #[error("Backup is not compatible with this version of Obnam: {0}.{1}")] + Incompatible(VersionComponent, VersionComponent), + + /// Error from a database + #[error(transparent)] + Database(#[from] DatabaseError), + + /// Error from generation metadata. + #[error(transparent)] + GenerationMeta(#[from] GenerationMetaError), + + /// Error from JSON. + #[error(transparent)] + SerdeJsonError(#[from] serde_json::Error), + + /// Error from I/O. + #[error(transparent)] + IoError(#[from] std::io::Error), +} + +/// A database representing a backup generation. +pub struct GenerationDb { + variant: GenerationDbVariant, +} + +enum GenerationDbVariant { + V0_0(V0_0), + V1_0(V1_0), +} + +impl GenerationDb { + /// Create a new generation database in read/write mode. + pub fn create<P: AsRef<Path>>( + filename: P, + schema: SchemaVersion, + checksum_kind: LabelChecksumKind, + ) -> Result<Self, GenerationDbError> { + let meta_table = Self::meta_table(); + let variant = match schema.version() { + (V0_0::MAJOR, V0_0::MINOR) => { + GenerationDbVariant::V0_0(V0_0::create(filename, meta_table, checksum_kind)?) + } + (V1_0::MAJOR, V1_0::MINOR) => { + GenerationDbVariant::V1_0(V1_0::create(filename, meta_table, checksum_kind)?) + } + (major, minor) => return Err(GenerationDbError::Incompatible(major, minor)), + }; + Ok(Self { variant }) + } + + /// Open an existing generation database in read-only mode. + pub fn open<P: AsRef<Path>>(filename: P) -> Result<Self, GenerationDbError> { + let filename = filename.as_ref(); + let meta_table = Self::meta_table(); + let schema = { + let plain_db = Database::open(filename)?; + let rows = Self::meta_rows(&plain_db, &meta_table)?; + GenerationMeta::from(rows)?.schema_version() + }; + let variant = match schema.version() { + (V0_0::MAJOR, V0_0::MINOR) => { + GenerationDbVariant::V0_0(V0_0::open(filename, meta_table)?) + } + (V1_0::MAJOR, V1_0::MINOR) => { + GenerationDbVariant::V1_0(V1_0::open(filename, meta_table)?) + } + (major, minor) => return Err(GenerationDbError::Incompatible(major, minor)), + }; + Ok(Self { variant }) + } + + fn meta_table() -> Table { + Table::new("meta") + .column(Column::text("key")) + .column(Column::text("value")) + .build() + } + + fn meta_rows( + db: &Database, + table: &Table, + ) -> Result<HashMap<String, String>, GenerationDbError> { + let mut map = HashMap::new(); + let mut iter = db.all_rows(table, &row_to_kv)?; + for kv in iter.iter()? { + let (key, value) = kv?; + map.insert(key, value); + } + Ok(map) + } + + /// Close a database, commit any changes. + pub fn close(self) -> Result<(), GenerationDbError> { + match self.variant { + GenerationDbVariant::V0_0(v) => v.close(), + GenerationDbVariant::V1_0(v) => v.close(), + } + } + + /// Return contents of "meta" table as a HashMap. + pub fn meta(&self) -> Result<HashMap<String, String>, GenerationDbError> { + match &self.variant { + GenerationDbVariant::V0_0(v) => v.meta(), + GenerationDbVariant::V1_0(v) => v.meta(), + } + } + + /// Insert a file system entry into the database. + pub fn insert( + &mut self, + e: FilesystemEntry, + fileid: FileId, + ids: &[ChunkId], + reason: Reason, + is_cachedir_tag: bool, + ) -> Result<(), GenerationDbError> { + match &mut self.variant { + GenerationDbVariant::V0_0(v) => v.insert(e, fileid, ids, reason, is_cachedir_tag), + GenerationDbVariant::V1_0(v) => v.insert(e, fileid, ids, reason, is_cachedir_tag), + } + } + + /// Count number of file system entries. + pub fn file_count(&self) -> Result<FileId, GenerationDbError> { + match &self.variant { + GenerationDbVariant::V0_0(v) => v.file_count(), + GenerationDbVariant::V1_0(v) => v.file_count(), + } + } + + /// Does a path refer to a cache directory? + pub fn is_cachedir_tag(&self, filename: &Path) -> Result<bool, GenerationDbError> { + match &self.variant { + GenerationDbVariant::V0_0(v) => v.is_cachedir_tag(filename), + GenerationDbVariant::V1_0(v) => v.is_cachedir_tag(filename), + } + } + + /// Return all chunk ids in database. + pub fn chunkids(&self, fileid: FileId) -> Result<SqlResults<ChunkId>, GenerationDbError> { + match &self.variant { + GenerationDbVariant::V0_0(v) => v.chunkids(fileid), + GenerationDbVariant::V1_0(v) => v.chunkids(fileid), + } + } + + /// Return all file descriptions in database. + pub fn files( + &self, + ) -> Result<SqlResults<(FileId, FilesystemEntry, Reason, bool)>, GenerationDbError> { + match &self.variant { + GenerationDbVariant::V0_0(v) => v.files(), + GenerationDbVariant::V1_0(v) => v.files(), + } + } + + /// Get a file's information given its path. + pub fn get_file(&self, filename: &Path) -> Result<Option<FilesystemEntry>, GenerationDbError> { + match &self.variant { + GenerationDbVariant::V0_0(v) => v.get_file(filename), + GenerationDbVariant::V1_0(v) => v.get_file(filename), + } + } + + /// Get a file's information given its id in the database. + pub fn get_fileno(&self, filename: &Path) -> Result<Option<FileId>, GenerationDbError> { + match &self.variant { + GenerationDbVariant::V0_0(v) => v.get_fileno(filename), + GenerationDbVariant::V1_0(v) => v.get_fileno(filename), + } + } +} + +struct V0_0 { + created: bool, + db: Database, + meta: Table, + files: Table, + chunks: Table, +} + +impl V0_0 { + const MAJOR: VersionComponent = 0; + const MINOR: VersionComponent = 0; + + /// Create a new generation database in read/write mode. + pub fn create<P: AsRef<Path>>( + filename: P, + meta: Table, + checksum_kind: LabelChecksumKind, + ) -> Result<Self, GenerationDbError> { + let db = Database::create(filename.as_ref())?; + let mut moi = Self::new(db, meta); + moi.created = true; + moi.create_tables(checksum_kind)?; + Ok(moi) + } + + /// Open an existing generation database in read-only mode. + pub fn open<P: AsRef<Path>>(filename: P, meta: Table) -> Result<Self, GenerationDbError> { + let db = Database::open(filename.as_ref())?; + Ok(Self::new(db, meta)) + } + + fn new(db: Database, meta: Table) -> Self { + let files = Table::new("files") + .column(Column::primary_key("fileno")) + .column(Column::blob("filename")) + .column(Column::text("json")) + .column(Column::text("reason")) + .column(Column::bool("is_cachedir_tag")) + .build(); + let chunks = Table::new("chunks") + .column(Column::int("fileno")) + .column(Column::text("chunkid")) + .build(); + + Self { + created: false, + db, + meta, + files, + chunks, + } + } + + fn create_tables(&mut self, checksum_kind: LabelChecksumKind) -> Result<(), GenerationDbError> { + self.db.create_table(&self.meta)?; + self.db.create_table(&self.files)?; + self.db.create_table(&self.chunks)?; + + self.db.insert( + &self.meta, + &[ + Value::text("key", "schema_version_major"), + Value::text("value", &format!("{}", Self::MAJOR)), + ], + )?; + self.db.insert( + &self.meta, + &[ + Value::text("key", "schema_version_minor"), + Value::text("value", &format!("{}", Self::MINOR)), + ], + )?; + self.db.insert( + &self.meta, + &[ + Value::text("key", "checksum_kind"), + Value::text("value", checksum_kind.serialize()), + ], + )?; + + Ok(()) + } + + /// Close a database, commit any changes. + pub fn close(self) -> Result<(), GenerationDbError> { + if self.created { + self.db + .create_index("filenames_idx", &self.files, "filename")?; + self.db.create_index("fileid_idx", &self.chunks, "fileno")?; + } + self.db.close().map_err(GenerationDbError::Database) + } + + /// Return contents of "meta" table as a HashMap. + pub fn meta(&self) -> Result<HashMap<String, String>, GenerationDbError> { + let mut map = HashMap::new(); + let mut iter = self.db.all_rows(&self.meta, &row_to_kv)?; + for kv in iter.iter()? { + let (key, value) = kv?; + map.insert(key, value); + } + Ok(map) + } + + /// Insert a file system entry into the database. + pub fn insert( + &mut self, + e: FilesystemEntry, + fileid: FileId, + ids: &[ChunkId], + reason: Reason, + is_cachedir_tag: bool, + ) -> Result<(), GenerationDbError> { + let json = serde_json::to_string(&e)?; + self.db.insert( + &self.files, + &[ + Value::primary_key("fileno", fileid), + Value::blob("filename", &path_into_blob(&e.pathbuf())), + Value::text("json", &json), + Value::text("reason", &format!("{}", reason)), + Value::bool("is_cachedir_tag", is_cachedir_tag), + ], + )?; + for id in ids { + self.db.insert( + &self.chunks, + &[ + Value::int("fileno", fileid), + Value::text("chunkid", &format!("{}", id)), + ], + )?; + } + Ok(()) + } + + /// Count number of file system entries. + pub fn file_count(&self) -> Result<FileId, GenerationDbError> { + // FIXME: this needs to be done use "SELECT count(*) FROM + // files", but the Database abstraction doesn't support that + // yet. + let mut iter = self.db.all_rows(&self.files, &Self::row_to_entry)?; + let mut count = 0; + for _ in iter.iter()? { + count += 1; + } + Ok(count) + } + + /// Does a path refer to a cache directory? + pub fn is_cachedir_tag(&self, filename: &Path) -> Result<bool, GenerationDbError> { + let filename_vec = path_into_blob(filename); + let value = Value::blob("filename", &filename_vec); + let mut rows = self + .db + .some_rows(&self.files, &value, &Self::row_to_entry)?; + let mut iter = rows.iter()?; + + if let Some(row) = iter.next() { + // Make sure there's only one row for a given filename. A + // bug in a previous version, or a maliciously constructed + // generation, could result in there being more than one. + if iter.next().is_some() { + error!("too many files in file lookup"); + Err(GenerationDbError::TooManyFiles(filename.to_path_buf())) + } else { + let (_, _, _, is_cachedir_tag) = row?; + Ok(is_cachedir_tag) + } + } else { + Ok(false) + } + } + + /// Return all chunk ids in database. + pub fn chunkids(&self, fileid: FileId) -> Result<SqlResults<ChunkId>, GenerationDbError> { + let fileid = Value::int("fileno", fileid); + Ok(self.db.some_rows(&self.chunks, &fileid, &row_to_chunkid)?) + } + + /// Return all file descriptions in database. + pub fn files( + &self, + ) -> Result<SqlResults<(FileId, FilesystemEntry, Reason, bool)>, GenerationDbError> { + Ok(self.db.all_rows(&self.files, &Self::row_to_fsentry)?) + } + + /// Get a file's information given its path. + pub fn get_file(&self, filename: &Path) -> Result<Option<FilesystemEntry>, GenerationDbError> { + match self.get_file_and_fileno(filename)? { + None => Ok(None), + Some((_, e, _)) => Ok(Some(e)), + } + } + + /// Get a file's information given its id in the database. + pub fn get_fileno(&self, filename: &Path) -> Result<Option<FileId>, GenerationDbError> { + match self.get_file_and_fileno(filename)? { + None => Ok(None), + Some((id, _, _)) => Ok(Some(id)), + } + } + + fn get_file_and_fileno( + &self, + filename: &Path, + ) -> Result<Option<(FileId, FilesystemEntry, String)>, GenerationDbError> { + let filename_bytes = path_into_blob(filename); + let value = Value::blob("filename", &filename_bytes); + let mut rows = self + .db + .some_rows(&self.files, &value, &Self::row_to_entry)?; + let mut iter = rows.iter()?; + + if let Some(row) = iter.next() { + // Make sure there's only one row for a given filename. A + // bug in a previous version, or a maliciously constructed + // generation, could result in there being more than one. + if iter.next().is_some() { + error!("too many files in file lookup"); + Err(GenerationDbError::TooManyFiles(filename.to_path_buf())) + } else { + let (fileid, ref json, ref reason, _) = row?; + let entry = serde_json::from_str(json)?; + Ok(Some((fileid, entry, reason.to_string()))) + } + } else { + Ok(None) + } + } + + fn row_to_entry(row: &rusqlite::Row) -> rusqlite::Result<(FileId, String, String, bool)> { + let fileno: FileId = row.get("fileno")?; + let json: String = row.get("json")?; + let reason: String = row.get("reason")?; + let is_cachedir_tag: bool = row.get("is_cachedir_tag")?; + Ok((fileno, json, reason, is_cachedir_tag)) + } + + fn row_to_fsentry( + row: &rusqlite::Row, + ) -> rusqlite::Result<(FileId, FilesystemEntry, Reason, bool)> { + let fileno: FileId = row.get("fileno")?; + let json: String = row.get("json")?; + let entry = serde_json::from_str(&json).map_err(|err| { + rusqlite::Error::FromSqlConversionFailure(0, rusqlite::types::Type::Text, Box::new(err)) + })?; + let reason: String = row.get("reason")?; + let reason = Reason::from(&reason); + let is_cachedir_tag: bool = row.get("is_cachedir_tag")?; + Ok((fileno, entry, reason, is_cachedir_tag)) + } +} + +struct V1_0 { + created: bool, + db: Database, + meta: Table, + files: Table, + chunks: Table, +} + +impl V1_0 { + const MAJOR: VersionComponent = 1; + const MINOR: VersionComponent = 0; + + /// Create a new generation database in read/write mode. + pub fn create<P: AsRef<Path>>( + filename: P, + meta: Table, + checksum_kind: LabelChecksumKind, + ) -> Result<Self, GenerationDbError> { + let db = Database::create(filename.as_ref())?; + let mut moi = Self::new(db, meta); + moi.created = true; + moi.create_tables(checksum_kind)?; + Ok(moi) + } + + /// Open an existing generation database in read-only mode. + pub fn open<P: AsRef<Path>>(filename: P, meta: Table) -> Result<Self, GenerationDbError> { + let db = Database::open(filename.as_ref())?; + Ok(Self::new(db, meta)) + } + + fn new(db: Database, meta: Table) -> Self { + let files = Table::new("files") + .column(Column::primary_key("fileid")) + .column(Column::blob("filename")) + .column(Column::text("json")) + .column(Column::text("reason")) + .column(Column::bool("is_cachedir_tag")) + .build(); + let chunks = Table::new("chunks") + .column(Column::int("fileid")) + .column(Column::text("chunkid")) + .build(); + + Self { + created: false, + db, + meta, + files, + chunks, + } + } + + fn create_tables(&mut self, checksum_kind: LabelChecksumKind) -> Result<(), GenerationDbError> { + self.db.create_table(&self.meta)?; + self.db.create_table(&self.files)?; + self.db.create_table(&self.chunks)?; + + self.db.insert( + &self.meta, + &[ + Value::text("key", "schema_version_major"), + Value::text("value", &format!("{}", Self::MAJOR)), + ], + )?; + self.db.insert( + &self.meta, + &[ + Value::text("key", "schema_version_minor"), + Value::text("value", &format!("{}", Self::MINOR)), + ], + )?; + self.db.insert( + &self.meta, + &[ + Value::text("key", "checksum_kind"), + Value::text("value", checksum_kind.serialize()), + ], + )?; + + Ok(()) + } + + /// Close a database, commit any changes. + pub fn close(self) -> Result<(), GenerationDbError> { + if self.created { + self.db + .create_index("filenames_idx", &self.files, "filename")?; + self.db.create_index("fileid_idx", &self.chunks, "fileid")?; + } + self.db.close().map_err(GenerationDbError::Database) + } + + /// Return contents of "meta" table as a HashMap. + pub fn meta(&self) -> Result<HashMap<String, String>, GenerationDbError> { + let mut map = HashMap::new(); + let mut iter = self.db.all_rows(&self.meta, &row_to_kv)?; + for kv in iter.iter()? { + let (key, value) = kv?; + map.insert(key, value); + } + Ok(map) + } + + /// Insert a file system entry into the database. + pub fn insert( + &mut self, + e: FilesystemEntry, + fileid: FileId, + ids: &[ChunkId], + reason: Reason, + is_cachedir_tag: bool, + ) -> Result<(), GenerationDbError> { + let json = serde_json::to_string(&e)?; + self.db.insert( + &self.files, + &[ + Value::primary_key("fileid", fileid), + Value::blob("filename", &path_into_blob(&e.pathbuf())), + Value::text("json", &json), + Value::text("reason", &format!("{}", reason)), + Value::bool("is_cachedir_tag", is_cachedir_tag), + ], + )?; + for id in ids { + self.db.insert( + &self.chunks, + &[ + Value::int("fileid", fileid), + Value::text("chunkid", &format!("{}", id)), + ], + )?; + } + Ok(()) + } + + /// Count number of file system entries. + pub fn file_count(&self) -> Result<FileId, GenerationDbError> { + // FIXME: this needs to be done use "SELECT count(*) FROM + // files", but the Database abstraction doesn't support that + // yet. + let mut iter = self.db.all_rows(&self.files, &Self::row_to_entry)?; + let mut count = 0; + for _ in iter.iter()? { + count += 1; + } + Ok(count) + } + + /// Does a path refer to a cache directory? + pub fn is_cachedir_tag(&self, filename: &Path) -> Result<bool, GenerationDbError> { + let filename_vec = path_into_blob(filename); + let value = Value::blob("filename", &filename_vec); + let mut rows = self + .db + .some_rows(&self.files, &value, &Self::row_to_entry)?; + let mut iter = rows.iter()?; + + if let Some(row) = iter.next() { + // Make sure there's only one row for a given filename. A + // bug in a previous version, or a maliciously constructed + // generation, could result in there being more than one. + if iter.next().is_some() { + error!("too many files in file lookup"); + Err(GenerationDbError::TooManyFiles(filename.to_path_buf())) + } else { + let (_, _, _, is_cachedir_tag) = row?; + Ok(is_cachedir_tag) + } + } else { + Ok(false) + } + } + + /// Return all chunk ids in database. + pub fn chunkids(&self, fileid: FileId) -> Result<SqlResults<ChunkId>, GenerationDbError> { + let fileid = Value::int("fileid", fileid); + Ok(self.db.some_rows(&self.chunks, &fileid, &row_to_chunkid)?) + } + + /// Return all file descriptions in database. + pub fn files( + &self, + ) -> Result<SqlResults<(FileId, FilesystemEntry, Reason, bool)>, GenerationDbError> { + Ok(self.db.all_rows(&self.files, &Self::row_to_fsentry)?) + } + + /// Get a file's information given its path. + pub fn get_file(&self, filename: &Path) -> Result<Option<FilesystemEntry>, GenerationDbError> { + match self.get_file_and_fileno(filename)? { + None => Ok(None), + Some((_, e, _)) => Ok(Some(e)), + } + } + + /// Get a file's information given its id in the database. + pub fn get_fileno(&self, filename: &Path) -> Result<Option<FileId>, GenerationDbError> { + match self.get_file_and_fileno(filename)? { + None => Ok(None), + Some((id, _, _)) => Ok(Some(id)), + } + } + + fn get_file_and_fileno( + &self, + filename: &Path, + ) -> Result<Option<(FileId, FilesystemEntry, String)>, GenerationDbError> { + let filename_bytes = path_into_blob(filename); + let value = Value::blob("filename", &filename_bytes); + let mut rows = self + .db + .some_rows(&self.files, &value, &Self::row_to_entry)?; + let mut iter = rows.iter()?; + + if let Some(row) = iter.next() { + // Make sure there's only one row for a given filename. A + // bug in a previous version, or a maliciously constructed + // generation, could result in there being more than one. + if iter.next().is_some() { + error!("too many files in file lookup"); + Err(GenerationDbError::TooManyFiles(filename.to_path_buf())) + } else { + let (fileid, ref json, ref reason, _) = row?; + let entry = serde_json::from_str(json)?; + Ok(Some((fileid, entry, reason.to_string()))) + } + } else { + Ok(None) + } + } + + fn row_to_entry(row: &rusqlite::Row) -> rusqlite::Result<(FileId, String, String, bool)> { + let fileno: FileId = row.get("fileid")?; + let json: String = row.get("json")?; + let reason: String = row.get("reason")?; + let is_cachedir_tag: bool = row.get("is_cachedir_tag")?; + Ok((fileno, json, reason, is_cachedir_tag)) + } + + fn row_to_fsentry( + row: &rusqlite::Row, + ) -> rusqlite::Result<(FileId, FilesystemEntry, Reason, bool)> { + let fileno: FileId = row.get("fileid")?; + let json: String = row.get("json")?; + let entry = serde_json::from_str(&json).map_err(|err| { + rusqlite::Error::FromSqlConversionFailure(0, rusqlite::types::Type::Text, Box::new(err)) + })?; + let reason: String = row.get("reason")?; + let reason = Reason::from(&reason); + let is_cachedir_tag: bool = row.get("is_cachedir_tag")?; + Ok((fileno, entry, reason, is_cachedir_tag)) + } +} + +fn row_to_kv(row: &rusqlite::Row) -> rusqlite::Result<(String, String)> { + let k = row.get("key")?; + let v = row.get("value")?; + Ok((k, v)) +} + +fn path_into_blob(path: &Path) -> Vec<u8> { + path.as_os_str().as_bytes().to_vec() +} + +fn row_to_chunkid(row: &rusqlite::Row) -> rusqlite::Result<ChunkId> { + let chunkid: String = row.get("chunkid")?; + let chunkid = ChunkId::recreate(&chunkid); + Ok(chunkid) +} + +#[cfg(test)] +mod test { + use super::Database; + use tempfile::tempdir; + + #[test] + fn opens_previously_created_db() { + let dir = tempdir().unwrap(); + let filename = dir.path().join("test.db"); + Database::create(&filename).unwrap(); + assert!(Database::open(&filename).is_ok()); + } +} diff --git a/src/engine.rs b/src/engine.rs new file mode 100644 index 0000000..d35281b --- /dev/null +++ b/src/engine.rs @@ -0,0 +1,123 @@ +//! Engine for doing CPU heavy work in the background. + +use crate::workqueue::WorkQueue; +use futures::stream::{FuturesOrdered, StreamExt}; +use tokio::select; +use tokio::sync::mpsc; + +/// Do heavy work in the background. +/// +/// An engine takes items of work from a work queue, and does the work +/// in the background, using `tokio` blocking tasks. The background +/// work can be CPU intensive or block on I/O. The number of active +/// concurrent tasks is limited to the size of the queue. +/// +/// The actual work is done in a function or closure passed in as a +/// parameter to the engine. The worker function is called with a work +/// item as an argument, in a thread dedicated for that worker +/// function. +/// +/// The need to move work items between threads puts some restrictions +/// on the types used as work items. +pub struct Engine<T> { + rx: mpsc::Receiver<T>, +} + +impl<T: Send + 'static> Engine<T> { + /// Create a new engine. + /// + /// Each engine gets work from a queue, and calls the same worker + /// function for each item of work. The results are put into + /// another, internal queue. + pub fn new<S, F>(queue: WorkQueue<S>, func: F) -> Self + where + F: Send + Copy + 'static + Fn(S) -> T, + S: Send + 'static, + { + let size = queue.size(); + let (tx, rx) = mpsc::channel(size); + tokio::spawn(manage_workers(queue, size, tx, func)); + Self { rx } + } + + /// Get the oldest result of the worker function, if any. + /// + /// This will block until there is a result, or it's known that no + /// more results will be forthcoming. + pub async fn next(&mut self) -> Option<T> { + self.rx.recv().await + } +} + +// This is a normal (non-blocking) background task that retrieves work +// items, launches blocking background tasks for work to be done, and +// waits on those tasks. Care is taken to not launch too many worker +// tasks. +async fn manage_workers<S, T, F>( + mut queue: WorkQueue<S>, + queue_size: usize, + tx: mpsc::Sender<T>, + func: F, +) where + F: Send + 'static + Copy + Fn(S) -> T, + S: Send + 'static, + T: Send + 'static, +{ + let mut workers = FuturesOrdered::new(); + + 'processing: loop { + // Wait for first of various concurrent things to finish. + select! { + biased; + + // Get work to be done. + maybe_work = queue.next() => { + if let Some(work) = maybe_work { + // We got a work item. Launch background task to + // work on it. + let tx = tx.clone(); + workers.push_back(do_work(work, tx, func)); + + // If queue is full, wait for at least one + // background task to finish. + while workers.len() >= queue_size { + workers.next().await; + } + } else { + // Finished with the input queue. Nothing more to do. + break 'processing; + } + } + + // Wait for background task to finish, if there are any + // background tasks currently running. + _ = workers.next(), if !workers.is_empty() => { + // nothing to do here + } + } + } + + while workers.next().await.is_some() { + // Finish the remaining work items. + } +} + +// Work on a work item. +// +// This launches a `tokio` blocking background task, and waits for it +// to finish. The caller spawns a normal (non-blocking) async task for +// this function, so it's OK for this function to wait on the task it +// launches. +async fn do_work<S, T, F>(item: S, tx: mpsc::Sender<T>, func: F) +where + F: Send + 'static + Fn(S) -> T, + S: Send + 'static, + T: Send + 'static, +{ + let result = tokio::task::spawn_blocking(move || func(item)) + .await + .unwrap(); + if let Err(err) = tx.send(result).await { + panic!("failed to send result to channel: {}", err); + } +} diff --git a/src/error.rs b/src/error.rs index d368763..928f258 100644 --- a/src/error.rs +++ b/src/error.rs @@ -1,25 +1,99 @@ -use crate::chunkid::ChunkId; +//! Errors from Obnam client. + +use crate::backup_run::BackupError; +use crate::chunk::ClientTrustError; +use crate::cipher::CipherError; +use crate::client::ClientError; +use crate::cmd::restore::RestoreError; +use crate::config::ClientConfigError; +use crate::db::DatabaseError; +use crate::dbgen::GenerationDbError; +use crate::generation::{LocalGenerationError, NascentError}; +use crate::genlist::GenerationListError; +use crate::label::LabelError; +use crate::passwords::PasswordError; use std::path::PathBuf; -use thiserror::Error; +use std::time::SystemTimeError; +use tempfile::PersistError; -/// Define all the kinds of errors any part of this crate can return. -#[derive(Debug, Error)] +/// Define all the kinds of errors that functions corresponding to +/// subcommands of the main program can return. +/// +/// This collects all kinds of errors the Obnam client may get, for +/// convenience. +#[derive(Debug, thiserror::Error)] pub enum ObnamError { - #[error("Can't find backup '{0}'")] - UnknownGeneration(String), + /// Error from chunk labels. + #[error(transparent)] + Label(#[from] LabelError), + + /// Error listing generations on server. + #[error(transparent)] + GenerationListError(#[from] GenerationListError), + + /// Error about client trust chunks. + #[error(transparent)] + ClientTrust(#[from] ClientTrustError), + + /// Error saving passwords. + #[error("couldn't save passwords to {0}: {1}")] + PasswordSave(PathBuf, PasswordError), + + /// Error using server HTTP API. + #[error(transparent)] + ClientError(#[from] ClientError), + + /// Error in client configuration. + #[error(transparent)] + ClientConfigError(#[from] ClientConfigError), + + /// Error making a backup. + #[error(transparent)] + BackupError(#[from] BackupError), + + /// Error making a new backup generation. + #[error(transparent)] + NascentError(#[from] NascentError), + + /// Error encrypting or decrypting. + #[error(transparent)] + CipherError(#[from] CipherError), + + /// Error using local copy of existing backup generation. + #[error(transparent)] + LocalGenerationError(#[from] LocalGenerationError), + + /// Error from generation database. + #[error(transparent)] + GenerationDb(#[from] GenerationDbError), + + /// Error using a Database. + #[error(transparent)] + Database(#[from] DatabaseError), + + /// Error restoring a backup. + #[error(transparent)] + RestoreError(#[from] RestoreError), - #[error("Generation has more than one file with the name {0}")] - TooManyFiles(PathBuf), + /// Error making temporary file persistent. + #[error(transparent)] + PersistError(#[from] PersistError), - #[error("Server response did not have a 'chunk-meta' header for chunk {0}")] - NoChunkMeta(ChunkId), + /// Error doing I/O. + #[error(transparent)] + IoError(#[from] std::io::Error), - #[error("Wrong checksum for chunk {0}, got {1}, expected {2}")] - WrongChecksum(ChunkId, String, String), + /// Error reading system clock. + #[error(transparent)] + SystemTimeError(#[from] SystemTimeError), - #[error("Chunk is missing: {0}")] - MissingChunk(ChunkId), + /// Error regarding JSON. + #[error(transparent)] + SerdeJsonError(#[from] serde_json::Error), - #[error("Chunk is in store too many times: {0}")] - DuplicateChunk(ChunkId), + /// Unexpected cache directories found. + #[error( + "found CACHEDIR.TAG files that aren't present in the previous backup, might be an attack" + )] + NewCachedirTagsFound, } diff --git a/src/fsentry.rs b/src/fsentry.rs index eae11b4..f31d6b5 100644 --- a/src/fsentry.rs +++ b/src/fsentry.rs @@ -1,10 +1,20 @@ +//! An entry in the file system. + +use log::{debug, error}; use serde::{Deserialize, Serialize}; use std::ffi::OsString; use std::fs::read_link; use std::fs::{FileType, Metadata}; -use std::os::linux::fs::MetadataExt; use std::os::unix::ffi::OsStringExt; +use std::os::unix::fs::FileTypeExt; use std::path::{Path, PathBuf}; +use users::{Groups, Users, UsersCache}; + +#[cfg(target_os = "linux")] +use std::os::linux::fs::MetadataExt; + +#[cfg(target_os = "macos")] +use std::os::macos::fs::MetadataExt; /// A file system entry. /// @@ -35,80 +45,245 @@ pub struct FilesystemEntry { // The target of a symbolic link, if any. symlink_target: Option<PathBuf>, + + // User and group owning the file. We store them as both the + // numeric id and the textual name corresponding to the numeric id + // at the time of the backup. + uid: u32, + gid: u32, + user: String, + group: String, +} + +/// Possible errors related to file system entries. +#[derive(Debug, thiserror::Error)] +pub enum FsEntryError { + /// File kind numeric representation is unknown. + #[error("Unknown file kind {0}")] + UnknownFileKindCode(u8), + + /// Failed to read a symbolic link's target. + #[error("failed to read symbolic link target {0}: {1}")] + ReadLink(PathBuf, std::io::Error), } #[allow(clippy::len_without_is_empty)] impl FilesystemEntry { - pub fn from_metadata(path: &Path, meta: &Metadata) -> anyhow::Result<Self> { + /// Create an `FsEntry` from a file's metadata. + pub fn from_metadata( + path: &Path, + meta: &Metadata, + cache: &mut UsersCache, + ) -> Result<Self, FsEntryError> { let kind = FilesystemKind::from_file_type(meta.file_type()); - Ok(Self { - path: path.to_path_buf().into_os_string().into_vec(), - kind: FilesystemKind::from_file_type(meta.file_type()), - len: meta.len(), - mode: meta.st_mode(), - mtime: meta.st_mtime(), - mtime_ns: meta.st_mtime_nsec(), - atime: meta.st_atime(), - atime_ns: meta.st_atime_nsec(), - symlink_target: if kind == FilesystemKind::Symlink { - Some(read_link(path)?) - } else { - None - }, - }) + Ok(EntryBuilder::new(kind) + .path(path.to_path_buf()) + .len(meta.len()) + .mode(meta.st_mode()) + .mtime(meta.st_mtime(), meta.st_mtime_nsec()) + .atime(meta.st_atime(), meta.st_atime_nsec()) + .user(meta.st_uid(), cache)? + .group(meta.st_uid(), cache)? + .symlink_target()? + .build()) } + /// Return the kind of file the entry refers to. pub fn kind(&self) -> FilesystemKind { self.kind } + /// Return full path to the entry. pub fn pathbuf(&self) -> PathBuf { let path = self.path.clone(); PathBuf::from(OsString::from_vec(path)) } + /// Return number of bytes for the entity represented by the entry. pub fn len(&self) -> u64 { self.len } + /// Return the entry's mode bits. pub fn mode(&self) -> u32 { self.mode } + /// Return the entry's access time, whole seconds. pub fn atime(&self) -> i64 { self.atime } + /// Return the entry's access time, nanoseconds since the last full second. pub fn atime_ns(&self) -> i64 { self.atime_ns } + /// Return the entry's modification time, whole seconds. pub fn mtime(&self) -> i64 { self.mtime } + /// Return the entry's modification time, nanoseconds since the last full second. pub fn mtime_ns(&self) -> i64 { self.mtime_ns } + /// Does the entry represent a directory? pub fn is_dir(&self) -> bool { self.kind() == FilesystemKind::Directory } + /// Return target of the symlink the entry represents. pub fn symlink_target(&self) -> Option<PathBuf> { self.symlink_target.clone() } } +#[derive(Debug)] +pub(crate) struct EntryBuilder { + kind: FilesystemKind, + path: PathBuf, + len: u64, + + // 16 bits should be enough for a Unix mode_t. + // https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html + // However, it's 32 bits on Linux, so that's what we store. + mode: u32, + + // Linux can store file system time stamps in nanosecond + // resolution. We store them as two 64-bit integers. + mtime: i64, + mtime_ns: i64, + atime: i64, + atime_ns: i64, + + // The target of a symbolic link, if any. + symlink_target: Option<PathBuf>, + + // User and group owning the file. We store them as both the + // numeric id and the textual name corresponding to the numeric id + // at the time of the backup. + uid: u32, + gid: u32, + user: String, + group: String, +} + +impl EntryBuilder { + pub(crate) fn new(kind: FilesystemKind) -> Self { + Self { + kind, + path: PathBuf::new(), + len: 0, + mode: 0, + mtime: 0, + mtime_ns: 0, + atime: 0, + atime_ns: 0, + symlink_target: None, + uid: 0, + user: "".to_string(), + gid: 0, + group: "".to_string(), + } + } + + pub(crate) fn build(self) -> FilesystemEntry { + FilesystemEntry { + kind: self.kind, + path: self.path.into_os_string().into_vec(), + len: self.len, + mode: self.mode, + mtime: self.mtime, + mtime_ns: self.mtime_ns, + atime: self.atime, + atime_ns: self.atime_ns, + symlink_target: self.symlink_target, + uid: self.uid, + user: self.user, + gid: self.gid, + group: self.group, + } + } + + pub(crate) fn path(mut self, path: PathBuf) -> Self { + self.path = path; + self + } + + pub(crate) fn len(mut self, len: u64) -> Self { + self.len = len; + self + } + + pub(crate) fn mode(mut self, mode: u32) -> Self { + self.mode = mode; + self + } + + pub(crate) fn mtime(mut self, secs: i64, nsec: i64) -> Self { + self.mtime = secs; + self.mtime_ns = nsec; + self + } + + pub(crate) fn atime(mut self, secs: i64, nsec: i64) -> Self { + self.atime = secs; + self.atime_ns = nsec; + self + } + + pub(crate) fn symlink_target(mut self) -> Result<Self, FsEntryError> { + self.symlink_target = if self.kind == FilesystemKind::Symlink { + debug!("reading symlink target for {:?}", self.path); + let target = read_link(&self.path) + .map_err(|err| FsEntryError::ReadLink(self.path.clone(), err))?; + Some(target) + } else { + None + }; + Ok(self) + } + + pub(crate) fn user(mut self, uid: u32, cache: &mut UsersCache) -> Result<Self, FsEntryError> { + self.uid = uid; + self.user = if let Some(user) = cache.get_user_by_uid(uid) { + user.name().to_string_lossy().to_string() + } else { + "".to_string() + }; + Ok(self) + } + + pub(crate) fn group(mut self, gid: u32, cache: &mut UsersCache) -> Result<Self, FsEntryError> { + self.gid = gid; + self.group = if let Some(group) = cache.get_group_by_gid(gid) { + group.name().to_string_lossy().to_string() + } else { + "".to_string() + }; + Ok(self) + } +} + /// Different types of file system entries. -#[derive(Debug, Copy, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Copy, Clone, Eq, PartialEq, Serialize, Deserialize)] pub enum FilesystemKind { + /// Regular file, including a hard link to one. Regular, + /// A directory. Directory, + /// A symbolic link. Symlink, + /// A UNIX domain socket. + Socket, + /// A UNIX named pipe. + Fifo, } impl FilesystemKind { + /// Create a kind from a file type. pub fn from_file_type(file_type: FileType) -> Self { if file_type.is_file() { FilesystemKind::Regular @@ -116,35 +291,39 @@ impl FilesystemKind { FilesystemKind::Directory } else if file_type.is_symlink() { FilesystemKind::Symlink + } else if file_type.is_socket() { + FilesystemKind::Socket + } else if file_type.is_fifo() { + FilesystemKind::Fifo } else { panic!("unknown file type {:?}", file_type); } } + /// Represent a kind as a numeric code. pub fn as_code(&self) -> u8 { match self { FilesystemKind::Regular => 0, FilesystemKind::Directory => 1, FilesystemKind::Symlink => 2, + FilesystemKind::Socket => 3, + FilesystemKind::Fifo => 4, } } - pub fn from_code(code: u8) -> anyhow::Result<Self> { + /// Create a kind from a numeric code. + pub fn from_code(code: u8) -> Result<Self, FsEntryError> { match code { 0 => Ok(FilesystemKind::Regular), 1 => Ok(FilesystemKind::Directory), 2 => Ok(FilesystemKind::Symlink), - _ => Err(Error::UnknownFileKindCode(code).into()), + 3 => Ok(FilesystemKind::Socket), + 4 => Ok(FilesystemKind::Fifo), + _ => Err(FsEntryError::UnknownFileKindCode(code)), } } } -#[derive(Debug, thiserror::Error)] -pub enum Error { - #[error("unknown file kind code {0}")] - UnknownFileKindCode(u8), -} - #[cfg(test)] mod test { use super::FilesystemKind; @@ -153,6 +332,9 @@ mod test { fn file_kind_regular_round_trips() { one_file_kind_round_trip(FilesystemKind::Regular); one_file_kind_round_trip(FilesystemKind::Directory); + one_file_kind_round_trip(FilesystemKind::Symlink); + one_file_kind_round_trip(FilesystemKind::Socket); + one_file_kind_round_trip(FilesystemKind::Fifo); } fn one_file_kind_round_trip(kind: FilesystemKind) { diff --git a/src/fsiter.rs b/src/fsiter.rs index a40ad34..ef2886d 100644 --- a/src/fsiter.rs +++ b/src/fsiter.rs @@ -1,37 +1,157 @@ -use crate::fsentry::FilesystemEntry; -use log::info; -use std::path::Path; -use walkdir::{IntoIter, WalkDir}; +//! Iterate over directory tree. + +use crate::fsentry::{FilesystemEntry, FsEntryError}; +use log::warn; +use std::path::{Path, PathBuf}; +use users::UsersCache; +use walkdir::{DirEntry, IntoIter, WalkDir}; + +/// Filesystem entry along with additional info about it. +pub struct AnnotatedFsEntry { + /// The file system entry being annotated. + pub inner: FilesystemEntry, + /// Is `entry` a valid CACHEDIR.TAG? + pub is_cachedir_tag: bool, +} /// Iterator over file system entries in a directory tree. pub struct FsIterator { - iter: IntoIter, + iter: SkipCachedirs, +} + +/// Possible errors from iterating over a directory tree. +#[derive(Debug, thiserror::Error)] +pub enum FsIterError { + /// Error from the walkdir crate. + #[error("walkdir failed: {0}")] + WalkDir(walkdir::Error), + + /// Error reading a file's metadata. + #[error("failed to get file system metadata for {0}: {1}")] + Metadata(PathBuf, std::io::Error), + + /// Error related to file system entries. + #[error(transparent)] + FsEntryError(#[from] FsEntryError), } impl FsIterator { - pub fn new(root: &Path) -> Self { + /// Create a new iterator. + pub fn new(root: &Path, exclude_cache_tag_directories: bool) -> Self { Self { - iter: WalkDir::new(root).into_iter(), + iter: SkipCachedirs::new( + WalkDir::new(root).into_iter(), + exclude_cache_tag_directories, + ), } } } impl Iterator for FsIterator { - type Item = Result<FilesystemEntry, anyhow::Error>; + type Item = Result<AnnotatedFsEntry, FsIterError>; fn next(&mut self) -> Option<Self::Item> { - match self.iter.next() { - None => None, - Some(Ok(entry)) => { - info!("found {}", entry.path().display()); - Some(new_entry(&entry)) - } - Some(Err(err)) => Some(Err(err.into())), + self.iter.next() + } +} + +/// Cachedir-aware adaptor for WalkDir: it skips the contents of dirs that contain CACHEDIR.TAG, +/// but still yields entries for the dir and the tag themselves. +struct SkipCachedirs { + cache: UsersCache, + iter: IntoIter, + exclude_cache_tag_directories: bool, + // This is the last tag we've found. `next()` will yield it before asking `iter` for more + // entries. + cachedir_tag: Option<Result<AnnotatedFsEntry, FsIterError>>, +} + +impl SkipCachedirs { + fn new(iter: IntoIter, exclude_cache_tag_directories: bool) -> Self { + Self { + cache: UsersCache::new(), + iter, + exclude_cache_tag_directories, + cachedir_tag: None, + } + } + + fn try_enqueue_cachedir_tag(&mut self, entry: &DirEntry) { + if !self.exclude_cache_tag_directories { + return; + } + + // If this entry is not a directory, it means we already processed its + // parent dir and decided that it's not cached. + if !entry.file_type().is_dir() { + return; + } + + let mut tag_path = entry.path().to_owned(); + tag_path.push("CACHEDIR.TAG"); + + // Tags are required to be regular files -- not even symlinks are allowed. + if !tag_path.is_file() { + return; + }; + + const CACHEDIR_TAG: &[u8] = b"Signature: 8a477f597d28d172789f06886806bc55"; + let mut content = [0u8; CACHEDIR_TAG.len()]; + + let mut file = if let Ok(file) = std::fs::File::open(&tag_path) { + file + } else { + return; + }; + + use std::io::Read; + match file.read_exact(&mut content) { + Ok(_) => (), + // If we can't read the tag file, proceed as if's not there + Err(_) => return, + } + + if content == CACHEDIR_TAG { + self.iter.skip_current_dir(); + self.cachedir_tag = Some(new_entry(&tag_path, true, &mut self.cache)); } } } -fn new_entry(e: &walkdir::DirEntry) -> anyhow::Result<FilesystemEntry> { - let meta = e.metadata()?; - let entry = FilesystemEntry::from_metadata(e.path(), &meta)?; - Ok(entry) +impl Iterator for SkipCachedirs { + type Item = Result<AnnotatedFsEntry, FsIterError>; + + fn next(&mut self) -> Option<Self::Item> { + self.cachedir_tag.take().or_else(|| { + let next = self.iter.next(); + match next { + None => None, + Some(Err(err)) => Some(Err(FsIterError::WalkDir(err))), + Some(Ok(entry)) => { + self.try_enqueue_cachedir_tag(&entry); + Some(new_entry(entry.path(), false, &mut self.cache)) + } + } + }) + } +} + +fn new_entry( + path: &Path, + is_cachedir_tag: bool, + cache: &mut UsersCache, +) -> Result<AnnotatedFsEntry, FsIterError> { + let meta = std::fs::symlink_metadata(path); + let meta = match meta { + Ok(meta) => meta, + Err(err) => { + warn!("failed to get metadata for {}: {}", path.display(), err); + return Err(FsIterError::Metadata(path.to_path_buf(), err)); + } + }; + let entry = FilesystemEntry::from_metadata(path, &meta, cache)?; + let annotated = AnnotatedFsEntry { + inner: entry, + is_cachedir_tag, + }; + Ok(annotated) } diff --git a/src/generation.rs b/src/generation.rs index 8a15363..477edc0 100644 --- a/src/generation.rs +++ b/src/generation.rs @@ -1,11 +1,43 @@ +//! Backup generations of various kinds. + use crate::backup_reason::Reason; use crate::chunkid::ChunkId; +use crate::db::{DatabaseError, SqlResults}; +use crate::dbgen::{FileId, GenerationDb, GenerationDbError}; use crate::fsentry::FilesystemEntry; -use rusqlite::Connection; -use std::path::Path; +use crate::genmeta::{GenerationMeta, GenerationMetaError}; +use crate::label::LabelChecksumKind; +use crate::schema::{SchemaVersion, VersionComponent}; +use serde::Serialize; +use std::fmt; +use std::path::{Path, PathBuf}; + +/// An identifier for a generation. +#[derive(Debug, Clone, Serialize)] +pub struct GenId { + id: ChunkId, +} + +impl GenId { + /// Create a generation identifier from a chunk identifier. + pub fn from_chunk_id(id: ChunkId) -> Self { + Self { id } + } -/// An identifier for a file in a generation. -type FileId = i64; + /// Convert a generation identifier into a chunk identifier. + pub fn as_chunk_id(&self) -> &ChunkId { + &self.id + } +} + +impl fmt::Display for GenId { + /// Format an identifier for display. + /// + /// The output can be parsed to re-created an identical identifier. + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{}", self.id) + } +} /// A nascent backup generation. /// @@ -13,90 +45,103 @@ type FileId = i64; /// finished yet, and it's not actually on the server until the upload /// of its generation chunk. pub struct NascentGeneration { - conn: Connection, + db: GenerationDb, fileno: FileId, } +/// Possible errors from nascent backup generations. +#[derive(Debug, thiserror::Error)] +pub enum NascentError { + /// Error backing up a backup root. + #[error("Could not back up a backup root directory: {0}: {1}")] + BackupRootFailed(PathBuf, crate::fsiter::FsIterError), + + /// Error using a local generation. + #[error(transparent)] + LocalGenerationError(#[from] LocalGenerationError), + + /// Error from a GenerationDb. + #[error(transparent)] + GenerationDb(#[from] GenerationDbError), + + /// Error from an SQL transaction. + #[error("SQL transaction error: {0}")] + Transaction(rusqlite::Error), + + /// Error from committing an SQL transaction. + #[error("SQL commit error: {0}")] + Commit(rusqlite::Error), + + /// Error creating a temporary file. + #[error("Failed to create temporary file: {0}")] + TempFile(#[from] std::io::Error), +} + impl NascentGeneration { - pub fn create<P>(filename: P) -> anyhow::Result<Self> + /// Create a new nascent generation. + pub fn create<P>( + filename: P, + schema: SchemaVersion, + checksum_kind: LabelChecksumKind, + ) -> Result<Self, NascentError> where P: AsRef<Path>, { - let conn = sql::create_db(filename.as_ref())?; - Ok(Self { conn, fileno: 0 }) + let db = GenerationDb::create(filename.as_ref(), schema, checksum_kind)?; + Ok(Self { db, fileno: 0 }) + } + + /// Commit any changes, and close the database. + pub fn close(self) -> Result<(), NascentError> { + self.db.close().map_err(NascentError::GenerationDb) } + /// How many files are there now in the nascent generation? pub fn file_count(&self) -> FileId { self.fileno } + /// Insert a new file system entry into a nascent generation. pub fn insert( &mut self, e: FilesystemEntry, ids: &[ChunkId], reason: Reason, - ) -> anyhow::Result<()> { - let t = self.conn.transaction()?; + is_cachedir_tag: bool, + ) -> Result<(), NascentError> { self.fileno += 1; - sql::insert_one(&t, e, self.fileno, ids, reason)?; - t.commit()?; + self.db + .insert(e, self.fileno, ids, reason, is_cachedir_tag)?; Ok(()) } - - pub fn insert_iter<'a>( - &mut self, - entries: impl Iterator<Item = anyhow::Result<(FilesystemEntry, Vec<ChunkId>, Reason)>>, - ) -> anyhow::Result<()> { - let t = self.conn.transaction()?; - for r in entries { - let (e, ids, reason) = r?; - self.fileno += 1; - sql::insert_one(&t, e, self.fileno, &ids[..], reason)?; - } - t.commit()?; - Ok(()) - } -} - -#[cfg(test)] -mod test { - use super::NascentGeneration; - use tempfile::NamedTempFile; - - #[test] - fn empty() { - let filename = NamedTempFile::new().unwrap().path().to_path_buf(); - { - let mut _gen = NascentGeneration::create(&filename).unwrap(); - // _gen is dropped here; the connection is close; the file - // should not be removed. - } - assert!(filename.exists()); - } } -/// A finished generation. +/// A finished generation on the server. /// -/// A generation is finished when it's on the server. It can be restored. +/// A generation is finished when it's on the server. It can be +/// fetched so it can be used as a [`LocalGeneration`]. #[derive(Debug, Clone)] pub struct FinishedGeneration { - id: ChunkId, + id: GenId, ended: String, } impl FinishedGeneration { + /// Create a new finished generation. pub fn new(id: &str, ended: &str) -> Self { - let id = id.parse().unwrap(); // this never fails + let id = GenId::from_chunk_id(id.parse().unwrap()); // this never fails Self { id, ended: ended.to_string(), } } - pub fn id(&self) -> ChunkId { - self.id.clone() + /// Get the generation's identifier. + pub fn id(&self) -> &GenId { + &self.id } + /// When was generation finished? pub fn ended(&self) -> &str { &self.ended } @@ -107,9 +152,51 @@ impl FinishedGeneration { /// This is for querying an existing generation, and other read-only /// operations. pub struct LocalGeneration { - conn: Connection, + db: GenerationDb, +} + +/// Possible errors from using local generations. +#[derive(Debug, thiserror::Error)] +pub enum LocalGenerationError { + /// Duplicate file names. + #[error("Generation has more than one file with the name {0}")] + TooManyFiles(PathBuf), + + /// No 'meta' table in generation. + #[error("Generation does not have a 'meta' table")] + NoMeta, + + /// Local generation uses a schema version that this version of + /// Obnam isn't compatible with. + #[error("Backup is not compatible with this version of Obnam: {0}.{1}")] + Incompatible(VersionComponent, VersionComponent), + + /// Error from generation metadata. + #[error(transparent)] + GenerationMeta(#[from] GenerationMetaError), + + /// Error from SQL. + #[error(transparent)] + RusqliteError(#[from] rusqlite::Error), + + /// Error from a GenerationDb. + #[error(transparent)] + GenerationDb(#[from] GenerationDbError), + + /// Error from a Database. + #[error(transparent)] + Database(#[from] DatabaseError), + + /// Error from JSON. + #[error(transparent)] + SerdeJsonError(#[from] serde_json::Error), + + /// Error from I/O. + #[error(transparent)] + IoError(#[from] std::io::Error), } +/// A backed up file in a local generation. pub struct BackedUpFile { fileno: FileId, entry: FilesystemEntry, @@ -117,8 +204,8 @@ pub struct BackedUpFile { } impl BackedUpFile { - pub fn new(fileno: FileId, entry: FilesystemEntry, reason: &str) -> Self { - let reason = Reason::from_str(reason); + /// Create a new `BackedUpFile`. + pub fn new(fileno: FileId, entry: FilesystemEntry, reason: Reason) -> Self { Self { fileno, entry, @@ -126,179 +213,204 @@ impl BackedUpFile { } } + /// Return id for file in its local generation. pub fn fileno(&self) -> FileId { self.fileno } + /// Return file system entry for file. pub fn entry(&self) -> &FilesystemEntry { &self.entry } + /// Return reason why file is in its local generation. pub fn reason(&self) -> Reason { self.reason } } impl LocalGeneration { - pub fn open<P>(filename: P) -> anyhow::Result<Self> + fn new(db: GenerationDb) -> Self { + Self { db } + } + + /// Open a local file as a local generation. + pub fn open<P>(filename: P) -> Result<Self, LocalGenerationError> where P: AsRef<Path>, { - let conn = sql::open_db(filename.as_ref())?; - Ok(Self { conn }) + let db = GenerationDb::open(filename.as_ref())?; + let gen = Self::new(db); + Ok(gen) } - pub fn file_count(&self) -> anyhow::Result<i64> { - Ok(sql::file_count(&self.conn)?) + /// Return generation metadata for local generation. + pub fn meta(&self) -> Result<GenerationMeta, LocalGenerationError> { + let map = self.db.meta()?; + GenerationMeta::from(map).map_err(LocalGenerationError::GenerationMeta) } - pub fn files(&self) -> anyhow::Result<Vec<BackedUpFile>> { - Ok(sql::files(&self.conn)?) + /// How many files are there in the local generation? + pub fn file_count(&self) -> Result<FileId, LocalGenerationError> { + Ok(self.db.file_count()?) } - pub fn chunkids(&self, fileno: FileId) -> anyhow::Result<Vec<ChunkId>> { - Ok(sql::chunkids(&self.conn, fileno)?) + /// Return all files in the local generation. + pub fn files( + &self, + ) -> Result<SqlResults<(FileId, FilesystemEntry, Reason, bool)>, LocalGenerationError> { + self.db.files().map_err(LocalGenerationError::GenerationDb) } - pub fn get_file(&self, filename: &Path) -> anyhow::Result<Option<FilesystemEntry>> { - Ok(sql::get_file(&self.conn, filename)?) + /// Return ids for all chunks in local generation. + pub fn chunkids(&self, fileid: FileId) -> Result<SqlResults<ChunkId>, LocalGenerationError> { + self.db + .chunkids(fileid) + .map_err(LocalGenerationError::GenerationDb) } - pub fn get_fileno(&self, filename: &Path) -> anyhow::Result<Option<FileId>> { - Ok(sql::get_fileno(&self.conn, filename)?) + /// Return entry for a file, given its pathname. + pub fn get_file( + &self, + filename: &Path, + ) -> Result<Option<FilesystemEntry>, LocalGenerationError> { + self.db + .get_file(filename) + .map_err(LocalGenerationError::GenerationDb) } -} -mod sql { - use super::BackedUpFile; - use super::FileId; - use crate::backup_reason::Reason; - use crate::chunkid::ChunkId; - use crate::error::ObnamError; - use crate::fsentry::FilesystemEntry; - use rusqlite::{params, Connection, OpenFlags, Row, Transaction}; - use std::os::unix::ffi::OsStrExt; - use std::path::Path; - - pub fn create_db(filename: &Path) -> anyhow::Result<Connection> { - let flags = OpenFlags::SQLITE_OPEN_CREATE | OpenFlags::SQLITE_OPEN_READ_WRITE; - let conn = Connection::open_with_flags(filename, flags)?; - conn.execute( - "CREATE TABLE files (fileno INTEGER PRIMARY KEY, filename BLOB, json TEXT, reason TEXT)", - params![], - )?; - conn.execute( - "CREATE TABLE chunks (fileno INTEGER, chunkid TEXT)", - params![], - )?; - conn.execute("CREATE INDEX filenames ON files (filename)", params![])?; - conn.execute("CREATE INDEX filenos ON chunks (fileno)", params![])?; - conn.pragma_update(None, "journal_mode", &"WAL")?; - Ok(conn) + /// Get the id in the local generation of a file, given its pathname. + pub fn get_fileno(&self, filename: &Path) -> Result<Option<FileId>, LocalGenerationError> { + self.db + .get_fileno(filename) + .map_err(LocalGenerationError::GenerationDb) } - pub fn open_db(filename: &Path) -> anyhow::Result<Connection> { - let flags = OpenFlags::SQLITE_OPEN_READ_WRITE; - let conn = Connection::open_with_flags(filename, flags)?; - conn.pragma_update(None, "journal_mode", &"WAL")?; - Ok(conn) - } - - pub fn insert_one( - t: &Transaction, - e: FilesystemEntry, - fileno: FileId, - ids: &[ChunkId], - reason: Reason, - ) -> anyhow::Result<()> { - let json = serde_json::to_string(&e)?; - t.execute( - "INSERT INTO files (fileno, filename, json, reason) VALUES (?1, ?2, ?3, ?4)", - params![fileno, path_into_blob(&e.pathbuf()), &json, reason,], - )?; - for id in ids { - t.execute( - "INSERT INTO chunks (fileno, chunkid) VALUES (?1, ?2)", - params![fileno, id], - )?; - } - Ok(()) + /// Does a pathname refer to a cache directory? + pub fn is_cachedir_tag(&self, filename: &Path) -> Result<bool, LocalGenerationError> { + self.db + .is_cachedir_tag(filename) + .map_err(LocalGenerationError::GenerationDb) } +} - fn path_into_blob(path: &Path) -> Vec<u8> { - path.as_os_str().as_bytes().to_vec() - } +#[cfg(test)] +mod test { + use super::{LabelChecksumKind, LocalGeneration, NascentGeneration, Reason, SchemaVersion}; + use crate::fsentry::EntryBuilder; + use crate::fsentry::FilesystemKind; + use std::path::PathBuf; + use tempfile::{tempdir, NamedTempFile}; - pub fn row_to_entry(row: &Row) -> rusqlite::Result<(FileId, String, String)> { - let fileno: FileId = row.get(row.column_index("fileno")?)?; - let json: String = row.get(row.column_index("json")?)?; - let reason: String = row.get(row.column_index("reason")?)?; - Ok((fileno, json, reason)) - } + #[test] + fn round_trips_u64_max() { + let tmp = tempdir().unwrap(); + let filename = tmp.path().join("test.db"); + let path = PathBuf::from("/"); + let schema = SchemaVersion::new(0, 0); + { + let e = EntryBuilder::new(FilesystemKind::Directory) + .path(path.clone()) + .len(u64::MAX) + .build(); + let mut gen = + NascentGeneration::create(&filename, schema, LabelChecksumKind::Sha256).unwrap(); + gen.insert(e, &[], Reason::IsNew, false).unwrap(); + gen.close().unwrap(); + } - pub fn file_count(conn: &Connection) -> anyhow::Result<FileId> { - let mut stmt = conn.prepare("SELECT count(*) FROM files")?; - let mut iter = stmt.query_map(params![], |row| row.get(0))?; - let count = iter.next().expect("SQL count result (1)"); - let count = count?; - Ok(count) + let db = LocalGeneration::open(&filename).unwrap(); + let e = db.get_file(&path).unwrap().unwrap(); + assert_eq!(e.len(), u64::MAX); } - pub fn files(conn: &Connection) -> anyhow::Result<Vec<BackedUpFile>> { - let mut stmt = conn.prepare("SELECT * FROM files")?; - let iter = stmt.query_map(params![], |row| row_to_entry(row))?; - let mut files = vec![]; - for x in iter { - let (fileno, json, reason) = x?; - let entry = serde_json::from_str(&json)?; - files.push(BackedUpFile::new(fileno, entry, &reason)); + #[test] + fn empty() { + let filename = NamedTempFile::new().unwrap().path().to_path_buf(); + let schema = SchemaVersion::new(0, 0); + { + let mut _gen = + NascentGeneration::create(&filename, schema, LabelChecksumKind::Sha256).unwrap(); + // _gen is dropped here; the connection is close; the file + // should not be removed. } - Ok(files) + assert!(filename.exists()); } - pub fn chunkids(conn: &Connection, fileno: FileId) -> anyhow::Result<Vec<ChunkId>> { - let mut stmt = conn.prepare("SELECT chunkid FROM chunks WHERE fileno = ?1")?; - let iter = stmt.query_map(params![fileno], |row| Ok(row.get(0)?))?; - let mut ids: Vec<ChunkId> = vec![]; - for x in iter { - let fileno: String = x?; - ids.push(ChunkId::from(&fileno)); + // FIXME: This is way too complicated a test function. It should + // be simplified, possibly by re-thinking the abstractions of the + // code it calls. + #[test] + fn remembers_cachedir_tags() { + use crate::{ + backup_reason::Reason, backup_run::FsEntryBackupOutcome, fsentry::FilesystemEntry, + }; + use std::{fs::metadata, path::Path}; + + // Create a `Metadata` structure to pass to other functions (we don't care about the + // contents) + let src_file = NamedTempFile::new().unwrap(); + let metadata = metadata(src_file.path()).unwrap(); + + let dbfile = NamedTempFile::new().unwrap().path().to_path_buf(); + + let nontag_path1 = Path::new("/nontag1"); + let nontag_path2 = Path::new("/dir/nontag2"); + let tag_path1 = Path::new("/a_tag"); + let tag_path2 = Path::new("/another_dir/a_tag"); + + let schema = SchemaVersion::new(0, 0); + let mut gen = + NascentGeneration::create(&dbfile, schema, LabelChecksumKind::Sha256).unwrap(); + let mut cache = users::UsersCache::new(); + + gen.insert( + FilesystemEntry::from_metadata(nontag_path1, &metadata, &mut cache).unwrap(), + &[], + Reason::IsNew, + false, + ) + .unwrap(); + gen.insert( + FilesystemEntry::from_metadata(tag_path1, &metadata, &mut cache).unwrap(), + &[], + Reason::IsNew, + true, + ) + .unwrap(); + + let entries = vec![ + FsEntryBackupOutcome { + entry: FilesystemEntry::from_metadata(nontag_path2, &metadata, &mut cache).unwrap(), + ids: vec![], + reason: Reason::IsNew, + is_cachedir_tag: false, + }, + FsEntryBackupOutcome { + entry: FilesystemEntry::from_metadata(tag_path2, &metadata, &mut cache).unwrap(), + ids: vec![], + reason: Reason::IsNew, + is_cachedir_tag: true, + }, + ]; + + for o in entries { + gen.insert(o.entry, &o.ids, o.reason, o.is_cachedir_tag) + .unwrap(); } - Ok(ids) - } - pub fn get_file(conn: &Connection, filename: &Path) -> anyhow::Result<Option<FilesystemEntry>> { - match get_file_and_fileno(conn, filename)? { - None => Ok(None), - Some((_, e, _)) => Ok(Some(e)), - } - } + gen.close().unwrap(); - pub fn get_fileno(conn: &Connection, filename: &Path) -> anyhow::Result<Option<FileId>> { - match get_file_and_fileno(conn, filename)? { - None => Ok(None), - Some((id, _, _)) => Ok(Some(id)), - } - } + let gen = LocalGeneration::open(dbfile).unwrap(); + assert!(!gen.is_cachedir_tag(nontag_path1).unwrap()); + assert!(!gen.is_cachedir_tag(nontag_path2).unwrap()); + assert!(gen.is_cachedir_tag(tag_path1).unwrap()); + assert!(gen.is_cachedir_tag(tag_path2).unwrap()); - fn get_file_and_fileno( - conn: &Connection, - filename: &Path, - ) -> anyhow::Result<Option<(FileId, FilesystemEntry, String)>> { - let mut stmt = conn.prepare("SELECT * FROM files WHERE filename = ?1")?; - let mut iter = - stmt.query_map(params![path_into_blob(filename)], |row| row_to_entry(row))?; - match iter.next() { - None => Ok(None), - Some(Err(e)) => Err(e.into()), - Some(Ok((fileno, json, reason))) => { - let entry = serde_json::from_str(&json)?; - if iter.next() == None { - Ok(Some((fileno, entry, reason))) - } else { - Err(ObnamError::TooManyFiles(filename.to_path_buf()).into()) - } - } - } + // Nonexistent files are not cachedir tags + assert!(!gen.is_cachedir_tag(Path::new("/hello/world")).unwrap()); + assert!(!gen + .is_cachedir_tag(Path::new("/different path/to/another file.txt")) + .unwrap()); } } diff --git a/src/genlist.rs b/src/genlist.rs index 10c614e..3a0d81a 100644 --- a/src/genlist.rs +++ b/src/genlist.rs @@ -1,33 +1,51 @@ +//! A list of generations on the server. + use crate::chunkid::ChunkId; -use crate::generation::FinishedGeneration; +use crate::generation::{FinishedGeneration, GenId}; +/// A list of generations on the server. pub struct GenerationList { list: Vec<FinishedGeneration>, } +/// Possible errors from listing generations. +#[derive(Debug, thiserror::Error)] +pub enum GenerationListError { + /// Server doesn't know about a generation. + #[error("Unknown generation: {0}")] + UnknownGeneration(ChunkId), +} + impl GenerationList { + /// Create a new list of generations. pub fn new(gens: Vec<FinishedGeneration>) -> Self { - let mut list = gens.clone(); + let mut list = gens; list.sort_by_cached_key(|gen| gen.ended().to_string()); Self { list } } + /// Return an iterator over the generations. pub fn iter(&self) -> impl Iterator<Item = &FinishedGeneration> { self.list.iter() } - pub fn resolve(&self, genref: &str) -> Option<String> { + /// Resolve a symbolic name of a generation into its identifier. + /// + /// For example, "latest" refers to the latest backup, but needs + /// to be resolved into an actual, immutable id to actually be + /// restored. + pub fn resolve(&self, genref: &str) -> Result<GenId, GenerationListError> { let gen = if self.list.is_empty() { None } else if genref == "latest" { let i = self.list.len() - 1; Some(self.list[i].clone()) } else { - let genref: ChunkId = genref.parse().unwrap(); + let genref = GenId::from_chunk_id(genref.parse().unwrap()); let hits: Vec<FinishedGeneration> = self .iter() - .filter(|gen| gen.id() == genref) - .map(|gen| gen.clone()) + .filter(|gen| gen.id().as_chunk_id() == genref.as_chunk_id()) + .cloned() .collect(); if hits.len() == 1 { Some(hits[0].clone()) @@ -36,8 +54,10 @@ impl GenerationList { } }; match gen { - None => None, - Some(gen) => Some(gen.id().to_string()), + None => Err(GenerationListError::UnknownGeneration(ChunkId::recreate( + genref, + ))), + Some(gen) => Ok(gen.id().clone()), } } } diff --git a/src/genmeta.rs b/src/genmeta.rs new file mode 100644 index 0000000..d5b14a3 --- /dev/null +++ b/src/genmeta.rs @@ -0,0 +1,62 @@ +//! Backup generations metadata. + +use crate::schema::{SchemaVersion, VersionComponent}; +use serde::Serialize; +use std::collections::HashMap; + +/// Metadata about the local generation. +#[derive(Debug, Serialize)] +pub struct GenerationMeta { + schema_version: SchemaVersion, + extras: HashMap<String, String>, +} + +impl GenerationMeta { + /// Create from a hash map. + pub fn from(mut map: HashMap<String, String>) -> Result<Self, GenerationMetaError> { + let major: VersionComponent = metaint(&mut map, "schema_version_major")?; + let minor: VersionComponent = metaint(&mut map, "schema_version_minor")?; + Ok(Self { + schema_version: SchemaVersion::new(major, minor), + extras: map, + }) + } + + /// Return schema version of local generation. + pub fn schema_version(&self) -> SchemaVersion { + self.schema_version + } + + /// Get a value corresponding to a key in the meta table. + pub fn get(&self, key: &str) -> Option<&String> { + self.extras.get(key) + } +} + +fn metastr(map: &mut HashMap<String, String>, key: &str) -> Result<String, GenerationMetaError> { + if let Some(v) = map.remove(key) { + Ok(v) + } else { + Err(GenerationMetaError::NoMetaKey(key.to_string())) + } +} + +fn metaint(map: &mut HashMap<String, String>, key: &str) -> Result<u32, GenerationMetaError> { + let v = metastr(map, key)?; + let v = v + .parse() + .map_err(|err| GenerationMetaError::BadMetaInteger(key.to_string(), err))?; + Ok(v) +} + +/// Possible errors from getting generation metadata. +#[derive(Debug, thiserror::Error)] +pub enum GenerationMetaError { + /// Missing from from 'meta' table. + #[error("Generation 'meta' table does not have a row {0}")] + NoMetaKey(String), + + /// Bad data in 'meta' table. + #[error("Generation 'meta' row {0} has badly formed integer: {1}")] + BadMetaInteger(String, std::num::ParseIntError), +} diff --git a/src/index.rs b/src/index.rs index d527839..42f1a95 100644 --- a/src/index.rs +++ b/src/index.rs @@ -1,65 +1,81 @@ +//! An on-disk index of chunks for the server. + use crate::chunkid::ChunkId; use crate::chunkmeta::ChunkMeta; +use crate::label::Label; use rusqlite::Connection; -use std::collections::HashMap; -use std::path::{Path, PathBuf}; +use std::path::Path; -/// A chunk index. +/// A chunk index stored on the disk. /// /// A chunk index lets the server quickly find chunks based on a -/// string key/value pair, or whether they are generations. +/// string key/value pair. #[derive(Debug)] pub struct Index { - filename: PathBuf, conn: Connection, - map: HashMap<(String, String), Vec<ChunkId>>, - generations: Vec<ChunkId>, - metas: HashMap<ChunkId, ChunkMeta>, +} + +/// All the errors that may be returned for `Index`. +#[derive(Debug, thiserror::Error)] +pub enum IndexError { + /// Index does not have a chunk. + #[error("The repository index does not have chunk {0}")] + MissingChunk(ChunkId), + + /// Index has chunk more than once. + #[error("The repository index duplicates chunk {0}")] + DuplicateChunk(ChunkId), + + /// An error from SQLite. + #[error(transparent)] + SqlError(#[from] rusqlite::Error), } impl Index { - pub fn new<P: AsRef<Path>>(dirname: P) -> anyhow::Result<Self> { + /// Create a new index. + pub fn new<P: AsRef<Path>>(dirname: P) -> Result<Self, IndexError> { let filename = dirname.as_ref().join("meta.db"); let conn = if filename.exists() { sql::open_db(&filename)? } else { sql::create_db(&filename)? }; - Ok(Self { - filename, - conn, - map: HashMap::new(), - generations: vec![], - metas: HashMap::new(), - }) + Ok(Self { conn }) } - pub fn insert_meta(&mut self, id: ChunkId, meta: ChunkMeta) -> anyhow::Result<()> { + /// Insert metadata for a new chunk into index. + pub fn insert_meta(&mut self, id: ChunkId, meta: ChunkMeta) -> Result<(), IndexError> { let t = self.conn.transaction()?; sql::insert(&t, &id, &meta)?; t.commit()?; Ok(()) } - pub fn get_meta(&self, id: &ChunkId) -> anyhow::Result<ChunkMeta> { + /// Look up metadata for a chunk, given its id. + pub fn get_meta(&self, id: &ChunkId) -> Result<ChunkMeta, IndexError> { sql::lookup(&self.conn, id) } - pub fn remove_meta(&mut self, id: &ChunkId) -> anyhow::Result<()> { + /// Remove a chunk's metadata. + pub fn remove_meta(&mut self, id: &ChunkId) -> Result<(), IndexError> { sql::remove(&self.conn, id) } - pub fn find_by_sha256(&self, sha256: &str) -> anyhow::Result<Vec<ChunkId>> { - sql::find_by_256(&self.conn, sha256) + /// Find chunks with a client-assigned label. + pub fn find_by_label(&self, label: &str) -> Result<Vec<ChunkId>, IndexError> { + sql::find_by_label(&self.conn, label) } - pub fn find_generations(&self) -> anyhow::Result<Vec<ChunkId>> { - sql::find_generations(&self.conn) + /// Find all chunks. + pub fn all_chunks(&self) -> Result<Vec<ChunkId>, IndexError> { + sql::find_chunk_ids(&self.conn) } } #[cfg(test)] mod test { + use super::Label; + use super::{ChunkId, ChunkMeta, Index}; use std::path::Path; use tempfile::tempdir; @@ -71,140 +87,113 @@ mod test { #[test] fn remembers_inserted() { let id: ChunkId = "id001".parse().unwrap(); - let meta = ChunkMeta::new("abc"); + let sum = Label::sha256(b"abc"); + let meta = ChunkMeta::new(&sum); let dir = tempdir().unwrap(); let mut idx = new_index(dir.path()); idx.insert_meta(id.clone(), meta.clone()).unwrap(); assert_eq!(idx.get_meta(&id).unwrap(), meta); - let ids = idx.find_by_sha256("abc").unwrap(); + let ids = idx.find_by_label(&sum.serialize()).unwrap(); assert_eq!(ids, vec![id]); } #[test] fn does_not_find_uninserted() { let id: ChunkId = "id001".parse().unwrap(); - let meta = ChunkMeta::new("abc"); + let sum = Label::sha256(b"abc"); + let meta = ChunkMeta::new(&sum); let dir = tempdir().unwrap(); let mut idx = new_index(dir.path()); idx.insert_meta(id, meta).unwrap(); - assert_eq!(idx.find_by_sha256("def").unwrap().len(), 0) + assert_eq!(idx.find_by_label("def").unwrap().len(), 0) } #[test] fn removes_inserted() { let id: ChunkId = "id001".parse().unwrap(); - let meta = ChunkMeta::new("abc"); + let sum = Label::sha256(b"abc"); + let meta = ChunkMeta::new(&sum); let dir = tempdir().unwrap(); let mut idx = new_index(dir.path()); idx.insert_meta(id.clone(), meta).unwrap(); idx.remove_meta(&id).unwrap(); - let ids: Vec<ChunkId> = idx.find_by_sha256("abc").unwrap(); + let ids: Vec<ChunkId> = idx.find_by_label(&sum.serialize()).unwrap(); assert_eq!(ids, vec![]); } - - #[test] - fn has_no_generations_initially() { - let dir = tempdir().unwrap(); - let idx = new_index(dir.path()); - assert_eq!(idx.find_generations().unwrap(), vec![]); - } - - #[test] - fn remembers_generation() { - let id: ChunkId = "id001".parse().unwrap(); - let meta = ChunkMeta::new_generation("abc", "timestamp"); - let dir = tempdir().unwrap(); - let mut idx = new_index(dir.path()); - idx.insert_meta(id.clone(), meta.clone()).unwrap(); - assert_eq!(idx.find_generations().unwrap(), vec![id]); - } - - #[test] - fn removes_generation() { - let id: ChunkId = "id001".parse().unwrap(); - let meta = ChunkMeta::new_generation("abc", "timestamp"); - let dir = tempdir().unwrap(); - let mut idx = new_index(dir.path()); - idx.insert_meta(id.clone(), meta.clone()).unwrap(); - idx.remove_meta(&id).unwrap(); - assert_eq!(idx.find_generations().unwrap(), vec![]); - } } mod sql { + use super::{IndexError, Label}; use crate::chunkid::ChunkId; use crate::chunkmeta::ChunkMeta; - use crate::error::ObnamError; use log::error; use rusqlite::{params, Connection, OpenFlags, Row, Transaction}; use std::path::Path; - pub fn create_db(filename: &Path) -> anyhow::Result<Connection> { + /// Create a database in a file. + pub fn create_db(filename: &Path) -> Result<Connection, IndexError> { let flags = OpenFlags::SQLITE_OPEN_CREATE | OpenFlags::SQLITE_OPEN_READ_WRITE; let conn = Connection::open_with_flags(filename, flags)?; conn.execute( - "CREATE TABLE chunks (id TEXT PRIMARY KEY, sha256 TEXT, generation INT, ended TEXT)", + "CREATE TABLE chunks (id TEXT PRIMARY KEY, label TEXT)", params![], )?; - conn.execute("CREATE INDEX sha256_idx ON chunks (sha256)", params![])?; - conn.execute( - "CREATE INDEX generation_idx ON chunks (generation)", - params![], - )?; - conn.pragma_update(None, "journal_mode", &"WAL")?; + conn.execute("CREATE INDEX label_idx ON chunks (label)", params![])?; + conn.pragma_update(None, "journal_mode", "WAL")?; Ok(conn) } - pub fn open_db(filename: &Path) -> anyhow::Result<Connection> { + /// Open an existing database in a file. + pub fn open_db(filename: &Path) -> Result<Connection, IndexError> { let flags = OpenFlags::SQLITE_OPEN_READ_WRITE; let conn = Connection::open_with_flags(filename, flags)?; - conn.pragma_update(None, "journal_mode", &"WAL")?; + conn.pragma_update(None, "journal_mode", "WAL")?; Ok(conn) } - pub fn insert(t: &Transaction, chunkid: &ChunkId, meta: &ChunkMeta) -> anyhow::Result<()> { + /// Insert a new chunk's metadata into database. + pub fn insert(t: &Transaction, chunkid: &ChunkId, meta: &ChunkMeta) -> Result<(), IndexError> { let chunkid = format!("{}", chunkid); - let sha256 = meta.sha256(); - let generation = if meta.is_generation() { 1 } else { 0 }; - let ended = meta.ended(); + let label = meta.label(); t.execute( - "INSERT INTO chunks (id, sha256, generation, ended) VALUES (?1, ?2, ?3, ?4)", - params![chunkid, sha256, generation, ended], + "INSERT INTO chunks (id, label) VALUES (?1, ?2)", + params![chunkid, label], )?; Ok(()) } - pub fn remove(conn: &Connection, chunkid: &ChunkId) -> anyhow::Result<()> { + /// Remove a chunk's metadata from the database. + pub fn remove(conn: &Connection, chunkid: &ChunkId) -> Result<(), IndexError> { conn.execute("DELETE FROM chunks WHERE id IS ?1", params![chunkid])?; Ok(()) } - pub fn lookup(conn: &Connection, id: &ChunkId) -> anyhow::Result<ChunkMeta> { + /// Look up a chunk using its id. + pub fn lookup(conn: &Connection, id: &ChunkId) -> Result<ChunkMeta, IndexError> { let mut stmt = conn.prepare("SELECT * FROM chunks WHERE id IS ?1")?; - let iter = stmt.query_map(params![id], |row| row_to_meta(row))?; + let iter = stmt.query_map(params![id], row_to_meta)?; let mut metas: Vec<ChunkMeta> = vec![]; for meta in iter { let meta = meta?; if metas.is_empty() { - eprintln!("lookup: meta={:?}", meta); metas.push(meta); } else { - let err = ObnamError::DuplicateChunk(id.clone()); + let err = IndexError::DuplicateChunk(id.clone()); error!("{}", err); - return Err(err.into()); + return Err(err); } } - if metas.len() == 0 { - eprintln!("lookup: no hits"); - return Err(ObnamError::MissingChunk(id.clone()).into()); + if metas.is_empty() { + return Err(IndexError::MissingChunk(id.clone())); } let r = metas[0].clone(); Ok(r) } - pub fn find_by_256(conn: &Connection, sha256: &str) -> anyhow::Result<Vec<ChunkId>> { - let mut stmt = conn.prepare("SELECT id FROM chunks WHERE sha256 IS ?1")?; - let iter = stmt.query_map(params![sha256], |row| row_to_id(row))?; + /// Find chunks with a given checksum. + pub fn find_by_label(conn: &Connection, label: &str) -> Result<Vec<ChunkId>, IndexError> { + let mut stmt = conn.prepare("SELECT id FROM chunks WHERE label IS ?1")?; + let iter = stmt.query_map(params![label], row_to_id)?; let mut ids = vec![]; for x in iter { let x = x?; @@ -213,9 +202,10 @@ mod sql { Ok(ids) } - pub fn find_generations(conn: &Connection) -> anyhow::Result<Vec<ChunkId>> { - let mut stmt = conn.prepare("SELECT id FROM chunks WHERE generation IS 1")?; - let iter = stmt.query_map(params![], |row| row_to_id(row))?; + /// Find ids of all chunks. + pub fn find_chunk_ids(conn: &Connection) -> Result<Vec<ChunkId>, IndexError> { + let mut stmt = conn.prepare("SELECT id FROM chunks")?; + let iter = stmt.query_map(params![], row_to_id)?; let mut ids = vec![]; for x in iter { let x = x?; @@ -224,20 +214,14 @@ mod sql { Ok(ids) } - pub fn row_to_meta(row: &Row) -> rusqlite::Result<ChunkMeta> { - let sha256: String = row.get(row.column_index("sha256")?)?; - let generation: i32 = row.get(row.column_index("generation")?)?; - let meta = if generation == 0 { - ChunkMeta::new(&sha256) - } else { - let ended: String = row.get(row.column_index("ended")?)?; - ChunkMeta::new_generation(&sha256, &ended) - }; - Ok(meta) + fn row_to_meta(row: &Row) -> rusqlite::Result<ChunkMeta> { + let hash: String = row.get("label")?; + let sha256 = Label::deserialize(&hash).expect("deserialize checksum from database"); + Ok(ChunkMeta::new(&sha256)) } - pub fn row_to_id(row: &Row) -> rusqlite::Result<ChunkId> { - let id: String = row.get(row.column_index("id")?)?; - Ok(ChunkId::from_str(&id)) + fn row_to_id(row: &Row) -> rusqlite::Result<ChunkId> { + let id: String = row.get("id")?; + Ok(ChunkId::recreate(&id)) } } diff --git a/src/indexedstore.rs b/src/indexedstore.rs deleted file mode 100644 index 0366013..0000000 --- a/src/indexedstore.rs +++ /dev/null @@ -1,57 +0,0 @@ -use crate::chunk::DataChunk; -use crate::chunkid::ChunkId; -use crate::chunkmeta::ChunkMeta; -use crate::index::Index; -use crate::store::Store; -use std::path::Path; - -/// A store for chunks and their metadata. -/// -/// This combines Store and Index into one interface to make it easier -/// to handle the server side storage of chunks. -pub struct IndexedStore { - store: Store, - index: Index, -} - -impl IndexedStore { - pub fn new(dirname: &Path) -> anyhow::Result<Self> { - let store = Store::new(dirname); - let index = Index::new(dirname)?; - Ok(Self { store, index }) - } - - pub fn save(&mut self, meta: &ChunkMeta, chunk: &DataChunk) -> anyhow::Result<ChunkId> { - let id = ChunkId::new(); - self.store.save(&id, meta, chunk)?; - self.insert_meta(&id, meta)?; - Ok(id) - } - - fn insert_meta(&mut self, id: &ChunkId, meta: &ChunkMeta) -> anyhow::Result<()> { - self.index.insert_meta(id.clone(), meta.clone())?; - Ok(()) - } - - pub fn load(&self, id: &ChunkId) -> anyhow::Result<(DataChunk, ChunkMeta)> { - Ok((self.store.load(id)?, self.load_meta(id)?)) - } - - pub fn load_meta(&self, id: &ChunkId) -> anyhow::Result<ChunkMeta> { - self.index.get_meta(id) - } - - pub fn find_by_sha256(&self, sha256: &str) -> anyhow::Result<Vec<ChunkId>> { - self.index.find_by_sha256(sha256) - } - - pub fn find_generations(&self) -> anyhow::Result<Vec<ChunkId>> { - self.index.find_generations() - } - - pub fn remove(&mut self, id: &ChunkId) -> anyhow::Result<()> { - self.index.remove_meta(id).unwrap(); - self.store.delete(id)?; - Ok(()) - } -} diff --git a/src/label.rs b/src/label.rs new file mode 100644 index 0000000..19d270a --- /dev/null +++ b/src/label.rs @@ -0,0 +1,138 @@ +//! A chunk label. +//! +//! De-duplication of backed up data in Obnam relies on cryptographic +//! checksums. They are implemented in this module. Note that Obnam +//! does not aim to make these algorithms configurable, so only a very +//! small number of carefully chosen algorithms are supported here. + +use blake2::Blake2s256; +use sha2::{Digest, Sha256}; + +const LITERAL: char = '0'; +const SHA256: char = '1'; +const BLAKE2: char = '2'; + +/// A checksum of some data. +#[derive(Debug, Clone)] +pub enum Label { + /// An arbitrary, literal string. + Literal(String), + + /// A SHA256 checksum. + Sha256(String), + + /// A BLAKE2s checksum. + Blake2(String), +} + +impl Label { + /// Construct a literal string. + pub fn literal(s: &str) -> Self { + Self::Literal(s.to_string()) + } + + /// Compute a SHA256 checksum for a block of data. + pub fn sha256(data: &[u8]) -> Self { + let mut hasher = Sha256::new(); + hasher.update(data); + let hash = hasher.finalize(); + Self::Sha256(format!("{:x}", hash)) + } + + /// Compute a BLAKE2s checksum for a block of data. + pub fn blake2(data: &[u8]) -> Self { + let mut hasher = Blake2s256::new(); + hasher.update(data); + let hash = hasher.finalize(); + Self::Sha256(format!("{:x}", hash)) + } + + /// Serialize a label into a string representation. + pub fn serialize(&self) -> String { + match self { + Self::Literal(s) => format!("{}{}", LITERAL, s), + Self::Sha256(hash) => format!("{}{}", SHA256, hash), + Self::Blake2(hash) => format!("{}{}", BLAKE2, hash), + } + } + + /// De-serialize a label from its string representation. + pub fn deserialize(s: &str) -> Result<Self, LabelError> { + if s.starts_with(LITERAL) { + Ok(Self::Literal(s[1..].to_string())) + } else if s.starts_with(SHA256) { + Ok(Self::Sha256(s[1..].to_string())) + } else { + Err(LabelError::UnknownType(s.to_string())) + } + } +} + +/// Kinds of checksum labels. +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub enum LabelChecksumKind { + /// Use a Blake2 checksum. + Blake2, + + /// Use a SHA256 checksum. + Sha256, +} + +impl LabelChecksumKind { + /// Parse a string into a label checksum kind. + pub fn from(s: &str) -> Result<Self, LabelError> { + if s == "sha256" { + Ok(Self::Sha256) + } else if s == "blake2" { + Ok(Self::Blake2) + } else { + Err(LabelError::UnknownType(s.to_string())) + } + } + + /// Serialize a checksum kind into a string. + pub fn serialize(self) -> &'static str { + match self { + Self::Sha256 => "sha256", + Self::Blake2 => "blake2", + } + } +} + +/// Possible errors from dealing with chunk labels. +#[derive(Debug, thiserror::Error)] +pub enum LabelError { + /// Serialized label didn't start with a known type prefix. + #[error("Unknown label: {0:?}")] + UnknownType(String), +} + +#[cfg(test)] +mod test { + use super::{Label, LabelChecksumKind}; + + #[test] + fn roundtrip_literal() { + let label = Label::literal("dummy data"); + let serialized = label.serialize(); + let de = Label::deserialize(&serialized).unwrap(); + let seri2 = de.serialize(); + assert_eq!(serialized, seri2); + } + + #[test] + fn roundtrip_sha256() { + let label = Label::sha256(b"dummy data"); + let serialized = label.serialize(); + let de = Label::deserialize(&serialized).unwrap(); + let seri2 = de.serialize(); + assert_eq!(serialized, seri2); + } + + #[test] + fn roundtrip_checksum_kind() { + for kind in [LabelChecksumKind::Sha256, LabelChecksumKind::Blake2] { + assert_eq!(LabelChecksumKind::from(kind.serialize()).unwrap(), kind); + } + } +} @@ -1,21 +1,38 @@ +//! Encrypted backups. +//! +//! Obnam is a backup program that encrypts the backups. This crate +//! provides access to all the functionality of Obnam as a library. + +#![deny(missing_docs)] + +pub mod accumulated_time; pub mod backup_progress; pub mod backup_reason; pub mod backup_run; -pub mod benchmark; -pub mod checksummer; pub mod chunk; pub mod chunker; pub mod chunkid; pub mod chunkmeta; +pub mod chunkstore; +pub mod cipher; pub mod client; pub mod cmd; +pub mod config; +pub mod db; +pub mod dbgen; +pub mod engine; pub mod error; pub mod fsentry; pub mod fsiter; pub mod generation; pub mod genlist; +pub mod genmeta; pub mod index; -pub mod indexedstore; +pub mod label; +pub mod passwords; +pub mod performance; pub mod policy; +pub mod schema; pub mod server; pub mod store; +pub mod workqueue; diff --git a/src/passwords.rs b/src/passwords.rs new file mode 100644 index 0000000..efc3f96 --- /dev/null +++ b/src/passwords.rs @@ -0,0 +1,102 @@ +//! Passwords for encryption. + +use pbkdf2::{ + password_hash::{PasswordHasher, SaltString}, + Pbkdf2, +}; +use rand::rngs::OsRng; +use serde::{Deserialize, Serialize}; +use std::io::prelude::Write; +use std::os::unix::fs::PermissionsExt; +use std::path::{Path, PathBuf}; + +const KEY_LEN: usize = 32; // Only size accepted by aead crate? + +/// Encryption password. +#[derive(Debug, Serialize, Deserialize, Clone)] +pub struct Passwords { + encryption: String, +} + +impl Passwords { + /// Create a new encryption password from a user-supplied passphrase. + pub fn new(passphrase: &str) -> Self { + let mut key = derive_password(passphrase); + let _ = key.split_off(KEY_LEN); + assert_eq!(key.len(), KEY_LEN); + Self { encryption: key } + } + + /// Get encryption key. + pub fn encryption_key(&self) -> &[u8] { + self.encryption.as_bytes() + } + + /// Load passwords from file. + pub fn load(filename: &Path) -> Result<Self, PasswordError> { + let data = std::fs::read(filename) + .map_err(|err| PasswordError::Read(filename.to_path_buf(), err))?; + serde_yaml::from_slice(&data) + .map_err(|err| PasswordError::Parse(filename.to_path_buf(), err)) + } + + /// Save passwords to file. + pub fn save(&self, filename: &Path) -> Result<(), PasswordError> { + let data = serde_yaml::to_string(&self).map_err(PasswordError::Serialize)?; + + let mut file = std::fs::File::create(filename) + .map_err(|err| PasswordError::Write(filename.to_path_buf(), err))?; + let metadata = file + .metadata() + .map_err(|err| PasswordError::Write(filename.to_path_buf(), err))?; + let mut permissions = metadata.permissions(); + + // Make readadable by owner only. We still have the open file + // handle, so we can write the content. + permissions.set_mode(0o400); + std::fs::set_permissions(filename, permissions) + .map_err(|err| PasswordError::Write(filename.to_path_buf(), err))?; + + // Write actual content. + file.write_all(data.as_bytes()) + .map_err(|err| PasswordError::Write(filename.to_path_buf(), err))?; + + Ok(()) + } +} + +/// Return name of password file, relative to configuration file. +pub fn passwords_filename(config_filename: &Path) -> PathBuf { + let mut filename = config_filename.to_path_buf(); + filename.set_file_name("passwords.yaml"); + filename +} + +fn derive_password(passphrase: &str) -> String { + let salt = SaltString::generate(&mut OsRng); + + Pbkdf2 + .hash_password(passphrase.as_bytes(), salt.as_salt()) + .unwrap() + .to_string() +} + +/// Possible errors from passwords. +#[derive(Debug, thiserror::Error)] +pub enum PasswordError { + /// Failed to make YAML when saving passwords. + #[error("failed to serialize passwords for saving: {0}")] + Serialize(serde_yaml::Error), + + /// Failed to save to file. + #[error("failed to save passwords to {0}: {1}")] + Write(PathBuf, std::io::Error), + + /// Failed read passwords file. + #[error("failed to read passwords from {0}: {1}")] + Read(PathBuf, std::io::Error), + + /// Failed to parse passwords file. + #[error("failed to parse saved passwords from {0}: {1}")] + Parse(PathBuf, serde_yaml::Error), +} diff --git a/src/performance.rs b/src/performance.rs new file mode 100644 index 0000000..29c2328 --- /dev/null +++ b/src/performance.rs @@ -0,0 +1,97 @@ +//! Performance measurements from an Obnam run. + +use crate::accumulated_time::AccumulatedTime; +use log::info; + +/// The kinds of clocks we have. +#[derive(Debug, PartialEq, Eq, Hash, Clone, Copy)] +pub enum Clock { + /// The complete runtime of the program. + RunTime, + + /// Time spent downloading previous backup generations. + GenerationDownload, + + /// Time spent uploading backup generations. + GenerationUpload, +} + +/// Collected measurements from this Obnam run. +#[derive(Debug)] +pub struct Performance { + args: Vec<String>, + time: AccumulatedTime<Clock>, + live_files: u64, + files_backed_up: u64, + chunks_uploaded: u64, + chunks_reused: u64, +} + +impl Default for Performance { + fn default() -> Self { + Self { + args: std::env::args().collect(), + time: AccumulatedTime::<Clock>::new(), + live_files: 0, + files_backed_up: 0, + chunks_reused: 0, + chunks_uploaded: 0, + } + } +} + +impl Performance { + /// Log all performance measurements to the log file. + pub fn log(&self) { + info!("Performance measurements for this Obnam run"); + for (i, arg) in self.args.iter().enumerate() { + info!("argv[{}]={:?}", i, arg); + } + info!("Live files found: {}", self.live_files); + info!("Files backed up: {}", self.files_backed_up); + info!("Chunks uploaded: {}", self.chunks_uploaded); + info!("Chunks reused: {}", self.chunks_reused); + info!( + "Downloading previous generation (seconds): {}", + self.time.secs(Clock::GenerationDownload) + ); + info!( + "Uploading new generation (seconds): {}", + self.time.secs(Clock::GenerationUpload) + ); + info!( + "Complete run time (seconds): {}", + self.time.secs(Clock::RunTime) + ); + } + + /// Start a specific clock. + pub fn start(&mut self, clock: Clock) { + self.time.start(clock) + } + + /// Stop a specific clock. + pub fn stop(&mut self, clock: Clock) { + self.time.stop(clock) + } + + /// Increment number of live files. + pub fn found_live_files(&mut self, n: u64) { + self.live_files += n; + } + + /// Increment number of files backed up this run. + pub fn back_up_file(&mut self) { + self.files_backed_up += 1; + } + + /// Increment number of reused chunks. + pub fn reuse_chunk(&mut self) { + self.chunks_reused += 1; + } + + /// Increment number of uploaded chunks. + pub fn upload_chunk(&mut self) { + self.chunks_uploaded += 1; + } +} diff --git a/src/policy.rs b/src/policy.rs index 032b851..8cdbd76 100644 --- a/src/policy.rs +++ b/src/policy.rs @@ -1,24 +1,40 @@ +//! Policy for what gets backed up. + use crate::backup_reason::Reason; use crate::fsentry::FilesystemEntry; use crate::generation::LocalGeneration; -use log::{debug, warn}; +use log::warn; +/// Policy for what gets backed up. +/// +/// The policy allows two aspects to be controlled: +/// +/// * should new files )(files that didn't exist in the previous +/// backup be included in the new backup? +/// * should files that haven't been changed since the previous backup +/// be included in the new backup? +/// +/// If policy doesn't allow a file to be included, it's skipped. pub struct BackupPolicy { new: bool, old_if_changed: bool, } -impl BackupPolicy { - pub fn new() -> Self { +impl Default for BackupPolicy { + /// Create a default policy. + fn default() -> Self { Self { new: true, old_if_changed: true, } } +} +impl BackupPolicy { + /// Does a given file need to be backed up? pub fn needs_backup(&self, old: &LocalGeneration, new_entry: &FilesystemEntry) -> Reason { let new_name = new_entry.pathbuf(); - let reason = match old.get_file(&new_name) { + match old.get_file(&new_name) { Ok(None) => { if self.new { Reason::IsNew @@ -42,14 +58,9 @@ impl BackupPolicy { "needs_backup: lookup in old generation returned error, ignored: {:?}: {}", new_name, err ); - Reason::Error + Reason::GenerationLookupError } - }; - debug!( - "needs_backup: file {:?}: policy decision: {}", - new_name, reason - ); - reason + } } } diff --git a/src/schema.rs b/src/schema.rs new file mode 100644 index 0000000..ae8c00b --- /dev/null +++ b/src/schema.rs @@ -0,0 +1,173 @@ +//! Database schema versions. + +use serde::Serialize; + +/// The type of schema version components. +pub type VersionComponent = u32; + +/// Schema version of the database storing the generation. +/// +/// An Obnam client can restore a generation using schema version +/// (x,y), if the client supports a schema version (x,z). If z < y, +/// the client knows it may not be able to the generation faithfully, +/// and should warn the user about this. If z >= y, the client knows +/// it can restore the generation faithfully. If the client does not +/// support any schema version x, it knows it can't restore the backup +/// at all. +#[derive(Debug, Clone, Copy, Serialize)] +pub struct SchemaVersion { + /// Major version. + pub major: VersionComponent, + /// Minor version. + pub minor: VersionComponent, +} + +impl SchemaVersion { + /// Create a new schema version object. + pub fn new(major: VersionComponent, minor: VersionComponent) -> Self { + Self { major, minor } + } + + /// Return the major and minor version number component of a schema version. + pub fn version(&self) -> (VersionComponent, VersionComponent) { + (self.major, self.minor) + } + + /// Is this schema version compatible with another schema version? + pub fn is_compatible_with(&self, other: &Self) -> bool { + self.major == other.major && self.minor >= other.minor + } +} + +impl std::fmt::Display for SchemaVersion { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}.{}", self.major, self.minor) + } +} + +impl std::str::FromStr for SchemaVersion { + type Err = SchemaVersionError; + fn from_str(s: &str) -> Result<Self, Self::Err> { + if let Some(pos) = s.find('.') { + let major = parse_int(&s[..pos])?; + let minor = parse_int(&s[pos + 1..])?; + Ok(SchemaVersion::new(major, minor)) + } else { + Err(SchemaVersionError::Invalid(s.to_string())) + } + } +} + +fn parse_int(s: &str) -> Result<VersionComponent, SchemaVersionError> { + if let Ok(i) = s.parse() { + Ok(i) + } else { + Err(SchemaVersionError::InvalidComponent(s.to_string())) + } +} + +/// Possible errors from parsing schema versions. +#[derive(Debug, thiserror::Error, PartialEq, Eq)] +pub enum SchemaVersionError { + /// Failed to parse a string as a schema version. + #[error("Invalid schema version {0:?}")] + Invalid(String), + + /// Failed to parse a string as a schema version component. + #[error("Invalid schema version component {0:?}")] + InvalidComponent(String), +} + +#[cfg(test)] +mod test { + use super::*; + use std::str::FromStr; + + #[test] + fn from_string() { + let v = SchemaVersion::from_str("1.2").unwrap(); + assert_eq!(v.version(), (1, 2)); + } + + #[test] + fn from_string_fails_if_empty() { + match SchemaVersion::from_str("") { + Err(SchemaVersionError::Invalid(s)) => assert_eq!(s, ""), + _ => unreachable!(), + } + } + + #[test] + fn from_string_fails_if_empty_major() { + match SchemaVersion::from_str(".2") { + Err(SchemaVersionError::InvalidComponent(s)) => assert_eq!(s, ""), + _ => unreachable!(), + } + } + + #[test] + fn from_string_fails_if_empty_minor() { + match SchemaVersion::from_str("1.") { + Err(SchemaVersionError::InvalidComponent(s)) => assert_eq!(s, ""), + _ => unreachable!(), + } + } + + #[test] + fn from_string_fails_if_just_major() { + match SchemaVersion::from_str("1") { + Err(SchemaVersionError::Invalid(s)) => assert_eq!(s, "1"), + _ => unreachable!(), + } + } + + #[test] + fn from_string_fails_if_nonnumeric_major() { + match SchemaVersion::from_str("a.2") { + Err(SchemaVersionError::InvalidComponent(s)) => assert_eq!(s, "a"), + _ => unreachable!(), + } + } + + #[test] + fn from_string_fails_if_nonnumeric_minor() { + match SchemaVersion::from_str("1.a") { + Err(SchemaVersionError::InvalidComponent(s)) => assert_eq!(s, "a"), + _ => unreachable!(), + } + } + + #[test] + fn compatible_with_self() { + let v = SchemaVersion::new(1, 2); + assert!(v.is_compatible_with(&v)); + } + + #[test] + fn compatible_with_older_minor_version() { + let old = SchemaVersion::new(1, 2); + let new = SchemaVersion::new(1, 3); + assert!(new.is_compatible_with(&old)); + } + + #[test] + fn not_compatible_with_newer_minor_version() { + let old = SchemaVersion::new(1, 2); + let new = SchemaVersion::new(1, 3); + assert!(!old.is_compatible_with(&new)); + } + + #[test] + fn not_compatible_with_older_major_version() { + let old = SchemaVersion::new(1, 2); + let new = SchemaVersion::new(2, 0); + assert!(!new.is_compatible_with(&old)); + } + + #[test] + fn not_compatible_with_newer_major_version() { + let old = SchemaVersion::new(1, 2); + let new = SchemaVersion::new(2, 0); + assert!(!old.is_compatible_with(&new)); + } +} diff --git a/src/server.rs b/src/server.rs index 4d5880e..ffd4009 100644 --- a/src/server.rs +++ b/src/server.rs @@ -1,9 +1,81 @@ +//! Stuff related to the Obnam chunk server. + use crate::chunk::DataChunk; use crate::chunkid::ChunkId; use crate::chunkmeta::ChunkMeta; use serde::{Deserialize, Serialize}; use std::collections::HashMap; use std::default::Default; +use std::path::{Path, PathBuf}; + +/// Server configuration. +#[derive(Debug, Deserialize, Clone)] +#[serde(deny_unknown_fields)] +pub struct ServerConfig { + /// Path to directory where chunks are stored. + pub chunks: PathBuf, + /// Address where server is to listen. + pub address: String, + /// Path to TLS key. + pub tls_key: PathBuf, + /// Path to TLS certificate. + pub tls_cert: PathBuf, +} + +/// Possible errors wittht server configuration. +#[derive(Debug, thiserror::Error)] +pub enum ServerConfigError { + /// The chunks directory doesn't exist. + #[error("Directory for chunks {0} does not exist")] + ChunksDirNotFound(PathBuf), + + /// The TLS certificate doesn't exist. + #[error("TLS certificate {0} does not exist")] + TlsCertNotFound(PathBuf), + + /// The TLS key doesn't exist. + #[error("TLS key {0} does not exist")] + TlsKeyNotFound(PathBuf), + + /// Server address is wrong. + #[error("server address can't be resolved")] + BadServerAddress, + + /// Failed to read configuration file. + #[error("failed to read configuration file {0}: {1}")] + Read(PathBuf, std::io::Error), + + /// Failed to parse configuration file as YAML. + #[error("failed to parse configuration file as YAML: {0}")] + YamlParse(serde_yaml::Error), +} + +impl ServerConfig { + /// Read, parse, and check the server configuration file. + pub fn read_config(filename: &Path) -> Result<Self, ServerConfigError> { + let config = match std::fs::read_to_string(filename) { + Ok(config) => config, + Err(err) => return Err(ServerConfigError::Read(filename.to_path_buf(), err)), + }; + let config: Self = serde_yaml::from_str(&config).map_err(ServerConfigError::YamlParse)?; + config.check()?; + Ok(config) + } + + /// Check the configuration. + pub fn check(&self) -> Result<(), ServerConfigError> { + if !self.chunks.exists() { + return Err(ServerConfigError::ChunksDirNotFound(self.chunks.clone())); + } + if !self.tls_cert.exists() { + return Err(ServerConfigError::TlsCertNotFound(self.tls_cert.clone())); + } + if !self.tls_key.exists() { + return Err(ServerConfigError::TlsKeyNotFound(self.tls_key.clone())); + } + Ok(()) + } +} /// Result of creating a chunk. #[derive(Debug, Serialize)] @@ -12,17 +84,18 @@ pub struct Created { } impl Created { + /// Create a new created chunk id. pub fn new(id: ChunkId) -> Self { Created { id } } + /// Convert to JSON. pub fn to_json(&self) -> String { serde_json::to_string(&self).unwrap() } } /// Result of retrieving a chunk. - #[derive(Debug, Serialize)] pub struct Fetched { id: ChunkId, @@ -30,31 +103,36 @@ pub struct Fetched { } impl Fetched { + /// Create a new id for a fetched chunk. pub fn new(id: ChunkId, chunk: DataChunk) -> Self { Fetched { id, chunk } } + /// Convert to JSON. pub fn to_json(&self) -> String { serde_json::to_string(&self).unwrap() } } /// Result of a search. -#[derive(Debug, Default, PartialEq, Deserialize, Serialize)] +#[derive(Debug, Default, Eq, PartialEq, Deserialize, Serialize)] pub struct SearchHits { map: HashMap<String, ChunkMeta>, } impl SearchHits { + /// Insert a new chunk id to search results. pub fn insert(&mut self, id: ChunkId, meta: ChunkMeta) { self.map.insert(id.to_string(), meta); } - pub fn from_json(s: &str) -> anyhow::Result<Self> { + /// Convert from JSON. + pub fn from_json(s: &str) -> Result<Self, serde_json::Error> { let map = serde_json::from_str(s)?; Ok(SearchHits { map }) } + /// Convert to JSON. pub fn to_json(&self) -> String { serde_json::to_string(&self.map).unwrap() } @@ -63,6 +141,7 @@ impl SearchHits { #[cfg(test)] mod test_search_hits { use super::{ChunkMeta, SearchHits}; + use crate::label::Label; #[test] fn no_search_hits() { @@ -73,7 +152,8 @@ mod test_search_hits { #[test] fn one_search_hit() { let id = "abc".parse().unwrap(); - let meta = ChunkMeta::new("123"); + let sum = Label::sha256(b"123"); + let meta = ChunkMeta::new(&sum); let mut hits = SearchHits::default(); hits.insert(id, meta); eprintln!("hits: {:?}", hits); diff --git a/src/store.rs b/src/store.rs index e6cc71f..185370e 100644 --- a/src/store.rs +++ b/src/store.rs @@ -1,7 +1,7 @@ +//! Store chunks on-disk on server. + use crate::chunk::DataChunk; use crate::chunkid::ChunkId; -use crate::chunkmeta::ChunkMeta; -use anyhow::Context; use std::path::{Path, PathBuf}; /// Store chunks, with metadata, persistently. @@ -13,6 +13,9 @@ pub struct Store { dir: PathBuf, } +/// An error from a `Store` operation. +pub type StoreError = std::io::Error; + impl Store { /// Create a new Store to represent on-disk storage of chunks.x pub fn new(dir: &Path) -> Self { @@ -38,34 +41,34 @@ impl Store { } /// Save a chunk into a store. - pub fn save(&self, id: &ChunkId, meta: &ChunkMeta, chunk: &DataChunk) -> anyhow::Result<()> { + pub fn save(&self, id: &ChunkId, chunk: &DataChunk) -> Result<(), StoreError> { let (dir, metaname, dataname) = &self.filenames(id); if !dir.exists() { - let res = std::fs::create_dir_all(dir).into(); - if let Err(_) = res { - return res.with_context(|| format!("creating directory {}", dir.display())); - } + std::fs::create_dir_all(dir)?; } - std::fs::write(&metaname, meta.to_json())?; - std::fs::write(&dataname, chunk.data())?; + std::fs::write(metaname, chunk.meta().to_json())?; + std::fs::write(dataname, chunk.data())?; Ok(()) } /// Load a chunk from a store. - pub fn load(&self, id: &ChunkId) -> anyhow::Result<DataChunk> { - let (_, _, dataname) = &self.filenames(id); - let data = std::fs::read(&dataname)?; - let data = DataChunk::new(data); + pub fn load(&self, id: &ChunkId) -> Result<DataChunk, StoreError> { + let (_, metaname, dataname) = &self.filenames(id); + let meta = std::fs::read(metaname)?; + let meta = serde_json::from_slice(&meta)?; + + let data = std::fs::read(dataname)?; + let data = DataChunk::new(data, meta); Ok(data) } /// Delete a chunk from a store. - pub fn delete(&self, id: &ChunkId) -> anyhow::Result<()> { + pub fn delete(&self, id: &ChunkId) -> Result<(), StoreError> { let (_, metaname, dataname) = &self.filenames(id); - std::fs::remove_file(&metaname)?; - std::fs::remove_file(&dataname)?; + std::fs::remove_file(metaname)?; + std::fs::remove_file(dataname)?; Ok(()) } } diff --git a/src/workqueue.rs b/src/workqueue.rs new file mode 100644 index 0000000..6b3ce80 --- /dev/null +++ b/src/workqueue.rs @@ -0,0 +1,62 @@ +//! A queue of work for [`crate::engine::Engine`]. + +use tokio::sync::mpsc; + +/// A queue of work items. +/// +/// An abstraction for producing items of work. For example, chunks of +/// data in a file. The work items are put into an ordered queue to be +/// worked on by another task. The queue is limited in size so that it +/// doesn't grow impossibly large. This acts as a load-limiting +/// synchronizing mechanism. +/// +/// One async task produces work items and puts them into the queue, +/// another consumes them from the queue. If the producer is too fast, +/// the queue fills up, and the producer blocks when putting an item +/// into the queue. If the queue is empty, the consumer blocks until +/// there is something added to the queue. +/// +/// The work items need to be abstracted as a type, and that type is +/// given as a type parameter. +pub struct WorkQueue<T> { + rx: mpsc::Receiver<T>, + tx: Option<mpsc::Sender<T>>, + size: usize, +} + +impl<T> WorkQueue<T> { + /// Create a new work queue of a given maximum size. + pub fn new(queue_size: usize) -> Self { + let (tx, rx) = mpsc::channel(queue_size); + Self { + rx, + tx: Some(tx), + size: queue_size, + } + } + + /// Get maximum size of queue. + pub fn size(&self) -> usize { + self.size + } + + /// Add an item of work to the queue. + pub fn push(&self) -> mpsc::Sender<T> { + self.tx.as_ref().unwrap().clone() + } + + /// Signal that no more work items will be added to the queue. + /// + /// You **must** call this, as otherwise the `next` function will + /// wait indefinitely. + pub fn close(&mut self) { + // println!("Chunkify::close closing sender"); + self.tx = None; + } + + /// Get the oldest work item from the queue, if any. + pub async fn next(&mut self) -> Option<T> { + // println!("next called"); + self.rx.recv().await + } +} diff --git a/subplot/client.py b/subplot/client.py index d450b4c..09b1556 100644 --- a/subplot/client.py +++ b/subplot/client.py @@ -3,15 +3,35 @@ import os import yaml +def start_obnam(ctx): + start_chunk_server = globals()["start_chunk_server"] + install_obnam(ctx) + start_chunk_server(ctx) + + +def stop_obnam(ctx): + stop_chunk_server = globals()["stop_chunk_server"] + stop_chunk_server(ctx) + uninstall_obnam(ctx) + + def install_obnam(ctx): runcmd_prepend_to_path = globals()["runcmd_prepend_to_path"] srcdir = globals()["srcdir"] # Add the directory with built Rust binaries to the path. - runcmd_prepend_to_path(ctx, dirname=os.path.join(srcdir, "target", "debug")) + default_target = os.path.join(srcdir, "target") + target = os.environ.get("CARGO_TARGET_DIR", default_target) + runcmd_prepend_to_path(ctx, dirname=os.path.join(target, "debug")) + ctx["server-binary"] = os.path.join(target, "debug", "obnam-server") -def configure_client(ctx, filename=None): +def uninstall_obnam(ctx): + runcmd_run = globals()["runcmd_run"] + runcmd_run(ctx, ["chmod", "-R", "u+rwX", "."]) + + +def configure_client_without_init(ctx, filename=None): get_file = globals()["get_file"] assert ctx.get("server_url") is not None @@ -20,42 +40,37 @@ def configure_client(ctx, filename=None): config = yaml.safe_load(config) config["server_url"] = ctx["server_url"] + logging.debug(f"client config {filename}: {config}") + dirname = os.path.expanduser("~/.config/obnam") + if not os.path.exists(dirname): + os.makedirs(dirname) + filename = os.path.join(dirname, "obnam.yaml") + logging.debug(f"configure_client: filename={filename}") with open(filename, "w") as f: yaml.safe_dump(config, stream=f) -def run_obnam_restore(ctx, filename=None, genid=None, todir=None): - genid = ctx["vars"][genid] - run_obnam_restore_with_genref(ctx, filename=filename, genref=genid, todir=todir) - +def configure_client_with_init(ctx, filename=None): + runcmd_run = globals()["runcmd_run"] + runcmd_exit_code_is_zero = globals()["runcmd_exit_code_is_zero"] -def run_obnam_restore_latest(ctx, filename=None, todir=None): - run_obnam_restore_with_genref(ctx, filename=filename, genref="latest", todir=todir) + configure_client_without_init(ctx, filename=filename) + runcmd_run(ctx, ["obnam", "init", "--insecure-passphrase=hunter2"]) + runcmd_exit_code_is_zero(ctx) -def run_obnam_restore_with_genref(ctx, filename=None, genref=None, todir=None): +def run_obnam_restore(ctx, genid=None, todir=None): runcmd_run = globals()["runcmd_run"] - runcmd_run( - ctx, - [ - "env", - "RUST_LOG=obnam", - "obnam", - "--config", - filename, - "restore", - genref, - todir, - ], - ) - - -def run_obnam_get_chunk(ctx, filename=None, gen_id=None, todir=None): + genref = ctx["vars"][genid] + runcmd_run(ctx, ["env", "RUST_LOG=obnam", "obnam", "restore", genref, todir]) + + +def run_obnam_get_chunk(ctx, gen_id=None, todir=None): runcmd_run = globals()["runcmd_run"] gen_id = ctx["vars"][gen_id] logging.debug(f"run_obnam_get_chunk: gen_id={gen_id}") - runcmd_run(ctx, ["obnam", "--config", filename, "get-chunk", gen_id]) + runcmd_run(ctx, ["obnam", "get-chunk", gen_id]) def capture_generation_id(ctx, varname=None): @@ -111,3 +126,12 @@ def stdout_matches_file(ctx, filename=None): stdout = runcmd_get_stdout(ctx) data = open(filename).read() assert_eq(stdout, data) + + +def stdout_contains_home_dir_path(ctx, path=None): + runcmd_get_stdout = globals()["runcmd_get_stdout"] + stdout = runcmd_get_stdout(ctx) + wanted = os.path.abspath(os.path.normpath("./" + path)) + logging.debug(f"stdout_contains_home_dir_path: stdout={stdout!r}") + logging.debug(f"stdout_contains_home_dir_path: wanted={wanted!r}") + assert wanted in stdout diff --git a/subplot/client.yaml b/subplot/client.yaml index b1f9b19..104f479 100644 --- a/subplot/client.yaml +++ b/subplot/client.yaml @@ -1,32 +1,70 @@ +- given: "a working Obnam system" + impl: + python: + function: start_obnam + cleanup: stop_obnam + - given: "an installed obnam" - function: install_obnam + impl: + python: + function: install_obnam + cleanup: uninstall_obnam - given: "a client config based on {filename}" - function: configure_client + impl: + python: + function: configure_client_with_init + types: + filename: file -- when: "I invoke obnam --config {filename} restore <{genid}> {todir}" - function: run_obnam_restore +- given: "a client config, without passphrase, based on {filename}" + impl: + python: + function: configure_client_without_init + types: + filename: file -- when: "I invoke obnam --config {filename} restore latest {todir}" - function: run_obnam_restore_latest +- when: "I invoke obnam restore <{genid}> {todir}" + impl: + python: + function: run_obnam_restore -- when: "I invoke obnam --config {filename} get-chunk <{gen_id}>" - function: run_obnam_get_chunk +- when: "I invoke obnam get-chunk <{gen_id}>" + impl: + python: + function: run_obnam_get_chunk - then: "backup generation is {varname}" - function: capture_generation_id + impl: + python: + function: capture_generation_id - then: "generation list contains <{gen_id}>" - function: generation_list_contains + impl: + python: + function: generation_list_contains - then: "file {filename} was backed up because it was new" - function: file_was_new + impl: + python: + function: file_was_new - then: "file {filename} was backed up because it was changed" - function: file_was_changed + impl: + python: + function: file_was_changed - then: "file {filename} was not backed up because it was unchanged" - function: file_was_unchanged + impl: + python: + function: file_was_unchanged - then: "stdout matches file {filename}" - function: stdout_matches_file + impl: + python: + function: stdout_matches_file + +- then: "stdout contains home directory followed by {path}" + impl: + python: + function: stdout_contains_home_dir_path diff --git a/subplot/data.py b/subplot/data.py index a24cd0c..583e52d 100644 --- a/subplot/data.py +++ b/subplot/data.py @@ -1,16 +1,41 @@ +import json import logging import os import random +import socket +import stat +import yaml -def create_file_with_random_data(ctx, filename=None): - N = 128 - data = "".join(chr(random.randint(0, 255)) for i in range(N)).encode("UTF-8") +def create_file_with_given_data(ctx, filename=None, data=None): + logging.debug(f"creating file {filename} with {data!r}") dirname = os.path.dirname(filename) or "." - logging.debug(f"create_file_with_random_data: dirname={dirname}") os.makedirs(dirname, exist_ok=True) with open(filename, "wb") as f: - f.write(data) + f.write(data.encode("UTF-8")) + + +def create_file_with_random_data(ctx, filename=None): + N = 128 + data = "".join(chr(random.randint(0, 255)) for i in range(N)) + create_file_with_given_data(ctx, filename=filename, data=data) + + +def create_unix_socket(ctx, filename=None): + fd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + fd.bind(filename) + + +def create_fifo(ctx, filename=None): + os.mkfifo(filename) + + +def create_cachedir_tag_in(ctx, dirpath=None): + filepath = f"{dirpath}/CACHEDIR.TAG" + logging.debug(f"creating {filepath}") + os.makedirs(dirpath, exist_ok=True) + with open(filepath, "w") as f: + f.write("Signature: 8a477f597d28d172789f06886806bc55") def create_nonutf8_filename(ctx, dirname=None): @@ -24,7 +49,7 @@ def chmod_file(ctx, filename=None, mode=None): def create_symlink(ctx, linkname=None, target=None): - os.symlink(linkname, target) + os.symlink(target, linkname) def create_manifest_of_live(ctx, dirname=None, manifest=None): @@ -46,7 +71,26 @@ def _create_manifest_of_directory(ctx, dirname=None, manifest=None): runcmd_run(ctx, ["find", "-exec", "summain", "{}", "+"], cwd=dirname) assert runcmd_get_exit_code(ctx) == 0 stdout = runcmd_get_stdout(ctx) - open(manifest, "w").write(stdout) + with open(manifest, "w") as f: + f.write(stdout) + + +def file_is_restored(ctx, filename=None, restored=None): + filename = os.path.join(restored, "./" + filename) + exists = os.path.exists(filename) + logging.debug(f"restored? {filename} {exists}") + assert exists + + +def file_is_not_restored(ctx, filename=None, restored=None): + filename = os.path.join(restored, "./" + filename) + logging.info(r"verifying that {filename} does not exist") + try: + exists = os.path.exists(filename) + except PermissionError: + exists = False + logging.debug(f"restored? {filename} {exists}") + assert not exists def files_match(ctx, first=None, second=None): @@ -57,3 +101,115 @@ def files_match(ctx, first=None, second=None): logging.debug(f"files_match: f:\n{f}") logging.debug(f"files_match: s:\n{s}") assert_eq(f, s) + + +def convert_yaml_to_json(ctx, yaml_name=None, json_name=None): + with open(yaml_name) as f: + obj = yaml.safe_load(f) + with open(json_name, "w") as f: + json.dump(obj, f) + + +def match_stdout_to_json_file_superset(ctx, filename=None): + runcmd_get_stdout = globals()["runcmd_get_stdout"] + assert_eq = globals()["assert_eq"] + assert_dict_eq = globals()["assert_dict_eq"] + + stdout = runcmd_get_stdout(ctx) + stdout = json.loads(stdout.strip()) + obj = json.load(open(filename)) + logging.debug(f"match_stdout_to_json_file: stdout={stdout!r}") + logging.debug(f"match_stdout_to_json_file: file={obj!r}") + + if isinstance(obj, dict): + stdout = {key: value for key, value in stdout.items() if key in obj} + assert_dict_eq(obj, stdout) + elif isinstance(obj, list): + obj = {"key": obj} + stdout = {"key": stdout} + assert_dict_eq(obj, stdout) + assert_dict_eq(obj, stdout) + else: + assert_eq(obj, stdout) + + +def match_stdout_to_json_file_exactly(ctx, filename=None): + runcmd_get_stdout = globals()["runcmd_get_stdout"] + assert_eq = globals()["assert_eq"] + assert_dict_eq = globals()["assert_dict_eq"] + + stdout = runcmd_get_stdout(ctx) + stdout = json.loads(stdout.strip()) + obj = json.load(open(filename)) + logging.debug(f"match_stdout_to_json_file: stdout={stdout!r}") + logging.debug(f"match_stdout_to_json_file: file={obj!r}") + + if isinstance(obj, list): + obj = {"key": obj} + stdout = {"key": stdout} + assert_dict_eq(obj, stdout) + elif isinstance(obj, dict): + assert_dict_eq(obj, stdout) + else: + assert_eq(obj, stdout) + + +def manifests_match(ctx, expected=None, actual=None): + assert_eq = globals()["assert_eq"] + assert_dict_eq = globals()["assert_dict_eq"] + + logging.debug(f"comparing manifests {expected} and {actual}") + + expected_objs = list(yaml.safe_load_all(open(expected))) + actual_objs = list(yaml.safe_load_all(open(actual))) + + logging.debug(f"there are {len(expected_objs)} and {len(actual_objs)} objects") + + i = 0 + while expected_objs and actual_objs: + e = expected_objs.pop(0) + a = actual_objs.pop(0) + + logging.debug(f"comparing manifest objects at index {i}:") + logging.debug(f" expected: {e}") + logging.debug(f" actual : {a}") + assert_dict_eq(e, a) + + i += 1 + + logging.debug(f"remaining expected objecvts: {expected_objs}") + logging.debug(f"remaining actual objecvts : {actual_objs}") + assert_eq(expected_objs, []) + assert_eq(actual_objs, []) + + logging.debug(f"manifests {expected} and {actual} match") + + +def file_is_readable_by_owner(ctx, filename=None): + assert_eq = globals()["assert_eq"] + + st = os.lstat(filename) + mode = stat.S_IMODE(st.st_mode) + logging.debug("file mode: %o", mode) + assert_eq(mode, 0o400) + + +def file_does_not_contain(ctx, filename=None, pattern=None): + data = open(filename).read() + assert pattern not in data + + +def files_are_different(ctx, filename1=None, filename2=None): + assert_ne = globals()["assert_ne"] + + data1 = open(filename1, "rb").read() + data2 = open(filename2, "rb").read() + assert_ne(data1, data2) + + +def files_are_identical(ctx, filename1=None, filename2=None): + assert_eq = globals()["assert_eq"] + + data1 = open(filename1, "rb").read() + data2 = open(filename2, "rb").read() + assert_eq(data1, data2) diff --git a/subplot/data.yaml b/subplot/data.yaml index 7659319..533237f 100644 --- a/subplot/data.yaml +++ b/subplot/data.yaml @@ -1,25 +1,99 @@ -- given: > - a file (?P<filename>\\S+) containing "(?P<data>.*)" - regex: true - function: create_file_with_given_data +- given: a file {filename} containing "{data:text}" + impl: + python: + function: create_file_with_given_data - given: "a file {filename} containing some random data" - function: create_file_with_random_data + impl: + python: + function: create_file_with_random_data + +- given: "a Unix socket {filename}" + impl: + python: + function: create_unix_socket + +- given: "a named pipe {filename}" + impl: + python: + function: create_fifo + +- given: a cache directory tag in {dirpath} + impl: + python: + function: create_cachedir_tag_in - given: "a file in {dirname} with a non-UTF8 filename" - function: create_nonutf8_filename + impl: + python: + function: create_nonutf8_filename - given: file {filename} has mode {mode} - function: chmod_file + impl: + python: + function: chmod_file - given: symbolink link {linkname} that points at {target} - function: create_symlink + impl: + python: + function: create_symlink - given: a manifest of the directory {dirname} in {manifest} - function: create_manifest_of_live + impl: + python: + function: create_manifest_of_live - given: a manifest of the directory {dirname} restored in {restored} in {manifest} - function: create_manifest_of_restored + impl: + python: + function: create_manifest_of_restored + +- given: "JSON file {json_name} converted from YAML file {yaml_name}" + impl: + python: + function: convert_yaml_to_json + +- then: "stdout, as JSON, exactly matches file {filename}" + impl: + python: + function: match_stdout_to_json_file_exactly + +- then: "stdout, as JSON, has all the values in file {filename}" + impl: + python: + function: match_stdout_to_json_file_superset + +- then: "file {filename} is restored to {restored}" + impl: + python: + function: file_is_restored + +- then: "file {filename} is not restored to {restored}" + impl: + python: + function: file_is_not_restored + +- then: "manifests {expected} and {actual} match" + impl: + python: + function: manifests_match + +- then: "file {filename} is only readable by owner" + impl: + python: + function: file_is_readable_by_owner + +- then: "file {filename} does not contain \"{pattern:text}\"" + impl: + python: + function: file_does_not_contain + +- then: "files {filename1} and {filename2} are different" + impl: + python: + function: files_are_different -- then: files {first} and {second} match - function: files_match +- then: "files {filename1} and {filename2} are identical" + impl: + python: + function: files_are_identical diff --git a/subplot/server.py b/subplot/server.py index 289e181..a604733 100644 --- a/subplot/server.py +++ b/subplot/server.py @@ -5,8 +5,6 @@ import random import re import requests import shutil -import socket -import time import urllib3 import yaml @@ -14,7 +12,7 @@ import yaml urllib3.disable_warnings() -def start_chunk_server(ctx): +def start_chunk_server(ctx, env=None): daemon_start_on_port = globals()["daemon_start_on_port"] srcdir = globals()["srcdir"] @@ -35,7 +33,7 @@ def start_chunk_server(ctx): "address": f"localhost:{port}", } - server_binary = os.path.abspath(os.path.join(srcdir, "target", "debug", "obnam-server")) + server_binary = ctx["server-binary"] filename = "config.yaml" yaml.safe_dump(config, stream=open(filename, "w")) @@ -44,22 +42,18 @@ def start_chunk_server(ctx): ctx["server_url"] = f"https://{config['address']}" daemon_start_on_port( - ctx, - name="obnam-server", - path=server_binary, - args=filename, - port=port, + ctx, name="obnam-server", path=server_binary, args=filename, port=port, env=env ) -def stop_chunk_server(ctx): +def stop_chunk_server(ctx, env=None): logging.debug("Stopping obnam-server") daemon_stop = globals()["daemon_stop"] daemon_stop(ctx, name="obnam-server") def post_file(ctx, filename=None, path=None, header=None, json=None): - url = f"{ctx['server_url']}/chunks" + url = f"{ctx['server_url']}/v1/chunks" headers = {header: json} data = open(filename, "rb").read() _request(ctx, requests.post, url, headers=headers, data=data) @@ -71,12 +65,12 @@ def get_chunk_via_var(ctx, var=None): def get_chunk_by_id(ctx, chunk_id=None): - url = f"{ctx['server_url']}/chunks/{chunk_id}" + url = f"{ctx['server_url']}/v1/chunks/{chunk_id}" _request(ctx, requests.get, url) -def find_chunks_with_sha(ctx, sha=None): - url = f"{ctx['server_url']}/chunks?sha256={sha}" +def find_chunks_with_label(ctx, sha=None): + url = f"{ctx['server_url']}/v1/chunks?label={sha}" _request(ctx, requests.get, url) @@ -86,14 +80,16 @@ def delete_chunk_via_var(ctx, var=None): def delete_chunk_by_id(ctx, chunk_id=None): - url = f"{ctx['server_url']}/chunks/{chunk_id}" + url = f"{ctx['server_url']}/v1/chunks/{chunk_id}" _request(ctx, requests.delete, url) def make_chunk_file_be_empty(ctx, chunk_id=None): chunk_id = ctx["vars"][chunk_id] chunks = ctx["config"]["chunks"] - for (dirname, _, _) in os.walk(chunks): + logging.debug(f"trying to empty chunk {chunk_id}") + for (dirname, _, filenames) in os.walk(chunks): + logging.debug(f"found directory {dirname}, with {filenames}") filename = os.path.join(dirname, chunk_id + ".data") if os.path.exists(filename): logging.debug(f"emptying chunk file {filename}") @@ -138,6 +134,33 @@ def json_body_matches(ctx, wanted=None): assert_eq(body.get(key, "not.there"), wanted[key]) +def server_has_n_chunks(ctx, n=None): + assert_eq = globals()["assert_eq"] + n = int(n) + files = find_files(ctx["config"]["chunks"]) + files = [x for x in files if x.endswith(".data")] + logging.debug(f"server_has_n_file_chunks: n={n}") + logging.debug(f"server_has_n_file_chunks: len(files)={len(files)}") + logging.debug(f"server_has_n_file_chunks: files={files}") + assert_eq(n, len(files)) + + +def server_stderr_contains(ctx, wanted=None): + assert_eq = globals()["assert_eq"] + assert_eq(_server_stderr_contains(ctx, wanted), True) + + +def server_stderr_doesnt_contain(ctx, wanted=None): + assert_eq = globals()["assert_eq"] + assert_eq(_server_stderr_contains(ctx, wanted), False) + + +def find_files(root): + for dirname, _, names in os.walk(root): + for name in names: + yield os.path.join(dirname, name) + + # Make an HTTP request. def _request(ctx, method, url, headers=None, data=None): r = method(url, headers=headers, data=data, verify=False) @@ -177,3 +200,17 @@ def _expand_vars(ctx, s): result.append(value) s = s[m.end() :] return "".join(result) + + +def _server_stderr_contains(ctx, wanted): + daemon_get_stderr = globals()["daemon_get_stderr"] + + wanted = _expand_vars(ctx, wanted) + + stderr = daemon_get_stderr(ctx, "obnam-server") + + logging.debug(f"_server_stderr_contains:") + logging.debug(f" wanted: {wanted}") + logging.debug(f" stderr: {stderr}") + + return wanted in stderr diff --git a/subplot/server.yaml b/subplot/server.yaml index 68f8f0c..cf57931 100644 --- a/subplot/server.yaml +++ b/subplot/server.yaml @@ -1,45 +1,104 @@ - given: "a running chunk server" - function: start_chunk_server - cleanup: stop_chunk_server + impl: + python: + function: start_chunk_server + cleanup: stop_chunk_server + +- given: "a running chunk server with environment {env:text}" + impl: + python: + function: start_chunk_server + cleanup: stop_chunk_server - when: "the chunk server is stopped" - function: stop_chunk_server + impl: + python: + function: stop_chunk_server - when: "I POST (?P<filename>\\S+) to (?P<path>\\S+), with (?P<header>\\S+): (?P<json>.*)" regex: true - function: post_file + types: + filename: word + path: word + header: word + json: text + impl: + python: + function: post_file -- when: "I GET /chunks/<{var}>" - function: get_chunk_via_var +- when: "I GET /v1/chunks/<{var}>" + impl: + python: + function: get_chunk_via_var -- when: "I try to GET /chunks/{chunk_id}" - function: get_chunk_by_id +- when: "I try to GET /v1/chunks/{chunk_id}" + impl: + python: + function: get_chunk_by_id -- when: "I GET /chunks?sha256={sha}" +- when: "I GET /v1/chunks?label={sha}" regex: false - function: find_chunks_with_sha + impl: + python: + function: find_chunks_with_label -- when: "I DELETE /chunks/<{var}>" - function: delete_chunk_via_var +- when: "I DELETE /v1/chunks/<{var}>" + impl: + python: + function: delete_chunk_via_var -- when: "I try to DELETE /chunks/{chunk_id}" - function: delete_chunk_by_id +- when: "I try to DELETE /v1/chunks/{chunk_id}" + impl: + python: + function: delete_chunk_by_id - when: "chunk <{chunk_id}> on chunk server is replaced by an empty file" - function: make_chunk_file_be_empty + impl: + python: + function: make_chunk_file_be_empty - then: "HTTP status code is {status}" - function: status_code_is + impl: + python: + function: status_code_is - then: "{header} is {value}" - function: header_is + impl: + python: + function: header_is - then: "the JSON body has a field {field}, henceforth {var}" - function: remember_json_field + types: + field: word + var: word + impl: + python: + function: remember_json_field - then: "the JSON body matches (?P<wanted>.*)" regex: true - function: json_body_matches + types: + wanted: text + impl: + python: + function: json_body_matches - then: "the body matches file {filename}" - function: body_matches_file + impl: + python: + function: body_matches_file + +- then: "server has {n:int} chunks" + impl: + python: + function: server_has_n_chunks + +- then: chunk server's stderr contains "{wanted:text}" + impl: + python: + function: server_stderr_contains + +- then: chunk server's stderr doesn't contain "{wanted:text}" + impl: + python: + function: server_stderr_doesnt_contain diff --git a/subplot/vendored/daemon.md b/subplot/vendored/daemon.md deleted file mode 100644 index 131dcb1..0000000 --- a/subplot/vendored/daemon.md +++ /dev/null @@ -1,38 +0,0 @@ -# Introduction - -The [Subplot][] library `daemon` for Python provides scenario steps -and their implementations for running a background process and -terminating at the end of the scenario. - -[Subplot]: https://subplot.liw.fi/ - -This document explains the acceptance criteria for the library and how -they're verified. It uses the steps and functions from the -`lib/daemon` library. The scenarios all have the same structure: run a -command, then examine the exit code, verify the process is running. - -# Daemon is started and terminated - -This scenario starts a background process, verifies it's started, and -verifies it's terminated after the scenario ends. - -~~~scenario -given there is no "/bin/sleep 12765" process -when I start "/bin/sleep 12765" as a background process as sleepyhead -then a process "/bin/sleep 12765" is running -when I stop background process sleepyhead -then there is no "/bin/sleep 12765" process -~~~ - - - ---- -title: Acceptance criteria for the lib/daemon Subplot library -author: The Subplot project -bindings: -- daemon.yaml -template: python -functions: -- daemon.py -- runcmd.py -... diff --git a/subplot/vendored/daemon.py b/subplot/vendored/daemon.py deleted file mode 100644 index febf392..0000000 --- a/subplot/vendored/daemon.py +++ /dev/null @@ -1,139 +0,0 @@ -import logging -import os -import signal -import socket -import subprocess -import time - - -# Start a daemon that will open a port on localhost. -def daemon_start_on_port(ctx, path=None, args=None, name=None, port=None): - daemon_start(ctx, path=path, args=args, name=name) - daemon_wait_for_port("localhost", port) - - -# Start a daeamon, get its PID. Don't wait for a port or anything. This is -# meant for background processes that don't have port. Useful for testing the -# lib/daemon library of Subplot, but not much else. -def daemon_start(ctx, path=None, args=None, name=None): - runcmd_run = globals()["runcmd_run"] - runcmd_exit_code_is = globals()["runcmd_exit_code_is"] - runcmd_get_exit_code = globals()["runcmd_get_exit_code"] - runcmd_get_stderr = globals()["runcmd_get_stderr"] - runcmd_prepend_to_path = globals()["runcmd_prepend_to_path"] - - argv = [path] + args.split() - - logging.debug(f"Starting daemon {name}") - logging.debug(f" ctx={ctx.as_dict()}") - logging.debug(f" name={name}") - logging.debug(f" path={path}") - logging.debug(f" args={args}") - logging.debug(f" argv={argv}") - - ns = ctx.declare("_daemon") - - this = ns[name] = { - "pid-file": f"{name}.pid", - "stderr": f"{name}.stderr", - "stdout": f"{name}.stdout", - } - - # Debian installs `daemonize` to /usr/sbin, which isn't part of the minimal - # environment that Subplot sets up. So we add /usr/sbin to the PATH. - runcmd_prepend_to_path(ctx, "/usr/sbin") - runcmd_run( - ctx, - [ - "daemonize", - "-c", - os.getcwd(), - "-p", - this["pid-file"], - "-e", - this["stderr"], - "-o", - this["stdout"], - ] - + argv, - ) - - # Check that daemonize has exited OK. If it hasn't, it didn't start the - # background process at all. If so, log the stderr in case there was - # something useful there for debugging. - exit = runcmd_get_exit_code(ctx) - if exit != 0: - stderr = runcmd_get_stderr(ctx) - logging.error(f"daemon {name} stderr: {stderr}") - runcmd_exit_code_is(ctx, 0) - - # Get the pid of the background process, from the pid file created by - # daemonize. We don't need to wait for it, since we know daemonize already - # exited. If it isn't there now, it's won't appear later. - if not os.path.exists(this["pid-file"]): - raise Exception("daemonize didn't create a PID file") - - this["pid"] = _daemon_wait_for_pid(this["pid-file"], 10.0) - - logging.debug(f"Started daemon {name}") - logging.debug(f" pid={this['pid']}") - logging.debug(f" ctx={ctx.as_dict()}") - - -def _daemon_wait_for_pid(filename, timeout): - start = time.time() - while time.time() < start + timeout: - with open(filename) as f: - data = f.read().strip() - if data: - return int(data) - raise Exception("daemonize created a PID file without a PID") - - -def daemon_wait_for_port(host, port, timeout=3.0): - addr = (host, port) - until = time.time() + timeout - while True: - try: - s = socket.create_connection(addr, timeout=timeout) - s.close() - return - except socket.timeout: - logging.error(f"daemon did not respond at port {port} within {timeout} seconds") - raise - except socket.error as e: - logging.info(f"could not connect to daemon at {port}: {e}") - pass - if time.time() >= until: - logging.error(f"could not connect to daemon at {port} within {timeout} seconds") - raise ConnectionRefusedError() - - -# Stop a daemon. -def daemon_stop(ctx, name=None): - logging.debug(f"Stopping daemon {name}") - ns = ctx.declare("_daemon") - logging.debug(f" ns={ns}") - pid = ns[name]["pid"] - signo = signal.SIGKILL - - logging.debug(f"Terminating process {pid} with signal {signo}") - try: - os.kill(pid, signo) - except ProcessLookupError: - logging.warning("Process did not actually exist (anymore?)") - - -def daemon_no_such_process(ctx, args=None): - assert not _daemon_pgrep(args) - - -def daemon_process_exists(ctx, args=None): - assert _daemon_pgrep(args) - - -def _daemon_pgrep(pattern): - logging.info(f"checking if process exists: pattern={pattern}") - exit = subprocess.call(["pgrep", "-laf", pattern]) - logging.info(f"exit code: {exit}") - return exit == 0 diff --git a/subplot/vendored/daemon.yaml b/subplot/vendored/daemon.yaml deleted file mode 100644 index 6165c62..0000000 --- a/subplot/vendored/daemon.yaml +++ /dev/null @@ -1,17 +0,0 @@ -- given: there is no "{args:text}" process - function: daemon_no_such_process - -- when: I start "{path}{args:text}" as a background process as {name}, on port {port} - function: daemon_start_on_port - -- when: I start "{path}{args:text}" as a background process as {name} - function: daemon_start - -- when: I stop background process {name} - function: daemon_stop - -- then: a process "{args:text}" is running - function: daemon_process_exists - -- then: there is no "{args:text}" process - function: daemon_no_such_process diff --git a/subplot/vendored/files.md b/subplot/vendored/files.md deleted file mode 100644 index 68ef1ac..0000000 --- a/subplot/vendored/files.md +++ /dev/null @@ -1,82 +0,0 @@ -# Introduction - -The [Subplot][] library `files` provides scenario steps and their -implementations for managing files on the file system during tests. -The library consists of a bindings file `lib/files.yaml` and -implementations in Python in `lib/files.py`. - -[Subplot]: https://subplot.liw.fi/ - -This document explains the acceptance criteria for the library and how -they're verified. It uses the steps and functions from the `files` -library. - -# Create on-disk files from embedded files - -Subplot allows the source document to embed test files, and the -`files` library provides steps to create real, on-disk files from -the embedded files. - -~~~scenario -given file hello.txt -then file hello.txt exists -and file hello.txt contains "hello, world" -and file other.txt does not exist -given file other.txt from hello.txt -then file other.txt exists -and files hello.txt and other.txt match -and only files hello.txt, other.txt exist -~~~ - -~~~{#hello.txt .file .numberLines} -hello, world -~~~ - - -# File metadata - -These steps create files and manage their metadata. - -~~~scenario -given file hello.txt -when I remember metadata for file hello.txt -then file hello.txt has same metadata as before - -when I write "yo" to file hello.txt -then file hello.txt has different metadata from before -~~~ - -# File modification time - -These steps manipulate and test file modification times. - -~~~scenario -given file foo.dat has modification time 1970-01-02 03:04:05 -then file foo.dat has a very old modification time - -when I touch file foo.dat -then file foo.dat has a very recent modification time -~~~ - - -# File contents - -These steps verify contents of files. - -~~~scenario -given file hello.txt -then file hello.txt contains "hello, world" -and file hello.txt matches regex "hello, .*" -and file hello.txt matches regex /hello, .*/ -~~~ - - ---- -title: Acceptance criteria for the files Subplot library -author: The Subplot project -template: python -bindings: -- files.yaml -functions: -- files.py -... diff --git a/subplot/vendored/files.py b/subplot/vendored/files.py deleted file mode 100644 index ec37b9d..0000000 --- a/subplot/vendored/files.py +++ /dev/null @@ -1,158 +0,0 @@ -import logging -import os -import re -import time - - -def files_create_from_embedded(ctx, filename=None): - files_create_from_embedded_with_other_name( - ctx, filename_on_disk=filename, embedded_filename=filename - ) - - -def files_create_from_embedded_with_other_name( - ctx, filename_on_disk=None, embedded_filename=None -): - get_file = globals()["get_file"] - with open(filename_on_disk, "wb") as f: - f.write(get_file(embedded_filename)) - - -def files_create_from_text(ctx, filename=None, text=None): - with open(filename, "w") as f: - f.write(text) - - -def files_file_exists(ctx, filename=None): - assert_eq = globals()["assert_eq"] - assert_eq(os.path.exists(filename), True) - - -def files_file_does_not_exist(ctx, filename=None): - assert_eq = globals()["assert_eq"] - assert_eq(os.path.exists(filename), False) - - -def files_only_these_exist(ctx, filenames=None): - assert_eq = globals()["assert_eq"] - filenames = filenames.replace(",", "").split() - assert_eq(set(os.listdir(".")), set(filenames)) - - -def files_file_contains(ctx, filename=None, data=None): - assert_eq = globals()["assert_eq"] - with open(filename, "rb") as f: - actual = f.read() - actual = actual.decode("UTF-8") - assert_eq(data in actual, True) - - -def files_file_matches_regex(ctx, filename=None, regex=None): - assert_eq = globals()["assert_eq"] - with open(filename) as f: - content = f.read() - m = re.search(regex, content) - if m is None: - logging.debug(f"files_file_matches_regex: no match") - logging.debug(f" filenamed: {filename}") - logging.debug(f" regex: {regex}") - logging.debug(f" content: {regex}") - logging.debug(f" match: {m}") - assert_eq(bool(m), True) - - -def files_match(ctx, filename1=None, filename2=None): - assert_eq = globals()["assert_eq"] - with open(filename1, "rb") as f: - data1 = f.read() - with open(filename2, "rb") as f: - data2 = f.read() - assert_eq(data1, data2) - - -def files_touch_with_timestamp( - ctx, - filename=None, - year=None, - month=None, - day=None, - hour=None, - minute=None, - second=None, -): - t = ( - int(year), - int(month), - int(day), - int(hour), - int(minute), - int(second), - -1, - -1, - -1, - ) - ts = time.mktime(t) - _files_touch(filename, ts) - - -def files_touch(ctx, filename=None): - _files_touch(filename, None) - - -def _files_touch(filename, ts): - if not os.path.exists(filename): - open(filename, "w").close() - times = None - if ts is not None: - times = (ts, ts) - os.utime(filename, times=times) - - -def files_mtime_is_recent(ctx, filename=None): - st = os.stat(filename) - age = abs(st.st_mtime - time.time()) - assert age < 1.0 - - -def files_mtime_is_ancient(ctx, filename=None): - st = os.stat(filename) - age = abs(st.st_mtime - time.time()) - year = 365 * 24 * 60 * 60 - required = 39 * year - logging.debug(f"ancient? mtime={st.st_mtime} age={age} required={required}") - assert age > required - - -def files_remember_metadata(ctx, filename=None): - meta = _files_remembered(ctx) - meta[filename] = _files_get_metadata(filename) - logging.debug("files_remember_metadata:") - logging.debug(f" meta: {meta}") - logging.debug(f" ctx: {ctx}") - - -# Check that current metadata of a file is as stored in the context. -def files_has_remembered_metadata(ctx, filename=None): - assert_eq = globals()["assert_eq"] - meta = _files_remembered(ctx) - logging.debug("files_has_remembered_metadata:") - logging.debug(f" meta: {meta}") - logging.debug(f" ctx: {ctx}") - assert_eq(meta[filename], _files_get_metadata(filename)) - - -def files_has_different_metadata(ctx, filename=None): - assert_ne = globals()["assert_ne"] - meta = _files_remembered(ctx) - assert_ne(meta[filename], _files_get_metadata(filename)) - - -def _files_remembered(ctx): - ns = ctx.declare("_files") - return ns.get("remembered-metadata", {}) - - -def _files_get_metadata(filename): - st = os.lstat(filename) - keys = ["st_dev", "st_gid", "st_ino", "st_mode", "st_mtime", "st_size", "st_uid"] - return {key: getattr(st, key) for key in keys} diff --git a/subplot/vendored/files.yaml b/subplot/vendored/files.yaml deleted file mode 100644 index be69920..0000000 --- a/subplot/vendored/files.yaml +++ /dev/null @@ -1,62 +0,0 @@ -- given: file {filename} - function: files_create_from_embedded - types: - filename: file - -- given: file {filename_on_disk} from {embedded_filename} - function: files_create_from_embedded_with_other_name - types: - embedded_filename: file - -- given: file {filename} has modification time {year}-{month}-{day} {hour}:{minute}:{second} - function: files_touch_with_timestamp - -- when: I write "(?P<text>.*)" to file (?P<filename>\S+) - regex: true - function: files_create_from_text - -- when: I remember metadata for file {filename} - function: files_remember_metadata - -- when: I touch file {filename} - function: files_touch - -- then: file {filename} exists - function: files_file_exists - -- then: file {filename} does not exist - function: files_file_does_not_exist - -- then: only files (?P<filenames>.+) exist - function: files_only_these_exist - regex: true - -- then: file (?P<filename>\S+) contains "(?P<data>.*)" - regex: true - function: files_file_contains - -- then: file (?P<filename>\S+) matches regex /(?P<regex>.*)/ - regex: true - function: files_file_matches_regex - -- then: file (?P<filename>\S+) matches regex "(?P<regex>.*)" - regex: true - function: files_file_matches_regex - -- then: files {filename1} and {filename2} match - function: files_match - -- then: file {filename} has same metadata as before - function: files_has_remembered_metadata - -- then: file {filename} has different metadata from before - function: files_has_different_metadata - -- then: file {filename} has changed from before - function: files_has_different_metadata - -- then: file {filename} has a very recent modification time - function: files_mtime_is_recent - -- then: file {filename} has a very old modification time - function: files_mtime_is_ancient diff --git a/subplot/vendored/runcmd.md b/subplot/vendored/runcmd.md deleted file mode 100644 index a9d4ed4..0000000 --- a/subplot/vendored/runcmd.md +++ /dev/null @@ -1,170 +0,0 @@ -# Introduction - -The [Subplot][] library `runcmd` for Python provides scenario steps -and their implementations for running Unix commands and examining the -results. The library consists of a bindings file `lib/runcmd.yaml` and -implementations in Python in `lib/runcmd.py`. There is no Bash -version. - -[Subplot]: https://subplot.liw.fi/ - -This document explains the acceptance criteria for the library and how -they're verified. It uses the steps and functions from the -`lib/runcmd` library. The scenarios all have the same structure: run a -command, then examine the exit code, standard output (stdout for -short), or standard error output (stderr) of the command. - -The scenarios use the Unix commands `/bin/true` and `/bin/false` to -generate exit codes, and `/bin/echo` to produce stdout. To generate -stderr, they use the little helper script below. - -~~~{#err.sh .file .sh .numberLines} -#!/bin/sh -echo "$@" 1>&2 -~~~ - -# Check exit code - -These scenarios verify the exit code. To make it easier to write -scenarios in language that flows more naturally, there are a couple of -variations. - -## Successful execution - -~~~scenario -when I run /bin/true -then exit code is 0 -and command is successful -~~~ - -## Failed execution - -~~~scenario -when I try to run /bin/false -then exit code is not 0 -and command fails -~~~ - -# Check output has what we want - -These scenarios verify that stdout or stderr do have something we want -to have. - -## Check stdout is exactly as wanted - -Note that the string is surrounded by double quotes to make it clear -to the reader what's inside. Also, C-style string escapes are -understood. - -~~~scenario -when I run /bin/echo hello, world -then stdout is exactly "hello, world\n" -~~~ - -## Check stderr is exactly as wanted - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hello, world -then stderr is exactly "hello, world\n" -~~~ - -## Check stdout using sub-string search - -Exact string comparisons are not always enough, so we can verify a -sub-string is in output. - -~~~scenario -when I run /bin/echo hello, world -then stdout contains "world\n" -and exit code is 0 -~~~ - -## Check stderr using sub-string search - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hello, world -then stderr contains "world\n" -~~~ - -## Check stdout using regular expressions - -Fixed strings are not always enough, so we can verify output matches a -regular expression. Note that the regular expression is not delimited -and does not get any C-style string escaped decoded. - -~~~scenario -when I run /bin/echo hello, world -then stdout matches regex world$ -~~~ - -## Check stderr using regular expressions - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hello, world -then stderr matches regex world$ -~~~ - -# Check output doesn't have what we want to avoid - -These scenarios verify that the stdout or stderr do not -have something we want to avoid. - -## Check stdout is not exactly something - -~~~scenario -when I run /bin/echo hi -then stdout isn't exactly "hello, world\n" -~~~ - -## Check stderr is not exactly something - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hi -then stderr isn't exactly "hello, world\n" -~~~ - -## Check stdout doesn't contain sub-string - -~~~scenario -when I run /bin/echo hi -then stdout doesn't contain "world" -~~~ - -## Check stderr doesn't contain sub-string - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hi -then stderr doesn't contain "world" -~~~ - -## Check stdout doesn't match regular expression - -~~~scenario -when I run /bin/echo hi -then stdout doesn't match regex world$ - -~~~ - -## Check stderr doesn't match regular expressions - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hi -then stderr doesn't match regex world$ -~~~ - - ---- -title: Acceptance criteria for the lib/runcmd Subplot library -author: The Subplot project -template: python -bindings: -- runcmd.yaml -functions: -- runcmd.py -... diff --git a/subplot/vendored/runcmd.py b/subplot/vendored/runcmd.py deleted file mode 100644 index a2564c6..0000000 --- a/subplot/vendored/runcmd.py +++ /dev/null @@ -1,252 +0,0 @@ -import logging -import os -import re -import shlex -import subprocess - - -# -# Helper functions. -# - -# Get exit code or other stored data about the latest command run by -# runcmd_run. - - -def _runcmd_get(ctx, name): - ns = ctx.declare("_runcmd") - return ns[name] - - -def runcmd_get_exit_code(ctx): - return _runcmd_get(ctx, "exit") - - -def runcmd_get_stdout(ctx): - return _runcmd_get(ctx, "stdout") - - -def runcmd_get_stdout_raw(ctx): - return _runcmd_get(ctx, "stdout.raw") - - -def runcmd_get_stderr(ctx): - return _runcmd_get(ctx, "stderr") - - -def runcmd_get_stderr_raw(ctx): - return _runcmd_get(ctx, "stderr.raw") - - -def runcmd_get_argv(ctx): - return _runcmd_get(ctx, "argv") - - -# Run a command, given an argv and other arguments for subprocess.Popen. -# -# This is meant to be a helper function, not bound directly to a step. The -# stdout, stderr, and exit code are stored in the "_runcmd" namespace in the -# ctx context. -def runcmd_run(ctx, argv, **kwargs): - ns = ctx.declare("_runcmd") - - # The Subplot Python template empties os.environ at startup, modulo a small - # number of variables with carefully chosen values. Here, we don't need to - # care about what those variables are, but we do need to not overwrite - # them, so we just add anything in the env keyword argument, if any, to - # os.environ. - env = dict(os.environ) - for key, arg in kwargs.pop("env", {}).items(): - env[key] = arg - - pp = ns.get("path-prefix") - if pp: - env["PATH"] = pp + ":" + env["PATH"] - - logging.debug(f"runcmd_run") - logging.debug(f" argv: {argv}") - logging.debug(f" env: {env}") - p = subprocess.Popen( - argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, **kwargs - ) - stdout, stderr = p.communicate("") - ns["argv"] = argv - ns["stdout.raw"] = stdout - ns["stderr.raw"] = stderr - ns["stdout"] = stdout.decode("utf-8") - ns["stderr"] = stderr.decode("utf-8") - ns["exit"] = p.returncode - logging.debug(f" ctx: {ctx}") - logging.debug(f" ns: {ns}") - - -# Step: prepend srcdir to PATH whenever runcmd runs a command. -def runcmd_helper_srcdir_path(ctx): - srcdir = globals()["srcdir"] - runcmd_prepend_to_path(ctx, srcdir) - - -# Step: This creates a helper script. -def runcmd_helper_script(ctx, filename=None): - get_file = globals()["get_file"] - with open(filename, "wb") as f: - f.write(get_file(filename)) - - -# -# Step functions for running commands. -# - - -def runcmd_prepend_to_path(ctx, dirname=None): - ns = ctx.declare("_runcmd") - pp = ns.get("path-prefix", "") - if pp: - pp = f"{pp}:{dirname}" - else: - pp = dirname - ns["path-prefix"] = pp - - -def runcmd_step(ctx, argv0=None, args=None): - runcmd_try_to_run(ctx, argv0=argv0, args=args) - runcmd_exit_code_is_zero(ctx) - - -def runcmd_try_to_run(ctx, argv0=None, args=None): - argv = [shlex.quote(argv0)] + shlex.split(args) - runcmd_run(ctx, argv) - - -# -# Step functions for examining exit codes. -# - - -def runcmd_exit_code_is_zero(ctx): - runcmd_exit_code_is(ctx, exit=0) - - -def runcmd_exit_code_is(ctx, exit=None): - assert_eq = globals()["assert_eq"] - assert_eq(runcmd_get_exit_code(ctx), int(exit)) - - -def runcmd_exit_code_is_nonzero(ctx): - runcmd_exit_code_is_not(ctx, exit=0) - - -def runcmd_exit_code_is_not(ctx, exit=None): - assert_ne = globals()["assert_ne"] - assert_ne(runcmd_get_exit_code(ctx), int(exit)) - - -# -# Step functions and helpers for examining output in various ways. -# - - -def runcmd_stdout_is(ctx, text=None): - _runcmd_output_is(runcmd_get_stdout(ctx), text) - - -def runcmd_stdout_isnt(ctx, text=None): - _runcmd_output_isnt(runcmd_get_stdout(ctx), text) - - -def runcmd_stderr_is(ctx, text=None): - _runcmd_output_is(runcmd_get_stderr(ctx), text) - - -def runcmd_stderr_isnt(ctx, text=None): - _runcmd_output_isnt(runcmd_get_stderr(ctx), text) - - -def _runcmd_output_is(actual, wanted): - assert_eq = globals()["assert_eq"] - wanted = bytes(wanted, "utf8").decode("unicode_escape") - logging.debug("_runcmd_output_is:") - logging.debug(f" actual: {actual!r}") - logging.debug(f" wanted: {wanted!r}") - assert_eq(actual, wanted) - - -def _runcmd_output_isnt(actual, wanted): - assert_ne = globals()["assert_ne"] - wanted = bytes(wanted, "utf8").decode("unicode_escape") - logging.debug("_runcmd_output_isnt:") - logging.debug(f" actual: {actual!r}") - logging.debug(f" wanted: {wanted!r}") - assert_ne(actual, wanted) - - -def runcmd_stdout_contains(ctx, text=None): - _runcmd_output_contains(runcmd_get_stdout(ctx), text) - - -def runcmd_stdout_doesnt_contain(ctx, text=None): - _runcmd_output_doesnt_contain(runcmd_get_stdout(ctx), text) - - -def runcmd_stderr_contains(ctx, text=None): - _runcmd_output_contains(runcmd_get_stderr(ctx), text) - - -def runcmd_stderr_doesnt_contain(ctx, text=None): - _runcmd_output_doesnt_contain(runcmd_get_stderr(ctx), text) - - -def _runcmd_output_contains(actual, wanted): - assert_eq = globals()["assert_eq"] - wanted = bytes(wanted, "utf8").decode("unicode_escape") - logging.debug("_runcmd_output_contains:") - logging.debug(f" actual: {actual!r}") - logging.debug(f" wanted: {wanted!r}") - assert_eq(wanted in actual, True) - - -def _runcmd_output_doesnt_contain(actual, wanted): - assert_ne = globals()["assert_ne"] - wanted = bytes(wanted, "utf8").decode("unicode_escape") - logging.debug("_runcmd_output_doesnt_contain:") - logging.debug(f" actual: {actual!r}") - logging.debug(f" wanted: {wanted!r}") - assert_ne(wanted in actual, True) - - -def runcmd_stdout_matches_regex(ctx, regex=None): - _runcmd_output_matches_regex(runcmd_get_stdout(ctx), regex) - - -def runcmd_stdout_doesnt_match_regex(ctx, regex=None): - _runcmd_output_doesnt_match_regex(runcmd_get_stdout(ctx), regex) - - -def runcmd_stderr_matches_regex(ctx, regex=None): - _runcmd_output_matches_regex(runcmd_get_stderr(ctx), regex) - - -def runcmd_stderr_doesnt_match_regex(ctx, regex=None): - _runcmd_output_doesnt_match_regex(runcmd_get_stderr(ctx), regex) - - -def _runcmd_output_matches_regex(actual, regex): - assert_ne = globals()["assert_ne"] - r = re.compile(regex) - m = r.search(actual) - logging.debug("_runcmd_output_matches_regex:") - logging.debug(f" actual: {actual!r}") - logging.debug(f" regex: {regex!r}") - logging.debug(f" match: {m}") - assert_ne(m, None) - - -def _runcmd_output_doesnt_match_regex(actual, regex): - assert_eq = globals()["assert_eq"] - r = re.compile(regex) - m = r.search(actual) - logging.debug("_runcmd_output_doesnt_match_regex:") - logging.debug(f" actual: {actual!r}") - logging.debug(f" regex: {regex!r}") - logging.debug(f" match: {m}") - assert_eq(m, None) diff --git a/subplot/vendored/runcmd.yaml b/subplot/vendored/runcmd.yaml deleted file mode 100644 index 48dde90..0000000 --- a/subplot/vendored/runcmd.yaml +++ /dev/null @@ -1,83 +0,0 @@ -# Steps to run commands. - -- given: helper script {filename} for runcmd - function: runcmd_helper_script - -- given: srcdir is in the PATH - function: runcmd_helper_srcdir_path - -- when: I run (?P<argv0>\S+)(?P<args>.*) - regex: true - function: runcmd_step - -- when: I try to run (?P<argv0>\S+)(?P<args>.*) - regex: true - function: runcmd_try_to_run - -# Steps to examine exit code of latest command. - -- then: exit code is {exit} - function: runcmd_exit_code_is - -- then: exit code is not {exit} - function: runcmd_exit_code_is_not - -- then: command is successful - function: runcmd_exit_code_is_zero - -- then: command fails - function: runcmd_exit_code_is_nonzero - -# Steps to examine stdout/stderr for exact content. - -- then: stdout is exactly "(?P<text>.*)" - regex: true - function: runcmd_stdout_is - -- then: "stdout isn't exactly \"(?P<text>.*)\"" - regex: true - function: runcmd_stdout_isnt - -- then: stderr is exactly "(?P<text>.*)" - regex: true - function: runcmd_stderr_is - -- then: "stderr isn't exactly \"(?P<text>.*)\"" - regex: true - function: runcmd_stderr_isnt - -# Steps to examine stdout/stderr for sub-strings. - -- then: stdout contains "(?P<text>.*)" - regex: true - function: runcmd_stdout_contains - -- then: "stdout doesn't contain \"(?P<text>.*)\"" - regex: true - function: runcmd_stdout_doesnt_contain - -- then: stderr contains "(?P<text>.*)" - regex: true - function: runcmd_stderr_contains - -- then: "stderr doesn't contain \"(?P<text>.*)\"" - regex: true - function: runcmd_stderr_doesnt_contain - -# Steps to match stdout/stderr against regular expressions. - -- then: stdout matches regex (?P<regex>.*) - regex: true - function: runcmd_stdout_matches_regex - -- then: stdout doesn't match regex (?P<regex>.*) - regex: true - function: runcmd_stdout_doesnt_match_regex - -- then: stderr matches regex (?P<regex>.*) - regex: true - function: runcmd_stderr_matches_regex - -- then: stderr doesn't match regex (?P<regex>.*) - regex: true - function: runcmd_stderr_doesnt_match_regex diff --git a/tutorial.md b/tutorial.md new file mode 100644 index 0000000..b20c84e --- /dev/null +++ b/tutorial.md @@ -0,0 +1,296 @@ +# Obnam tutorial + +With the help of this tutorial, you're going to set up Obnam, make your first +backup, and check that you can restore files from it. + +In Obnam, **a client** is a computer whose data is being backed up, and **a +server** is a computer that holds the backup. A single computer can serve both +roles, but don't put your backup onto the same disk as you're backing up; if +that disk breaks, the backup won't do you any good. Consider using +an USB-attached disk, or better yet, some network-attached storage. + + + +# Setting up a server + +For this, you'll need: + +* Git and Ansible installed on your local machine +* a Debian host with plenty of space to keep your backups (this can be the same, + local machine) + +On your local machine, clone the Obnam repository: + +``` +$ git clone https://gitlab.com/larswirzenius/obnam.git +$ cd obnam/ansible +``` + +The next command depends on where your Obnam server is hosted: + +- if the server is accessible from the Internet, run the following commands, + replacing `obnam.example.com` with the domain name of the host: + + ``` + $ printf '[server]\nobnam.example.com\n' > hosts + $ ansible-playbook -i hosts obnam-server.yml -e domain=obnam.example.com + ``` + + The above gets a free TLS certificate from [Let's Encrypt][]. + +- if it's a private server or just the same machine as the Obnam client, run the + following: + + + ``` + $ printf '[server]\nprivate-vm\n' > hosts + $ ansible-playbook -i hosts obnam-server.yml + ``` + + This uses a pre-created self-signed certificate from `files/server.key` and + `files/server.pem`, and is probably only good for trying out Obnam. You may + want to generate your own certificates instead, e.g. using [OpenSSL] + something like this: + + ``` + $ openssl req -x509 -newkey rsa:4096 -passout pass:hunter2 \ + -keyout key.pem -out cert.pem -days 365 -subj /CN=localhost + ``` + + Put the generated keys into `/etc/obnam` (the location can be configured + with `tls_key` and `tls_cert` keys in `/etc/obnam/server.yaml`, which we + are about to describe). + +Check that the server is installed and running: + +``` +$ sudo systemctl is-active obnam +active +``` + +Ansible created a directory, `/srv/obnam/chunks`, that will contain the +backed-up data. If you want to use a different directory, you have to stop the +service, move the existing directory to a new location, and update Obnam's +configuration: + +``` +$ sudo systemctl stop obnam +$ sudo mv /srv/obnam /the/new/location/ +$ sudoedit /etc/obnam/server.yaml +``` + +In the editor, you'll see something like this: + +``` +address: 0.0.0.0:443 +chunks: /srv/obnam/chunks +tls_key: /etc/obnam/server.key +tls_cert: /etc/obnam/server.pem +``` + +Paths to TLS files might be different if you're using Let's Encrypt. Anyway, you +have to edit `chunks` key to point at the new location. Once you're done, save +the file and start the server again: + +``` +$ sudo systemctl start obnam +$ sudo systemctl is-active obnam +active +``` + +Half the job done, another half to go! Let's set up a client now. + +[Let's Encrypt]: https://letsencrypt.org/ +[OpenSSL]: https://www.openssl.org/ + + + +# Setting up a client + +There is a Debian package built by CI from every commit. It works on Debian 10 +(buster) and later. You can run a script to install it: + +``` +$ curl -s https://gitlab.com/larswirzenius/obnam/-/raw/main/install-debian.sh | sudo bash +``` + +If you'd rather not download a script from the Internet and run it as +root (kudos!), you can do the same steps manually. Add the following +to `/etc/apt/sources.list.d/obnam.list`: + +``` +deb http://ci-prod-controller.vm.liw.fi/debian unstable-ci main +``` + +Then save the following PGP public key as `/etc/apt/trusted.gpg.d/obnam.asc`: + +``` +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBFrLO7kBEADdz6mHstYmKU5Dp6OSjxWtWaqTDOX1sJdmmaIK/9EKVIH0Maxp +5kvVO5G6mULLAjv/kLG0MxasHPrq8I2A/y8AqKAGVL8QelwLjQMIFZ30/VbGQPHS ++T5TZXEnoQtNce1GUhFwJ38ZyjjwHBFV9tSec7rZ2Q3YeM3nNnGPf6DacXGfEOPO +HIN4sXAN2hzNXNjKRzTIvxQseb6nr7afUh/SlZ3yhQOCrIzmYlD7tP9WJe7ofL0p +JY4pDQYw8rT6nC2BE/ioemh84kERCT1vCe+OVFlSRuMlqfEv+ZpKQ+itOmPDQ/lM +jpUm1K2hrW/lWpxT/ZxHKo/w1K36J5WshgMZxfUu5BMCL9LMqMcrXNhNjDMfxDMM +3yBPOvQ4ls6fecOZ/bsFo1p8VzMk/w/eG8vPs5yuNa5XxN95yFMXoOHGb5Xbu8D4 +6yiW+Af70LbiSNpGdmNdneiGB2fY38NxBukPw5u3S5qG8HedSmMr1RvSr5kHoAAe +UbOY+BYaaKsTAT7+1skUW1o3FJSqoRKCHAzTsMWC6zzhR8hRn7jVrrguH1hGbqq5 +TZSCFQZExuTJ7uXrTLG0WoBXIjB5wWNcSeXn8myUWYB51nJNF4tJBouZOz9JwWGl +kiAQkrHnBttLQWdW9FyjbIoTZMtpvVx+m6ObGTGdGL1cNlLAvWprMXGc+QARAQAB +tDJJY2sgQVBUIHJlcG9zaXRvcnkgc2lnbmluZyBrZXkgKDIwMTgpIDxsaXdAbGl3 +LmZpPokCTgQTAQgAOBYhBKL1uyDoXyxUH3O717Wr+TZVS6PGBQJayzu5AhsDBQsJ +CAcCBhUICQoLAgQWAgMBAh4BAheAAAoJELWr+TZVS6PGB5QQANTcikhRUHwt9N4h +dGc/Hp6CbqdshMoWlwpFskttoVDxQG5OAobuZl5XyzGcmja1lT85RGkZFfbca0IZ +LnXOLLSAu51QBkXNaj4OhjK/0uQ+ITrvL6RQSXNgHiUTR/W2XD1GIUq6nBqe2GSN +31S1baYKKVj5QIMsi7Dq8ls3BBXuPCE+xTSaNmGWjes2t9pPidcRvxsksCLY1qgw +P1GFXBeMkBQ29kBP87SUL15SIk7OiQLlEURCy5iRls5rt/YEsdEpRWIb0Tm5Nrjv +2M3VM+iBhfNXTwj0rJ34mlycF1qQmA7YcTEobT7z587GPY0VWzBpQUnEQj7rQWPM +cDYY0b+I6kQ8VKOaL4wVAtE98d7HzFIrIrwhTKufnrWrVDPYsmLZ+LPC1jiF7JBD +SR6Vftb+SdDR9xoE1yRuXbC6IfoW+5/qQNrdQ2mm9BFw5jOonBqchs18HTTf3441 +6SWwP9fY3Vi+IZphPPi0Gf85oMStgnv/Wnw6LacEL32ek39Desero/D8iGLZernK +Q2mC9mua5A/bYGVhsNWyURNFkKdbFa+/wW3NfdKYyZnsSfo+jJ2luNewrhAY7Kod +GWXTer9RxzTGA3EXFGvNr+BBOOxSj0SfWTl0Olo7J5dnxof+jLAUS1VHpceHGHps +GSJSdir7NkZidgwoCPA7BTqsb5LN +=dXB0 +-----END PGP PUBLIC KEY BLOCK----- +``` + +After that, run the following commands to install Obnam: + +``` +$ sudo apt update +$ sudo apt install obnam +``` + +Now verify that everything is installed correctly: + +``` +$ obnam --version +obnam-backup 0.3.1 +``` + +The version might be different, but at least there should **not** be any errors. + + + +# Making a backup + +To create a backup, client needs to know three things: where the backup server +is, where the live data is, and what key to use for encryption. To tell the +client about the first two, create a file `~/.config/obnam/obnam.yaml` with +contents like this: + +```yaml +server_url: https://obnam.example.com:443 +roots: + - /home/joe + - /home/ann + - /etc + - /var/spool +``` + +Adjust the server address to match what you previously configured on the server. +The `roots` key is a list of all the directories that Obnam should back up. Make +sure that the roots are accessible to the user who would be doing the backup — +the user has to be able to read their contents to back them up. + +To generate an encryption key, run `obnam init` and type a passphrase. You don't +need to remember it — it's just an additional random input to aid key +generation; Obnam will not prompt you for it ever again. The generated key will +be saved into `~/.config/obnam/passwords.yaml`, and *that's* the file you should +not lose: you can't make or restore backups without it. Consider copying it +somewhere separate from your main backup. + +With that, you're ready to make your first backup! Run the following command, +and watch Obnam go through all the files in your roots: + +``` +$ obnam backup +elapsed: 7s +files: 3422/0 +current: /home/ann/music/Beethoven/1.flac +``` + +Depending on how much data you have under the roots, this might take a while. +But once Obnam is done, it will print out a report like this: + +``` +status: OK +duration: 85 +file-count: 1223 +generation-id: 3905a0ad-9971-413c-ac81-ca8587c5f8c2 +``` + +That's how you know you've got a backup! Hold off the celebration, though; the +backups are only as good as your ability to use them, so let's check if you can +recover the files you just backed up. + + + +# Restoring a backup + +Let's imagine that your disk crapped out. In that case, you probably want to +just grab the latest backup. In other cases, you might find that a file you +thought useless and deleted long ago is actually important. To restore it, you +need to find the backup that still has it. + +The first order of business is to restore your `passwords.yaml`. If you already +have it on your current machine, great; if not, you'll have to restore it from +some *other* backup before you can use Obnam to restore everything else. It's +impossible to recover any data without knowing the key, since it's all +encrypted. + +Got the `passwords.yaml` in place? Good. Let's get a list of all your backups +with `obnam list`: + +``` +$ obnam list +6d35e3dd-3264-4269-a9d3-74fbd354c90e 2021-01-13 02:32:50.482465724 +0300 +e4387899-d1dd-4e42-bc57-f56e6097d235 2021-01-14 02:36:00.029561204 +0300 +9acde8d9-c167-4ad0-86b6-560c711713e1 2021-01-18 02:45:56.865274252 +0300 +708db71e-d863-47e6-92c3-679041e25c8e 2021-01-20 02:49:50.664349817 +0300 +0f3a63d0-d992-42ff-ab77-7e2457745a40 2021-01-22 03:00:56.902063598 +0300 +028ce888-4a5b-438c-978c-0812646165cf 2021-02-07 16:18:19.008757980 +0300 +481bb25f-5377-4e41-b824-4e60fda8f01c 2021-02-08 19:04:44.072710112 +0300 +5067e10e-2d4d-4ff4-a9a0-568ed008dd2c 2021-02-11 20:26:06.589610566 +0300 +3905a0ad-9971-413c-ac81-ca8587c5f8c2 2021-02-12 22:35:20.431081194 +0300 +``` + +That second-to-last backup, 5067e10e-2d4d-4ff4-a9a0-568ed008dd2c, looks like +it's old enough. Let's see what files it contains: + +``` +$ obnam list-files 5067e10e-2d4d-4ff4-a9a0-568ed008dd2c +``` + +You might need to `grep` the result to check for specific files. Anyway, suppose +this backup it exactly what you need. Let's restore it to a directory called +"yesterday": + +``` +$ obnam restore 5067e10e-2d4d-4ff4-a9a0-568ed008dd2c yesterday +``` + +Obnam will print out a progress bar and some stats. Once the restoration is +done, you can look under `yesterday/` to find the file you needed. Easy! + +Now you're prepared for the worst. (Unless *both* your primary and backup disks +break. Or your backup server is inaccessible. Or you lose all copies of +`passwords.yaml`. Or there is no electrical grid anymore to power your devices. +Or the zombies are trying to break in, distracting you from reading this +tutorial. Look up "disaster recovery planning"—oh right, no electricity.) + + +# Where to go from here + +Obnam is still at the alpha stage, so it's likely that the instructions above +didn't quite work for you. If so, please [open issues][issue-tracker] and help +us improve Obnam! + +If you're interested in more details, and especially in how Obnam works +internally, take a look at [the Subplot file](obnam.html). It not just explains +things, but also contains acceptance criteria and tests for them. Great stuff! + + +[issue-tracker]: https://gitlab.com/larswirzenius/obnam/-/issues |