docs: start draft for next planning meeting

1 files changed, 168 insertions, 0 deletions

diff --git a/blog/2021/04/25/meeting.md b/blog/2021/04/25/meeting.md
new file mode 100644
index 0000000..d87224d
--- /dev/null
+++ b/blog/2021/04/25/meeting.md
@@ -0,0 +1,168 @@
+[[!meta title="Iteration planning: April 25—May 9"]]
+[[!tag meeting]]
+
+[[!toc levels=1]]
+
+# Assessment of the iteration that has ended
+
+[previous iteration]: /blog/2021/04/18/planning
+
+The goal for the [previous iteration][] was:
+
+> The main goal of this iteration is to add at least rudimentary
+> encryption of chunks, before they're uploaded to the server, and
+> decryption and validation after they're downloaded. This should use
+> the encryption keys stored by `obnam init`. This is done only if the
+> client configuration says encryption is turned on, to allow an opt-in
+> approach to encryption for now. Later on, encryption won't be
+> optional.
+>
+> At the same time, work needs to start on using more concurrency in the
+> client, and that means that Lars needs to learn more about Rust,
+> specifically async Rust.
+>
+> Additionally, some smaller issues will be worked on, to tackle the
+> back log of open issues.
+
+That goal was not reached. Lars didn't get [[!issue 28]] (iterators
+instead of large vectors), [[!issue 109]] (release), or [[!issue 110]]
+(chunk encryption) done, partly due to life and work keeping him busy,
+and partly because attempting to do [[!issue 28]] took up almost all
+the time he'd allocated for Obnam. The issues Lars did work on have
+outstanding merge requests, because of the multi-day review period
+process being used. Further, Lars learned a bit about async Rust and
+has a plan for concurrency in the client.
+
+Those issues will be carried to the next iteration.
+
+# Discussion
+
+## Concurrency in the client
+
+Lars experimented with async Rust, using
+[tokio](https://crates.io/crates/tokio), on a branch of the
+[Summain](https://crates.io/crates/summain) program. His conclusion is
+that async will work well for those parts of programs that do I/O or
+other blocking operations that allow switching between async tasks.
+Such tasks are executed in tokio's worker threads. For CPU intensive
+tasks, tokio's blocking tasks, executed in its blocking threads, work
+fine.
+
+Blocking tasks are necessary to use all the available CPU: tokio only
+schedules normal tasks in a co-operative way, limiting the amount of
+CPU cycles used for CPU intensive computation without inhibiting
+progress of other tasks. Blocking tasks run in their own threads,
+allowing more CPU to be used without interfering with normal async
+task.
+
+Lars was able to make a version of Summain where all the metadata
+gathering of files is done in async tasks, but the checksum
+computations are done in blocking tasks. This allowed all the CPU on
+the test host used for checksum computation. The resulting program
+was reasonably clear and obvious, a definite plus.
+
+For Obnam, Lars proposes the following: the Obnam client will execute
+in four phases:
+
+1. Download the latest generation from server. Until this is done,
+   nothing can be decided about whether a file needs to be backed up
+   or not. If there are no generations (initial backup), invent a
+   dummy, empty generation to be used instead.
+   
+2. Scan the file system to files to be backed up, excluding based on
+   client configuration (e.g., `CACHEDIR.TAG`), and checking against
+   the previous generation. Insert metadata of any not-excluded files
+   into a new nascent generation. For each inserted file, mark it as
+   unchanged, changed, or new.
+   
+3. Scan the nascent generation for regular files that are changed or
+   new. For each such file, split it into chunks, compute the checksum
+   for the chunk, and look up the checksum on the server or upload the
+   chunk and insert the chunk id into the nascent generation.
+   
+4. Upload the nascent generation to the server.
+
+Of these, phases 1 and 4 are synchronous, in that phase 1 must finish
+before 2 can start, and phase 3 must finish before 4 can start;
+however, otherwise they can use async Rust. Phases 2 and 3 are fully
+async, and can be interleaved. The checksum computation part of phase
+3 is a blocking task, to allow all the available CPU used for it.
+
+## Iteration length
+
+This was a one-week iteration. In principle it went OK, but Lars
+failed to get much of the issues resolved he'd taken on. A longer
+iteration would allow more flexibility, but can be problematic in
+other ways.
+
+A bigger issue is that if a change needs three days of review, by
+default, a one-week iteration means all the work would need to be done
+very early in the iteration, or else not enough time to get changes
+reviewed.
+
+For now, Lars suggests going back to two-week iterations.
+
+## This iteration
+
+This iteration covers April 25 though May 9. To allow participation
+and feedback of the meeting plan, the meeting won't be closed until
+Tuesday, April 27.
+
+
+# Goals
+
+## Goal for 1.0 (not changed this iteration)
+
+The goal for version 1.0 is for Obnam to be an utterly boring backup
+solution for Linux command line users. It should just work, be
+performant, secure, and well-documented.
+
+It is not a goal for version 1.0 to have been ported to other
+operating systems, but if there are volunteers to do that, and to
+commit to supporting their port, ports will be welcome.
+
+Other user interfaces is likely to happen only after 1.0.
+
+The server component will support multiple clients in a way that
+doesn’t let them see each other’s data. It is not a goal for clients
+to be able to share data, even if the clients trust each other.
+
+## Goal for the next few iterations (not changed for this iteration)
+
+The goal for next few iterations is to have Obnam support encryption
+well. This will involve having a documented threat model, which has
+been reviewed by all stakeholders participating in the project, and
+Obnam defending against all the modeled threats.
+
+## Goal for the iteration that is starting
+
+The main goal of this iteration is to add at least rudimentary
+encryption of chunks, before they’re uploaded to the server, and
+decryption and validation after they’re downloaded. This should use
+the encryption keys stored by `obnam init`. This is done only if the
+client configuration says encryption is turned on, to allow an opt-in
+approach to encryption for now. Later on, encryption won’t be
+optional.
+
+Additionally, work will continue on using iterators instead of
+potentially enormous vectors when querying the SQLite database.
+
+# Commitments for this iteration
+
+New [[!milestone 9]] represents this iteration on GitLab.
+
+This is a two-week iteration. For Lars, that means a time budget of
+8 hours. Lars is committed to resolving the following issues:
+
+- [[!issue 28]] - (2.75h; time boxed: give up if it takes longer)
+- [[!issue 109]] - (0.25h)
+- [[!issue 110]] - (1h)
+- [[!issue 113]] - (4h)
+
+That is a total of 8 hours, rough estimate. Additionally, finishing
+off the merge requests from the April 18—25 iteration.
+
+
+# Meeting participants
+
+* Lars Wirzenius