To: ick-discuss@ick.liw.fi
From: Steve Kemp <steve@steve.org.uk>
Date: Wed, 20 Feb 2019 15:54:23 +0000
Message-ID: <1550678063.32630.0@ssh.steve.org.uk>
In-Reply-To: <20190216144009.GA5870@exolobe1.liw.fi>
References: <5e2b278847b76dd0311d5050b3455b12b4dd3077.camel@liw.fi>
 <1549688875.1308.1@ssh.steve.org.uk> <20190216144009.GA5870@exolobe1.liw.fi>
Subject: Re: Plan for using Muck for the Ick controller
Precedence: list
Sender: ick-discuss-bounces@ick.liw.fi
Errors-To: ick-discuss-bounces@ick.liw.fi

> (Sorry about taking so long to answer: I was travelling for work.)

  No worries.

> You're right, this scales badly to long build logs, because of the
> large number of HTTP requests required. (The size of the logs is so
> much less of an issue I'm going to ignore that for now.)

> For now, I think I'm OK with that, if it's fast enough for
> not-very-long build logs. From my personal Ick instance, the longest
> log files are about 14000 lines.

  I suspect this depends upon what you're building; for complicated
 reasons at work I'm creating images which build nginx, php, curl,
 and similar things from source.  The build output from such jobs
 is pretty volumous.  At home I tend to build linux-kernels which
 also have a good few thousand lines of output.

> This is not excellent. Is it tolerable? Not very tolerable for
> interactive use. I'm not much worried, for now, about the time it
> takes to create the log file snippets, that's going to be drowned by
> the actual build. The time to get the list of snippets is fast enough,
> I think, even in the current prototype written in Python without
> indexes.

  That's probably fair.

> The full log assembly time is bad. As you say, it's because there's a
> lot of HTTP requests. It might be tolerable for a short while, while
> we improve things, just to get the Ick controller to use Muck instead
> of having local state, but let's discuss how we can improve it.

  That's probably a good approach.  When it comes to builds in my
 current system I have to say that I tend to scroll straight to
 the bottom to look for an "All OK", or "ERROR/FAIL" message.

  The stuff in the middle I don't care about so much very often.
 I'd not suggest that having a ring-buffer was a good idea, but if
 it were possible to keep track of the last update that might be
 a compromise worth making.

> I don't like the idea of keeping the log file locally, either on the
> worker-manager or the controller, as it means that one can't do the
> equivalent of "tail -f" while the build is running. Updating the
> controller's view of the log while the build is running, without
> lagging much from actual output, is a thing I'd really like to keep.
  
  Sure.

  I suspect when everything is done the build-log belongs with the
 output artifacts, in the object-store.  That might be personal
 bias at play of course.

> * new step: if there are more than M short snippet objects for a
>   build, the controller would construct a combined snippet object, and
>   delete the short snippet objects
> * new step: at the end of the build, the controller would comine all
>   short snippet objects

  That seems like a good idea.

Steve
-- 
https://www.steve.org.uk/

_______________________________________________
ick-discuss mailing list
ick-discuss@ick.liw.fi
https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi