summaryrefslogtreecommitdiff
path: root/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/new/1550678347.M29990P29481Q1.koom
blob: c2b16e7e762e0b9c2ee5fe02e620300ca6d7f473 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
Return-Path: <ick-discuss-bounces@ick.liw.fi>
X-Original-To: distix@pieni.net
Delivered-To: distix@pieni.net
Received: from yaffle.pepperfish.net (yaffle.pepperfish.net [88.99.213.221])
	by pieni.net (Postfix) with ESMTPS id 0EF9942E76
	for <distix@pieni.net>; Wed, 20 Feb 2019 15:58:13 +0000 (UTC)
Received: from platypus.pepperfish.net (unknown [10.112.101.20])
	by yaffle.pepperfish.net (Postfix) with ESMTP id D0EC04111E;
	Wed, 20 Feb 2019 15:58:12 +0000 (GMT)
Received: from ip6-localhost.nat ([::1] helo=platypus.pepperfish.net)
	by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian))
	id 1gwUGC-0004QQ-Q6; Wed, 20 Feb 2019 15:58:12 +0000
Received: from [10.112.101.21] (helo=mx3.pepperfish.net)
 by platypus.pepperfish.net with esmtps (Exim 4.80 #2 (Debian))
 id 1gwUGB-0004Q4-Br
 for <ick-discuss@ick.liw.fi>; Wed, 20 Feb 2019 15:58:11 +0000
Received: from mail.steve.org.uk ([176.9.183.102] helo=ssh.steve.org.uk)
 by mx3.pepperfish.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <steve@steve.org.uk>) id 1gwUG9-0000Ru-FW
 for ick-discuss@ick.liw.fi; Wed, 20 Feb 2019 15:58:11 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
 d=steve.org.uk; s=20150726; h=References:In-Reply-To:Message-ID:Date:Subject:
 From:To:Sender:Reply-To:Cc:MIME-Version:Content-Type:
 Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:
 Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:
 List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=EOZG+lzSbTnZF4rkMB8QR/12qPYYuySwLoXgRIrMaBU=; b=WHdvz7fjyteIxpAOINHZDJX6/
 LaGsMqyUpPrdhNeaY0Hb6pUs0iW7ZZva8wzAbvtmx4mVwpEoRsl/fxA1BmPrTBoX49bWi2gQslnXh
 iCyAXZJHsLwWwgBtBjfc2u6aqkjabkAADuxIz/zCcGyZ9G574sPlzffscP6cvOusRWPh0=;
Received: from steve by ssh.steve.org.uk with local (Exim 4.89)
 (envelope-from <steve@steve.org.uk>) id 1gwUG2-00006g-Uq
 for ick-discuss@ick.liw.fi; Wed, 20 Feb 2019 15:58:03 +0000
To: ick-discuss@ick.liw.fi
From: Steve Kemp <steve@steve.org.uk>
Date: Wed, 20 Feb 2019 15:54:23 +0000
Message-ID: <1550678063.32630.0@ssh.steve.org.uk>
In-Reply-To: <20190216144009.GA5870@exolobe1.liw.fi>
References: <5e2b278847b76dd0311d5050b3455b12b4dd3077.camel@liw.fi>
 <1549688875.1308.1@ssh.steve.org.uk> <20190216144009.GA5870@exolobe1.liw.fi>
X-added-header: steve.org.uk
X-Pepperfish-Transaction: 0681-bd93-c8ba-26b1
X-Spam-Score: -2.1
X-Spam-Score-int: -20
X-Spam-Bar: --
X-Scanned-By: pepperfish.net, Wed, 20 Feb 2019 15:58:11 +0000
X-Spam-Report: Content analysis details: (-2.1 points)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
 -0.1 DKIM_VALID_EF          Message has a valid DKIM or DK signature from
 envelope-from domain
 -0.1 DKIM_VALID_AU          Message has a valid DKIM or DK signature from
 author's domain
 0.1 DKIM_SIGNED            Message has a DKIM or DK signature, not necessarily
 valid
 -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-ACL-Warn: message may be spam
X-Scan-Signature: 78c34a7242bc3b0cdb4980a2406d4978
Subject: Re: Plan for using Muck for the Ick controller
X-BeenThere: ick-discuss@ick.liw.fi
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: discussions about the ick CI system <ick-discuss-ick.liw.fi>
List-Unsubscribe: <https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi>,
 <mailto:ick-discuss-request@ick.liw.fi?subject=unsubscribe>
List-Archive: <http://listmaster.pepperfish.net/pipermail/ick-discuss-ick.liw.fi>
List-Post: <mailto:ick-discuss@ick.liw.fi>
List-Help: <mailto:ick-discuss-request@ick.liw.fi?subject=help>
List-Subscribe: <https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi>,
 <mailto:ick-discuss-request@ick.liw.fi?subject=subscribe>
Sender: ick-discuss-bounces@ick.liw.fi
Errors-To: ick-discuss-bounces@ick.liw.fi

> (Sorry about taking so long to answer: I was travelling for work.)

  No worries.

> You're right, this scales badly to long build logs, because of the
> large number of HTTP requests required. (The size of the logs is so
> much less of an issue I'm going to ignore that for now.)

> For now, I think I'm OK with that, if it's fast enough for
> not-very-long build logs. From my personal Ick instance, the longest
> log files are about 14000 lines.

  I suspect this depends upon what you're building; for complicated
 reasons at work I'm creating images which build nginx, php, curl,
 and similar things from source.  The build output from such jobs
 is pretty volumous.  At home I tend to build linux-kernels which
 also have a good few thousand lines of output.

> This is not excellent. Is it tolerable? Not very tolerable for
> interactive use. I'm not much worried, for now, about the time it
> takes to create the log file snippets, that's going to be drowned by
> the actual build. The time to get the list of snippets is fast enough,
> I think, even in the current prototype written in Python without
> indexes.

  That's probably fair.

> The full log assembly time is bad. As you say, it's because there's a
> lot of HTTP requests. It might be tolerable for a short while, while
> we improve things, just to get the Ick controller to use Muck instead
> of having local state, but let's discuss how we can improve it.

  That's probably a good approach.  When it comes to builds in my
 current system I have to say that I tend to scroll straight to
 the bottom to look for an "All OK", or "ERROR/FAIL" message.

  The stuff in the middle I don't care about so much very often.
 I'd not suggest that having a ring-buffer was a good idea, but if
 it were possible to keep track of the last update that might be
 a compromise worth making.

> I don't like the idea of keeping the log file locally, either on the
> worker-manager or the controller, as it means that one can't do the
> equivalent of "tail -f" while the build is running. Updating the
> controller's view of the log while the build is running, without
> lagging much from actual output, is a thing I'd really like to keep.
  
  Sure.

  I suspect when everything is done the build-log belongs with the
 output artifacts, in the object-store.  That might be personal
 bias at play of course.

> * new step: if there are more than M short snippet objects for a
>   build, the controller would construct a combined snippet object, and
>   delete the short snippet objects
> * new step: at the end of the build, the controller would comine all
>   short snippet objects

  That seems like a good idea.

Steve
-- 
https://www.steve.org.uk/

_______________________________________________
ick-discuss mailing list
ick-discuss@ick.liw.fi
https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi