summaryrefslogtreecommitdiff
path: root/devsetup.mdwn
blob: 52fa9096d69522df561a22dd25cb8b644cede2b0 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
# Development setup phase

The next three weeks are overtly busy, but not actually
productive. Robin spends a lot of time with the SmartHome
developers to gain an understanding of what their gadgets
actually do, so we can help them re-design things for our
platform. Andy and Bert read everything about the hardware
they can find, and set up a cross-building environment and
start preparing a Linux kernel for the hardware. I iterate
over contracts with Sam, and eventually get them signed by
both parties. We agree that we'll work mainly from our own
office. However, Nina is still waiting for hardware, which
means that Andy and Bert work blind.

We like our own office. It's actually an apartment near the
city centre, in a building from 1905, which we rent. There's
private rooms for everyone, except Andy and Bert prefer to
share a room. What was meant to be a living room is the
meeting room. The kitchen remains as a kitchen, and there's
even a small room we use as a server room. We've spent a
fair bit of effort on making the apartment into a good
office for us. There's privacy for everyone, extremely
comfortable sofas, good networking and electricity
everywhere, good heating, a real bathroom (both shower and
tub), basically all the comforts of a modern office, except
it feels more like a home. We've put sound isolation
everywhere, both external and internal walls, plus ceiling,
so that it's quiet. It was pretty quiet even before, since
it's an old building with quite thick stone walls. We can
have an actual shouting match in the meeting room without
disturbing the neighbours or those working in their own
rooms.

I sit at my desk, and call Sam to ask what's up. "Hey Sam,
it's Anna. We're still waiting for the ten SmartHome hubs,
and we must have them to really start development. It's been
several weeks, is there a problem?"

"Hi, Anna, let me check and call back to you."

Fifteen minutes pass. My phone makes a noise, it's Sam
calling back. I answer. "We seem to have a problem. Some of
our guys think one or two devices should be enough for you.
We don't have ten in the office."

"Sorry, but it's not your decision how many we need. If we
have to share devices for development, that will create
congestion, and block work that would otherwise progress
smoothly."

"I see. Very well. I'll what see we can find."

"Wait. I'm curious: these devices, can we use your normal
mass produced SmartHome hubs for development? The ones you
have manylots available for sale in your web store, and
on Amazon?"

"Yeah, those would work. Unfortunately, due to tax and
accounting reasons, we can't just go get them from the
warehouse. To start with, we've outsourced the sales and
warehousing."

"I understand. Let's do this: we'll order what we need,
overnight delivery, and bill you for them as part of our
expenses? That is a bit of a roundabout way to get the
stuff, but it unblocks us and since this is a life-and-death
project, we have to avoid being blocked."

Sam is silent. "Uh, yeah, that would work. As you say, it's
quite roundabout."

"Good. Can you send one of your techies to our office
tomorrow to help Nina get them set up for development, get a
serial console attached, and flash a new kernel that we've
built, and so on? We have copies of the relevant
documentation, but it always helps to have someone who's
done the thing before."

------

The following day, Nick from SmartHome arrives early in the
work day. Nina parks him in the kitchen, and shows him how
to use the coffee machine, while they wait for the devices
to arrive by courier.

The courier arrives by noon. Nina signs for the delivery,
and carries the boxes to her lab, a large room with plenty
of shelves and desks so there's room for spreading hardware
around. It is her sanctum, the rest of aren't supposed to
enter without asking for permission.

The lab is filled with furniture and bits and pieces of
electronics. The walls are all covered by bookcases, Lundia
shelves, cupboards, chests of drawers, and other storage.
Part of one wall is covered by a board with hooks for tools.
There are a lot of tools hung on the board: every
conceivable sort of screwdriver, hammer, pliers, knives,
even an axe and, most curiously of all, a half-meter long
rubber baton, and a 20-pound sledgehammer. Near the door are
several large garbage bins, labelled for recycling: WEEE,
metal, toxic, and misc.

In addition to a very large, sturdy table in the middle of
the room, similar to an island table in a kitchen, there is
a lab bench along one wall, with an oscilloscope, soldering
irons, a large magnifier, and other tools for electronics.
There are three bar stools around the island table, and at
the lab bench.

The room is well lit, and both the lab bench and the island
table have large lamps hanging over them from the ceiling.

Nick does not know he's entering a special place. He follows
Nina into the room, and promptly collides with a table,
causing a spar desktop machine fall to the floor. There is a
terrible noise, and parts fly across the floor to all
corners of the room.

"You! Sit!" Nina points her finger at Nick, and uses her
loud voice. Nick is startles, but luckily doesn't cause any
more havoc. He looks around for a chair.

"Sit! Now! NOW! On the floor!" Nick drops to the floor, with
his buttocks first, as if he were a well-trained dog. He
looks startled and confused at his reaction.

"Look, Nick. This is my lab. I need your help, and that's
why you're allowed in here, but you're not allowed to move,
or touch anything, and if you break anything else, I'll
throw you out. Do you understand?"

"I'm sorry, I'm just clumsy. I hope I didn't break anything
important?" Nick has dropped his shoulders, and speaks in a
soft, calm voice. He resembles nothing as much as a dog
caught doing something bad. If he was wagging a tail and
pleading for mercy using his big, glistening brown puppy
eyes, nobody would be surprised.

"Nah, it was just a spare PC I was going to re-purpose as an
extra backup server. Don't worry about it."

Nina takes one SmartHome cardboard box, puts it in the
middle of the big table, and stacks the rest on an empty
shelf on the wall. Carefully, with slow hand movements, the
unpacks the gadget from the box, then takes her phone and
takes a photograph of all the contents.

"All right, I have a hub. I guess I need to open it to gain
access to the serial port, right?"

"Yes, Miss Nina."

Nina frowns at Nick. "Why are you calling me Miss Nina? Are
you being sarcastic?"

"No, Miss Nina. I don't want to anger you any further."

"Good boy." Nick smiles.

"To open the case, I need to open these four screws" - Nina
points at them - "And then use a bit of force to get the
cover apart, right?"

"Yes, Miss Nina. Would you like me to do it for you?"

"No, thanks, I need to know how to do this." Nina goes to
the wall, where a ton of tools are hanging, and chooses an
electric screwdriver, and returns to the table. Deftly, she
opens the four screws. She then goes back to the walls,
returns the screwdriver to its place, and takes a small
crowbar. She inserts the end of the crowbar between the
halves of the case, and wrenches it open with a sudden, fast
movement resulting in a crunch.

Nick is surprised, again, but stays put. "Oh."

Nina smiles a small, tight smile, and starts picking out the
various bits of electronics from inside the remains of the
case she's ruined. "This is the mainboard, I assume the
header with the pins for the console serial port are here.
Is that also how we flash a new kernel?"

"Yes, that's right."

Nina turns on the lamp hanging from the ceiling over the
desk. The desk, and the room, is bathed in a bright, cold
white light. She bows down and looks at the now-exposed
mainboard of the opened hub. She raises her glasses and puts
them on top of her head, picks up the mainboard, and squints
at it quite closely.

"I should have suitable connector, somewhere, let me see."
She goes to rummage in a cupboard. She pulls out a cable
that's clearly handmade. "This should do." She plugs the
cable to the mainboard, pulls out a laptop from under a heap
of stuff on another table, and plugs the other end of the
cable to the laptop. She connects power to the mainboard,
and soon the laptop screen starts showing a stream of text.

"Hmm. Okay, that boots with the factory kernel. Now, let's
see if I can flash the kernel that Andy and Bert have built.
It's the same version of the kernel, and build from the same
source code version you use, it should work."

Several minutes pass, as Nina uploads the new kernel to the
device over a 115200 bit/s serial connection, and then
writes it to the internal flash storage in the device. Nick
gets bored and offer to fetch them both coffee. "No liquids
in the lab." is Nina's curt answer.

"May I go have coffee in the kitchen?"

"Sure, go ahead."

Nick gets up from the floor, and goes to the kitchen. Robin
is there, sipping a cup of tea. "If you'd like tea, there's
plenty left in the pot, milk's in the fridge."

"Thank you. I'll sit here. Nina says I can't bring liquids
into the lab."

"The lab? Nina let you into the lab?"

"Er, yeah? Why do you ask? She went in and I followed. And
then I was clumsy and a computer fell on the floor and
broke."

"Oh dear. You should probably be told that the lab is Nina's
special place. She's spent months of work time setting it
up, and it's where she spends almost all of her time. The rest
of us do not enter without an explicit invitation each time.
She doesn't usually tolerate other people there, lest they
make a mess there. If she didn't shout at you, she must like
you."

"Well, she did shout at me to sit on the floor after I was
clumsy."

"That's understandable. You did break a computer. Not to
worry, she'll be able to fix it, and if it was something
expensive, we'll bill your company for it. Hah."

"Oh. Okay, I guess."

Nick sits for a while, finishing his cup of tea. "Thank you
for the tea. I will go back to help Nina now."

Robin nods. Nick gets up, washes his cup under the tap, and
puts it in the dishwasher. Just as he's about to open the
kitchen door, Robin speaks up.

"Oh, boy, just to be clear, when I said Nina must like you,
I didn't mean romantically. I don't want there to be any
misunderstanding."

Nick gets to the lab door, which is standing open. He
politely knocks on the door frame. "Miss Nina, may I enter?"

Nina looks up from the laptop, and nods. "Pull up that chair
and get seated next to me so you can see the laptop.
Actually, let me move the monitor here." Nina gets up, goes
to the lab desk, and unplugs the 24" monitor sitting on one
end. She carries the monitor to the island desk, and plus in
cable end a display cable. "There, that'll make it easier
for both of us to see what's happening."

Nick picks up a bar stool on the other side of the island
table, and carries it near Nina's stool, and sits down.

"So, I tried to flash another kernel onto the board, but it
seems to not boot correctly. Let me show you. See if you can
spot anything strange."

Nina presses a button the power supply for the board, and
the monitor has quickly streaming text again. Soon, the
stream stops for about thirty seconds, and then a final
screenful splurges.

Nina points at the final splurge, and reads aloud. "Oops.
Kernel panic. Unable to handle kernel NULL pointer
dereference. Does that location look familiar to you?"

"I'm not sure. Which kernel image is this?"

"It's an image that Andy and Bert built from the SmartHome
source code dump, using the included instructions. They say
it boots up fine on another board, and in Qemu, but
obviously we haven't been able to test it on real hardware
until now."

"If it boots on other hardware, my guess would be something
specific to our hardware. Perhaps a device driver?"

"Perhaps. I better go get Andy to look at this. He'll be
thrilled."

------------------------------------------------------------

The next morning I have tea with Nina and Andy in the
kitchen. We don't have regular stand up-meetings in person,
but I try to talk to everyone at least a few times every
week. What we do have is a kind of written stand ups, where
everyone writes up what they've done each day in the
internal blog, and Nina and Andy had written about the
kernel booting problems. I'd pinged them on IRC so I could
discuss this more efficiently in person. I made a pot of
tea, Lapsang Souchong, since I like the smokey flavor.

"I understand the SmartHome devices arrived yesterday. Is
everything working now?"

Nina shakes her head, while holding her tea mug between her
hands, as if to warm them . "No, not at all. We haven't got
a working kernel build yet."

"Oh? Andy?"

Andy looks like he'd be vexed, if he wasn't his usual
perpetually calm self. "Yeah. We've set up a cross-building
environment, but the kernel image we build doesn't boot
correctly. We spent some time debugging that yesterday with
Nick from SmartHome, but he wasn't all that much help. I
don't think he does this kind of thing usually."

"I see. Do we know what the problem is?"

"Not yet. It looks like the normal embedded Linux vendor
inability to provide complete and corresponding source code
for their kernel. Getting their kernel to build, install,
and run shouldn't be this hard."

"I agree. What can we do to solve it this time?"

"I'll get one of the SmartHome boards from Nina, and let her
keep the rest. She can continue to build a mini-rack, but
Bert and I can use the one to debug. That way we're not
waiting on Nina, and she's not waiting on us. If Bert and I
are still stuck in the afternoon, I think you should call
SmartHome and arrange for a couple of their best kernel
engineers to come help us."

I nod. We've learnt, as a team, that it's important to ask
for help if you're stuck, while it's also important to not
ask for help until you've given the problem a good try
yourself. "I'll call Sam after lunch, unless you tell me
it's all working."

"Nina, does that sound OK to you? And are you doing OK with
the mini-rack build?"

Nina takes a gulp of tea, and nods. "Yeah, it's going well.
I've unpacked all the devices, and extracted the boards from
the cases. That's great fun. The cases are now waste, but we
don't need them anyway. I'll need to go pick up some
hardware bits for the rack so that we can install all the
boards, but that's no problem."

We all have company credit cards, but especially Nina. She
often needs to go buy computer parts, and given that we do
not usually use off-the-shelf computers, they're often
esoteric parts. Luckily, she knows where to find anything we
need.

Nina departs, taking the keys for the company van from the
key safe. Andy withdraws to the room he shares with Bert, so
they can together unlock the mysteries of the kernel that
wouldn't boot. I'm not worried about that. We've had this
situation before, where an embedded Linux vendor doesn't
share the actual kernel source code they use, or are missing
some build step from their documentation, or do something
else stupid. We know it's possible to get Linux booting, and
we have access to the people who made it happen. I'm not
worried, but I am irritated that this particular problem
keeps coming up over and over again.

I take me time to finish my cup of tea. When I'm done, I put
the cup in the dishwasher. Like all of us, I avoid liquids at
my desk, even if Nina's lab is the only place where liquids
are explicitly forbidden.

------------------------------------------------------------

In the afternoon Andy comes to my room, and tells me they're
still stuck with the kernel. I call Sam.

"Hi, Sam, it's Anna. We're having some trouble building and
booting the Linux kernel. Could we have one or two of your
kernel engineer visit out office?"

"Hi, Anna. Of course. Is it urgent? Do you need them today,
or is tomorrow morning OK?"

"Tomorrow morning at nine would be perfectly fine." I look
at Andy with my eyebrows lifted, for confirmation, and he
nods. "Can you ask them to use the door phone, and call
Andy if there's any problem getting in?"

"Sure thing."

------------------------------------------------------------

Nina returns from a shopping trip, with large cardboard
boxes and an army surplus 100-liter duffel bag filled with
things. She carries them into her lab, and closes the door.
She's rigged up an "ON AIR" sign outside the lab door, which
she got from a radio station. We've never asked her how.

------------------------------------------------------------

The following morning two SmartHome kernel engineers push
the button on the door phone at exactly nine o'clock. Andy
lets them in, and they join Bert in his and Andy's room. I
arrive soon after, and see from IRC that Andy has requested
them to be left alone, so I don't go and introduce myself
and tell them they're welcome.

Three hours later, they exit the Andy and Bert room, and
shake hands and say goodbyes. They're gone before I get out of
my room. "Good morning, Andy and Bert. I guess those were
the SmartHome kernel engineers leaving? How are things?"

"Very good. We have a booting kernel now, and we know what
changes SmartHome has made to the stock Linux version. It
should be plain sailing from here."

Bert stands silent, but restless. It's a clear sign that he
disagrees with something, but isn't sure he should say
anything. "Bert, what's up?"

"They're horrible. Their kernel changes are horrible!
Utterly despicable! And we're going to have to use their
code, since we can't get the specs to write the necessary
drivers ourselves."

"How much of a problem is that going to be?"

"Who knows! It's crap code, it may blow up in our faces at
any time. They have busy-waits and arbitrary sleeps all over
the place. It's no wonder they don't upstream the code.
Upstream kernel maintainers would laugh in their faces. Linus
would be sarcastic at them."

To upstream changes means to send your changes to the person
or project who maintains the code in public, and get them to
include the changes in their code. It can be a lot of work,
and it opens you up for criticism, but the benefit is much
less maintenance work for you in the long run. Up-streaming
is considered to be the mark of a full participant of open
source, and something a lot of companies fail to do.

"I see. Andy, do you agree?"

"The code is not great, but it's not the worst embedded
kernel code I've seen, in or out of Linux. I think we'll
manage. It might be good to be prepared, though, and try to
get documentation for the SoC they use. Just so that we can
debug and improve things if things do blow up in our faces."

An SoC is a system on a chip, meaning it's basically a
single computer chip that contains all the usual parts of a
computer, except interfaces for communicating with the
outside world. Most embedded systems are built around an
SoC, and there's a number of big companies developing their
own SoC chips, and the SoCs are all different, and for
reasons nobody else understands, the SoC vendors are
reluctant to publish all the documentation to program them.
Lack of documentation makes it difficult to write reliable
software for them, and that in turn makes is less attractive
to choose their SoC for products. One may liken this to
wearing shoes made out of dynamite, but the vendors seem
supremely confident in being able to grow new legs on
demand.

"Yes, I agree, that's a good idea. I assume you asked those
kernel engineers for them?"

"Yeah, but they have them under NDA, and can't share them
with us."

"Typical hardware vendor BS, in other words. It's probably
no use asking Sam, either. Can you email the particular,
make and model number of the SoC, and I'll send out some
feelers? Oh, and please email Nina where she can find the
working kernel, and work with her to setup a continuous
integration job to build it?"

"Of course."

I go to my room to compose a thank-you email to Sam, for the
help his kernel engineers gave us. I don't mention the code
quality issue, or the lack of SoC documentation. There's no
point in ruffling feathers at this point. I know people
around the industry, and I may be able to get someone to
leak me a copy of the SoC documentation. That's a little
unethical, of course. I will first, however, contact the SoC
vendor directly and see if they'll give me a copy of the
documentation for a promise that we'll take care of
up-streaming any necessary kernel changes. We're likely to
have some time after the SmartHome project, and the
up-streaming would keep Andy and Bert productively busy
during the lull.

------------------------------------------------------------

We do not see much of Nina for the next two days. She's
there, when the rest of us arrive in the morning, and still
there, when everyone else has left. We only see her when she
comes to the kitchen to drink or eat, and though she seems
happy, we do not engage her in conversation. If she needs
help, she'll tell us, but otherwise she's clearly deep in
her work, and gets quite terse if she has to talk about
anything.

All of us are allowed to be as eccentric as we want, as long
as we do not harm the others.

When Nina finally emerges from the lab, it's early
afternoon. She opens the lab door, and turns off the "ON
AIR" sign. She looks satisfied, but tired. Via IRC, she
announces that the mini-rack with SmartHome devices is
ready, and that she'd be happy to give a demonstration and
tutorial after she's had lunch. She goes out, and returns
after fifteen minutes with take-out Indian food, and takes
the food to the kitchen. She takes a carton of milk from a
cupboard, and sits down at the kitchen table to feat.

Nina doesn't usually care much what food she eats, but
when she finishes a build or other project, she celebrates
with her favourites. It's always the same Vindaloo, from the
same takeout restaurant, and with a liter of cold milk to
calm her taste buds. Some years ago, the restaurant she had
been using went out of business, and Nina spent a week to
find a new favoured restaurant, eating Vindaloo three times a
day, until she found one she liked. It was a very aromatic
week for the rest of us.

The whole team is gathered in the corridor for the
demonstration. Nina comes from the kitchen and ushers us
into the lab itself, and tells us to take position on one
side of the island table. She goes to the other side, and
starts. A luggage trunk is on her side of the table, made
out of hard, black plastic. The trunk has handles on all
sides for carrying, and opens along a vertical piano hinge
along one side. Inside is 10U of server rack.

Nina has a pointing stick in her hand.

"Welcome. This is is the SmartHome development trunk. It is
similar to what we've used in previous projects." The stick
strikes the side of the trunk. "It has nine boards from
SmartHome hubs, plus a controller, which gets the serial
console from each hub, plus Ethernet and power to each hub,
which are both programmatically controlled." The stick
touches the various parts of the rack.

"The controller also has a wifi access point to which the
hubs connect. All of us have ssh access to the controller,
and can manage each of the hubs. We'll co-ordinate manually
over IRC, as before, to decide who uses what at any time. To
simplify things, I've named the hubs nina-01, andy-01, and
so on. We each have a dedicated hub, for now, plus CI has
three of its own. That leaves only one spare, but we can add
more if needed."

Nina pauses to take a breath.

"Our normal development rack now has a continuous integration
system set up, which uses the three hubs to run tests on.
The tests are currently very simple: install a newly build
kernel, and reboot, and verify that root can log in via the
serial port within 60 seconds. We'll expand on that as we
start development. In addition, the rack has SmartHome
lighting gadgets, with a light bulb each. Each hub has its
own light bulb. The light bulbs are inside the trunk, and
there's a webcam so you can watch them remotely. Questions?"

The four of us are standing quite for a moment, until Andy
speaks. "Nina, you'll have to excuse our silence. We're used
to you being excellent, but you've outdone yourself. This is
a great setup, and it'll help the rest of us work faster."

I have a question. "What shall we call this trunk?" Nina
smiles a wicked smile. "How about the torture chamber? It's
where we put the SmartHome stuff to the test."

There is a murmur of approval, and much nodding. Nina looks
satisfied. She goes to her laptop, opens up Inkscape, and
quickly makes a drawing with the words "SmartHome Torture
Chamber" laid out nicely and using a Germanic font. She
punches a key combination, and the printer in corner of the
lab spits out a few sheets. Nina gets the sheets, which turn
out to be A4 size labels, with the Inkscape design now
printed on them. She peels off the labels and attaches them
to four sides and the top of the trunk.

"OK then, there's documentation in the internal wiki, and if
your question's not answered by that, ask on IRC, though I'll
be at home sleeping for the next 24 hours or so. Now, who'd
like to carry this to the server cabinet?"

We have a few servers we run in the office. Cloud computing
is all very well, but it requires good Internet connectivity
to use it for development, and when we need to work from
customer premises, that doesn't always exist. We keep the
servers in a small room that may have been a child's or
servant's bedroom in prehistoric times. Andy and Bert,
our two strongest members with the least back problems,
carry the SmartHome trunk there. They place it next to our
development infrastructure trunk, and connect power and
Ethernet.

The development infrastructure trunk is outwardly similar to
the SmartHome one that Nina has built, but the contents are
quite different. The rack contains several 1U servers, on
which we can run quite computationally intensive tasks, such
as compilation, automated tests, continuous integration.
There's also plenty of storage. We have a git server in
there, plus space for backups of each of our work laptops.
This is important so that if we're stuck on a client site
with bad connectivity, we can have everything we need
locally in our trunk. There's even a local copy of
Wikipedia, Stack Overflow, and the Debian package archive.

As much as possible, anything we may need while at the
customer site, we try to have a local copy of. The trunk
also contains a 4G modem, plus a router, so that we can, if
necessary, use that for Internet access.

When we do go to work on-site at a customer, we bring the
development infrastructure trunk with us, take over a
meeting room, and set it up there. It needs power, plus
optionally an Ethernet connection, and we're ready to start
work.

The trunks were originally Nina's idea. Before that, we
tried to have much of the same functionality and data on
laptops, plus USB drives, but it was always quite
cumbersome. The trunks work much better, especially after
the first couple of iterations of the design.

--------------------------------------------

After the demo, when Nina has gotten the new trunk installed
and connected it up in the server room, I quietly ask her to
step into my office.

"Good work with the torture chamber, Nina. I guess you
worked around the clock for two days, is that right?"

"Yeah, Anna, that's right. I went a little overboard. But
don't worry, I'm going to go home and rest and recover
next."

"Good, that's what I want to hear. While we do not track
work hours here, we do not ever want to see someone burn
out. I and Robin have both done that at least a couple of
times each, and we won't let it happen to anyone in the
Team."

"Understood."

"So I don't want to see you working at all before Monday,
OK?"

"Yeah. I'm going home, I'll have some food, and sleep and
binge on the new season of Agent Carter. See you on Monday!"

Nina gets up and walks to the door. "Wait! What did you
think of the young man from SmartHome, Nick?" Nina turns
around. "He's OK, I guess. A bit clumsy, maybe a bit
inexperienced, but at least he seems to not have a big ego
that ruins everything. I can work with him again."

I'm surprised. That's the most positive appraisal I've ever
heard Nina give of someone outside the Team. "Good to hear.
Bye now."

-------------------------------------------------------

Next day in the afternoon I walk to Andy's and Bert's room.

"Hi guys, how is it going?" 

Andy looks up and takes his hands off the keyboard. "All
good now. We've gotten the torture chamber to build and
install a kernel from our own git repository to a test
board, and run some tests against that board. All going
well, now we just need to finish adding the rest of the
board support, missing drivers, etc."

"So it's plain sailing now?" 

"Yep, until we hit the first unanticipated and unforeseeable
snag. You know, the submerged trap that universe always
likes to throw at us."

"Yeah, those. Can't help them, just have to allow for some
of them in the schedule."

"If we don't hit on too many snags, I'm sure we'll have a
sufficiently working kernel end of next week, and after that
we'll start porting our platform to the SmartHome board."

"Excellent. Tell me at once if you hit a snag, OK?"

"Will do."

Andy turns back to his laptop, and lifts his hands on the
keyboard, even though he doesn't start to type yet. I know
what he's doing. I interrupted his concentration when I came
in and interrupted the two, and now they need to get back
all the details of what they were working into their
short-term memory. "Enter the flow" or "go into hack mode"
as we call it. Technically, I should've asked, on IRC, Andy
and Bert to come visit my office, when they're next taking a
break from concentration anyway. I did wrong.

I go out, closing the door quietly behind me. I go to the
kitchen and put in a five euro note into the jar marked
"unwarranted hack mode interruptions". It's a large glass
jar, almost half-full with five euro notes. When it's full,
we'll take out the money and have an office party.

-------------------------------------------------------

When I get back to my own desk, and check my email inbox,
there's a mail from Sam. He's used the subject line "Is RPM
a possibility?" and he wants to know if we could support RPM
packaging for their code, instead of Debian packaging. He
says their guys are familiar and comfortable with RPM and
say it is more secure, because packages can be digitally
signed, unlike packages in the Debian format.

This is a snag. This is the client having ideas of things
beyond their understanding. It usually happens in every
project. While they do not mean ill, the end result is still
that if we accommodate them, it's not just shooting our
collective foot with a shotgun, it's surgically amputating
both feet, while running, and putting in prosthesis made
out of dynamite, with the fuses lit.

The problem isn't that the client has ideas and opinions and
requests and suggestions. The problem is that the client
doesn't know what they're talking about, and doesn't even
suspect that they don't know.

This particular type of snag needs to be stopped early and
thoroughly. It will need to be handled with careful
diplomacy so that they walk away from the confrontation
feeling good about themselves. It'd be easy to just show how
utterly wrong they are and drag their self esteem through a
pile of dung and dance a fandango on their ego with high
heels until they weep, but that would just make it harder to
finish the project together. Every client project needs the
client involved and eager to help if it is to succeed. Also,
they need to think they have all the control, even if they
can't be allowed to have any control. A project actually
controlled by the client is doomed. They should have a
little influence, but their control should be limited to
fairly loose setting of scope and constraint.

I'm going to have to deal with this, but it's getting late,
and I'm tired. Diplomacy requires a lot of energy and time.
I send a quick reply to Sam, so he doesn't worry if my
response takes a while, but I don't say anything except that
I'll respond properly tomorrow. I tell everyone I'm leaving
for the day on the company IRC channel, shut down my laptop,
tidy up my desk, and go home.

I too want to see the new season of Agent Carter. Maybe with
some take-out Indian food, curled up under a blanket on my
couch, with my cat purring in my lap, or on my shoulder.