Another day, another six-hour moving extravaganza. And then an unplanned nap. There's nothing wrong with showing up at the office at 8:30pm, though it's a little less cool when your boss has taken to asking around on IRC to try and find you. I left voice mail! I swear! (My previous boss — who, I am flattered to discover, actually reads this — is perhaps not surprised by these events.)
I upgraded to Psyche today, and it's OK so far. The install is very pretty, though some of the post-installation configuration tools are a bit rough. OK, quite a bit rough. And their gnome-terminal build occasionally just goes blank until I resize it. I think I'll grab the Ximian GNOME 2 snapshots when they start working with 8.0. And man, do I ever miss having MP3 support in xmms. Ah well, that's the price I pay for wanting pretty fonts everywhere. (Except for in Mozilla, because of a huge, embarrassing mess. Don't get me started; blizzard will save us soon enough.)
I found my keys, though. And our TV will arrive on Wednesday, which I guess is sort of exciting. I think that's a bigger deal for Tyla than for me, though I do certainly love shiny new toys with power cables.
I should do a fair bit of work tonight, but I'm afraid that
- I'll be too tired to work out when I get home, and it is a Monday, and;
- I'll totally screw my fragile sleep cycle, making the depressive effects of the onset of fall that much more impressive.
Now, of course, I can't reach our CVS server. I think it's time to go home, set up the new printer, and spend some time reading about the synchronization and recovery techniques that Peter and Phil used in InterMezzo. Tomorrow, though, the code had better look out.
And I'm so behind on replying to email. Sigh.
Another morning of climbing, another serious reinforcement of my belief that I need to start doing chinups or something. Were I a little bit shorter, I could use the door frames here. Not sure what I'll do, but I need to do something if I'm going to beat that silly 60-foot 5.6. (Harder, no doubt due to the length of the climb, than the 5.8 I flew up after failing on my 5.6 nemesis. Go with the figuring.)
After climbing and lunching — it's OK to drink a beer with your first meal of the day, if that meal starts at 1pm and has followed at least 2 hours of athletic activity, by the by — I napped like I was trying to qualify for the Narcolepsy Nationals. Tyla went off to inspect the quality of the offerings at the local futon store, and came back typically well-informed.
When I woke up after my nap-of-over-achievement, Tyla was cleaning the office, and I was reminded that tomorrow morning we're helping Emily move into her new, temporary digs. And there we find me now, pondering dinner and a bath.
I'm also tempted to play more with XML-RPC tonight, as a nice Sunday-evening hobby-hack, but Tyla's listening to music, so I can't reboot the machine right now. Eat a frog, bounce a cheetah.
Of all the movies in the house, Mehmet wanted to watch our wedding videos (the videos of Tyla's and my wedding, to clarify). They still make me a bit weepy, but nothing outrageous. I wish I'd only been a bit weepy at the real event, because then I could have actually spoken to our assembled guests, instead of just blubbering like someone had run over my puppy — I mean, it was the happiest day of my frigging life!
In related news, Tyla and I were both a lot slimmer in high school. It is definitely time for some biking or running or home liposuction with box cutters and a dustbuster.
Dinner was, if I do say so myself, a rousing success. Mark Bittman's cauliflower/eggplant/potato curry, my bourbon-and-peppercorn grilled salmon, "Tyla's" apple/pear/walnut salad and some fresh fruit to round it all out. And, of course, some lovely company, in the form of Mehmet and Ken and Joan. If only we had some house-elves to clean up the kitchen for us, it would have been a perfect night.
We didn't get the XBox. All those surprised, please raise your hand and pay closer attention. Ah well, maybe next time.
I did play a little bit today with XML-RPC and the Blogger API, in hope that I could use mozblog without losing the geeky, obsessive control over my diary that I currently enjoy. Didn't make huge progress, but learned lots.
If we're going to get tropical storm rains to drench us and our belongings, I think it would only be fair for us to grow a tropical beach somewhere on the waterfront. Really, is that so unreasonable?
More can't-happen chasing today, still uncaught. Andreas tells me that there's a busy-wait in the recovery code, so I'll try and track that down today as well.
Meanwhile, back at the ranch, Tyla has been dutifully searching for a good deal on our new television. As I type this, she's using her feminine wiles to talk the Future Shop guy down a few hundred dollars. I just know it. (I have very good Tyla-feminine-wiles detectors.)
Yay! I finally found the rest of the leaked Red Hat 8 ISOs, and got them burned. I won't have time to finish the install on my laptop, though, because it'll take ages for my silly slow laptop drive to be checked for bad blocks. Ah well.
Time to leave and go buy some groceries for dinner. What a non-productive day. I hope phil doesn't read this entry.
Urgh: forgot to publish this before actually leaving for grocery shopping.
Forty-five minutes. Ken said that lunch would take forty-five minutes. Well, after an hour lunch, we had to go to MEC, where I had to buy a harness and chalk bag. So I lost two hours in the middle of the day. Oh well.
For pretty much the rest of the work day, I chased this bizarre crash that phil was seeing. It's one of those "can't happen" things, so I think I'm going to have to reproduce it under the debugger and see what happens. A puzzler, that's for sure.
More climbing tonight. Man, do I need more grip strength. I think I need to hire some disobedient minion, so I can spend a few hours each day crushing his windpipe. Or maybe I could just get one of those squeezy balls. That's probably cheaper.
I'm all nicely tired now, so I think I'm going to go to bed. G'night!
The season premiere of Buffy was good too, though our little backup TV doesn't really do it any favours. I need to get on the "buy a new TV" thing at once.
Got more of the lock-revocation stuff working today, and cleaned up some old bugs that had been bothering me for a while. That was nice, and soon I hope to hear back from HP about their test plan, so that I can make sure they're testing the right stuff. Or that I'm writing the right stuff, I guess.
We might be the only people in the theatre, but I bet I can get Mark and Aven to accompany me to Mel Gibson's latest project.
I had a fantastically difficult time getting out of bed this morning, and it wasn't the first time this week. I hope this passes soon, because it's getting really old. I'm supposed to go watch the premiere of The West Wing at Alasdair's tonight, but I'm not sure I feel up to it anymore. Maybe I'll just tape it instead. Or get phik to digitize it for me, so that I can watch it over lunch tomorrow.
He's no fun. At least Ken loves me.
Maybe it's just because I watched it after returning from the cinematic abortion that is Ballistic, but the first episode of CSI: Miami was pretty darned good. I'm looking forward to seeing more, and it appears that the CSI folk have done a much, much better job than whoever it was that decided to make all the lame Law & Order spinoffs.
Tyla and I went to get our health cards today, which went pretty well. I had to explain to the woman at the counter that, in spite of my frequent jokes to the contrary, Quebec is a part of Canada, so, no, we didn't have a "break in health coverage", and, no, we didn't risk our Ontario coverage if we were out of the province for more than 30 days total in the next 5 months. (Which I will almost certainly be.) Apparently, our cards are supposed to arrive by the first of October, which will impress me a fair bit.
I got another decent chunk of recovery working today (we cancel any locks that a dead client had, so that other members of the cluster can get on with their getting on), which was a nice bonus on top of not having to participate in the twice-weekly conference calls any more. Certainly improved my mood.
My headphones started to die today, so I'm borrowing Fixy's. (I should send him mail and tell him that.) I need to buy some new ones, but maybe I can get Bay Bloor to throw some in along with a TV purchase? Worth considering, at least.
Push-ups are pretty good exercise, and apparently they're quite good at strengthening a bunch of muscles used in climbing, but there's basically no way I can add them to my current exercise routine. I just have nothing left in my arms by the time I get done, even after letting them rest during the sit-ups. Maybe I'll do grip exercises and push-ups on the off nights — those off nights that don't involve actual climbing, that is. Thursday is currently designated as a climbing night, as is Sunday morning, so that really only leaves Tuesday and Saturday. Still, having more stuff that I can do just as well when I'm travelling would be good, since taking a week off during our Calgary trip made the subsequent few sessions pretty harsh.
Further, Google News is outstanding. Go now, go often.
Now, to head home and watch some Buffy.
I know it's pretty and all, and it does include things like Thanksgiving, my wife's birthday, and the Cocktail Party, but I really don't like fall all that much. And given how I seem to always get a moderate-to-severe case of the blahs at this time of year, I could probably convince someone that fall didn't like me all that much either. If they weren't paying very close attention, anyway.
Alasdair and I saw Ballistic (I refuse to compound the misrepresentation found in the fuller, colon-laden title) tonight, and it was, well, tremendously bad. Deliciously bad, perhaps, but it left a somewhat sour aftertaste. Or maybe the metaphor I'm reaching for — stretching, you might say, and I would not raise a cry in protest — involves more mildew. It did give Al and I quite a bit of befuddled and nonplussed dinner conversation afterwards, but other than that, Al is quite correct: the most satisfying thing about the movie was that when it ended, one could go and urinate. Not recommended, unless perhaps you are Chris Blizzard, in which case it will only help improve your opinion of Reindeer Games.
Phil and I are going to work together on recovery, which really means that for the next few days we are going to run a race in parallel lanes, trying to avoid tripping each other. On Friday, we will merge into one lane, discover that one of us is skiing and the other pole-vaulting, and scramble to make sense of what we've done. Or, perhaps, it might all work out, but given how much utter shite I wrote and then removed from my editor today, I'm allowed to be a bit code-grumpy. So there.
Tomorrow is the season premiere, so of course Tyla has been brushing up on her lore. A scholarly (?) journal devoted to Buffy the Vampire Slayer was an unexpected find, but I must confess that I, too, am interested to learn more about how "as Spike comes closer and closer to the Scoobies he proves himself an able ironicist". No, really. (I can feel the power of Alasdair's eyes rolling as I type this very paragraph.)
I'd like to start with a special thanks to my mother, whose encouraging words regarding the card puzzle warmed my heart tremendously:
Subject: magic my ass!
...Read your latest web entry and tried out the magic trick. You really don't get it?? You're kidding, right?
That's the kind of support only a mother can provide. Anyway, yeah, I didn't get it right away. It's not like I spent an hour scratching my head — though I guess it might have turned out that way; sort of glad I didn't try, now — so you can stop hiding my picture in shame, Mom. Sheesh.
Yesterday's entry was waylaid by an unexpected 11.5 hour sleep, beginning unexpectedly at 10pm and continuing against expectations until 9:30 this morning. Had it survived, it would have detailed such excitement as:
- the purchase of some nice climbing shoes, which fit so nicely that it didn't really matter that they were the only things in my size and would have been acclaimed regardless;
- the purchase of two books;
- the non-purchase of a new television, aborted due to uncharacteristically poor service at Bay Bloor Radio (what do you mean, "I can read about it on the Sony web site"?); and
- an absolutely fantastic meal cooked by Tyla, featuring Cantaloupe-Tomato Gazpacho, Roast Salmon with Spicy Soy Dressing and steamed sugar snap peas.
Today, climbing occurred. We had a very good and thorough introductory lesson at Rock Oasis, and then did some climbs with Madhava afterwards. I did relatively well on the last 5.7, but didn't make it to the top because I didn't have enough strength left in my grip or arms for the last move. Time to add some forearm and wrist work into my exercise routine, clearly.
I'm trying pretty hard to avoid doing work this weekend, but I see that I have a voice mail from Peter, so I'm going to cave and call him back. I'm so weak.
(And I'm glad I did call him back, because we hashed out a huge amount of the recovery protocol in a very pleasant way. Duplicate locks due to partition and MDS reboot? No problem. Client reconnection racing with a revocation of locks and filehandles? Pfft. Duplicate client UUIDs in a NAT configuration contending for a connection? I laugh at thee. Now we just have to write the code. The best part is that I get to work with phik very closely next week, as we making lock replay work, and that will be a tremendous amount of fun.)
The last few entries sucked, even ignoring the sub-par orthography, but that's just truth in advertising: the last few days kinda sucked, too.
Today was a little bit better. I knocked a few things off my TODO list, and combined with last night's ten hours of sleep, that really improved my mood. Of course, something was wrong with the cluster today ("missing kernel source" gets the stamp, for the people out there still playing MCR-failure bingo), so I blew some time on that. It wouldn't be a day at work without something along those lines.
I was thinking of seeing Ballistic today, in spite of the fact that it's apparently really bad — or perhaps because it's apparently really bad — but my partner in cinematic slumming is out of town this week. How will I endure?
As regards the previous entry's use of the phrase "Brian and I were both wrote", please be advised that it was one of those days. Furthermore: bah.
Brian and I were both wrote: the Lustre parts didn't work correctly. I've found and fixed one bug, but at least one more remains. Argh.
Did manage to have a nice lunch with Miriam out in the prairies, though. Yum.
Well, we ran our test today. Finally. And it didn't work, because Kimberlite wasn't happy with something. Brian and I both think that the Lustre parts worked correctly, though, so I'm going to leave that in his capable hands and try to get some work done this week.
Mom finally called me today, which is so shameful I can't really bear to talk about it. But she did have some truly fantastic news: her decade-long battle to get CCRA to follow their own damned rules about taxability of support payments has come to a close, and she is victorious. Woo-hoo!
Tyla made a fantastic dinner tonight, whilst I was lounging in the bath with a glass of wine. Yummy summer-sausage-and-lentil soup, and a stunning salad of apples and pears in a walnut dressing. My god. It made me totally forget about the incredible misadventures that befell our poor cluster today.
If I go bed now, I can get to the office and get some work done before a meteor strikes the cluster, or whatever tomorrow's disaster will be. And then I can take a nice long lunch to see Miriam during her layover, too. Whee.
Work today was a series of wrestling matches so overwrought that I half-expected to find out that I was being shown on pay-per-view. CVS timestamps, configuration fragility in Lustre and my workstation locking up: I took all comers.
And I pretty much got my ass handed to me, so we're going to try again tomorrow.
On the other hand, I did manage to sell the television, make a nice lasagna dinner for Emily, Tyla, and myself, and only melt a little bit of plastic by not checking the oven contents before pre-heating it. So the day wasn't a total wash.
Looks like I'm going to be sitting on a panel at issre 2002 on the topic of open source software reliability, or something. I hope I come up with interesting things to say before then.
A lazy Sunday, as they should perhaps all be. It takes some of the thrill out of getting up at 11 when you go to bed at 4, but it's still a bit of a treat.
We went grocery shopping, so I'm making Barry's Killer Lasagna tomorrow, and Mark Bittman's Shrimp, Roman-style on Wednesday. Except that I'm eating out Wednesday with the local geek group. Dammit, why did I just remember that?
After our grocery shopping, we went to visit Dad-and-Lisa-and-the-girls for dinner. We got to see their new pool and landscaping, which is, I must say, a lot prettier than I had expected. (I generally find backyard pools to be a bit ugly, and a waste of a perfectly good backyard, but this was neither. Hurrah.) I also decided what I should get them for their anniversary present this year. So cunning!
Two nice men are coming by to see the TV tomorrow, independently, so I have the highest of hopes that it will be out of my life quite shortly.
In other parental news, it's now too late to call Mom. Sorry, Mom, you raised a total loser.
I'm still having quite a bit of fun in Anarchy Online. Wonder how long that will last.
Random fact: as of today, and not counting the menus and template stuff, I've written 274472 bytes of HTML for this diary.
While I did play some video games today — Anarchy Online has its hooks in me relatively deeply at this point — I also had lots of non-electronic fun.
Both Tyla and I were up late last night, so it was a bit of a rush to get to the movie, but it was well worth the effort. A genuinely fun movie, which didn't require any more concentration or analysis than we were willing to invest before lunch.
Lunch was more of a disaster, since the service was slow, forgetful and inaccurate at Café Nervosa. At least the food was expensive.
We were tired, inexplicably, and so we went back to the house for quick naps before Alasdair, Kristen and Mike joined us for dinner. We had Mexican at the great little place down Bloor, and then wandered down to Rancho Relaxo to see Craig's band. We all got tired before their late start time, so I chatted with him briefly and then we retired.
We're all such pansies.
The cluster was off-line today for a two — no, wait: seven — I mean, ten hour maintenance exercise, so I didn't get to run my test after all. We Lustre folk were going to take the day off anyway, other than my foiled test, so it didn't take much convincing to get me to spend the day lazing about the house and playing video games. Not that it really ever does.
I also put an ad for the TV in the paper today, which means that some nice person might come and give us money and then take the television away, which is really the best part.
Dinner at Insomnia trumped the more adventurous Brunswick House plans, and then we rented Zoolander for entertainment. I hadn't realized that it was based on two skits from VH1, but it sure puts the crap that gets scraped off of Saturday Night Live to shame.
Queen Video and I are currently in the middle of a very mild, and exceedingly polite, disagreement about whether or not I returned Sexy Beast to them. I think I did, and Alasdair and Madhava also recall in no small detail the trip in question, but the kind folks at Queen Video can't locate the DVD, and they have no record of my returning it. They said they'd "turn the place upside down", in case it was misfiled, and then we could split the cost of the replacement if neither of us could find it in a week or two. Eminently reasonable.
Emily, who joined us this evening, says that she has an excellent photo of Tyla and me. Hard though it may be to believe, we apparently both look great in this photo. We'll see, I suppose.
Tomorrow, Tyla and I are going to see Whasango (a.k.a. Volcano High) at one of the film festival screenings, and then hang out with Kristen and Mike. In the evening, a friend's band is playing at Rancho Relaxo, so we might pop by there for dining and/or drinking as the night wears on.
For a day that I really didn't spend at work — I was feeling kinda crappy, and the morning conference call didn't really fill me with visions of unfettered productivity — I sure managed to make a lot of progress. Our contract officer at LLNL decided, in a maneuver that I think is both wise and convenient, that our outrageous successes of the last week were good enough to demonstrate the awesomeness of our software, so we're considered to have passed our 15TB test. Brian got the shared-storage magic box back in the magic sharing-of-storage game today, but silly configuration brainos on my part kept us from running our test. That's what Friday is for, clearly.
Andreas believes, and I have no reason to doubt him, that on Monday night Lustre mounted the largest filesystem ever, well, mounted under Linux, at 85 terabytes. And Peter thinks that our 220 megabytes-per-second single-client write rate puts us at the top of networked filesystem performance history. So that LLNL guy probably does know what he's talking about.
I suspect, deep down in my suspectothalmus, that tomorrow I'm going to help Brian do the MDS failover test a few times, we're going to celebrate a bit, and then I'm going to play Anarchy Online with my boss until dinner time. Speaking only for myself, I can't wait. (Might be that some extra tuning of the OSTs is in order as well. "Sir? Sir! I think you have that in backwards.")
Alasdair is quite right that my work-focused entries are a bit boring to read, though of course I find that those days are anything but boring to experience. I now make a solemn vow: this weekend, I will do something interesting, which does not involve computers other than in optional, supporting roles like payment or telephony-assisted friend notification. Further, I will write no fewer than 3 meaty, quip-laden paragraphs on the topic of such exploits, primarily for my own amusement. I will also mail my mother, whom I love very much, but for whom I seem to have some sort of electronic-mail block, as though my emotional optic nerve attached to the retina of my INBOX in such a way as to defeat even the very bestest of filial-communication intentions.
My good friend Phil is not feeling well, so people should think get-well thoughts on his behalf. I will be, so you don't have to feel self-conscious about it.
This entry skipped in honour of — oh, who am I kidding? We spent the whole day travelling, and I'm too tired to write an entry that will be worth the typing.
Glad to be home, though.
Today, I am wimping out and not even using complete sentences. Sosumi.
- DDN still totally "casters-up", ergo no MDS test.
- Lunch, then walk back from downtown Canmore. Where is my afternoon?
- Phil makes data write quickly. So quickly. Now he needs to take a vacation before I have him abducted.
- Back home tomorrow. Kinda nice.
Today was going to be the first "real-life" test of my MDS failover work on the cluster, but a critical piece of hardware developed a serious case of the sucks, so we'll try for tomorrow.
Instead, after doing all the groundwork I could, I hacked up a crappy little test-reporting tool so that the good people at LLNL can run massive test sets for us. That was sort of fun, but man, is my Perl rusty.
After dinner, Tyla and Peter and I took a little walk in the mountains. We didn't see any bears, but there's always tomorrow. I wish I spoke better French, because English doesn't really have words to describe how scandalously nice this house and its surroundings are.
It's starting to feel like we're going to be able to pass this big test in the near future, if enough of the cluster can stay up.
You have: 220 megabytes/sec
You want: terabytes/hour
I don't know how these kids do it. After a lovely two-hanky ceremony, a wonderful reception (featuring probably the best speech I've ever heard at such an event, delivered by Edgar's father Edgar Ricardo), dinner and dancing, everyone but me was still raring to go when the proverbial shop was packed up around 1am. I am clearly getting too old for this sort of thing.
Today we're going to a little open house at Janice's brother's place, and then heading off to beautiful Canmore for some fresh air, lovely wilderness and Peter-assisted recovery domination. Should be a pile of fun.
First, though, I need some food. Onward! (Happily, it's a fair bit more summery outside today.)
"Good morning, Mr. Shaver. It's 8AM, and 7 degrees." As far as I'm concerned, both numbers in this morning's wake-up call were too small. At least there isn't any snow.
The trip to Calgary was mostly uneventful, once Tyla removed her Swiss Army knife, tweezers and nail file from her carry-on. Where, you may ask, has she been for the past year? A fine question. I think I was nearby for almost all of it, but things like this do make me wonder just a touch.
Tyla's off visiting some relatives, and I'm still hacking on recovery. Right now, though, I'm going to hop in the shower and then rustle up some lunch. (This is Calgary, after all.)
Recovery progresses: we no longer crash, but neither do we seem to complete successfully. More debugging after lunch.
Post lunch (Vietnamese, quite nice) update: Recovery works. Not a great screen shot, but a lot of effort went into the last line in "Virtual Console #1 (client)" that reads:
That means that we were able to shut down the MDS, restart a new one to take over, connect to the new MDS, replay all the state it needed to catch up, and then let the cat process continue, oblivious to it all. Whee!
Now don't get me wrong, here. There are some nice things about being a tall guy. I can play mediocre basketball without really any skill. I have never needed to stand on a stool to get something out of a freezer. Nobody will ever know if I start to go bald. (Except, I guess, Chris-at-the-office, who towers over me a little.)
But when I watch my much-smaller wife pack, and realize that she can fit a week's worth of clothing into a Kleenex box, I start to covet petiteness just a little. I certainly need a larger garment bag, to avoid Terminal Suit Crumpling Syndrome. And then I could put a pair of shoes in the bottom of a garment bag without having to break new ground in tessellation. To dream!
Speaking of dreaming, I had a strange dream last night. I had been given the task of resurrecting our recovery code so that we could run an important deliverable-demo. I fought tooth and nail against time and software, and managed to get through the Hard Part with a day and a half to spare. When I turned my attention to the remaining, ostensibly-Easy Part, though, I found that it was anything but Easy. And I cried like a child.
Only I wasn't dreaming. Sigh. I guess I'm hacking on the plane!
Here's a sign that maybe I work with really big computer systems a little too much:
<shaverHome> phik: so what's the biggest FS statfs can report on Linux?
<phik> 16TB, I think
<shaverHome> that's outrageously small
<phik> no kidding
I'm going to mail my Mom right now, while I wait for my laundry to dry. Then I'm going to go to the office, pack my laptop, lock my screen, rush back home, and take off to the airport. Calgary, here we come.
I went to sleep last night (OK, early this morning; did I mention that Anarchy Online was fun?) worrying about how we were going to get around our various network-layer problems in time for this big deliverable/demo/test, but when I got to work today, Eric made me feel a whole lot better:
<eeb> looks like I found the netregression problem :)
I finally brought my new keyboard to work today, and it's taking some getting-used-to, as the kids say. The keys require a fair bit more pressure than on my previous keyboard, but at least the #@$@^%!@ backslash key is in the right spot.
How bad a person am I? I haven't replied to any of my mother's last three emails. That's pretty outrageously bad. I should be replying instead of typing this entry. Why are you reading this? Don't encourage me!
I think I sort of forgot that Tyla and I are going to Calgary this weekend, when I agreed to handle this MDS-failover demonstration — or, at least, failed to object violently to handling it. Tomorrow will be, I predict, a very busy and exciting day, since it took all of today to just get to the point where the client reconnects to the MDS, and I have not yet even started on the replaying of connection state. I have a record, somewhere, of Phil telling me that it shouldn't be too hard, which will be handy if we every end up in court.
I'm sort of enjoying Anarchy Online these days, which is pleasantly surprising; I had a horrible experience when I played it at launch, as did pretty much everyone else who tried it then. Of course, I don't have a lot of time for it right now, what with all the hours devoted to debugging on the wrong copy of my test suite and scratching my head over random iozone failures.
They finally put some good content up on the web describing the MCR cluster we're shipping Lustre Lite onto. The pictures are pretty impressive, if you're into that sort of thing.
Chris is coming to visit tonight, on the way to canoeing fun. I wish I got to go on canoe trips, but I don't think I'm going to have time at any point this month. Or, say, next month.
OK, so almost everyone's mail was working. If you weren't named Madhava, you were probably pretty happy with the migration. Once you stopped sending mail to your grandparents with "bitchcake" in the header, that is.
But if you were Madhava — and therefore, in all likelihood, still are — oh, woe unto thee. First, you got screwed because I had two /etc/passwd entries for you, which caused no end of confusion on the part of our beloved Postfix. (If one /etc/passwd entry is good, two must be great! Or, as it turns out, quite not.)
After that, you got to be the first person to run into some Postfix limit on mailbox size, which we remedied shortly. And after that, a few last pieces of mail bounced because postfix isn't set up to create new mailboxes, just append to them. What a joke. Again, Madhava, my apologies.
(I took a break in the middle of fixing this stuff, after I thought I'd fixed it the first time, to describe the new recovery infrastructure to Peter and sit in on the weekly MCR status call. Nothing tremendously exciting, though I'm glad Peter likes the new recovery design as much as I do.)
In the afternoon, I chased some bugs that were introduced in my huge landing, and phil and I killed the one that was keeping us from proceeding towards our next huge goal: phil and others are going to make sure we can write 15 terabytes (trillions, yes, of bytes) of data to a cluster of storage nodes, while I make sure that the client doesn't get too put off by the fact that someone is going to abduct the metadata server halfway through. For our deliverable in December, I think we have to pull the same trick while blindfolded and juggling cats.
Today's product endorsement is of Pete's Tofu 2 Go, in the Lemon Pepper variety. Very nummy, nicely filling, and the Mango Chipotle Sauce is quite nice as well. I should really get some fruit for the office, but in the meantime this stuff is way better than the other crap I was putting in my mouth today.
Will I ever be able to justify an X-box? There is now a glimmer of geeky hope.
For a holiday, today was a lot of work. We got to have a lot of fun first, though, starting with a rushed but pleasant breakfast to see Miram off on her way to Ottawa via Peterborough and Kingston. After that, we hustled to the CNE grounds to catch the Red Arrows' performance, which was quite impressive. We next caught a very interesting gun run show, and then visited some agricultural animal exhibit things. All very fun, and the Snowbirds were a wonderful ending.
But, just as we were preparing to leave for home, I noticed that I had a voice mail message from Amos. Neon had been brought back to life after some nasty heat-related (we think) downtime, and it was time to migrate everyone. That's just about done now, thanks to the tireless efforts of Amos and Phil, though there are a few gigs of web-mirror stuff to pull the wrong way down Amos's DSL. And I'm certain in no uncertain way that we've forgotten something, but mail and ssh seem to work, so whatever.
During all that, I managed to get my monstrous recovery + network-rearchitecture + mild-debugging-and-cleanup patch ready to check in:
35 files changed, 629 insertions(+), 444 deletions(-)
Well, mostly ready. We currently crash at unmount because we attempt recovery on a connection that has already been freed, but there are mysteries piled atop secrets here, so I'm going to check in anyway. Phil's going to need to live in that tree for a while to help me figure it out, and I had a conflicting-change warning shot fired by Andreas, so in I go!
Now it's time to go home. Yay for home, where I have a lovely wife, and some weights to heft before bed.
It took quite a while, truth be told, for Miriam to show up last night. All was quickly forgiven because, hey, it's Miriam and we love her dearly, but it meant that I had chicken wings and beer for dinner at 11pm. Makes Tyla's Cinnabon breakfast look pretty respectable, I suppose.
Today we got dim sum at the Kowloon, as we often do when there are people in town on a Sunday, and then I came to work. A phone call to Peter to clear up some handle/export/connection confusion resulted in the two of us savaging the existing request-processing infrastructure. He wasn't as concerned about the state of recovery as I was, and this will make things dramatically nicer, so I'm already excited about the coming week. I forgot to bring my new keyboard in, because I'm a moron.
The plan — Miriam's, at least — is/was to see Y Tu Mamá También this evening, which is reported to feature quite a lot of nudity. I think I'm going to give it a pass, though, so that I can get this networking stuff back up off the ground tonight. Peter promised to redocument once we were done, if I sent him a tree, and I hope very much to meet my end of that deal.
At long last, I found my copy of Salt, so I'll be starting into that again once I finish From Beirut to Jerusalem. Maybe I'll take tomorrow off, after Al and I hit the Ex to see the Red Arrows, and just read and lounge. That does sound awfully nice, doesn't it? Especially if I do it beside my Dad's new pool. Hmmm.