S11E06: Love, even when it’s difficult

I have previously mentioned that our cat is currently in full design crit mode. Worse still, he’s not a collaborative design critter: he just pisses on things.

That’s not a metaphor. Our flat is…unpleasantly stinky right now.

And it’s making me reflect on the nature of unconditional love, and self-mastery, and trying to let go of mundane things. It is not easy. Curtains are just things and cats are beautiful and delightful creatures, and yet sometimes I find myself wondering whether the curtains are worth more to me. I don’t like feeling this way. But sometimes I do.

I’m also reflecting on how – honestly, how humiliating I find it for someone to damage my things and to then be forced to play with them. I’m yeeted back to childhood, having to make up and play nice with a bully who pushed me over. I don’t like it. It doesn’t feel good. But I have to remember that, unlike the bully, Salem doesn’t know that what he’s doing has upset me. All he knows is that he’s trying to mark his territory – to guard against what terrifying creatures I dread to think – and we keep stubbornly removing the markers. He likely can’t figure out what the hell we’re doing. I like to imagine in his mind he’s the zombie apocalypse prepper and we’re stubbornly unplugging his forcefield and emptying all of his weapons, and he can’t figure out why and he can’t communicate with us. But he keeps trying nonetheless.

We love each other, it seems, but without communication we’re driving each other up the wall. There’s…probably a lesson there. Not for me though. I’m going to keep trying to convince him that he doesn’t need to put up a forcefield across the curtains.

Work

I had an interview recently for a senior role and I don’t know how it went. I never do. I hope I’ll find out this week whether I got it. I did get a lovely bit of recognition from a dear colleague though, and I feel really proud of myself for that. It comes with a little gift, and that’s nice, but it’s less important than knowing that the unseen, non-dev bits of my work are also being seen and appreciated. I built something quite cool recently, and it worked on the second try, and I got to talk about it at our fortnightly Show and Tell (which my partner lovingly tells me features a voice suited for gameshow hosting).

My side-project, my corporate objective, bustles onwards. This week I finished up some code that meets the outcome I’ve been trying to achieve for about seven years.

It is a process that identifies and pairs off people on our graudate development programme. It does so on the basis of weighted scores, and using last year’s anonymised data we managed to pair off 726 people with well-matched roles. Not everyone got what they wanted, but nobody got anything they really didn’t want, and that’s positive. You’ve never seen anyone so pleased to see a message this simple printed onto their screen:

There were 843 bids in total against 726 total candidates
Task completed in 12449.306011199951 ms

This process used to take about two weeks of staff-time, or about 266,400 seconds. I’ve managed to get it down to 13 seconds: an improvement of about 99.995%. In fact, I think it used to take two people two weeks of staff-time, so it’s really an improvement of 99.997%.

(This number has completely thrown me. Apparently after 99% improvements everything is marginal. I’m genuinely a little disappointed by that.)

Anyway. I hope this means it’ll get broader uptake, and I’m genuinely curious to see whether anyone else running a graduate scheme will pick it up. Are there conferences for this sort of thing? Can I get up on stage and say “Look, this’ll save you one week, six days, seven hours, twenty-six minutes, forty-five seconds every time you do this. Please…I don’t know, buy it?”

(It doesn’t work like that, which is why I’m not allowed to do sales. It’s complicated, and every use-case is slightly different, so you either have to change your process so that it meets what the software does or I have to build a bunch of costly customisations, and it’ll just be expensive for you, so why not just use the same process everyone else does?

Again, this is why I’m not in sales, but also why I’ve never had a developer quit on me.)

I also did something quite clever (and possibly quite over-engineered!) to encode this. Please, if you’re a Python dev, check it out and tell me how I can make it simpler.

Otherwise…tips for getting cat pee out of fabrics are most welcome.

S11E05: Snackered

Last week I was tired. This week I’m exhausted. There’s a lot going on.

MSc update

Gang don’t – don’t try to do an MSc alongside a full-time job, especially when that MSc is in a proper academic subject like securing computers. And the Internet. And the various other Internet-connected things we’ve stupidly built. The people who started thinking about securing computers were the sorts of people who thought real deep about math and then built thinking engines. It’s so damn complicated and it’s so mathematical and it’s so wrong. It’s as if someone sat down and thought all the way through what a perfect meal would contain and never once considered that it would interact with a human.

Don’t get me wrong – it’s fascinating. But in my rush to perform perfectly, to continue being the guy who can do everything, I may have gone slightly beyond my means. If I could I would rewind and maybe just take this one module at a time. But I’m halfway through the session, after which the workload drops by half. And I will get an evening or two back and I am so excited. I’m going to take people on dates. I’m going to sit down of an evening and not fret that I should be studying.

My notes continue to take shape, a messy constellation of linked concepts. Not unlike the Internet; not unlike people. We design systems that reflect ourselves. As above, so below.

A messy graph of nodes and lines. The nodes are of differing sizes, and everything is interlinked in a web

And this incoherent mess is how I feel at the moment, with so much pulling me all over the place. In a few weeks I’m going to become the sole maintainer of my team’s platform. Within 18 months I need a promotion, or my move to what’s parochially called “The Regions” (I’ve been here only six months and yet I bristle like the best of them: as if London isn’t a region!) will result in a chunky pay cut. The relationships around me evolve in ways I wasn’t expecting (but that I always hoped for!) and they must be managed. Salem has become a design critic, and is manifesting his unhappiness with peach curtains by spraying them with urine.

And my attempts to automate a problem – a piece of work I have been struggling with for seven years! – are starting to come apart at the seams as I fight with the same problem I accused my academics of up above: resenting that my beautiful mathematical model is being sullied by humans and their needs. It is the most fascinating, the most crunchy of problems. It is exhausting.

And despite this! Despite my sullenness and my cave-dwelling; my lack of energy; my frankly embarrassing obsession with a problem nobody else cares about (would that I had got into Dungeons and Dragons and could put forth Opinions on the OGL!); my need to end a sentence with a joke that punctures the tension built into the rising tempo; my self-referential bullshit –

Despite this I am surrounded by love. Love that is unconditional. Love that is still there when I don’t have the energy to match it. Love that overwhelms me and keeps me afloat. This is what it’s all about pals.

Work update

At work I continue to play with AWS. It’s so exciting and so powerful, and yet so incredibly badly documented. We also made a wrong decision about 2 years ago, and we’re kind of stuck with it for the interim, or at least until I spend two days re-writing the thing by hand. Nonetheless even working with that less-than-preferable approach, I’ve figured out a really neat way to get some code to run on a remote server in a way that’s deterministic and predictable, and that we can test.

I’m hoping to convince my organisation to pay for me to do a bit of training in this, because the more I use it the better I understand how well-architected cloud solutions like this can really cut costs. I think in one area I can probably decrease it by about 75%. It also means we’re locked into the vendor, but right now I can live with that.

My corporate objective is crunching along. I’ve got access to hundreds of data points, and now I need to figure out how to make them match properly. The model I’ve developed works on a small scale, but unfortunately at a larger scale things stop working. I think it means we’ll need to flex some rules, and that really disappoints me. I was so confident that if I could just find the right…configuration, the right algorithm, this intractible problem could be solved. But it can’t. At least, it can’t at the moment. There are a few tweaks I can still make. I’d like to start by flexing what’s disqualifying, and additionally figuring out what to do with people who have really strict constraints. There’s more work to do there, and it remains really interesting.

I’ve got an interview for a promotion on Monday. Wish me luck.

S11E04: Will nobody rid me of this turbulent flat

I know why you all come here. You come here to enjoy my fun little rants about my flat. Well it’s over, you hear? Over. There’ll be no more of that. From here on in it’s purely professional.

FLAT NEWS (for the last time)

I am writing this on Thursday in the hopes that if I write it will happen: I will complete the sale of my flat tomorrow. It has been a very exciting journey, and generated heaps of hashtag content for me, but I will be very, very pleased to have the damn thing off my plate. Today’s hilarity was a sequence of questions at C-minus-21-hours, all of which could have been answered by reading the things I have sent several times by email. I do not appreciate being used as the search function on whatever terrible implementation of Sharepoint they’re using, even if it is faster.

Once I finally have everything offloaded I get to do a bunch of offboarding from services. My electricity was very easy and is already lined up. The council’s website was…less good, and I’ve had to take a pause because I’m not at all convinced that I’ve pressed all the buttons and made the appropriate sacrifices in the right order.

And then – at long last – the poem I wrote will have a new home. And I’ll need to find something new to write about.

Writing code

This is going to be a little less interesting for folks, but.

I’m working on some code to match Fast Streamers, and I’m now getting into a nice crunchy question of how we calculate scores and things. As we do, I’m trying to keep in mind that I’ll need to separate things out later. Some schemes will want to score different things, and there’ll be different inputs. At the moment I have a certain amount of inheritance going on, but I’m keen to figure out better ways to write this. At the moment there’s a sort of buzzy sense around composition, and some factories maybe? Like each scheme might have its own scoring approach and if not it can fall back on the default…I think. Comments welcome.

I’m also very aware that my colleague John has been doing some work on our shared mentoring labour, and I have things to fix up. I suspect that we left it alone for too long and some library we rely on hasn’t been updated or has rotted away. Such is the nature of writing code in the 21st century, alas. John: if you’re reading this, the moment I’ve sold my fucking flat, I promise I’ll do some work on the mentoring code.

Being in love news

It continues to be awesome and I recommend it to everyone. I write poems for the people I love, which means being loved by me is double awesome, although then you do have to do literary critique as homework.

Terror of getting older news

I saw my parents over the weekend and we had a conversation. And it was frank and open and honest, which I love about my parents. From them I get a really strong principle of right and wrong; heaps of compassion; a willingness always to help those in need; and autism.

The autism means I know exactly how that last sentence will land and crafted it specifically, because one of my hyperfixations is words and language. The phrases get longer as the sentence go on and then abruptly stop, ending in something that’s not like the other. It’s why I could come up with a diatribe against what I believe is their joint unwillingness to accept they’re getting older (even as I, hypocrite that I am, have not done proper exercise in weeks! I eat fast food at least once a week!) off the cuff. I can…feel? I think? The shape of the words that need to come next.

But we stayed up late talking, and that was good. I am slowly getting to a better place with my parents. Step by step, I’m treating them less like my parents, and more like my peers. And that’s sad. It’s sad because they become necessarily a little less when they step down from the lofty perch on which we put our parents. My dad is still the same person, but he looms a little less. He is a guy. He is a guy with whom I have so much in common and with whom I love to hang out. But he’s also just a guy. I don’t have to take all of his opinions seriously, and I don’t have to argue with all the ones I don’t agree with.

And if he’s just a guy – if I can keep the image of the parent separate to the people – then when they die I will not be as angry, or as broken. Because folks will die. People I know will pass. But my dad will always be with me, and my oldest guy friend will die, and I can keep these separate.

Because I think deep down the reason we struggle so much with our parents leaving us is because they promised that they wouldn’t. We are deep down the children we were, and as we look at their bodies failing we feel their strong hands slip from our little ones in a crowd of strangers. And we are once again alone, and terrified, and surrounded by strangeness that cannot be made sensible.

Our parents aren’t supposed to leave us, and they do.

S11E03: I am good at things (and bad at others)

This week saw the beginnings of an end of year review, the lesson that function is more important than form, and envy, sort of.

FLAT NEWS!

A small amount of progress. I am going to put down in words my experience of this ordeal, because I think it is very, very funny. For the small price of one (1) non alcohol beer I will perform it at a social function of your choosing. Here we go.

I am selling my flat. When selling a flat your buyer does some searches, and one search on my property turned up an agreement made in 1990. It was a section 52 agreement, where the developer agreed to commit some money – around £5,000 – to build a car park. Fast forward to today, and this agreement is still on the land where my flat is built. Given that it’s still appearing, the buyer’s lawyers say, there’s no way we can get the money for the mortgage.

This naturally surprised me, given I have a mortgage.

Do you have any evidence, they asked, that this liability has not been passed down through every person who’s owned the land until, finally, it was invested in your flat?

No, I said. Because that would be insane. Because the counter-argument – that there was indeed a conspiracy to designate my apartment as the sole inheritor of this ancient liability – would have some paper trail. There would be at least one email saying, “At last, our evil plan has come to fruition: now the fool will surely be on the hook for a car park, a-ha-ha-ha.”

Now listen, they said. We’re responsible to the lender for making sure we’ve done all our checks. The lender’s interests are not protected if the buyer takes on this extra liability.

I pointed out that the lenders were offering the buyer a quarter of a million quid, and probably wouldn’t protest too much at the risk of another couple of grand, particularly because a quarter of a million at 3% a year is (as they say in the biz) a nice little earner.

They clocked off for the weekend, which means I win. Hopefully, having answered all their questions, I will be able to retrieve the golden fleece and sell this damn flat – though, tragically, it means I’ll need to find something new to write about.

Jobs news

I had an interesting mix of feedback this week. Most of it was good. One piece was that despite the above, I still ran our show and tell with my usual mix of wit, charm, crowd control, melee attacks, and slides with very few words on them. Like this one, which says “Rotation, rotation, rotation”.

(I was presenting some work I’ve done on rotating Fast Streamers. Not like…turning them around. Moving them around. You know what? It doesn’t matter.)

However, I also got some really good feedback from my product manager. The codebase I’m working on was inherited from another team and they…did the best they could. Their best was just not as good as I’d like it to be. However, bitching and moaning about it (one of my all-time favourite past-times) isn’t much use at all. I’ve got to get to a place of radical acceptance. Acceptance, because re-writing the whole thing is impossible, and radical, because radical is cool (except in the context of trans-exclusionary radical feminists: they are not cool at all). It is going to be a struggle but I think it was really something I needed told.

The last time I did something for the first time

Maria guided me through patching up my new coat, whose pocket got caught on a seat. I haven’t sown anything at all since a cub scout badge, and it was a really pleasant experience to mend something. I’ve not done a particularly good job with it, because it’s my first time, but I did something for the first time and it was good. I’m happy I did it. I’m excited to see how long it lasts and if there’s anything I learn for the next time. But right now, it works. It’s patched and it works, and that’s good.

S11E01: Starting again

Hello everyone. It’s been a while. I have been…busy. I’ve had a lot on. A lot of things have happened.

In October I started a distance learning MSc with the University of London – specifically with Royal Holloway. It’s a huge amount of work. I’m learning a lot. I’m learning a lot in 90-minute blocks at the beginning and end of my working day and 4-hour blocks on Saturdays and Sundays. The course front-loads two sessions with two modules each, after which it calms down and I’ll have a little bit more thinking/breathing/being space.

We had our first two exams this past week. I really like the way they’ve been designed – we get 24 hours to complete them, which means folks can still work them around other commitments if they need to. They took me about 4-5 hours each, though, so I’m writing this from a brain that is completely wrung out. There’s no rest, either: we’re straight on to our next session, which started this week.

This session we’re covering computer and network security, and so far my takeaways are:

  • the Internet was a bad idea
  • computers were also a bad idea
  • realistically anything beyond the abacus shouldn’t be allowed

Thankfully all of these things exist, mostly in terrifyingly badly configured ways, so I have a job for as long as the robots permit me to live life.

I took a couple of weeks off to prepare and sit these exams, and it’s been interesting. Even with the deadlines looming, I seem to get to a point of 75% knowledge and then just stop. I can’t learn more unless I start putting it into practice, so I need to find new ways to put my theoretical education into practical situations.

One of the most interesting parts of doing an international MSc is the variety of people I get to meet. There’s folks on this course doing all kinds of security and security-adjacent roles, most of them in languages that aren’t English. I am blown away by anyone trying to deal with academic English when it’s not their first language – I know I struggle sometimes – and so I’ve been working really hard to avoid idioms, sarcasm, and to speak plainly on the Slack instance we all share. Already there are fascinating little conflicts and resolutions, the kind of thing I see every day at work. What software to use for note-taking; what time to arrange study groups; whether to take a coaching or directorial approach to questions. We even have a little study group, and I confess I have a little crush on one of my classmates. It’s nice that I’m getting the full university experience, even remotely.

At work, which I am still doing five days a week, both my junior and my senior have found new jobs. This is interesting for me. I could apply for that senior role. In fact, I hope it’ll surprise some people reading that I’m not a senior. Obviously not you, my friend. You’ve read every episode I’ve ever written. But it’ll surprise some people.

Becoming more senior will come with more responsibility, and I’m not sure I want that. At least not right now, when I need to be able to start and finish at specific times so that I can get in the studying that I need. On the other hand, if I want that to be a way of being senior, it’s kind of up to me to model it.

At the same time – perhaps these things are connected, perhaps not – someone senior I trust approached me about a role in their organisation. I’ve not been looking to move, and my organisation has funded me to do the MSc, as well as coughed up a fair bit to help me move house to sunny Manchester. If it weren’t this guy asking, I wouldn’t have given it a second thought. But it is this guy asking. I don’t know. Right now, with no team around me, and the prospect of a contractors coming in…the idea of starting again is tempting.

(I like working with contractors on specific outcomes, but I find it hard to build a team around them, because I can never be certain they’ll be here tomorrow. Equally neither can they, that’s the nature of the gig, but I’ve found it a little bit harder to build a team dynamic.)

I’m going to wind up here. These were harder than expected to write. I think I’m out of the habit. Here’s hoping it comes back to me.

S10E10: Success!

I’ve been feeling overwhelmingly pleased with myself. I wrote code that extended the behaviour of some existing code, which is like…object-oriented developer 101. And it’s the first time I’ve ever done it in a real project! And it worked! It’s reassured me that I mostly know what I’m doing.

I’ve also talked folks through some of my work with the AWS CDK, and was reminded that I need to hurry up and finish the big AWS rewrite I started a little while ago. It’s almost finished, I think, but it’s not actually been deployed yet. And that’s inevitably where the problems will start.

Moving is inching ahead. We’ve got a couple of weeks left, and we’re at the awkward in-between stage where we have packed enough to feel confident, but the pace is wrong and our burndown charts are projecting past the end of our timebox…

Kanban is a superb way to organise your moving experience, by the way.

I was offered a position as a senior engineer. It’s hugely exciting for me because it means I’m definitely ready for that level of work. On the other hand, I really like my team and the work – and this week has cemented that feeling for me. So maybe I’ll leave it for a little while and come back to it in a year or two.

Continue reading

S10E09: more on constructing clouds

Sorry for folks who were lost last week. This week might not be better. But I’ve only just started writing these, so who knows? Maybe there’ll be good, generally applicable lessons. In general nothing I do feels pliable, but I can’t always tell.


At last our flat has been surveyed, which means the slow march of selling can continue. Everyone so far seems content with the price that we’ve agreed on, so all that remains now is to wait for everything else to happen.

This is true about everything, actually.

I took out an equity loan with the Help To Buy scheme, which means I have to deal with a third-party agency that won the contract from Homes England to administer the scheme. Reading their Trustpilot reviews is…a bad idea, if you have an equity loan. So far my experience has been okay: they told me quickly that one of my forms was wrong, and I was able to get that sorted by the next day. According to GOV.UK, they’ll respond in “7-10” days. It’s been 6 calendar days, or five working days. Stay tuned for good news…I hope.


I’ve done more AWS work! I do plenty in my day job, but my day job doesn’t publish their code, for perfectly sensible reasons. I am publishing all my code for the mentor-matcher, so let’s take a look at that.

The core of this code is something called a State Machine, also known as a finite state machine. Here is a diagram:

There are a few gotchas I’m finding with AWS. One of these I think is specific to event-driven architectures, which is: events must be small. Small events trigger big things. This leads to some behaviour that I don’t love, such as code like this:

def read_write_s3(function):
    @functools.wraps
    def wrapped_func(event, context):
        data_uuid = event["data_uuid"]
        step = event.get("step", 0)
        data = s3_resource.Object(bucket_name, f"{data_uuid}/{str(step)}.json")
        file_content = data.get()["Body"].read().decode("utf-8")
        json_content = json.loads(file_content)

        output = function(json_content)

        data_for_next_step = s3_resource.Object(
            bucket_name, f"{data_uuid}/{str(step + 1)}"
        )
        data_for_next_step.put(Body=(bytes(json.dumps(output).encode("UTF-8"))))
        return {"data_uuid": data_uuid, "step": step + 1}

    return wrapped_func


@read_write_s3
def async_process_data_event_handler(event: dict[str, int | list[dict]], context):
    """
    Event handler that calls the `tasks.async_process_data` function.
    :param event: A dictionary with an event from AWS. Must have the "mentees" and "mentors" keys
    :param context: The AWS context
    :return:
    """
    unmatched_bonus = event.get("unmatched bonus", 6)
    mentees = event["mentees"]
    mentors = event["mentors"]
    matched_mentors, matched_mentees, bonus = async_process_data(
        mentors, mentees, unmatched_bonus
    )
    return {
        "mentors": [mentor.to_dict_for_output() for mentor in matched_mentors],
        "mentees": [mentee.to_dict_for_output() for mentee in matched_mentees],
        "unmatched bonus": bonus,
    }

where I’ve written a decorator to deal with the continuous writing-in-and-out of S3, leaving the function to do the single thing I need it to do.

This is a perfectly good approach to take, I just think it’s a bit ugly. But it gets me lots of attention from recruiters, and what boy can say no to that?

(This one. I’ve even updated my profile to tell people I’m happy where I am.)

This code takes great big chunks of data and, when a specific keyword is passed in at the top:

  • Creates a number of copies of it
  • Applies, or maps, the ProcessData function over each of them
  • Aggregates the results and reduces it to a single answer

This approach takes advantage of the immense scalability of the cloud platform to do several million parallel calculations, and then reduce them down to the best solution. It takes about two minutes, and costs about a dollar. Maybe two dollars if it’s real tricky.