I’ve been stuck on a bug that somebody before me last tried to fix at 2am. It is the kind of bug that compels you. It is the kind of bug that makes you sit up at 2am, convinced you’ve fixed it, only for it to remain stubbornly unmoving.

I have included a precis of it below, for anyone who knows about these things and would like to join me in the frustration.

When you define a thing, you lock it into being. Sometimes that’s useful, but oftentimes it’s not.

Sorry for folks who were lost last week. This week might not be better. But I’ve only just started writing these, so who knows? Maybe there’ll be good, generally applicable lessons. In general nothing I do feels pliable, but I can’t always tell.

At last our flat has been surveyed, which means the slow march of selling can continue. Everyone so far seems content with the price that we’ve agreed on, so all that remains now is to wait for everything else to happen.

This is true about everything, actually.

I took out an equity loan with the Help To Buy scheme, which means I have to deal with a third-party agency that won the contract from Homes England to administer the scheme. Reading their Trustpilot reviews is…a bad idea, if you have an equity loan. So far my experience has been okay: they told me quickly that one of my forms was wrong, and I was able to get that sorted by the next day. According to GOV.UK, they’ll respond in “7-10” days. It’s been 6 calendar days, or five working days. Stay tuned for good news…I hope.

I’ve done more AWS work! I do plenty in my day job, but my day job doesn’t publish their code, for perfectly sensible reasons. I am publishing all my code for the mentor-matcher, so let’s take a look at that.

The core of this code is something called a State Machine, also known as a finite state machine. Here is a diagram:

There are a few gotchas I’m finding with AWS. One of these I think is specific to event-driven architectures, which is: events must be small. Small events trigger big things. This leads to some behaviour that I don’t love, such as code like this:

def read_write_s3(function):
    @functools.wraps
    def wrapped_func(event, context):
        data_uuid = event["data_uuid"]
        step = event.get("step", 0)
        data = s3_resource.Object(bucket_name, f"{data_uuid}/{str(step)}.json")
        file_content = data.get()["Body"].read().decode("utf-8")
        json_content = json.loads(file_content)

        output = function(json_content)

        data_for_next_step = s3_resource.Object(
            bucket_name, f"{data_uuid}/{str(step + 1)}"
        )
        data_for_next_step.put(Body=(bytes(json.dumps(output).encode("UTF-8"))))
        return {"data_uuid": data_uuid, "step": step + 1}

    return wrapped_func


@read_write_s3
def async_process_data_event_handler(event: dict[str, int | list[dict]], context):
    """
    Event handler that calls the `tasks.async_process_data` function.
    :param event: A dictionary with an event from AWS. Must have the "mentees" and "mentors" keys
    :param context: The AWS context
    :return:
    """
    unmatched_bonus = event.get("unmatched bonus", 6)
    mentees = event["mentees"]
    mentors = event["mentors"]
    matched_mentors, matched_mentees, bonus = async_process_data(
        mentors, mentees, unmatched_bonus
    )
    return {
        "mentors": [mentor.to_dict_for_output() for mentor in matched_mentors],
        "mentees": [mentee.to_dict_for_output() for mentee in matched_mentees],
        "unmatched bonus": bonus,
    }

where I’ve written a decorator to deal with the continuous writing-in-and-out of S3, leaving the function to do the single thing I need it to do.

This is a perfectly good approach to take, I just think it’s a bit ugly. But it gets me lots of attention from recruiters, and what boy can say no to that?

(This one. I’ve even updated my profile to tell people I’m happy where I am.)

This code takes great big chunks of data and, when a specific keyword is passed in at the top:

• Creates a number of copies of it
• Applies, or maps, the ProcessData function over each of them
• Aggregates the results and reduces it to a single answer

This approach takes advantage of the immense scalability of the cloud platform to do several million parallel calculations, and then reduce them down to the best solution. It takes about two minutes, and costs about a dollar. Maybe two dollars if it’s real tricky.

I spent some time with my mentee this week – though, truth be told, she’s more senior than me now. Mostly we just hang out and do cool stuff. I’ve also picked up another mentee, and we’re going on a functional programming journey together. He’ll be talking to his colleagues with deadly earnestness about monads before long, just you wait.

Three things:

1. Advent of Code got really difficult, although at least partly because I’m forcing it to be difficult
2. Work started to slow down, just as I started to get to grips with it
3. I got some honest feedback that was the brutal reminder I needed to be more considerate

In reverse order:

I am really frustrated with myself that someone had to give me this feedback, because I’ve banged on about this endlessly myself. Senior people, I have written crossly on these very pages, should really think before they speak. They should remember that they’re senior people before they remember that they’re experts.

As with sex and food, good intentions are simply not enough. I fell a long way short of my intentions, stepping into someone else’s domain without proper consideration for the time they’d spent working on a problem. I jumped in already assuming that I knew more than they did, and with a pinch of wanting to prove that I could be valuable. I’m going to cut that out.

On the plus side, I’m so lucky that someone felt comfortable giving me that feedback. It shouldn’t be necessary, but at least I’ve had it, and now I can correct myself. So that’s good.

Work is starting to slow down for everyone, I think, because of the impending festivities. So emails are starting to bounce back from polite but firm out of offices (outs of office?). So out of a desire to write a little code, and to be useful, I’m covering a little information service. So far I’m not writing any backend code, but I am re-familiarising myself with GitHub Actions and the language we use to describe how we deploy code into a production environment. The language is called YAML, which stands for YAML Aint Markup Language. The first YAML stands for YAML Aint Markup Language. The first…

You get the picture. It’s recursive, which is a topic I covered last week and will likely cover again. In fact, it’s come up as I’ve been writing Python in a functional style. The other thing we insist on in functional programming is immutability: that things can’t change or be changed. You can create a copy, but that copy too can’t be changed. It forces you to think about things quite differently.

I’m finding myself applying that thought process to deploying code. The binary package that we deploy should ideally be immutable: nobody, even me as the developer, should be able to poke around inside it. If there’s a problem with it, we should fix the code once at source and then redeploy the binary. This is at odds with the traditional approach, which is caring for servers like they’re pets and jumping into them every so often to mess about with the environment or similar.

“Redeploy” is a weasel word here that actually covers a lot of work. The binary has to be properly tested, and once it’s tested, we want to promote it into a testing environment so people on our team can try their best to break it. From there, we want to promote exactly the same artefact into production. The slower we do this loop, the longer it takes to get changes into the hands of users.

This is why people like me obsess over small changes and automating tests as much as possible. This is what I’ll be working on over Christmas, I think.

Advent of Code got impossible this weekend, and it really killed my enthusiasm. I’m also finding I have less time? less desire? in these very dark evenings. Instead I’m just getting cosy with m’buddy Salem and drinking a lot of tea.

Maybe in January, eh.

I’ve started something new this week: a role at a new department. I’m spending the first week poking my nose into things, meeting people, and getting my hardware/software/note-taking approach sorted.

I’m currently getting stuck into Obsidian, which I use to take notes in Markdown. It’s got a lovely feature whereby it links together ideas, so I can tag notes about #Architecture and then see at a glance who else I’ve spoken to about it. I’m hopeful that this will help me make sense of the whole thing, as well as keeping a good record of what I’m actually doing.

So far I’ve met a lot of people and generated multiple pages of notes. The constellation of my connected thoughts is currently a little haphazard, but I’m looking forward to seeing how that’ll grow over the next couple of weeks.

My new department is very keen on apprenticeships and so, inspired by Terence, I’m going to sign myself up to something. Before long I’ll have as many letters after my name as are in my name.

Also this week: I wrote a tiny little bit of code and then used Twilio to set up my familial secret Santa. For any rational human being, this is a task achieved through the use of small slips of paper and a hat/bag/inside-out umbrella. My family – my extended family – is closer to a clan, and with the addition of rules such as “No one in your immediate family”, I wondered if perhaps we could get a robot to do it.

Of course we could. Much geekery ahead:

Let’s start with some data:

The parent_names property is useful, because we can use it to identify siblings, children, and parents – so that nobody ends up with someone in their immediate family. This can’t be done purely on family_name, because although Rickard and Arya share the same family_name they’re not in the same immediate family.

Here’s some code to match people together:

def score(person_one: dict, person_two: dict):
  """Good scores are low, bad scores are high"""
  if person_one.get("parent_names") == person_two.get("parent_names") 
  # these would be siblings or partners - for the purposes of this
  # code partners were modelled as siblings
  # hence my use of Game of Thrones as an example...
  or person_one.name in person_two.parent_names or person_two.name in person_one.parent_names:
  # these are parent-child or child-parent relationships
    return 1
  else:
    return 0

and here’s some code to calculate how every person matches with every other person:

import csv
import munkres
import itertools
import math


with open(family_data.csv) as the_clan:
  everyone = [score(*pairing) for pairing in itertools.product(csv.DictReader(the_clan), repeat=2)]
  square_size = math.sqrt(len(everyone))
  matrix = [lst[i:i + square_size] for i in range(0, len(everyone), square_size)]
  matches = munkres.Matrix(matrix).solve()
  # this returns pairings that match the ids in the table above
  return matches

I used this basic guide from Twilio to help write my function to send the messages.

# Download the helper library from https://www.twilio.com/docs/python/install
import os
from twilio.rest import Client

def send(recipient_number: str, gifter_name: str, giftee_name: str):
    account_sid = os.environ['TWILIO_ACCOUNT_SID']
    auth_token = os.environ['TWILIO_AUTH_TOKEN']
    client = Client(account_sid, auth_token)
    message_body = f"Dear {gifter_name}, you're going to be buying a present for {giftee_name} this year. " \
                   f"Don't forget to put some ideas in the WhatsApp group for your secret Santa! Lots of love, Santa"

    message = client.messages \
                    .create(
                         body=message_body,
                         from_='Santa',
                         to=recipient_number
                     )

    print(message.sid)

One cool feature of Twilio is that, if you chuck them £20 for an “upgraded” account, you can send texts where the “from” field can be a string, rather than a number. The upshot of this is that I could send messages that seemed to come from Santa!