Overengineering our assessment sending process

Recently, we decided to review our assessment sending process. In the final step of our interviewing process, we send candidates a small assessment in which they have to solve a small case study. When we first started this process, we chose the easiest way to do it: sending a zipped version of the assessment per email to the candidate and expecting an updated zip file back per mail. For years this solution was sufficient, but as we grew, it became more and more apparent that this solution did not scale well. For example, we ran into issues such as:

• It was hard to see which changes a candidate made.
• It took a lot of manual work to run linters and other code quality tests.
• It was hard to make changes to assignments and share these with the hiring team who sent these emails.

When we reviewed these issues, we realized that we could solve most of these issues using Git. Ideally, we wanted to make a new repo for each candidate and let them submit their solution via a Pull Request since this would give us a clear overview of their changes. However, the main question that remained for this approach was: Can we automate this repo creation process while keeping it accessible even for our non-technical users? The answer is: yes, we can if we use the GitHub API and a custom-made Slackbot.

Solution

Our hiring team now starts the assignment sending process by entering the slack command: /gdd_assignment_form. This command opens a form within the Slack UI, asking for the candidate’s GitHub username and the assignment they need to receive. After submitting this form, the Slackbot will create a private new repo within our new godatadriven-assessments GitHub organization and invite the candidate as a contributor to this repo. When the Slackbot is done, it will inform the person that submitted the form using a direct message. The only thing left to do for the hiring team is to send the repo link to the candidate.

How does it work?

We mainly use two Python libraries in our codebase, namely PyGitHub and Bolt for Python. PyGitHub makes it extremely easy to authenticate with GitHub and read/write information within a GitHub organization. We use this library to find all the template repos within our assessment GitHub organization and create new repos for our candidates. The other library, Bolt for Python, makes listening and reacting to any Slack event easy using its wide array of Python decorators. We use these decorators to listen and respond to the /gdd_assignment_form slack command and listen for the Slack UI form interactions. The Bolt for Python library also makes deploying your Slackbot on serverless deployment solutions extremely easy. Since the Slackbot only needs to be active during the assignment sending process, we decided to deploy it on a Google Cloud function. We chose this option because, for a Google Cloud function, you only pay for what you use, so in the end, our Slackbot deployment only costs us a few cents per month.

Benefits

We have been using this new Slackbot-based workflow for a couple of months now, and it has been a great success. This new workflow provides benefits for everybody involved in the workflow.

Candidates

For the candidate, the process has also become much smoother. They only need to click on the repo link they received from the hiring team. This link brings them to a page that tells them that the godatadriven-assessment-bot has invited them as a collaborator to a repo created explicitly for their assessment. After accepting this invitation, they will be brought to the readme page of the repo, which describes the project setup, the practicalities of the assessment, and the problem they have to solve. Once they are done, they have to submit their solution as a pull request. This submission method makes it easy to see which changes the candidate has made. It also makes it possible to run additional automated quality checks on the submission. These changes make it feel for a candidate like they contribute to a small open source project while also making it much easier for us to assess the candidate’s submitted solution.

Assessment makers

From the assessment-making process point of view, it has also become much easier to make changes to the assessment and sync them with the hiring team. When the hiring team interacts with the Slackbot, the bot provides an overview of all the template repos in the assessment organization. The hiring team only needs to pick one of the templates, and the Slackbot will take care of the rest. The advantage of these template repos is that they make it possible to quickly create a new repo that is unaware of the Git history of the original repo, which helps to prevent accidental solution leakage. So by moving all our assessments into private GitHub template repos, we can still easily change them using a typical Git workflow. At the same time, the Slackbot ensures that the hiring team is always using the latest version.
Creating new assessments has also become much easier. Our Slackbot provides our hiring team with an overview of all the template repos in the godatadriven-assessments organization. Thus, when we want to add a new assignment, we only need to add a new template repo to this organization. So with this new workflow, we basically have created a continuous deployment pipeline for our assessments.

GoDataDriven ❤️ Cool engineer solutions

Whenever someone encounters a problem at GoDataDriven, we always encourage them to create a cool solution and see it through to the end. This project was the first project I did at GoDataDriven. By seeing it through to the very end, I not only redesigned our assignment sending process within my first few weeks at GoDataDriven, but it also led to my first PyData talk, which was an incredible experience for me.

Do you recognize this drive to have an impact, and do you want colleagues who push and help to see your idea through to the very end? I have good news. We are hiring in Amsterdam, Eindhoven, and even in Zürich!