Interview Signals: Can They Code?

This article collects a bunch of my thoughts over the years around interview signals. I’ve been a part of many hiring committees; in fact, I’ve been involved in hiring for almost every job I’ve ever had. I’ve also played the role of hiring manager numerous times for large companies and for my own endeavors. One thing that has always gnawed at me when discussing a candidate’s performance is the myriad of vague declarations made by interviewers: “The candidate can’t code” or “She didn’t communicate well” or even “I just didn’t get enough positive signal from the candidate” (Signal for what?)

Writing this article is not timely. It’s amazing to me that the world has changed so much since LLMs have become entrenched in tech. Hiring has not been spared and many of us hear tales of hiring processes that feel alien and would be unheard of just one or two years ago. Still, interviews will always be about discovering signal from a candidate; it’s only what the signals are that has changed. I argue that fluency in coding is still a very valuable signal today, at least for now.

Signals

I find signals in interviews the most ill defined aspects of hiring. Just what exactly are we testing the candidate on anyway? Is it how to program? How to design a system? The ability to explain oneself? The ability to lead? I’ve heard all of these as wants from hiring managers (including myself,) but these are vague and abstract concepts subject to personal bias. We have to break these concepts down if we ever want to have a shared idea of what they mean with our teams.

You can’t talk about signals without noise. For interviews, this would be anything that doesn’t contribute to what you want to evaluate. Different questions are better at capturing specific signals or can introduce noise (e.g. a question can require too much context to understand). We should obviously strive to reduce noise and this can often be done by carefully choosing questions.

Let’s consider tic tac toe. The game is ubiquitous enough that everybody understands how to play it. If a candidate insists they don’t know the game, they very likely do and a quick game or two will clarify the rules for them. The fact that the game is ubiquitous removes a source of false negatives from the interview process since the candidate can start off with some previous knowledge and confidence instead of anxiously reading a wall of text explaining a contrived problem. The choice of Tic Tac Toe doesn’t matter in particular. The game is just a normalized setting that allows crafting problems that gather different signals from the candidate. I find that questions around games tend to have that normalizing quality to them.

Can they code?

In the context of a software engineering role, whether a candidate can code is one of the most important signals to uncover and should be prioritized early in the process. It is expensive to run a candidate through the rigmarole of an entire interview process for both the candidate and the engineering team. Team members have to take time away from other priorities to take part in the process, departments may have to pay to fly and provide accommodations for candidates, and candidates have to take time from their own lives to participate. This is why it is imperative to design a process where this signal is established first - it’s a good reason why phone screens exist.

Let’s break down testing if a candidate can code using tic tac toe:

This problem is indeed very basic, but it tests one of the most important tenets of programming - iteration. If you learned programming around other people (such as a classroom environment,) you may remember that iteration is one of the first topics where some people have trouble getting the concept of programming. Beforehand, it’s all just simple variable assignments and control flow. Once you can repetitively do something your programming universe becomes much larger.

 1# consider a board to be an array of arrays containing strings
 2# e.g. game_board = [
 3#     ["x", "o", "x"],
 4#     ["o", "o", "x"],
 5#     ["o", "x", "x"],
 6# ]
 7
 8
 9def display1(board):
10    for row in board:
11        print(" ".join(row))
12
13
14def display2(board):
15    # index based iteration
16    for i in range(len(board)):
17        s = ""
18        for j in range(len(board[i])):
19            s += board[i][j] + " "
20        print(s)
21
22
23def display3(board):
24    for row in range(len(board)):
25        print(board[row])

This is one of the most fundamentals signals you can uncover about a candidate. It’s a signal they have cleared one of the very first “difficult” things to learn about programming. If a candidate fails at this, then it provides an early exit ramp from the interview process.

While the first problem can demonstrate that the candidate can iterate over data structures or other simple usages of loops, it does not demonstrate the ability to wield iteration as a tool to solve problems. This happens to be another skill that is developed in the early phases of learning to build software. A student begins to see problems as nails that can be solved using their shiny new hammer (iteration).

 1def hasWon(player, board):
 2
 3    # horizontal win
 4    row_count = 0
 5    for i in range(len(board)):
 6        for j in range(len(board[i])):
 7            if board[i][j] == player:
 8                row_count += 1
 9        if row_count == len(board):
10            return True
11        row_count = 0
12
13    # vertical win
14    col_count = 0
15    for i in range(len(board)):
16        for j in range(len(board[i])):
17            if board[j][i] == player:
18                col_count += 1
19        if col_count == len(board):
20            return True
21        col_count = 0
22
23    # diagonal win - left to right
24    diag_count = 0
25    for i in range(len(board)):
26        if board[i][i] == player:
27            diag_count += 1
28    if diag_count == len(board):
29        return True
30
31    # diagonal win - right to left
32    rev_diag_count = 0
33    for i in range(len(board)):
34        row = i
35        col = len(board) - i - 1
36        if board[row][col] == player:
37            rev_diag_count += 1
38    if rev_diag_count == len(board):
39        return True
40
41    # checked all win conditions and didn't win
42    return False

The solution above is written so that it separates the constituent win condition checks to illustrate the different sub-problems in this question. The question itself is not “tricky” (that is never the goal by the way,) but you can see even visually that there is a heightened level of complexity and higher level of competence with iteration and problem solving that is required.

You may be able to see how the candidate generalizes throughout their solution to the above problem. For example, the candidate may decide that tic tac toe is always played on a 3x3 grid and write:

1def hasWon(player, board):
2    # horizontal win
3    row1_win = player == board[0][0] and player == board[0][1] and player == board[0][2]
4    # ... and so forth

This is not technically wrong. The interviewer needs to steer the candidate towards a more generalized solution during the course of the session. “What if the board is bigger than 3x3?”, “Can you use loops instead? I would like to see how you do with iteration.” and so forth.

To directly test for generalizing, we always need a problem that can be very quickly expanded without adding so much complexity that the candidate is starting miles away from an acceptable solution (e.g. don’t switch from tic tac toe to chess - this throws away the current mental frame and any built up confidence). For Tic Tac Toe in particular, there are many ways to expand the problem. The most trivial one is to extend the board size to NxN and change the game to connect N.

Note that the sample solution to the Signal 3 example already generalizes to any NxN board by virtue of checking counts against the size of the board instead of a hard coded value (3). It’s always a pretty good sign when a candidate writes solutions that generalize by default. When this happens, it can be a sign that the candidate has a sense of writing things that scale.

A very common issue I have observed over the years is that students (and consequently candidates) will mistake run-time complexity with the shape of a solution. For example, a candidate might see two nested loops and immediately jump to the conclusion that run-time complexity is quadratic. It’s not hard to see why people would develop such heuristics, but it’s a deep misunderstanding. A candidate is generally in good shape if they can tell you how parts of their code affect the total run-time complexity and why. We’re generally looking for an explanation like:

If we have a board of size M=N^2, my solution traverses the board entirely for checking row and column wins contributing O(N^2) for each. Checking for diagonals does not require traversing the entire board and only contributes O(N) for each win condition. So the total worst case run-time is in the order of O(N^2) or O(M) - however we want to express that.

Similar things could be said about candidate’s understanding space complexity. An unsurprising observation I’ve made is that candidates who excel at run-time analysis will also excel at analyzing space complexity.

Other than the game board, my hasWon implementation only uses a few fixed variables to keep track of the player and counters to keep track of winning. We can represent them as constant space of O(1).

Nobody writes perfect code, especially not under duress. It is almost a universal guarantee that the candidate will write flawed code. A candidate should be able to walk through their code and identify and correct the problem (using test cases, printing to the console, whatever). Some of the best candidates I’ve seen catch their own mistakes before the interviewer has to explain what went wrong as well as the fix. I tend to consider it a good signal when the candidate forms a theory of what the error actually is before doing anything - it shows a structured approach as opposed to guessing.

Testing can also yield some valuable information, although I wouldn’t expect a candidate to build a formal test suite during an interview. Special attention should be paid to the test cases that candidates rely on. Do they only horizontal and vertical wins but overlook diagonals? Do they test a board without a winner? Exhaustive coverage should not be the expectation here but candidates should show some knowledge of what constitutes something worth testing.

For any question that relates to games, you can usually extend the question in any direction you like. It’s important not go overboard trying to measure completely useless signals in these cases. One of the reasons I’ve seen interviewers go really far beyond their initial ask is that they didn’t bring a question appropriate enough to fill the time for the interview. I suppose the candidate could be incredibly brilliant (or an AI) and finish exceedingly fast, but I think this is usually a symptom of not planning. In any case, we should ask what we need for whatever signal we are trying to get but not too far into strange territory.

That’s it. That’s a simple way to determine if somebody knows how to code using Tic Tac Toe. There are many things that this exercise does not test (e.g. recursion,) but it’s intention is not support every conclusion we might need to draw about a candidate. It’s designed to answer if the candidate can code and whether you should bring them in for the rest of the interview process. It does this by breaking the core question down into several smaller, but more observable signals. The same approach can be built around many different questions - kudos if you use questions about games.