A few days ago I watched How Computers Learn talk by Peter Norvig. In this talk, Peter talked about how Google did machine learning and at one point he mentioned that at Google they also applied machine learning to hiring. He said that one thing that was surprising to him was that being a winner at programming contests was a negative factor for performing well on the job. Peter added that programming contest winners are used to cranking solutions out fast and that you performed better at the job if you were more reflective and went slowly and made sure things were right.

Watch the relevant video fragment from the lecture:

Peter Norvig says that being good at programming competitions correlates negatively with being good on the job at Google.
Video URL: https://www.youtube.com/watch?v=DdmyUZCl75s.

You can watch the full lecture here:

How Computers Learn - Vienna Gödel Lecture 2015 by Peter Norvig.
Video URL: https://www.youtube.com/watch?v=T1O3ikmTEdA.

I extracted the fragment from the QA session at 1h 11m 50s.


Simon Permalink
April 05, 2015, 08:43

Doesn't come as much of a surprise, honestly. Programming contests teach you about working under time pressure, which is good, but they encourage short-term thinking... you do whatever it takes to solve your problem, because you don't need to worry about the code debt you're building up, the ongoing cost of maintaining the horrible hack you've put in place.

It's fine to go for the short-term hack when you're being called at 3am and that'll be enough to get your client's business running again. But at the same time, you've got to be thinking about how you fix it properly - and that's the bit that contests effectively discourage.

rfk Permalink
April 05, 2015, 16:44

Someone should sponsor a programming contest where, for full points, you have to go back and modify your code 6 months later to meet new requirements.

April 05, 2015, 18:01

I love the idea :) Let's start a crowdfunding campaign? :)

@pkrumis thanks for the extract - now I have a legitimate reason not to participate in these :)

Robert Zeurunkl Permalink
April 06, 2015, 00:11

Or, a modified version that doesn't require the six month wait. One half of the team has to write the code, and then immediately the other half of the team has to implement the changes.

Joe Morris Permalink
April 07, 2015, 12:18

And to make the contest more realistic, there should be no communication between the first half of the team and the second half except in the code and machine-readable documentation, all of which must be delivered at the changeover point (i.e., no consultation). And let's say that the first half of the team must implement a set of original specs and the second half implement a set of change orders (all teams have the same specs and change orders but the first half doesn't know what's in the change orders) - and there is a total time limit, but each team can determine when to switch.

Oh yes...how about points off for poor (or non-existent) end-user documentation?

Shed_Dweller Permalink
April 13, 2015, 07:29

And the programs is judged at change over and at full time against their respective requirements. Each half of the team gaining up to half of the available points.
So the first team can't just create boilerplate stubs for the second team and then hand off.

Andrew Permalink
April 06, 2015, 01:25

As someone who has hired their fair share of programmers, I think I can understand why. The kind of people I have known who like to frequent these kinds of competition’s are hard to work with. Their egos are out of control, and they are often douchebags. They are what I call a team grenade because they blow everyone apart. They have zero interest in building up the tea, company, and product, and just want to be known as the smartest. And if they don’t know anything, they will never admit it.

DMan Permalink
April 06, 2015, 02:39

You certainly make unfair generalizations. Sure, there seems to be a correlation, but I wouldn't go that far and judge *all* those qualities.

Ben Permalink
April 10, 2015, 08:24

Building up the tea is very important in British offices (especially smaller companies). No-one likes a colleague that doesn't do their fair share of tea making.

April 06, 2015, 03:36

Quote from video:

I think what it meant was that everybody who gets hired at Google is pretty good. So if I just had to pick someone off the street, I really want that contest winner, I got to take him every time, or her. But if it's somebody who passed the hiring bar, then they're all pretty much at the same level. And maybe the ones used to winning the contest, they're really used to going really, really fast and cranking the answer out and then moving on to the next thing and you perform better on the job if you're a little more reflective and go slowly and make sure you get things right.

This sounds the same as how height isn't positively correlated with "productivity" among NBA basketball players, except that no one writes linkbait-y articles titled "Being tall isn't correlated with being good at basketball".

Being taller is pretty obviously correlated with being good at basketball. But if being taller were correlated with being a better NBA player it wouldn't actually be a sign that tall people are better at basketball. It would be a sign that basketball coaches are making mistakes in drafting and that, at the margin, they should favor selecting slightly taller players with slightly worse skills. The exact same logic would apply if height were negatively correlated with NBA performance, but with the sign flipped.

The information that performance at Google is negatively correlated to programming contest performance only tells you that, at the margin, Google should weight programming contests less positively. This says basically nothing for companies that aren't literally copying Google's entire hiring process, including having access to the same applicants.

Miroslav Balaz Permalink
April 09, 2015, 08:37

What if the reasoning behind is that people who tend to risk have higher chance to win programming competition at least once, but in long term they might do more mistakes(therfore worse performance on job). So if you take sample from good competition programmers, the ones that have one at least once (with exception to superstars) are the ones that are willing to take more risk.

In that case it would be not dependent on the sample being from google employees.

I thing the study was not targeted for association between competition programming skills and job performance, so the correlation measured most probably have low predictability value.

Mikhail Goncharov Permalink
April 06, 2015, 07:16

I think that data is only tells that it's easier to people with contest background to be hired. So the hiring process is bit biased and one should be quite better (at dev. scale) to pass an interview w/o contest background (and Peter told about that "I got to take [winner] every time... but if...").
I am sure if one take overall dev. population (s)he will see that there is a _positive_ correlation between "being good developer" and "winner" (and that will suggest to make it a part of hiring process ;) )

April 06, 2015, 15:09

That's surprising but surely there is some element of truth behind it. I think main reason of this is programming contests are mostly far from the work you do at real work, but you can't say to all programming contests. There are some of them at TopCoder which are more or less present the challenge you face in real world.

April 07, 2015, 17:36

Speaking as someone who has done pretty well at programming contests in the past, this actually seems somewhat plausible.

I think that programming contests can be good, but only up to a point. At lower levels, practicing for programming contests is a good way to build up your coding efficiency and basic algorithmic knowledge. However as you go farther along, it becomes more of a self serving game. Practicing for things at very elite levels has little to do with the way you would actually program things in the real world.

There are also huge chunks of computer science that you almost never see in programming competitions. Due to their complexity you hardly ever see any problems on more advanced (and practical!) topics like geometry, AI, operating systems, computer graphics, networks, databases or data structures. It is very easy if you are studying for competitions like ICPC or TopCoder to get too hyper focused on silly "puzzle" questions which have cute and easily implementable solutions. Doing this to an extreme is pathological, and leaves you with a toxic combination of gaping holes in your knowledge plus the overweening self confidence that comes with all the praise from such competitions. The result is that you can end up with a degraded sense of empathy and warped perception of reality.

I think it was around my second year in graduate school I realized that programming competitions were no longer a useful way for me to spend my time and so I stopped having anything to do with. I still don't regret the time I spent on them as an undergraduate, but I do wish that I had reached this enlightenment sooner.

Mary Wolken Permalink
December 17, 2015, 11:03

What is the difference between a competitive coding test and a coding test used for hiring programmers, such as the ones from TestDome.com? The idea seems somewhat similar, yet companies massively use these coding tests and they are generally a good first step in the selection process. The phrase here is "first step." I'd think that any coding test is a good indicator of skill level, but that's just one part of the screening process. Hiring someone just because they won a competition without any other screening is the problem here, I'd think.

Leave a new comment

(why do I need your e-mail?)

(Your twitter name, if you have one. (I'm @pkrumins, btw.))

Type the word "rocket_460": (just to make sure you're a human)

Please preview the comment before submitting to make sure it's OK.