Should Developers Be Allowed to Use AI in Technical Interviews? [2024 Research]
Many developers are adopting AI tools to enhance productivity and code quality. This shift brings forth challenges in accurately evaluating technical skills.
As interviewers looking to assess which developers are the best fit for a given position, we have an important choice to make: Should we allow candidates to use AI during their live coding exercises?
We began by discussing the issue internally, and the consensus was that all things being equal, we would prefer to allow the use of AI at some level. The reasons for this are simple:
- We want to see how developers solve realistic problems in a realistic situation
- Developers already use many external tools while coding (such as search engines and code examples from the web)
- How a developer leverages tools such as AI is an important part of their skill set that will ultimately affect their productivity
- The technology landscape will continue to change, and a less rigid, rules-based approach to vetting will lead to more flexibility and adaptation
That said, we recognized the risk that allowing AI use could end up making it more difficult for us to accurately evaluate candidates, leading to a lower placement success rate.
To help us make an informed internal decision on our AI policy, we embarked on a research project. Our exploration centered on the potential benefits and challenges of allowing candidates to leverage AI tools during technical interviews. In this article, we’re sharing some of our core findings and how we’re refining our vetting process to remain robust and relevant.
Table Of Contents
A Note on AI and the Technical Vetting Process
Before moving into our research, we want to acknowledge that a broader debate exists in the AI and technical vetting sphere; namely, relying solely on AI to evaluate developers from a technical standpoint. We want to assert that our viewpoint is distinct: we advocate for human-led vetting, because only humans can supply the nuanced judgment and contextual analysis of a developer’s performance in an interview.
Overview of our Research Project and Findings
Our methodology entailed a two-pronged approach: a developer survey and live interviews. The core aim was to dissect the implications of AI tool adoption among developers during technical interviews and how this aligns with (or diverges from) industry expectations and client perspectives.
Part 1: Developer Survey
The survey targeted developers with varied programming backgrounds, soliciting their candid reflections on AI tool usage, perceived productivity gains, and integration into their workflows. We designed this survey to capture nuanced data beyond general statistics, striving to understand the qualitative aspect of AI integration in daily developer practices.
Part 2: Live Coding Interviews
Our talent team conducted an experiment that complemented the survey. We evaluated the actual performance of developers—spanning diverse proficiency levels in Node.js, React, and TypeScript—within a controlled setting. Here, we looked at differences between developers, and within developers. That is:
- How did developers using AI perform compared to developers who did not use AI in the technical exercise?
- For developers that used AI, how did they perform in parts of a technical interview where AI was not permitted?
In all, we conducted 55 technical interviews. On average, participants were intermediate-level programmers, with an average of 6.31 years of experience.
Results: What Our Research Showed About Developers & AI Usage
We broke down our results into key areas:
- developers’ thoughts, perceptions, and current experience around AI;
- the actual impact on performance with and without AI;
- correlation between seniority and AI usage; and
- developer perception on the use of AI in technical interviews.
Detailed Breakdown of Live Coding Interview Participants
| Seniority | Years Experience | %Total | %Used AI (Y/N) |
| Junior | 0-2 years | 23% | 91% |
| Intermediate | 3-5 years | 27% | 92% |
| 6-10 years | 38% | 89% | |
| Senior | 11-15 years | 6% | 100% |
| 16-20 years | 2% | 0% | |
| 20+ years | 4% | 100% |
Developer Survey
Our survey revealed nuanced attitudes toward the use of AI in technical interviews among developers. Overall, 65% of respondents supported the inclusion of AI tools in technical tests. However, this acceptance varied notably across different experience levels and familiarity with AI.
Developers with 3-5 years of experience were the most receptive, with 83% endorsing AI usage, compared to only 48% of those with 1-3 years of experience. Developers who had never used AI tools were the most resistant, suggesting that familiarity with these tools correlates with acceptance.
Regarding the structure of AI integration in interviews, opinions were divided. The most favored approach (40% of respondents) advocated for a hybrid model where AI tools could be used in certain parts of the interview but not others.
Additionally, a significant majority believe that developers should be evaluated on their proficiency in using AI tools effectively, suggesting an acknowledgment of these tools’ growing importance in the software development landscape.
Live Coding Interviews
Developer desire to use AI in interviews
Before the interview, we surveyed participants to understand their thoughts and experience with AI. The results here mirrored those seen in the talent experiment. Unsurprisingly, we found that most developers had used AI; however, The desire to leverage AI tools during interviews varied notably across different levels of programming and AI experience.
For instance, a substantial majority of junior developers (81.8%) expressed interest in using AI, a sentiment slightly less prevalent among semi-senior (77.4%) and notably lower among senior developers (50.0%).
There are a few interpretations of this variance. First, junior developers might be more comfortable with emerging technologies, and therefore more inclined or open to integrating AI into their problem-solving strategies. Second, it could reflect a perception that AI is a means to augment their capabilities.
Impact on Performance
The impact of AI on test outcomes was multifaceted. We explored this across two domains: test completion rate and performance, with performance having a more important weight in assessing whether a developer passes or fails.
Developers who used AI tools showed a marginally higher test completion rate (22.2%) compared to those who did not use AI (17.7%). The time taken to complete tests did not differ significantly between the two groups, with AI users completing tests just slightly quicker on average.
When we dug deeper into the results, it does appear that AI is more helpful for some tasks than others. Specifically, candidates using AI showed higher scores in:
- Code Structure: 4.25 vs. 3.83
- Optimization: 4.25 vs. 3.5
- Debugging: 4 vs. 3.5
These differences suggest that AI usage may aid in producing cleaner code, better optimization, and effective debugging.
While there was a slight difference in test completion rates and times between AI and non-AI groups, it wasn’t statistically significant.
Seniority, AI Experience, and Test Outcomes
We did observe a relationship between seniority and test performance, though. And for more senior developers, AI appeared to confer an advantage: senior developers that used AI scored higher than more junior developers (and marginally higher than their non-AI counterparts), indicating that senior developers were likely able to use AI more effectively.
To understand whether AI on its own provided an advantage, each participant had to complete a 15-minute segment without the use of AI tools. Here, seniority was the only factor that influenced performance.
However, the variance in AI’s impact based on seniority and AI experience was pronounced. Senior developers, despite being less inclined to use AI, demonstrated a higher completion rate for additional tasks when they did engage with AI tools.
How Scalable Path is Incorporating Findings Into Technical Interviews
1. Structuring Interviews Around Fundamental Skills
We continue to design technical assessments around fundamental coding skills and problem-solving abilities, independent of AI assistance. That way, we focus on what’s important: how they will perform in real-world situations. But now, developers have a choice to use AI or not.
2. Evaluating AI Proficiency if Developers Choose to Use It
If a developer chooses to use AI, our interviewers will assess a candidate’s AI proficiency as they go through the technical exercise.
3. Talent Team Composed of Senior Developers
Our findings underscore the need to prioritize human expertise to truly evaluate candidates. It’s only this way that we can gain deeper insights into a candidate’s problem-solving approach and creative thinking.
We’ve always had a team of skilled developers evaluating candidates. And this will continue to be the case.
4. Interviewer Training & Feedback Loop
We conduct ongoing training with our interviewers to enhance their understanding of AI tools, potential biases and limitations of AI, and evaluating developers in this new landscape. This ensures our evaluation criteria remains stringent yet fair, and that our interviewers are able to discern genuine skill from superficial proficiency.
At Scalable Path, this means:
- recognizing the nuances of AI-assisted solutions,
- distinguishing between adept use of AI and over-reliance on it, and
- assessing a candidate’s ability to integrate AI tools into their problem-solving process intelligently.
We remain committed to continually refining our vetting process based on ongoing feedback and emerging trends in AI.
Conclusion
We embarked on this research project to challenge our own assumptions, better understand the current technological landscape, and most importantly, refine our vetting process to be more modern and relevant. We still believe in the importance of a human perspective when it comes to evaluating a developer’s ability. And we also continue to put evaluation of the developer’s raw ability at the center of our vetting. But we believe it’s critical to understand how a developer will perform in the real world – and as of now, the “real world” means AI as a near-constant companion in the developer workplace.