Coding assessments in the age of AI: what still measures skill
Updated 27 October 2026 · 7 min read
Short answer
Since AI assistants can answer most multiple-choice questions and many coding problems, a coding assessment now needs a step the candidate must own: questions about their own answer. Keep the quiz short as a filter, grade code on hidden tests that never reach the browser, report large pastes to the recruiter, and ask follow-up questions on the candidate's written reasoning. In the EU, AI used to evaluate candidates is classified as high-risk, so keep a human decision and tell candidates what is assessed.
What AI assistants changed
A multiple-choice question can be pasted into an AI assistant in seconds. Short, well-specified coding problems can often be solved the same way. Tests built on these formats alone now measure, in part, access to a tool rather than skill.
That does not make them useless: a short timed quiz still filters out candidates who lack the basics. But it can no longer be the main signal.
Formats compared
| Format | What it measures | Weakness with AI |
|---|---|---|
| Multiple-choice quiz | Knowledge of a technology's basics | Easy to answer with an assistant |
| Coding problem with tests | Ability to produce a program that passes tests | Code can be generated; needs hidden tests and signals on how it was written |
| Written case study | Reasoning, architecture choices, priorities | Slower to grade |
| Follow-up questions | Understanding of one's own answer | Only works after a written answer |
Limiting cheating without surveillance
- Server-side timers: a browser timer can be paused or bypassed.
- Hidden tests: grade code on tests that never reach the browser, not only on the visible examples.
- Paste signals: count large pastes and show them to the recruiter, and tell the candidate before they start.
- Follow-up questions: ask about what the candidate wrote, under a time limit.
Webcam proctoring and screen recording are the usual answer. They are intrusive, easy to get around with a second device, and they put off the candidates you most want. Hirlya uses none of them: neither the screen nor the camera is recorded.
A format that holds up
- Technical quiz (about 20 min) as a filter, balanced by level.
- Case study (about 30 min): a real situation from the job. See our case study interview guide.
- Follow-up questions (15 min): three questions about the candidate's own answer.
- Coding exercise (30 to 45 min, optional): one or two problems in a business context, graded on hidden tests.
- Interview: start from the answers and the signals collected.
What EU law says about AI in hiring
The EU Artificial Intelligence Act (Regulation (EU) 2024/1689) lists in Annex III, as high-risk, AI systems intended to be used for the recruitment or selection of people, in particular to analyse and filter job applications and to evaluate candidates. In practice: tell candidates what is assessed and how, keep a human who decides, and be able to explain a score.
In Hirlya, AI pre-assesses written answers and gives its reasons; a recruiter reviews every result and makes the decision.
Questions
Can you stop candidates from using ChatGPT in a coding test?
Not reliably. It is more effective to keep tests hidden on the server, report large pastes, and ask follow-up questions about the candidate's own answer.
Is AI-based candidate evaluation allowed in the EU?
Yes, but the AI Act classifies AI systems used to evaluate candidates as high-risk (Annex III), which comes with obligations such as human oversight and transparency.