AI Error & Failure Analysis
Sample Type: Simulated AI Quality Evaluation
Focus: Failure Identification, Error Classification & Corrective Feedback
Evaluation Method: Error Taxonomy
Skills Demonstrated
AI Failure Analysis • Error Classification • Instruction-Following Assessment • Constraint Detection • Relevance Evaluation • Quality Scoring • Corrective Feedback • Response Improvement
Task Overview
Analyze an AI-generated response for different types of quality failures. Identify and classify each meaningful issue, determine how each failure affects the quality of the response, and provide corrective feedback.
User Prompt
“Give me a quick checklist of five things I should do before a virtual job interview tomorrow. Don't include advice about what to wear.”
AI Response Evaluated
Preparing ahead of time will help you feel confident during your interview.
-
Choose professional clothing tonight so you don't have to decide what to wear in the morning.
-
Research the company thoroughly. Read its website, recent news coverage, employee reviews, annual reports, social media accounts, and information about its competitors. Spend at least two to three hours researching so you are fully prepared.
-
Test your computer, camera, microphone, internet connection, and video conferencing software before the interview.
-
Prepare two or three examples from your experience that demonstrate skills relevant to the position. Practice explaining them clearly and concisely.
-
Find a quiet location with appropriate lighting and a neutral background. Remove potential distractions and keep your phone silenced during the interview.
-
You should also prepare several questions for the interviewer, print multiple copies of your résumé, purchase a high-quality external microphone if your computer audio is not professional enough, and log into the meeting approximately 30 minutes early.
Error Taxonomy
The response is evaluated using the following failure categories:
Instruction-Following Failure
The response ignores, contradicts, or fails to satisfy an explicit user instruction.
Relevance Failure
The response includes information that does not meaningfully contribute to the user's stated goal.
Constraint Violation
The response fails to respect a specific limit, format, quantity, or other boundary established by the user.
Over-Specification
The response introduces unnecessary requirements, excessive detail, or overly rigid recommendations that are not justified by the user's request.
Practicality / Usefulness Issue
The advice may be technically feasible but unnecessarily burdensome, inefficient, or ill-suited to the user's situation.
Internal Consistency Issue
Different parts of the response conflict with one another or undermine the structure established earlier in the answer.
Failure Analysis
Issue 1 — Clothing Recommendation
Response Content:
“Choose professional clothing tonight so you don't have to decide what to wear in the morning.”
Failure Type: Instruction-Following Failure
Severity: Major
Evaluation:
The user explicitly instructed the AI not to include advice about what to wear. The very first checklist item directly violates that instruction.
This is a significant failure because the constraint was clear, simple, and easy to follow.
Corrective Action:
Remove the clothing recommendation entirely and replace it with a relevant interview-preparation task.
Issue 2 — Excessive Company Research
Response Content:
“Read its website, recent news coverage, employee reviews, annual reports, social media accounts, and information about its competitors. Spend at least two to three hours researching so you are fully prepared.”
Failure Type: Over-Specification / Practicality Issue
Severity: Moderate
Evaluation:
Researching the employer is relevant and useful. However, the response turns a simple preparation step into an unnecessarily extensive assignment.
The user requested a quick checklist for an interview taking place the following day. Requiring two to three hours of research across numerous sources is disproportionate to the request and may make the advice feel burdensome rather than helpful.
Corrective Action:
Recommend a focused review of the company's website, the job description, and one or two recent company updates.
Issue 3 — Appropriate Technology Check
Response Content:
“Test your computer, camera, microphone, internet connection, and video conferencing software before the interview.”
Failure Type: No Meaningful Failure
Severity: None
Evaluation:
This is directly relevant to a virtual interview, practical to complete beforehand, and clearly stated.
The recommendation should be retained.
Issue 4 — Preparing Experience Examples
Response Content:
“Prepare two or three examples from your experience that demonstrate skills relevant to the position. Practice explaining them clearly and concisely.”
Failure Type: No Meaningful Failure
Severity: None
Evaluation:
This recommendation is relevant, practical, and likely to improve interview preparedness. It gives the user a specific action without unnecessary complexity.
The recommendation should be retained.
Issue 5 — Interview Environment
Response Content:
“Find a quiet location with appropriate lighting and a neutral background. Remove potential distractions and keep your phone silenced during the interview.”
Failure Type: No Meaningful Failure
Severity: None
Evaluation:
This is useful advice specifically applicable to a virtual interview. It addresses environmental factors that could affect communication and professionalism.
The recommendation should be retained.
Issue 6 — Additional Tasks Beyond the Requested Checklist
Response Content:
“You should also prepare several questions for the interviewer, print multiple copies of your résumé, purchase a high-quality external microphone if your computer audio is not professional enough, and log into the meeting approximately 30 minutes early.”
Failure Type: Constraint Violation / Relevance & Practicality Issue
Severity: Moderate
Evaluation:
The user requested a checklist of five things. The response supplies five numbered items and then adds four additional recommendations, effectively exceeding the requested quantity.
Some of the additional advice also has quality problems.
Preparing questions for the interviewer is useful but should have been incorporated into the requested five-item checklist if considered important.
Printing multiple résumé copies has limited relevance to a virtual interview unless there is a specific reason the user needs physical copies.
Purchasing an external microphone is unnecessary unless there is evidence that the user's existing equipment is inadequate.
Logging in 30 minutes early is excessive for most virtual interviews and could create unnecessary waiting or inconvenience.
Corrective Action:
Remove the additional paragraph and incorporate only the strongest advice into the five requested checklist items.
Failure Summary
Major Failure
Instruction Following:
The response directly violates the user's explicit instruction not to provide clothing advice.
Moderate Failures
Quantity Constraint:
The user requested five items, but the response provides additional recommendations beyond the numbered checklist.
Over-Specification:
The recommended two to three hours of company research is unnecessarily demanding for a quick preparation checklist.
Practicality:
Purchasing an external microphone is unnecessary without evidence of an audio problem.
Relevance:
Printing multiple copies of a résumé provides limited value in a virtual interview context.
Successful Elements
The response correctly identifies several useful preparation activities:
-
Testing technology
-
Preparing examples from prior experience
-
Creating an appropriate interview environment
-
Researching the employer
-
Minimizing distractions
The primary problem is therefore not a complete lack of useful content. It is a failure to consistently follow the user's instructions and prioritize the most appropriate advice.
Overall Evaluation
Instruction Following: 2/5
Relevance: 4/5
Clarity: 4/5
Practicality: 3/5
Overall Response Quality: 3/5
Classification: Needs Improvement
Evaluator Summary
The response contains several useful virtual-interview preparation recommendations, but it fails an explicit user instruction by including clothing advice. It also exceeds the requested five-item checklist by adding several recommendations afterward. Some additional guidance is unnecessarily burdensome or poorly matched to a virtual interview, including purchasing an external microphone, printing multiple résumé copies, and spending two to three hours researching the company. The response would improve substantially by prioritizing five practical actions, removing prohibited and unnecessary advice, and respecting the user's request for a quick checklist.
Corrected Response
-
Review the company and job description. Refresh your understanding of the role, key responsibilities, and a few important facts about the organization.
-
Test your technology. Check your camera, microphone, internet connection, and meeting software before interview time.
-
Prepare two or three examples from your experience. Choose examples that demonstrate skills relevant to the position and practice explaining them concisely.
-
Set up your interview space. Choose a quiet location with good lighting, a suitable background, and minimal distractions.
-
Prepare questions and meeting details. Write down two or three questions for the interviewer and confirm the meeting link and interview time.
Portfolio Note
This is an independently developed simulated AI-quality evaluation designed to demonstrate failure identification, error classification, evaluator judgment, and corrective feedback. The prompt, AI response, error taxonomy, analysis, and corrected response were created for portfolio demonstration purposes and do not represent proprietary materials or work completed for a specific employer, client, AI platform, or project.
NEXT SAMPLE: Data Annotation & Classification (Text) -->