Data Annotation & Classification
Sample Type: Simulated Text Annotation Task
Focus: Intent Classification, Label Accuracy & Guideline Adherence
Evaluation Method: Single-Label Intent Classification
Skills Demonstrated
Data Annotation • Text Classification • Intent Recognition • Taxonomy Application • Guideline Adherence • Ambiguity Resolution • Quality Control • Contextual Judgment
Task Overview
Review a set of customer-support messages and assign each message to the most appropriate intent category using a predefined annotation taxonomy.
The objective is to demonstrate consistent label application, contextual interpretation, and appropriate handling of messages that could reasonably fit more than one category.
Annotation Guidelines
Assign one primary label to each message based on the user's main intent.
Available Labels
ORDER_STATUS
The customer wants information about the current location, progress, or expected arrival of an existing order.
CANCELLATION
The customer wants to cancel an order that has already been placed.
RETURN_REFUND
The customer wants to return an item, receive a refund, or asks about return/refund eligibility.
PRODUCT_INFORMATION
The customer is requesting information about a product before or independently of making a purchase.
ACCOUNT_SUPPORT
The customer needs assistance accessing, updating, or managing an account.
PAYMENT_BILLING
The customer's primary issue involves payment, charges, billing, or a payment method.
OTHER
The message does not reasonably fit any of the available categories.
Annotation Results
Item 1
Customer Message:
“Can you tell me when my package is supposed to arrive? The tracking hasn't updated since Monday.”
Assigned Label: ORDER_STATUS
Confidence: High
Rationale:
The customer's primary intent is to obtain information about the delivery status of an existing order. The tracking issue provides context but does not change the central intent.
Item 2
Customer Message:
“I ordered the wrong size about an hour ago. Can you stop the order before it ships?”
Assigned Label: CANCELLATION
Confidence: High
Rationale:
Although the customer mentions ordering the wrong size, the requested action is to stop an existing order before shipment. Cancellation is therefore the most appropriate primary label.
Item 3
Customer Message:
“I received the sweater yesterday, but the sleeves are too short. How do I send it back and get my money back?”
Assigned Label: RETURN_REFUND
Confidence: High
Rationale:
The customer explicitly wants to return the product they received and obtain a refund. Both actions fall within the same defined category.
Item 4
Customer Message:
“Is this backpack waterproof, and will a 15-inch laptop fit inside?”
Assigned Label: PRODUCT_INFORMATION
Confidence: High
Rationale:
The customer is requesting product specifications and compatibility information. There is no indication of an existing order or post-purchase issue.
Item 5
Customer Message:
“I changed my email address, and now I can't log in. How can I update the email on my account?”
Assigned Label: ACCOUNT_SUPPORT
Confidence: High
Rationale:
The primary issue concerns account access and the ability to update account information. No order, payment, or product issue is involved.
Item 6
Customer Message:
“My card was declined, but I can see a pending charge from your company. Was I actually charged?”
Assigned Label: PAYMENT_BILLING
Confidence: High
Rationale:
The central concern is whether a payment was processed after the card was declined. The message directly concerns a charge and payment status.
Item 7
Customer Message:
“My order hasn't shipped yet, and I've changed my mind. I don't want it anymore.”
Assigned Label: CANCELLATION
Confidence: High
Rationale:
This message contains information about shipping status, which may initially appear as ORDER_STATUS. However, the customer's actual requested outcome is to stop the order because it is no longer wanted. Under the guideline to classify by primary intent, CANCELLATION is the stronger label.
Item 8
Customer Message:
“The shoes arrived damaged. Can I get another pair instead of a refund?”
Assigned Label: RETURN_REFUND
Confidence: Medium
Rationale:
The customer is requesting a replacement rather than a refund. However, the taxonomy does not provide a separate exchange or replacement label. Because the issue concerns resolution after receiving an unacceptable product, RETURN_REFUND is the closest available category.
This item demonstrates the need to apply the existing taxonomy rather than invent an unsupported label.
Item 9
Customer Message:
“I forgot my password, and the reset email never showed up.”
Assigned Label: ACCOUNT_SUPPORT
Confidence: High
Rationale:
The customer cannot access the account and needs assistance with password recovery. This directly matches the account-support category.
Item 10
Customer Message:
“Do you have any stores in San Diego where I can see your products in person?”
Assigned Label: OTHER
Confidence: Medium
Rationale:
The customer is asking about retail locations rather than requesting information about a particular product. Because the taxonomy does not include a store-location or general-business-information category, OTHER is the most appropriate available label.
Quality Review
Total Items: 10
Labels Applied: 7
Ambiguous / Boundary Cases Identified: 2
Primary-Intent Rule Applied: Yes
Unsupported Labels Created: No
Boundary Cases
Item 7 — ORDER_STATUS vs. CANCELLATION
The mention of an unshipped order is secondary context. The desired action—stopping the order—determines the primary classification.
Item 8 — RETURN_REFUND vs. Missing Replacement Category
The user requests a replacement, but no replacement or exchange label exists. The closest available category is used rather than creating a new label outside the taxonomy.
Overall Evaluation
Annotation Quality: 5/5
Classification: Consistent Guideline Application
Evaluator Summary
The dataset was classified according to the defined taxonomy and primary-intent rule. Straightforward messages were assigned directly to the corresponding categories, while ambiguous cases were evaluated based on the customer's desired outcome rather than on isolated keywords. Items involving cancellation and replacement demonstrate the importance of contextual judgment when categories overlap or the taxonomy does not contain an exact label. No unsupported categories were introduced, preserving consistency with the provided annotation guidelines.
Portfolio Note
This is an independently created simulated data annotation and classification sample designed to demonstrate structured labeling, taxonomy application, and evaluator judgment. The dataset, taxonomy, customer messages, labels, and evaluation were created for portfolio demonstration purposes and do not represent proprietary data or work completed for a specific employer, client, AI platform, or project.
NEXT SAMPLE: Guideline-Based AI Response Evaluation -->