Best Practices for Managing Data Annotation Projects
3
VII. What to Include in Annotation Guidelines
Summary ........................................................................................................................................................................ 17
Keep Complexity and Length in Mind ....................................................................................................................... 17
Include Both Tool and Annotation Instructions ......................................................................................................... 17
Illustrate Labels with Examples ................................................................................................................................... 18
Consider Including the End Goal ............................................................................................................................... 18
Maintain Consistency with Other Written Documentation ...................................................................................... 18
VIII. Creating, Testing and Maintaining Annotation Guidelines
Summary ........................................................................................................................................................................ 19
Designate a Single Person to “Own” the Guidelines ................................................................................................ 19
Employ a Limited Pilot Period to Test the Guidelines .............................................................................................. 19
Display the Guidelines in the Annotation Tool ........................................................................................................ 20
Communicate Changes to the Guidelines................................................................................................................. 21
IX. Staffing and Training the Workforce
Summary ....................................................................................................................................................................... 22
Use the Guidelines for Training ................................................................................................................................. 22
Encourage Real-Time Questions ............................................................................................................................... 23
Assess Quality During the Training Period ............................................................................................................... 23
Provide Written Feedback During the Training Period ........................................................................................... 25
X. Managing the Annotation Process
Summary ....................................................................................................................................................................... 26
Review the Annotation Tool and the Workforce ...................................................................................................... 26
Set Realistic Quality, Timeliness and Productivity Targets ...................................................................................... 26
Implement a Quality Assurance Process ................................................................................................................... 27
Explicitly Address Annotator Collaboration ............................................................................................................. 28
Provide the Workforce with Written Feedback for Erroneous Work Items ........................................................... 28
Promptly Respond to Questions from the Workforce ............................................................................................. 29
Track Quality, Timeliness, and Productivity Statistics .............................................................................................. 29
Ensure Technical Support for the Annotation Tool ................................................................................................. 30
