notation.md

Problem Notation

Supervised Scenario

• Consider an input pattern observed with probability distribution and a ground-truth label observed with conditional probability distribution .
• Given a finite sample , where .
• Objective: estimate a predictive model that maps or learn statistics of , where .

Crowdsourcing scenario

• Same objective that supervised scenario, but the ground-truth labels corresponding to the input patterns are not directly observed.
• Consider labels that do not follow the ground-truth distribution . Instead, they are generated from an unknown process that represents the annotator ability to detect the ground truth.

Individual

• Consider multiple noise labels given by annotators.
• These annotations come from a subset of the set of all the annotators participating in the labelling process. ( )
• The annotator identity could be define as a input variable: , with
  • Then
• Given a sample

Global

• Consider that we do not known or do not care which annotators provided the labels: we know but not
• Consider the number of times that all the annotators gives each possible labels:
• Given a sample .

Focus

In this implementation, we study the pattern recognition case, that is, we let be a small set of K categories or classes .

---

One also can define two scenarios based on the annotation density and assumptions:

• Dense:
  • All the annotators labels each data:
  • The implementation is simpler since fixed size matrices are assumed.
• Sparse:
  • The number of labels collected by data point and annotator varies:
  • An appropiate implementation lead to computational efficiency.

---

Confusion Matrices

• Individual confusion matrix (for an annotator t):

• Global confusion matrix (for all the annotations):