The aggregation feature for bounding boxes enables generating an average high-confidence result from multiple judgments. The origin (0,0) of each bounding box will be the top left corner. The results can be seen in the “annotation” column of the aggregated report.
Fig. 1: Aggregated report of a bounding box job
The aggregated box is available in the Aggregate Report on the Results page.
Aggregation consists of two steps.
- This step involves identifying the set of boxes from different judgments that potentially represent the same object.
- This is accomplished by clustering boxes based on the degree of overlap computed by the ratio of intersection over union (IoU).
- Each cluster has a maximum of one box from each contributor that worked on the image.
- The degree of overlap is configurable for each job.
- This step computes the union of intersections from the component boxes in each cluster.
- By default, any sections that do not overlap with at least one other box are discarded. This can be changed using report options to increase recall.
- An aggregate bounding box is generated from the sections that are part of at least two component boxes.
- Boxes that don’t have sufficient overlap with any cluster are ignored during aggregation.
Note: Aggregation cannot be applied to jobs with one judgement per unit.
The default setting for aggregation is “all”. Bounding boxes aggregation can be enabled by in the following ways.
- Results Options:
Select Box Aggregation (‘bagg_x’) in the Aggregation section for the annotation response. Degree of overlap can be specified as a number between 0 to 1, 0 represents no overlap, and 1 represents 100% overlap between boxes.
By default, boxes from a contributor that did not match another box from other contributors are discarded. To improve recall by including such unclustered boxes, check the “Include low confidence boxes” checkbox.
2. Job Design: This value can also be set within CML using the aggregation attribute on a CML element.
Applicable to cml:boundingbox
Returns aggregated bounding boxes from constituent contributor responses with degree of overlap greater than the specified value for x. x can be any floating point number between 0 & 1. Always prefix '0' for decimals; for example, bagg_0.4 will return all responses with overlap greater than 40%.
A confidence score is assigned to each box based on the measure of agreement.
- Agreement is computed as the ratio of intersection over union of the component boxes (Note: Unlike other aggregation methods, confidence for bounding boxes aggregation is not weighted by trust).
- In addition to each box, each row has a confidence score that is the average of confidence of all aggregate boxes in the row
- The confidence score ranges between 0 and 1.
- A value closer to 1 represents higher confidence.