English
 
Help Privacy Policy Disclaimer
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT
  ImageNet suffers from dichotomous data difficulty

Meding, K., Schulze Buschoff, L., Geirhos, R., & Wichmann, F. (2021). ImageNet suffers from dichotomous data difficulty. In NeurIPS 2021 Workshop on ImageNet: past, present, and future (pp. 1-27).

Item is

Basic

show hide
Genre: Conference Paper

Files

show Files

Locators

show
hide
Locator:
https://openreview.net/pdf?id=-TMrjGZmnUC (Publisher version)
Description:
-

Creators

show
hide
 Creators:
Meding, K, Author
Schulze Buschoff, LM1, Author              
Geirhos, R, Author
Wichmann, FA, Author              
Affiliations:
1External Organizations, ou_persistent22              

Content

show
hide
Free keywords: -
 Abstract: "The power of a generalization system follows directly from its biases" (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems---but to what degree have we understood how their inductive bias influences model decisions? We here attempt to disentangle the various aspects that determine how a model decides. In particular, we ask: what makes one model decide differently from another? In a meticulously controlled setting, we find that (1.) irrespective of the network architecture or objective (e.g. self-supervised, semi-supervised, vision transformers, recurrent models) all models end up with a similar decision boundary. (2.) To understand these findings, we analysed model decisions on the ImageNet validation set from epoch to epoch and image by image. We find that the ImageNet validation set suffers from dichotomous data difficulty (DDD): For the range of investigated models and their accuracies, it is dominated by 46.3% "trivial" and 11.3% "impossible" images. Only 42.4% of the images are responsible for the differences between two models' decision boundaries. The impossible images are not driven by label errors. (3.) Finally, humans are highly accurate at predicting which images are "trivial" and "impossible" for CNNs (81.4%). Taken together, it appears that ImageNet suffers from dichotomous data difficulty. This implies that in future comparisons of brains, machines and behaviour, much may be gained from investigating the decisive role of images and the distribution of their difficulties.

Details

show
hide
Language(s):
 Dates: 2021-12
 Publication Status: Published online
 Pages: -
 Publishing info: -
 Table of Contents: -
 Rev. Type: -
 Identifiers: -
 Degree: -

Event

show
hide
Title: NeurIPS 2021 Workshop on ImageNet: past, present, and future
Place of Event: -
Start-/End Date: 2021-12-13 - 2021-12-14

Legal Case

show

Project information

show

Source 1

show
hide
Title: NeurIPS 2021 Workshop on ImageNet: past, present, and future
Source Genre: Proceedings
 Creator(s):
Affiliations:
Publ. Info: -
Pages: - Volume / Issue: - Sequence Number: - Start / End Page: 1 - 27 Identifier: -