The 20BN-something-something Dataset


Introduction

The 20BN-SOMETHING-SOMETHING dataset is a large collection of densely-labeled video clips that show humans performing pre-defined basic actions with everyday objects. The dataset was created by a large number of crowd workers. It allows machine learning models to develop fine-grained understanding of basic actions that occur in the physical world. It is available free of charge for academic research. Commercial licenses are available upon request.

A paper with supplementary material can be found here.

Poking a stack of cans so the stack collapses
Plugging cable into charger
Closing dishwasher

Data format

The video data is provided as one large TGZ archive, split into parts of 1 GB max. The total download size is 25.2 GB. The archive contains directories numbered from 1 to 108499. Each directory corresponds to one video and contains JPG images with height 100px and variable width. The JPG images were extracted from the orginal videos at 12 frames per seconds. The filenames of the JPGs start at 00001.jpg. The number of JPGs varies as the length of the original videos varies.

Terms of use

This dataset is made availble under the Creative Commons Attribution 4.0 International license CC BY-NC-ND 4.0. It can be used for academic research free of charge. If you seek to use the data for commercial purposes please contact us.

Download Dataset

Please register or log in to download the dataset.


20BN-SOMETHING-SOMETHING-DATASET
Total number of videos
108,499
Training Set
86,017
Validation Set
11,522
Test Set (w/o labels)
10,960
Labels
174
986
Holding something
979
Turning something upside down
924
Turning the camera left while filming something
914
Turning the camera right while filming something
914
Stacking number of something
888
Opening something
885
Approaching something with your camera
877
Picking something up
873
Pushing something so that it almost falls off but doesn't
864
Folding something
863
Moving something away from the camera
858
Closing something
850
Moving away from something with your camera
845
Turning the camera downwards while filming something
841
Pushing something so that it slightly moves
839
Turning the camera upwards while filming something
838
Pretending to pick something up
838
Showing something to the camera
833
Moving something up
830
Plugging something into something
830
Unfolding something
828
Putting something onto something
827
Showing that something is empty
825
Taking something from somewhere
825
Pretending to put something on a surface
824
Putting something next to something
821
Moving something towards the camera
820
Showing a photo of something to the camera
815
Pushing something with something
808
Throwing something
802
Pushing something from left to right
801
Something falling like a feather or paper
801
Throwing something in the air and letting it fall
796
Throwing something against something
793
Lifting something with something on it
788
Taking one of many similar things on the table
785
Showing something behind something
781
Putting something into something
780
Tearing something just a little bit
779
Moving something away from something
778
Tearing something into two pieces
777
Holding something next to something
777
Pushing something from right to left
776
Putting something, something and something on the table
775
Moving something closer to something
775
Pretending to take something from somewhere
774
Pretending to put something next to something
773
Uncovering something
772
Something falling like a rock
772
Putting something and something on the table
772
Pouring something into something
769
Moving something down
769
Pulling something from right to left
767
Throwing something in the air and catching it
763
Tilting something with something on it until it falls off
762
Putting something in front of something
760
Pretending to turn something upside down
759
Putting something on a surface
757
Pretending to throw something
756
Showing something on top of something
756
Covering something with something
753
Squeezing something
752
Putting something similar to other things that are already on the table
751
Lifting up one end of something, then letting it drop down
749
Taking something out of something
747
Moving part of something
745
Pulling something from left to right
744
Lifting something up completely without letting it drop down
743
Holding something in front of something
743
Putting something behind something
743
Moving something and something closer to each other
743
Attaching something to something
742
Pushing something so that it falls off the table
735
Holding something over something
734
Pretending to open something without actually opening it
732
Removing something, revealing something behind
729
Hitting something with something
727
Moving something and something away from each other
727
Touching (without moving) part of something
724
Pretending to put something into something
724
Showing that something is inside something
721
Lifting something up completely, then letting it drop down
720
Pretending to take something out of something
709
Holding something behind something
707
Laying something on the table on its side, not upright
700
Poking something so it slightly moves
699
Pretending to close something without actually closing it
698
Putting something upright on the table
690
Dropping something in front of something
687
Dropping something behind something
685
Lifting up one end of something without letting it drop down
682
Rolling something on a flat surface
677
Throwing something onto a surface
671
Showing something next to something
668
Dropping something onto something
668
Stuffing something into something
662
Dropping something into something
662
Piling something up
660
Letting something roll along a flat surface
658
Twisting something
643
Spinning something that quickly stops spinning
636
Putting number of something onto something
634
Moving something across a surface without it falling down
634
Putting something underneath something
628
Plugging something into something but pulling it right out as you remove your hand
627
Dropping something next to something
606
Poking something so that it falls over
593
Spinning something so it continues spinning
588
Poking something so lightly that it doesn't or almost doesn't move
585
Wiping something off of something
582
Moving something across a surface until it falls down
580
Pretending to poke something
570
Putting something that cannot actually stand upright upright on the table, so it falls on its side
566
Pulling something out of something
565
Scooping something up with something
562
Pretending to be tearing something that is not tearable
543
Burying something in something
542
Tipping something over
533
Tilting something with something on it slightly so it doesn't fall down
528
Pretending to put something onto something
522
Bending something until it breaks
512
Letting something roll down a slanted surface
509
Trying to bend something unbendable so nothing happens
505
Bending something so that it deforms
503
Digging something out of something
502
Pretending to put something underneath something
497
Putting something on a flat surface without letting it roll
479
Putting something on the edge of something so it is not supported and falls down
471
Spreading something onto something
471
Pretending to put something behind something
466
Sprinkling something onto something
463
Something colliding with something and both come to a halt
462
Pushing something off of something
453
Putting something that can't roll onto a slanted surface, so it stays where it is
451
Lifting a surface with something on it until it starts sliding down
433
Trying but failing to attach something to something because it doesn't stick
433
Pretending or failing to wipe something off of something
427
Pulling something from behind of something
423
Pushing something so it spins
420
Pouring something onto something
416
Pulling two ends of something but nothing happens
413
Moving something and something so they pass each other
413
Pretending to sprinkle air onto something
405
Putting something that can't roll onto a slanted surface, so it slides down
395
Something colliding with something and both are being deflected
386
Pretending to squeeze something
367
Pulling something onto something
362
Putting something onto something else that cannot support it so it falls down
358
Pouring something out of something
358
Lifting a surface with something on it but not enough for it to slide down
346
Moving something and something so they collide with each other
341
Tipping something with something in it over, so something in it falls out
339
Letting something roll up a slanted surface, so it rolls back down
318
Pretending to scoop something up with something
311
Pretending to pour something out of something, but something is empty
294
Pulling two ends of something so that it gets stretched
290
Failing to put something into something because something does not fit
288
Pretending or trying and failing to twist something
282
Trying to pour something into something, but missing so it spills next to it
277
Something being deflected from something
273
Poking a stack of something so the stack collapses
267
Spilling something onto something
245
Pulling two ends of something so that it separates into two pieces
229
Pouring something into something until it overflows
220
Pretending to spread air onto something
219
Twisting (wringing) something wet until water comes out
217
Poking a hole into something soft
207
Spilling something next to something
206
Poking a stack of something without the stack collapsing
183
Putting something onto a slanted surface but it doesn't glide down
170
Pushing something onto something
141
Poking something so that it spins around
121
Spilling something behind something
77
Poking a hole into some substance

Leaderboard

If you have been successful in creating a model based on the training set and it performs well on the validation set, we encourage you to run your model on the test set (which is published without any class labels, as you might have noticed). Please prepare a .csv file with the video's id in the first column and your predicted class label (as a string matching the wording used in the training and validation sets). As a separator, please use a semicolon. You can then upload your .csv file here (user login required) to be ranked in the leaderboard and to benchmark your approach against that of other machine learners. We are looking forward to your submission.

Rank
Name
Approach
Accuracy
1
33d7b883bf
Debidatta Dwibedi
29 days ago
35.36 %
2
Yue Guo
about 1 month ago
34.71 %
3
Anonymous
about 1 month ago

TRN_v2

33.6 %
4
Yana Hasson
4 days ago
32.1 %
5
Cff25a29ed
Eren Gölge
about 2 months ago

Besnet

31.66 %
6
Guillaume Berger
about 2 months ago
30.48 %
7
Raghav Goyal
3 months ago
27.23 %
8
6d45d16e95
Harrison.AI
3 months ago
26.38 %
9
Valentin (esc) Haenel
15 days ago

https://github.com/esc/smth-smth-baseline/tree/train_004

smth-smth-baseline with dropout

24.55 %
10
Xiu-Shen Wei
2 months ago
22.13 %
11
Peng
about 2 months ago
21.93 %
12
Peter_CV
about 2 months ago
19.68 %
13
Aengus Tran
3 months ago
15.49 %
14
Eduardo Monauni
3 months ago
1.43 %
15
Aaryahi Roshan
4 months ago
1.39 %



Feedback