Repository navigation
Expand file tree
/
Copy pathREADME.html
More file actions
467 lines (466 loc) · 20.4 KB
/
Copy pathREADME.html
File metadata and controls
467 lines (466 loc) · 20.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
<h1 id="epic-kitchens-dataset"><a href="https://epic-kitchens.github.io/2018">EPIC Kitchens Dataset</a></h1>
<!-- start badges -->
<!-- end badges -->
<blockquote>
<p><a href="https://epic-kitchens.github.io/">EPIC-Kitchens</a> is the largest dataset in first-person (egocentric) vision; multi-faceted non-scripted recordings in native environments - i.e. the wearers' homes, capturing all daily activities in the kitchen over multiple days. Annotations are collected using a novel `live' audio commentary approach.</p>
</blockquote>
<h2 id="authors">Authors</h2>
<p>Dima Damen (1) Hazel Doughty (1) Giovanni Maria Farinella (3) Sanja Fidler (2) Antonino Furnari (3) Evangelos Kazakos (1) Davide Moltisanti (1) Jonathan Munro (1) Toby Perrett (1) Will Price (1) Michael Wray (1)</p>
<ul>
<li>(1 University of Bristol)</li>
<li>(2 University of Toronto)</li>
<li>(3 University of Catania)</li>
</ul>
<p><strong>Contact:</strong> <script type="text/javascript">
<!--
h='bristol.ac.uk';a='@';n='uob-epic-kitchens2018';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'">'+e+'<\/'+'a'+'>');
// -->
</script><noscript>uob-epic-kitchens2018 at bristol dot ac dot uk</noscript></p>
<h2 id="citing">Citing</h2>
<p>When using the dataset, kindly reference:</p>
<pre><code>@article{Damen2018EPICKITCHENS,
title={Scaling Egocentric Vision: The EPIC-KITCHENS Dataset},
author={Damen, Dima and Doughty, Hazel and Farinella, Giovanni Maria and Fidler, Sanja and
Furnari, Antonino and Kazakos, Evangelos and Moltisanti, Davide and Munro, Jonathan
and Perrett, Toby and Price, Will and Wray, Michael},
journal={arXiv preprint arXiv:1804.02748},
year={2018}
}</code></pre>
<p>(Check publication <a href="https://epic-kitchens.github.io">here</a>)</p>
<h2 id="dataset-details">Dataset Details</h2>
<h3 id="ground-truth">Ground Truth</h3>
<p>We provide ground truth for action segments and object bounding boxes.</p>
<ul>
<li><strong>Objects:</strong> Full bounding boxes of narrated objects for every annotated frame.</li>
<li><strong>Actions:</strong> Split into narrations and action labels:
<ul>
<li>Narrations containing the narrated sentence with the timestamp.</li>
<li>Action labels containing the verb and noun labels along with the start and end times of the segment.</li>
</ul></li>
</ul>
<h3 id="dataset-splits">Dataset Splits</h3>
<p>The dataset is comprised of three splits with the corresponding ground truth:</p>
<ul>
<li>Training set - Full ground truth.</li>
<li>Seen Kitchens (S1) Test set - Start/end times only.</li>
<li>Unseen Kitchens (S2) Test set - Start/end times only.</li>
</ul>
<p>Initially we are only releasing the full ground truth for the training set in order to run action and object challenges.</p>
<h3 id="important-files">Important Files</h3>
<ul>
<li><code>README.md (this file)</code></li>
<li><code>README.html</code></li>
<li><code>README.pdf</code></li>
<li><a href="#license"><code>license.txt</code></a></li>
<li><a href="EPIC_train_action_labels.csv"><code>EPIC_train_action_labels.csv</code></a> (<a href="#epic_train_action_labelscsv">Info</a>).</li>
<li><a href="EPIC_train_action_labels.pkl"><code>EPIC_train_action_labels.pkl</code></a> (<a href="#epic_train_action_labelscsv">Info</a>).</li>
<li><a href="EPIC_train_action_narrations.csv"><code>EPIC_train_action_narrations.csv</code></a> (<a href="#epic_train_action_narrationscsv">Info</a>).</li>
<li><a href="EPIC_train_object_labels.csv"><code>EPIC_train_object_labels.csv</code></a> (<a href="#epic_train_object_labelscsv">Info</a>).</li>
<li><a href="EPIC_test_s1_timestamps.csv"><code>EPIC_test_s1_timestamps.csv</code></a> (<a href="#epic_test_s1_timestampscsv">Info</a>).</li>
<li><a href="EPIC_test_s1_timestamps.pkl"><code>EPIC_test_s1_timestamps.pkl</code></a> (<a href="#epic_test_s1_timestampscsv">Info</a>).</li>
<li><a href="EPIC_test_s1_timestamps.csv"><code>EPIC_test_s2_timestamps.csv</code></a> (<a href="#epic_test_s2_timestampscsv">Info</a>).</li>
<li><a href="EPIC_test_s1_timestamps.pkl"><code>EPIC_test_s2_timestamps.pkl</code></a> (<a href="#epic_test_s2_timestampscsv">Info</a>).</li>
<li><a href="EPIC_noun_classes.csv"><code>EPIC_noun_classes.csv</code></a> (<a href="#epic_noun_classescsv">Info</a>).</li>
<li><a href="EPIC_verb_classes.csv"><code>EPIC_verb_classes.csv</code></a> (<a href="#epic_verb_classescsv">Info</a>).</li>
</ul>
<p>We direct the reader to <a href="https://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d">RDSF</a> for the videos and rgb/flow frames.</p>
<p>We provide html and pdf alternatives to this README which are auto-generated.</p>
<h2 id="files-structure">Files Structure</h2>
<h3 id="epic_train_action_labels.csv">EPIC_train_action_labels.csv</h3>
<p>CSV file containing 14 columns:</p>
<table>
<thead>
<tr class="header">
<th align="left">Column Name</th>
<th align="left">Type</th>
<th align="left">Example</th>
<th align="left">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left"><code>uid</code></td>
<td align="left">int</td>
<td align="left"><code>6374</code></td>
<td align="left">Unique ID of the segment.</td>
</tr>
<tr class="even">
<td align="left"><code>video_id</code></td>
<td align="left">string</td>
<td align="left"><code>P03_01</code></td>
<td align="left">Video the segment is in.</td>
</tr>
<tr class="odd">
<td align="left"><code>narration</code></td>
<td align="left">string</td>
<td align="left"><code>close fridge</code></td>
<td align="left">English description of the action provided by the participant.</td>
</tr>
<tr class="even">
<td align="left"><code>start_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:23:43.847</code></td>
<td align="left">Start time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="odd">
<td align="left"><code>stop_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:23:47.212</code></td>
<td align="left">End time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="even">
<td align="left"><code>start_frame</code></td>
<td align="left">int</td>
<td align="left"><code>85430</code></td>
<td align="left">Start frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="odd">
<td align="left"><code>stop_frame</code></td>
<td align="left">int</td>
<td align="left"><code>85643</code></td>
<td align="left">End frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="even">
<td align="left"><code>participant_id</code></td>
<td align="left">string</td>
<td align="left"><code>P03</code></td>
<td align="left">ID of the participant.</td>
</tr>
<tr class="odd">
<td align="left"><code>verb</code></td>
<td align="left">string</td>
<td align="left"><code>close</code></td>
<td align="left">Parsed verb from the narration.</td>
</tr>
<tr class="even">
<td align="left"><code>noun</code></td>
<td align="left">string</td>
<td align="left"><code>fridge</code></td>
<td align="left">First parsed noun from the narration.</td>
</tr>
<tr class="odd">
<td align="left"><code>verb_class</code></td>
<td align="left">int</td>
<td align="left"><code>3</code></td>
<td align="left">Numeric ID of the parsed verb's class.</td>
</tr>
<tr class="even">
<td align="left"><code>noun_class</code></td>
<td align="left">int</td>
<td align="left"><code>10</code></td>
<td align="left">Numeric ID of the parsed noun's class.</td>
</tr>
<tr class="odd">
<td align="left"><code>all_nouns</code></td>
<td align="left">list of string (1 or more)</td>
<td align="left"><code>['fridge']</code></td>
<td align="left">List of all parsed nouns from the narration.</td>
</tr>
<tr class="even">
<td align="left"><code>all_nouns_class</code></td>
<td align="left">list of int (1 or more)</td>
<td align="left"><code>[10]</code></td>
<td align="left">List of numeric IDs corresponding to all of the parsed nouns' classes from the narration.</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above.</p>
<h3 id="epic_train_action_narrations.csv">EPIC_train_action_narrations.csv</h3>
<p>CSV file containing 5 columns:</p>
<p><em>Note: The start/end timestamp refers to the start/end time of the narration, not the action itself.</em></p>
<table>
<thead>
<tr class="header">
<th align="left">Column Name</th>
<th align="left">Type</th>
<th align="left">Example</th>
<th align="left">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left"><code>participant_id</code></td>
<td align="left">string</td>
<td align="left"><code>P03</code></td>
<td align="left">ID of the participant.</td>
</tr>
<tr class="even">
<td align="left"><code>video_id</code></td>
<td align="left">string</td>
<td align="left"><code>P03_01</code></td>
<td align="left">Video the segment is in.</td>
</tr>
<tr class="odd">
<td align="left"><code>start_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:23:43.847</code></td>
<td align="left">Start time in <code>HH:mm:ss.SSS</code> of the narration.</td>
</tr>
<tr class="even">
<td align="left"><code>stop_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:23:47.212</code></td>
<td align="left">End time in <code>HH:mm:ss.SSS</code> of the narration.</td>
</tr>
<tr class="odd">
<td align="left"><code>narration</code></td>
<td align="left">string</td>
<td align="left"><code>close fridge</code></td>
<td align="left">English description of the action provided by the participant.</td>
</tr>
</tbody>
</table>
<h3 id="epic_train_object_labels.csv">EPIC_train_object_labels.csv</h3>
<p>CSV file containing 6 columns:</p>
<table>
<thead>
<tr class="header">
<th align="left">Column Name</th>
<th align="left">Type</th>
<th align="left">Example</th>
<th align="left">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left"><code>noun_class</code></td>
<td align="left">int</td>
<td align="left"><code>20</code></td>
<td align="left">Integer value representing the class in noun-classes.csv.</td>
</tr>
<tr class="even">
<td align="left"><code>noun</code></td>
<td align="left">string</td>
<td align="left"><code>bag</code></td>
<td align="left">Original string name for the object.</td>
</tr>
<tr class="odd">
<td align="left"><code>participant_id</code></td>
<td align="left">string</td>
<td align="left"><code>P01</code></td>
<td align="left">ID of participant.</td>
</tr>
<tr class="even">
<td align="left"><code>video_id</code></td>
<td align="left">string</td>
<td align="left"><code>P01_01</code></td>
<td align="left">Video the object was annotated in.</td>
</tr>
<tr class="odd">
<td align="left"><code>frame</code></td>
<td align="left">int</td>
<td align="left"><code>056581</code></td>
<td align="left">Frame number of the annotated object.</td>
</tr>
<tr class="even">
<td align="left"><code>bounding_boxes</code></td>
<td align="left">list of 4-tuple (0 or more)</td>
<td align="left"><code>"[(76, 1260, 462, 186)]"</code></td>
<td align="left">Annotated boxes with format <code>(<top:int>,<left:int>,<width:int>,<height:int>)</code>.</td>
</tr>
</tbody>
</table>
<h3 id="epic_test_s1_timestamps.csv">EPIC_test_s1_timestamps.csv</h3>
<p>CSV file containing 7 columns:</p>
<table>
<thead>
<tr class="header">
<th align="left">Column Name</th>
<th align="left">Type</th>
<th align="left">Example</th>
<th align="left">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left"><code>uid</code></td>
<td align="left">int</td>
<td align="left"><code>1924</code></td>
<td align="left">Unique ID of the segment.</td>
</tr>
<tr class="even">
<td align="left"><code>participant_id</code></td>
<td align="left">string</td>
<td align="left"><code>P01</code></td>
<td align="left">ID of the participant.</td>
</tr>
<tr class="odd">
<td align="left"><code>video_id</code></td>
<td align="left">string</td>
<td align="left"><code>P01_11</code></td>
<td align="left">Video the segment is in.</td>
</tr>
<tr class="even">
<td align="left"><code>start_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:00:00.000</code></td>
<td align="left">Start time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="odd">
<td align="left"><code>stop_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:00:01.890</code></td>
<td align="left">End time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="even">
<td align="left"><code>start_frame</code></td>
<td align="left">int</td>
<td align="left"><code>1</code></td>
<td align="left">Start frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="odd">
<td align="left"><code>stop_frame</code></td>
<td align="left">int</td>
<td align="left"><code>93</code></td>
<td align="left">End frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above.</p>
<h3 id="epic_test_s2_timestamps.csv">EPIC_test_s2_timestamps.csv</h3>
<p>CSV file containing 7 columns:</p>
<table>
<thead>
<tr class="header">
<th align="left">Column Name</th>
<th align="left">Type</th>
<th align="left">Example</th>
<th align="left">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left"><code>uid</code></td>
<td align="left">int</td>
<td align="left"><code>15582</code></td>
<td align="left">Unique ID of the segment.</td>
</tr>
<tr class="even">
<td align="left"><code>participant_id</code></td>
<td align="left">string</td>
<td align="left"><code>P09</code></td>
<td align="left">ID of the participant.</td>
</tr>
<tr class="odd">
<td align="left"><code>video_id</code></td>
<td align="left">string</td>
<td align="left"><code>P09_01</code></td>
<td align="left">Video the segment is in.</td>
</tr>
<tr class="even">
<td align="left"><code>start_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:00:01.970</code></td>
<td align="left">Start time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="odd">
<td align="left"><code>stop_timestamp</code></td>
<td align="left">string</td>
<td align="left"><code>00:00:03.090</code></td>
<td align="left">End time in <code>HH:mm:ss.SSS</code> of the action.</td>
</tr>
<tr class="even">
<td align="left"><code>start_frame</code></td>
<td align="left">int</td>
<td align="left"><code>118</code></td>
<td align="left">Start frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
<tr class="odd">
<td align="left"><code>stop_frame</code></td>
<td align="left">int</td>
<td align="left"><code>185</code></td>
<td align="left">End frame of the action (WARNING only for frames extracted as detailed in <a href="#video-information">Video Information</a>).</td>
</tr>
</tbody>
</table>
<p>Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above.</p>
<h3 id="epic_noun_classes.csv">EPIC_noun_classes.csv</h3>
<p>CSV file containing 3 columns:</p>
<p><em>Note: a colon represents a compound noun with the more generic noun first. So pan:dust should be read as dust pan.</em></p>
<table>
<thead>
<tr class="header">
<th align="left">Column Name</th>
<th align="left">Type</th>
<th align="left">Example</th>
<th align="left">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left"><code>noun_id</code></td>
<td align="left">int</td>
<td align="left"><code>2</code></td>
<td align="left">ID of the noun class.</td>
</tr>
<tr class="even">
<td align="left"><code>class_key</code></td>
<td align="left">string</td>
<td align="left"><code>pan:dust</code></td>
<td align="left">Key of the noun class.</td>
</tr>
<tr class="odd">
<td align="left"><code>nouns</code></td>
<td align="left">list of string (1 or more)</td>
<td align="left"><code>"['pan:dust', 'dustpan']"</code></td>
<td align="left">All nouns within the class (includes the key).</td>
</tr>
</tbody>
</table>
<h3 id="epic_verb_classes.csv">EPIC_verb_classes.csv</h3>
<p>CSV file containing 3 columns:</p>
<table>
<thead>
<tr class="header">
<th align="left">Column Name</th>
<th align="left">Type</th>
<th align="left">Example</th>
<th align="left">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left"><code>verb_id</code></td>
<td align="left">int</td>
<td align="left"><code>3</code></td>
<td align="left">ID of the verb class.</td>
</tr>
<tr class="even">
<td align="left"><code>class_key</code></td>
<td align="left">string</td>
<td align="left"><code>close</code></td>
<td align="left">Key of the verb class.</td>
</tr>
<tr class="odd">
<td align="left"><code>verbs</code></td>
<td align="left">list of string (1 or more)</td>
<td align="left"><code>"['close', 'close-off', 'shut']"</code></td>
<td align="left">All verbs within the class (includes the key).</td>
</tr>
</tbody>
</table>
<h2 id="file-downloads">File Downloads</h2>
<p>Due to the size of the dataset we provide three scripts for downloading the <a href="https://github.com/epic-kitchens/download-scripts/blob/master/download_videos.sh">videos</a>, <a href="https://github.com/epic-kitchens/download-scripts/blob/master/download_frames_rgb_flow.sh">frames</a> or <a href="https://github.com/epic-kitchens/download-scripts/blob/master/download_object_detection_images.sh">object annotation images</a> separately.</p>
<p><em>Note: These scripts will work for Linux and Mac. For Windows users a bash installation should work.</em></p>
<p>These scripts replicate the folder structure of the dataset release, found <a href="https://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d">here</a>.</p>
<p>If you wish to download part of the dataset instructions can be found <a href="https://github.com/epic-kitchens/download-scripts">here</a>.</p>
<h2 id="video-information">Video Information</h2>
<p>Videos are recorded in 1080p at 59.94 FPS on a GoPro Hero 5 with linear field of view. There are a minority of videos which were shot at different resolutions, field of views, or FPS due to participant error or camera. These videos identified using <code>ffprobe</code> are:</p>
<ul>
<li>1280x720: <code>P12_01</code>, <code>P12_02</code>, <code>P12_03</code>, <code>P12_04</code>.</li>
<li>2560x1440: <code>P12_05</code>, <code>P12_06</code></li>
<li>29.97 FPS: <code>P09_07</code>, <code>P09_08</code>, <code>P10_01</code>, <code>P10_04</code>, <code>P11_01</code>, <code>P18_02</code>, <code>P18_03</code></li>
<li>48 FPS: <code>P17_01</code>, <code>P17_02</code>, <code>P17_03</code>, <code>P17_04</code></li>
<li>90 FPS: <code>P18_09</code></li>
</ul>
<p>The GoPro Hero 5 was also set to drop the framerate in low light conditions to preserve exposure leading to variable FPS in some videos. If you wish to extract frames we suggest you resample at 60 FPS to mitigate issues with variable FPS, this can be achieved in a single step with FFmpeg:</p>
<pre><code>ffmpeg -i "P##_**.MP4" -vf "scale=-2:256" -q:v 4 -r 60 "P##_**/frame_%010d.jpg"</code></pre>
<p>where <code>##</code> is the Participant ID and <code>**</code> is the video ID.</p>
<p>Optical flow was extracted using a fork of <a href="https://github.com/feichtenhofer/gpu_flow"><code>gpu_flow</code></a> made <a href="https://github.com/dl-container-registry/furnari-flow">available on github</a>. We set the parameters: stride = 2, dilation = 3, bound = 25 and size = 256.</p>
<h2 id="license">License</h2>
<p>All files in this dataset are copyright by us and published under the Creative Commons Attribution-NonCommerial 4.0 International License, found <a href="https://creativecommons.org/licenses/by-nc/4.0/">here</a>. This means that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not use the material for commercial purposes.</p>
<h2 id="changelog">Changelog</h2>
<ul>
<li>09/04/18: Initial Release</li>
</ul>