This repository was archived by the owner on Feb 29, 2024. It is now read-only.
Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
421 lines (369 loc) · 21.3 KB
/
Copy pathindex.html
File metadata and controls
421 lines (369 loc) · 21.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" media="all">
body {font-family: 'Droid Sans', helvetica,Arial,sans-serif;}
a:link, a:visited, a:active {text-decoration:none}
div.container{width:85%; margin:2%; line-height:140%;}
div.right{float:right;width:150px; margin:1em; padding:1em;}
div.content{margin-left:4%; padding:1em;}
</style>
<title>Ole Winther</title>
<meta name="description" content="Academic page of Ole Winther">
<meta name="keywords" content="Deep Learning, Gaussian processes, Variational Inference">
<meta name="resource-type" content="document">
<meta name="distribution" content="global">
<meta http-equiv="Content-Style-Type" content="text/css">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- <script type="text/javascript" async="" src="./assets/ga.js"></script><script type="text/javascript" async="" src="./assets/ga(1).js"></script>
<script type="text/javascript">
function trackOutboundLink(link, category, action) {
try {
_gaq.push(['_trackEvent', category , action]);
} catch(err){}
}
</script>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-8635368-2']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script> -->
<style type="text/css">
div.foo p {
margin-bottom:0.0em;
margin-top:0.0em;
}
</style>
</head>
<body data-new-gr-c-s-check-loaded="14.1012.0" data-gr-ext-installed="">
<div class="container">
<div class="right"><img alt="" width="250px" src="./assets/ole_page.jpg">
</div>
<div class="content">
<h1 style="text-align: left;">Ole Winther</h1>
<b><font size="4">
<!--<a href="#preprints" onClick="_gaq.push(['_trackEvent', 'withinpage', 'preprints', 'textlink']);">Preprints</a>-->
<!--| <a href="#courses" onclick="_gaq.push(['_trackEvent', 'withinpage', 'courses', 'textlink']);">Courses</a>| -->
<!--
<a href="#publications" onclick="_gaq.push(['_trackEvent', 'withinpage', 'publications', 'textlink']);">Publications</a>
| <a href="#videos" onclick="_gaq.push(['_trackEvent', 'withinpage', 'videos', 'textlink']);">Videos</a>
| <a href="#other" onclick="_gaq.push(['_trackEvent', 'withinpage', 'other', 'textlink']);">Misc</a>
| <a href="#talks" onclick="_gaq.push(['_trackEvent', 'withinpage', 'talks', 'textlink']);">Talks</a>
-->
</font></b>
<!-- <img alt="" src="./pictures/text.png"> <br> -->
<br>
<p>
Professor in High dimensional biological data analysis/Machine learning at <a href="https://www1.bio.ku.dk/english/research/scarb/">Section for Computational and RNA Biology</a>, <a href="https://www1.bio.ku.dk/" target="_blank">Department of Biology</a>, <a href="https://www.ku.dk/english/" target="_blank">University of Copenhagen</a> and professor in Data science and complexity at <a href="https://www.compute.dtu.dk/english/research/research-sections/cogsys" target="_blank">Section for Cognitive Systems</a>, <a href="https://www.compute.dtu.dk/" target="_blank">DTU Compute</a>,
<a href="https://www.dtu.dk/english" target="_blank">Technical University of Denmark</a>.
<br>
<br>
Current and former interests:
</p>
<ul>
<li><a href="https://www.nature.com/articles/s41587-021-01156-3" target="_blank">Biological sequence analysis</a> </li>
<li><a href="https://arxiv.org/abs/2310.19789" target="_blank">Diffusion models</a> </li>
<li><a href="https://arxiv.org/abs/2207.08143" target="_blank">Large language models</a> </li>
<li><a href="https://iopscience.iop.org/article/10.1088/2632-2153/aca23e" target="_blank">AI for science</a> </li>
<li><a href="https://academic.oup.com/bioinformatics/article/36/16/4415/5838187" target="_blank">Single cell gene expression analysis</a> </li>
<li><a href="https://arxiv.org/abs/1902.02102" target="_blank">Deep Generative Models</a> </li>
<li><a href="https://arxiv.org/abs/1605.07571" target="_blank">Variational Inference</a></li>
<li><a href="https://arxiv.org/abs/1602.02282" target="_blank">Structured Mean-field Approximations</a></li>
<li><a href="https://academic.oup.com/bioinformatics/article/33/21/3387/3931857?login=true" target="_blank">Gene Regulation and Sequence Bioinformatics</a></li>
<li><a href="https://pubmed.ncbi.nlm.nih.gov/23462700/" target="_blank">Health informatics and NLP for large-scale search</a></li>
<li><a href="http://videolectures.net/abi07_winther_lsb/" target="_blank">Matrix Factorization for Collaborative Filtering</a></li>
<li><a href="https://direct.mit.edu/neco/article/12/11/2655/6421/Gaussian-Processes-for-Classification-Mean-Field" target="_blank">Gaussian Processes</a></li>
<li><a href="https://www.jmlr.org/papers/volume6/opper05a/opper05a.pdf" target="_blank">Expectation Propagation</a></li>
</ul>
<p></p>
I'm CRO (chief research officer) and co-founder of <a href="http://www.raffle.ai/" target="_blank">raffle.ai</a>, where we use modern NLP for enterprise search.
<br>
I'm CTO and co-founder of <a href="https://www.findzebra.com/" target="_blank">FindZebra</a>, a search engine for rare diseases.
<br>
I'm Head of the <a href="https://ellis-cph.dk/" target="_blank">ELLIS Unit Copenhagen</a> and a co-PI of the <a href="https://www.mlls.dk/" target="_blank">Machine Learning for Life Science (MLLS) Center</a>.
<p></p>
<table style="width:85%">
<tbody>
<tr>
<td>
<a href="https://scholar.google.com/citations?user=7VAwhzUAAAAJ&hl=en" target="_blank"><img src="./assets/gscholar.png" width="45px" height="45px"></a>
<a href="https://github.com/DeepLearningDTU" target="_blank"><img src="./assets/github.svg" width="45px" height="45px"></a>
<a href="https://twitter.com/olewinther1" target="_blank"><img src="./assets/twitterblack.svg" width="45px" height="45px"></a>
<a href="https://dk.linkedin.com/in/owinther" target="_blank"><img src="./assets/linkedin.jpg" width="45px" height="45px"></a>
<p>
<!-- <a href="" onclick="_gaq.push(['_trackEvent', 'downloads', 'resume', 'pdf']);">Curriculum Vitae</a></p> -->
<p>
<br>
<p></p>
E-mail: <a href="mailto:ole.winther@bio.ku.dk">olwi@dtu.dk</a> ,<a href="mailto:olwi@dtu.dk">olwi@dtu.dk</a> and <a href="mailto:ole.winther@gmail.com">ole.winther@gmail.com</a>
</p>
Teaching:<ul>
<li><a href="https://kurser.dtu.dk/course/02456" target="_blank">02456 at DTU: Deep Learning</a></li>
</ul>
<br>
</td>
<td>
Current research group:
<ul>
<li><a href="" target="_blank">Jun Wang</a>, PhD, KU </li>
<li><a href="" target="_blank">Beatrix M. G. Nielsen</a>, PhD, DTU </li>
<li><a href="" target="_blank">Rachael M. DeVries</a>, PhD, KU with Novo Nordisk</li>
<li><a href="" target="_blank">Felix G. Teufel</a>, PhD, KU with Novo Nordisk</li>
<li><a href="" target="_blank">Irene R. Rodriguez</a>, postdoc, DTU</li>
<li><a href="" target="_blank">Anders Christensen</a>, PhD, DTU (ELLIS PhD, host <a href="https://eml-unitue.de/people/zeynep-akata" target="_blank">Zeynep Akata</a>)</li>
<li><a href="https://georgosgeorgos.github.io/" target="_blank">Giorgio Giannone</a>, PhD, DTU</li>
<li><a href="" target="_blank">Christopher Heje Grønbech</a>, PhD, KU and <a href="https://qlucore.com" target="_blank">Qlucore</a></li>
<li><a href="https://orbit.dtu.dk/en/persons/raluca-alexandra-jalaboi" target="_blank">Raluca Jalaboi</a>, Industrial PhD, DTU (<a href="https://omhu.com" target="_blank">Omhu</a>)</li>
<li><a href="https://orbit.dtu.dk/en/persons/jacob-mathias-schreiner" target="_blank">Mathias Schreiner</a>, PhD, DTU</li>
</ul>
<a href="./past_students.html" target="_blank">Former group members</a>
<!-- <br>
<a href="http://cogsys.imm.dtu.dk/staff/winther/" target="_blank">Old website</a> -->
<!-- <ul>
<li><a href="https://casperkaae.github.io/" target="_blank">Casper Sønderby</a>, PhD, KU, now Research Scientist at Google Brain</li>
<li><a href="https://dk.linkedin.com/in/s%C3%B8ren-kaae-s%C3%B8nderby-64810865" target="_blank">Søren Kaae Sønderby</a>, PhD, KU, now Machine Learning Engineer at Apple</li>
<li><a href="https://www.unumed.com/index.html" target="_blank">Marco Fraccaro</a>, PhD, DTU, now Chief Scientific Officer at Unumed</li>
<li><a href="https://www.corti.ai/" target="_blank">Lars Maaløe</a>, PhD, DTU, now CTO at Corti</li>
</ul>
<br> -->
<!-- <a href="https://www.dtu.dk/english/about/job-and-career/vacant-positions?inst=55004752#jobFilter" target="_blank">How to join</a> -->
</td>
</tr>
</tbody>
</table>
<div class="foo">
<!--<H2><a id=preprints>Preprints</a></H2>
<table border="0" cellspacing="4" cellpadding="2">
</tbody></table>
<br>
<p>
-->
<p></p><h2><a id="publications">Selected publications</a></h2>
<table border="0" cellspacing="2" cellpadding="2"><tbody>
<tr>
<td valign="top">
<a href="https://academic.oup.com/bioinformatics/article/36/16/4415/5838187?login=true" target="_blank">
<!-- <img src="./assets/" alt="Link" width="130px"> -->
</a>
</td>
<td valign="top">
<b>scVAE: variational auto-encoders for single-cell gene expression data</b>
<p>
</p>
<a href="" target="_blank">Christopher Heje Grønbech</a>,
<a href="" target="_blank">Maximillian Fornitz Vording</a>,
<a href="" target="_blank">Pascal N Timshel</a>,
<a href="" target="_blank">Casper Kaae Sønderby</a>,
<a href="" target="_blank">Tune H Pers</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>Bioinformatics</em>, 2020
<!-- <br>
<a href="https://arxiv.org/abs/1512.09300" target="_blank">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://arxiv.org/abs/2008.01998" target="_blank">
<img src="./assets/ovis.png" alt="Link" width="130px">
</a>
</td>
<td valign="top">
<b>Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Bounds</b>
<p>
This paper introduces novel results for the score function gradient estimator of the importance weighted variational bound (IWAE).
We prove that in the limit of large K (number of importance samples) one can choose the control variate such that the Signal-to-Noise ratio (SNR) of the estimator grows as √K.
This is in contrast to the standard pathwise gradient estimator where the SNR decreases as 1/√K.
Based on our theoretical findings we develop a novel control variate that extends on VIMCO.
Empirically, for the training of both continuous and discrete generative models, the proposed method yields superior variance reduction, resulting in an SNR for IWAE that increases with K without relying on the reparameterization trick.
The novel estimator is competitive with state-of-the-art reparameterization-free gradient estimators such as Reweighted Wake-Sleep (RWS) and the thermodynamic variational objective (TVO) when training generative models.
</p>
<a href="" target="_blank">Valentin Liévin</a>,
<a href="" target="_blank">Andrea Dittadi</a>,
<a href="" target="_blank">Anders Christensen</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>NeurIPS</em>, 2020
<!-- <br>
<a href="https://arxiv.org/abs/1512.09300" target="_blank">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://arxiv.org/abs/1902.02102" target="_blank">
<img src="./assets/biva.png" alt="Link" width="130px">
</a>
</td>
<td valign="top">
<b>BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling</b>
<p>
With the introduction of the variational autoencoder (VAE), probabilistic latent variable models have received renewed attention as powerful generative models.
However, their performance in terms of test likelihood and quality of generated samples has been surpassed by autoregressive models without stochastic units.
Furthermore, flow-based models have recently been shown to be an attractive alternative that scales well to high-dimensional data. In this paper we close the performance gap by constructing VAE models that can effectively utilize
a deep hierarchy of stochastic variables and model complex covariance structures. We introduce the Bidirectional-Inference Variational Autoencoder (BIVA), characterized by a skip-connected generative model and an inference network formed by
a bidirectional stochastic inference path.
</p>
<a href="" target="_blank">Lars Maaløe</a>,
<a href="" target="_blank">Marco Fraccaro</a>,
<a href="" target="_blank">Valentin Liévin</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>NeurIPS</em>, 2019
<!-- <br>
<a href="https://arxiv.org/abs/1512.09300" target="_blank">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://academic.oup.com/bioinformatics/article/33/24/4049/4191344" target="_blank">
<!-- <img src="./assets/" alt="Link" width="130px"> -->
</a>
</td>
<td valign="top">
<b>DeepLoc: prediction of protein subcellular localization using deep learning
</b>
<p>
</p>
<a href="" target="_blank">Jose Juan Almagro Armenteros</a>,
<a href="" target="_blank">Casper Kaae Sønderby</a>,
<a href="" target="_blank">Søren Kaae Sønderby</a>,
<a href="" target="_blank">Henrik Nielsen</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>Bioinformatics</em>, 2017
<br>
<!-- <a href="https://arxiv.org/abs/1605.07571">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://arxiv.org/abs/1602.02282" target="_blank">
<img src="./assets/ladder.png" alt="Link" width="130px">
</a>
</td>
<td valign="top">
<b>Ladder Variational Autoencoders
</b>
<p>
Variational Autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models.
We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network.
We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models.
</p>
<a href="" target="_blank">Casper Kaae Sønderby</a>,
<a href="" target="_blank">Tapani Raiko</a>,
<a href="" target="_blank">Lars Maaløe</a>,
<a href="" target="_blank">Søren Kaae Sønderby</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>NeurIPS</em>, 2016
<br>
<!-- <a href="https://arxiv.org/abs/1602.02282">paper</a>
| <a href="">code</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://arxiv.org/abs/1605.07571" target="_blank">
<img src="./assets/srnn.png" alt="Link" width="130px">
</a>
</td>
<td valign="top">
<b>Sequential Neural Models with Stochastic Layers
</b>
<p>
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model's posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the state of the art results on the Blizzard and TIMIT speech modeling data sets by a large margin, while achieving comparable performances to competing methods on polyphonic music modeling.
</p>
<a href="" target="_blank">Marco Fraccaro</a>,
<a href="" target="_blank">Søren Kaae Sønderby</a>,
<a href="" target="_blank">Ulrich Paquet</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>NeurIPS</em>, 2016, Oral
<br>
<!-- <a href="https://arxiv.org/abs/1605.07571">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://arxiv.org/abs/1602.05473" target="_blank">
<img src="./assets/avae.png" alt="Link" width="130px">
</a>
</td>
<td valign="top">
<b>Auxiliary Deep Generative Models
</b>
<p>
Deep generative models parameterized by neural networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep generative models with auxiliary variables which improves the variational approximation. The auxiliary variables leave the generative model unchanged but make the variational distribution more expressive. Inspired by the structure of the auxiliary variable we also propose a model with two stochastic layers and skip connections. Our findings suggest that more expressive and properly specified deep generative models converge faster with better results. We show state-of-the-art performance within semi-supervised learning on MNIST, SVHN and NORB datasets.
</p>
<a href="" target="_blank">Lars Maaløe</a>,
<a href="" target="_blank">Casper Kaae Sønderby</a>,
<a href="" target="_blank">Søren Kaae Sønderby</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>ICML</em>, 2016
<br>
<!-- <a href="https://arxiv.org/abs/1605.07571">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://arxiv.org/abs/1512.09300" target="_blank">
<img src="./assets/beyond_pixels.png" alt="Link" width="130px">
</a>
</td>
<td valign="top">
<b>Autoencoding beyond pixels using a learned similarity metric</b>
<p>
We present an autoencoder that leverages learned representations to better measure similarities in data space.
By combining a variational autoencoder (VAE) with a generative adversarial network (GAN) we can use learned feature representations in the GAN discriminator as basis for the VAE reconstruction objective.
Thereby, we replace element-wise errors with feature-wise errors to better capture the data distribution while offering invariance towards eg translation.
</p>
<a href="" target="_blank">Anders Boesen Lindbo Larsen</a>,
<a href="" target="_blank">Søren Kaae Sønderby</a>,
<a href="" target="_blank">Hugo Larochelle</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>ICML</em>, 2016
<!-- <br>
<a href="https://arxiv.org/abs/1512.09300" target="_blank">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
<tr>
<td valign="top">
<a href="https://jmlr.org/papers/v14/opper13a.html" target="_blank">
<!-- <img src="./assets/beyond_pixels.png" alt="Link" width="130px"> -->
</a>
</td>
<td valign="top">
<b>Perturbative Corrections for Approximate Inference in Gaussian
Latent Variable Models</b>
<p>
Expectation Propagation (EP) provides a framework for approximate inference. When the model under consideration is over a latent Gaussian field, with the approximation being Gaussian, we show how these approximations can systematically be corrected. A perturbative expansion is made of the exact but intractable correction, and can be applied to the model's partition function and other moments of interest. The correction is expressed over the higher-order cumulants which are neglected by EP's local matching of moments. Through the expansion, we see that EP is correct to first order. By considering higher orders, corrections of increasing polynomial complexity can be applied to the approximation. The second order provides a correction in quadratic time, which we apply to an array of Gaussian process and Ising models. The corrections generalize to arbitrarily complex approximating families, which we illustrate on tree- structured Ising model approximations. Furthermore, they provide a polynomial-time assessment of the approximation error. We also provide both theoretical and practical insights on the exactness of the EP solution.
</p>
<a href="" target="_blank">Manfred Opper</a>,
<a href="" target="_blank">Ulrich Paquet</a>,
<a href="" target="_blank">Ole Winther</a>
<br>
<em>JMLR</em>, 2013
<!-- <br>
<a href="https://arxiv.org/abs/1512.09300" target="_blank">paper</a> -->
</td>
</tr>
<tr><td height="10px"></td></tr>
</tbody></table>
</div>
</div>
</div>
</body></html>