DataLoader With Multiple Workers Leaks Memory PyTorch Developer podcast

Artwork

Tech PyTorch Edward Yang Team PyTorch Deep Learning Machine Learning

Kandungan disediakan oleh PyTorch, Edward Yang, and Team PyTorch. Semua kandungan podcast termasuk episod, grafik dan perihalan podcast dimuat naik dan disediakan terus oleh PyTorch, Edward Yang, and Team PyTorch atau rakan kongsi platform podcast mereka. Jika anda percaya seseorang menggunakan karya berhak cipta anda tanpa kebenaran anda, anda boleh mengikuti proses yang digariskan di sini https://ms.player.fm/legal.

PyTorch Developer Podcast « »
DataLoader with multiple workers leaks memory

3+ y ago 16:38

Kongsi

MP3•Laman utama episod

Kandungan disediakan oleh PyTorch, Edward Yang, and Team PyTorch. Semua kandungan podcast termasuk episod, grafik dan perihalan podcast dimuat naik dan disediakan terus oleh PyTorch, Edward Yang, and Team PyTorch atau rakan kongsi platform podcast mereka. Jika anda percaya seseorang menggunakan karya berhak cipta anda tanpa kebenaran anda, anda boleh mengikuti proses yang digariskan di sini https://ms.player.fm/legal.

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.

Further reading.

A nice summary of the full issue https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
DataLoader architecture RFC https://github.com/pytorch/pytorch/issues/49440
Cinder Python https://github.com/facebookincubator/cinder

… continue reading

83 episod

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Artwork

DataLoader with multiple workers leaks memory

PyTorch Developer Podcast

32 subscribers

published 3+ y ago

Kongsi

MP3•Laman utama episod

Kandungan disediakan oleh PyTorch, Edward Yang, and Team PyTorch. Semua kandungan podcast termasuk episod, grafik dan perihalan podcast dimuat naik dan disediakan terus oleh PyTorch, Edward Yang, and Team PyTorch atau rakan kongsi platform podcast mereka. Jika anda percaya seseorang menggunakan karya berhak cipta anda tanpa kebenaran anda, anda boleh mengikuti proses yang digariskan di sini https://ms.player.fm/legal.

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.

Further reading.

A nice summary of the full issue https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
DataLoader architecture RFC https://github.com/pytorch/pytorch/issues/49440
Cinder Python https://github.com/facebookincubator/cinder

… continue reading

83 episod

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Semua episod

×

Selamat datang ke Player FM

Player FM mengimbas laman-laman web bagi podcast berkualiti tinggi untuk anda nikmati sekarang. Ia merupakan aplikasi podcast terbaik dan berfungsi untuk Android, iPhone, dan web. Daftar untuk melaraskan langganan merentasi peranti.

Dengarkan lebih 500+ topik

Panduan Rujukan Pantas

Podcast Teratas

Bantuan/Soalan Lazim | Naik taraf | Iklankan

Seni|Perniagaan|Komedi|Ekonomi|Hiburan|Berita|Politik|Agama

Sains|Bolasepak|Sukan|Bercerita|Teknologi|True Crime

Hak Cipta 2025 | Peta Laman | Dasar Privasi | Syarat Perkhidmatan | | hak cipta

Dengar rancangan ini semasa anda meneroka