Artwork

Kandungan disediakan oleh Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. Semua kandungan podcast termasuk episod, grafik dan perihalan podcast dimuat naik dan disediakan terus oleh Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci atau rakan kongsi platform podcast mereka. Jika anda percaya seseorang menggunakan karya berhak cipta anda tanpa kebenaran anda, anda boleh mengikuti proses yang digariskan di sini https://ms.player.fm/legal.
Player FM - Aplikasi Podcast
Pergi ke luar talian dengan aplikasi Player FM !

185: Workflow Orchestrators

1:32:02
 
Kongsi
 

Manage episode 517701083 series 8393
Kandungan disediakan oleh Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. Semua kandungan podcast termasuk episod, grafik dan perihalan podcast dimuat naik dan disediakan terus oleh Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci atau rakan kongsi platform podcast mereka. Jika anda percaya seseorang menggunakan karya berhak cipta anda tanpa kebenaran anda, anda boleh mengikuti proses yang digariskan di sini https://ms.player.fm/legal.

Intro topic: Asymmetric Returns

News/Links:

Book of the Show

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
  continue reading

186 episod

Artwork

185: Workflow Orchestrators

Programming Throwdown

7,601 subscribers

published

iconKongsi
 
Manage episode 517701083 series 8393
Kandungan disediakan oleh Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. Semua kandungan podcast termasuk episod, grafik dan perihalan podcast dimuat naik dan disediakan terus oleh Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci atau rakan kongsi platform podcast mereka. Jika anda percaya seseorang menggunakan karya berhak cipta anda tanpa kebenaran anda, anda boleh mengikuti proses yang digariskan di sini https://ms.player.fm/legal.

Intro topic: Asymmetric Returns

News/Links:

Book of the Show

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
  continue reading

186 episod

Semua episod

×
 
Loading …

Selamat datang ke Player FM

Player FM mengimbas laman-laman web bagi podcast berkualiti tinggi untuk anda nikmati sekarang. Ia merupakan aplikasi podcast terbaik dan berfungsi untuk Android, iPhone, dan web. Daftar untuk melaraskan langganan merentasi peranti.

 

Panduan Rujukan Pantas

Podcast Teratas
Dengar rancangan ini semasa anda meneroka
Main