))
Gradient Hacking: Definitions and Examples
Siri yang diarkibkan ("Suapan tidak aktif" status)
When?
This feed was archived on February 21, 2025 21:08 (
Why? Suapan tidak aktif status. Pelayan kami tidak dapat untuk mendapatkan kembali suapan podcast yang sah untuk tempoh yang didapati.
What now? You might be able to find a more up-to-date version using the search function. This series will no longer be checked for updates. If you believe this to be in error, please check if the publisher's feed link below is valid and contact support to request the feed be restored or if you have any other concerns about this.
Manage episode 424087970 series 3498845
Kandungan disediakan oleh BlueDot Impact. Semua kandungan podcast termasuk episod, grafik dan perihalan podcast dimuat naik dan disediakan terus oleh BlueDot Impact atau rakan kongsi platform podcast mereka. Jika anda percaya seseorang menggunakan karya berhak cipta anda tanpa kebenaran anda, anda boleh mengikuti proses yang digariskan di sini https://ms.player.fm/legal.
Gradient hacking is a hypothesized phenomenon where:
- A model has knowledge about possible training trajectories which isn’t being used by its training algorithms when choosing updates (such as knowledge about non-local features of its loss landscape which aren’t taken into account by local optimization algorithms).
- The model uses that knowledge to influence its medium-term training trajectory, even if the effects wash out in the long term.
Below I give some potential examples of gradient hacking, divided into those which exploit RL credit assignment and those which exploit gradient descent itself. My concern is that models might use techniques like these either to influence which goals they develop, or to fool our interpretability techniques. Even if those effects don’t last in the long term, they might last until the model is smart enough to misbehave in other ways (e.g. specification gaming, or reward tampering), or until it’s deployed in the real world—especially in the RL examples, since convergence to a global optimum seems unrealistic (and ill-defined) for RL policies trained on real-world data. However, since gradient hacking isn’t very well-understood right now, both the definition above and the examples below should only be considered preliminary.
Source:
https://www.alignmentforum.org/posts/EeAgytDZbDjRznPMA/gradient-hacking-definitions-and-examples
Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.
---
A podcast by BlueDot Impact.
Learn more on the AI Safety Fundamentals website.
Bab
1. Gradient Hacking: Definitions and Examples (00:00:00)
2. RL credit hacking examples (00:01:26)
3. Gradient descent hacking examples (00:03:15)
85 episod
Kongsi
