SHRED: Retain-Set-Free Unlearning
via Self-Distillation with Logit Demotion

Zizhao Hu, Ameya Godbole, Johnny Tian-zheng Wei, Mohammad Rostami, Jesse Thomason, Robin Jia

GLAMOUR Lab · Allegro Lab · University of Southern California · June 2026

AI chatbots helping plan bioweapon attacks
AI models contain entire Harry Potter book

Why Machine Unlearning?

LLMs memorize training data — this creates real-world risks

🔒 Privacy

Personal data leakage, GDPR compliance

⚠️ Hazardous Knowledge

Bioweapons, exploits, chemical synthesis

©️ Copyright

Verbatim text reproduction, IP violations

🏛️ Government & Intelligence

Classified data exposure, national security

📰 Misinformation

Outdated facts, false claims at scale

📊 Test-Set Contamination

Inflated benchmarks, unreliable LLM evals

Goal: Remove specific knowledge while preserving everything else

Problem Definition

🤖
Contaminated LLM
Memorizes forget set + retain set
📄 Forget Request
Facts, documents, or entities to remove
Unlearning
Algorithm
→
🤖 ✓
Unlearned LLM
✓ Retains general knowledge & utility
✓ Preserves retain set performance
✗ Cannot recall forget set content

The Core Challenge

Retraining is expensive · Model editing is destructive

🎯 Gold Standard: Retrain

Remove forget data from training set, retrain from scratch

✓ Perfect unlearning guarantee

✗ Extremely costly — full retraining from scratch

✗ Not scalable — must retrain again for each new request

vs.

⚡ Approximate Unlearning

Modify the model post-hoc to remove specific knowledge

✓ Fast — minutes instead of weeks

✓ Scalable — handles sequential requests

⚠ Risk of destroying model utility

Method inspiration

Self-Distillation Enables Continual Learning

Motivation: Not All Tokens Are Equal

Memorized content has a distinctive probability signature

High-P Tokens (common structure)

The author 's full name is

Function words, common patterns — NOT memorization

Low-P Tokens (memorized facts)

…is Hsiao Yun-Hwa

Rare tokens, specific names/facts — THIS is memorization

SHRED Principle

Target only the bottom P% tokens by probability — leave everything else untouched

SHRED: How It Works

For each position in the forget set, create a target distribution

P = 50% → target the top 50% highest-entropy positions for demotion
Q: Who wrote "The Golden Compass"? A:
The author 's name is Hsiao Yun -H wa .
■ Low entropy: keep original distribution (implicit retain) ■ High entropy: demote ground-truth logit → −∞
▼ "name" — low entropy → keep
▼ "Hsiao" — high entropy → demote
"name" — Low Entropy → Keep ✓
0.78
name
0.08
title
0.06
pen
0.08
…
No change → implicit retain
"Hsiao" — High Entropy (Original)
0.42
Hsiao
0.20
Philip
0.15
the
0.23
…
Content token dominates ❌
→
After Demotion (KL Target)
0.00
Hsiao
0.35
Philip
0.26
the
0.39
…
Redistributed naturally ✓
Training Objective

L = KL(qtarget ‖ pmodel) — single loss, no separate retain set needed

Training Pipeline

A New Forget–Utility Frontier — No Retain Set

SHRED (circles) is the only method in the top-left corner on all four benchmarks

Forgetting That Survives Relearning

200 fine-tune steps on 40 forget examples: SHRED +0.15 vs RMU +0.67 · NPO +0.47 (oracle +0.05)

Robust Against Membership-Inference (MIA) Attacks

Small-batch SHRED reaches PrivLeak ≈ 0 (the oracle); retain-set baselines stay near −100

Graceful Under Sequential Requests

Utility drops only ~0.08 by round 3 — vs ~0.15 for NPO and RMU

One Knob (P) Controls the Trade-off

Optimal at P = 50–75% (default 50%); P = 100% (UNDIAL-like) is strictly worse

Deployment-Friendly Training

8-bit optimizer is free · small batch preserves utility · LoRA costs ~8 utility pts

Unlearning Improves Utility — Above Base and Oracle

Utility rises above both Full (base) and the retrained Target oracle. The Target was never trained on the forget set — so SHRED doesn't just revert fine-tuning, it reduces hallucination in general.

Thank You!

Questions & Discussion

Next Steps

Efficient training · Localization · Hallucination reduction

Zizhao Hu · University of Southern California · 2026