Agent memory needs more than trusted-looking sources

cannot safely trust saved memory or retrieved documents just because the source looks d. An core tested four defenses: scoring records by value, requiring , letting newer records replace older ones in a fixed way, and giving credit to records that seemed to lead to success. All four failed when the attacker knew how the defenses worked.

The weak point was that the record itself supplied the signals used to judge it, so the attacker could write fake value, fake support, convenient timing, or fake success. and cost are harder for the writer to fake, but only proves where something came from, not whether it is true. False memory can still enter through a real, d session and pass a source check.

Related tests point to the same pattern: RAG can quote the right text while pointing to the wrong source about 30% of the time, source and metadata can be forged, and a cached fact can become stale even though it still fits the context perfectly. The safer rule is to make memory cheap to store but expensive to use for broader influence, by requiring separate anchored evidence before it shapes an answer outside its original scope.

Key points

  • Four common memory defenses failed against an attacker who knew the rules.
  • proves the source path, not the truth of the content.
  • RAG can return correct text while attaching the wrong .
  • Source and metadata are weak if the writer can fill them in.
  • Memory can cut , but important uses need separate evidence checks.

Sources covering this story (7)

Read original