Did more reading. I think I understand, but by no means does this make me an expert on this particular topic.
It's important to note there are two kinds of ACLs:
The issue here appears to be that -- under the hood -- Linux doesn't have a competent extended attribute engine yet. It's using getfactl/getxattr/setfacl/setattr; the native chown/chmod utilities on Linux don't even support extended attributes yet. Andreas Gruenbacher -- the author of the utilities -- clearly did a great job implementing something that doesn't have great support in the kernel.
Richacls is at present the most promising candidate to bring true, decent ACL support into the Linux kernel. SuSE has supported it since 11.3, but I'm not aware of any others that do.
So what does xattr=sa do? It simply enables handling of extended attributes the way ZFS was designed to do. I strongly suspect the default of "xattr=on" is a bug leftover from early testing that wasn't cleaned up.
I was wondering, will setting xattr=sa have changed the xattr storage method for all files within the dataset (including those created when I was using xattr=on)?
Changing it to xattr=sa will change it for all subsequently-written objects in the dataset, but not existing ones unless modified.
Is there a way that I can check which xattr storage method ZFS is using for particular files?
zdb. However, most people for this kind of issue don't really want to "view" which ones are incompliant and which are not. There's a trivial way to force data to be stored the way you want: migrate it to a filesystem with the appropriate properties set from the get-go. While inconvenient, it's more convenient than trying to implement Block Pointer Rewrite to try to modify data in place...