Just to follow up to this, I put in an 8400GS with 512MB of ram instead, and I can run VDPAU with no artifacts. I made no other changes. It seems to be a problem with myth and the 9500GT and 1GB of video memory as mplayer played the videos just fine.