Well, the memory does acount for something. But you have to realize that AGP is a different port, running at different speeds. Any AGP card will beat any PCI card. PCI just uses a really old interface, and AGP is still of good use, although dying, and rather quickly I might add.
So the AGP will win. However, maybe if you compared a 256MB PCI card to a 32MB AGP card, you would be able to compare, but not a 256MB PCI against a 128MB AGP card.