it will apply both.
they would be effectively merged, just like if you were applying two regular ADM policies or two ADMX policies.
but remember that deny takes precedence, so if you deny something in the ADM policy then allow it in the ADMX policy it'll still be denied to windows vista and windows 7 machines as they would read both policies.
if you don't like the idea of applying both, and letting the machine merge them, (perhaps because it'll increase the boot time as it will see two policies to apply, then...
you can limit the application either through the use of different OUs to keep windows 7 and XP machines separate in your AD structure.
create an OU for xp machines and apply the ADM policies just to this OU
create an OU for w7 machines and apply the ADMX policies just to this OU.
That's the way that I'd recommend,
or you could change the security settings on the policy to say that some machines have no read access to the policy, (so you let Xp machines see the ADM policy and deny access to windows 7 machines) visa versa for the ADMX policies.
that's the way I wouldn't recommend (because you'll have to create and maintain long lists of machines in the security policy, or in groups), but it is useful if you want to apply a policy to just a single machine inside an OU with lots of machines.
(you can do that per machine, or create groups of machines and put a group in the security settings).
