Just dropping a link to this blog which has a solution, but not the most simple one 😅
https://luke.geek.nz/azure/accessing-keyvault-azure-devops/
Essentially they are suggesting obtaining the IP of the build agent when it runs, then add it as an allowed IP via the 'az' client, and then remove it after. Jeez!
I'm going to
role roll with opening up the firewall to get things moving and putting trust in the IAM access security.
I think the way forward will be to have a hosted build agent with a static IP... but open to other suggestions 🙂