Using Azure can open up a can of worm around security and many customers have many concerns

Microsoft do a lot of things to keep your Team Service project safe and secure, refer to this link for details: Visual Studio Team Services Data Protection Overview.

You can deploy your own build agent which you can have full control and easy to configure your machines to only accept the deployment from that build agent.

Another URL link I found from Microsoft Virtual Learning which might be useful:

Getting Started with Azure Security for the IT Professional

Do IT security concerns keep you up at night? You’re not alone! Many IT Pros want to extend their organization’s infrastructure but need reassurance about security. Whether you are researching a hybrid or a public cloud model with Microsoft Azure, the question remains the same: Does the solution meet your own personal and your organization’s bar for security, including industry standards, attestations, and ISO certifications? In this demo-filled course, explore these and other hot topics, as a team of security experts and Azure engineers takes you beyond the basic certifications and explores what’s possible inside Azure. See how to design and use various technologies to ensure that you have the security and architecture you need to successfully launch your projects in the cloud. Dive into datacenter operations, virtual machine (VM) configuration, network architecture, and storage infrastructure. Get the information and the confidence you need, from the pros who know, as they demystify security in the cloud.

This article is very useful is you need to deploy from a remote server

Configuring on-premises Build server for Visual Studio Online