Remote access android phone without permission

Can someone really access my Android phone without permission, and how do I stop it?