For me "auf jemanden schwören" means that you trust someone when he or she says something, that you value their opinion, that you would do what they say and so on and so on. That's why I took "listen to you" as a translation. The other one (swear by you) is more of a literal way to put it
