You drag them down by having a population of 117 (or whatever). You drag them down by being raided. You drag them down by being there, basically.
Travian is a game where your appearance matter. And you made the CF alliances look weak.
No one has any entitlement to be in an alliance. Period. Just because we all happened to come from CF, we don't have to stick together over there. Sure, it'd be great. But if some people want to win the game, why shouldn't they try to?
It isn't like you weren't offered a place in another alliance. You were just offered a place in an alliance for the not-so-serious players. The ones who don't really, really want to try to win. If that doesn't fit your profile, you're more than capable of proving yourself, and moving to the other alliance.