Cam, in my opinion, you've misevaluated the military/civilain rate of the Hidden Villages. IMO, they'ed be more likely to have a military value of closer to 30-40% of the city population (assuming all Hidden Villages are the same size, which they aren't). You have to remember that the Villages' main income maker is mercenary work, not farming or industry based like most nations in the real world, which would demand that a greater ratio of Mil:Civ existed to continue and increase the income. Even using a low 25% rate of Mil:Civ, to have a 80,000 man army ready, you'd only need a combined population of at least 320,000 to support it.
Correct me if I'm wrong, but isnt that 'neutral' village also supplying it's chakra samurai into the army, further skewing the needed population of the Ninja Villages? Factor that in and the Villages might have even less than 320,000 combined population.
One last thing, you are confusing the Ninja Hidden Village population as the ONLY population for the entire country they reside in. This is completely incorrect as each of the ruling Daimyo has a standing army of samurai and we know of other places like a warrior monk temple which the immortal duo slaughtered. Remember that Fire Nation is a big country in the Naruto Universe and for it just to have only Kohona in it... well, that's not much of a hidden village then is it (course it's not really hidden when everyone has to know where it is to hire ninja...)