One of the biggest philosophical questions of all time – which has been a great stumbling block to many on their spiritual journey – is the problem of why an all-powerful, all-loving god allows suffering and evil to exist in the world. The usual reply supplied by explicitly religious sources is that God gives us the capability of free will, and if he were to impose limits on our will it then it would cease to be free at all.
But this then gives rise to the equally unfathomable questions: is free will really so much more important than the creation of a wholesome society? Couldn’t God at least have created us with morally superior natures, to enable us to freely choose more wisely than we do?
One of the great minds of the 20th century, Thomas Troward, gives us the answer, in his book The Word and the Law. The following is a substantial quote and may seem overly philosophical or intellectual to modern readers. I urge you to read it nevertheless, as it is the only satisfactory answer to the question that I have ever come across. My own very brief summary will appear at the end.
“Then the question very naturally suggests itself: Why did not God create us so that we could not think negative or destructive thoughts? And the answer is: Because He could not. There are some things which even God cannot do. He cannot do anything that involves a contradiction in terms. Even God could not make twice two either more or less than four. Now I want the student to see clearly why making us incapable of wrong-thinking would involve a contradiction in terms, and would therefore be an impossibility. To see this we must realize what is our place in the Order of the Universe. The name "Man" itself indicates this. It comes from the Sanskrit root MN, which, in all its derivatives, conveys the idea of Measurement, as in the word Mind, through the Latin mens, the faculty which compares things and estimates them accordingly; Moon, the heavenly body whose phases afford the most obvious standard for the periodical measurement of time; Month, the period thus measured; "Man," the largest of the Indian weights; and so on. Man therefore means "The Measurer," and this very aptly describes our place in the order of evolution, for it indicates the relation between Personal Volition and Immutable Law.
If we grant the truth of the maxim "Nature unaided fails" the whole thing becomes clear, and the entire progress of applied science proves the truth of this maxim. To recur to an illustration I have employed in my previous books, the old ship-builders thought that ships were bound to be built of wood and not of iron, because wood floats in water and iron sinks; but now nearly all ships are made of iron. Yet the specific gravities of wood and iron have not altered, and a log of wood floats while a lump of iron sinks, just the same as they did in the days of Drake and Frobisher. The only difference is, that people thought out the underlying principle of the law of flotation, and reduced it to the generalized statement that anything will float, the weight of which is less than that of the mass displaced by it, whether it be an iron ship floating in water, or a balloon floating in air. So long as we restrict ourselves to the mere recollection of observed facts, we shall make no progress; but by carefully considering why any force acted in the way it did, under the particular conditions observed, we arrive at a generalization of principle, showing that the force in question is capable of hitherto unexpected applications if we provide the necessary conditions. This is the way in which all advances have been made on the material side, and on the principle of Continuity we may reasonably infer that the same applies to the spiritual side also.
We may generalize the whole position thus. When we first observe the working of the Law under the conditions spontaneously provided by Nature, it appears to limit us; but by seeking the reason of the action exhibited under these limited conditions, we discover the principle, and true nature, of the Law in question, and we then learn from the Law itself, what conditions to supply in order to give it more extended scope, and direct its energy to the accomplishment of definite purposes. The maxim we have to learn is that "Every Law contains in itself the principle of its own Expansion," which will set us free from the limitation which that Law at first appeared to impose upon us. The limitation was never in the Law, but in the conditions under which it was working, and our power of selection and volition enables us to provide new conditions, not spontaneously provided by Nature, and thus to specialize the Law, and disclose immense powers which had always been latent in it, but which would for ever remain hidden unless brought to light by the co-operation of the Personal Factor. The Law itself never changes, but we can specialize it by realizing the principle involved and providing the conditions thus indicated. This is our place in the Order of the Universe. We give definite direction to the action of the Law, and in this way our Personal Factor is always acting upon the law, whether we know it or not; and the Law, under the influence thus impressed upon it, is all the time re-acting upon us.
Now we cannot conceive any limit to Evolution. To suppose a point where it comes to an end is a contradiction in terms. It is to suppose that the Eternal Life Principle is used up, which is to deny its Eternity; and, as we have seen, unless we assume its Eternity, it is impossible to account either for our own existence or that of anything else. Therefore, to say that a point will ever be reached where it will be used up, is as absurd as saying that a point will be reached where the sequence of numbers will be used up. Evolution, the progress from lower to higher modes of manifestation of the underlying Principle of Life, is therefore eternal, but, in regard to the human race, this progress depends entirely on the extent to which we grasp the principles of the Law of our own Being, and so learn to specialize it in the right direction. Then if this be our place in the Universal Order, it becomes clear that we could not occupy this place unless we had a perfectly free hand to choose the conditions under which the Law is to operate; and therefore, in order to pass beyond the limits of the mineral, vegetable and animal kingdoms, and reach the status of being Persons, and not things, we must have a freedom of selection and volition, which makes it equally possible for us to select either rightly or wrongly; and the purpose of sound teaching is to make us see the eternal principles involved, and thus lead us to impress our Personality upon the Law, in the way that will bring out the infinite possibilities of good which the Law, rightly employed, contains. If it were possible to do this by an automatic Law, doubtless the Creative Wisdom would have made us so. This is why St. Paul says: "If there had been a law given which could have given life, verily righteousness should have been by the law" (Gal. iii, 21). Note the words "a law given," that is to say, imposed by external command; but it could not be. The laws of the Universe are Cosmic. In themselves they are impersonal, and the infinite possibilities contained in them, can only be brought out by the co-operation of the Personal Factor. It is only as we grasp the true relation between Jachin and Boaz, that we can enter into the Temple either of our own Individuality, or of the boundless Universe in which we live. The reason, therefore, why God did not make us mechanically incapable of wrong thinking, is simply because the very idea involves a contradiction in terms, which negatives all possibility of Creation. The conception lands us in a reductio ad absurdum.”
In summary, as Troward has already explained at length in his other philosophical tomes – primarily The Edinburgh Lectures on Mental Science – we must firstly understand the true nature of God in order to understand man’s place in creation. Troward explains that God or the “Universal Spirit” is not only infinite, but entirely impersonal and undifferentiated. It is infinitely intelligent, infinitely powerful, infinitely loving and so on – but it is without any personal volition of its own. Because it is life in itself, it seeks only one thing – the communicating and increasing of that life. This is in its very nature, but to attribute any other specific motives to it would be to place a limitation upon the limitless. We cannot therefore talk about “God’s will” in any manner other than the communication of itself in new forms of life, and the eternal evolution of that life. The Universal Spirit cannot be said to make decisions in its infinite form, because to prefer one course of action would mean that it must repudiate the opposite course of action. Doing so would lead us to the conclusion that God prefers one thing to another – this would, of course, be placing a limitation on the infinite because if God prefers A, then God is not someone who prefers B. Anything which is limited in any way cannot be infinite, therefore the Universal, unmanifested Spirit cannot be partial in any of its dealings.
The only way in which the eternal limitless can make such decisions is by becoming limited – that is, by taking on physical incarnation. After various incarnations through the animal kingdom, eventually a soul becomes sufficiently evolved to take on human incarnation. It is at this point that it the soul is invested with the capability of free will, rather than the pure instinct that drives the lower beasts. By being granted this new ability, we become the decision making faculty of God. As the Universal Spirit cannot limit itself by making decisions while in infinite, unmanifested form, it delegates the role to its creation, man – the measurer. If man’s faculty of free will were taken away, he would be no man at all – but would be effectively still in the realm of the animals. His entire mission – that of being the decision-making faculty of God – would be nullified by removing the faculty of free will.
A soul’s transition from animal to human must come in the relatively early stages of its evolution, because a necessary part of attaining human perfection is to be incarnated multiple times in a human body. To impose human perfection as an entry requirement for incarnation would be a contradiction in terms – it would be like trying to improve academic grades by refusing anyone entry to school until they had already graduated.
If you did not fully appreciate the Troward quote, with this summary in mind I urge you to go back and read it again for a deeper understanding of the matter. Search all the philosophy of all the world and you'll never find a better explanation than his.