Worldly life, that is to say, a life in pursuit of material aims brings its own challenges that are mainly outer. The spiritual life brings another set of challenges, not from outside but from within. These challenges are subtler and often difficult even to detect because the enemy here is within.
Yes it is true that spiritual life, when lived sincerely brings abiding peace and happiness which is again independent of all outer conditions. The happiness of a person living a life for material gains is fleeting because it is dependent upon outer objects that are, by their nature transient and temporary. The material life is like drinking something that is sweet in the beginning but becomes bitter as you reach the end. The spiritual life is more difficult in the beginning but becomes sweeter and sweeter as you reach the end.
Affectionately,
Alok Da