Have patience and trust in the Grace. What you are having are some experiences on the way. The Divine realisation is a far off thing. It is not like a fruit or reward one gets for one’s effort. The Divine knows what is best is best for us and it is He alone who decides when to give what. Learn to truly surrender. If one is afraid of difficulties or impatient to get realisations then the delay will be much more. Why even notice the difficulties. They are there and may stay for a long time. But the seeker keeps moving forward regardless of their presence. They drop away if you do not give them too much space and importance.
Also human nature does not change by meditation and peace alone. One has to work for the Divine as well as love and surrender. Meditation and experiences do not change much and hence the difficulties tend to return.
Affectionately,
Alok Da