Yes it is a common experience once one enters the path of Yoga and takes it in real earnest. One likes to be either alone or in the company of those who strengthen the aspiration. It is so because the average human being is full of desires and leads, more often than not, a superficial life centered around small, ignoble things. When a genuine spiritual aspirant meets such people an interchange follows, a mixture of consciousnesses takes place and as a consequence one loses the spiritual gains. It is best therefore to minimise these social interchanges. But whenever one has to, it is best to try and stay conscious in the heart centre and be a smiling witness to the strange and devious ways of the world with its values turned upside down.
Affectionately,
Alok Da