First thing needed is to simplify the coordinates under consideration. People being nice and pay being good must come later. It is also helpful to dissociate your job with swadharma. Swadharma does not impel you to leave one job and take up another. It makes us aware of the divine impulsion in us, the inner direction that would be the expression of the Divine within us. Your swadharma is calling you to undertake challenges in the fields of knowledge. The outer details, the job etc are not the real thing. You can see if your present job gives you the scope of this challenge. Otherwise, instead of throwing away a job which is giving you financial stability you can take up the study of physics on you own and see first whether this is what you truly want to engage full-time, in studying this field. Job, money are later considerations. Abandoning all dharmas comes when you are being called to undertake the Divine Work directly. If that be your goal, to discover and serve the Divine then, yes, you could continue with this job, any job for that matter and practice equanimity and nishkama karma, do it as an offering and in the spirit of Divine Service.
Affectionately,
Alok Da
