This is just a personal story and it might be a disturbing, but I think it gets the point across. My dad, who passed away several years ago, had cancer. When he saw the eye specialist for a regular checkup, the doctor could see that the cancer which had originally been in his prostate, had spread to his brain. He could see the tumour on his optic nerve when he looked into his eyes. It was a very traumatic day because we (mum and I) knew he didn't have long to live. Anyway, my mother, had a checkup for glaucoma about a year later and I was very anxious leading up to the appointment. I had no idea why. I was edgy, irritable, etc. Once I got in there (I needed to drive my mother there), my anxiety subsided. I relaxed. I still didn't know why.
Looking back, I realised that my subconscious mind had associated the eye specialist's rooms with being told that a loved one was about to depart this world in a few months. It was just my subconscious mind playing tricks on me. That's why I was so scared of going back.
Having said all that, do you what you're comfortable with. If people care about you, they'll understand.
