I can't say I've ever noticed or been bothered by it (I don't like the way sandalwood smells, either, by the way).
I've got my signiture rose scent but most people seem so HAPPY when they run into me after a time and say "you smell just like you used to!" so I figure no one's bothered by it.

I wouldn't recommend doing away with yours entirely, though. A signiture scent is nice. I would look into that soap option - that sounds nifty.