I've worn mine hiking, caving, camping, gardening, everywhere (still do). I also have this idea of not removing my ring, ever, for any reason (even when I had surgery, I made the nurses tape it on). I've only had it off my finger once since my wedding day sixteen years ago, and that was on my tenth anniversary to get it cleaned and inspected.
Anyway, if it's a good quality setting with plenty of prongs to hold the diamond (mine has six) and the diamond doesn't stick out like sore thumb (I got mine set close in so it wouldn't snag on anything) you should be fine. But I remember when I got mine and worried about it when I went caving with all the grime and goo. So I understand.