I keep mine on, for both the stack-o-leaves issue, and the sinking in soft ground issue. I have lost a basket before, but I have also found some along the trail. Periodic checking to make sure they are tight has become routine.
If on the AT, when picking up others' lost baskets, rather than trashing them out, leave them at the next shelter you reach. This way, others who have lost a basket may be able to use it. To some this may be trail magic while to others, it's leaving trash behind. Since I like having baskets on my poles, I try to think of it as trail magic.