If the horses are getting spooked by a hat, they too are unqualified to be on the trail.
Horses tend to shy from things they dont recognize... Usually no matter how strange you look, SPEAKING,,, greating of the day for instance, will eliminate 90% of the human/horse interaction. They hear your voice and recognize it as human,, so they ignore you. The ones that wont shouldnt be on trail.
Goes both ways,, other trail users have some responsibilities, but so do the animal owners.
One of the things I like about endurance horses,,, most have seen it all and arnt phased. I do all my training in public parks with trails shared by bikes, joggers, women pushing those 3 wheeled jogging baby carriages. Those are freaky to alot of horses.
Alot of people want to freeze and step off trail,,, usually that makes things worse.

Sadly I do have a few aquainteneaces in my riding circles that expect the world to walk on egg shells because they are on a horse.... Honestly you cant expect that on a public trail, and said rider/horse combo shouldnt be there.

Vast majority of dog issues I have had on trail, dogs wernt really being aggressive but horse doesnt know that and can create a very dangerous situation when the dogs run at or under the horses. Seems they always want to run around to their back legs, Some horses will try to spin to keep them head on, some like mine have really good rearview aim,
Have had more than 1 conversation with an off leash dog owner that went like this.

Me: "get your dog he's sposed to be on a leash"
"Oh fluffy wont bite"

WACK !!!!! dog gets drop kicked end over end.

Me: I wasnt worried about my horse getting bit.