I heard this sentiment from co-workers and friends who didnt grow up eating goat. I think it boils down to two different things:
1. "meat musk" (lack of a better word, lol): People dont realize how pungent beef can be, its just that most Americans are accustomed to it. You can really smell this with grass-fed beef. I think goat/lamb meat is "musky" for the same reasons, as they eat mainly from foraging/hay.
2. preparation/cooking: If you plan on cooking goat like a steak, its not a gonna taste good. Generally, most dishes require that the goat is simmered or slow-cooked to make the meat more tender. Spices also make a big difference here with the dish.