[1] Holthuijsen, L. H. Waves in oceanic and coastal waters. Cambridge University press, 2010.
Waves are ubiquitous, except on land ;-). Waves in the open sea are a mix — a superposition — of waves in different directions.1 The dominating direction of large waves is, after a while, the wind direction; but that's not absolute. You have some omnidirectional background "noise" of chaotic movement as well as waves which have traveled a long distance and have nothing to do with local conditions.
What you see at the shore is the part of the waves on sea which run towards the shore. That's of course less when the wind is off-land, but it is not zero. In addition you have refraction on the coast line; the waves "bend around" the coast line.
In essence, that you see a dominating wave direction towards the land at any wind direction is a result of zero waves emanating from the land.
Other answers have already covered the mechanics of waves better - but I thought this additional misunderstanding in the question was worth clearing up.