Any suggestions on how to get this information from the landsea mask grid?
Thank you in advance.
To rephrase your question: How can I determine the orientation of the coastline in a gridded dataset?
Well, one way is using mathematics. Let's call the boolean landmask variable $L$, where $L=\{^{0 \text{ if land}}_{1\text{ if water}}$ Then the gradient is $$\nabla L =\{^{<\frac{\delta L}{\delta x},\frac{\delta L}{\delta y}> \text{ at the coastline}}_{\vec{0} \text{ not at the coastline}}$$ Let's look at just the coastline. The angle is $\arctan2(\frac{\delta L}{\delta x},\frac{\delta L}{\delta y})$.
Now if you want to find out the angle difference between the coast and the wind, then you can use the properties of the dot product to find the angle difference. That is $$\delta \phi=\arccos\left(\frac{\nabla L \cdot \vec{v}}{|\nabla L||\vec{v}|}\right)=\arccos\left(\frac{u\frac{\delta L}{\delta x}+v\frac{\delta L}{\delta y}}{\sqrt{u^2+v^2}\sqrt{\left(\frac{\delta L}{\delta x}\right)^2+\left(\frac{\delta L}{\delta y}\right)^2}}\right)$$