What are the high field strength and large ion lithophile (HFS or HFSE & LIL or LILE) elements?
What was the likely composition of Earth's early crust (how did crustal composition evolve)?
Why silicon is abundant in earth surface?
Why do felsic materials have lower melting points than mafic?
In earth's (upper) mantle, most Mg is inside the mineral olivine, with some in other minerals such as clinopyroxene and orthopyroxene. In contrast, most Al is in the minerals plagioclase, spinel, and garnet. There could be a tiny amount of Al in the pyroxenes as well.
When you melt the solid mantle rock that is composed of all of the minerals I listed above, some minerals melt at lower temperatures than the others. So the magma composition that is formed more closely resembles those minerals that melted.
As you can guess, the Mg-rich minerals are refractory (that is, they are harder to melt). The Al-rich minerals are usually very easy to melt. So whenever you have magma that comes from the mantle into the crust, it will have more Al than Mg. Over the billions of years of Earth's crustal evolution, quite a lot of Al (along with Na, K, Fe, Ca, and Si) accumulated in the crust, relative to the amount that was in the parental mantle.