Temperature gradient from crust to core. Recent research have improved the accuracy, but the principle in the figure remains (Annual Review of Earth and Planetary Sciences, 24(1), 15–40)

So why do you experience the temperature colder underground when you are digging?

The temperature at depth is the average temperature for day and night or even the year. The energy from the sun reaches only the uppermost soil layer that absorbs heat but the thermal conductivity is relatively low and it will not transport the heat down. The ground is reluctant to change its temperature regardless of variations on the surface.

How strongly you experience the temperature difference depends on where in the world you are and the local conditions. In some areas, the ground remains frozen, known as permafrost, even as the temperatures in the air can be pleasant during the summer and the surface layer melts. The depth of the suggests that it's been formed over a long time. It reaches down to the depth where the mean temperature over hundreds or thousands of years is 0°C. At larger depths, the heat from the Earth's interior takes over and melts the frost.

Temperature in the active layer and permafrost. The thickness of active layer is less than a couple of meters and the thickness of the permafrost depends on its age, the bedrock and the distance to the coast. (By German Wikipedia contributor, "HylgeriaK")

See the plot, linked by @JeopardyTempest in the comments to better understand the variations. Plotted on another scale and depth, annual variations would probably be seen as well.