These weather records allow other kinds of examination besides getting a global average temperature - mapping of regional changes, geographic ones and all the way down to the conditions at individual sites, including their suitability for inclusion in global temperature record calculations. Badly sited ones can be excluded, step temperature changes from changing location or changes to existing location can be adjusted for to achieve better accuracy.
These records don't exist in isolation - there is a broad understanding of climate and weather processes that goes with them. They are also supported by satellite data, such as microwave sounding that allow remote estimation of atmospheric temperatures; whilst they tend to show higher response to El Nino Southern Oscillation, the rates of warming are broadly consistent with weather station records.
It is a reliable observation that in between nearby weather stations sharing similar geography there are similar weather conditions, including similar maximum and minimum temperatures. Consistently different conditions in the spaces between would be noticed and we cannot expect such differences to be confined only in those between spaces; if there were significant variations in the temperatures and rates of change over time in the spaces between locations it would be happening at those locations as well and that would show up in the weather station records. What we find in those records is consistent with global warming.