Cinnabar and other mercury minerals need some care. Any liquid mercury which is spilt gets into cracks and crevices and is essentially impossible to remove. This is dangerous because the metallic mercury literally evaporates into the air and contaminates the air.

Another set of minerals that would make me somewhat nervous are radioactive minerals. In general even such specimens can be handled safely with a few precautions since the raw samples have low radioactivity. You just are not going to find big crystals of uranium-235 metal while mineral collecting.

If you can afford a small upfront expense micromount collecting is a neat way to go. A large collection of minerals will take up a small space. Micromounters swap and give away specimens much more generously than collectors who want cabinet size specimens. Active mines are virtually impossible to get into. However lots of old mines have tailing piles where good micromount specimens can be found. Lastly I'll also point out that there are several thousand minerals. Only a couple of hundred have good cabinet size specimens.