For example, olivine is defined in your chart as "no streak, cannot be scratched, green, grainy". I know of many other minerals which are exactly like that (epidote is one example). I also know of olivine which doesn't satisfy these conditions - not all of them are "grainy" (whatever that even means?).

You can write a custom flowchart for the minerals in that specific box, but that would be rather boring because most of them are just silica in various forms. Anything more complex than that would lead to false positives and false negatives for anything that slightly deviates from the rubric (which is almost everything).