How to Choose the right Drivers

I have read hundreds of tutorial on speaker building but non of them give a clear description on how to choose the right drivers. I have a feeling that it is possible to make a High Quality with any kind of driver but I want to make sure there isn't a process or anything about how to choose the right driver. I want to make a 3-way loudspeaker, Sub-woofer, Mid-range, Tweeter, for my dorm room and don't want to spend much over $100. I understand frequency response graphs to some extent and I was under the impression that I should be looking for a driver whose frequency response graph had a steady linear increase and as flat as possible at its peak, but there isn't a single driver whose near that perfect. even those $400-500 dollar ones.

I decided to go and look at other peoples builds and look at all there specs they had on their drivers and loudspeaker and I found found guy who make a granite loudspeaker and their frequency response graph for the loudspeaker look very ideal so I went and looked at spec sheet and the graph for the tweeter they used and it looked horrendous. It was wicked jumpy and kinda made a parabola

This is the granite speaker build I looked at.
DIY Granite Speakers with Seas G17REX/P and 27 TFFC Drivers

This is the specs for the tweeter they used in the build.
H0881-06 27TFFC

could someone help me make sense of this so I can have a clear set answer on what drivers I should choose for my build I want to do.

Thank You