You know, I’d suspect you’d be fine. When over 99.99% of Macs have these security features turned on, what are the odds that a bad actor is going to take the time to target the 0.01% that don’t?
You’d be like that one kid at school whose crazy parents didn’t have him vaccinated – as long as everyone else is still properly immunized, there’s just not going to be anything out there nasty for him to catch.
I’d like to know just how much faster this makes your Mac, though. I’m not going to strip security out of my machine just off of some guy’s general impressions.