So an all pass filter is actually a phase only filter, it doesnt do anything to the frequency response at all. The only thing an all pass filter does is rotate the phase through a time dependent adjustment, frequency wise the signal remains unchanged.
If you remove everythign from the mid range part of the crossover and replace it with a high pass and low pass filter in series you will get a much simpler crossover network and a more easy to work with frequency response.
As a starting point I would match the high and low pass filters you have put on the tweeter and woofers e.g. if the woofers are low passed at 400hz add a high pass at 400hz to the mid drivers. Likewise if the tweeter is high passed at 3000hz add a matching low pass of 3000hz to the mid drivers.
Try to get clear separation of the different frequency bands so most frequencies only come out from one driver range and frequencies at and near the crossover range only come from 2 driver sets. At the moment it looks like the woofers are playing up high enough that they overlap the bottom and of the tweeter meaning the upper mid range is being played by all 3 driver sets, this will add a lot of distortion to that frequency band
Give this alteration suggestion a go or if you want me to take a look at the crossover for you reach out via message on here
Knowing your own ears and not wasting time trying to fix something you wont notice is perfectly valid and a sensible approach to managing time and workload