First, I think it is a really cool idea! The large baffle will actually help you not lose as much bass via baffle step. So that is a big plus, as it’ll help keep crossover costs down. The only thing I would think that would be problematic is finding thin enough speakers. If it is 4″ deep You probably will only have about 3.25″ to work with (about 0.25″ for the recess and 0.5″ for the rear material. Plus you will want it to breathe, so a little less than that (3″ maybe). I think that will be the most limiting factor of a build like this.
I did a project a few years ago that was only about 3″ deep that might work. If this is anything like you are thinking. Obviously I am only talking about the front part of that speaker.