The optimization tactic worked maybe even better than expected. Levels that don't have procedurally generated hills are now effectively loaded instantly:
All those raised flat surface and walls have been reduced to a few quads and box colliders, and those rounded edges are created as single objects as well.