One of two things worked, i installed the iso to a usb key, and also disconnected the ilo from the network (when i was expecting the auto vs interactive prompt it was "communicating with ilo") and this time it downgraded properly.
Crazy that we have to jump through these hoops to install an OS on a server...