Early nerves among a sell-out crowd were made worse as Oxford took the lead and news filtered through of a goal for Wycombe Wanderers at Torquay.
The sending off of goalscorer Williams, however, sparked a Town fightback and they went in at half-time in front thanks to goals from Marquis and Toney.
Kouo-Doumbe made the game safe just after the break to cap a superb achivement by Chris Wilder since his arrival from Oxford earlier in the year.
Wycombe kept their side of the bargain at Torquay which means Bristol Rovers drop into the Conference after defeat at home to Mansfield.