The board voted, 4-2, to fire Downs, who had been on administrative leave since December.