Simon Hughes told BBC Breakfast that parties in government were often punished in by-elections.