George Adam, the minister for parliamentary business, has joined calls for the restrictions to be relaxed as ‘things have moved on’