It's an uncomfortable fact that leaving the European Union in a way that is true to the referendum result is becoming harder and harder to deliver.