Brexit doesn't guarantee that migration will come down.