Brexit fate 'in the hands of the British'