Prime minister says he has never ruled out visiting the southern neighbour and will travel where it is convenient.