Can a neighbour refuse a party wall agreement?

No, provided the correct procedures are followed your neighbour cannot stop you undertaking works which you are authorised to undertake by the Party Wall Act.