At the very least, you should always put the scope and amount of the retainer in writing, subject to very few exceptions that allow for different methods of confirming services.
It is equally important that if you decide not to represent a prospective client, to send a non-engagement letter. The purpose of a non-engagement letter is to dispel any notion in the mind of the prospective client that you represent him or her. Be sure to reaffirm potential limitations that you brought to the individual's attention so they are not placed in a disadvantaged situation. Non-engagement letters should not include an opinion regarding the merits of the matter you have declined.