Late fee clauses are popular with landlords—they motivate the tenant to pay on time, and they compensate the landlord for the value of his time spent demanding the late rent (and the loss of use of the rent money while it’s still unpaid). But to be valid and enforceable, late fees must meet two requirements: They must be reasonable, which means they must closely approximate the actual losses the landlord suffers when the rent is late; and they must be part of the lease or rental agreement.
Your question initially concerns the second requirement. After landlords and tenants sign a lease or rental agreement, any variations or additions should be done in writing, signed by both sides. To drive that part home, most rental documents contain a clause that says so—any changes must be written down and signed.
If your agreement is missing this clause, you may find that your landlord will argue that his oral modification of the agreement (he “mentioned” late fees after the lease was signed) is enforceable, because you didn’t protest.
Your landlord is not likely to prevail on this argument. First, in some states, by law any late fee policy must be written. Even if this isn’t so in your state, the lack of a written policy will hinder the landlord. The “modification” (addition of a late policy) is to your detriment, which means that a judge will be hesitant to apply it to you on the word of the landlord alone (if the modification were in your favor, the landlord’s attempts might be met with more success). And secondly, the landlord’s failure to enforce his clause over three years is telling—had he really intended to be compensated for late rent, he should have acted each month when you were tardy. Finally, $5,000 in late fees over three years is an enormous sum, which cannot possibly bear a reasonable relationship to the landlord’s actual losses when you paid late. That alone should defeat the landlord’s attempt to collect it, either by using your deposit to cover it, or by doing that plus suing you in small claims court for the balance.
Most states provide for an orderly method for landlords to return deposits, which must be used only for unpaid rent and damage beyond wear and tear. (Late fees are sometimes called “additional rent” in leases, which enables landlords to use the deposit to cover late fees, but as explained above, your landlord’s policy probably won’t pass legal scrutiny.)
You’ll need to check the security deposit laws of your state to learn what you must do when a landlord refuses to return a deposit. Start with a demand letter, then sue in small claims court for the return of the deposit if the landlord isn’t forthcoming.