That completely depends on the circumstances.
If their T&Cs say that they charge a fee for counting coins, e. g. when they are over a certain amount or number, then sure, this is lawful.
If the fee depends on whether you are a customer with them or not, and they clearly say so, as well.
If you weren't aware of that fee and they already started processing the coins before you was aware about that fee, they might not be able to stop the process.
While I've never seen anything obviously posted in a bank about their policies on accepting coins, I think the wisest course of action when you go into ANY bank you've never dealt with before or don't have an account at would be to ask at the teller window before attempting a transaction.
This is especially the case with cashing checks - many banks charge you a fee to cash a check drawn on one of their accounts if you're not a customer of the bank, so it's best to clarify that before conducting your transaction. That way you know up front and can make an informed decision.
For the record, it isn't unusual for banks to have some kind of fee associated with loose coins (your post doesn't specify so I'm assuming they weren't rolled) because you're taking up a teller's time and bank resources to roll and package them.
As an alternative, just about every big grocery store in America now has those kiosks for exchanging coins for cash. The fees are posted up front (they're not terrible considering they're saving you the time and aggravation of counting and rolling them yourself) and it's pretty quick.
If the bank promised to give you the full value of your change, and then intentionally failed to fulfill that promise, then that's fraud, which is a crime.