Do you think it's a bit much for charities to want you to give your bank details, so they can 'direct debit"?

Sounds like he was trying to scam you. You made the right choice not to provide your banking information. Even though he showed you papers that seemed official, chances are that they were not. Anyone can have documents printed up to make it look real. Even fake identification.