It's an informal expression often used to describe someone who suddenly becomes silent and unwilling to talk, especially when they are expected to or being pressed for information. The image it evokes is that of a clam closing its shell tightly, making it impossible to get anything out of it.
For example:
When the police started asking questions, he suddenly clammed up.
