I find all this public key and private key concept is confusing to general people. A private key is an actual key that you can use to unlock an encrypted message. A public key is actually a lock that you gave someone so that he can use to encrypt message and only you can open it.
It's still "my" key even though you're signing or rather encrypting a message with it.