~/.ssh/authorized_keys is more than just a file listing valid public keys which can be used for server access. Each key can be preceded by a list of options:
~/.ssh/authorized_keys