Background

Sometimes the /root/.ssh/authorized_keys and/or /etc/ssh/ssh_config files are hosed/corrupted by a user or possibly by software. This locks authorized users out of the instance.

Answer

Because RightLink does not use SSH, you can repair the SSH configuration by running a RightScript.

Note: Only EC2 is supported at this time, but you could use these scripts to adapt for other environments. Also, because the authorized_keys file is restored with the EC2 instance's key only, you will need to reboot the server to re-setup managed login for users with server_login.

  1. Import the RightScripts from the marketplace:

    Print sshd configuration
    Repair sshd configuration

  2. Run the Print sshd configuration as an Any Script. The output should look similar to:

    ********************************************************************************
    *RS> RightScript: 'Print sshd configuration' ****
    05:08:55: Directory listing of of /root/.ssh:
    05:08:55: --
    05:08:55: /root/.ssh:
    total 12K
    drwx------ 2 root root 4.0K Jun 6 04:53 .
    drwx------ 6 root root 4.0K Jun 6 05:05 ..
    -rw------- 1 root root 388 Jun 6 04:53 authorized_keys
    05:08:55: --
    --
    Directory listing of /etc/ssh:
    05:08:55: /etc/ssh:
    total 168K
    drwxr-xr-x 2 root root 4.0K Jun 6 04:55 .
    drwxr-xr-x 96 root root 4.0K Jun 6 05:08 ..
    -rw-r--r-- 1 root root 123K Apr 2 11:48 moduli
    -rw-r--r-- 1 root root 1.7K Apr 2 11:48 ssh_config
    -rw------- 1 root root 672 Jun 6 04:55 ssh_host_dsa_key
    -rw-r--r-- 1 root root 611 Jun 6 04:55 ssh_host_dsa_key.pub
    -rw------- 1 root root 227 Jun 6 04:53 ssh_host_ecdsa_key
    -rw-r--r-- 1 root root 183 Jun 6 04:53 ssh_host_ecdsa_key.pub
    -rw------- 1 root root 1.7K Jun 6 04:55 ssh_host_rsa_key
    -rw-r--r-- 1 root root 403 Jun 6 04:55 ssh_host_rsa_key.pub
    -rw-r--r-- 1 root root 302 Jan 10 2011 ssh_import_id
    -rw-r--r-- 1 root root 2.5K Apr 24 00:38 sshd_config
    05:08:55: --
    Contents of /etc/ssh/ssh_config:
    --
    05:08:55: # This is the ssh client system-wide configuration file. See
    # ssh_config(5) for more information. This file provides defaults for
    # users, and the values can be changed in per-user configuration files
    # or on the command line.
    
    # Configuration data is parsed as follows:
    # 1. command line options
    # 2. user-specific file
    # 3. system-wide file
    # Any configuration value is only changed the first time it is set.
    # Thus, host-specific definitions should be at the beginning of the
    # configuration file, and defaults at the end.
    
    # Site-wide defaults for some commonly used options. For a comprehensive
    # list of available options, their meanings and defaults, please see the
    # ssh_config(5) man page.
    
    Host *
    # ForwardAgent no
    # ForwardX11 no
    # ForwardX11Trusted yes
    # RhostsRSAAuthentication no
    # RSAAuthentication yes
    # PasswordAuthentication yes
    # HostbasedAuthentication no
    # GSSAPIAuthentication no
    # GSSAPIDelegateCredentials no
    # GSSAPIKeyExchange no
    # GSSAPITrustDNS no
    # BatchMode no
    # CheckHostIP yes
    # AddressFamily any
    # ConnectTimeout 0
    # StrictHostKeyChecking ask
    # IdentityFile ~/.ssh/identity
    # IdentityFile ~/.ssh/id_rsa
    # IdentityFile ~/.ssh/id_dsa
    # Port 22
    # Protocol 2,1
    # Cipher 3des
    # Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
    # MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
    # EscapeChar ~
    # Tunnel no
    # TunnelDevice any:any
    # PermitLocalCommand no
    # VisualHostKey no
    # ProxyCommand ssh -q -W %h:%p gateway.example.com
        SendEnv LANG LC_*
        HashKnownHosts yes
        GSSAPIAuthentication yes
        GSSAPIDelegateCredentials no
    05:08:55: --
    --
    Contents of /var/spool/cloud/meta-data/public-keys-0-openssh-key:
    05:08:55: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzNbV8eLt8krYY2xoC1vbEpbW8zx1s4aRyJ0dOgT1AIlgLccE3uxpy1ec0x8csMIIT1tiDa5qNfNDCfQ27KSZQD0in5hz0x71EYGs3ofsUKsrAzQ2C81KHVJc7oX2RgCOVqHLJrT9jx7aDfoNgsHbs9vX9Yc/A8MIGTyZSCNiI36QVB97qZYTanrajKhtNnevKhYumuSWBcbYwAPW89nKCkJ/Lt5vQY2jCENqvAqDLziQ8CBV0E0mj3UHGABeAn8bFUSxFZ2hXV9X5HCxnb1bIH9MeIhWpS4z1MmWaMfOCf1me8UI7BwBBRZmRoGeCr6+yGE8f1WbQAkkKXCz4H1bF oss-ap
    05:08:55: --
    Contents of /root/.ssh/authorized_keys:
    --
    05:08:55: csh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzNbV8eLt8krYY2xoC1vbEpbW8zx1s4aRyJ0dOgT1AIlgLccE3uxpy1ec0x8csMIIT1tiDa5qNfNDCfQ27KSZQD0in5hz0x71EYGs3ofsUKsrAzQ2C81KHVJc7oX2RgCOVqHLJrT9jx7aDfoNgsHbs9vX9Yc/A8MIGTyZSCNiI36QVB97qZYTanrajKhtNnevKhYumuSWBcbYwAPW89nKCkJ/Lt5vQY2jCENqvAqDLziQ8CBV0E0mj3UHGABeAn8bFUSxFZ2hXV9X5HCxnb1bIH9MeIhWpS4z1MmWaMfOCf1me8UI7BwBBRZmRoGeCr6+yGE8f1WbQAkkKXCz4H1bF oss-ap
    05:08:55: --
    05:08:55: Script exit status: 0
    05:08:55: Script duration: 0.416071
    05:08:55: Chef Run complete in 0.43177 seconds
    *RS> Duration: 5.18 seconds
    *RS> completed: Print sshd configuration
    

    Notice how the public key does not much (I intentionally put a typo in the key type, csh-rsa instead of ssh-rsa.

  3. Now, run the Repair sshd configuration as an Any Script. This will replace the key from the meta-data cache and overwrite /etc/ssh/ssh_config with a stock setup for PKI; example output:

    ********************************************************************************
    *RS> RightScript: 'Repair sshd configuration' ****
    05:16:35: Restoring public key for instance
    05:16:35: `/var/spool/ec2/meta-data/public-keys-0-openssh-key' -> `/root/.ssh/authorized_keys'
    05:16:35: Restoring a stock-standard ssh_config
    05:16:35: `/var/cache/rightscale/right_scripts_content/rs_attach70199601467640/ssh_config' -> `/etc/ssh/ssh_config'
    05:16:35: Restarting sshd.
    05:16:35: sshd: unrecognized service
    05:16:35: ssh stop/waiting
    05:16:35: ssh start/running, process 20717
    05:16:35: Done.
    05:16:35: Script exit status: 0
    05:16:35: Script duration: 0.40479
    05:16:35: Chef Run complete in 0.419857 seconds
    *RS> Duration: 5.21 seconds
    *RS> completed: Repair sshd configuration
    

If either of these files were indeed the issue with logging in via SSH, you should now be able to login again (reboot the server if you need to update managed login).