Ssh: connect to host github.com port 22: operation timed out mac

There's nothing worse than losing precious development time thanks to some random sporadic tooling error. It's amazing how difficult it was to find an actual fix for this, although due to the network-related nature of the issue I can imagine there are at least 400 million different fixes and causes.

Out of the blue I was unable to pull or push to any repos. Since I use SSH with Github, I was able to circumvent the issue briefly by changing the origin to https then providing my user/pass when pushing and pulling but all of a sudden that stopped working completely as well. What gives?

So first up, the error. After remaining idle for quite a while, the command would fail with the following:

$ git push
ssh: connect to host github.com port 22: Operation timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Alright... After some Googlin' I was able to find a Stack overflow post mentioning the same issue, with the accepted answer saying to change the repo URL from SSH to https. That was a no-go, but the first command to double check came in handy:

$ ssh -T git@github.com

The post mentioned that should timeout, and it did. However with my repo already on https and it still not working, I had to explore other avenues to fix it. After restarting 4 times with no luck I did more searching and found myself on the following page on Github Help: Using SSH over the HTTPS port.

Hot. More things to try. The first was an altered version of the first command that essentially ran the same check but through a different port. I was supposed to receive a friendly success message back, so I went ahead and gave it a go:

$ ssh -T -p 443 git@ssh.github.com
Hi aesinv! You've successfully authenticated, but GitHub does not provide shell access.

It was like finding a radio linked directly to the forrest rangers after being lost in the forrest for 2 weeks, it was absolutely beautiful. I went ahead and followed the second step listed on the article, which was to update my ssh config to route github connections through the HTTPS port.

# ~/.ssh/config
Host github.com
    Hostname ssh.github.com
    Port 443

Now for the moment of truth. To test the routing was complete I ran the same command I had run before, $ ssh -T [email protected], and saw the same friendly greeting I had before. Hope swelled throughout my being. Then I tried pushing the changes that were waiting politely to be transported safely over to the repository, and what I was greeted with was a truly incredibly sight to behold.

$ git push
Warning: Permanently added the RSA host key for IP address '[192.30.252.151]:443' to the list of known hosts.
Counting objects: 59, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (48/48), done.
Writing objects: 100% (59/59), 6.54 KiB | 0 bytes/s, done.
Total 59 (delta 28), reused 0 (delta 0)
To git@github.com:Organization/repo.git
   2a7b8b5..0c8a46c  feature-hero -> feature-hero

I may have cried a tear of happiness.

Hopefully this can help someone else who gets tormented by this awful issue. I still am not sure what exactly caused it as things were working just great yesterday, but I'm glad I was able to find a fix. It took nearly 3 hours, but we can gloss over that.

I'm receiving "Connection refused" or "Connection timed out" errors when trying to connect to my Amazon Elastic Compute Cloud (Amazon EC2) instance using SSH.

Short description

Error message: "ssh: connect to host ec2-X-X-X-X.compute-1.amazonaws.com port 22: Connection timed out". This error message comes from the SSH client. The error indicates that the server didn't respond to the client and the client program gave up (timed out). The following are common causes for this error:

• The security group or network ACL doesn't allow access.
• There's a firewall on the instance's operating system.
• There's a firewall between the client and the server.
• The host doesn't exist.

Error message: "ssh: connect to host ec2-X-X-X-X.compute-1.amazonaws.com port 22: Connection refused". This message comes remotely from a host. The following are common causes for this error:

• The host reached the instance but there was no service listening on the SSH port.
• A firewall blocked and was set to reject the package instead of dropping it.

Resolution

Note: The last two verification steps require OS-level access of the instance.

For the “Connection refused” error, verify the following

• There's no firewall on the instance rejecting the SSH connection.
• The SSH daemon (sshd) is running and listening on port 22.

Note: Both verification steps require OS-level access of the instance.

If the instance passes both health checks, use one of the following four listed methods with your configuration

• Method 1: Use the EC2 Serial Console for Linux.
• Method 2: Use AWS Systems Manager Session Manager.
• Method 3: Run the AWSSupport-TroubleshootSSH automation runbook.
• Method 4: Use a user data script.

Method 1: Use the EC2 Serial Console for Linux

If configured, you can use the EC2 Serial Console for Linux to troubleshoot OS-level issues on supported Nitro-based instance types. The serial console allows troubleshooting of boot issues, network configuration, and SSH configuration issues. The serial console is accessible using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).

Before using the serial console, grant access to it at the account level. Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.

Note: Each instance using the serial console must include at least one password-based Linux user with sudo access.

For more information on configuring the EC2 Serial Console for Linux, see Configure access to the EC2 Serial Console. If there isn’t a Linux account with a login password configured, you must run ssm-user to reset the password for an account with sudo access. For more information on running ssm-user commands, see the section Managing ssm-user sudo account permissions on Linux and macOS.

After configuration, connect to the EC2 instance through the EC2 serial console using a password-configured Linux user.

Next, run the following commands. These commands verify that the SSH connections aren't being blocked by the OS firewall or TCP wrapper. The commands also verify that the sshd service is running and listening on port 22.

1.    If you have iptables rules configured, then run following command to add a rule in iptables accepting all SSH connections on default port 22:

$ sudo iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT

Because it’s a best practice to use security groups instead of an OS-based firewall, the firewall can be deactivated altogether. To deactivate the OS-based firewall, use one of the following set of commands, depending on your operating system:

Important: The following commands flush all main iptables rules. They also add a rule allowing incoming SSH connections. Additionally, they change the default policy of the main chain to ACCEPT so that flushing the iptables rule doesn’t affect instance network connectivity. Because these commands flush all main iptables, they also flush any existing rules.

Distributions that use UFW (Ubuntu, Debian)

$ sudo iptables -F
$ sudo iptables -P INPUT ACCEPT
$ sudo ufw disable

Distributions that use firewalld (Red Hat, CentOS)

$ sudo iptables -F 
$ sudo iptables -P INPUT ACCEPT
$ sudo systemctl disable firewalld

Note: After you regain access to your instance, review your firewall configuration (UFW, firewalld, iptables).

2.    Verify that SSH is running and verify that the SSH TCP port (22) is in listening state:

$ sudo systemctl restart sshd
$ sudo ss -tpln | grep -iE '22|ssh'
LISTEN 0 128 *:22 *:* users:(("sshd",pid=1901,fd=3))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1901,fd=4))

Note: If your system doesn't have the ss command, you can use the legacy netstat command with the same syntax shown in the preceding example.

3.    Make sure that the TCP wrapper isn’t blocking an SSH connection:

$ if [[ $( cat /etc/hosts.[ad]* | grep -vE '^#' | awk 'NF' | wc -l) -ne 0 ]];\
then sudo sed -i '1i sshd2 sshd : ALL: allow' /etc/hosts.allow; fi

4.    Next, connect to the instance using SSH.

5.    Disconnect the EC2 Serial Console session if it's no longer required.

Method 2: Use AWS Systems Manager Session Manager

Note: To use this method, the instance must be an SSM managed instance and its SSM agent ping status must be Online. For more information on Session Manager and a complete list of prerequisites, see Setting up Session Manager.

To confirm that SSH connections aren't being blocked by the firewall or TCP wrapper and that the sshd service is running and listening on port 22:

1.    Open the AWS Systems Manager.

2.    Start a session for the instance using Session Manager.

3.    Follow steps 1 - 4 from Method 1: Use EC2 Serial console for Linux.

4.    Close the Session Manager session if it's no longer required.

Method 3: Run the AWSSupport-TroubleshootSSH runbook

The AWSSupport-TroubleshootSSH automation runbook installs the Amazon EC2Rescue tool for Linux on the instance. This tool checks for and attempts to fix issues that prevent a remote connection Linux host though SSH.

To run the AWSSupport-TroubleshootSSH runbook:

1.    Open the AWSSupport-TroubleshootSSH page.

2.    Select Run this Automation (console).

To learn more, see How can I use the AWSSupport-TroubleshootSSH Automation workflow to troubleshoot SSH connection issues?

Method 4: Use a user data script

If none of the methods described are suitable in your environment, then use an EC2 user-data script. The EC2 user-data script turns off the OS-level firewall and the TCP wrapper, and then restarts the sshd service.

Important:

• This procedure requires a stop and start of the EC2 instance. If the instance has any data stored on instance store volumes, that data is deleted after stopping the instance.
• If the instance is part of an Amazon EC2 Auto Scaling group, then terminating the instance might also stop instances within the Auto Scaling group.
• If the instance is launched by services that use AWS Auto Scaling, then terminating the instance might also stop instances within the Auto Scaling group.
• Instance termination depends on the instance scale-in protection settings for the Auto Scaling group. If the instance is part of an Auto Scaling group, temporarily remove the instance from the Auto Scaling group before starting the resolution steps.
• Stopping and starting the instance changes the public IP address of the instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to the instance.

Follow these steps to configure user-data for the instance:

1.    Open the Amazon EC2 console.

2.    Choose Instances from the navigation pane, and then select the instance that you plan to connect to.

3.    Stop the instance.

4.    Choose Actions, Instance Settings, Edit User Data.

5.    Copy the following user data script into the Edit User Data dialog box, and then choose Save.

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
 
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
 
#cloud-config
cloud_final_modules:
- [scripts-user, always]
 
--//
Content-Type:
    text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
 
#!/bin/bash
iptables -P INPUT ACCEPT
iptables -F
systemctl restart sshd.service || service sshd restart
if [[ $( cat /etc/hosts.[ad]* | grep -vE '^#' | awk 'NF' | wc -l) -ne 0 ]];\
then sudo sed -i '1i sshd2 sshd : ALL: allow' /etc/hosts.allow; fi
--//

6.    Connect to the instance using SSH.

7.    The preceding user data script is set to run on every reboot of the instance. After regaining access to the instance, remove the user data script.

Note: The preceding command flushes all main iptables rules. After regaining access to the instance, review the firewall configuration for accuracy (for example, UFW, firewalld, iptables).

How do I fix SSH connected to host port 22 connection timed out?

Why is SSH connection timed out?

How do I change my SSH key in GitHub?

How do I clone using SSH?