Your First SSH Connection to a Linux Server

What SSH Actually Is and Why It Matters

SSH stands for Secure Shell. It is a network protocol that lets you connect to a remote computer over an encrypted connection and execute commands as if you were sitting at that machine's keyboard. Every Linux server you will ever manage is accessed primarily through SSH. Understanding how it works and how to use it properly is a foundational skill for anyone managing servers or working with remote infrastructure.

The encrypted part is what makes SSH reliable for internet-based connections. SSH encrypts everything between your computer and the server, including what you type and what the server sends back. This means you can safely manage a server in a data centre across the country without worrying about someone intercepting your password or session data.

Most server management, whether for websites, applications, or infrastructure, happens over SSH. Getting comfortable with it early saves time and reduces the risk of mistakes when you need to make changes quickly.

Making Your First SSH Connection on Windows

Windows 10 and Windows 11 include a built-in OpenSSH client, so you do not need to install anything extra. Open PowerShell or Command Prompt and type your connection command:

ssh username@server_ip_address

Replace username with your server user and server_ip_address with the server's IP address. If SSH is running on the default port 22, you do not need to specify it. If your administrator has changed the port, use the -p flag:

ssh username@server_ip_address -p 2222

Windows users who prefer a graphical interface often use PuTTY. Download it from the official website at chiark.greenend.org.uk. PuTTY requires the server IP address, the SSH port number, and your username. If you are using key-based authentication, you will also need to point PuTTY to your private key file.

To use a private key in PuTTY, you first need to convert it using PuTTYgen, which comes bundled with the PuTTY installation. Open PuTTYgen, load your existing private key file, and save it with the .ppk extension. Then in PuTTY, navigate to Connection, SSH, Auth, and browse to your .ppk file before connecting.

Making Your First SSH Connection on macOS and Linux

Both macOS and Linux ship with OpenSSH pre-installed. Open your terminal application and connect using the same syntax:

ssh username@server_ip_address

For non-default ports, specify with the -p flag:

ssh username@server_ip_address -p 2222

The first time you connect to a new server, SSH will display a warning about the server's host key and ask if you want to trust it. Type yes to add the key to your known hosts file. This step is important because it protects you from man-in-the-middle attacks where someone attempts to impersonate the server you are trying to reach.

How the SSH Connection Process Works

When you connect to a server via SSH, several things happen in sequence. The client and server first negotiate an encrypted connection using Diffie-Hellman key exchange. Then the server proves its identity by presenting its host key. Finally, you authenticate using either a password or a public key.

Each server has a unique host key pair. The private key stays on the server and must never be shared. The public key is distributed to clients, who compare it against the stored host key to verify they are connecting to the genuine server and not an imposter.

The known hosts file stores host keys for servers you have connected to before. On Linux and macOS, this file is located at ~/.ssh/known_hosts. If the host key changes because the server was rebuilt, you reinstalled the operating system, or there is a potential security incident, SSH will refuse to connect and show a warning like "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED".

If you are certain the change is legitimate, remove the old key using the ssh-keygen command:

ssh-keygen -R server_ip_address

You can then reconnect and accept the new host key. This is a useful command to remember when working with servers that get rebuilt or reimaged regularly.

Password Authentication vs Key-Based Authentication

Password authentication is straightforward but carries risks. Passwords can be guessed through automated attacks, phished, or intercepted if the connection is somehow compromised. Key-based authentication is significantly more secure and eliminates the need to type a password each time you connect.

To set up key-based authentication, you generate a cryptographic key pair on your local machine. The private key stays on your local machine and must be protected. The public key is added to the server's ~/.ssh/authorized_keys file. When you connect, the server sends a challenge that only your private key can answer correctly.

Generate a key pair on your local machine:

ssh-keygen -t rsa -b 4096 -C "[email protected]"

Accept the default file location and set a strong passphrase when prompted. The passphrase adds a layer of protection: even if someone gains access to your private key file, they cannot use it without the passphrase. Choose something memorable but not easily guessed.

Copy the public key to the server:

ssh-copy-id username@server_ip_address

This appends your public key to the server's ~/.ssh/authorized_keys file. Test the connection to confirm key-based authentication is working:

ssh username@server_ip_address

If you are not prompted for a password, key-based authentication is active. For guidance on further hardening your SSH setup once keys are in place, a look at securing SSH on Ubuntu covers the next steps.

Simplifying Connections with the SSH Config File

If you connect to the same servers repeatedly, the SSH config file saves significant typing and reduces mistakes. Create or edit the file at ~/.ssh/config on your local machine:

Host myserver
    HostName 192.168.1.100
    User admin
    Port 2222
    IdentityFile ~/.ssh/id_rsa

Now you can connect with a simple command using the alias you defined:

ssh myserver

The SSH config file also supports connection multiplexing, which reuses a single TCP connection for multiple SSH sessions. This is particularly useful when you are opening several terminal windows connected to the same server, as it significantly reduces connection overhead.

Transferring Files with SCP and SFTP

SSH includes two built-in tools for transferring files securely. SCP, or Secure Copy, works directly from the command line and handles both files and directories:

scp file.txt username@server_ip_address:/home/username/
scp -r my_directory username@server_ip_address:/home/username/

The -r flag makes SCP recursive, copying entire directories. SFTP, the SSH File Transfer Protocol, opens an interactive session similar to traditional FTP clients:

sftp username@server_ip_address

Inside the SFTP prompt, common commands include get to download files, put to upload, and ls to list the remote directory contents. Type help to see all available commands.

Running Single Commands Remotely

SSH can run a single command on the remote server without opening an interactive session. This is useful for quick checks and automation scripts:

ssh username@server_ip_address "df -h"
ssh username@server_ip_address "systemctl status nginx"
ssh username@server_ip_address "tail -20 /var/log/syslog"

Quotes around the remote command are important. They prevent your local shell from interpreting special characters before the command reaches the remote server. Without quotes, characters like * or $ would be expanded locally rather than on the server where you intended them to run.

Hardening Your SSH Access

Two of the most impactful hardening steps for SSH are changing the default port and disabling password authentication. The default port 22 is constantly scanned by automated bots searching for vulnerable servers. Moving to a non-standard port reduces the noise of automated login attempts significantly.

Disable password authentication once you have key-based authentication confirmed working. Edit the server's SSH configuration file:

sudo nano /etc/ssh/sshd_config

Set these values to enforce key-only login and prevent direct root access:

Port 2222
PasswordAuthentication no
PermitRootLogin no
PubkeyAuthentication yes

After saving the file, restart the SSH service to apply changes:

sudo systemctl restart sshd

Test before logging out: Keep your current SSH session open and test the new configuration in a separate terminal window. If something is misconfigured, the original session lets you fix it without losing access to the server.

Beyond basic hardening, SSH security is part of a broader approach to server protection. Regular maintenance, monitoring, and updates all play a role in keeping access secure over time. If you are managing multiple servers, establishing a solid maintenance schedule helps ensure security settings are reviewed periodically rather than set once and forgotten.

Troubleshooting Common SSH Problems

Connection refused errors usually mean the SSH service is not running or the port is blocked by a firewall. Check that the service is active:

sudo systemctl status sshd

If it is not running, start it:

sudo systemctl start sshd

Permission denied errors typically indicate a problem with your key file or incorrect username. Verify that your public key is in the server's ~/.ssh/authorized_keys file and that the file has the correct permissions:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Connection timing out often points to a network-level issue. Check that the server's firewall allows connections on the SSH port and that you have the correct IP address.

Documenting Your SSH Setup

As you work with more servers, keeping track of connection details, usernames, ports, and key locations becomes important. Maintaining a simple, secure record of your server access details saves time and prevents lockouts.

Your SSH config file itself serves as useful documentation of how you connect to each server. Beyond that, noting which keys access which servers, any non-standard ports in use, and any jump hosts or proxies involved in your setup helps when troubleshooting or handing off management to someone else. Good documentation practices apply beyond SSH; if you want to read more about keeping IT documentation practical and actually useful, there is a guide on IT documentation that people actually read worth reviewing.

Moving Forward with SSH

SSH is the foundation of server management on Linux systems. Once you are comfortable connecting, using key-based authentication, and transferring files, you have the basics covered for everyday server work. From there, hardening your setup, using the SSH config file to stay organised, and understanding how to troubleshoot common problems will serve you well as you manage more servers over time.

If you are working with servers that need a security review or help setting up proper access controls, preparing details about your current setup, server count, and any existing security measures helps when reaching out for assistance. Whether you are starting out or need a hand reviewing an existing configuration, you can get in touch to discuss what is needed.