A way out (onto the internet)

To make a long story short, I used to run a server directly from my own private Internet connection, using a static public IP address, things were good, I could even send mail from the mailserver.. Things changed, a lot. Internet connections are now crippled and everything is terrible. here's my way out.

I still want to run my server from my home, I still want some measure of control. I still don't want to trust a VPS provider.

I ended up buying the cheapest VPS I could find with flatrate data and a static public IP.

I place as little trust as possible on then VPS: I installed an ssh public key and a small script to route ports through a tunnel.

This is how my network setup looks:

                                            ┌────────────────┐
                                   ┌────────► Public Internet◄─────────────────┐
           ┌───────────────────┐   │        └────────────────┘                 │
           │Router under CG-NAT├───┘                                           │
           └────────▲──────────┘                                               │
                    │                                                          │
       ┌────────────┴┐                                                         │
       │LAN under NAT│                                                         │
       └─────▲───────┘                                                         │
             │                                                      ┌──────────┼─────────────────┐
┌────────────┼──────────────┐                                       │ VPS      │                 │
│ Server     │              │                                       │      ┌───┴──┐              │
│         ┌──┴───┐          │                                       │      │ eno1 │◄─────┐       │
│     ┌───► eno1 │          │                                       │      └──────┘      │       │
│     │   └──────┘          │                                       │                    │       │
│     │                     │                                       │                    │       │
│     │ ┌─────────┐      ┌──┴───────────────────────────────────────┴────┐               │       │
│     └─┤Routes to│      │                                               │               │       │
│       │LAN+VPS  │      │                                               │               │       │
│       └─────────┘      │                                               │               │       │
│                        │                                               │               │       │
│        SSH-Client──────┤                                               ├───►SSH-Server │       │
│                        │  ┌───────────┐                   ┌──────────┐ │               │       │
│        Default GW──────┼─►│tun1       │◄── pointopoint ──►│      tun1│ │    ┌──────────┴─┐     │
│                        │  │10.10.10.1 │                   │10.10.10.2│ │    │Ip forward  │     │
│        Services ◄──────┼──┤           │                   │          │◄├────┤Port forward│     │
│                        │  └───────────┘                   └──────────┘ │    └────────────┘     │
│                        │                                               │                       │
│                        └──┬───────────────────────────────────────┬────┘                       │
│                           │                                       │                            │
└───────────────────────────┘                                       └────────────────────────────┘

So it's a lot simpler than I had thought it could be. I'm treating the VPS as simply a gateway to the public Internet.

The only "tricky" part of this setup was getting traffic to flow back through the tun device on the server.

My initial attempt used masquerade out of tun1 on the vps, but this resulted in all incoming connections to the services looking like they came from the .2 address, which I didn't want. But making the tun device be the default gateway for the server fixed this.

VPS Config

The only configuration I did on the VPS SSH-Server was to set PermitTunnel yes in sshd_config.

I then placed a script on the VPS: router.sh

#!/bin/bash

echo "Tun setup ip"
ifconfig tun1 10.10.60.2 pointopoint 10.10.60.1 netmask 255.255.255.0
echo "Tun setup mtu 1448"
ip link set dev tun1 mtu 1448

function clearRules {
        iptables -P INPUT ACCEPT
        iptables -P FORWARD ACCEPT
        iptables -P OUTPUT ACCEPT
        iptables -t nat -F
        iptables -t mangle -F
        iptables -F
        iptables -X
        echo "Rules cleared."
}

clearRules

IP=10.10.10.1

function forward {
        PROT="$2"
        PORT="$1"
        echo "Forwarding port $PORT/$PROT"
        iptables -A PREROUTING -t nat -p $PROT -i eth0 --dport $PORT -j DNAT --to-destination $IP:$PORT
        iptables -A FORWARD -p $PROT -d $IP --dport $PORT -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
}

forward 70 tcp
forward 80 tcp
forward 443 tcp

# Linux router
echo 1 > /proc/sys/net/ipv4/ip_forward
EXT=eth0
INT=tun1

# NAT
iptables -t nat -A POSTROUTING -o $EXT  -j MASQUERADE 
# Allow traffic from internal to external
iptables -A FORWARD -i $INT -o $EXT -j ACCEPT
# Allow returning traffic from external to internal
iptables -A FORWARD -i $EXT -o $INT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Drop all other traffic that shouldn't be forwarded
iptables -A FORWARD -j DROP
                                                                                                                                                
# Allow outgoing connections
iptables -A OUTPUT -j ACCEPT

echo "Routing done."

The routing setup script is executed upon connection from the server under nat.

Server under NAT config

This is not more difficult, but it's worth noting that if you mess it up, you might lose SSH connectivity, so be careful if you don't have physical/console access to the server.

I made a "fixup.sh" script which will be called after the SSH client establishes connection.

#!/bin/bash

# Configure the tun interface
ifconfig tun1 10.10.60.1 pointopoint 10.10.60.2 netmask 255.255.255.0
ip link set dev tun1 mtu 1448

ip route delete default

# Add special routes for my internal LAN and also for the specific IP of the VPS
# It's important that these go out via the servers real NIC.
ip route add 192.168.20.0/24 dev eno0
ip route add the.vps.ip.address  via 192.168.40.1 dev eno0

# Let everything else flow through the tun interface.
ip route add default via 10.10.60.2 dev tun1

echo "TUN setup done."

That script is called when the SSH connection has been established.

And the final piese of the puzzle, is the SSH client command that sets up the initial connection with the TUN devices.

I made it as a systemd unit file, I called it /etc/systemd/system/sshvpn.service and here it is:

[Unit]
Description=SSH VPN To VPS
After=network.target

[Service]
ExecStart=/usr/bin/ssh -w 11:11 -i /home/dusted/.ssh/ssdvps-delta_id_rsa -o PermitLocalCommand=yes -o LocalCommand="/home/dusted/fixout.sh" -o ServerAliveInterval=10 -o ServerAliveCountMax=10 -o TCPKeepAlive=yes root@dusted.dk './router.sh'
RestartSec=5
Restart=always

[Install]
WantedBy=multi-user.target

To use it:

sudo systemctl daemon-reload
sudo systemctl enable sshvpn.service
sudo systemctl start sshvpn.service

P.S. Mind the MTU

Note how I've set the MTU size on the TUN interfaces in both scripts, that made traffic go from, very slow and not always working at all, to about 91% of the raw bandwith.

For OpenVPN, I experimentially determined that adding "mssfix 1300" to the servers config gave the best improvement in speed, it's about 74% of the raw speed.