OpenConnect server, also known as ocserv, is a VPN server that communicates over SSL. By design, its goal is to become a secure, lightweight, and fast VPN server. OpenConnect server uses the OpenConnect SSL VPN protocol. At the time of writing, it also has experimental compatibility with clients that use the AnyConnect SSL VPN protocol.

Why AnyConnect? Although the AnyConnect protocol is relatively easy to detect, it is widely used by many large corporations with significant economic impact. Therefore, it is currently safer and more reliable than PPTP, OpenVPN, and some other legacy VPN protocols.

2026 Update & Modern Recommendations

The manual compilation and SysVinit scripts detailed in this guide are primarily for legacy systems or custom builds. If you are setting up ocserv on a modern Debian/Ubuntu system today, it is strongly recommended to use standard package tools:

1. Install via APT: Skip source compilation and install directly using sudo apt update && sudo apt install ocserv.
2. Use Systemd: Service management is handled automatically via Systemd (sudo systemctl enable --now ocserv).
3. Use Trusted Certificates (Let’s Encrypt): Instead of self-signed certificates (which trigger client security warnings), use Certbot to issue trusted certificates:

   sudo apt install certbot
   sudo certbot certonly --standalone -d vpn.yourdomain.com
   

   Then reference /etc/letsencrypt/live/vpn.yourdomain.com/fullchain.pem and privkey.pem in ocserv.conf.

Update On Jan 2018:

A Docker image has been created for fast deployment, so you can skip reading the lengthy content below if you prefer.

Here, we will mainly talk about how to set up ocserv on a Debian system.

1. Compile ocserv

The official website of ocserv is http://www.infradead.org/ocserv/ . Currently (as of 2015/09/30), the website is down for unknown reasons. However, a backup mirror is available at https://github.com/fanyueciyuan/ocserv-backup. The latest version in this archive is 0.10.8.

The required packages for compiling ocserv are pkg-config and libgnutls28-dev. Since ocserv has many additional features, it is better to install all dependencies at once to avoid re-compiling later:

apt-get install build-essential pkg-config libgnutls28-dev libwrap0-dev libpam0g-dev libseccomp-dev libreadline-dev libnl-route-3-dev liblz4-dev

Next, extract and configure:

tar xvJf ocserv-0.10.8.tar.xz
cd ocserv-0.10.8
./configure --prefix=/usr/local/ocserv --sysconfdir=/etc/ocserv/

To keep the system tidy, we install it into a dedicated directory. As a result, we need to create symbolic links (ln) for the binary and man pages:

sudo ln -s /usr/local/ocserv/share/man/man8/occtl.8 /usr/local/share/man/man8/occtl.8
sudo ln -s /usr/local/ocserv/share/man/man8/ocpasswd.8 /usr/local/share/man/man8/ocpasswd.8
sudo ln -s /usr/local/ocserv/share/man/man8/ocserv.8 /usr/local/share/man/man8/ocserv.8
sudo ln -s /usr/local/ocserv/sbin/ocserv /usr/local/bin/ocserv
sudo ln -s /usr/local/ocserv/bin/occtl /usr/local/bin/occtl
sudo ln -s /usr/local/ocserv/bin/ocpasswd /usr/local/bin/ocpasswd

Now compile and install:

make
sudo make install

2. Configure ocserv

Sample configuration files are located in the doc directory. Before configuring the server, we need to generate CA certificates and server certificates.

First, install gnutls-bin:

cd ~
apt-get install gnutls-bin
mkdir certificates
cd certificates

Create the CA template file ca.tmpl:

cn = "VPN CA"
organization = "Chillrain Node"
serial = 1
expiration_days = 3650
ca
signing_key
cert_signing_key
crl_signing_key

Generate the CA private key and CA certificate:

certtool --generate-privkey --outfile ca-key.pem
certtool --generate-self-signed --load-privkey ca-key.pem --template ca.tmpl --outfile ca-cert.pem

Next, create the server template file server.tmpl:

cn = "your domain name or ip"
organization = "Chillrain Node"
expiration_days = 3650
signing_key
encryption_key
tls_www_server

Generate the server private key and server certificate:

certtool --generate-privkey --outfile server-key.pem
certtool --generate-certificate --load-privkey server-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template server.tmpl --outfile server-cert.pem

Copy the certificates and the sample configuration file to /etc/ocserv:

mkdir /etc/ocserv
sudo cp server-cert.pem server-key.pem /etc/ocserv
cp ~/ocserv-0.10.8/doc/sample.config /etc/ocserv/ocserv.conf

Edit /etc/ocserv/ocserv.conf. Key settings to update:

# Default to plain password mode
auth = "plain[/etc/ocserv/ocpasswd]"
try-mtu-discovery = true

# Server certificate paths
server-cert = /etc/ocserv/server-cert.pem
server-key = /etc/ocserv/server-key.pem

# Ports: Using 1443 to avoid conflicts if standard HTTPS port 443 is used by a web server
tcp-port = 1443
udp-port = 1443

# Security best practice: Do not run as root
run-as-user = nobody
run-as-group = daemon

# Local IP subnet allocation
ipv4-network = 10.8.0.0
ipv4-netmask = 255.255.255.0

# DNS server
dns = 8.8.8.8

# Push route entries
route = 0.0.0.0/128.0.0.0
route = 128.0.0.0/128.0.0.0
cisco-client-compat = true

Note on Default Routing

Regarding default routes, the official documentation suggests commenting out all route pushes if you want all client traffic routed through the VPN. However, when using Cisco AnyConnect 3.0, doing so causes the client to remove all LAN and broadcast routes, making the VPN server itself unreachable.

Pushing the split default routes resolves this issue:

• 0.0.0.0/128.0.0.0 = 0.0.0.0/1 (Covers 128.0.0.0 to 255.255.255.255)
• 128.0.0.0/128.0.0.0 = 128.0.0.0/1 (Covers 0.0.0.0 to 127.255.255.255)

3. Create User & Configure NAT Routing

Create a user account:

ocpasswd -c /etc/ocserv/ocpasswd username

Add iptables NAT rules for packet forwarding:

sudo iptables -t nat -A POSTROUTING -o eth0 -s 10.8.0.0/24 -j MASQUERADE

To automatically apply iptables rules on startup:

cat << EOF | sudo tee -a /etc/network/if-pre-up.d/iptables
iptables -t nat -A POSTROUTING -o eth0 -s 10.8.0.0/24 -j MASQUERADE
EOF
sudo chmod a+x /etc/network/if-pre-up.d/iptables

Enable IPv4 forwarding in /etc/sysctl.conf:

net.ipv4.ip_forward=1

Apply the sysctl settings:

sysctl -p /etc/sysctl.conf

4. System Init Script (SysVinit)

Create the startup script at /etc/init.d/ocserv:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          ocserv
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

PATH=/bin:/usr/bin:/sbin:/usr/sbin
DAEMON=/usr/local/bin/ocserv
PIDFILE=/var/run/ocserv.pid
DAEMON_ARGS="-c /etc/ocserv/ocserv.conf"

case "$1" in
start)
    if [ ! -r $PIDFILE ]; then
        echo -n "Starting OpenConnect VPN Server Daemon: "
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
        $DAEMON_ARGS > /dev/null
        echo "ocserv."
    else
        echo -n "OpenConnect VPN Server is already running.\n\r"
        exit 0
    fi
    ;;
stop)
    echo -n "Stopping OpenConnect VPN Server Daemon: "
    start-stop-daemon --stop --quiet --pidfile $PIDFILE --exec $DAEMON
    echo "ocserv."
    rm -f $PIDFILE
    ;;
force-reload|restart)
    echo "Restarting OpenConnect VPN Server: "
    $0 stop
    sleep 1
    $0 start
    ;;
status)
    if [ ! -r $PIDFILE ]; then
        exit 3
    fi
    PID=$(cat $PIDFILE | sed 's/ //g')
    EXE=/proc/$PID/exe
    if [ -x "$EXE" ] && [ "$(ls -l "$EXE" | cut -d'>' -f2,2 | cut -d' ' -f2,2)" = "$DAEMON" ]; then
        exit 0
    elif [ -r $PIDFILE ]; then
        exit 1
    else
        exit 3
    fi
    ;;
*)
    echo "Usage: /etc/init.d/ocserv {start|stop|restart|force-reload|status}"
    exit 1
    ;;
esac
exit 0

Set executable permissions and configure boot startup:

sudo chmod a+x /etc/init.d/ocserv
sudo update-rc.d ocserv defaults

Start the ocserv service:

sudo /etc/init.d/ocserv start

5. Alternative: Client Certificate Authentication

If you prefer using client certificates for authentication instead of passwords, generate user certificates as follows:

certtool --generate-privkey --outfile user-key.pem

cat << EOF > user.tmpl
cn = "VPN"
unit = "VPN"
expiration_days = 365
signing_key
tls_www_client
EOF

certtool --generate-certificate --load-privkey user-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template user.tmpl --outfile user-cert.pem

# Export to PKCS#12 (.p12) format for client import
openssl pkcs12 -export -inkey user-key.pem -in user-cert.pem -certfile ca-cert.pem -out user.p12 -password pass:

Plain-text SOCKS5 traffic is typically intercepted and filtered by deep packet inspection (DPI) systems like the GFW. Recently, however, I noticed a strange phenomenon: certain unencrypted packets containing sensitive domain keywords (such as google.com) were able to pass through without being blocked.

After capturing and inspecting the network traffic, I found that this behavior is directly tied to how the SOCKS5 connection is initialized.

1. SOCKS5 Handshake Protocol

When establishing a SOCKS5 connection, the client first sends a Method Selection Request detailing the authentication methods it supports.

A typical request frame looks like this:

05 02 00 02

The byte breakdown is as follows:

• 05: Protocol version (SOCKS5).
• 02: NMETHODS — the number of authentication methods supported (in this case, 2).
• 00 02: METHODS — the supported method identifiers:
  • 0x00: No authentication
  • 0x02: Username/Password

Standard SOCKS5 Authentication Method Identifiers

2. The Anomaly

The anomaly occurs when the client advertises support for GSSAPI (0x01) during the initial negotiation.

Even if the server ultimately selects 0x00 (No Auth) or 0x02 (Username/Password) and GSSAPI is never actually used for the connection, the presence of 0x01 in the client’s initial request causes the GFW to bypass DPI scanning on subsequent payload traffic.

Specifically, when the handshake payload contains:

05 03 00 01 02

(SOCKS5, 3 methods supported: No Auth, GSSAPI, Username/Password)

The GFW appears to ignore the connection from that point forward.

3. Conclusion

This behavior is quite bizarre, though it holds true in recent tests. However, relying on protocol quirks in plain-text SOCKS5 is inherently fragile and unreliable for long-term use. For consistent security and uninhibited traffic flow, encapsulating connection data using encrypted solutions like stunnel or TLS wrappers remains the recommended approach.