A rather interesting task came up recently: following the example of https://www.howsmyssl.com/, get a page listing the cipher suites that a client passes in its ClientHello during the TLS handshake.
And at the same time, sketch out a tool that will work with other header types, in particular Proxy-Connection.
The main web server I work with is Nginx — more precisely, its OpenResty build with the LuaJIT interpreter for Lua scripts.
The core problem is that the web server has very little direct access to anything involved in the TLS handshake procedure. So it makes sense to intercept the handshake at a lower level.
Some side issues inevitably come up — for example, the Session ID from the client header will not equal the Session ID of the server response or the ngx.var.ssl_session_id variable.
In principle, you could take a much simpler path:
- Capture packets with Tshark for the client hello and the server response, and pipe the output to a Python script:
tshark -i eth0 -Y 'ssl.handshake.type == 1 or ssl.handshake.type == 2 ' -l -V | python3 tcpparse.py - Implement logic in the Python script to determine which of the two packets arrived. Match the client and server greetings by the Acknowledgment number of the client packet, which will equal the Sequence Number of the server packet. Take the Session ID from the server packet and cache the results.
But since the task implied some groundwork for the future, I decided not to use tshark and to stop at the stream module in Nginx:
stream {
server {
listen 443;
content_by_lua_file "/lua/simpleproxy.lua";
}
}
local function proxy_data(client_sock, server_sock)
local ok1, err = ngx.thread.spawn(read_from_socket_client, client_sock, server_sock)
if not ok1 then
ngx.log(ngx.ERR, "failed to spawn read_from_socket_client thread: ", err)
return
end
local ok2, err = ngx.thread.spawn(read_from_socket_server, server_sock, client_sock)
if not ok2 then
ngx.log(ngx.ERR, "failed to spawn read_from_socket_server thread: ", err)
return
end
ngx.thread.wait(ok1, ok2)
end
local function proxy_handler_stream()
local client_sock, err = ngx.req.socket(true)
if not client_sock then
ngx.log(ngx.ERR, "failed to get client socket: ", err)
return ngx.exit(ngx.ERROR)
end
local server_sock = ngx.socket.tcp()
local ok, err = server_sock:connect("127.0.0.1", 8443)
if not ok then
ngx.log(ngx.ERR, "failed to connect to the server: ", err)
return ngx.exit(ngx.ERROR)
end
proxy_data(client_sock, server_sock)
end
proxy_handler_stream()
The proxy_handler_stream function starts reading data from the client socket, initializes the connection to the backend, and passes control to the proxy_data function, which uses the "Light threads" mechanism — one of the implementations of coroutines in Lua.
Minimally, to relay packets between the client and the server, the code of the read_from_socket_client and read_from_socket_server functions is enough:
local function read_from_socket_client(from_sock, to_sock)
from_sock:settimeout(0)
to_sock:settimeout(0)
while true do
local data, err, partial = from_sock:receive("1")
if not data and err == "closed" then
break
end
data = data or partial
if data and #data > 0 then
local bytes, send_err = to_sock:send(data)
if not bytes then
return false, "failed to send data: " .. send_err
end
end
end
end
local function read_from_socket_server(from_sock, to_sock)
from_sock:settimeout(0)
to_sock:settimeout(0)
while true do
local data, err, partial = from_sock:receive("1")
if not data and err == "closed" then
break
end
data = data or partial
if data and #data > 0 then
local bytes, send_err = to_sock:send(data)
if not bytes then
return false, "failed to send data: " .. send_err
end
end
end
end
Our task, however, isn't limited to proxying, so let's start with some ritual dances.
Let's modify read_from_socket_client and read_from_socket_server. Of the data passed over the sockets we need to drop the first 5 bytes (honestly, I don't remember what they are for, sorry). The 6th byte holds the header type (01 — client hello, 02 — server response), followed by 3 bytes of message length. Beyond that we don't need to read and aggregate data, so we simply continue relaying:
local accumulated_data_client = ""
local accumulated_data_server = ""
local counter_client = -5
local counter_client_stop = 4096
local counter_server = -5
local counter_server_stop = 4096
local function read_from_socket_client(from_sock, to_sock)
from_sock:settimeout(0)
to_sock:settimeout(0)
while true do
local data, err, partial = from_sock:receive("1")
if not data and err == "closed" then
break
end
data = data or partial
if data and #data > 0 then
counter_client = counter_client + 1
if counter_client > 0 and counter_client <= counter_client_stop then
accumulated_data_client = accumulated_data_client .. data
if counter_client == 4 then
local last_three_bytes = accumulated_data_client:sub(2, 4)
local result = 0
for i = 1, #last_three_bytes do
result = result * 256 + last_three_bytes:byte(i)
end
counter_client_stop = result + 4
end
end
local bytes, send_err = to_sock:send(data)
if not bytes then
return false, "failed to send data: " .. send_err
end
end
end
end
local function read_from_socket_server(from_sock, to_sock)
from_sock:settimeout(0)
to_sock:settimeout(0)
while true do
local data, err, partial = from_sock:receive("1")
if not data and err == "closed" then
break
end
data = data or partial
if data and #data > 0 then
counter_server = counter_server + 1
if counter_server > 0 and counter_server <= counter_server_stop then
accumulated_data_server = accumulated_data_server .. data
if counter_server == 4 then
local last_three_bytes = accumulated_data_server:sub(2, 4)
local result = 0
for i = 1, #last_three_bytes do
result = result * 256 + last_three_bytes:byte(i)
end
counter_server_stop = result + 4
end
end
if counter_server == counter_server_stop then
infoFromHandshake(accumulated_data_client, accumulated_data_server)
end
local bytes, send_err = to_sock:send(data)
if not bytes then
return false, "failed to send data: " .. send_err
end
end
end
end
In the code above, when counter_server reaches the end of the header payload, the infoFromHandshake function is called. It finally receives the variables holding the client request and the server response to it.
The server response does contain the SessionID in cleartext, but the client request carries no explicit cipher suite information. And the simplest solution I know of for decoding the headers further is to use the cryptobyte library, written in Go.
So for now, let's quickly wire up data transfer over a pipe to our upcoming Go program, write the response to Redis, and leave Lua alone:
local function infoFromHandshake(data_client, data_server)
local timestamp = os.time()
local randomPart = math.random(10000, 99999)
local tempInputClientFileName = string.format("/tmp/hs_client_input_%d_%d.txt", timestamp, randomPart)
local tempInputServerFileName = string.format("/tmp/hs_server_input_%d_%d.txt", timestamp, randomPart)
local tempInputClientFile = io.open(tempInputClientFileName, "w")
tempInputClientFile:write(data_client)
tempInputClientFile:close()
local tempInputServerFile = io.open(tempInputServerFileName, "w")
tempInputServerFile:write(data_server)
tempInputServerFile:close()
local go_program_client = assert(io.popen("/ciphersuites/clienthello < " .. tempInputClientFileName, "r"))
local clienthello = assert(go_program_client:read("*a"))
go_program_client:close()
os.remove(tempInputClientFileName)
local go_program_server = assert(io.popen("/ciphersuites/clienthello < " .. tempInputServerFileName, "r"))
local sessionid = assert(go_program_server:read("*a"))
go_program_server:close()
os.remove(tempInputServerFileName)
if sessionid and clienthello then
setRedisKeys(sessionid, clienthello)
else
return
end
end
I'm leaving out the setRedisKeys function; it is trivially simple.
In the Go application we read the data from the pipe and determine the header type from the first byte:
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"os"
)
func main() {
var buf bytes.Buffer
_, err := io.Copy(&buf, os.Stdin)
if err != nil {
fmt.Fprintln(os.Stderr, "Ошибка при чтении:", err)
return
}
data := buf.Bytes()
if len(data) < 1 {
fmt.Fprintln(os.Stderr, "Нет данных для чтения")
return
}
switch data[0] {
case 0x01:
ClientDecode(data)
case 0x02:
ServerDecode(data)
default:
fmt.Fprintln(os.Stderr, "Неизвестный заголовок")
}
}
func ClientDecode(data []byte) {
clientInfo := UnmarshalClientHello(data)
clientInfoJson, err := json.Marshal(clientInfo)
if err != nil {
fmt.Println("Ошибка при кодировании в JSON:", err)
return
}
fmt.Printf(string(clientInfoJson))
}
func ServerDecode(data []byte) {
server_info := UnmarshalServerResponse(data)
serverSessionID := server_info.SessionIDHex
fmt.Printf(string(serverSessionID))
}
The basis for the Go decryption code is this article: https://www.agwa.name/blog/post/parsing_tls_client_hello_with_cryptobyte
package main
import (
"bytes"
"encoding/json"
"encoding/hex"
"crypto/cryptobyte"
"fmt"
"io"
"os"
)
func main() {
var buf bytes.Buffer
_, err := io.Copy(&buf, os.Stdin)
if err != nil {
fmt.Fprintln(os.Stderr, "Ошибка при чтении:", err)
return
}
data := buf.Bytes()
if len(data) < 1 {
fmt.Fprintln(os.Stderr, "Нет данных для чтения")
return
}
switch data[0] {
case 0x01:
ClientDecode(data)
case 0x02:
ServerDecode(data)
default:
fmt.Fprintln(os.Stderr, "Неизвестный заголовок")
}
}
func ClientDecode(data []byte) {
clientInfo := UnmarshalClientHello(data)
clientInfoJson, err := json.Marshal(clientInfo)
if err != nil {
fmt.Println("Ошибка при кодировании в JSON:", err)
return
}
fmt.Printf(string(clientInfoJson))
}
func ServerDecode(data []byte) {
server_info := UnmarshalServerResponse(data)
serverSessionID := server_info.SessionIDHex
fmt.Printf(string(serverSessionID))
}
We need the following types. The ServerHelloInfo struct could have been left out, since we only need SessionID / SessionIDHex, and ClientHelloInfo could have been simplified, but let's keep them as groundwork for the future.
The CipherSuites array for the MakeCipherSuite function is generated by parsing the CSV at https://www.iana.org/assignments/tls-parameters/tls-parameters-4.csv; an example of that generation, as well as a full example of the parsing implementation, can be seen in the code from the article "Parsing a TLS Client Hello with Go's cryptobyte Package". Here is a shortened version of the parsing, cut down to my needs:
type ProtocolVersion uint16
func (v ProtocolVersion) Hi() uint8 {
return uint8(v >> 8)
}
func (v ProtocolVersion) Lo() uint8 {
return uint8(v)
}
func (v ProtocolVersion) MarshalJSON() ([]byte, error) {
return json.Marshal([2]uint8{v.Hi(), v.Lo()})
}
type CompressionMethod uint8
func (m CompressionMethod) MarshalJSON() ([]byte, error) {
return json.Marshal(uint16(m))
}
type CipherSuite struct {
Code [2]uint8 `json:"code"`
Name string `json:"name,omitempty"`
Grease bool `json:"grease,omitempty"`
}
func (c CipherSuite) CodeUint16() uint16 {
return (uint16(c.Code[0]) << 8) | uint16(c.Code[1])
}
func MakeCipherSuite(code uint16) CipherSuite {
hi := uint8(code >> 8)
lo := uint8(code)
return CipherSuite{
Code: [2]uint8{hi, lo},
Name: CipherSuites[code].Name,
Grease: CipherSuites[code].Grease,
}
}
type ClientHelloInfo struct {
Raw []byte `json:"raw"`
Version ProtocolVersion `json:"version"`
Random []byte `json:"random"`
SessionID []byte `json:"session_id"`
CipherSuites []CipherSuite `json:"cipher_suites"`
CompressionMethods []CompressionMethod `json:"compression_methods"`
Extensions []Extension `json:"extensions"`
Info struct {
ServerName *string `json:"server_name"`
SCTs bool `json:"scts"`
Protocols []string `json:"protocols"`
JA3String string `json:"ja3_string"`
JA3Fingerprint string `json:"ja3_fingerprint"`
} `json:"info"`
}
type ServerHelloInfo struct {
Raw []byte `json:"raw"`
Version ProtocolVersion `json:"version"`
Random []byte `json:"random"`
SessionID []byte `json:"session_id_string"`
SessionIDHex string `json:"session_id"`
CipherSuite CipherSuite `json:"cipher_suite"`
CompressionMethod CompressionMethod `json:"compression_method"`
Extensions []Extension `json:"extensions"`
Info struct {
SelectedProtocol *string `json:"selected_protocol"`
SCTs bool `json:"scts"`
} `json:"info"`
}
And the actual header parsing itself:
func UnmarshalClientHello(handshakeBytes []byte) *ClientHelloInfo {
info := &ClientHelloInfo{Raw: handshakeBytes}
handshakeMessage := cryptobyte.String(handshakeBytes)
var messageType uint8
if !handshakeMessage.ReadUint8(&messageType) || messageType != 1 {
return nil
}
var clientHello cryptobyte.String
if !handshakeMessage.ReadUint24LengthPrefixed(&clientHello) || !handshakeMessage.Empty() {
return nil
}
if !clientHello.ReadUint16((*uint16)(&info.Version)) {
return nil
}
if !clientHello.ReadBytes(&info.Random, 32) {
return nil
}
if !clientHello.ReadUint8LengthPrefixed((*cryptobyte.String)(&info.SessionID)) {
return nil
}
var cipherSuites cryptobyte.String
if !clientHello.ReadUint16LengthPrefixed(&cipherSuites) {
return nil
}
info.CipherSuites = []CipherSuite{}
for !cipherSuites.Empty() {
var suite uint16
if !cipherSuites.ReadUint16(&suite) {
return nil
}
info.CipherSuites = append(info.CipherSuites, MakeCipherSuite(suite))
}
if !clientHello.Empty() {
return nil
}
return info
}
func UnmarshalServerResponse(handshakeBytes []byte) *ServerHelloInfo {
info := &ServerHelloInfo{Raw: handshakeBytes}
handshakeMessage := cryptobyte.String(handshakeBytes)
var messageType uint8
if !handshakeMessage.ReadUint8(&messageType) || messageType != 2 {
return nil
}
// Skip the message length (3 bytes)
if !handshakeMessage.Skip(3) {
return nil
}
// Read the protocol version (2 bytes)
if !handshakeMessage.Skip(2) { // the version is usually skipped
return nil
}
// Read and skip the random number (32 bytes)
if !handshakeMessage.Skip(32) {
return nil
}
var sessionIDBytes cryptobyte.String
if !handshakeMessage.ReadUint8LengthPrefixed(&sessionIDBytes) {
return nil
}
info.SessionIDHex = hex.EncodeToString(sessionIDBytes)
return info
}
In total, we have an application skeleton that reads our header bytes from the pipe and returns clientInfoJson and serverSessionID.
In Nginx we set ssl_session_cache none;, ssl_session_tickets off;, keepalive_timeout 0; — this gets rid of tickets in SSL sessions and guarantees that a session is created for every request. Otherwise everything works with curl, but Chrome will surely come back for the favicon, and we won't see a correct Session ID value.
All that remains is the small matter of getting the data out of Redis: into the location we need in our HTTP server listening over SSL, we insert a content_by_lua_block or content_by_lua_file with roughly this content:
local function getclientinfo()
local redis_key = ngx.var.ssl_session_id
local clientinfo
local redis_err
local ok, red = pcall(redis.connect, { host = redis_local_server, port = redis_local_port, timeout = redis_local_connect_timeout })
if not ok then
redis_err = true
end
if not redis_err then
local res, err = red:get(redis_key)
if err then
elseif not res then
else
clientinfo = res
end
end
local jsonData = json.decode(clientinfo)
ngx.header.content_type = 'application/json; charset=utf-8'
ngx.say(jsonData)
ngx.exit(200)
end
getclientinfo()
If the full output isn't needed, use ngx.say(jsonData.cipher_suites).
That's about it. I tried to explain the main points and show the approaches. The code was written in a hurry and needs more work — it's a sketch. While researching across the internet, apart from Andrew Ayer's article, I found plenty of questions and very little that was even somewhat systematic or useful. I hope this saves someone time.