Passive fingerprinting leverages the information leaked about the client's browser (from the HTTP and TLS layers) and operating system (from the TCP and IP layers). Importantly, passive fingerprinting is not detectable by the client, unlike active fingeprinting such as fingerprint2js. While there is a wealth of infomation online on how to bypass active fingerprinting, passive fingerprinting doesn't get the same support.
How does browser fingerprinting work? Browsers leak information from both the order of the HTTP they send and in the metadata of the ClientHello packet used to establish a TLS session with a web server.
RFC 2616 Hypertext Transfer Protocol -- HTTP/1.1 does not enforce a specific header order, so browsers and tools will implement them in different ways. The headers relevant to fingerprinting are:
The ClientHello packet sent when the browser initiates a TLS handshake with the web server is tied to TLS/SSL library used to make that request. Seeing as browsers typically use their own libraries, this can be a somewhat reliable way to determine the actual browser behind a request. The security industry largely uses the JA3 fingerprinting method to share signatures.
How does operating system fingerprinting work? Each operating system flavor implements their TCP/IP stack in a specific way and with non-standard default values, which can be used for fingerprinting. The critical components of the signaturable metadata are: the IP version, the initial TTL (which has to be computed), the length of the IPv4 options (if any), the maximum segment size, the window size, the window scaling factor, the TCP options and their order, "quirks" or other oddities within the packet, and the payload class.