Openssl on Guardian Project

Quick set up guide for Encrypted Client Hello (ECH)

Fri, 10 Nov 2023 00:00:00 +0000

The Encrypted Client Hello (ECH) mechanism draft-spec is a way to plug a few privacy-holes that remain in the Transport Layer Security (TLS) protocol that’s used as the security layer for the web. OpenSSL is a widely used library that provides an implementation of the TLS protocol. The DEfO project has developed an implementation of ECH for OpenSSL, and proof-of-concept implementations of various clients and servers that use OpenSSL, and other TLS libraries, as a demonstration and for interoperability testing. DEfO is funded by the Open Technology Fund (OTF).

This guide is aimed at those who would like to try out ECH with our experimental Debian unstable packages for OpenSSL, curl and nginx. The DEfO project is making packages temporarily available so people can carry out such experiments before upstream maintainers include ECH in standard releases.

This guide describes two set ups: one assumes that you want to setup everything (i.e. a web server and associated DNS authoritative server) on one virtual machine and a second describes a way to set up an ECH-enabled web server if you have an existing DNS configuration elsewhere. Many other configurations are of course possible. Comments/questions are welcome via email or as GitLab issues/merge requests.

In summary this guide shows how to: - find and install our Debian unstable packages - configure a minimal DNS setup (using dnsmasq or bind) that enables use of ECH from browsers - configure a minimal nginx setup for an ECH enabled web sites - test your DNS set up and that ECH is working

We assume the reader is comfortable with simple DNS management and system administration tasks. When commands shown below should be run as root they are prepended with a ‘#’, if not they are prepended with a ‘$’.

Install packages

You need to first have access to a virtual machine running Debian unstable. Many online resources describe many ways to do this.

We start with an empty (minimal) Debian unstable system and install OpenSSL from the DEfO apt repository.

# apt install ca-certificates
# echo "deb [trusted=yes] https://defo.ie/debian/ ./" > /etc/apt/sources.list.d/defo.list
# apt update
# apt install openssl

Pick some DNS names to use

In the text below we assume that the web server for which we wish to benefit from ECH is called hidden.example.com

ECH also involves a so-called public_name - that’s present in the outer ClientHello when ECH is used and hence is visible to a network observer, unlike the server name we include inside the Encrypted Client Hello. The public_name we use below is example.com`

You should obviously substitute your chosen DNS names.

Generate an ECH key pair

For ECH to work, we need a new asymmetric key pair. (Different from those used for TLS server authentication), so next we generate an ECH key pair:

# mkdir /etc/echkeydir/
# openssl ech -public_name example.com -pemout /etc/echkeydir/example.pem.ech

Note that we need this file later to set up DNS as well as nginx. Nginx will make use of the private key from the ECH key file, whilst the DNS will be used to publish the corresponding public key in an HTTPS resource record (that’s where ECH-enabled browsers will look for ECH public keys).

The file name chosen should end with .pem.ech

Set up DNS

There are many ways in which one can set up the DNS records required to enable experimenting with ECH. We document two ways here, the first is where the main DNS name with which you’re experimenting is newly registered DNS name and has no existing DNS set up. The second describes a case where a DNS name exists but we need to add some new resource records to enable ECH.

In both cases, <IP> should be the IPv4 address for the host running the web server. (We leave IPv6 handling as an exercise for the reader, but there’s nothing ECH-specific required for IPv6.)

Standalone/New setup : dnsmasq as a authoritative DNS server to serve the ECH key (on the same VM)

In this case we also need to pick some DNS nameserver names, we assume the authoritative primary nameserver for both web server DNS names is ns.example.com with a secondary of ns2.example.com`

The DNS set up to use starting from a clean dnsmasq install:

# apt install dnsmasq
# cat <<EOF > /etc/dnsmasq.d/example.conf
no-resolv
no-hosts
auth-server=ns.example.com,ns2.example.com
auth-zone=example.com
auth-sec-servers=ns2.example.com
auth-soa=42,admin.example.com
host-record=example.com,<IP>
host-record=hidden.example.com,<IP>
dns-rr=hidden.example.com,65,<ascii-hex encoded HTTPS rdata>
EOF
# systemctl restart dnsmasq

Note that you need to bump the SOA version (42 in the above) every time you change the config and make sure the zone distributes to ns2.

The <ascii-hex encoded HTTPS rdata> will be published as the HTTPS resource record (type == 65) for hidden.example.com and can be produced from the file we generated earlier via a shell script you can download and use as follows:

$ _curl_ https://raw.githubusercontent.com/sftcd/openssl/ECH-draft-13c/esnistuff/pem2rr.sh -o pem2rr.sh
$ chmod u+x pem2rr.sh
$ ./pem2rr.sh /etc/echkeydir/example.pem.ech
0001000005003b0039fe0d0035db0020002059907d619054c907a1f296ceb63dde1d57f72f15db172601a2f6b55e66e7cd0f00040001000100066261722e69650000
$

Let’s assume you have an existing bind-based DNS setup for example.com, but are moving the IP address for that to a new VM that'll run the ECH-enabled web server for bothexample.comandhidden.example.com`. The change you might then make using the bindnsupdate` command on the authoritative DNS server would then look like:

Change/add IP address records for our DNS names.

$ sudo nsupdate -l
> update delete example.com a
> update delete example.com aaaa
> update add example.com 300 a <IP>
> update add hidden.example.com 300 a <IP>
> send
> quit
$

Add new HTTPS resource record for hidden.example.com

First we need to get the base64 encoded public from our /etc/echkeydir/example.pem.ech file:

$ tail -2 /etc/echkeydir/example.pem.ech | head -1
ADr+DQA2mQAgACAF7cLT+KtK0oR2DrRCGXmzBWM1eHgJgDEGLqL644/OcAAEAAEAAQAHaG9iYS5pZQAA
$

That base64 encoded value is what we need to publish in the DNS…

$ sudo nsupdate -l
> update delete hidden.example.com HTTPS
> update add hidden.example.com 300 HTTPS 1 . ech=<base64-encoded-value>
> send
> quit
$

Checking your DNS set up

You can check that value is correctly published in the DNS e.g. using the dig command`:

$ dig +short https hidden.example.com
1 . ech=ADr+DQA2mQAgACAF7cLT+KtK0oR2DrRCGXmzBWM1eHgJgDEGLqL644/OcAAEAAEAAQAHaG9iYS5pZQAA
$

If your version of dig is older and doesn’t know about HTTPS resource records, then you may need to check via:

$ dig +short +unknownformat -t TYPE65 hidden.example.com
\# 67 0001000005003C003AFE0D0036990020002005EDC2D3F8AB4AD28476 0EB4421979B30563357878098031062EA2FAE38FCE70000400010001 0007686F62612E69650000
$

The 2nd-last invocation of dig above shows the presentation format version of the HTTPS resource record. The last invocation above shows the equivalent in ascii-hex, which (minus the spaces) is what dnsmasq needs in it’s configuration file.

We assume dnsmasq will likely accept presentation format for HTTPS resource records in future as bind tooling already.

Set up nginx to serve an outer and an inner (hidden) website

Install nginx-light from the DEfO repo, as above:

# apt install _nginx_-light certbot python3-certbot-_nginx_

If you don’t already have a web server config for example.com then you can create a basic pair of sites-enabled configuration files via the following commands:

# rm /etc/_nginx_/sites-enabled/default
# cp /etc/_nginx_/sites-available/default /etc/_nginx_/sites-enabled/example.conf
# sed -i 's/server_name _;/server_name example.com;/' /etc/_nginx_/sites-enabled/example.conf
# echo "ssl_echkeydir /etc/echkeydir/;" >> /etc/_nginx_/sites-enabled/example.conf
# cp /etc/_nginx_/sites-available/default /etc/_nginx_/sites-enabled/hidden.example.conf
# sed -i -e 's/server_name _;/server_name hidden.example.com;/' -e sed 's/\(listen.*\)default_server;/\1;/' /etc/_nginx_/sites-enabled/hidden.example.conf
# systemctl restart _nginx_

It’s possible the sed commands above may not work for you, e.g. if default files change, or perhaps you won’t use those if you have an existing web server config for example.com. In that case you can simply edit the config files to ensure the following, before runningcertbot``:

you have example.com and hidden.example.com servers set up with correct server_name`
optionally change the DocRoot (root) in these configuration files to serve different content
there’s a line within the http stanza as follows: ssl_echkeydir /etc/echkeydir/;`
the ssl_echkeydir line can in _nginx_.conf within the http stanza or within either of the files in sites-enabled when it must be outside the server stanza

Run certbot to get a TLS server certificate

Next you need to rRun certbot to get TLS server public keys certificates for both DNS names (example.com` and `hidden.example.com). Those can be in the same certificate (or not) for the purposes of this experimental setup.

# certbot --nginx
...iteractions...
#

certbot may prompt you for e.g. an email address or approval and if you have an existing certificate for example.com you may be prompted as to whether you want to add hidden.example.com to that certificate.

Use curl to test the set up

Install our ECH-enabled curl package from the DEfO repo and run:

$ curl -v --ech true --doh-url https://1.1.1.1/dns-query https://hidden.example.com/ |& grep Succeeded

You should see this in the output:

ECH: result: status is Succeeded, inner is hidden.example.com, outer is example.com

Check with browsers

If you’re running a recent browser version (chromium-based since version 105, firefox since before then:-) you can enable ECH in the browser by following our client configuration instructions. Unfortunately, browsers don’t provide user interface to show if ECH has succeeded or not, but you can verify that your browser is able to use ECH if you visit https://defo.ie/ech-check.php.

Feedback

All going well, you should now have an ECH-enabled web site and be able to extend/play with that as you like. We’d appreciate feedback on this guide if you have a chance. As before, comments/questions are welcome via email or as GitLab issues/merge requests.

DEfO - Developing ECH for OpenSSL (round two)

Thu, 09 Nov 2023 00:00:00 +0000

Encrypted ClientHello (ECH) plugs a privacy-hole in TLS, hiding previously visible details from network observers. The most important being the name of the web-site the client wishes to visit (the Server Name Indication or SNI). This can be a major privacy leak, like when accessing a dissident news source hosted on a Content Delivery Network (CDN). A visible domain name also provides a straightforward method for censors to block websites and internet services. Tolerant Networks Limited and the Guardian Project successfully ran the OTF-funded DEfO project that developed interoperable implementations of ECH for OpenSSL, Conscrypt and, via those libraries, a range of ECH-enabled web servers and clients. This second funded project, DEfO-2, is a timely continuation of that project from the same the team. As needed for disambiguation, we use DEfO-1 to refer the completed project and DEfO-2 for this current project. When there’s no ambiguity, we use the DEfO acronym to cover both past and future work related to ECH for OpenSSL, related applications and other TLS stacks.

As the IETF standard for ECH completes, our key objectives are to:

Upstream DEfO code
Integrate ECH into more clients and servers
Gain and document operational experience
Submit key code for red team audits
Publish open-source ECH provisioning tools

The key challenges we expect to face in meeting those objectives are: firstly, dealing with the OpenSSL and other upstream code bases (e.g. nginx, Apache HTTP Server) - satisfying upstream developers when dealing with complex code changes, as are involved here, has proven to be quite time and effort consuming. Secondly, it is a challenge to arrange the trials we have envisaged for DEfO-2 but doing so should help to demonstrate that web sites can easily and safely enable ECH without putting themselves at risk of interoperability failures or adverse attention from censors and without further centralising the web. Lastly, there are some remaining technical challenges not addressed in DEfO-1 (proprietary TLS ClientHello extension handling, interactions between TLS Hello Retry Request and ECH, and privacy analyses of split-mode ECH deployments) that we plan to address in DEfO-2.

The key challenges we aim to mitigate for users is the ease with which user activity can be tracked and blocked based on clear text SNI. Secondarily, our focus on web-server integrations and provisioning mechanisms for ECH addresses Internet centralisation (which itself poses potential risks for censorship) by ensuring this technology can be easily deployed without having to depend on entities such as global-scale CDNs.

The primary gaps addressed by DEfO are: the privacy-leak that is clear text SNI in TLS and secondly that nobody else has been developing an ECH implementation for OpenSSL, which is one of the most widely used TLS stacks, particularly for web servers. That situation has not changed since DEfO-1 started. Arguably filling that gap has become more pressing as some browsers now ship with ECH support.

ECH is designed to contribute to the safety of users by removing one the the main remaining aspects of the web that allows network observers to easily monitor and censor web traffic based on either client DNS queries (browsers typically only use ECH when DoH has been used) and the Server Name Indication (SNI) in the TLS handshake, which is encrypted via ECH. The eventual goal is that use of ECH becomes near ubiquitous, and that goal is very achievable for web sites that make use of a CDN. DEfO however also has a focus on ECH support in various web servers and proxies (Apache, nginx, lighttpd, HAProxy) so that users of deployments that don’t use a commercial CDN can also benefit from ECH. The result of using ECH should be that neither the DNS query nor the TLS exchange leak the name of the web server with which the browser is establishing contact, thus taking away a still-easy opportunity for monitoring and censorship.

Censors however, especially at the nation-state level, might choose to block all uses of ECH, which is something that is to be expected. The main mitigation for that envisaged is that browsers, even while not using ECH, will emit “fake” (or GREASEd) ECH values, thus increasing the costs if a censor decides to block all use of ECH. The extent to which GREASEing will be an effective mitigation for blocking all ECH will essentially end up as a political/commercial decision for censors, browser makers, and web sites, but what we can say is that for now at least, browser makers and the larger CDNs do seem committed to making use of ECH. So we can have some hope that even the most capable censors might have to think hard before blocking all ECH. In DEfO-2 we are also planning some significant-scale trials that, if successful, should go a long way towards helping other significant web sites overcome fears related to enabling ECH. Overcoming a fear that one’s web site may be blocked if one deploys ECH will be a valuable result of DEfO-2 should our trials come to fruition as we hope.

We do see a number of usability issues for those deploying web servers that need to be addressed, and that we plan to address in DEfO-2. Our approach is to aim for the same level of usability for web server administrators as has been achieved by certbot as it interacts with Let’s Encrypt or other CAs. Making it easy to enable ECH, especially for “smaller” web properties is high priority for DEfO.

The outcome for which we hope is the upstreaming of ECH into important code bases, and to have demonstrated that one can deploy ECH easily at either small or large scale. The impact we expect is that we continue to significantly contribute to the use of ECH becoming near ubiquitous.

Timeliness

The time is now ripe for DEfO-2:

Firefox now supports ECH by default.
Chrome supports ECH in 10% of stable releases as of August 2023.
Brave now also supports ECH, behind the same flags as Chromium
These browser developments, plus the server code developed by DEfO-1, now enable us to plan real-world experiments
Cloudflare has beta support for enabling ECH.
During the run-time of DEfO-2 we expect to be in a position to run trials with significant players that could significantly assist with the goal of making use of ECH common for large web sites.
The DEfO-2 project timeline should also cover the finalisation of the IETF specification for ECH, significant progress on ancillary specifications (e.g. for provisioning) and provide sufficient time for upstreaming of DEfO code
DEfO-2 benefits from the same team as DEfO-1 – Tolerant Networks and the Guardian Project having co-operated successfully on DEfO-1 are looking forward to continuing that collaboration and to extending the team for DEfO-2
HPKE, a core part of ECH, but with broader applicability, is now RFC9180 and the OpenSSL maintainers merged our DEfO code for HPKE in November 2022.

Our development projects

The DEfO project implemented Encrypted ClientHello (ECH) support for OpenSSL and Conscrypt, carried out interoperability testing of those implementations, and also used those libraries to ECH-enable various web servers and clients. We deployed services using these web servers and the DNS infrastructure required to support automated key upated for the HTTPS RRs associated with those services. Here we provide a short overview of that work in order to help with larger scale experiments and with further development of the ECH specification.

Libraries

As part of the DEfO project, we ECH-enabled two important TLS libraries:

Conscrypt is a Java Security Provide (a library) that provides a Java “wrapper” for the C++ language boringssl library. Conscrypt is commonly used as the TLS provider for applications running on Android devices and is thus an attractive target to allow many clients to be ECH-enabled. (We do not target browser clients in DEfO as work on ECH-enabling those is being done by browser-makers.) The authors of boringssl (Google) have added ECH support to a version of their code, and we used that to enhance Conscrypt to call the new borinssl APIs required to use ECH and to provide mechanisms for applications to default to, or signal use of, ECH.
OpenSSL is a long-lived library providing cryptographic and TLS services that is used by many applications, including many web servers and hence is an attractive target for ECH-enabling, especially for server-side functionality. Our ECH-enabled fork of OpenSSL is here.
We have and will continue to work with the developers of other TLS libraries (e.g. wolfSSL) to assist in ECH-enabling their code, mostly via interoperability testing.

Clients

We ECH-enabled implemented the following TLS client applications:

OpenSSL s_client - this client application comes as part of the OpenSSL build but is commonly used for testing and as an extremely simple scriptable TLS client.
curl is a widely-used command line web client that can use OpenSSL for TLS support, so we ECH-enabled that.
F-Droid is an Android client application that provides an installable catalogue of FOSS applications and that uses Conscrypt. We made an ECH-enabled build called DEfO ECH Apps

Servers

We ECH-enabled implemented the following web servers:

The Apache HTTP Server is one of the two most commonly used web servers today, you can try our ECH-enabled fork.
nginx is the other web server in the “top two.”, you can try our ECH-enabled fork.
lighttpd is a web server that is commonly used on smaller devices such as home routers, you can try our ECH-enabled fork.
HAProxy is widely used as an HTTP ingress proxy and so is a good target for exploring ECH split-mode, you can try our ECH-enabled fork.
OpenSSL s_server - this example server application is part of the OpenSSL build and is commonly used for testing and experimentation.

Test tools

Amongst the test tooling we developed are:

Comprehensive OpenSSL make test targets for HPKE and ECH.
ECHInteropTest is a Java client for interoperability testing on Android.
echdnsfuzz is a catalogue of “interesting” ECHConfigList values that could cause issues for clients combined with a service to randomly select one of those for publication in the public DNS every 30 minutes. This is a useful part of fuzz-testing an ECH-enabled client application.

Issues Arising

We saw the following issues that could benefit from further work to ease deployment of ECH:

For HAProxy, since DEfO-1 we have achieved support for HelloRetryRequest in ECH split-mode but have further work to do on ECH key rotation.
There can sometimes be a lack of clarity as to which software component should be responsible for choosing to attempt real (i.e. non-GREASEd) ECH, and hence to be responsible for the additional DNS queries required to acquire an ECHConfig. For libraries like OpenSSL and boringssl it clearly only makes sense for that decision (and hence any new DNS handling code) to be outside the library. For a browser, it as clearly makes sense for that code to be in the browser application layer. With “middleware” though, such as OkHTTP or Conscrypt it can be hard to know which is the correct decision.
The new DNS code required for handling HTTPS RRs is not too complex but the full generality of SVCB is extremely complex. Adding such complexity (and associated caching) is a major change for clients like curl that have to date only had to have a very simple model for DNS - essentially only querying A/AAAA and having almost trivial caching in the application itself.
The client implementation of ECH is relatively complex in that it “touches” the TLS state machine in many ways, and hence requires changes in a lot of places. As well as increasing the cost of implementing this also increases the costs associated with testing and upstreaming.
Achieving the same level of usability as certbot for web server administrators may be challenging, but is an important goal to make it easy for web server administrators to be able to easily deploy ECH.

Conclusions

ECH is demonstrably implementable and can be deployed. We don’t yet know if new issues will become apparent as larger-scale experiments are carried out, but we should find out during the run-time of DEfO-2.

IETF116 Conference Report: Tuesday March 28, 2023

Wed, 29 Mar 2023 00:00:00 +0000

Day Two of the 116th IETF meeting in Yokohama Japan. For the rundown on Day One, see my daily report.

The OHAI Working Group has submitted the core draft of Oblivious HTTP Application Intermediation to the RFC Editor for editorial finalization and publication. OHAI is designed to support transational uses of the HTTP protocol that seek IP address privacy (by means of a relay pair, one associated with the client and one associated with the target resource). The target resource is, thus, said to be oblivious to the requester’s IP address. While the initially-imagined use case for OHAI was access to the DNS service (with some in the IETF feeling DNS-over-HTTP did not go far enough to protect user privacy), the dominant use case imagined today is telemetry - monitoring vendor-, application- or operating system-defined usage parameters on centralized systems.

It’s fair to ask how OHAI-capable services are to be discovered. The OHAI Working Group is proposing to use DNS Service Binding Records (SVCB, defined here and well-described here). TLS 1.3 Encrypted Client Hello is among the other IETF standards leveraging SVCB. SVCB records, similar to the (also new) HTTPS records, allow a host of define multiple ways to make connection with parameterization using just the DNS lookup (as opposed to the multiple round-trip mechanism of HTTP’s Alt-Svc header). The draft - Discovery of Oblivious Services via Service Binding Records has been under Working Group discussion since IETF113 and is close to submission for last call (technical completion).

Speaking of TLS 1.3 and Service Binding Records, Encrypted Client Hello ECH was, in fact, the reason SVCB records were defined. DNS lookup is the only point at which certain cryptographic information can be made available before all the connection encryption starts to happen. Unfortunately, the SVCB draft is stuck in the RFC Editor work queue behind ECH, even though there is language in ECH that references SVCB. The Working Group is therefore required to remove all the SVCB language from the ECH draft so that it may proceed. This procedural detail - and accompanying delay - has important consequences: the major open source software package that requires modification before ECH can expect wide adoption (OpenSSL) is waiting for ECH to become a full-fledged RFC before the submitted pull requests will be granted. Meanwhile there is plenty of experimentation with ECH happening on the live Internet with Mozilla/Firefox and Cloudflare running experiments.

It turns out there’s an alternative (or perhaps parallel) proposal for TLS 1.3 - CompactTLS - an effort begun in 2019 and now in it’s eighth revision. cTLS proposes to save bandwidth by trimming obsolete material, tighter encoding, a template-based specialization technique, and alternative cryptographic techniques. cTLS is not directly interoperable with TLS 1.3 or DTLS 1.3 since the over-the-wire framing is different. cTLS is undergoing formal analysis and implementation work proceeds, though no results are currently available.

The HTTPbis Working Group formally adopted the HTTP Unprompted Authentication specification in the month before IETF116. As a reminder, Unprompted Authentication allows a server to offer authenticated services without advertising that it does so (meaning the authenticated resources can not be actively probed). A number of semantic elements are under discussion, mostly to understand how this new feature is unique among existing HTTP features (and can, or can not, be aligned with adjacent capabilities). An action item for IETF117 is a formal analysis of the cryptography used, with several alternatives being proposed.

IETF: Year End Review 2021

Thu, 23 Dec 2021 00:00:00 +0000

In terms of potential impact on Internet Freedom, it’s been a banner year at the Internet Engineering Task Force (IETF). QUIC (featuring the improved privacy and security of TLS1.3) reached Proposed Standard status, with implementations and rollouts from every major vendor on both server and client, and with multiple open source toolkit options for developers. Encrypted Client Hello for TLS1.3 gained traction via the DEfO project that, through pull requests, makes a huge privacy enhancement easily available to the major security library (OpenSSL) underpinning the Internet’s most important service engines (nginx, apache, lighttpd, haproxy on the server, even curl on the client). IP address privacy got new attention with a working group formed around Oblivious HTTP Application Intermediation (OHAI), as did Privacy-Preserving Measurement (PPM) which seeks to drastically reduce the amount of personal information swept up in the pervasive monitoring of all public Internet activity. Meanwhile, the Internet Research Task Force (IRTF) has focused on developing new cryptographic techniques to serve these rapidly-evolving privacy-focused activities. IRTF also fosters work on truly-global Internet access and, in a sense, serves as the IETF’s conscience through it’s work on the human rights implications of protocol design.

Yet, it’s valid to ask: will the Internet be more free when/if these proposals become widely-deployed standards? While there’s sound reason for hope, I see two major concerns that could significantly reduce the positive impact of this work.

The first concern is extreme pushback from the major state actors who want more control over how their citizens use the Internet. While QUIC, TLS1.3 and ECH appear to be headed for full adoption in regions where the Internet is already nominally free, they amount to a royal flush for the good guys in the regions where it is not. The reaction? Stop playing the game (the major state actors have already blocked QUIC). What’s going to be the reaction when ECH makes ubiquitous domain-fronting possible? Will that just increase the number of states participating in blocking QUIC? It’s important to remember, state actors have options.

The second concern is centralization. The Internet has gotten more centralized over time, both in the way the breadth of its content is hosted and in the number of services users visit to view that content. Further, national gateways have seen massive consolidation in the last five years as most states reduce exposure in the name of terrorism prevention and reducing the flow of disinformation. This centralization has worsened the pervasive monitoring all users suffer under and has made it difficult (at best) to trust service providers with even the most benign-seeming of our personal data. Yet, the privacy solutions under development at IETF - OHAI and PPM, along with Privacy Pass and others - all rely on layers of new infrastructure operated by these same providers, seeming to require us to put even more trust in them. Directionally, is that sustainable? It’s important to remember, we users also have options.

During 2022, I’ll be looking more critically at how IETF addresses these concerns. IETF continues to be the most open and responsive standards organization determining the Internet’s future. It feels like we’re reaching a point where Internet Freedom isn’t any longer a niche interest, but rather the central focus of progress for mankind in terms of our intercommunication as citizens of the planet. The best option would seem to be an IETF at the forefront of that progress.

Implementing TLS Encrypted Client Hello

Tue, 30 Nov 2021 00:00:00 +0000

As part of the DEfO project, we have been working on accelerating the development Encrypted Client Hello (ECH) as standardized by the IETF. ECH is the next step in improving Transport Layer Security (TLS). TLS is one of the basic building blocks of the internet, it is what puts the S in HTTPS. The ECH standard is nearing completion. That is exciting because ECH can encrypt the last plaintext TLS metadata that it is possible to encrypt. So ECH will bring some real improvements in privacy and censorship resistance.

ECH is built on top of TLSv1.3 and completes unfinished work from the TLSv1.3 effort, now that private DNS is a thing. ECH is now in draft-13 and there are many implementations that are interoperating. ECH is working for OpenSSL, BoringSSL, nginx, Apache HTTPD, lighttpd, HAProxy, Conscrypt, curl, and more. There is work underway in Firefox and Chromium. It has been sketched out for OkHTTP. Draft versions of ESNI and ECH have been deployed in Firefox releases and some production web services.

The main downside of ECH is that it is complex, and most of that is unavoidable. Large scale websites have complicated server-side setups, and ECH has to work in those setups. That is where ECH’s complexity comes from. DNS SVCB and HTTPS RR Types is a related standard to encapsulate the complexity of large scale websites a single DNS lookup. It adds additional complexity since it might include the TCP port number for the service, while TLS implementations mostly assume that the port number is known before doing a DNS lookup.

ECH was formerly known as Encrypted SNI (ESNI), which it replaces. So do not expect updates to ESNI unless ECH proves too complicated for implementers. ECH is more complex to implement in the TLS library than ESNI. Once it is implemented in the TLS library, then implementing the next level of managing the keys and hostnames is quite similar between ESNI and ECH. The rest of this post will be diving into details about that complexity and what it takes to implement ECH.

Do Not Stick Out

ECH implementers must also work to ensure that their implementation does not create packets that look look different than other implementations. How ECH is implemented will affect fingerprintability. Right now, BoringSSL, OpenSSL, and Firefox all put the ECH extension in different places, so it is easy for a network observer to tell which of them is serving as the client in a TLS session. Ideally, all implementations would create network packages that look identical to the network observer. To get an idea of this problem in the real world, see this detailed technical discussion of metadata ordering in DTLS as used in WebRTC.

Getting the keys

ECH encrypts the first stage of the negotiation between the client and server, the ClientHello packet using a new type of key and related configuration. This is known as an “ECH Config”. This negotiation is what sets up the encrypted TLS connection, so to encrypt the negotiation, so the keys can not be derived from the certificates and negotiation itself. ECH also needs new methods distributing this ECH Config. HTTPS/SVCB therefore includes ECH Config Lists, and DNS is the recommended way to get ECH Config Lists. Other distribution methods are also possible, and ECH implementation should provide a method to directly supply ECH Config Lists.

When using ECH in conjunction with HTTPS/SVCB, the DNS needs to be handled differently than the common, decades old practices that most TLS stacks are built on. The ECH Config List must match the IP address so that multi-CDN setups still work. That ensures the server gets the ECH Config that matches its ECH private keys, not the one for a separate server. If the A/AAAA lookup is done separately from the HTTPS/SVCB lookup, they might not match since the results could come from different caches, etc. It is not yet clear where the right place to handle that DNS is, but it is clear that clients that implement HTTPS/SVCB and ECH will need to consider that DNS will touch different parts of the stack.

Which part of the stack will handle the HTTPS/SVCB look up depends on whether the code is acting as a library or more as the client itself. For example, libraries like OkHTTP or Conscrypt can both act as a TLS library, providing the blocks for implementing TLS in an app. They can also act as the TLS client itself, the app just needs to open the hostname and port to get a connection. This is especially true for OkHTTP, which can also handle DNS itself with its own DNS-over-HTTPS implementation. Most apps will expect to just open a URL and have it work without providing anything else. In the case of HTTP libraries like OkHTTP or Volley, Conscrypt becomes the provider of TLS functions rather than the client. OkHTTP’s default setup will probably want to treat Conscrypt like the client and do zero config. But there are use cases where apps want to handle DNS in OkHTTP, in that case Conscrypt is a library. Another way this could be broken down would be to have Conscrypt only be the TLS library, then leave the DNS to the HTTP stacks. Then each HTTP library would have to have their own ECH and HTTPS/SVCB handling (In Android, that would mean HttpURLConnection, OkHTTP, Volley, Apache HTTP Client, etc). Python provides another example with HTTP libraries like Requests working in combination with the core ssl module.

One thing that is clear here: whichever piece initiates the HTTPS/SVCB DNS query should also handle the ECH Config, and setting up the lower levels. On top of this, ECH has GREASE and “Retry Configs”. GREASE is a key part of ensuring that ECH connections do not stick out. GREASE makes the network observer see TLS packets that look the same as connections that successfully used an ECH Config to encrypt. Retry Configs are sent by the server if it cannot decrypt what the client sent. Whichever piece of the code handles the ECH Config List should then also handle GREASE and the Retry Configs.

Outer and Inner ClientHello

This post began by saying that ECH was complex. Most of that complexity stems from the concept of ClientHelloOuter and ClientHelloInner. They are sort of parallel versions of the TLS ClientHello, with ClientHelloOuter being plaintext and ClientHelloInner being encrypted. Many of the same bits of metadata can be set in either ClientHelloOuter or ClientHelloInner, or even in both. And the outer and inner values can be different. For example, the outer plaintext SNI could be set to a generic domain name, while the inner could be set to a different domain name that the client wants to protect. Whenever the client needs to manage what is set in “inner”, “outer”, etc., then extra complexity is exposed all the way up to the app using the ECH client. Also, so many possible configuration options also means so many ways that the negotiation might fail.

BoringSSL’s SSL_set1_ech_config_list() provides a good example of this complexity:

If a supported ECHConfig is found, ssl will encrypt the true ClientHello parameters. If the server cannot decrypt it, e.g. due to a key mismatch, ECH has a recovery flow. ssl will handshake using the cleartext parameters, including a public name in the ECHConfig. If using SSL_CTX_set_custom_verify, callers should use SSL_get0_ech_name_override to verify the certificate with the public name. If using the built-in verifier, the X509_STORE_CTX will be configured automatically.

If no other errors are found in this handshake, it will fail with SSL_R_ECH_REJECTED. Since it didn’t use the true parameters, the connection cannot be used for application data. Instead, callers should handle this error by calling SSL_get0_ech_retry_configs and retrying the connection with updated ECH parameters. If the retry also fails with SSL_R_ECH_REJECTED, the caller should report a connection failure.

A walk through which pieces needs to do what

Let us return to the Python example to go through the pieces needed to implement ECH there. Starting at the lowest level, the Python ssl is built on top of OpenSSL, so the new ECH functions in OpenSSL need to be included and exposed in ssl’s API. Unlike with a plain TLSv1.3 connection, ssl would not set up an ECH connection without additional configuration as long as ssl does not include the additional DNS requirements needed to fetch the ECH Config. The DNS could instead be implemented in the HTTP library. Requests is built on top of urllib3, so urllib3 could implement HTTPS/SVCB handling. Then Requests would automatically get ECH connections. Since it can be necessary to handle the ECH Config outside of DNS, urllib3 and Requests could provide methods for the app to submit the ECH Config. This would mean overriding any HTTPS/SVCB handling in Requests and urllib3.

Based on this outline, these are the pieces that need to be implemented:

HTTPS/SVCB DNS queries for getting ECH Config Lists, IP, and other config.
A way to provide ECH Config Lists as bytes directly to the ssl module.
A method to ensure encrypted DNS is used so all metadata is encrypted.
A method to enable and disable GREASE, ideally with GREASE default on.
A callback that gets called whenever ECH negotiation fails and the server offers a “Retry Config”.
Potentially also controls of what to send in the ClientHelloOuter versus the ClientHelloInner.

From the point of view of the client implementation, there is a big difference between TLS before and after ECH. This new complexity may look scary but this is a clear path to providing strong privacy protections with TLS that can be as widely deployed as TLSv1.3 is now.

Security in a thumb drive: the promise and pain of hardware security modules, take one!

Fri, 28 Mar 2014 16:54:39 -0400

Hardware Security Modules (aka Smartcards, chipcards, etc) provide a secure way to store and use cryptographic keys, while actually making the whole process a bit easier. In theory, one USB thumb drive like thing could manage all of the crypto keys you use in a way that makes them much harder to steal. That is the promise. The reality is that the world of Hardware Security Modules (HSMs) is a massive, scary minefield of endless technical gotchas, byzantine standards (PKCS#11!), technobabble, and incompatibilities. Before I dive too much into ranting about the days of my life wasted trying to find a clear path through this minefield, I’m going to tell you about one path I did find through to solve a key piece of the puzzle: Android and Java package signing.

For this round, I am covering the Aventra MyEID PKI Card. I bought a SIM-sized version to fit into an ACS ACR38T-IBS-R smartcard reader (it is apparently no longer made, and the ACT38T-D1 is meant to replace it). Why such specificity you may ask? Because you have to be sure that your smartcard will work with your reader, and that your reader will have a working driver for you system, and that your smartcard will have a working PKCS#11 driver so that software can talk to the smartcard. Thankfully there is the OpenSC project to cover the PKCS#11 part, it implements the PKCS#11 communications standard for many smartcards. On my Ubuntu/precise system, I had to install an extra driver, libacr38u, to get the ACR38T reader to show up on my system.

So let’s start there and get this thing to show up! First we need some packages. The OpenSC packages are out-of-date in a lot of releases, you need version 0.13.0-4 or newer, so you have to add our PPA (Personal Package Archive) to get current versions, which include a specific fix for the Aventra MyEID: (fingerprint: F50E ADDD 2234 F563):

sudo add-apt-repository ppa:guardianproject/ppa
sudo apt-get update
sudo apt-get install opensc libacr38u libacsccid1 pcsc-tools usbutils

First thing, I use lsusb in the terminal to see what USB devices the Linux kernel sees, and thankfully it sees my reader:

$ lsusb
Bus 005 Device 013: ID 072f:9000 Advanced Card Systems, Ltd ACR38 AC1038-based Smart Card Reader

Next, its time to try pcsc_scan to see if the system can see the smartcard installed in the reader. If everything is installed and in order, then pcsc_scan will report this:

$ pcsc_scan 
PC/SC device scanner
V 1.4.18 (c) 2001-2011, Ludovic Rousseau <&#x6c;u&#x64;ov&#x69;c.&#x72;o&#x75;&#x73;s&#x65;au&#x40;f&#x72;&#x65;e&#x2e;fr>
Compiled with PC/SC lite version: 1.7.4
Using reader plug'n play mechanism
Scanning present readers...
0: ACS ACR38U 00 00

Thu Mar 27 14:38:36 2014
Reader 0: ACS ACR38U 00 00
  Card state: Card inserted, 
  ATR: 3B F5 18 00 00 81 31 FE 45 4D 79 45 49 44 9A
[snip]

If pcsc_scan cannot see the card, then things will not work. Try re-seating the smardcard in the reader, make sure you have all the right packages installed, and if you can see the reader in lsusb. If your smartcard or reader cannot be read, then pcsc_scan will report something like this:

$ pcsc_scan 
PC/SC device scanner
V 1.4.18 (c) 2001-2011, Ludovic Rousseau <&#x6c;&#x75;&#x64;&#x6f;&#x76;&#x69;c.rousse&#x61;&#x75;&#x40;&#x66;&#x72;&#x65;e.fr>
Compiled with PC/SC lite version: 1.7.4
Using reader plug'n play mechanism
Scanning present readers...
Waiting for the first reader...

Moving right along… now pcscd can see the smartcard, so we can start playing with using the OpenSC tools. These are needed to setup the card, put PINs on it for access control, and upload keys and certificates to it. The last annoying little preparation tasks are finding where opensc-pkcs11.so is installed and the “slot” for the signing key in the card. These will go into a config file which keytool and jarsigner need. To get this info on Debian/Ubuntu/etc, run these:

$ dpkg -S opensc-pkcs11.so
opensc: /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
$ pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so \
>     --list-slots
Available slots:
Slot 0 (0xffffffffffffffff): Virtual hotplug slot
  (empty)
Slot 1 (0x1): ACS ACR38U 00 00
  token label        : MyEID (signing)
  token manufacturer : Aventra Ltd.
  token model        : PKCS#15
  token flags        : rng, login required, PIN initialized, token initialized
  hardware version   : 0.0
  firmware version   : 0.0
  serial num         : 0106004065952228

This is the info needed to put into a opensc-java.cfg, which keytool and jarsigner need in order to talk to the Aventra HSM. The name, library, and slot fields are essential, and the description is helpful. Here is how the opensc-java.cfg using the above information looks:

name = OpenSC
description = SunPKCS11 w/ OpenSC Smart card Framework
library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
slot = 1

Now everything should be ready for initializing the HSM, generating a new key, and uploading that key to the HSM. This process generates the key and certificate, puts them into files, then uploads them to the HSM. That means you should only run this process on a trusted machine, certainly with some kind of disk encryption, and preferably on a machine that is not connected to a network, running an OS that has never been connected to the internet. A live CD is one good example, I recommend Tails on a USB thumb drive running with the secure persistent store on it (we have been working here and there on making a TAILS-based distro specifically for managing keys, we call it CleanRoom).

HSM plugged into a laptop

First off, the HSM needs to be initialized, then set up with a signing PIN and a “Security Officer” PIN (which means basically an “admin” or “root” PIN). The signing PIN is the one you will use for signing APKs, the “Security Officer PIN” (SO-PIN) is used for modifying the HSM setup, like uploading new keys, etc. Because there are so many steps in the process, I’ve written up scripts to run thru all of the steps. If you want to see the details, read the scripts. The next step is to generate the key using openssl and upload it to the HSM. Then the HSM needs to be “finalized”, which means the PINs are activated, and keys cannot be uploaded. Don’t worry, as long as you have the SO-PIN, you can erase the HSM and re-initialize it. But be careful! Many HSMs will permanently self-destruct if you enter in the wrong PIN too many times, some will do that after only three wrong PINs! As long as you have not finalized the HSM, any PIN will work, so play around a lot with it before finalizing it. Run the init and key upload procedure a few times, try signing an APK, etc. Take note: the script will generate a random password for the secret files, then echo that password when it completes, so make sure no one can see your screen when you generate the real key. Alright, here goes!

code $ git clone https://github.com/guardianproject/smartcard-apk-signing
code $ cd smartcard-apk-signing/Aventra_MyEID_Setup
Aventra_MyEID_Setup $ ./setup.sh 
Edit pkcs15-init-options-file-pins to put in the PINs you want to set:
Aventra_MyEID_Setup $ emacs pkcs15-init-options-file-pins
Aventra_MyEID_Setup $ ./setup.sh 
Using reader with a card: ACS ACR38U 00 00
Connecting to card in reader ACS ACR38U 00 00...
Using card driver MyEID cards with PKCS#15 applet.
About to erase card.
PIN [Security Officer PIN] required.
Please enter PIN [Security Officer PIN]: 
Using reader with a card: ACS ACR38U 00 00
Connecting to card in reader ACS ACR38U 00 00...
Using card driver MyEID cards with PKCS#15 applet.
About to create PKCS #15 meta structure.
Using reader with a card: ACS ACR38U 00 00
Connecting to card in reader ACS ACR38U 00 00...
Using card driver MyEID cards with PKCS#15 applet.
Found MyEID
About to generate key.
Using reader with a card: ACS ACR38U 00 00
Connecting to card in reader ACS ACR38U 00 00...
Using card driver MyEID cards with PKCS#15 applet.
Found MyEID
About to generate key.
next generate a key with ./gen.sh then ./finalize.sh
Aventra_MyEID_Setup $ cd ../openssl-gen/
openssl-gen $ ./gen.sh 
Usage: ./gen.sh "CertDName" [4096]
  for example:
  "/C=US/ST=New York/O=Guardian Project &#x54;e&#x73;&#x74;/&#x43;N=&#x74;es&#x74;.&#x67;&#x75;a&#x72;di&#x61;np&#x72;o&#x6a;&#x65;c&#x74;&#x2e;i&#x6e;fo&#x2f;em&#x61;i&#x6c;&#x41;d&#x64;re&#x73;s=&#x74;e&#x73;&#x74;@&#x67;&#x75;a&#x72;di&#x61;np&#x72;o&#x6a;&#x65;c&#x74;.i&#x6e;fo"
openssl-gen $ ./gen.sh "/C=US/ST=New York/O=Guardian Project Te&#x73;t&#x2f;C&#x4e;=&#x74;e&#x73;t&#x2e;g&#x75;ardi&#x61;n&#x70;r&#x6f;j&#x65;c&#x74;.&#x69;n&#x66;o/em&#x61;i&#x6c;A&#x64;d&#x72;e&#x73;s&#x3d;t&#x65;st&#x40;g&#x75;a&#x72;d&#x69;a&#x6e;p&#x72;o&#x6a;e&#x63;t.&#x69;n&#x66;o"
Generating key, be patient...
2048 semi-random bytes loaded
Generating RSA private key, 2048 bit long modulus
.......................................+++
..................................................+++
e is 65537 (0x10001)
Signature ok
subject=/C=US/ST=New York/O=Guardian Project Test/&#x43;&#x4e;&#x3d;&#x74;&#x65;st.gu&#x61;&#x72;&#x64;&#x69;&#x61;nproj&#x65;&#x63;&#x74;&#x2e;&#x69;nfo/e&#x6d;&#x61;&#x69;&#x6c;&#x41;ddres&#x73;&#x3d;&#x74;&#x65;&#x73;t@gua&#x72;&#x64;&#x69;&#x61;&#x6e;proje&#x63;&#x74;&#x2e;&#x69;&#x6e;fo
Getting Private key
writing RSA key
Your HSM will prompt you for 'Security Officer' aka admin PIN, wait for it!
Enter destination keystore password:  
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
[Storing keystore]
Key fingerprints for reference:
MD5 Fingerprint=90:24:68:F3:F3:22:7D:13:8C:81:11:C3:A4:B6:9A:2F
SHA1 Fingerprint=3D:9D:01:C9:28:BD:1F:F4:10:80:FC:02:95:51:39:F4:7D:E7:A9:B1
SHA256 Fingerprint=C6:3A:ED:1A:C7:9D:37:C7:B0:47:44:72:AC:6E:FA:6C:3A:B2:B1:1A:76:7A:4F:42:CF:36:0F:A5:49:6E:3C:50
The public files are: certificate.pem publickey.pem request.pem
The secret files are: secretkey.pem certificate.p12 certificate.jkr
The passphrase for the secret files is: fTQ*he-[:y+69RS+W&+!*0O5i%n
openssl-gen $ cd ../Aventra_MyEID_Setup/
Aventra_MyEID_Setup $ ./finalize.sh 
Using reader with a card: ACS ACR38U 00 00
Connecting to card in reader ACS ACR38U 00 00...
Using card driver MyEID cards with PKCS#15 applet.
Found MyEID
About to delete object(s).
Your HSM is ready for use! Put the secret key files someplace encrypted and safe!

Now your HSM should be ready for use for signing. You can try it out with keytool to see what is on it, using the signing PIN not the Security Officer PIN:

smartcard-apk-signing $ /usr/bin/keytool -v \
>     -providerClass sun.security.pkcs11.SunPKCS11 \
>     -providerArg opensc-java.cfg \
>     -providerName SunPKCS11-OpenSC -keystore NONE -storetype PKCS11 \
>     -list
Enter keystore password:  

Keystore type: PKCS11
Keystore provider: SunPKCS11-OpenSC

Your keystore contains 1 entry

Alias name: 1
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: &#x45;&#x4d;&#x41;&#x49;&#x4c;&#x41;&#x44;&#x44;RESS=test@g&#x75;&#x61;&#x72;&#x64;&#x69;&#x61;&#x6e;&#x70;&#x72;oject.info, CN=test.guardianproject.info, O=Guardian Project Test, ST=New York, C=US
Issuer: E&#x4d;A&#x49;LA&#x44;D&#x52;ES&#x53;=&#x74;e&#x73;&#x74;@&#x67;u&#x61;rd&#x69;a&#x6e;pr&#x6f;j&#x65;ct&#x2e;i&#x6e;f&#x6f;, CN=test.guardianproject.info, O=Guardian Project Test, ST=New York, C=US
Serial number: aa6887be1ec84bde
Valid from: Fri Mar 28 16:41:26 EDT 2014 until: Mon Aug 12 16:41:26 EDT 2041
Certificate fingerprints:
	 MD5:  90:24:68:F3:F3:22:7D:13:8C:81:11:C3:A4:B6:9A:2F
	 SHA1: 3D:9D:01:C9:28:BD:1F:F4:10:80:FC:02:95:51:39:F4:7D:E7:A9:B1
	 SHA256: C6:3A:ED:1A:C7:9D:37:C7:B0:47:44:72:AC:6E:FA:6C:3A:B2:B1:1A:76:7A:4F:42:CF:36:0F:A5:49:6E:3C:50
	 Signature algorithm name: SHA1withRSA
	 Version: 1


*******************************************
*******************************************

And let’s try signing an actual APK using the arguments that Google recommends, again, using the signing PIN:

smartcard-apk-signing $ /usr/bin/jarsigner -verbose \
>     -providerClass sun.security.pkcs11.SunPKCS11 \
>     -providerArg opensc-java.cfg -providerName SunPKCS11-OpenSC \
>     -keystore NONE -storetype PKCS11 \
>     -sigalg SHA1withRSA -digestalg SHA1 \
>     bin/LilDebi-release-unsigned.apk 1
Enter Passphrase for keystore: 
   adding: META-INF/1.SF
   adding: META-INF/1.RSA
  signing: assets/busybox
  signing: assets/complete-debian-setup.sh
  signing: assets/configure-downloaded-image.sh
  signing: assets/create-debian-setup.sh
  signing: assets/debian-archive-keyring.gpg
  signing: assets/debootstrap.tar.bz2
  signing: assets/e2fsck.static
  signing: assets/gpgv
  signing: assets/lildebi-common
[snip]

Now we have a working, but elaborate, process for setting up a Hardware Security Module for signing APKs. Once the HSM is setup, using it should be quite straightforward. Next steps are to work out as many kinks in this process as possible so this will be the default way to sign APKs. That means things like figuring out how Java can be pre-configured to use OpenSC in the Debian package, as well as including all relevant fixes in the pcscd and opensc packages. Then the ultimate is to add support for using HSMs in Android’s generated build files like the build.xml for ant that is generated by android update project. Then people could just plug in the HSM and run ant release and have a signed APK!