Passing the Torch – My last Root DNSSEC KSK Ceremony as Crypto Officer 4

November 24, 2025 by Cnznews

Many years ago, when I was an infant, the first computers were connected over the ARPANET – the seminal computer network that eventually evolved to become the Internet. Computers were big and expensive at that time; In fact the first version of NCP – the predecessor to TCP/IP – represented only about 250 computers on the network.

The mapping of names (human friendly) to network addresses (computer friendly) on this network was maintained through a “host file” – literally a flat file of ordered pairs, creating a relationship between host (computer friendly) name and address.

So it continued as computers became less expensive and proliferation increased, more institutions wanted to connect to the ARPANET due to network effects. In response, TCP/IP was developed, with support for orders of magnitude more connected computers. Also, military users of the network split into their own networks, and in the early 1980s we had the beginnings of the Internet, or “Katenet” as it was sometimes called at the time – a network of networks.

Clearly, as we went from “a few hundred computers” to “capacity of billions”, a centrally managed hosts file was not going to scale, and in the early 1980s development began on a distributed database to replace the centrally managed file. The name of this distributed database was the Domain Name System or DNS.

It is important to know that at that time, access to the Network of Networks was still limited to a select few – higher education, research institutions, military organizations and the military-industrial complex (ARPA, later DARPA, was, after all, an activity of the United States Department of Defense), and a few companies that were strongly linked to one or more of these constituencies. Widespread public commercial access to the Internet will persist for many years into the future.

It was in this environment that DNS emerged. Academics, military researchers, university students – a pretty collegial environment. Paleo-cybersecurity practices are not mentioned – in fact the term “cybersecurity” has not even been coined yet, although the notion of “computer security” dates back to the early 1970s.

I mention this brief “history of the early Internet” to answer the question that inevitably arises: Why weren’t better security built into DNS? The answer is two-fold: firstly, it was not based on the environment in which it was developed, and secondly, even if it were, the security practices would have been firmly rooted in 1980s best practices, which would certainly have been inadequate by modern standards.

The IETF began development on Domain Name System Security Extensions (DNSSEC) in 1995 due to the discovery of security flaws in the 1990s. Early versions were difficult to deploy. Later editions improved somewhat. But inertia is one thing, the status quo persists, and there was very real concern that DNSSEC would result in a net reliability zero (security vs. availability could be a tricky circle to square), concentrate power in undesirable ways, and have other unanticipated negative effects.

At the end of the day, as is often the case, there was a crisis in actually moving the ball. In 2008, Dan Kaminsky discovered a fundamental flaw in DNS that simplified cache poisoning – essentially making it possible for an attacker to misdirect users to arbitrary web sites.

In less than two years, DNS roots will be cryptographically signed – allowing people who want to sign their domains to create a cryptographic chain of trust that authenticates their DNS lookups. This is non-repudiation, not non-disclosure – DNS queries and responses clearly occurred. But this time, the responses came back with digital signatures, courtesy of DNSSEC.

David Huberman at ICANN created a great slide deck explaining how it all works.

Trust in a system requires more than technical correctness. This includes confidence in the execution of the system being run. That’s why ICANN decided to create a framework to facilitate trust and transparency. Among other things it includes:

Keeping cryptographic material at two highly secure sites, one near Washington DC and one in Los Angeles (geographic diversity)
Creating a multi-layered security system that requires multiple people to have access to a key management feature
Storing cryptographic materials in an offline HSM, which uses Shamir’s secret sharing, requires a quorum of at least 3 of the 7 crypto authorities to “wake them up”.
Trusted community representative with the roles of crypto official and recovery key share holder
Highly scripted (and therefore auditable) function related to handling cryptographic material
Live streaming of all events
Hosting external witnesses from the community who have expressed interest in attending a ceremony in person

When the call came for volunteers to be trusted community representatives, I was no stranger to community involvement, having served for several years on the ARIN Advisory Council and doing community work (and later board work) for NANOG. I was hired by a top tier domain operator, and I submitted my CV and expressed my interest.

This is how I found myself in Culpeper, Virginia in 2010 as Crypto Officer 4 at the first DNSSEC root signing ceremony. I didn’t know that I would still be able to do this fifteen years later. I was the last of the original Crypto Officers for KMF-East, and second last overall – behind only Subramaniam “SM” Moonesamy, who is Crypto Officer 7 for KMF-West.

It has been an adventure. I’ve been a small participant in Root Key Rolls, been put in a B-roll appearance on BBC Horizon (S53E13), made friends with many of the people I served with, overseen ceremonies remotely during the COVID lockdown, and witnessed an amazing pivot by ICANN staff who managed to select, test, integrate and deploy the new HSM on only 8 months’ notice, an achievement that amazes me.

I was an early advocate for improving trust in our process by taking advantage of natural turnover and backfilling existing TCRs with people selected from a broader group of qualified individuals than the Fellowship of DNS protocol experts, operators, and developers. I am grateful that our voices were heard.

On November 13, 2025, I passed the torch to Lodrina Cherney who is now Crypto Officer 4 for KMF-East. Lodrina is a security researcher, an educator with an emphasis on digital forensics, and works in security engineering at a large cloud provider. I am honored to have him as my successor.

Several people have contacted me to ask what prompted me to step back from ICANN volunteer work. Those expecting some kind of sordid mess or scandal were sorely disappointed – quite the contrary, it’s a huge success story and I’m glad I was able to play my small part. With one of them being a direct cut and paste from a Slack log like this:

What prompted you to withdraw from ICANN?

several things:

This was considered a 5 year commitment. I have been doing this for over 15 years.
Several years ago (more than a decade ago) there was broad consensus among the group that people from more diverse backgrounds than just the DNS-old-boy-network (which was the original group of TCR) was a good thing.
Many people set out first by bicycle; I was happy to let those for whom the journey was more disgusting go first. But this is only practical and it is really good for the system to issue one TCR per function.
Covid delayed it. Kaminsky’s untimely death and subsequent replacement as recovery major shareholder (RKSH) delayed this.
Another delay was caused by the sudden EOL of the AEP Keeper HSM and transition to the Thales Luna HSM. After 8 months of research, development and implementation it went off without a hitch – a record that amazes me and is a true testament to the skill and commitment of the ICANN PTI team. ICANN expressed its desire for continuity among long-serving COs after that milestone; Frederico Neves (fellow original crypto officer) and I were willing to extend our stay for this.

So in short it was time to pass the torch. Everyone is doing everything right. At the end of Ceremony 59 I commented that when we started doing this 15 years ago, success was not guaranteed; We needed the Kaminsky bug to actually deploy it. Today, major Unix DNS resolvers come with DNSSEC validation enabled. All major public DNS resolvers (Google, Quad9, Cloudflare) perform DNSSEC validation. I thanked all those responsible for the security, integrity and credibility of this process for putting their personal credibility on the line and said I was honored to be able to play a small role in making this happen.

Epilogue:

I won’t be attending most East Coast ceremonies from here, but I wouldn’t rule out an occasional appearance as an outside witness, especially at KMF-West where I’ve never been to in person.

Here are scans of our ceremony scripts from both Ceremony 59 and the previous day’s administrative ceremony.

Root DNSSEC KSK Function 59

Root DNSSEC KSK Administrative Function 59 Backup HSM Acceptance Test

Root DNSSEC KSK Administrative Function 59 Safe #2 Credentials Safe Deposit Box Lock Assembly Programming

kskm-ksrsigner-logs.pdf

<a href