Who do you trust to tell you who to trust?

The big security news of the past few days is the story of the compromise of the DigiNotar Certificate Authority and the subsequent issuing of fraudulent SSL certificates, leading to actual Man in the Middle attacks against Gmail users in Iran. If that previous sentence was gobbledygook, this post should explain some of it. It will also give you instructions on how to tell your Mac to no longer trust certificates issued by DigiNotar.

Who’s in immediate danger?

IRI to Gmail MITM

Some seemingly secure communication over SSL between computer users in Iran and google.com domains is being intercepted and read by a third party, presumably the government of Iran.

Any readers connecting to Gmail (or have other sensitive communication with google.com domains) from networks in Iran, update all your browsers immediately, only use Safari if you follow the instructions below, and pay extremely close attention to warnings from web browsers about certificates, and once you have a trustworthy connection to Google, change your Gmail passwords.

Not a Gmail compromise

It is important to note that although communication between users and Gmail has been compromised, this was not an actual break-in at Gmail. Nor is it a failure of the web browsers people use, nor is it a failure of the cryptography involved. Instead it is a failure of the “trust” infrastructure that we all rely on.

This matters to everyone

The demonstration of how the trust infrastructure can fail matters for everyone, even if they are not using Gmail from Iran. DigiNotar has acknowledged that there may be other, undetected, fraudulent certificates out there as well. Although this issue doesn’t tie in to any of our products at AgileBits, I wanted to help people understand what is going on and why it matters. I’ve actually been surprised this hasn’t been bigger news earlier.

The Trust Infrastructure of Secure (HTTPS) Web Browsing

Browsing the web using SSL/TLS (that is where the URL begins with “https://”) does two things for you. It makes sure that the communication between your web browser and the web server it is talking to is very well encrypted. The other thing that is does is give you assurance that the site you are talking to is the site that it claims to be from its network address.

If someone could trick your browser or your bit of the Internet to go to a their own site when you connect to agilebits.com, then they could create a malicious version of 1Password that you might download. But because we use SSL for crucial parts of our downloads and for our updating processes, you are assured that it really is us you are talking to. Certificate snippet

If you visit https://agilebits.com using Safari and click on the little lock icon in the upper right corner, you will see something that looks like the figure on the right. This tells you that the people claiming to be agilebits.com (that’s us!) have proved ourselves sufficiently for PositiveSSL to certify that our website certificate really does belong to us. PositiveSSL in turn has been checked out by User Trust Network (UTN) which has said that PositiveSSL can be trusted to issue certificates like the kind that they issued to us.

Now why should be trust UTN to make such declarations? They are at a root of this chain of trust, so who has checked them out to make sure that they only issue certificates that should be issued? The answer is that the people who built your web browser have made that decision. UTN is one of a couple hundred or so root Certification Authorities (CA) that are trusted by most web browsers and operating systems.

Through the magic of Public Key Cryptography (a topic for another day), these certificates are impossible to counterfeit without knowing a secret key used in the creation of the certificates. When someone at PositiveSSL creates a certificate, they have to enter a password that unlocks a particular key that is used in signing a certificate.

Man in the Middle

If someone, let’s call her Ewa, can get a certificate that says that she is google.com in a way that a user, let’s call him Azada, would trust that certificate then Ewa can intercept communication between Azada and Gmail even though that communication is encrypted. What Ewa does is pretend to be Gmail when talking to Azada, and at the same time she pretends to be Azada when talking to Gmail. When Azada thinks he is sending his username and password to Gmail, Ewa will get that username and password. Ewa will quickly re-encrypt that username and password in her communication with Gmail. So both Azada and Gmail see that they have an encrypted connection, and they think that they are talking directly to each other. But Ewa is our man in the middle. (Please note that “Man in the Middle” attacks are distinct from “Meet in the Middle” attacks even though they share the same abbreviation, MITM.)

What happened to DigiNotar?

DigiNotar is another root certification authority. On July 9 their systems were compromised. Hundreds of certificates were created by those who broke in. DigiNotar detected the break-in and revoked most, but not all, of the certificates generated by the attackers. For reasons that aren’t yet clear, they failed to issue a revocation for the google.com certificate. That’s the certificate that has been used for Man In The Middle attacks in Iran. It is also worth noting that DigiNotar did not notify anyone about the break-in at the time.

Certificate revocations are tricky thing. Browsers don’t always fetch the latest Certificate Revocation Lists (CRLs). Typically a revocation is issued when a site loses its certificate or when a business changes its name so is no longer meets the conditions under which the certificate was issued. (Our name change from Agile Web Solutions to AgileBits has played havoc with certificates for our old domain names for exactly this reason.) On problem with certificate revocations is that they don’t tell us whether the actual CA was compromised, as happened in the DigiNotar case.

No one outside of DigiNotar or the attackers knew that a CA had been compromised until some problems were reported in Iran in the last few days of August. Only since then has DigiNotar and their parent company Vasco acknowledged that there had been a breach and a large number (more than 500) bogus certificates issued. Keep in mind that because DigiNotar is a trusted root CA, they can issue certificates for any domain possible, not just for their customers. In practice DigiNotar is a small provider, but because they are (well actually, had been) trusted by browsers fraudulent certificates could be issued for any site.

What’s happened since

This is an on-going story. But Microsoft, Google, and Mozilla have removed DigiNotar from the list of trusted root certification authorities in updates to Internet Explorer, Chrome, and Firefox respectively. Untrusted certificate warning
What this means is that if you visit a site that actually uses a certificate signed by DigiNotar using those browsers your browser will give you a warning. Customers of DigiNotar are scrambling to get their certificates signed by other certification authorities, but this will take some time. After all a CA needs to verify that the person making the request for a certificate really is who they say they are.

This should not be a practical problem for those who don’t browse sites based in The Netherlands. But the Dutch government has many sites whose certificates have been signed by DigiNotar.

Untrusting DigiNotar on the Mac

[Update: September 9: Use Software Update on OS X 10.7.1 or OS X 10.6.8 to get Apple’s Security Update 2011-005. The remainder of this section on untrusting DigiNotar is moot in light of this security update from Apple.]

Apple has not (yet) removed DigiNotar from its list of trusted certification authorities, but you can do this yourself if you wish. Unless you are in Iran there is no real security gain from doing so (unless of course other fraudulent DigiNotar certificates emerge). However there is little harm in doing so unless you visit Dutch government websites.

The certificates are in a system keychain on OS X; so you will need to use Keychain Access.app to make changes.
Select DigiNotar in Keychain Access
Keychain Access.app is in the Utilities folder in the system Applications folder. Once you have launched Keychain Access, you need to

  1. Select “Certificates” in the lower left panel of Keychain Access
  2. Search for “diginotar” using the search field
  3. Double click on DigiNotar Root CA in the main panel

After you double click on DigiNotar you will be presented with a window that allows you to change the trust relationship you have with that certificate.

You will need to click on the expansion triangle by “Trust” and there you will be presented with the ability to change “When using this certificate” from “Use System Defaults” to “Never Trust”. You will be prompted for an administrator username and password to make this kind of change. You will also need to log out of OS X and back in again before the change takes effect.

[Update September 8: Because of a bug in Apple’s trust system, some certificates (EV certificates) will be trusted even if the signing certificate is “untrusted”. Therefore it is necessary to actually remove the DigiNotar certificate instead of merely setting trust to “Never”. TUAW has a nice set of instructions for removing the certificate.

Unfortunately, even removing the DigiNotar root CA certificate doesn’t entirely eliminate the problem on the Mac (though it does work better than merely untrusting the certificate). For those who are particularly adventurous see the instructions at ps | Enable ]

Again, let me stress that this is of no practical importance unless you have reason to suspect that you are on a network which where entities are trying to use fraudulent certificates to spy on you.

On the other hand, I have “de-trusted” DigiNotar, not so much for any added sense of security but because I feel that DigiNotar failed to live up to the absolutely enormous responsibility it undertook when becoming a root CA. For all we know, they just have suffered from bad luck. Maybe other certification authorities are just as vulnerable. But there is immense power in being a root CA trusted by browsers. DigiNotar’s failure to disclose the breach and their failure to revoke all of the bad certificates means to me that they shouldn’t be trusted with that power.


Pay attention to browser warnings

Man in the middle attacks are real, not just theoretical. If people routinely ignore the warnings, that MITM attacks will become more common. In this case, it was a vigilant user who noticed an inconsistency and raised the alarm by asking questions. One concern about browsers no longer trusting DigiNotar certificates is that users of those sites will “learn” to ignore warnings. People need to consider this in the decision to remove DigiNotar from the list of trusted root authorities.

Just as you shouldn’t ignore SSL error messages about untrusted certificates, you shouldn’t panic every time you one. There are lots of ways for things to be misconfigured that lead to these errors. Instead you should try to investigate by other means whether the site really is who they say it is. Report the error to the operators of the site. (Before you all report to us that agile.ws comes up with an error, we know. The certificate was revoked when we changed our business name. Just use our agilebits.com domain.)

The doomsayers may have been right

When the whole trust structure for SSL was devised, there were many people who worried that it gave too much power to certification authorities. In this instance we had one that suffered a security breach, but imagine if there were a corrupt one. With hundreds of trusted certification authorities, each with the power to issue certificates for any domain, the scope for abuse is substantial. I was one of those worriers.

It is easy to say that I don’t like the system, it is much much harder to present an alternative that works better and doesn’t burden users with the task of performing their own audits of certificates or authorities. There are a number of proposals out there, and discussion of them has certainly kicked off again over the past few days. My post here has already been long enough, so I will redirect readers to a post by Mike Caldwellwho proposes an idea I haven’t seen before (as well as linking to other proposals).

Browsers need to take revocations more seriously

I mentioned above that revocation lists are a tricky process. A browser isn’t going to fetch these lists every time it connects to a site. And so so certificate revocation doesn’t propagate to users immediately. From what we’ve seen over the past week, Safari on the Mac appears to be the laggard in updating from CRLs. I expect that every browser maker is looking closely at the handing of certificate revocation.

The impending death of DigiNotar shows the system working

DigiNotar’s days as a CA are numbered. Their customers will be extremely angry with them, and it is unclear how they can survive, although their parent company, Vasco, might. This will be a lesson not just to everyone else in the business, but to potential customers. Customers will want to know that a CA’s security is strong enough that they will not have to worry about them becoming untrusted. This lesson may seem to contradict the one above, but we need to look at all angles.

24 replies
  1. poof
    poof says:

    The information here seems to be a bit outdated. As of last friday (2-9-2011) every root certificate in control of DigiNotar has been flagged as untrusted because they all may have been compromised. This also includes the root certificate that is being used for PKIoverheid. Mind you, DigiNotar is 1 of 6 parties that control root certificates for PKIoverheid. The Dutch government is now switching over to certificates from the remaining 5 parties (it has already done so for digid.nl; this uses one from Getronics).

    Fox-IT did some research regarding DigiNotar and these root certificates. Main aim was to get some answer to what happened. They have release that report last friday. It seems that DigiNotar made numerous mistakes regarding security and that their root certificate for PKIoverheid may also have been compromised. The report in English can be found at the following link: http://tweakimg.net/files/upload/Operation+Black+Tulip+v1.0.pdf The Department of Justice is now starting a new investigation into DigiNotar. The story is not over yet. Things are a big mess in the Netherlands (at a technical level due to certificates needing to be replaced as well as political because politicians have ignored experts for years; security and IT was not something that was on politicians agendas until now).

    Over the weekend there has also arisen a bug in OS X regarding EV certificates and the Keychain trust settings. It seems that setting the root certificate to untrusted will be ignored when using an EV certificate. Websites with these kind of certificates will be marked as trusted by OS X. Therefore using Safari is probably something that is NOT advisable at the moment. This bug needs to be resolved first. Firefox seems to be a good alternative because that one does play nicely. However, the newer Firefox versions require the new 1Password plugin which lacks some features from the old plugin (main one being no support for HTTP auth login windows; lots of home routers will use this; you need to copy-paste manually).

    • Jeff
      Jeff says:

      You are absolutely correct that this is a rapidly developing story. A great deal has been learned in the time that I started to right it and eventually posted. (For example there are reports today that the Dutch government has persuaded Microsoft to delay their update to IE in The Netherlands.

      I’ve been trying to reproduce the bug with EV (Extended Validation) bug. I did include a link to an article that is tracking the Safari bug, but you are correct that I didn’t mention it in this article for a number of reasons. One is the post I published is about half the length of an earlier draft. The second is that I haven’t reproduced the buggy behavior to my satisfaction.

      There is and has been excellent coverage and discussion of this case among people who are familiar with CAs, CRLs, and Online Certificate Status Protocol. And that discussion is keeping up to date. That wasn’t my audience for this. Originally I wasn’t going to write anything because I thought it would be better covered elsewhere. But I noticed that there was a lack of explanation of what is happening for the ordinary user. (You will notice that I played fast and loose with terms like “issuing a certificate” and “signing a certificate”.

      One of the most interesting bits of information is what is on the Tor blog. They’ve examined all of the revoked certificates and found that intruder basically left a calling card. Tor was a specific target, as it appears that the government (or whoever) of Iran really does not like that anonymizing tool. (The FBI doesn’t like TOR either as I know from personal experience, but that is a very different story.)

      One thing that using 1Password allows (and I should have thought of this plug earlier) is switching browsers. I know that most users are fiercely loyal in their browser choice, but when I started using 1Password it was greatly liberating as I could get my passwords in (almost) all of the browsers that I used. (Prior to using 1Password, I used a combination of the password manager in Firefox and also used Password Gorilla.)

      You are not the only person to miss HTTP Authentication handling in 1Password. I deal with enough network gear that it matters to me, too. We are looking at and pushing for APIs for browser extensions that will let us do what we need to do. There are some promising hints of possibilities with some browsers, but nothing we can base any promises on.

      OK. I’m rambling, so I’ll stop now.



    • poof
      poof says:


      Thanks for the reply. The story has been developing quite rapidly since last Friday. A lot of the information is in Dutch over at various parts of rijksoverheid.nl (the Dutch government site for everything about the government) and website as tweakers.net (where that report in pdf came from) and Bits for Freedom (the Dutch EFF: http://www.bof.nl). The latter is very active on twitter with tweeting and retweeting all the information (mostly in Dutch though). The English sites seem to be a bit behind but that is quite obvious (lots of stuff happening on Dutch tv and Dutch sites but it all needs to come to them and be translated first).

      Btw, personally I don’t miss the HTTP Auth stuff but a lot of other people do as you can seen in the forums. The toolbar, contextmenu and HTTP Auth are 3 of the main features people miss from the old extension. I thought I’d just mention it for those who are unaware of this. I also know that you guys are trying to get this all back into the new extensions. Thanks for telling us that because I forgot to add that part. Whoops.

    • poof
      poof says:

      In addition to that I have a Google Translate link (prepare for crappy English) for the main article about the DigiNotar problems on rijksoverheid.nl, might provide some more info:

    • Jeff
      Jeff says:

      Thanks. The Fox-IT report that you linked to earlier was a very interesting, and somewhat disturbing read.

      Although I don’t read Dutch at all, one of our team, Stefan, is Dutch. So we’ve been hearing how this is playing out in news there.



  2. Asgeir S. Nilsen
    Asgeir S. Nilsen says:

    In my opinion using TLS server certificates to fix an untrustworthy DNS is a hack that backfired. Rather than patching this hack focus could be brought back to the original problem — an untrustworthy DNS.

    • Jeff
      Jeff says:

      I agree, Asgeir, but we have to work with the systems we have.

      But note that even if we have DNSSEC everywhere, there would still be a need for this kind of authentication. Someone manipulating router tables (which an ISP or telecom can easily do) would still be able to launch a MITM attack even if the DNS lookup were fully trustworthy.

      We (well, I) don’t know whether the attack in Iran involved DNS manipulation or routing manipulation.



  3. Ben F
    Ben F says:

    Nice post. I’m a bit surprised that this hasn’t been picked up on the krebsonsecurity blog; BK has reported on CA trust issues in the past.

    When I went into Keychain Access to make the change you describe, I discovered that Digitar was not configured as “System Default,” but as “Always Trust.” Is this normal, or a cause for concern? I’m running OS X 10.6.8 with all Apple software updates installed.

    • Jeff
      Jeff says:

      That’s not a problem, Ben. Change it either from “Always Trust” or “Use System Defaults” to “Never Trust”.

      I’ve done enough meddling with my system over the years to not be starting in the same place that others are. I should have tested on a cleaner installation.



    • poof
      poof says:

      Normally you can ask the software company (Apple, Mozilla, Google, etc.) to add your root certificate to their list. The software company will do some checks and when they don’t see any objection you’ll get added to the list. Every root certificate that has been added will be trusted by default. There’s a certain point to that: you add the root certificate so the system recognises it and won’t bug the user with an error message. If you were to set it to untrusted this will still cause the error message. The biggest problem is that in the first case the error will state that it doesn’t know the root certificate, in the second it will say that the root certificate isn’t trusted. That means that something funny is going on with that root certificate and you as a user should not trust them. That is in fact the wrong message to get across. You only distrust them when they proved you can not trust them like DigiNotar did. I don’t think it would matter if it were set to “system defaults” because that would come down to “always trust” anyway.

  4. Seth Bromberger
    Seth Bromberger says:


    I’m the (re)discoverer of the EV bug. As of last Friday, DigiNotar reissued the certs for the site I was using to confirm the bug. I did create a screencast of it here: http://www.youtube.com/watch?v=g2eGfFa-gNY

    Note that it had been reproduced by lots of people, including experts – and one (Ryan Sleevi) actually pinpointed the bug in the code that causes the system to ignore trust settings when an EV cert is in the mix.

    Hope this helps. Let me know if you need more details.

    • Jeff
      Jeff says:

      Thanks Seth!

      Thanks for your work on identifying and documenting this Safari bug. I now see why I failed to reproduce it (I didn’t have an EV certificate in the mix).

      So let me summarize for readers here. Because of a bug in Safari (or the OS X Keychain), the instructions that I gave above to “detrust” DigiNotar are not sufficient. If there is an EV (Extended Validation) certificate in the chain of trust, then even if you told the system to “Never trust” DigiNotar, Safari will trust it anyway.

      The recommended solution is to not merely to set “Never Trust” on the certificate, but to remove it from the system keychain entirely.

      Details about how to do that are here:


      One disadvantage of removing, versus “revoking” the certificate in your keychain is that a Software Update from Apple might restore it. I think that Apple is unlikely to do that, but they (as usual) remain silent on this whole thing.



  5. Seth Bromberger
    Seth Bromberger says:


    You’ve got it exactly right. You’re also right that Apple’s been silent so far – which is distressing for a number of reasons: not only have they not pushed out root certificate updates, but they haven’t addressed the bug that results in a “Never Trust” setting not being respected in the first place. Combined, these two omissions are causing me great concern about Apple’s commitment to security. It’s been almost a week since the original alert, and all major browsers have fixed the problem. Hell, even Microsoft(!) has fixed the problem. What’s taking Apple so long? Are they unaware of the significance of this issue?

    • Jeff
      Jeff says:

      It’s really frustrating that Apple haven’t commented on the bug, but it isn’t surprising. They never comment on security issues until they have an update. So we can’t read anything at all into the fact that they haven’t commented. It really is a worrying bug, and I would hate to think that someone at Apple had confused validity of a certificate with trustworthiness. So I’m going to take an optimistic approach and not think that.

      Apple’s lack of action on updating root certificates is more puzzling. Apple really is the odd one out. IE, Firefox, and Chrome have all been updated. (Though the IE changes, according to some reports, aren’t being rolled out in The Netherlands at the request of the Dutch government, or they are set to trust an intermediate certificate issued by DigiNotar and used by government servers.) I have no idea of what is going on inside Apple, so all I can do is speculate.

      I think that Lion and Safari 5.1 really boosted Apple’s credibility in their approach to security, as these introduce security architectures that Apple had fallen behind on. I would hate to see some of that regained credibility eroded by their handling of this. But, as I said, their policy of silence for such things is a long standing one, and we shouldn’t read too much into their silence.

      Another thing to note is that as every hour passes since the DigiNotar CA has been removed/revoked by users of any browsers, the less likely it is that there are undetected fraudulent DigiNotar certificates out there. So the practical security concern is minimal as long as enough people have already stopped trusting DigiNotar. Firefox, Chrome and IE users (assuming they get the updates) will rogue certificates quickly.

      My real worry is that if one CA (or two, considering the Comodo case a few months back) can be compromised this way, how many others have been without detection. For me, the big news in the whole thing was that something that I’ve always worried about as a “theoretical” possibility actually happened. We don’t know when the MITM attack with google.com was first launched, so we don’t know if it was working successfully since early July or whether it started soon before it was detected. I suspect that we never will know.



    • Seth Bromberger
      Seth Bromberger says:


      Given that the Comodo hacker has claimed credit for DigiNotar and also claims to have compromised several other CAs, continued vigilance is warranted. IMHO, Apple needs to step up to the plate here and start removing these certs via update as soon as they become aware of the issue. (they also need to fix their code but I’m willing to give them some time to do that – see next paragraph for why they can’t have it both ways, though).

      As for Lion boosting Apple’s security cred, I wouldn’t go that far. The recent LDAP auth fiasco – exclusive to Lion – shows me that they’re not performing adequate testing of new code. That it affects enterprise users – a group already traditionally marginalized by Apple – is a double blow.

    • Jeff
      Jeff says:

      Obviously I have nothing good to say about the LDAP fiasco. That is a truly terrible bug, and you are right. IT departments in enterprises will (correctly) hold that against Apple for a long time.

      The credibility I was referring to is the introduction of Address Space Layout Randomization. OS X has been a laggard in these sorts of architectural things. I think that for too long Apple was resting on its Unix laurels when compared to XP or older Windows systems. But as it was resting on these laurels, competitors leapt ahead in basic OS architecture with respect to security. I see Lion as Apple finally doing something about that.

      Apple has some breathing room because the cybercrime gangsters haven’t developed the tool kits and expertise for attacking OS X. Their skills and tools are heavily invested into attacking Windows. Had the LDAP issue happened on Windows, it would have been actively exploited as a part of other automated attacks within hours.

      So I fully recognize that Apple has some real catching up to do, I also think that architecture in Lion and the sandboxing requirements of things sold through the Mac App Store indicate that they really are moving forward in security issues despite the issues that have come up in this discussion (trusting untrustworthy EV certificates, failure to push an update removing DigiNotar, and the hard to fathom LDAP bug in Lion).

    • Seth Bromberger
      Seth Bromberger says:

      I think we’re in violent agreement on most things here, except that you’re willing to give Apple credit for implementing an OS feature (randomization) that’s been in Linux since — 2005? possibly earlier? To be fair, it doesn’t appear that it’s standard in FreeBSD (OS X’s closest relative).

      Anyway, interesting discussion; thanks! If I could make one more suggestion – some (most?) folks won’t read this far down; perhaps an update to the “Never Trust” instructions in the body of the blog post describing the problem with this approach is in order?



    • Jeff
      Jeff says:


      Yes. We are in agreement for the most part. One reason that I didn’t write about the LDAP issue is that I try to keep profanity out of what I say publicly when I am writing here. As you can image, it would be a struggle to discuss that issue honestly without using some choice language.

      ASLR (and other features of Security Enhanced Linux) was made available to Linux kernels as early as 2003, however it is turned off by default in the major distributions. So it really is Microsoft that gets the credit for first introducing this to the public with Vista.

      I’ll update the post telling people to actually remove the DigiNotar certificate due to the still unpatched bug in Safari/Keychain.

      I think that our disagreement is that I see signs that Apple is working to get better, and I consider that laudable. We both agree that it is a laggard in terms of security design.



  6. Jeff
    Jeff says:

    Apple has just released a security update (Security Update 2011-005) which removes all trust for DigiNotar signed certificates. I do not know (yet) if this fixes the more general problem of trust of EV certificates.

    So run Software Epdate from the latest versions of Lion or Snow Leopard to get the security updates.



Comments are closed.