Certificates, Provisioning Profiles, and Expiration Dates: The Perfect Storm

As you may have read, this weekend was a little hectic for us and some of our app developer friends1. On Saturday we got word that users of 1Password for Mac were seeing the app fail to launch correctly. It took a few hours, but we diagnosed the problem and released an update that corrected the issue. This issue will only have affected users that downloaded 1Password for Mac directly from our website, so if you downloaded it from the Mac App Store you had a much more calm weekend than we did.

But alas, that story has already been told. Now it’s time for the nitty gritty technical details about all the forces that aligned against us that had us staring up a giant wall of crashing water like George Clooney and Mark Wahlberg.

Prologue: Not All Certificates Are Created Equal

There’s a lot of information to unpack in this post, but before I get started, I’d like to address an assumption I’m seeing far too many people making: that what happened to us was simply an issue of an expired certificate and that all we needed to do was create a new one, just like you do for SSL certificates.

That’s simply not true.

Developer certificates are much different than SSL certificates and serve a very different purpose. Unlike a simple SSL certificate, our developer certificate is used to sign 1Password and needs to be valid during build time. The expiry time of a certificate or provisioning profile should have no impact on whether or not macOS will allow an app to launch or not.
An analogy may be helpful here: if you think of the developer certificate as a carton of eggs, and 1Password as a cake, then it is important not to use expired eggs to make the cake. The fact that the eggs may expire a few days after making the cake should have no effect on the cake itself. After all, the cake is already made and delivered.

Jumping out of the galley and back into our developer world, an expired certificate typically doesn’t affect us until the next time we need to do a release, which would have been this week with our next betas. Certificates control our ability to sign new apps. They don’t affect existing released apps.

For example, we have some users still using 1Password 3 for Mac (hey there, if that’s you, you should really consider upgrading to a 1Password membership as soon as possible!). The first release of 1Password 3 was in 2009, around 8 years ago. Assuming a user is happy with 1Password 3, how long should they expect to be able to continue using the software they paid for? The only acceptable answer to that question is: as long as they feel like it.

Obviously there’s plenty of reasons for why a user would want to upgrade to newer versions, but the fact of the matter is that a user shouldn’t be reliant on us to keep providing updated builds of an unmaintained app just to keep it running. Unlike an SSL certificate, this isn’t something we can simply fix from our end. Fixing the issue we ran into this weekend is a matter of creating a new build of the app and having users update to the new version.

Taking a Tour of the Engine Room

iCloud Sync

To properly understand what happened, let’s take a step back and look at the different parts of this.

In Mac OS X 10.7 Apple introduced Gatekeeper. Gatekeeper is really quite awesome as it gives users control over what software is allowed to run on their system. The default is to allow software from verified and trusted developers: those apps that have been uploaded to the Mac App Store, or those signed with Developer ID certificates made available to the developer by Apple.

Gatekeeper ensures that apps that have been tampered with will refuse to run, and also provides Apple with a way to revoke certain certificates if a developer has been found to be doing harm (i.e. distributing Developer ID signed malware). These simple steps stop a wide variety of attack vectors and we think the world of Apple for having implemented this.

The next layer is the Provisioning Profile. Provisioning Profiles provide information about what the app can do, as well as who can run it. There are certain services on the Mac that require that the app include a Provisioning Profile. In our case, we needed to start using a Provisioning Profile when we added support for unlocking 1Password using Touch ID.

To be clear, Touch ID itself doesn’t necessitate the profile, but in order to unlock your vault we need to store a secret and we choose to store it the OS X keychain. The specific configuration we’re using for that requires declaring that we want access to a specific keychain access group, which needs to be declared in a provisioning profile. The provisioning profile is included in the app bundle and cannot be updated independently of the app.

Next up… XPC. We use XPC to communicate between the 1Password main app and 1Password mini – the little 1Password that runs in your menu bar – and it’s really quite awesome. 1Password mini acts as the brains of the whole operation, and the larger app is mostly just responsible for displaying information. The reason we love XPC so much is because it’s an inter process communication tool that actually provides us the building blocks we need to perform mutual authentication. What this means is that 1Password mini will refuse to communicate with the main app unless it can prove that it’s signed by us. The inverse is true as well.

Storm Clouds Gather

clouds-gathering@2xAt around 3pm EST on February 18th we started getting reports of failures in 1Password for Mac. Folks were seeing an error appear that 1Password was unable to connect to 1Password mini.

Unable to start 1Password

This initial failure occurred due to the fact that the provisioning profile embedded in 1Password mini had an expiration date. Expiration dates seem to be required, and due to the fact that the expiration date elapsed, Gatekeeper decided that 1Password mini was no longer safe to run. We’ve filed a bug with Apple as we feel that this shouldn’t be the case (rdar://30631939 for those of you reading along inside the Mothership).

Only 1Password mini contains the Provisioning Profile as all Touch ID operations happen within that process. This meant that Gatekeeper was deciding that our main 1Password app could launch. Upon launching, 1Password performs its start up sequence which includes asking the system to launch 1Password mini if it’s not already running. When doing so, the system would log the following to the console:

com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): Binary is improperly signed.
com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): removing service since it exited with consistent failure reason When validating /Applications/1Password 6.app/Contents/Library/LoginItems/2BUA8C4S2C.com.agilebits.onepassword4-helper.app/Contents/MacOS/2BUA8C4S2C.com.agilebits.onepassword4-helper:
Code has restricted entitlements, but the validation of its code signature failed.
Unsatisfied Entitlements:
com.apple.xpc.launchd[1] (com.apple.ReportCrash[11041]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash

The 1Password main app detected the failure and provided an error panel telling the user that it couldn’t connect to mini.

Due to the expired Provisioning Profile, 1Password mini wouldn’t launch. And without mini running, 1Password itself was unable to startup successfully. Both mini and 1Password itself were signed with the same Developer ID certificate. Gatekeeper allowed 1Password to run, but due to the different rules for apps with provisioning profiles, it would not allow mini to run.

As far as we can tell, the only way to correct this problem is to provide a new build of the app with an updated provisioning profile with a new expiration date. Within a few hours we were able to publish a new version which did exactly this. As of 6.5.4, we had an app that users could download and run again.

The Eye Of The Storm

eye-of-the-storm@2xAfter this initial bout of terror, death defying feats, and mad scrambles we figured the technical portion of this exercise was finished and had begun transitioning into customer support mode; helping allay the fear, uncertainty, and doubt that this event had caused.

Little did we know at the time, we were only in the eye of the storm – the calm center before things would get rough again.

1Password for Mac includes an updater within the app so that users can easily upgrade to the latest versions as they become available. This updater validates downloads before performing the update to ensure that the updated app is in fact from AgileBits. One of the steps taken during validation is looking at the code signature of the downloaded app and ensuring that it satisfies the following security requirement:

anchor apple generic and identifier com.agilebits.onepassword4 and certificate leaf[subject.CN] = “Developer ID Application: Agilebits Inc.”

This check has worked really well for us. It’s simple and does the trick.

This check is also extremely specific about the common name2 it looks for. When we generated our updated provisioning profile we also needed to generate a new Developer ID certificate. We didn’t realize it at the time, but the common name of newly created certificates now include the team identifier in addition to the company name;  “Developer ID Application: AgileBits Inc. (2BUA8C4S2C)” vs. “Developer ID Application: AgileBits Inc.”. Close. Super close. But we weren’t looking for a “close” match.

The result of this new common name was that even though our app would now launch, the automatic updater would never run successfully because as far as it was concerned the update being provided wasn’t valid and therefore needed to be rejected. This is what users who could still run 6.5.3 and tried to update to 6.5.4 saw.

Once we discovered this problem we had no choice but to pull the 6.5.4 update and issue a 6.5.5 update that included a modified security requirement check. Sadly this didn’t address the fact that users running 6.5.3 and earlier are not able to automatically update to 6.5.5.

Moving Forward and Heading Home

heading-home@2xThis was painful for everyone. We lost sleep over the weekend, but worse than that… our users temporarily lost access to some of their most important information. This is unacceptable to us and we want to make sure this doesn’t happen again.

We’ve reached out to Apple for help and guidance on what we can do to avoid this happening again in the future. Our new provisioning profile doesn’t expire until 2022, but we’ll make sure that this is resolved far before then so that you need not worry about that happening.

If you’re a developer of a Developer ID signed app, we recommend that you check to see if your app includes a provisioning profile. Since that’s mostly handled automatically by Xcode, it’s likely that there are apps out there whose developers aren’t even aware of the inclusion of the provisioning profile. Check the expiration date, and ensure that you release an updated build with an updated provisioning profile well before the expiration date is hit so your users have time to update.

We’ve also filed an enhancement request with Apple asking that developers be notified via email of impending distribution certificate or provisioning profile expirations with explanations of repercussions. This was filed as rdar://30631968.

If you have questions about any of this, please don’t hesitate to ask us in the comments below.

Love,
The 1Password Mac Team
❤️

P.S. Happy 5th Birthday to Gatekeeper! ? We were one of the first apps to sign with Developer ID certificates, use XPC, and leverage the entitlements required for Touch ID. It’s always exciting being on the cutting edge of technology but we wouldn’t have it any other way. ?

Further Reading

This was the second post in a three part series. See the exciting prequel and sequel here:

Part 1 : 1Password for Mac 6.5.5: Manual update required

Part 3 : PSA for macOS Developers: Renew Your Certificates & Provisioning Profiles


  1. The exact same perfect storm appears to caused our friends at Smile to hit the same rough seas that we had. You can see Adam Engst’s story in TidBITS for details on how this affected PDFPen. 
  2. The Common Name is the subject.CN part of the security requirement. As our Chief Defender of the Dark Arts often says of Common Names: they are often very uncommon. The name is inherited from older identify management systems. I don’t need to say much more as Jeff loves explaining things, so let’s all sit back and watch what he says in his comment that I’m sure he’ll be adding soon. 
51 replies
« Older CommentsNewer Comments »
  1. rlobrecht
    rlobrecht says:

    Thank you Rick, Dave, and the 1Password team. This kind of transparency and explanation is one of the reasons I trust your software with my secrets. Keep it up.

    Reply
    • Dave Teare
      Dave Teare says:

      Thank you so much! ❤️

      To make a point about transparency, I tried to write this response in a transparent ink, but decided not to write the entire message that way ?

      Except for this little bit where I tell you how much I love you. I didn’t want my wife to see :)

      Cheers! ?

      ++dave;

  2. Brian Shim
    Brian Shim says:

    Much love, you guys! Stuff like this can happen to anyone, but you guys have established such a commitment to your users that this is just a small blip in an amazing track record. Keep up the great work, we hope to see you around for many, many years to come!

    Reply
    • Dave Teare
      Dave Teare says:

      Thank you so much for saying so, Brian! I really appreciate you taking the time to share these kind words with us. ❤️

      I’m expecting 1Password and our team to be around for many many many more years to come. After all, 1Password has awesome users like yourself, and for me and the team, we have Ray Kurzweil’s Singularity right around the corner. ?

      Take care,

      ++dave;

  3. Brian
    Brian says:

    Thanks for all the details. I wish more folks would this when things go wrong.

    Followup question: Today, Software Update is still telling me that I have to manually download and install the update (version 6.6.1) instead of installing it like normal Software Update updates.

    I don’t need the touchbar stuff, nor any of the fixes it mentions, and I’m gun-shy now about updating.

    What’s the reason why Software Update updates won’t work, and when will that part get back to normal?

    Reply
    • Dave Teare
      Dave Teare says:

      Hi Brian,

      You’re very welcome. I’m glad to hear you enjoyed the details as much as we enjoyed writing them ?

      As for the 6.6.1 update, you do indeed need to manually update to this new version. This is expected. You can read the The Eye Of The Storm section of this blog post for all the nitty details on why a manual update is required.

      Even if you don’t have a fancy new MacBook with a TouchBar, you’ll want to manually update so you can get future updates installed automatically. Part of the 6.6.1 release is an update to the updater so that future updates can be installed automatically. Whew, that’s a mouthful to say, isn’t it? I hope it’s easier to read than it is to say ?

      Take care and enjoy the rest of your weekend.

      ++dave;

  4. J Barton Elliott
    J Barton Elliott says:

    Puzzled, but I did not see a problem until Sat 2/25 am. I use 1Password continuously on my various devices. Something must have updated to make the bug appear. Also, all my updates came from the Apple App Store. Do you suppose there is a different way this manifests itself?

    Thanks for the great software. I could not live without it !

    Reply
    • Dave Teare
      Dave Teare says:

      Good morning! ☀️ And thank you so much for the kind words! They are music to our ears ?

      Having this issue appear is rather puzzling. I’m not overly surprised that the issue didn’t happen until now as macOS will cache things and doesn’t re-verify things while 1Password mini is running. So it’s possible to run for days or weeks (or years?) without ever seeing the problem.

      The part that is puzzling me however is you mention you’re running from the Mac App Store. This is odd because the way things work is Apple resigns 1Password when we upload it using their own certificate. So in theory Mac App Store users should not be affected at all.

      Can you please verify that you are indeed running the Mac App Store version? You can read this article for details on how to do that.

      Please let me know.

      ++dave;

  5. J Barton Elliott
    J Barton Elliott says:

    Dave,

    I am indeed NOW running the latest version from the web, which fixed the problem.
    1Password 6
    Version 6.6.1 (661010)
    AgileBits Store

    The IOS version is 6.5.2 App Store.

    Maybe I am wrong and have been updating the Mac this way all along? One gets numb to these things after a while. You explanation of cashing is probably exactly what happened. Mostly i am using the mini versions from Safari, going for days (weeks?) without opening the main app.

    Thanks for the clarification.

    -te

    Reply
  6. xz4gb8
    xz4gb8 says:

    “Curiouser and curiouser!” Cried Alice, as she read this blog entry.

    Guillaume Boudreau command let me to another use of an embedded.provisionprofile:

    $ find

    /Applications/SimpleFloatingClock.app/Contents/embedded.provisionprofile

    <date>2013-11-02T17:37:20Z</date>

    I’ve used Greg Weston’s SimpleFloatingClock.app since migrating to Leopard and continuing to the most recent Sierra Beta. Expired profiles apparently do not matter in some cases.

    Obligatory kiss-up: One of the reasons I originally decided on 1Password about three years ago was the willingness of the AgileBits team to respond to my questions. Since then, I have had many conversations with them and have been well satisfied with both the team and the product, often learning something new and always appreciating the product and its continuing improvement.

    Reply
    • Dave Teare
      Dave Teare says:

      Thanks for the obligatory kiss-up! I appreciate it ?

      As for the expired provisioning profile in SimpleFloatingClock.app, my guess is Greg’s not using it for anything critical or perhaps he didn’t wire it up to be used. I’m unsure about that though so don’t quote me on it. You could always as Greg to see if he can solve this mystery for us both ?

      Take care,

      ++dave;

  7. xz4gb8
    xz4gb8 says:

    My previous post may have been the victim of both spell check and automatic editing by the blog interface:

    Should have been “command led me”,

    The find command should have show a placeholder for the deleted command text. The “–” from the command output was omitted.

    There is some editing magic that I can not find. Help would be appreciated.

    Reply
    • Dave Teare
      Dave Teare says:

      If it’s any consolation, I didn’t even notice the typo ?

      As for formatting on this blog, you can use HTML if you’d like. But the plain text formatting came out readable enough for me.

      ++dave;

    • Jeffrey Goldberg
      Jeffrey Goldberg says:

      Hi David,

      First of all let me thank you for asking. People should be suspicious when they get a message to update some security system in an unusual manner. Asking people to manually update a security product is not something that should be taken lightly. We don’t want people to grow comfortable with clicking on links in some email to fetch a new copy of 1Password.

      The fact that you (and others) are asking questions that are all various forms of “how do I know that these unusual update instructions aren’t trying to trick me into installing something I shouldn’t be installing” is reassuring to me. So thank you for asking!

      Now onto the substance of your question.

      We do publishes hashes, but we don’t really publicize that we publish them to avoid confusing clutter on download pages. These hashes are mostly used as a first checksum by our updater (more on that below) and aren’t really designed for human consumption.

      But for humans who do wish to consume them you will find that for any of these downloads, there is a corresponding .sha256 file.

      So for example, https://cache.agilebits.com/dist/1P/mac4/1Password-6.6.1.pkg will get you 1Password for Mac 6.6.1 and https://cache.agilebits.com/dist/1P/mac4/1Password-6.6.1.pkg.sha256 will get you its SHA256 hash.

      Manually check signatures

      Of course those hashes are published in the same place the downloads come from so they may not be serving all of your security requirements. Their primary use is by our updater as a first checksum of a download. But you can check that an installed version of 1Password is signed by AgileBits. Note that these instructions do not apply to installations through the Mac App Store.

      You can verify that it is signed by us by running

      codesign -v -d /Applications/1Password\ 6.app

      in a Terminal window and looking at the Identifier.

      If you want to check the identifier and also verify that the signature is valid (thus duplicating what the operating system does) you can run this command

      codesign -vvv R="identifier com.agilebits.onepassword4 and anchor trusted" /Applications/1Password\ 6.app

      Why no PGP or DSS?

      In anticipation of a question that people sometimes ask, I would like to say a few words about why we are not also publishing PGP or DSS signature files for each of these downloads. There are a number of reasons:

      • The manual codesigning check above has all of the security properties of a PGP or DSS signature check.
      • Anyone who has the knowledge and motivation to check a PGP or DSS signature would find it no more difficult to run the codesing checks described above.
      • The timestamp on a codesigning signature is far more trustworthy than the timestamp on a PGP signature
      • Managing two codesiging secret keys (the PGP secret key and the Apple codesigning secret key) offers two avenues of attack for creating forgeries than just having one, as it is very unlikely that people will manually check both. So a successful compromise of either of those two keys would be a problem.

      So a PGP signature would add pretty much nothing to actual security as it is possible to check the signature validity and identify of signer through what is already available on the operating system. But producing a PGP signature would require that our build process have to handle yet another secret.

      PGP signatures are an important thing for software that is installed on systems that don’t have full code signing checks. It is also useful for software that is distributed over a large number of channels. But 1Password is not distributed that way or for such systems, so we can make use of technology that addresses the same problem better while putting less burden on the end user.

      Our own updater

      Our own updater uses TLS as a first layer and will reject updates on a TLS security problem. That use of TLS is for all three of the download itself, the hash summary, and the release notes update message. Indeed, the latter is so that when the updater message tells you that you may need to manually update and gives you a link, you can be sure that that message comes from us.

      After fetching the update, it checks the SHA256 hash as described above.

      Finally it unzips it (updates come through .zip files while first installations are now through .pkg files) and then runs a code signing check similar to what is described above. Indeed, it was our “Common Name” that changed when we got a new certificate which is why the manual update was necessary. (We have since fixed this to just compare the Team Identifier; so that future changes in the format of the CN should produce this problem, while still providing a very strict check that it really is our update that we have fetched.)

      All of these checks that we do are in addition to the checks that the operating system performs.

  8. Janusz Buda
    Janusz Buda says:

    It’s a pity I’m retiring in a few weeks. The way AgileBits has dealt with the ‘perfect storm’ would have provided an excellent case study for my graduate school course in how to deal (and how not to deal) with serious corporate problems. Over the years I’ve collected dozens of cases of bad, sometimes disastrous, responses to catastrophic failures. The number of good responses is very small. Yours meets all the criteria for how to do it right, and then some. Not just the handling of the ‘perfect storm’ incident, but prompt, polite, helpful and, most important, continuing support of users in need, and an extra-mile presence on user forums that some developers never visit.

    Drat! Writing the above paragraph has triggered yet another earworm. Now I’ve got Dr. Feelgood’s “She Does It Right” racing around my head.

    All the best to a wonderful company that does it right.

    Reply
    • Jeffrey Goldberg
      Jeffrey Goldberg says:

      Thank you so much Janusz!

      You know that retirement doesn’t mean you need to stop writing.

      (And I’m sure you know that once someone mentions an earworm, others are infected. This is a nice was to have stuck in my head.)

      Cheers,
      -j

« Older CommentsNewer Comments »

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *