Blog Header@2x

Certificates, Provisioning Profiles, and Expiration Dates: The Perfect Storm

As you may have read, this weekend was a little hectic for us and some of our app developer friends1. On Saturday we got word that users of 1Password for Mac were seeing the app fail to launch correctly. It took a few hours, but we diagnosed the problem and released an update that corrected the issue. This issue will only have affected users that downloaded 1Password for Mac directly from our website, so if you downloaded it from the Mac App Store you had a much more calm weekend than we did.

But alas, that story has already been told. Now it’s time for the nitty gritty technical details about all the forces that aligned against us that had us staring up a giant wall of crashing water like George Clooney and Mark Wahlberg.

Prologue: Not All Certificates Are Created Equal

There’s a lot of information to unpack in this post, but before I get started, I’d like to address an assumption I’m seeing far too many people making: that what happened to us was simply an issue of an expired certificate and that all we needed to do was create a new one, just like you do for SSL certificates.

That’s simply not true.

Developer certificates are much different than SSL certificates and serve a very different purpose. Unlike a simple SSL certificate, our developer certificate is used to sign 1Password and needs to be valid during build time. The expiry time of a certificate or provisioning profile should have no impact on whether or not macOS will allow an app to launch or not.
An analogy may be helpful here: if you think of the developer certificate as a carton of eggs, and 1Password as a cake, then it is important not to use expired eggs to make the cake. The fact that the eggs may expire a few days after making the cake should have no effect on the cake itself. After all, the cake is already made and delivered.

Jumping out of the galley and back into our developer world, an expired certificate typically doesn’t affect us until the next time we need to do a release, which would have been this week with our next betas. Certificates control our ability to sign new apps. They don’t affect existing released apps.

For example, we have some users still using 1Password 3 for Mac (hey there, if that’s you, you should really consider upgrading to a 1Password membership as soon as possible!). The first release of 1Password 3 was in 2009, around 8 years ago. Assuming a user is happy with 1Password 3, how long should they expect to be able to continue using the software they paid for? The only acceptable answer to that question is: as long as they feel like it.

Obviously there’s plenty of reasons for why a user would want to upgrade to newer versions, but the fact of the matter is that a user shouldn’t be reliant on us to keep providing updated builds of an unmaintained app just to keep it running. Unlike an SSL certificate, this isn’t something we can simply fix from our end. Fixing the issue we ran into this weekend is a matter of creating a new build of the app and having users update to the new version.

Taking a Tour of the Engine Room

iCloud Sync

To properly understand what happened, let’s take a step back and look at the different parts of this.

In Mac OS X 10.7 Apple introduced Gatekeeper. Gatekeeper is really quite awesome as it gives users control over what software is allowed to run on their system. The default is to allow software from verified and trusted developers: those apps that have been uploaded to the Mac App Store, or those signed with Developer ID certificates made available to the developer by Apple.

Gatekeeper ensures that apps that have been tampered with will refuse to run, and also provides Apple with a way to revoke certain certificates if a developer has been found to be doing harm (i.e. distributing Developer ID signed malware). These simple steps stop a wide variety of attack vectors and we think the world of Apple for having implemented this.

The next layer is the Provisioning Profile. Provisioning Profiles provide information about what the app can do, as well as who can run it. There are certain services on the Mac that require that the app include a Provisioning Profile. In our case, we needed to start using a Provisioning Profile when we added support for unlocking 1Password using Touch ID.

To be clear, Touch ID itself doesn’t necessitate the profile, but in order to unlock your vault we need to store a secret and we choose to store it the OS X keychain. The specific configuration we’re using for that requires declaring that we want access to a specific keychain access group, which needs to be declared in a provisioning profile. The provisioning profile is included in the app bundle and cannot be updated independently of the app.

Next up… XPC. We use XPC to communicate between the 1Password main app and 1Password mini – the little 1Password that runs in your menu bar – and it’s really quite awesome. 1Password mini acts as the brains of the whole operation, and the larger app is mostly just responsible for displaying information. The reason we love XPC so much is because it’s an inter process communication tool that actually provides us the building blocks we need to perform mutual authentication. What this means is that 1Password mini will refuse to communicate with the main app unless it can prove that it’s signed by us. The inverse is true as well.

Storm Clouds Gather

clouds-gathering@2xAt around 3pm EST on February 18th we started getting reports of failures in 1Password for Mac. Folks were seeing an error appear that 1Password was unable to connect to 1Password mini.

Unable to start 1Password

This initial failure occurred due to the fact that the provisioning profile embedded in 1Password mini had an expiration date. Expiration dates seem to be required, and due to the fact that the expiration date elapsed, Gatekeeper decided that 1Password mini was no longer safe to run. We’ve filed a bug with Apple as we feel that this shouldn’t be the case (rdar://30631939 for those of you reading along inside the Mothership).

Only 1Password mini contains the Provisioning Profile as all Touch ID operations happen within that process. This meant that Gatekeeper was deciding that our main 1Password app could launch. Upon launching, 1Password performs its start up sequence which includes asking the system to launch 1Password mini if it’s not already running. When doing so, the system would log the following to the console:

com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): Binary is improperly signed.
com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): removing service since it exited with consistent failure reason When validating /Applications/1Password 6.app/Contents/Library/LoginItems/2BUA8C4S2C.com.agilebits.onepassword4-helper.app/Contents/MacOS/2BUA8C4S2C.com.agilebits.onepassword4-helper:
Code has restricted entitlements, but the validation of its code signature failed.
Unsatisfied Entitlements:
com.apple.xpc.launchd[1] (com.apple.ReportCrash[11041]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash

The 1Password main app detected the failure and provided an error panel telling the user that it couldn’t connect to mini.

Due to the expired Provisioning Profile, 1Password mini wouldn’t launch. And without mini running, 1Password itself was unable to startup successfully. Both mini and 1Password itself were signed with the same Developer ID certificate. Gatekeeper allowed 1Password to run, but due to the different rules for apps with provisioning profiles, it would not allow mini to run.

As far as we can tell, the only way to correct this problem is to provide a new build of the app with an updated provisioning profile with a new expiration date. Within a few hours we were able to publish a new version which did exactly this. As of 6.5.4, we had an app that users could download and run again.

The Eye Of The Storm

eye-of-the-storm@2xAfter this initial bout of terror, death defying feats, and mad scrambles we figured the technical portion of this exercise was finished and had begun transitioning into customer support mode; helping allay the fear, uncertainty, and doubt that this event had caused.

Little did we know at the time, we were only in the eye of the storm – the calm center before things would get rough again.

1Password for Mac includes an updater within the app so that users can easily upgrade to the latest versions as they become available. This updater validates downloads before performing the update to ensure that the updated app is in fact from AgileBits. One of the steps taken during validation is looking at the code signature of the downloaded app and ensuring that it satisfies the following security requirement:

anchor apple generic and identifier com.agilebits.onepassword4 and certificate leaf[subject.CN] = “Developer ID Application: Agilebits Inc.”

This check has worked really well for us. It’s simple and does the trick.

This check is also extremely specific about the common name2 it looks for. When we generated our updated provisioning profile we also needed to generate a new Developer ID certificate. We didn’t realize it at the time, but the common name of newly created certificates now include the team identifier in addition to the company name;  “Developer ID Application: AgileBits Inc. (2BUA8C4S2C)” vs. “Developer ID Application: AgileBits Inc.”. Close. Super close. But we weren’t looking for a “close” match.

The result of this new common name was that even though our app would now launch, the automatic updater would never run successfully because as far as it was concerned the update being provided wasn’t valid and therefore needed to be rejected. This is what users who could still run 6.5.3 and tried to update to 6.5.4 saw.

Once we discovered this problem we had no choice but to pull the 6.5.4 update and issue a 6.5.5 update that included a modified security requirement check. Sadly this didn’t address the fact that users running 6.5.3 and earlier are not able to automatically update to 6.5.5.

Moving Forward and Heading Home

heading-home@2xThis was painful for everyone. We lost sleep over the weekend, but worse than that… our users temporarily lost access to some of their most important information. This is unacceptable to us and we want to make sure this doesn’t happen again.

We’ve reached out to Apple for help and guidance on what we can do to avoid this happening again in the future. Our new provisioning profile doesn’t expire until 2022, but we’ll make sure that this is resolved far before then so that you need not worry about that happening.

If you’re a developer of a Developer ID signed app, we recommend that you check to see if your app includes a provisioning profile. Since that’s mostly handled automatically by Xcode, it’s likely that there are apps out there whose developers aren’t even aware of the inclusion of the provisioning profile. Check the expiration date, and ensure that you release an updated build with an updated provisioning profile well before the expiration date is hit so your users have time to update.

We’ve also filed an enhancement request with Apple asking that developers be notified via email of impending distribution certificate or provisioning profile expirations with explanations of repercussions. This was filed as rdar://30631968.

If you have questions about any of this, please don’t hesitate to ask us in the comments below.

Love,
The 1Password Mac Team
❤️

P.S. Happy 5th Birthday to Gatekeeper! 🎂 We were one of the first apps to sign with Developer ID certificates, use XPC, and leverage the entitlements required for Touch ID. It’s always exciting being on the cutting edge of technology but we wouldn’t have it any other way. 🙂

Further Reading

This was the second post in a three part series. See the exciting prequel and sequel here:

Part 1 : 1Password for Mac 6.5.5: Manual update required

Part 3 : PSA for macOS Developers: Renew Your Certificates & Provisioning Profiles


  1. The exact same perfect storm appears to caused our friends at Smile to hit the same rough seas that we had. You can see Adam Engst’s story in TidBITS for details on how this affected PDFPen. 
  2. The Common Name is the subject.CN part of the security requirement. As our Chief Defender of the Dark Arts often says of Common Names: they are often very uncommon. The name is inherited from older identify management systems. I don’t need to say much more as Jeff loves explaining things, so let’s all sit back and watch what he says in his comment that I’m sure he’ll be adding soon. 
49 replies
Newer Comments »
  1. Rick Fillion
    Rick Fillion says:

    In case you’re wondering how to tell when a provisioning profile will expire you can run security cms -D -i on the Terminal to have it output information about a profile. For example, here’s how to check 1Password’s:

    security cms -D -i /Applications/1Password\ 6.app/Contents/Library/LoginItems/2BUA8C4S2C.com.agilebits.onepassword4-helper.app/Contents/embedded.provisionprofile

    It will output the xml plist which will contain:

    <key>ExpirationDate</key>
    <date>2022-02-17T23:59:55Z</date>

    In our case the provisioning profile is pretty far down into our bundle since it’s inside of 1Password mini, but for most apps it would be something like

    security cms -D -i /Applications/Example.app/Contents/embedded.provisionprofile

    Reply
  2. Jeffrey Goldberg
    Jeffrey Goldberg says:

    I am going to defy Dave and talk instead about how different sorts of certificates server different purposes and so need to be treated differently.

    Part 1: What is a certificate?

    Certificates are for saying who a signature belongs to. You (or your computer) can look at 1Password.app and look at the signature that comes with it. To make the signature, we need to use a signing secret that only we have. The signature says which signing secret it was made with (but it doesn’t say what the secret is). So when your computer sees a signature on a file, it can check to see whether the signature matches the file and it knows which signing secret was used, but so far it doesn’t know who that signing secret belongs to (and again it does not know what the signing secret is).

    A certificate is a bunch of information about the signing secret, and particularly about who it belongs to, what it should be used for (e.g., is it just for signing, or can it be used for encryption as well), when it was created, and when it expires. From what I have said so far a certificate could be a pack of lies. Anyone can create any certificate that they want. So the first thing that the creator of a signing secret will do is sign their own certificate with the signing secret it describes. Certificates need to be self-signed so that we can even know that the certificate actually belongs to the signing key.

    But a self-signed certificate isn’t enough. It proves that whoever knows the signing secret created the certificate, but it does not tell you that any of the other information in the certificate is true. I could create a signing secret along with a certificate claiming that it belongs to Cleopatra Cleopatra VII Philopator, Ptolemaic Queen of Egypt. And I could use the signing secret to sign that certificate.

    As it turns out, I am not Cleopatra. So while I may have proven that the signing secret I’m using is tied to that particular certificate, I have not proved that any of the other information in the certificate is actually true. So again, the certificate really isn’t of much use.

    Now if I were indeed Cleopatra, I would go to someone who you trust and prove to them through my knowledge of Greek, Egyptian, and how not to be assassinated by my relatives that I am Cleopatra. Once I have proven to them who I am, I would ask them to sign my certificate with their own signing secret. Such an entity, that is (supposedly) trusted and signs other certificates this way is called a Certifying Authority.

    As you can imagine, there are loads of problem with knowing which Certifying Authorities to trust, but I don’t want to get into that. If you want to read more about that, take a look at an old blog post here, Who do you trust to tell you who to trust?

    Now we, AgileBits, have loads of different certificates. We have a certificate for the domain name 1password.com and puts the “S” in “HTTPS”. We have a codesigning certificate signed by Apple, and we have a provisioning profile certificate, also signed by Apple. Most of use individually have S/MIME certificates used for signing email and receiving encrypted email.

    All of these certificates have expiration dates, but those expiration dates mean different things based on what the certificate is used for. And that is what I hope to get into in Part II, but I need to have dinner first and walk the dogs.

    Reply
    • Dave Teare
      Dave Teare says:

      Well this certainly isn’t where I expected you to take things, but I must say, it’s a much more fascinating discussion than very often uncommon Common Names I was expecting. 🙂

      This was certainly 🍿 worthy, and I can’t wait for Part II. 😀

      ++dave;

    • Jeffrey Goldberg
      Jeffrey Goldberg says:

      So now onto Part II, which is just an expansion of Dave’s essential point that not all certificates are created equal.

      The certificates in question are all about signing secrets. And the expiry date roughly means, “don’t trust anything signed with that certificate after its expiry time”. Now that sentence is ambiguous. Does it mean after the expiry date you shouldn’t trust any of the signatures or does it mean that you shouldn’t trust any of the signatures that are created after the expiry date? The answer is that it depends on the certificate. The certificate itself with specify some fine details about how it should be used.

      But let’s just continue with the cases where the certificate says that signatures last forever even if the certificate expires (and all certificates expire). You can continue to trust old valid signatures as long as you want, but you shouldn’t trust signatures that were created after expiry date.

      That meaning works out well in general, even for TLS website certificates. (TLS is the successor of SSL.) When your web browser talks to a secure web server it will actually create a message that it sends to the server and says, “sign this random gibberish.” The server signs it, proving that it has access to the signing secret. So the server creates a new signature each time you go it in your browser. This means that once the server’s signing certificate expires your browser will reject any of those signatures. This is what we want to happen. And this can happen because there is an active dialogue between the browser and the server at the time that the browser is checking the server’s certificate.

      Now for contrast, let’s look at an S/MIME certificate. These are used for signing and encrypting email. When Morgan le Chien (one of my dogs) sends an email to Chena the Psycho Dog (the other dog) and she has her mailer set up to cryptographically sign the message, the signature will also sign a statement about the time it was created.

      So Morgan sends email to Chena saying, “I’ve got your doll, and I am running around with it in the back yard. Please come and chase me. It will be fun.” Morgan sends this when her S/MIME certificate is valid and creates a valid signature for the message. When Chena reads the message, she will see that the message is indeed signed by Morgan. It is not, as she feared, a trick by the coyote to lure Chena into the back yard on her own.

      Now suppose a year later after Morgan’s certificate has expired (because email addresses change frequently it is hard to get a very long lived S/MIME certificate), Chena wants to make a record of all of the toys that Morgan stole. So she goes through her old email. Morgan’s email should still be marked as valid as the signature was made when the certificate was valid even though Chena is reading it after the certificate has expired. This is generally how things work with S/MIME certificates. The annoying thing with S/MIME is that you might have to keep a bunch of expired decryption secrets around to decrypt older mail to you, but that is a different story.

      The astute reader may not that there is nothing stopping someone from signing with an expired S/MIME certificate and lying about the time the signature was made. As the S/MIME standard’s doc says

      The signing-time attribute is used to convey the time that a message was signed. The time of signing will most likely be created by a message originator and therefore is only as trustworthy as the originator.

      This may be good enough for email, but it isn’t good enough for code signing. We don’t want someone who gets hold of an expired signing secret to be able to create backdated signatures for software. So the rules for code signing certificates is that the signatures that they create must be timestamped by a trusted time stamper. The Certificate Authorities (CAs) run specific timestamping servers that will sign the signature with its knowledge of the time. So unlike S/MIME timestamps, the timestamps on code signatures are as trustworthy as the CA’s time stamping service.

      This whole timestamping thing is one of the things that makes codesigning certificates (and codesigning) more complicated most other timestamping certificates.

      Now it wasn’t the expiration of our codesigning certificate that triggered the problem; it was the expiration of our provisioning profile that triggered last Saturday’s mishap. So the question is how should and how do all of these things work with provisioning profiles. For that you will have to wait for further investigation. We still don’t know exactly why an expired provisioning profile had the effect that it had.

  3. Chris Ridd
    Chris Ridd says:

    Parsing parts of the X.509 subject DN seems awfully fragile and error prone, as you’ve just unluckily discovered. Doing this for HTTPS verification has been deprecated by the IETF for a loooong time.

    For HTTPS the solution has been to use subject alternate name certificate extensions, which work well and are more flexible. Multiple names, IP addresses, email addresses are all allowed, and they are extensible so you could put your shoe size in there too if you were inclined. (Yay X.509.)

    Is there any provision (sorry) for setting your own custom SAN in Apple developer certs?

    Reply
    • Dave Teare
      Dave Teare says:

      Hi Chris, 👋

      You’re right, there are some better ways to verify the downloaded zip we download is indeed 1Password. Rick actually had a paragraph in his original draft talking about this that was left on the editing room floor as the post was already quite lengthy.

      Since you brought it up, I went and found what he had written:

      We’re going to make our updater better and change how we perform validation of the download. Over the years we’ve learned a lot about validating code signatures and we can do better than the simple check demonstrated above. If the check worked more like how we validate our XPC connections we wouldn’t have run into the updater problem.

      I don’t have much more to add than that as I’m not exactly sure what Rick was alluding to to be honest. Maybe if Jeff or Rick find some time they can elaborate, but given we’re still in firefighting mode here, we both may need to wait awhile 🙂

      Take care,

      ++dave;

  4. bvankuik
    bvankuik says:

    Thanks for this detailed explanation. I applaud the fact that you show what went wrong so we can all learn from it. Raises my already considerable faith in the company.

    Reply
    • Dave Teare
      Dave Teare says:

      Thanks! I’m glad you enjoyed this post. And I’m happy to hear your applause! 👏 Please don’t stop. 🙂

      If we ever find some time to sleep, we might have a third article to expand on a few more things we’ve learned. But sleep must come first! 😴

      ++dave;

  5. Kamila
    Kamila says:

    Thank you for this article! This is an wonderful example on how to treat your customers. I love the human explanations and honesty! As a PR expert I say one word: Congratulations!

    Reply
    • Dave Teare
      Dave Teare says:

      Thank you, Kamila! I’m glad you enjoyed this article.

      It was a fun one to be involved with and I really liked how it turned out as well. Albeit I think we could find better things to spend our time on to get PR. 😉

      Take care,

      ++dave;

    • Shiner
      Shiner says:

      Hey Pat,

      Thanks so much, it’s great to hear that you enjoyed reading the details.

      I have to say that I also learned a lot from Rick and his post. While provisioning profiles and developer certificates are something that we deal with quite often, some of what they do and how they function can be complex and mysterious. Having someone talk through them in detail definitely gave me a greater understanding of their specific roles and how they all fit together.

  6. Guillaume Boudreau
    Guillaume Boudreau says:

    Thanks for the interesting post.

    I decided to check the expiration date of all the apps I have installed that use a provisioning profile.
    Looks like many of them will expire end of July / beginning of August 2017. I would guess that would coincides with the expiration date of the profiles created when Apple started required profiles..?

    $ find /Applications -name '*.provisionprofile' -exec echo "{}" \; -exec security cms -D -i "{}" 2>/dev/null \; -exec echo \; | egrep -A 1 "\.provisionprofile$|ExpirationDate" | egrep -v 'xml version|<key>'

    /Applications/1Password 6.app/Contents/Library/LoginItems/2BUA8C4S2C.com.agilebits.onepassword4-helper.app/Contents/embedded.provisionprofile
    --
    <date>2022-02-17T23:59:55Z</date>
    --
    /Applications/Boxcryptor.app/Contents/embedded.provisionprofile
    --
    <date>2017-07-24T09:05:30Z</date>
    --
    /Applications/Numi.app/Contents/embedded.provisionprofile
    --
    <date>2017-07-31T08:55:00Z</date>
    --
    /Applications/Parallels Desktop.app/Contents/embedded.provisionprofile
    --
    <date>2018-10-22T12:11:00Z</date>
    --
    /Applications/TunnelBear.app/Contents/embedded.provisionprofile
    --
    <date>2017-08-03T21:51:00Z</date>

    Reply
    • Dave Teare
      Dave Teare says:

      Ah, interesting! Thanks for sharing, Guillaume.

      I was trying to find another app that had an embedded provisioning profile so I Rick could use something other than “Example” in his post. I was in a bit of a rush and didn’t find one so I’m glad you found some! 🙂

      I’m not sure why the expiry dates coincide. I guess it’s for the same reason Smile’s expired on the same day we did: I’m guessing it’s around the time Apple sent out an email asking developers to sign up.

      But it’s just a guess 🙂

      ++dave;

    • Jeffrey Goldberg
      Jeffrey Goldberg says:

      I had been wondering about the coincidence of our expiry being the same day as Smile Software’s PDFPen Pro. I suspect that as long time Apple developers, these may have been signed pretty much in a batch.

      And running your wonderful pipeline there, I see that KeybaseInstaller.app also has a July 2017 expiry for their provisioning profile. <date>2017-07-12T22:54:55Z</date> Thanks for looking into this and the data.

    • Martin Machacek
      Martin Machacek says:

      I’ve used Guillaume’s pipeline to verify apps on my Mac. Interestingly enough, the provisioning profile embedded with OmniGraffle has expired more than a month ago, but the application still launches and works as normal (AFAICT).

      $ find /Applications -name ‘*.provisionprofile’ -exec echo “{}” \; -exec security cms -D -i “{}” 2>/dev/null \; -exec echo \; | egrep -A 1 “.provisionprofile$|ExpirationDate” | egrep -v ‘xml version|’

      /Applications/1Password 6.app/Contents/Library/LoginItems/2BUA8C4S2C.com.agilebits.onepassword4-helper.app/Contents/embedded.provisionprofile

      2017-02-18T20:31:30Z

      /Applications/OmniGraffle.app/Contents/embedded.provisionprofile

      <date>2017-01-08T00:03:35Z</date>

    • Jeffrey Goldberg
      Jeffrey Goldberg says:

      That is very interesting, Martin.

      I could speculate that Omnigraffle’s provisioning profile is not provisioning something that OmniGraffle depends on. Or, perhaps, the certificate within the their profile hasn’t expired.

      I could speculate, or I could take a look. Hold on … No luck, I have the Mac App Store version of OmniGraffle, and the Mac App Store versions aren’t affected.

  7. Oliver Dunk
    Oliver Dunk says:

    Dave asked me to write up my experience of the weekend, and so here I am doing just that. I ‘fell down the rabbit hole’ at around 9PM GMT on Saturday, when I noticed a sudden increase in the number of tickets heading to our urgent box. I of course investigated, and after a tip off from a helpful customer (you know who you are!) it became clear that the issue related to the provisioning profile mentioned above. Another team member noticed the rise in tickets but worryingly I was the first to realise how serious it might be. I knew that it could affect the majority of our Mac customers and that something needed to be done.

    On one of my own projects, I’d have hopped into the code, made a few changes here and there and then released a new version. I’m not a Mac developer though, and it turns out the 1Password Mac developers fancy a two day break once a week. Crazy! I quickly posted the problem internally and am happy to say that the team scrambled together in a way that makes me truly admire their commitment to the product.

    It was a busy night, and I helped out where I could spending the rest of my time keeping track of what was happening so that I could feed things back to people as they arrived. I didn’t get to sleep until late, but compared to the rest of the team, had quite a long nap while they worked to make the changes they needed to.

    That’s the story over, but I’d like to quickly finish things off by mentioning the wonderful 1Password community. Not only did you help us discover the issue as quickly as possible, but your kindness in responding with sympathy for the situation was greatly appreciated.

    Reply
  8. Bruce Dillahunty
    Bruce Dillahunty says:

    This points (to me) to a weakness in the app design… not that you had the cert and stuff expire, but the reliance on Mini to get into the main app.

    I’m sure I don’t understand everything, but so far we’ve had one instance recently in the Beta stream where an issue caused problems between Mini and the main app, and now this.

    (The beta issue was here: https://discussions.agilebits.com/discussion/75129/6-6-beta-6-wont-launch-failed-to-connect-to-mini)

    Anyhow, wonder if there is some way to let the main app still open even if mini won’t. That would let people at least get to their data. I know there are pro’s and con’s, but I would have thought I could at least copy/paste my passwords around manually to keep going (or even export somehow if desperate) while you guys slave away fixing things.

    Reply
    • Dave Teare
      Dave Teare says:

      Hi Bruce,

      This is a great question. There is no doubt that there are some pros and cons to our current approach with 1Password mini, and we very well might change things in the future, but having a single app here would not have helped at all in this instance.

      The only reason 1Password itself was able to launch was because the embedded provisioning profile only existed in 1Password mini. If we had a single app, the profile would have been within the main 1Password bundle and macOS wouldn’t have launched it at all. In other words, it would have been less resilient in this scenario.

      At least with the current approach users were able to see an error message and could then Google for a solution.

      And unless I’m missing something, the forum discussion that you linked to was caused by the exact same issue that was discussed in this post. Where it would help immensely is when there are multiple copies of 1Password installed. At the moment this causes 1Password to be unable to open, but again, we are able to detect this error and walk users through how to solve the issue (delete the other copy).

      So I agree things aren’t perfect, but I think the current approach with mini provides a pretty good balance when all things are considered.

      I hope that helps shed some more light on things,

      ++dave;

  9. jaimebabo
    jaimebabo says:

    What a great blog report! Now, I am energized to run some On apps that are as crucial to my OS10.10.5 as 1PW is. That includes iParallels from Coriolus in U k. I assume they must follow the certificate route as you all do. See what an enormous benefit you are to us?
    I’m very impressed an will remain a long time user.
    Jim B

    Reply
    • Dave Teare
      Dave Teare says:

      Great to hear, Jim! I’m glad we could energize you. 🙂 ⚡️

      You’re right, all developers need to have a Developer ID like we do if they want to release macOS apps outside of the Mac App Store. Not all apps require an embedded provisioning profile, however, so don’t be surprised if you don’t find one in every app.

      Take care,

      ++dave;

  10. rlobrecht
    rlobrecht says:

    Thank you Rick, Dave, and the 1Password team. This kind of transparency and explanation is one of the reasons I trust your software with my secrets. Keep it up.

    Reply
    • Dave Teare
      Dave Teare says:

      Thank you so much! ❤️

      To make a point about transparency, I tried to write this response in a transparent ink, but decided not to write the entire message that way 😉

      Except for this little bit where I tell you how much I love you. I didn’t want my wife to see :)

      Cheers! 🍻

      ++dave;

  11. Brian Shim
    Brian Shim says:

    Much love, you guys! Stuff like this can happen to anyone, but you guys have established such a commitment to your users that this is just a small blip in an amazing track record. Keep up the great work, we hope to see you around for many, many years to come!

    Reply
    • Dave Teare
      Dave Teare says:

      Thank you so much for saying so, Brian! I really appreciate you taking the time to share these kind words with us. ❤️

      I’m expecting 1Password and our team to be around for many many many more years to come. After all, 1Password has awesome users like yourself, and for me and the team, we have Ray Kurzweil’s Singularity right around the corner. 🙂

      Take care,

      ++dave;

  12. Brian
    Brian says:

    Thanks for all the details. I wish more folks would this when things go wrong.

    Followup question: Today, Software Update is still telling me that I have to manually download and install the update (version 6.6.1) instead of installing it like normal Software Update updates.

    I don’t need the touchbar stuff, nor any of the fixes it mentions, and I’m gun-shy now about updating.

    What’s the reason why Software Update updates won’t work, and when will that part get back to normal?

    Reply
    • Dave Teare
      Dave Teare says:

      Hi Brian,

      You’re very welcome. I’m glad to hear you enjoyed the details as much as we enjoyed writing them 🙂

      As for the 6.6.1 update, you do indeed need to manually update to this new version. This is expected. You can read the The Eye Of The Storm section of this blog post for all the nitty details on why a manual update is required.

      Even if you don’t have a fancy new MacBook with a TouchBar, you’ll want to manually update so you can get future updates installed automatically. Part of the 6.6.1 release is an update to the updater so that future updates can be installed automatically. Whew, that’s a mouthful to say, isn’t it? I hope it’s easier to read than it is to say 😉

      Take care and enjoy the rest of your weekend.

      ++dave;

  13. J Barton Elliott
    J Barton Elliott says:

    Puzzled, but I did not see a problem until Sat 2/25 am. I use 1Password continuously on my various devices. Something must have updated to make the bug appear. Also, all my updates came from the Apple App Store. Do you suppose there is a different way this manifests itself?

    Thanks for the great software. I could not live without it !

    Reply
    • Dave Teare
      Dave Teare says:

      Good morning! ☀️ And thank you so much for the kind words! They are music to our ears 🙂

      Having this issue appear is rather puzzling. I’m not overly surprised that the issue didn’t happen until now as macOS will cache things and doesn’t re-verify things while 1Password mini is running. So it’s possible to run for days or weeks (or years?) without ever seeing the problem.

      The part that is puzzling me however is you mention you’re running from the Mac App Store. This is odd because the way things work is Apple resigns 1Password when we upload it using their own certificate. So in theory Mac App Store users should not be affected at all.

      Can you please verify that you are indeed running the Mac App Store version? You can read this article for details on how to do that.

      Please let me know.

      ++dave;

  14. J Barton Elliott
    J Barton Elliott says:

    Dave,

    I am indeed NOW running the latest version from the web, which fixed the problem.
    1Password 6
    Version 6.6.1 (661010)
    AgileBits Store

    The IOS version is 6.5.2 App Store.

    Maybe I am wrong and have been updating the Mac this way all along? One gets numb to these things after a while. You explanation of cashing is probably exactly what happened. Mostly i am using the mini versions from Safari, going for days (weeks?) without opening the main app.

    Thanks for the clarification.

    -te

    Reply
  15. xz4gb8
    xz4gb8 says:

    “Curiouser and curiouser!” Cried Alice, as she read this blog entry.

    Guillaume Boudreau command let me to another use of an embedded.provisionprofile:

    $ find

    /Applications/SimpleFloatingClock.app/Contents/embedded.provisionprofile

    <date>2013-11-02T17:37:20Z</date>

    I’ve used Greg Weston’s SimpleFloatingClock.app since migrating to Leopard and continuing to the most recent Sierra Beta. Expired profiles apparently do not matter in some cases.

    Obligatory kiss-up: One of the reasons I originally decided on 1Password about three years ago was the willingness of the AgileBits team to respond to my questions. Since then, I have had many conversations with them and have been well satisfied with both the team and the product, often learning something new and always appreciating the product and its continuing improvement.

    Reply
    • Dave Teare
      Dave Teare says:

      Thanks for the obligatory kiss-up! I appreciate it 🙂

      As for the expired provisioning profile in SimpleFloatingClock.app, my guess is Greg’s not using it for anything critical or perhaps he didn’t wire it up to be used. I’m unsure about that though so don’t quote me on it. You could always as Greg to see if he can solve this mystery for us both 🙂

      Take care,

      ++dave;

  16. xz4gb8
    xz4gb8 says:

    My previous post may have been the victim of both spell check and automatic editing by the blog interface:

    Should have been “command led me”,

    The find command should have show a placeholder for the deleted command text. The “–” from the command output was omitted.

    There is some editing magic that I can not find. Help would be appreciated.

    Reply
    • Dave Teare
      Dave Teare says:

      If it’s any consolation, I didn’t even notice the typo 🙂

      As for formatting on this blog, you can use HTML if you’d like. But the plain text formatting came out readable enough for me.

      ++dave;

    • Jeffrey Goldberg
      Jeffrey Goldberg says:

      Hi David,

      First of all let me thank you for asking. People should be suspicious when they get a message to update some security system in an unusual manner. Asking people to manually update a security product is not something that should be taken lightly. We don’t want people to grow comfortable with clicking on links in some email to fetch a new copy of 1Password.

      The fact that you (and others) are asking questions that are all various forms of “how do I know that these unusual update instructions aren’t trying to trick me into installing something I shouldn’t be installing” is reassuring to me. So thank you for asking!

      Now onto the substance of your question.

      We do publishes hashes, but we don’t really publicize that we publish them to avoid confusing clutter on download pages. These hashes are mostly used as a first checksum by our updater (more on that below) and aren’t really designed for human consumption.

      But for humans who do wish to consume them you will find that for any of these downloads, there is a corresponding .sha256 file.

      So for example, https://cache.agilebits.com/dist/1P/mac4/1Password-6.6.1.pkg will get you 1Password for Mac 6.6.1 and https://cache.agilebits.com/dist/1P/mac4/1Password-6.6.1.pkg.sha256 will get you its SHA256 hash.

      Manually check signatures

      Of course those hashes are published in the same place the downloads come from so they may not be serving all of your security requirements. Their primary use is by our updater as a first checksum of a download. But you can check that an installed version of 1Password is signed by AgileBits. Note that these instructions do not apply to installations through the Mac App Store.

      You can verify that it is signed by us by running

      codesign -v -d /Applications/1Password\ 6.app

      in a Terminal window and looking at the Identifier.

      If you want to check the identifier and also verify that the signature is valid (thus duplicating what the operating system does) you can run this command

      codesign -vvv R="identifier com.agilebits.onepassword4 and anchor trusted" /Applications/1Password\ 6.app

      Why no PGP or DSS?

      In anticipation of a question that people sometimes ask, I would like to say a few words about why we are not also publishing PGP or DSS signature files for each of these downloads. There are a number of reasons:

      • The manual codesigning check above has all of the security properties of a PGP or DSS signature check.
      • Anyone who has the knowledge and motivation to check a PGP or DSS signature would find it no more difficult to run the codesing checks described above.
      • The timestamp on a codesigning signature is far more trustworthy than the timestamp on a PGP signature
      • Managing two codesiging secret keys (the PGP secret key and the Apple codesigning secret key) offers two avenues of attack for creating forgeries than just having one, as it is very unlikely that people will manually check both. So a successful compromise of either of those two keys would be a problem.

      So a PGP signature would add pretty much nothing to actual security as it is possible to check the signature validity and identify of signer through what is already available on the operating system. But producing a PGP signature would require that our build process have to handle yet another secret.

      PGP signatures are an important thing for software that is installed on systems that don’t have full code signing checks. It is also useful for software that is distributed over a large number of channels. But 1Password is not distributed that way or for such systems, so we can make use of technology that addresses the same problem better while putting less burden on the end user.

      Our own updater

      Our own updater uses TLS as a first layer and will reject updates on a TLS security problem. That use of TLS is for all three of the download itself, the hash summary, and the release notes update message. Indeed, the latter is so that when the updater message tells you that you may need to manually update and gives you a link, you can be sure that that message comes from us.

      After fetching the update, it checks the SHA256 hash as described above.

      Finally it unzips it (updates come through .zip files while first installations are now through .pkg files) and then runs a code signing check similar to what is described above. Indeed, it was our “Common Name” that changed when we got a new certificate which is why the manual update was necessary. (We have since fixed this to just compare the Team Identifier; so that future changes in the format of the CN should produce this problem, while still providing a very strict check that it really is our update that we have fetched.)

      All of these checks that we do are in addition to the checks that the operating system performs.

  17. Janusz Buda
    Janusz Buda says:

    It’s a pity I’m retiring in a few weeks. The way AgileBits has dealt with the ‘perfect storm’ would have provided an excellent case study for my graduate school course in how to deal (and how not to deal) with serious corporate problems. Over the years I’ve collected dozens of cases of bad, sometimes disastrous, responses to catastrophic failures. The number of good responses is very small. Yours meets all the criteria for how to do it right, and then some. Not just the handling of the ‘perfect storm’ incident, but prompt, polite, helpful and, most important, continuing support of users in need, and an extra-mile presence on user forums that some developers never visit.

    Drat! Writing the above paragraph has triggered yet another earworm. Now I’ve got Dr. Feelgood’s “She Does It Right” racing around my head.

    All the best to a wonderful company that does it right.

    Reply
    • Jeffrey Goldberg
      Jeffrey Goldberg says:

      Thank you so much Janusz!

      You know that retirement doesn’t mean you need to stop writing.

      (And I’m sure you know that once someone mentions an earworm, others are infected. This is a nice was to have stuck in my head.)

      Cheers,
      -j

Newer Comments »

Leave a Reply

Want to join the discussion?
Feel free to contribute!

What's on your mind?