Developers: How we use SRP, and you can too

Donkey: Oh, you both have layers. Oh. You know, not everybody likes onions. Cake! Everybody loves cake! Cakes have layers!
Shrek: I don’t care what everyone likes! Ogres are not like cakes.
Donkey: You know what else everybody likes? Parfaits! Have you ever met a person, you say, “Let’s get some parfait,” they say, “Hell no, I don’t like no parfait”? Parfaits are delicious!

1Password uses a multi-layered approach to protect your data in your account, and Secure Remote Password (SRP) is one of those very important layers. Today we’re announcing that our Go implementation of SRP is available as an open source project. But first, I’d like to show you the benefits SRP brings as an ingredient in the 1Password security parfait.

Parfaits: delicious and secure

The first layer of security in 1Password, your Master Password, protects your data end to end – at rest and in transit – but we wanted to go further. The second layer is something we call Two-Secret Key Derivation. It combines your Secret Key with your Master Password to greatly improve the strength of the encryption. Thanks to your Secret Key, even if someone got your data from our servers, it would be infeasible to guess your Master Password.

That still wasn’t enough for us, though. When we first started planning how we were going to securely authenticate between 1Password clients and server, we had a wish list. We wanted to ensure that:

  • your Master Password is never transmitted or stored on the server.
  • eavesdroppers can’t learn anything useful.
  • the identity of user and server are mutually authenticated.
  • the authentication is encryption-based.

There was actually one other requirement that wasn’t exactly part of the list but applied to every item in the list: we didn’t want to roll our own solution. We know better than to roll our own crypto, and we wanted to find a proven solution that’s been around and has stood the test of time.

We wanted this layer to be just right.

SRP: a hell of a layer

It took us a while to find what we needed for this layer. (Apparently the marketing department of augmented password-authenticated key agreement protocols is underfunded.) But we eventually found SRP, which ticked all our boxes. SRP is a handshake protocol that makes multiple requests and responses between the client and the server. Now, that may not sound very interesting – and I’m not one to show excitement easily – but SRP is a hell of a layer. With SRP we can:

  • authenticate without ever sending a password over the network.
  • authenticate without the risk of anyone learning any of your secrets – even if they intercept your communication.
  • authenticate both the identity of the client and the server to guarantee that a client isn’t communicating with an impostor server.
  • authenticate with more than just a binary “yes” or “no”. You actually end up with an encryption key.

All this makes SRP a great fit for 1Password, and it keeps your data safe in transit. As an added bonus, because SRP is encryption-based, we end up with a session encryption key we can use for transport security (the fourth layer) instead of relying on just Transport Layer Security (TLS).

So that’s four layers of protection for your 1Password account: Master Password, Secret Key, SRP, and TLS. Now I’d love to show you how SRP works step by step in 1Password.

How 1Password uses SRP

Before any authentication can be done, the account needs to be enrolled. 1Password does this when you create an account. To use SRP, you’ll need a couple different things:

  • a key derivation function (KDF) that will transform a password (and, in our case, also your Secret Key) into a very large number. We’ve chosen PBKDF2.
  • an SRP group consisting of two numbers: one very large prime and one generator. There are seven different groups; we’ve chosen the 4096-bit group.

Enrollment

To enroll, the client sends some important information to the server, and the server saves it:

  1. The client generates a random salt and Secret Key.
  2. The client asks the user for a Master Password.
  3. The client passes those three values to the KDF to derive x.
  4. The client uses x and the SRP group to calculate what’s called a verifier.
  5. The client sends the verifier, salt, and SRP group to the server.
  6. The server saves the verifier and never transmits it back to the client.

Now the account is ready for all future authentication sessions.

Authentication

To authenticate, the client and server exchange non-secret information. Then the client combines that with a secret that only it knows and the server combines it with a secret only it knows:

  1. The client requests the salt and the SRP group from the server.
  2. The client asks the user for the Master Password and Secret Key.
  3. The client passes those three values (minus the SRP group) to the KDF to derive x.
  4. The client:
    1. Generates a random secret number a.
    2. Uses the SRP group to calculate a non-secret counterpart A.
    3. Sends A to the server.
  5. The server:
    1. Generates a random secret b.
    2. Uses the SRP group to calculate a non-secret counterpart B.
    3. Sends B to the client.

So A and B are exchanged, but a and b remain secrets. Through the power of math, the client (with x, a, B) and the server (with the verifier, A, b) can both arrive at the same very large number using different calculations. This is the number that 1Password uses as a session encryption key.

Verification

The last step is verification. After all, no amount of fancy math will help if the numbers don’t match up between client and server.

To verify, the client and server exchange encrypted messages:

  1. The client encrypts a message with the session encryption key and sends it to the server.
  2. The server decrypts the message and verifies it.
  3. The server encrypts its own message with the same session encryption key and sends it to the client.
  4. The client decrypts the message and verifies it.

If the client and server both used the correct inputs then they’ll both have the same session encryption key, which allows them to decrypt and verify the message. If they don’t use the correct inputs, everything fails.

The verification process proves to the server that the client has x, which can only be derived using the correct Master Password and Secret Key. It also proves to the client that the server has the verifier, which ensures that the client is communicating with the 1Password server, not an impostor.

Now that it has been verified, the session encryption key can be used to encrypt every message between the client and server going forward.

As you can see, it’s critical to remember your Master Password and keep your Secret Key safe if you ever want to authenticate your 1Password account. They’re also very important layers, after all. 🙂 And, like any good parfait, everything comes together to create something better than the individual layers alone.

Implement SRP in your own app

We love SRP so much that we want to see it used in more of the apps we use. That’s why we want to help you get started with SRP in your own project. Our Go implementation of SRP is available as an open source project:

This package provides functions for both clients and servers, and you only need to BYOKDF (Bring Your Own Key Derivation Function, like any good party). It’s the same code we’re using on our server and in the 1Password command-line tool, so we welcome security researchers to take a look and report any issues through the 1Password Bugcrowd bug bounty program.

SRP is one of the less appreciated parts of 1Password, and I hope I’ve explained it well enough for you to implement it in your own project. You can read more about how we use SRP in the 1Password Security Design White Paper. Leave a comment if you have any questions, or file an issue if you see something that can be improved.

Until next time, enjoy that parfait!

Terraforming 1Password

A few days ago I posted this tweet:

The tweet generated quite a bit of interest from people running or managing their services, and I thought I would share some of the cool things we are working on.

This post will go into technical details and I apologize in advance if I explain things too quickly. I tried to make up for this by including some pretty pictures but most of them ended up being code snippets. 🙂

1Password and AWS

1Password is hosted by Amazon Web Services (AWS). We’ve been using AWS for several years now, and it is incredible how easy it was to scale our service from zero users three years ago to several million happy customers today.

AWS has many geographical regions. Each region consists of multiple independent data centres located closely together. We are currently using three regions:

  • N. Virginia, USA us-east-1
  • Montreal, Canada ca-central-1
  • Frankfurt, Germany eu-central-1

In each region we have four environments running 1Password:

  • production
  • staging
  • testing
  • development

If you are counting, that’s 12 environments across three regions, including three production environments: 1password.com, 1password.ca, and 1password.eu.

Every 1Password environment is more or less identical and includes these components:

  • Virtual Private Cloud
  • Amazon Aurora database cluster
  • Caching (Redis) clusters
  • Subnets
  • Routing tables
  • Security roles
  • IAM permissions
  • Auto-scaling groups
  • Elastic Compute Cloud (EC2) instances
  • Elastic Load Balancers (ELB)
  • Route53 DNS (both internal and external)
  • Amazon S3 buckets
  • CloudFront distributions
  • Key Management System (KMS)

Here is a simplified diagram:

env

As you can see, there are many components working together to provide 1Password service. One of the reasons it is so complex is the need for high availability. Most of the components are deployed as a cluster to make sure there are at least two of each: database, cache, server instance, and so on.

Furthermore, every AWS region has at least two data centres that are also known as Availability Zones (AZs) – you can see them in blue in the diagram above. Every AZ has its own independent power and network connections. For example, Canadian region ca-central-1 has two data centres: ca-central-1a and ca-central-1b.

If we deployed all 1Password components into just a single Availability Zone, then we would not be able to achieve high availability because a single problem in the data centre would take 1Password offline. This is why when 1Password services are deployed in a region, we make sure that every component has at least one backup in the neighbouring data centre. This helps to keep 1Password running even when there’s a problem in one of the data centres.

Infrastructure as Code

It would be very challenging and error-prone to manually deploy and maintain 12 environments, especially when you consider that each environment consists of at least 50 individual components.

This is why so many companies today switched from updating their infrastructure manually and embraced Infrastructure as Code. With Infrastructure as Code, the hardware becomes software and can take advantage of all software development best practices. When we apply these practices to infrastructure, every server, every database, every open network port can be written in code, committed to GitHub, peer-reviewed, and then deployed and updated as many times as necessary.

For AWS customers, two major languages could be used to describe and maintain the infrastructure:

CloudFormation is an excellent option for many AWS customers, and we successfully used it to deploy 1Password environments for over two years. At the same time we wanted to move to Terraform as our main infrastructure tool for several reasons:

  • Terraform has a more straightforward and powerful language (HCL) that makes it easier to write and review code.
  • Terraform has the concept of resource providers that allows us to manage resources outside of Amazon Web Services, including services like DataDog and PagerDuty, which we rely on internally.
  • Terraform is completely open source and that makes it easier to understand and troubleshoot.
  • We are already using Terraform for smaller web apps at AgileBits, and it makes sense to standardize on a single tool.

Compared to the JSON or YAML files used by CloudFormation, Terraform HCL is both a more powerful and a more readable language. Here is a small example of a snippet that defines a subnet for the application servers. As you can see, the Terraform code is a quarter of the size, more readable, and easier to understand.

CloudFormation

"B5AppSubnet1": {
    "Type": "AWS::EC2::Subnet",
    "Properties": {
        "CidrBlock": { "Fn::Select" : ["0", { "Fn::FindInMap" : [ "SubnetCidr", { "Ref" : "Env" }, "b5app"] }] },
        "AvailabilityZone": { "Fn::Select" : [ "0", { "Fn::GetAZs" : "" } ]},
        "VpcId": { "Ref": "Vpc" },
        "Tags": [
            { "Key" : "Application", "Value" : "B5" },
            { "Key" : "env", "Value": { "Ref" : "Env" } },
            { "Key" : "Name", "Value": { "Fn::Join" : ["-", [ {"Ref" : "Env"}, "b5", "b5app-subnet1"]] } }
        ]
    }
},

"B5AppSubnet2": {
    "Type": "AWS::EC2::Subnet",
    "Properties": {
        "CidrBlock": { "Fn::Select" : ["1", { "Fn::FindInMap" : [ "SubnetCidr", { "Ref" : "Env" }, "b5app"] }] },
        "AvailabilityZone": { "Fn::Select" : [ "1", { "Fn::GetAZs" : "" } ]},
        "VpcId": { "Ref": "Vpc" },
        "Tags": [
            { "Key" : "Application", "Value" : "B5" },
            { "Key" : "env", "Value": { "Ref" : "Env" } },
            { "Key" : "Name", "Value": { "Fn::Join" : ["-", [ {"Ref" : "Env"}, "b5", "b5app-subnet2"]] } }
        ]
    }
},

"B5AppSubnet3": {
    "Type": "AWS::EC2::Subnet",
    "Properties": {
        "CidrBlock": { "Fn::Select" : ["2", { "Fn::FindInMap" : [ "SubnetCidr", { "Ref" : "Env" }, "b5app"] }] },
        "AvailabilityZone": { "Fn::Select" : [ "2", { "Fn::GetAZs" : "" } ]},
        "VpcId": { "Ref": "Vpc" },
        "Tags": [
            { "Key" : "Application", "Value" : "B5" },
            { "Key" : "env", "Value": { "Ref" : "Env" } },
            { "Key" : "Name", "Value": { "Fn::Join" : ["-", [ {"Ref" : "Env"}, "b5", "b5app-subnet3"]] } }
        ]
    }
},

Terraform

resource "aws_subnet" "b5app" {
  count             = "${length(var.subnet_cidr["b5app"])}"
  vpc_id            = "${aws_vpc.b5.id}"
  cidr_block        = "${element(var.subnet_cidr["b5app"],count.index)}"
  availability_zone = "${var.az[count.index]}"

  tags {
    Application = "B5"
    env         = "${var.env}"
    type        = "${var.type}"
    Name        = "${var.env}-b5-b5app-subnet-${count.index}"
  }
}

Terraform has another gem of a feature that we rely on: terraform plan. It allows us to visualize the changes that will happen to the environment without performing them.

For example, here is what would happen if we change the server instance size from t2.medium to t2.large.

Terraform Plan Output

#
# Terraform code changes
#
# variable "instance_type" {
#    type        = "string"
# -  default     = "t2.medium"
# +  default     = "t2.large"
#  }


$ terraform plan 
Refreshing Terraform state in-memory prior to plan...

...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ module.b5site.aws_autoscaling_group.asg (new resource required)
      id:                                 "B5Site-prd-lc20180123194347404900000001-asg" =>  (forces new resource)
      arn:                                "arn:aws:autoscaling:us-east-1:921352000000:autoScalingGroup:32b38032-56c6-40bf-8c57-409e9e4a264a:autoScalingGroupName/B5Site-prd-lc20180123194347404900000001-asg" => 
      default_cooldown:                   "300" => 
      desired_capacity:                   "2" => "2"
      force_delete:                       "false" => "false"
      health_check_grace_period:          "300" => "300"
      health_check_type:                  "ELB" => "ELB"
      launch_configuration:               "B5Site-prd-lc20180123194347404900000001" => "${aws_launch_configuration.lc.name}"
      load_balancers.#:                   "0" => 
      max_size:                           "3" => "3"
      metrics_granularity:                "1Minute" => "1Minute"
      min_size:                           "2" => "2"
      name:                               "B5Site-prd-lc20180123194347404900000001-asg" => "${aws_launch_configuration.lc.name}-asg" (forces new resource)
      protect_from_scale_in:              "false" => "false"
      tag.#:                              "4" => "4"
      tag.1402295282.key:                 "Application" => "Application"
      tag.1402295282.propagate_at_launch: "true" => "true"
      tag.1402295282.value:               "B5Site" => "B5Site"
      tag.1776938011.key:                 "env" => "env"
      tag.1776938011.propagate_at_launch: "true" => "true"
      tag.1776938011.value:               "prd" => "prd"
      tag.3218409424.key:                 "type" => "type"
      tag.3218409424.propagate_at_launch: "true" => "true"
      tag.3218409424.value:               "production" => "production"
      tag.4034324257.key:                 "Name" => "Name"
      tag.4034324257.propagate_at_launch: "true" => "true"
      tag.4034324257.value:               "prd-B5Site" => "prd-B5Site"
      target_group_arns.#:                "2" => "2"
      target_group_arns.2352758522:       "arn:aws:elasticloadbalancing:us-east-1:921352000000:targetgroup/prd-B5Site-8080-tg/33ceeac3a6f8b53e" => "arn:aws:elasticloadbalancing:us-east-1:921352000000:targetgroup/prd-B5Site-8080-tg/33ceeac3a6f8b53e"
      target_group_arns.3576894107:       "arn:aws:elasticloadbalancing:us-east-1:921352000000:targetgroup/prd-B5Site-80-tg/457e9651ad8f1af4" => "arn:aws:elasticloadbalancing:us-east-1:921352000000:targetgroup/prd-B5Site-80-tg/457e9651ad8f1af4"
      vpc_zone_identifier.#:              "2" => "2"
      vpc_zone_identifier.2325591805:     "subnet-d87c3dbc" => "subnet-d87c3dbc"
      vpc_zone_identifier.3439339683:     "subnet-bfe16590" => "subnet-bfe16590"
      wait_for_capacity_timeout:          "10m" => "10m"

-/+ module.b5site.aws_launch_configuration.lc (new resource required)
      id:                                 "B5Site-prd-lc20180123194347404900000001" =>  (forces new resource)
      associate_public_ip_address:        "false" => "false"
      ebs_block_device.#:                 "0" => 
      ebs_optimized:                      "false" => 
      enable_monitoring:                  "true" => "true"
      iam_instance_profile:               "prd-B5Site-instance-profile" => "prd-B5Site-instance-profile"
      image_id:                           "ami-263d0b5c" => "ami-263d0b5c"
      instance_type:                      "t2.medium" => "t2.large" (forces new resource)
      key_name:                           "" => 
      name:                               "B5Site-prd-lc20180123194347404900000001" => 
      name_prefix:                        "B5Site-prd-lc" => "B5Site-prd-lc"
      root_block_device.#:                "0" => 
      security_groups.#:                  "1" => "1"
      security_groups.4230886263:         "sg-aca045d8" => "sg-aca045d8"
      user_data:                          "ff8281e17b9f63774c952f0cde4e77bdba35426d" => "ff8281e17b9f63774c952f0cde4e77bdba35426d"


Plan: 2 to add, 0 to change, 2 to destroy.

Overall, Terraform is a pleasure to work with, and that makes a huge difference in our daily lives. DevOps people like to enjoy their lives too. 🙌

Migration from CloudFormation to Terraform

It is possible to simply import the existing AWS infrastructure directly into Terraform, but there are certain downsides to it. We found that naming conventions are quite different and that would make it more challenging to maintain our environments in the future. Also, a simple import would not allow us to use the new Terraform features. For example, instead of hard-coding the identifiers of Amazon Machine Images used for deployment we started using aws_ami to find the most recent image dynamically:

aws_ami

data "aws_ami" "bastion_ami" {
  most_recent = true
  
  filter {
    name   = "architecture"
    values = ["x86_64"]
  }
  filter {
    name   = "name"
    values = ["bastion-*"]
  }
  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
  name_regex = "bastion-.*"
  owners     = [92135000000]
}

It took us a couple of weeks to write the code from scratch. After we had the same infrastructure described in Terraform, we recreated all non-production environments where downtime wasn’t an issue. This also allowed us to create a complete checklist of all the steps required to migrate the production environment.

Finally, on January 21, 2018, we completely recreated 1Password.com. We had to bring the service offline during the migration. Most of our customers were not affected by the downtime because the 1Password apps are designed to function even when the servers are down or when an Internet connection is not available. Unfortunately, our customers who needed to access the web interface during that time were unable to do so, and we apologize for the interruption. Most of the 2 hours and 39 minutes of downtime were related to data migration. The 1Password.com database is just under 1TB in size (not including documents and attachments), and it took almost two hours to complete the snapshot and restore operations.

We are excited to finally have all our development, test, staging, and production environments managed with Terraform. There are many new features and improvements we have planned for 1Password, and it will be fun to review new infrastructure pull requests on GitHub!

I remember when we were starting out we hosted our very first server with 1&1. It would have taken weeks to rebuild the very simple environment there. The world has come a long way since we first launched 1Passwd 13 years ago. I am looking forward to what the next 13 years will bring! 😃

Questions

A few questions and suggestions about the migration came up on Twitter:

By “recreating” you mean building out a whole new VPC with Terraform? Couldn’t you build it then switch existing DNS over for much less down time?1

This is pretty much what we ended up doing. Most of the work was performed before the downtime. Then we updated the DNS records to point to the new VPC.

Couldn’t you’ve imported all online resources? Just wondering.2

That is certainly possible, and it would have allowed us to avoid downtime. Unfortunately, it also requires manual mapping of all existing resources. Because of that, it’s hard to test, and the chance of a human error is high – and we know humans are pretty bad at this. As a wise person on Twitter said: “If you can’t rebuild it, you can’t rebuild it“.

If you have any questions, let us know in the comments, or ask me (@roustem) and Tim (@stumyp), our Beardless Keeper of Keys and Grounds, on Twitter.

1Password command-line tool 0.2: Tim’s new toys

Some of you may know Tim, our Beardless Keeper of Keys and Grounds here at AgileBits. Tim and his team keep everything running smoothly. The servers are serving happily and the networks are flowing gracefully. Tim is also the administrator of our company team on 1Password.com.

Tim can script and automate with the best of them, and from the moment he got a preview of op, the DevOps team began bombarding us with feedback. One of the first things he asked for was the ability to create vaults, so we added that right away. But we knew we could still do more for Tim – after all he was on the nice list this year – so we got him some new toys to play with. If you’re too excited to read more, you can just start playing with op 0.2 now. To find out more, read on.

Vault into the new year

Our first gift to Tim was more control over vault access. He can now use op to add users to vaults, remove users from vaults, and even delete vaults.

So when Dave told Tim about a new project (codenamed Honey Badger), it was easy to set things up.

Dave needed two developers, Chris and Betty, as well as one of our designers, Matt, involved in the project. With the command-line tool, Tim can switch to his terminal and do this right away. After he signs in, he can create the vault needed for the project:

op create vault "Honey Badger"

But this is old news! He’s been creating vaults for months now. What’s new is that he can now give everyone involved access to that vault:

op add "Chris Meek" "Honey Badger"
op add "Betty Da" "Honey Badger"
op add "Matt Davey" "Honey Badger"

Tim can even create a script to take a list of email addresses and add everyone to the vault at once:

#!/bin/bash
# Usage: add-everyone.sh "Honey Badger" < emailaddresses.txt
while read p; do
    op add $p $1
done

After Matt is done designing project Honey Badger, it’s just as simple to remove him from the vault:

op remove "Matt Davey" "Honey Badger"

When everyone is done with the project, Tim can use op delete vault "Honey Badger" and move on to his next gift.

New year, new groups

The next gift we gave Tim was control over group membership. He can now use op to create and delete groups and choose who belongs to them.

When Dave told Tim that Wendy was moving from the support team to the design team, Tim just casually sipped his cocoa. He knew this would be trivial. We already have groups set up for both teams, so he just ran two commands:

op remove "Wendy Appleseed" "Support"
op add "Wendy Appleseed" "Design"

Tim can also create and remove groups with op create group and op delete group if ever he needs to.

Resolve to level up your skills

The holidays may be over, but we have a feeling Tim will be playing with his new toys for many days to come. If you want to level up your own skills, head over to download this latest release and read the full documentation on our support site.

Level up with op 0.2!

Then pop in to the 1Password Support forum to let us know what you think. You’re all on our nice list, and we love hearing from you. Your feedback after the initial public beta was instrumental in shaping this release.

We’re incredibly excited to continue work on this tool, as it gives you access and control over your 1Password data in a way that’s never been possible before.

Integrate 1Password into your Android apps

Account sign-up and sign-in is a critical piece of your app, but it’s also the part that is most likely to create friction for new users. It might be necessary for your app to collect information such as an email address and password, but let’s face it, those details are rather tedious to type out.

Here’s something that happens far more often than it should. I’m out with a friend and they’ve just told me about this fantastic new app. I get fired up and download it to my phone, only to be confronted with a wall of text fields. I need to register for a new account before I can continue, and that seems like a lot of work. So I tell myself that I’ll do it when I get home and of course, I forget all about it. By the time I look at the app again, I’ve lost that initial excitement and more often than not, I just delete the app.

If I do give the app another chance, I still need to go through the hassle of creating a new account. Of course, I have 1Password to help me here, but I still need to switch between apps, copying and pasting the values as I go. Wouldn’t it be so much better if the app could just ask 1Password to create my credentials and then sign me in?

I’m very happy to say that this is no longer just wishful thinking! We’ve been working closely with Google and other leading password managers on a protocol for Android called OpenYOLO. This new protocol provides the kind of seamless integration that would have saved me so much of the frustration above. The best part is that implementing the protocol is simple, and you can get started on integrating it into your Android app today!

Getting started with OpenYOLO


You Only Login Once (YOLO) is Google’s internal code name for their Smart Lock for Passwords API on Android. OpenYOLO is its open-source successor that enables client apps to communicate directly with supported providers like 1Password. Using OpenYOLO, your app can ask 1Password to create, save, retrieve, or delete its user credentials.

It’s almost effortless to integrate 1Password with your app using OpenYOLO! Let me show you just how easy it is to get started…

Import the library

Add this line to your app’s build.gradle file to import the OpenYOLO API:

dependencies {
    compile 'org.openyolo:openyolo-api:0.3.1'
}

Get the client

You’ll need an instance of CredentialClient in order to interact with OpenYOLO. It provides convenience methods for sending requests and retrieving responses.

mCredentialClient = CredentialClient.getInstance(this);

Prepare the request

For each operation, your app needs to prepare a request and retrieve an Intent for handling it. You can then hand off the request to OpenYOLO with startActivityForResult(). OpenYOLO will take the request and determine if there are any credential providers like 1Password available to handle it. If multiple providers are available, the user will be prompted to select which one to use to complete the request.

Handle the response

Once the chosen provider has completed the request, a response will be returned to your app in onActivityResult(). You can use CredentialClient to pull the relevant information from the response.

Working with 1Password to simplify users’ experiences

Signing up new users

To avoid another story like mine above, you can assist users with new account registration in your app. Before prompting me to manually enter my email address and choose a unique password, your app can make a request to have those credentials generated.

HintRetrieveRequest request = HintRetrieveRequest.fromAuthMethods(AuthenticationMethods.EMAIL);

Intent hintIntent = mCredentialClient.getHintRetrieveIntent(request);

startActivityForResult(hintIntent, HINT_REQUEST_CODE);

In my case, 1Password will prompt me to confirm that I want to create new credentials and OpenYOLO will return details to your app’s onActivityResult().

if (requestCode == HINT_REQUEST_CODE) {
    HintRetrieveResult result = mCredentialClient.getHintRetrieveResult(data);

    if (result.isSuccessful()) {
        Hint hint = mCredentialClient.getHintRetrieveResult(data).getHint();

        signUpUser(hint.getIdentifier(), hint.getGeneratedPassword());
    }
}

Saving user credentials

My account details are all filled in, and I’m ready to start exploring the app, but of course I want to make sure those new account details are stored in 1Password. This will come in handy the next time I need to sign in to your app. You can help here too with a simple save request to OpenYOLO.

Credential credential = new Credential.Builder(mEmail, AuthenticationMethods.EMAIL,
AuthenticationDomain.getSelfAuthDomain(this))
.setPassword(mPassword)
.build();

Intent saveIntent = mCredentialClient.getSaveIntent(CredentialSaveRequest.fromCredential(credential));

startActivityForResult(saveIntent, SAVE_REQUEST_CODE);

Now I am ready to go! Only a couple of seconds after downloading your app, I am all signed up for an account and more importantly, I can begin using it!

Signing in with existing credentials

Okay, now the fun part! My brand new and shiny Pixel 2 XL arrives and I download all of my favourite apps to it. I still need to sign in to my apps, but you’ve gone and saved me all the hassle by once again using OpenYOLO.

CredentialRetrieveRequest request = CredentialRetrieveRequest.fromAuthMethods(AuthenticationMethods.EMAIL);

Intent retrieveIntent = mCredentialClient.getCredentialRetrieveIntent(request);

startActivityForResult(retrieveIntent, RETRIEVE_REQUEST_CODE);

In response to this request, 1Password will prompt me to select my sign-in credentials and OpenYOLO will return them to your app in onActivityResult():

if (requestCode == RETRIEVE_REQUEST_CODE) {
    CredentialRetrieveResult result = mCredentialClient.getCredentialRetrieveResult(data);

    if (result.isSuccessful()) {
        Credential credential = mCredentialClient.getCredentialRetrieveResult(data).getCredential();

        signInUser(credential.getIdentifier(), credential.getPassword());
    }
}

Less noise, more fun!

That’s it! With a few changes to your code, your app now integrates with 1Password using OpenYOLO. This decreases noise from your app and more importantly saves your users time and effort so that they can get to the really important part – using your app.

If a solution like this is in place the next time my buddy gets me to try out your app, I would be able to create an account with a few taps, and they would have the opportunity to show me why they love the app!

DroidCon London 2017

At DroidCon London today, Google announced that OpenYOLO is now ready for developers like yourself to leverage its powers and bring it into the hands of users. If you are also attending the conference, send a wave at my colleague, Michael Verde! He will be available for a hands-on session to help integrate OpenYOLO into your apps.

Reach out to us

For further details on integrating OpenYOLO into your Android app, take a look through the open-source project available on Github. The repository also includes a handful of sample apps to help with development. If you’re curious about the protocol itself, you can read all about it in the OpenYOLO for Android specification.

If you have any questions, feedback or want to let us know that your Android app supports integration with 1Password through OpenYOLO, please reach out to us at support+android@agilebits.com. We look forward to hearing from you!

Announcing the 1Password command-line tool public beta

Here at AgileBits, we’ve been working hard over the last few months to bring power users, developers, and administrators more powerful ways to interact with 1Password. We’re proud to announce that we have something that fits the bill. It’s called the 1Password command-line tool, and we can’t wait to see what you build with it. Let me take this opportunity to walk you through the exciting potential.

Introducing op

1Password apps are available on just about every platform, but they’ve always had the same dependency: a graphical interface. Now all of 1Password is available with just two characters: op.

The 1Password command-line tool makes your 1Password account accessible entirely from the command line. A simple op signin will securely authenticate you with the 1Password service and give you access to a wide range of capabilities:

Getting usernames and passwords from items:

$ op get item OpenProxy | jq '.details.fields[] | select(.designation=="password").value'

"genuine-adopt-pencil-coaster"

Creating new items and vaults:

$ op create item login $(cat aws.json | op encode) --title="AWS"

{"uuid":"5hinhvejl7wtmbeorfts7ho3di","vaultUuid":"i5imjpvdivbsxo56m2ap2n66gy"}
$ op create vault devops

{"uuid":"ny5khay7t3lmhrp4pjsxl4w34q"}

Working with documents:

$ op create document ./devops.pdf --vault=devops --tags=architecture

{"uuid":"i3rsiwjfh7aryvbu5odr4uleki","vaultUuid":"ny5khay7t3lmhrp4pjsxl4w34q"}

If you’re a team administrator, you can also manage other users and shared vaults — all without leaving your terminal:

op suspend john@acmecorp.com

One of the most frequent requests we receive from 1Password Teams customers is the ability to export the Activity Log. With the Pro plan, op list events makes it easy to ingest activity data into the application of your choosing. Be it Splunk, Kibana, Papertrail, or your own tool, op outputs JSON, so it’s simple to work with.

But we didn’t just build the tool to solve specific requests. It’s flexible enough to handle use cases we haven’t even thought of. The possibilities are endless, and we know you’ll come up with something amazing.

🎶 Rock, robot rock (solid) 🎶

The command structure is similar to tools you already use, providing easy integration with your workflow. Now automated systems can have access to secure credentials without ever storing them in plaintext. Here at AgileBits, for example, we’ve been using op for the last few months as part of our automated build systems. It’s been super useful for fetching secure keys and tokens required for building and deploying 1Password. After a secure op signin, we have a script that fetches the appropriate signing key from a shared vault and automatically signs new builds.

The tool was written from the ground up with the battle-tested Go programming language, the very same we used to build the 1Password service itself. As with every 1Password client, all encryption and decryption is done on your machine locally, ensuring the highest level of security best practices you’ve come to expect from the entire family of 1Password apps.

Get yo’ *NIX on

Our dreams of late have been filled with penguins. Two weeks ago we shared a treat with Linux users, and this week it becomes a feast. You might have already tried 1Password for Linux and Chrome OS, but we know what really makes developers salivate: a CLI. You can download op for macOS, Linux, FreeBSD, OpenBSD, and NetBSD on i386, ARM, and AMD64 architectures. Oh, and our Windows friends can play too!

What’s next?

If you’re as excited as we are about this, here’s everything you need to get started:

We highly value the thoughts of people using the beta in the real world, so we can continue improving the tool for you. As we work toward a stable release and eventually open source, please bear in mind that there may be breaking changes down the line, but we’re more than happy to work with you to resolve any issues. We look forward to working together to create some truly useful and powerful tools, and we can’t do it without you.

Now let’s get ready to 🎶 pipe it, grep it, cat it, sed it 🎶

PSA for macOS Developers: Renew Your Certificates & Provisioning Profiles

Welcome to Part 3 in a three-part series of posts that go in-depth on recent events that caused macOS to prevent 1Password for Mac from launching on our customer’s machines. In this thrilling conclusion we’ll go into what we’ve learned and what the rest of the developer community needs to do to prevent this same sort of pain in their own apps.

In case you need to catch up on your reading:

Part 1 : 1Password for Mac 6.5.5: Manual update required

Part 2 : Certificates, Provisioning Profiles, and Expiration Dates: The Perfect Storm

We never take for granted that 1Password is an integral part of our customer’s workflows. It’s an app that has engendered a great deal of trust and any time we stumble and hurt our customers, we spend as much time as needed to fully understand what happened and make sure we cover our bases for the future. The events of this past week are no exception.

We’ve learned a fair amount over the last week, so let’s dive in.

Who This Affects

provisioning-bandaids@2xWe went over this a bit in part 2, but we’ve been able to confirm that the issue we ran into is one that affects any Developer ID signed application also containing a Provisioning Profile. If your app has declared any codesign entitlements there’s a good chance you’ve got a provisioning profile. Often developers think of codesign entitlements only in the context of sandboxing an application, but they’re used for other things as well. In our case it is used to declare a keychain access group.

The presence of the provisioning profile will depend on your use of app services, which you can see in the Capabilities pane in the project editor when viewing the target in Xcode. If any of these options are set, there’s a relatively good chance that your app is shipping with a provisioning profile.

terminal-icon@2xAs a user, you can see if an app contains a provisioning profile by right clicking on the app in Finder, and choosing “Show Package Contents”. Then navigating to Contents to see if there’s a “embedded.provisionprofile” file. Seeing its expiration date requires that you open Terminal and use the security cms -D -i command followed by the path to embedded.provisionprofile file. It will output the xml plist which will contain something that looks like this:

<key>ExpirationDate</key>

<date>2022-02-17T23:59:55Z</date>

Generally, this provisioning profile is set to expire at the same time as your Developer ID certificate. One of the hallmarks of 1Password is that it tends to adopt the latest and greatest technologies that Apple has to offer right on day one. For this reason our provisioning profile was generated relatively early on and therefore we are one of the first ones to experience this pain.

We urge all developers that distribute an app outside of the Mac App Store to check whether their app ships with a provisioning profile, and to verify its expiration date.

 

Short Term Fix

short-term-fix@2xWhen we generated our new provisioning profile last week we also created a new Developer ID certificate. Both this new certificate and the associated provisioning profile expire in 2022. In the short term this buys us a bit of time.

By the time you read this 1Password 6.6.1 will have been published on our website (with a major new version in the Mac App Store as well). This new version will help some users who have been having issues with the manual update process and also comes with a load of other goodies.

 

longterm-fix@2xLonger Term Fix

Apple has posted a thread on their Developer Forum indicating they’ve made changes to the developer center to help with this problem. Newly generated Developer ID Provisioning Profiles are now valid for 18 years instead of 5. That takes us up to 2035, just in time for us to start worrying about y2k38 bugs. If our customers are still using 1Password 6.6.1 in 2035 then they’ve certainly missed a few update notifications. ?

Apple recommends developers generate new provisioning profiles to obtain one that has the longer expiration date. We’ll be doing this on our side shortly.

In practical terms, this solves the issue for our customers.

 

Proper Long Term Fix

Ideally there would be no expiration that affects users. A few years ago I resurrected a system from 1988 and set up an operating system from 1994 on it. Expiration dates on software would have made this impossible. It pains me to think of someone being unable to run 1Password in the future out of curiosity because of arbitrary limits such as this.

The issue we’ve filed with Apple (rdar://30631939) regarding the inability to run apps with expired provisioning profiles remains open. We will continue to advocate for this to be changed and recommend that all developers of affected software do the same (please dupe the rdar). We’ll keep you updated if this changes.

 

out-of-the-storm@2x

Certificates, Provisioning Profiles, and Expiration Dates: The Perfect Storm

As you may have read, this weekend was a little hectic for us and some of our app developer friends1. On Saturday we got word that users of 1Password for Mac were seeing the app fail to launch correctly. It took a few hours, but we diagnosed the problem and released an update that corrected the issue. This issue will only have affected users that downloaded 1Password for Mac directly from our website, so if you downloaded it from the Mac App Store you had a much more calm weekend than we did.

But alas, that story has already been told. Now it’s time for the nitty gritty technical details about all the forces that aligned against us that had us staring up a giant wall of crashing water like George Clooney and Mark Wahlberg.

Prologue: Not All Certificates Are Created Equal

There’s a lot of information to unpack in this post, but before I get started, I’d like to address an assumption I’m seeing far too many people making: that what happened to us was simply an issue of an expired certificate and that all we needed to do was create a new one, just like you do for SSL certificates.

That’s simply not true.

Developer certificates are much different than SSL certificates and serve a very different purpose. Unlike a simple SSL certificate, our developer certificate is used to sign 1Password and needs to be valid during build time. The expiry time of a certificate or provisioning profile should have no impact on whether or not macOS will allow an app to launch or not.
An analogy may be helpful here: if you think of the developer certificate as a carton of eggs, and 1Password as a cake, then it is important not to use expired eggs to make the cake. The fact that the eggs may expire a few days after making the cake should have no effect on the cake itself. After all, the cake is already made and delivered.

Jumping out of the galley and back into our developer world, an expired certificate typically doesn’t affect us until the next time we need to do a release, which would have been this week with our next betas. Certificates control our ability to sign new apps. They don’t affect existing released apps.

For example, we have some users still using 1Password 3 for Mac (hey there, if that’s you, you should really consider upgrading to a 1Password membership as soon as possible!). The first release of 1Password 3 was in 2009, around 8 years ago. Assuming a user is happy with 1Password 3, how long should they expect to be able to continue using the software they paid for? The only acceptable answer to that question is: as long as they feel like it.

Obviously there’s plenty of reasons for why a user would want to upgrade to newer versions, but the fact of the matter is that a user shouldn’t be reliant on us to keep providing updated builds of an unmaintained app just to keep it running. Unlike an SSL certificate, this isn’t something we can simply fix from our end. Fixing the issue we ran into this weekend is a matter of creating a new build of the app and having users update to the new version.

Taking a Tour of the Engine Room

iCloud Sync

To properly understand what happened, let’s take a step back and look at the different parts of this.

In Mac OS X 10.7 Apple introduced Gatekeeper. Gatekeeper is really quite awesome as it gives users control over what software is allowed to run on their system. The default is to allow software from verified and trusted developers: those apps that have been uploaded to the Mac App Store, or those signed with Developer ID certificates made available to the developer by Apple.

Gatekeeper ensures that apps that have been tampered with will refuse to run, and also provides Apple with a way to revoke certain certificates if a developer has been found to be doing harm (i.e. distributing Developer ID signed malware). These simple steps stop a wide variety of attack vectors and we think the world of Apple for having implemented this.

The next layer is the Provisioning Profile. Provisioning Profiles provide information about what the app can do, as well as who can run it. There are certain services on the Mac that require that the app include a Provisioning Profile. In our case, we needed to start using a Provisioning Profile when we added support for unlocking 1Password using Touch ID.

To be clear, Touch ID itself doesn’t necessitate the profile, but in order to unlock your vault we need to store a secret and we choose to store it the OS X keychain. The specific configuration we’re using for that requires declaring that we want access to a specific keychain access group, which needs to be declared in a provisioning profile. The provisioning profile is included in the app bundle and cannot be updated independently of the app.

Next up… XPC. We use XPC to communicate between the 1Password main app and 1Password mini – the little 1Password that runs in your menu bar – and it’s really quite awesome. 1Password mini acts as the brains of the whole operation, and the larger app is mostly just responsible for displaying information. The reason we love XPC so much is because it’s an inter process communication tool that actually provides us the building blocks we need to perform mutual authentication. What this means is that 1Password mini will refuse to communicate with the main app unless it can prove that it’s signed by us. The inverse is true as well.

Storm Clouds Gather

clouds-gathering@2xAt around 3pm EST on February 18th we started getting reports of failures in 1Password for Mac. Folks were seeing an error appear that 1Password was unable to connect to 1Password mini.

Unable to start 1Password

This initial failure occurred due to the fact that the provisioning profile embedded in 1Password mini had an expiration date. Expiration dates seem to be required, and due to the fact that the expiration date elapsed, Gatekeeper decided that 1Password mini was no longer safe to run. We’ve filed a bug with Apple as we feel that this shouldn’t be the case (rdar://30631939 for those of you reading along inside the Mothership).

Only 1Password mini contains the Provisioning Profile as all Touch ID operations happen within that process. This meant that Gatekeeper was deciding that our main 1Password app could launch. Upon launching, 1Password performs its start up sequence which includes asking the system to launch 1Password mini if it’s not already running. When doing so, the system would log the following to the console:

com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): Binary is improperly signed.
com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): removing service since it exited with consistent failure reason When validating /Applications/1Password 6.app/Contents/Library/LoginItems/2BUA8C4S2C.com.agilebits.onepassword4-helper.app/Contents/MacOS/2BUA8C4S2C.com.agilebits.onepassword4-helper:
Code has restricted entitlements, but the validation of its code signature failed.
Unsatisfied Entitlements:
com.apple.xpc.launchd[1] (com.apple.ReportCrash[11041]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash

The 1Password main app detected the failure and provided an error panel telling the user that it couldn’t connect to mini.

Due to the expired Provisioning Profile, 1Password mini wouldn’t launch. And without mini running, 1Password itself was unable to startup successfully. Both mini and 1Password itself were signed with the same Developer ID certificate. Gatekeeper allowed 1Password to run, but due to the different rules for apps with provisioning profiles, it would not allow mini to run.

As far as we can tell, the only way to correct this problem is to provide a new build of the app with an updated provisioning profile with a new expiration date. Within a few hours we were able to publish a new version which did exactly this. As of 6.5.4, we had an app that users could download and run again.

The Eye Of The Storm

eye-of-the-storm@2xAfter this initial bout of terror, death defying feats, and mad scrambles we figured the technical portion of this exercise was finished and had begun transitioning into customer support mode; helping allay the fear, uncertainty, and doubt that this event had caused.

Little did we know at the time, we were only in the eye of the storm – the calm center before things would get rough again.

1Password for Mac includes an updater within the app so that users can easily upgrade to the latest versions as they become available. This updater validates downloads before performing the update to ensure that the updated app is in fact from AgileBits. One of the steps taken during validation is looking at the code signature of the downloaded app and ensuring that it satisfies the following security requirement:

anchor apple generic and identifier com.agilebits.onepassword4 and certificate leaf[subject.CN] = “Developer ID Application: Agilebits Inc.”

This check has worked really well for us. It’s simple and does the trick.

This check is also extremely specific about the common name2 it looks for. When we generated our updated provisioning profile we also needed to generate a new Developer ID certificate. We didn’t realize it at the time, but the common name of newly created certificates now include the team identifier in addition to the company name;  “Developer ID Application: AgileBits Inc. (2BUA8C4S2C)” vs. “Developer ID Application: AgileBits Inc.”. Close. Super close. But we weren’t looking for a “close” match.

The result of this new common name was that even though our app would now launch, the automatic updater would never run successfully because as far as it was concerned the update being provided wasn’t valid and therefore needed to be rejected. This is what users who could still run 6.5.3 and tried to update to 6.5.4 saw.

Once we discovered this problem we had no choice but to pull the 6.5.4 update and issue a 6.5.5 update that included a modified security requirement check. Sadly this didn’t address the fact that users running 6.5.3 and earlier are not able to automatically update to 6.5.5.

Moving Forward and Heading Home

heading-home@2xThis was painful for everyone. We lost sleep over the weekend, but worse than that… our users temporarily lost access to some of their most important information. This is unacceptable to us and we want to make sure this doesn’t happen again.

We’ve reached out to Apple for help and guidance on what we can do to avoid this happening again in the future. Our new provisioning profile doesn’t expire until 2022, but we’ll make sure that this is resolved far before then so that you need not worry about that happening.

If you’re a developer of a Developer ID signed app, we recommend that you check to see if your app includes a provisioning profile. Since that’s mostly handled automatically by Xcode, it’s likely that there are apps out there whose developers aren’t even aware of the inclusion of the provisioning profile. Check the expiration date, and ensure that you release an updated build with an updated provisioning profile well before the expiration date is hit so your users have time to update.

We’ve also filed an enhancement request with Apple asking that developers be notified via email of impending distribution certificate or provisioning profile expirations with explanations of repercussions. This was filed as rdar://30631968.

If you have questions about any of this, please don’t hesitate to ask us in the comments below.

Love,
The 1Password Mac Team
❤️

P.S. Happy 5th Birthday to Gatekeeper! ? We were one of the first apps to sign with Developer ID certificates, use XPC, and leverage the entitlements required for Touch ID. It’s always exciting being on the cutting edge of technology but we wouldn’t have it any other way. ?

Further Reading

This was the second post in a three part series. See the exciting prequel and sequel here:

Part 1 : 1Password for Mac 6.5.5: Manual update required

Part 3 : PSA for macOS Developers: Renew Your Certificates & Provisioning Profiles


  1. The exact same perfect storm appears to caused our friends at Smile to hit the same rough seas that we had. You can see Adam Engst’s story in TidBITS for details on how this affected PDFPen. 
  2. The Common Name is the subject.CN part of the security requirement. As our Chief Defender of the Dark Arts often says of Common Names: they are often very uncommon. The name is inherited from older identify management systems. I don’t need to say much more as Jeff loves explaining things, so let’s all sit back and watch what he says in his comment that I’m sure he’ll be adding soon. 

Having a blast at AGConf[7]

Every year since I joined the AgileBits family, I’ve looked forward to our annual reunion. It’s a rare opportunity to see the faces and hear the voices that only exist in my head and in text most of the time. AGConf gives us the chance to get together, discuss, plan, collaborate, joke, sing, dance, and occupy the same space for one glorious week.

Suffice it to say that AGConf[7] was a blast. We all met in Fort Lauderdale, Florida, where we boarded the Independence of the Seas for a five day Caribbean cruise. We visited Labadee, Haiti, and Falmouth, Jamaica, before making the return trip and parting ways.

Seeing the same old friendly faces

Most of us work remotely, and with the exception of the odd trip to the mothership (Toronto), we don’t get to see each other. The first day is filled with hellos and hugs–and an abundance of both. We always keep our eyes out for the official AGConf tees, and this year’s 007 themed one was a big hit!

Seeing the shiny new faces

In addition to seeing our old friends and colleagues, it’s always fun to meet the new Bits on the team! It seems like we’re growing exponentially these days, and by some wonderful Agilemagic, everyone fits in just right. I’m already looking forward to meeting more new Bits next year!

Getting some work done

Getting such great minds together always makes for some interesting discussion. It’s fun to put our heads together to help customers, solve problems, and make plans for the future. It’s also one of the few times during the year where we can have face to face discussions. Any big plans we’ve been working on in secret can finally be revealed. Any questions we’ve been wanting to ask get answered, sometimes with a full lecture.

Beyond all of the enlightening and productive internal discussion, we also spend a lot of time supporting our customers. At AgileBits, we take customer support very seriously, and everyone takes part in it. On the high seas, we all have each other to bounce ideas off of, or help with tricky issues we may not be too sure about. Need a developer? Just grab one. Have a security question? Goldberg’s got you covered. Working during AGConf almost doesn’t feel like work at all.

Having lots of fun

And of course, when we get together, we have fun. We play games, soak in the hot tubs, drink Labadoozies and rum punches galore, get dressed up fancy for dinner, sing karaoke, eat lots of dessert, get late night pizza, do yoga, watch the sun rise and set…I could go on. There’s so much to do, and so many people to see, it’s hard to make sure to get enough sleep! For a bunch of computer geeks, we’re a pretty rad crew.

 

Parting is such sweet sorrow

Like all other good things, AGConf, too, must come to an end. We came, we saw, we conquered, and we did some other stuff too. Now it’s time to get back to developing software we’re passionate about, and supporting customers we love. Until next time, Bits!

AgileBits Presents: AgileCloudSDK

Today we’re pleased to announce that AgileCloudSDK is officially open source and available on GitHub. AgileCloudSDK is a framework that we’ve built for the purpose of bringing iCloud sync to the AgileBits Store version of 1Password. AgileCloudSDK allows us to take our existing iCloud sync solution and make it work outside the Mac App Store.

Get the Code

Enough with the pleasantries, let’s get to it! All of the code is up on GitHub. The repository includes the framework, two sample apps (one using CloudKit, the other using AgileCloudSDK), and instructions on how to get set up.

Compatibility

Using both CloudKit JS and CloudKit REST Services, AgileCloudSDK is a drop-in replacement for the official CloudKit framework. Its core functionality is compatible enough that for our sync code, the only difference between it and CloudKit is which framework header we import. The gap is a little wider for sync setup, because OS X and iOS handle all the authentication automatically when using CloudKit.

When using CloudKit JS or the REST Services, authentication is not handled automatically. In this case, AgileCloudSDK brings the user to the CloudKit authentication page in their browser. After a successful login, an authentication token is returned to the app.

AgileCloudSDK auth

The differences in authentication required us to modify how sync is configured in 1Password, but thankfully the changes we had to make were fairly minimal.

One of the many benefits of using AgileCloudSDK is that developers can target either the development or production CloudKit container, which is invaluable for debugging issues in a production environment.

Getting Here

The first stage of AgileCloudSDK development was to simply build this thing for our own purposes. It was important to us that we explain to our users how we managed to get iCloud sync outside the Mac App Store; announcing the framework a few months ago was our way of doing that.

The next phase was to get more real-world experience with the framework and to fix issues as they came up. We’ve been refining AgileCloudSDK and we’re really happy with how it’s working out so far, and we think it’s ready for others to use it.

Going Forward

We’re now entering the third phase of AgileCloudSDK development. This is where we would love all you app developers to get involved. AgileCloudSDK contains implementations of every CloudKit class that 1Password uses, which is the majority of them. See our GitHub project page for a list of classes that are not implemented. We’d love for AgileCloudSDK to implement 100% of the CloudKit API.

We used both CloudKit JS and CloudKit REST Services to implement AgileCloudSDK. CloudKit JS is almost always the quickest way to get something built, but the REST services allow more flexibility. We’d love to see more code converted to using only the REST services, with the goal of one day breaking the dependence on any main-thread processing.

Feedback

For general feedback, leave a comment below or e-mail us at support+agilecloudsdk@agilebits.com. If you have questions or comments on implementation, or if you find a bug, please file an issue in the repo.

AgileCloudSDK: iCloud Sync Gets Its Wings

At the last WWDC, Apple announced some changes to CloudKit, the technology that enables an app to sync with iCloud. As many of you know, it was previously impossible for non-Mac App Store apps to sync with iCloud. The changes that Apple made to CloudKit have opened up some really exciting possibilities, and today, we’re happy to announce that we have been able to implement iCloud sync in the AgileBits Store version of 1Password.

1Password 6 for Mac Sync Preferences (iCloud)

Wait…what?

1Password uses the CloudKit API to sync your data with iCloud. In OS X 10.10 Yosemite, the CloudKit framework provided by Apple did all the heavy lifting by communicating with Apple’s servers for the app, but it was only available for apps that were codesigned by the Mac App Store. This meant that only the Mac App Store version of 1Password could sync with iCloud.

You can read more about this in the post that Roustem wrote last fall.

What’s changed?

CloudKit is still the way that developers access the iCloud database, but Apple has provided a brand new way of accessing their CloudKit servers: CloudKit web services. CloudKit web services allows apps to access CloudKit via a really nice web framework called CloudKit JS. When these changes were announced at WWDC, we were excited to start working with this new framework to see what possibilities it would present. Ultimately, CloudKit JS enabled us to support iCloud sync in our AgileBits Store version of 1Password.

[Update] If you are a Mac app developer, and would like to use AgileCloudSDK so that your app can sync with iCloud, your app must also be in the App Store. Your customers, however, will now be able to choose whichever version they want.

How did we do it?

To make sync as seamless and stable as possible we wanted to make very few changes to the existing sync code in 1Password. We felt the best way to do that was to make a framework that looks and acts like Apple’s native CloudKit framework, but uses the CloudKit web services. This means 1Password can use Apple’s native CloudKit framework in the Mac App Store version, and our new AgileCloudSDK framework in the AgileBits Store version.

There are a lot of internal differences between native CloudKit and CloudKit web services. CloudKit web services relies on JavaScript but native CloudKit uses Cocoa classes for data storage, so we knew that we couldn’t reuse the existing code without something to translate the requests and data back and forth.

In order to talk to Apple’s servers, we needed a mediator. Adam Wulf and I created a class that takes native CloudKit API calls, translates them to web service API calls, and translates the responses back to native Cocoa code. The 1Password sync code is now completely ignorant as to whether it’s connecting to native CloudKit or CloudKit web services. This means that 1Password can find your data in iCloud, whether you’re using the Mac App Store version or the AgileBits Store version. We’re extremely pleased with this outcome!

From our customers’ point of view, iCloud sync in the AgileBits Store version of 1Password will look a little bit different during the initial setup. CloudKit JS does not use the iCloud settings from OS X, so to authenticate with Apple, 1Password will prompt you to log in to your iCloud account by displaying the iCloud login page in your default web browser. Once you have logged in to your iCloud account, CloudKit web services sends an authentication token back to 1Password, which it then stores (securely, of course). This enables 1Password to sync with iCloud without having to reauthenticate each time. Since the iCloud login for CloudKit JS is completely separate from the iCloud settings in OS X System Preferences, you can even use a completely different iCloud account if you like!

1Password 6 for Mac iCloud auth

Share the knowledge

One of the challenges we faced when developing this framework was that no one else seemed to be working on this particular problem yet. We want AgileCloudSDK to continue to grow and improve and we can think of no better way to ensure that than to release it as open source. We are currently busy prepping it for release and plan to have more information (including a release date) soon. If you’re interested in learning more about this framework, reach out to us at support+agilecloudsdk@agilebits.com.

To everyone at Apple who worked hard to make this new functionality possible: thank you. You’re awesome.