How to sync user accounts with G Suite push notifications


If your organization uses G Suite to manage your users, you can synchronize those users’ accounts with other systems. This is a little different to just using Google Sign-in with your own custom app. Let me explain why

By Merlin Carter

We were working with the team at sennder, one of our portfolio companies, to synchronize a backend system with G Suite. Their system has a private codename, so I’m just gonna call it “Reinhardt” because it is based on Django (a web framework named after Django Reinhardt).

For our project, it wasn’t enough to have people sign into Reinhardt with their Google accounts. Reinhardt is a huge system with many roles and permissions levels. The goal was to centrally manage user accounts and permissions from within the G Suite and automatically create matching user accounts within Reinhardt.

To help you understand how to do this yourself, I’m going to break down our process with code examples.

We used the push notification feature of Google’s Directory API to perform the following major tasks:

  • Have Google notify Reinhardt when someone creates a user in G Suite
    Whenever someone creates a new user in the G Suite, we have Google automatically notify Reinhardt by sending a request to a webhook address like this one: 
    https://reinhardt.sennder.com/gsuite/user-webhook
  • Create the user in Reinhardt based on the data that G Suite provides us:
    The webhook process reads the request, extracts the primary email, then uses the primary email as a key to retrieve the full user details from G Suite. Once we have the full user details, we create the user in Reinhardt’s database.

If you want to try this yourself, make sure that you’ve read my companion article, “Beginning a G Suite integration — what to prepare before you start coding”. It gives you an overview of all the prerequisites, such as service accounts and permission scopes.

Once you have that out of the way, you’re ready to start.


Installing the Google APIs Python Client

To interact with Google APIs, it’s easiest to use one of their client libraries. In our case, we used the Python client library for Google APIs, so we’ll be using Python examples from our implementation.

You can install the Google APIs Client by entering the following commands in a bash shell.

pip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install google-api-python-client
Code language: HTML, XML (xml)

Installing the Django and the Django REST Framework

If you want to follow along and try out the examples, you’ll need to install Django and the accompanying REST framework which we use for our webhook receiver.

You can install Django with the following command:

python -m pip install Django

And the REST framework is like so:

pip install djangorestframework

You’ll also need to add the REST framework to your Django INSTALLED_APPS configuration like this:

INSTALLED_APPS = [
    ...
    'rest_framework',
]
Code language: JavaScript (javascript)

If you need more help with the Django-related stuff, check out the documentation on configuring Django apps and the Django REST framework.


Creating a G Suite client wrapper

In our case, we created a wrapper around the standard Google APIs client library. This gave us a convenience class that was more tailored to working with the G Suite APIs. We called it the GSuiteClient, and here’s what it looks like:

from typing import List
from google.oauth2 import service_account
from googleapiclient.discovery import build
class GSuiteClient:
    def __init__(
        self, permission_file_path: str, permission_scopes: List[str], admin_user: str
    ):
        self._permission_file_path = permission_file_path
        self._permission_scopes = permission_scopes
        self._admin_user = admin_user
        self._user_service = None
        self._credentials = None
    @property
    def user_service(self):
        if self._user_service is None:
            self._user_service = build(
                "admin",
                "directory_v1",
                credentials=self.credentials,
                cache_discovery=False,
            )
        return self._user_service.users()
    @property
    def credentials(self):
        if self._credentials is None:
            credentials = service_account.Credentials.from_service_account_file(
                filename=self._permission_file_path, scopes=self._permission_scopes
            )
            self._credentials = credentials.with_subject(self._admin_user)
        return self._credentials

⚠️ Beware of the API version parameter

  • The API version parameter required for the Directory API endpoint is not what we expected it to be.
  • According to Google’s official documentation, you’re supposed to initialize an API with the following syntax:
service = build(‘api_name’, ‘api_version’, …)
  • Like us, you might assume the directory API is initialized like so:
service = build(‘Directory’, ‘v1’, …)
  • But actually, the API name “Directory” is appended to the version parameter like this:
service = build(‘admin’, ‘Directory_v1’, …)

It took us a while to figure this one out. If you look at the reference documentation, a POST request to the directory API looks like this:

POST https://www.googleapis.com/admin/directory/v1/users/watch?customer=my_customer&event=add
Code language: JavaScript (javascript)

We didn’t anticipate that we’d need to concatenate the API name with the version number.


Initializing the G Suite client settings

Our G Suite client wrapper expects three main settings:

  • A path to the credentials that you’re supposed to download when you create a service account. This step is covered in a companion article.
  • The permissions scopes that your integration needs (also covered in the same article).
  • The primary email of a G Suite user who has administrative privileges for your G Suite instance. The integration must always be associated with a “real “user.

We give it these settings when we instantiate our GSuiteClient like this:

# initlialize our GSuite client wrapper with the required settings
gsuite_client = GSuiteClient(
    permission_file_path="C:/gs_example/gsuite-integration-211111-627e77777b0a.json",
    permission_scopes=["https://www.googleapis.com/auth/admin.directory.user"],
    admin_user="[email protected]",
)
Code language: PHP (php)

Creating a notification channel to watch for changes

As Google’s documentation acknowledges, the word “channel” is a bit ambiguous and has a different meaning across Google’s product portfolio. Here is how they define the concept of “channel” in the content of push notifications.

A channel specifies routing information for notification messages. As part of the channel setup, you identify the specific URL where you want to receive notifications. Whenever a channel’s resource changes, the Directory API sends a notification message as a POST request to that URL.

Creating a Channel Generator Class

To create a channel, we first wrote a class that could be instantiated later on. It calls the Users: watch method based on the parameters that you provide it.

Here’s what the class looks like:

from uuid import uuid4
from gs_example.gsuite_part.gsuite_client import GSuiteClient
class GSuiteChannelGenerator:
    def __init__(self, client: GSuiteClient):
        self._client = client
    def create_channel(
        self, url: str, domain: str, event: str, time_to_live: int
    ) -> None:
        param = {
            "domain": domain,
            "event": event,
            "body": {
                "type": "web_hook",
                "id": str(uuid4()),
                "address": url,
                "params": {"ttl": time_to_live},
            },
        }
        self._client.user_service.watch(**param).execute()

Calling the Channel Generator

Our logic to call the channel generator was somewhat complex, so we’ve simplified it for this article. The complex part involved keeping the channel alive, but I’ll get into that in a moment.

In this example, we’re creating the channels when we start the Django server. We’ve updated Django’s standard URL configuration file that initializes when the server starts.

Note: Don’t do this in production because it will create channels every time you start the server. We’ve done it to simplify the example. The Google API sends a sync message as soon as you create a channel, and if it gets a 404 (i.e. your server isn’t yet running), it will not send notifications. That’s why we create the channels during the server initialization.

Here’s what our URL configuration looks like:

from django.contrib import admin
from django.urls import path
from gs_example.gsuite_part.channel_generator import GSuiteChannelGenerator
from gs_example.gsuite_part.gsuite_client import GSuiteClient
from gs_example.gsuite_part.web_hook import WebHookReceiver
urlpatterns = [
    path("admin/", admin.site.urls),
    path("gsuite/user-webhook", WebHookReceiver.as_view()),
]
# initlialize our GSuite client wrapper with the required settings
gsuite_client = GSuiteClient(
    permission_file_path="C:/gs_example/gsuite-integration-268011-627e32157b0a.json",
    permission_scopes=["https://www.googleapis.com/auth/admin.directory.user"],
    admin_user="[email protected]",
)
channel_generator = GSuiteChannelGenerator(client=gsuite_client)
channel_generator.create_channel(
    url="https://reinhardt.sennder.com/gsuite/user-webhook",
    domain="sennder.com",
    event="add",  # for this case we watch user creation
    time_to_live=60 * 60 * 8,
)
channel_generator.create_channel(
    url="https://reinhardt.sennder.com/gsuite/user-webhook",
    domain="sennder.com",
    event="update",  # for this case we watch user edit
    time_to_live=60 * 60 * 8,
)
Code language: PHP (php)
  • The variable urlpatterns defines the path that triggers our webhook receiver (again, we’ll get to the webhook receiver a bit later).
  • The variable gsuite_client instantiates our GSuiteClient with our required settings.
  • The variable channel_generator instantiates the GSuiteChannelGenerator with our preconfigured GSuiteClient.
  • We then call the create_channel function twice so that we get two channels:
    One sends notifications about new users, and another sends notifications about updated users. After you’ve successfully created channels, comment out the two channel_generator.create_channel calls. You won’t need them until the channels expire.
  • The time_to_live argument sets each channel to expire after 8 hours.

After each channel is created, you should get a response that resembles the following example:

{
  "kind": "api#channel",
  "id": "01234567-89ab-xxx-012BAX6789ab", # ID generated for the channel.
  "resourceId": "B4ibMJiIhTx123xf2K2bexk8G4", # ID of the watched resource.
  "resourceUri": "https://www.googleapis.com/admin/directory/v1/users?domain=sennder.com&event=add", # Version-specific ID of the watched resource.
  "token": "target=GSuiteCreateUser", # Present only if one was provided.
  "expiration": 1384823632000, # Actual expiration time as Unix timestamp (in ms), if applicable.
}
Code language: PHP (php)

After Google has created the channel, it will send your webhook receiver a sync message to indicate that it’s ready to send notifications. This message includes the header parameter X-Goog-Resource-State: sync.

In the companion article, I talked about how to set up a test server with the help of ngrok. Here’s how a sync message looks in the ngrok dashboard.

Screenshot: Sync message in the ngrok dashboard showing the Google sync message with sync parameter
Google sync message with sync parameter

How long can a channel live?

We couldn’t really find a definitive answer to this question in Google’s documentation on the Directory API, but we compared it with the documentation for the Drive API and found this note.

For the Drive API, the maximum expiration time is 86400 seconds (1 day) after the current time for File resources and 604800 seconds (1 week) for Changes. If you don’t set the expiration property in your request, the expiration time defaults to 3600 seconds after the current time.

Since Google Drive is part of G Suite, we assumed that channels for watching user accounts would also expire after a week. But we didn’t want our channels to last that long because…

⚠️ Currently, you can’t stop a notification channel

  • This is a bug. Suppose that you want to temporarily stop notifications (eg: you want to fix a bug in the callback receiver). Google provides the stop endpoint for doing just that.
  • However, we found this Stack Overflow issue where a user reported that they get a 404 when they make a stop request in the client or call the stop endpoint directly (there is also general confusion on the correct address of the stop endpoint). 
    There are multiple reports of this issue in various client support forums, and at the time of writing, it doesn’t appear to be fixed. For that reason, we didn’t want a channel to live too long in case we needed to stop notifications.
  • Judging by this bug report for the G Suite Admin SDK, it looks like this issue won’t be fixed any time soon.

To keep a channel alive, we had a process that ran every hour and checked to see if the channel had expired. If it was due to expire within 30mins, we created a new channel. Whenever we created a channel, we stored its metadata in a Redis cache (we didn’t use Memcached because we needed the cache to say alive after a process ended).

Our scheduled process could then check the channel expiry date without making another request to Google.


Acting on a notification

When someone created a user in G Suite, Google sent a request to our webhook receiver with the following headers and body.

POST https://reinhardt.sennder.com/gsuite/user-webhook # Your receiving URL.
Content-Type: application/json; utf-8
Content-Length: 189
X-Goog-Channel-ID: addChannel
X-Goog-Channel-Token: 245tzzzzzzzzzz333
X-Goog-Channel-Expiration: Mon, 09 Dec 2013 22:24:23 GMT
X-Goog-Resource-ID: B4ibMJzzzzzzzzzzbexk8G4
X-Goog-Resource-URI: https://www.googleapis.com/admin/directory/v1/users?domain=sennder.com&event=add&alt=json
X-Goog-Resource-State: delete
X-Goog-Message-Number: 226440
  
{
 "kind": "admin#directory#user",
 "id": "111225555555555555558702",
 "etag": "\"Mf8RAmnAzzzzzzzzzzzzzzdRE/evLIDzzzzzzzzzzzz7Pzq8UAw\"",
 "primaryEmail": "[email protected]"
}
Code language: JavaScript (javascript)

We then used this information to recreate the user in the other system. Here’s the code of the webhook receiver that performs this task.

from django.contrib.auth.models import User
from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView
from .gsuite_client import GSuiteClient
from .gsuite_user_fetcher import GSuiteUserFetcher
# initlialize our GSuite client wrapper with the required settings
gsuite_client = GSuiteClient(
    permission_file_path="C:/gs_example/gsuite-integration-211111-627e77777b0a.json",
    permission_scopes=["https://www.googleapis.com/auth/admin.directory.user"],
    admin_user="[email protected]",
)
gsuite_client_fetcher = GSuiteUserFetcher(user_service=gsuite_client.user_service)
class WebHookReceiver(APIView):
   def post(self, request: Request) -> Response:
       data = request.data
       if "primaryEmail" in data:
           user_email = data["primaryEmail"]
           user = gsuite_client_fetcher.fetch_user_by_email(user_email=user_email)
           try:
               # Check if the user already exists in DB
               u = User.objects.get(username=user.email)
               print('User exists, updating user: ', user)
               # Refresh all user data:
               # We're just refreshing the first/last names here
               # but you can add more fields where necessary..
               u.first_name = user.first_name
               u.last_name = user.last_name
               u.save()
           except User.DoesNotExist:
               print('User did not exist previously')
               User.objects.create_user(username=user.email,
                                        first_name=user.first_name,
                                        last_name=user.last_name,
                                        email=user.email,
                                        password='default_password')
       else:
           print("We got a Sync from Google!!")
       # we return 200 to Google as confirmation that we received the notification.
       return Response(status=200)
  • First, we instantiate the GSuiteUserFetcher class, which we’ll use to retrieve the user details based on their primary email.
  • The function in the WebHookReceiver class checks to see if the notification contains the primary email (if not, we assume it’s just a standard sync message from Google).
  • Once we have confirmed that the primary email is present, we use it to fetch the full details of the user (by calling the fetch_user_by_email method of the G Suite Client Fetcher).

The response from Google resembles the following example:

{
    "kind": "admin#directory#user",
    "id": "10455555555555552791",
    "etag": "enlFCt4L0-k8zzzzzzzzzz/3hRG05XNzzzzzzzzzzzobrHi6Erg",
    "primaryEmail": "[email protected]",
    "name": {
        "givenName": "Hugobert",
        "familyName": "Dinkelbaum",
        "fullName": "Hugobert Dinkelbaum"
    },
    "isAdmin": false,
    "isDelegatedAdmin": false,
    "lastLoginTime": "1970-01-01T00:00:00.000Z",
    "creationTime": "2020-02-03T10:54:01.000Z",
    "agreedToTerms": false,
    "suspended": false,
    "archived": false,
    "changePasswordAtNextLogin": true,
    "ipWhitelisted": false,
    "emails": [
        {"address": "[email protected]", "primary": true},
        {"address": "[email protected]"}
    ],
    "nonEditableAliases": ["[email protected]"],
    "customerId": "C0zzzzzpm",
    "orgUnitPath": "/",
    "isMailboxSetup": true,
    "isEnrolledIn2Sv": false,
    "isEnforcedIn2Sv": false,
    "includeInGlobalAddressList": true,
    "customSchemas": {
        "PM_Teams": {
            "team_name1": true,
            "team_name2": true
        },
        "Roles": {
            "role_name1": true,
            "role_name2": true
        },
        "User_Level": {"Team_Lead": true, "Admin": true, "Basic": true},
    }
}
Code language: JSON / JSON with Comments (json)

You can see that the response contains a number of custom G suite fields, such as “User_Level”. We these fields to determine the equivalent permissions when creating the user entry in Reinhardt.

After we receive the response, we use Django’s built-in serializer to serialize the user details into an object called “UserEntity” like this:

class GSuiteUserSerializer(Serializer):
    primaryEmail = CharField()
    name = DictField()
    isAdmin = BooleanField()
    def create(self, validated_data) -> UserEntity:
        return UserEntity(
            email=validated_data["primaryEmail"],
            first_name=validated_data["name"]["givenName"],
            last_name=validated_data["name"]["familyName"],
            is_admin=validated_data["isAdmin"],
        )

Then, we check if the user already exists in the Reinhardt database. If they are a newly created G Suite user, we call the standard Django create_user() method to insert the user entry into the Reinhardt database as well.

if "primaryEmail" in data:
        user_email = data["primaryEmail"]
        user = gsuite_client_fetcher.fetch_user_by_email(user_email=user_email)
        try:
            # Check if the user already exists in DB
            u = User.objects.get(username=user.email)
            print('User exists, updating user: ', user)
            # Refresh all user data:
            # We're just refreshing the first/last names here
            # but you can add more fields where necessary..
            u.first_name = user.first_name
            u.last_name = user.last_name
            u.save()
        except User.DoesNotExist:
            print('User did not exist previously')
            User.objects.create_user(username=user.email,
                                     first_name=user.first_name,
                                     last_name=user.last_name,
                                     email=user.email,
                                     password='default_password')
Code language: PHP (php)

A note about passwords: unsurprisingly, we can’t get the user password from the Directory API. The problem is Django’s create_user() method requires a password.

To get around this, we just randomly generate a password and require the user to reset their password when they first log in to Reinhardt.

If the notification relates to an existing user, we make sure that the user already exists in the Reinhardt database, and we then update the entry, field by field.

We use Django helper functions to update all the fields in the database with the new data G Suite admin API. It doesn’t matter if 1 field or 5 fields were changed, all fields are updated with the latest values from G Suite.


Synchronizing user accounts saves a lot of time

Before we created this integration, sysadmins had to create each user twice. Once in G Suite and again in Reinhardt. This was particularly cumbersome at the beginning of the month when large batches of new hires would need to be onboarded. For one user, it doesn’t take that long to create duplicate accounts. But it’s more laborious and error-prone when you have to do it for 20 consecutive accounts.

Although this integration turned out to be much more complex than we expected, we’re happy that we could spare people some monotonous busywork. And hopefully, we’ve also made it easier for you to do the same thing — but without the “gotchas” that slowed us down.