Webhooks

Using Webhooks

Webhooks are a way for web apps to get real-time notifications when users' files change in Dropbox.

Once you register a URI to receive webhooks, Dropbox will send an HTTP request to that URI every time there's a change in any of the accounts connected to your app. (Note that if there are no accounts connected to your app, your webhook URI will not receive any webhook notifications. For information on how to connect accounts, refer to the OAuth Guide.)

In this tutorial, we'll walk through a very simple example app that uses webhooks to convert Markdown files to HTML. Thanks to webhooks, the app will convert files as soon as they're added to Dropbox. The sample app is running at mdwebhook.herokuapp.com, and the full source code is available on GitHub.

The verification request

To set up a new webhook, find your app in the App Console, and add the full URI for your webhook (e.g. https://www.example.com/dropbox-webhook) in the "Webhooks" section. Note that the URI needs to be one publicly accessible over the internet. For example, 127.0.0.1 and localhost URIs will not work, since Dropbox's servers will not be able to contact your local computer.

Once you enter your webhook URI, an initial "verification request" will be made to that URI. This verification is an HTTP GET request with a query parameter called challenge. Your app needs to respond by echoing back that challenge parameter. In order to avoid introducing a reflected XSS vulnerability, set the following headers in your response to the verification request:

'Content-Type: text/plain'
'X-Content-Type-Options: nosniff'

The purpose of this verification request is to demonstrate that your app really does want to receive notifications at that URI. If you accidentally entered the wrong URI (or if someone maliciously entered your server as their webhook), your app would fail to respond correctly to the challenge request, and Dropbox would not send any notifications to that URI.

The following is Python code (using the Flask framework) that responds correctly to a verification request:

@app.route('/webhook', methods=['GET'])
def verify():
    '''Respond to the webhook verification (GET request) by echoing back the challenge parameter.'''

    resp = Response(request.args.get('challenge'))
    resp.headers['Content-Type'] = 'text/plain'
    resp.headers['X-Content-Type-Options'] = 'nosniff'

    return resp

If your app responds correctly to the challenge request, Dropbox will start sending notifications to your webhook URI every time one of your users adds, removes, or changes a file. If your app fails to respond correctly, you'll see an error message in the App Console telling you about the problem.

Requesting the appropriate scope for your app

If your app is scoped, webhooks require your app to have files.metadata.read authorized by the user in order to receive webhook notifications, so please remember to request the scope during OAuth flow.

Also, see the OAuth Guide for more information about OAuth 2 scopes at Dropbox

Receiving notifications

Once your webhook URI is added, your app will start receiving "notification requests" every time a user's files change. A notification request is an HTTP POST request with a JSON body. The JSON has the following format:

{
    "list_folder": {
        "accounts": [
            "dbid:AAH4f99T0taONIb-OurWxbNQ6ywGRopQngc",
            ...
        ]
    },
    "delta": {
        "users": [
            12345678,
            23456789,
            ...
        ]
    }
}

Note that the payload of the notification request does not include the actual file changes. It only informs your app of which users have changes. You will typically want to call /files/list_folder/continue to get the latest changes for each user in the notification, keeping track of the latest cursor for each user as you go.

Every notification request will include a header called X-Dropbox-Signature that includes an HMAC-SHA256 signature of the request body, using your app secret as the signing key. This lets your app verify that the notification really came from Dropbox. It's a good idea (but not required) that you check the validity of the signature before processing the notification.

Below is Python code (again using Flask) that validates a notification request and then calls a function process_user once for each user ID in the payload:

from hashlib import sha256
import hmac
import threading

@app.route('/webhook', methods=['POST'])
def webhook():
    '''Receive a list of changed user IDs from Dropbox and process each.'''

    # Make sure this is a valid request from Dropbox
    signature = request.headers.get('X-Dropbox-Signature')
    if not hmac.compare_digest(signature, hmac.new(APP_SECRET, request.data, sha256).hexdigest()):
        abort(403)

    for account in json.loads(request.data)['list_folder']['accounts']:
        # We need to respond quickly to the webhook request, so we do the
        # actual work in a separate thread. For more robustness, it's a
        # good idea to add the work to a reliable queue and process the queue
        # in a worker process.
        threading.Thread(target=process_user, args=(account,)).start()
    return ''

Typically, the code you run in response to a notification will make a call to /files/list_folder/continue to get the latest changes for a user. In our sample Markdown-to-HTML converter, we're keeping track of each user's OAuth access token and their latest cursor in Redis (a key-value store). This is the implementation of process_user for our Markdown-to-HTML converter sample app:

def process_user(account):
    '''Call /files/list_folder for the given user ID and process any changes.'''

    # OAuth token for the user
    token = redis_client.hget('tokens', account)

    # cursor for the user (None the first time)
    cursor = redis_client.hget('cursors', account)

    dbx = Dropbox(token)
    has_more = True

    while has_more:
        if cursor is None:
            result = dbx.files_list_folder(path='')
        else:
            result = dbx.files_list_folder_continue(cursor)

        for entry in result.entries:
            # Ignore deleted files, folders, and non-markdown files
            if (isinstance(entry, DeletedMetadata) or
                isinstance(entry, FolderMetadata) or
                not entry.path_lower.endswith('.md')):
                continue

            # Convert to Markdown and store as <basename>.html
            _, resp = dbx.files_download(entry.path_lower)
            html = markdown(resp.content)
            dbx.files_upload(html, entry.path_lower[:-3] + '.html', mode=WriteMode('overwrite'))

        # Update cursor
        cursor = result.cursor
        redis_client.hset('cursors', account, cursor)

        # Repeat only if there's more to do
        has_more = result.has_more

Best practices

Always respond to webhooks quickly

Your app only has ten seconds to respond to webhook requests. For the verification request, this is never really an issue, since your app doesn't need to do any real work to respond. For notification requests, however, your app will usually do something that takes time in response to the request. For example, an app processing file changes will call /files/list_folder/continue and then process the changed files. (In our Markdown example, we needed to download each Markdown file, convert it to HTML, and then upload the result.) It's important to keep in mind that list_folder payloads can sometimes be very large and require multiple round-trips to the Dropbox API, particularly when a new user first links your app and has a lot of files.

To make sure you can always respond within ten seconds, you should always do your work on a separate thread (as in the simple example above) or asynchronously using a queue.

Manage concurrency

When a user makes a number of changes in rapid succession, your app is likely to receive multiple notifications for the same user at roughly the same time. If you're not careful about how you manage concurrency, your app can end up processing the same changes for the same user more than once.

For some applications, this is not a serious issue. In our Markdown-to-HTML conversion app, if the same changes are processed more than once, a file just ends up getting overwritten with the same content, so no harm is done. Work that can be repeated without changing the outcome is called idempotent. If your app's actions are always idempotent, you don't need to worry much about concurrency.

Unfortunately, not every app can be made idempotent easily. For example, suppose you have an app that sends email every time a certain file is changed. To avoid sending duplicate emails, you need to make sure that your app never processes the same user on multiple threads/processes at the same time. The simplest solution is to use leases. When a thread or process starts processing a certain user, it will first acquire a lease on that user, giving it exclusive access.

Webhooks Overview

With webhooks, your web app can be notified when your users' files change, rather than requiring your app to frequently poll Dropbox for changes. Every time a user's files change, the webhook URI you specify will be sent an HTTP request telling it about the changes. You can register webhook URIs via the App Console. To learn more about how to use webhooks, read the webhooks tutorial.

Permissions for scoped apps

For scoped apps, the app needs to be authorized with files.metadata.read scope by the user to be able to see the webhook notifications.

See here for more information about OAuth2 scopes at Dropbox

Handling the verification request

The first request to your new webhook URI will be a verification request to confirm that Dropbox is communicating with the right service.

The verification request will be a GET request with a challenge parameter, which is a random string (e.g. https://www.example.com/dropbox-webhook?challenge=abc123). Your app should echo back the challenge parameter as the body of its response. Once Dropbox receives a valid response, the endpoint is considered to be a valid webhook, and Dropbox will begin sending notifications of file changes. Your app has up to ten seconds to respond to the verification request. Dropbox will not perform automatic retry for verification requests.

Receiving notifications

Once you have successfully added your webhook endpoint, Dropbox will begin sending POST requests to your webhook endpoint when any user linked to the app has a file change. If multiple users make changes at the same time, they may be included in the same notification.

You can verify the authenticity of the request by looking at the X-Dropbox-Signature header, which will contain the HMAC-SHA256 signature of the entire request body using your app secret as the key.

When sending change notifications to your app, Dropbox will handle errors with exponential backoff. Each request that results in a non-200 response code or times out will be re-attempted over the course of about ten minutes. After that, a failure will be reported, which you'll be able to see in the App Console.

If your webhook reports more than 35 errors in the past 10 minutes and exceeds a 4.5% failure rate, your webhook will be disabled and you will receive an email notification. Please note that these thresholds are subject to change. To re-enable the webhook go to the webhooks section in the App Console.

The body of the HTTP request will be JSON with details about the changes.

Notification format

For file changes, your endpoint will receive JSON with the following format:

{
  "list_folder": {
    "accounts": [
      "dbid:AAH4f99T0taONIb-OurWxbNQ6ywGRopQngc",
      ...
    ]
  },
  "delta": {
    "users": [
      12345678,
      23456789,
      ...
    ]
  }
}

The list_folder.accounts list is a list of API v2 account IDs. The delta.users list is a list of API v1 user IDs.

Note that only the user IDs and accounts with file changes are provided. Your app is expected to call /files/list_folder/continue to find out what files changed using the latest cursor your app previously stored for that account.

Dropbox Business notifications

There are also webhooks for the Dropbox Business API. Find out more about Dropbox Business API webhooks.

More information

To see an example of working with webhooks, check out the webhooks tutorial.