RESTful Realtime

RealCrowd is an online investment platform for commercial real estate. Investors can purchase equity in institutional quality properties offered by top notch investment companies (called operators). These operators identify the asset they want to buy, secure loans, and raise money from investors to buy it. RealCrowd lets operators do the money raising part easily online. Because of the nature of real estate transactions there is a limited time window for raising funds and investors need to decide fairly quickly if they are going to participate.

When we send out the announcement for a new investment listing we get a stampede of visitors. These visitors will usually sit on the investment page for 10 minutes or more as they review detailed financial documents and other information about the investment, contemplating a significant decision. Often times investments are made by others when an investor has the page open. We wanted to avoid the poor user experience of an “out of stock” situation at investment time so we keep the critical investment information up to date in real time. An investor on the site today sees the updated amount available even as investments are made by other people. We also plan to support various social features in the future, which could mean user-to-user communication or other interactions, and these kinds of features will require realtime behavior, so we are setting ourselves up to easily expand into that later.

Approach

Nowadays there are about 1,000 different services and components you can use to do realtime. We decided to use GRIP to realtime-enable our API. GRIP was created by the folks at Fanout and is a simple yet flexible way of designing a realtime system. Basically, client connection management is isolated to a dedicated proxy (such as Pushpin) and our API communicates with that proxy to send updates to users. As we’re an “API first” company, we found this approach more attractive than alternatives like Socket.IO because our public interface stays clean, simple, and under our control.

Since our backend code is primarily written in C#, we created a GRIP library for .NET. It is packaged in NuGet as RealCrowd.Grip. We hope you find it useful!

Implementation

Our investment API now supports long-polling for straightforward and durable realtime push. In sticking with our RESTful philosophy, we use the ETag and If-None-Match headers to detect for changes. If the client provides an If-None-Match header that matches the data known by the server, then the request hangs open until the data changes, at which point the server can respond with new data instantly. If enough time passes with no change, the server responds with code 304 Not Modified.

What’s great about this protocol flow is that it’s identical to a traditional check for resource modification. Some browsers like Chrome will even include If-None-Match on AJAX requests if an ETag was received in an earlier response, making client side code super simple.

For example, a client may request investment information against our realtime domain stream.realcrowd.com:

GET https://stream.realcrowd.com/api/v1/investment/fortdaviscenter

Since the If-None-Match header was not supplied, the server responds immediately:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "320000000"

{
  "data": {
    "minInvestment": 0,
    "investmentIncrement": 0,
    "investmentId": "fortdaviscenter",
    "commitmentsDueDate": 0,
    "totalEquityAmount": 330000000,
    "totalUserCommitment": 0,
    "description": null,
    "projectedIrrHigh": 0.0,
    "targetAmount": 330000000,
    "totalCommitments": 320000000,
    ...
  }
}

The client can check for updates by performing the same request again, but this time including an If-None-Match header containing the value of the ETag header received in the previous response from the server.

GET https://stream.realcrowd.com/api/v1/investment/fortdaviscenter
If-None-Match: "320000000"

At this time we’re simply using the commitment level as the ETag. The ETag value is opaque to the client, though, so we can always make a more advanced value in the future if we choose (such as a digest of the entire object).

If no commitment change occurs for awhile, the server times out the request and responds with 304:

HTTP/1.1 304 Not Modified
Content-Length: 0
ETag: "320000000"

Otherwise, if more money is committed, then the server will respond with the latest data. Here we’ve gone from $3.2M committed to $3.3M:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "330000000"

{
  "data": {
    "minInvestment": 0,
    "investmentIncrement": 0,
    "investmentId": "fortdaviscenter",
    "commitmentsDueDate": 0,
    "totalEquityAmount": 330000000,
    "totalUserCommitment": 0,
    "description": null,
    "projectedIrrHigh": 0.0,
    "targetAmount": 330000000,
    "totalCommitments": 330000000,
    ...
  }
}

If a client receives a 304, it simply makes the same request again. This endpoint is being used today on RealCrowd property pages to ensure the latest commitment information is displayed.

Code

Making the investment endpoint realtime capable only took a couple of changes. First, near the start of our endpoint handler, we now check for the If-None-Match header and respond with an application/grip-instruct message if it matches the value on the server. This tells the GRIP proxy to keep the client on the line in a long poll, but doesn’t require our API servers to maintain that connection. Otherwise, we set the ETag on the response we are building and continue processing as normal, in which case the GRIP proxy will simply return the response to the client.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
var response = new HttpResponseMessage(HttpStatusCode.OK);
 
...
 
var serverETag = '\"' + data.ETag + '\"';
var clientETag = this.Request.Headers.IfNoneMatch.FirstOrDefault();
 
if (clientETag != null && clientETag.ToString() == serverETag)
{
    var gripPrefix = Configuration.Current.GripPrefix;
    var gripProxies = RealCrowdApi.Configuration.Current.GripProxies;
 
    if (gripProxies != null && this.Request.CheckGripSignature(gripProxies))
    {
        var instruct = new RealCrowd.Grip.Instruct();
        instruct.CreateResponseHold(gripPrefix + "investment-" + id, data.ETag,
            "application/json", "");
        instruct.Response.Code = 304;
        instruct.Response.Headers.Add(new KeyValuePair<string, string>("ETag", serverETag));
        response.Content = new StringContent(instruct.ToString(), Encoding.UTF8,
            "application/grip-instruct");
    }
    else
    {
        response.StatusCode = HttpStatusCode.NotImplemented;
        response.Content = new StringContent("Error: Realtime endpoint not supported.\n");
    }
 
    return response;
}
 
response.Headers.ETag = new EntityTagHeaderValue(serverETag);
 
...

If the request should be held open, we bind it to a channel name specific to the investment id. Channels are a lot like namespaces in Socket.IO. We also specify the response information that the GRIP proxy should use in the event of a timeout. The CheckGripSignature() method is handy, as we can use it to detect if an incoming request actually came from a GRIP proxy. If the request didn’t, then we respond with code 501 Not Implemented. We use this for our local workstation development environment so the client hitting that endpoint will not attempt any polling unless the developer has the GRIP proxy running and is working with the realtime components.

The above code is enough to make requests hang open and timeout properly. This brings us to the next part: publishing asynchronous updates. We already had a background job that ran anytime investment data changed, so all we needed to do was publish HTTP response data to the GRIP proxy from the job handler.

1
2
3
4
5
6
7
8
9
10
11
...
var gripPrefix = Configuration.Current.GripPrefix;
 
var id = Convert.ToString(data.TotalCommitments);
var responseFormat = new RealCrowd.Grip.HttpResponseFormat("application/json",
    await content.ReadAsStringAsync());
responseFormat.Headers.Add(new KeyValuePair<string, string>("ETag", data.ETag));
 
await gripPublisher.PublishAsync(gripPrefix + "investment-" + commitment.InvestmentId, id,
    null, responseFormat);
...

In the above code, content is an ObjectContent that we must read from to create an HttpResponseFormat to publish. That’s it for the server side.

On the browser side, we’re using Pollymer to access our API. It’s a JS library that contains various AJAX conveniences for long-polling. Here’s the code we run on every investment page:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
var req = new Pollymer.Request({ maxTries: -1, errorCodes: '500,502-599' });
var uri = realtimeApiRoot + 'investment/' + investmentId;
req.on('finished', function (code, result, respHeaders) {
    if ((code >= 200 && code <= 299) || code == 304) {
        var etag = null;
        for (var key in respHeaders) {
            if (!respHeaders.hasOwnProperty(key)) {
                continue;
            }
            if (key.toLowerCase() == 'etag') {
                etag = respHeaders[key];
            }
        }
        if (!etag) {
            console.log("failed to get ETag: headers=" + JSON.stringify(respHeaders));
            return;
        }
 
        process(result);
        reqHeaders = { 'If-None-Match': etag };
        req.start('GET', uri, reqHeaders);
    }
});
req.start('GET', uri);

What this does is create a request object that is configured to retry failed requests indefinitely (except code 501). On success, process the result, and of course deal with any ETag-related things and start another poll. Notice that the request object is reused. This ensures we benefit from Pollymer’s randomized delay behavior, preventing connection stampedes on events where clients would get simultaneously disconnected such as load balancer, server, or router restarts.

Conclusion

Real estate may be an old industry but we pride ourselves in building a cutting edge service. What do you think of our RESTful realtime API approach? If you’re using .NET and want to do the same thing, be sure to check out our GRIP library for .NET.

Discuss on Hacker News