BrightCloud Web Service Overview

Purpose

BrightCloud's Hosted Internet Security Service provides access to the largest, most accurate, and most up to date repository of categorized Web sites in the industry. It is a URL content and security classification service that provides access to classifications for more than 13 billion URLs (and growing). It provides comprehensive, accurate, and up to date URL classifications for both the most popular and most obscure websites.

BrightCloud Web Service (BCWS) is the web interface to this most comprehensive URL classification service from BrightCloud.

What can you do using BrightCloud Web Service?

Using BrightCloud Web Service, you can

  • Access a list of all the categories made available by BrightCloud. In addition to providing you with a dynamic way to create a comprehensive category list (for reporting or data structure creation), this saves you space and bandwidth, improving speed and decreasing infrastructure costs.
  • Access the most comprehensive classification information for any URL. This classification information can then be used as a basis for creating a high quality web filtering system, proxy caching system, safe browsing browser plug-in, or countless other applications! Additionally, because each of a given URL's categories have a confidence score, a rich policy model can be built for any integration.
  • Immediately access any new/updated classifications that BrightCloud discovers. This can be used for getting real time updates for URLs whose categories have changed, for example from benign to malicious because they've been hacked and are downloading malware. This is possible for applications which make BCWS calls every time or even for integrations maintaining a local cache of URL classifications that should be updated for any important categorization changes on the BrightCloud master database.
  • Report any URLs to BrightCloud that haven't been categorized yet. The corresponding sites are then categorized by BrightCloud and our service is updated. Therefore, by submitting URLs to us which we don't know about, you'll be automatically making sure the service improves to make sure your application knows about it in the future.
  • Suggest alternate classifications for a URL. The number of URLs we classify every day is huge. We inevitably make mistakes. Therefore, we allow you to programmatically suggest that we take another look at how we classified a particular URL. We take these seriously, so you can improve the service for everyone by suggesting good categories whenever you find a mistake!

 

REST API

Introduction

The REST interface to BrightCloud Web Service is the recommended way of accessing the web service. It is more in line with the World Wide Web (www) architecture, which is true for any carefully designed REST interface. This interface uses URLs to address all the logical resources made available by the web service, and a fixed set of HTTP methods to query/add/update these resources.

Security Considerations

Every REST request to the BrightCloud Web Service needs to be signed using a special key/secret pair. BrightCloud Web Service uses the 2-legged variation of the OAuth standard protocol for authenticating REST requests. The OAuth protocol not only uses the key/secret pair but also a timestamp and a random "nonce" value for signing the requests to provide a high degree of safety against a variety of attacks.

BrightCloud web service clients are required to generate a security string in accordance with the OAuth protocol (using their key/secret pair) for every REST request and send it in the HTTP Authorization header value in the request. A number of readymade OAuth consumer libraries are available on the internet for quite a few popular languages (including PHP, Java, Ruby, .NET) that one can use for signing the requests with minimal effort.

The key/secret pair is unique for each end user. You can generate your own key/secret pair by registering at the BrightCloud Web Services website (bcws.brightcloud.com/signup). You can find more information about this in our online article OAuth Integration for BrightCloud Web Services.

Rest URLs and Methods

The following table lists all combinations of resources, URLs, and allowed HTTP methods that BCWS supports for it's REST interfaces.

Resource URL HTTP Methods
URI List http://thor.brightcloud.com/rest/uris GET, POST
URI Info http://thor.brightcloud.com/rest/uris/{ URL } GET, PUT
Category List http://thor.brightcloud.com/rest/uris/categories GET

Retrieve BrightCloud categories

Resource: Category List
URL: http://thor.brightcloud.com/rest/uris/categories
HTTP Method: GET

Sample Request Sample Response
GET /rest/uris/categories HTTP/1.1
HOST: thor.brightcloud.com
Authorization: OAuth
    realm="http://thor.brightcloud.com/rest",
    oauth_version="1.0",
    oauth_nonce="dbe9e4311c72dad530b7afe47ec50ceb",
    oauth_timestamp="1248962133",
    oauth_consumer_key="dpf43f3p2l4k3l03",
    oauth_token="",
    oauth_signature_method="HMAC-SHA1",
    oauth_signature="lWdn1ll3bAsKWMgpdheBvyt4bcQ%3D"
HTTP/1.1 200 OK
Transfer-encoding: chunked
Content-type: text/xml

< bcap >
  < seqnum >1< /seqnum >
  < response >
    < status >200< /status >
    < statusmsg >OK< /statusmsg >
    < categories >
      < cat >
        < catid >68< /catid >
        < catname >Abortion< /catname >
        < catgroup >Legal Liability< /catgroup >
      < /cat >

      < cat >
        < catid >46< /catid >
        < catname >Abortion- Pro Choice< /catname >
        < catgroup >Legal Liability< /catgroup >
      < /cat >
      .
      .
      .
    < /categories >
  < /response >
< /bcap >

Retrieve categories for a URL

Resource: URI Information
URL: http://thor.brightcloud.com/rest/uris/{ URL }
HTTP Method: GET

Sample Request Sample Response
GET /rest/uris/www.google.com HTTP/1.1
HOST: thor.brightcloud.com
Authorization: OAuth
    realm="http://thor.brightcloud.com/rest",
    oauth_version="1.0",
    oauth_nonce="dbe9e4311c72dad530b7afe47ec50ceb",
    oauth_timestamp="1248962133",
    oauth_consumer_key="dpf43f3p2l4k3l03",
    oauth_token="",
    oauth_signature_method="HMAC-SHA1",
    oauth_signature="lWdn1ll3bAsKWMgpdheBvyt4bcQ%3D"
HTTP/1.1 200 OK
Transfer-encoding: chunked
Content-type: text/xml

< bcap >
  < seqnum >1< /seqnum >
  < response >
    < status >200< /status >
    < statusmsg >OK< /statusmsg >
    < uri >www.google.com< /uri >
    < categories >
      < cat >
        < catid >50< /catid >
        < conf >85< /conf >
      < /cat >
    < /categories >
    < bcri >88< /bcri >
    < a1cat >1< /a1cat >
  < /response >
< /bcap >

Retrieve real-time updates

Resource: URI List
URL: http://thor.brightcloud.com/rest/uris
HTTP Method: GET

Sample Request Sample Response
GET /rest/uris HTTP/1.1
HOST: thor.brightcloud.com
Authorization: OAuth
    realm="http://thor.brightcloud.com/rest",
    oauth_version="1.0",
    oauth_nonce="dbe9e4311c72dad530b7afe47ec50ceb",
    oauth_timestamp="1248962133",
    oauth_consumer_key="dpf43f3p2l4k3l03",
    oauth_token="",
    oauth_signature_method="HMAC-SHA1",
    oauth_signature="lWdn1ll3bAsKWMgpdheBvyt4bcQ%3D"
HTTP/1.1 200 OK
Transfer-encoding: chunked
Content-type: text/xml

< bcap >
  < seqnum >1< /seqnum >
  < response >
    < status >200< /status >
    < statusmsg >OK< /statusmsg >
    < updatecdn >false< /updatecdn >
    < updatertu >false< /updatertu >
    < updatetime >2009/7/30 5:50:4< /updatetime >
    < cdnlist >< /cdnlist >
    < rtulist >< /rtulist >
  < /response >
< /bcap >

Report unknown/uncategorized URLs to BrightCloud

Resource: URI List
URL: http://thor.brightcloud.com/rest/uris
HTTP Method: POST

Sample Request Sample Response
POST /rest/uris HTTP/1.1
HOST: thor.brightcloud.com
Content-Type: text\xml
Content-Length: 80
Authorization: OAuth
    realm="http://thor.brightcloud.com/rest",
    oauth_version="1.0",
    oauth_nonce="dbe9e4311c72dad530b7afe47ec50ceb",
    oauth_timestamp="1248962133",
    oauth_consumer_key="dpf43f3p2l4k3l03",
    oauth_token="",
    oauth_signature_method="HMAC-SHA1",
    oauth_signature="lWdn1ll3bAsKWMgpdheBvyt4bcQ%3D"

< uncathits >
  < uri >{ URL1 }< /uri >
  < hits >{ No. of hits for URL1 }< /hits >

  < uri >{ URL2 }< /uri >
  < hits >{ No. of hits for URL2 }< /hits >
  .
  .
  .
< /uncathits >
HTTP/1.1 200 OK
Transfer-encoding: chunked
Content-type: text/xml

< bcap >
  < seqnum >1< /seqnum >
  < response >
    < status >200< /status >
    < statusmsg >OK< /statusmsg >
  < /response >
< /bcap >

Suggest alternate categories for a URL

Resource: URI Information
URL: http://thor.brightcloud.com/rest/uris/{ URL }
HTTP Method: PUT

Sample Request Sample Response
PUT /rest/uris/www.google.com HTTP/1.1
HOST: thor.brightcloud.com
Content-Type: text\xml
Content-Length: 80
Authorization: OAuth
    realm="http://thor.brightcloud.com/rest",
    oauth_version="1.0",
    oauth_nonce="dbe9e4311c72dad530b7afe47ec50ceb",
    oauth_timestamp="1248962133",
    oauth_consumer_key="dpf43f3p2l4k3l03",
    oauth_token="",
    oauth_signature_method="HMAC-SHA1",
    oauth_signature="lWdn1ll3bAsKWMgpdheBvyt4bcQ%3D"

< newcats >
  < categories >
    < cat >{ Category1 }< /cat >
    < cat >{ Category2 }< /cat >
    .
    .
    .
  < /categories >
< /newcats >
HTTP/1.1 200 OK
Transfer-encoding: chunked
Content-type: text/xml

< bcap >
  < seqnum >1< /seqnum >
  < response >
    < status >200< /status >
    < statusmsg >OK< /statusmsg >
  < /response >
< /bcap >

 

Next steps

Get setup and get going quickly