Developer Documentation

Help with using our API

1 - Linked Data API

1.1 URI Dereferencing

Following the standard practices for Linked Data, we distinguish between a 'real-world' resource and documents about that resource. Identifiers (URIs) for the resources follow the pattern:

http://data.digitalsocial.eu/id/{...}

When you look them up you get redirected with an HTTP status code of 303 ("See Other") to the corresponding document about that thing. The document URLs follow the pattern:

http://data.digitalsocial.eu/doc/{...}

In cases where a URI identifies something that is essentially a document (an 'information resource') then we respond with a 200, as their URI and document page URL are one and the same. This includes Datasets as well as ontology terms and concept schemes.

For example dereferencing the resource with the following URI, responds with HTTP status 200 ("success").

http://data.digitalsocial.eu/data/organizations-and-activities

1.2 Resource Formats

You can specify what format you want the resulting document to be in. By default you get HTML in a human-readable form, but you can also ask for the document in one of several RDF formats: RDF/XML, N-triples, Turtle or JSON-LD.

There are two ways to specify which format you want: you can append a format extension to the document page's URL or you can use an HTTP Accept header with the resource's URI or document page's URL.

Format	Extensions	Accept Headers
RDF/XML	.rdf	application/rdf+xml
n-triples	.nt, .txt, .text	application/n-triples, text/plain
Turtle	.ttl	text/turtle
JSON-LD	.json	application/ld+json, application/json

1.3 Example: Dereferencing URIs with Ruby

Here's an example of dereferencing a URI using the RestClient library. Similar approaches can be taken in other languages. This assumes you already have Ruby set up on your system. Also, if you don't already have it, you'll need to install the gem:

$ gem install rest-client.

... and require it in your script.

require 'rest-client'.

1.3.1 Specifying the format in an accept header - in this case RDF/XML

If you're using the accept header, you can directly request the URI. This involves two requests, because doing an HTTP GET on the resource identifier gives you a 303 redirect to the appropriate document page. RestClient looks after that for you.

RestClient.get 'http://data.digitalsocial.eu/id/organization/51302748-844d-ae4b-6657-b1162fda0a05', :accept=>'application/rdf+xml'

1.3.2 Specifing the format as an extension - in this case JSON

RestClient.get 'http://data.digitalsocial.eu/id/organization/51302748-844d-ae4b-6657-b1162fda0a05.json'

1.4 Example: Dereferencing URIs with cURL

Here's an example of dereferencing a URI using the widely available cURL command line program.

1.4.1 Specifying the format in an accept header - in this case Turtle

curl -L -H "Accept: text/turtle" http://data.digitalsocial.eu/id/organization/51302748-844d-ae4b-6657-b1162fda0a05.json

1.4.2 Specifing the format as an extension - in this case N-triples

curl http://data.digitalsocial.eu/id/organization/51302748-844d-ae4b-6657-b1162fda0a05.nt

2 - Other Resource APIs

Alongside the URI dereferencing we offer the following additional ways of accessing data in the system. Please be sure to read the Options and Limits section, for some background information which applies to all these APIs, such as details on data formats and pagination.

Some examples of accessing the data from our APIs using different languages follow at the end of this section.

2.1 Individual Datasets

Dataset identifiers take the form http://data.digitalsocial.eu/data/{dataset-short-name}, where {dataset-short-name} is a URI section that uniquely identifies the dataset. The short name can contain lower-case letters, numbers, slashes, and hyphens.

Dereferencing a dataset identifier responds with HTTP status code 200 and provides metadata about the dataset, including a link to where the dataset contents can be downloaded.

e.g. http://data.digitalsocial.eu/data/organizations-and-activities

Please also see the Use of Named Graphs section, for how the dataset data and metadata is stored in the database.

2.2 Themes

There are no themes in this linked data site, due to the small number of datasets.

2.3 Collections of Datasets

A list of all datasets is available at: http://data.digitalsocial.eu/data, paginatable with page and per_page.

2.4 Individual Resources

As well as using URI dereferencing to access information about individual resources, you can use the following URL pattern:

http://data.digitalsocial.eu/resource?uri={resource-uri}

This is especially useful for resources for which we have information in our database, but which aren't in the data.digitalsocial.eu domain (i.e. so you can't dereference them in our site).

If using a format extension to request a particular format for the resource, the extension is added immediately after '/resource', for example to get a JSON-LD version.

http://data.digitalsocial.eu/resource.json?uri={resource-uri}

2.5 Collections of Resources

Collections of resources can be retrieved from /resources by supplying filters. For now, we just support filters for dataset and type_uri.

Filter parameter	Expected value	Behaviour
dataset	The short name of a dataset (see above).	Filters the results to only include resources in the named graph of that dataset.
type_uri	The URI of a resource type.	Filters the results to only include resources of the type identified by that URI.

e.g. http://data.digitalsocial.eu/resources?dataset=organizations-and-activities&type_uri=http%3A%2F%2Fwww.w3.org%2Fns%2Forg%23Organization

2.6 Options and Limits

2.6.1 Formats

Resources accessed via our resource APIs can be accessed in the same choice of formats as for URI dereferencing (via both format extensions or HTTP Accept headers).

2.6.2 Pagination

For any APIs which return collections of things, the list can be paginated using page (default 1) and per_page (default 1000) query-string parameters. The maximum allowable page size will initially be set to 1000, but we may consider increasing this (as well as the default) in the future.

2.6.3 Response Size Limits

All requests to our APIs are subject to the response size limits.

2.7 Example: Using Ruby to get a filtered list of resources

2.7.1 Basic Example

Here we use Ruby to retrieve a list of all the organization resources in the Organizations and Activities Dataset as N-triples.

The short name for that dataset is organizations-and-activities, and the URI for the organization type is http://www.w3.org/ns/org#Organization, so the URL we need to call is as follows. (See the Collections of Resources section).

http://data.digitalsocial.eu/resources?dataset=organizations-and-activities&type_uri=http%3A%2F%2Fwww.w3.org%2Fns%2Forg%23Organization

If you visit that URL in your browser, you'll see a paginated list of the resources. We want to get it in N-triples format, so we'll add the .nt extension. (See the Formats section).

The following Ruby code assigns a string of N-triples into the ntriples_data variable. Note that as the maximum page size is 1000, and there are over 1000 resoures of that type in the dataset, we'll need to make multiple requests.

We use the RestClient here, which you can install with $ gem install rest-client.

require 'rest-client'

url = "http://data.digitalsocial.eu/resources.nt"

ntriples_data = ""
page = 1
done = false

while !done
  puts "requesting page #{page}..."
  response = RestClient.get url, {:params =>
    {
      :page => page,
      :per_page => 1000,
      :dataset => "organizations-and-activities",
      :type_uri => "http://www.w3.org/ns/org#Organization"
    }
  }

  if response.length > 0
    ntriples_data += response
    page += 1
  else
    puts "no more data"
    done = true
  end
end

puts "data:"
puts ntriples_data

2.7.2 Extension: parsing the n-triples into an array of statements.

The ruby-rdf library is useful for parsing various rdf formats. Install it with $ gem install rdf. The following code reads our string of ntriples data into an array of RDF::Statements.

require 'rdf'

statements = []
RDF::Reader.for(:ntriples).new(ntriples_data) {|r| r.each {|s| statements << s}}

puts "parsed #{statements.length} triples"

Note: If you're doing a lot of work with RDF in Ruby, you might want to look at using Swirrl's open-source SPARQL ORM for Ruby, Tripod.

2.8 Example: Using JavaScript to get a filtered list of resources

Here we use jQuery to retrieve a list of all the organization resources in the Organizations and Activities Dataset as N-triples.

http://data.digitalsocial.eu/resources?dataset=organizations-and-activities&type_uri=http%3A%2F%2Fwww.w3.org%2Fns%2Forg%23Organization

If you visit that URL in your browser, you'll see a paginated list of the resources. We want to get it in N-triples format, so we'll add the .nt extension. (See the Formats section).

The following HTML page uses JavaScript to request the data as JSON and add it to the results array. Note that as the maximum page size is 1000, and there are over 1000 resoures of that type in the dataset, we'll need to make multiple requests.

<!DOCTYPE html>
<html>
  <head>
    <script src="http://code.jquery.com/jquery-1.9.1.min.js"></script>
  </head>
  <body>
    <script type="text/javascript">
      var perPage = 100;
      var typeUri = "http://www.w3.org/ns/org#Organization";
      var dataset = "organizations-and-activities";

      var baseUrl = "http://data.digitalsocial.eu/resources.json?"
      baseUrl += "per_page=" + perPage.toString();
      baseUrl += "&dataset=" + dataset;
      baseUrl += "&type_uri=" + encodeURIComponent(typeUri);

      var page = 1;
      var results = [];

      function callAjaxPaging() {
        console.log("trying page: " + page.toString());
        url = baseUrl + "&page=" + page.toString();

        $.ajax({
          dataType: 'json',
          url: url,
          success: function(pageOfData) {
            results = results.concat(pageOfData);
            console.log("got " + results.length.toString() + " so far");

            if (pageOfData.length == perPage) {
              // this page was full. There might be more.
              page += 1;
              console.log("trying next page");
              callAjaxPaging();
            } else {
              // no more pages.
              alert('finished with ' + results.length.toString() + " results");
            }
          }
        });
      }

      alert('press OK to begin');
      callAjaxPaging();
    </script>
  </body>
</html>

3 - SPARQL

3.1 Introduction to SPARQL

The most flexible way to access the data is by using SPARQL. Pronounced "sparkle", SPARQL stands for Sparql Protocol and RDF Query Language. It's a query language, analagous to SQL for relational databases, for retrieving and manipulating data from triple-stores like ours. We support SPARQL 1.1 query syntax.

To submit a SPARQL query from your code, issue an HTTP GET request to our endpoint:

http://data.digitalsocial.eu/sparql?query={URL-encoded query}

For example, to run this simple query...

SELECT * WHERE {?s a <http://www.w3.org/ns/org#Organization>}

...and get the results as JSON, you could GET the following URL (note the .json extension):

http://data.digitalsocial.eu/sparql.json?query=SELECT+%2A+WHERE+%7B%0D%0A%3Fs+a+%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Forg%23Organization%3E%7D

See below for details of SPARQL Results Formats and examples in a variety of languages (at the end of this section).

3.2 SPARQL Results formats

As with other aspects of our API, to get the data in different formats, you can use either format extensions or HTTP Accept headers.

The available formats depend on the type of SPARQL query. A SPARQL query can be one of four main forms: SELECT, ASK, CONSTRUCT or DESCRIBE.

Query Type	Format	Extension	Accept Headers
SELECT	xml	.xml	application/xml, application/sparql-results+xml
	json	.json	application/json, application/sparql-results+json
	text	.txt, .text	text/plain
	csv	.csv	text/csv
ASK	json	.json	application/json, application/sparql-results+json
	xml	.xml	application/xml, application/sparql-results+json
	text	.txt, .text	text/plain
CONSTRUCT	RDF/XML	.rdf	application/rdf+xml
	N-triples	.nt, .txt, .text	text/plain, application/n-triples
	Turtle	.ttl	text/turtle
DESCRIBE	RDF/XML	.rdf	application/rdf+xml
	N-triples	.nt, .txt, .text	text/plain, application/n-triples
	Turtle	.ttl	text/turtle

3.3 SPARQL Results Pagination

We will accept page and per_page query-string parameters for paginating the results of SELECT queries.

For requests made through the website (i.e. HTML format), the page size is defaulted to 20.

For requests to our sparql endpoint for data formats (i.e. non-HTML), there will be no defaults for these parameters (i.e. results are unlimited).

For SELECT queries, for convenience you can optionally pass the pagination parameters and we will use them to apply LIMIT and OFFSET clauses to the query. For other query types (i.e. DESCRIBE, CONSTRUCT, ASK), pagination like this doesn't make so much sense, so those parameters will be ignored.

Please also refer to the Response Size Limits section below, and the examples at the end of this section.

3.4 SPARQL Errors

If you make a SPARQL request with a malformed query in a data format (i.e. non-HTML), then we will respond with HTTP status 400, with a helpful message in the response.

Additionally, please note the Response Size Limits, which apply to all API calls, as well as the Errors section.

3.5 JSON-P

If you're requesting SPARQL results as JSON, you can additionally pass a callback parameter and the results will be wrapped in that function. This is useful for getting around cross-domain issues if you're running JavaScript on older browsers. (Please also see the Cross-Origin Resource Sharing section).

For example:

http://data.digitalsocial.eu/sparql.json?callback=myCallbackFunction&query=SELECT+%2A+WHERE+%7B%0D%0A%3Fs+a+%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Forg%23Organization%3E%7D

Or to make a JSON-P request with jQuery, you can omit the callback parameter from the url and just set the dataType to jsonp.

queryUrl = 'http://data.digitalsocial.eu/sparql.json?query=SELECT+%2A+WHERE+%7B%0D%0A%3Fs+a+%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Forg%23Organization%3E%7D

$.ajax({
  dataType: 'jsonp',
  url: queryUrl,
  success: function(data) {
    // callback code here
    alert('success!');
  }
});

3.6 Use of Named graphs

3.6.1 Dataset Data

The data for each dataset is contained within a separate named graph. The dataset itself has a URI, for example

http://data.digitalsocial.eu/data/organizations-and-activities

The web page for the dataset provides the named graph that contains the dataset, in this case

http://data.digitalsocial.eu/graph/organizations-and-activities

The graph name for the dataset is contained in the dataset metadata, using a predicate called http://publishmydata.com/def/dataset#graph and can be obtained by a query like this:

SELECT ?graph
  WHERE {
    <http://data.digitalsocial.eu/data/organizations-and-activities> <http://publishmydata.com/def/dataset#graph> ?graph
  }

3.6.2 Dataset Metadata

The metadata we store about the dataset itself (that is returned by dereferencing its URI), is stored its own separate graph, for example:

http://data.digitalsocial.eu/data/organizations-and-activities/metadata

We also use named graphs for each concept scheme and ontology.

3.7 Parameter Interpolation

You can parameterize your SPARQL by including %{tokens} in your queries, and providing values for the tokens on the url query string.

http://data.digitalsocial.eu/sparql?query=URL-encoded-SPARQL-query?token1=value-for-token1&token2=value-for-token2

3.8 Example: Using Ruby to request data from the SPARQL Endpoint

This sample Ruby makes a request to our SPARQL endpoint for all resources of type Unitary Authority (as JSON) and then puts the result in a Hash.

require 'rest-client'
require 'json'

query = 'SELECT * WHERE {?s a <http://www.w3.org/ns/org#Organization>}'

url = "http://data.digitalsocial.eu/sparql.json"

results_str = RestClient.get url, {:params => {:query => query}}
results_hash = JSON.parse results_str
results_array = results_hash["results"]["bindings"]

puts "total number of results: #{results_array.length}"

Note: If you're doing a lot of work with RDF in Ruby, you might want to look at using Swirrl's open-source SPARQL ORM for Ruby, Tripod.

3.8 Example: Using JavaScript to request data from the SPARQL Endpoint

This example HTML page uses jQuery to make a request to our SPARQL endpoint for all resources of type Unitary Authority (as JSON) and log the results.

<!DOCTYPE html>
<html>
  <head>
    <script src="http://code.jquery.com/jquery-1.9.1.min.js"></script>
  </head>
  <body>
    <script type="text/javascript">
      var query = "SELECT * WHERE {?s a <http://www.w3.org/ns/org#Organization>}";
      var url = "http://data.digitalsocial.eu/sparql.json?query=";
      url += encodeURIComponent(query);
      $.ajax({
        dataType: 'json',
        url: url,
        success: function(data) {
          alert('success: ' + data.results.bindings.length + ' results');
          console.log(data);
        }
      });
    </script>
  </body>
</html>

Note: See the Cross-Origin Resource Sharing section for a note about accessing data from from other domains.

4 - General

4.1 Response Size Limits

For all requests to our API, if the request issues a request to the database which causes more than 5MB of data to be returned, we will respond with HTTP status code 400, with the a message in the response body including the phrase Response too large. Note that full pre-canned dumps of all datasets will be available (in zipped n-triples format) at URLs defined in the dataset metadata.

4.2 Errors

Error type	HTTP status code	Notes
Response too large	400	We will include a text message in the response body including the phrase "Response too large."
SPARQL Syntax Error	400	We will include a text message in the response body with details of the error.
Resource Not Found	404	Returned if you request a resource or URL that doesn't exist
Not Acceptable	406	Returned if you request a non-supported data format
Unexpected Errors	500
Query Timeouts	503	The timeout for requesting data from our database will initially be set to 10 seconds.

4.3 Cross-Origin Resource Sharing (CORS)

Our web server is configured to allow access from all domains (by adding the following line to our nginx configuration):

add_header Access-Control-Allow-Origin "*";

This means that if you're writing JavaScript to request data from our server in to a web page hosted on another domain, your browser should check this header and allow it.

A Note about Browser Support for CORS:

Modern browsers (such as recent versions of Internet Explorer Firefox, Chrome and Safari) have full CORS support. It is not supported in Internet Explorer 6 and 7. Versions 8 & 9 of Internet Explorer offer limited support. If you need to support older browsers, consider making requests for data via SPARQL, with JSON-P.

This page is available as
HTML