Introduction 1/2

Garlik is an online identity company. We are the online personal identity experts.
Our aim is to make our customers comfortable in a digital world.
By educating people about their digital identity
Founded by

Mike Harris, founding CEO of Egg plc
Egg CIO Tom Ilube
Professor Nigel Shadbolt, University of Southampton/Web Science/Data.gov.uk

With an Advisor Board Including:

Sir Tim Berners-Lee
Professor Dame Wendy Hall (President of the ACM)

Introduction 2/2

And a handful of Semantic Web Geeks

Steve Harris
Nick Lamb
Cezary Biernacki
Me :)

Funded by three of UK's leading blue chip Venture Capitalist firms

Encore/DFJ Esprit
Doughty Hanson
Noble Venture Finance

Introduction to this Talk

Garlik’s motivation, interest, and experiences of using Semantic Web (Linked Data) technologies in its online services
Methodologies and approaches that were taken in order to deploy online services to hundreds of thousands of users.
DataPatrol, a consumer-centric personal information protection product.
QDOS a Linked Data service that is used to measure peoples’ online activity.

Some Screenshots (1/3) ...

Some Screenshots (2/3) ...

Our Motivations for Developing/Employing Semantic Web Technologies (1/2)

Flexibility

In capturing, and indexing Personal Information, whether it be specific data sources or information collected from the World Wide Web/compromised data feeds
Garlik needs to be able to act in an agile manner, if new data is available we have to search/index it quickly

Scalability

Needed to meet the demands of a high-volume consumer facing services
Developed our own clustered RDF storage solutions : 4store and 5store

Our Motivations for Developing/Employing Semantic Web Technologies (2/2)

Allows for Data Centric: Ontology/Schema Driven Software Engineering (will come back to this)

Must be able to index/search over new data quickly
Garlik makes use the expressive power of RDF by implementing schema driven software solutions that allow for functionality to be added and removed by virtue of adding and removing triples from an RDF schema.

Use and Promotion of Standards
The adoption and promotion of Web Standards have always been high on Garlik’s agenda, this is evident by the decision to release and support SPARQL compliant 4store under the GPLv3 open-source license
Allows for trivial updates to backend platforms

We are W3C members

Steve is involved in the SPARQL 1.1 Working Group
I am involved in the Social Web Incubator Group

In short ...

Trust me, we are a Semantic Web Technology Company

We write Triplestores

4store
5store

We use Linked Data
We use SPARQL, HTTP, RDF over HTTP in our software architectures

Our give to the Semantic Web Technology Stack

Now to talk about 4store and 5store, before talking about applications of these products
Some notes about the two

Garlik’s RDF stores provide facilities over the HTTP protocol to allow access to information held within them.
Already mentioned the SPARQL protocol which allows queries to be run over HTTP, but in addition 4store and 5store allow data to be added and removed using the HTTP PUT and DELETE verbs
More info on: http://4store.org/trac/wiki/SparqlServer
HTTP allows us to use existing, widely implemented HTTP features for data management means that updates can be made by lightweight, asynchronous, loosely coupled parser processes (will come back to this...)
They are both Quad stores, and not Triple stores
The schema-less nature of our use of RDF allows these new data sources to be imported into the RDF store without changes to any schemas, or affecting the operation of existing processes. RDF namespacing is used to distinguish new types of data, so existing queries, designed before the inclusion of the new data items will not be presented with new results.

4store

Written in ANSI C99
In 2006, when Garlik was started, 4store had to store 1 billion triples (1GT), and answer queries in milliseconds
In order to minimise downtime, import speed was critical, so as to meet SLAs Garlik have with partners
Sustained import speed of seventy thousand triples per second (70kT/s) is required in order to allow the dataset to be restored from backup data in four hours.
Fault tolerance, data replication was designed from outset
Holds quad-graph in RAM, clustered, using avahi, it is not a front-end to a RDBMS
DataPatrol has ~15(GT), and until last year (June 2009), it was powered by 4store, 9 x (4 gig RAM x86-64 CentOS) machine cluster
Since going open-source, numerous bugs and have reported, and fixed.
GPLv3 source code repo : http://github.com/garlik/4store

4store's Network Topology

4store: http://4store.org/

4store, an efficient, scalable and stable RDF database
Run the Knowledge Bases that powers QDOS, and the foaf.qdos subdomain of QDOS in an 8 node cluster
For info/help come on irc://irc.freenode.net/#4store
Or if you don't do IRC, try our mailing list [email protected]
For tutorial on how to use 4store http://4store.org/presentations/bbc-2009-09-21/slides.html
Reading if you like symbols :Harris, S., Lamb, N., Shadbolt, N.: 4store: The design and implementation of a clustered RDF store. In: ISWC 2009 SSWS Workshop 09. Washington DC (2009)

5store

As 4store’s capacity was stretched at 15GTs, and the import performance, of around 150 kT/s was felt to be insufficient, we decided to build 5store
Extreme scalability in mind. It is designed to operate with clusters of many hundreds of machines, and approaches double the triples per machine density of 4store
Strengths are performance, scalability and stability in the domain of RDF storage and SPARQL queries.
NO RDFS reasoning, and NO REASONING ... (will come back to this)
As both 4store and 5store adhere closely to the SPARQL standards, upgrading is trivial
Compared to working with RDBMS system, upgrading is said to be alot easier, i.e. no need to write new code

5store

5store was designed from scratch for clusters of up to 1,000 machines.
The goal was to scale over 1 Trillion (10^12) triples
This should be easy achievable given a large enough cluster.
Given the new indexing algorithms deployed, there is no effective limit on the number of triples stored
Rather it is constrained by the practicalities of cluster size, network speed and so on
This is something we will be address very shortly
5store is implemented in ANSI C99 and C++. It makes use of custom discovery and clustering technologies and proprietary, novel indexing algorithms. It is primarily intended to run on 64-bit Linux platforms with high RAM density

5store

Import Performance and SLAs

Time taken to reimport data into a system is becoming even more important. 5store comfortably achieves sustained import speeds of a million triples per second on a 16 node cluster, allowing a ten billion triple store to be restored in under three hours.
Commodity lost cost hardware, will standard single SATA disks
Could be improved by using more disks

5store

Has been running our live production DataPatrol instance since June 2009
Information will *hopefully* be found in the "Linking Enterprise Data Book" due out in the next few months
Contact Steve or Mischa
The Stationary Office (TSO), who maintain and manage most of the http://data.gov.uk/ are the first people to have bought 5store TSO Press Release Regarding the Sale of 5store

What DataPatrol does for its customers

It helps by protecting its users from the following

DataPatrol's non-exclusive Partner Companies

We are currently operating in Germany, US, UK, and we are expanding ...
As well as: CPP Germany, Sovereign Bank, Geek Squad, Selftrade

DataPatrol Technologies used

5store
Java
Jetty
SPARQL
SPARQL Protocol
HTTP+RDF (turtle)

DataPatrol's Software Architecture Enables

Extensive Management Information (MI) capabilities, for our data providers
Based on Provenance information attached to all data via Named Graphs in quad-store

The Companies House Register
The Births, Deaths, and Marriages Register
Euro Direct Marketing Database
The Royal Mail Redirects Database (only third party to access)
The WWW
Our sources of compromised data (continuous updates)
No need to a MI system, as all data is quad-store is always indexed, no mysql related problems...
Some partners require:

Data to be deleted after a given duration
That we pay them per "hit" on their data-source

DataPatrol's Software Architecture Enables

Web Service based Architecture (use of RESTful APIs)

Take advantage of mature technology, http caches, http libraries
Allows for simple debugging
Staged deployment
No complicated test scripts, Object Models, horrible database abstraction technology
Easy/known entity for modern day developers
By passes connection-pooling issues, short lives connection to the backend store

Internally accessible Linked Data for CS team
Allows for debugging of customer related questions

DataPatrol's Software Architecture

QDOS

We all have a presence in the online world whether we use the internet or not and protecting yourself from identity fraud is just one side of the story
Our digital presence also increasingly opens up new opportunities and influences real world decisions made about us
Recent research has shown that more and more British citizens are making decisions based on digital status. Already 16% have chosen their new home based on how their prospective neighbours appear online. 1 in 5 (20%) have researched a prospective boss online before accepting a job and 32% have searched online to find out more about trades people and professionals, from plumbers to lawyers, before hiring them to do a job.
Garlik commissioned PCP market research consultants to conduct research among 2000 UK adults in November 2007

QDOS

More public example
We CAN NOT expose DataPatrol's internals due to privacy reasons
10 million FOAF files
And there will be more soon!
Linked Open Data App
Part of the Linked Data Cloud for a while now
Conneg, and Schema Driven Software Deployment (will come to this very shortly)
With a bunch of RESTFul Web-services on http://foaf.qdos.com/

QDOS's software Architecture

Schema Driven

Flexibility to deploy, with no service downtime - this makes us agile
This is used in DataPatrol for our compromised data-feeds
QDOS shows this in pseudo public manner
QDOS's backend "Score calculators" runs of this method
This is visible in the QDOS Schema
User data is visible here http://qdos.com/user/5acc361496df109a7c2967760d5d9792/rdfxml"
Data is visible here http://qdos.com/user/5acc361496df109a7c2967760d5d9792-ext/rdfxml

foaf.qdos.com

Making FOAF useful
Bi-directional distributed social graph
Linked Data needs indexes, and crawl based, distributed querying will not cut it
Public RESTful webservices which output RDF

FOAF : The Friend-Of-A-Friend Ontology

'The Friend of a Friend (FOAF) project began early in 2000 as an "experimental linked information project", created by Dan Brickley and Libby Miller.'
RDF : http://xmlns.com/foaf/0.1/
HTML : http://www.foaf-project.org/
FOAF makes it possible for people to build large scale, decentralised, open social networks that are controlled by individual users.
Sites such as : LiveJournal, hi5, vox, friendfeed?, google???,...
Reflectors : twitter, flickr, lastfm, delicious, ...

The FOAF Landscape as per Q1 2009

Foaf blob

Overview (a Steve-ism)

Viewer (1): http://foaf.qdos.com/

HTML view of a FOAF PersonalProfileDocument
Includes a list of friends
Includes a view of who claims to know the given foaf:Person
Caters for searching on FOAF IFPs (what is an IFP?)

Viewer (2): http://foaf.qdos.com/find/?q=http://planb.nicecupoftea.org/

Builder (1): http://foafbuilder.qdos.com/

RAP
AJAR(ish)
SPARQL
SPARQL-PROTOCOL to 4-store
A Foaf Editor which allows you to :
- Search for a FOAF file
- Build and store a FOAF file
- Edit their own existing FOAF file
- Manage privacy and permissions
- Provision of a service that allows people to search and browse millions of FOAF files in a user friendly way
- Edit / Store Personal Info in both a Public and Private Manner

Builder (2): http://foafbuilder.qdos.com/

Get your location via GUI

Builder (2): http://foafbuilder.qdos.com/

Write to WebDAV, save the file to your machine, ...

Builder (3): http://foafbuilder.qdos.com/

Get the RDF out of the FOAF builder

Private foaf.qdos.com http://private.qdos.com/

Get your own FOAF OAuth endpoint with your OpenID URI
Allows for private FOAF data to be stored and retrieved from an OAuth endpoint

Forward Search: http://foaf.qdos.com/forward/

Entering a unique identifier for a foaf:Person (i.e. email address, mbox sha1sum, homepage, weblog, or a FOAF URI) will return FOAF RDF listing foaf:Persons which the identifier claims to know.
Parameters : path and ifp
Example 1: http://foaf.qdos.com/forward?path=http://danbri.org/&ifp

<http://users.ecs.soton.ac.uk/nrs/foaf.rdf#me>
    a foaf:Person ;
    foaf:knows [
        a foaf:Person ;
        foaf:mbox_sha1sum "965c47c5a70db7407210cef6e4e6f5374a525c5c" ;
        foaf:name "Tim Berners-Lee"
    ], [
        a foaf:Person ;
        foaf:mbox_sha1sum "26a374ad4de937252c044f7dd45aa48ecbdb4f16" ;
        foaf:name "Steve Harris"
    ], [
        a foaf:Person ;
        foaf:mbox_sha1sum "28339a9a36ee4e330da9613958efb89e2961c6cb" ;
        foaf:name "Nick Gibbins"
    ], [
        a foaf:Person ;
        foaf:mbox_sha1sum "ca2d238d1ade1efb7128874e99e84fdef5a3c447" ;
        foaf:name "Tom Heath"
    ] .

Example 2: http://foaf.qdos.com/forward?path=http://moustaki.org/foaf.rdf%23moustaki

SameAs Search: http://foaf.qdos.com/sameas/

To get owl:sameAs triples about a given foaf:Person according to their FOAF presence enter their URI, email address, mbox sha1sum, homepage or blog URL above and click 'Go'.
Entering a unique identifier for a foaf:Person (i.e. email address, mbox sha1sum, homepage, weblog, or a FOAF URI) will return FOAF RDF listing the URIs we think are owl:sameAs.
Parameters : path and ifp
Example 1: http://foaf.qdos.com/sameas?path=http://danbri.org/&ifp

<http://danbri.org/foaf.rdf#danbri> a foaf:Person .
<http://danbri.org/foaf.rdf#danbri> owl:sameAs _:genid1 .
_:genid1 a foaf:Person .
_:genid1 foaf:homepage <http://danbri.org/> .
<http://foafbuilder.qdos.com/people/danbri.org/foaf.rdf#danbri> a foaf:Person .
<http://foafbuilder.qdos.com/people/danbri.org/foaf.rdf#danbri> owl:sameAs _:genid2 .
_:genid2 a foaf:Person .
_:genid2 foaf:homepage <http://danbri.org/> .
<http://www.advogato.org/person/danbri/foaf.rdf#me> a foaf:Person . 
<http://www.advogato.org/person/danbri/foaf.rdf#me> owl:sameAs _:genid3 .
_:genid3 a foaf:Person .
_:genid3 foaf:homepage <http://danbri.org/> .

Example 2: http://foaf.qdos.com/sameas?path=http://plugin.org.uk/swh.xrdf#me
Similar in vain to Hugh/Ian's http://sameas.org/

Reverse Search: http://foaf.qdos.com/reverse/

Entering a unique identifier for a foaf:Person (i.e. one of the IFPs email address, mbox sha1sum, homepage, weblog, or a FOAF URI) will return FOAF RDF listing the people which claim to know the identifier presented.
Parameters : path and ifp
Example 1: http://foaf.qdos.com/reverse?path=http://danbri.org/&ifp
Example 2: http://foaf.qdos.com/reverse?path=http://tomheath.com/id/me

<http://tomheath.com/id/me> a foaf:Person .

<http://nxg.me.uk/norman>
    a foaf:Person ;
    foaf:homepage <http://nxg.me.uk>, <http://www.astro.gla.ac.uk/users/norman/> ;
    foaf:knows <http://tomheath.com/id/me> ;
    foaf:mbox_sha1sum "0254b43a4c00a8c5eb9093d9820353640ee02786", "14f2d3a469303d0faca13f73064845ddda4d2dbd", 
     "58a587e433b1e0c22aa026eeee7703ffcee6262c","bfeee75c2025815392014a95cfd4b6aad117c07f", 
     "84612b3540a255bb11de3db579e02533ff2631c5", "baa162e14a20b3fafbccbf0fceaa1159a3a042d1",     
     "c9994fd3713561d0d9a468cd30c11a0f3cb3055a", "e6578555a8108ea83ee7f1d60556e63fd83bcc5e" ;
    foaf:name "Norman Gray" .

[]
    a foaf:Person ;
    foaf:homepage <http://kmi.open.ac.uk/people/mikele> ;
    foaf:knows <http://tomheath.com/id/me> ;
    foaf:mbox_sha1sum "97ddc048d1a7df5e6fcb8f530777896a395d3dc1" ;
    foaf:name "Michele Pasin" .

<http://www.aelius.com/njh#me>
    a foaf:Person ;
    foaf:homepage <http://www.aelius.com/njh/> ;
    foaf:knows <http://tomheath.com/id/me> ;
    foaf:mbox_sha1sum "3f361af12a473b48852819e3ff9153783149a26a", "a8476e8096db2bde20a734c8446bb3d60befea2b" ;
    foaf:name "Nicholas J Humfrey" .

So how is this useful..?

Try adding the following triples to your foaf file:

<#me> rdfs:seeAlso <http://foaf.qdos.com/forward?path=http://foo.com/blog&ifp>
<#me> rdfs:seeAlso <http://foaf.qdos.com/reverse?path=http://foo.com/blog&ifp>
<#me> rdfs:seeAlso <http://foaf.qdos.com/sameas?path=http://foo.com/blog&ifp>

OpenID Social Verifier (1): http://foaf.qdos.com/verify-about

The service is implemented as a RESTful HTTP service.
You send requests to the service with an identifier for you and the OpenID you wish to verify.
If the OpenID is within 2 people of you then the service returns YES, and the distance between you and them, 0 being an ID for you, and 2 being 2 people away, for example:
```
  <verify>
  <result degree="2">YES<result>
  <verify>
  
```
The endpoint of the REST service is http://foaf.qdos.com/verify and the arguments are action, qdosuri, ifp, ifpval and openid.
action Check or List

OpenID Social Verifier (2): http://foaf.qdos.com/verify-demo

Validator: http://foaf.qdos.com/validator/

Checks to whether a RDF document makes correct use of the FOAF ontology

Future Direction in the foaf.qdos.com world

Full Text Search over FOAF files
Grab all of it !
Server to Server updates : XMPP JQbus?
OAuth which Interacts with 4store code to be released
Write Data out of FOAF builder to any OAuth
More Robust Triangulation option
FOAF+SSL for sign-on
Continual Update
Release code snipbits for people to ping to the FOAF KB http://foaf.qdos.com/ping
Release stats on connectivity and richness of FOAF data from different sites
Find time and Write a paper :)

Conclusions

Industrial Strength Semantic Web Applications are NOT the future, we have them NOW
Practicality The capabilities of the SPARQL 1.0 query language are somewhat limited compared to typical SQL implementations. This often means that more processing work is required in the application layer, and can mean that more queries are required to answer the same question, compared to a SQL-based implementation. However, this is offset by the convenience of the high degree of standardisation between stores, and the high degree of flexibility in data representation.
HTTP+RDFAll of our services communicate using RDF, the consequences of this are that we achieve a high degree of flexibility with respect to staged deployments incorporating new capabilities.
Cost Effectiveness Despite the high engineering effort involved in developing two cluster-based RDF stores, we feel that overall this effort was well justified. The data volumes that can be held by 5store in particular are similar to those held in Key-Value stores, but yet we have access to a standard, and relatively sophisticated query language that supports very efficient join operations. On cheap commodity hardware
Reliability and Uptime There is inevitably some concern when deploying software on top of a newly developed database system, however we have been operating our production system on 4store and 5store for several years now, while retaining high uptime, and without data loss. Engineering of the storage system for robustness, and the use of redundant clusters has helped in this area, and we do not feel that we suffered any more downtime that if we had gone for a more traditional RDBMS.

Garlik and Enterprise Semantic Web Technologies

Presentation Overview

Introduction 1/2

Introduction 2/2

Introduction to this Talk

Some Screenshots (1/3) ...

Some Screenshots (2/3) ...

Some Screenshots (2/3) ...

Our Motivations for Developing/Employing Semantic Web Technologies (1/2)

Our Motivations for Developing/Employing Semantic Web Technologies (2/2)

In short ...

Our give to the Semantic Web Technology Stack

4store

4store's Network Topology

4store: http://4store.org/

5store

5store

5store

5store

What DataPatrol does for its customers

DataPatrol's non-exclusive Partner Companies

DataPatrol Technologies used

DataPatrol's Software Architecture Enables

DataPatrol's Software Architecture Enables

DataPatrol's Software Architecture

QDOS

QDOS

QDOS's software Architecture

Schema Driven

foaf.qdos.com

FOAF : The Friend-Of-A-Friend Ontology

...

The FOAF Landscape as per Q1 2009

Overview (a Steve-ism)

Viewer (1): http://foaf.qdos.com/

Viewer (2): http://foaf.qdos.com/find/?q=http://planb.nicecupoftea.org/

Builder (1): http://foafbuilder.qdos.com/

Builder (2): http://foafbuilder.qdos.com/

Builder (2): http://foafbuilder.qdos.com/

Builder (3): http://foafbuilder.qdos.com/

Private foaf.qdos.com http://private.qdos.com/

Forward Search: http://foaf.qdos.com/forward/

SameAs Search: http://foaf.qdos.com/sameas/

Reverse Search: http://foaf.qdos.com/reverse/

So how is this useful..?

OpenID Social Verifier (1): http://foaf.qdos.com/verify-about

OpenID Social Verifier (2): http://foaf.qdos.com/verify-demo

Validator: http://foaf.qdos.com/validator/

Future Direction in the foaf.qdos.com world

Conclusions