Presentation Overview

• Introduction - Who are Garlik?
• Introduction to Talk
• Garlik and the Semantic Web: DataPatrol and QDOS
• 4store
• 5store
• DataPatrol
• QDOS
• foaf.qdos

Introduction 1/2

• Garlik is an online identity company. We are the online personal identity experts.
• Our aim is to make our customers comfortable in a digital world.
• By educating people about their digital identity
• Founded by
• Mike Harris, founding CEO of Egg plc
• Egg CIO Tom Ilube
• Professor Nigel Shadbolt, University of Southampton/Web Science/Data.gov.uk
• With an Advisor Board Including:
• Sir Tim Berners-Lee
• Professor Dame Wendy Hall (President of the ACM)
• ...

Introduction 2/2

• And a handful of Semantic Web Geeks
• Steve Harris (CTO)
• Nick Lamb
• Cezary Biernacki
• Monika Stepinska
• Vaidas Jablonskis
• Me :)
• Funded by three of UK's leading blue chip Venture Capitalist firms
• Encore/DFJ Esprit
• Doughty Hanson
• Noble Venture Finance

Introduction to this Talk

• Garlik’s motivation, interest, and experiences of using Semantic Web (Linked Data) technologies in its online services
• Methodologies and approaches that were taken in order to deploy online services to hundreds of thousands of users.
• DataPatrol, a consumer-centric personal information protection product.
• QDOS a Linked Data service that is used to measure peoples’ online activity.

Some Screenshots (1/3) ...

Some Screenshots (2/3) ...

Some Screenshots (2/3) ...

Our Motivations for Developing/Employing Semantic Web Technologies (1/2)

• Flexibility
• In capturing, and indexing Personal Information, whether it be specific data sources or information collected from the World Wide Web/compromised data feeds
• Garlik needs to be able to act in an agile manner, if new data is available we have to search/index it quickly
• Scalability
• Needed to meet the demands of a high-volume consumer facing services
• Developed our own clustered RDF storage solutions : 4store and 5store
• ...

Our Motivations for Developing/Employing Semantic Web Technologies (2/2)

• Allows for Data Centric: Ontology/Schema Driven Software Engineering (will come back to this)
• Must be able to index/search over new data quickly
• Garlik makes use the expressive power of RDF by implementing schema driven software solutions that allow for functionality to be added and removed by virtue of adding and removing triples from an RDF schema.
• Use and Promotion of Standards
• The adoption and promotion of Web Standards have always been high on Garlik’s agenda, this is evident by the decision to release and support SPARQL compliant 4store under the GPLv3 open-source license
• Allows for trivial updates to backend platforms
• We are W3C members
• Steve is involved in the SPARQL 1.1 Working Group
• Was involved in the Social Web Incubator Group
• And am involved in the WebID Incubator Group
• And both Steve and I will be on the RDF Next Steps WG

In short ...

• Trust me, we are a Semantic Web Technology Company

• We write Triplestores
• 4store
• 5store
• We use Linked Data
• We use SPARQL, HTTP, RDF over HTTP in our software architectures

Our give to the Semantic Web Technology Stack

• Now to talk about 4store and 5store, before talking about applications of these products
• Some notes about the two
• Garlik’s RDF stores provide facilities over the HTTP protocol to allow access to information held within them.
• Already mentioned the SPARQL protocol which allows queries to be run over HTTP, but in addition 4store and 5store allow data to be added and removed using the HTTP PUT and DELETE verbs
• More info on: http://4store.org/trac/wiki/SparqlServer
• HTTP allows us to use existing, widely implemented HTTP features for data management means that updates can be made by lightweight, asynchronous, loosely coupled parser processes (will come back to this...)
• They are both Quad stores, and not Triple stores
• The schema-less nature of our use of RDF allows these new data sources to be imported into the RDF store without changes to any schemas, or affecting the operation of existing processes. RDF namespacing is used to distinguish new types of data, so existing queries, designed before the inclusion of the new data items will not be presented with new results.

4store

• Written in ANSI C99
• In 2006, when Garlik was started, 4store had to store 1 billion triples (1GT), and answer queries in milliseconds
• In order to minimise downtime, import speed was critical, so as to meet SLAs Garlik have with partners
• Sustained import speed of seventy thousand triples per second (70kT/s) is required in order to allow the dataset to be restored from backup data in four hours.
• Fault tolerance, data replication was designed from outset
• Holds quad-graph in RAM, clustered, using avahi, it is not a front-end to a RDBMS
• DataPatrol has ~15(GT), and until last year (June 2009), it was powered by 4store, 9 x (4 gig RAM x86-64 CentOS) machine cluster
• Since going open-source, numerous bugs and have reported, and fixed.
• GPLv3 source code repo : http://github.com/garlik/4store

4store's Network Topology

4store: http://4store.org/

• 4store, an efficient, scalable and stable RDF database
• Run the Knowledge Bases that powers QDOS, and the foaf.qdos subdomain of QDOS in an 8 node cluster
• For info/help come on irc://irc.freenode.net/#4store
• Or if you don't do IRC, try our mailing list [email protected]
• For tutorial on how to use 4store http://4store.org/presentations/bbc-2009-09-21/slides.html
• Reading if you like symbols :Harris, S., Lamb, N., Shadbolt, N.: 4store: The design and implementation of a clustered RDF store. In: ISWC 2009 SSWS Workshop 09. Washington DC (2009)

5store

• As 4store’s capacity was stretched at 15GTs, and the import performance, of around 150 kT/s was felt to be insufficient, we decided to build 5store
• Extreme scalability in mind. It is designed to operate with clusters of many hundreds of machines, and approaches double the triples per machine density of 4store
• Strengths are performance, scalability and stability in the domain of RDF storage and SPARQL queries.
• NO RDFS reasoning, and NO REASONING ... (will come back to this)
• As both 4store and 5store adhere closely to the SPARQL standards, upgrading is trivial
• Compared to working with RDBMS system, upgrading is said to be alot easier, i.e. no need to write new code

5store

• 5store was designed from scratch for clusters of up to 1,000 machines.
• The goal was to scale over 1 Trillion (10^12) triples
• This should be easy achievable given a large enough cluster.
• Given the new indexing algorithms deployed, there is no effective limit on the number of triples stored
• Rather it is constrained by the practicalities of cluster size, network speed and so on
• This is something we will be address very shortly
• 5store is implemented in ANSI C99 and C++. It makes use of custom discovery and clustering technologies and proprietary, novel indexing algorithms. It is primarily intended to run on 64-bit Linux platforms with high RAM density

5store

• Import Performance and SLAs
• Time taken to reimport data into a system is becoming even more important. 5store comfortably achieves sustained import speeds of a million triples per second on a 16 node cluster, allowing a ten billion triple store to be restored in under three hours.
• Commodity lost cost hardware, will standard single SATA disks
• Could be improved by using more disks

5store

• Has been running our live production DataPatrol instance since June 2009
• Information will *hopefully* be found in the "Linking Enterprise Data Book" due out in the next few months
• Contact Steve or Mischa
• The Stationary Office (TSO), who maintain and manage most of the http://data.gov.uk/ are the first people to have bought 5store TSO Press Release Regarding the Sale of 5store

What DataPatrol does for its customers

• It helps by protecting its users from the following
• 

DataPatrol's non-exclusive Partner Companies

• We are currently operating in Germany, US, UK, and we are expanding ...
• 
• As well as: CPP Germany, Sovereign Bank, Geek Squad, Selftrade, T Mobile, Direct Line, a Chrome App

DataPatrol Technologies used

• 5store
• Java
• Jetty
• SPARQL
• SPARQL Protocol
• HTTP+RDF (turtle)

DataPatrol's Software Architecture Enables

• Extensive Management Information (MI) capabilities, for our data providers
• Based on Provenance information attached to all data via Named Graphs in quad-store
• The Companies House Register
• The Births, Deaths, and Marriages Register
• Euro Direct Marketing Database
• The Royal Mail Redirects Database (only third party to access)
• The WWW
• Our sources of compromised data (continuous updates)
• No need to a MI system, as all data is quad-store is always indexed, no mysql related problems...
• Some partners require:
• Data to be deleted after a given duration
• That we pay them per "hit" on their data-source

DataPatrol's Software Architecture Enables

• Web Service based Architecture (use of RESTful APIs)
• Take advantage of mature technology, http caches, http libraries
• Allows for simple debugging
• Staged deployment
• No complicated test scripts, Object Models, horrible database abstraction technology
• Easy/known entity for modern day developers
• By passes connection-pooling issues, short lives connection to the backend store
• Internally accessible Linked Data for CS team
• Allows for debugging of customer related questions

DataPatrol's Software Architecture

QDOS

• We all have a presence in the online world whether we use the internet or not and protecting yourself from identity fraud is just one side of the story
• Our digital presence also increasingly opens up new opportunities and influences real world decisions made about us
• Recent research has shown that more and more British citizens are making decisions based on digital status. Already 16% have chosen their new home based on how their prospective neighbours appear online. 1 in 5 (20%) have researched a prospective boss online before accepting a job and 32% have searched online to find out more about trades people and professionals, from plumbers to lawyers, before hiring them to do a job.
• Garlik commissioned PCP market research consultants to conduct research among 2000 UK adults in November 2007

QDOS

• More public example
• We CAN NOT expose DataPatrol's internals due to privacy reasons
• 10 million FOAF files
• And there will be more soon!
• Linked Open Data App
• Part of the Linked Data Cloud for a while now
• Conneg, and Schema Driven Software Deployment (will come to this very shortly)
• With a bunch of RESTFul Web-services on http://foaf.qdos.com/

QDOS's software Architecture

Schema Driven

• Flexibility to deploy, with no service downtime - this makes us agile
• This is used in DataPatrol for our compromised data-feeds
• QDOS shows this in pseudo public manner
• QDOS's backend "Score calculators" runs of this method
• This is visible in the QDOS Schema
• User data is visible here http://qdos.com/user/5acc361496df109a7c2967760d5d9792/rdfxml"
• Data is visible here http://qdos.com/user/5acc361496df109a7c2967760d5d9792-ext/rdfxml

foaf.qdos.com

• Making FOAF useful
• Bi-directional distributed social graph
• Linked Data needs indexes, and crawl based, distributed querying will not cut it
• Public RESTful webservices which output RDF

FOAF : The Friend-Of-A-Friend Ontology

• 'The Friend of a Friend (FOAF) project began early in 2000 as an "experimental linked information project", created by Dan Brickley and Libby Miller.'
• RDF : http://xmlns.com/foaf/0.1/
• HTML : http://www.foaf-project.org/
• FOAF makes it possible for people to build large scale, decentralised, open social networks that are controlled by individual users.
• Sites such as : LiveJournal, hi5, vox, friendfeed?, google???,...
• Reflectors : twitter, flickr, lastfm, delicious, ...

...

The FOAF Landscape as per Q1 2009

Overview (a Steve-ism)

Viewer (1): http://foaf.qdos.com/

• HTML view of a FOAF PersonalProfileDocument
• Includes a list of friends
• Includes a view of who claims to know the given foaf:Person
• Caters for searching on FOAF IFPs (what is an IFP?)

• foaf:homepage
• foaf:mbox_sha1sum
• foaf:jabberID
• foaf:isPrimaryTopicOf
• foaf:weblog
• ...

Viewer (2): http://foaf.qdos.com/find/?q=http://planb.nicecupoftea.org/

Builder (1): http://foafbuilder.qdos.com/

• RAP
• AJAR(ish)
• SPARQL
• SPARQL-PROTOCOL to 4-store
• A Foaf Editor which allows you to :
  • Search for a FOAF file
  • Build and store a FOAF file
  • Edit their own existing FOAF file
  • Manage privacy and permissions
  • Provision of a service that allows people to search and browse millions of FOAF files in a user friendly way
  • Edit / Store Personal Info in both a Public and Private Manner

Builder (2): http://foafbuilder.qdos.com/

• Get your location via GUI

Builder (2): http://foafbuilder.qdos.com/

• Write to WebDAV, save the file to your machine, ...

Builder (3): http://foafbuilder.qdos.com/

• Get the RDF out of the FOAF builder

Private foaf.qdos.com http://private.qdos.com/

• Get your own FOAF OAuth endpoint with your OpenID URI
• Allows for private FOAF data to be stored and retrieved from an OAuth endpoint

Forward Search: http://foaf.qdos.com/forward/

• Entering a unique identifier for a foaf:Person (i.e. email address, mbox sha1sum, homepage, weblog, or a FOAF URI) will return FOAF RDF listing foaf:Persons which the identifier claims to know.
• Parameters : path and ifp
• Example 1: http://foaf.qdos.com/forward?path=http://danbri.org/&ifp

• <http://users.ecs.soton.ac.uk/nrs/foaf.rdf#me>
      a foaf:Person ;
      foaf:knows [
          a foaf:Person ;
          foaf:mbox_sha1sum "965c47c5a70db7407210cef6e4e6f5374a525c5c" ;
          foaf:name "Tim Berners-Lee"
      ], [
          a foaf:Person ;
          foaf:mbox_sha1sum "26a374ad4de937252c044f7dd45aa48ecbdb4f16" ;
          foaf:name "Steve Harris"
      ], [
          a foaf:Person ;
          foaf:mbox_sha1sum "28339a9a36ee4e330da9613958efb89e2961c6cb" ;
          foaf:name "Nick Gibbins"
      ], [
          a foaf:Person ;
          foaf:mbox_sha1sum "ca2d238d1ade1efb7128874e99e84fdef5a3c447" ;
          foaf:name "Tom Heath"
      ] .
  

• Example 2: http://foaf.qdos.com/forward?path=http://moustaki.org/foaf.rdf%23moustaki

SameAs Search: http://foaf.qdos.com/sameas/

• To get owl:sameAs triples about a given foaf:Person according to their FOAF presence enter their URI, email address, mbox sha1sum, homepage or blog URL above and click 'Go'.
• Entering a unique identifier for a foaf:Person (i.e. email address, mbox sha1sum, homepage, weblog, or a FOAF URI) will return FOAF RDF listing the URIs we think are owl:sameAs.
• Parameters : path and ifp
• Example 1: http://foaf.qdos.com/sameas?path=http://danbri.org/&ifp

• <http://danbri.org/foaf.rdf#danbri> a foaf:Person .
  <http://danbri.org/foaf.rdf#danbri> owl:sameAs _:genid1 .
  _:genid1 a foaf:Person .
  _:genid1 foaf:homepage <http://danbri.org/> .
  <http://foafbuilder.qdos.com/people/danbri.org/foaf.rdf#danbri> a foaf:Person .
  <http://foafbuilder.qdos.com/people/danbri.org/foaf.rdf#danbri> owl:sameAs _:genid2 .
  _:genid2 a foaf:Person .
  _:genid2 foaf:homepage <http://danbri.org/> .
  <http://www.advogato.org/person/danbri/foaf.rdf#me> a foaf:Person . 
  <http://www.advogato.org/person/danbri/foaf.rdf#me> owl:sameAs _:genid3 .
  _:genid3 a foaf:Person .
  _:genid3 foaf:homepage <http://danbri.org/> .
  

• Example 2: http://foaf.qdos.com/sameas?path=http://plugin.org.uk/swh.xrdf#me
• Similar in vain to Hugh/Ian's http://sameas.org/

Reverse Search: http://foaf.qdos.com/reverse/

• Entering a unique identifier for a foaf:Person (i.e. one of the IFPs email address, mbox sha1sum, homepage, weblog, or a FOAF URI) will return FOAF RDF listing the people which claim to know the identifier presented.
• Parameters : path and ifp
• Example 1: http://foaf.qdos.com/reverse?path=http://danbri.org/&ifp
• Example 2: http://foaf.qdos.com/reverse?path=http://tomheath.com/id/me

<http://tomheath.com/id/me> a foaf:Person .

<http://nxg.me.uk/norman>
    a foaf:Person ;
    foaf:homepage <http://nxg.me.uk>, <http://www.astro.gla.ac.uk/users/norman/> ;
    foaf:knows <http://tomheath.com/id/me> ;
    foaf:mbox_sha1sum "0254b43a4c00a8c5eb9093d9820353640ee02786", "14f2d3a469303d0faca13f73064845ddda4d2dbd", 
     "58a587e433b1e0c22aa026eeee7703ffcee6262c","bfeee75c2025815392014a95cfd4b6aad117c07f", 
     "84612b3540a255bb11de3db579e02533ff2631c5", "baa162e14a20b3fafbccbf0fceaa1159a3a042d1",     
     "c9994fd3713561d0d9a468cd30c11a0f3cb3055a", "e6578555a8108ea83ee7f1d60556e63fd83bcc5e" ;
    foaf:name "Norman Gray" .

[]
    a foaf:Person ;
    foaf:homepage <http://kmi.open.ac.uk/people/mikele> ;
    foaf:knows <http://tomheath.com/id/me> ;
    foaf:mbox_sha1sum "97ddc048d1a7df5e6fcb8f530777896a395d3dc1" ;
    foaf:name "Michele Pasin" .

<http://www.aelius.com/njh#me>
    a foaf:Person ;
    foaf:homepage <http://www.aelius.com/njh/> ;
    foaf:knows <http://tomheath.com/id/me> ;
    foaf:mbox_sha1sum "3f361af12a473b48852819e3ff9153783149a26a", "a8476e8096db2bde20a734c8446bb3d60befea2b" ;
    foaf:name "Nicholas J Humfrey" .

So how is this useful..?

• Try adding the following triples to your foaf file:

<#me> rdfs:seeAlso <http://foaf.qdos.com/forward?path=http://foo.com/blog&ifp>
<#me> rdfs:seeAlso <http://foaf.qdos.com/reverse?path=http://foo.com/blog&ifp>
<#me> rdfs:seeAlso <http://foaf.qdos.com/sameas?path=http://foo.com/blog&ifp>
   

OpenID Social Verifier (1): http://foaf.qdos.com/verify-about

• The service is implemented as a RESTful HTTP service.
• You send requests to the service with an identifier for you and the OpenID you wish to verify.
• If the OpenID is within 2 people of you then the service returns YES, and the distance between you and them, 0 being an ID for you, and 2 being 2 people away, for example:

    <verify>
    <result degree="2">YES<result>
    <verify>
    

• The endpoint of the REST service is http://foaf.qdos.com/verify and the arguments are action, qdosuri, ifp, ifpval and openid.
• action Check or List

OpenID Social Verifier (2): http://foaf.qdos.com/verify-demo

Validator: http://foaf.qdos.com/validator/

• Checks to whether a RDF document makes correct use of the FOAF ontology

Future Direction in the foaf.qdos.com world

• Full Text Search over FOAF files
• Grab all of it !
• Server to Server updates : XMPP JQbus?
• OAuth which Interacts with 4store code to be released
• Write Data out of FOAF builder to any OAuth
• More Robust Triangulation option
• FOAF+SSL for sign-on
• Continual Update
• Release code snipbits for people to ping to the FOAF KB http://foaf.qdos.com/ping
• Release stats on connectivity and richness of FOAF data from different sites
• Find time and Write a paper :)

Conclusions

• Industrial Strength Semantic Web Applications are NOT the future, we have them NOW
• Practicality The capabilities of the SPARQL 1.0 query language are somewhat limited compared to typical SQL implementations. This often means that more processing work is required in the application layer, and can mean that more queries are required to answer the same question, compared to a SQL-based implementation. However, this is offset by the convenience of the high degree of standardisation between stores, and the high degree of flexibility in data representation.
• HTTP+RDFAll of our services communicate using RDF, the consequences of this are that we achieve a high degree of flexibility with respect to staged deployments incorporating new capabilities.
• Cost Effectiveness Despite the high engineering effort involved in developing two cluster-based RDF stores, we feel that overall this effort was well justified. The data volumes that can be held by 5store in particular are similar to those held in Key-Value stores, but yet we have access to a standard, and relatively sophisticated query language that supports very efficient join operations. On cheap commodity hardware
• Reliability and Uptime There is inevitably some concern when deploying software on top of a newly developed database system, however we have been operating our production system on 4store and 5store for several years now, while retaining high uptime, and without data loss. Engineering of the storage system for robustness, and the use of redundant clusters has helped in this area, and we do not feel that we suffered any more downtime that if we had gone for a more traditional RDBMS.