Wilted buttercup, grey skies, and geek

I was attempting to install 4store on my fedora 12 box, and I found that I couldn’t run the tests which are shipped with 4store. I got the following output when attempting to run “make test” :


[root@blanket 4store]# make test
(cd tests && make -w test)
make[1]: Entering directory `/usr/local/src/4store/tests'
(cd query && pwd && ./setup.sh --autorun)
/usr/local/src/4store/tests/query
4store[9702]: backend-setup.c:176 erased files for KB query_test_mmt04r
4store[9702]: backend-setup.c:301 created RDF metadata for KB query_test_mmt04r
../../src/frontend/4s-import: error while loading shared libraries: librasqal.so.1: cannot open shared object file: No such file or directory
Preparing for tests...
../../src/frontend/4s-delete-model: error while loading shared libraries: librasqal.so.1: cannot open shared object file: No such file or directory
[FAIL] add-and-delete
[FAIL] distinct-predicate
[FAIL] foaf-all-limit
[FAIL] foaf-bnode-vs-variable
[FAIL] foaf-construct
[FAIL] foaf-disjunctive-filter
[FAIL] foaf-distinct
[FAIL] foaf-graph-all
[FAIL] foaf-graph-pred
[FAIL] foaf-knows-name
[FAIL] foaf-knows-name-sha1
[FAIL] foaf-knows-sha1
[FAIL] foaf-knows-sha1-xml
[FAIL] foaf-multi-disjunctive-filter
[FAIL] foaf-nested-optional
[FAIL] foaf-nothing
[FAIL] foaf-optional-order
[FAIL] foaf-optional-pair
[FAIL] foaf-optional-regex
[FAIL] foaf-repeat-var
[FAIL] graphs
[PASS] integrity
[FAIL] null-optional
[FAIL] null-optional-double
[FAIL] optimiser-disjunction
[FAIL] select-bnodes
[FAIL] select-order
[FAIL] select-unused
[FAIL] size
[FAIL] tiger-broadway
[FAIL] tiger-explosion
[FAIL] tiger-fail-optional
[FAIL] tiger-harold-ave
[FAIL] tiger-landmarks
[FAIL] tiger-mixed-optional
[FAIL] tiger-reverse
[FAIL] tiger-sugar-hill
[FAIL] tiger-sugar-hill-filter
[FAIL] tiger-typical
[FAIL] tiger-water-names
Tests completed: passed 1/40 (39 fails)
make[1]: Leaving directory `/usr/local/src/4store/tests'


This struck me as odd given that when I ran the “configure” script I got the following output :

[root@blanket 4store]# ./configure
[OK ] pkg-config installed
[OK ] raptor installed
[OK ] rasqal installed
[OK ] glib2 installed
[OK ] libxml2 installed
[OK ] pcre installed
[OK ] ncurses installed
[OK ] readline installed
[OK ] z installed
[OK ] avahi installed


So these are the setup step which I had to perform in order to get 4store working and running the tests:

1. After compiling the code, used ldd to see which libs were missing

ldd src/frontend/4s-query
[root@blanket 4store]# ldd !$2
ldd src/frontend/4s-query
linux-gate.so.1 => (0x00d9a000)
librasqal.so.1 => not found
libraptor.so.1 => /usr/lib/libraptor.so.1 (0x00678000)
libxml2.so.2 => /usr/lib/libxml2.so.2 (0x04f2f000)
libavahi-common.so.3 => /usr/lib/libavahi-common.so.3 (0x051d0000)
libavahi-client.so.3 => /usr/lib/libavahi-client.so.3 (0x051bd000)
libavahi-glib.so.1 => /usr/lib/libavahi-glib.so.1 (0x004b8000)
libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0x00da9000)
libpcre.so.0 => /lib/libpcre.so.0 (0x00110000)
....


2. Fix my ld.so configuration. The issue was that I had to add the following line, which points to my librasqal.so.1, to the following empty file
/etc/ld.so.conf.d/local.conf :

/usr/local/lib


A big yay to working 4store :

[root@blanket 4store]# make test
(cd tests && make -w test)
make[1]: Entering directory `/usr/local/src/4store/tests'
(cd query && pwd && ./setup.sh --autorun)
/usr/local/src/4store/tests/query
4store[10887]: backend-setup.c:176 erased files for KB query_test_mmt04r
4store[10887]: backend-setup.c:301 created RDF metadata for KB query_test_mmt04r
removing old data
Reading
into
Pass 1, processed 63 triples (63)
Reading
into
Pass 1, processed 764610 triples (764547)
Reading
into
Pass 1, processed 764658 triples (48)
Pass 2, processed 764658 triples, 26769 triples/s
Updating index
Index update took 17.133335 seconds
Imported 764658 triples, average 19375 triples/s
Preparing for tests...
[PASS] add-and-delete
[PASS] count
[PASS] distinct-predicate
[PASS] foaf-all-limit
[PASS] foaf-bnode-vs-variable
[PASS] foaf-construct
[PASS] foaf-disjunctive-filter
[PASS] foaf-distinct
[PASS] foaf-graph-all
[PASS] foaf-graph-pred
[PASS] foaf-knows-name
[PASS] foaf-knows-name-sha1
[PASS] foaf-knows-sha1
[PASS] foaf-knows-sha1-xml
[PASS] foaf-multi-disjunctive-filter
[PASS] foaf-nested-optional
[PASS] foaf-nothing
[PASS] foaf-optional-order
[PASS] foaf-optional-pair
[PASS] foaf-optional-regex
[PASS] foaf-repeat-var
[PASS] graphs
[PASS] integrity
[PASS] null-optional
[PASS] null-optional-double
[PASS] optimiser-disjunction
[PASS] select-bnodes
[PASS] select-order
[PASS] select-unused
[PASS] size
[PASS] tiger-broadway
[PASS] tiger-explosion
[PASS] tiger-fail-optional
[PASS] tiger-harold-ave
[PASS] tiger-landmarks
[PASS] tiger-mixed-optional
[PASS] tiger-reverse
[PASS] tiger-sugar-hill
[PASS] tiger-sugar-hill-filter
[PASS] tiger-typical
[PASS] tiger-water-names
[PASS] union-nobind
Tests completed: passed 42/42 (0 fails)
make[1]: Leaving directory `/usr/local/src/4store/tests'


Hello All,

I had one of my hard-drives fail miserably a couple of weeks ago, and have only now recovered all of my blog content from Google’s and Yahoo’s caches. I have recreated all of the content, and have set up a cronjob to backup my sql tables, I had copies of most of the important stuff on my machine, and only lost my sql tables … d’oh….

I have set the following 301 redirects from all of the old URLs, I don’t think I have missed any out. Do shout if you find any old broken URLs on my site.

redirect 301 /blog/2009/04/02/barcamp-09/ /blog/2009/07/07/barcamp-09/
redirect 301 /blog/2009/03/24/time-machine-to-a-linux-box/ /blog/2009/07/07/timemachine-to-a-linux-box/
redirect 301 /blog/2009/03/23/signing-a-public-key/ /blog/2009/07/08/signing-someone’s-public-key/
redirect 301 /blog/2009/03/20/making-foaf-useful/ /blog/2009/07/08/making-foaf-useful/
redirect 301 /blog/2009/03/19/yum_64bit_binaries/ /blog/2009/07/07/configuring-yum-to-only-install-64bit-binaries/
redirect 301 /blog/2009/03/19/webdav_webid/ /blog/2009/07/08/enabling-a-writable-webid-with-webdav/
redirect 301 /blog/2009/03/18/foafwot/ /blog/2009/07/08/enabling-trust-in-a-foaf-document/
redirect 301 /blog/2009/03/17/signingverifyinggpg/ /blog/2009/07/08/signing-and-verifying-documents-using-gnupg/
redirect 301 /blog/2009/02/05/ah-good-work-tobyink/ /blog/2009/07/07/ah-good-work-tobyink-…/
redirect 301 /blog/2009/01/28/duplicate-dylib-libiconv2/ /blog/2009/07/07/duplicate-dylib-libiconv-2-dylib/
redirect 301 /blog/2008/12/31/ld-duplicate-symbol-mac-osx/ /blog/2009/07/07/ld-duplicate-symbol-_g_bit_nth_lsf-mac-osx-leoparddarwin/


I have put together these notes on installing 3Store on linux boxes. I should thank Dave Dupplaw for his notes on installation, this blog is also a good point of call.

Pre-Install

Before you install 3Store you must ensure that you have raptor and rasqal libraries installed. You should install these from source! This is not a necessity but what we have found
it best after a number of installations.

Rasqal and Raptor are both libraries produced by Redland, and can be downloaded from their site: http://librdf.org.

Unpack each tar ball, configure, make, and make install each one.

Rasqal and Raptor both install their package files into /usr/local/lib/pkgconfigg but 3Store does not expect the files there. To fix this set the environment variable PKG_CONFIG_PATH to point to that directory. If that directory does not exist, locate rascal.pc to find its location and place that path in your PKG_CONFIG_PATH environment variable. For example, on a 64 bit architecture, these files should be found in /usr/local/lib64/pkgconfig.

In [t]csh this is:

setenv PKG_CONFIG_PATH /usr/local/lib/pkgconfig/

In bash this is:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/

It should be noted that 3store also requires glib, as well as the Berkeley DB libraries. You should be able to install these via your package manager, i.e. apt-get, yum, or through emerge.

3Store Installation

Installation is from source, so unpack the source and enter the 3store directory.

Use ./configure to configure the compilation scripts.

Make the scripts with the make command. Install 3Store with make install (you will have to be root to do this).

Troubleshooting

Package rascal.pc does not exist

If you get this error:

checking for RASQAL… Package rasqal was not found in the pkg-config
search path.

Perhaps you should add the directory containing rasqal.pc to the

PKG_CONFIG_PATH environment variable.

Other issue:

No package ‘rasqal’ found configure: error: Package requirements (rasqal >= 0.9.11) were not met:

Then check that you have correctly installed rascal and set up the PKG_CONFIG_PATH environment variable.

Library libidn.so does not exist

If you get this error:

gcc: /usr/lib/libidn.so: No such file or directory

Then locate libidn and make a soft link to the necessary library.
For example, if libidn.so.11 exists, link it to the filename libidn.so:

ln -s /usr/lib/libidn.so.11 /usr/lib/libidn.so

The make script will then be able to find the file.

glibc 386 libraries missing RedHat

If you are working on x86 platform, you may still require the i386 libraries to do
the compilation. This error may be typified by the following error message during compilation:

/usr/bin/ld: crti.o: No such file: No such file or directory

It is easiest if you are signed up to the RedHat network to do update your glibc libraries to the latest versions:

up2date --showall # (optional to find latest versions)

up2date --get glibc-devel-2.3.4-2.19.i386

rpm -Uvh /var/spool/up2date/glibc-devel-2.3.4-2.19.i386.rpm

Alternatively, you can check your original discs for the i386 glibc-devel library rpm. RedHat Enterprise Linux comes with the i386 versions of the libraries on disc 3. Install them using the last command given above.

Wrong Architecture/File Format

There is a bug in the RedHat Enterprise installer that means even if you tell if not to install MySQL it will do. Unfortunately, in either case it will install the 32 bit version, which causes an error similar to this:

/usr/local/lib/librasqal.so: could not read symbols: File in wrong format

If this is the case, you need to ensure that the compilation knows where your 64-bit MySQL installation is. If you do not have one you will have to download an appropriate version (or build from source).

The installation works by calling the mysql-config script which tells the compilation how to compile-in the MySQL libraries. To make it work, ensure that the script in your path is the correct version:

In [t]csh:

setenv PATH /usr/local/mysql/bin:${PATH}

Or in bash:

export PATH=/usr/local/mysql/bin:$PATH

Libraries not found

When you run a 3store tool (e.g. ts-query) you get a message similar to the following:

ts-query: error while loading shared libraries: lib3store.so.0: cannot open shared object file: No such file or directory

This means the 3Store tool cannot find the 3Store libraries. You can locate lib3store to find the location of the libraries (which by default are put into /usr/local/lib) and add this to your LD_LIBRARY_PATH:

In [t]csh:

setenv LD_LIBRARY_PATH /usr/local/lib

Or in bash:

export LD_LIBRARY_PATH=/usr/local/lib

Better still, you should add this path to your ld configuration so that it will always
work. To do this create a new file in /etc/ld.so.conf.d/ called 3store.conf. Put /usr/local/lib on its own at the top of the file. Save it, and run /sbin/ldconfig, to update your configuration. The tools should now be able to find the libraries, and will be able to for anyone who uses the machine.

This is fixed in another manner if you are using Gentoo Linux. There is no /etc/ld.so.conf.d/ directory. In gentoo you should add the path to the
LDPATH environment variable (not LD_LIBRARY_PATH!), this can be done like so:

Edit /etc/env.d/00basic

And add the LIBDIR to the line:

LDPATH="/usr/local/lib"

On a 64 bit architecture it may look like so:

LDPATH="/usr/local/lib:/usr/local/lib64"

Then make sure you run the env-update script in gentoo.

Berkeley DB library not found

So if you get the following error during the configure process.

“configure: error: Cannot find Berkeley DB library version 4″

You either don’t have the Berkeley DB libraries installed, or there are not where 3store expects them to be.

If you don’t have it installed, grab it and cd to the directory, then:

cd build_unix
../dist/configure

make
sudo make install

So now, if you are in the position where you have Berkeley DB 4 installed, but you are still getting the above error then 3Store is assuming it is somewhere which it isn’t! So now we have to make a soft link from where the db.h file actually resides, and where 3store thinks it should be.

su
mkdir /usr/include/db4
cd /usr/include/db4
ln -s /usr/local/BerkeleyDB.4.5/include/db.h
./configure LDFLAGS=-L/usr/local/BerkeleyDB.4.5/lib

If you are using a version 4.5 or above of Berkeley DB, you make have to edit the configure script as follows:

Edit line 21151, which looks like so:

for ac_lib in db-4.4 db-4.3 db-4.2 db-4.1 db; do

and add your version of db to it, like so:

for ac_lib in db-4.5 db-4.4 db-4.3 db-4.2 db-4.1 db; do

And now you should be able to make and make install.

Problems during compilation “make”

If you had problems with the configure script and Berkeley DB, then you may have to edit some on the source code in the src directory. In order to get it to work I have had to:

Change the source code and which had #include <db.h> to #include<PATHTO/db.h>

This had to be done in the following files:

Datatypes.h

Bulkimport.c

Problems with ts-util!

If you are having problems when running the command:

ts-util optimise "KBNAME"

You will have to edit line number 164 in ts-util.c in the src directory, and recompile.

Line 164 is missing an “AND”, and should look like so

ts_mysql_query(c->db, "DELETE FROM "T_S" WHERE NOT EXISTS (SELECT * FROM "T_T" WHERE model=hash) AND NOT EXISTS (SELECT * FROM "T_T" WHERE subject=hash) AND NOT EXISTS (SELECT * FROM "T_T" WHERE predicate=hash) AND NOT EXISTS (SELECT * FROM "T_T" WHERE object=hash)");