Comments on: The unholy legacy of databases

By: Rickard

Rickard — Mon, 05 May 2008 08:02:34 +0000

Part of the point of splitting storage and query is that it becomes easier to do cross-storage queries. If you store objects in many places, but index/query them in one (again, the website vs Google analogy), it becomes supertrivial to query stuff in different places. The LDAP database problem you outline is one of the cases I had in mind when I designed these API’s in Qi4j, because I want to be able to do the same thing. In Qi4j our primary indexer is going to be Sesame2 (i.e. RDF), with SPARQL as the main query language (although it’s usually hidden under a domain-oriented Java API). Will be very interesting to see how it works out.

By: stephan

stephan — Sat, 03 May 2008 05:51:28 +0000

@Laird: Your query builder doesn’t sound too ugly, but I haven’t seen the code :-)

“I was hoping that someone somewhere smarter than me had figured out how to bridge querying disparate systems in a better way.”

Uh, smarter, than I’m most possibly not the right person.

Perhaps the joins are only needed for reporting, if that is the case it would be best to write the data also into a OLAP for reports.

Like http://mondrian.pentaho.org/

By: Laird Nelson

Laird Nelson — Fri, 02 May 2008 20:45:29 +0000

I surely don’t. :-) My main problem is that I need both–the ability to join LDAP information (Student information, say), as it happens, with additional information relevant to it from a database (classes they’re taking).

The decidedly brute-force, ugly, smelly, hairy, nasty and yet strangely cool approach I took at one point was to make a kind of query builder that, in conjunction with some simple lookup and filtering facilities implemented on Storage instances–but again, not true searches–whittled down the sets of items from each Storage to be combined (so the Student Storage was able to filter using a simple where clause/predicate, and the Class Storage was able to do the same). Then I loaded those sets into a temporary database (H2–obviously could have been anything) and did the more complicated joining there (minus the simple where clauses/filters that were used to get me the candidate sets). (A tip of my hat to a former colleague for first exploring this approach.) The result, of course, was dog slow and could not be used on enormous datasets, but performance was never a priority and the client understood that they would pay dearly in performance costs for this approach. I was hoping that someone somewhere smarter than me had figured out how to bridge querying disparate systems in a better way.

Thanks for the links to the papers; very interesting reading.

By: stephan

stephan — Fri, 02 May 2008 20:08:36 +0000

@Laird: I’m not sure if this is possible. You’re giving stuff up when you try this approach. But you also gain something. I guess it depends on the application you have. If flexibility in the backend is needed, than this is a good approach. If a RDBMS is all you need, then this approach is overengineered.

Have you read the transaction apostate paper and the Amazon dynamo paper? Sometimes it seems it isn’t even possible to have data on one machine to join it.

http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

http://www-db.cs.wisc.edu/cidr/cidr2007/papers/cidr07p15.pdf

But if you have new insights and a solution to the problem of data joining of disparate stores, please drop me a line.

By: Laird Nelson

Laird Nelson — Fri, 02 May 2008 18:18:20 +0000

I have tried this sort of thing as well and I really really like the concept. The only thing that stops my poor, slow, small brain from really seeing it through to its logical conclusion is the sometimes-requirement to join attributes from one Thing stored with one Storage mechanism with the attributes of another Thing stored with another Storage mechanism. Do you have suggestions here?

Obviously, a Compass/Lucene-type search handles a huge number of cases–people tend to like to search by keywords, and so a coarse-grained search/locate strategy like that makes a lot of sense. But in some of the applications I work on, careful targeted queries that join bits of two entities together–a classic SQL join–are also needed.

Have you found a convenient way to expose a *common* SQL-like query mechanism across items that use different Search implementations?