wrapping up old issues

http://agilewiki.wiki.sourceforge.net/CowDb

Release cowdb0.8.10 is out and symbolic links are now fully functional. There are also programming guides now for symbolic links, well known names and deep copies.

pathnames, well known names, well known table

The next release (cowdb0.8.9) has been cleaned up quite a bit and some new concepts have been included.

First Class Objects -- These are objects which are under an index, though the system index is also a first class object and all indexes are first class objects. Only a first class object supports the fork operation.

Well known name -- Only first class objects can have a well known name. As forked objects can not change name (because the name is held by the fork object), a well known name provides an alternative which can be changed.

Well known table -- Every index has a well known table by which first class objects can be located using their well known name. The well known table of the system index then holds the well known names of the indexes of all the top-level branches. I.E. The top-level well known table replaces the old branch table but now there is symetry at every level.

Pathnames -- Every object now has a pathname by which it can be located. However, all the top-level branch indexes (the branch roots) have the same pathname: "/". Pathnames can be absolute (start with "/") or relative. However ".." notation is not supported, as pathname resolution is aggressive. Also when a pathname for an object is constructed, well known names (where available) are given preference over ordinary object names.

The display logic has also been enhanced to display each object's pathname.

Backlinks have still not been implemented--that's a release all by itself.

backlinks--taking cowdb to the next level

In the current release of cowdb, we take the "What me worry?" approach to symbolic links--they are the responsibilit of the application programmer to maintain. And this can be a significant source of complexity--especially when manual maintenance is allowed, and multi-level deep copies are supported. So we need to automate things a bit more. Especially when we start supporting pathname references.

A large table of backlinks maintained at the top-level branch will provide the necessary information to automate the maintenance of symbolic links. And as any Wiki user knows, backlinks are useful to have around.

But note now the divergence from a more traditional OODBMS. CowDb already has built-in support for large hashes and large tables. And yes, we are taking a small step here towards a Rolonic database. But only a very small step.

B-Trees and a User Guide

Release cowdb0.8.7 is out, and it includes b-trees and range iterators. I've also added a basic user guide to the Wiki. Check it out at
http://agilewiki.wiki.sourceforge.net/CowDb

Oh, and here's the updated feature list:

• Light Weight A small memory footprint, but with a configurable cache size.
• Simple Operation Each database has a single disk file. No log files; no before image files.
• Highly robust Fully transactional. Recovery is fast and fully automated.
• Branches (virtual deep copies) can be done at any level.
• Extensible Hashes for fast access to large collections.
• B-Trees for ordered access to large collections. Includes range iterators.
• Long Running Queries An advanced threading model allows long running queries to continue on one branch while updates execute in other branches in the same database.
• Extensible Almost every object used in the implementation has an interface. And the factory pattern is used throughout. New application objects can be defined by subclassing PersistentObject (and extending the factory) or objects can be stored in a container for Serializable POJOs. Byte arrays are also supported.
• Framework Free Injection Of Control (IOC) is used in place of a complex framework. This significantly reduces the complexity as well as making it easier to understand.
• Conditional Compression Large objects are compressed; small objects are not.
• Common Context Object methods and methods both operate within the context of an index, so the same programming style can be used.
• Navigatable Context A transaction can easily migrate from one context to another by selecting another index.
• Symbolic Links Both direct references (disk offset) and symbolic links (named access via an index) are supported.

Finally! N-Level Branching

Only now have I finally implemented n-level branching (i.e. deep copies)--and it was a lot harder than I had thought, at each and every step of the way.

Found a lot of bugs in the process, so overall it was a good thing. And the latest version of the API (updated to SVN but not yet released) is quite interesting as transactions can navigate from index to index (using the setIndex method). Also, if you want to abort a transaction, you just throw the AbortException.

So what's next? Looks like it is time to implement b-trees. I expect this will not take anywhere near the time it took to implement n-level branching!

Turtles all the way down (or is it up?)

In today's release of cowdb0.8.4, persistent objects now all have names. Turns out, this simplifies things, as branches and forks already had names. It also lays some interesting groundwork for making cowdb much easier to work with, aside from providing the basis for symbolic references (via handles) and deep copies. But the first thing we need to do is to introduce some new terminology--yet to be implemented.

The most important term in a COW DB is index, for it is only via an index that structures other than trees can be built. What was the fork table will become the primary index, and top will become the secondary index. All persistent objects will have access (via control or via an association with the current thread using a weak reference hash) to their index, and all logical blocks will be accessible via their index using their name as the key. (Logical blocks being persistent objects which reside in a physical block on disk, while other persistent objects must make do with residing in containers which themselves are logical blocks.)

Notice now that we have introduced the idea of levels, where system objects are at the first level and other objects are at higher levels. Branches then will be in the primary index along with the fork objects which manage disk space allocation for a branch. And with careful symmetry then, we can have branches at any level--which gives us deep copies.

So now that I have everything working so well (0.8.4 was primarily a bug release), it is time to dig in and do a bunch of refactoring. Cheers!

release 0.8.2: LargeHash

Release 0.8.2 is out, and it is looking great. Speedup has been better than 10X. And with large hash tables being supported, CowDB is now actually useful for larger applications.

priorities

Cowdb now supports handle tables, which have half the disk footprint of persistent tables. Also, PersistentTable and HandleTable now support HashMap as an alternative to TreeMap.

I've started to move from constructor-based IOC to method-based. The code will be cleaner, and this way we can more easily implements cloning of persistent objects. Cloning can then be used to reduce the need for disk reads:

1. When a writable object is needed, it can be copied from the read-only version of the same object.
2. When writing to disk, we can also save a read-only copy for use by subsequent queries & transactions.

I'm still interested in deep copies--this should be a very noteworthy feature. But I fear it will overload the fork table. So next on the list should be support for large hashes.

After deep copies, a b-tree implementation should wrap everything up nicely. Then it's on to phase 2: a Rolonic database.

Feature tracking can be found here: http://sourceforge.net/tracker/?group_id=106672&atid=645208

deep copies

Release 8.1 has been posted, the transactional API has been revised, the new threading model has been put in place and tests have been devised for exercising the new threading model. Time to move on to the next item.

I've been thinking a lot about deep copies, which should be very quick on a COW database. It should only be a matter of refactoring some of the existing code which handles the forking of a branch.

Deep copies are something you want to be able to do in any case in an object store, only the Copy-On-Write technology should make it very efficient. And it seems like a very important feature--how else can you implement undo/redo?

Updated CowDb documentation

See CowDb for an updated description of the COW Db.