COWDB2 release 0.1

o Bug fix: Transaction Abort is now working.
o ForkBranch is now working, which means that all utilities (except RunScript and Browse) have now been implemented and are working.

COWDB2 release 0.0.7

o Adjacent free and available blocks are now being consolidated. This solves a disk fragmentation problem and reduced the footprint on disk significantly.
o Validate was raising bogus db corruption errors--fixed.

COWDB2 release 0.0.6

o The DeleteBranch utility is now working.
o Disk Space Fragmentation has been addressed.

COWDB2 release 0.0.5

o ListBranches, CreateBranch, ListOpenBranches, and CloseBranch utilities are now working.
o ChangeControlEvent and FreeEvent have been added to the object model.
o Control classes have been refactored.
o Convenience methods added to CObject and the code base updated to use them, making the code easier to read and write.

Your input is requested

I have just revised the home page for AgileWiki at http://agilewiki.wiki.sourceforge.net/ and very much desire any suggestions you may have. I believe that a clear and compelling value proposition is important if we are to continue to grow the development team.

Thanks in advance.

Bill

COWDB2 release 0.0.4

Validate is now working.

When initializing a database, the trunk branch is now created.

Aspect factories now take a fully qualified class name when instantiating a capability. As a result, COWDB2 is now fully extensible.

change attribute events

For a very long time now I have not been happy with attributes. On the one hand, they have been very dumb and on the other hand they can effect major changes. In COODBMS, only the admin was allowed to set attributes arbitrarily--the mechanism in place was completely unsafe.

So my thought is to create a change attribute even and send it to the appropriate element aspects, which can then raise an exception if there is a problem, so it can be secured and yet made available to users as a fast way to make changes to an element.

Again, events look like a great way for the aspects of an element to coordinate.

going off line for PC repair

Taking my thinkpad in for repair. The display no longer works and, while I can work using a monitor, I'm thinking I should get it fixed.

Not sure how long I will be off line. :-(

COWDB2 release 0.0.3

* Unblock was not updating disk.
* UnblockUpdates, listOpenDbs and closeDb utilities are now working.

disk space management

In COWDB2 we should be able to implement a much more scalable and fragmentation resistant memory management system.

First, we can round up allocation requests to the nearest Fibonacci number. This will prevent disk space fragmentation.

Second, we can take a 2-tier approach to space management. The first tier will be a system which is embedded in the root element itself. It will not be scalable and will only be used for managing the space used by the second tier. At issue here is that the root element must not grow beyond an anticipated size.

The second tier will [eventually] be a manager which uses a b-tree. Space for the elements of the b-tree will be managed by the first tier manager, while the second tier manager handles everything else. By using a b-tree for managing space in the second tier manager, we achieve the desired scalability.

COWDB2 release 0.0.2

The following utilities are now working:
- tracebackLoggingOn,
- tracebackLoggingOff,
- listDbs and
- blockUpdates.

Aspect Programming with fine-grained Aggregation and IOC

I am simply delighted with the way the new object model is progressing. More powerful and flexible than anticipated. At some point I expect to make much more extensive use of element aggregates in the framework but for now I think we will make do with implementing all the transactions as elements.

I've ported over the low-level database logic and have integrated the role element property file with the database configuration property file. So there is some progress besides the work being done on the object model. Still not much is working, but what does work is being tested regularly.

One thing which has frustrated me for years is trying to map rolonic structures into a single class heirarchy. At a minimum you need mixins. COWDB's simple aggregation wasn't used to address this--not enough aspects had been broken out. But aggregation in COWDB2 is much more fine-grained and really works nicely.

You know you are doing something right when...

In the Rolonics world view, everything has for major aspects: wholeness, partness, structure and stream. Now these are the four packages under the elements package in the COWDB2 object model. Seems fitting.

I've always wondered why I could not implement a Rolonics database in a Rolonic fashion. I've tried for years to incorporate Rolonics into the implementation. Finally, it looks like I'm getting there. I just seemed to me that if Rolonics was so all-encompassing, I should be able to use it no matter what I was programming. The turning point was in implementing elements as aggregates, where each of the objects in the aggregate was for a different aspect of the element.

Now the exciting thing about all this is that the functionality of the elements divides up so nicely into different aspects, where normally aspects are all interwoven through the code base. This is just another indication to me that there is something very very right Rolonics.

COWDB2 release 0.0.1

The new COWDB2 server can now [only] process shutdown requests.

a 2-pronged approach

Made some good progress on the object model for COWDB2, but still there is a long way to go.

I also started working on migrating the COODBMS server code over to COWDB2. One very fine thing here is that we will have only one object model, rather than the 2 in COWDB, as the new object model supports both persistent and non-persistent elements. And that removes a major learning hurdle.

Now as I'm spreading myself so thin, progress now will slow even more. But while I'm trying to write nice clear code for the new object model, the framework is being ported "as is." So it will go a bit more quickly. But once I get a bit further I'll start testing with the "old" Browse program. So again things will slow. But that's all fine, so long as COWDB2 comes out of the development process in a form that is easier to understand.

As the (past) Director of Engineering at Fiorano, I was always saying that working code is simply not good enough. So I've talked the talk and now I'm walking the walk. But I am also hopeful that the new object model will be interesting in its own right. I must admit, I've not done much with aggregation before. But then I've never had to deal with anything quite so flexible.

A good start on COWDB2

It is the beginning of monsoon season here. We get rains in the afternoon and evening. And like every year in Raipur, the rains bring frequent and repeated power outages. Fortunately this condition doesn't last for too long. After a while the electric company gets things under control again and power becomes stable. Indeed, one of the advantages of living in Raipur is that the state capital has a surplus of electricity. It's only a temporary distribution problem.

I have however made a good start on COWDB2 today, though there is a very long road ahead. It is fun though, rationalizing the API. You can follow my progress at http://agilewiki.wiki.sourceforge.net/COWDB2

COODBMS, a qualified success

I count COODBMS as an overall success. It includes a significant number of AgileWiki3's capabilities, despite having a very different implementation. It is significantly more robust than AgileWiki3 and solved the problem of having an ever growing footprint on disk--AgileWiki3 had no equivalent to COODBMS's PurgeHistory. Still it is a partial implementation of AgileWiki3 and there is much to be done if we were to continue this implementation.

The failure of COODBMS is a general lack of clarity. This is partly because it was a layered implementation which did not anticipate the all of the requirements. Partly it is because the implementation is so very different from AgileWiki3. And partly because an important capability--time navigation--was added as an afterthought. Considering the size of the effort remaining, I believe it best to begin again with COWDB2, though hopefully there will be many large chunks of code that can easily be moved to the new implementation.

The rewrite will have a very similar overall architecture, with only minor changes. Where I expect to have the most changes are in the object model. The old implementation had 2 object models in the server and a third in the client. And as the implementation progressed, it was the object model which suffered the most. Otherwise I expect most of the ideas and techniques to carry over, all be it as a cleaner and clearer implementation.

As released, COWDB release 4.1 and COODBMS release 0.19.1 are fully functional and have no known bugs. In that reguard I can only hope that COWDB2 does as well.

COODBMS release 0.19.1

There are no known bugs as of this release, which includes fixes for caching problems, memory leaks and navigation of past time. Multi-threaded stress testing has also been completed successfully.

COWDB release 4.1

Dirty cache issues and memory leaks have now been addressed. There are no known bugs.

COODBMS cleans up nicely

After that last run, I ran purge history. Here's the result:

Branch Index Count = 123
Well Known Table Count = 4
Links Table Count = 38
Backlinks Table Count = 38
Transaction Index Count = 22

Available blocks = 40.
Blocks in use=556.
Internal nodes=20.

So it cleans up very nicely.

COODBMS is working

I increased the dirty cache size from 750 to 4096 and the memory leaks vanished. I'll note that the new algo for managing the dirty cache does require a larger cache size.

Branch Index Count = 79
Well Known Table Count = 702
Links Table Count = 1412
Backlinks Table Count = 1412
Transaction Index Count = 349

Available blocks: 1056.
Blocks in use: 43,613.
Internal nodes: 102.

one step forward, two back

The bug in dirty cache is fixed. I set the max entries down to 64 to add further stress on the system and the scripts ran longer but still I am getting a memory leak.

At the time of the memory leak, the number of blocks in use is 20,576, and 47 internal nodes. but the number of available blocks is 405. No problem here, but my concern is that in production the number of available blocks would be very much higher--and the table of available blocks is kept in a single block. Transaction count is 154 (I was not running purge history).

Here's the script I was running, where each run is processed as a single transaction:

//exercise removeWellknownName
createWellknown a1
go /a1
removeWellknownName

//removing a link
createWellknown a1
createWellknown b1
go /a1
cu
addOrderedSection l
go l
addLink b /b1
go /b1
go /a1/CU/l
remove b
go /b1
removeWellknownName
go /a1
removeWellknownName

//adopting a child
createWellknown a1
createWellknown b1
go /a1
adoptChild b /b1
go b
removeWellknownName
go /a1
removeWellknownName

//fork a child
createWellknown a1
createWellknown b1
go /a1
adoptFork b /b1
removeWellknownName
go /b1
removeWellknownName

//set well known name
createWellknown a1
go /a1
createChild b
go b
setWellknownName b1
go /a1
removeWellknownName
go /b1
removeWellknownName

//well known fork
createWellknown a1
go /a1
createChild b
go b
wellknownFork b1
go /a1
removeWellknownName
go /b1
removeWellknownName

//reorder parents
createWellknown a1
createWellknown b1
go /a1
createChild ac
go /b1
adoptChild bc /a1/ac
go bc
moveParentFirst /b1
go /b1
removeWellknownName
go /a1/ac
go /a1
removeWellknownName

//change child name
createWellknown a1
go /a1
createChild b
changeChildName b c
go c
go /a1
removeWellknownName

//change key
createWellknown a1
go /a1
lu
addOrderedSection x
addOrderedSection y
changeKey x z
go /a1
removeWellknownName

//change Classifier Section name
createWellknown a1
createWellknown b1
go /a1
cu
addOrderedSection l
go l
addLink b /b1
cu
changeKey l links
go /b1
removeWellknownName
go /a1
removeWellknownName

wellknown

shaking down nicely

COODBMS runs consistently until it is about 3 Mb in size. Then the caching problem nails it every time. But that was long enough a run time to complete some nice multi-threading stress tests, which it passed with flying colors.

So there is still the odd bug or 3, but we are getting close to a production release. We also need one final feature--load/dump to external files. This will be essential for migration to new releases as the internal structures evolve.

COODBMS release 0.19

o Bug found/fixed: validate throwing spurious dup allocation exceptions.
o Bug found/fixed: validate was closing the database.
o Bugs found/fixed: the the remove methods of sorted and sorted iterator had memory leaks.
o The first argument of adoptChild and adoptFork are now optional.
o Bug found/fixed: the well known table was occasionally deleting itself.

COWDB release 4

o Database validation is greatly enhanced.
o Disk space management of branches has been fixed.
o A pending table has been added for tracking space which has been freed in one branch but not in another.

things are going well

With release 0.18.9, the only significant known error remaining is a memory leak. The work done these last two days was not easy, but putting an end to those pending errors was a great step forward.

I'll note that memory leaks only occur when there are internal b-tree nodes, so that gives me a great starting point. And the memory leaks are easy enough to force--just let the database grow and they start happening. So it should only be a matter of investing the time/labor to find the leak. And as other errors are not occurring, it is easy now to focus on this one bug. The downside, of course, is that finding memory leaks is never a fun process. Sigh.

COODBMS release 0.18.9

o Reworked how freed blocks are processed--the code was not working when there were multiple branches/snapshots. This fixes all known pending errors.
o A validation table was added to prevent pending errors from corrupting the database. Free validates against this table.
o Small bugs fixed relating to syntax error handling and time navigation.

finally caught my tail!

I've been chasing my tail all day trying to track down pending and dup allocation errors. I did narrow it down to a chained fork issue. Then after a few more turns I finally realized the flaw in the logic.

So I expect finally to make some progress tomorrow. Today has been... "educational." I always say, if you pound your head against the wall long enough and hard enough, eventually something will give way. Fortunately I'm a tad hard headed. (A trait which is not always appreciated!) :-)

back from Delhi, Ruby

Back from a trip to Delhi, and a bit tired. I did manage to read a book on Delhi and I like the language, especially blocks. I am thinking that display logic can be written in Ruby (or Python), stored in DSecs in the database and accessed as needed. This will keep the front end thin and application independent.

But first there are a few bugs that need fixing.

Bill