in Bangalore

Flew to Bangalore yesterday. Had to change terminal buildings in Mumbai. 33C and humid.

Manged to check my email this morning and post this, but that's about it. Should be back on line on Thursday.

COODMS release 0.18

o Internal Rolon names now include a timestamp.
o Snapshots are created when a Rolon is updated. These are linked to by the transaction Rolon.
o New snapshots command allows navigation to the snapshots of a Rolon.
o Various fixes and enhancements made to wellknown and move parent commands.
o Includes the cowdb3.4.3 fix for forks.

CowDb release 3.4.3

Bug found/fixed in the fork logic.

A Great Day

Small change of plans--I'm flying out to Bangalore tomorrow. But I should still be back on Thursday.

Found some bugs relating to wellknown and moveParent. Old code was destroying the links and backlinks tables. Fixed.

Right now I'm ready for a release but having internet problems. BSNL broadband is down. (I doubt that anyone is in the office on a Sunday.) and TATA Indicom is incompatible with editing wiki pages right now. Probably the server is overloaded and can't handle the SSL with all the BSNL traffic coming over. Sigh. So no release right now.

I am very happy with the progress. In about 2 weeks we should have the old AgileWiki3 time navigation capabilities back. Still there is a critical bug somewhere relating to forks/snapshots.

big day, back Thursday

Lots of heavy lifting today. COODBMS release 0.18 has come a long way, but it still is not ready for release.

I did find a design flaw today in the fork logic (keeping an old reference long after it was invalid) that was causing intermittent errors. Fixed the flaw and now the problems occur much less often. :-(

As I may have said before, I'm going to Bangalore tonight and should be back in Raipur and online on Thursday. Wish me luck!

good day, bad ending, travel plans

It was a good day. Basic snapshots are done. Timetravel a la AgileWiki3 will be handled in 18.1.

The day had a bad ending. Getting memory leaks. But not so bad as they are very reproducible. I'll dig into it tomorrow.

And my much delayed trip back to Bangalore is back on. Finally got confirmed tickets to Bangalore. (Return tickets are not quite confirmed yet, but I guess we'll just manage.) Leaving tomorrow night. It is summer here now, so who wants to travel? Everyone it seems. It is also summer vacation for the kids. (Summer comes before monsoon. Monsoon comes when we're having summer in the US. I keep thinking it must be August because everyone says it is summer and it sure feels like August to me!)

Called for service on my thinkpad today. Got a nice clear connection and was told that this was NOT the service number. Called the service number twice and both times the connection was so bad I could only hang up. So I sent an email.

Oh yes, I should be back on line on Thursday, or later if there is a problem with those return tickets. :-)

Journal

In this last release the user journal now is not used so heavily--it will now only reflect the changes made to the account and other account-related activity. To see the activity of a user, go to their account and enter the command rbacklinks USER. This will list all the transactions submitted by that user.

The journal of a rolon still reflects the specific changes made to it. In addition, the command rbacklinks JSECS will list all the transactions which have changed it.

The next step will be to modify the [internal] name of all rolons. We just need to append a '9' to the existing names. The Journal's snapshots will have a timestamp appended to the name (in place of the '9'), causing them to occur before the original (timestamps beginning yyyy).

Each transaction rolon will now have a link to the snapshots created. Snapshots will be "before images", as this avoids problems with the purge history utility.

Link resolution is now a bit more complicated. You need to extract the timestamp for the time being navigated and append it (in place of that '9' or timestamp) to the name of the rolon in the link being accessed. A match will be greater or equal, but must have the same first 36 characters (the UUID).

We will need a new command to view the snapshots of a rolon. You can then select a past rolon and follow the links to view the state of the branch at the time the snapshot was made. Backlinks should also work for past time.

It is all going to take some time, to program and debug. Release 0.17.2 will likely be the last release for a few days.

COODBMS release 0.17.2

release 0.17.2:

• New home command navigates to the user's account.
• The rbacklinks command now lists the links in a nice order.
• New rlinks and linkCSecs commands work like rbacklinks and backlinkCSecs.
• Browse now uses main.cowdb's trunk as the default branch. The section Completing the Instalation of COODBMS has been updated as well.
• JSecs are no longer added to the user account when the user runs a transaction, nor is there now a JSECS link to the user account. Rather, there is only a single USER link in the transaction Rolon. This significantly reduces the size of the user's JUnit.

Note that the next release will not be backward compatible.

why not keep the database stupid?

Dmitri raised an interesting point--why not keep the application logic out of the database? Until now I'd been thinking of COODBMS as an OODBMS, so ofcourse I was thinking that the application logic should be in the database. But what if we make COODBMS dumb backend and put application processes in front of it?

The advantage of having a single process is speed, of course. But that is not always the most pressing concern. Having application processors means we can spread the system over multiple machines for greater scalability. It also means that application bugs will be [largely] confined to the application server--a good thing.

Now for Aye, we need to add notifications so that Aye can maintain its own cache. Why not then focus on notifications and a cache implementation next?

COODBMS release 0.17.1

o Scripts containing only query commands are run as a query, increasing throughput on busy servers.
o Comments are now supported and preserved in the transaction Rolon.
o The rbacklinks command now displays the Rolons containing the link to the selected Rolon as well as having an optional argument for selecting backlinks by CSec name.
o New command, backlinkCSecs, identifies the Classifier Sections containing links to the selected Rolon.

COODBMS release 0.17

* The createWellknown bug has been fixed.
* The runScript utility now works with any branch in any database.
* The static.cowdb trunk branch now holds command descriptions, which is now used by Browser. A script is provided for loading this information.
* The check register sample application now has a script for defining its commands.

http://agilewiki.wiki.sourceforge.net/CoodbmsReleases

CowDb release 3.4.2

Methods PElement.addPathnameReference was calling MapApp.makeSymbolic, which does not work for wellknown names due to the side effect of deleting the handle in the wellknown table when a wellknown name is removed. These methods now call addSymbolic and the makeSymbolic methods (now having no usages) have been dropped.

more thoughts on Journal

First, the design has gotten to be a bit much, while still being inadequate. In AgileWiki3 we had the ability to navigate links in past time and we should again be able to do that.

One simplifier is the mandate: keep Rolons small. As soon as you start trying to do do much in a Rolon, you start down the same path of oo and end up needing a lot of custom code. Building networks of Rolons yields a far more accessible structure. Rolons are simple trees and you should not push them too hard. (Case in point--I'm creating a structure to describe commands and could easily define new attributes to identify queries and scriptable. But by using multiple parents I can achieve the same thing without needing new attributes. And then I also have fast access to all queries, etc.)

Now if we assume rolons are small (without putting any particular limit on it--we still allow for very scalable sections), then we do not need to be able to view all the changes specific to a section--we just view the Rolon's journal for all the changes for the journal. This greatly simplifies the proposed design.

Snapshots created when a transaction is processed need to be named what-when, where what is the name of the updated rolon and when is the timestamp of the change. Now we can include the classifier in the snapshot, so long as the same snapshot is added to all the links, extending the rolon name being linked to. Link resolution then is just a tad more complicated and is very reminiscent of AgileWiki3--resolution will be to any snapshot of the same time or earlier.

Oh, and there is one other constraint. When purging, we must always keep the oldest snapshot of those being purged. Otherwise link resolution will fail.

This will not be an easy change, but it will give us all the capabilities of AgileWiki3 for navigating time.

Desprately seeking Suzan. Also Kumar and Bob.

http://agilewiki.wiki.sourceforge.net/Openings

If you have an interest in learning more about Rolonics, or working on COODBMS, or doing a generic GUI, or would like to participate in any other way, then please see the above link.

a chat with Norm

Just had a great chat with Norm. A few things came up that I'd like to share.

First (on my mind) is journaling. AgileWiki3 did this right, but there were problems in the approach which I believe are flawed from a production perspective. It did not have state in any sense but implemented everything in the Journal and played it backwards to get the state for any given time.

The current approach is to have cheap snapshots of Rolon descriptors and ledgers (still to be implemented) and to present them to the same effect. Workable if we can keep the snapshots cheap enough, but in some sense a cludge in contrast to playing the journal.

One comment by Norm which I find very interesting is that classifiers/descriptors/ledger/journal may be the only representation to date capable of modeling systems theory (including systems of systems I assume). In my mind this adds a lot of validity to what I am trying to do, even if it is only partly true. (What he actually said was "the quadrivium is the first useful structure for modeling system theory.")

TiddlyWiki and RememberTheMilk were also discussed. Each has its strength and both may best be described using Rolonics--for then you can see how their capabilities could be implemented as a single extrodinary system.

Aye, not MOODBMS, trip delay

Our tickets were not confirmed, so the trip is delayed. I did take most of the day off yesterday, got some papers organized and spent some time at the bank. And I needed the break.

I am thinking now that MOODBMS is not a proper layer over COODBMS. It is "just" an application and the Aye GUI. So the name should be Aye.

Which means notifications and plugin metadata are all part of COODBMS, though perhaps not version 1.0.

COODBMS release 0.16.4

Several bug fixes and enhancements have been implemented relating to file inport/export.

http://agilewiki.wiki.sourceforge.net/CoodbmsReleases

travel plans, versioning

Making a quick trip to Bangalore, so I will be off-line and without a computer until Thursday. Before I go, I'll release the changes to COODBMS made so far. (I'll likely be busy today on other things.)

I've been thinking about version control, which is a nice capability to build applications on, if done reasonably well. We will have this capability once I manage to do the planned enhancements to the Journal, but there is another small hole--the import commands. As implemented, the import commands do not update an existing entry, so we need another set of commands for updating documents held by COODBMS. THEN we can put together some journal commands which will give us access to the history, and past versions, of a document. And yes, this all needs to be a part of COODBMS 1.0.

Of course, it will not be a SVN replacement unless we can put together a client that provides reasonably good support. But I'll leave that as a project for someone else. (I've got enough to do already.)

loading data into COODBMS is still too difficult

I've been thinking about a static.cowdb database which would hold help descriptions, application plugins and other information which changes infrequently. The advantage of having a dedicated database for this, just as we have a dedicated database for logins, is that it reduces thread blocking and consequently improves both throughput AND response time. And in this case, where queries are the norm, multiple threads can access the database at the same time. Another plus here is that, because the cache space is shared, the increase in the memory footprint is not significant.


But there is a small complication. How to distribute the contents or provide a means of upgrading it? Note that commands like bind and importText can not currently be part of a RoleML script, as these scripts run on the server and the source file is on the client. We could use a browse script and run browse against an input file, but then if there is an error we can't simply fix the problem and rerun--we've got to undo things manually. Not a good approach.

One alternative is to make the run command (and the runScript utility) smart enough to recognize these commands and pass the file contents along with the script.

COODBMS release 0.16.3

o Cyclic check was broken, allowing a rolon to adopt a parent--fixed.
o Logging is now done using java.util.logging. Client-side utilities are provided to set the logging level for capturing tracebacks.
o With the 3.4.1 release of CowDb, the move commands are now working.

http://agilewiki.wiki.sourceforge.net/CoodbmsReleases

CowDb release 3.4.1

Method PElement.getKey was not handling the simple case of direct links--fixed.

Questions about COODBMS vs MOODBMS, 9 day fast

The 9 day fast is over and we are all tired. This is the tenth day, victory day, and we had a big fire puja for the Goddess Durga's blessing. So it is a good time to start the preliminaries for MOODBMS.

My first thought is that notifications should be part of MOODBMS, as the COODBMS client is dumb but the MOODBMS clients will need to have a local cache. On the other hand, the MOODBMS help is not open ended. So we need to start a commands set in COODBMS and have help implemented on the server side. This way an application can define its own help. (I am a big believer that clients should be application-independent and back-end driven.)

COODBMS release 0.16.2

o DTDs written for RoleML scripts and responses.
o New logoff command for Browse.
o The browse commands elements, setWellknownName, keywordMatches, adoptFork and wellknownFork have all been fixed.

more thoughts on rolon shapshots in the Journal

I've been thinking that there is no need for the snapshot made when a rolon is updated to contain the JU. This is needless overhead.

The CU should also not be copied in the snapshot, as it will complicate a lot of things. The CU in the snapshot will then only contain the reference to the parent of the snapshot (the transaction rolon) and pairs of links to the original rolon which was updated.

slippage, script DEnts

I've just pushed the expected release date for COODBMS back to June 1st. I am not happy about this, but I really want COODBMS to be a commercial grade product and too many things have come up--like the 4 days lost working on forks.

I've also been considering having scripts stored in a DEnt and running them there, but finally realized that it is best done as part of MOODBMS.

Reworking the Journal

Had a great chat with Norm on Saturday. I have been concerned about the Journal, as the use cases were not clear in my mind. What I am providing is a JEnt for each change made to a Rolon, a JEnt in the user JU to capture the changes made by a user, a wrapper JSec for all the JEnts in a Rolon created by a transaction and a link from the transaction rolon to all those JSecs. The idea is that you can look at a Rolon's JU and see all the changes which were authored by it or applied to it.

Now we add 2 additional requirements: we want to see all the changes made to a given element and we want to capture a snapshot of all the updated rolons.

Now it is easy enough to create the snapshots of the updated rolons and make them children of the transaction rolon. The snapshots can then have a CSec which links to the elements changed in the updated rolon. And we can then view, in chronological order, the backlinks of an element from the rolons which capture the change.

That is almost enough, but we have no association between the JEnt and the snapshot. And the JEnt contains the description of the change, as well as the rolon-relative link (in the Context attribute) to the element in the snapshot which was changed. So we need another link from the snapshot to the JEnt. And as multiple elements in the same rolon may be changed by one transaction, the snapshot may reflect the changes to changes to more than one element. So the link to the JEnt and the link to the element that was changed need to be coordinated somehow. One way would be to have a CSec in the snapshot for each change, and have only these 2 links together in the same CSec.


So in the snapshot we can have a CSec called SOURCE and under this CSec we could have a CSec for each change. Each change CSec then would have a link to the element in the original rolon which was changed and a link to the JEnt in the same rolon which describes the change and gives the location in the snapshot of the element that was changed. (This is important as the rolon-relative pathname of an element may change over time through the use of the changeKey operation.)

COODBMS release 0.16

* New utilities for database integrity validation.
* Includes the CowDb 3.4 bug fixes.

CowDb release 3.4

o Sorted nodes were not freeing space when the node was freed--fixed.
o Nodes were marked dirty when read--fixed.
o Fork logic was corrupting the database--fixed.
o New validation class checks database integrity.

progress

I can now create branch x,
fork branch x y,
delete branch x and then
delete branch y.

This works great, after a whole lot of effort. Also I now have a solid validation tool for checking for duplicate space usage and memory leaks in the database. In addition, I've sped things up by fixing a bug which forced every object read to be rewritten. So the effort has been very worthwhile.

But when I create branch y z, i.e. fork a fork, I still have problems. So I'm close, but no cigar.

Some things are working

I've decided to go back and add the validation logic to the CowDb samples. Got through the first five and, aside from a bug in the validate logic, found no problems.

big memory leak

A COODBMS deleteBranch opens up a huge hole--we've got a grade A #1 memory leak here.

Here's some sample output from the validate utility after running createBranch/deleteBranch a few times. The format is simple, block location + length = starting address of unaccounted memory. Basically I just built a table of all available, freed and in use memory and combined adjacent blocks. When things are working, there should only be one big block with a starting location of 512.

512 + 320 = 832
85312 + 64 = 85376
89920 + 192 = 90112
127744 + 64 = 127808
169792 + 64 = 169856
211776 + 4480 = 216256
216448 + 128 = 216576

validation utility is written

I should have written this utility a long time ago. It is less than 200 lines of code.

First I tried it on what I thought was a good database. Oh, there is a memory leak. Or perhaps a bug in the utility.

Then I ran createBranch on a new database. That was fine.

Obviously I'm going to need to play with this. Likely there is more than one problem. :-(

COODBMS release 0.15.1

o Bug fixed: move parent operations must change pathname of the context.
o Opcodes now match command names.
o Command processing pushed into eval to simplify the addition of application code.

database corruption

The COODBMS 15.1 release is ready and I'm going to ship it out. But I've managed to corrupt the database, likely through a series of forkBranch and deleteBranch operations. So I will be shifting my attention to CowDb for a while, hopefully not for very long.

COODBMS release 0.15

Changes:
o Queries are now scriptable and extensible--responses to all commands are now standardized, so you can add queries without changing the client code.
o Dropped several obsolete opcodes in the server.
o Elements query no longer displays links.
o The listUsers utility has been replaced by a script.
o A migration script is include for upgrading existing branches to work with this release.

progress, slow but steady

Nearly done reworking the queries. And now that we have scriptable queries, I've been using scripts a lot more for informal testing. I find that when I can only test with a UI that the quality of the code crumbles over time. I've really got to revisit those old COODBMS tests and convert them to scripts and use them as a test suite.

I've also spent a bit of time on the output from those same queries. Small changes, but quite helpful I suspect.

CowDb release 3.3.5

Bug found/fixed in CowDb: forkBranch was saving the new branch processor in the branch processor table under the old branch name.

release notes:
http://agilewiki.wiki.sourceforge.net/CowDb#Releases
download:
http://sourceforge.net/project/showfiles.php?group_id=106672&package_id=253852

a good start on reworking queries

The logic for the rbacklinks and show queries have been migrated from Browse to the server. So things are pretty much in place and there is a good chance that we'll be done with reworking the queries tomorrow.

I'll note that there is not a lot left to do in COODBMS. Notifications are next. But then there is performance testing, and that is worth putting some serious effort into. Performance should be slower than CowDb, what with all that journaling. But in any case, it will be good to beat on COODBMS a bit.

I should probably also create some test scripts to replace all the old COODBMS samples. (Those samples are still on SVN--I just didn't include them in the last release.) And of course the top-level wiki pages will need to be reviewed. (I did finally add some navigational aids for to the wiki today.)

And once all that is in place, it will really be time to work on MOODBMS. On the one hand, I really want to have COODBMS in good shape before moving on. But on the other hand, it will be nice to get started on the next layer of AgileWiki.

MOODBMS should be quite exciting as it will be focused on virtually eliminating the need for application-specific UIs. And while we will only be looking at a non-graphical user interface, this should lay the groundwork for the wiki.

COODBMS release 0.14

* There is now a wiki page describing every Browse command.
* The sample application, check register, has been updated as well as its wiki page. This application now demonstrates how to extend the COODBMS server.
* New utility, runScript.

Good progress on converting check register

Check register is now half working. And I must say, it is nice to be back in the code.

Browse commands documented

http://agilewiki.wiki.sourceforge.net/BrowseCommands

See the above link--there is now a wiki page for each command.

documentation of commands half done

http://agilewiki.wiki.sourceforge.net/BrowseCommands

Creating the Wiki pages for these commands is a slow process, but I am now more than half done.

Safely home in Raipur

Everything went well on the trip from Bangalore to Raipur. Travel time: 27 hours. And our cat Sassy was as good as gold.

so many pages, so little time --> dead links

Not a lot of time left before leaving for Raipur. I may get back online this weekend, with any luck. I just created a BrowseCommands page in the wiki. It and the script page now have bunches of links to all the pages, yet to be created, describing all the commands. Not the best stopping point. :-(

The movers came yesterday and I was quite impressed. Everything was wrapped in cardboard and then 3 layers of plastic, including the couch, stuffed chairs, refigerator, washing machine, A/C, TV, PC, monitor, etc, etc, etc. And every dish and cup was wrapped in 2 layers of newspaper. There were 8 packers and it took them about 8 hours to do all this. But remember, it is an 8-day trip by rough roads in an open truck and they will get some rain on the way. Total piece count: 70. Total cost: Rs24,000 or about 2 months rent in Bangalore.

Tonight's adventure: a 2-day train trip with Rupali and our cat, Sassy. Hopefully Sassy will not be forced to ride in freight. She's already freaked out by the empty rooms in the appartment.