The AgileWiki Blog

New Home Page for AgileWiki

2009-03-10T14:43:00.006+05:30

With the release going out today, the database is now fully scalable. Time for a home page which sports a feature list: http://agilewiki.sourceforge.net

The big change included in today's release, 13.1.0, is that element text content (text or binary) is now implemented using a b-tree. No problems with content that is hundreds of megabytes.

If interested, be sure to join the AgileWiki community at http://agilewiki.ning.com/

Bill

P.S. I just moved this page because of an access control issue.

Putting Magic into Software

2009-03-03T12:32:00.000+05:30

http://agilewiki.ning.com/profiles/blogs/putting-magic-into-software

Web3.0 Business Model

2009-02-28T23:02:00.000+05:30

http://agilewiki.ning.com/profiles/blogs/web30-business-model

AgileWiki on Ning

2009-02-27T17:04:00.001+05:30

I've just created a ning network for AgileWiki: http://agilewiki.ning.com/

You are welcome to join.

Bill

Rolon Basics

2009-02-24T17:55:00.002+05:30

Just completed a new section to the User Guide, Rolon Basics:

Headline - A Brief Description

Context - Rolon Context Drives Everything

Names - Names Depend on Context

Journals - Recording Changes and Other Events

Realms - For Organizing Rolons, and More

UserAccounts - An area for Each User

Relations - Building Complex Structures

AccessControl - Privacy and Control

UserGroups - Advanced Access Control and Common Work Areas

Rolonic Software Engineering in Brief

2009-02-18T13:55:00.000+05:30

Common Data Structure - Everything is implemented as a Rolon. This includes commands, journal entries (which capture activity) and services. All Rolons, and only Rolons, are user objects.
Flexible Structures - Structures of Rolons are built using parent/child relations. Additional, arbitrary relations can be defined and used to interconnect Rolons. One such relation, Applicable Context is used to extend the namespace of a Rolon beyond the context implicit in the Rolon's location.
Open and Extensible - A Rolon can be expressed as an XML document and is implemented by binding elements to Java classes. Plugins are used to define and extend these bindings. All Rolons implement the same life cycle, enabling the composition of complex services from structures of Rolons.

Rolonic Software Engineering provides a systematic and comprehensive approach to the implementation of fully interoperable services, which is exactly what we need to implement Web 3.0.

ElementModel release 12.4.0

2009-02-09T12:11:00.001+05:30

Access control is now in place.
Relations reworked to support incremental inferencing.

Home Page

The end of programming as we know it

2009-01-24T11:35:00.001+05:30

I've just added a new page to the wiki which I think you will enjoy. It may well be the ultimate disruptive technology. :-)

Everything Just Works Together

Open letter to AgileWiki developers

2009-01-14T11:02:00.000+05:30

Everyone,

I do hope you all can find something of interest below that you to contribute to. This list is just off the top of my head. I'd be more than glad to discuss other projects as well. For example, we need to be able to mirror the content of one system in others. Lots more coming, you can count on that.

Bill

Password Management --Bill

Problems with a p2p network of extensible nodes is that there is no central authority that can be trusted. You end up with too many passwords, as usual, for access control.

But you can trust your own client software, and if you use a public web service, you likely trust that as well. So why not have your point of entry into the network manage your passwords?

Likely also, you can group some of those hosts together and use a common password for them. Passwords then are assigned to individual hosts or to groups of hosts. And when you change passwords, that change should be communicated silently to all the hosts in the same group.

So we end up with something very much like single-signon, but where only the user needs to trust the point of entry. And we can use passwords to sign the messages as well.

I expect to have this complete (except for signing messages) in a day or two.

Web Front End --Naji

This is a huge project. Integration with Jetty, replacement of Browse with Comet. To date, a simple hello world has been released. I expect to participate in this project as well, but not any time soon. I expect that there will be pleanty of work here to keep any number of developers busy.

I already have an ISP where we can run this.

Access Control --Bill

With Web 3.0, you no longer need large capital expenditures to provide a service--the number of computers providing the service grows with the number of users. But think about it. If I'm keeping important information in my ark/repository/database, why would I let just anyone come and change it? And some of that information is confidential too!

For users to open their systems and really collaborate, we need access control. I've done this before and with relations already built in, it will not be hard.

Wiki --Bill

This is the beginning of semantics. As the repository is somewhat hierarchical, there are inherent problems in building a wiki on it--hierarchical wikis are always a good idea gone wrong. The fix is adding an applicative context, which gives you control over the namespace each page can use for referencing other pages.

I'd rather be writing docs than working on this project, so yes, I'd welcome the help here.

RSS Aggregator

The idea here is to feed the ark with content from multiple sources (information fusion), do a keyword search on the content and organize the incoming material on that basis. Norm (the father of Rolonics) has as an example over 200 feeds which he monitors. This would be a big help for him (our test user) and many others like him.

This project could be the first real application of AgileWiki--nothing else is needed. It is time to start this project now. First step, determine which rss aggregator(s) to integrate with.

Who wants to get started on this?

JXTA / JXTA 2 Integration

The current com package (nio sockets) might work in a business, but not out in the wilds of the internet. JXTA looks like the way to go. I see this as a necessary minimum to be a real Web 3.0 player.

I'd like to finish reworking the message formats (adding digital signatures) before this project gets too far along. But that shouldn't take too long.

Who wants to get started on this? I already have an ISP where we can run the core servers for this.

Developer Docs --Bill

By the end of March I hope to start working on this full time. But it would be very helpful if I could get a bunch of questions from everyone so I have some idea where to begin. I mean, the code all reads so clearly for me, but I suspect that is not the case for everyone else. To do a good job when writing, you need to understand the needs of the reader.

Everyone please, help me out on this. Thanks!

Email Client

I've worked in companies where everyone spent 6 hrs a day (no exageration!) reading and organizing their email, and a lot of those email were "things you might need to know" that were sent to everyone. Very few emails actually needed to be read immediately, but they all needed to be sorted out and organized. What a waste of time!

Integrating an email client into AgileWiki means that you can share not only the content, but the organizational structure. Again, this is all about information fusion and creating a useful organization of content. This could be a big time saver and an important application.

Having done this before, I'll note that the hard part is in presenting the email nicely. Seems like every email client structures the email slightly differently.

Any takers for this project?

Inferencing --Bill

Reasoning systems really aren't that hard to do. The tricky part is making them incrimental, so as the information changes, the inferences can be adjusted without having to start everything over again.

The relations already built into AgileWiki will be a starting point. And applicative context, which is part of the wiki implementation, is done, that will be the next part.

This will be a fun project, but we'll not be ready to start it for a while.

Get an email ID as yourname@ymail.com or yourname@rocketmail.com. Click here.

Why everything "just works together"

2009-01-12T10:37:00.001+05:30

With most software projects, if you have N options/features, the order of complexity is on the order of 2**N. So long as N is small, things go well, but as you add options and features, the complexity quickly gets out of hand--integration then becomes a black art. In contrast, the complexity of this project is on the order of b+N, where b is the complexity of the base system. That b is a bit large, so it makes it hard to come up to speed, but as the project grows, the overall complexity is proportional to the number of options/plugins. This is an astonishing result.

The technique we use to achieve this is by simply binding classes (and other configuration data) to element types. Each plugin defines new element types and also extends the configuration of pre-existing element types. The binding is all done in the .cfg file, which is generated from the .plugin file.

Now when the program first starts (empty database), it creates the ark root element, which contains configuration data about various sub-elements and child rolons which are to be created. These are created in turn, recursively, to generate all the architectural rolons needed to run the ark with a given set of plugins. Note that plugins never delete a pre-existing element type, nor do they ever change the binding and configuration data of an element--they only add additional configuration data. This is key, as it provides most of the decoupling we need to have a truly open and extensible platform where everything just works together.

http://agilewiki.wiki.sourceforge.net/

Twitter for AgileWiki development progress

2009-01-09T09:48:00.002+05:30

We've started using Twitter on the AgileWiki project.

The links to the twitter accounts of the developers can be found at the bottom of this page: http://agilewiki.wiki.sourceforge.net/

Identity for web 3.0

2008-12-29T15:34:00.002+05:30

http://agilewiki.wiki.sourceforge.net/Identity

I've been looking at web 3.0 and find that I very much agree with the ideas. I would categorize web 3.0 as being a p2p system which supports multiple points of entry (home, office, cyber cafe), where the nodes support information fusion, have pluggable applications and some native reasoning. And this is exactly what I've been working on for so many years now.

Today I've been thinking about an easy way to validate user identity in such a network. See the above link for details. It is reasonably easy to do and I've introduced only one new idea--categories of passwords.

Element Model release 12.0.0

2008-12-22T15:04:00.002+05:30

The renameChild command is working again.
Dump no longer writes out prebuilt rolons.
Changes to the basic architecture are now restricted, as changes not preserved when migrating to a new release via dump/load are no longer permitted.
New Plugin interface.
New ConfigurationManager converts a list of plugins into a configuration file.
Keywords are now broken out as a separate plugin.

With the new plugin facility it is now trivial to configure a program. And for each new service/facility/application, you create a new plugin.

The utilities have been revamped as well.

Rolonic Software Engineering

2008-12-11T16:42:00.001+05:30

Finally completed the operational theory of the element model. I do hope you find it helpful.

Roadmap for ElementModel and AgileWiki4

2008-12-05T08:21:00.002+05:30

A comprehensive road map for Element Model and AgileWiki4 has been added to the home page. Inferencing, a mail client and versioning are included.

The development team (2) is far too small for the work planned. If you are a Java developer, your participation would be most welcome!

Element Model release 11.0.0

2008-11-22T16:33:00.001+05:30

Element model databases are now crash-proof.

no design time

2008-10-24T09:51:00.002+05:30

After a restless night I've decided not to do the design time thing. Two reasons:
1. Every application class would need to support an additional constructor.
2. There is a one-to-many relationship between element types and element instances.

So instead we're going with configuration classes which will embrace [most of] the parameters of an element type definition. This can be implemented incrementally as well, which is important at this point.

design time

2008-10-23T20:11:00.002+05:30

Today I was working on configuration. Configuration is never easy, but in the Element Model it seemed extra difficult because it is spread across too many classes. Finally decided to make use of the JavaBean design time idea. Application logic should support its configuration.

So obvious. Not even original. But I think it will help a lot.

element model release 10.0.0, goals for 11

2008-10-22T08:16:00.002+05:30

New validate utility checks the integrity of a database.
Additional convenience methods added to AbstractApplication.
Javadoc rewritten for the elementmodel package.

This completes basic persistence. For release 11 there are 3 objectives:
- Employ weak references (in handles) and a cache to reduce the memory footprint.
- Use variable-length blocks in place of fixed-length blocks.
- Extend browse to the point where it operates as a simple wiki.

A light-weight EJB replacement

2008-10-20T17:20:00.000+05:30

The Element model has come a long way and release 9-11 is out.

Peer Proxies are complete.
Basic persistence is done.
Dump/Load commands can be used to migrate data from one release to the next.
Naji has started work on integration with Jetty.

Download

The Rolonic Framework is now Complete

2008-09-25T17:47:00.002+05:30

http://agilewiki.wiki.sourceforge.net/

Release 8-0 of the element model completes the Rolonic framework. The next step will be to add Peer Ark Proxies.

element model framework now supports toolbars

2008-09-04T22:12:00.001+05:30

Release 5-3:

New setHeadline command.
[up] and [home] have been added to the toolbar of GBrowse.
The framework now supports toolbars.

Contributors to this release: Bill, Naji.

element model 5-2

2008-09-03T12:57:00.001+05:30

Fixed a critical bug introduced in 5-1 which prevented most sample programs from running.

Added configuration support for headlines.

ToolTipText is now based on the element's headline.

element model release 5-1

2008-09-02T18:31:00.001+05:30

http://agilewiki.wiki.sourceforge.net/GFrameManager
http://agilewiki.wiki.sourceforge.net/ExitOptionOperation

Release 5-1 is out. The focus this time was on cleaning up the frame management classes--which have now been reduced to just GFrameManager.

Also, catch the two new wiki pages above.

element model release 5-0

2008-09-01T16:50:00.000+05:30

The element model now supports the development of fully decoupled Swing components. Layout can now be separated from behavior, making for a much more fluid architecture.

Added a new page on Swing, too.

element model release 4-9

2008-08-31T19:47:00.001+05:30

RsBrowse is replaced by GBrowse.
The old rswing package is gone.

The new GBrowse program uses BoxLayout in place of GridLayout and behaves better.

element model release 4-7

2008-08-30T16:36:00.002+05:30

New samples package.
ArkLocks are no longer reentrant.
Configuration of multiple top-level rolons is now supported.
New getCompoundPathname for navigating multiple arks.
Responses are now returned using the new ROUTE operation.
A contextMap is now used with commands and route operations.

I'll note that by using the route operation to return command responses, sockets will be fully bi-directional with no session tracking on the "server" side.

element model release 4-6

2008-08-24T18:44:00.001+05:30

-New CommandStatusEvent for tracking command completion.
-[non-GUI] Browse now supports asynchronous commands.
-New swingcapabilities.CommandManager class supports command completion.

element model release 4-5

2008-08-23T15:40:00.000+05:30

Any program can now be run using a generic Main.main.
A new integration with Swing is included.
ThreadPool is now part of ArkRoot for reduced locking overhead and greater flexibility.

element model release 4-4

2008-08-21T21:38:00.000+05:30

o New up and destroy commands.
o The sleep command now demonstrates an asynchronous response.
o A thread pool is now used in place of dedicated threads.
o Command API now includes a pathname of the target.
o All commands rewritten for improved thread safety.

element model release of version 4-3

2008-08-19T17:22:00.000+05:30

o Synchronization locks have been replaced by ReentrantLocks.
o RsBrowse now runs commands on a separate thread.
o Browse has been cleaned up and fully documented.

element model release 4-2

2008-08-18T10:40:00.000+05:30

o Configuration role and op ascension numbers are now generated.
o Command applicability bug fixed.
o Commands now optionally return an operation request.

element model release 4-1

2008-08-17T16:51:00.001+05:30

API polish, code cleanup and 2 more "How To" docs: Sub-Arks and Commands.

element model release 4-0

2008-08-17T07:16:00.002+05:30

Compound pathnames have been introduced for navigation, and command execution, in subordinate arks and other contexts.

In the top-level ark you a sub-ark, ark1.

Now you can enter that ark using the pathname /ark1//

This applies recursively. So if in ark1 you create ark1.1, then you can enter it with the pathname /ark1//ark1.1//

Once you have entered an ark, all the commands are processed in that ark. This includes help, which lists only the commands available in the ark you have entered.

element model release 3-6

2008-08-13T14:39:00.000+05:30

The execution model has been redone for greater flexibility and simplicity.

element model release 3-5

2008-08-12T16:24:00.000+05:30

o ServiceRequests replaced by the more powerful and role-based OperationRequests.
o RsBrowse now uses a combo box for command entry.

Contributors: Bill, Naji

element model release of version 3-4

2008-08-10T21:10:00.000+05:30

o Services must now register for the service requests they can process.
o Framework services are no longer distinguished from other services and are fully extensible.
o Service requests now replace many of the operations and events in Browse and RsBrowse.

element model release 3-3

2008-08-09T19:23:00.000+05:30

o Enhanced configuration properties creation and management.
o Cleaned up browse.

element model release of version 3-2

2008-08-08T18:41:00.001+05:30

o Reworked the way Swing is mapped into the element model.
o RsBrowse is now 3 fully decoupled components.
o Multiple RsBrowse windows can be opened in multiple arks.

Download

Write once, use in everything

2008-08-05T19:31:00.002+05:30

My vision from the 60's has always been what you could call "write once, use in everything," which is not quite the same as "write once, run everywhere." Finally after all these years, I seem to be making progress.

Bill

element model 3-1

2008-08-05T17:22:00.001+05:30

o Browse and RsBrowse now support the go and express commands.
o A navigation bar has been added to RsBrowse.
o A simple threading model has been implemented.

element model release 3-0

2008-08-04T22:32:00.000+05:30

o The element model is now integrated with Swing.
o There is now a Swing version of Browse.
o Ordered lists of elements are now supported.

Contributors to this release: Bill & Naji.

element model release 2-4

2008-08-02T21:13:00.000+05:30

o ServiceRequest object are now used in place of a control object for framework requests.
o Constructor IOC is now used for better integration with Swing.

Download

element model release 2-3

2008-08-02T07:43:00.002+05:30

For each framework service there is now a subclass of ServiceRequest. Service requests then are passed to the element, which forwards them to an application object in the framework which implements Services. Service request processing is synchronous, allowing for the results to be returned by the call for the request.

download

element model release 2-2

2008-08-01T08:07:00.000+05:30

o Entries have been added to the containers package.
o Requests to rolons are now via action events.
o Browse now supports createArk and arks commands.

Contributors to this release:
Bill
Naji

granularity

2008-07-29T10:34:00.002+05:30

Granularity is the key concept that opens our understanding of software. In AgileWiki we have 4:

1. Methods apply to capability objects. These are grouped by aspect and elements are aggregates of, at most, one capability for each aspect.

2. Events apply to elements. Each event is routed to the capability objects of the applicable aspects.

3. Operations apply to rolons, which are DOMish trees of elements.

4. Commands apply to arks, which are a collection of rolons.

element model release 2-1

2008-07-29T10:03:00.000+05:30

Browse is now working and supports help, echo and exit commands.

https://sourceforge.net/project/showfiles.php?group_id=106672&package_id=285239

element model release 2-0

2008-07-28T14:13:00.002+05:30

Release 2-0 completes the element model. A Hello World! program is also included.

Download

element model release 1-0

2008-07-26T17:33:00.001+05:30

In this release we have implemented a simple ark and event processing.

elementModel release 0-2

2008-07-25T13:46:00.002+05:30

http://sourceforge.net/project/showfiles.php?group_id=106672&package_id=285239

The element model, which is really an extensible DOM, has now been roughed out in a minimalistic sense--only the necessary items have been added. It can express itself now as XML, but that was a feature that was helpful for testing.

The idea here is to provide the minimum of code to introduce the concepts. What we have are 4 interfaces and 10 classes, none of them very long.

The download (see the link above) includes the javadocs as before. This time it also includes the source code. For your reading pleasure.

Element Model release 0-1

2008-07-24T10:50:00.002+05:30

What is the point of a framework that no one understands? So we begin again, this time with a simplified element model.

http://agilewiki.wiki.sourceforge.net/

a layered approach

2008-07-23T14:35:00.002+05:30

One thing I learned when working on COWDB and COODBMS is that it is very difficult to implement things in layers--you really need to know what you are doing.

One thing I learned when working on COWDB2 is that it is very difficult to explain a monolithic implementation. People need to learn things layer by layer.

An important goal in this project is to build a team that can work on, support, and expand on what I have learned these last few years. So a layered approach is the only real option. I have learned a great deal these last few months while working on COWDB2. Hopefully I have learned enough to be able to use the layered approach.

Layer 0 -- This will be the element model. There will be no persistence object (that is moving into the streaming aspect) and this layer will have no knowledge of the aspects, excepting only the Application objects which are what are passed instead of the element.

Layer 1 -- This will be a business object model. Business objects map into XML documents. Multiple business objects will be supported, with links between them. We will implement attributes, containers and links. (The approach taken will be to implement the minimum necessary for each layer. The implementation of a rich set of capability objects can be delayed until and as they are needed.) In this layer, the framework will support the creation/deletion/retrieval of business objects. But there will be no persistence.

Layer 2 -- Here we add well known business, parent/child relationships, operations and roles. Again, a light weight minimal implementation with no persistence. The framework will provide access to well known business objects.

The objective here is to develop the basics and provide the necessary documentation. These three layers will then comprise the core for a more serious effort.

COWDB2 release 2-0.3

2008-07-09T17:14:00.000+05:30

o Application objects now need only implement an interface with one getter and one setter to be used in COWDB2.
o The diskspacemanagement package is now complete, including javadocs.

COWDB2 release 2-0.2

2008-07-07T06:48:00.002+05:30

In this release we have made a big change to the object model. We now pass references to the Application object place of references to the aggregate element, making the object model largely transparent. So when using the API, you pass application objects as arguments and the returned values can simply be cast appropriately. Internally we now also have strong typing, as everything is no longer simply a CElement.

One consequence of this is that for every element need an application object, so we've had to add a lot of Application classes for the various element roles used by the framework.

There has also been a change to the implementation of transactions. They are now simply facods, carrying a reference to their associated element. Facods are strictly transient and are not aggregated into the element.

COWDB2 release 0.1.3

2008-07-05T18:33:00.000+05:30

The control package is now complete, including the Javadocs.

Contributors to this release:
--Naji
--Bill

COWDB2 release 0.1.2

2008-07-03T11:40:00.000+05:30

o Read and write caches are now in place.
o Threading issue addressed in handle.
o Javadocs done for the contents and documents packages.
o New Bytes capability added.

Contributers to this release:
- Naji
- Bill

COWDB2 release 0.1.1

2008-07-01T07:51:00.002+05:30

o The Java Docs for the elements package been completed.
o Some minor changes/enhancements made to the API.

COWDB2 release 0.1

2008-06-29T07:40:00.000+05:30

o Bug fix: Transaction Abort is now working.
o ForkBranch is now working, which means that all utilities (except RunScript and Browse) have now been implemented and are working.

COWDB2 release 0.0.7

2008-06-27T13:37:00.001+05:30

o Adjacent free and available blocks are now being consolidated. This solves a disk fragmentation problem and reduced the footprint on disk significantly.
o Validate was raising bogus db corruption errors--fixed.

COWDB2 release 0.0.6

2008-06-25T10:32:00.000+05:30

o The DeleteBranch utility is now working.
o Disk Space Fragmentation has been addressed.

COWDB2 release 0.0.5

2008-06-23T14:28:00.000+05:30

o ListBranches, CreateBranch, ListOpenBranches, and CloseBranch utilities are now working.
o ChangeControlEvent and FreeEvent have been added to the object model.
o Control classes have been refactored.
o Convenience methods added to CObject and the code base updated to use them, making the code easier to read and write.

Your input is requested

2008-06-21T12:20:00.003+05:30

I have just revised the home page for AgileWiki at http://agilewiki.wiki.sourceforge.net/ and very much desire any suggestions you may have. I believe that a clear and compelling value proposition is important if we are to continue to grow the development team.

Thanks in advance.

Bill

COWDB2 release 0.0.4

2008-06-21T10:00:00.000+05:30

Validate is now working.

When initializing a database, the trunk branch is now created.

Aspect factories now take a fully qualified class name when instantiating a capability. As a result, COWDB2 is now fully extensible.

change attribute events

2008-06-20T11:54:00.002+05:30

For a very long time now I have not been happy with attributes. On the one hand, they have been very dumb and on the other hand they can effect major changes. In COODBMS, only the admin was allowed to set attributes arbitrarily--the mechanism in place was completely unsafe.

So my thought is to create a change attribute even and send it to the appropriate element aspects, which can then raise an exception if there is a problem, so it can be secured and yet made available to users as a fast way to make changes to an element.

Again, events look like a great way for the aspects of an element to coordinate.

going off line for PC repair

2008-06-17T17:37:00.001+05:30

Taking my thinkpad in for repair. The display no longer works and, while I can work using a monitor, I'm thinking I should get it fixed.

Not sure how long I will be off line. :-(

COWDB2 release 0.0.3

2008-06-17T08:26:00.000+05:30

* Unblock was not updating disk.
* UnblockUpdates, listOpenDbs and closeDb utilities are now working.

disk space management

2008-06-15T13:28:00.002+05:30

In COWDB2 we should be able to implement a much more scalable and fragmentation resistant memory management system.

First, we can round up allocation requests to the nearest Fibonacci number. This will prevent disk space fragmentation.

Second, we can take a 2-tier approach to space management. The first tier will be a system which is embedded in the root element itself. It will not be scalable and will only be used for managing the space used by the second tier. At issue here is that the root element must not grow beyond an anticipated size.

The second tier will [eventually] be a manager which uses a b-tree. Space for the elements of the b-tree will be managed by the first tier manager, while the second tier manager handles everything else. By using a b-tree for managing space in the second tier manager, we achieve the desired scalability.

COWDB2 release 0.0.2

2008-06-15T09:39:00.000+05:30

The following utilities are now working:
- tracebackLoggingOn,
- tracebackLoggingOff,
- listDbs and
- blockUpdates.

Aspect Programming with fine-grained Aggregation and IOC

2008-06-13T19:58:00.002+05:30

I am simply delighted with the way the new object model is progressing. More powerful and flexible than anticipated. At some point I expect to make much more extensive use of element aggregates in the framework but for now I think we will make do with implementing all the transactions as elements.

I've ported over the low-level database logic and have integrated the role element property file with the database configuration property file. So there is some progress besides the work being done on the object model. Still not much is working, but what does work is being tested regularly.

One thing which has frustrated me for years is trying to map rolonic structures into a single class heirarchy. At a minimum you need mixins. COWDB's simple aggregation wasn't used to address this--not enough aspects had been broken out. But aggregation in COWDB2 is much more fine-grained and really works nicely.

You know you are doing something right when...

2008-06-12T16:18:00.002+05:30

In the Rolonics world view, everything has for major aspects: wholeness, partness, structure and stream. Now these are the four packages under the elements package in the COWDB2 object model. Seems fitting.

I've always wondered why I could not implement a Rolonics database in a Rolonic fashion. I've tried for years to incorporate Rolonics into the implementation. Finally, it looks like I'm getting there. I just seemed to me that if Rolonics was so all-encompassing, I should be able to use it no matter what I was programming. The turning point was in implementing elements as aggregates, where each of the objects in the aggregate was for a different aspect of the element.

Now the exciting thing about all this is that the functionality of the elements divides up so nicely into different aspects, where normally aspects are all interwoven through the code base. This is just another indication to me that there is something very very right Rolonics.

COWDB2 release 0.0.1

2008-06-12T07:38:00.001+05:30

The new COWDB2 server can now [only] process shutdown requests.

a 2-pronged approach

2008-06-09T19:56:00.002+05:30

Made some good progress on the object model for COWDB2, but still there is a long way to go.

I also started working on migrating the COODBMS server code over to COWDB2. One very fine thing here is that we will have only one object model, rather than the 2 in COWDB, as the new object model supports both persistent and non-persistent elements. And that removes a major learning hurdle.

Now as I'm spreading myself so thin, progress now will slow even more. But while I'm trying to write nice clear code for the new object model, the framework is being ported "as is." So it will go a bit more quickly. But once I get a bit further I'll start testing with the "old" Browse program. So again things will slow. But that's all fine, so long as COWDB2 comes out of the development process in a form that is easier to understand.

As the (past) Director of Engineering at Fiorano, I was always saying that working code is simply not good enough. So I've talked the talk and now I'm walking the walk. But I am also hopeful that the new object model will be interesting in its own right. I must admit, I've not done much with aggregation before. But then I've never had to deal with anything quite so flexible.

A good start on COWDB2

2008-06-08T21:09:00.003+05:30

It is the beginning of monsoon season here. We get rains in the afternoon and evening. And like every year in Raipur, the rains bring frequent and repeated power outages. Fortunately this condition doesn't last for too long. After a while the electric company gets things under control again and power becomes stable. Indeed, one of the advantages of living in Raipur is that the state capital has a surplus of electricity. It's only a temporary distribution problem.

I have however made a good start on COWDB2 today, though there is a very long road ahead. It is fun though, rationalizing the API. You can follow my progress at http://agilewiki.wiki.sourceforge.net/COWDB2

COODBMS, a qualified success

2008-06-08T07:11:00.001+05:30

I count COODBMS as an overall success. It includes a significant number of AgileWiki3's capabilities, despite having a very different implementation. It is significantly more robust than AgileWiki3 and solved the problem of having an ever growing footprint on disk--AgileWiki3 had no equivalent to COODBMS's PurgeHistory. Still it is a partial implementation of AgileWiki3 and there is much to be done if we were to continue this implementation.

The failure of COODBMS is a general lack of clarity. This is partly because it was a layered implementation which did not anticipate the all of the requirements. Partly it is because the implementation is so very different from AgileWiki3. And partly because an important capability--time navigation--was added as an afterthought. Considering the size of the effort remaining, I believe it best to begin again with COWDB2, though hopefully there will be many large chunks of code that can easily be moved to the new implementation.

The rewrite will have a very similar overall architecture, with only minor changes. Where I expect to have the most changes are in the object model. The old implementation had 2 object models in the server and a third in the client. And as the implementation progressed, it was the object model which suffered the most. Otherwise I expect most of the ideas and techniques to carry over, all be it as a cleaner and clearer implementation.

As released, COWDB release 4.1 and COODBMS release 0.19.1 are fully functional and have no known bugs. In that reguard I can only hope that COWDB2 does as well.

COODBMS release 0.19.1

2008-06-07T18:07:00.000+05:30

There are no known bugs as of this release, which includes fixes for caching problems, memory leaks and navigation of past time. Multi-threaded stress testing has also been completed successfully.

COWDB release 4.1

2008-06-06T10:45:00.000+05:30

Dirty cache issues and memory leaks have now been addressed. There are no known bugs.

COODBMS cleans up nicely

2008-06-06T09:47:00.002+05:30

After that last run, I ran purge history. Here's the result:

Branch Index Count = 123
Well Known Table Count = 4
Links Table Count = 38
Backlinks Table Count = 38
Transaction Index Count = 22

Available blocks = 40.
Blocks in use=556.
Internal nodes=20.

So it cleans up very nicely.

COODBMS is working

2008-06-06T09:42:00.002+05:30

I increased the dirty cache size from 750 to 4096 and the memory leaks vanished. I'll note that the new algo for managing the dirty cache does require a larger cache size.

Branch Index Count = 79
Well Known Table Count = 702
Links Table Count = 1412
Backlinks Table Count = 1412
Transaction Index Count = 349

Available blocks: 1056.
Blocks in use: 43,613.
Internal nodes: 102.

one step forward, two back

2008-06-06T08:07:00.002+05:30

The bug in dirty cache is fixed. I set the max entries down to 64 to add further stress on the system and the scripts ran longer but still I am getting a memory leak.

At the time of the memory leak, the number of blocks in use is 20,576, and 47 internal nodes. but the number of available blocks is 405. No problem here, but my concern is that in production the number of available blocks would be very much higher--and the table of available blocks is kept in a single block. Transaction count is 154 (I was not running purge history).

Here's the script I was running, where each run is processed as a single transaction:

//exercise removeWellknownName
createWellknown a1
go /a1
removeWellknownName

//removing a link
createWellknown a1
createWellknown b1
go /a1
cu
addOrderedSection l
go l
addLink b /b1
go /b1
go /a1/CU/l
remove b
go /b1
removeWellknownName
go /a1
removeWellknownName

//adopting a child
createWellknown a1
createWellknown b1
go /a1
adoptChild b /b1
go b
removeWellknownName
go /a1
removeWellknownName

//fork a child
createWellknown a1
createWellknown b1
go /a1
adoptFork b /b1
removeWellknownName
go /b1
removeWellknownName

//set well known name
createWellknown a1
go /a1
createChild b
go b
setWellknownName b1
go /a1
removeWellknownName
go /b1
removeWellknownName

//well known fork
createWellknown a1
go /a1
createChild b
go b
wellknownFork b1
go /a1
removeWellknownName
go /b1
removeWellknownName

//reorder parents
createWellknown a1
createWellknown b1
go /a1
createChild ac
go /b1
adoptChild bc /a1/ac
go bc
moveParentFirst /b1
go /b1
removeWellknownName
go /a1/ac
go /a1
removeWellknownName

//change child name
createWellknown a1
go /a1
createChild b
changeChildName b c
go c
go /a1
removeWellknownName

//change key
createWellknown a1
go /a1
lu
addOrderedSection x
addOrderedSection y
changeKey x z
go /a1
removeWellknownName

//change Classifier Section name
createWellknown a1
createWellknown b1
go /a1
cu
addOrderedSection l
go l
addLink b /b1
cu
changeKey l links
go /b1
removeWellknownName
go /a1
removeWellknownName

wellknown

shaking down nicely

2008-06-05T19:59:00.002+05:30

COODBMS runs consistently until it is about 3 Mb in size. Then the caching problem nails it every time. But that was long enough a run time to complete some nice multi-threading stress tests, which it passed with flying colors.

So there is still the odd bug or 3, but we are getting close to a production release. We also need one final feature--load/dump to external files. This will be essential for migration to new releases as the internal structures evolve.

COODBMS release 0.19

2008-06-05T14:51:00.000+05:30

o Bug found/fixed: validate throwing spurious dup allocation exceptions.
o Bug found/fixed: validate was closing the database.
o Bugs found/fixed: the the remove methods of sorted and sorted iterator had memory leaks.
o The first argument of adoptChild and adoptFork are now optional.
o Bug found/fixed: the well known table was occasionally deleting itself.

COWDB release 4

2008-06-04T21:53:00.000+05:30

o Database validation is greatly enhanced.
o Disk space management of branches has been fixed.
o A pending table has been added for tracking space which has been freed in one branch but not in another.

things are going well

2008-06-03T15:37:00.003+05:30

With release 0.18.9, the only significant known error remaining is a memory leak. The work done these last two days was not easy, but putting an end to those pending errors was a great step forward.

I'll note that memory leaks only occur when there are internal b-tree nodes, so that gives me a great starting point. And the memory leaks are easy enough to force--just let the database grow and they start happening. So it should only be a matter of investing the time/labor to find the leak. And as other errors are not occurring, it is easy now to focus on this one bug. The downside, of course, is that finding memory leaks is never a fun process. Sigh.

COODBMS release 0.18.9

2008-06-03T14:12:00.000+05:30

o Reworked how freed blocks are processed--the code was not working when there were multiple branches/snapshots. This fixes all known pending errors.
o A validation table was added to prevent pending errors from corrupting the database. Free validates against this table.
o Small bugs fixed relating to syntax error handling and time navigation.

finally caught my tail!

2008-06-02T21:49:00.002+05:30

I've been chasing my tail all day trying to track down pending and dup allocation errors. I did narrow it down to a chained fork issue. Then after a few more turns I finally realized the flaw in the logic.

So I expect finally to make some progress tomorrow. Today has been... "educational." I always say, if you pound your head against the wall long enough and hard enough, eventually something will give way. Fortunately I'm a tad hard headed. (A trait which is not always appreciated!) :-)

back from Delhi, Ruby

2008-06-01T13:48:00.002+05:30

Back from a trip to Delhi, and a bit tired. I did manage to read a book on Delhi and I like the language, especially blocks. I am thinking that display logic can be written in Ruby (or Python), stored in DSecs in the database and accessed as needed. This will keep the front end thin and application independent.

But first there are a few bugs that need fixing.

Bill

corruption, fragmentation and scalability

2008-05-24T08:57:00.005+05:30

Disk space management is the foundation on which AgileWiki is being built. And there are three which need to be addressed: corruption, fragmentation and scalability. Until these are addressed there is little point in continuing.

The form of corruption which concerns me most is pending errors. This occurs when one leg of a
fork frees a block which is still in use by the other and then a system-level free of the block occurs. Unfortunately the API lends itself to this error. But it is likely quite repairable--the block is no longer in use and only we have not removed the block from the pending list. On the other hand, since it is an easy error to make, we need to be able to catch this error prior to completing the transaction which does the system-level free of the block. This can be done by maintaining a database-level list of all pending blocks and checking this list prior to doing a free. Other forms of database corruption being experienced just need to be shook out of the existing code base through the usual test/fix process.

Fragmentation is another concern, especially as the list of all free blocks is maintained in memory. Currently this is being addressed by forcing all disk space allocations to be a multiple of 64. Increasing this number has a significant impact on fragmentation and the number of blocks in the free block list, but is expensive in terms of small blocks. But there are other approaches which can be used. Remember the old buddy allocation scheme where blocks were allocated with a length which was always a power of 2? Fragmentation is avoided, but at a cost increasing disk space requirements by 100%. I am thinking that there is a less costly approach. We only allocate blocks whose size is a Fibonacci Number. This should also avoid fragmentation while increasing disk space requirements by only 60%.

Finally, there is a scalability issue in the pending lists held by the fork blocks. This list can get very large. We need to move these lists out of the fork block and into another database-level table.

COODBMS release 0.18.8

2008-05-23T19:09:00.002+05:30

o Validate now logs numerous counts helpful in assuring the integrity of the database.
o PurgeHistory is now removes all old links and backlinks.
o Backlinks are now correct--fixed a bug in JTransacton.

There will be no subsequent release for the next week--I'm traveling.

--b

COODBMS release 0.18.7

2008-05-21T17:44:00.001+05:30

o loggers have been split out for fine-grain control and the basic disk management events have been added.

o Validate no longer reports a bogus memory leak.

o JUnit internal node configuration has been fixed.

o RunScript now accesses static.cowdb with a query rather than a transaction.

more bugs

2008-05-21T07:32:00.001+05:30

I just realized that there are several places where internal b-tree nodes are transparent where they should not be. Gotta fix this.

good progress today

2008-05-20T20:42:00.003+05:30

The memory leak which occurs after the backlinks table splits are the two new subordinate nodes of the backlinks table root node. This is because validate does not see the subordinate nodes--they are transparent to the normal iterator. Basically this is a bug in validate.

COODBMS release 0.18.6

2008-05-20T08:45:00.000+05:30

o Enhanced disk space management consolidates adjacent blocks sooner and has a new method, notFree, which confirms that a block has not been freed.

o Additional test scripts included in this release.

the new AvailableApp

2008-05-19T09:04:00.002+05:30

The new AvailableApp is working just great and sports a method, notFree, which can be used by Fork to catch that intermittent (and likely debugging-related) database corruption bug just a bit earlier.

One advantage of the new code is that it consolidates adjacent free blocks sooner. Another advantage is that you can check to see if a block has already been freed sooner and more easily.

On the down side, the validate utility is now broken and needs to be fixed. And the new code offers no help in the latest database corruption bug which occurs after a [b-tree] split, as that is a memory leak and can only be detected by the validate utility by walking through all structures.

(The sourceforge site is down as I write this, though the new code has been checked in.)

COODBMS release 0.18.5

2008-05-18T16:50:00.000+05:30

o Reworked PurgeHistory;
o Found/fixed a bug in Rolon.free;
o Browse run command now handles full-line comments; and
o 6 test scripts added.

a rough spot

2008-05-18T16:08:00.002+05:30

There are now two bugs causing database corruption. One occurs when doing debugging, the other occurs when the backlinks table splits. And as the code stands, these are not easy bugs to find.

Database integrity is validated by a separate utility and can not currently be called as needed while processing a transaction. But the logic which manages disk space can be enhanced so we can check db integrity at any time--so long as only one thread is running. Seems to me like a very worthwhile project.

shaken, not stirred

2008-05-17T12:17:00.002+05:30

Just got in. Left Delhi at 4:55 AM on Friday and then spent the next 30 hours being rocked back and forth. I am told that I will need to return again to Delhi in 2+ weeks. Meanwhile I have US and Indian taxes to work on. Life is what you make of it, yes?

So there I was, thinking about 2-time. It finally dawned on me that "it is much simpler than I had been thinking." Indeed, there is nothing further that needs to be implemented. Consider a check register application. Let us say that each check entry is a Rolon and we have a ledger Rolon whose children are the checks--and the check Rolons are named by the Ledger Rolon with a key which begins with the date the check was written, i.e. the effective date. This is a 2-time check register.

The first time axis is operational time. This is the time when a check is entered into the system or the time when the check entry rolon is updated. Operational time is also what we navigate with the "when" command. So we can easily go back to before then last set of entries/changes were added to the system and view the ledger (the children of the ledger rolon) for some past time.

The second time axis is effective time and is the prefix of the keys used to name the check entries in the ledger rolon. We see things in effective time order when we list the children of the ledger rolon. And we can process them in effective order when we iterate through the children of the ledger rolon.

And that's 2-time. Is simple.

off line for a few days

2008-05-11T19:51:00.001+05:30

Heading out to Delhi by train tomorrow morning to apply for a new visa. I'll be back Thursday.

COODBMS release 0.18.4

2008-05-11T19:50:00.000+05:30

o Multiple sessions are now supported.
o Numerous bug fixes have been added.
o Purge history is now working but remains incomplete.

retrodiction, when?

2008-05-10T13:22:00.002+05:30

Retrodiction, the ability to change our perception of past events, is a key concept in Rolonics. Every living thing has this capability and many applications in the insurance industry, to give an example, require this capability. But this capability was not part of AgileWiki3, nor is it a part of COODBMS--these time machines only allow past state to be navigated.

Now depending on how it is implemented, retrodiction can be very expensive. Normally you simply reprocess all activity since the last valid snapshot. But while this may work for an application with well defined scope, it does not work in the general case unless we have well defined worksets--and even then it can be expensive.

Working with virtual snapshots, as COODBMS does, has a similar problem when it comes to retrodiction--how many snapshots need to be created, i.e. which journal entries need to be revisited? Again, it would work ok for an application with very narrow scoping.

On the other hand, deductive inferencing systems, good ones any way (not AgileWiki3) track the dependencies of each deduction and allow changes. (All X were smaller than i, but now we have a new X which is larger than i.) Ah hah! Here we have the basis for a scalable retrodiction system.

But there's still a trick. Retrodiction is about viewing past state and changes over time. If we implement it as part of the inferencing layer, then we need to support views of deductions in 2-time. So for example, if at time t4 we enter a policy with an effective date of t1 and a claim for time t2 is entered at time t3, then we need to conclude at time t3 that the claim is invalid but revise at conclusion at time t4, for at time t4 we see the retrodicted history of having a policy at time t1 and a claim at t2.

So I am thinking we can have a scalable retrodiction system by implementing it in the inference layer. But our inference engine will need to work in 2-time.

Now let us consider a check register. We want to view the checks in the order they were written, not in the order they were entered. And we want to know the account balance when each check was issued. Now we deduce the balance from the balance after the previous activity completed. But my wife and I have a joint account and I am especially bad at entering the checks I have written. So for a given check, the prior check will change over time. This then changes what the account balance was when the check was written and impacts subsequent checks as well. But this is nothing that a good inverence engine can't handle.

Now here I am talking about using inferencing as a means of implementing a scalable retrictive system. The problem is that inferencing systems don't scale. Fortunately this is one issue which was addressed in AgileWiki3. In the semantic layer (namespaces) we define worksets and use those worksets to restrict the scope of any given inference. Bingo!

COODBMS release 0.18.3

2008-05-09T16:10:00.001+05:30

o Bug fix: snapshot backlinks were incorrect.
o Bug fix: links were accessible prior to their creation.