A picture

Here's a picture which shows how a date gets resolved: http://compstrm.sourceforge.net/cal.pdf

OK, its really only one slide. But its a start!

only a few more paragraphs

High-level documentation comes slowly for me. So far I've only managed a few more paragraphs at compstrm.sourceforge.net.

As for another slide presentation, I've been thinking of focusing on using the Ark to manage email. I can then use the presentation at work to explain what the Ark is good for. In the department where I work we get a lot of emails that contain a wealth of information, but there are so many that trying to organize them all would be quite time consuming.

better colors for the UI

Rupali returned from her parents home yesterday. The train was a few hours late, so she arrived at 4AM. (Its a 36 hour train ride.) Needless to say, I didn't sleep well. We had to go grocery shopping--the pantry was quite empty, and then enjoyed a rather long nap in the afternoon. So all in all, I didn't get to spend too much time working on the Ark.

But I did manage to do a few things:

• I changed the colors in the new UI to various shades of grey, to prevent clashes with other colors. I also made the grey a bit lighter to ease readability, along with some other minor changes. Its now been posted on www.compstrm.org as a sneak preview.
• Added a new introductory paragraph on the compstrm home page, compstrm.sourceforge.net.
• Updated the project summary description on Sourceforge: An agile tool for managing both knowledge structures and information streams. A rich environment supporting diverse relationships between topics, name scoping, time navigation, tables, email and binary data.

I still want to do some more writing, as well as a new set of slides, before getting back to the code.

2.0.5 is complete, a long chat with Norm

Version 2.0.5 is done, but there were some substantial changes, and likely some things got slower. Before moving on, I want to take some time to understand the deep implications of the changes that have been made. I also want to review some of the high-level documentation and perhaps work up a better set of slides. The capabilities of version 2.0.5 are significant enough that they need to be included.

On the UI, which I thought was so very fine, Norm says it is better but will take him some time to get used to. And it is blocky. Oh well, at least it is better.

I was talking with Norm about adding citations to the Topic display and he said not to do it--in a well connected structure, there can be LOTS of citations. Keep them separate!

Norm also made a request for queries which use inherited tags. These would behave like the d, f and p commands except that matching would be allowed on inherited tags as well. Currently to achieve this he is duplicating folder tags on the child pages, so there is a real need here.

sneak preview

I've put the new UI on www.compstrm.org. All I did was reformat the top part of the page but, it makes a huge difference. I'm sure you will like it. The Ark is starting to look like a product. :-)

making citations visible topics

In a eariler version of the Ark written for Norm, citations were included in the list of visible topics. This made the Ark much more organic, as a reference by a topic to a second topic made the first topic visible to the second. But in the current version of the Ark, citations are "hidden"--they are only displayed when you use the cit command.

Now if we make citations visible topics, then citations to the current topic should be included in its display. This all sounds good to me, but I'm thinking that I'd like to make some progress on descriptors before taking another side trip.

release 2.0.5.2

This is perhaps one of the most significant releases in a long while, as it makes it much easier to work with unstructured data. I plan to make good use of this capability to manage all my work-related email.

I did change the order of items displayed however. Children are grouped together rather than intermixed with topic matches, named topics and included topics. This was done because otherwise the next and prev commands would have become far less intuitive.

good progress on rewriting the Topic display

I am very close to finishing the new Topic display. Includes, names and matches are all listed together with the Topic's children. I am also happy with the transparency it is bringing to the Ark. This should help a lot in making the Ark more usable.

Now if only I could improve the UI. The top of the display is just too cluttered. Perhaps if I added some borders to visualy organize it. ;-)

a display order issue

By extending the topic display to include includes, names and tag matches as well as children, we must deal with how to order Pages and LSecs when they are not being sorted. The answer is to list all the children first, then the includes, followed by names and then tag matches.

I expect that the second item in each row will identify the relationship between the current topic and the topic being listed.

I suspect things may be particularly interesting when listing LSecs, which can be nested.

But the more I reflect on this, the more convinced I am that this is an important change and should be included in 2.0.5. It should greatly enhance the value of tag matches and names.

immediate context

With 2.0.5 now in beta, I've been thinking about its impact on the utility of the standard topic display.

Currently we only display the children of a topic. I'm thinking instead that it should display the immediate context (in contrast to the complete context displayed by the t command). Which is to say that it should display the names and the tag matches.

Wouldn't it be nice, for example, when viewing a month to see the days of the month without having to use the cls command?

early alpha release 2.0.5.0

This early alpha release does NOT add matching Topics to the list of VisibleTopics. It just lays the groundwork. The documentation has been updated, as well as the cls display and the post command. And the match command has been implemented.

So if you want to see the documentation (CompStrm Cabinet) or look at the code, then by all means, download it!

The next release should be much more interesting, both from the perspective of added utility and in terms of the difficulty of the code changes. After that, everything else is just frosting on the cake.

started work on 2.0.5

It was a pleasure to get back to code, though with dirty dishes to wash, laundry and unpacking, I didn't get too much more done than the match command (previously tagmatch).

But it went well. Next step is to enhance the cls display to show the matches, and then (the critical piece) to update the logic for resolving topics and for listing visible topics.

I expect 2.0.5 will be in beta pretty quickly--perhaps by this weekend.

Back, if not up-to-speed

I arrived back in India last night and made it to work this morning, though I'm not yet up-to-speed.

I'll also note that next week is a 2-day work week due to various holidays here. So I get a 4-day weekend followed by a 3-day weekend. My hope/desire is to spend some of that time writing code for the Ark. ;-)

Meanwhile, Norm has experienced some pain. After using the Ark intensly for a day, he deleted all but the last 2 log files (my standard practice) prior to migrating to his laptop. Subsequently he was unable to restart the wiki server. Ouch!

The www.compstrm.org server has been stable. Of course, this means I have not captured anything on why it hung last week. ;-(

almost home

I'm on my way back home. 8 hour layover in Hong Kong. The wierd thing is that the blogger web site is localized to chinese characters. Not helpful.

Looking forward to writing code for the Ark. I've been thinking over Descriptors in general and everything seems fine. One thought I've had is to allow references to descriptors in the CompStrm Cabinet--it would prevent a lot of needless duplication.

Enjoying California

I am very much enjoying the climate here in California, though many of my coworkers are finding it a bit cold, especially mornings/evenings.

Talks all day and into the evening--we're kept pretty well occupied. But I did manage to go shopping yestersay afternoon instead of going to the picnic. And Saturday is a free day.

No great thoughts here. Just looking forward to working on the Ark again. The break has been refreshing, but don't expect anything for a week or so--I'll need to recover from the trip and likely catch up on other things.

tag matching and ApplicativeContex

To date, the Ark is really great for information which is reasonably well structured, with tags and the sCmd the only means of identifying content which was not so well structured.

Meanwhile, Norm has been using tags extensively to classify much of the content he is adding to his own Ark, and has suggested a significant enhancement that would give the Ark a tremendous capability for working with unstructured (but tagged) content. (Yes, I'm excited!)

The idea is to add tag matching to the ApplicativeContext. VisibleTopics then would include all the appropriatly tagged Topics. This gets around the need to structure everything. For many things, simply adding the appropriate tags will be good enough.

posting vs DescriptorUnits

A Topic must always use a DescriptorUnit in the same Cabinet. This makes it easy to move Cabinets between Arks. But a copy of a Topic can be posted to another Cabinet. Access controls complicate things--the user requesting the post may not have permission to create a new DescriptorUnit in the destination Cabinet. Indeed, what we really want to do is to use the corrisponding DescriptorUnit in the destination Cabinet.

The flip side is that we probably want to allow DescriptorUnits to be reorganized and renamed. So we can't just match by Cabinet-relative pathname, its too restrictive. Instead, we want to establish a trans-Cabinet DescriptorUnit identity. And a Tag is looking like a great way to do this. The post operation can then look for a DescriptorUnit in the destination Cabinet which has the same value.

A few problems remain. First, we may change a DescriptorUnit enough that we want to change its identity--it is no longer the same DescriptorUnit that is in the other Cabinets. So we need to be able to change the identity.

Second, when we use post to make a copy of the DescriptorUnit in the same Cabinet, we probably want the new copy to have a different identity.

Third is the question of what to do when there is no matching DescriptorUnit. Currently, the post operation just drops the DescriptorUnit when doing a cross-Cabinet post. For now, we can continue this practice in this case.

As for the Tag name, well, we can just us DescriptorUnit. Currently this Tag takes only the value True. All we need to do is to provide a command which sets it to a unique identifier.

breaking up is hard to do

Well, what started out as 2.0.4--restricting header and lsec operations--has now become 3 releases:

• 2.0.4 splits header and text content operations
• 2.0.5 changes how LEnt content is recognized and restricts access to LEnt commands
• 2.0.6 restricts access to header commands.

Unfortunately, the CompStrm Cabinet now reflects 2.0.6. :-( Call it programming by intention--I've gotten into the habit of updating the documentation first.

At least 2.0.4 is in beta. I'm installing it now on www.compstrm.org.

Things are looking Up!

It looks like 2.0.4 is going to be a nice enhancement to the Ark, and a big step in actually making use of DescriptorUnits. The catch is that LSecs with LEnts must be manually converted by assigning the appropriate DescriptorUnit.

I've also submitted my Visa extension application. I've been told to check back in about a month.

Today's a holiday, which is great 'cause I've been stressing over that visa application. Time to kick back a bit.

I also got my plane ticket to California. Leaving late Friday night. Next week I'm NOT working on the Ark. (I've probably gotten a bit stale, working on the Ark so long without a break!)

Where do we go from here?

I've been rather busy lately working on my visa extension. I'm close, unless the officer comes up with additional paperwork to be submitted, again. But I've no room for complaint when I consider what it takes to get an employment visa in the US.

This Friday I'm leaving for California, and will be there for a bit more than a week. Its MDE Week at Sun Microsystems, and all of MDE from around the globe gets together. I will not be working on CompStrm during the trip. (I probably need some time off, anyway.)

As for access controls, I really think the cart has gotten in front of the horse. Before working any more on access controls, the Ark should be making better use of DescriptorUnits.

It is DescriptorUnits which we have no experience with, though we already have several descriptors: sortls, sortp and the csv=LEnts header. The latter is a good case in point. An LSec which holds LEnts should have a DescriptorUnit which declares this, not a header. And the LEnt-related commands should not be applicable to LSecs which do not hold LEnts.

I've also been thinking about headers. Some topics should allow header updates, and others should not--and that should, again, be declared in the DescriptorUnit. Then the header commands will only be applicable to topics which allow header updates.

One benifit here is that we can reduce the number of applicable commands for most topics, and that should make the Ark a bit less confusing.

3 steps backward, it seems

Last night I was working on the specs for what I thought was going to be a simple access control system. But it got complex fast. I think there are some underlying issues that I need to address first. But more to the point, I'm thinking I need to put DescriptorUnits to good use BEFORE working any more on access control. Why have a big fancy lock on an empty house???

After retiring (in India), Norm then spent the day (in the US) trying to use the Ark. He ran into some problems (reported on the forum). I did the quick fix to 2.0.3, but I want to replace the ch command so we don't always have these non-header header problems.

So lets toss the schedule. Its time to admidt that I need to work on basics before I can actually project a schedule. The difficulties I am facing here are all due to the fact that version 2 is quite different from version 1, and I still need to come to grips with that.

doing small steps

Gosh, 2.0.3--cabinet-level access control--went quick. I've just inserted a new release, 2.0.4--write access control--that should go just as quick.

In 2.0.4, some areas of a Cabinet are Tagged as Public, and any logged-in user can make changes. For the remainder of the Cabinet, only users with a role of admin can make changes.

This is a pretty simple approach that seems to have a lot of value.

access control priorities

Having finished 2.0.2, its time to start working on Access Contol in earnest. But this is a biggie, and it might be reasonable to intermix other projects--access control does not by itself add to much value.

I think cabinet-level controls are a good place to start. Don't want someone cluttering up the cabinet namespace, mm? It seems reasonable to have this entirely in the domain of the AdMin user.

The second priority is to restrict updates of selected things. Specificly, it would be nice to restrict who could change the content of a DescriptorUnit.

A third priority might be to limit read-access to selected items. This is vital for when users become Topics. We will also want to restrict read-access of who has what privileges.

Fourth, it would be great if we could limit the commands available to most users in most cabinets, just to keep things easier to learn. Ultimately, it would be great if a user could self-declare their skill level and make some commands accessible on that basis.

Finally, being able to have private cabinets adds a lot of value to the Ark. Not everything needs to be available for public consumption! This would restrict read-access of content as well as restricting the use of s, t and other commands to view Topic names and Tags.

All this is going to take a while. Frankly, I'd love to be playing with other uses of Descriptors besides access control.

Can we make the Ark simpler?

Headers now have much more limited uses than they had in version 1 of the Ark. They are used to describe email content, to indicate that an LSec contains LEnts, and in DescriptorUnits (where, I expect, they may be used heavily).

One thought then is to indicate in a Topic's DescriptorUnit the applicability of Headers. We can also replace the existing ch command with two commands, one for editing headers and the other for editing TextContent. Now most user will not need to know what a Header is.

new strategy on access control

Access control is easy to get wrong. And if it is wrong, its a real mess. It is unlikely that I can get it right all in one go, and the specs still have not stabalized.

So I'm going to try implementing access control in easy stages. The first step is securing the Ark-level. Only AdMin can create Cabinets, post Cabinets and rename Cabinets.

The second step will be to secure the Cabinet-level. A Cabinet with a tag of "Private" can not be viewable or in any way accessible except to users known to the Cabinet. I think I can do this with reasonable speed, though t, s and other commands will be a bit slower.

I'm not sure yet how we can split up the rest of access control at this point, but perhaps breaking out these two steps will be a good start.

What cost is reasonable for Security?

I am concerned about the overhead of adding security, especially when it comes to the citations display and other queries where a list of Topics is generated.

I am also concerned about the impact to content management if you can not view all the citations of a Topic.

So I am thinking that, if you want a truely private Cabinet, it should probably be in a private Ark. Access controls then should be limited to the ability to make various kinds of changes, and to viewing content. And citations, Tags and Topic names should always be public. Otherwise the cost of access controls gets pretty high very fast.

3 conclusions

I had a delightful chat last night, and then some intense IM (Yahoo) with Norm this morning. He findes the 4-number versioning I've proposed acceptable. (He had not liked the mixed versioning system I have been using to date in version 2.) So I will be using the 4-number system, outlined in my previous blog, going forward.

I've also been mulling over tag references and the new du property. And I've reached a few conclusions. The first is that I like the default DescriptorUnits as I've outlined for 2.0.2. But when the DescriptorUnit for a Topic is set explicitly, since am now restricting the selection to a DescriptorUnit (Topic) in the same Cabinet, then the property should be a non-symbolic (UUID) reference. This will add some speed exactly where it will be needed most.

The other conclusion I've reached deals with references by headers and tags, though tags may not contain the '{' or '}' characters. References and citations should otherwise work the same for both, but when a header or tag value contains one or more '/' characters, the value should also be treated as a reference. (Currently only when a Header value contains a ProperWikiName or when all or some of the value is enclosed in braces is the value checked for references.)

thoughts on version numbers

I'm thinking of going to a 4-number versioning system:

1. The first number is the generation. Here I am thinking that Cabinet sharing and Cabinet proxies should be moved to generation 3. That way, g2 focuses on service configuration while g3 focuses on the web of Arks.
2. The second number reflects a change in data structure, typically requiring a conversion.
3. The third number is tied to a set of, often related, intentions.
4. And then the fourth number identifies a particular release.

The first two numbers would be reflected in both the top-level directory and the .dmp file names; the last two numbers then being the extension to identify a particular release.

Comments?

Can I work on 2.0.2 now?

This conversion has been difficult. I released 2.0.1-0539b, figuring I was done and in the process of doing the release, a significant bug shows up. I've now fixed that bug in release 2.0.1-0539c and installed the release on www.compstrm.org.

Is this conversion over yet??? Norm just told me that he finished running the conversion on his Ark, so I am hopeful. I'm looking forward to working on 2.0.2 and 2.0.3, descriptors and access controls. If I can get through these next two versions, that really opens up a lot of applications to the Ark.

searching

The new s command now lets you search the entire Ark by Topic name and Tag values.

The logic is not very fast--there's a lot of room for improvement--but it seems to be pretty good even so.

Now, for example, you can search for an email that has been filed in the wrong place.

Note that you can even do intersection searches on the same Tag. Consider the command "s subject=*friend* subject=*earth*". This would match on the subject "The friends of earth" and on the subject "The earth has no friends here".

(Intersection searches also work with t, c, d, f, p, ls and le commands.)