integrity checking

I've added integrity checks to the snap and snapCabinet utilities. This will be included in the next release.

much faster is still slow

I did a quick review of postEvent in wikilib.py and found some pretty bad code. No wonder the Ark felt like something had broken every time you logged in.

Mind, we still have slow updates. And we create a login event with every login attempt. But now in 2.0.15.1, logins are only as slow as similar operations.

a fine first day at FOSS

On the first day the non-commercial exibitors were given a separate area. Traffic was low in this area, but I still had pleanty of folk to talk to who were interested in learning something about the Ark. With any luck, we may get some participation on the compstrm forum.

By the end of the day, plans had changed. We were all moved into the main area, where the commercial exibitors are located. There was room there and a lot more traffic, so why not? So today may be quite interesting.

time off

After releasing 2.0.15.0, I decided to take some time off. Got a book (scifi).

Tuesday is the start of http://FOSS.in and I'm taking 4 days off from work to man a booth there. Perhaps I'll be able to work at the conference, perhaps not. Hopefully I'll be too busy taking about the Ark to get much else done.

its too slow!

The Ark is slow, and I have some ideas for making it faster. But logging in is especially slow. So I want to take some time out and review that code.

I also want to look at the snap logic and see if it can correct (drop) badly formed structures.

more on tables

Sometime tomorrow, 2.0.15--extending table descriptor units--should be completed and released as a beta. Table descriptor units will each have a Columns table with two columns: column name and a brief description.

So what's next? How about an addrow command that uses the Columns table to generate a HTML form? Could be fun! :-)

name magic explained

I just wanted to do a quick review of name magic--which begins with automaticly determining the DescriptorUnit (read "type") to associate with a newly created Topic.

Lets say we are in some Topic (other than the Data Topic). This current topic, which is where we are about to create a subordinate topic, has an associated DescriptorUnit--either the du was explicitly assigned or a default du is used.

Now if the DescriptorUnit has a NewTopicNameMap table (LSec), then when you create a supordinate topic, the new topic name is applied to the NewTopicNameMap to determine which du to assign.

For example, in the Drawer DescriptorUnits, we (will) have the following NewTopicNameMap:

Topic, DescriptorUnit
---------------------
*TableDescriptorUnit, TableDescriptorUnitDescriptorUnit
*DescriptorUnit, DefaultDescriptorUnitDescriptorUnit

Now when we create a folder called MyTableDescriptorUnit in Drawer DescriptorUnits, the crf command knows enought to give it the TableDescriptorUnitDescriptorUnit as its du. Then it knows that MyTableDescriptorUnit is a du for a table and creates a Columns table.

So once we've defined all the descriptorunits and properly configured the Ark, all subsequent [type] assignments are handled automaticly, as long as the users follow the naming convention.

Could prove interesting.

TableDescriptorUnitDescriptorUnit

I'll confess, I always enjoyed playing with things like the class class. I've just finished writing the specs for 2.0.15 http://www.compstrm.org/wiki/uuid/EQ7UHDZ4oq9pdO12+NFgCR96 and now we have both DefaultDescriptorUnitDescriptorUnit and TableDescriptorUnitDescriptorUnit.

The nice thing here is that the meta knowledge is defined in Topic structures, rather than interwoven into the code. And it takes a very small driver, once all the structures are created.

I think we're nearing the end of the ugly duckling stage. 2.0.15 is still pretty heavy for the small functionality it gives you, but I think we will see this quickly reverse, with small incriments in structure/specs will yield a whole lot of capability. And if we do it right, that capability can be used to make the Ark easier to use, and perhaps even make it easier to learn.

TableDescriptorUnits

In 2.0.14, we are now automaticly generating the NewTopicNameMap LSec when a DescriptorUnit is created. Consider that a sample of things to come.

Now when we create a table, it would be quite nice if we also populated the first line of the table containing the column names. So we need to put that information into the table's DescriptorUnit. I'll observe that the DescriptorUnits of tables are different from other DescriptorUnits. Ipso facto, table DescriptorUnits should have their own DescriptorUnit--lets call it TableDescriptorUnitDescriptorUnit. We'll also want a name pattern for TableDescriptorUnits--lets say "*TableDescriptorUnit".

Now, how do we recognize a table DescriptorUnitDescriptorUnit? --By the value of the descriptorunit header! Rather than have a value of "True", the descriptorunit header of a TableDescriptorUnitDescriptorUnit can be set to "Table". This opens the doors to some new capabilities:

• Automatic generation of the textcontent header in the table DescriptorUnit,
• Automatic generation of a Columns LSec in the table DescriptorUnit and
  
• Inhibit the generation of the NewTopicNameMap.
  

no more mkdu

With the release of 2.0.13.0, you will rarely need to use the mkdu command.

In the DescriptorUnits drawer, you just create a topic with a name ending in "DescritorUnit", and "Prest-O Change-O", it is recognized as a DescriptorUnit and the DescriptorUnit tag has been assigned.

OK, so its not a lot of flash considering all the plumbing. (Need more flash powder, next time.) But remember that most of that plumbing is pretty generic.

one step towards an Ark that is easier to use

Well, 2.0.12.1 is out. The gencal utility has been updated and the crc command has been revamped.

Be sure to rerun "gencal 05" and to rerun crc on all cabinets other than cal and CompStrm.
(This is spelled out on the getting started page in the data migration section.)

So what's next? How about:

1. A restrict header in DescriptorUnits for a bit more flexibility.
2. A DescriptorUnit header in DescriptorUnits of DescriptorUnits. Now when creating a DescriptorUnit, the du command can be run automagicly.
3. An LSec in DescriptorUnits which gives the column names to be used when an LSec table is created. Great for when you're creating a lot of tables with the same columns.

Small improvements, but I suspect there will be a cumulative effect on ease of use. Eventually we should have forms for adding a row to a table. (One small step at a time, OK?)

a turn for the better

Well, setup magic has just become crc magic. The crc command no longer creates duplicate cabinets, but refreshes it. Which is to say it adds the standard DescriptorUnits if they are missing or out-of-date. (This is a lot like the way gencal works when rerun for the same year.)

So we don't need a setup utility--just recreate your existing cabinets in place. (No data should be lost.)

Meanwhile, I've been having fun refactoring code, moving code from gencal to role and simplifying webi.

The DescriptorUnit identifier

The value of the DescriptorUnit tag is what distinguishes a DescriptorUnit from others, possibly with the same name. The post command uses this, least moving a topic from one Cabinet to another result in the assignment of the wrong DescriptorUnit.

Now the mkdu command sets the DescriptorUnit tag to a nice unique id. And that could be quite important when dealing with multiple arks, multiple communities, and a need to uniquely identify the right DescriptorUnits.

But there's a flip side here. Some DescriptorUnits are the same for all cabinets and all arks. I.E. the DefaultTopicDescriptorUnit. So why not set the DescriptorUnit tag to the topic name in these cases--this will help post get it right!

And yes, this is an issue for the new setup utility.

jotspot

http://www.jot.com/index.php

Wiki applications, now there's an idea! Email support, tagging, access control. And a whole lot more that you will not find in CompStrm. Looks very cool!

setup magic: delay is not good

There are a number of enhancements that I'd like to add to DescriptorUnits to make things easier to use, and these will require changes in the new setup utility, so the temptation is to further delay the initial implementation of the setup utility. But I've come to realize that this is NOT a good idea.

Without proper administration, all these new features to make things easier for (non-administrative) users simply don't work. And right now it is just too much of a pain to manually do all the setup. So the setup utility should be done sooner rather than later.

This means of course that I'll be making minor changes to the setup utility (which creates as many DescriptorUnits as might be helpful) with almost every enhancement to DescriptorUnits. Oh well. At least they should be minor changes!

reordering intentions

I'm delaying setup magic a bit, as there are now a few items which will impact its implementation. Case in point, replacing the newtopicdescriptorunit header with a much more powerful NewTopicNameMap LSec. Version 2.0.11 then becomes NewTopicNameMap.

NewTopicNameMap should help make DescriptorUnits a bit more transparent to the user.

name magic for DescriptorUnit selection

I realy do want the Ark to be easy to use--it should "just work". And the hope here is that DescriptorUnits have enought potential power that this is doable.

Now we really don't want a user assigning DescriptorUnits to the Topics they create--that's exposing too much complexity. One thought then is to assign DescriptorUnits when a Topic is created based on its name. This can be controled via an LSec Table in the DescriptorUnit of the parent Topic.

The LSec would have two columns, TopicName and DescriptorUnit, where the TopicName column contains a regex and the DescriptorUnit column would name a DescriptorUnit.

The cr_ command would then match the name of the new topic to determine the DescriptorUnit.

Hmm. A third column, Role, could also be used to restrict creation to a particular group of users for a given topic name pattern.

changing intentions

I've now written up (briefly) the imatch specs under the intentions page at www.compstrm.org. I've also split setup magic and user Folders into separate versions--the specs for user Folders may not be stable yet.

I've also been thinking about how the calendar should be secured. Yes, you could easily use the restrict command (it will take some time) to restrict write access to the entire calendar. But that's a rather large hammer. What if you want to allow updates to days and weeks, but not to months? What about preventing the creation/destruction of days? This is a big topic and I'm not even going to try to address it all in one go.

Now I like the new logic in the create commands, where we copy the parent's restrict tag onto the new child. But I think there is also a role here to be played by the du command. Lets define a new header in DescriptorUnits, "restrict". This header would name a role. And when du is used to assign a DescriptorUnit to a Topic, if the restrict header is present in that DescriptorUnit, then the restrict tag on the topic would be assigned the role named by that restrict header.

We can then also modify the restrict command to check for the presense of the restrict header in the DescriptorUnit of each topic it encounters and apply the header when present. (And propagate the new value down to subtopics as appropriate.) Of course, we'll want similar behavior in gencal and the new ark setup utilities.

This will give us a bit more flexibility in managing write-access, but will only come into play when there is a restrict header in a DescriptorUnit.

Any comments?

But first, a few minor improvements; imatch

Before launching into 2.0.11 proper, I'm taking time out for a few minor enhancements:

• The chnmCmd now redisplays the Topic's content.
• Words like McBride, which should not be treated as WikiNames, can now be enclosed in {{double braces}}.
• All regex's used in the Ark have been extended to support "¦". Ex: "*BiLaf*¦*NoKas*" will now match strings containing either BiLaf or NoKas.

These changes have all been completed and will be included in the first release of 2.0.11.

There is also another change that I've discussed with Norm, but its a bit more than minor: imatch. (See below.) I expect imatch will be implemented in a separate version.

Currently what TagMatch does is add any matching topics to the list of visible topics. (I.E. They show up when you use the t command.) TagMatches of this kind are created using the match command from the cls display.

An alternate form of TagMatch will be created using the imatch command. This new kind of TagMatch will include any matching topics to the list of applicative topics. This is kind of like having wildcards in the pathname of an include, except that it works faster and uses global scope for matching.

Please remember when using tags that they are global across all cabinets. Use long names for tags. Topic names can be short because they are scoped based on structure, but tag names are not scoped and must be unique. (I've broken this rule in www.compstrm.org in several instances, unfortunately. Case in point, use of the desc tag on days in the calendar. Such tags are of limited use for tag matching.)

setup magic; user folders

Its time! Ark administration needs a bit of magic. And it is about time for every user to get their own Folder, which only they can update. Check out the new specs, 2.0.11:

http://www.compstrm.org/wiki/uuid/2zQLfkmGVeR1VHnnGnTtVZqm

more slides

I've just written a set of slides on the Ark's Data Security.

See compstrm.sourceforge.net/security.pdf

The Users Table

Norm concurs that the Users table needs to be external to the Cabinets. So we'll need an Ark Cabinet to hold this (and other) data.

The User table should now likely be 3 columns: user,cabinet,role and while we're at it, lets make the cabinet column a regex. Here then is a sample in csv form that I'll probably use on compstrm.org:

user,cabinet,role
BillLaForge,*,admin

a booth at FOSS.IN

If you're in Bangalore the end of this month, stop by the FOSS conference. I'm taking 4 day's vacation from Sun and will be spending the time manning a booth, giving demos of the Ark.

FOSS = Free and Open Source Software

Progress on write access

You may have noticed that 2.0.10 is now in alpha. Hopefully it will be in beta sometime this weekend.

I'm thinking now that after 2.0.11 (custom and config roles), its time to redo the GUI a bit. I'd like to list the applicable commands on the left side of the display, along with off-site links and the like. A narrow column, but it will reduce the width available for editing and content display somewhat.

Meanwhile, I'll note that there is an OSS conference here in Bangalore the end of this month: foss.in

priorities

2.0.10, limiting write access, is coming along nicely. But gosh, it would be nice to be adding other features which use descriptors besides access control.

On the other hand, the custom and config roles will help organize commands. Could make the system easier to learn if I build the right documentation around it.

I'll note also that we need private topics before we can implement 2.1.

mmm... Maybe I can just push back a bit on the custom and config roles. There is a real issue with creating a topic and then setting the descriptor unit. I'd like see a create operation where you first name the descriptor unit (or let it default). A form could then be displayed for topic name, usage (Drawer, ..., LSec), and tags.

Or perhaps support customized queries? (Perhaps this one should wait until after the custom role.)

Descriptor Units bring a lot of power to the Ark. Guess I just want to show it off a bit. Access control has real value, but its not very flashy. ;-)

Cabinet Classifiers

I've been thinking about moving/sharing Cabinets between Arks and it occurs to me that different Arks will have different users with different role assignments in their various Cabinets.

So it makes sense to move the Users LSec, which is really classifier data for the Cabinet, out of the Cabinet itself. Perhaps we can create another Cabinet (which will typically not be shared) to hold Cabinet Classifier data as well as daemon descriptors (like email configuration data, which is currently in the wiki.tac file).

Well, if I do the API right, I can address this later.

started writing code for access control :-)

As I refined the specs for access control, the specs for remarks shrank considerably--a good sign. So I've folded them into the specs for limiting write access.

I've also included specs for a new command, restrict, which would populate the restrict tag across Topic sub-trees. That should take a lot of the pain out of using the restrict tag.

Meanwhile I've started writing the code this morning, building an API to make it all easy. It is going well. Finally!

changing access control, again

One thing has made me uneasy--having a restrict header in the DescriptorUnit which names the group that has write-access. This is Classifier information and may vary from Topic to Topic with no other change to the DescriptorUnit. Wrong!

So lets make restrict a Tag (which is type of Classifier) on the Topic instead. And for convenience, when a Topic is created we just copy the Tag from the parent, when present.

Now I'm happy. :-)

access control--more to come

I've reviewed the spec for limiting write access (still 2.0.10), and made only a few minor changes:

http://www.compstrm.org/wiki/uuid/TTsWzQipFgyV9bJv92bgwenX

One remaining item is specs for Remarks. You should be able to create a Remark even on a write-protected Topic. And only the author should be able to change the content!

limiting updates

I'm pushing the config and custom roles (was 2.0.10) back to 2.0.11, as I think I've finally figured out a simple way of limiting write access. The (new) 2.0.10 spec simply defines a restrict header in DescriptorUnits which names the role required to update Topics which use that DescriptorUnit.

Now why did it take me so long to come up with this?

http://www.compstrm.org/wiki/uuid/TTsWzQipFgyV9bJv92bgwenX

admin, config and custom roles

Release 2.0.9.0 is out, and its beta. This is a worth while release, as it effectively limits the use of header commands to DescriptorUnits where they are needed.

So I've also been working on the specs for 2.0.10. I've beefed it up a bit--it now defines admin, config and custom roles. Basicly a user assigned the role of admin in a Cabinet can do anything in that Cabinet.

The job of a user assigned the role of config is to define DescriptorUnits, while the job of a custom user is to customize the classifiers of the Topics in a Cabinet, as needed. Ordinary users then only see commands applicable to building content.

Note that even with 2.0.10, we will not have effective access control--any user can change the CabinetDescriptorUnit and assign themselves the admin role. (This will likely change in 2.0.11.) Rather, for now we are doing little more than restricting the visibility of commands to roles where they are applicable.

Of course, the Ark doesn't really look simpler to the user until the documentation becomes role-based.

The Ark is an ugly duckling!

Access controls are getting closer. I just wrote the specs for 2.0.10, which limits some operations on DescriptorUnits to users with the config role. I'll note that this is the shortest set of access control specs that I've written to date, and pretty easy to implement. Of course, I've still got to implement 2.0.9, first.

Version 2.0.8 and 2.0.9 are important, as we are now further limiting the applicable command set. This reduces the apparent complexity of the Ark to a new user. Unfortunately, it also adds complexity. I feel like I just grafted a long neck onto a baby duck. Its an ugly duckling, but hopefully it will mature to become a graceful swan.

The Ark is now officially in the ugly duckling phase of development!

DescriptorUnit complications

With the release of 2.0.8, working with LEnts just got a bit more complicated, as the LSec holding the LEnts now requires an appropriate DescriptorUnit.

The upcoming 2.0.9 restricts the use of headers, which is fine except that you want to be able to use headers in DescriptorUnits. As a header is used to allow the use of headers, it becomes a bit of a catch-22 situation. Of course, you could always post the initial DescriptorUnits from CompStrm or cal.

Rather than require such tricks, perhaps it would be best to always allow headers in DescriptorUnits.

cal is now fully typed

Big holiday here, so I got the day off from Sun. Had a root canal done (oh joy!). But at least the price was right--US$30.

In reworking gencal, just about everything gets a DescriptorUnit--29 DescriptorUnits in all. Right now there isn't a lot of benifit to doing that. Yes, I'm playing a bit here. But as the capabilities goverened by DescriptorUnits grows, this becomes much more worth while. But yes, there's still quite a ways to go yet.

gmail labels

Yesterday, Thomas pointed out that gmail allows users to attach labels to emails. This is a lot like tagging emails in the Ark. And of course, for every label there is a virtual folder which is the equivalent of a TagMatch on a Topic. Cool stuff.

Thanks Thomas!

changes to d, f and p commands

In addition to being able to match based on inherited tags, it makes sense for the d, f and p commands to list all the inherited tags when there are no name$ arguments.

This may make for a lot of columns. Here's my thoughts on that:

• I expect that multiple tags will occur most often at the Page level, and the fewest at the Cabinet level, as higher levels are likely more focused within a single realm of knowledge while lower levels will have greater diversity. So inherited tags typically might add only a few additional columns.
  
• You can always use name$ arguments to select the interesting columns. When command macros are added (an obvious DescriptorUnit feature), this becomes quite reasonable.
• We can enhance the display so that the less commonly used tags (for a particular query) are combined in a single final column, the (4?) most popular tags each having their own column.

Inherited tags

Release 2.0.6 is out, moving us one step closer to a typed system of topics. At least now the post command is a bit more DescriptorUnit-friendly. This is going to be a long, dragged-out process, so I'm happy to intersperse some other functunality--like TagMatches--which have immediate utility. On the other hand, I'm quite happy to have made some progress on DescriptorUnits. It's been like a black cloud hanging over my head.

Next up is 2.0.7--inherited tags. There is a real need for this, as otherwise tags need to be replicated to subordinate topics to be able to do intersection searches.

My first thought was to have 4 new commands, it, dit, fit and pit. The it command would list the inherited tags, while dit, fit and pit would work very much like d, f and p, except for allowing matches on the tags of ancestor topics.

My current thinking is to avoid having more commands, enhance the Drawer, Folder and Page displays to include inherited tags, and just move the added functionality into the d, f and p commands. This way we avoid some of the complexity, and the added functionality shouldn't interfere with the normal usage of d, f and p.

Slides: EMailOverload

The cal.pdf picture has been replaced with a complet set of slides:
http://compstrm.sourceforge.net/EMailOverload.pdf

So now its back to working with code. :-)