access controls

I've revamped the intention for access controls. See:

http://www.compstrm.org/wiki/uuid/ms6ad8Pnf+n3R4-u0X58ut+q

www.compstrm.org is now stable

I just confirmed that slurp, the Yahoo web crawler, is still hitting www.compstrm.org. In spite of this, the site is stable. A welcome change, and due to the bug fix in 2.0.0. (Currently running 2.0.1-0539a)

lookin' good!

With 2.0.1 in beta, the problems introduced by 2.0.0 are past us. (I say this with fingers crossed.) And I'm happy with the result, though there has been an increase in complexity. (What, with headers, tags and LEnts having 3 distinct types of name/value pairs!)

So I'm looking at the enhancements remaining in 2.0.1. And looking at tags in particular.

Can a tag value be a reference? I don't want to identify which values are references and which are not. Nor do I want to introduce scope resolution--remember that with the new s command, tags are matched across the entire Ark, not just from within a context.

So I'm suggesting something simple for tag value reference identification--if the value names a Topic anywhere in the Ark, then it is a reference. And tag value citations would be added to the cit display.

So if there is a xx-x Topic, then a tag value set to same should, when displayed, be clickable.

Where I hesitate is that this adds even more complexity, as TextContent and Header value references follow very different rules. but it seems reasonable.

For example, if I have a tag value which is a user name, anywhere in the Ark, these references should be accessible. It seems OK for tags to have different rules--they're classifiers, after all.

Now, what about path names? Well, if references are not resolved from within a context, then path names don't seem to make too much sense.

What are the alternatives?

1. don't consider tag values as references--as implemented now, or
2. use the full context resolution mechanism as we do for header values and TextContent.

Comments?

progress on the www.compstrm.org bug

I've been unable to reproduce this problem on my laptop. But after monitoring www.compstrm.org with a "tail -f twistd.log", it looks like the Yahoo web crawler is causing problems when attempting to access an outdated UUID--something I probably didn't test. :-(

more good news

The cvt seems to be working. Next (likely tomorrow) I'll release it as an alpha--it becomes beta after I've updated the CompStrm Cabinet.

yet another small milestone :-)

I'm happy to report that I just finished testing the conversion of email from 1.7 to 2.0.0. Turned out, it was a lot easier than I thought it would be.

Next, of course, is to work out the conversion logic for LEnts, which changed quite a bit in 2.0.0.

a small milestone

Start simple!

I created a very simple-minded routine to convert from 1.7 to 2.0.0. And it works! Well enough, anyway, to handle the converstion of the CompStrm Cabinet.

The next step, of course, is to start dealing with those messy details relating to email headers.

1.7 bug; 2.0.0 beta 2

The 2.0.0 beta 2 is out. LEnts have been reworked--they are no longer Topics.

• Use lents to set up a LSec for LEnts, then
• Use ch to add/change/remove LEnts from the text content of that LSec.

LEnts should now be very much faster to work with. And yes, if you include several LSecs, then the le command can display the consolidated set of LEnts.

Meanwhile, I'm starting to get a handle on that bug in 1.7 that I reported in the CompStrm forum.

notes on release 2.0.0-0538a

This is the alpha1 release of 2.0.0. Some things to keep in mind:

1. Tags are fully implemented, but LEnts have not been reimplemented.
2. The conversion utility for data migration from 1.7 to 2.0.0 has not yet been written.
3. There is no release content (Cabinet CompStrm).

All of these issues will be addressed in the beta release.

bug found, templates

Well I found the problem I was having snaping/restoring tags. As I expected, the tag dictionary was missing the Internal attribute, so its contents were not being saved. So I'm retesting now. If all goes well, then I'll release 2.0.0 alpha. :D

Meanwhile I've been thinking about templates. Instead of creating something from scratch, its often easier to copy and modify. Or, in Rolonic terms, post and change. But I'm also thinking that the collection wherein we are creating a new Topic might suggest a default template--in its DescriptorUnit, of course. And likely there would be different templates suggested depending on the usage--one for LSecs, one for Pages, etc.

Failing a suggested template in the DescriptorUnit, we may further want to have Cabinet- or Ark-level templates. Of course, for creating Cabinets, we would only have the Ark-level template.

This all seems especially handy for initializing the DescriptorUnit of the new Topic. Guess I need to work it into the schedule.

some debugging remains

I finished updating gencal and everything looked great! All ready for the release of 2.0.0 alpha1!

But then I did a snap/restore. Woops! All the tags are gone!

So I think some work remains, even before the first alpha release. :-(

But methinks I'm getting close. ;-)

Slippage

I was out sick a few days this month, but the impact on the Ark was for more like 2 weeks.

Bad code in the refactored email processing delayed things further.

Add to this the addition of the reimplimentation of LEnts to 2.0.0, and you can see why I've slipped the schedule by a Month.

On the other hand, a Sun-related trip to the US in mid-October will likely have a 2-week impact on the Ark as well.

For the revised schedule, please see
http://www.compstrm.org/wiki/uuid/TRZsvhqmYwiArfBDE6V6bUrJ

(Note that the above topic now also details out the expectations for what will be included in version 2.)

email is working, faster

Email is working fine now, finally. Looks like I was writing code on a bad day when I should have been avoiding the computer--it took a while to find all the bugs.

I'm especially happy to report, again, that a long list of emails (84 today) displays quite quickly. So its fast enough to be usable, whereas it wasn't before.

I guess the next thing to work on is updating calendar.

some progress on 2.0.0 email this AM

Got back to the code, finally. Reviewed the 2.0.0 email logic, found some problems and made fixes. Still want to do a bit more cleanup, and then its time to test, test, test. But I think I'm pretty close.

I have a feeling that the conversion logic is going to be just as tough, as it must be able to convert old emails. :-(

renumbering those intentions again!

Everything I've layed out for future releases should be called version 2.-, not version 3 or version 4. Perhaps I should use a 3-level version number, the second number indicating when a conversion is required, rather than the first number.

My "Essays on Arkness", then, all apply to version 2.

I also don't like the way 2.0 (now 2.0.0?) degrades performance of LEnts. So I'd like to rearrange things so that this release actually enhances LEnt performance. I.E. LEnts will no longer be Topics. Otherwise no one, including myself, will want to put it into production.

configuration management

Once you introduce configurable services, configuration management becomes an issue. This is where, I believe, the Ark will have a significant advantage.

http://www.compstrm.org/wiki/uuid/w7YhBRG8N9cG5fpYZrUpXYpw

paints, canvas, brushes do not make a painting

I am the plumber of the Ark. My focus is on making it work and making it usable. So my vision of the Ark is one of how the various parts need to work together. The best I've been able to come up with to describe what the Ark is intended to become is a platform for configurable services:

http://www.compstrm.org/wiki/uuid/v5Via5fQX6qr5qpkxcZ5DrYs

Norm's perspective is considerably different. He knows what he wants to be able to do, and the Ark is a tool that might just make it easier for him to work. With every capability that I add to the ark, he has usage patterns and practices which have been waiting for it.

So I speak of a platform for configurable services, while Norm speaks of a collection of interrelated services which enable usage patterns quite different from how we normally expect to be interacting with a computer.

pounding on email

Normally I don't usually have any problems when refactoring code. (Though I remember back in 1980 when I first was able to edit code using a monitor instead of punched cards--refactoring was a tough nut then. Experience helps a lot!) But the email code has been giving me problems.

Its now working for most cases (which is quite irritating). So I got a book (sf) and took a break yesterday. I need to change my testing methodology and start again fresh. The code was a bit more fragile than I had thought.

On a more positive note, I am happy to report that a display of 50 emails is significantly faster, which is important, as it was just too slow before. Some of the speedup is from not having to read the body of all those emails, and some is from not checking tag values for WikiNames.

AccessControl

Turns out (long discussion with Norm) that AccessControl is implimented using both classifiers and descriptors. Owner is a classifier. Defining user groups is handled by classifiers. Identifying the accessible commands is done in the descriptor.

So it looks like I can still use a Topic as a DescriptorUnit. I just need to define more classifiers. (Classifiers seem to pop up everywhere. This is because a classifier is specific to a type of relationship.)

See http://www.compstrm.org/wiki/uuid/ms6ad8Pnf+n3R4-u0X58ut+q for details.

But what is a Descriptor Unit?

Gosh, I thought I knew. The DescriptorUnit defines behavior. So its kinda like the type of Topic. And that probably works for a lot of things, but now I'm looking at access controls.

Is the "owner" of a topic a descriptor? I think so.

Now access controls may be based on type in many cases. For example, only the owner of a Remark can change it.

So, perhaps I just need to specify the owner and a "common" DescriptorUnit. I'm not sure.

I think know Norm's answer--every Topic should likely have its own DU.

happy with tags, unhappy with LEnts

I LIKE tags the way they are now--there is a potential there for speeding up queries and other processing. I've decided that its the implimentation of LEnts that I don't like.

LEnts should not be Topics. Rather, the text content of a LSec should be a csv file with a header identifying it as such. We then eliminate the chlec command and just use ch. And it will be faster than chlec ever was.

very unhappy with tags as implimented

The problem with tags as now implemented is that they are fully indexed and yet, the indexing is not used. Better to create a document and stuff them all there, one line per tag.This way they will be no slower than before.

The objective here is to gain some speed (especially when processing emails where the body of the email was included with the tags) and to distinguish between tags (classifier) and headers (ledger).

back on track with 2.0

I'm happy to report that I've fixed the design flaw that I previously reported and retested everything. Things are just a bit faster than before, except chlsc, which is now so slow as to be unusable. Oh well, its mostly queries that tags participate in and slow update is the cost.

(At some point I really need to work on speeding up updates--there's a lot there that could be done.)

a chicken or egg problem?

The problem with adding descriptors is that you really want them under access control. And to implement access control, you really want descriptors.

So I'm thinking of decoupling access controls and making users first-class objects (Topics). Converting users to Topics can wait. But before we do too much with descriptors, we need access controls.

Then the first thing we will find in a Descriptor unit is perhaps a list of who can make changes to a Topic and who can change the DescriptorUnit of a Topic. The rest can wait.

Two steps forward, one back

Ouch! TKCS dictionaries accept only UUIDs for values, not strings.

So now I've got to go back and re-implement tags. :-(

Well, at least the bad code was unreleased.

got started on 2.0

Implemented the tag command: "tag name value"
or: "tag name". One command does it all--like name.

I am now listing tag name=value in a fourth column at the top of the page (under the help line).

And the t/c/d/f/etc. commands are now listing tags rather than headers.

A good start.

Adding a User View

Interesting day. Got a chance to use TWiki at work. The attraction there seems to be (among many other things) a view that can be customized for each site.

And yes, I think we need a user view for the Ark. See http://www.compstrm.org/wiki/uuid/8ET8fpIfmddiRF3eVStARi-L for my thoughts on that.

bug speculation is a bad habit

Found the problem with snap. My timestamp logic assumes Python time has a granularity of only 1/256th of a second--this is not true when running under Solaris. So it was just a porting bug.

(Fixed, undergoing final tests but still unreleased.)

too fast a machine?

I'm having trouble running snap/restore on my desktop at the office--a Blade2500 with dual SPARC processors and 2GB RAM.

It occurs to me that a lot of my logic depends on having unique timestamps. I do have a small supplimental counter for when clock granularity is too small. But its a SMALL counter. Perhaps it overflowed.

(Python gives you a granularity of only 1/256th of a second.)

Cabinets are Important

Feeling a bit better today, but still caughing a lot. And now my wife is down with it.

I did manage to get the (final?) release of 1.7 out today, which means the next step is to start version 2. But do try out the improved help line at www.compstrm.org. I think its a big improvement.

I also want to do some essays on Arkness. Here's the first one, "Cabinets are Important": http://www.compstrm.org/wiki/uuid/izJa8UMsjk+j2oZo5AoPp14Z

The ups and downs of the day

This is a national holiday in India, Lord Ganish's birthday. So I had the chance to get a lot done on the Ark.

Came down with a cold, had a miserable night.

Still I got the snap and snap cabinet utilities completed and fixed the bugs that I had reported. A bit more cleanup is left on the Unix scripts and then I'm ready to release. Perhaps tomorrow.

With snap you will see a substantial reduction in file sizes and in the time needed to do a restore. I am very happy with the result and looking forward to applying it to www.compstrm.org and also to the Arks at work.

Thomas came for a visit

Thomas is a very dear friend, an ex-employee of mine, and a co-worker at Sun. He comes to visit from time to time, and this was one of those special nights.

I had a chance to review the CompStrm slides with him, and I now find them quite disapointing. I've also been looking over past blog entries and find there is a general lack of continuity. Simply too much is left unsaid. And then, of course, there is the documentation in general. Perhaps obscure is too kind a word. But there is hope.

As work on the Ark progresses, it is slowly becoming more consistent with Rolonic theory. And it becomes easier and easier to show what I've been leading up to all along. I'm not working on a jazzed up Wiki, but on something new. Its called an Ark. And I am starting to understand something about what an Ark is and what it enables you to do.

new schedule, code review

Yesterday I briefly spec'ed out a number of major versions: v2--DescriptorUnits, v3--Journals and v4--access control. Each of these will involve a conversion.

I've also streached out the schedule. Part of the problem is that I'll be spending a week in October in California, plus travel time from India, and I don't expect to be writing any code at that time. I also want to allow time for the maturing of each of these versions, as v3 and v4 in particular have very brief and incomplete descriptions of the work to be done.

As much as possible, I want to move major features to V3 or V4, as each of the conversions will open up important capabilities. So, for example, email filters (2.3) will likely be delayed unless there is an immediate pressing need.

This morning I took the opportunity to review the code for post, dump and load. This is all working code and it looks good, if somewhat obscure. While doing the code review, I've worked out how to snap the current state of an Ark out to a .dmp file. This will be pretty straight forward, and without the need to make any changes to load/restore. So I expect the development of 1.7 to be complete very soon now.

1.8 becomes 2.0

I'm thinking that 1.7 is a good place to close out version 1. Once it is complete, then on to version 2.

There are several reasons for this. Next on deck is a data conversion, with potential destabalization hazards. And then we start defining Topic types and adding type-specific logic, making the Ark more of an application server than a Wiki. But once we have snapdump capability, version 1 is complete and sustainable.

So 1.8 becomes 2.0. (I've made a few other changes as well.)

lots of dumps/restores

Using the new utilities, I've been able to shrink all the Arks I'm maintaining (currently 4). I'm especially happy that the log files for the one dropped from 540MB to 260MB, largely by dropping and then regenerating the calendar. And this now includes the CompStrm Cabinet, where before it did not. (Of course, by regenerating the calendar, I dropped a LOT of unwanted emails which I had already posted elsewhere.)

The down side is that the weekend is over, and I've written no code since 1.7-0535c. :-(

fixing the problem

I really do dislike having to fall back to Alpha. Its also generally difficult to fix a bug which only occurs on especially large datasets. But I may just be lucky this time.

I suspect that the problem has to do with restore not creating the Ark Topic. I've now added a check to restore to create it if needed, and I'm reloading now. The reload will take a few hours.

Wish me luck!

dump utilities, then header conversion

I think it might be best to split off the header conversion into a separate version, with 1.7 now tightly focused on the new dump capabilities. The (new) 1.8 would then deal almost entirely with header conversion.

Getting 1.7 with the new dump capabilities into beta and shipped will be helpful for both Norm and I, as we are managing moderately large Arks. Also, I want these utilities stable well before releasing header conversion.