structures, usage namespaces, promotion/demotion

Well, I've done (and released) all the structures now. Usages are:

• Ark
• Cabinet
• Drawer
• File
• Page
• Ledger and
• Remark.

Each usage has its own color (but I probably need better colors).

So, next is usage namespaces. When you select a topic displayed by the ledger or remarks commands, for example, you don't want the option of a page by the same name.

Also, a ccp command (works like cc, but restricted to matching with a page) would be handy. And if I'm in a LedgerEntry, a ccp with no arguments should take me to the Page the LedgerEntry is a part of. (Or the Folder, Drawer, Cabinet or Ark the LedgerEntry is a part of.)

My expectation is that names will generally be short, and resolved by context. So being able to further restrict a selection by usage should be handy.

After that I'd like to look into promotion/demotion. Promotion would convert a LedgerEntry to a Page or a Drawer to a Cabinet.

A bit further out, I should be able to post (copy) a LedgerEntry/Page/Remark/etc to another part of the Ark.

Backfilling a bit, I still need a move command to move things between Cabinets. (Should also serve as a shortcut for add/rm.)

Still so much to do to wrap up CompStrm1.

All in a morning

I finished Cabinets today. Also fixed the cc bug and disallowed arguments on the jnl command. Still a few bugs to work on before I want to release. Perhaps tomorrow some time.

Oh, and did you catch Norm's latest blog entry?

Bugs all over the place

Last night IST / this morning EDT, Norm gave the Wiki a first trial run while I was on the phone with him. Ran into 4 or more bugs, including things I had only just tested a few releases back. He even found 2 ways to crash it.

My sincerest thanks, Norm. --Bill

Sharing Cabinets

It occurred to me last night that I can restrict Roles (specificly Drawers, Folders and Pages) from being under (in) more than one Cabinet at the same time. Then I can easily support the sharing of Cabinets--the edge conditions are eliminated.

Cabinets also seems to be a good level for restricting access. So I'm inclined to disallow Pages, Folders and Drawers at the Ark level as well.

I'll note that sharing remains out-of-scope for CompStrm1--due to problems with list and destroy operations. But I might as well put these restrictions into CompStrm1, rather than try to impose them later.

Ark, Folders and Pages

Data is now an Ark (usage).

Folders are now also a distinct usage from page.

A page does not have subordinate pages.

The ark and Folders can have subordinate Folders and Pages.

One thing is wrong, I'm thinking. In the Ark display, folders and pages should be listed separately. (Once I add cabinets, folders will not have subfolders.)

Another observation: By unfolding pages into ark, cabinets, folders and pages, its getting to be a lot more like type rather than usage. ;-(

non-recursive structures

I've been thinking about the advantages of having pages, remarks and ledgers. And I've been thinking about a conversation Norm and I had several years ago about having only three layers of realms.

Simply put, if everything is a page, its hard to find the page you want.

So pages are in folders, folders are in drawers and drawers are in cabinets. OK, we add some flexibility by also allowing pages in drawers and by allowing pages and folders in cabinets. Now you can look at the pages, folders, drawers or cabinets visible from the current location. And you allow promotion/demotion: a page can become a folder (promotion) or a cabinet (a 1-drawer cabinet) can become a drawer (demotion).

This is a much more "natural" system, easier for folk to learn and use.

Remarks

DataSets now have two uses: Pages and Remarks.

New terms:

• One page can be under another, as a sub-topic;
• A remark can be on a page or on another remark.

The topics command does not normally list remarks, except when the current context is a remark.

The command set for pages and remarks differ, somewhat.

Rolonicly, a Remark would be a non-performing journal entry.

alpha, comments and ledgers

Bad migration proceedures are forcing CompStrm-1 back to alpha.

Meanwhile, comments and ledgers are coming along. (I'd rather have stayed in beta!)

sharing domains

Most, if not all the issues I have with sharing have to do with edge conditions. In previous implementations this was avoided by sharing at the realm level, where realms were non-overlapping collections of roles.

We can do the same in CompStrm. Under the root, Data, we can define various domains, each with its own access and sharing rules. Then no more edge conditions--a page may not be under two pages in different domains. Applicative context and references can still cross domains, so we will not loose much.

Feeling better!

What's wrong with Destroy!

A comment is attached to only one thing. Likewise a ledger. But a page may be a sub-topic under more than one other page. So there are no side-effects when I destroy a comment or a ledger.

But consider a situation where in my ark, the page Swimming is under Exercise, but in your ark the same page is under both Exercise and FunActivities.

Now if I destroy Swimming and share my updates for Exercise, then on your ark Swimming is destroyed--so it is no longer under FunActivities. Oops! this may not be a good thing.

Alternatively, supose I just remove Swimming from Exercise and let Swimming be garbage collected. Ater sharing, you still have swimming under FunActivities.

The reverse is interesting too. Lets say you remove Swimming from Exercise, share the updates, then reinstate Swimming under Exercise and share the updates a second time.

Now Swimming gets garbage collected on my system after the first update and I'm not sure at all about what happens when the second update is received. Maybe Swimming should not have been garbage collected?

I like the implicit destruction of a page when it is not under any other page. Otherwise I have zombies which get included in queries.

Perhaps I need a tks-level mechanism for reinstatement. ;-(

Anyway, sharing isn't in CompStrm-1 any more. I've obviously got a lot of issues to deal with here.

Balancing Applicative and Structural Context

Before starting CompStrm, I had been working on an Ark for Norm which lacked deep structures, but which had extensive Applicative Context capabilities. So even without much structural context we could build up various contexts in which the roles operated.

In contrast, the CompStrm1 Wiki supports complex structural context, with no applicative context at present and only a very simple mechanism planned for release this weekend. Which is the better approach?

There's no question in my mind that when it comes to crafting a context, structural context is easy to use but simply incomplete. Structure implies more of a relationship than simply usage. (In this role I want to be able to refer to those roles, but there is no other relationship between those roles and this one.) So you need to add some explicit applicative context.

In converse, you can build any arbitrary context using an explicit applicative context--it is complete. But then, you are left doing all the heavy lifting. (It is explicit.) So I like the mix, with an emphasis on structural context.

My thought here is that perhaps the CompStrm1 Wiki will not need applicitive context mechanisms that were as powerful as I had previously implemented, and that the result will be more natural/intuitive. As I said to Norm last night, "using it will change the way you work."

What is a System?

Catch the latest blog entry at http://ontologist.blogspot.com/ where Norm addresses the key flaw in modern software architectures.

public hosting of CompStrm1 Wiki?

With the CompStrm1 Wiki back in beta, it would be great if we could get it hosted on the internet. I for one would be glad to add some instructional pages for new users. Anyone?

forum

I'd like to remind folks (Frank, you too) that I would prefer that you post to the CompStrm forum at http://sourceforge.net/forum/forum.php?forum_id=367038 rather than email me directly (though that's really OK, too). Note that the link to CompStrm Form is on my blog page, listed under links.

citations, includes and excludes

The cit command lists all pages where the current page (or the name given as an argument) are referenced. A very handy command when renaming or moving a page.

The cit command now also lists pages which explicitly include or exclude the current page.

In the process of updating the cit command, I found/fixed a minor bug. ;-)

At this point, all the easy code for includes and excludes is done. Next step is to update the visibility and referencing logic--the fun stuff!

ledgers--partness

Ledgers are another potential use of DataSet objects. They differ from pages only slightly, so it is important to clarify when ledgers and when sub-pages should be used. The distinction is all about partness.

Pages are whole things. A page is never a part of another page, but can be "in" more than one page--think of a page as having a collection of sub-topics, as a sub-topic can be in more than one collection.

In contrast, a ledger is a part of a page or a part of another ledger. Unlike pages, ledgers are organized into pure tree structures, with a page being the root. But like pages, comments can also be on a ledger. (Comments can be on/about anything!)

The page display lists pages it is in, comments, ledgers and sub-parts--making the structural relationships of the page visible.

And visible ledgers? They are not listed in the topics page, but rather in a ledgers page. When the context is a page, the top-level ledgers of the pages it is in (recursively) are visible. And when in a ledger, the child ledgers of the ledger it is in (recursively) are also visible.

One big gain here is that the ledgers, though referenceable from other pages, do not cluter the topics page. Between this and the structural (and applicative) restrictions on the visibility of topics, keeping the list of visible topics to a reasonable subset of the entire Wiki should be a managable task. And that is a very big benifit indeed.

Oh yes, nothing new on syntax. Qualified page names and ledger names all look the same. (Qualified comment names are not allowed.)

comments

In a traditional Wiki, comments are to be added to the end of a page. But in the CompStrm-1 Wiki we are separating content from structure. So we should be able to attach comments to a page. Note however that comments are distinct from sub-pages.

Now lets say that a Wiki Page is one use of a DataSet. A comment would be another use. But the rules are different for how these DataSet object can interact with each other.

First, a comment can not be a sub-page in another page. Nor can a page be a comment on another page. However, a comment can be on another comment.

A comment can not be referenced by any page except within the page it is on, nor is it visible except in the list of comments. But a comment can reference another comment on the same page or comment.

The content of the comments then are quite distinct from the content of the page they are on. Further, comments do not clutter the list of visible topics.

usage vs type, progress on includes/excludes

I had said that I would not be introducing page types (i.e. application types) into CompStrm-1. Well, I've been thinking that usage is orthogonal to types and there are things I can do with usage.

The idea here is that a DataSet can be used for more than just pages, specificly for comments (non-performing journal entries) and ledgers. (More on these two new uses in subsequent blogs.)

The idea here is that type defines wholeness (descriptor), while usage defines partness (classifier). So different uses of a given type may give rise to different rules about how an object relates to other objects that are not a part of it, while type controls how the various parts of an object work together. Specificly, for operations which effect how an object works with other objects, usage will dictate applicability (in the simplest case) and or might even define such operations.

Now type is something that does not change in the lifetime of an object. But usage can change. A ledger can become a page (promotion), or a page may become a ledger in another page (demotion).

Meanwhile progress has been good on includes and excludes. The commands work (inc, exc, rmc) and the page display now lists includes and excludes. Next I want to update the cit command to list include/exclude citations. But I still need to update topics and name resolution to use includes and excludes, so the fun part still needs to be done.

But before moving the CompStrm1 Wiki from development to maintenance, I am seriously considering adding comments and ledgers.

includes and excludes

Includes and excludes will be sets of (optionally qualified) names.

Pages named in the excludes will be used as visible pages (topics) when resolving references.

Pages named in excludes will be excluded from topics and will not be used when resolving references.

Includes are "inherited" by sub-pages, but can be effected by "closer" includes and excludes.

Should be fun.

beta

With sharing no longer a part of CompStrm1, I'd say that we are definately in beta. And with today's release of the order command for arranging subtopics, CompStrm1 is nearing completion.

And while I also want to have includes and excludes inCompStrm1, I am also inclinded to give testing and cleanup a bit more emphasis. For example, with an improved sort for subtopics, I'm thinking that the same sort logic should be used with topics. And does the cr command allow the creation of pages with spaces in the topic names?

no sharing in CompStrm1

The ability to share datasets between Wikis is a very useful capability, especially when one of your computers is a laptop. Unfortunately destroy operations, and especially list operations, break this capability as implemented in CompStrm1. The problem is that dump/load/restore are implimented at the tks level, even though the code is in tkcs. The .dmp files contain the changes (which for list operations is the entire revised list) rather than intents. This breaks data integrity when not all datasets are on all computers or when only selected datasets are being shared. (It fails in edge conditions.)

So sharing is not going to be part of CompStrm1. ;-( On the other hand, that means that CompStrm1 is very close to being finished. ;-) I'm currently working on the order command, which is used to order an unsorted list of subtopics. Then there are includes and excludes (applicitive context) to work on. And that's it.

compstrm1, sub-topics

I've changed the top-level directory to compstrm1. The idea is to allow for maintenance releases of this first-generation wiki and start development of the second generation. Hmm. That's still a ways off, methinks.

SubTopics are now, by default, listed in chronological order. If you want to sort them, use the sort command. (Then the subtopics for that page are sorted when it is displayed.) If you want to turn sorting off, use unsort.

The next step then is to allow the user to arrange subtopics. Perhaps tomorrow?

0515a and beyond

I just wanted to note that release 0515a is not backward compatible.

Also, while sub-topics are now lists, they are still displayed in alphabetical order.

Next step is to allow explict control over subtopic order.

more rolonics: structures and streams

Catch Norm's latest bog entry at http://ontologist.blogspot.com/2005/04/streams-and-structures.html

some progress with list, CompStrm1 vs CompStrm2

Almost everything is working again. The two exceptions are the add and rm commands.

I'm thinking of changing the compstrm directory to compstrm1. The first "product" then would be the CompStrm1 Wiki. This would be my first step in forking the code base, CompStrm2 then being the start of a type-based Wiki.

In CompStrm1, there is limited support for the todo, relations and phonebook examples. By adding support for different types of pages, I can begin customizing the displays and command sets by type and then add better support for these examples.

Now once list is working, things will begin to stabalize. Adding includes (applicative context) will be an addition, so it should not break backward compatibility. Of course, I may still need to change the dump file format when I fix load--but that remains to be seen.

some progress

OK, I've got the examples working again.

The problem with the wiki is that I need to call some api (for lists) which does not exist. SoI need to refactor some tkcs operations to create the api.

not a good morning

The todo example is working, but not much else.

Probably some small bug in relations and phonebook. But before I can fix the Wiki, I need to refactor some tkcs-level code. ;-(

typed wiki, the next generation

I'm thinking that there is only so far I can go with the current wiki without introducing different types of pages. That seems to be a natural break between generations.

I can still do ordering/arranging of subpages, and even introduce a simple include mechanism (Applicative Context) to suppliment the structural context. I might even spend time on page formatting. ;-/

There is just a whole lot more that can be done once page types are introduced. So it seems logical to stabalize this wiki, implementing everything that does not depend on type.

Some progress on lists of sub-topics

DataSets now have a list of WellKnowns, rather than a set.

Everything seems to be working, though I found some unrelated problems in the phonebook example. (Still to be fixed.)

So far, the change to a list is transparent to the Wiki. Next thing is to turn sub-topic sorting on and off on a per-page basis. And then add some arrangement commands.

second thoughts

Ah, the technology calls.

I don't think I'll be driving the current Wiki all the way to production. It should be fairly usable and stable, but I'll not support conversion to advanced versions.

You should be able to use it, once its stabalized. A nice set of training wheels, as it were. We need to explore what you can do with it, and what not.

My thought is that this wiki should

• be effective in documenting a system in flux;
• be a bit friendlier than a standard wiki for group efforts;
• be able to handle a diverse, but interrelated topics;
• with some structuring, be able to restrict the number of visible topics to a significant, and related, subset of all topics in the wiki; and
• support sharing pages and collections of pages between personal wikis.

Perhaps that's enough. ;-)

Cleaning up and reorganizing

Well, I've rearranged the web pages in release 0514e. Didn't do any writing, but even so it should be a lot easier for a newbie now to install CompStrm--less intimidating anyway.

Changed the web package to be the wiki package. Purged ds.py, moving what little code that is still used into webi.py.

Three things are left:

1. better documentation for the wiki,
2. ordered/arranged sub-pages and
3. fix load.

reorganizing blues

Woke up this morning and realized that I had broken crark.bat. ;-(

Fixd it. Tested it by doing a release/install. Repeated until I got it right.

Compstrm Wiki now a major focus

I've been thinking about putting a major effort into the CompStrm Wiki.

1. First, some cleanup--especially web pages.
2. Implement sub-page ordering. This will/should/must be the final change to the existing data structures.
3. Fix dump file loading, especially command ids and file destroys. (The wiki command rm, like de, also needs to be handled at a higher level.)
4. Enhance page formating.
5. Port to Solaris. (Which means it is likely that it will also work on anything except perhaps a Mac.)

The idea is to drive the CompStrm Wiki to beta or even production, and release it as a version 1.0. Then fork the development, with all but bug fixes going into a new version.

My thought is that the existing technology, in the form of a Wiki, is good enough or better. And at this point significant benifits can be gained by making the Wiki usable, rather than producing an endless stream of technology enhancements.

command cleanup, ordered sub-topics?

Today's release, 0514c, cleaned up the rm, chnm and de commands. Can I now say that the CompStrm Wiki is user friendly? (Well, perhaps just a little more than it was before.)

I'm also thinking of making subordinate pages a list rather than a set. (The DataSetItem property then becomes the index to a file list property.) Then the list of sub-topics displayed on a page could be arranged by the user(s).

Maintaining an ordered list of topics can always be done on the wiki page itself. But with a heirarchical structure, it doesn't seem right to have to maintain both the structure and the list on the page.

At least by having a list instead of a set, the list will always be accurate. (New sub-topics can always be added to the end.)

some polish?

I'm thinking that it would be good not to move on too quickly.

I want to check the de and chnm commands--they should not apply to the Data "page".

I'm also looking at the summary page, which talks about a Wiki, and the home page, which talks about an anything box. I think the emphasis should now be on the Wiki.

Maybe change the web package to be the wiki package.

I think there is some dead (old wiki) code in ds.py that should be purged.

And when a page is displayed, I'd like to also list the subordinate pages, if any. Note that this would be a subset of the pages named by the topics command, which lists all "visible" pages.

Oh yes, you did notice the new release today?

rls 0514b
The CompStrm Wiki has been dropped and the web interface is now named "The CompStrm Wiki".

To further add to the confusion, Wiki pages have also been dropped, and DataSets--at theuser level, anyway--are now called Wiki Pages.

The Wiki Commands have also been renamed a bit:

cc: Display the content of a given topic. The
two optional arguments are topic and time.

ch: Changes the content of a page.

chnm: Changes the topic name for a given page.

cit: Displays the Citations for a given
topic. The optional argument is a topic or URL.

cr: Create a new page.

ct: Change or clear the time context. The
optional argument is time.

de: Destroy a Topic.

add: Add to another page. The one required
argument is the topic this page is being
added to.

rm: Remove from another page. The one required
argument is the topic this page is being added to.

help: Displays this list.

jnl: Displays the Journal for a given topic.
The two optional arguments are topic and time.

topics: List all topics for a given time
visible from this page.

short vs long term goals

The web access has gotten to be a nice wiki. Future intentions are grand and glorious, but I'm thinking that it would benifit tremendously from a bit of cleanup, even at the risk of sub-optimizing from the perspective of those long term goals.

Case in point: pages & data sets?

Why have both? OK, yes, there are likely some long-term reasons. But for right now, a dataset does everything a page does, and more. So lets drop the pages for now. Datasets can be used as pages, so why not use datasets for everything? (The Data dataset being a minor exception here, as it is not the same kind of dataset as all the rest.)

So the crpage and crds commands become just cr, for now. And while I'm at it, chtnm should just be chnm.

So the short term goal is to make Web into a nice wiki, and nothing more than a nice wiki (with a built-in time machine, structural context, qualified references, et all). OK, it should be a really super wiki. But its a really short term goal, and mostly to do with documentation. (I.E. I'm likely not going to add extended formatting for now.)

Then I need to focus for a while on .dmp files. If the CompStrm Web is to be a great wiki, then we gotta get out of Alpha! And more than .dmp files in the narrow sense, I want to fix things so sharing works well. (Gosh, it would be nice to browse someone's wiki and download a .dmp file of a selected dataset. ;-)

After that, the focus will probably shift to making it more than a wiki. Like private, internal and external dataset members, explicit context (includes), perspectives, type-specific displays and other commands, ledgers, etc.

cit command

With the 0514a release, the web interface is now usable as a wiki. So it is time for a bit of cleanup--I want to drop the old wiki code and reorganize some things.

(I should note that 0514a fixed the cit command and that there are still problems with load. We remain in alpha.)

citations

If pages AaAa and BbBb both have a reference to CcCc, then the citations of page CcCc are AaAa and BbBb. This is what we want to achieve.

However, it is a bit more complicated because

• there may be more than one page named CcCc,
• CcCc might not be referencable from every page and
• a reference may use a qualified name, like MyDS/CcCc.

So, how do we determine if page AaAa is actually referencing the same CcCc page that is our current context? We just need to make sure that the current page is referencable from page AaAa.

Handling qualified names also isn't too hard, given a big enough hammer! We just calculate every possible qualified name the current page might be referenced by and check for them.

Load modifies CommandID

I figured out that I like what I've done, locally, and to make dump/load work, all I need to do is to have load change the command id.

The Command Id is the timestamp of when the command started. Load just needs to map all occurrances of the same command id to the time when processing started on that command. It may be just that simple, or I may need to sort things by originating timestamp (if I don't already). In any case, the solution is now isolated to load processing and it can wait until I focus on dump/load/restore again. (OK, I've also got to update the navigating through time tutorial--its using a UUID rather than a timestamp.)

Command files, dumps and data set item properties

I've been thinking a lot about internal structures. I'm thinking now that everything is almost right and that I should only make minor enhancements.

The disruptive item is commands. When sharing realms, I've decided that it is OK if not all the transactions/operations of a command are included. However, I will need a command file which is then referenced by transaction files. As it will be implemented at such a low level, it needs to be done with care. So the creation time of the command file will be the start of the command in the journal.

Oh! Hmm. If a command's transactions are split into two different dump files which are then loaded at different times, then the command really becomes two commands, as the two sets of transactions each have different start/end times.

Looks like I still have some thinking to do. ;-) I think I'll stick to plan and work on citations next.

maintenance commands done and released

Not a bad day--got out 2 releases.

Release 0513b contains the maintenance commands crds, dsadd and dsrm. Now you can create and reorganize complex structures of data sets.

Just be sure to "run wiki/install.tki" in the TKI shell before trying to use the web interface.

Well, that's it for today. Next is working on citations, which may take a while as I want to handle qualified names. After that, its time to go back and clean up the internal structures. ;-(

one step forward, two steps back

Well, I've been reflecting on the new Journal and my conclusion is that it is a very good thing that we are in Alpha!

Dumps/loads/restores are all broken.

The new structure is fragile. I need to make command a file and move some transaction data to it. As a cludge, I'm using timestamps to identify the transactions associated with a command. This breaks on load/restore and is risky to share as well.

Further, unless commands are broken up, you can't dump a dataset without dumping them all. The property on datasets which references the well knowns that are in it needs to be inverted.

My conclusion is to just say dumps are broken, ignore the low-level stuff, and focus on look & feel for a while. We'll stay alpha for some time. I want to add some maintenance commands first. Then fix citations. Then work on dumps.

I'll note that the reason why I put the property connecting datasets to wellknowns was because it was the right answer for a previous implementation--I had done it the other way initially had had to reverse it. Oh well.

In any case, I still intend to do a release today (Sunday).

journal, release, citations

Well, I've finally finished the Journal, and very happy with the result.

The web command, jnl, now lists only user-level commands. It also allows you to navigate to either when the command started (but before any changes were made) or to when the command completed. This is ever so much nicer than only being able to navigate to when an internal operation completed--now the state is consistent across files.

I expect that I'll be doing a release tomorrow. Still a little cleanup and testing to do.

Next on the list will be citation processing. Then I've got to enhance the command set so you can move pages between data sets and create new data sets. Ledgers (secondary user objects) are on hold.

journal

The journal in TKS is at a very low level--it is the raw stream of property changes associated with a file. This is very differrent from a Rolonic Journal.

In its simplest form, a Rolonic Journal is a sequence of informing or performing Journal Entries. Informing JEs are just notes and perhaps documents posted "For Your Information". Performing JEs on the other hand are user commands which effect some change in a Role or Ledger. Sharing a Journal with another Ark then allows the changes effecting a Role or Ledger to be shared.

The first problem encountered with the TKS journal is that sharing has its drawbacks--activities like destruction may require more operations on the destination than at the source, as the destination may have more references to the item being destroyed.

The current effort in CompStrm is focused on grouping the low-level changes which relate to a particular user command. This makes the Journal much more comprehendable.