sharing vs .dmp files (Security Issues)

TKS (Time Knowledge Store) is the backend of TKCS, and keeps a transaction log of property changes, rather than property values, where each transaction deals with the changes for a single file (TKS record). Using these changes and related change indexes, TKS can quickly calculate the value(s) of a property for any given time.

TKCS dump files (.dmp) are little more than a collection of transactions. Restoring a .dmp file is done by reprocessing the transactions to update the TKS indexes. But there is virtually no validation, so .dmp files are useful only when the source is trusted.

Dump file loading is almost good enough for distributing applications (scripts and metadata), that really can't be validated very effectively in any case. I would only add a mechanism to restrict a load operation to files in selected directories, these directories being declared in the file header and viewable before starting the load.

But will this be good enough for sharing data between users? Here the element of trust may be entirely lacking. How well can we validate a .dmp file? I have a mixed answer.

We can restrict load processing to specific directories, as mentioned above. And TKCS has been carefully designed to clearly separate metadata from ordinary data, as metadata changes made by an untrused source are far more risky. However, a user might use TKI to make changes that an application can't handle. And there is currently in TKCS no validation of state. So sharing data by way of .dmp files could result in strange data that causes unexpected behavior. Is this good enough?

I think it is good enough. Scripts and other logic must be prepared for bad data, which can be caused by any number of causes, including bugs. But it is still a long way from perfect. Not every user will be able to use TKI to clean up bad data. But then, you can always dump the good data and restore it to a new database OR an advanced user can share a .dmp file that fixes the problem.

Now a better solution might be to share user-level commands rather than low-level changes. These commands can be validated as they are executed. Data fixes can still be handled by sharing .dmp files with a trusted user, of course. But this all starts to get a bit messy when the data is build by a combination of user commands, TKI commands, conversion scripts and shared .dmp files.

My conclusion here is that .dmp files are a reasonable, if not perfect, means of sharing data between users--so long as we can restrict updates to selected directories.