About Tag

Fish Fixes and setuptools

2012-01-25T22:39:00.003Z

I have a few pieces of good news. And one piece of not-so-good news.

I’ve pushed Fish 4.33 to Github. This version includes two main bug fixes and some useful reorganization. The bug fixes are:
One test was failing under Python 2.7 (but not 2.6 or 2.5) That turns out to be because I was lazily doing some json encoding “by hand” and failing to percent-encode non-ASCII characters. httplib in the earlier versions didn’t seem to mind, but in Python 2.7 it does. I am now using a proper json serializer and things are better.
It transpires that starting Fish interactively was failing for new users. This is because the interactive version of Fish (the one you get if you just type fish <return>) was doing a sync, and that sync assumes that you have a tag called .fish/alias. New users tend not to have that. Obviously, the solution is to create it, which Fish now does. (It makes it private, too. You can change it to public if you prefer.
fish perms public .fish/alias
will do the job.)
My thanks to Rodrigo Barnes (@rodrigobarnes) for noticing and pointing out both problems.
I have switched the default/preferred HTTP library used by Fish from httplib2, which had always caused me problems, to Kenneth Reitz’s requests. I don’t know whether httplib2 is just broken or whether it’s a case of user error, but I have never managed to persuade it to work reliably. It took under 15 minutes to swap it out and replace it with requests, and the process fixed all the problems I knew about with httplib2, so that was pretty fantastic.

I have talked to various people about requests and the most common response has been “It’s awesome”. Do not misunderstand me when I say, I disagree. My experience merely that requests is well-designed, functional, pythonic and easy-to-use. In those respects it is like most Python libraries. I suspect that the main reason requests generates reactions of awe is that all the other main Python libraries for handling HTTP requests are sub-par [1]. As Kenneth Reitz himself says (specifically of urllib2):

Things shouldn’t be this way. Not in Python.

Requests is excellent and I strongly recommend you use it, with one caveat:

Version 0.9.2, which I used to make the change, works perfectly. Use that.

I have not yet been able to make version 1.0.1 work with Fish. It is entirely possible that is because I have done something stupid. But at this point, using 1.0.1 will not move you forward.

The way I have upgraded Fish is first to try to import requests and then to check that the version is less that 1.0.0. If requests is unavailable, or you have 1.0.0 or newer, Fish falls back (silently) to httplib2. To be fair, that mostly works; but requests works better.

[“What does that mean?”, you ask. Well, I have two problems with httplib2. First, it consistently fails with request bodies over slightly less than 64KB. This is a problem with some file uploads to Fluidinfo. Secondly, when performing bulk uploads, particularly if I use multiple threads, I get occasional failures with httplib2. Retrying sometimes, but not always, works. (This was a major pain uploading, for example, the 2.5 million items in the British National Bibliography. The upload spent about 50% of the time uploading the first 98%, and the other 50% retrying the troublesome 2%. I have yet to see any random failures with requests.]
I finally got round to packaging up Fish with setuptools. So if you download it now, you should be able to do a standard
python setup.py install
in the package directory and standard, good things will happen. These include installing the fish script, so if you used an alias or specially added it to your path before, this is no longer necessary.

I also cleaned up the import structure a bit to make this work.

So much for the good news.

The bad news is that I can’t upload it to PyPI, which had been my intention, because there is already a package called fish on PyPI.

Animating fish (and birds) for progress bars

Obviously, I could rename Fish (for a second time), but that seems undesirable when it is about to appear in print in the form of an O’Reilly book (blog post upcoming). So I guess we’ll have to do things the old way.

Needless to say, in the process of all these lovely upgrades, it is possible I broke something. Please let me know if I did and I will endeavour to fix it.

[1]	Funny term, “subpar”. Every golfer aspires to shoot under par: in golf, less is definitely more. But I didn’t feel using the term “super-par” would really have conveyed my meaning.

Tags, The Tagosphere and Every Thing: Towards an rcp for Fish

2012-01-22T19:16:00.000Z

In the first of this series of articles we discussed the analogy between the Unix File System and Fluidinfo. The Fluidinfo Shell, Fish, is largely a working through of that analogy. A new insight from that article, upon which we will now expand, is that the objects in Fluidinfo can usefully be viewed as analogues of host computers in a computer network.

In the second article, we discussed how to augment Fish with Unix-like copy (cp) and move (mv) commands for Fluidinfo’s abstract tags and namespaces.

In this third, and—with luck—final part of the trilogy, we will tackle the more complex question of how to copy and move data within and among objects and concrete tags in Fluidinfo. Just as we leant on the behaviour of Unix’s cp and mv commands when considering similar commands for abstract tags in Fluidinfo, we will look to rcp, rsync and ftp as sources of wisdom, guidance and precedent as we try to design similar commands for concrete tags in Fluidinfo.

Recapitulation¶

In the previous article in this series, we concluded that we could sensibly model cp and mv commands for namespaces and abstract tags in Fluidinfo on their Unix counterparts provided we resolve three issues:

How should we handle ambiguity, given that a tag and a namespace can share the same name (and path) in Fluidinfo?

How should we handle permissions on copied and moved tags, given the rather different way permissions work in Fluidinfo and Unix?

How destructive should overwriting behaviour be? If we copy a (whole, abstract) tag “on top” of another, should all old values of that tag be destroyed, or should the result be a union in which the new values take precedence, but the old values remain on objects that were not tagged with the “source” tag?

Our candidate solutions are (respectively):

Where paths are ambiguous for either source or destination items, we will demand disambiguation by adding a trailing slash (/) to denote a namespace or a trailing dot (.) to denote a tag.

We propose a reverse Facebook principle, whereby whenever the degree of access granted to data unclear, we should grant the smallest set of rights that the user might reasonably expect, the logic being that while overly restrictive permissions can easily be relaxed, unintentional release of private information is potentially irreversible and more harmful.

We didn’t really resolve overwrite behaviour beyond a meal-mouthed suggestion that we might need to provide options. While we should do that, this doesn’t answer the question, because unless we force the user to make an explicit choice on a case-by-case basis, we will have to decide on the default behaviour, and that will be the main behaviour people will experience (unless we make such a poor choice that it is normally overridden). I am currently leaning toward the less descructive behaviour as the default.

The Jobs to be Done¶

Thus far, we have considered moving and copying whole tags (abstract tags together with all their concrete instances). But at least as important, and probably more common in practice, will be the need to move or copy just some concrete tags. Concentrating first on the movement and copying within Fluidinfo (as opposed to transfers between Fluidinfo and the local file system), some examples of things Alice might wish to do include:

Copy one or more tag(s) from on object to another. For example:

Copy Alice’s description tag from paris to city:paris.

Move Alice’s description tag from paris to city:paris.

Move Alice’s rating tag from paris to city:paris while moving it into her private namespace (on the destination object).

Rename or duplicate tags on one an object or a set of objects. For example:

Rename Alice’s rating tag on city:paris as star-rating

Copy Alice’s rating tag on city:paris to star-rating

Move Alice’s rating tag on city:paris to private/rating

Perform any of the above operations on all objects Alice has tagged with alice/personal.

Systematically move tags from objects using on about tag convention to those using another. For example:

Move all Alice’s rating tags on objects she has tagged with alice/city to corresponding objects with the same about tag, but preceded by city: (e.g. from paris to city:paris).

Alice might also be interested in exchanging information between her local file system and Fluidinfo. (In this case, we would probably support only copying, not movement, in much the same way as there is no rmv command, and indeed even drag on drop usually copies rather than moves when source and destination are different volumes or hosts.) Examples of upload and download to and from Fluidinfo might include

Copy a local set of files to an object.

Copy the files and directories in Alice’s ~alice/blog directory as corresponding tags and namespaces on the Alice's Blog object in Fluidinfo.

The same operation, but now taking the files from ~alice/blogs/drinkingblog/html and placing them in a new alice/blog namspace on the same nominated object

Conversely, Alice might wish to download a blog, stored as a collection of tags and namespaces in Fluidinfo as a set of local files—the precise inverse of the cases above.

Download tags from a number of objects in Fluidinfo to different parts of the local file system.

Alice might have a description tag on objects corresponding to each element of the periodic table and wish to download them either to different local directories or different files within a directory. For example, she might want the description from the element element:mercury to go to elements/mercury/description.txt or to elements/mercury-description on the local file system.

Again, conversely, Alice might wish to upload a directory structure, taking one or more of the upper levels of the directory structure either as object specifiers or tags in some systematic way.

It is unlikely that the first version of any rcp analogue in Fish will support all of these fully, but it is useful to think about what our aspirations might be.

Still another case might be to map between the contents of a local file (e.g. a CSV file) and a set of objects in Fluidinfo. However, I currently regard that as a separate kind of task, and not really suitable for analogues of cp and mv.

Lessons from rcp, rsync, scp etc.¶

I have argued that Fluidinfo objects (which represent things) can usefully be viewed as analogues of hosts (computers) with Unix File systems. That immediately suggests that we might usefully look to rcp and its variants (rsync, scp and possibly ftp) for inspiration.

The basic addressing scheme in rcp simply identifies a remote file as

host:path/to/file.ext

Fr example, to copy a file drink.me from a host wonderland, Alice can say, for example:

rcp wonderland:drink.me .

In fact, in normal use, rcp can replace cp, solong as your file names don’t contain colons. In the case of Fluidinfo, we won’t introduce a different command at this stage but will simply extend cp (though later we will propose adding an rcp for different purposes).

Remembering that we are mapping hosts to Fluidinfo objects, which are usually identified by their about tags, this suggests we might naïvely consider a directly similar rcp-like cp command for Fluidinfo. The first use case we suggested for Alice was copying her description from the object for paris to the object for city:paris. So that might be:

cp paris:description city:paris:description

Unfortunately, in this particular case the “solution” doesn’t work well because the target about tag includes a colon. Obviously, in principle, we could escape the colon in some way. If we did the ‘obvious’ thing of escaping with a backslash, and were typing into an interactive Fish shell (rather than from the command line), that would mean we would need to type:

cp paris:description city\:paris:description

which is already tedious. However, if we were working from a Unix shell, which itself uses backslash for escaping, we would need to use either

cp paris:description city\\:paris:description

or perhaps

cp paris:description 'city\:paris':description

This is OK in principle, but would get very tiresome (and be rather error-prone) in practice, particularly since colons are widely used in about tags.

Another option we might consider is to use Fluidinfo’s full path convention. Recall that the value for alice/description on paris is available at

http://fluiddb.fluidinfo.com/about/paris/alice/description

So it might seem promising to base things on that. Our cp command would then look something like:

cp /about/paris/alice/description /about/city:paris/alice/description

Unfortunately, that doesn’t look too promising either. First, Fish already uses a leading slash to introduce absolute paths to other users’ tags (allowing Alice can omit alice/ from her own tags, a significant convenience, that it would be awkward to retain using this scheme).

Perhaps worse, however, is that one of the most common kinds of about tags is a URL. That would cause real pain getting an about tag through the shell with the slashes protected as part of a path such as above. Imagine having to say:

cp /about/http:\\/\\/example.com\\/foo\\/bar/description /about/http:\\/\\/example.com\\/foo\\/bas

This is not really a problem for the API, where the about tag typically gets passed into some function separately and escaped automatically by the machine, but it would be a significant inconvenience for a Fish user.

An alternative that might work better is to swap the host and the path. A colon still wouldn’t make a good separator since colon is a legal character in a tag name, and it would be confusing to reverse rcp‘s convention anyway. We could, however, use @, which would perhaps seem more natural in that order. So Alice might say:

cp description@paris description@city:paris

I think I remember using that as an alternative form with some version of rcp, but I can’t see it in the documentation for any version I’m currently using, so I may be mistaken. Nevertheless, the form does feel reasonably natural from both @‘s use in email addresses and its more general use to mean “at”. Although the @ is used in reasonably common about tags (particularly Twitter names), it doesn’t really cause a problem, since it is not a valid character in a tag name. It would be fine to say:

cp chess@@RedQueen chess@@WhiteQueen

since the second @, in each case, it clearly part of the object identifier. This feels like the first contender.

Another possibility we might mention is specifying objects using -a, -i or -q, as is possible with other Fish commands such as tag. The issue with that is that it only works well if all the tags are on the same object. We could have either repeated flags or different names flags for source and destination, but if we want to allow Unix-like

mv src1 src2 ... srcN dest

that will be less good. But it might be useful for some cases, particularly when the source and destination objects are the same.

Let’s try some other examples from our shopping list, keeping in mind the possibility of using -a, -q etc. as well as the @ convention:

Move Alice’s description tag from paris to city:paris.
cp description@paris description@city:paris
or potentially:
cp description@paris @city:paris
Move Alice’s rating tag from paris to city:paris while moving it into her private namespace (on the destination object).
cp rating@paris private/rating@city:paris
Rename Alice’s rating tag on city:paris as star-rating
mv rating@city:paris star-rating@city:paris
mv -a city:paris rating star-rating
mv -q 'fluiddb/about matches "city:paris"' rating star-rating
We could also allow a trailing at to mean ‘the same as last time’ on the second or subsequence use, so that
mv rating@city:paris star-rating@
might be acceptable, or even just
mv rating@city:paris star-rating
Copy Alice’s rating tag on city:paris to star-rating
cp rating@city:paris star-rating@city:paris
cp rating@city:paris star-rating@
cp rating@city:paris star-rating
Move Alice’s rating tag on city:paris to private/rating
mv rating@city:paris private/rating@city:paris
mv rating@city:paris private/rating@
mv rating@city:paris private/rating
mv -a 'city:paris' rating private/rating
Perform any of the above operations on all objects alice has tagged with alice/personal. For example, let’s take the moving of rating to private/star-rating
mv -q 'has alice/personal' rating private/star-rating
We could also do the same thing with namespaces, though I didn’t list that as an example. But it seem fairly clear that
cp -q 'has alice/personal' private secret
or perhaps
cp -q 'has alice/personal' -r private secret
would mean copy all the tags under personal to a new secret namespace (if it doesn’t exist) or to under secret if it does. [The -r would mean recurse, as with cp, but whether it should be required I’m not sure.]
Move all Alice’s rating tags on objects she has tagged with city to corresponding objects with the same about tag, but preceded by city: (e.g. from paris to city:paris).

This one is completely different and is probably something we need to admit defeat on for now. Though it is not the same, it reminds me of a Unix command I’ve always thought was missing, which is a pattern-based bulk rename (and bulk copy). I actually have to allow systematic renaming of files using commands like:
bmv foo@.txt bar@.txt
which in which the @ is a wildcard and on the right has the same value as that on the left (like a tagged regular expression). So if a directory had food.txt and fool.txt, this would match rename food.txt as bard.txt, and fool.txt as barl.txt. Systematic construction of about tags could follow a similar pattern, but useful though it would be, is probably out of scope here.

I can, however, imagine allowing a more manual version. For example, we might allow braces and a syntax like:
cp rating@{foo,bar,baz} star-rating@{Foo,Bar,Baz}
to move and rename tags from things like rating on foo to star-rating on Foo etc.

Although perhaps not pretty, this @ syntax, combined with -a, -i and -q flags (for when the same object is to be used both sides of the cp or mv) seems to get us a long way. Adding in paired lists in the {a,b,c} form would go a step further.

Upload and Download¶

Ironically, although we modelled our extended cp on rcp, we have concluded that we don’t really need the “r” but can just use cp for all copying and movement within Fluidinfo. Curiously, this leaves open the possibility of using rcp to perform upload and download. The idea would be that paths without an @ in the rcp command would be taken as paths on the local file system while paths that include an @ refer to tags on an object in Fluidinfo.

In case it’s not clear, the key point is that since Fluidinfo tags can have arbitrary MIME types, they can be used to store files: the analogy we have been pursuing is not merely structural. For example, the documentation for Fish is stored in Fluidinfo, as a collection of tags on the object with the about tag fish, with its index.html file at

http://fluiddb.fluidinfo.com/about/fish/fish/index.html

The be completely clear, this is not a URL pointing to a static HTML file: this is the fish user’s tag index.html on the object with the about tag fish, being retrieved directly from the Fluidinfo database.

A partial visualization of the tags on that object is shown below.

The point of commands for uploading to Fluidinfo is to make it easy to publish a tree of files such as this.

Let’s see how the example use cases for upload and download might work in this case. I am going to assume that we have no difficult cases with ambiguous paths in Fluidinfo, and that if we were to we would either require disambiguation or the commands would fail. (This would obviously be an issue only on download.)

Copy a local set of files to an object.
Copy the files and directories in Alice’s ~alice/blog directory as corresponding tags and namespaces on the Alice's Blog object in Fluidinfo.
rcp blog blog@"Alice's Blog"
The same operation, but now taking the files from ~alice/blogs/drinkingblog/html and placing them in a new alice/blog namspace on the nominated object
rcp ~alice/blogs/drinkingblog/html blog@"Alice's blog"
Conversely, Alice might wish to download a blog, stored as a collection of tags and namespaces in Fluidinfo as a set of local files, the precise inverse of the cases above.
rcp blog@"Alice's Blog" blog
rcp blog@"Alice's blog" ~alice/blogs/drinkingblog/html
Download tags from a number of objects in Fluidinfo to different parts of the local file system.

Alice might have a description tag on objects corresponding to each element of the periodic table and wish to download them either to different local directories or different files within a directory. For example, she might want the description from the element element:mercury to go to elements/mercury/description.txt or to elements/mercury-description on the local file system.

This is a more interesting and difficult case, and again is fundamentally about constructing paths semi-programmatically. Although it is not too hard to think of ways it might be done (perhaps with regular expressions), we might be moving beyond what it’s reasonabe to expect Fish to do. That goes for the final example too.

These examples suggest that modelling a Fish rcp command on a modified Unix-like rcp could work well. It’s almost as if we are considering our host machine as part of Fluidinfo, with its file system representing a concrete tag hierarchy on an anonymous local object. There are certainly other issues to consider, including MIME types and files for potential omission (dot files? backup files ending in ~, symbolic links etc.), but we seem to have a promising way forward.

I was going to consider ftp as an alternative source of inspiration, but its model seems more cumbersome in general, and better suited to situations where you can considering only one host, whereas in Fluidinfo it is normal to consider many objects, so I think we can probably forget ftp.

Summary¶

Perhaps surprisingly, we have largely got there.

Over the last three articles, we have probed and extended the analogy between Fluidinfo and the Unix file system and come up with plausible syntaxes for building mv and cp commands that give us pretty powerful ways of moving and copying data within Fluidinfo, drawing on ideas from cp, rcp, mv and rsync as well as the current Fish.

Despite borrowing from rcp in designing Fish’s cp command, we had left rcp itself free and can now use it as the basis for moving data between a local file system and Fluidinfo, using almost identical conventions save for the interpretation of a path that does not contain an @ (“the same as object as before” for cp, and “the local file system” for rcp).

There are definitely holes to fill, but it feels like there is a reasonable outline spec. If people see problems, or have better ideas or comments, do let me know.

Otherwise, all the remains is the trivial task of implementation. How hard could that be?

Movement and Copying in Fluidinfo

2012-01-19T14:52:00.000Z

In the previous article, we discussed the analogy between the Unix File System and Fluidinfo’s tag hierarchy. This analogy forms the basis and inspiration for the Fluidinfo Shell, Fish. But a file system without move and copy commands would be a sad and contemptible thing, and at the moment Fish, like Fluidinfo, is impoverished by the lack of such basic functionality as cp and mv. Here we will try to design such functionality, building on the analogy.

Copying and Moving¶

In Unix we can

copy files with the cp command

copy directories (and their contents) with cp -R

move files to a different location with mv

move directories (and their contents) to a different location with mv

rename files, also with the mv command

rename directories with mv

delete files with rm (“remove”)

delete empty directories with the rmdir command and delete directories together with their contents with the rm -r command.

In general, the functionality of mv is conceptually equivalent to copying and then removing an item.

We can also copy files and directories between different machines using the rcp and rsync commands, which are both similar to cp but understand a host: prefix. An alternative to these commands is the ftp command, which operates in a very different manner, and uses different mechanisms, to ultimately similar effect.

Fish, today, offers no commands for moving, renaming or copying tags or namespaces, but does provide an rm command that performs the combined functions of rm and rmdir (also requiring a -r and -f flags in some cases). It is worth noting, briefly, that in some sense Fish’s rm goes much further than rm on Unix, in that the command

fish rm rating

removes not only the abstract rating tag, but every occurrence of that tag in Fluidinfo, potentially on millions of objects. This is why Fish requires the -f flag to force the removal of a tag that is in use. In the prevous article, we argued that Fluidinfo’s objects play the natural analogues of computers in a network. From that perspective, if we think of rcp as a more powerful version remote version of cp, Fish’s rm command is more like a remote rm (presumably rrm) that allows you to remove all files with a given path on all hosts simultaneously. It’s as if you could say something like:

rrm -f *:~/.bashrc

to remove the .bashrc in your home directory on every machine on which you have an account. Indeed, if Linus Torvalds were not merely Linux’s creator but the superuser on all copies of the OS, with such a command he could remove everything on all Linux hosts with

rrm -rf *:/

Let’s think about the Unix commands cp and mv, and their possible generalizations to the realm of Fluidinfo. Recall that, when we want to be precise, we need to distinguish between two different senses of the word “tag”. Ordinarily when we attach a tag, possibly with a value, to an object, we create what we might variously call a “tag instance” or a “concrete tag”. Fluidinfo, however, maintains a user’s tag hierarchy independent of whether tags are actually in use. When discussing tags in this sense, independent of objects, I call them abstract or platonic tags. These are quite real and can persist even when the tag is not in use. The diagram below shows the abstract tag hierarchy for Alice, on the right, and her file system, on the left. Note carefully that in her private namespace Alice has both a tag and a namespace called moments.

In what follows, assume Alice’s current working directory on Unix is her home directory /home/alice, and for concreteness, assume that her shell is the Bourne Again Shell, Bash.

Copying a file or a tag¶

On Unix, Alice can copy her has-drunk file into her private directory by saying any of the following:

cp has-drunk private/has-drunk
cp has-drunk private/
cp has-drunk private

Obvious though it is, let us spell out what this means. After running one of these commands, Alice will have two files, /home/alice/has-drunk and /home/alice/private/has-drunk, where previously she had only one, and each will contain a separate copy of the same data.

We could plausibly adopt any or all of these in Fish to copy Alice’s has-drunk tag to her private namespace. But what would that do? I think the most obvious action would be to create a new abstract tag called alice/private/has-drunk and then tag all the objects currently tagged with alice/private/has-drunk with the new alice/private/has-drunk tag, copying their values if any. We would need to consider quite carefully how to handle permissions when performing such copying. The case is rather different from Unix, because in Unix permissions are hierarchical in the sense that a file with public read permission in an fully private directory cannot be read. This is an important detail. On Unix, let’s assume that Alice’s private/things file has open read permissions (644):

alice$ ls -l private/things
-rw-r--r--  1 alice  staff   0 17 Jan 18:57 things

and that she then locks her private directory so that only even she can look at it.

alice$ chmod 700 private

If Bert now trys to look at alice/private/things he will find that he cannot:

bert$ cat ~alice/private/things
cat: /home/alice/private/things: Permission denied

In particular, on Unix this means that if Alice moves a non-private file to a private directory (by which I mean, one with neither read nor execute permission) it becomes unreadable.

In Fluidinfo, the permissions hierarchy is consulted only when new tags and namespaces are created. So if Alice creates a new tag in her private namespace, it will default to being private; if we copy the permissions of a tag when copying the tag, its permissions will be unaltered, and potentially different from if we created the tag afresh in the new location.

The correct behaviour is not clear, and either way there is potential for surprising the user in unpleasant ways, most obviously by making public data that the user intended to be private. We have seen above how by copying permissions with tags we could violate a (reasonable) assumption that moving a tag into a private namespace would make it private. If we fail, however, to copy permissions, copying a private tag to a non-private namespace would result in a non-private tag, which might also be a nasty surprise.

My first inclination here is to do a “reverse Facebook” by, when in doubt, setting the permissions on the destination to the more restrictive of the two possibilities, on the assumption that revealing data that Alice wanted to keep private is both worse and less correctable than making data more private than intended, given the inability to make people unsee (or even, uncopy) things. Needless to say, we could also have options to allow the user to choose what behaviour she wants.

Q1. How should permissions behave when tags and namespaces are copied or moved? Should we go for:

The permission of the destination is copied from the source?

We follow mv = rm + cp + rm and create the new tag or namespace in the new location according to default rules?

Maximum privacy: apply the more restrictive of the permissions suggested by a. and b. (or, if necessary, their most restrictive intersection).

Moving or renaming a file or a tag¶

Going back to Unix, Alice can move her has-drunk file from her home directory to her private directory with any of these commands:

mv has-drunk private/has-drunk
mv has-drunk private/
mv has-drunk private

Again, we could plausibly adopt all of these forms in Fish to move Alice’s has-drunk name to her private namespace. In this case, there seems no real issue about what should happen. We can’t sensibly “move” the abstract tag but not the concrete ones. Using our rule of thumb that

move = copy to new location + remove the original

mv src dest = cp src dest; rm src

this reinforces the case for making cp copy all the concrete tags as well as the abstract tag.

Renaming really raises no extra issues: just as in Unix Alice can rename here has-drunk tag to drunk with a simple

mv has-drunk drunk

she should be able to rename her has-drunk abstract tag as drunk with the same command in Fish, and in the process rename all its concrete instances.

Copying and Moving and Renaming Directories¶

What about copying a directory in Unix? We use cp for that, but now we need to use -R to force the directory and all its contents to be copied recursively: without this -R, we can’t even copy and empty directory such as things:

$ cp things thangs
cp: things is a directory (not copied).

But with -R, we can copy a directory hierarchy as easily as a file. Let’s suppose Alice wants a duplicate of her private directory in her things directory. She can use any of

cp -R private things
cp -R private things/
cp -R private things/private
cp -R private things/private/

and the result will be a duplicate:

$ ls -RF things
private/

things/private:
moments/      things          thoughts/

things/private/moments:

things/private/thoughts:

Move works essentially the same way and needs no example. Again, so far there seems to be no reason why we shouldn’t build analogous functionality in Fish for copying and moving namespaces and their contents. We would certainly allow the -R flag, but might not require it, and would certainly allow -r to be used as a synonym. As with copying simple files, and following our mv = cp + rm dictum, concrete tags in the hierarchy would be copied, together with their values, on all objects to which they are attached.

Clobbering¶

Now consider the following commands on Unix, in the context of the same directory structure shown in the original figure:

cp has-drunk private/things
mv has-drunk private/things

The destination, private/things is a file that already exists: it will be clobbered (overwritten) by both cp and mv. The same would be true if Alice copied her private/moments/things file to her private directory with any of

cp private/moments/things private
cp private/moments/things private/
cp private/moments/things private/things

or their mv counterparts. So in Unix, the rule is

When the destination specified is a directory, move or copy the source into that directory. If there was already a file with that name in the directory, delete it first.

When the destination specified is a file, first remove that file if it exists, then copy or move the source to that destination.

Except that this isn’t quite true: you can’t clobber a file with a directory. So

$ cp -R things private/things
cp: private/things: Not a directory
cp: private/things: Not a directory
cp: private/things: Not a directory
cp: private/things: Not a directory

(the four failures being as each of the entities in private fails to be be copied), and

$ mv things private/things
mv: rename things to private/things: Not a directory

Why can’t a hulking great directory clobber a puny file? I don’t know. Unix has many wonderful attributes, but consistency is not foremost among them. To my surprise, even adding -f cannot persuade the system to do it. Whether Fish should copy this apparently anomalous behaviour is not completely clear to me: logic suggests not, but fidelity to Unix conventions suggests maybe so. The point may be moot anyway, as there’s a good chance I will require a -f to clobber even a tag, just as I do with rm, if it is in use. This is because whereas on Unix, clobbering a single file removes a single entity, however big. In Fluidinfo, a single abstract tag could have a million instances or more, and I feel requiring a -f flag to encourage the user to confirm her intent before engaging in such (potentially) wide-spread destruction is not unreasonable.

These minor exceptions notwithstanding, the way files get clobbered suggests that we might extend our recipe to include the rule:

If dest is a file:
   mv src dest = rm -f dest; cp -R src dest; rm -r src

Does that make sense for tags in Fish?

This, I think, is an interesting question. We could certainly make Fish remove all the abstract destination tag and all its concrete tags before moving or copying another tag. But it also seems reasonable to consider the possibility of replacing those concrete tags present in the source, but not those absent in the source.

To make this real: suppose Alice says (in Fish)

$ cp has-drunk moments

and at the time she does the state of her has-drunk and drunk tags is as follows:

$ fish show -q 'has alice/has-drunk' /about
2 objects matched
Object d440c5cf-9680-4748-b70e-56f07f35ca09:
  /fluiddb/about = "drink me (not poison)"
Object ec430756-e110-4bc4-b882-544afda1cce8:
  /fluiddb/about = "drink me"

$ fish show -q 'has alice/drunk' /about
2 objects matched
Object d440c5cf-9680-4748-b70e-56f07f35ca09:
  /fluiddb/about = "drink me (not poison)"
Object 49126b6d-18bd-457f-af55-a251cf400fc9:
  /fluiddb/about = "drink me not"

or, diagramatically:

It seems clear that the value of the has-drunk tag on "drink me (not poison)" (no value) should be replaced with the value of the drunk tag (true), and that a new has-drunk tag should be placed on "drink me not", also with the value true. It is less clear, however, that the has-drunk tag on "drink me" needs to be deleted. We will be moving on to discuss selective copying and moving later anyway, but we have certainly formed a question:

Q2. If a tag is clobbered by a mv or cp command, should all of its instances be clobbered, or only those necessary to make way for the tag values from the source?

Shared Paths¶

Where things get more interesting is when the source or destination is ambiguous, because it specifies both a tag and a namespace. This can’t occur in Unix, because each path resolves unambiguously to either a file or a namespace. Let’s think through the cases with the aid of our diagram of Alice’s tag structure above, which I’ll repeat for easier reference (sod the cost!)

What would Alice reasonably expect to happen if she issued the following commands?

mv private/moments private/thoughts
mv moments private
mv moments private/moments
mv private/moments /alice

There is ambiguity everywhere. Both the source and the destination can be ambiguous, so we have to consider all of:

unambiguous source, unambiguous destination

unambiguous source, ambiguous destination

ambiguous source, unambiguous destination

ambiguous source, ambiguous destination

Only the first of these is straightforward.

In the case of the existing ls and rm commands in Fish, I have taken that view that an ambiguous path refers to both the possible targets, but while this seems unobjectionable in the case of ls, it clearly leads to the possibility of removing more than the user intended. I plan to reconsider that in the light of these ruminations on cp and mv.

It seems to me that probably a better way forward than taking an ambiguous specification as referring to both its targets is to demand disambiguation. The question is: how would that be achieved?

I have made a point, in the examples above, of listing subtly different alternative forms for some commands, e.g.

mv has-drunk private
mv has-drunk private/

Over recent years (particularly with the rise of tab completion in shells) it has become increasing common to allow directories to be specified including a trailing slash, and no harm derives from this practice. The question is we can we exploit this tend this as a way of disambiguating between tags and namespaces.

In Unix shells and commands, in most cases, the inclusion or omission of the slash makes no difference to behaviour, though I am aware of at least one case where this is not so—rsync.

To quote from the man page for rsync (on Mac OS 10.6.8):

A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning “copy the contents of this directory” as opposed to “copy the directory by name”, but in both cases the attributes of the containing directory are transferred to the contain- ing directory on the destination. In other words, each of the follow- ing commands copies the files in the same way, including their setting of the attributes of /dest/foo:

I personally find this behaviour bizarre and it always make me slightly nervous, using rsync, which is otherwise superior in every way to rcp. However, the idea of using a trailing slash to specify the namespace (cf. directory) rather than the tag is different: it is not changing the behaviour of the command according to which of two unanambiguous specifications of a directory (namespace) is used, but rather using the slash to disambiguate; this seems less objectionable.

The question then would be, how would unambiguously specify the tag? It would seem very ill advised to require that namespaces should always be specified with a trailing slash, so we cannot sensibly say that a path not ending a slash will be taken to be a tag: that way madness lies.

My temptation is to use a trailing dot. I like this as a solution partly because I can recall no case, in nearly thirty years of Unix use, of ever meeting a file (other than the directories . and ..) whose name ended in a dot. I also feel that, while files do not always have extensions, and directory names may include them, by convention most filenames do contain a period and most directory names do not. Admittedly, this is not true of tags, but for me, at least, some association between dots-in-tag-names and “file-ness” survives through the analogy on which Fish is built. If we adopt this idea, our problems all but disappear. We can imagine:

$ mv private/moments private/thoughts
Error: private/moments is ambiguous; use private/moments. or private/moments/

$ mv private/moments. private/thoughts
# moves the tag private/moments to the tag private/thoughts/moments

$ mv private/moments/ private/thoughts
# moves the namespace private/moments to the namespace private/thoughts/moments

$ mv private/moments/ private/moments. private/thoughts
# moves private/moments. to private/thoughts/moments.
# and private/moments/ to private/thoughts/moments/

If this were adopted in Fish, I think there would be an overwhelming case for making rm work the same way; the current behaviour of ls might stay the same, as it is not descructive, or might change in the interest of slavish consistency.

That concludes our discussion of cp and mv for abstract tags in Fish. In the third and perhaps final part of this “trilogy”, we will discuss moving and copying tags and namespaces in the context of the object hierarchy, i.e., how we might copy or move tags from one object to another, or within or among objects.

Like ℵ0 File Systems for Everyone

2012-01-17T21:03:00.000Z

If the ideal blog post is but a screenful or two, this one fails rather badly. So badly, in fact, that I’ve decided to split it into three, each of which will itself be amply proportioned. This post discusses the analogy between the Unix File System and Fluidinfo, as a way of preparing the ground for the following posts. The second post will build on that to discuss options for syntax of potential cp and mv commands in Fish. The final post in the series will discuss upload and download, or maybe rcp, rsync and ftp; as well as possibly fsync, i.e., commands for copying parts of a local file system to or from Fluidinfo.

Everyone loves the Unix File System, even through the expletives we utter at those moments when our ardour is temporarily diminished.

The Fluidinfo Shell, Fish, is largely built on the idea of mapping Fluidinfo namespaces to Unix directories, tags to files, and tag values to the contents of files. Fish’s commands are mostly constructed with close and deliberate reference to corresponding Unix commands.

Fluidinfo and Unix diverge in several small and one large way when viewed through the lens of this analogy.

In Unix, a path (such as /Users/njr/foo) resolves either to a file or to a directory, but not both; in Fluidinfo, a file and a namespace may share a path. This difference causes a number of complications in mapping Unix commands to Fluidinfo.

In Unix, full paths begin with a slash (/) and the Unix kernel works only with such full paths. The shell maintains a notion of a current working directory and allows the user to refer to files relative to that. The Fluidinfo Shell, Fish, does something similar in the context of Fluidinfo, but effectively clamps the working directory to the user’s home namespace (njr, in my case).

To a first approximation, Unix provides only a single hierarchical file structure for each user: Fluidinfo provides an infinite number of them—one for every possible about tag text and one for each UUID. (There are 2¹²⁸, or some 340 decillion = 3.4 ⨉ 10³⁸, UUIDs). It is this infinite number of potential about tags motivates the title of this piece: ℵ₀ (“aleph zero”) is the smallest infinity—the number of counting numbers. Non-mathematicians may be unaware, and skeptical, of the idea that there can be infinities of different sizes, but thanks the remarkable work of Georg Cantor, we can be quite confident that this is so. Infinite sets require some care, and have surprising properties: for example, ℵ₀ is not only the number of positive whole numbers, but also the number of whole numbers (positive and negative). Even the number of rational numbers (fractions) is the same. It is only when we add in the irrationals, to form the real numbers, which are vastly more numerous than the rationals, that we get a larger cardinal, ℵ₁. Needless to say, while Fluidinfo does in principle offer ℵ₀ tag hierarchies to each user, just as real-world Turing machines are limited by the inability of Maxell to supply an infinite tape, the maximum possible size of Fluidinfo itself is limited by the physical properties of the universe, not to mention the finite capacity of even Amazon’s Simple Storage System, S3.

It is this last difference—a tag hierarchy for each Fluidinfo object—that is most significant. One way of thinking about it is this: on Unix (at least considering a single host, with a single file system, in isolation) there is a single directory hierarchy containing everything, as illustrated on the left in the diagram below.

In contrast, we can think of Fluidinfo as having a tag hierarchy, with (potentiall) different tag values for the same tag, on each of a very large number of objects.

Some tags in the tag heirarchy are present on many objects, others on only one, or even none (since there is a notion of an abstract tag in Fluidinfo, separate from any concrete instances of it actually attached to objects).

The tag heirarchies on all the objects can be viewed as a single structure, only part of which is present on any object. I’ve attempted to show this with dark tags and directories for instantiated parts of the Fluidinfo hierarchy on each objects, with parts instantiated elsewhere left pale in the diagram below. I’ve illustrated two whiskies (which use the same parts of the overall hierarchy) and one city, which uses mostly different tags.

We might call this way of thinking about the Fluidinfo tag structure as the the many tag hierarchies view. Sometimes this seems like the natural way to think about Fluidinfo’s tag hierarchy, with the tag structure being in some sense inferior to the objects. The objects might then be likened to different hosts in a computer network, and tags could be thought of as having a single value in each separate hierarchy, just as the contents of a single file is well-defined and unambiguous on a specific computer.

At other times, it seems more natural to think of the tag hierarchy as the primary entity with the with tags having different values on different objects. I’ve tried to illustrate this below, with different collections of objects hanging off tags in a single hierarchy.

We might call this the single tag hierarchy view.

Neither of these views is really more correct that the other—the underlying storage mechanism looks little like either, though is perhaps slightly closer to the second view, in the sense that all the values for a given tag are stored together in the database.

Fluidinfo’s addressing mechanism, however, is very like the “many tag hierarchies” view. After all, the address to access Alice’s whisky/age tag on Ardbeg is

http://fluiddb.fluidinfo.com/about/Ardbeg/alice/whisky/rating

Ignoring the base URL (http://fluiddb.fluidinfo.com) we then have something remarkably familiar from Unix, and comparable either to a host or a volume (/about/Ardbeg), followed (after a separating slash) by the tag path (/alice/whisky/rating). Notice how similar this is, for example, to the way we address files across systems using Unix’s remote copy, rcp. If Alice wanted to copy a file from the host ardbeg on her local area network, she might say

rcp ardbeg:whisky/rating ./whisky/ardbeg-rating

or:

rcp ardbeg:/Users/alice/whisky/rating ./whisky/ardbeg-rating

or even, if she were on host talisker, and wished to be more explicit

rcp ardbeg:/Users/alice/whisky/rating talisker:/Users/talisker/whisky/ardbeg-rating

Similarly, if she has multiple file systems mounted on her machine, Alice might say something like:

cp /Volumes/whiskydata/whisky/ardbeg/rating /Users/alice/whisky/ardbeg-rating

where now /Volumes/whiskydata is a disk (or file system).

We could also note that an alternative way of specifying the previous rcp command would be:

rcp /Users/alice/whisky/rating@ardbeg ./whisky/ardbeg-rating

This is entirely equivalent, but looks slightly more like the “single tag hierarchy” view, at least in the sense that the disambiguating host comes after—rather than before—the tag path.

However we lay things out, it feels fairly clear that there is a good case for extending our analogy to include either volumes, or more likely hosts as the file-system analogues of Fluidinfo’s objects.

That concludes the introduction to this series. In the next part, I’ll discuss how these alternative views might help us to decide the best way to implement mv and cp commands in Fish.

Updates to the Web Apps Shell-Fish and About Tag

2012-01-15T11:20:00.001Z

I just pushed new versions of Shell-Fish (the online version of Fish) and the About Tag app.

There are some quite significant changes, which I will summarize briefly.

Shell-Fish has been updated to the latest version 4.26. It has sat at 4.00.0 for a ridiculously long time, and that became self-sustaining because non-trivial changes were needed to the web version to make it use the new one.

The About Tag app now includes a clone of Shell-Fish. Click the bottom tag on the logo to use this.

The search functionality in About Tag has been overhauled and remodelled on the much better version used at yet another of my sites, Art of Tagging, which is hosted on the intriguing PythonAnywhere platform. Changes include:

When you click a link, the diagram now appears inline

If you re-click, it refreshes

If you’re signed in, you can tag items and the diagram updates.

The changes are quite major, and I haven’t tested them as thoroughly as I would wish, so I’ve probably broken some thing. Sorry. Let me know and I’ll try to fix them.

The three sites all use the same model of requiring your Fluidinfo credentials; unfortunately they don’t share databases, so if you want to use all three, you need put them in three times. This is crazy; all three sites should merge, but that will take time.

Although the search functionality on the About Tag site mostly works well, Google’s hard 10-second time limit on queries means that queries that match a lot of objects tend to time out. The version of Art of Tagging doesn’t suffer from this limit, but unfortunately that site is quite often down for maintenance.
There are many small changes to Shell-Fish but two really major bits of new functionality:
Aliases and syncing now work. If you create aliases on a local version, type sync in Shell-Fish and they will appear there, and vice-versa. Among other things, this makes it easy to use sequences and access them from both sites.
You can drop the -i and -a in almost all cases. So (in the most common case), rather than:
fish> tag -a "artist:melody gardot" rating=10
you can simply use:
fish> tag "artist:melody gardot" rating=10
though the old form will, of course, continue to work, along with -q and the new -@ (for anonymous objects).
My process of turning everything from XHTML to HTML5 continues. Shell-Fish and the About Tag we app are now pure HTML5, like this blog. Previously, they were all XHTML, which worked for everything under the Sun except Microsoft Browsers; at least now Internet Explorer 9 should be able to use some of the functionality. Users of Internet Explorer 1–8, and therefore of Windows XP, remain out of luck (in more ways than I can possibly enumerate, most of which are rather more significant than not being able to use these web sites).

Dial @ for Anonymous

2012-01-12T22:36:00.000Z

I have added a new flag to Fish in version 4.25, which I have just pushed to the Fish repository on Github. The feature allows you to ask Fluidinfo to usher into existence a brand new, tagless object with no about tag, and is accessed using the new -@ flag as follows:

$ fish tag -@ private/note="Dial @ for Anonymous"

Tagged object d679b99d-fe5e-43e9-88dc-47334df776c7 with private/note = "Dial @ for Anonymous"

Although Fish usually exhibits an almost monastic silence when it successfully completes a non-reporting task you have set it, in this case, there seems quite a high likelihood that you will want to know the ID of the object it has created, so it tells you.

The new flag also works with the tag command’s recently added -f flag to accept input from a file or from standard input. In particular, this provides a convenient way of attaching a multi-line note to an anonymous object:

$ fish tag -@f private/note
Everyone needs a little privacy sometimes.
And you don't get much more private than a private tag
on an anonymous object.
^D

Tagged object afcf1f23-7c4b-44ea-a251-1bc99e959436 with private/note = "Everyone needs a little privacy sometimes.
And you don't get much more private than a private tag
on an anonymous object.
"

It will not have escaped the notice of the attentive reader of this About Tag blog that its author has spent perhaps five years proselytizing on behalf of Fluidinfo’s about tag (fluiddb/about) as the one true way to identify an object in Fluidinfo, benefiting as it does from the system-guaranteed properties of uniqueness and persistence. Nothing has changed: I still believe that data that is to have any social aspect, i.e., which might ever be shared with someone else, or associated with someone else’s data, is almost always better placed on an object with an about tag.

For personal data, however, particularly when private, I think the situation reverses. If the data I am writing is not intended to be in any way social, it may be that it is much better to put it on an anonymous object. It also provides an conventient way of guaranteeing that an object, at least at creation time, currently has no other tags.

I plan shortly to revamp the sequence command to allow it to use anonymous objects, which probably fit better with some of its use cases.

From Sets to Lists

2012-01-09T22:58:00.002Z

The Fluidinfo API has just been updated from version 1.13 to version 1.14. There is a single change to the API: where previously it supported sets of strings as a primitive type for tag values, it now instead supports lists of strings. The differences are:

in lists, the order of elements is significant, whereas is sets it is not

in lists, repetition is allowed, whereas in sets an element is either present of absent.

In mathematics, sets are conventionally denoted using braces {...} whereas lists (sequences) are more often denoted using either square brackets [...] or parentheses (...), and this is also true in some more mathematical programming languages, such as Python.

The following pair of interactions with Fish illustrates the difference. Here is the old behaviour, with a slightly older version of Fish:

$ fish --version
fish 4.20

$ fish show fluidinfo /fluiddb/version /fluiddb/release-date
Object with about="fluidinfo":
  /fluiddb/api-version = "1.13"
  /fluiddb/release-date = "2011-12-02T02:28:17Z"

$ fish tag Paris airports='{"Orly", "Charles de Gaulle", "Orly"}'

$ fish show Paris airports
Object with about="Paris":
  /njr/airports = {
    "Charles de Gaulle",
    "Orly"
  }

Notice how the set specified as {"Orly", "Charles de Gaulle", "Orly"} has been deduplicated and reordered, and that Fish uses braces to denote the value.

Here is the new behaviour, with a new version of Fish that I’ve just pushed to Github:

$ fish --version
fish 4.24

$ fish show fluidinfo /fluiddb/version /fluiddb/release-date
Object with about="fluidinfo":
  /fluiddb/api-version = "1.14"
  /fluiddb/release-date = "2012-01-10T00:34:00Z"

$ fish tag Paris airports='["Orly", "Charles de Gaulle", "Orly"]'

$ fish show Paris airportsObject with about="Paris":
  /njr/airports = [
    "Orly",
    "Charles de Gaulle",
    "Orly"
  ]

In this case, order has been preserved, and duplicates in the list have been retained.

For the moment, with an eye to backwards compatibility, Fish allows you to use either braces or square brackets when specifying a compound tag value, but always reports the result using square brackets.

For people interfacing with Fluidinfo directly through the API, there is no syntactic change since compound values were already sent to and returned from the API using JSON lists. The difference is behavioural, with order and duplicates now being preserved in lists sent to Fluidinfo where previously they were not.

Existing compound values will be returned in an unspecified order.

Fish 4.23

2012-01-09T09:40:00.001Z

I’ve just pushed a new version, 4.23, of Fish to Github.

It contains three changes:

It now shows non-primitive tag values when they are textual, where before it simply showed their MIME type and size.

It allows standard input (stdin) to be used to specify long string values for tags, which are then written, by default, with MIME type text/plain.

Where previously it used {...} to denote tag values that are sets of strings (both for input and output), it now uses [...] instead, reflecting a forthcoming change to the API. The change is that Fluidinfo’s primitive compound values will transmogrify from unordered sets of strings to ordered lists of strings. I will blog separately about that when the release occurs. (The change to Fish is backwards compatible, and braces will still be allowed on input.)

I’ll explain a little more about the first two changes.

Non-primitive text types¶

In the early days of Fluidinfo, almost all tag values were primitive, i.e. they were either numbers, strings, booleans, sets of strings or valueless (NULL, if you prefer). However, the API has always supported tag values with arbitrary MIME types.

Fish largely ignored the MIME type in the early days, and since almost all values were primitive, this was not a great problem. However, as Fluidinfo has developed, ever more non-primitive tag values are used, and the result has been that using Fish’s show, tags and get commands has caused binary data to be dumped to terminals with at best meaningless, and at worst destructive consequences.

As a result, I previously changed Fish to show only information about the size and MIME type for non-primitive tag values. For example:

$ fish show abouttag image/red-spinner.gif
Object with about="abouttag":
  /njr/image/red-spinner.gif = <Non-primitive value of type image/gif (size 3208)>

While this is good for MIME types such as GIF, it is less useful for textual MIME types. For example, instead of using primitive strings for Fluidinfo’s record of its API version and release date, the team decided to use MIME type text/plain so that these show up nicely in browers, as you can verify by visiting:

http://fluiddb.fluidinfo.com/about/fluidinfo/fluiddb/version

http://fluiddb.fluidinfo.com/about/fluidinfo/fluiddb/release-date

But with the old Fish behaviour, this meant that the result was the following, less than helpful, output:

$ fish show fluidinfo /fluiddb/version /fluiddb/release-date
Object with about="fluidinfo":
  /fluiddb/version = <Non-primitive value of type text/plain (size 4)>
  /fluiddb/release-date = <Non-primitive value of type text/plain (size 20)>

So with the change in 4.23, Fish now provides more helpful output whenever it recognizes a MIME type as textual:

$ fish show fluidinfo /fluiddb/version /fluiddb/release-date
Object with about="fluidinfo":
  /fluiddb/version = "1.13"
  /fluiddb/release-date = "2011-12-02T02:28:17Z"

Setting Tag Values from Standard Input¶

I added the -f flag to Fish’s tag command the other day to allow the value of a tag to be set from a file. This left the slightly anomalous situation that when -f was used, any value specified would be taken as a filename, but any valueless tags specified were simply set as old-style valueless tags.

Since that useful change, I was reflecting on two further anomalies. First, in Unix, it is usually possible to use standard input instead of a file, which is helpful both for pipelining data and for typing multi-line input. Fish provided no mechanism for this. Secondly, there is no easy way in Fish of directly specifying a multi-line string value.

I realised that all three of these anomalies can be pleasingly remedied by changing things so that when a valueless tag is given to Fish’s tag command, in combination with the -f flag, Fish will read stdin to get a value for the tag. (If more than one tag is specified this way, the same value from stdin is be used for all of them.) By default, the MIME type will be set to text/plain, though the -M flag can be used to specify something different.

So that’s what I’ve done. Here is an example:

$ fish tag -f foo foo
This is line one.
This is line two.
...and that's your lot!
^D

$ fish show foo foo
Object with about="foo":
  /njr/foo = "This is line one.
This is line two.
...and that's your lot!
"

For those less familiar with the Unix command line, here I typed fish tag -f foo foo at the command promt ($). Fish then read what I typed until I terminated the input with the end-of-file character (CTRL+D on Unix, including Mac OS X, CTRL+Z on Windows). The input was then used as the tag value.

Here is an example using a pipe:

$ cat > foo.txt
One
Two
Buckle my shoe

$ cat foo.txt | fish tag -f foo foo

$ fish show foo foo
Object with about="foo":
  /njr/foo = "One
Two
Buckle my shoe
"

Writing Files to Fluidinfo with Fish 4.21

2012-01-01T19:24:00.000Z

I pushed a new version of Fish to Github. It includes experimental new functionality to allow files to be written to Fluidinfo using Fish’s tag command.

This functionality is invoked by the -f flag; when this is used, parameter values are taken to be file names. For example, suppose the Fluidinfo user njr has a file in the /tmp directory called spinner.gif and wants to upload this as the tag njr/image/red-spinner.gif on the object whose about tag is abouttag. Assume that we are on a Unix-like system and that the current working directory is /tmp. Then any of the following will work:

$ fish tag -f abouttag image/red-spinner.gif=spinner.gif
$ fish tag -f abouttag image/red-spinner.gif=./spinner.gif
$ fish tag -f abouttag image/red-spinner.gif=/tmp/spinner.gif
$ fish tag -f -M image/gif abouttag image/red-spinner.gif=/tmp/spinner.gif

This red-spinner.gif file has actually been written to this location, using a command similar to those above, and can be viewed at:

http://fluiddb.fluidinfo.com/about/abouttag/njr/image/red-spinner.gif

Notice that the correct MIME type is served, so the browser can show the image.

Points to note:

The -f flag indicates that all tag values in the command are filenames, i.e. you can’t mix file-based values and literals in a single command.

The filename has to be included even if it is indentical to the tag name (though this may change at a later date) and there is no recursive option for writing directories as tag bundles at the moment.

By default, MIME types for certain common files are inferred from the file extensions as detailed below.

If you want to specify the MIME type explicitly, use the -M flag to do this. This will apply to all tags set in the command.

Text files are assumed to be encoded as UTF-8.

On unix-like systems, tildes will be expanded, so ~ and ~njr etc. may be used.

The file extensions for which MIME types are inferred fall into two groups. First are the textual MIME types (not necessarily types that begin text/, but ones whose contents are textual). These are:

TEXTUAL_MIMES = {
    'txt': None,
    'csv': 'text/plain',
    'html': 'text/html',
    'xml': 'text/xml',
    'htm': 'text/html',
    'css': 'text/css',
    'js': 'text/javascript',
    'vcf': 'text/vcard',
    'plain': 'text/plain',
    'vsg': 'image/svg+xml',
    'ps': 'application/postscript',
    'eps': 'application/postscript',
    'rss': 'application/rss+xml',
    'atom': 'application/atom+xml',
    'xhtml': 'application/xhtml+xml',
}

Files with these extensions are read as non-binary (open("foo.text")) and interpreted as ‘UTF-8’.

Notice that .txt files are not given a MIME type, meaning that they are written as primitive (unicode) strings to Fluidinfo.

The extensions for which non-textual types are inferred are listed below. These files are read as binary files (open("foo.text", 'rb'))

BINARY_MIMES = {
    'png': 'image/png',
    'jpeg': 'image/jpeg',
    'jpg': 'image/jpg',
    'gif': 'image/gif',
    'tif': 'image/tiff',
    'tiff': 'image/tiff',
    'ico': 'image/vnd.microsoft.icon',
    'pdf': 'application/pdf',
    'zip': 'application/zip',
    'gz': 'application/x-gzip',
    'json': 'application/json',
    'mp3': 'audio/mpeg',
    'mp4': 'audio/mp4',
    'ogg': 'audio/ogg',
    'wav': 'audio/vnd.wave',
    'tar': 'application/x-tar',
    'rar': 'application/x-rar-compressed',
}

Use the -M flag to set the MIME type where this mapping is not correct or where the MIME type is not listed above.

This feature is experimental and may change in various ways as experience with it is gained.

Lies, Damned Lies and Progress Indicators

2011-12-30T13:15:00.000Z

Here are some you may have met. (Click start, in a modern non-Microsoft* browser, on each one.)

The perfect progress indicator (rarely seen in the wild).

The software installer progress bar

The software uninstaller progress bar

The media-download progress bar.

The software update progress bar.

No wonder so many cop out and use one of these:

* [Why non-Microsoft? Before going all HTML5 and SVG on this blog I made a point of testing things with Internet Explorer 9 and was delighted to find that everything seemed to work fine there. Unfortunately, I didn’t think to test SMIL-based SVG animation, which works (you guessed it) on Chrome, Safari, Firefox, Opera, iPhone, iPad and even newer Androids, but not, in fact, Internet Explorer 9.]

The British Library Catalogue / British National Bibliography

2011-12-27T15:54:00.000Z

I have added to Fluidinfo information on approximately 2.5 million books drawn from the roughly 3 million records in the British National Bibliography, which documents the British Library’s Catalogue.

As ever, I have used the book-u convention (implemented using the Python abouttag library) to select about tags for the objects, and have tagged the books in Fluidinfo under the book user. Data specific to the British National Biography (BNB) is stored in the namespace book/bnb, while more generic data (derived from the information contained in the Bibliography) is stored directly in the book namespace.

Here is an example of a book that has been augmented with data from the British National Library. The book is George Orwell’s Animal Farm, and it is illustrated using the About Tag visualizer. (If you can’t see the picture below, upgrade to the latest version of your browser or see here for information on why you might be having trouble.) The green tags are the new ones.

Notice that, because of the careful normalization inherent in the book-u convention, where the book is already in Fluidinfo, the new data has generally been added to the existing object corresponding to that book, as in the case above.

The core data that should almost always be present is:

the about tag fluiddb/about, normalized using the book-u convention:

book:animal farm (george orwell)
the book/author tag, containing the best author information I was able to extract, in this case

George Orwell

Where there is more than one author, they are generally shown separated by commas, with the last joined with an and (with no Oxford Comma). For example, The Feynman Lectures on Physics, by Feynman, Leighton and Sands has
$ fish show -a 'book:the feynman lectures on physics (richard p feynman; robert b leighton; matthew l sands)'
/book/author

Object with about="book:the feynman lectures on physics (richard p feynman; robert b leighton; matthew l sands)":
  /book/author = "Richard P. Feynman, Robert B. Leighton and Matthew L. Sands"
or, graphically:

The book/author tag has had a lot of processing done to it, as described below.
the book/title field, which is usually almost identical to that in the BNB data. In this case it is:

Animal farm

I have not altered the capitalization, which is therefore generally consistent with some entry in the BNB database (though I would really prefer it were in Title Case).

the book/source tag shows where the base data was taken from. This tag’s value is a set of strings, each of which corresponds an entry in one of the 17 files from which the BNB data was extracted. The entries consist of

the name of the file (always BNBrdfdcNN.xml) where NN runs from 01 to 17

a dash -

the datestamp on that file (always 20101115 at present)

the digit zero (0) and a # sign

the record number in the file, starting from 1, with six digits.

Since multiple bibliographic entries can correspond to the same work, there is sometimes more than one of these.
the book/r tag is a pseudo-random floating point value with 0.0 ≤ book/r < 1.0.

Some of the raw data has also been added, with almost no cleaning up, under the book/bnb namespace. The BNB data uses the Dublin Core metadata standard, and includes:

bnb/creator, which is the person or organization primarily responsible for the creation of the work. This is sometimes blank, and is stored as a single string value.

bnb/contributors, which is a list of contributors, sometimes including the creator and sometimes not.

bnb/dewey is the set of Dewey Decimal classifications found on the records corresponding to this book.

bnb/isbn is the set of international standard book numbers found on the records corresponding to this book.

bnb/id is the set of British Library IDs found on the records corresponding to this book. (I’m not entirely clear what this identifier is, but it appears to be important and well populated.)

Other information is available in the data (including classification information), and I will probably extract this and add it at a later time.

Finding, Inspecting and Tagging Books in Fluidinfo¶

There are multiple ways of retrieving book data from Fluidinfo and of tagging it.

Probably the easiest and most general method is to go to http://artoftagging.com and do a search that involves a book and some keywords from the title and/or author. A list of results should come back and you can see a visualization of any of them by clicking the link If you have a Fluidinfo account, you can create an account at artoftagging.com and then save your Fluidinfo details there. Once logged in, you will then be able to add your own tags to any object you find.
If you just want to construct the about tag for a book, you can do that using the online version of the Fluidinfo Shell, Fish. Once there, type, for example:
fish> about book "Animal Farm" "George Orwell"
book:animal farm (george orwell)

fish> about book "The Feynman Lectures on Physics' 'Richard P. Feynman"
"Robert B. Leighton" "Matthew L. Sands"
book:the feynman lectures on physics (richard p feynman; robert b leighton; matthew l sands)
(The quotes tell Fish that “Animal Farm” is the title and “George Orwell” a single author.) Alternatively, you can download and install Fish on your own machine. (It is available from Github.) You can then type the same commands, after fish, e.g.:
$ fish> about book "Animal Farm" "George Orwell"
book:animal farm (george orwell)
You can then use any Fluidinfo tool, including the new Object Browser, to work with that object, signing in with Twitter if you like.
Another easy way of finding an about tag for a book is to find it on Amazon (US or UK, for now) and use the az-fish bookmarklet available at the top of the online Fish (drag it to your browser’ toolbar). The bookmarklet will take the item on the current Amazon page and issue the appropriate Fish command to find the about tag. (You don’t need to log into Fish or Fluidinfo to do this.)

The Hierarchy of Books: Works and Manifestations¶

The International Federation of Library Associations (IFLA) describes a hierarchy of four kinds of “book” entities in its report Functional Requirements for Bibliographic Records. These are:

works

expressions

manifestations

items.

Quoting from that report:

“The entities defined as work (a distinct intellectual or artistic creation) and expression (the intellectual or artistic realization of a work) reflect intellectual or artistic content. The entities defined as manifestation (the physical embodiment of an expression of a work) and item (a single exemplar of a manifestation), on the other hand, reflect physical form.”

Loosely, a work is the conceptual book, usually described by the combination of a title and author—Animal Farm by George Orwell.

The report describes an expression of a work as “the intellectual or artistic realization of a work in the form of alpha-numeric, musical, or choreographic notation, sound, image, object, movement, etc., or any combination of such forms.” Thus George Orwell’s Animal Farm can be translated into different languages, laid out differently, typeset on pages, or in digital form, or recorded as spoken words, and these correspond to different expressions of that same book. There may also be different editions, printings etc., which may have slightly different content. Again, these are different expressions of the same conceptual work. (Occasionally, expressions may encompass several works, such as in the case of compendia.)

Moving down the hierarchy, a manifestation is a particular rendering of a work into physical form — “the physical embodiment of an expression of a work.” Note that “[A]s an entity, manifestation represents all the physical objects that bear the same characteristics, in respect to both intellectual content and physical form.” Thus, all the copies of the same printing of the same edition of Animal Farm that are essentially indistinguishable collectively correspond to a manifestation of George Orwell’s Animal Farm.

Finally, an item is an individual copy of a book: “a single exemplar of a manifestation.”

The entries in the British Library’s catalogue correspond literally to items, but conceptually to manifestations, but the objects to which I have attached the data in Fluidinfo correspond to works. This is why the c. 3 million records reduce to c. 2.5 million Fluidinfo objects, and why some of the objects have multiple ISBNs etc. It is entirely possible to create further objects at the level of manifestations (and even items, if someone really wants to do so), and even more so at the level of expressions, but I have not done this yet.

The reason I have concentrated on works rather than manifestations is that this seems much the most important level to represent in a system like Fluidinfo: with important exceptions, when people want to rate or comment on a book, it is most often the work, rather than the manifestation, that they are interested in. Moreover, collecting together information about the different ISBNs associated with a single work is positively helpful. That is not to say that there isn’t a case for creating other objects at the level of expressions or manifestations.

Further Work¶

There is a great deal more that can be usefully done with the fabulous data from the British Library. While I am not committing to doing these, tasks on my list list include:

Authors. Creating an object corresponding to each creator/author/contributor. I plan to use about tags of the form author:normalized name (birth-year) for these, e.g. author:George Orwell (1903). The required data is largely available in the BNB dataset. I would then plan to add a book/related-authors tag to each book, pointing to its authors’ objects and, on the author objects, corresponding sets of book/related-books tags pointing back to their works.
Upload Checking. Checking the everything uploaded OK. I count 2,558,738 unique books (as works) in the BNB dataset, and I appeared to upload all of these successfully (getting HTTP 204 statuses back from Fluidinfo). However, when I count objects having a book/r tag, I get only 2,468,661, a shortfall of 90,077.

Whether this indicates a problem or not is unclear, as if I count the number of books with a book/source but no book/r, with the query
has book/source except has book/r
it reports 18,921 such books, but as far as I can tell, all those it finds in fact have a book/r, so it appears that Fluidinfo is having some difficulty executing some queries correctly at the moment.
About Tag Checking. I had to use some fairly hairy code to coerce the BNB data into the correct form to generate canonical about tags in the book-u convention, and it has definitely failed in some cases. For example, I have seen at least one example where the surname of an author in the BNB data preceded the forename but without a comma, so that forename and surname will have been reversed. To the extent that I can detect these problems, I will try to fix them.

Recent additions. I believe the British Library has issued updates with recent additions (since November 2010); I certainly plan to get that data and import it in a similar fashion, and then to set up a CRON job to do that regularly. In this way, I hope the dataset will be living and always current.

Categorizations. The BNB data includes subject categories for the records, which I have not imported thus far. I will do so.

Year information. There is information about publication dates in the BNB data, but it is not in a very structured form. If I am able to extract it with a satisfactory degree of reliability, I will get this too. Obviously, different manifestations will have different publication dates, so this will probably be a set-valued tag.

Enjoy the data, and let me know if you find problems.

I expect I will write a number of other posts on issues associated with this data.

About Tag Goes HTML5 with Embedded SVG: Browser Requirements

2011-12-27T14:58:00.000Z

This page uses HTML5 and an embedded Scalable Vector Graphics (SVG) diagram, and acts as a test. From this point on (27th December 2011) my plan is to use HTML5 with embedded SVG as the default format for posts, and therefore if you use a browser (or feed reader) that does not support this, you will miss out.

You should see an elegant diagram below. If you not, it probably means that you are not using a modern, standards-compliant HTML5 Browser.

I have tested this page on the following:

A Macbook Pro running OS X 10.7.2 (Lion) with Safari 5.1.2, Chrome 16.0.912.63, Firefox 8.0.1, Opera 11.6
A Mac Pro running OS X 10.6.8 (Snow Leopard) with the same browsers.
An iPad running iOS 5.0.1 (in Safari, naturally)
An iPhone running iOS 5.0.1 (again, Safari, of course)
A Sony Vaio running Windows 7 with Internet Explorer 9
A prehistoric Dell latitude laptop running Chrome 16 and Firefox 9

and everything looks good. It does *not* work with Internet Explorer 8, but then, what does? It also does not work with many older versions of Firefox, Chrome, Safari or Opera. So this is a good time to upgrade. I don't know which Android or Linux growers it works with, but I would guess it will work with some.

I imagine the diagrams will not show up in most feed readers, and that is unfortunate, but I think this is a time to push forward, so that is what I am doing.

Among the other marvellous benefits of SVG, it is scalable (just like the S says), which means that if you zoom the browser (generally command-plus on macs, and control-plus on Windows) the diagram will scale too, without looking terrible. Wonder of wonders.

Fish 4.18 Released: Aliases, Sequences, Duck Typing and More

2011-12-19T19:00:00.000Z

It has been an unconsionably long time since I last pushed a version of Fish to Github. The reason for this is mostly that, while adding various features, I broke a few things and wanted to get them all fixed before inflicting them on people. I believe they are now fixed (but do disabuse me of this notion if you discover otherwise.)

There is much that is new, though almost all changes are backwards compatible. Everything is documented in the new green-themed documentation, available (as usual) in Fluidinfo itself at

http://fluiddb.fluidinfo.com/about/fish/fish/index.html

The main changes are as follows:

You can omit the -a or -i on the tag, untag, show, get and tags commands. If you do so, Fish will use the first argument instead, assuming that if it looks like a UUID, it is a UUID, and if not, that it isn’t. For these purposes, UUIDs must be expressed with lower-case hex digits and include the dashes in 88888888-4444-4444-4444-cccccccccccc formation. So now the following both work where before they would have generated errors:
$ fish show Paris rating /about /id
Object with about="Paris":
  /njr/rating = 10
  /fluiddb/about = "Paris"
  /id = "17ecdfbc-c148-41d3-b898-0b5396ebe6cc"

$ fish show 17ecdfbc-c148-41d3-b898-0b5396ebe6cc rating /about /id
Object 17ecdfbc-c148-41d3-b898-0b5396ebe6cc:
  /njr/rating = 10
  /fluiddb/about = "Paris"
  /id = "17ecdfbc-c148-41d3-b898-0b5396ebe6cc"
Needless to say, the longer -a and -i forms work, and are useful if you want to tag multiple objects in one command (since they may be repeated).
The tags command now displays the tags in alphabetical order, except for the about tag, which is always listed first.
Fish now supports simple aliases, which effectively allow you to add commands to Fish. A simple example is:
fish alias eiffel 'show -a "Eiffel Tower"'
which allows commands like eiffel rating to be used in place of show -a "Eiffel Tower" rating.

Aliases are stored in Fluidinfo, with private tags on objects whose about tag is the name of the alias. For example, with the alias definition above, the object with about tag paris has a tag njr/.fish/alias added to it with its value set to the expansion text for the alias:
$ fish alias paris
paris:
  njr/.fish/alias = "show -a "Paris""
(Obviously, the quoting here is slightly unfortunate; I will fix that some time.)

Aliases are also cached locally in the file-system; the cache is updated from Fluidinfo using the new sync command or whenever Fish is entered in interactive mode (by typing Fish).

Because aliases are stored in Fluidinfo, they can be shared between multiple copies of Fish, and also with the online version Shell-Fish.

The cache can be viewed with showcache.
Support for sequences has been added. Sequences provide a convenient way of storing a numbered collection of items that are added to over time. They are described in a previous blog post (Sequences in Fluidinfo).

Briefly, in the simplest case, a sequence of remarks is defined by saying:
$ fish mkseq remark
This creates two new aliases:
remark is used to add a new remark, using the alias
$ fish alias
remark:
  njr/.fish/alias = "seq /njr/remark"
remarks is used to look at (or search) remarks, using the alias remarks:
njr/.fish/alias = "listseq /njr/remark"
Thus if we say:
$ fish mkseq remark
Next remark number: 0

$ fish remark "Isn't this a remarkable first remark"
0: Isn't this a remarkable first remark
2011-12-18

$ fish remark "...and this only slightly less remarkable"
1: ...and this only slightly less remarkable
2011-12-18
then we will see:
$ fish remarks
0: Isn't this a remarkable first remark
2011-12-18

1: ...and this only slightly less remarkable
2011-12-18
By default, sequences are public, but you can easily make them private by specifying a tag in a private namespace (typically private); you can also specify the plural form. To set up the sequence as private, and use myremarks to list and search remarks, you would instead say:
$ mkseq remark remarks private/remark
For more details, see the previous blog post or the documentation.
Non-primitive types are now shown more sensibly (by show and tags). Previously, Fish would attempt to print even non-primitive types, with sometimes unfortunately consequences both in terms of the volume of output and its effects on terminals. For non-primitive types, output is now shown as below:
$ fish show fish /fish/index.html
Object with about="fish":
  /fish/index.html = <Non-primitive value of type text/html (size 8907)>
Display of set-valued tags is also improved, e.g.:
$ fish tags 'artist:led zeppelin'
Object with about="artist:led zeppelin":
  /fluiddb/about = "artist:led zeppelin"
  /musicbrainz.org/artist
  /musicbrainz.org/artist/end-date = "1980-09-25"
  /musicbrainz.org/artist/members = {
    "Jimmy Page"
    "John Bonham"
    "John Paul Jones"
    "Robert Plant"
  }
  /musicbrainz.org/artist/name = "Led Zeppelin"
  /musicbrainz.org/artist/sort-name = "Led Zeppelin"
  /musicbrainz.org/artist/start-date = "1968-01-01"
  /musicbrainz.org/artist/type = "group"
  /musicbrainz.org/mbid = "678d88b2-87b0-403b-b63d-5da7465aecc3"
The Fish API has been updated to take account of the renaming of FluidDB to Fluidinfo, and various tests have been changed to use more esoteric unicode characters.

Some operations are faster (because more use is made of the /values endpoint).

Finally, when Fish starts it checks the environment for the presence of the variable FISHUSER. If this is defined, the credentials in the startup file identified by the string specified in FISHUSER will be used, rather than the default ones. (This is mainly helpful if you want to use Fish with different Fluidinfo accounts in different shells concurrently.) Thus, if FISHUSER is set to foo (on UNIX), the credentials from ~/.fluidDBcredentials.foo will be used, rather than those in ~/.fluidDBcredentials.

So, obviously, there are quite a lot of changes, and though I’ve been using it for a while, some things might have broken. (I fixed some bugs yesterday; always dangerous!)

Fragmentation and URL Normalization

2011-12-15T13:11:00.000Z

I have updated the abouttag.py library to use a new, better convention for normalizing URLs. The two main changes people will notice are:

URLs that represent directories will now include, rather than exclude, a trailing slash:
http://fluidinfo.com/
rather than
http://fluidinfo.com
There is now a dependency on the excellent urlnorm.py, by Jehiah Czebotar.

The Issue: Fragmentation¶

The twin evils that the abouttag.py library and this blog exist to fight are fragmentation and overloading.

Fragmentation occurs in Fluidinfo when different users store information about the same thing on different objects, while overloading occurs when people store information about different things on the same object. In general, both of these are undesirable. Fragmentation reduces data sharing and makes it harder to extract information from the system, whereas overloading creates ambiguity and confusion.

One of the more common uses for Fluidinfo is for tagging web pages, and it is very natural to use the URL as the about tag, as almost everyone does. There is not much of a problem with overloading in this case (except to the extent that URLs point to web pages that change over time), but there is definitely fragmentation.

I would distinguish between two kinds of fragmentation in the case of URLs.

Different representations of the same URL. Perhaps the most obvious example is the trailing slash on many URLs. Punctilious persons with good knowledge of W3C standards (and in particular RFC3986) prefer the inclusion of a trailing slash on URLs (and more generally, on URIs) where appropriate, and thus prefer
```
http://fluidinfo.com/
```
to the more colloquial
```
http://fluidinfo.com
```
Technically, these are different URLs, but web servers so routinely and uniformly redirect the latter to the former that they can be considered for all practical purposes the same. It seems highly desirable for any convention for about tags for URLs to map these two forms, along with other similar representational variants, to a common about tag.
Different URLs that may or may not represent the same web page. The most obvious example of this is the www. that used to be de rigeur and is now commonly (but not reliably) redundant. Most right-thinking webmasters (webmistresses?) routinely redirect these to the same place, there is no general guarantee that the www. form (http://www.fluidinfo.com/) and the bare form (http://fluidinfo.com/) will produce the same page, nor even that they should both work.

Standardizing this would therefore seem to be a normalization too far.

The Old and New Behaviour of abouttag.py¶

Fluidinfo is far from the only system with an interest in developing a canonical or normalized form for URLs. Search engines and social bookmarking sites (such as Pinboard and Delicious) work better if different URLs representing the same resource are collapsed, and as mentioned above, there is even a standard (RFC3986) for how to perform the canonicalization.

The relevant Wikipedia page describes six normalizations that preserve URL semantics. These are:

Converting the scheme and host to lower case. (HTTP:// → http:// and FLUIDINFO.COM → fluidinfo.com).

Capitalizing letters in escape sequences (%3a → %3A)

Decoding percent-encoded octets of unreserved characters (%7E → ~)

Adding a trailing slash where appropriate (http://fluidinfo.com → http://fluidinfo.com/)

Removing the default port (http://fluidinfo.com:80/ → http://fluidinfo.com/)

Removing dot-segments (http://fluidinfo.com/accounts/./new/ → http://fluidinfo.com/accounts/new/)

Happily, libraries to perform these normalizations already exist and are freely for a number of programming languages, including Python. As noted above, Jehiah Czebotar’s urlnorm.py performs the task admirably in Python, so in the version of abouttag.py that I just pushed to Github (version 0.6) I have made added a new convention, uri-2, corresponding to this behaviour and have made that the default. So now:

>>> from abouttag.uri import URI

>>> URI(u'http://fluidinfo.com')
u'http://fluidinfo.com/'

>>> URI(u'HTTP://FLUIDINFO.com:80')
u'http://fluidinfo.com/'

>>> URI(u'HTTP://FLUIDINFO.com:80')
u'http://fluidinfo.com/'

>>> URI(u'http://fluidinfo.com/a/./b/?arg=%7Ealice')
u'http://fluidinfo.com/a/b/?arg=~alice'

This is different from the old behaviour, which can be obtained by explicitly adding a convention argument of ‘uri-1’:

>>> URI(u'http://fluidinfo.com', convention=u'uri-1')
u'http://fluidinfo.com'
# note no trailing slash

>>> URI(u'HTTP://FLUIDINFO.com', convention=u'uri-1')
u'http://fluidinfo.com'
# Same downcasing, but again no trailing slash

>>> URI(u'http://fluidinfo.com:80', convention=u'uri-1')
u'http://fluidinfo.com:80'
# uri-1 didn't strip default ports

>>> URI(u'http://fluidinfo.com/a/./b/?arg=%7Ealice', convention='uri-1')
u'http://fluidinfo.com/a/./b/?arg=%7Ealice'
# nor did it undo unnecessary %-encoding or strip . & .. path segments.

Both the new and the old versions perform one additional normalization, which is to add a leading http:// if no scheme is present in the input. This is not because there is not a distinction between a domain and a URL, but rather because by calling the URI function the user is clearly indicating that this is a URI, which requires a scheme, and http:// is clearly the appropriate default scheme:

>>> URI(u'fluidinfo.com')
u'http://fluidinfo.com/'

Why...?¶

The reader may be wondering why I did not adhere to the RFC previously, and issued forth older versions of the abouttag library with the altogether inferior behaviour of uri-1. Ignorance, pure and simple.

Siri: The Command Line for Everyone Else

2011-12-10T17:38:00.001Z

Perhaps the biggest difference between the way in which “real” people use computers and geeks use computers is this:

Real people use Graphical User Interfaces because they find them intuitive and efficient.

Geeks generally prefer the command line, which they find easier, more precise and faster than GUIs. For geeks, GUIs tend to get in the way, limit and interfere.

Here’s a GUI for my files:

It has all the advantages of clickability, a set of icons for actions, and a few extra things hidden on menus and right-click (“content”) menus. But it is very limited.

Here, in contrast, is a command line, in its spare, minimalist glory:

If you don’t know what to type, the command line is intimidating, unhelpful and limiting. But if you do know, you can do almost anything: far from limiting, the command line is open and alive with virtually unbounded possibilities. Instead of having to nagivate menus and finders and buttons and icons, the command line allows you to access almost anything the machine can do, all from one place, just by typing.

There are two major things that stop real people from benefitting from the command line and its liberating possibilities:

People don’t know the commands or the right syntax for them.

People don’t like typing. [1]

Enter Siri¶

Just like a command line, Siri has the potential to allow me to access anything my phone can do with no navigation: if I want to call Alex, I say “call Alex”. If I want to find out the height of Mount Everest, I just say “How high is Mount Everest?”. If I want to send a Tweet, I say “text Bird” and it will send a Tweet for me. (OK, this last one is a hack: if I say “Tweet this”, Siri knows exactly what I mean, but refuses saying “Sorry, Nicholas, I can’t help you with Twitter.” But I can send a tweet as a text message by saying “text Bird” because I have the Twitter short-code listed under Bird Parker. “Why not under Twitter?”, you ask? Because Siri still refuses if I list it under Twitter! Go figure!)

Of course, unlike the command line, I don’t have to get the syntax right with Siri. I just issue commands in plain English, and a reasonable proportion of the time it “understands” me.

Before Siri, the nearest thing to a syntax-free command line for real people was Google—a little box into which you can type anything in the reasonable hope that the search will return some relevant information. But Google is largely a one-trick pony, and even though it’s a good trick, it’s nothing like as powerful as when the software makes an attempt to understand the command and has the ability to take actions. (By offering a list of the top dozen or so “hits”, Google also hedges its bets, getting the user to pick the best-looking “answer”: quite apart from speech recognition and “comprehension”, Siri goes for broke by putting all its money on a single interpretation of what you said, only asking for clarification occasionally.)

Horace Dediu, his podcast Getting to Know You makes the case that the significance of Siri is that it allows Apple to learn much more about its users, allowing a new level of lock-in, power and service. That’s an interesting and important perspective that may prove to be right. But after a day with Siri, I think the more direct and immediate consequence is exactly that Siri could bring all the power of the command line to the masses.

[1]	Actually, most geeks don’t really like typing either, and have myriad ways to reduce typing, from globbing (wild-card expansion) to command-line completion; but the basic point stands.

Microsoft, iPads and the Innovator's Dilemma

2011-11-26T16:33:00.001Z

This article was motivated by listening to Episode 44 of Hypercritical, the episode of John Siracusa‘s weekly podcast that focused on the question “What Ails Microsoft?”; listen to it for context. I agreed with most of Siracusa’s analysis, but thought he missed a few key insights and perspectives.

You should listen to Siracusa’s podcast, but here are some of the key points in his analysis of what ails Microsoft are:

Microsoft consistently refuses bet-the-company radical changes that will be good for the user and its own long-term business prospects because it are scared of damaging its cash cows (primarily Windows and Office, but also servers, Exchange etc.);

Microsoft serves primarily PC vendors, IT departments, backward-looking developers and perhaps Intel rather than its end-users; this leads to poor user experiences;

Microsoft follows rather than leads and so is always behind the curve (think Bing, XBox, Zune, Windows Phone etc.)

Microsoft underestimates its own position of strength, which would in fact allow it to upset its customers more (to everyone’s long-term benefit) for fear of losing what it has;

The demands of its core customers for a roadmap mean Microsoft always overpromises and underdelivers, has low marketing impact and never surprises competitors etc.

Apple is the reverse of all this, repeatedly taking bet-the-company risks, always focusing on the end user, repeatedly canibalising its own products, being secretive and never publishing roadmaps, constantly leading and redefining categories (without necessarily being first mover), all in manner of what Steve Denning calls Radical Management, which has led to its current position as the world’s most valuable company.

While I agree with most of these points, here is what I think Siracusa missed.

Clayton Christensen and The Innovator’s Dilemma¶

Clayton Christensen‘s The Innovator’s Dilemma is the best business book I’ve ever read. Unusually, it contains a thesis that can’t be reduced to a single sentence. His interest is in how great companies get overthrown by disruptive innovators. His key ideas are as follows:

Christensen defines a disruptive technology as one that is worse than the incumbent technology on the key metrics that are usually used to measure quality in that space, but better in some other, traditionally less important metrics.

Although he offers several examples, Christensen’s clearest example is disk drives. Here, the two traditional key metrics are speed and capacity. Disk technologies have been replaced in waves, first with 8” disks being replaced with 5.25” disks, then 3.5” disks, then 2.5” disks then 1” disks. (Solid-state drives are now gradually starting to replace rotating disks.)

Christensen argues that incumbent leaders almost always succeed with sustaining (non-disruptive) innovations that improve the performance of the technology against the standard metrics, but almost always fail to bring to market new disruptive technologies, even though these are often first developed by the market-leading company. He says this happens primarily because leading companies tend to be “well managed”, and are strongly influenced by their best customers and partners, who are, almost by definition, mostly bought into the existing metrics. So when, for example, Winchester (the leading 8” disk manufacturer) asks its customers “would you be interested in lower power, physically smaller disk that has lower speed and less capacity they say “no, that’s a terrible thing, we need speed and capacity”.

New entrants, often start-ups, see an opportunity to serve new markets, often consisting of people not using the incumbent technology, for whom the alternative metrics (in this case, size and power consumption) are more important than the traditional ones. For example, 8” disks didn’t work for PCs but 5.25” disks did; 5.25” disks didn’t work for laptops but 3.5” disks did (and then 2.5”); 2.5” disks didn’t work for iPods but 1” disks did. Now solid-state memory, which is fast but expensive/lower capacity, works for phones, cameras, tablets etc. in a way that even 1” disks didn’t.

A key point Christensen makes is that the new market, of non-consumption, is often unattractive to the incumbent leader, who typically sees it as small and offering low margins, but is highly attractive for newcomers, who typically hone themselves on lower margins as they serve it.

Over time, sustaining improvements to the new technology tend to improve it against the traditional metrics as well as the new ones: current 3.5” disks have much larger capacities and better latencies than did early ones. As they improve, they become more viable in increasing parts of the “old” market, and the old leader tends to be reduced to ever smaller, more niche parts of the market. Eventually, the new technology tends to get good enough for mainstream use and at this point the advantages of the new technology start to be more interesting to old customers. (“So I can enough speed and capacity, but with a smaller footprint, less power consumption and a lower price: well sure!”) If it survives at all, the previous leader ends up serving only the very high end where the extremes of the old metrics are required.

iPads, PCs and the Innovator’s Dilemma¶

Apple was not the first to come up with the idea of a Tablet PC. In fact, Alan Kay came up with many of the key ideas in his remarkable 1972 paper on the Dynabook. But in the more recent past, Microsoft (especially Bill Gates) championed tablet computers and brought them to market a decade before Apple built the iPad. Microsoft, however, saw a tablet, through the ever-present and distorting lens of its Windows cash cow, as an enhancement to a traditional Windows PC: you add a touch-screen (and a stylus) to traditional laptop running (of course) Windows and voilà, a tablet is born.

The iPad received a very luke-warm reception when it was launched, and was widely derided as (merely) a giant iPod Touch. It was criticized for being underpowered, closed, not running even standard Mac applications, let alone Windows software, not supporting “true” multi-tasking or windowing and more besides. Yet it quickly sold in the tens of millions and is clearly now replacing PCs for some people.

With some caveats, this fits Claytonsen’s model very well. The iPad is a worse general-purpose computer against the traditional metrics. It has slower hardware (though rarely feels slow), few ports, no user accounts, comparatively little storage, no hardware keyboard, limited, vetted software and (cough) no true multitasking, no Flash, no replacable battery and limited upgrade options.

But look at the alternative new metrics, that show all the ways in which it is better for some people and purposes. It is extremely small and light. It is fantastically easy to use. Thanks to Apple’s control-freakery, installing software is simple and worry-free. It has a touch screen. It has no significant issues with viruses etc. Its battery genuinely lasts over 10 hours even when you use the machine intensively. It has stores for software, books, music and videos built in (and it probably already knows your credit card number). It has numerous sensors (cameras, microphones, accelerometers, gyroscopes and more). Software for it tends to be really cheap and some of it is of fantastic quality. It is supremely relaxing to use.

For people who mostly surf the web, do light email, play games, watch films, read books etc., the iPad is not just a “good enough” alternative to a laptop or even a desktop PC: it may actually be signficantly better. The iPad 2 (and iOS 5) followed the pattern of sustaining improvements, both on the new metrics (usability, weight, size, sensors etc.) and the old (speed, capacity, ability to link to an external monitor, multitasking etc.).

Crucially, while Microsoft saw a tablet as a way to extend the PC, and added Touch features to Windows and made its tablet PCs full Windows PCs “with added Touch”, Apple redesigned all the upper layers of the operating system to give the best possible experience for the iPad as a new class of device. It didn’t worry about disrupting sales of its own Mac laptops, still less (naturally) those of Windows PCs: it just made the iPad as good as it could, in its own right.

Risk and Perfect 20-20 Hindsight¶

The other major point I feel Siracusa failed to make, and many people are missing, is that Steve Jobs’s and Apple’s Radical Management is a genuinely high risk strategy: it can fail as well as succeed, and frequently does so. I think we need to separate out two ideas that I feel are being conflated. The first is “betting the company” on an uncertain new thing, which isn’t necessarily a good idea for a leading company, but makes more sense for a struggling company. The second is the the aggressive development and marketing of new technologies that might canibalize your existing business; this probably is a good idea, even if the new business is lower margin or lower value, because almost certainly someone will do it, and it’s better for the leader to do it to itself than for a competitor to do so.

[Image: Flippin’ Coins , by Pauli Antero on Flickr, Creative Commons, some rights reserved.]

In terms of the risk side, a comparison I like to make is with finding lucky people. Contrast two situations. If I say to you “Give me a coin and I’ll toss it ten times and get heads each time”, and then I do it, you’ll probably think that is quite impressive and either very lucky or (more likely) manipulated. But if I take a thousand people and get each of them to flip a coin repeatedly, and after each round of flipping I get all the people who got tails to stop, after 10 rounds I might well have a single person who got a sequence of 10 heads. But there would be nothing odd about that, and it certainly doesn’t require the person to have special powers or be “lucky” (in any non-scientific sense).

Apple now has the largest market capitalization of any company in the world, with massive success and profits, after a series of audacious, high-risk moves that worked out. I’m not saying for a moment that this is pure luck, or that Apple is like the one-in-a-thousand kid who got ten heads in a row, but I do think that the world has annointed Apple after the fact when many other companies have made “audacious” moves that didn’t work out and, in some cases, sent them under. (Time-Warner’s merger with AOL was certainly audacious.) I think the modern Apple has made a series of good moves, and combined those with backtracking where necessary (allowing native apps, giving Final Cut Pro a stay of execution, allowing various apps and books into the stores after poorly judged bans etc.), and has won for reasons that combine skill and luck; but even if Steve Jobs had lived, that doesn’t mean he didn’t have more lemons in him, and that these might not have come out and eventually hobbled or even killed the company.

Even today, when Microsoft is routinely left off the list of the key tech companies (now typically Apple, Google, Facebook, Amazon), or is at least seen as the laggard among these, it remains massively profitable and powerful. I think it is in terminal decline, and deserves to be, but it is far from irrelevant yet.

Everyone Needs a Little Privacy

2011-11-02T10:48:00.001Z

I’m delighted to announce two new features in Fluidinfo—the existence of a private namespace for everyone, and the creation of a readable Fluidinfo version number. Thanks are due to Jamu (@jkakar) and Manuel (@ceronman) for implementing these changes. They were both my requests/suggestions, so blame me if you hate them.

The private namespace¶

If you have an existing Fluidinfo account, you will now find that you now have a top-level namespace called private. If you had such a namespace anyway, it won’t have been touched, but if you did not, it has been created, and all of its permissions have been set, as the name suggests, so that only you can access it.

Using Fish (or the online version at http://shell-fish.appspot.com), you can see this by using the ls command. So for me (logged in as njr), I can get a Unix-style listing of the permissions with:

$ fish ls -ld private
nrwc------   njr/private/

or a more detailed, Fluidinfo-style listing by saying

$ fish ls -Gd private
njr/private/:

NAMESPACE (/namespaces)
  Read
    list (read):        policy: closed; exceptions = [njr]
  Write
    create (create):    policy: closed; exceptions = [njr]
    update (metadata):  policy: closed; exceptions = [njr]
    delete (delete):    policy: closed; exceptions = [njr]
  Control
    control (control):  policy: closed; exceptions = [njr]

New users registering from now on will also have this namespace created for them, with the permissions set to private.

We have given everyone a private namespace by default for two main reasons:

It means that, without having to know anything about how the permissions system works, all Fluidinfo users have any easy way to create both private and non-private information: if you want your data to be public, by default all your tags not under the private namespace are readable by everyone (but writable only by you); where you want data to be private, just use a tag in your private namespace.

Just as importantly, we expect that many Fluidinfo applications will now place some data in a user’s private namespace (as appropriate) and some at the top level, or in other namespace that aren’t private by default. So an application to allow users to share clipboard between devices, for example, would probably use private tags by default, but a rating application would be more likely to use tags that are public by default.

Just as most people never change any file permissions, we expect that most Fluidinfo users will not want to be messing around with tag permissions, and this change means they should not have to do so.

FAQ¶

Q1: I hate this: I don’t want a private namespace! Can I get rid of it?
A1: Of course. You can safely delete the private namespace. In Fish, assuming you haven’t added anything to it you can do this simply by saying:
```
$ fish rm private
```
(The fish part of all these commands can be omitted if you are using the online version of Fish.)

Or, if you prefer, you can alter the permissions on the namespace to make it non-private. For example, if you say
```
$ fish perms default private
```
that will reset the permissions on the namespace private to the defaults (making it readable by everyone, but writable only by you). I wouldn’t particularly recommend this course, but it’s up to you.
Q2: I already had a namespace called private, but its permissions aren’t set correctly. How can I fix them up?
A2: You can do this with Fish’s perms command:
```
$ fish perms private private
```
Note that this only changes the permissions on the private namespace itself. If you have tags or namespaces under it, their permissions will not be affected, but new tags and namespace that you create under private after this change will be affected. (I will add a -r option to Fish soon, for recursive permissions changes, but it isn’t there at the time of writing.)
Q3: I used private as a tag: have you just trashed all my private tags?

A3: No. Fluidinfo is quite happy for you to have a tag and a namespace with the same name. If you had a private tag, this hasn’t been affected at all by the new private namespace.
Q4: Is everything under my private namespace private?

A4: If we’ve just created the private namespace for you, yes; but if you had one already, it depends:

A few months ago we made a change to Fluidinfo so that when a new tag or namespace is created, it inherits permissions from its parent namespace. So everything works as you would expect, and if you create new tags and namespaces under your private namespace, everything will work as any reasonable person would expect.

It is important to note, however, that the Fluidinfo permissions system is not hierarchical, in the sense that changing the permissions on a parent namespace has no effect on its existing child namespaces and tags.
Q5: Will I run into problems if I buck the trend and don’t have a private namespace called private or if I make other tags and namespace private?

A5: No. It is obviously possible that Fluidinfo applications you use might make assumptions that your private namespace is private, or that it exists, or even that tags not under your private namespace are public, but such apps would be making unjustified assumptions.

There should certainly be no problem at all with making any tag or namespace you like private, whether it is under the private namespace or not. Indeed, you can make your top-level namespace private if you like (though again, doing so will only affect new top-level tags and namespaces).

But if you don’t have a strong philosophical objection, your life probably be simpler if you go along with having a namespace called private, and probably by making its contents private too.

Fluidinfo API Version and Release Date¶

The other small but useful change is that Fluidinfo is now publishing information about the versions of the code and API deployed.

Every time a new version of the live Fluidinfo code is deployed, the tag fluiddb/release-date on the object with about tag fluidinfo will be updated. This is a plain text string (MIME type text/plain) in ISO8601 extended format. Because it is a plain-text string, you can view it in a browser at

http://fluiddb.fluidinfo.com/about/fluidinfo/fluiddb/release-date.

or with Fish by saying

$ fish show -a fluidinfo /fluiddb/release-date
Object with about="fluidinfo":
  /fluiddb/release-date = "2011-11-04T17:40:27Z"

The intention and commitment from the team is that every time the live Fluidinfo code changes, this release date will be updated (even if there is no intended change to the API). This obviously allows library and application writers to make statements such as:

Fish version 4.12 has been tested with the Fluidinfo release of 2011-11-04T17:40:27Z and all tests pass.

(which is true).

There is also an API version published using the tag fluiddb/api-version on the same object. Again this has MIME type text/plain, and can be viewed at

http://fluiddb.fluidinfo.com/about/fluidinfo/fluiddb/api-version

It can also be displayed using Fish with the command

$ fish show -a fluidinfo /fluiddb/api-version
Object with about="fluidinfo":
  /fluiddb/api-version = "1.13"

This is updated whenever a deliberate change to the API is made. If, at some point in future, multiple APIs are supported, the intention is to extend this to be a space-separated list of APIs supported in the current release.

The API change log can be viewed at

http://doc.fluidinfo.com/fluidDB/api/changelog.html

and I’m guessing that from this point forward the change log will include the API version numbers for new features.

Tagging Music and Bands (or Anything Else) in Fluidinfo

2011-10-10T23:36:00.001+01:00

I’ve put a modified version of the About Tag web site up at ArtOfTagging.com.

Here’s a short video that illustrates its use, specifically with the data from musicbrainz.org that Eric Seidel (@gridaphobe) heroically imported into Fluidinfo.

The new ArtOfTagging.com allows you to do search-engine search over the Fluidinfo About Tags, to look up a specific, known about tag, to tag objects (if you log in and assiciate a Fluidinfo account) and to run Fluidinfo queries. It also includes the latest incarnation of the object visualizations, now inline and set up so that they update if you tag something.

If you want to know more about the about-tag conventions used in this, see The Music of Fluidinfo II.

Python Anywhere (featuring Emacs Anywhere!)

2011-10-10T19:32:00.001+01:00

Python Anywhere is a fascinating new web service (currently in beta) from the good folks at Resolver Systems. I’m using it to host the Django port of my AboutTag web application, used, among other things, for searching Fluidinfo, but I’ll blog separately about that.

I first came across Resolver Systems a few years ago when their main focus was on Resolver One a spreadsheet that you program in Python. This is so obviously a good idea I was strongly rooting for Resolver, but competing successfully against Excel, whatever the merits of the alternative, was always going to be a tall order, and last time I heard Microsoft still had the edge in the spreadsheet market.

Python Anywhere grew out of a project Resolver ran to produce a web service—Dirigible—based around the same fundamental concept of a python-programmable spreadsheet. Digirible is also notable for its support for grid computing, so it offers potential to perform really large calculations while exploiting a spreadsheet structure and all the goodness of Python, Numpy and parallelism.

Python Anywhere retains Dirigible’s concept of writing python in a browser, but dispenses with the spreadsheet. So in a nutshell, Python Anywhere is marketed as a Python console you can access through a browser. But it’s actually more than that.

When you log into the Python Anywhere site, your primary choice is to launch any of half a dozen different kinds of interactive Python consoles (Python 2.7, Python 2.6, Python 3.2, IPython 2.7, IPython 2.6 or IPython 3.2). These are all well configured in terms of libraries, and seem to work well, and they absolutely allow you just to use Python interactively. But so far, I’ve made very little use of any of those, for the same reason that most of the time when I write Python on my own computers I spend comparatively little time in the interactive Python consoles, and I when I do, I prefer to launch them from a terminal (or Emacs).

With Python Anywhere, I almost always use the seventh option, which is to launch a Bash console. From there, I have access to all the same Pythons, and most of the Unix (Linux?) tools you’d expect.

Including Emacs.

Including Emacs.

I am one of those people who spends more time in Emacs than in all the rest of my applications put together. And while Emacs in a console in a browser isn’t perfect, the achievement in making it work at all (and quite well) is remarkable, and the result is pretty good. Even if Python Anywhere did not actually offer Python, for me the availability of Emacs, from (almost) any device, through a browser would be very interesting.

But while I prefer to do it from bash than in a Python console, the availability of well-configured Pythons through a browser is also fascinating.

One clear use case is access from iPads, iPhones and other devices that don’t allow you to run Python on them. Although this isn’t working particularly well yet, this is clearly in Resolver’s sights, and I think will be most useful. (Even I have to admit, however, that Emacs and the iPad/iPhone don’t seem like a match made in heaven; I can see I might have to dust off my vi skills, though maybe the lack of an escape key on an iPad is a problem even for vi? Presumably it can be rebound?)

The Python Anywhere Platform also offers other services. There’s Dropbox integration for transferring files, and good access to things like Github. There’s the ability to schedule jobs (cron, essentially) and they are trialling hosted web applications through Django and other WSGI-compatible frameworks. My port of AboutTag (the new version is available at ArtOfTagging.com, which redirects to Python Anywhere) is running on this, and looks great. Now obviously there are lots of web hosting platforms around, with various pros and cons, but I’m not aware of any others that you console access through a browser, which I can see being handy for all those critical fixes when your only available device is a friend’s machine, or a phone or an internet café.

I don’t know what the future holds for Python Anywhere, but it’s a fascinating project from some very smart and friendly people, so if you like Python (or Emacs, (or vi) or Bash, or any of that command-liney goodness), head on over and request an invitation to the beta.

Sequences in Fluidinfo

2011-08-12T08:09:00.002+01:00

I’ve added a powerful new feature to Fish to allow it manage arbitrary sequences. I’ll introduce the feature first, and then there are a couple of sections later on that motivate them, for anyone interested.

Sequences in Fluidinfo¶

A Fish sequence is simply a numbered collection of textual items that is added to over time. For example, I have a sequence called thoughts in Fish. When I have a new thought, I say something like:

$ fish thought "Perhaps savethewords.org would like to add threatened words and their definitions to Fluidinfo"

to which Fish responds:

10: Perhaps savethewords.org would like to add threatened words and their definitions to Fluidinfo
2011-08-11

This provides confirmation that the thought has been recorded (since this data is read back from Fluidinfo after the write), that it is thought #10 and that it was recorded on 11th August.

I can look at my last five thoughts by using this command:

$ fish thoughts 5

6: The Art of Tagging & the Tagging of Art
2011-07-31

7: Integrated fish history across all machines and shell-fish
2011-08-02

8: We need a word for words like suBtle, that are self-descriptive in non-onomatopoeic ways.
2011-08-08

9: A website featuring dodrantal things
2011-08-10

10: Perhaps savethewords.org would like to add threatened words and their definitions to Fluidinfo
2011-08-11

It defaults to the last 20 if I don’t specify. I can also filter by putting search terms on the command line:

$ fish thoughts art
6: The Art of Tagging & the Tagging of Art
2011-07-31

and I can specify a range of thoughts:

$ fish thoughts 8-9

What I’ve built into Fish isn’t, of course, a thought command, but a general capability to create sequences. The base command is mkseq, which makes a new sequence, and whose general form is:

mkseq sequence-name [plural-form [tag]]

The thought and thoughts commands were made using the command:

mkseq thought thoughts private/thought

The first argument is the name of the command to add an item to the sequence. The second argument, which is optional, is the plural form, which is used to list items in the sequence; if it is not specified, an s is added to the singular form. The final argument, also optional, is the tag to use for storing the sequence. In this case, I wanted to keep my thoughts private, so I chose to use the tag njr/private/thought; if not specified, the name of the sequence item is used for the tag, in the user’s top-level namespace, i.e. njr/thought in this case.

[Readers with intimate knowledge of Fluidinfo’s permissions system will spot a flaw with this scheme: since Fluidinfo’s permissions are not inherited, simply using the tag njr/private/thought won’t make my thoughts private, even though my private namespace has its permissions set to deny access to others. However, that will change next week, when Fluidinfo will gain create-time inheritance of permissions; and I’m not going to release this functionality until then.]

The mkseq command creates two aliases, as we can see using the alias command:

$ fish alias
thought:
  njr/.fish/alias = "seq private/thought"

$ thoughts:
  njr/.fish/alias = "listseq private/thought"

This reveals that the other two commands I’ve added are seq and listseq. The seq command adds to a sequence using the tag specified, and the listseq command lists recent items from the sequence with the tag specified.

Because Fish stores aliases in Fluidinfo itself, if I use Fish from more than one place, I’m but a sync away from being able to record thoughts from any of those places.

The organization of sequence data¶

There are a number of ways that the trinity of sequence commands above could have been implemented, but this is how I’ve done it. I’m going to use thoughts as the exemplar, here, but what follows applies to any Fish sequence.

The first question is: what object or objects should be used to store the sequence? In Fluidinfo, it is very natural to use one object per thought. I could have used anonymous objects (one with no about tag), and that’s not a ridiculous idea in this case, where I’m not expecting sequences to make significant use of Fluidinfo’s social aspects. In fact, however, what I decided to to was to use the integers. So my first thought is stored on the object whose about tag is 0, the second is on the object with about tag 1 and so forth.

Thus, if I use Fish’s tags command, I should see my thought number 10:

$ fish tags -a 10
Object with about="10":
  /fluiddb/about = "10"
  /musicbrainz.org/related-albums = {"album:10 (asleep at the wheel)", "album:10 (box set)", "album:10 (cali gari)", "album:10 (divididos)", "album:10 (dj db)", "album:10 (enuff znuff)", "album:10 (freestyle)", "album:10 (harri marstio)", "album:10 (i8u)", "album:10 (johan johansson)", "album:10 (john anderson)", "album:10 (k s choice)", "album:10 (kate rusby)", "album:10 (linea 77)", "album:10 (liroy)", "album:10 (ll cool j)", "album:10 (mcguffey lane)", "album:10 (mercyme)", "album:10 (miskolci ütősök)", "album:10 (perplex)", "album:10 (sharon kips)", "album:10 (suat suna)", "album:10 (supersilent)", "album:10 (the dramatics)", "album:10 (the fools)", "album:10 (the stranglers)", "album:10 (user)", "album:10 (various artists)", "album:10 (wet wet wet)", "album:10 (тент)"}
  /musicbrainz.org/related-artists = {"artist:10"}
  /njr/index/about
  /njr/private/thought = "Perhaps savethewords.org would like to add threatened words and their definitions to Fluidinfo"
  /njr/private/thought-date = 20110811.0735
  /njr/private/thought-number = 10
  /wordtools/gcide = "10 \10\ adj.
1. denoting a quantity consisting of one more than nine and
one less than eleven; -- representing the number ten as
Arabic numerals

Syn: ten, x
[WordNet 1.5 +PJC]"

And indeed, in amongst other data, you see three njr-thought tag and couple of others. The thought itself is stored on the tag njr/private/thought, as a string. (Don’t try this at home though: njr/private is private, notwithstanding publication of select extacts on this blog, and will be invisible to users who are not njr.)

Its number is stored, as a numeric value, on the tag njr/thought-number: this might seem redundant, given that the about tag for the object also stores that information, but the fact it is numeric is extremely useful, allowing me to query based on the thought number using inequalities.

Finally, the date is stored as a numeric value. Fluidinfo doesn’t support a date type, so the way I decided to store dates was as a numeric value in the form YYYYMMDD.hhmmss. This again makes it easy to do queries based on a date range if I want to. I am just using the local time and not worrying about time zone issues. Using Greenwich Mean Time (GMT; or Coordinated Universal Time, UTC, as I believe modern anti-imperialists are supposed to call it) would have ensured that the time ordering of the datestamps was more consistent with the sequence item numbers, but at the cost of making querying less natural, especially for people not lucky enough to live near longitude 0º.

In addition to using these three tags to store the sequence itself, Fish uses one further tag to record the next sequence number to allocate, so as to avoid the need to perform a complex query to know this. The tag it uses is, in this case, njr/private/thought-next and it stores this on the object for the user njr. Taking advantage of command substitution through left-quoting, and Fish’s integration of the abouttag library, we can see this by saying:

$ fish show -a "`fish abouttag fi-user njr`" private/thought-next
Object with about="Object for the user named njr":
/njr/private/thought-next = 11

In case this isn’t clear, Fish first executes the command

$ fish abouttag fi-user njr

which returns the about tag Fluidinfo uses for the user object for njr. The show command then becomes:

$ fish show -a "Object for the user named njr" private/thought-next

which yields the expected result, 11.

That about tag convention¶

I imagine some readers might be raising an eyebrow at my use of the objects with about tags corresponding to the integers. Wouldn’t I normally argue that about tags should be specific, encourage sharing and discourage pollution of useful objects with irrelevant information?

All these things are true, but I think that, for the most part, they don’t apply in this case. I think most of these sequences will be private anyway, and in that sense there is no anti-social pollution involved in using the integer about tags. More fundamentally, however, even if they are not private, I don’t think there is any great need for these items to be social. Where data is not social, and especially when private, the choice of object in Fluidinfo is purely for the convenience of the user. There is both logic and convenience in using the integers.

There is a partial down-side to using the same objects for different sequences, and it is that any extra information needs to be stored in sequence-specific tags. This is one of the reasons that I used njr/private/thought-date rather than a more generic njr/datestamp. It also means that if I wanted to add tags to the object, they would also have to be similarly qualified. It may be that it will turn out that these are strong reasons to move away from the integers, but for the moment I feel the benefits outweight the disadvantages. I may add support later for sequences using either anonyous objects or ones with more specific about tags (like thought-10 or even njr/thought-10). But for now, this is it.

As always, I’ll be interested in any thoughts; I hope to roll this out next week semo time, after the API update.

I’ll finish with a couple of sections describing where the motivation for sequences in Fluidinfo and Fish came from, in case anyone is interested.

The Importance of the Palm Pilot¶

Before 1999, I had always had a problem organizing information. Wherever and however I recorded it, I would struggle to find it or access it when I needed it. The information came in many shapes and sizes—books I wanted to buy, recipes, phone numbers, quotations, thoughts, plans, ideas, locations of items I had found “a safe place” for and so forth. None of it was very voluminous, but I wanted access to it everywhere and I wanted to be able to search it and organize it.

In 1999, I bought a Palm V, and from that point forward it’s not much of an exaggeration to say that my problems with information disappeared. The Palm became the place to record all (low-volume) information. I carried it almost everywhere when I was out, and I synchronized it to my computer, and this meant I almost always had access to all of it

The Palm V was brilliant in a number of ways. It had four primary applications—notes, to-do lists, the address book, and the calendar. That doesn’t sound like much, but when combined with a couple of core feaures of Palm OS, and a little imagination, it became a tremendously powerful system for organizing little pieces of information. For me, the two extra crucial features were search and categorization. In 1999, Palm OS had full text search across all data in the major applications, and it worked superbly. It was actually better than most so called full-text search today. (As a trivial example, my iPhone today still won’t search photograph names.)

The second core feature of Palm OS that added to its organizational prowess was categories. Everything could be categorized, and though they were a little mean in allowing (I think) only 12 categories, that was just enough. I could organize my little bits of information into categories like books, food, travel, work and so forth. Even more importantly, I could keep separate to-do lists in a dozen or so categories.

Perhaps most interesting application to me was the Address Book, which I subverted as a database for anything to do with people. I kept a few categories of contacts in there (friends, family, business etc.), but I also had categories like books, music and quotations. For example, I stored quotations under their originator in the quotations category. I stored information about books under their author in the books category and so forth. It was mad, but worked remarkably well.

By being almost always with me, both at my desk and on the go, and providing categorization and full-text search, the data in the Palm could genuinely function as my central repository for little bits of information. A problem I had always had went away.

I moved from a Palm V to a Vx, then to a Sony Clié (also running Palm OS) and eventually to a Palm T|X, all of which were fine machines, and the data came with me. But four years ago, the iPhone was born, and there was not a moment’s doubt that this was the device that would finally allow me to move from carrying a phone and a PDA to a single device. (The Handsprings were never tempting, and nothing before or since, I would argue, comes close to the iPhone in overall power and utility.) But marvellous as the iPhone is, despite its ever-improving synchronization methods, the iPhone + Mac combination is not as powerful for organizing little bits of information as the Palm was.

The Power of Logging¶

I am not particularly happy about organizations collecting ever-increasing quantities of data about me, though like most people I put up with it in some cases because of the benefits that come with, for example, carrying a device that geolocates me day and night.

I am, however, much happier about the idea of recording information myself, for my benefit, under my control.

For the last four years or so, I have been working on a the Artists Suite, a set of data analysis tools whose lead component is Miró. From very early on, I was clear that I wanted the software to log almost everything it did. Miró is a command-driven system, and among things, it records every command issued to it (on a per-session basis) and all the results it generates. Miró never deletes any of its history (though users can, obviously) and as a result I now have nearly four years of logs detailing how I’ve used Miró myself—logs of well over 10,000 analysis sessions. This has turned out to be even more useful than I expected. It helps, I suppose, that as an analysis tool, Miró can read, search and analyse its own logs. But the fundamental power lies simply in recording everything, with no effort required on my part. Miró annotates the data, of course, with datestamps and sequence numbers, and records how the session was invoked and so forth, and I regularly go back and use the logs to reconstruct what I did previously in a way I simply would be unable to do otherwise. This had made a poerful impression on me.

Gigs in Fluidinfo

2011-08-11T14:47:00.003+01:00

I’ve written previously on about-tag conventions for music in Fluidinfo, and if you search for almost any artist or album in Fluidinfo you’ll see the great progress that Eric Seidel (@gridaphobe) has already made in this area, with more to come.

So far, the main conventions I’ve discussed concern artists, albums and tracks, which are exemplified by

artist:pink floyd

album:the dark side of the moon (pink floyd)

track:speak to me (pink floyd)

What about gigs?

My first thoughts are the following:

Most gigs are primarily defined by an artist (or sometimes artists) and a date.

Very occasionally an artist may play two gigs in one day; in that case, adding a venue or a time ought to be sufficient to disambigate it

As usual, these things are easy to know if you attend the gig, have a ticket or a listing, and are pretty unambiguous.

I therefore think there are two prime options for defining gigs. The first would to use only the artist and date, in the main case, and to qualify when necessary with a disambiguating time (this being less ambiguous than place). The second option is to use artist, date and start time in every case. (I suppose time zone considerations could mess up the non-ambiguity of full datestamps, but I don’t care.)

The former has the considerable advantage of simplicity, and I would imagine will be unique well over 99.9% of the time; the latter is more precise and uniform, but has the significant disadvantage that start times are harder to know and less clearly defined.

On balance, therefore, I am tempted to suggest that gigs be exemplified by the following:

gig:dean friedman (2011-08-10)

and, where an artist performs more than once on the same day, and when it is important to distinguish:

gig:dean friedman (2011-08-10:21:00)

(I think sub-minute accuracy will probably not be required in this case.

As an example, Dean Friedman, surely one of the finest songwriters and performers playing today, performed last night, as part of the Edinburgh Fringe, at the Music Box, at Stevenson College. Here is the object for that gig

gig:dean friedman (2011-08-10)

and my here is the address of my njr/review tag for it:

http://fluiddb.fluidinfo.com/about/gig:dean friedman (2011-08-10)/njr/review

It’s value is:

$ fish show -a 'gig:dean friedman (2011-08-10)' review
Object with about="gig:dean friedman (2011-08-10)":
  /njr/review = "http://fluiddb.fluidinfo.com/about/gig:dean%20friedman%20%282011-08-10%29/njr/review.html"

That URL, points to the review itself, which is stored in Fluidinfo, in the tag njr/review.html, which can be view at http://fluiddb.fluidinfo.com/about/gig:dean friedman (2011-08-10)/njr/review.html

A reasonable alternative to gig be concert, which I suspect will be used in preference for classic music, but the ubiquity of the term gig for non-classical music (and even, colloquially, for classical concerts) suggests that gig is the better starting point.

Alias Fish and Sync

2011-08-09T07:56:00.002+01:00

I’ve been making a number of changes to the Fluidinfo shell, Fish. I haven’t pushed them to GitHub yet or the online version, Shell-Fish, yet, for various reasons, but I can start to document them.

Aliases¶

The provision of aliases is a fairly basic part of a shell that Fish has lacked until now. I have added a simple form of aliasing that is exemplified by the following example:

fish alias plp 'show -q "has njr/lastpage" /about'

This creates an alias called plp that expands to the text given so that plp will show the about tags for any objects tagged with the njr/lastpage tag. If I run this command now I get:

$ fish plp
1 object matched
Object f79d5ea3-50c1-4c9e-b98e-7bbe46b69ee1:
  /fluiddb/about = "http://www.guardian.co.uk/"

because the page tagged with njr/lastpage is currently the Guardian’s website.

A couple of syntactic details to note about these aliases:

The alias only applies to the first (non-flag) word in the command.
Any arguments that follow the aliased term are added to the substituted command. So, for example, the alias
alias parisrating 'show -a "Paris" rating'
if invoked as
fish parisrating /alice/rating
will show both my rating of Paris (as specified in the alias), and Alice’s, as specified in the command, viz:
$ fish parisrating /alice/rating
Object with about="Paris":
  /njr/rating = 9
  /alice/rating = "smelly"
There is no provision for using positional arguments such as $1 yet. This will change, as surely as night follows day.

So far, so boring. But here’s something slightly more interesting: where should Fish store its aliases?

Storing Aliases in Fluidinfo¶

The question need only be posed for the answer to present itself: obviously, Fish should store aliases in Fluidinfo; as it does. This turns out to be quite interesting.

Fish stores aliases on objects whose about tag is the alias; in other words, the alias parisrating is stored on the object whose about tag is parisrating:

$ fish tags -a parisrating
Object with about="parisrating":
  /fluiddb/about = "parisrating"
  /njr/.fish/alias = "show -a "Paris" rating"

As you can see, the alias is stored in a tag called njr/.fish/alias, and the value of that tag is simply the expansion text. (I know, I know: the output is confusing, embedding, as it does, double quotes in a double-quoted string, without any escaping. There is an excuse, but not one that’s worth the electrons.)

Both the .fish namespace and the .fish/alias tag are private, by default, as you can see:

$ fish ls -ld .fish .fish/alias
nrwc------   njr/.fish/
trwc------   njr/.fish/alias

or if you prefer your permissions Fluidinfo-style:

$ fish ls -Ld .fish .fish/alias

njr/.fish/:
     read: policy: closed; exceptions = [njr]
    write: policy: closed; exceptions = [njr]
  control: policy: closed; exceptions = [njr]


njr/.fish/alias:
     read: policy: closed; exceptions = [njr]
    write: policy: closed; exceptions = [njr]
  control: policy: closed; exceptions = [njr]

Of course, the user can change this.

Storing the alias this way involves a small leakage of potentially private information, in the sense that people can see that an object with the about tag parisrating exists, though not that anyone is using it as an alias, or who is doing so, or what the alias expands to. I could have avoided this by using anonymous objects and another .fish tag instead of the about tag, but I think the approach I’ve adopted is better overall. If you share my philosphical perspective that objects for every possible about tag already exist, but are lazily instantiated, there is no leakage, but obviously in the non-platonic Amazon server farm where the data is hosted, Plato’s writ does not run. (This is probably a good thing; imagine what Amazon would charge to host Fluidinfo’s data if we admitted that every possible string is stored.)

The principal advantage of storing the alias in Fluidinfo is that it is available from anywhere. So if I have Fish on several machines (and I do) I can create the alias once and use it from everywhere. In principle, I can also use it from the online version of Fish (Shell-Fish), but that isn’t implemented yet, which is one of the reasons I haven’t committed this to GitHub yet.

There is also a downside to storing aliases in Fluidinfo, which is that retrieving the definitions requires a Fluidinfo query. Given that alias expansion precedes command matching (so that built-in Fish commands can be replaced), this lookup is required before each command is evaluated; for Fish’s single-shot mode, where a single Fish command is entered from a Unix or Windows prompt, that introduces a delay I am not keen to accept.

For this reason, the Fish cache has been born.

Caching Fluidinfo for Fish¶

The Fish cache is simply a dump of a Fish’s internal representation of certain Fluidinfo objects to local storage. Specifically, on Unix, Fish writes a file (a pickle file) to ~/.fishcache.username where username is the authenticated user’s username. When Fish is invoked with arguments, it reads the cache, which includes all the objects used for aliases.

When an alias is created (or deleted), Fish first makes the change in Fluidinfo, then updates the cache.

There is then a new sync command, which updates the cache from Fluidinfo. It is important to be clear that this synchronization does not involve any kind of mediation: the cache is cleared and updated from Fluidinfo; Fluidinfo is the source of truth.

This means that in order to get any aliases (or alias deletions) performed by a different copy of Fish, you need to perform a sync operation.

When Fish is invoked without arguments, Fish performs a sync before accepting any input.

The plan for the online version of Fish is very similar to the interactive version except that instead of using local files, the online version will store its cache in the “local” database (which, in the case of Google App Engine, means the Data Store).

I think this approach holds great promise, not only for aliases, but for other important Fish data, probably including configuration parameters.

Summary of the alias, unalias and sync and showcache commands¶

alias

The alias command is summarized as follows:

alias [name [expansion-text]]

With no parameters, alias lists all aliases and their expansions.

With a single parameter, alias lists the expansion for the alias specified (if it exists).

With two or more parameters, alias defines (or redefines) an alias. It is best to quote the expansion text as a single parameter to stop Fish from interpreting it, though in simple cases this is strictly unnecessary. Here are some examples:

$ fish alias book 'abouttag book'

$ fish book 'Fugitive Pieces' 'Anne Michaels'
book:fugitive pieces (anne michaels)

$ fish alias book
book:
  njr/.fish/alias = "abouttag book"

$ fish alias
book:
  njr/.fish/alias = "abouttag book"

parisrating:
  njr/.fish/alias = "show -a "Paris" rating"

plp:
  njr/.fish/alias = "show -q "has njr/lastpage" /about"

unalias

There is also an unalias command. You can probably guess how it works. To remove the three aliases above you would say:

fish unalias book parisrating plp

showcache and sync

Finally, the showcache command can be used to show the contents of the cache, and the sync command can be used to update it from Fluidinfo.

In this particular case, I deleted the parisrating alias on another machine with this result:

$ sync

$ showcache
Cache:

  fluiddb/about="plp":
      njr/.fish/alias = "show -q "has njr/lastpage" /about"

  fluiddb/about="book":
      njr/.fish/alias = "abouttag book"

Today, the cache stores only aliases, so the output from showcache tends to look rather similar to that from alias, but as other types of data start to be cached, a sharper distinction will be drawn.

I have a good feeling about this.

Post Script

If you are using Fluidinfo directly, and are not taking advantage of the /values API, you really should check it out: it is dramatically faster. I have been a bit slow to upgrade Fish to use it, but that is now happening incrementally, and is yielding very significant and welcome performance improvements everywhere. Think of the difference between drinking beer with a straw and glugging it straight from the glass; there is simply no comparison.

[Thanks to @joannescrub for taking the time to point out a number of typos in this post.]

The End of the Fish Wars: The Fluidinfo Shell Version 4.0

2011-07-31T19:44:00.002+01:00

A funny thing happened. Terry (Jones; Fluidinfo creator, founder, CEO etc.) started using Fish and declared that he preferred using Unix-style paths in it. Given that the whole reason for adding support for non-Unix-style paths was the distress Unix-style paths appeared to cause Terry, this was a happy surprise. It turns out that I slightly misunderstood his position.
So with his complete blessing, the latest version of Fish defaults to Unix-style paths, i.e. except in queries, if a tag path does not start with a leading /, it is assumed to be in the authenticated user’s namespace; paths for tags and namespaces of other users are introduced with a leading /. So if I use fish, my rating tag is rating, but Terry’s is /terrycojones/rating.
If you already use fish, this affects you only if you have been using Fluidinfo-style paths without setting that explicitly in your ~/.fluidDBcredentials file. If that’s the case, and you want to continue using Fluidinfo-style long paths, add a third line to that file as follows:

username
password
unix-style-paths false

Users of te online version, Shell-Fish, won’t see any difference when they’re logged in as the preference will be remembered. But if you use the fish user, it now uses Unix-style paths.
The new consensual goal is to make Fish report as much as it can using Fluidinfo-style paths, but to default to unix-style for input. So the abbreviated form is seen primarily as a shortcut for input (which is, indeed, what it is). The complication is that this makes it harder for Fish to consume its own output; so I will make changes slowly, trying to ensure that things like command substitution with left quoting are facilitated as much as possible.
There are a few minor improvements to Fish in 4.00 as well as the paths change. It’s gained exit and quit commands to leave the interactive shell (though ^D (unix) and ^Z (Windows) still work). And it’s gained a get command, which is like show, but more minimal, returning the information you request with no extras. Compare and contrast:

$ fish show -a Paris rating /about
Object with about="Paris":
  /njr/rating = 10
  /fluiddb/about = "Paris"

$ fish get -a Paris rating /about
10
"Paris"

The get form is motivated partly, again by, a desire to make it as simple as possible for Fish (and other things) to consume the output.
There’s also better mobile support in the online version, Shell-Fish, but that’s another blog post (coming shortly).

Movement and Renaming In Fluidinfo and Fish

2011-07-26T20:49:00.000+01:00

The Fluidinfo Shell, Fish, already includes Unix-inspired commands for many tag-management tasks including listing tags and namespaces (ls), removing them (rm), creating them (touch and mkns/mkdir) as well an analogue of the Unix chmod permissions-changing command (perms). What about copying and renaming?

The Unix command for renaming files is mv (short for move). I imagine this is briefly confusing for newcomers to Unix, but the command is actually well named when all its functions are considered. In Unix, to rename a file from foo to bar one says:

mv foo bar

The same form can be used to rename a directory, say from private to secret:

mv private secret

The true appropriateness of the name mv becomes apparent only when the command is used for it third form, in which the destination is an existing directory. If etc is such an existing directory then

mv foo etc

will move foo into the directory etc; or, if you prefer, it renames foo as etc/foo. The same thing happens with a directory, i.e.

mv private etc

will move the directory private into etc as etc/private. We can also move several files at the same time:

mv foo private etc

will move both foo and private to etc., which is handy.

Should Fish do something similar?

Moving Tags and Values in Fish¶

There seems to be every reason to support a similar mv command in Fish. As with files, we may wish to rename a tag or namespace, or to move it within the tag hierarchy, and Unix provides a well-thought-through, powerful and convenient template for this. Tags would behave like files, and namespaces like directories. But there is a complication, as we shall see.

In Fluidinfo, a named tag, such as njr/rating is an abstract entity: it doesn’t really store any significant information itself. The tag may be attached to many objects, with different values or none on each. We can think of those tags-attached-to-objects as concrete tags or tag instances.

So if Fish implemented an mv command as above, then if authenticated as user njr (and using Unix-style paths in Fish), the command

mv rating star-rating

would change the name of the tag njr/rating to njr/star-rating not just in one place, but everywhere it occurs in the system. Similarly, if njr/private is a namespace,

mv private secret

would rename not only the tags in njr/private, but on every object to which those tags are attached.

This is not so much a problem as an opportunity. There is definitely a need for an easy way to rename and move whole named abstract tags and namespaces, but it would also be useful to be able to move and rename individual concrete tags. For example, maybe I want to rename my njr/rating tag on Citizen Kane from njr/rating to njr/star-rating (preserving its value). (After all, film ratings are all about stars, right?)

The obvious thing to do there would be to add a -a to specify the about tag for the object:

mv -a 'film:citizen kane (1941)' rating star-rating

It could also accept -i, to specify by ID, or -q to choose a set of objects for which to apply the renaming with a Fluidinfo query.

So far, so good. But what if it is a different kind of movement that I desire? What if I want to move the tag to a different object? Perhaps I’ve used the wrong object, or wish to swap to a different convention for about tags. As an example, let’s suppose I missed out the year required in the film-u convention when constructing the about tag for the film Citizen Kane. How would I move the tag from one object to another?

Perhaps the most obvious approach would be to repeat the -a flag to specify two different about tags. The library Fish uses for processing command-line options does allow them to be repeated, so the command could be

mv -a 'film:citizen kane' -a 'film:citizen kane (1941)' rating

Notice that there tag appears only once, because here, it’s not being renamed. However, I really don’t like repeated tags; they are quite rare in Unix and feel sloppy to me. I’d be more tempted to use -b for the second one (from a to b; geddit?). So then it would be:

mv -a 'film:citizen kane' -b 'film:citizen kane (1941)' rating

You could also perform a renaming at the same time:

mv -a 'film:citizen kane' -b 'film:citizen kane (1941)' rating star-rating

which you might prefer to express as

mv -a 'film:citizen kane'/rating -b 'film:citizen kane (1941)' star-rating

We could do something similar with -i (-j for the second, I suppose); it doesn’t really apply in the case of -q since it would be optimistic in the extreme to expect pairs of about tags from two different queries to align correctly.

What I’ve described so far seems fairly workable, and is my current best guess at what I’ll try to implement. A cp command would be fairly similar, and would probably require a -R flag for recursion.

But there is an alternative that seems worth discussing, even if only to reject it.

The endpoint approach¶

I generally think of tags as being attached to objects, sometimes with values, pretty much exactly as the diagrams generated by http://abouttag.appspot.com show them. Here’s an example for Citizen Kane.

(You can generate this live, in a modern browser, by visiting here).

And I think of the (abstract) tags and namespaces as living in a tree, exactly like the directory structure in a hierarchical file system. Something like this, where the yellow folder icons represent namespaces and the tags are red.

The analogy can even be extended further by thinking of the value on a tag as being like a file’s contents; this seems particularly apt in cases where the tag values really do correspond to file contents, as is the case with Fish’s documentation. Using a pair of tricks stolen from Nicholas Tollervery (@ntoll), the documentation is actually stored in Fluidinfo, with the content of the index.html file stored in a tag called index.html under the fish namespace. So the full path for the tag that stores the front page of Fish’s documentation is fish/index.html and this (concrete) tag is stored on the object with the about tag fish. The URL for the documentation is then http://fluiddb.fluidinfo.com/about/fish/fish/index.html. The second “trick” is to set the MIME type of the value of fish/index.html on the fish object to text/html, which causes Fluidinfo to serve the content with that MIME type, so that as far as the internet is concerned, http://fluiddb.fluidinfo.com/about/fish/fish/index.html is regular HTML web content. The same applies to all the other pages. It will come as no surprise to learn that the fish documentation is published programmatically, and that I have vague plans to add a publish command to Fish to allow a directory tree of files to be uploaded to Fluidinfo as a set of tags on namespaces on a nominated object.

It will not have escaped your attention in this digression, that a crucial difference between a file in a filesystem and a tag in Fluidinfo is that the tag may have arbitrarily many different values, each attached to a different object. It’s like having your file system partially replicated on every computer in the world, with different content on the subset of files stored on each.

This suggests a different way of thinking about the relationship between the tag hierarchy and objects, which is really the way that Fluidinfo’s end-points use them. In this view, each object has a partial copy of the (abstract) tag hierarchy instantiated on it, with values that are specific (and hopefully appropriate) to that object.

This is neatly illustrated by looking at a recursive listing of the fish user’s top-level namespace:

$ fish ls -R

fish:
_static/            h                   p                   tags.html
a                   help.html           perms.html          test.html
about.html          i                   pwd.html            touch.html
abouttag.html       index.html          pwn.html            u
amazon.html         install.html        q                   unixlike.html
b                   j                   r                   untag.html
c                   k                   rm.html             v
cli.html            l                   s                   version.html
commands.html       ls.html             search.html         w
count.html          m                   searchindex.js      whoami.html
d                   mkdir.html          shell-fish.html     x
e                   mkns.html           show.html           y
f                   n                   su.html             z
g                   normalize.html      t
genindex.html       o                   tag.html

fish/_static:
basic.css           default.css         fish-doc-logo.png   pygments.css
def.css             doctools.js         jquery.js           searchtools.js

All the tags that look like filenames (the ones containing a dot) are tags that exist in exactly one place—on the object with the about tag fish. (They could occur in multiple places, however. For example, I could publish the full set of documentation to version-numbered objects fish v3.12, fish v3.11 etc., rather than constantly replacing the old with the new, as I do now.)

The other tags (those with single-letter names) are there to allow anyone playing with the online version of Fish (shell-fish) to tag things with a set of different tags, and might exist on any set of objects in Fluidinfo.

In this view, the full path for a concrete tag can be expressed either in terms of its about tag (if it has one) or its ID. In the case of fish/index.html, the only concrete instantiation taht exists today lives under the /about endpoint, at

/about/fish/fish/index.html

or, for those of you who prefer UUIDs, under the /objects endpoint, at

/objects/0e7b94c7-1b37-41c2-bd7f-1a376786aa4e/fish/index.html

This suggests an alternative possible syntax for moving concrete tags. We could recast

mv -a 'film:citizen kane' /rating -b 'film:citizen kane (1941)' star-rating

mv '/about/film:citizen kane/njr/rating' '/about/film:citizen kane (1941)/star-rating'

This approach is not without its attractions. It is rather elegant and unambiguous, and it completely eliminates the need for flags in this case, which can hardly fail to be a benefit. On balance, however, I think it’s probably not the way to go. It feels somewhat inconsistent with thre rest of Fish, it forces me to put the username in (which I don’t really want to do) and—to me—it feels slightly inverted (if logical). There’s also a potential problem that there’s technically ambiguity with respect to a potential user called about, but that probably isn’t such a worry.

But the more important reason is that I dont want to have to repeat the object specification if I do something like

mv '/about/film:citizen kane (1941)/njr/rating' '/about/film:citizen kane (1941)/star-rating'

In that case, the -a tag feels much better, and that seems like a strong reason for with with either -a and -b or repeated -a flags when shifting between different objects.

So at the moment I’m leaning strongly towards the first suggestions introduced above, using -a and -b as required. As ever, however, I’m interested in opinions. I expect it will be a while before I get around to implementing this anyway.

Command Substitution and the Interactive Fish Shell

2011-07-21T10:06:00.001+01:00

The Fluidinfo Shell, Fish (documented here) has a gained a few important new features. In summary, these are

Interactive Shell. If invoked without any parameters,
Fish will start an interactive session that allows you to issue a sequences of Fish commands (without prefixing them by fish).
$ fish
This is fish version 3.12.
> whoami
alice
> ls
books/              favourite-things    private/            test-fish/
comment             has-read            rating              to-read
> show -a 'book:animal farm (george orwell)' alice/rating
Object with about="book:animal farm (george orwell)":
  alice/rating = 2
> show -a "`about book 'animal farm' 'george orwell'`" alice/rating
Object with about="book:animal farm (george orwell)":
  alice/rating = 2
> ^D
$
Points to note include:
If the readline library is available to python, it will be used, allowing Emacs key bindings to be used to edit the command line (as well as arrow keys) and saving a command history, which may be navigated with CTRL-P and CTRL-N or up and down arrows. You can also view the history by typing history or h.
The interactive session is exited by typing the end-of-file
character—CTRL-D on Unix and CTRL-Z on Windows.

Command substitution with left quoting (“backticks”) is supported
as illustraed in the last example of the interactive session shown (explanation below).
Single and double quotes may be used interchangeably, and
are protected by each other. Quotes and other special characters may be escaped using backslash, which also escapes itself. So, for example, Alice's Adventures in Wonderland may be passed as a parameter with
"Alice's Adventures in Wonderland"
or
'Alice\'s Adventures in Wonderland'
Obviously, on Unix, when using the interactive shell, the
Unix shell does not process the command line, meaning, among other things, that there is no globbing (wildcard expansion).
Command Substitution. Unix users will be familiar with
command substitution, which now comes to Fish. The way this works in Unix shells is that any part of a command enclosed in left quotes (“backticks”) is evaluated before the enclosing command, and its output replaces it in that enclosing command. This is particularly relevant to the commands that build Fluidinfo about tags—abouttag, amazon and the new normalize command (see below). In Unix shells, it has always been possible to say:
$ fish show -a "`fish abouttag book 'animal farm' 'george orwell'`" alice/rating
The shell first evaluates the subcommand, which produces the output book:animal farm (george orwell) and then, after substitution, the shell executes:
$ fish show -a "book:animal farm (george orwell)" alice/rating
In Unix, double quotes are weaker than single quotes, and command substitution can still occur inside them, which is exactly what we need in this case. Fish now makes almost identical functionality available in its new interactive mode. In the interactive shell you can say
> show -a "`about book 'animal farm' 'george orwell'`" alice/rating
This command is identical except that the leading fish is no longer needed within the substitution command (though it can be used), just as it not needed in the outer show command.
Command substitution in Shell-Fish. The online version of
Fish, Shell-Fish also benefits from the new command substution capabilities, but does not yet include the history feature. Whether readline-like command line editing is available is a feature of the browser (today).
The new normalize command. A new normalize command has
been added to Fish. This simply performs some standardization on text, consisting primarily of mapping to lower case, removing most punctuation and standardizing whitespace; accents are preserved. If multiple arguments are passed, the components are joined with colons. For example:
$ fish normalize 'A *very* strange (and    wonderful) fish'
a very strange and wonderful fish

$ fish normalize golfer 'Darren Clarke'
golfer:darren clarke

Happy fishing!

Fish 3.07: More compact longer listings

2011-07-11T08:36:00.002+01:00

I made a minor update to Fish, available from Github. This fixed a rather nasty bug that had broken many aspects of the perms command and which also modified the behaviour of the ls -L command (longer listing, including Fluidinfo-style permissions summary). The behaviour that was previously generated by ls -L is still available with ls -G, which is exemplified by

$ fish ls -G njr/rating

njr/rating:

ABSTRACT TAG (/tags)
  Write
    update (metadata):   policy: closed; exceptions = [njr, miro]
    delete (delete):     policy: closed; exceptions = [njr, miro]
  Control
    control (acontrol):  policy: closed; exceptions = [njr, miro]

TAG (/tag-values)
  Read
    read (read):         policy: open; exceptions = []
  Write
    create (tag):        policy: closed; exceptions = [njr, miro]
    delete (untag):      policy: closed; exceptions = [njr, miro]
  Control
    control (tcontrol):  policy: closed; exceptions = [njr, miro]

The new behaviour for ls -L will present a more compact summary in the large majority of cases where the four write permissions for tags are identical and the two control permissions are the same. This simplified view is:

$ fish ls -L njr/rating

njr/rating:
     read: policy: open; exceptions = []
    write: policy: closed; exceptions = [njr, miro]
  control: policy: closed; exceptions = [njr, miro]

The longer view is generated when the permissions cannot be shown accurately in this more compact form. Analogous behaviour occurs for namespaces when their three write permissions are identical. For example:

$ fish ls -Gn njr/fi

njr/fi/:

NAMESPACE (/namespaces)
  Read
    list (read):        policy: closed; exceptions = [njr, miro]
  Write
    create (create):    policy: closed; exceptions = [njr, miro]
    update (metadata):  policy: closed; exceptions = [njr, miro]
    delete (delete):    policy: closed; exceptions = [njr, miro]
  Control
    control (control):  policy: closed; exceptions = [njr]

reduces to:

$ fish ls -Ln njr/fi

njr/fi/:
     read: policy: closed; exceptions = [njr, miro]
    write: policy: closed; exceptions = [njr, miro]
  control: policy: closed; exceptions = [njr]

The very perceptive reader will also notice that the longest listing now includes the write update (metadata) permission that controls access to writing the namespace description; this was missing in previous releases as a result of a typographical error in the source. (Computers can be so picky.)

About Tags In Fish

2011-07-07T08:27:00.001+01:00

I’ve added a new command to fish (and updated the online version, Shell-Fish accordingly) to allow easy construction of standardized about tags using the conventions from the abouttag library. They make use of a new abouttag function, available in the new generic.py file in the abouttag library, which takes the object type as its first parameter, and the usual parameters as a variable parameter list.
The new fish command is abouttag, though can also be abbreviated to about and its general form is:

fish abouttag <object type> <object specifiers>

The object type is something like book, album or fi-user and the object specifiers are the key parameters used to describe that object, in the same order as they are used in the corresponding function from the abouttag library.
The easiest way to illustrate and define these is with examples. The following examples are taken from a Unix system; on Windows, use double quotes rather than single around parameters. In the online version (Shell-Fish), and on Unix, single or double quotes work. In the online version, you don’t need the fish prefix (though it does work).
I should note that part of the motivation for adding this functionality is a desire to allow the command to be used to specify objects without knowing the exact form of their about tags. In Unix-like systems (Linux, Mac OS X, Solaris etc.), this is possible by using left quotes, which can be placed inside double quotes. Thus, the following, slightly ungainly command (using all three forms of quote) works, at least in bash:

$ fish show -F -a "`fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'`" njr/rating
Object with about="book:gödel escher bach an eternal golden braid (douglas r hofstader)":
  njr/rating = 10

I will leave it to the reader to judge whether this is easier than using cut and paste. For those who don’t know about left quotes in Unix shells, a command enclosed in left quotes within another command is evaluted before its enclosing command; its output replaces the left-quoted phrase on the original command line. So in the case above, we first run the command

fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'

which generates

book:gödel escher bach an eternal golden braid (douglas r hofstader)

as its output. In effect, the outer command is then transformed to

fish show -F -a "book:gödel escher bach an eternal golden braid (douglas r hofstader)" njr/rating

I hope to extend shell-fish, the on-line version of fish, to support left quotes, but that may take a little while.
The following examples are taken from the fish documentation, which is available online from http://fluiddb.fluidinfo.com/about/fish/fish/index.html.

Books and related items using the book-u convention (book, author)

$ fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'
book:gödel escher bach an eternal golden braid (douglas r hofstader)

$ fish abouttag book 'The Feynman Lectures on Physics' 'Richard P. Feynman' 'Robert B. Leighton' 'Matthew Sands'
book:the feynman lectures on physics (richard p feynman; robert b leighton; matthew sands)

$ fish abouttag book 'The Oxford English Dictionary: second edition, volume 3', 'John Simpson', 'Edmund Weiner'
book:the oxford english dictionary second edition volume 3 (john simpson; edmund weiner)

$ fish abouttag author 'Douglas R. Hofstadter' 1945 2  15
author:douglas r hofstadter (1945-02-15)

Music-related items (track, album, artist, isrc-recording)

$ fish abouttag track 'Bamboulé' 'Bensusan and Malherbe'
track:bamboulé (bensusan and malherbe)

$ fish abouttag album 'Solilaï' 'Pierre Bensusan'
album:solilaï (pierre bensusan)

$ fish abouttag artist 'Crosby, Stills, Nash & Young'
artist:crosby stills nash & young

$ fish abouttag isrc-recording 'US-PR3-73-00012'
isrc:USPR37300012

URLs and URIs (URI, URL)

$ fish abouttag uri FluidDB.fluidinfo.com
http://fluiddb.fluidinfo.com

$ fish abouttag url https://FluidDB.fluidinfo.com/one/two/
https://fluiddb.fluidinfo.com/one/two

$ fish abouttag URI http://fluiddb.fluidinfo.com/one/two/
http://fluiddb.fluidinfo.com/one/two

$ fish abouttag URL 'http://test.com/one/two/?referrer=http://a.b/c'
http://test.com/one/two/?referrer=http://a.b/c

Fluidinfo objects (fi-user, fi-namespace, fi-tag)

$ fish abouttag fi-user njr
Object for the user named njr

$ fish abouttag fi-namespace njr/misc
Object for the namespace njr/misc

$ fish abouttag fi-ns njr/private
Object for the namespace njr/private

$ fish abouttag fi-tag terrycojones/private/rating
Object for the attribute terrycojones/private/rating

Database components (db-table, db-field)

$ fish abouttag db-table 'elements'
table:elements

$ fish abouttag db-field 'name' 'elements'
field:name in table:elements

Miscellaneous (planet, element)

$ fish abouttag planet 'Mars'
planet:Mars

$ fish abouttag element 'Helium'
element:Helium

Like A Scrolling Terminal

2011-06-25T14:04:00.000+01:00

I have just updated Shell-Fish, the on-line version of the Fluidinfo Shell, fish, so that it acts rather more like a scrolling terminal, and rather less like a search engine.

Instead of being a “one-shot” application, where each time you type a fish command the screen is wiped and you lose all previous context, fish now simply scrolls the output, just a like a terminal. If you’re a command-line kind-of-a person, it will all look deeply familiar, except for the merest hint of AJAX spinniness.

The original inspiration for this was Stefan Grothcop’s wonderful Google Shell, goosh, which I’ve long used as my main interface to Google on Firefox. If you haven’t tried goosh, what are you waiting for?

Although the new scrolling version of Shell-Fish appears to work fine, I expect there are bugs: let me know if you find any. It still lacks a few basics, most obviously a history, but I plan to add that. Globbing too.

All in good time.

Securing shell-fish with fish: Even More on Fluidinfo Permissions

2011-06-19T16:39:00.002+01:00

I mentioned in my last post that I’ve extended the range of permissions that can be set with the perms command to cover the whole gamut. Here, I’ll describe this in the context of showing how I have set up permissions for the fish user to make them suitable for use with the online version of fish—shell-fish.

My goal was this. Shell-fish, the online version of fish, allows users to log in with their Google account and effectively link it to one or more Fluidinfo accounts. Users who do this can use fish to perform arbitrary Fluidinfo operations using their own tags. But I also wanted to allow users to try out Fluidinfo and fish without setting up an account. Initially, I did this using the Fluidinfo test user, but that’s probably not a good long-term solution. What I’d prefer to do is to provide a locked down account that anyone can use for certain things—not only for reading but also to try out tagging. The process of locking down a Fluidinfo account is also useful as an illustration of the extended perms command in fish, so that is the subject of this post.

I created the Fluidinfo fish user and switched to it in fish.

$ fish su fish
Credentials set to user fish.

$ fish ls -ld fish
nrwcr--r--   fish/

As you can see the permissions on fish’s namespace are the defaults, with the user having read, write and control permission, and everyone else having read only. (See this post or the fish documentation for the ls command).

The first thing I wanted to do was to transfer control permission from the fish user to me (njr) so that a user of Shell-Fish can’t do arbitrary things. This wasn’t possible with the original perms command, but I’ve added three new forms of the command, each of which follows the general template:

fish perms perms-class [open|closed] [except list+of+users] list of tags or namespaces

where perms-class is one of read, write or control. This simply lets the user set the explicit FLuidinfo permissions, as open or closed, with an optional exception list. The exception list is specfied as a list of users, separated by + signs. The command only changes the permissions class specified and changes all the permissions in that class for the tags or namespaces given. So valid examples operating on a tag fish/rating and a namespace fish/private include:

fish perms read closed except fish fish/rating fish/private
fish perms write closed fish/rating fish/private
fish perms control closed except fish+njr fish/rating fish/private

To transfer the control permissions to njr for the top-level fish namespace, I used the command (still as the fish user at this point):

$ fish perms -f control closed except njr fish

The only extra thing to notice here is the -f flag, which forces fish to make a change it would otherwise resist. Any change to permissions that results in the owner of a tag or namespace not having control permissions falls into this category, and since this is usually undesirable, fish requires the -f flag to force this to happen. (Tranferring permission to njr is, of course, irreversable for fish.)

Having made this change, if we repeat the ls command we see this:

fish ls -ld fish
(denied)    fish/

Fluidinfo allows only users with control permission to see the permissions on tags and namespaces, so the fish user can no longer see them. (I suppose this is slightly odd: it means that you can’t even see that you have write permission if you don’t have control permission. But that’s not a huge problem.)

Let’s switch to njr and try.

$ fish su njr
Credentials set to user njr.

$ fish ls -ld fish
nrw-r-cr--   fish/

This is as expected. The owner—fish—no longer has control permission on the fish namespace, but someone in a group does. We know that someone is njr. (We could verify this with ls -L, as we will after making some more changes.)

So far so good.

In terms of writing, my initial idea is to provide a set tags with single-letter names (a to z) in fish‘s top-level namespace that fish can write, but not to allow other tags or namespace to be written.

First, let’s create those tags as the fish user.

$ fish su fish
Credentials set to user fish.

$ fish -U touch a b c d e f g h i j k l m n o p q r s t u v w x y z

The touch command on a non-existent tag creates the tag, and the -U says use Unix-style paths, i.e. assume they’re in the authenticated user’s namespace unless introduced with a leading /. So this creates the 26 single-letter tags I want.

Next, we need to remove write permission from fish on the fish namespace, so that it can’t create new tags or namespaces.

$ fish su njr
Credentials set to user njr.

$ fish perms write closed except njr fish

$ fish ls -ld fish
nr--rwcr--   fish/

Now let’s think about the permissions on the tags we have just created. There is currently nothing hierarchical about Fluidinfo’s permissions. So switching back to the fish user, we see:

$ fish su fish
Credentials set to user fish.

$ fish ls -l
trwcr--r--   fish/a
trwcr--r--   fish/b
...
trwcr--r--   fish/z

So even though njr has control of the fish namespace, the fish user has control of the tags it created. I don’t want that, so let’s transfer them over:

$ fish -U perms -f control closed except njr a b c d e f g h i j k l m n o p q r s t u v w x y z

$ fish ls -l a
trw-r-cr--   fish/a

[Again, the -U saves me having to prefixed each tag with njr/. Of course, adding globbing (wild-carding) to fish would be even more helpful; I’ll probably do that soon, though interaction with the Unix shell’s globbing will be a slight issue there.]

So this is pretty good: the fish user has read and write permissions, njr has read and control, and everyone else has read only.

The one remaining problem is that there are several different write permissions on tags—permission to tag things with the tag (in native Fluidinfo API terms, create permission on /tag-values), permission to untag objects currently tagged with the tag (delete permission on /tag-values) permission to change the description of the tag (update permission on /tags) and permission to delete the tag itself (delete permission on /tags). The last of these is the problem: I don’t really want a random user of the Shell-fish tags to be able to delete any of fish/a to fish/z. To start with, fish couldn’t recreate them (because I’ve removed write permission from fish on the fish namespace) and if it did, fish would get control permission over that tag, which is what I’ve been trying to avoid. So I need to remove the fine-grained Fluidinfo permission to annihilate the tags from fish.

In order to allow this, I added a -X option to the perms command. This “eXtra” specification restricts which particular permissions within the class specified should actually be modified. -X is followed by the fish name for a low-level Fluidinfo permission, as shown in the ls -L “longer” listing. Let’s look at the detailed permissions on one of our tags:

$ fish su njr
Credentials set to user njr.

$ fish ls -L /fish/a

fish/a:

ABSTRACT TAG (/tags)
  Write
    update (metadata):  policy: closed; exceptions = [fish]
    delete (delete):    policy: closed; exceptions = [fish]
  Control
    control (acontrol): policy: closed; exceptions = [njr]

TAG (/tag-values)
  Read
    read (read):        policy: open; exceptions = []
  Write
    create (tag):       policy: closed; exceptions = [fish]
    delete (untag):     policy: closed; exceptions = [fish]
  Control
    control (tcontrol): policy: closed; exceptions = [njr]

Here, the Fluidinfo name for the low-level permission is shown first, and then parentheses, the fish name is shown. The fish names are unique for each kind of permission that exists on tags, and the one we are concerned with is the ability to delete the tag itself—the abstract tag. This has the same name, delete, in both systems. So I used the command:

$ fish perms -f write -X delete closed except njr /fish/a

$ fish ls -L fish/a

fish/a:

ABSTRACT TAG (/tags)
  Write
    update (metadata):  policy: closed; exceptions = [fish]
    delete (delete):    policy: closed; exceptions = [njr]
  Control
    control (acontrol): policy: closed; exceptions = [njr]

TAG (/tag-values)
  Read
    read (read):        policy: open; exceptions = []
  Write
    create (tag):       policy: closed; exceptions = [fish]
    delete (untag):     policy: closed; exceptions = [fish]
  Control
    control (tcontrol): policy: closed; exceptions = [njr]

As you can see, this changed only the delete permission from the specified write class, which is what we wanted. (If I had wanted to change several, I could have repeated the -X option.) I repeated this using the other 25 tags. If we now look at the result using ls -g, we get this:

$ fish ls -g /fish
tr/-r/cr--   fish/a
tr/-r/cr--   fish/b
...
tr/-r/cr--   fish/z

Reading across:

The t means that this is tag

The r/- means that the tag’s owner, fish, has read and some write permissions on the tag—in this case, all except the delete permission on the abstract tag

The r/c means that there is some group of users with control and read permissions, but only some write permissions. That group is in fact the singleton [njr], and he has delete permission only. (I could have actually just made the delete policy closed—no one needs to be able to delete the tag, and I have control anyway. Equally, I could have given myself all the other write permissions; but I didn’t.)

The final r-- means that the world has read permission, but not write or control.

So that’s it. The fish user can now tag and untag things using the tags fish/a through fish/z but has no ability to create other tags or namespaces, or delete the tag itself, and does not have control permission. The fish user can also in principle change tag descriptions, though fish doesn’t provide a way of doing that for existing tags right now.

I have just switched over shell-fish to use the fish user for non-authenticated users, so you can try this out now. In passing, if you use ls to list the fish namespace, you’ll see this:

_static/        genindex.html   mkdir.html      s               unixlike.html
a               h               mkns.html       search.html     untag.html
b               help.html       n               searchindex.js  v
c               i               o               show.html       version.html
cli.html        index.html      p               su.html         w
commands.html   install.html    perms.html      t               whoami.html
count.html      j               pwd.html        tag.html        x
d               k               pwn.html        tags.html       y
e               l               q               test.html       z
f               ls.html         r               touch.html
g               m               rm.html         u

If you’re wondering what all these tags-that-look-like-web-pages are, they’re tags that I use to store the HTML fish documentation in Fluidinfo itself. If you look carefully at the documentation path — http://fluiddb.fluidinfo.com/about/fish/fish/index.html — you will see that its root is a tag on the Fluidinfo object having the about tag fish. The name of that tag is fish/index.html, and its content is the HTML for the index page. The trick lies partly in the tag name and partly in setting its content type to text/html.

I have a script that does that, which will probably become a fish command called Something like upload or publish or files2fi.

Fishier Still: rm, extra perms and more with fish 3.01

2011-06-18T20:43:00.002+01:00

I’ve written a bit about the changes I’ve made to fdb as it became fish. I wanted to test it a bit more myself before pushing it to GitHub, but have now done so as well as making some more changes.

Key things to know:

fish 3.0 is simply a renamed version of fdb, reflecting the change in FluidDB’s name to Fluidinfo. It’s available from https://github.com/njr0/fish.

I haven’t changed the names of things like the credentials file (though its default position has moved to c:\fish\credentials.txt on Windows, as I said it would). I also haven’t changed the name of the FluidDB class internally, though many other internal names have changed to fish.

There are a few commands available in fish 3.01. The most significant changes are the addition of an rm command for removing tags and namespaces, and extensions to the perms command to allow more detailed control if required. (In fact, I believe all possible permissions can now be set with it.)

There are some new minor commands as well—touch and mkns/mkdir.

I’ve changed the error handling a bit, which should improve things, at least for some kinds of errors.

In a minor change, I’ve decided to allow -R and -r to be used interchangably to specify recursive descent. At the moment, that means that ls -r and rm -R will work as well as the more conventional ls -R and rm -r. In future, perms and a planned cp will probably support these options too. Even though, Unix users tend to know which is which, there seems no reason to insis that the “right” one be used in fdb.

The documentation is in a new (but even more natural) place in Fluidinfo — http://fluiddb.fluidinfo.com/about/fish/fish/index.html.

I haven’t removed fdb from GitHub. It’s more stable and the rename is a pain, o I’ll probably leave it there for 6–12 months before pointing it to fish, which is a new project. Fish is at http://github.com/njr0/fish.

The new features are all documented in the online documentation, but I’ll outline most fo the changes here as well, except for the extended perms options functionality, which I’ll cover in a separate post about how I used them to “secure* the fish user for use with the online version of fish—shell-fish.

rm¶

The most significant new command is rm. If you’re familiar with Unix, you’ll feel right at home, though there’s no globbing (wildcarding) yet. The rm command can be used to remove one or more empty namespaces or unused tags simply by giving the paths or paths. For example, given a tag njr/rating and a namespace private, the following work:

$ rm njr/rating
$ rm njr/private
$ rm njr/rating njr/private

If you want to remove a non-empty namespace, use -r to do a recursive removal. This will delete all the tags (removing them from objects in the process) and sub-namespaces before removing the nominated namespaces itself:

$ rm -r njr/private

If I followed the Unix analogy slavishly, fish would let you remove a tag even if it were in use (after all, files don’t have to be empty to delete them); however, the potential scale of inadvertant destruction of information is, in our case, large enough that I feel some proection is in order. So to remove a tag that’s in use, fish requires use of either the -r (recurse) or -f (force) flag.

As on Unix, -f is intended to encourage fish just to get on and remove things if humanly (fishily?) possible, and not to complain if the things you’re asking it to remove don’t exist. In time, this will extend to changing permissions, where possible, but right now if you don’t have the relevant permissions removal will fail.

There is no rmns or rmdir command, but fish will suggest you use rm if you find yourself typing them inadvertantly. (I never understood why you can’t remove an empty direcory with a plain rm on Unix anyway.)

touch, mkns and mkdir¶

Although the Fluidinfo API requires tags to be created before they are used, fish (like its predecessor fdb) has always created tags on first use, including required namespaces, but there are still a couple of situations in which you might want to create them explicitly. One is if you want to pre-permissions; the other is that you might wish to specify the description that both tags and namespaces can have. The relevant commands are:

$ touch user/tag
$ mkns user/ns
$ touch -m "tag description" user/ns/tag
$ mkns -m "namespace description" user/ns1/ns2

The first creates a tag called tag in user‘s namespace, the second creates a namespace ns in user‘s namespace, the third creates a tag called ns/tag in user‘s user/ns namespace, creating user/ns if required, and setting the description (metadata) to “tag description”, and the last does something similar for a namespace user/ns1/ns2.

C:> Using fdb On Windows

2011-06-16T21:55:00.001+01:00

It has become clear that fdb hasn’t been tested much on Windows. [1]

However, Over the last few days, fdb has been used a bit more on Windows, and as a result I’ve made some changes to make the experience on Windows a littler better.

Here are a couple of things that might be worth knowing if you’re using it on Windows. [2]

Credentials. In the past, fdb would look for a credentials file (containing username, password and optionally a tag-path preference) in the user’s home directory, but to do that reliably required a non-standard python library. That seems like an unnecessary dependency. So I’ve changed this. Now, on Windows, fdb consult the environment variable FDB_CREDENTIALS_FILE to find the location of said file; if it’s not set, the credentials file is assumed to be c:\fdb\credentials.txt (soon to become, c:\fish\credentials.txt).
Quoting Queries. Fluidinfo expects string values in queries to be quoted using double quotes. On Unix, the usual way to handle that is to use single quotes around the query and put double quotes within. For example:
$ fdb show -q 'fluiddb/about matches "café"' fluiddb/about
On Windows, that doesn’t work. Instead, you need to use double quotes in both places. Because that means having double quotes within double quotes, the inner double quotes have to be escaped, and on Windows, the way this is achieved is with “stuttering”, i.e. repeating the quoted quote. So the Windows equivalent of the command above is:
C:> fdb show -q "fluiddb/about matches ""café""" fluiddb/about

I guess I should also say that the above is based on Windows XP. I haven’t tried it on Vista or Windows 7. If you try it and run into different behaviour, or other problems, let me know.

[1]	Windows hates me and has always made it abundantly clear that it considers me to be an idiot, so I try to avoid it much myself.

[2] When I say “Windows”, what I really mean in using the Windows shell. I’m not entirely what the right name for that is. I mean thing that you get if you select Run... from the Start menu and then type cmd. The thing the used to be called the (MS)-DOS shell. The Windows command line. C:>! If you’re using Cygwin, it’s more-or-less the same as on Unix-like systems.

Of Fish, Shell-Fish and Fish Py

2011-06-15T09:21:00.000+01:00

FluidDB is dead; long live Fluidinfo.

Whither fdb?

Obviously, fdb should become fi; it’s perfect. Thirty-three-and-a-third per cent shorter is 33⅓% better for a command-line command. And fi is just so beautiful. It could almost become a ligature: how perfect would ﬁ be?

Except, of course, there’s one tiny problem. In Unix shells, fi is reserved as the closing counterpart to if. Even if I could make fi kinda, sorta work, I wouldn’t want to. The closing counterpart to if should be fi; it’s part of the cosmic order.

So what to do? The procrastinator’s dictum to the rescue:

Why put off till tomorrow that which doesn’t really need to be done until the day after that?

Why does fdb need to change at all? It could be a throwback, a reminder of glories past, a piece of Fluidinfo’s cultural legacy (along with the fluiddb superuser).

That’s what I thought.

Until I decided to put fdb into the sky. I’ve long thought it would be cool to have a browser-based version of fdb that anyone could simply use without installation. “No software”, as Salesforce.com likes to say.

For better or for worse, I tend to use Google’s App Engine to write web apps at the moment, so I went to register a new app there.

In Search of a Google App Engine App Name¶

Unfortunately, registering a new app is a bit like picking a domain; most of the desirable onare are gone already, not helped by the fact that all google usernames are considered taken. Add to that a minimum-of-six-characters requirement, and fdb looks to be in trouble.

As I was doing all this, I was chatting online with Terry (@terrycojones), who is the leading advocate of taking the DB out of FluidDB, and who, while entirely willing for me to plough my own furrow, had a very clear preference for expunging the db from fdb too.

Clearly, in reality, fdb is a shell for Fluidinfo. Unix has a long history of shells. In roughly chronological order I have used sh (the original “Bourne” shell), csh (the C shell), tcsh, ssh (Simon’s shell; not the Secure Shell; though I use that daily too), and now, always, nearly exclusively, bash, the truly wonderful Bourne-Again Shell. I’ve also dabbled with ksh, zsh and no doubt various others that fall into the large things-I-used-to-know category.

So give this, what would you call a shell for Fluidinfo? It just has to be fish. It’s screaming out to be fish. The only problem is that it’s thirty-three per cent worse (33% more typing).

Well, the fact that it’s 33% worse and a bit fishy.

Well, the fact that it’s 33% worse and a bit fishy and isn’t actually long enough to be a Google App Engine ID. (Too long and yet too short; not long enough and yet altogether too long. Such a paradox.)

FDB’s Fate Sealed by a Typo¶

As I was checking availability on App ID after App ID, I eventually got to shell-fish. Now shell-fish is just silly. I mean, it’s redundant (shell-Fluidinfo shell?). It’s hyphenated. It’s even fishier than fish. It sounds like a drunken version of selfish. Clearly, no person in his right mind was never going to choose shell-fish.

But then, instead of clicking the “Check Availability Button”, I typed return. And discovered that shell-fish was available. And that I had registered it.

Now, give me some credit. I do appreciate that this wasn’t really it. I could have changed it. But it could have been worse. I checked availability of fishnet (taken) and fish-net (available) and countless dozens I’ve have to go back to the IRC logs to recall so memorable were they. But in the end, I wasn’t convinced that I was gong to do better than shell-fish. And it does lock in fish, which is the perfect name for the Fluidinfo Shell—well, except for being 33% worse, and fishy, and not available as a Google App Engine ID, and . . .

Shell-Fish¶

So there it is. If you wish to be a guinea pig, head on over to http://shell-fish.appspot.com, where you can try fish online. It’s mostly the same as fdb was, and fish is, except that

you don’t need to prefix commands with fdb (or fish), obviously.

you are subject to the Google App Engine, 5–10 second maximum for an HTTP request. This can be an issue; timeouts are not uncommon, especially for complex queries, and when Fluidinfo is under load.

It’s almost certainly buggy and subject to change.

Right now, if you don’t log in, you will use the Fluidinfo test user. Before too long (when registrations are fixed), it’ll be a different user. But you can log in using your own Fluidinfo credentials if you like.

The way you do that is that you log into the appliction using a Google Account. (My app doesn’t get to see your Google password.)

Then, if you go into settings, you can add one or more Fluidinfo accounts by specifying your username and password. (You can also choose whether to use the default, Fluidinfo-style full paths for all tags and namespaces (njr/rating etc.) or whether your own tags and namespaces will be abbreviated to rating etc., at the cost of having to use a leading / for other people’s (/ntoll/rating etc.).

IMPORTANT: PASSWORD SECURITY

If you register a Fluidinfo username and password, shell-fish will store these in Google’s data store. I’m not particularly comfortable either with asking people for their passwords or with storing them, but I don’t think there’s much alternative at the moment. (There may be in an OAuth future.)

Obviously, before you hand over your password, you need to consider a few things:

Do you trust me? I could steal your password.

Do you trust fish? Even if you think me worthy of your trust, do you consider me competent? [Disclosure: sometimes, I make mistakes. See the discussion above on how shell-fish got its name.]

Are you happy with your password living in Google’s data store?

On the last point, I have taken what might be called minimal precautions. I do not store your password in plain text, partly so that should anyone happen to gain access to Google’s data store, they won’t just be able to read your password, and even more so that if I browse the shell-fish Google data store, I won’t inadvertantly see your password. (I’d have to decide to be evil.)

But I should also admit that what I’ve done to the password, while presumably technically qualifying as encryption, would probably be more accurately termed obfuscation. Let’s put it this way: it’s better than ROT-13, but it’s not as strong as PGP.

The other thing to know is that when you remove a user from your shell-fish settings, I simply the datastore (well, shell-fish tells the data store) to delete the record. I certainly don’t have access to it after that; whether a DrEvil@google.com could recover it, I know not.

Of Fish, Shell-Fish and Fish Py¶

So there it is.

I am in the process of changing the name of fdb to fish. (In fact, I’ve done it locally, but I don’t plan to push it to Github for a few days.)

The web app shell-fish is available at http://shell-fish.appspot.com for brave early adopters. I’m fiddling with it all the time. It will change a lot (most importantly, I hope it will end up looking more like a scrolling terminal than a one-shot search engine—think Goosh rather than Google. But right now, it’s very Google.

As fdb becomes fish, so will fdb.py become fish.py (geddit?).

One more thing . . . Amazon Product Pages¶

I’ll blog about this separately, but even if you don’t want to use fish per se, you might be interested in one neat little experimental feature.

At the top of the page in shell-fish, there’s a link called az-fish (though it might become amazon-fish). This is bookmarklet. Don’t click on it on the page; rather drag it to your tool bar. (It works even if you don’t give fish details of your Fluidinfo account.)

Then, click it when you are on an Amazon product page for a book, an eBook, a CD or an MP3 track. (I’ve only tested it on Amazon UK and Amazon US; it will probably need to tweaked for others, especially for non-English others.) It will take the URL and give it to fish (at shell-fish), which will attempt to figure out the about tag for the corresponding book, album or track in Fludiinfo, using its new amazon command (which may eventually become a thing command.)

How cool is that?

Permissions and Permissiveness

2011-06-12T11:44:00.000+01:00

It is a truth universally acknowledged, that a single namespace in possession of a good tag bundle must be in want of a good privacy policy.

Over recent weeks I’ve written quite a lot about the Fluidinfo permissions system, concentrating on what permissions Fluidinfo maintains, how we might represent those, list them and change them. Today I want to look at how they work, in practical terms, from a user’s perspective. There were some surprises for me as I tried them out. If you don’t want to read all the detail you can cut the chase by just reading the Take-aways

In what follows, I will assume you are familiar with the structure of Fluidinfo permissions and how fdb represents them and sets them.

If you haven’t already read the post on listing and setting permissions, you will probably find it helpful to do so before continuing.

Privacy¶

In what follows, I’m generally going to change permissions in one namespace, as user njr0 and explore the implications of this using the user miro, which I also control. If you want to do the same exploration and only have access to a single account, bear in mind that you can always user the test user (username test, password test) to try things out.

Let’s set things up first:

$ fdb su njr0
Credentials set to user njr0.

$ fdb -F tag -a http://google.com njr0/secret/password="very secret"

$ fdb -F ls -ld njr0/secret njr0/secret/password
nrwcr--r--   njr0/secret/
trwcr--r--   njr0/secret/password

$ fdb -F perms private secret

$ fdb -F ls -ld njr0/secret
nrwc------   njr0/secret/

So here we

set our credentials to njr0

Tag http://google.com with a password using a tag called secret/password

Check the permissions on the namespace (njr0/secret) and the password tag (njr0/secret/password)

See that they’re open, so change the permissions on the namespace to private. (In raw Fluidinfo-terms, means setting all permissions to closed with exception lists consisting of the owner—njr0—only.

Verify that only I as the owner (njr0) have any permissions on my secret namespace.

[If you’re wondering what’s up with all the -F flags, these tell fdb to use full, Fluidinfo-style paths; I have fdb configured to use unix-style relative paths that, by default, I don’t need to put in the leading njr0/; if you don’t change the default, you can omit the -F.]

Now let’s switch to Miró, and see how things look.

$ fdb su miro
Credentials set to user miro.

$ fdb -F ls -ld njr0/secret

$ fdb -F show -a http://google.com njr0/secret/password
Object with about="http://google.com":
  njr0/secret/password = "very secret"

There is no output from the first ls command because Miró does not have read permission (“list” permission, as it is officially known) on the namespace. But the last command (the show command) might come as a bit of a surprise.

Despite the fact that njr0 removed read permission on the namespace njr0/secret, miro can still read it.

Fluidinfo permissions are NOT heirarchical.

If you want to stop tags being read, you have to remove read permission on each one individually.

It’s not that the permissions system doesn’t work; it’s just a bit more—er—permissive than you might expect. (Terry calls it “bloody-minded”.) In effect, Fluidinfo’s read/list permission on namespaces is a bit like execute permission on a directory in Unix, with the difference that there isn’t another permission to close up the namespace completely.

Let’s round this part of the discussion off by closing down the permission on the password tag.

$ fdb su njr0
Credentials set to user njr0.

$ fdb -F perms private njr0/secret/password

$ fdb -F ls -ld njr0/secret/password
trwc------   njr0/secret/password

$ fdb su miro
Credentials set to user miro.

$ fdb -F show -a http://google.com njr0/secret/password
Object with about="http://google.com":
  (error code 401 (UNAUTHORIZED) attempting to read tag njr0/secret/password)

You might wonder whether this UNAUTHORIZED error is confirming that the tag is present on this object, but it isn’t. We get the same error if we try look looking for the tag on http://yahoo.com:

$ fdb -F show -a http://yahoo.com njr0/secret/password
Object with about="http://yahoo.com":
(error code 401 (UNAUTHORIZED) attempting to read tag njr0/secret/password)

$ fdb su njr0
fdb su njr0
Credentials set to user njr0.

$ fdb -F show -a http://yahoo.com njr0/secret/password
fdb -F show -a http://yahoo.com njr0/secret/password
Object with about="http://yahoo.com":
  (tag njr0/secret/password not present)

The current situation is plainly unsatisfactory.

A key reason for not wanting to make permissions hierarchical in Fluidinfo is that this would require checking up the hierarchy when reading them, which could be slow. We are, however, discussing a proposal that whenever a tag or namespace is created, the hierarchy be checked and the permissions be set in a way that is consistent with the enclosing namespace hierarchy. If that were the case, most of the practical problems would dissipate. Under this proposal, if you were to create a namespace, and then make it private, everything created under it would then become private. And if you decided to make an existing namespace private, you would only have to change the permissions on the subnamespaces and tags already in it to ensure that the whole namespace would be private. Indeed, if I were to add a -R option to the perms command in fdb, to descend the hierarchy recursively, this would be a single command. I will probably do that.

I hope this proposed change, or something similar, that will be made in Fluidinfo.

Losing Control: Beware Write Permission on Namespaces¶

Let’s now turn our attention to write permissions.

Fluidinfo actually maintains a plethora of write permissions, but fdb normally shows and maniplates them together for simplicity. Let’s see what happens if njr0 gives write permission on a namespace to miro. Specifically, let’s set things so that miro can read and write the secret namespace

$ fdb su njr0
Credentials set to user njr0.

$ fdb -F perms group miro njr0/secret

$ fdb -F ls -gd njr0/secret
nrwcrw----   miro   njr0/secret/

$ fdb -F ls -l njr0/secret
trwc------   njr0/secret/password

As you can see, njr0 retains read, write and control permissions on the namespace njr0/secret, but is now allowing miro to read and write the namespace as well. We also see that this hasn’t changed the permissions on the njr0/secret/password tag, which remains private to njr0.

So let’s get Miró to tag something for us. In particular, let’s get it to give Amsterdam an njr0/secret/rating of 2.

$ fdb su miro
Credentials set to user miro.

$ fdb -F tag -a Amsterdam njr0/secret/rating=2

$ fdb -F ls -l njr0/secret/rating
tr--rwcr--   njr0/secret/rating

Woah! There’s a permissions structure you don’t expect to see very often. Let’s expand it using -g

$ fdb -F ls -g njr0/secret/rating
tr--rwcr--   r:(world)  w:miro   njr0/secret/rating

Let’s read across the permissions and see what they say.

The leading t just means that njr0/secret/rating is a tag.

The next block of three (r--) shows the permissions that the owner—njr0—has for this tag. The owner only has read permission—no write permissions on the tag, and no control permissions. (The owner is not a happy owner at this point.)

Skipping to the final block of three (r--), this shows that the world (everyone, barring any group exceptions) has read permission on the tag, but no write or control permissions.

The “middle” block of three (rwc) shows the group permissions, and when listed used the -g opion, shows us that the read group is world (shown as (world)) and the write group is miro.

So the read permission is as expected. But the write permissions are set so that miro can write the tag, but the owner, njr0, cannot.

That’s worth saying again:

When someone other than the owner (the person in whose top-level namespace the tag exists) creates a tag, by default the write permissions are set so that the creator has write permissions, and the owner does not. The same goes for namespaces created by someone other than the owner.
If that weren’t surprising enough, look at the control permissions. The owner doesn’t have control permissions but the group does. The ls -g command in fdb doesn’t even show you who, because, to be honest, I wasn’t expecting control to be given away very much. I should probably make it so that it lists the control group, when there is a group exception.) The suspicion would have to be that the creator also gained control permissions, but let’s see by using ls -L, to give the longer Fluidinfo view of the permissions, in all their glory.
fdb -F ls -L njr0/secret/rating

njr0/secret/rating:

ABSTRACT TAG (/tags)
  Write
    update (metadata):  policy: closed; exceptions = [miro]
    delete (delete):    policy: closed; exceptions = [miro]
  Control
    control (control):  policy: closed; exceptions = [miro]

TAG (/tag-values)
  Read
    read (read):        policy: open; exceptions = []
  Write
    create (tag):       policy: closed; exceptions = [miro]
    delete (untag):     policy: closed; exceptions = [miro]
  Control
    control (control):  policy: closed; exceptions = [miro]
As suspected, miro has expropriated not only write permission, but also control permissions (the bounder!). Miró now has it, and there’s not a thing njr0 can do about it, other than appealing to Miró’s better nature.

Since Miró has control and write access, and njr0 doesn’t, njr0 can’t take back control of or delete the tag. You might think njr0 could take the drastic step of deleting the njr0/secret namespace, but he can’t even do that because a namespace has to be empty before it can be deleted, and njr0 can’t make it empty.

Again, this is worth emphasizing:

Be very careful granting write permissions on namespaces in Fluidinfo at the moment. If you let someone else write in your namespace, they will gain exclusive write and control permissions over any tags and namespaces they create under the namespace for which you gave them write permissions, and you can’t take them back.

“You give an inch, they take a mile.”

Obviously, when I say you can’t take them back, I mean that the Fluidinfo API won’t let you take them back. There really is a superuser in Fluidinfo (fluidddb) and if you were to explain your predicament to the Fluidinfo team, I’m guessing they’d be sympathetic. But even so . . .

This is an extraordinary and bad situation which I’m sure will be remedied. Clearly, it can’t be right that miro can end up have complete and exclusive control of a tag or subnamespace in my namespace without my ever having granted miro control permissions on anything.

In effect, at the moment, the creator of the tag is becoming its virtual owner. Indeed, the word “owner” becomes slightly problematical here. In the unix-like ls -l way of showing permissions, I have used the first group of three control indicators to show the permissions for the person whose Fluidinfo username is the first part of the full path to the tag or namespace—i.e., njr0 in the case of the tag njr0/secret/rating. I think this is natural and right, and I don’t plan to change it. I would even argue that it is consistent with Fluidinfo’s view, because Fluidinfo doesn’t maintain any separate notion of the owner of a tag or namespace. In fact, in some sense, it doesn’t have any notion of an owner at all: tags and namespaces only have permissions, and those consist of a policy (open or closed) and an exception list (a list of users to whom the policy is reversed).

Fluidinfo is perfectly content for no one to have any permissions for a tag or namespace (000). Or for everyone in the world except the person in whose top-level namespace it resides to have control permissions (policy: open; exceptions [njr0]—es exemplified by the James Bond 007 permission, or even 001 permissions).

“Yes, I do do see your predicament, old boy, but I’m afraid you really only have yourself to blame. I suggest you be more careful next time. Toodle pip!”

Bloody-minded isn’t the half of it. The team is aware of the situation, and again, I’m confident things will change.

To round this off, let’s just confirm that Miró can change things back.

$ fdb su miro
Credentials set to user miro.

$ fdb -F perms default njr0/secret/rating

$ fdb su njr0
Credentials set to user njr0.

$ fdb -F ls -g njr0/secret/rating
trwcr--r--   (world)   njr0/secret/rating

Sanity restored. Unlike Fluidinfo, when you set the permissions to default with the command

fdb perms default njr0/secret/rating

fdb takes default to be the default for the owner of the tag (whoever is root namespace the tag lives in), not the creator of the tag. Therefore, permissions revert to the default. (In this case, private might have made more sense.)

This has has the side effect of removing Miró’s write permission, since the perms command doesn’t have a way, at the moment, of saying give back control and write, but keep the exception on write. I might add such a thing, since it’s clearly useful in the present situation, or I might just implement chmod, which would probably let you do it by saying fdb chmod 760 njr0/secret/rating. To achieve full resporation, njr0 now has to re-grant the appropriate permissions, and indeed remove world read. In this case, the relevant commands are

$ fdb su njr0
Credentials set to user njr0.

$ fdb -F perms group miro njr0/secret/rating

$ fdb -F ls -g njr0/secret/rating
trwcrw----   miro   njr0/secret/rating

Here, we’re back where we expected to be. But it shouldn’t have been that hard, even though the last step was only necessary because the perms command doesn’t offer a way to hand back control permissions to the owner while keeping write permissions (because, when I implemented perms, I didn’t know just how bloody-minded Fluidinfo was.)

Take-aways¶

Here are the key take-aways:

Permissions in Fluidino are entirely non-hierarchical.

As a consequence of point 1, removing read permission from a namespace does not stop people reading tags or namespace under it. Any existing tags and namespaces have to be changed explicitly to remove read access, and any new tags or namespace created also have to be protected individually.

(You can change your defaults for new tags and namespaces using /policies if you like; but /policies affect all new tags and namespaces, not just a subset. It’s a bit like an non-inverted version of a Unix umask.)

When a new tag is created, its permissions are set according to the default permissions the creator, not the owner. This includes control permissions.

The direct consequence of point 3 is that, in normal circumstances, if someone else creates a tag or namespace somewhere in your namespace (which they can, of course, do only if you grant them the appropriate write permissions), they not you will end up with exclusive write and control permissions unless they explicitly change the permissions to that you have access.

Practically speaking, I think the consequence of this that you want to be very cautious granting write permissions on namespaces. Granting them on individual tags is much safer, because permissions are at the level of the whole tag (the abstract tag, if you will), rather than at the level of a particular tag on an object. As far as I can see, letting someone else write one of your tags only allows them exactly what you would expect: they can write that tag, delete it, change it and so forth, but not wrest control from you when you’re not looking.

The Music of Fluidinfo II

2011-06-09T09:10:00.005+01:00

I pushed an update to the abouttag.py library last night; it now includes some conventions and normalization for some music-related items. This supports work by Eric Seidel (@gridaphobe), who is looking at importing data from MusicBrainz to Fluidinfo.

These are similar (but not identical) to the ideas I proposed previously in the post The Music of FluidDB I: Albums, Tracks and Songs.

The first three kinds of things covered are works—named albums and named tracks respectively. As with books, this conceptual work seems like the single most important level to represent in Fluidinfo. So there may be many different issues, editions and pressings of The Dark Side of the Moon by Pink Floyd, but there is only one work. Even more clearly, Billie Holliday may have recorded God Bless the Child a number of times (I’m playing through several as I type this) , but there is only one conceptual work God Bless the Child sung by Billie Holliday.

Albums (as works). The convention for this, called album-u, is similar, but not identical to the convention for books, having the general form
album:name of album (artist)
The name of the album and the artist are normalized using the usual normalize function from abouttag.py, removing some punctuation, regularizing spacing and converting to lower case, but not removing accents. (I think it’s increasingly clear that removing accents was a mistake in book-1; I’m now recommending using the relatied book-u conventions, which is like book-1 except that it preserves accents.)

The other main difference between the book-u convention and the album-u convention is that in the case of books, multiple authors are consolidated into a standard list, separated by semicolons (in fact, a semicolon followed by a space). This works less well for music, where artists take more different forms Diana Ross and the Supremes, Pink Floyd, John Renbourn and Stefan Grossman, Crosby, Stills, Nash & Young etc. Since MusicBrainz, in particular, has a well-defined artist field, which is supposed to be the official recording credit, Eric is just planning to standardize that.

Example usage is:
from abouttag.music import album

print album(u"Solilaï", u'Pierre Bensusan')
print album(u"Déjà   Vu", u'Crosby, Stills, Nash & Young')
producing
album:solilaï (pierre bensusan)
album:déjà vu (crosby stills nash & young)
Of course, we may also create objects for paricular releases of an album, but those would use a different convention.
Named Tracks / Recorded Songs. These are have a form that is identical to albums except that they use track: as the prefix. Again, two different recordings of the same song get consolodated into a single (conceptual) track (convention track-u).

Example usage is:
from abouttag.music import track

print track(u'Bamboulé', u'Bensusan and Malherbe')

print track(u'''Archie Campbell/Marjorie Campbell/Miss Lyall's '''
            u'''Strathspey/Miss Lyall's Reel/The St Kilda Wedding''',
            u'The Cast'),
producing
track:bamboulé (bensusan and malherbe)
track:archie campbell marjorie campbell miss lyalls strathspey miss lyalls reel the st kilda wedding (the cast)
Recordings. In the case of conceptual tracks, it is particularly clear that same artist may record the same track several times. Happily, there is a standard identifier for such recordings of tracks, the International Standard Recording code. This is 12-character code, usually formatted for print as CC-XXX-YYY-NNNNN, where CC is a registrant country code, XXX is a registrant code, UU is the last two digits of the registration year and NNNNN identifies the recording.

Despite minor misgivings on my part (I would have chosen to keep the dashes, since Fluidinfo generally favours humans over machines), we have chosen to standardize this in the form isrn:CCXXXYYYNNNNN.

Examples:
from aboutag.music import isrc_recording

print isrc_recording(u'US-PR3-73-00012')
print isrc_recording(u'uspr37300012')
produces
isrc:USPR37300012
isrc:USPR37300012
Artist. The artist is simply identified by artist:name, where name is normalized as usual. Since accents are preversed, metal fans need not fear for their heavy metal umlauts (a.k.a. röck döts). For example:
from aboutag.music import artist

print artist(u"Crosby, Stills, Nash & Young")
print artist(u"Motörhead")
produces:
artist:crosby stills nash & young'
artist:motörhead'

Disambiguation, search and the related-to tag¶

I’ll do a longer post on this shortly, but an interesting an useful metaconvention is emerging within Fluidinfo.

I think there is a slightly reluctant but growing consensus within the Fluidinfo community that it makes sense to use about tags that exhibit pretty good uniqueness—I’ve suggested that at worst we should probably aim for conventions that mean that when we deal with a million objects we are unlikely to get a clash in about tags, not just within the list but with anything else anyone is ever likely to put into Fluidinfo.

At the same time, there is natural desire to make it easy for humans to navigate Fluidinfo via the natural about tags. Mercury the planet and mercury the element share the natural about tag mercury, but there is clearly a problem if we just stick information (like mass, or radius) on the object with about tag mercury. The about tags planet:mercury and element:mercury are much less ambiguous. (Unfortunately, in the early days I actually used planet:Mercury and element:Mercury; in my copious spare time, I’ll probably move the data over to the lower-case versions).

The emerging meta-convention is:

Store data on the most specific, unambiguous object reasonably available

book:animal farm (george orwell)

album: dark side of the moon (pink floyd)

element:mercury etc.

Help humans by also using the more natural, ambiguous objects

animal farm

dark side of the moon

mercury

as disambiguation nodes (à la Wikipedia), adding appropriate related-to tags that point to the various more specific items. The convention for these is that they are sets of about tag strings.

As an example, MusicBrainz might add a tag

musicbrainz.org/related-to=["album:dark side of the moon (pink floyd)"]

to the object with about tag dark side of the moon.

Similarly, an the Fluidinfo object with the about tag money might find itself tagged with (for example):

musicbrainz.org/related-to=["track:money (pink floyd)", "track:money (the flying lizards)",
                            "track:money (michael jackson)", "album:money (kfmdm)"]
miro/books/related-to=["book:money (martin amis)"]
imf.org/related-to=["economics:money"]

and so forth. This looks like it could have legs.

The key principle is: put the data on specific object at the right conceptual level, disambiguating as early as possible, i.e. don’t start by assuming that money could only refer to the Pink Floyd song, and use a convention that works well for all objects in the class of interest. Similarly, if you want put data that’s only applicable to a particular recording of that song, stick it on a more specific object (in this case, perhaps the isrn: object), probably adding a related-to tag pointing from the track: object to the various isrn: objects.

My only problem with this convention is that the term related-to sounds symmetrical, whereas it’s being use in a fairly directed way. Perhaps could-refer-to would be better.