Joseph Corneli The content of this file is available under your choice of the GNU Free Documentation License (FDL) and the GNU General Public License (GPL). If the FDL is selected, be advised that this document has no Invariant Sections, no Front-Cover Texts, and and no Back-Cover Texts.
(December 27, 2020 (draft))

A scholia-based document model for commons-based peer production informs an advanced Emacs-based hypertext system.



This document may look like a paper, but it is also runable code. The code implements a hypertext system. You can use this hypertext system to browse and edit the code and documentation that describes the system itself.

To do that, when viewing the document’s source code with Emacs, evaluate this form (in place):

(require ’cl)
  (let ((beg (search-forward "\\begin{verbatim}"))
        (end (progn (search-forward "\\end{verbatim}")
                    (match-beginning 0))))
    (eval-region beg end)

1 Prelude

Note 1.1 (The philosophy).

The main concern here is to model systems for collaborative authorship. What freedoms and limitations are embodied in such systems? Working to understand how to empower users will hopefully enable us to build a powerful and agile system; indeed, our goals here extend beyond “authorship”, to “readership” and query-answering; that is, from hypertext to artificial intelligence.

Note 1.2 (SBDM and CBPP).

The idea of a scholia-based document model (SBDM) for commons-based peer production (CBPP) is thematic for the work. Definitions for these terms suitable to the usage made within this document are approximately as follows:

  • CBPP: the processes by which shared information resources are produced by multiple co-authors.

  • SBDM: a framework for understanding the ways in which documents are put together out of smaller inter-referential components.

Note 1.3 (Bits and pieces).

Scholia-based documents tend to be comprised of small pieces whose semantic content has been illuminated through an ongoing process of annotation. This helps with refactoring, and with the process of producing new, meaningful, derivative works. Rich annotation also opens the way to increasingly powerful computer-based inference about the content of the document. This document should be a good example.

Author and summarizing example (
Figure 1: Author and summarizing example (après L. Weiner).

2 Introduction

Note 2.1 (A scholia-based document model for commons-based peer production).

The fundamental data structure used in this report is a simple database made up of articles, each of which has several freeform components. An article’s name is an arbitrary LISP object, typically a string (two articles with the same name are considered to be the same). The articles can contain text from a string, buffer, or file (as well as from other sources). An article is supplied with about and type metadata which say, respectively, what other pieces of the document the articles are about, and generally how they are to be treated by the system. A collection of articles can be viewed as a semantic network, with links from which about data is constituted comprising the edges between nodes (Note 3.11). (There are other sorts of links in the system as well, cf. Note 8.3.) Linking is how scholia are attached to other articles; the collection of all articles is “the commons”; and the rules for interacting with this collection defines the commons’s regulatory system.

Note 2.2 (Inspirations).

One of the first things that got me thinking about this stuff was the question of how to combine the contents of two digital libraries in a useful and non-disruptive way11 1 From here, the more technical question of how to do bidirectional updating between two digital libraries which share content and are changing in real time arose (see Section 10.1). But probably the most basic motivation comes from thinking about knowledge representation and human interface for the Hyperreal Dictionary of Mathematics project. The aim of this project is to get the wolrd’s mathematical knowledge into a coherent machine-readable and humanly-useful form. I anticipate that the first major applications of Arxana will be those directed towards building the HDM (see Section 12.7).

After thinking about these things for a while, I wrote up a few pages that expressed some ideas I later learned to be the central notions in Ted Nelson’s Project Xanadu (see Note 2.8). My interest in AI explains some of the differences between the two designs; but the fact that I’ve chosen to use Emacs for development probably explains even more.

Certainly the powerful text processing tools available in Emacs (ranging from text properties and overlays to self-modifying code) make it a great place to develop a next-generation hypertext system. Existing Emacs-based hypertext systems, including the emacs-wiki family of projects, and of course the built-in info and help subsystems, give some sense of how useful Emacs-based hypertext can be.

Much of the attitude towards AI developed here inherits from Minsky’s “Society of Mind”. In addition, a certain amount of experience working with contemporary AI systems (Cyc and KM) both got me to think about what an AI system is and what I would like to see “AI work” become.

The style of writing is inspired by Knuth’s “TeX: the program”. Here, instead of a typesetting program that can be used to typeset a description of itself, we have a hypertext system that can be used to browse and edit a description of itself. (Emacs had in fact pulled that one off previously, and to great effect, but in a rather different way.)

User freedom is probably the premier philosophical concern underlying this endeavor. My ideas about online freedom have something to do with what I know about MUDs (which isn’t a whole lot; mainly that they are sometimes user-hackable and that they resemble the text-based single-user games with which I am more familiar) and with my experience as a participant in various CBPP environments.

Section 13 delves further into these matters.

Note 2.3 (Straightforward applications).

Although I am envisioning the system being used to work on projects of grand scope and vision, it can also be used in some down-to-earth ways that are relevant to many people: for instance, the system can be used by document co-authors to annotate and comment on successive drafts of an evolving paper. Features to aid in this process have been present in MS Word for some time, to the best of my knowledge. (And in ASCII you can do it with brackets [ins: like this :ins], OR WITH CAPS.)

(Getting things set up to make quick and easy side-comments on a given text would be nice.)

Note 2.4 (Acknowledgements).

Aaron Krowne both helped inspire the initial design (through a series of discussions about PlanetMath, the HDM project, and bi-directional updating between digital libraries) and helped to give it a coherent initial expression22 2 Joseph A. Corneli and Aaron P. Krowne, A scholia-based document model for commons-based peer production. Proceedings of the Emory University Symposium on Free Culture and the Digital Library, 2005..

Ray Puzio helped to put the project on a sound theoretical and philosophical foundation as it developed.

2.1 Overview of the system

Note 2.5 (Scholia).

A “scholium” (plural “scholia”) is a comment on something else. We speak of a scholia-based document model (or SBDM for short) because speech is typically about something. We also refer to elements of the larger document as “articles”. If an article isn’t about anything (in the technical sense described in Note 3.7) then then it is considered to be degenerate as a scholium. We will also talk about “an article and its attached scholia” (regardless of whether or not the article is itself a scholium attached to some other article). Scholia need not be pieces of text: they can also be code or markup telling the reader (or the computer) how to process the article(s) they are written about.

Note 2.6 (Generalized links).

The about features described in Note 2.5 naturally associate documents or parts of documents with one another. This generalizes the notion of link familiar from the web (and other places). These familiar links are pieces of text with special markup which say that are about some other piece of text:

<a href="document this link is about">text</a>

In the scholium system, links can overlap and can point at more different kinds of targets than in typical hypertext systems. Links can also be given rich semantic properties by pairing them with additional type data and appropriately responsive procedures.

It is important to understand the fact that, in our system, markup and metadata are typically stored as scholia linked to appropriate targets.

Note 2.7 (Modeling the commons).

In addition to being capable of modeling many different kinds of documents, the system developed here should be capable of modeling and/or supporting the relevant features of the CBPP processes that create these documents. The SBDM is actually intended as a sort of meta-model: a model for modeling models, including models of documents, commonses, social contracts, culture building, and culture change. Cf. Section 12 and Note 13.41.

Note 2.8 (Relationship to Xanadu).

The system presented here isn’t meant to be an implementation of the Xanadu idea, per se, although it provides some of the features one would expect from a “Xanadu implementation.”

I should probably mention that I haven’t read all of Nelson’s stuff extra-closely, although I’ve looked through his books and enjoyed what I saw there. As the system here moves towards greater usability, I assume I’ll find the time to review the details from Nelson’s writing more closely and flesh out this comparison. And I hope to have a chance to review other systems that follow in the Xanadu lineage, e.g. token_word33 3 (which supposedly implements “almost every core feature of Xanadu”). I guess it’s the hacker in me that makes me want to write the code first; and certainly, the nature of the code I’ve produced thus far lends itself to being helpful with literature review tasks (cf. Note 12.11). I’m pretty sure that this work represents an advance, or at least steps in a new direction, for Emacs-based hypertext.

One clear difference between this system and the Xanadu system is that here articles are not supposed to be presented in a pay-to-access fashion. The idea of working on complicated systems for royalties or license maintenance doesn’t strike me as particularly compelling. Also, in this system, articles can be deleted (see Note 7.34), which as I understand it, is discouraged in Xanadu (more investigation is warranted and the system for deletion here needs finishing!).

To sum up, my overall sense is that Arxana, combined with simple internet protocols could, very nicely, facilitate something Xanadu-like, suitable for the current age.

Note 2.9 (When implementing something new).

It is good to have an illustration of why the system is special and different, and why it is also compatible with existing ways of working. This is true whether the system is based on code or mores (e.g. why are principles of freedom important online?). I’ll be devoting more time to making these illustrations in the future. I expect that such illustrations will do a lot to underscore the value of the framework developed in this document (and, indeed, value is often most obvious in the more concrete manifestations and examples of a more general system). Cf. Note 2.17.

Note 2.10 (What doesn’t work or needs debugging).

Here are some more details on things that we’re planning to provide at some point but haven’t finished yet.

  • We should support threaded followups and make other improvements to the display (see Note 2.34).

  • “Labels” work but “namespaces” haven’t been implemented yet (see Note 2.37).

  • We don’t yet have support for “actionable” articles (see Note 3.43).

  • Various browsing features could be improved and extended (e.g. to enable deletion or editing of some subset of articles en masse), and probably debugged (especially the code from Section 6.4). The history mechanism could probably be debugged by displaying it.

  • We haven’t fully finished the code for deleting (Section 7.6).

  • Committing could stand to be beefed up with versioning (see Note 7.49).

  • We don’t yet have support for saving articles selectively. In particular, we hope to add support for saving only new or modified articles soon (but of course also for other predicates).

  • Support for derivative articles is somewhat lacking (Section 8); in particular, I’ve done some work and sketching for transclusion and identification (Sections 8.2 and 8.4) but I’m not yet happy with the results and have instead had to seek a temporary work-around . These are some of the trickest technical problems tackled in this document (see Note 8.16). However, they should also be some of the “killer aps” when they are finished.

  • Support for working in a distributed environment is lacking. I plan to add ‘‘native’’ support for CVS and/or GNU Arch soon44 4

  • Several of the experiments we describe haven’t been attempted yet.

  • Increased standardization and various other improvements should be effected throughout.

  • We need to check and expand support for buffers and files within the scholium system; see 2.17. The question of scholia attached to the Scholia Display buffer may prove to be an especially annoying case; where will these be displayed? More generally and less confusingly, what should be done with scholia attached to buffers that have frequently-changing content? Perhaps these scholia can somehow be associated with those changes, and swapped in and out programmatically whenever the changes occur. We should debug the ‘display-scholia-about-current-buffer’ function. See Note 5.71 for an example of the sort of problem we have to deal with here. (By keeping articles and buffers separate, we’ve made the first step towards insuring orthogonality, but we want to be sure of usefulness, too, e.g. by improving default loading of scholia for buffers.)

  • We need to check how this loads against a plain-vanilla Emacs, and probably provide some easier means of getting started.

  • Make it so that ‘sch-plain-text’ (Note 5.5) can display links. In fact, we might want to always have it take in links, or the unlistified contents of links. See Note 2.61.

  • We should finish the functions for browsing parents; cf. Note 6.66.

  • Has the problem with committing a document that has a links at the start of its display been resolved?

  • The system should be documented with use cases for all of its features; at least the basic ones.

Given that the basic system does work and hopefully provides an exciting glimpse of what is possible here, now seems to be a reasonable time to invite other people to help with some of these issues.

Note 2.11 (Streamlined versions of this document).

As mentioned in Note 1.3, the system makes it easy to produce certain kinds of derivative versions of a document. Two examples that may be helpful particularly helpful to readers are a version that focuses on code (see Note 12.34), and a version that steers clear of code (see Note 12.35). Compare “weave” and “tangle”. (Of course, I do suggest reading everything at some point; see Note 2.12.)

Note 2.12 (A motto).

Certain issues cannot be resolved without writing code.

2.2 Design issues

Note 2.13 (Introduction to design issues).

Some of the key development issues have been sketched in the previous section. Here we discuss the issues that come in at the level of design.

2.2.1 Design principles

Note 2.14 (Principle of minimality).

This system could be infinitely complex. At each turn, we try to stick to minimal implementations first and then work out from there.

Note 2.15 (Principle of order).

Functions and variables should not be used before they are defined. (There is a related principle of context that is harder to express: the idea there is that things should be defined in places that make sense.) Any exceptions to this rule should be carefully considered.

Note 2.16 (Principle of transparency).

It would be nice not to have features that disrupt the easy and completely general flow of the code, but this seems very hard to accomplish. We can at least try to be aware of the places where such losses of generality appear, and make some assessment of the consequences (and possible future remedies), and certainly try to avoid losses of generality whenever possible.

Note 2.17 (Principle of orthogonality).

In order to make this system work well with the rest of Emacs, it should be roughly orthogonal to the rest of Emacs. It should not require you to go off into your own little world, nor should it require redefining basic functions, or otherwise change what people are used to in terribly significant ways.

2.2.2 System architecture

Note 2.18 (The principles of the architecture).

The system is built on the principle of a semantic network. Articles correspond roughly to the nodes of a graph, and link-backlink pairs to the edges. Both of these types of objects can bear labels with certain structured information on them. (Actually, labels on links are sub-frames of the labels on articles; and labels on backlinks are derived from data about the linking article (I guess you could call that an “induced subframe” or “derivative subframe” – the issues having to do with where the knowledge is stored, e.g., where you might store the query that tells you which thing is a derivative, or one particular sort of derivative, are in my opinion a philosophical issue beyond the scope of what we wish to get involved with in this paragraph) and anyway, all information in the system is stored on articles. But we don’t really need to go there now either.) We could just say “frame” and “subframe” and keep it simple. But note that these are the internal frameworks of the system, not the content frameworks or whatever you want to call that. That really just about sums up the architecture of the system itself.

Note 2.19 (Metaprinciples of the architecture).

Eventually we want to get things like code that defines the system to be “understood” by the system in some way. These are semantic networks, so saying that they can understand something is moderately appropriate. Right now, the code is just imported into the system in a bulk fashion. But eventually the functions could go into the article table. Really, the fact of the matter is that later we want to put the fundamental object arrays for Emacs into the article table too, so that the whole computation that is Emacs becomes a massive and somewhat complicated scholium based document evolving in time. So, now we get a sense of what it means to be self-documenting to the extreme. (Now, per usual, we just modeling something that is already there, and is already documented to a certain extent by itself, maybe to a richer extent then you might think.) Getting Emacs embedded in the scholium system is the next step after getting the system somewhat roughly embedded in itself. (So, this might be an interesting thing for the LISP crowd to think about; and indeed, it provides a sort of metaphor for the structure of the HDM itself.)

To sum that up, and to conclude! We want to make the system increasingly cognisant of itself. If it doesn’t make sense to look at all of the processes going on inside of Emacs, for example, then we don’t look at all of these processes. But to the extent that it is useful to look at the details of a given process, it would be helpful to track that process in the scholium system. (One question we might ask ourselves is when exactly is it useful to track a given process, noting that we are often tracking the process through time, if it is a computation, and then most likely examining the trace later. However, you could also set things up so that you could look at the trace from different parts of it that are evolving. Of course, you can’t look into the future, you can only look at nodes that have already been computed. But you could get a look-ahead effect going within your computation that would know where to look ahead of time, so that its journey to retrieve the answer begins before the answer has actually been recieved, and the query process meets up with the answer just in time. And that would be fairly tricky!

Note 2.20 (Architectural implementation).

The system is currently implemented as a hash table of articles; the format for these is explained later on in this document, mainly in Section 3.7.

2.2.3 Stylistic considerations

Note 2.21 (Backend and frontend).

To the extent that it is possible, we should distinguish between the backend and the frontend of the system. We could rewrite the backend in scheme, for example, and rewrite the frontend in various web-programming languages to make a system that is easy to use over the internet.

Note 2.22 (Abstracting access).

Since we may eventually support both lists and hash tables (and perhaps other things), we presumably should have an abstract means of accessing articles. (Unless we were to decide to just do things with hash tables; however, it seems likely to me that even if we use a hash table as the main library, people may want to use lists for sublibraries; if the same functions for access are supposed to be used in both full and sub library scenarios, we’ll want the abstract access regardless of how the full library is set up.) A theoretical discussion of the benefits of hash tables can be found on PlanetMath55 5

Note 2.23 (The Emacs Way).

Scholia about files and about buffers are probably most of what would make this system useful to people who are collaborating on papers or coding projects. See Note 5.12, Note, 2.52.

Note 2.24 (Intuitive interface).

The bread and butter of the system’s user interface is a set-up with side by side buffers (see Note 5.11). Other display styles (e.g. as described in Note 2.33, or threading, as in Note 2.34, or a display spread across several frames, etc.) should be straightforward to add.

Note 2.25 (Text properties and overlays).

We essentially need to have scholia encoded locally as text properties, because we need to be able to cut and paste the elements and have the metadata transfered along. However, at least one aspect of text properties is already “owned” by fontlock, namely faces. To use faces in the scholium system, we have to hack an override in – namely, we mark the buffer up using text properties and then mark them up again with overlays. In general, text properties are used to track text, and overlays are just used as a visual aide. See section 5.4, also Note 7.46.

Note 2.26 (Managing text properties in buffers).

We store information about all attached scholia on text property lists. We should also be storing the name of the article on all of the text throughout the buffer, so that if the contents of that buffer are cut and pasted, the text can be tracked. This requires a somewhat complex data structure, and makes markup with overlays (as well as some other details, e.g. text marked up with references which is subsequently cut in half) tricky either in practice or conceptually. See Note 7.46.

Note 2.27 (Selection methods).

Various selection facilities need to be implemented. A little predicate matching thing would make things extensible! (See also Note 5.69 and Note 6.16.)

Note 2.28 (Local navigation).

It is important to have convenient ways to move around from scholium to marked region and back, from marked region to marked region, and so on. See Section 6.2.

Note 2.29 (Color and style of display).

We could offer the user a choice between type-based coloration (indicating what sort of scholium is being marked up), owner-coloration (who created the scholium) and connected-coloration (which what I’ve initially coded up). We could use some combination of color and other visual indicators (see Note 2.30) to provide more information.

Similarly, text written by a given author in a shared file could have a special color. Another type of display would indicate how many scholia had been created about a given region (like a rainfall chart). Different styles can be generated essentially indefinitely. Another user-friendly feature would be to be able to easily change the color associated with some particular set of data (rather than changing the overall scheme whereby colors are associated with data).

Note 2.30 (The monochrome lifestyle).

For the benefit of people working with and without fontlock, three display modes to point to attached scholia from Window A will be available: one that uses font lock; one that uses intangible tokens [N:] and [:N] where N is a number to demarcate the beginning and end of the Nth scholium (or some other customizable pattern, e.g. based on the bookkeeping information); and a third “relaxed” mode uses no markup at all. A number of intuitive ideas for inserting indicators (e.g. in the fringe) could be tried. See 5.67 for another monochrome-friendly approach. Another simple idea, similar to the one described above, would be to use symbols (e.g. *, #, §) to associate scholia with the beginning(s) of the passage(s) they are attached to, and use narrowing to screen out other material.

Note 2.31 (User-crafted semantics).

As a generalization of Note 2.29, we will eventually want some essentially completely different styles of rendering. In every case we should also move to support user-crafted semantics. See also Note 3.32.

Note 2.32 (Contextual display).

An interesting feature would be to restrict to showing only the scholia that are attached to the current portion of the main article that is currently displayed, or only those associated with the character position of point. In order to pull this off, we will need to look some into page and cursor motion in Emacs66 6 Thanks to Stephen Monnier and Eli Zaretskii for telling me what to look for, on the emacs-devel list, 2006-04-04. The relevant tips were to look at: ‘window-scroll-functions’ (when a window’s starting pos changes); ‘fontification-functions’ (when a piece of text needs to be displayed); ‘before-’ and ‘after-change-functions’ (when a piece of text is modified); ‘window-start’ and ‘window-end’; ‘pre-’ and ‘post-command-hook’ (to run before/after each command, so you can detect cursor motion)..

Alternatively, we could write a new motion function, something like ‘move-to-next-region-with-scholium’ (cf. Note 6.4), and combine this with a selective scholia display that will show whatever scholia are found at the new point. The closest thing I could find in Emacs is ‘calendar-move-hook’.

Yet another thought along these lines would be to have scholia appear in the same buffer as the rendered main article, but, for example, in columns beyond column 72 (or whatever the rightmost character in the display of the main article is).

Compare Note 5.68 for thoughts on automatic contextualization of an article at display time.

Note 2.33 (References).

Traditional hypertext links are a useful specialization of the general sort of links we’re developing here (Note 2.6). In an effort to reduce confusion, when we wish to be precise, we will refer to the scholia that are used to make a piece of text point at other piece of text as references. This usage is consonant with use in other hypertext systems, in which “links” are pieces of text that are marked up with references to other texts.

Despite the time it may take to get used to the name, the familiar feel of references should make it easy for people familiar with other hypertext systems to transition to this system.

References are an example of a type of scholium that appears as markup instead of appearing “alongside” the article they are attached to (which is what will typically happen with other scholia). References may have special semantics associated with them, see Note 12.68 and Note 4.30.

It may be convenient to offer allow switching between our overlayed references and the more familiar inline, wiki-like, references, or at the very least, some other way to easily edit reference targets (and similarly for other sorts of links). See Section 12.5 for various information on simulating a wiki with the scholium system.

Note 2.34 (Threading).

We can also provide facilities for showing all attached scholia together with their attachments, and so on, within some fixed graph distance, beyond which we just include links to lower levels (as yet unimplemented - but a good idea). This is essentially what’s done on Slashdot, and we would want to use a similar indenting (and, ergo, threading) mechanism. (We could be even more Slashdot-like and do scoring and the whole bit.) The thing to bear in mind is that we dont’ have to be constrained to follow threads along one “follow-up-to” axis; we can choose arbitrary conditions to constrain the way we view a local hierarchy (cf. Note 13.39). The simplest one to state is the one already mentioned, namely, viewing all attached scholia to some depth.

Note 2.35 (Ancestors).

We typically think about showing a article together with articles that are about it (its scholia) but we can just as well show it together with articles that it is about (its referents). Indeed, views that take into consideration the article’s parents and grandparents have obvious applications, e.g., when attempting to follow a conversation. Particularly if an article has no children itself, it may be handiest to display it together with its parent (multiple parents could potentially complicate the matter). These issues have been taken up to some degree in various places in this document, including Note 5.78 and Note 6.65.

Note 2.36 (Expand and collapse backlink context).

We typically show an article together with the text of its scholia, but we could optionally simply say that such-and-such an article links here. Compare Note 2.32 and Note 2.35.

We may also want to provide a hierarchy of additional scholia, mapping along the links-to and the linked-to directions, see Note 2.34.

Note 2.37 (Labels and namespaces).

A label is a predicate extent that has (typically) been maintained on the fly (or otherwise made concrete). A namespace is a place to put articles; articles in different namespaces can have the same names but not be the same. An object can have many labels but can only live in one namespace. (See Section 3.5, and Note 3.69 in particular.)

Note 2.38 (Filtering).

Filtering is closely related to the topic of subcollections mentioned in Note 2.37 (and it also relates to the issue of predicate mapping; see Note 12.21 and Note 6.40). The point to keep in mind is that different people are going to want to see different things when looking at a document.

Note 2.39 (Transcluding and outline views).

Note that instead of text stating where the URL is, we could have an “transclude” scholium; this would cause the latest, current, version of the target document to appear in the current document when the latter is rendered. One should be able to look at the current document as an outline (hierarchy) showing these “transclusion” relationships explicitly. Similarly, one should be able to collapse and expand different parts of the document (i.e., in the collapsed view, only the title of the transcluded document would appear, but in the expanded view all of the content would appear). These sorts of ideas are handy whenever thinking about container documents (Note 13.16; oppose Note 13.17). When examining an outline, one should be able to jump to the article in its place inside of the document that is being outlined, or as a stand-alone item.

Note 2.40 (Rendered and source views).

A more mainstream example than the expanded and collapsed views from Note 2.39 are rendered and source views. These could be relevant when working with LaTeX or HTML (e.g., when browsing this document, it would be nice to have things detexed). Presumably edits would take place in the source view (though not necessarily; compare the Emacs Muse77 7 Switching in and out of rendered views would be an extra step when initiating edits (see Section 7.1).

One simple feature that would be a nice part of a reasonable rendering would be to keep the text within a certain width by filling at render time (this would be both for articles and for the scholia display). Presumably exported text would also have to be filled if it is to look good.

Much more generally, it would be nice to have a full-fledged LaTeX renderer (along the lines of LaTeX-to-HTML). A renderer is in many ways similar to a parser, so a simple parser is a reasonable first step. Some of the work related to Emacspeak88 8 is, I think, relevant to this problem.

Note 2.41 (Page numbers).

If we could figure out how to do some truly device-independent rendering of LaTeX documents, one particular thing we’d presumably have come up with be a correspondence between page numbers in the printed version of a document and in the electronic version. In the mean time, maybe we can come up with some easy way to solve this using some specially logging when LaTeX processes “notate” environments? (E.g., it wouldn’t be hard to use special commands to make an index of the notate environments in the document; and presumably Emacs could read the source of that index.)

Note 2.42 (Alternate names for buffers).

Should buffers have special names stored in buffer-local variables, or should we search the library for buffers and check to see if they are the same (regardless of what name they go by)? Or some other approach to deal with named buffers? See section 5.5.

Note 2.43 (Saving and reading).

The user can save and read back a given collection of scholia. As an example, arbitrary files can be opened in ‘scholia-mode’ so that all scholia stored in the same directory as the file will appear automatically as annotations.

Note 2.44 (Versioning).

In order to edit and stay kosher, we should provide support for versioning. This version information will be stored on the bookkeeping slot (or, alternatively, we should store sufficient information to recover the versions; if such information isn’t already available elsewhere in the system). Note that we could just go ahead and make functions for editing the text and then add functions for saving the previous version later, in a hook. (This relates to the slightly more general problem of making editing work in general, see Section 7.)

In order to deal with about data that terminates on a given article version instead of on the latest version, we will want to have an additional optional column available in links that is reserved for the version number.

(Note that some objects may not need to have versions stored about them.)

Note 2.45 (CVS/Arch/filesytem support).

We need to figure out how to integrate external information resources into the system. One issue with CVS and Arch is that we (sometimes) will want to only look at files that are new or have changed upon reloading. This means that the version management software will probably have to be integrated fairly closely into the system. Another option would be to keep a record of the files in any directory that is being used, and check this record against that file’s listing whenever the user wants to reload the content from that file. This may be the only option we really have for using the filesystem directly, but for the case of CVS, proper integration would probably be better.

Note that Arch has its own way of doing version management (as do all version control systems, of course). It might be best if we could swap different methods for version control in and out. Fundamentally, we need to know where to look for articles – they don’t necessarily have to be in the filesystem (perhaps they could be assembled in real time in some buffer, for example; similarly, I think Arch builds various patched versions in real time).

Note 2.46 (Responsiveness to editing).

As editing takes place, the information in the system needs to be kept up-to-date and coherent with itself. The main developments in this section are presented in Section 7.4.

Note 2.47 (Unique identifiers for individual pieces of markup).

When committing edits, we reparse the buffer to find the places where the scholia are attached (Section 7.3). For things to work properly, each piece of markup should have a unique identifier (the article name and link number should suffice, see Note 3.10). This will make it possible tell whether a piece of markup has been completely deleted, or swapped with another piece, or split into several pieces, or whatever (in theory, a region can even be moved between buffers, which would require an additional level of reparsing, we won’t worry about that here for now; see Note 8.60).

Note 2.48 (Manipulation of windows).

The somewhat aggressive manipulation of windows used in this program could get annoying, especially in the case of editing multiple articles at once. (Which is one possible reason for editing things inside one buffer, see Note 8.33.)

Note 2.49 (Sorting scholia).

We want scholia to appear in some reasonable order relative to the order of marked regions. Section 5.3 is supposed to do this. Note also that sorting should not be destructive; see Note 3.17.

Note 2.50 (Complexity issues).

Some features simply would take too long to run when there are lots of scholia or articles involved. We need to be aware of these sorts of complexity issues. (It usually doesn’t take much more than a simple test to make one fully aware of the problem, when dealing with terrible complexity scenarios!) Section 3.5 contains a few thoughts on how to limit complexity.

Note 2.51 (Ease of use).

In addition to being more featureful than the wiki medium, I hope that this system will be easier to use – wikis as I know them are fairly laborious; lots of steps are needed to edit and save. The system should be designed in a way that requires few if any “extra” steps from authors.

Note 2.52 (Return to base representation).

It might be nice to have a way to return from a scholium to its “actual” base representation, i.e., if it is associated with a buffer, we should find that buffer, or if it is associated with a file we switch to a buffer visiting that file. Indeed, I think we have thing set up now so that an article stored in a buffer will always be attached to that buffer. We might want to be more flexible, allowing various different ways to attach and reattach (compare Note 3.108).

Note 2.53 (Code transparency).

It is good to have easy-to-understand names for functions that will appear over and over again; for example, we provide functions to access the bits and pieces of scholia (see Note 3.16). But we should try to do this all the time, for all of the data structures we build.

See also Note 13.23 for further thoughts on why it is a good idea to do things this way.

Note 2.54 (Creating a scholium about several articles).

I’m not sure we have sufficiently strong mechanism for making scholia that apply to several other articles. That would be one potential use of the list browser from the previous section; we could mark articles and make a scholium about marked articles quickly, or mark articles and then make scholia about sections from other articles. Also, it would be good to have a way to break out of the region selection routine and start selecting regions from some article fluidly when working on making scholia that way.

This is useful as long as we are able to gather all of the articles we want to make a scholium about into one listing, or if we can find a way to use several listings together intelligently.

Note 2.55 (Space versus time tradeoffs).

We value time over space tradeoffs. If the data set becomes very large, this judgement may have to change (e.g. if the system gets popular, we won’t be able to hold all of the available articles in memory, or on disk, and we’ll have to get clever about swapping things around).

Note 2.56 (Support for systems in which passages have special meanings).

RMAIL, info, and EDB are just a few examples of Emacs subsystems whose most “meaningful” objects are smaller than the files in which these objects live. Support for these objects would be good to add to the scholium system, so that we can e.g. record mail messages or info pages that match some particular property.

Note 2.57 (Undoable editing).

It would be nice to be able to undo the editing actions that we support.

Note 2.58 (Support for narrowing).

Frequently the function ‘point’ is used here; I’m assuming that documents will be widened properly. But it would be good to be able to support narrowing (see Note 2.56).

Note 2.59 (Hooks and ‘cond’ forms).

At some earlier point, it seemed useful to think about transforming a ‘cond’ form with an unquoted variable that contains extra cases (that are to be defined as needed). This seemed like a good opportunity to use LISP self-modifiability, and furthemore, it seemed like a good way write things in order (see Note 2.15). However, typically the same thing can be achieved with hooks. The function ‘sch-plain-text’ is a good example of this (Note 5.5).

Note 2.60 (Multidimensional visual browsing).

It might be handy to have a small control pannel at the bottom of the main article display that could be used to go backwards and forwards according to the layout of the document and also according to the order in which things had been browsed so far (and anything else that the user might dream up to put there).

Note 2.61 (Using datatypes).

In order to be able to pass different kinds of objects around and have them interpreted correctly, it might be advantageous to have our data structures typed; I’m not quite sure how types work in Emacs – but I can say that datatypes are obviously scholia. For example, we might want there to be an article datatype (right now, there are only types of articles, like “references” for example). Another entity that could have a special datatype is the link.

The question of datatypes for forms seems interesting; it basically boils down to the question as to whether we add scholia to arbitrary lisp structures (see Note 3.42).

Note 2.62 (Hooks as scholia).

Hooks are an easy-to-see example of scholia in the Lisp paradigm (compare Note 3.42, 8.6).

2.3 Elisp requirements

Note 2.63 (Packages).

I’m not sure we actually need align any more, but cl is of course very helpful.

(require ’align)
(require ’cl)
(load "cl-seq")
(load "cl-extra")

2.4 Lisp preliminaries

Note 2.64 (On ‘add-or-replace’).

It seems that this function is no longer used.

(defun add-or-replace (lis elt pred)
  (let ((found nil)
        (n 0)
        (len (length lis)))
    (while (and (not found)
                (< n len))
      (when (funcall pred (nth n lis))
        (setcar (nthcdr n lis) elt)
        (setq found t))
      (setq n (1+ n)))
    (if found
        (1- n)
      (nconc lis (list elt))
Note 2.65 (On ‘add-to-or-start-list’).

This is is used to start store an element on a “possible-list”; unlike ‘add-to-list’, it works even when list is nil, or even (for now) when possible-list is not a list at all. (Unlike with ‘add-to-list’, possible-list should not be the name of a list.)

This function is used by ‘put-type-labels’, ‘put-backlinks’ and ‘label-article’. Note that you have to save the result if you want to this function to have a lasting effect.

Like ‘add-to-list’, this function will only store one copy of element.

(defun add-to-or-start-list (possible-list element)
  (cond ((null possible-list)
         (list element))
        ((listp possible-list)
         (if (member element possible-list)
           (append possible-list (list element))))
        (t (list possible-list element))))
Note 2.66 (On ‘next-single-property-change+predicate’).

Like ‘next-single-property-change’ except instead of automatically judging changes in the text property by the ‘eq’ predicate, it allows the user to supply a test.

(defun next-single-property-change+predicate (pos prop &optional test)
  (let ((starting (get-text-property pos prop))
        (index pos)
        (cmpfct (or test
      (while (and (setq index (1+ index))
                  (< index (point-max))
                  (funcall cmpfct
                           (get-text-property index prop)))))
    (when (< index (point-max))
Note 2.67 (On ‘delete-all-dups’).

This function differs from ‘delete-dups’ in being non-destructive and in keeping none of several ‘equal’ occurrences of an element in list as opposed to one; otherwise, it manifests a similar idea.

(defun delete-all-dups (list)
  (let ((tail list)
    (while tail
      (cond ((member (car tail) (cdr tail))
             (setq tail (delete (car tail) tail)))
             (setq ret (cons (car tail) ret)
                   tail (cdr tail)))))

(defun set-difference (A B)
  (delete-all-dups (append A B)))

(defun zip (A B)
  (let ((ret (make-hash-table)))
    (while A
      (puthash (car A)
               (car B)
      (setq A (cdr A)
            B (cdr B)))

(defun flatten (list-structure)
  (apply ’concatenate ’list (list list-structure)))

3 Scholia-based documents

3.1 The digital library

Note 3.1 (On ‘article-table’).

We begin with an empty library.

(defvar article-table
  (make-hash-table :test ’equal))
Note 3.2 (Structure of articles).

Every article has a name, which can be any LISP object (but is most frequently a string). They also have various other standard sorts of data associated with them; see Note 3.7 for the particulars. This data is referred to by ‘put-article’ collectively as the article’s value (but this name is somewhat misleading, because the name too is an important part of the article). One special sort of article is a “library subcollection” – see Section 3.5 for more information on these things.

Note 3.3 (On ‘put-article’).

This destructively adjusts the values on the hash table (see documentation for ‘puthash’). To intelligently manage the values in the table takes more work, so we typically ‘get-article’ before to make various comparions and adjustments to existing values before making changes.

(defun put-article (name value)
  (puthash name value article-table))
Note 3.4 (On ‘get-article’).

Basically just ‘gethash’, but since the article table is indexed by name and we don’t want to pass name around all the time, we add it to the recalled value.

(defun get-article (name)
  (let ((content (gethash name article-table)))
    (when content
      (cons name content))))
Note 3.5 (On ‘name-of-current-article’).

The “current article” is of special importance for display (an article is made “current” when ‘display-article’ displays it). The name of this article is stored in this variable.

(defvar name-of-current-article nil)

(defun current-article ()
  (get-article name-of-current-article))
Note 3.6 (What about using the file system?).

If the library is being updated by many people at different times, it may also make sense to read documents from a file system on the fly. Indeed, the program does support files and buffers. But the article table is still a decent model – similar in nature to a library’s catalog.

3.2 Creating scholia

Note 3.7 (Component pieces of scholia).

The ‘scholium’ function maintains a catalog of articles, indexed by name (Section 3.1). Each article contains text, an indication of what the article is about, its type, and bookkeeping information which gives the article’s edit history and ownership information. Articles must always follow this pattern.

Note 3.8 (Format of ‘name’ field).

Any LISP object can be the name of a scholium. Scholia are indexed by name: names are unique within namespaces (see section 3.5), and the combination of name and namespace is how an article is looked up. (At present, names are typically rendered as strings without any newlines, but this is a front-end matter and essentially arbitrary from the point of view of internal format.)

Note 3.9 (Format of ‘text’ field).

Just about anything can go into the text field, but anything “weird” needs to be backed up by appropriate type metadata and appropriate handlers (see Section 5).

Note 3.10 (Format of ‘about’ field).

The about field is a list of links (in the sense of Note 2.6). See Note 3.11 for details.

We could at some point investigate making about data (and perhaps other sorts of data) typed so that we can more easily the genesis of a given datum.

Note 3.11 (Links).

Each link is associated with precisely one article. Links can have additional link-type information stored about them. The general format of a link is:

(<target article’s name> &rest <link-types>)

The design here maintains partial symmetry between the treatment of article types and link types; the cdr of a link can be processed by the same typedata processing functions as the type data from articles.

For example, the link

’(foo (passage 10 15) mistaken)

indicates that the region from character position 10 to 15 in the article named “foo” is “mistaken”, whereas the link

’(foo mistaken)

indicates simply that the article foo is mistaken.

Note 3.12 (Links indicate at most one passage).

It seems reasonable to me to assert that each link will indicate at most one passage. This is indeed an assumption made in the code, see the function ‘typedata-includes-passage’ (Note 3.29)).

Note 3.13 (Format of ‘type’ field).

The type field is supposed to be a symbol or list of symbols. If it is nil, the article is assumed to be normal text.

Note 3.14 (Format of ‘bookkeeping’ field).

The bookkeeping field has a special format, very similar to the format of metadata articles (see Section 3.3.1); one might expect to see something like

((owner <owner>)
 (ACL <ACL>)

Remember that the bookkeeping is owned, so we probably shouldn’t put anything there that can’t be edited by the user, and so in particular, things that are needed for system consistency should go into metadata articles instead.

Do notice that various schemes for access and ownership can be considered. (E.g. maybe anyone on the access control list (ACL) can edit the text of the article, but only the owner is allowed to edit the ACL itself.)

It would be worth looking at existing strategies (e.g. as found on PlanetMath) for handling complicated ownership and access arrangements. Actual implementation of useful bookkeeping features will have to come later (for a sort of silly placeholder for the meantime, see Note 4.3).

Note 3.15 (Relationship of bookkeeping data and metadata article).

Bookkeeping data is user editable, but metadata article generally isn’t, or, when it is, it is typically treated as common property. This is important! See Note 3.46.

Note 3.16 (Access functions).

A few simple functions to get pieces of (just aliases to help with coding; see Note 2.16). Note 3.18 talks about something similar for links.

(defalias ’scholium-name ’first)
(defalias ’scholium-text ’second)
(defalias ’scholium-about ’third)
(defalias ’scholium-type ’fourth)
(defalias ’scholium-bookkeeping ’fifth)
Note 3.17 (Order and interpretation of ‘about’ field).

The order of links in the about field is not arbitrary.

The order of links determines link-ids, which are used for both backlinks (Section 3.3.2) and markup (Section 5.4).

Note 3.18 (Link accessors).

Links have exactly one linked-to-article together with any number of different link-types.

(defalias ’linked-to-article ’first)
(defalias ’link-type ’cdr)
Note 3.19 (Special link accessors).

First we need a function that will give us the type-element associated with a given type. Then, we have some important examples.

The function ‘link-type-accessor’ will produce the (first) specific type-element from link that matches the given type, if one exists.

Note that the format of version access should follow the ‘link-version’ function given here. (Passages don’t have versions, so it doesn’t make sense to put a version number as an additional element of a “passage” element.)

(defun link-type-accessor (link type)
  (car (member-if (lambda (type-elt)
                    (or (eq type-elt type)
                        (and (listp type-elt)
                             (eq (car type-elt) type))))
                  (cdr link))))

(defun link-beginning (link)
  (second (link-type-accessor link ’passage)))

(defun link-end (link)
  (third (link-type-accessor link ’passage)))

(defun link-version (link)
  (second (link-type-accessor link ’version)))
Note 3.20 (Link creators).

It might be nice to have a function that is in charge of link creation, so we can quickly identify the places where links are being created.

Note 3.21 (Link clusters).

In previous versions of this work, I grouped links into “segments” by default. I subsequently realized that adding this extra default layer was ugly, and that it would limit the attractiveness of the system. Nevertheless, the initial intention was, I think, good. The idea with segments was to group certain links together semantically. For example, if the region that a reference was attached to got split in two after editing, one might want to maintain that one reference as being about two regions “in the first place” while continuing to be about only the one target article “in the second place”. (On references, see Note 2.33.)

In the case of references, we can get away with making making the “link text” correspond to all of the links besides the last one. And in many other cases, similarly, we will be able to specify semantics for about data in the type data. However, in some cases, we may need to put together link clusters that are separate from the article entirely – in other words, we need to be able to attach scholia to about data (as well as, e.g., to text data).

This flexibility assures us that there can be as many different kinds of clusters as we can come up with. By contrast, if we were using segments, the actual semantics would either have to be more limited, or would end up building extremely complex about fields.

More work on the subject of link clusters will follow in due course. Right now, I think they are just conceptually useful. (Which is another reason for getting rid of segments… thank goodness they never had a chance to become useful!)

Note 3.22 (Uses for ‘type’ field).

One use of types is to maintain subcollections; see Note 3.73. Articles of different types can be rendered or otherwise interpreted differently.

Note 3.23 (Hooks for new and modified scholia).

If the scholium is new, run ‘new-scholium-hook’, otherwise, run individual hooks depending on which pieces of the scholium were modified (if any). We use separate hooks in part because we need to be careful about what sorts of changes we make. If we adjusted the value of some scholium every time the value of a scholium was adjusted, we would expect to encounter infinite loops pretty quickly!

The standard settings for these hooks will be given in the section 3.5; these settings facilitate e.g. the automatic creation of backlinks (Note 3.56).

(defvar new-scholium-hook nil)
(defvar scholium-modified-text-hook nil)
(defvar scholium-modified-about-hook nil)
(defvar scholium-modified-type-hook nil)
(defvar scholium-modified-book-hook nil)
Note 3.24 (The ‘scholium’ function).

This function is foundational for the system. Lots of other functions will use it, but because it is destructive, users should typically not access it directly. Several interactive interfaces to this function appear in Section 4. Note that the conditional structure here shouldn’t be a ‘cond’; we really mean to run several hooks if several parts of the scholium have been modified.

(defun scholium (name text &optional about type book)
  (let ((old-version (get-article name)))
    (put-article name (list text about type book))
    (when (not (metadata-override))
      (if (not old-version)
          (run-hooks ’new-scholium-hook)
        (when (not (equal (scholium-text old-version) text))
          (run-hooks ’scholium-modified-text-hook))
        (when (not (equal (scholium-about old-version) about))
          (run-hooks ’scholium-modified-about-hook))
        (when (not (equal (scholium-type old-version) type))
          (run-hooks ’scholium-modified-type-hook))
        (when (not (equal (scholium-bookkeeping old-version) book))
          (run-hooks ’scholium-modified-book-hook))))))
Note 3.25 (On ‘metadata-override’).

We don’t want to run the usual hooks when the article being processed is a metadata article or a “fake” article. Fake articles appear when we create fake references that they don’t show up in the scholia display and that aren’t permanently attached to anything. See Note 6.64.

(At least at one point I had some misgivings about this override, but it seems to get the job done. I don’t see why we would want to have weird automatic thing happen after creating a metadata article or a “fake” article, so it may be that the design here is perfectly fine.)

(defun metadata-override ()
  (when (or (typedata-includes type ’meta)
            (typedata-includes type ’fake))

3.2.1 Simple scholium derivatives

Note 3.26 (On ‘article-names-from-about-data’).

The idea here is to build a list with just the names of the articles that the about data features. Names are uniquified.

(defun article-names-from-about-data (about)
  (delete-dups (mapcar (lambda (elt) (car elt)) about)))
Note 3.27 (On ‘typedata-includes’).

Read typedata in the format used in scholia, and say whether it matches a specific-type. Notice that when typedata and specific-type are both is ‘nil’, we return a positive match.

(defun typedata-includes (typedata specific-type)
  (cond ((eq typedata specific-type) t)
        ((and (listp typedata)
              (member specific-type typedata)) t)
        (t nil)))
Note 3.28 (On ‘typedata-includes-element-of-list’).

Like ‘typedata-includes’ (Note 3.27), but only requires one type out of those listed to match in order to return ‘t’.

(defun typedata-includes-element-of-list (typedata list-of-types)
  (let (ret)
    (while (and list-of-types
                (not ret))
      (when (typedata-includes typedata (car list-of-types))
        (setq ret t))
      (setq list-of-types (cdr list-of-types)))
Note 3.29 (On ‘typedata-includes-passage’).

A specific test to see if typedata includes type “passage” (noting that the list element that indicates this type should come together with some suffix that tells you which passage is being indicated).

As far as I can tell, this function will always apply to typedata coming from links (but who knows).

(defun typedata-includes-passage (typedata)
  (let (ret)
    (when (listp typedata)
      (while (and typedata
                  (not ret))
        (when (eq (car (car typedata)) ’passage)
          (setq ret (car typedata)))))

3.2.2 Examples

Note 3.30 (Examples of the ‘scholium’ function in use).

As an example of how the ‘scholium’ function might be applied, here is a scholium about the article I’m writing now.

(scholium "Remark On Scholium Definition"
          "This is an Emacs Lisp function."
          ’(((passage "sbdm4cbpp.tex" 49078 49738))))

Actually, the article that I’m writing right now won’t be in the article list unless we explicitly put it there. So the scholium that we added here is actually attached to a fictitious article with the name “sbdm4cbpp.tex”. That’s OK, but for completeness, I’ll add the current buffer as a scholium with the appropriate name:

(scholium "sbdm4cbpp.tex"

Notice that we’ve supplied a buffer instead of a string for the text field. We now add another scholium, attached to the previous two, communicating to the user a fact that the system already knows:

(scholium "This is a scholium about part of the SBDM document"
          "This scholium is attached to this region."
          ’(((article "Remark On Scholium Definition"))
            ((passage "sbdm4cbpp.tex" 31091 31744))))

By setting the type data appropriately, we might make this last example display when “Remark On Scholium Definition” is displayed, but not when “sbdm4cbpp.tex” is displayed. It might make sense to mark up the words “this region” with a reference to “sbdm4cbpp.tex”. Tricky things like this will be coming in later sections.

Note 3.31 (Mixed scholia).

Designing functions to build increasingly complicated about structures easily seems to become increasingly challenging.

Note 3.32 (Types of text).

There could be other kinds of “text” that one might want to be able to use – symbols, processes, web pages, etc.; basically, any sort of information-containing thing could be something that one would wish to add to the scholium system. In addition, plain-text scholia can be miniature database entries (in some label) if they are given a specific type. For example, a function object might have a descriptive note, a doc string, and a definition. Lists are another example of a special plain-text article (see Note 4.17).

3.2.3 Further notes on creating scholia

Note 3.33 (The division of scholia into pieces).

The division is somewhat arbitrary. Ownership could certainly be maintained separately from edit history; however, it makes some sense to set access controls when the article is first saved. Things can get complicated if different persons or groups have different permissions; e.g. some may be able to change the access control list and some may not.

Note 3.34 (Completeness of the scholium function’s arguments).

I think I’ve basically covered everything I’d be interested in with these freeform types. The idea is that you can put whatever you want into these fields, and then write whatever sort of accessor functions you like to use that information! However, if someone wanted to add some new fields, this would be easy; they would only have to change the scholium function and the add-props function.

Note 3.35 (How to assert that a relationship exists).

The relationship between different articles presumably has to be added with scholia that point to both articles and say something special about them in the “type” field. On the other hand, it would be useful to have that particular type be read both ways, so that scholia about both the high-level document and lower-level document explain the relationship of the two documents to one another. (I think backlink registries would help with this.) We’ll talk more about how to actually assert relationships between articles in Section 7.

Note 3.36 (Freedom to create derivative versions).

In general, this can be specified in the bookkeeping information - i.e., the licensing terms (or royalty terms or whatever). For just about any article, it is important to be able to track derivatives! The insistance on tracking makes our system not so different from “regular” version management.

Note 3.37 (Types and modes).

Oftentimes, articles are not so different from buffers. Type data is a way of specifying things like filetype or editing mode. (Maybe a bit more general, because articles need not be very buffer-like.)

Note 3.38 (Extended access).

It might be a good idea to let about be filled by a function that will select a sequence of character positions? Well, I suppose that even if the capability isn’t explicitly coded into the system, people can write their own code that does the job (expands such functions before calling ‘scholium’)?

Alternatively, if we had sub-libraries in this library, such a function might be more complicated. It would be worth thinking about examples of how this sort of thing might be used, exactly.

Of course, if the library contains heterogeneous materials, like the case that is being discussed in the “superimposed information” articles; there, selecting a region may mean different things for different articles. It seems like the case of Word docs and PDFs is simpler than the case of whole libraries (e.g. filesets?) but I’d need to think more about the issues here to come up with good examples. In which case would one rather add a library than all of the documents? (Probably there are some cases.) Well, I guess the GCIDE example that I talked about could be relevant here, although the idea of this GCIDE thing isn’t quite the same as the usual web-like “scholium thing” we’re considering elsewhere in this work. Also, one might think of Google or whatever as being a “library” that you have to interact with in a special way to get at the documents – the reasons for not adding them are that they aren’t free (for one thing) and there are just too many of them (for another!).

Of course, scholia for really “weirdly” formatted documents – including very rich things, like pictures or videos – may have very different sorts of semantics. Its nice of the people who are working on this “superimposed information” stuff to invest time and energy into this sort of thing, but my own feeling is that its important to get some good, working, deployments for text-based document collections going asap!

Note 3.39 (Type creation).

The AI issue relates to the issue of type creation. We should be talking about the relationship of semantic net ideas (an up-to-date reference on semantic nets would be helpful; maybe SNePs).

Note 3.40 (Formulaic scholia).

One obvious sort of type is the form, i.e., an object with certain fields to be filled in. Compare HyperCard99 9; more interesting ideas can be gotten there too, I think.

Note 3.41 (Transformative scholia).

It is interesting to think of transformative scholia; attachments that change the way a given source looks. This is a topic that has recently been getting some play in the web browser community. Simply being able to annotate web pages would be swell, because then you could search through them as if they were part of your own local web, which, in fact, they probably would be.

Note 3.42 (Functional scholia).

Functional scholia are similar to (and generalize) the transformative scholia of Note 3.41. Reimplementing LISP inside the scholium system would be an example of this sort of thing; functions and variables as nodes in a web, programs as clusters of nodes that have been linked together in certain ways. One way this might be done would be to use the ordered about list as an argument list for a function and put the function itself on either the type field or, more likely, the text field (a specification of the expected arguments would go on the type field). Perhaps something like this could be used to build documentation for LISP functions automatically; needless to say, anything we can do scholia-wise can be done LISP-wise (reasoning by set-inclusion, if nothing else), however there may be some advantages to thinking about code scholiumifically. (Functions and their subroutines can easily be regarded as forming a network, and so, code can easily be thought of as a scholium-based document under this arrangement.)

Note 3.43 (Interactive and actionable scholia).

One of the big goals for the system is to be able to write MUD-like programs and text-based games; to do this we need, interactive scholia. More generally, scholia that do something e.g. when browsed or when encountered by a given process; we call elements of this broader category actionable scholia.

Note 3.44 (Simulating features from single-buffer editing).

In Note 3.42, we mentioned a certain feature that should be replicated in the scholium system to make it more useful (‘get-function’); however, some systematic study of the features that are useful from single-buffer editing should be made, and those features re-implemented for the scholium system.

Note 3.45 (Handling functions).

A simple feature request somewhat related Note 3.42 is that we should be able to jump to the scholium associated with a certain function’s name; the analogue of my ‘get-function’ command. The text that documents a certain function should also show up as a docstring when one runs ‘describe-function’ in Emacs.

3.3 Metadata

3.3.1 Metadata articles

Note 3.46 (Metadata article overview).

Metadata articles constitute a merger between “local” and “holographic” data storage (Note 3.101). Metadata articles are essentially local, but they are used to maintain information about data drawn from all over (in particular, backlinks are recorded on metadata articles). A metadata article is always attached to some other article, and is used to record common information about this article. They are updated automatically when certain editing events take place. When an article is rendered, certain fields in the metadata article can take precedence over corresponding information in the article itself (see Note 3.54).

Note 3.47 (Metadata article design).

We exploit the facts that articles (a) have unique names, (b) any LISP object can be the name of an article, to produce the rule that any non-metadata article with name name to which we need to associate a metadata article will have a metadata article named “(meta name)”. The text of this article is the metadata associated with the base article; its format is that of a list of lists, with each sub-list headed by a tag describing the contents of that sub-list (e.g. backlinks). Metadata articles are not considered to be about anything; we get the picture by looking at the article’s name. (They may however contain updated about information for the article they are “meta” to; see Note 3.54.) Also, for now at least, we give them only very minimal bookkeeping information.

A “metadata” label is maintained that allows us to easily keep track of the collection of metadata articles (see Note 3.73, et seq.).

This design makes it easy to set metadata “fields” with ‘assoc’ and ‘setcdr’; see Note 3.51.

(It would be good to say a bit more about the different fields that are stored in metadata articles here.)

Note 3.48 (Maybe phasing out use of pseudo-namespace).

So far we’ve been able to “afford” the use of a pseudo-namespace for metadata articles. However, we may need to use a more legitimate approach at some point. We’ll look at this issue again if it ever seems important.

Note 3.49 (On ‘metadata-article-name-p’).

So, like I said, we’re losing some generality in the set of available names by doing things the way we’ve been doing it.

(defun metadata-article-name-p (name)
  (when (and (listp name)
             (eq (car name) ’meta))

(defun base-name-from-metadata-name (name)
  (second name))
Note 3.50 (On ‘metadata-article’).

This function retrieves the metadata article associated with the article named name.

(defun metadata-article (name)
  (get-article (list ’meta name)))
Note 3.51 (On ‘put-metadata-field’).

This can be can be used to fill a metadata field with the given value, in the metadata article pertaining to the article named name. If the field has not yet been set, it is created anew with the appropriate value.

(defun put-metadata-field (field value name)
  (let* ((metadata (metadata-article name))
         (old-value (assoc field (scholium-text metadata)))
    (if old-value
        (setq new-text (progn (setcdr old-value value)
                              (scholium-text metadata)))
      (setq new-text ‘((,field . ,value))))
    (scholium (list ’meta name)
Note 3.52 (On ‘get-metadata-field’).

This can be used to retrieve any particular metadata field pertaining to the article named name.

(defun get-metadata-field (field name)
  (cdr (assoc field (scholium-text (metadata-article name)))))
Note 3.53 (Similarity between metadata and text properties).

It makes some sense that metadata about articles and metadata about characters would be treated somewhat similarly. Anyway, the similar idioms seem to arise.

Note 3.54 (Updating link extent).

If an article A is a scholium attached to some other article B, and B’s text is edited, then A’s about data, if taken literally, may be quite inaccurate. The solution to this problem is to examine the text properties indicating which regions of B A is supposed to be about, and store the updated information where it will be useful: A’s metadata article “(meta A)”. The information stored in (meta A) “masks” individual links; see Note 5.47 for more information.

Note 3.55 (Updated ‘about’ fields in the distributed case).

In the distributed case, it would probably be best to use the metadata pertinent to the current checked-out instance. To facilitate this, contributed about data should probably be stored together with the id of its source. We’re not quite there yet, but hopefully soon will be! (See also Note 7.29

3.3.2 Backlinks (who links here?)

Note 3.56 (Theory of backlinks).

One particularly interesting sort of object is the “backlink”. These complement the generalized links discussed in Note 2.6, and they play a fundamental role in the system. Backlinks make some operations faster (compare Note 5.72).

Unlike regular links, which live in articles, an article’s backlinks live in its “public” metadata article. See 3.58 for further information on the lifecycle of a backlink.

See Note 12.12 for a description of tricks that can be done with backlinks. Its worth mentioning that backlinks not seen in the typical display, but they could be used in special displays (see Note 2.35). Backlinks are going to be important for AI applications; they provide the information a given node needs to “know” its context within the broader semantic network.

Note that Howm is an already existing Emacs package for both forward (‘‘goto’’) and backward (‘‘come-from’’) linking1010 10 The behavior of Howm’s come-from links is somewhat different from the backlinking features implemented in the scholium system, but it shouldn’t be too hard to add Howm-like features (see Section 12.4 for some words on how we might approach this).

(This note should probably just state the definition of backlinks; anyway, as it is it seems to vague and too long.)

Note 3.57 (Implementation of backlinks).

Whereas links are typed (Note 3.11), backlinks are not. Insteady, backlinks indicate an article and a certain link in that article. The backlink data is useful because it allows us to easily figure out which articles link to a given article (without doing any mapping). The basic use of backlink data is to display an article together with its scholia.

We had considered using typed backlinks (mirroring the type of the link that each backlink corresponds to), but this seemed excessive, since the corresponding information can easily be found by examining the link corresponding to a given backlink.

Note 3.58 (Overview of linking process).

When about data is created or modified, ‘put-backlinks’ is called to update the backlinks in the metadata articles associated with the linked-to articles. A backlink points at the specific link that it corresponds to, by way of the linking article and the unique link number that identifies the corresponding link.

Note 3.59 (On ‘delete-backlink’).

This function is called by ‘put-backlinks’ to prune away an old backlink.

Also notice that in this function, we know that some metadata and backlinks already exist, so in particular the ‘metafield’ exists. Hence we don’t need the complicated conditions found in the ‘add-backlink’ function.

(I don’t suppose we actually need to delete the “backlink” field in the case in which we delete the last backlink.)

(defun delete-backlink (linked-to-article link-number)
  (let* ((metaarticle (metadata-article linked-to-article))
         (metatext (scholium-text metaarticle))
         (metafield (assoc ’backlinks metatext))
         (backlinks (remove (list name link-number)
                            (cdr metafield))))
    (setcdr metafield backlinks)
    (scholium (list ’meta linked-to-article)
Note 3.60 (On ‘add-backlink’).

This function is called by ‘put-backlinks’ to add a new backlink.

Note that the relevant metadata article may not exist prior to the time this function runs and even if it exists, it may not contain any backlink data. So we have to run the relevant tests to deal with these cases. This accounts for differences between the implementation of ‘add-backlink’ and ‘delete-backlink’ (Note 3.59).

(defun add-backlink (linked-to-article link-number)
  (let* ((metaarticle (metadata-article linked-to-article))
         (metatext (scholium-text metaarticle))
         (metafield (assoc ’backlinks metatext))
         (backlinks (add-to-or-start-list
                     (cdr metafield)
                     (list name link-number))))
      (setcdr metafield backlinks))
      (setcdr metatext
              ‘(((backlinks . ,backlinks)))))
      (setq metatext
            ‘((backlinks . ,backlinks)))))
    (scholium (list ’meta linked-to-article)
Note 3.61 (On ‘put-backlinks’).

The basic use for this function is to run it within the scope of the ‘scholium’ function, to build and store new (or updated) backlink data. Backlinks are added to each of the metadata articles corresponding to articles that this article is about.

The algorithm used here (zap all existing backlinks, then place all new backlinks) seems likely to be woefully inefficient! If so, this can be improved later.

Notice that we don’t take into account name changes with this function (there is no relevant “old-name” variable, at least, not at present).

(defun put-backlinks ()
  (let ((old-about (scholium-about old-version))
        (link-number 0))
    (dolist (link old-about)
      (setq link-number (1+ link-number))
      (delete-backlink (car link) link-number)))
  (let ((link-number 0))
    (dolist (link about)
      (setq link-number (1+ link-number))
      (add-backlink (car link) link-number))))

(add-hook ’new-scholium-hook ’put-backlinks)
(add-hook ’scholium-modified-about-hook ’put-backlinks)
Note 3.62 (Generality of ‘put-backlinks’).

The function used to be written in a general way so that it can store backlinks in different fields. For example, I had set things up to add to a ‘derives-to’ metadata field when the function ‘include-article’ ran (see Note 8.24). However, it wasn’t clear to me that this approach was really useful or necessary.

Nevertheless, storing new or updated mask data has to be done at some point; and it may be that we’ll want a generalized version of ‘put-backlinks’ after all (perhaps with a better name, speaking to its new general purpose).

Note 3.63 (Turning backlinking off on demand sometimes?).

It might be nice to turn backlinking off in certain situations. A rather silly example is the situation in which you don’t want the owner of a certain object to know that you are talking about it. There may be some more relevant situations. (Compare the use of fake links, as documented in Note 3.25).

3.4 Consistency conditions

Note 3.64 (Why we maintain consistency).

This discussion might go into Section 13 or Section 13.1, but given that it is fairly foundational for the system, it is worthwhile to at least include an overview early on.

When something changes in the system – whether it is the addition of an article, or the deletion of an article, or edits to an article – very often the effects are not purely local; or at least, they shouldn’t be. Someone may want to know when a certain kind of article is created. Articles that referred to an article that gets deleted may want to refer to something else. If an article changes, references to specific regions of that article may have to be re-directed or, in some cases, eliminated.

Many of these operations can be taken care of automatically by a sufficiently powerful hypertext system. Indeed, particularly in the case of multi-user editing, the consistency conditions (considered as a very general category) essentially constitute the social contract associated with the work.

Note 3.65 (When we maintain consistency).

As we mentioned in Note 3.64, in the broadest sense, we’re always maintaining some sort of consistency. But the three major categories, adding, deleting, and editing, give a nice first-run division of the cases. (There may be some other situations that will come up, and will not fit immediately into any of these three groups, e.g. maintaining some state that isn’t officially part of an article per se, but at least ideally, we can decompose the space this way.)

Sometimes users won’t want certain “weird” things to happen to their articles automatically. For example, maybe someone really means to point at the region between character position 5 and character position 31 in some particular article, and not to the string that currently appears in this region at the time when the link is created. In this case, the person creating the link should say so – and the system should be set up to adhere to the reasonable wishes of its users. In particular, it should be entirely possible for two users to have vastly different views of the document’s contents!

(Notice that in the distributed case, it we may have to adopt somewhat different notions of consistency; see Section 10.)

Note 3.66 (Consistency conditions for adding).

This is a fairly simple case, relatively speaking. It includes such things as automatic labeling (cf. Section 3.5.1) as well as the automatic addition of backlinks to appropriate articles (Note 3.61).

Note 3.67 (Consistency conditions for deleting).

The most important feature associated with deletion is to delete or redirect backlinks. See Section 7.6 for details.

Note 3.68 (Consistency conditions for editing).

This is by far the most complicated of the three cases. When an article is edited, links that point to its various components may have to be updated. In particular, when an article’s about data is edited, anything that pointed at this data must change – this includes everything that uses link-id’s (Note 5.39), i.e., backlinks in other articles, masks in the linking article, and markup.

Changes to an article’s text can have repercussions of similar complexity, especially when derivative works are taken into account; see Section 8 for details.

3.5 Subcollections

3.5.1 Labels

Note 3.69 (Cost of maintaining labels).

To maintain a label, we have to do predicate testing as we go along. This is a way of amortizing the cost of lookup. (An example would be nice, I’m sure.) The thought is that some overall savings can be obtained this way – but in general they don’t come automatically; we have to be intelligent about limiting the tests we run (at least, I don’t think we get savings for free by doing things this way). For example, we don’t add backlinks to metadata articles – but there should be other intelligent things to do too. Another idea in a similar theme is to have several functions for creating different kinds of articles; these functions can instructions for applying certain labels hard-coded into them.

Note 3.70 (Labels as made up of trivial clusions).

Given that the design of labels stores a list of link-like elements in the text field of a article, and the fact that we would like to have backlinks from the labeled articles to the label itself (Note 12.25), it would appear to me that labels are somewhat like trivial clusions (Note 8.3). Assuming that we are going to go ahead with the backlinking from labeled articles (which seems to be useful when we think about exploring or exporting a hierarchy), then we may want to treat labels in the same way we treat clusions, or alternatively, switch to putting their contents onto the about field instead.

Note 3.71 (Comparison with labels in other systems).

Various folksonomy-oriented environments allow people to use labels. The usage in these systems is certainly similar to usage in Arxana, however, I’m not completely sure that the ideas are the same. This would bear further investigation.

Note 3.72 (Label terminology).

Following on Note 3.71, it seems possible that the term “label” would be somewhat confusing if the label was thought of as a set, and not a list. So to be clear, what we really are doing is keeping track of various things on separate lists. (This is relevant e.g. for ‘label-article-insert-before’, Note 3.75).

Note 3.73 (On ‘modified-type-labels’).

An alist of types to look for paired with the corresponding label to apply. Each new or modified article is tested to see if it matches any of the given types; if so, then the corresponding label(s) will be applied. (Note that the “label” label is not treated specially.)

(defvar modified-type-labels ’((nil . plain)
                               (label . label)
                               (meta . metadata)
                               (list . list)
                               (reference . reference)
                               (section . section)))
Note 3.74 (On ‘label-article’).

This function is responsible for the mechanics of editing labels, and creates them as needed.

(defun label-article (name article-label)
  (scholium article-label
            (add-to-or-start-list (scholium-text
                                   (get-article article-label))
Note 3.75 (On ‘label-article-insert-before’).

Like ‘label-article’ (Note 3.74), but takes an additional argument before-this to say which entry the new name should be inserted just before.

(defun label-article-insert-before (name before-this article-label)
  (let* ((contents (scholium-text (get-article article-label)))
         (before-this-headed (member before-this contents)))
    (when before-this-headed
      (let ((len (- (length contents)
                    (length before-this-headed)
        (if (> len -1)
            (setcdr (nthcdr len contents)
                    (cons name before-this-headed))
          (setq contents (cons name before-this-headed))))
      (scholium article-label
Note 3.76 (On ‘put-type-labels’).

This function runs within the scope of ‘scholium’. It might make more sense to be able to add lots of different labels for a new article; the ‘cond’ function doesn’t work that way however.

(defun put-type-labels ()
  (dolist (type-label modified-type-labels)
    (when (typedata-includes type (car type-label))
      (label-article name (cdr type-label)))))

(add-hook ’new-scholium-hook ’put-type-labels)
(add-hook ’scholium-modified-type-hook ’put-type-labels)
Note 3.77 (Generalizations of the autolabeler).

At the basic level, what we want is a way to maintain a list of predicates, such that any scholium that satisfies a given predicate gets the corresponding label. We could maintain several such lists, but it is (theoretically) important to be general, and the design should be evaluated to determine how easy it is to add new predicates and labels.

Note 3.78 (Could take subcollection as optional argument).

In general, the functions that operate on all articles should probably take a subcollection as an optional argument. This might be a namespace, or a label, or a list that has been assembled by some novel means. Some of our functions already have something like this working, but it hasn’t really been standardized in any way.

3.5.2 Namespaces

Note 3.79 (Namespace implementation).

Namespaces have to be implemented in a nicely self-similar way. Whereas an article can bear any number of labels, it can be only in one namespace at a time. (Since namespaces can be nested, we might have ABa, so that the article a isn’t a member of A, but limiting a recursive search to A would still help one to find a.)

Actually, namespaces are a lot like the filesystem, so I can imagine that we might provide the ability to symlink elements from one namespace into another (similar to textual identification, see Section 8.4). So maybe best to take what I said about only being in one namespace at a time with a grain of salt. What is probably the case is that the object will only be in one namespace at a time, but that it can have “virtual copies” in other namespaces.

If we go this route then we have two things to implement, roughly self-similar namespaces themselves, and a method for doing identification between them. I hope we can follow (in one direction or another) the implementation of labels; we’ll check that out now.

The first step will probably be to make a namespace article-type, with a specially formatted text field (a list or table), which is to be displayed in a special way. We’ll most likely want a variant on ‘read-article-name’ that recurs if the name that is read in names a namespace.

Display and deletion will both have to have some work done on them to make them namespace-compatible.

Note 3.80 (Why namespaces).

Frequently one will be working with a data structure which has some symmetries that you don’t want to collapse. For example, you might have a Note and a Section with the same name in a given article. Or you might have several sections with the same name in several different articles. Or you might be working with a database like structure in which every item has a foo-type attachment, and you want these to be represented as different scholia, not one scholium with type foo. Furthermore even in cases where you aren’t expecting overlap, it might be handy to keep different collections strongly separated from each other. (E.g. if you have a bunch of articles coming from one resource and a bunch of very different articles coming from another resource.) In this case, it is probably often just as good to use labels and keep the different collections only weakly separated – indeed, when there isn’t overlap in names, a sufficiently well-developed system for working with labels will probably give you everything that you would have wanted from a system with namespaces. Be that as it may, overlap is prevalent enough to make namespaces useful. (Another approach to overlap is disambiguation, which is what Wikipedia uses; but having to use different names all the time when disambiguating could get very tedious.)

Note 3.81 (Using namespaces for importing).

In Note 3.80, we mentioned that namespaces could be useful when working with several articles with similar structure. Underscoring that point, it seems we may typically want to use namespaces when importing any secondary documents into the scholium system (e.g. the HDM Manifesto or Noosphere’s documentation). They may also be useful for articles that are stored remotely (e.g. we might maintain one or more “web” namespaces). 3z z

Note 3.82 (Maintain the ‘label’ article as a namespace?).

As one example, it might be advantageous to maintain the ‘label’ article as a namespace!

Note 3.83 (On ‘generalized-put-article’).

Like ‘put-article’ (Note 3.3), but you specify a path as well as a name. The path should specify the path to the namespace wherein the name, value pair is to be stored.

(defun generalized-put-article (path name value)
  (puthash name value (generalized-get-article path)))
Note 3.84 (On ‘generalized-get-article’).

Like ‘get-article’ but takes a path. The path specifies a list of nested namespaces, and possibly, a final non-namespace item to be found in the last namespace. Optional argument current keeps track of the “current” namespace as the path is digested. If only one item appears in the path and namespace isn’t given, the path points to an item in the main article table, and we return the corresponding item.

Be advised that there is a little bit of potentially destructive ambiguity here. But to make up for it, notice the doubly-clever use of ‘and’ and ‘cond’ (to deal with the case that we’re generalizing).

(defun generalized-get-article (path &optional current)
  (let ((first-step (get-article path)))
     ((and (not current) first-step)
     ((cdr path)
      (if (typedata-includes (get-article (car path)) ’namespace)
          (generalized-get-article (cdr path) (car path))
        (error "Path element not a namespace")))
     ((and (car path) current)
      (gethash current (car path)))
     ((car path) (get-article (car path))))))
Note 3.85 (Self-symmetry).

If we’re going to have a set-up where things look roughly the same on every level, for one thing, we may going to want to have a “current path” variable, so we can move from level to level and populate that level properly.

We may want to have a different class of labels at every level in the hierarchy. Whether we could also have labels that reach across levels is another question. And as for metadata articles, should they be stored in the top level or should they be stored in the same namespace as the article that they apply to? Storing them in the same namespace sounds good (it seems “more distributed”), however remember that it may not always be possible to access a given namespace to store metadata. Perhaps the best option is to mirror whatever distributed resources one draws upon in the local implementation; then storing metadata in the namespace local to the article being treated would make some sense. Or we could have one single namespace for metadata articles (I’ve considered this option before) which would take data coming from all other namespaces. Seems like an interesting approach, too. These matters will require more thought.

Notice that with namespaces we’re pretty likely to end up repeating some of the work people have done with file systems. On the other hand, it is probably possible to use a simplified approach, with a function to accomplish “add article to namespace at the end of this path”; like working with the file system but always from the root directory. Eventually we can add the convenience functions (really not that complicated) to prepend a default path to the root. (We seem to already have the non-interactive elements of this arrangement with ‘generalized-put-article’ (Note 3.83) and ‘generalized-get-article’ (Note 3.84).)

3.6 Data access and conversion

Note 3.86 (Introduction to data access and conversion).

The functions in this section give easy access to things we’re often interested in later on. The functions here are simple, but it is useful to have names for them anyway (Note 2.16).

(defun label-to-list (label)
  (mapcar (lambda (name)
            (format "%s" name))
          (scholium-text (get-article label))))

(defun label-to-propertized-list (label)
  (mapcar (lambda (name)
            (propertize (format "%s" name) ’name name))
          (scholium-text (get-article label))))
Note 3.87 (On ‘get-backlinks’).

This code produces the backlinks associated with name. See Note 3.57 for details. Notice that throughout the code I have been applying this idiom

(lambda (backlink)
  (get-article (car backlink)))

to the element of the returned list in order to come up with the names of the backlinked articles. However, I recently converted the backlink format to be richer than it was before so that we knew which link was doing the linking, not just which article. So presumably we should be using this additional data at least some of the time.

(defun get-backlinks (name)
  (get-metadata-field ’backlinks name))

(defun get-links (name)
  (cdr (assoc ’links (scholium-text (metadata-article name)))))
Note 3.88 (On ‘read-article-name’).

We frequently have to read the name of an article from a list. We may be able to do this in several different more intelligent ways than the one we have here! For one thing, we could use ‘turn-article-table-into-names’ (which see). Another thing to do would be to allow some other source of input, for example, some particular namespace could be specified (this might be best handled with another function, ‘read-article-name-from-namespace’, say). We could offer some recursion if a namespace is selected.

(defun read-article-name ()
  (let* ((completion-ignore-case t)
         (strings (turn-article-table-into-list))
         (string-found (completing-read
                        "Article: "
         (place (- (length strings)
                   (length (member string-found strings)))))
    (nth place (turn-article-table-into-names))))
Note 3.89 (On ‘read-article-path’).

Like ‘read-article-name’ but reads a path to an article through namespaces (cf. ‘generalized-get-article’, Note 3.84).

(defun read-article-path (&optional namespace path)
  (let* ((completion-ignore-case t)
         (strings (if namespace
                      (turn-namespace-into-list namespace)
         (string-found (completing-read
                        "Article: "
         (place (- (length strings)
                   (length (member string-found strings))))
         (ret (nth place (if namespace
                             (turn-namespace-into-names namespace)
    (setq path (append path (list ret)))
    (if (and (typedata-includes ret ’namespace)
             (y-or-n-p "Read further? "))
        (read-article-path namespace path))
Note 3.90 (Reading from non-default namespaces).

Perhaps there should be a “read from this namespace” variable that can be swapped for the default in ‘read-article-name’. This sort of overload system should probably be used whenever we are reading from a list of articles (just for example, see Note 4.33).

An alternative perspective on this matter is that it depends on the behavior of ‘get-article’. That function is responsible for finding the article with the given name, regardless of where it resides. So, ‘get-article’ needs to know where to look. Perhaps ‘name’ will contain some instructions, or, more likely, perhaps this function should take an optional argument that would help it to limit the search.

Note 3.91 (On ‘link-about-article-p’).

This provides a quick way to tell whether one of the elements of an about list is actually about the article named article-name.

(defun link-about-article-p (link article-name)
  (equal (linked-to-article link) article-name))

3.7 Further notes on scholia-based documents

Note 3.92 (Following the design of metadata articles to make namespaces?).

Note that we could in theory follow a similar pattern with the meta prefix to make additional “namespaces”, i.e., the article “(foo bar)” would be the article named “bar” within the namespace “foo”. We would maintain a label on these articles, as with the “metadata” label. But it seems more desirable to create a namespace type, elements of which would have the property that they could contain articles which aren’t elements of the main article list.

Note 3.93 (Design of real namespaces).

We currently only have one real “namespace” (see Note 3.92 for a way to create a fake namespace). A real namespace would probably have a list or a hash table in the text field and a “namespace” token into the type field.

Note 3.94 (Namespace narration).

Presumably a lot of verbose discussion of namespaces can be trimmed away when we actually have an implementation.

Note 3.95 (Finding namespaces by searching the namespace hierarchy).

We should sometimes be able to find namespaces, by searching through the namespace hierarchy for their name. (I think I may have recently seen an Emacs package posted to gnu.emacs.sources for doing something similar with directories or files.)

Note 3.96 (Versioned backlinks?).

Note that backlinks are outside of the editorial control of the person whose article is being linked to. Nonetheless, it may be handy to maintain versioned backlink repositories, possibly to correspond to different versions of linked-to and the linked-from articles (this could be very complicated).

Note 3.97 (Holographic data).

There are a number of different strategies for working with “holographic data”, i.e., data that represents a document but that contains different bits and pieces all over, which must be assembled to construct any particular view into the document. Some of these strategies involve maintenance of datastructures and some involve search. (See Note 3.101 for some further comments.) For a pictoral illustration of something vaguely similar, check out ‘animate-string’.

Note 3.98 (Simulating a unified catalog using many namespaces).

The argument for namespaces says that we could simulate a full catalog by having functions that trace through all of the namespaces and gather all of the articles that have been found there.

Note 3.99 (Are namespaces out-modded by labels?).

I’m not even sure if we should be using namespaces at all, but one category of cases that seems to motivate their use that in which you have objects that contain sub-objects. Namespaces seem like a natural model, even if we decide never to use them explicitly.

Note 3.100 (Remote subcollections).

We might want to have one subcolection for each source of articles in a distributed system. For a built-in web browser, we might also want to maintain directions for re-loading everything that’s been downloaded from the internet, probably as its own subcollection (with instructions on where to find the articles on disc and remotely).

Note 3.101 (Holographic data storage).

This is the idea of a document in which data is stored “all over”, and our views into the document get different bits and pieces of this data. For example, in Table 1, the string “the quick brown fox” is stored in a somewhat non-intuitive way, namely, indexed by letter of the alphabet rather than character position. If we look at a neighborhood of the letter “o” (in the natural metric coming from this index), we might see something like


which gives us some idea of what the document as a whole has to say. (As would any neighborhood of any element, hence the appellation “holographic”.)

Similarly, an article displayed together with its scholia gives a view into the document it is a part of. In general, its scholia may come from “all over”. We use backlinks (Note 3.56) to speed the location of scholia necessary for presenting such views.

(Perhaps something from the theory of labels or something similar can be used to speed things up too; e.g., I suppose that in addition to generating this table, we could generate a list of the words in the text too. Something to think about I guess.)

character positions
spc 4, 10, 16
b 11
c 8
e 3
h 2
i 7
k 9
n 15
o 13, 18
q 5
r 12
t 1
u 6
w 14
x 19
Table 1: Lexicographic storage for “the quick brown fox”
Note 3.102 (World-writable components versus metadata articles).

If we’re going to maintain a collection of backlinks for each linked-to article (which seems to be an exceptionally good idea), we have to decide whether to maintain such a list as a part of the article itself or as part of some other article (presumably an article that contains nothing but metadata about the article that we’re working with). This is an aesthetic consideration – we’re interested in maintaining ownership over articles, so for this reason, we don’t want them to contain world-editable components. However, it may not be so nice to have the article list filled up with system-owned metadata articles, either. Perhaps this is one case in which it would be good to use a separate namespace. (It is even conceivable that users would want different behavior in different circumstances; e.g. if all the articles are owned by one person, making portions of the articles “world-editable” may not be such a problem.)

Note 3.103 (Savings afforded by backlinks).

Here we assume that we use metadata articles of Note 3.102 and that we use a hash table to store the document. If we maintain a list of backlinks for each article, the added look-up cost of adding a link is O(1) per linked-to article (i.e. we need to find the appropriate place to store the backlink metadatum), and the cost to display an article is O(1) per linked-from article (i.e. we use the list of backlinks to find each scholium attached to the article being displayed). On the other hand, if we don’t maintain backlinks, there is no look-up cost associated with adding a link, but there is a cost of O(#Articles) associated with document display, since we have to test every article to see if it is about the article being displayed.

Note 3.104 (Private information).

This is one potential use of namespaces (we could have a public and a private namespace… though actually this sounds a bit more like something that we would want to use labels for). Privacy is generally considered to be an important right.

Note 3.105 (Reading articles into a given namespace).

If a bunch of scholia have been saved to a file, it would be convenient to have a way to read them all into one specific namespace.

Note 3.106 (Intelligent search).

One topic we must concern ourselves with from time to time is the scope of propagation. In a large system, we don’t want to have to look at everything after any given change. Even if we maintain a list of backlinks (see Note 3.56), we might well have to search through everything in order to find the articles that are backlinked to! (Which may not be such a bad thing if we use a hash table.) Another possible solution is to try to search intelligently, by using namespaces to guide the search. If each backlink tells you roughly where the corresponding linking article can be found, then you can recover that article much more efficiently. I’m pretty sure this relates to the system for IP addressses used on the web and also to the “humber” system used in Xanadu. (The Wikipedia page about the IP system probably contains important background). See also Note 3.46 for related thoughts on using namespaces to limit search.

Note 3.107 (Possible relationship between labels and ‘type’ data).

The type field complements labels; labels can be applied by anyone, but type data can only be changed by the object’s owner. When you render an article, you look on its type field first to get the basic type data specified by the author. Then you look at its metadata article to find backlinks; if any of these backlinks are to labels, you may find additional rendering information there.

Note 3.108 (Movable scholia).

At various places in the lifecycle of a scholium, we’ll want it to move around, either concretely (from one base of attachment to another) or ephemerally (appearing in some new place while continuing to be attached in its original location). We already have various features associated with some of these processes, for example, transclusion (Section 8.2) is about viewing a given article through some new window. However, we’ll want to do more, e.g., one thing that should be coming soon is the ability to pass scholia “through” a compilation (Section 8.7). We’ll eventually want to be able to do the reverse as well. In general, support for the various concrete operations (e.g. passing a scholium through a reference and making it stick) are lacking at present.

4 Adding articles to the digital library

Note 4.1 (Introduction to adding articles in the digital library).

The code in this section provides various wrappers for the ‘scholium’ function, making it easy to create scholia. Section 4.1 details the process of adding appropriate bookkeeping information to new articles; articles created using the functions in this section have standard bookkeeping information added to them automatically. Section 4.2 concerns adding scholia based on already-existing sources of text. Section 4.3 presents functions that the user would call to create articles in which the text field is filled in by the user.

For editing articles that have already been added to the library, see Section 7.

Note 4.2 (Setting ‘type’ field at creation time).

The user might want to set the type field at creation time, not just text and about (indeed, we could probably do a better with about in a few places, e.g. throughout Section 4.2). But if it is not provided immediately, type can still be edited in later.

4.1 Supplying initial bookkeeping information

Note 4.3 (On ‘sch-book’).

Interactive functions that call ‘scholium’ should always provide bookkeeping information.

The following definition can be extended to add any sort of initial bookkeeping information we might like to maintain. The bookkeeping field will be revised when the article is edited (see Section 7).

(defun sch-book ()
  ‘((created ,user-login-name ,(current-time-string))))

4.2 Adding text from an existing source

4.2.1 Adding text from buffers

Note 4.4 (On ‘make-current-buffer-into-article’).

We store the object representing the buffer in the text field. There is no particular reason to set the type (at least, one has yet to appear).

An alternative approach would be to make an article’s text reflect the current contents of the buffer (i.e., save the buffer as a string) – but we currently don’t do this. However, it would probably be nice to have that option, or even just the option to make a certain selection into an article.

(defun make-buffer-into-article (&optional buffer name)
  (let* ((buffer (or buffer
                     (get-buffer (read-buffer
                                  "Buffer: "
         (name (or name
                   (read-string "Name: "
                                (buffer-name buffer)))))
    (scholium name buffer nil nil (sch-book))))

(defun make-current-buffer-into-article (name)
  (interactive (list (read-string
                      (concat "Name (default "
                              (buffer-name (current-buffer)) "): ")
                      (buffer-name (current-buffer)))))
  (make-buffer-into-article (current-buffer) name))
Note 4.5 (Arbitrary buffer over current buffer?).

We could write ‘make-buffer-into-article’ and ‘make-current-buffer-into-article’ in such a way that the former calls the latter and uses ‘save-excursion’. Both approaches seem to have the same end result.

4.2.2 Adding text from files

Note 4.6 (On ‘make-file-into-article’).

This function adds a file to the article list. Again, the text field is not filled in with a string representing the text directly, but rather, with the name of the file that holds the text. A token is added to the type field explaining that this is a file.

(defun make-file-into-article (path name)
   (let* ((bufn (buffer-file-name))
          (pth (read-file-name
                (if bufn
                    (concat "File (default: " bufn "): ")
                  (concat "File: "))
                (if bufn
          (dnme (file-name-nondirectory pth))
          (nme (read-string (concat "Name (default: " dnme "): ")
     (list pth nme)))
  (scholium name path nil ’(file) (sch-book)))

4.3 Interactively supplying text

Note 4.7 (Global varibles describing new scholia).

The variables ‘new-scholium-name’ and ‘new-scholium-about’ are used to build new articles interactively. For now, type is ignored by these functions. Also, we don’t have a ‘new-scholium-text’ field, since we get the text another way. It could be that an approach with a ‘new-scholium-text’ variable would allow us to unify the treatment here with the one used in previous subsections.

(defvar new-scholium-name nil "Name of our new scholium.")
(defvar new-scholium-about nil "What the new scholium is about.")
Note 4.8 (On ‘new-scholium-mode’).

This mode is invoked by ‘make-scholium’ in the buffer in which the new scholium’s text is to be supplied.

(define-minor-mode new-scholium-mode
  "Mode for composing a new scholium.
  :init-value nil
  :keymap ’(("\C-c\C-c" . escape-scholium-creation)))
Note 4.9 (On ‘escape-scholium-creation’).

Once the new scholium’s text has been supplied, this function creates a scholium from that text and other data the user has supplied. It then restores the window configuration that was active before ‘make-scholium’ ran. It also nullifies ‘new-scholium-name’ and ‘new-scholium-about’, since we’re done with these things.

(defun escape-scholium-creation ()
  (scholium new-scholium-name
            (buffer-substring-no-properties (point-min) (point-max))
  (kill-buffer (concat "Editing scholium: " new-scholium-name))
  (set-window-configuration sch-win-config)
  (setq new-scholium-name nil
        new-scholium-about nil))
Note 4.10 (On ‘make-scholium’).

This function is called every time the user makes a scholium with new text (i.e. text that is typed in on the fly). Functions for making scholia about articles, parts of articles, buffers, etc., are given in this document and all use this function.

(defun make-scholium ()
  (setq sch-win-config (current-window-configuration))
  ;; we allow this to be set elsewhere
  (unless new-scholium-name
    (setq new-scholium-name (read-string "Scholium name: ")))
  (set-buffer (get-buffer-create "Scholia Display"))
  (other-window -1)
  (other-window 1)
  (switch-to-buffer (get-buffer-create (concat "Editing scholium: "
Note 4.11 (Redisplay after running ‘make-scholium’).

It would probably be fairly convenient to redisplay whatever object was being displayed after ‘make-scholium’ has run. (It would be nice to have the configuration of windows be 100% consistent!) In order to make this be efficient, we would probably want to a system for incremental markup, as described in Note 5.27. Presumably this just means adding ‘redisplay-article’ to an appropriate hook.

Note 4.12 (Add a ‘make-scholium-hook’?).

We could add a hook here if we wanted to make it easy to do interesting things after a scholium is added, like update the display. (We might want to make a section on default settings – and we might want to make more hooks throughout the scholium system code.)

4.3.1 Scholia attached to the current article

Note 4.13 (On ‘make-scholium-about-current-article’).

This function makes a scholium about the article as a whole.

(defun make-scholium-about-current-article ()
  (when name-of-current-article
    (setq new-scholium-about ‘(((,name-of-current-article))))
Note 4.14 (On ‘make-scholium-about-part-of-current-article’).

This function makes a scholium about one specific portion of the article.

This function makes the (somewhat unrealistic seeming) assumption that the current article and the current buffer are the same thing. This situation should be resolved.

However, if no article is current yet, then perhaps we should offer to make a scholium about the current buffer?

(defun make-scholium-about-part-of-current-article (beg end)
  (interactive "r")
  (if name-of-current-article
        (setq new-scholium-about
        (message (concat (format "%s--%s" beg end) " added.")))
    (message "Make some article current first.")))
Note 4.15 (The use of ‘deactivate-mark’ in this code).

Since regions are being selected interactively, unless we ‘deactivate-mark’, we will be left with a highlighted region after some of the scholium-creating function have run. This doesn’t seem clean to me.

Note 4.16 (On ‘make-scholium-about-current-line’).

Here is a little convenience wrapper for working with lists.

(defun make-scholium-about-current-line ()
  (make-scholium-about-part-of-current-article (line-beginning-position)

(defun make-scholium-about-current-line-quickly ()
  (setq new-scholium-name (buffer-substring-no-properties
  (make-scholium-about-part-of-current-article (line-beginning-position)

(defun make-scholium-about-current-line-quickly-and-completely ()
  (scholium (buffer-substring-no-properties

(defun make-scholium-about-current-sentence ()
  (make-scholium-about-part-of-current-article (line-beginning-position)
Note 4.17 (Semantics of lists).

Lists may benefit some special semantics. In particular, it might be useful to only display the scholia that are attached to the current line (or item more generally). Also, an indicator to show the existence of scholia attached to other lines (when such scholia do in fact exist).

Note 4.18 (Scoring).

Maybe the way to go with scoring would be to make a change to the type field to indicate that the thing has type “score 10” or whatever. Or perhaps we would want to attach a scholium?

Note 4.19 (Locally-stored data versus indexed data).

This reminds me of the different models of data storage. We could keep everything local (a score field for each scholium) or store things in an index (look-up each scholium to see if it has a score). The magic of hash tables says that lookup is “linear” but is the factor lower if the data is kept locally? Maybe on average.

Of course, if you have to search a collection to find the extent of some predicate, that will take some time; if everything matching a certain description has been stored on a certain list, then you already have the predicate’s extent without search.

There’s really no reason not to store some things locally if we know that search is going to always be local (e.g. the score of a certain article is probably fine to store locally) and we have permission to store that data locally. See Note 3.97 for more on this general topic – the issues here have been pretty much resolved with in the design of metadata articles (Note 3.46).

Note 4.20 (Local list of scholia).

One example of how local lists could come in handy (like we were talking about in Note 4.19) would be to keep a local list of all the scholia that are attached to the current article. Then, if changes have to be propagated, we just look at the things on this list, rather than having to search through all of the articles again to figure out which scholia are attached to the current article. Of course, the local list could be kept updated as the set of scholia attached to the current article changes. This would (I think) be an optimization, and isn’t essential for building a working system.

4.3.2 Creating a scholium that applies to several regions

Note 4.21 (On ‘reading-regions-mode’).

This mode is invoked by ‘make-scholium-about-several-parts-of-current-article’, and adds an editing mode in the buffer containing the current article that enables the user to select regions that the scholium will be about.

(define-minor-mode reading-regions-mode
  "Mode for reading in regions.
  :init-value nil
  :keymap ’(("\C-c\C-c" . add-region)
            ("\C-c\C-g" . escape-reading-regions-mode))
  (message "C-c C-c to add regions; C-c C-g to make scholium."))
Note 4.22 (On ‘add-region’).

This function adds regions to ‘new-scholium-about’.

(defun add-region (beg end)
  (interactive "r")
  (setq new-scholium-about
        (cons ‘(,name-of-current-article
  (message (concat (format "%s--%s" beg end) " added.")))
Note 4.23 (On ‘escape-reading-regions-mode’).

When all of the regions desired have been selected, this function calls ‘make-scholium’ to finish things off.

(defun escape-reading-regions-mode ()
  (reading-regions-mode -1)
Note 4.24 (On ‘make-scholium-about-several-parts-of-current-article’).

This function makes a scholium that applies to several portions of the article, using the mode and so on featured in this section.

(Note, it doesn’t seem that this is displayed quite right; I’m getting two copies of the scholium’s text in the Scholia Display buffer.)

(defun make-scholium-about-several-parts-of-current-article ()
  (let ((article (get-article name-of-current-article)))
    (if (article-buffered article)
        (switch-to-buffer (get-buffer (scholium-text article)))
      (switch-to-buffer "Main Article Display"))
    (setq new-scholium-about nil)
    (reading-regions-mode 1)))

4.3.3 Scholia about the current buffer

Note 4.25 (Making scholia about the current buffer).

We present facilities for making new scholia about the current buffer or file that this buffer is visiting. Corresponding functions for displaying scholia that are attached to the current buffer appear in section 5. This function ensures that the buffer is associated with on article on the article list; otherwise, we can’t make a scholium about it.

Note 4.26 (On ‘call-if-user-adds-current-buffer-to-article-list’).

This is used by functions that require the current buffer to be an article; typically they recall themselves after the buffer has been added. It is used by ‘make-scholium-about-current-buffer’ and ‘display-scholia-about-current-buffer’.

(defun call-if-user-adds-current-buffer-to-article-list (fct)
  (when (y-or-n-p "Buffer not an article, add to list? ")
         (read-string (concat "Name (default: "
                               (current-buffer)) "): ")
                      (buffer-name (current-buffer))))
        (funcall fct)))
Note 4.27 (On ‘make-scholium-about-current-buffer’).

This function makes a scholium about the current buffer, requiring that it be an article. (Maybe we should just add the current buffer to the article list transparently, rather than giving the prompt in ‘call-if-user-adds-current-buffer-to-article-list’.)

(defun make-scholium-about-current-buffer ()
  (let ((article (get-article (buffer-name (current-buffer)))))
    (if (not article)
      (setq new-scholium-about
            ‘(((,(buffer-name (current-buffer))))))
Note 4.28 (Scholium creation modalities).

We developed all sorts of creation facilities for generic articles, shouldn’t we have them all over again for buffers? It seems only fair that there would be some symmetry between the way articles and buffers are handled. Maybe there is some way to reuse the work.

On the other hand, maybe we don’t need a lot of complex functions for specially handling buffers, since once the buffer has been made into an article, all of the functions for working with articles apply. (Contrast Note 5.12!)

Note 4.29 (Making a scholium about a file.).

Doing something like what ‘make-scholium-about-current-buffer’ does that makes a scholium about the current buffer as a file shouldn’t be very different at all. I’ll want to think a little more about the relevance of doing things this way: it may not really matter if we run the exporting and importing routines just right. (We could also write a function to make scholia about a file without displaying the file.)

4.3.4 Creating references

Note 4.30 (Fancy references).

In Note 2.33 it isn’t made completely clear what the set of reference targets is comprised of. One might be led to assume that it is only possible to reference to an article. In the current implementation, that assumption is correct; ‘follow-reference’ calls ‘display-article’, which in turn only knows how to display articles. However, in theory we could just as well reference any object that the system knows how to display; so, if ‘display-article’ was overloaded to work with regions as well as articles, for example, there would be no problem with referencing regions (although the code for ‘make-reference-in-current-article’ would have to be adjusted slightly). It would be nice to add this feature, since HTML already has targets that are approximately this general (well, more like “go to a specific point within the given document”). It shouldn’t be too much to ask to enable following a reference and then performing a given action (e.g. “find the first occurance of foo in the referenced document”).

Another relevant thing to point out is that we can have different kinds of references appear in different colors (Note 2.29); this is true of other sorts of linking as well (Note 2.6). Compare Note 13.30.

Note 4.31 (Abstract connections).

References have two about fields, one comprising the reference’s target, and the other, the region from which the reference originates. This shows how an abstract connection between two documents can be formed by a third element, rather than such connections being the exclusive domain of about data itself. This is an important feature for semantic networks; larger-scale structures can be expressed using basic pieces.

Note 4.32 (On ‘genref’).

Instead of using ‘gensym’ (which causes problems) we do something similar to generate references with unique ids. Note that this might cause some problems when we go to the distributed way of doing things (or even just save and restore articles with references in them after making some intervening edits), since references with the same names might mean different things. But of course, this is true of all scholia with the same names, so I propose not to worry about it too much right now.

It may turn out to be advantageous to use a reference counter that is local to each article.

Actually, the scheme proposed here seems pretty weak; saving files and then reading them back in after Emacs has been shut down could cause problems. It might be much better to have a reference counter in each metadata article, so that the collection of references associated with a given article is always unique. Furthermore, references associated with a given article should possibly be recorded on that article’s metadata explicitly as references.

(defvar *reference-counter* 0)

(defun genref ()
  (setq *reference-counter* (1+ *reference-counter*)))
Note 4.33 (On ‘make-reference-in-current-article’).

This function is similar to ‘make-scholium-about-part-of-current-article’ (Note 4.14) except that the type is set to “reference”. References currently don’t have any text, but we could later set things up to let them have a docstring or something like that.

(Are backlinks working properly in an article that has several references associated with it?)

(defun make-reference-in-current-article (beg end &optional target)
  (interactive "r")
  (let ((target (or target
    (if name-of-current-article
        (when target
          (scholium ‘(reference ,name-of-current-article ,(genref))
      ;; Maybe the message should instead be an offer to make
      ;; a scholium about the current buffer?
      (message "Make some article current first."))))
Note 4.34 (Speedy reference creation).

It would be handy to be able to make a reference to an article and automatically create and follow the reference, all at the same time. (Maybe also subsets of these actions, like create but don’t follow.) Probably we could do something similar for generalized references.

(defun create-follow-reference ()
  (let ((name (buffer-substring (point) (mark))))
    (unless (get-article name)
        (set-buffer (get-buffer-create name))
        (make-current-buffer-into-article name)))
    (make-reference-in-current-article (min (point) (mark))
                                       (max (point) (mark))
    (display-article name)))
Note 4.35 (Come-from references).

As weird as it might sound, it could be handy to have a function for adding references to the current article from some other article (or class of articles). See Note 3.56 for further comments. (It seems possible, though maybe not too likely, that we would be able to use the outline of ‘make-reference-in-current-article’ to create a generic function for both “come-from” and “goto” references.)

Note 4.36 (Inspecting or changing reference target, deleting reference, changing “alt text”).

Various operations that you might like to do with a reference need should to be supported. changing the “alt text” should be supported automatically – just edit the text – but there is the chance that the user could screw things up and accidentally delete the whole scholium if they do the wrong thing, so we may want to provide some assistance.

Note 4.37 (Reference component access).

These functions give easy access to information specifying the referenced article (the target), the region the reference applies to, and the name of the article the reference lies in. These functions rely on the formulaic nature of the type data of references, namely, a link to the passage wherein the reference is made is given, followed by a link to the referenced article.

Notice that these functions would have to change if we later allow multiple sources (presumably, regions) to be part of the same reference.

(defun reference-source-link (reference)
  (first (scholium-about reference)))

(defun reference-from-article (reference)
  (car (reference-source-link reference)))

(defun reference-to-article (reference)
 (car (second (scholium-about reference))))
Note 4.38 (Justification of the representation of references).

It might seem excessive to have backlinks to a reference on the reference target and the reference source, but remember that we want to be able to display references as scholia. Now, one thing that might be the case is that instead of storing a standard backlink on the reference target, we might want to store a special “referenceed-by” backlink. But that is probably excessive, at least, given our rendering scheme (see Note 5.54).

4.3.5 New undirected articles

Note 4.39 (On ‘make-new-undirected-article’).

Although one could simply create a new buffer and add that buffer to the article list (as in Section 4.2.1), sometimes it may be more intuitive to simply add a new undirected article directly to the article list.

(defun make-new-undirected-article ()
  (setq new-scholium-about nil)

4.3.6 Followups

Note 4.40 (Introduction to followups).

This section is about making scholia about other already-existing scholia. It relies on markup features from Section 5, and its presence here is a bit ugly (Note 2.15). Perhaps it will be moved somewhere else later.

Note 4.41 (Complex “about” features for scholia).

More complicated features like creating scholia directly about several regions of a given scholium or like creating a scholium that relates to several regions of several different scholia can be coded up soon, following the style of section 4.3.1.

Note 4.42 (On ‘name-of-current-scholium’).

The function can be called from anywhere; “current” is defined relative to the position of ‘point’ in the Scholia Display buffer. (It is natural to assume that there is only one current scholium, given the way the contents of this buffer have been put together.)

Note that scholia appear once in the Scholia Display buffer, so there is only one “current scholium” (with one name) when we look at things this way. If we were going to do something like this for the main article buffer, then we’d need to do a bit more. (In fact, various functions to associate marked regions with scholia need just this sort of special touch.)

(defun name-of-current-scholium ()
    (set-buffer (get-buffer-create "Scholia Display"))
    (let ((ret (car (scholia-named-at-point))))
      (if ret
          (message (format "%s" ret))
        (message "Position cursor on a scholium in Scholia Display."))

(defun make-scholium-about-current-scholium ()
  (when (equal (buffer-name (current-buffer)) "Scholia Display")
    (let ((cur (name-of-current-scholium)))
      (when cur
        (progn (setq new-scholium-about ‘(((,cur))))

4.4 Further notes on adding articles to the digital library

Note 4.43 (Forbid the creation of “empty” scholia?).

Perhaps ‘make-scholium’ should disallow (or at least display an idiot box about) the creation of “empty” scholia – they seem somewhat silly.

Note 4.44 (Transcluded strings).

We’d should have some function to insert an auto-updating string (like the name for the buffer as an article). We would typically want such auto-updating portions to appear in a specially colored face.

Note 4.45 (Multi-reference creation?).

This section should probably cover: the event of interactively making a scholium that relates to several different pieces of text from several different articles. Note that you could probably do things like this by using the generic list, and marking things in different ways that show their relationship together, the create the new scholium from that list.

Note 4.46 (Anonymous scholia).

Creating “anonymous” scholia should probably be made possible; by which I mean, the names of these scholia would be filled in automatically but would be non-significant (compare ‘gensym’). Of course, if we could have automatic names that were significant that would be cool too.

Note 4.47 (Merging editorial comments).

It would be nice if there was an easy way to merge editorial comments into the main document, corresponding to the ability to make such comments in the first place (Note 2.3).

5 Rendering articles

5.1 Formatting articles for display

Note 5.1 (“Formatting” versus “display” versus “rendering”).

The term “formatting” seems to capture a specific notion of producing a user-visible version of an arbitrary article, that is then “displayed”. “Rendering” is meant to capture a more general idea, which includes non-user-visable features (e.g. invisible text properties). Some sorts of articles will require further work to “render” (see Note 3.41 and Note 3.42).

Note 5.2 (Generality, or lack thereof, in the rendering system developed here).

It woulb be great to have all sorts of different kinds of rendering systems. (For one example additional to those mentioned in 5.1, see Note 3.43; we could supply other examples!) For now, we have limited ourselves to something pretty basic.

Note 5.3 (Producing plain text).

The function ‘sch-plain-text’ uses various criteria to render scholium system objects as a string (the collection of acceptable objects is the same as the collection of link targets, plus articles specified simply by name). Its behavior can be fine-tuned by changing the variable ‘sch-plain-text-hook’.

Note 5.4 (On ‘sch-plain-text-hook’).

Alternative ways of setting the return value of ‘sch-plain-text’.

(defvar sch-plain-text-hook nil)
Note 5.5 (On ‘sch-plain-text’).

This function is called by ‘mark-up-scholium’ to render scholia that are to be displayed alongside the main article, and also by ‘transclude-article’ (see Note 8.18) and other functions in Section 8.2.

Its goal is to turn arbitrary articles into strings. This will be done in different ways depending on the sort of article in question. (And could be done in other ways depending on other stuff.) Compare Note 2.31.

Here’s how it works: different kinds of “objects” are to be distinguished from one another by simple tests – is the ‘car’ equal to passage? Does looking up the object in the article table produce anything? Then act as appropriate, grabbing the text that is specified. Currently it works on input article names or input articles. (If input is both the name of an article and an article, it will be treated as a name.)

Another thing that might be handy to be able to process is simple strings, which aren’t article names, or articles (obviously).

Note that links are rendered differently depending on context.

Links can’t actually run from arbitrary object to arbitrary object within the current version of the system. That can probably be fixed easily.

Converting the function to render scholium system objects in general is a step towards answering the request in Note 3.5 about extending the purview of ‘display-article’. At that point, it will take more than just a title as argument, certainly; and it will need a “triage” phase in order to figure out what sort of object it has been applied to; various modifications will have to be made so that it can be applied to various sorts of objects.

(defun sch-plain-text (title-or-article)
  ;; this overloading of the input & reliance upon ‘get-article’ to
  ;; sort things out... could probably be revised into something
  ;; better
  (let* ((obj (or (get-article title-or-article)
         (text (scholium-text obj))
         (type (scholium-type obj))
    ;; This seems like the correct way to override a ‘cond’ form.
    (run-hooks ’sch-plain-text-hook)
    (when (not ret)
       ((bufferp text)
        (save-excursion (set-buffer (get-buffer text))
                        (setq ret (buffer-string))))
       ((typedata-includes type ’file)
        (let ((bufs (buffer-list))
              (live nil))
          (while (and bufs (not live))
            (when (equal (buffer-file-name (car bufs))
                         (expand-file-name text))
              (setq live (car bufs)))
            (setq bufs (cdr bufs)))
          (if live
              (progn (set-buffer live)
                     (setq ret (buffer-string)))
            (find-file text)
            (setq ret (buffer-string))
            (kill-buffer (current-buffer)))))
       ;; these quoted things should presumably themselves be rendered
       ;; as links (and we probably don’t need the crazy markup for
       ;; things about the whole buffer, ever)
       ((typedata-includes type ’reference)
        (if (equal (reference-to-article obj)
            (setq ret
                  ;; it might be kind of cool to at least include a
                  ;; snippet of the context of this link, say 3 lines
                  ;; worth
                  (format "\"%s\" links here."
                          (reference-from-article obj)))
          (setq ret
                (format "This is a link from \"%s\" to \"%s\"."
                        (reference-from-article obj)
                        (reference-to-article obj)
       ((stringp text)
        (setq ret text))
        (setq ret (format "%S" text)))))
Note 5.6 (Turn list of “links here” items into listing).

It would be nice to be able to view all of the “links here” articles in a listing (as in Section 6.3), and from there in a compilation (Note 8.53).

Note 5.7 (Rendering special list forms).

With a small loss of transparency (note 2.16), we can set up some specially-encoded text fields. See, e.g., Section 8.2.

Note 5.8 (Strategy for ‘cond’s).

Any time we see a ‘cond’ in this code, we could wrap it in a backquote and unquote a variable that says how to deal with things that come up later in the code (see Note 2.15).

Note 5.9 (Actionable references).

When displaying a reference as the main article, it would be nice to have the names of the articles (or other objects) themselves rendered as references to the articles (or other objects) in question. These wouldn’t officially be “references” from the point of view of the system (unless they were temporary ones), but they would act like references for the user.

Some examples of this sort of thing appear in Section 6.5, and are hooked in by ‘scholia-display-extras-hook’ (Note 5.76).

Note 5.10 (Render the names of articles?).

Depending on the value of a variable like ‘sch-use-names’ or something of that sort, this could print the names of the scholia above their text contents. We might also want to print a list of articles that this article is “about” (assuming there is more than one).

5.2 Managing windows and buffers

Note 5.11 (Side-by-side buffers).

The first display style shows the main article in one buffer and scholia about this article in another buffer. The scholia may be attached to specific regions of the main article, in which case, this connection is shown graphically. Other display styles (e.g. threading, see Note 2.34) are in the works.

Note 5.12 (Special treatment of buffers and files).

Buffers and files need some special treatment for display, even if they don’t need to be given much attention at scholium-creation time (Note 4.28). For example, it would be nice to set up automatic display of scholia associated with a given file if some spceial variable is set. One class of buffers that requires special treatment are the killed ones (see Note 5.71). This issue relates to Note 2.23.

Note 5.13 (Simpler side-by-side annotation).

A simpler method for doing side-by-side editing has been discussed recently on the help-gnu-emacs mailing list1111 11 This gives local features similar to some of those provided by the scholium system, but none of the global features. (We should make sure that we actually can simulate this behavior, I think it should be no problem.)

Note 5.14 (A physical scholium system).

Working on this paper, I’ve taken to printing the document out single-sided and binding the drafts along the margin, so that each page has a blank facing page that I then use for notes. Sounds familar, doesn’t it? This approach is helping me understand how to use paper better. And it is giving me some ideas about useful features to work into the electronic version. See also Note 5.15.

Note 5.15 (Working with large-form scholium systems).

With larger pieces of paper, or other media, it is possible to put together scholium-based documents with more content and more than two columns. With a couple of screens, or a larger screen, useful things along these lines could be accomplished with Emacs too. (As it is, two side-by-side windows is about all that can be managed on my computer.)

Working with large format documents, I noticed some interesting effects. On the left, I might see a document assembled out of several smaller articles (say, by identification), and on the right, I would see all of the comments that applied to any of these articles, while a third columns would give comments that applied to text in the other columns, or to the document as a whole, or to any of the other articles that had gone into building previous columns.

Even with just a couple of columns to work with, various complex things like this should be possible to do on the computer (e.g. with different colors or different levels of indentation).

(The point is, eventually should be able to make more interesting sort compilations than those described in Section 8.7.)

Note 5.16 (Avoiding information overload).

A tricky option would be to just display the scholia that relate to portions of the document that are on the current screen. The paper version of Note 5.14 does this by default. But I’m not sure how to make Emacs behave this way.

Note 5.17 (Context).

Context is important. This is one of the reasons that scholia themselves are important – they get comments situated in a certain context. The notion of context seems to be different for text and hypertext. In text, everthing appears in context, whereas in hypertext, nearby things sometimes don’t appear at all; they reside on the other side of a link, and may never be seen by a reader. But this duality is somewhat misleading, in particular, it is sometimes thecase that “information overload” would occur if everything appeared on the same page or in the same place. And sometimes the context of the page can be misleading; regular text frequently relies on global contextual features. (And of course, readers might not actually see everything that is on the page!) It is an interesting feature of reality and human psychology that contexts are often decipherable – we can figure out how things fit together. A “good” hypertext model should presumably include features that reflect this aspect of our thinking, and allow users to explicitly recontextualize things as they see fit. For some technical suggestions on this matter, see Note 5.68.

Note 5.18 (Dot dot dot).

How should scholia in later generations be displayed? In the default mode, without threading, a “dot dot dot” would be useful. (And in the case of scholia that bridge the gap between two articles, the “dot dot dot” probably needs to have a special meaning.) Anyway, note that these dot-dot-dotted things could be found by a depth-first search through the document collection. (Find anything about the current article, then find anything about that, etc., then find the next thing about the current article, etc.)1212 12 Compare the 2nd figure in Corneli and Krowne (2005) (cf. Footnote 2, Page 2)..

(defvar pre-sch-win-config nil "Saved window configuration.")
(defvar sch-win-config nil "Saved window configuration.")

(defvar buffer-associated-with-current-article nil)
Note 5.19 (On ‘article-buffered’).

Here’s a little convenience function that tells you whether the article’s text lives in a buffer or not.

(defun article-buffered (article)
  (bufferp (scholium-text article)))
Note 5.20 (Make new buffered article).

We should provide some command to make a new article in a new buffer, all in one shot.

Note 5.21 (Detaching articles from buffers).

Perhaps it should be possible to detach a buffered article from the buffer that contains it.

Note 5.22 (On ‘scholia-overwhelm-display’).

This function displays scholium stuff. It is called by ‘display-article’ (Note 5.88) and ‘display-scholia-about-current-buffer’ (Note 5.92). If there is a main article buffer (i.e. the article to be displayed lives in its own buffer), we use that buffer to display the article; otherwise, we use the “Main Article Display” buffer. Scholia that match the appropriate set of conditions in ‘mark-things-up’ will be rendered to the “Scholia Display” buffer.

(defun scholia-overwhelm-display (text)
  (unless pre-sch-win-config
    (setq pre-sch-win-config (current-window-configuration)))
  (if rendering-target-buffer
      (pop-to-buffer rendering-target-buffer t)
    (switch-to-buffer (get-buffer-create "Main Article Display") t))
  (insert text)
  (goto-char (point-min))
  (setq buffer-associated-with-current-article (current-buffer))
  (other-window 1)
  (switch-to-buffer (get-buffer-create "Scholia Display") t)
Note 5.23 (Switching between views).

We offer a few convenient functions for switching between the article-plus-scholia browsing display and whatever came before. It would also be nice to offer a function for switching between the article display and the generic listing display.

(defun back-to-normal ()
  (setq sch-win-config (current-window-configuration))
  (set-window-configuration pre-sch-win-config)
  (setq pre-sch-win-config nil))

(defun back-to-other-view ()
  (setq pre-sch-win-config (current-window-configuration))
  (set-window-configuration sch-win-config)
  (setq sch-win-config nil))

5.3 Sorting scholia for markup purposes

Note 5.24 (Introduction to sorting scholia).

The main function in this section is ‘sort-scholia-by-beg-position’, which is called by ‘mark-things-up’.

The use for this function is to put scholia in the order in which they will be rendered; the order is given by sorting the scholia on their about data. Since there are several different kinds of about data, the sorting criteria are a little complicated. For one thing, each scholium may be about several different articles, but here we only want to sort relative to their relationship to one given article.

The criterion we use when sorting is: scholia that are about the whole article go first; then, we rank any other scholia according to their first beginning position. (Thus, any scholia that are about several regions within the same article need to have a subsidiary sort done to order the indicated regions; this is accomplished, as needed, by the function ‘first-beginning-about-article’, below.)

Note 5.25 (On ‘first-beginning-about-article’).

Return 0 if about is about all of article, i.e., otherwise return the character position of the first region within article that about is actually about. If there is none, return nil. (Actually, let me note that in the usage we have established so far, we would already know that in the case that about is a string, it is about the article here, and in the case that about is a one-layer list, similarly; so some of the tests we do here are as yet unneeded.)

Here we’re assuming that if there is some link to the article that isn’t a passage link, then we treat the link as about the article as a whole. The case in which the link is about the whole article and some part of the article might possibly be better treated some other way; but I’ll leave that case for subsequent work.

(defun first-beginning-about-article (about article)
   ;; condition for the scholium to be about the article as a whole.
   ;; The condition is that there is _some_ link to the article that
   ;; is NOT a ‘‘passage’’-type link.  In this case, we just return 0.
   ((member-if (lambda (link)
                 (and (equal (car link) article)
                      (not (link-type-accessor link ’passage))))
   ;; else, collect the regions of ‘article’ that ‘about’ indicates,
   ;; and sort them.
    (let* ((marking-links
            (let (marked)
              (mapc (lambda (elt)
                      (when (and
                             (typedata-includes-passage (link-type elt))
                             (equal (linked-to-article elt) article))
                        (setq marked (cons elt marked))))
            (car (sort
                  (lambda (link1 link2)
                    (< (link-beginning link1)
                       (link-beginning link2)))))))
      (link-beginning earliest-link)))))
Note 5.26 (On ‘sort-scholia-by-beg-position’).

This function orders scholia according to the magnitude of the smallest beginning a region of article that the input scholia mark. All links are considered when finding the first marked region.

(defun sort-scholia-by-beg-position (scholia article)
   (sort scholia
         (lambda (scholium1 scholium2)
           (let ((beg1 (first-beginning-about-article
                        (scholium-about scholium1)
                 (beg2 (first-beginning-about-article
                        (scholium-about scholium2)
             (and beg1
                  (< beg1 beg2)))))))
Note 5.27 (Incremental markup).

Instead of sorting everything each time we display, it would be nice to handle things incrementally: just add new markup for the newly created scholium, and store things in their natural order after reparsing. I think that this approach should work, but it would involve making a number of changes to the code. (Note that in the setup we’re using, updates to metadata may have to be wholesale, even if updates to text properties are incremental.)

5.4 Marking things up

Note 5.28 (Introduction to marking things up).

As mentioned in the introduction (Note 2.25): when articles are displayed, both overlays and text properties are used for markup. Overlays make it possible to put visible markup into font locked buffers. However, when text is cut and pasted or whatever, overlays disappear, so we use text properties too, to add a needed degree of permanence to the markup.

5.4.1 Faces for overlays

Note 5.29 (A more abstract face function).

Perhaps ‘new-simple-scholium-face’ could take an argument – say the scholium – and provide a different face for each user, or a different face for each type, or whatever. Probably there should be a variable called ‘scholium-face-function’ that points you to the actual function that is going to be in use. Using this sort of abstraction throughout would certainly facilitate user customization, and so, would be a good thing.

Note 5.30 (On ‘new-simple-scholium-face’).

Adapted from ttn’s ricette-mode.el. Nota bene: new faces are automatically customizable, so if you don’t like the way they look, you can change them.

(defmacro new-simple-scholium-face (num color doc)
  (let ((prop (intern (concat "sch-face-" (int-to-string num)))))
       (defvar  ,prop ’,prop)
       (defface ,prop
         ’((t (:foreground ,(symbol-name color)
               :underline ,(symbol-name color)))) ,doc))))
Note 5.31 (Underlining versus foreground).

It is worth considering using the underline attribute instead or in addition to the foreground attribute: underlining would be less obtrusive in documents that already use faces. We could have two variants; one with underlining for the main article, one foreground for the scholia display. We could also give users some immediately-customizable options. (Do the default colors I picked out work well?)

(new-simple-scholium-face 1 maroon1 "First scholium face.")
(new-simple-scholium-face 2 aquamarine1 "Second scholium face.")
(new-simple-scholium-face 3 IndianRed1 "Third scholium face.")
(new-simple-scholium-face 4 yellow1 "Fourth scholium face.")
(new-simple-scholium-face 5 firebrick1 "Fifth scholium face.")
(new-simple-scholium-face 6 plum1 "Sixth scholium face.")
Note 5.32 (Reference face).

A special face for references. We don’t currently have a special face for visited references, but this can be added if/when we start keeping track of which references have been visited (see Note 6.58). We might also want to record and display information about visited articles in general.

(defface sch-reference-face
  ’((t (:foreground "red" :underline "red")))
  "Face for references in the scholium system.")
Note 5.33 (Special face for scholia about whole article?).

It might be good to have a special face (and perhaps other special treatment, like a special section of the scholia display buffer) for scholia that apply to the whole of the main article.

Note 5.34 (Special face for transclusions that have been collapsed).

We’re going to have to have a special face for transclusions that have been collapsed (see Note 2.39).

5.4.2 Masks and miscellaneous markup mechanics

Note 5.35 (Color by number).

The ‘scholia-count’ variable keeps track of how many scholia have been displayed. The ‘scholium-face’ function selects a face to use when displaying the next scholium according to this count. This simple display mechanism seems sort of lame (hence, it is turned on with the ‘use-crazy-font-lock’ variable); better things may come later, see, e.g. Note 2.29.

Note that turning off “crazy font lock” makes it so that no text properties are added to the buffer, but of course it would be best if text properties were added and overlays omitted. On the other hand, we need to do some thinking to sort out the use of text properties versus the use of overlays. Cutting and pasting multiple copies of some markup in the same buffer may cause some trouble at commit time. See Section 8.2; also, compare Note 8.60.

(defvar scholia-count 0 "Number of scholia about the article found.")

(defun scholium-face ()
  (let ((short-count (mod scholia-count 6)))
    (cond ((eq short-count 0)
          ((eq short-count 1)
          ((eq short-count 2)
          ((eq short-count 3)
          ((eq short-count 4)
          ((eq short-count 5)

(defvar use-crazy-font-lock t)
(defvar main-article-overlays nil)
(defvar scholia-overlays nil)
Note 5.36 (Overlays and text properties adjust when editing).

By default, overlays and text properties will move appropriately when text is added to the buffer. This is important!

Note 5.37 (Access to rendering target buffer).

The rendering target buffer is either “Main Article Display” if the current article isn’t buffered, or whatever buffer the article lives in if it is buffered. (Well, actually, we can always redirect rendering to any buffer of our choice; but the preceding statement accurately describes the default operation.) We use ‘get-rendering-target-buffer’ as a shorthand when we grab the rendering target buffer, and ‘set-buffer-to-rendering-target-buffer’ to make that buffer current for editing. (Note that we could go about this in a slightly different way, namely set the ‘rendering-target-buffer’ variable to “Main Article Display” instead of ‘nil’ when there is nothing to override the default – but what would be the point?)

(defun get-rendering-target-buffer ()
  (get-buffer (or rendering-target-buffer
                  (get-buffer "Main Article Display"))))

(defun set-buffer-to-rendering-target-buffer ()
  (set-buffer (get-rendering-target-buffer)))
Note 5.38 (The ‘scholia’ property).

At render time, we use a text property (which we call ‘scholia’) to associate regions with the scholia that are attached to them. A region can be associated with more than one scholium, so the ‘scholia’ text property stores a list. Specifically, a list identifying the links that associate various scholia with this region (see Note 5.39).

This information helps us restore order to shifted markup when committing edits (see Section 7.3).

Note 5.39 (Link-id’s).

The elements of stored in the ‘scholia’ text property’s list are ordered pairs (two-element lists), each containing information of the following form: (1) the name of a scholium associated with this buffer position; paired with, (2), the link number corresponding to (one of) the link(s) through which said scholium came to be associated with the particular region being marked up.

Lists of this form are a fairly common idiom in this work, and they get a special name, the link-id.

Note 5.40 (Experiment with many ‘scholium’ properties).

All else equal, it might be advantageous to use independent ‘scholia’ properties instead of just one ‘scholium’ property (see Note 7.2).

Accordingly, I tried switching over to using one ‘scholium’ property for each marked-up region. These properties were given somewhat complicated names – namely, the link-ids that are currently stored as elements of the ‘scholia’ property (Note 5.38). However, this doesn’t work, as suggested by the following example.

(progn (put-text-property (point) (1+ (point)) ’(foo 1) t)
       (put-text-property (point) (1+ (point)) ’(foo 1) nil)
       (text-properties-at (point)))

This is because1313 13 Thanks Andreas Schwab, help-gnu-emacs, 2005/12/12.

(eq ’(foo 1) ’(foo 1)) ;=> nil

The Emacs text property engine is uniformly ‘eq’-based instead of ‘equal’-based. (So, if we happened to have an association between link-id’s and integers, this would have worked, but that seems like a silly kludge.)

Indeed, this ‘eq’ preference is pretty much fatal to the utility of non-symbol text properties (unless we managed to hang directly onto the actual link-ids that would be used for the names of the text properties, which seems infeasible; or alternatively used some other sort of weird work-around, as above). A further difficulty is associated with the fact that these various ‘scholium’ text properties would be indiscriminately mixed in with any other text properties that happened to be stored at point, requiring filtering for any useful en masse processing.

I have some ideas that could possibly improve the prospects for doing away with the ‘scholia’ property and replacing it with several ‘scholium’ properties, but it seems like anything realistic would hacking Emacs C. But since it seems that there are no immediate problems associated with using just one ‘scholia’ property, we plunge ahead that way.

Note 5.41 (On ‘add-to-scholia-property-within-region’).

For each character in the region between beg and end, this function grabs the ‘scholia’ property and replaces it with a version that has been modified to include the input value (a link-id; see Note 5.38).

The same property is used in the Scholia Display buffer, although that may be a bit of an abuse (cf. Note 5.58); something simpler would work for our needs there – but the current way is expedient. Also, it seems to be suggestive – perhaps in the future we’ll be able to treat the Scholia Display buffer as a proper scholium-based article itself, presumably by using transclusion and identification, as appropriate (see Section 8).

(defun add-to-scholia-property-within-region (start end value)
  (while (< start end)
    (put-text-property start (1+ start) ’scholia
                        (get-text-property start ’scholia)
    (setq start (1+ start))))
Note 5.42 (On ‘current-markup’).

This variable will be used to record the regions of the article being displayed which have scholia attached to them and are, consequently, marked up. This information will be stored at markup time by ‘mark-up-region’ (Note 5.44). It is important to have this information on record so that we have something to compare to after editing takes place (see Section 7.4).

Its format is a list of elements of the form

((<name> <link number>) <beg> <end>)

where ‘beg’ and ‘end’ denote the beginning and end of the region marked up via the specified link (but see Note 5.43!).

(defvar current-markup nil)
Note 5.43 (Investigating ‘current-markup’ in the context of masks).

If the link in question is being masked, then the format of ‘current-markup’ changes from the form described in Note 5.42 to the following:

((mask (<name> <link number>)) <beg> <end>)

where ‘beg’ and ‘end’ denote the beginning and ending of one particular region coming from the mask for the indicated link.

I’m not sure this is really sufficient information. Should we know which part of the mask we’re dealing with? I.e., use something like a link-id, but for mask components?

Note 5.44 (On ‘mark-up-region’).

Like ‘add-to-scholia-property-within-region’, but also adds to ‘current-markup’ (see Note 5.42).

(defun mark-up-region (start end value)
  (add-to-scholia-property-within-region start end value)
  (setq current-markup
        (add-to-or-start-list current-markup
                              (list value start end))))
Note 5.45 (On ‘non-printing-types’).

This variable will record a list of the types of scholia that we typically don’t want to print when displaying an article. The ‘derives-from’ type is an example of a scholium that we typically don’t want to print; see Note 8.24.

User should temporarily set ‘non-printing-types’ to ‘nil’ before ‘mark-things-up’ runs whenever they wish to display all scholia.

(defvar non-printing-types nil)
Note 5.46 (Masking links).

It is sometimes necessary to use links that point at something other than the thing that they were pointed at originally – or more typically, the same thing, but found in a different place (Note 3.54).

For example, consider the common case of a link pointing at some region of text found in a given article. If the linked-to article gets edited, the link under consideration may have to be adjusted if it is going to continue to point at the “correct” region.

Thus, when a linked-to article changes, it will typically have to communicate changes to back to the linking article; or rather, to the metadata article associated with the linking article (Note 3.46). These changes take the form of masks, which cover and re-route links.

A given mask may continue to change as the linked-to article changes. The strategy for keeping masks up-to-date is as follows. When we discover (at commit time) that a link needs to be redirected, an entry in the “masks” metadata field of the article containing the link is created or adjusted. Specifically, the new link(s) designed to replace the old link are stored as an entry on the “masks” list, as an ordered pair consisting of: (1) old link-id; followed by, (2) the new link(s) replacing the old one. Subsequent changes in the linked-to article cause old masks to be deleted and new masks to be swapped in; we never do “recursive masking”.

Mask-bearing links are clearly distinguished as such (at markup time), so that we can easily identify them later (when parsing markup). The function that generates the mask-identifying tags is ‘compute-usable-about-data’ (Note 5.49), and the function that picks these tags out from among the others at commit time is ‘store-link-masks’ (Note 7.16). Indeed, ‘store-link-masks’ is responsible for creating new masks as well.

At the other end of the process, the function ‘compute-usable-about-data’ examines each link from the about data of the linking article together with any corresponding masks, and replaces any link that is masked with the data coming from that mask (Note 5.49).

(Changes to the linking article’s about data can change link-id’s, or do away with the requirement for a particular mask altogether – this will have to be dealt with when we have mechanisms for changing about data!)

Note 5.47 (Masks).

In light of the comments in Note 5.46, it is possible to think of a mask as a map that takes a given link to to a region, and replaces it with a list of several regions.

The format of a mask, then, is

((<name> <link number>) &rest regions)

where the regions are pairs (two-element lists), each giving the beginning and end of a region that the link is being redirected to.

This format lacks generality! Certainly a link that is pointing to something other than a region may need to be redirected, for example, a link pointing at a page, when the page gets deleted. For the time being, these interesting cases can be dealt with through ad hoc measures. Eventually we’ll want to make a study of all of the different redirection cases; but getting the platform working reasonably well takes priority. (See also Note 5.48.)

Finally, observe that the format used here is related to the format of the ‘current-markup’ variable (Note 5.42).

For an interesting meditation on masking perceptions, see Lem1414 14 Stanislaw Lem, The Futurological Congress.

Note 5.48 (Generality of masks).

Once we have more general sorts of masks (Note 5.47), we’ll have a host of new and interesting conditions to consider. Different types of links will presumably have to be masked in different ways, and under different conditions.

Furthermore, we will eventually be masking text data as well as about data.

Note 5.49 (On ‘compute-usable-about-data’).

This function runs within the scope of ‘mark-things-up’ (Note 5.54). Its role is to identify the regions to be marked up, based on the about data expressed by attached scholia, and augmented by any masks associated with this data. In short, this is the mask-applying part of the algorithm described in Note 5.46.

The way it works is as follows. We look through all of the links (although we really don’t need to do this anymore, given that we have specifically identified the relevant links by using link-id’ed backlinks in the article being displayed; the change would have to come in at the level of ‘mark-things-up’ or higher) – and if the link is relevant, then it will be reflected in the value returned by ‘compute-usable-about-data’. However, if the link is masked, it will be the mask that is reflected, instead of the link itself.

Recall that we can’t ‘assoc’ the link across the masks; we need to ‘assoc’ the link-id instead.

The format of the return value is kind of ugly; we could almost certainly do without the extra ‘list’ layer.

Note that in the return value, the name of the scholium is supplied, not the name of the linked-to article – we’ll subsequently know (when applying markup) that all of the links apply to the linked-to article, whereas we won’t know where these links are coming from unless we record that specially.

(defun compute-usable-about-data ()
  (let (usable-data
        (about (scholium-about scholium))
        (masks (get-metadata-field ’masks
        (link-number 0))
    (dolist (link about)
      (setq link-number (1+ link-number))
      (when (link-about-article-p link name-of-current-article)
        (let* ((link-id (list (scholium-name scholium) link-number))
               (mask (assoc link-id masks)))
          (if mask
              (dolist (reg (cdr mask))
                (setq usable-data
                       ‘((mask ,(car mask)) ,@reg))))
            (setq usable-data
                   (append (list (list (scholium-name scholium)
                           (let ((beg (link-beginning link)))
                             (if beg
                                 (list beg (link-end link))
                               (list nil))))))))))
Note 5.50 (On ‘mark-things-up-customizations’).

This may not have to be used.

(defvar mark-things-up-customizations nil)
Note 5.51 (On ‘scholia-display-pre-update-hook’).

This gives us a chance to do various customizations to the environment before the main part of ‘mark-things-up’ runs.

I imagine that the role that it will play will be similar to the one played by the functions that run right in ‘pre-mark-up’, namely to zap variables and settings that we don’t want to have around anymore.

(defvar scholia-display-pre-update-hook nil)
Note 5.52 (On ‘pre-mark-up’ ).

This function generally zaps things, to prepare for markup.

(defun pre-mark-up ()
  (setq current-markup nil)
    (remove-list-of-text-properties (point-min)
  (mapcar #’delete-overlay main-article-overlays)
  (setq main-article-overlays nil)
  (setq scholia-count 0))
Note 5.53 (On ‘mark-things-up-hook’).

For unconditional customizations to ‘mark-things-up’. The functions added to this hook run after everything else ‘mark-things-up’ does is done.

(defvar mark-things-up-hook nil)
Note 5.54 (On ‘mark-things-up’).

This function assembles text, text properties, and overlays: it is the main rendering engine, called by ‘display-article’ to get scholia and appropriate markup onto the screen. It is important that ‘raw-scholia’ be defined (and be a list of scholia) for this function to work properly.

The first part of the plot is to delete the old markup; we call ‘pre-mark-up’ to take care of this.

A list of “raw scholia” is expected to be present in the context in which this function runs; typically this data is provided by ‘display-article’ (see Note 5.88), but it can be supplied by any stand-in (e.g. ‘display-scholia-about-current-buffer’ of Note 5.92).

The raw scholia are first sorted using ‘sort-scholia-by-beg-position’ (Note 5.26) and then translated into a usable form by ‘compute-usable-about-data’ (Note 5.49).

The function ‘mark-things-up’ can handle different sorts of scholia differently (e.g., references are marked up in the conventional way); see Note 5.55.

In order to selectively display scholia, the ‘raw-scholia’ variable should be modified before this function runs.

(defun mark-things-up ()
  (let ((scholia (sort-scholia-by-beg-position
    (dolist (scholium scholia)
      (unless (typedata-includes-element-of-list
               (scholium-type scholium)
        (let ((usable-about-data (compute-usable-about-data))
              (current-position-in-scholia-display (point)))
             (typedata-includes (scholium-type scholium) ’reference)
             (equal (reference-from-article scholium)
  (run-hooks ’mark-things-up-hook))
Note 5.55 (Ancillary functions for ‘mark-things-up’).

The functions that assemble and apply the markup, and in particular the value stored on the ‘scholia’ property, run within the scope of ‘mark-things-up’. The ‘scholia’ property is built according to the description from Note 5.38.

References are rendered by ‘mark-up-reference’; regular scholia are rendered by ‘mark-up-scholium’. Note that there are only two different nontrivial kinds of markup at present.

Some very ancillary functions are described in Note 5.56.

Note 5.56 (Functions for adding overlays).

These function run within the scope of ‘mark-up-scholium’ to add overlays to the display.

(defun add-overlays-in-scholia-display-buffer ()
  (setq scholia-overlays
         (make-overlay current-position-in-scholia-display
                       (get-buffer "Scholia Display")
  (overlay-put (car scholia-overlays)
               ’face (scholium-face)))

(defun add-overlays-in-rendering-target-buffer ()
  (setq main-article-overlays
         (make-overlay (second elt)
                       (third elt)
  (overlay-put (car main-article-overlays)
               ’face (scholium-face)))
Note 5.57 (On ‘mark-up-reference’).

We call this function from ‘mark-things-up’ (Note 5.54) to render a scholium if it has reference type and its about data indicates that that the reference originates from (i.e., appears in) the current article. See Note 4.37 for a description of the format of the return value of the function ‘reference-source-link’ used here.

References should perhaps be rendered differently depending on their sub-type (in particular, a different face could be used for references with different sub-types).

(Do we really want to loop through all of the elements of ‘usable-about-data’? Mightn’t there be something there corresponding to the linked-to article?)

(defun mark-up-reference ()
    (dolist (elt usable-about-data)
      (mark-up-region (second elt)
                      (third elt)
                      (first elt))
      (when use-crazy-font-lock
        (setq main-article-overlays
               (make-overlay (second elt)
                             (third elt)
        (overlay-put (car main-article-overlays)
                     ’face ’sch-reference-face)))))
Note 5.58 (Using the ‘scholia’ property for everything!).

Notice that we use the same ‘scholia’ property for references, and not some special property. This seems to make reparsing easier (see Section 7.3) – but we can also easily filter the references out of the collection of all scholia when needed (Note 6.12).

Note 5.59 (Alternate reference display).

In Note 2.33 we asserted that references will be displayed as markup. However, we could certainly set things up so that we could switch between the normal view and a view where references are displayed as “proper scholia”.

Note 5.60 (On ‘mark-up-scholium’).

This inserts scholia and marks them up, together with the marked up regions (when these exist; the function does both whole-article scholia and region-specific scholia).

The ‘scholia-count’ variable is set for purposes of face selection; see Note 5.35.

(defun mark-up-scholium ()
  ;; this part takes place in the scholia display buffer
  (insert (sch-plain-text (scholium-name scholium)))
   ;; add a list to make it possible to reuse the scholium property
   (list (scholium-name scholium)))
  (when use-crazy-font-lock
  (insert "\n\n")
  ;; this part is relevant to the buffer containing the main article
  (dolist (elt usable-about-data)
    (when (second elt)
        (mark-up-region (second elt)
                        (third elt)
                        (first elt))
        (when use-crazy-font-lock
  ;; adjust count once everything else is done here, so same count
  ;; applies in both buffers (useful for coloration purposes)
  (setq scholia-count (1+ scholia-count)))
Note 5.61 (Markup in the Scholia Display buffer).

There is no technical reason to use overlays here, because we assume that this buffer is not subject to font-lock (i.e. we could use text properties for everything). But, for the sake of uniformity, we always use overlays for fontification. Also, there’s no particular reason to maintain the ‘scholia’ property as a list in this buffer, but we do that too.

Note 5.62 (Make “Scholia Display” read-only?).

Maybe ‘mark-things-up’ should set and unset read-only status for the Scholia Display buffer. Some of the functions we’ve written so far do toggle read-only status. We should make a firm decision one way or the other!

At present, it seems to me that it is best to have it be editable and have code ready to propagate changes back to their sources. Indeed, this would make a good example of the power of this system. See Note 8.32.

5.4.3 Further notes on marking things up

Note 5.63 (Highlighting the current scholium).

Note that it may prove to be useful to highlight the current scholium in some way, or current scholia with some kind of color-coded depth showing how many scholia are about the current piece of text. (“Current” here comes from where the point is in the main article display, but I suppose it might be reasonable to do something similar with the scholia display buffer.)

Note 5.64 (Semantics of color).

The actual markup behavior should depend on the semantics that are currently en vogue, see Note 2.29 and Note 2.31. Maybe we should be using a hook or a redefinable function to handle the “when use-crazy-font-lock” case.

Note 5.65 (Lots of different overlay styles possible).

In theory we could have a lot of different overlay styles. Implementing these styles seems a little tricky, but isn’t too impossible. Note that some of the different overlay setups may benefit from having more information than just the name stored locally (assuming that it takes a lot of time to look things up by name to find more properties, which isn’t necessarily a fair assumption in the case of hash tables).

One example of something that would be nice would be to have nested scholia display well. (I’m sure they would even work consistently at present.) It may be that delimiters would be a more effective way of illustrating such relationships.

Note 5.66 (Sometimes we don’t want overlays at all).

Another point is that sometimes we don’t want overlays to show up at all – they could be very distracting when attached to code, for example. Whether or not an overlay is shown in a given instance may depend on global state variables, properties of the article that is being marked up, or specific data that has been stored as part of a scholium. (We’ll have to add more code to handle these sorts of criteria.)

Note 5.67 (Emacs task: blinking?).

This is a pretty minor issue, but it would be cool to hack a display independent blink property into emacs. Blinking already works for cursors. So, how to do it for text? The point is that if there was a “blink” feature in Emacs, the “move to region” things could make the text they find blink (this would work for monochrome displays of Note 2.30 just as well as any other).

Note 5.68 (Displaying nearby scholia).

A nice display system will put some of the scholia nearby the current document onscreen. The default operation of ‘mark-things-up’ (performed through the offices of ‘sort-scholia-by-beg-position’) is to show an article together with its scholia in order; Note 2.35 talks about going the other way. But we can do still more complicated things.

People like having nearby things to look at. This is one reason that print dictionaries, for example, are kind of fun. You look at the dictionary, and you can see words that are nearby lexicographically; sometimes they are related and sometimes they aren’t.

In different contexts, different sorts of “closeness” may be relevant, and different sorts semantics would help make the contextual display useful.

5.5 Display interface

Note 5.69 (Selective Displays).

More selective displays could be offered. For example, we might display only the articles that are “about” the current article and that are of some specified “type”. We could also take bookkeeping information into account, or “meta-level” information like what subcollection (or namespace?) of the digital library the article is located in. Metalevel information (maybe by definition) comes from outside of the system itself, for example, we might display only scholia from a certain directory. These sorts of deployments are where the real power of this system lies. (See Note 2.27 for general comments along these lines, and Note 2.35 for somewhat similar thoughts along a diverging line.)

5.5.1 Different treatment for buffered and non-buffered articles

Note 5.70 (On ‘rendering-target-buffer’).

If a given name is associated with a buffered article, then that buffer will be where the article is displayed. If the article to be displayed is in a buffer, this variable will be set that buffer object. Otherwise it will be set to nil.

(defvar rendering-target-buffer nil)
Note 5.71 (Dealing with killed buffers).

For some reason, ‘rendering-target-buffer’ gets set to the “killed buffer” object when the buffer containing the main article is killed. That isn’t nice; presumably we’ll need to add a function to the ‘kill-buffer-hook’ that will ensure that if this buffer is killed, this variable will be set back to nil (or whatever it should in fact be set to). We will presumably also have to remove this function from the hook when the main article changes.

Note 5.72 (Better way to name (or access) buffered articles).

Either we should search on the text field for the current buffer, and ignore the name, or we should store the “official” name of the buffer as a buffer-local variable. (E.g. this buffer should be called “SBDM for CBPP” but ‘display-scholia-about-current-buffer’ can, at present, only access the buffer by its ‘buffer-name’. I think that adding a buffer-local variable at creation time would probably be very straightforward, then we just need to look-up that variable instead of using ‘buffer-name’. (Of course, we could make ‘buffer-name’ be the default for the “buffer article name”.) Note that unless the article is created by the approved means, we won’t have a chance to store the variable; this suggests that the display section should come after the creation section (which makes plenty of sense). Note that everywhere ‘buffer-name’ appears in this paper things will have to be modified if we make this sort of switch.

5.5.2 Display mechanics

Note 5.73 (What just happened to the screen?).

The simple display mechanism we use here will overwhelm your whole display (see Note 5.22); use M-x back-to-normal to return to the pre-display window configuration.

Note 5.74 (Making ‘sch-plain-text’ work with weird types).

We should either make sure that ‘sch-plain-text’ can handle all articles with many different types.

Note 5.75 (On ‘scholia-display-post-update-hook’).

This provides a way to customize the article (and scholia) being displayed.

(defvar scholia-display-post-update-hook nil)

(add-to-list ’non-printing-types ’fake)
Note 5.76 (On ‘scholia-display-extras-hook’).

This hook is run at the end of ‘scholia-display-extras’ (Note 5.77) and allows for easy tweaking of the scholia display (for example, displaying special sorts of scholia for articles with certain types).

Some basic additions to this hook appear in Section 6.5.

Note 5.77 (On ‘scholia-display-extras’).

This is used to add some extra stuff to the Scholia Display for purposes of navigation and establishing context. It is called by ‘display-article’ (Note 5.88) and ‘display-scholia-about-current-buffer’ (Note 5.92).

It might be nice to have this in the main article window, but that could also be confusing, especialy if the main article is associated with some buffer; so I’m taking the route of caution here. Also note that according to the principle of order 2.15 this stuff should probably appear much later on in the document, perhaps in Section 6.5.

It might be more appropriate to have some of these extra features display in or above the main article buffer; the info system has a nice un-editable bar for display of various navigational data.

(defun scholia-display-extras ()
  ;; this setting is useful for presentations.
  (goto-char (point-min))
  ;; Be careful that this is ignored when the article is saved or
  ;; otherwise processed internally.
   "Title: "
   (propertize (format "%s" name-of-current-article) ’face ’italic)
  (goto-char (point-max))
  (run-hooks ’scholia-display-extras-hook)
  (goto-char (point-min)))
Note 5.78 (More than one parent).

We’ll want to allow more than one parent eventually; compare what happens when following a scholium/reference in a region that has more than one of these things written about it. For the purposes of markup, “parent” may become a variable (so add ‘(reference-to-parent foo)’ for the foo parent) and maybe, if the various reference scholium objects created here are actually helpful, make them include lists of references? Multi-references haven’t been done anywhere in this code so far, but they seem like a reasonable idea. See also Note 6.66, Note 6.67.

Note 5.79 (Text for display purposes only).

I think the setup will work so that individual sections that bear identification markup will all have their changes propagated properly, i.e., independently. However, I think we may have to do a little more work to make sure that the name of the article (currently shown in italics whenever an article is displayed, though it would probably be nicer to emulate the style used by info) and any other after-the-fact textual additions are ignored whenever the article is saved.

5.5.3 Display styles

Note 5.80 (On ‘display-style’).

As discussed in Section 2.2.3, there are a number of different styles we’d like to offer users to choose between when they go to display an article. For now, the relevant settings for the display style variable are ‘plain’ (the default, displaying all scholia) and ‘contextual’ (which causes only those scholia associated with the region being displayed to appear; see Note 2.32).

There are a few problems here: if we go with the contextual display, what about scholia that apply to the article as a whole? That’s a little tricky. I guess for now we just leave them out?

Set this variable with ‘set-display-style’, not manually.

(defvar display-style ’plain)
Note 5.81 (On ‘window-displayed-substring’).

This variable will hold the string that is being shown through a given window at a given point in time.

(defvar window-displayed-substring nil)
Note 5.82 (On ‘set-display-style’).

Use this function to control the setting of ‘display-style’ (Note 5.80). Relevant arguments are ‘plain’, or ‘contextual’.

(Actually, anything but ‘contextual’ will set ‘display-style’ to its default setting. Eventually, we might want to be able to have contextual display together with some other non-default features, at which point we’ll have to adjust this function appropriately.)

(defun set-display-style (style)
  (cond ((eq style ’contextual)
         ;; we may need a "double hook" here, so we get the local
         ;; hooks set up in the correct window
         (add-hook ’display-article-hook
         (setq display-style ’contextual))
         (remove-hook ’display-article-hook
         (setq display-style ’plain))))
Note 5.83 (On ‘initiate-contextual-updating’).

This function gets the rendering target buffer set up to do live-updating of scholia.

For now, this function doesn’t do anything for labels – since for now, labels typically don’t have scholia on their text. However, this can be undone later, if necessary.

In addition, we don’t yet have any code for getting rid of contextual updating in an individual buffer OTF.

(remove-hook ’window-scroll-functions
             ’set-window-displayed-substring t)
(defun initiate-contextual-updating ()
  (unless (typedata-includes (scholium-type article) ’label)
    (save-excursion (set-buffer rendering-target-buffer)
                    (add-hook ’window-scroll-functions
                              ’maybe-update-scholia-display nil t))))
Note 5.84 (On ‘maybe-update-scholia-display’).

In order for this to work optimally, we’d might to keep track of a list of the current scholia that are being displayed. But for now, I suppose (contrary to this function’s name) we can just redisplay all the scholia we encounter every time.

(defun maybe-update-scholia-display ()
  ;; this is just a ridiculous thing to run -- for testing purposes
  ;; only.  Eventually, we’ll actually want to analyse the text,
  ;; figure out which scholia are relevant, and update the scholia
  ;; display.
  (save-excursion (set-buffer (get-buffer-create "*scratch*"))
                  (insert window-displayed-substring)))
Note 5.85 (On ‘set-window-displayed-substring’).

This function will be an element of the ‘window-scroll-functions’ hook when the display mode is set to ‘contextual’.

(defun set-window-displayed-substring
  (setq window-displayed-substring
        (buffer-substring (window-start)

5.5.4 Displaying articles

Note 5.86 (On ‘display-article-hook’).

We will later do some “interesting” things with this function, so we add a hook. For example, one use for this hook is to maintain a history of articles that have been displayed; see Section 6.4.

(defvar display-article-hook nil)
Note 5.87 (On ‘raw-scholia-selector’).

This function is here both to select raw scholia for display when ‘display-article’ runs, and to provide for a choice between different ways of selecting raw scholia. It returns a list of scholia.

This function is set up to run within the scope of ‘display-article’ (Note 5.88).

We select slightly different scholia when the article that is being displayed is a label; in particular, we don’t want to display scholia that indicate the current article as a ‘parent’, since these scholia will already be listed in the label’s text itself.

If we’re trying to display scholia contextually, then this function should probably be doctored with some, to make it so that the initial set of scholia that are displayed are contextually appropriate, i.e., are just those associated with the on-screen portion of the buffer that is being displayed.

(defun raw-scholia-selector (&optional what-is-displayed)
  (cond ((eq what-is-displayed ’label)
          (lambda (scholium)
            (member-if (lambda (link)
                         (member ’parent (cdr link)))
                       (scholium-about scholium)))
          (mapcar (lambda (backlink)
                    (get-article (car backlink)))
                  (get-backlinks name-of-current-article))))
        (t (mapcar (lambda (backlink)
                     (get-article (car backlink)))
                   (get-backlinks name-of-current-article)))))

;; (eq display-style ’contextual)
;;          (save-excursion (set-buffer (get-buffer-create "*scratch*"))
;;                          (erase-buffer)
;;                          (insert window-displayed-substring))
;;          nil
Note 5.88 (On ‘display-article’).

Display article found via path (which can just be the name of an article in the main article tabel), if said article exists. If the article lives in a buffer, that buffer will be where the article is displayed; otherwise the article is displayed in the “Main Article Display” buffer. Note that this function can also be used to display labels (it calls ‘display-label’; but see Note 5.103, since some other approach might be valuable sometimes).

I don’t want to have all of the children appear as scholia when a label is browsed, at least not by default; but it is kind of neat to know that they can be made to appear; if we don’t do the ‘remove-if’ then the scholia display will be the OTF-assembled compilation mentioned in Note 1.3.

It makes more sense to always run ‘display-article-hook’ after the ‘cond’, and to put things that are conditional upon being in this specific branch into ‘scholia-display-post-update-hook’ or, barring that, some additional branch-specific hook. This of course means that the parent will be added to the history list if that is where we browse from; this is intended.

If it turns out to be needed here, we could reuse the trick of running a hook and then making a test before the ‘cond’, which we’re familiar with from e.g. ‘sch-plain-text’ (Note 5.5).

It would be good to make the buffer disposition (left? right?) is consistent when we use this function to display labels; I think we have it sorted out properly for the display of “normal” articles.

We may want to treat namespaces and labels together in the same ‘cond’ branch (currently namespaces aren’t handled specially by this function).

Since we are now reading in paths, these paths have to be parsed. This is the job of ‘generalized-get-article’ (see Note 3.84).

(defun display-article (path)
  (interactive (list (read-article-path)))
  (let* ((article (generalized-get-article path))
         (name (scholium-name article)))
    (if (not article)
        (error "No article by that name found")
      (setq name-of-current-article name)
       ((typedata-includes (scholium-type article) ’label)
        (display-label name)
        (setq rendering-target-buffer "*Generic List*")
        (switch-to-buffer (get-buffer "*Generic List*"))
        (setq buffer-read-only nil)
        (switch-to-buffer (get-buffer-create "Scholia Display") t)
        (let ((raw-scholia (raw-scholia-selector ’label)))
        (switch-to-buffer (get-buffer "*Generic List*"))
        (setq buffer-read-only t)
        (pop-to-buffer "Scholia Display")
        (other-window -1))
        (if (article-buffered article)
            (setq rendering-target-buffer (scholium-text article))
          (setq rendering-target-buffer nil))
        (run-hooks ’scholia-display-pre-update-hook)
        (scholia-overwhelm-display (sch-plain-text article))
        (let ((raw-scholia (raw-scholia-selector)))
        (pop-to-buffer (get-rendering-target-buffer))
        (run-hooks ’scholia-display-post-update-hook)))
      (run-hooks ’display-article-hook))))
Note 5.89 (Argument of ‘display-article’ is typically a string).

Note that for typical interactive use, a string is expected, which means a string is produced, which means that when ‘get-article’ is called in the next line, bad things will happen for articles whose names are not strings – unless we do something about it.

Note 5.90 (Would have liked to associate a type with paths).

It would be nice to use a special type of object to distinguish between paths and names. (I.e., if ‘read-article-path’ returns an object of type path when it actually builds a path and ‘generalized-get-article’ looks for this type before deciding what to do, we could be OK just passing in an article name to the generalized function.) But I’m not sure how to assert that a given form has a given type (or even whether this is possible). Maybe by working with structs one could do it, but I don’t understand structs (their documentation seems to be lacking).

Note, the idea of associating a type with a form seems very scholiumific. But I’ve forgotten whether Lisp has anything for doing this.

Note 5.91 (On ‘redisplay-article’).

This accomplishes a simple task.

It would be nice if we could get the point restored to its original position after this thing runs.

Also, it would be good to have the function run automatically after scholia have been added about the document.

(defun redisplay-article ()
  (display-article name-of-current-article))

5.5.5 Displaying scholia about a given buffer

Note 5.92 (On ‘display-scholia-about-current-buffer’).

This function is similar to ‘display-article’ (5.88), but it works directly on the buffer level. (Stylistically this function should probably just be a thin wrapper, but we’re running with it for now.)

This function should perhaps check to see whether the current buffer has been edited since the last time this function (or similar, through other means) was executed. If the buffer has been edited, the user should probably be prompted, and asked whether to reparse (Section 7.3) before redisplaying.

(defun display-scholia-about-current-buffer ()
  (let ((article (get-article (buffer-name (current-buffer)))))
    (if article
          (setq rendering-target-buffer (current-buffer))
          (run-hooks ’scholia-display-pre-update-hook)
          (scholia-overwhelm-display (sch-plain-text article))
          (let ((raw-scholia
                 (mapcar (lambda (backlink)
                          (get-article (car backlink)))
                         (get-backlinks name-of-current-article))))
          (pop-to-buffer (get-rendering-target-buffer))
          (run-hooks ’scholia-display-post-update-hook)
          ;; weird!
          (run-hooks ’display-article-hook))
Note 5.93 (Running ‘display-article-hook’ from ‘display-scholia-about-current-buffer’?).

It seems a bit little weird to have to run this hook here.

Note 5.94 (Automatically display scholia about any found buffer).

It might be handy to integrate the scholium system into day-to-day Emacs operation by adding ‘display-scholia-about-current-buffer’ to the ‘find-file-hook’.

Note 5.95 (Displaying a generic buffer together with scholia).

You should be able to simply call ‘display-scholia-about-current-buffer’ after loading the buffer (untested). To turn it on all the time, add it to the appropriate hook. Note that this won’t add any new scholia to the system. (To add new scholia at file load time, we’ll need some additional code.)

5.5.6 Displaying labels

Note 5.96 (On ‘display-label-hook’).

This provides a way to customize the behavior of ‘display-label’.

(defvar display-label-hook nil)
Note 5.97 (On ‘currently-displayed-label’).

This gives us a handle on the most recently displayed label. This facilitates only the simplest improvement to reverting behavior.

(defvar currently-displayed-label nil)
Note 5.98 (On ‘display-label’).

The function ‘display-label’ uses the catalog browsing feature of Section 6.3. If we want to do other interesting rendering things with articles that have special types, we can follow the usage of ‘display-label’ in ‘display-article’. It might be good for this function to run its own hook, e.g., for maintaining a special history (see 6.17).

(defun display-label (name)
    (let* ((completion-ignore-case t)
            (scholium-text (get-article ’label)))
           (label-strings (mapcar (lambda (name)
                                    (format "%s" name))
           (string-found (completing-read
                          "Label: "
           (place (- (length label-strings)
                     (length (member string-found label-strings)))))
      (nth place label-names))))
  (article-menu-listing (label-to-propertized-list name))
  (run-hooks ’display-label-hook))

(add-hook ’display-label-hook (lambda ()
                                (setq currently-displayed-label name)))

(defalias ’list-label ’display-label)
Note 5.99 (On ‘display-intersection-of-labels’).

For displaying everything bearing every one of the input labels.

(defun display-intersection-of-labels (&rest labels)
  (let ((intersection (label-to-list (car labels)))
        (ctn (cdr labels)))
    (while ctn
      (setq intersection (intersection intersection
                                       (label-to-list (car ctn))
                                       :test ’equal))
      (setq ctn (cdr ctn)))
    (article-menu-listing (turn-list-into-propertized-list
Note 5.100 (On ‘display-difference-of-labels’).

For purposes of simplicity, this is set up to work with two labels only, for the time being.

(defun display-difference-of-labels (label-A label-B)
  (article-menu-listing (turn-list-into-propertized-list
                         (set-difference (label-to-list label-A)
                                         (label-to-list label-B)))))
Note 5.101 (Label derivatives).

It might be the case that someone would want to add a new label to the elements of an intersection or difference (as would be found by internal routines of ‘display-intersection-of-labels’ and ‘display-difference-of-labels’).

Note 5.102 (Display type in another column).

Probably it would be good to have another accessor that would show the article’s type in a second column. This could be especially useful in a context in which a section can contain both notes and subsections (just for example).

Note 5.103 (Display scholia about labels too, at least sometimes).

It would probably be good to be able to be able to compose and display scholia about labels, since they are, after all, also articles. Probably this should be optional, since there are times when we just want to see the list and not think of it as an article. And in addition, if we do plan to display scholia about labels only sometimes, we’ll have to decide what to do with the contents of the scholia display buffer at times when we just want to display the list and not its scholia. One trick might be to make the listing prefer to appear in the window wherein the scholia are displayed in such a case, if it exists; this way both windows wouldn’t confusingly appear at the same time. (Compare Note 4.11.)

5.5.7 Finding marked regions

Note 5.104 (On ‘find-marked-regions’).

This function is used in the subsequent section (Section 5.5.8) to locate the parts of the buffer that have scholia written about them. Note that this scheme might have been outmoded by stuff in Section 7.3.

(defun find-marked-regions ()
  (let (names-and-positions
        (next-change-point (point-min)))
    (while next-change-point
      (let ((next-region (find-next-marked-region)))
        (when next-region
          (setq names-and-positions
                (cons next-region
Note 5.105 (On ‘find-next-marked-region’).

This function is within the scope of ‘find-marked-regions’. Should be able to find all the regions associated with any scholium. Right now, this function is working in a simplified universe in which scholia and regions are mapped to each other in 1-1 way!

(defun find-next-marked-region ()
  (let* ((beg (next-single-property-change next-change-point
         (end (when beg
                (next-single-property-change beg
    (setq next-change-point end)
    (when end
       (get-text-property beg ’scholia)
       (list (cons beg end))))))

5.5.8 Turning overlays off and on

Note 5.106 (Overlays exercises).

As an exercise with text properties and overlays, here is some code for turning the overlays on and off in the Scholium Display and main article buffers.

Note 5.107 (Turning overlays off).

It is easy enough to turn overlays off; this is accomplished for the main article buffer and the Scholia Display buffer by ‘sch-turn-main-article-overlays-off’ and ‘sch-turn-scholia-overlays-off’, respectively.

(defun sch-turn-main-article-overlays-off ()
  (mapcar #’delete-overlay main-article-overlays)
  (setq main-article-overlays nil))

(defun sch-turn-scholia-overlays-off ()
  (mapcar #’delete-overlay scholia-overlays)
  (setq scholia-overlays nil))
Note 5.108 (Turning overlays on).

Turning overlays on is a bit trickier. In order to be able to turn scholia on, we need to be able to find all the regions that have scholia attached to them. This is accomplished (for the main article buffer only, I think) by ‘find-marked-regions’.

(defun sch-turn-main-article-overlays-on ()
  ;; to save from potential overlap weirdness
    (let ((names-and-positions (find-marked-regions)))
        (dolist (info names-and-positions)
    (let* ((name (car info))
           (marked-regions (cadr info))
           (scholium (get-article name))
           (beg (point)))
        (dolist (reg marked-regions)
          ;; add "transient" overlay
          (when use-crazy-font-lock
            (setq main-article-overlays
                   (make-overlay (car reg)
                                 (cdr reg)
            ;; using ‘scholium-face’ here is a bit weird
            (overlay-put (car main-article-overlays)
                         ’face (scholium-face))))))))))

(defun sch-turn-scholia-overlays-on ()
Note 5.109 (Using these functions as a subroutine for initial markup?).

It doesn’t make a whole lot of sense to use these functions for initial markup, because they involve a bit of search that would be unneeded at that stage. However, it would be good to take another look at ‘mark-things-up’ and see if we can get any further insights into how to write this function from there. (Presumably this function was initially based on that one already…)

6 Browsing

6.1 Scholia browsing

Note 6.1 (Introduction to scholia browsing).

The point of this section is to associate scholia with the regions of the article that they apply to, and to enable the user to quickly find and navigate between the marked-up regions in the displayed version of this article and their associated scholia.

Note 6.2 (Moving without reparsing).

The functions for scholia browsing are likely to move you to the “wrong place” unless the buffer has been parsed and the scholia locations are up-to-date (i.e., it will move you to the specified region, but this will probably be the wrong region).

Note 6.3 (Problem with the design of the ‘move-to…-scholium’ functions).

These probably shouldn’t be using overlays as the source of their information about where scholia reside, since overlays are not necessarily going to be there!

(I guess this means “if the content has been cut and pasted.” I’m guessing that there are cases where we would like to use this function under conditions in which cutting and pasting may have taken place. I’m not sure exactly what those cases would be, since I’m not sure exactly when this function is called. Seems it is, so far, only called in the case of scrolling the main article display to a certain region, namely a region with an attached scholium. It might be reasonable just to use overlays in that case (?). This is a somewhat curious point, and since it actually deals with something sort of fundamental in Emacs, maybe it would be best to give it more thought later, especially since this function is probably useful right now.)

They can be turned off altogether (Section 5.5.8) or one by one (Note 5.65).

(That is a good point. Audience, what do you think?)

Anyway, the logic of the function is probably going to be similar no matter what guts we use.

Note that there is a funny case of two references that begin at the same point (and, possibly, end at the same point as well). I think this case isn’t handled well by the function the way it is written.

This is a general matter that goes beyond these functions – and should probably be noted as such somewhere.

Note 6.4 (On ‘move-to-next-region-with-scholium’ and ‘move-to-previous-region-with-scholium’).

This moves the point to the beginning of the next region that has a scholium about it (if there is one).

(I’m noticing a bug when the function is used interactively with the binding selected in Section 11 and the cursor is positioned on a right paren; calling the function with M-x in this case doesn’t result in the same problem.)

Note these two functions are not quite symmetrical, because we want the cursor to end up at the beginning of the marked region. (I think there will be a problem if we try to go to the beginning of a scholium that is attached at the beginning of the article, but that isn’t such a big deal.)

(defun move-to-next-region-with-scholium ()
  (let ((change (next-overlay-change (point))))
    (if (overlays-at change)
        (progn (goto-char change)
               (list change (next-overlay-change change)))
      (if (overlays-at (next-overlay-change change))
          (progn (goto-char (next-overlay-change change))
                 (list (next-overlay-change change)
                       (next-overlay-change (next-overlay-change
        (message "No subsequent regions with scholia about them.")

(defun move-to-previous-region-with-scholium ()
  (let ((change (previous-overlay-change (point))))
    (if (overlays-at change)
        (progn (goto-char change)
               (list change (previous-overlay-change change)))
      (if (overlays-at (previous-overlay-change change))
          (progn (goto-char (previous-overlay-change change))
                 (list (previous-overlay-change change)
                       (previous-overlay-change (previous-overlay-change
        (message "No previous regions with scholia about them.")

(defun scroll-article-display-to-next-region-for-current-scholium ()
    (set-buffer buffer-associated-with-current-article)
    (let (found
          (curpoint (point)))
      (while (and (not found)
                  (not (eobp))
        (mapc (lambda (overlay)
                (when (equal (overlay-get overlay ’scholia)
                  (setq found t)))
              (overlays-at (point)))
        (if found
      (when (not found)
        (goto-char curpoint)
        (message "Scholium not about further regions in buffer.")))))

(defun scroll-article-display-to-previous-region-for-current-scholium ()
    (set-buffer buffer-associated-with-current-article)
    (let (found
          (curpoint (point)))
      (while (and (not found)
                  (not (bobp))
        (mapc (lambda (overlay)
                (when (equal (overlay-get overlay ’scholia)
                  (setq found t)))
              (overlays-at (point)))
        (if found
      (when (not found)
        (goto-char curpoint)
        (message "Scholium not about previous regions in buffer.")))))
Note 6.5 (Cycling through regions).

It wouldn’t be too hard to make a variant of ‘scroll-article-display-to-previous-region-for-current-scholium’ that cycles through the regions that the current scholium is about.

Note 6.6 (On ‘move-to-first-region-for-scholium’).

Move you to the beginning of the region marked up by the scholium named name. Should this be made interactive?) At present, it is only called by ‘move-to-first-region-for-current-scholium’.

(defun move-to-first-region-for-scholium (name)
  (pop-to-buffer (get-buffer rendering-target-buffer))
  (let ((beg (point-max))
        (about (scholium-about (get-article name))))
    (dolist (link about)
      ;; this should be revised in light of the
      ;; fact that a link can be multiply typed
      (when (and (typedata-includes-passage (link-type elt))
                 (equal (linked-to-article link)
                 (< (link-beginning link) beg))
        (setq beg (link-beginning link))))
    (unless (equal beg (point-max))
      (goto-char beg))))
Note 6.7 (On ‘move-to-first-region-for-current-scholium’).

This uses the function ‘move-to-first-region-for-scholium’ from section 6.1; the thought behind including the function here is that it establishes a relationship between the Scholia Display buffer and the main article buffer (however it could probably go in section 6.1 equally well).

This should probably be complemented by a function ‘move-to-last-region-for-current-scholium’.

Also, it should probably have some intelligent message (not to say “error message”) if the scholium applies to the article as a whole.

(Gives some error, complaining about ‘elt’ being void. Can this run in the Scholia Display buffer as well as the main article buffer?)

(defun move-to-first-region-for-current-scholium ()
  (let ((current (name-of-current-scholium)))
    (move-to-first-region-for-scholium current)))

6.2 Local browsing

Note 6.8 (Introduction to local browsing).

This section will describe purely local navigation commands that require little in the way of additional data structures. The main idea for the functions in this section is to make it so that focus can move from the current article to adjacent articles (either articles that are about the current article or articles that the current article is about), or more generally, to articles within some neighborhood of the current article (articles that we can reach by stepping from adjacent article to its adjacent articles, etc.). We re-use the listing mechanism from the previous section as needed.

Note 6.9 (On ‘read-scholia-property-at-point’).

Suppose we simply want to make the current scholium into the new current article. That’s what the next function is for.

(defun read-scholia-property-at-point ()
  (get-text-property (point) ’scholia))
Note 6.10 (On ‘scholia-named-at-point’).

I think that this should strip out the “mask” tags from the link-ids, but leave the name parts. Since it is only used by interactive functions, this seems fine, and appropriate.

(defun scholia-named-at-point ()
  (mapcar (lambda (id)
            (if (eq (car id) ’mask)
                (car (second id))
              (car id)))
Note 6.11 (On ‘follow-scholium’).

This causes the current scholium to become the current article.

Eventually we’ll want to be able to run this command with a mouse-click.

(defun follow-scholium ()
  (let ((current (name-of-current-scholium)))
    (when current
      (display-article current))))
Note 6.12 (On ‘follow-reference’).

This reads the ‘scholia’ property at point and either follows the reference at point (if there is only one) or allows the user to choose between references (if there are several).

We might want to provide an additional function for following links in general; basically the strategy for that is, just don’t do the ‘remove-if’ filtering.

(defun follow-reference ()
  (let ((references
         (remove-if (lambda (name)
                      (not (typedata-includes
                            (scholium-type (get-article name))
     ((equal (length references) 1)
      (let* ((ref (get-article (car references)))
             (to-article (reference-to-article ref)))
        (if (equal to-article name-of-current-article)
            (display-article (reference-from-article ref))
          (display-article to-article)))
      ;; maybe ‘display-article’ should be returning
      ;; some non-‘nil’ value so that we don’t have to do this.
      ;; this sort of disambiguation is really only needed if the
      ;; references have different targets.  Two distinct references
      ;; to the same thing overlaying each other could be treated as
      ;; one for simple following purposes.
      (list-articles references))
      (message "No reference at point.")

(defun follow-reference-or-scholium ()
  (unless (follow-reference)

(defun display-an-article-that-current-article-is-about ()
  (let ((abouts (scholium-about
                 (get-article name-of-current-article))))
     ((equal (length abouts) 1)
      (display-article (car abouts)))
      (list-articles abouts))
      (message "Article isn’t about any other articles.")))))
Note 6.13 (On ‘current-scholium-is-about’).

This function is similar to the previous one, but it applies to scholia. Since one presumably knows that the current scholium is about the current article, this is most useful when a scholium is about several different articles, as it allows the user to move “down” to any of them.

(defun current-scholium-is-about ()
  (scholium-about (get-article (name-of-current-scholium))))

(defun display-an-article-that-current-scholium-is-about ()
  (let ((abouts (current-scholium-is-about)))
     ((equal (length abouts) 1)
      (display-article (car abouts))
      (message "Note: scholium is only about current article."))
      (list-articles abouts))
      (message "Article isn’t about any other articles.")))))

6.3 Catalog browsing

Note 6.14 (Content-free browsing).

A popular style of browsing ignores the content of the objects themselves and looks only at their metadata. Examples include library catalogs or lists of search engine results. Since you aren’t actually browsing the articles directly, this is “browsing a catalog” more than it is “browsing the library”.

Using this technique, one should be able to selectively choose subcollections to look at, e.g. all the works by a given author or what have you. In this section, we provide code for browsing by article name. The code is an interface to a generic browsing/listing command based on Emacs’s built-in ‘list-buffers’ command. This setup is designed with extensibility in mind, however it may be useful to rewrite the selection mechanism (or add an alternate version) in order to make each line in the current display into a bibliography-style or index-card style reference. This would make the information fit on the screen better (and such modifications should be easy to make later).

See Note 6.24 for a description of the data that are displayed along with names in the listing. We may eventually want to be able to switch between several alternative data-listings, e.g., we might want name, type, and namespace; or name, article-about list, about-article list. With if complicated lists like this are used, we may want to have more than one line associated with each name.

We currently provide only very basic functions for working with these listings.

Note 6.15 (Catalog browsing in the scholium system).

This code provides a generic way to perform actions on rich objects from a list. It is assumed that each object has several attributes that a user may be interested in. “Name”, “size” and “mode” are familiar examples from ‘list-buffers’; here there can be an arbitrary number of arbitrary attributes. This makes this mechanism useful in some cases where ‘completing-read’ would not provide the user with enough information.

The way attributes are obtained from objects is left up to the user, as are the actual actions that can be performed on the objects.

Generality comes at an obvious, unavoidable, cost, namely any given deployment of this interface requires some extra set up – examples are provided.

Note 6.16 (Predicate support for listing).

Just for example, one thing people might want to do would be to list the articles in the vicinity of the article being browsed or in a certain section from a paper, and so on. The function will work with any set that we can select; what we need are a bunch of functions for doing the selection.

Note 6.17 (History support for listing).

It would be good to maintain a special history list for this display, that way if you browse a succession of nested catalogs, you can go backwards and forwards to choose the best catalog. (Catalogs are similar to directories from dired, gopher, etc.) See Section 6.4 for a discussion of history support in general (start with Note 6.55).

Note 6.18 (Purpose of generality).

Different menus may be useful for different purposes. A collection of article names may frequently be insufficient data from which to choose an article. Different data may be useful for different times and purposes, so we’ve made the system extensible. It may be useful to change the overall style of the display as well (e.g. to select from a list of multi-line records rather than a list of lines); modifications at this level are on hold until such time as they are needed.

Note 6.19 (On ‘make-generic-menu-mode’).

This takes the name of the menu (as a space-separated string) together with a list of bindings to be used in that particular menu mode.

This should provide a docstring for the mode it creates.

(defmacro make-generic-menu-mode
  (mode bindings)
  (let* ((modedash (downcase (replace-regexp-in-string " " "-" mode)))
         (modesymbol (intern (concat modedash "-mode")))
         (mapname (intern (concat modedash "-map"))))
       (defvar ,mapname)
       (setq ,mapname (make-keymap))
       (suppress-keymap ,mapname t)
       (dolist (binding ,bindings)
         (define-key ,mapname (car binding) (cdr binding)))
       (defun ,modesymbol ()
         (use-local-map ,mapname)
         (setq major-mode (quote ,modesymbol))
         (setq mode-name ,mode)
         (setq truncate-lines t)
         (setq buffer-read-only t)))))
Note 6.20 (On ‘generic-menu-noselect’).

The basic idea of this is that we have some objects and some functions to map across the objects to extract the information from the objects (accessors). The functions must be set up to produce strings as their output. The functions correspond to columns in the display; individual objects correspond to rows. This is the same idea no matter what the source of the objects is. (In particular, it might be a good idea for a later version of this function to accept either a hash table or a list as the source of the objects; see Note 6.45.) Note that columns of the display are assumed to be as wide as their widest item!

(defun generic-menu-noselect (objects accessors)
  (let (cols)
    (dolist (get-this accessors)
      ;; if we built this front-to-back rather than back-to-front,
      ;; that would be better
      (setq cols (cons
                  (cons (car get-this)
                        (mapcar (cdr get-this) objects))
    ;; find the width of the columns.
    (let ((lens (mapcar (lambda (col)
                          (let ((len 0))
                            (dolist (str col)
                              (let ((candidate (length str)))
                                (when (> candidate len)
                                  (setq len candidate))))
      (with-current-buffer (get-buffer-create "*Generic List*")
        (setq buffer-read-only nil)
        (while cols
          (goto-char (point-min))
          (goto-char (line-end-position))
          (dolist (str (car cols))
            (insert str " ")
            ;; fill with spaces to make up for lost space
            (insert-char 32 (- (car lens) (length str)))
            (unless (equal (forward-line) 0)
            (goto-char (line-end-position)))
          (setq cols (cdr cols))
          (setq lens (cdr lens)))
        (goto-char (point-min))
Note 6.21 (Improved sorting of article listing).

Sorting could be done using autocompletion on the name of the column to sort on, too, which would be kind of nice. Or we could make a command to sort on the current column, or reorder columns according to the values on the current line or just about anything you might like.

But I don’t think sorting is going to work at all until we have a consistent way of identifying the fields to sort; in the case of strings with spaces in them, ‘sort-fields’ won’t work. Since we compute the width of each of the columns in ‘generic-menu-noselect’, if we were to store this info (perhaps as a text property attached to each column heading), we could adroitly divide the text up to find the strings we’re trying to sort. See Note 6.33.

(defun Generic-menu-sort (col)
  (interactive "P")
    (sort-fields (or col 1) (progn (goto-line 2)
Note 6.22 (Hierarchies for browsing).

Hierarchies are somewhat similar to menus, e.g., we might want to use hierarchies to display sub-library relationships. (This is like KM’s ontology browser.) Of course, graph browsing would be even more generic (but display of these things seems a bit far removed from what we can do right now).

Note 6.23 (Reuse of names from article listing).

It would be nice if names were displayed together with quote marks making the printed versions of the names appropriate for direct re-use.

Note 6.24 (On ‘standard-article-menu-accessors’).

Note that it would be easy to provide more metadata – just revise this variable with additional fields as desired. See Note 6.20 for a description of how accessors work.

(defvar standard-article-menu-accessors
  ’(("Name" . identity)
    ("C"    . (lambda (elt) " "))))
Note 6.25 (Control panels).

A little control panel on the left of the article listing comprised of various toggle-state elements provides a way to interactively manipulate the articles individually or en masse (see section 7.5). (The only feature in the control panel for now is an indicator showing which article is current; this is indicated by a dot in the “C” column.)

Note 6.26 (On ‘article-menu-listing-hook’).

This is here to customize the behavior of the article menu listing. Currently it is used to offset the activities invoked by ‘currently-displayed-label-hook’; whereas we want ‘currently-displayed-label’ to be defined when the listing is used to display a label, we’d rather it be ‘nil’ when something other than a label has been displayed, since anything else could be misleading.

(defvar article-menu-listing-hook nil)
Note 6.27 (On ‘article-menu-listing’).

The optional input subset is a list of article names to pump into the generic menu; it defaults to the list of “plain” articles as recorded on the corresponding label. The optional input accessors specifies the functions to use to extract information from the articles named by subset; it is in the format of, and defaults to, ‘standard-article-menu-accessors’. (Note that function plays a similar role to ‘Buffer-menu-revert’ from buff-menu.el.)

(defun article-menu-listing (&optional subset accessors)
    ;; maybe this should always handle propertizing itself?
    (or subset
        (label-to-propertized-list ’plain))
    (or accessors
  ;; note, this runs every time, even if the current article
  ;; isn’t on the list.
  (run-hooks ’article-menu-listing-hook))

(add-hook ’article-menu-listing-hook (lambda ()
                                       (setq currently-displayed-label
Note 6.28 (On ‘turn-article-table-into-list’).

I hate to actually use this function the way it is used in ‘display-article’… probably we should just select from the plain articles there, and write a separate function to display other articles. In short, there really shouldn’t be any need to use this function, except maybe for debugging purposes or explicit listing of all the articles (whenever that really needs to be done).

(defun turn-article-table-into-list ()
  (let ((names (list t)))
    (maphash (lambda (name value)
               ;; It might be nice to have %S here, but
               ;; I don’t know if it would be _useful_
               (nconc names (list (format "%s" name))))
    (cdr names)))
Note 6.29 (On ‘turn-article-table-into-names’).

This is a variant of ‘turn-article-table-into-list’ that produces the actual names of the articles; probably it should supercede the other, since we could get the print names by mapping over the output of this function, obviously.

(defun turn-article-table-into-names ()
  (let ((names (list t)))
    (maphash (lambda (name value)
               (nconc names (list name)))
    (cdr names)))
Note 6.30 (On ‘turn-article-table-into-propertized-list’).

This combines the best of both ‘turn-article-table-into-list’ and ‘turn-article-table-into-names’. Compare ‘label-to-propertized-list’.

(defun turn-article-table-into-propertized-list ()
  (let ((names (list t)))
    (maphash (lambda (name value)
               (nconc names
                       (propertize (format "%s" name) ’name name))))
    (cdr names)))
Note 6.31 (On ‘turn-list-into-propertized-list’).

Turns an arbitrary list into a propertized list.

(defun turn-list-into-propertized-list (lis)
  (let (names)
    (mapc (lambda (name)
            (setq names (cons
                         (propertize (format "%s" name) ’name name)
Note 6.32 (Special-purpose listings).

Here are a few functions to list special collections of articles. The function ‘article-menu-list-labels’ is perhaps particularly noteworthy; browsing labels seems to be a powerful way of organizing and retrieving information, see Note 6.45.

(defun article-menu-list-plain-articles ()

(defun article-menu-list-all-articles ()
  (article-menu-listing (turn-article-table-into-propertized-list)))

(defun article-menu-list-metadata-articles ()
  (article-menu-listing (label-to-propertized-list ’metadata)))

(defun article-menu-list-labels ()
  (article-menu-listing (label-to-propertized-list ’label)))

(defun list-articles (lis)
  (article-menu-listing (turn-list-into-propertized-list lis)))
Note 6.33 (Using text properties in the article listing).

The approach that I’ve been using so far, searching for spaces, won’t work if the names of the columns have spaces in them. Something more sophisticated (probably using text properties) would work better in general. It would also be handy to tuck the articles “true names” in as text properties attached to the print names.

Note 6.34 (On ‘article-menu-display-article’).

The point is to grab the name of the article on the currrent line of the listing and display it. This needs to be checked a bit in the multicolumn case (which itself needs to be explored).

(defun article-menu-display-article ()
  (when (> (line-number-at-pos) 1)
      (goto-char (line-beginning-position))
      (search-forward-regexp "[. >] .")
      (setq name-of-current-article
            (get-text-property (point) ’name))
      (display-article name-of-current-article))))
Note 6.35 (On ‘article-menu-mark-article’).

Use this to mark the article mentioned on this line.

(defun article-menu-mark-article ()
  (setq buffer-read-only nil)
  (when (> (line-number-at-pos) 1)
    (goto-char (line-beginning-position))
    (delete-char 1)
    (insert ">")
    (when (< (line-number-at-pos)
             (progn (save-excursion (goto-char (point-max))
      (forward-line 1)))
  (setq buffer-read-only t))
Note 6.36 (On ‘article-menu-unmark-article’).

Use this to remove any mark on the article mentioned on this line.

(defun article-menu-unmark-article ()
  (setq buffer-read-only nil)
  (when (and (> (line-number-at-pos) 1)
             (not (save-excursion (goto-char (line-beginning-position))
                                  (looking-at "\\."))))
    (goto-char (line-beginning-position))
    (delete-char 1)
    (insert " ")
    (when (< (line-number-at-pos)
             (progn (save-excursion (goto-char (point-max))
      (forward-line 1)))
  (setq buffer-read-only t))

(defun article-menu-unmark-all-articles ()
  (setq buffer-read-only nil)
  (save-excursion (goto-line 2)
                  (goto-char (line-beginning-position))
                  (while (re-search-forward "^>" nil t)
                    (replace-match " ")))
  (setq buffer-read-only t))
Note 6.37 (On ‘article-menu-point-out-current-article’).

This is a non-interactive function that puts a dot in front of the current article when the article listing is generated.

(defun article-menu-point-out-current-article ()
  (goto-char (point-min))
  (setq buffer-read-only nil)
  (save-excursion (when (search-forward-regexp "^\\." nil t)
                    (replace-match " ")))
  (when (and
          ;; maybe this ‘(format "%s" name-of-current-article)’
          ;; stuff should be stored as some kind of function,
          ;; like ‘print-name-of-article’ or something like that
          (concat "^[. ] " (regexp-quote
                            (format "%s" name-of-current-article)))
          nil t))
    (replace-match (concat ". " (substring (match-string 0) 2))))
  (goto-char (line-beginning-position))
  (setq buffer-read-only t))
Note 6.38 (Features of the article menu).

The only “actionable” feature of the current listing is display. See also Note 6.25.

(make-generic-menu-mode "Article Menu"
                        ’(("g" . article-menu-listing)
                          ("m" . article-menu-mark-article)
                          ("u" . article-menu-unmark-article)
                          ("U" . article-menu-unmark-all-articles)
                          ("q" . quit-window)
                          ("\C-m" . article-menu-display-article)))
Note 6.39 (Additional actions).

We also want to have a function that will give the semantics for “select” and other sorts of things that we might be able to do to the objects that we’ve listed. (Maybe we should pass this in along with the objects, and set it up as a local variable that can be called later.)

Note 6.40 (Search engine interface).

It will be useful to interface this browsing mechanism with a simple predicate matching mechanism. Sometimes we’ll want to restrict search (Note 6.43), but for now, we might as well write a function that searches everything.

A more advanced search engine would apply general predicates to create the list (e.g. it would allow the user to restrict to scholia of a certain type, or that have scholia of a given type, or that match other computed properties).

Other basic advances would be to have a second column that showed the actual match, and to display the matches in order according to the position of the articles in question within some specific linearization of the document (e.g. the main one).

The main functions in the “search engine” so far are ‘article-menu-list-articles-matching-regexp’ (Note 6.41) and ‘article-menu-list-articles-matching-predicate’ (Note 6.42)

Note 6.41 (On ‘article-menu-list-articles-matching-regexp’).

This makes a listing showing all of the articles that match regexp.

(It would be nice if we could leave out meta, reference, code, and label types of articles in the default version of this.)

(defun article-menu-list-articles-matching-regexp (regexp)
  (interactive "MRegexp: ")
  (let ((matches (mapcar
                  (lambda (name)
                    (propertize (format "%s" name) ’name name))
                   (lambda (elt)
                        ;; rough, not bothering with
                        ;; ‘sch-plain-text’
                        (let ((article (get-article elt)))
                           (format "%s"
                                   (scholium-name (get-article elt)))
                           (format "%s"
                                   (scholium-text article)))
                          (goto-char (point-min))
                          (search-forward-regexp regexp
                                                 nil t)))))
    (if matches
        (article-menu-listing matches)
      (message "No hits."))))
Note 6.42 (On ‘article-menu-list-articles-matching-predicate’).

This is a way to pick out all of the articles in the library that match a given predicate.

(defun article-menu-list-articles-matching-predicate (pred)
  (let ((matches (mapcar
                  (lambda (name)
                    (propertize (format "%s" name) ’name name))
                  (remove-if (lambda (elt) (not (funcall pred elt)))
    (if matches
        (article-menu-listing matches)
      (message "No hits."))))
Note 6.43 (Restricting search).

If we have lots of subcollections, then we will sometimes need to search through all of them, and sometimes, only through some subset. When doing search, there are two efficient ways to limit things (and these are very similar): either add subcollections that match some predicate, or remove subcollections that match some predicate.

Note 6.44 (Saving search results).

It would be handy to be able to store the listings generated by search in some workspace. As a prior step, one needs to be able to turn the arbitrary listing into an article (specifically, a label)!

Note 6.45 (Browsing subcollections).

Compare Section 5.5.6 (should that stuff go here?). Also note that functionality for displaying sub-objects made out of hash tables should probably be worked in. Note that browsing from list to list is similar to the Gopher experience.

One special sort of subcollection to browse would be one generated by search. A variant of ‘multi-occur’ that resolves at the scholium level would be very handy to have (see Note 6.40).

Note 6.46 (Display all matches in full).

Instead of just providing the listing, sometimes it could be handy to display a concatenation of all matching texts (maybe with the matching words highlighted). Compare Note 8.1.

Note 6.47 (On ‘display-article-listing’).

This just puts the cursor in the article listing.

(defun display-article-listing ()
  (pop-to-buffer "*Generic List*"))
Note 6.48 (Arrangement of listing window).

The generic list should always go on the left in our display, and similarly with files that have their own buffers. This is especially relevant whenever we’re treating the listing as an article, cf. Note 5.98. It is possible, although perhaps unlikely, that sometimes we’ll want article listings to be mapped into the Main Article Display set to a Generic List Mode.

Note 6.49 (Static catalog display ).

It might be advantageous to not have the display disappear – which presumably means, display the articles that it lists as scholia attached to it; otherwise, they display and their scholia display, and there isn’t enough room on the display for the catalog. Something like this should be easy enough to do.

Note 6.50 (Electric catalog).

Probably we could write things so that sections display directly when the cursor moves over the relevant line in the catalog. This would make browsing entries very speedy.

Note 6.51 (Scholia attached to sections).

When we import the system (Note 12.23), an article is created for each section that is displayed as a listing. Such articles are currently displayed together with all of the articles in the section shown as attached scholia. It would probably be nice to be able to turn this “feature” off (also perhaps on a browsing-style-dependant basis).

6.4 Temporal browsing

Note 6.52 (History models).

I had the idea to make a generic browser that could be used as the backend for both the scholium system’s temporal browser and for my lynx-based browser nero. (It may prove to be most convenient to simply embed a new nero in the scholium system, indeed, this seems fairly likely to happen.) But I think it makes more sense at this point just to write a working scholia browser. That sort of abstractification can come later; the design will feature some effort in this direction, but perhaps largely in the form of notes about how to make things fully general and abstract.

What is a temporal browser? Typically, a list of history elements that can be added to when new pages are encountered, and from which elements are deleted as new paths through the network get trod.

For example, if the user browser pages A, B, and C in order, there will be a history list H=(ABC). Upon going back, the history list becomes H=(AB), and a future list is created, F=(C). If the user then visits a new page D, the history list becomes H=(ABD) and F=().

In order to make access quick, one obvious design strategy is to store pages that have been browsed locally (i.e., rather than re-downloading them each time they are browsed). Of course, if the pages are already stored locally, this issue is moot. And in this case, to save storage space (at a small time cost), we should be recording metadata (names or pointers) on the history list, not the pages themselves.

There are a couple of other models to consider.

First, we can store our history data in a tree. In this “istree” model, what we would see in the above scenario is:


I.e., each level of the tree stores the parent node as the car, and a list of children as the cdr (children are stored as atoms if they themselves are childless, or as the head of lists if they have children). To keep track of where we are in the tree, we need a list of directions (cars and cdrs). In the current case, the list would be (in the order of application): cdr, car, cdr, cdr. (For purposes of maintaining a symmetry which will be seen momentarily, we always return a list in which the current node is the car.) If the user moves back from D to its parent B, the supplementary list becomes cdr, car, in other words, we find the most recent parent (car) in the list. If the user moves back again from B, the supplementary list becomes nil, which means we are at the first node.

The data structure is simple, but confessedly, the explanation is a bit complicated! Another simple model keeps track of the full history. In this case, the model evolves as follows:


This model has the benefit of retaining data about loops (e.g. the short loop B, C, B). Loops can be lopped off by the following algorithm: starting at the end of the list (i.e. the current node), make a list of nodes that have been encountered, and trace backward. If the previous node hasn’t been encountered, put it at the beginning of our growing loop-free list. If it has been encountered, search backward for the earliest appearance of the last novel node, and add the element preceding that one to the loop-free list. Moving “back” in the sense of the truncated history list of traditional browsers can be accomplished by finding antecedent of the earliest appearance of the current node. E.g., if the user wants to move back, we add the antecedent B to the list, to obtain T=(ABCBDB). If the user wants to move back again, we obtain T=(ABCBDBA), since A is the earliest immediate prequel to B. Similarly, to move forward at this point, we find the latest immediate sequel to the current node; in this case, B again, and put T=(ABCBDBAB).

We can also recover the data stored in the istree model. This is left as an easy exercise to the reader, but the algorithm is given in the code; for reasons of generality (and simplicity!), we are adopting the full history model as the basis of this browser.

A variety of interesting novel browser actions can be performed using this information, e.g., finding all the pages you looked at before or after this one.

Note 6.53 (Science (fiction) tie-in).

Users can tap into the lived experience of a time lord (assuming we retain the “temporal” metaphor), or the experience of a Wolframian particle (compare NKS) if we consider a spatial metaphor instead. We’ll let you sort out the paradoxes. (Hint: you never step in the same river twice.)

Note 6.54 (Tabbed browsing).

Users of the full history model may be in less immediate need of “tabs” (in the form of multiple histories) than users of the truncated history model. Alternate browsing paths are (close to) natively available. Futhermore, a page can be marked as a divider between one “tab” and another. Indeed, this gives us an analogy to the “tabs” than stick out of file folders in a physical filing cabinet. (The only difference is that pages can be in multiple files at the same time, which is as it should be.) This means that multiple histories can be used (in a read-only mode) as virtual cabinets for organizing information. The actual system of cabinetry need not be maintained here; implementing extra browsing features is a good activity for some rainy day.

On the other hand, see Note 6.55.

Note 6.55 (Thematic histories).

It might be useful to keep a record of several different sorts of histories, e.g. of editing events or “catalog pages” (Note 6.17). These alternate histories wouldn’t have much to do with the temporal browser per se. Emacs does this in some cases (e.g. recording the input history for various interactive functions separately).

If we’re going to do this, probably the code in this section should be written in a somewhat more generic way.

(defvar sb-history nil)

(add-hook ’display-article-hook
          (lambda ()
            (sb-add-article-to-history name-of-current-article)))
Note 6.56 (Instead of adding the name).

Maybe we should be adding directions on how to get the article. The name might not be sufficient if we end up working with nested structures.

Note 6.57 (Complexities could be handled by ‘display-article’).

Of course, additional sorts of access instructions (like those alluded to in Note 6.56) could be handled by ‘display-article’. This section does not depend on article names being stored as strings.

(defun sb-add-article-to-history (article)
  (setq sb-history (nconc sb-history (list article))))

(defun sb-back ()
  (let ((current (car (last sb-history)))
        (n 0)
    (while (not found)
      (if (equal (nth n sb-history) current)
          (progn (setq found t)
                 (if (> n 0)
                     (display-article (nth (1- n) sb-history))
                   (message "Already at beginning of history.")))
        (setq n (1+ n))))))

(defun sb-forward ()
  (let* ((current (car (last sb-history)))
         (max (1- (length sb-history)))
         (n (1- max))
    (while (not found)
      (if (equal (nth n sb-history) current)
          (progn (setq found t)
                 (if (< n (1- max))
                     (display-article (nth (1+ n) sb-history))
                   (message "Already at end of future")))
        (setq n (1- n))))))
Note 6.58 (Keeping track of visited references).

It seems like we might want to keep track of information about visited references in some data structures maintained by the temporal browser. Alternatively, we could put this information into the metadata articles attached to the articles that have been visited (which would make it easy to report the number of visits to the article owners). See Note 5.32.

Note 6.59 (Function to go all the way back).

Should be able to go all the way back to the beginning of the history list in one go.

Note 6.60 (On ‘sb-previous’).

This gives the most recently browsed article besides the current one.

(defun sb-previous ()
  (car (last sb-history 2)))
Note 6.61 (Be careful to avoid cycles).

I’m not sure that the formulation of temporal browsing given in this section actually works! It needs a critical review to make sure that the future actually has an end, for example. I seem to be discovering problems by using display-only text (see Note 5.79).

6.5 Linear browsing

Note 6.62 (Browsing through the contents of a label).

In Note 2.60, the idea of being able to move forward through the document’s natural structure was discussed. One of the easiest ways to define this sort of structure is relative to the contents of some label. A more advanced mode would stitch together several labels, e.g., the sections and subsections of a document, with some simple logic for moving from label to label. In addition, a given article might appear as part of several labels, in which case, the user could be presented with possible directions to travel in. This is in some sense an extension of Section 6.2, since it gives more local directions to go in. It is also similar to the idea of following through a set of slides (and presumably would make it easy to implement such an arrangement).

The key thing seems to be identification of the “parent” label(s) to use for this purpose. Maybe this info is already available in the form of backlinks? Although, note that we might not always want to turn on this sort of label browsing for all labels all the time.

Note 6.63 (Disposition of code for linear browsing).

Some code for additional graphical features has been added to the scholia display by ‘scholia-display-extras’. It would be good to add some commands to do a semantically similar sort of navigation non-graphically, here. When the multi-parent browsing ideas from Note 5.78 have become fixed, they should of course be available here too.

Note 6.64 (On ‘add-visible-back-temporal-link’).

This makes a link to the previous-browsed page appear in the Scholia Display. It runs within the scope of ‘scholia-display-extras’ (Note 5.77).

It like we should potentially be able to change the strategy here, so as to add more information to the text properties and not actually create any corresponding article. I think this would require some adjustments to the way text properties are handled.

(defun add-visible-back-temporal-link ()
  ;; this should be marked up to become a fake link (one that isn’t
  ;; displayed in the main buffer)
  (when (sb-previous)
    (insert "\n\n"
            "Back (temporal): ")
    (let ((beg (point)))
      (insert (propertize
               (format "%s" (sb-previous)) ’face ’sch-reference-face))
      (scholium ’reference-to-previous-article
                ’(reference fake))
       ’(reference-to-previous-article 1 1)))))
Note 6.65 (On ‘add-visible-parent-and-sibling-links’).

This adds links to the parent and nearest siblings in the Scholia Display buffer. It runs within the scope of ‘scholia-display-extras’ (Note 5.77).

(defun add-visible-parent-and-sibling-links ()
  ;; identify the link to the parent, if it exists.
  ;; (this assumes that there is only one parent)
  (let ((link-to-parent (car (member-if (lambda (link)
                                          (member ’parent (cdr link)))
                                         (scholium-about article)))))
    (when link-to-parent
      (let* ((parent (get-article (first link-to-parent)))
              (scholium-text parent))
             (this-name-headed (member name-of-current-article
             (next (cadr this-name-headed))
             (prev (car (last (butlast parent-data
                                       (length this-name-headed))))))
        (when parent
          (insert "\n\n"
                  "Parent: ")
          (let ((beg (point)))
            (insert (propertize (format "%s" (scholium-name parent))
                                ’face ’sch-reference-face))
            (scholium ’reference-to-parent
                        (,(scholium-name parent)))
                      ’(reference fake))
             ’(reference-to-parent 1 1))))
        (when prev
          (insert "\n\n"
                  "Back (in parent): ")
          (let ((beg (point)))
            (insert (propertize (format "%s" prev) ’face ’sch-reference-face))
            (scholium ’reference-to-previous-article-in-parent
                      ’(reference fake))
             ’(reference-to-previous-article-in-parent 1 1))))
        (when next
          (insert "\n\n"
                  "Forward (in parent): ")
          (let ((beg (point)))
            (insert (propertize (format "%s" next)
            (scholium ’reference-to-next-article-in-parent
                      ’(reference fake))
             ’(reference-to-next-article-in-parent 1 1))))))))

(add-hook ’scholia-display-extras-hook
(add-hook ’scholia-display-extras-hook
Note 6.66 (On ‘forward-in-parent’).

A quick command to get the label identified as this object’s parent, and move to the next mentioned after this one in this label. (It would also be possible to follow the next reference in a page, i.e., use a page like a label; but this function doesn’t do that.)

It would be nice to have this function select the “next cousin” (or other suitable relative) if we are out of siblings.

We’ll also want a quick command to hop to an article’s parent.

(defun forward-in-parent ()
Note 6.67 (On ‘backward-in-parent’).

Like ‘forward-in-parent’ (Note 6.66) but moves backward instead.

(defun backward-in-parent ()

7 Editing and deleting

7.1 Initiating edits

Note 7.1 (How to initiate edits).

Buffers are frequently non-read-only; if these buffers have been marked up with scholia, then they can be edited directly. The Main Article Display buffer (for scholia with no other or better home) may as well be non-read-only as well. The point being that to start editing the text of an article, one just needs to begin. (In the current implementation, articles in the Scholia Display buffer need to be made current with ‘follow-scholium’, or by some other means, but that is easy enough.)

However, the other parts of an article (Note 3.7) are not immediately editable without some intermediate step to call them up and expose them to the user. Since bookkeeping data is maintained by the system itself, the only parts of articles that need special initial steps to edit are the name, about, and type fields.

Note 7.2 (Inserting text in the middle of a marked up region).

When someone inserts text into the middle of a marked up region, does the new text take on the markup properties of the surrounding text, or does it not, or can we make it an option, to possibly be exercised in different ways in different places? I assume we will want different behavior at different times. (Cf. Note 8.9). The reference to consider is the ‘‘Sticky Properties’’ node from the Elisp manual1515 15 (info "(elisp)Sticky Properties").

There they tell us that

By default, a text property is rear-sticky but not front-sticky; thus, the default is to inherit all the properties of the preceding character, and nothing from the following character.


If a character’s ‘rear-nonsticky’ property is ‘t’, then none of its properties are rear-sticky. If the ‘rear-nonsticky’ property is a list, properties are rear-sticky unless their names are in the list.

Thus, we have nothing to worry about in terms of front-stickiness, but we must maintain an appropriate rear-nonsticky list throughout the regions in which we want some of the properties to be non-sticky.

Something like this (enhanced, of course) will work:

(progn (put-text-property (line-beginning-position) (line-end-position)
       ’face ’italic)
       (put-text-property (line-beginning-position) (line-end-position)
       ’rear-nonsticky ’(face)))

7.2 Editing ‘about’ data

Note 7.3 (The joys and sorrows of editing ‘about’ data).

As noted in 3.68, changes to about data can have broad ramifications. In addition to masking any explicit links that point at this data, we will have to change everything that uses link-id’s.

In particular, backlinks corresponding to the set of edited links will have to change. I think that this part at least can be accomplished by running ‘scholium-modified-about-hook’ (cf. Note 3.23 and Note 3.61).

We’ll have to decide whether it is worth our trouble to change markup (it would seem to be inavoidable; however, as a quick work-around, perhaps other renderings can simply be blanked out when about data is being edited).

7.3 Finding revised ‘about’ data by parsing text properties

Note 7.4 (Mini-overview of parsing text properties).

This section concerns the topic of finding changes to about data of scholia that are induced by changes to the text of an article that they are about.

Note 7.5 (Overview of parsing text-properties).

By finding the places in a marked-up buffer where the ‘scholia’ property changes and examining its value before and after the change point, the beginning and end of each marked region can be found. (See the description of this property in Note 5.38.)

More specifically: (1) we must find all the places where the ‘scholia’ property changes; (2) then figure out what was new or what went away at each change point; (3) and update a list of beginnings and endings in the approximate form of an about list. This is a first step in the lengthier process of committing edits (see Section 7.4).

Note 7.6 (Indicator functions).

We can have marked regions that overlap to any depth; overlapping scholia can also have non-trivial non-overlapping regions.

In a mathematical abstraction, we might see scholia like χ[0,3], χ[1,10], χ[2,5]′′, χ[8,9]′′′ (as pictured in Figure 2). We could even have different regions from the same scholium overlap.

It would be nice to be able to indicate arbitrary subsets of the text that we’re presented with. The actual set of available indicators is a σ-algebra, if I remember correctly.

For LISP code, picking out arbitrary bits of list structure would be handy (see Note 8.6).

Note 7.7 (On ‘scholia-property-at-changepoints’).

This returns a list of pairs; points where the ‘scholia’ property changes paired with the value of the ‘scholia’ property at those points.

Figure 2: Markup simulated with indicator functions
(defun scholia-property-at-changepoints ()
  (let ((next-change-point (point-min))
    (while next-change-point
      (setq change-points (cons (list next-change-point
      (setq next-change-point (next-single-property-change+predicate
    (when (not (null (get-text-property
                      (caar change-points)
      (setq change-points (cons (point-max) change-points)))
    (reverse change-points)))
Note 7.8 (On ‘detect-scholia-structure’).

This function says which passages of the current article have been marked up with which scholia. Typically, this function will be run after editing, to recover the modified markup information and propagate it to attached scholia (see Section 7.4).

The way the function works is as follows. At each of the change points, we find the the annotations that have either just appeared or just disappeared, by using the ‘set-difference’ function to compare the value of the ‘scholia’ property at the upcoming change point with the value at the current one. We then sort the elements of this difference into “opening” and “closing” sets based on whether they are present at the upcoming change point, or the current one. In order to ensure that there is an adequate basis for comparison in all cases, exogenous points at “-” and “” are introduced, with no scholia attached to them.

Every “opening” element corresponds to some as-yet-unseen marked-up region; when encountering such an item, we add a term to the return value that looks like “(VALUE BEG)” where VALUE and BEG are, respectively

  • the value of the ‘scholia’ property – which will be the name of a scholium together with a unique link identifier, as described in Note 5.38;

  • and the point where this link starts to attach to the current article.

Similarly, every “closing” element corresponds to the end of some already-seen marked-up region. When encountering such elements, we modify the corresponding previously added term in the return value to read “(VALUE BEG END)”, END here being the point where the link in question stops attaching to the current document.

Note that since ‘scholia-property-at-changepoints’ gives us the values in increasing order, new “opening” items are encountered and added to the list in increasing order as well. This property is important for later.

(defun detect-scholia-structure ()
  (let ((begs-and-ends
         (cons ’(-infinity nil)
                (list ’(infinity nil)))))
    (while (> (length begs-and-ends) 1)
      (let ((difference (set-difference (second
                                         (first begs-and-ends))
                                         (second begs-and-ends))))
        (when difference
          (dolist (elt difference)
            (if (member elt (second (second begs-and-ends)))
                (setq opening (cons elt opening))
              (setq closing (cons elt closing))))
          (dolist (elt opening)
            (setq marked-regions
                   ‘(,elt ,(first (second begs-and-ends))))))
          (dolist (elt closing)
            (let ((found (member-if (lambda (pass)
                                      (equal (first pass) elt))
              (setcdr (cdr (car found))
                      (list (first (second begs-and-ends)))))))
        (setq begs-and-ends (cdr begs-and-ends))))
Note 7.9 (New scholia added while editing).

I don’t think that new scholia added while editing will need to be treated any differently during the reparsing process from scholia that were created before editing started. Scholium-creation should be a self-contained process that gets everything stored properly; the only special concern is that text properties would need to be added to the buffer that is being edited, but that’s obvious enough.

Note 7.10 (Finding changes in scholia about the whole article).

Presumably this data will be stored in a relevant metadata article, rather than in text properties (contrast Aaron’s picture). We probably won’t have to do reparsing to handle scholia like this, but we will have to do something.

Note 7.11 (Make ‘scholia’ a variable?).

It would be easy enough to make ‘scholia’ as used above into a variable instead of hard code, so that we could read from another text property instead. Probably we would want to complement this with similar changes to the code that adds text properties!

7.4 Committing edits

Note 7.12 (Themes for committing).

(1) Store stuff to metadata articles; (2) Prefer these for rendering purposes.

Note 7.13 (Committing implications).

Edits to any part of a scholium (in particular, to its name, text, about, or type data) need to be saved and otherwise processed. Sometimes changes in one article will result in changes to other articles, e.g., changing about data may lead to changes in backlinks; and changes to text may lead to changes in one or more attached scholium’s about data (Note 7.15).

Note 7.14 (Bookkeeping and version managing).

When articles have been modified, bookkeeping data needs to be updated. In particular, we are supposed save sufficient data to move back and forth between article versions. This is true of both text data and the other data that make up articles (name may be handled somewhat differently, but in general we need to store enough data to move between all versions of the article).

Note 7.15 (Setting updated ‘about’ information).

The idea of preferring metadata articles when rendering has been discussed in the section on metadata articles (Section 3.3.1 and Note 3.54 in particular), and is reflected in the way ‘mark-things-up’ is written (Note 5.54). For this all to work, it is necessary for ‘commit-edits’ to store updated about data when it runs.

Note 7.16 (On ‘store-link-masks’).

This function runs within the scope of ‘commit-edits’, after ‘adjusted-markup’ has been found by parsing (Note 7.21). Indeed, so far, this is the only significant default action of ‘commit-edits’ (cf. Note 7.15).

The function compares the elements of ‘current-markup’ (Note 5.42) to the elements of ‘adjusted-markup’. When changes are present in the markup data, ‘store-link-masks’ adds or changes masks.

More precisely, for each element of ‘current-markup’, we select from ‘adjusted-markup’ those elements with the same link-id (or maybe mask-id). These are the ‘matching-regions’ corresponding to that markup element. Unless there is only one matching region and this matching region happen to point to extend over exactly the range specified by the original link, we execute an operation on masks.

This operation is to swap in a replacement mask, if the markup element was coming from a mask already, or to create a brand new mask otherwise. If we’re doing a replacement, we look for the one mask that corresponds to the link-id borne by the current markup element. We then substitute all of the matching regions, for whatever was previously stored in this mask.

Otherwise, we just create a new mask with the appropriate link-id, containing the relevant regions.

(defun store-link-masks ()
  (dolist (link-ext current-markup)
    (let* ((matching-regions (remove-if
                              (lambda (markup-item)
                                (not (equal (car markup-item)
                                            (car link-ext))))
           (num-matching-regions (length matching-regions))
           (name-of-linking-article (if (eq (first
                                             (car link-ext)) ’mask)
                                        (first (second
                                                (first link-ext)))
                                      (first (first link-ext)))))
      (unless (and (eq num-matching-regions 1)
                   (equal (cdr (first matching-regions))
                          (cdr link-ext)))
        (if (eq (caar link-ext) ’mask)
            ;; swap in a different mask
            (let* ((masks
                    (get-metadata-field ’masks
                    (member-if (lambda (item)
                                 (equal (second (car link-ext))
                                        (car item)))
              (setcdr (car current-mask-headed)
                      (mapcar (lambda (ided-reg)
                                (cdr ided-reg))
              (put-metadata-field ’masks
          ;; create a new mask
          (let ((masks (or (get-metadata-field
                           (list ’masks))))
               (cdr masks)
               ‘(,(car link-ext)
                 ,@(mapcar (lambda (ided-reg) (cdr ided-reg))
Note 7.17 (Adding new markup to rendered articles).

Adding new markup to rendered articles will introduce a discrepancy between ‘adjusted-markup’ and ‘current-markup’, i.e., items will be found that weren’t there before. This shouldn’t, be a problem, because ‘store-link-masks’ iterates through the members of ‘current-markup’ looking for matches. So, anything completely new in ‘adjusted-markup’ will be ignored by the ‘store-link-masks’ phase of the commit process. This is relevant to Section 8, and any other code that modifies the markup of rendered articles.

Note 7.18 (On ‘commit-edits-hook’).

This is the single point of customization for the commit function, to be used in future sections. It runs within the scope of ‘commit-edits’, after ‘adjusted-markup’ has been found by parsing. Thus, functions that are added to this hook can take advantage of this knowledge about the current state of markup.

(defvar commit-edits-hook nil)
Note 7.19 (On ‘after-committing-edits-hook’).

This facilitates additional actions after the main committing routines are finished.

(defvar after-committing-edits-hook nil)
Note 7.20 (Summary of idea behind ‘commit-edits’).

When you edit the buffer, the text properties showing where scholia are attached move around. When you run ‘commit-edits’, metadata indicating the new position of these regions is stored; it will be used subsequently so that the correct regions are highlighted. (These suggestions coming can later be incorporated into the scholia by their authors.) Furthermore, ‘commit-edits’ stores a version of the article for subsequent browsing.

Note 7.21 (On ‘commit-edits’).

Anything run by the ‘commit-edits-hook’ that changes the text of the article being committed should store the desired value on ‘adjusted-text’. It is going to take a bit of thinking about how to make several functions (particularly functions having to do with derivatives) run properly, in serial. (But I’m pretty sure it can be done.)

Note that when committing we currently make sure that exactly one newline is attached to the end of the page when the text is saved. This has a beneficial effect on exporting the scholium system (see 12.38).

We may at some point want one or more commit functions that will store more different kinds of edits from more different places. For example, we’ll want to be able to commit edits that take place in the Scholia Display buffer. Similarly for edits to ‘about’ data, and so forth (see Note 7.2).

(defun commit-edits ()
  (let* ((adjusted-markup (copy-tree (detect-scholia-structure)))
         (old-contents (get-article name-of-current-article))
         (adjusted-text (buffer-substring-no-properties (point-min)
         (old-text (scholium-text old-contents)))
    (run-hooks ’commit-edits-hook)
    ;; here is where the criterion for checking that some change has
    ;; actually been made would be inserted
    (when (not (equal adjusted-text old-text))
      (scholium name-of-current-article
                (with-temp-buffer (insert adjusted-text)
                                  (goto-char (point-max))
                                  (insert "\n")
                ;; of course, for more advanced versions of
                ;; this code, these won’t be static
                (scholium-about old-contents)
                (scholium-type old-contents)
                ;; this should be reformatted to store the old
                ;; version!
                (scholium-bookkeeping old-contents)))
    (run-hooks ’after-committing-edits-hook)))
Note 7.22 (After committing, redisplay article).

Hopefully this won’t cause any problems. The one concerning point I can think of might come up when committing fully rendered and identified compilations (Section 8.7), but we haven’t worked out the major details of that yet, so there’s no reason to worry about this one minor one.

(add-hook ’after-committing-edits-hook ’redisplay-article)
Note 7.23 (Shortcomings of the basic commit implementation).

Old comments should continue to apply verbatim to the old version of the document – and also, tentatively, to the new version. Maybe masks should contain information about which version of the article they apply to. A given link could have a collection of masks, each relevant to a different version of the article being linked to (perhaps recycling content when the masks are known to be the same).

And, of course, commiting is going to have to store a version of the article on the bookkeeping list (or equivalent; see Note 2.45). We will then want support for version browsing – hopefully the browsing functions can be made to “just work” on data coming in with an explicit version number.

And, in addition, we should be updating backlinks as needed.

(These items should be assessed to see how critical they are; it seems to me that we can probably work with things roughly as they are, if we aren’t doing anything too wild. It would be interesting to document the process of coming to rely more heavily on the system as time goes by.)

Another nice thing would be to make sure that there is a newline at the end of the committed text.

Note 7.24 (Assigning masks after reparsing).

We send a message to each of the scholia that have been redirected after editing has taken place (cf. Note 5.47). In order to do this, we have to compare the new markup data with the old markup data to figure out what will change. We may want to keep a buffer-local record of the original markup data in order to figure out precisely what has changed. (I think for now we just assume that there is one rendered article at a time, so we only keep track of one set of old markup data. But don’t quote me on that.)

Alternatively, we could mask everything, regardless of whether or not this data has actually changed, but this seems rather silly, and potentially very inefficient besides. So we don’t do that.

Note 7.25 (Similarity of deleting and committing).

Note that deleting is probably going to be similar to committing, in terms of the way changes to about data are reported and stored. (See Section 7.6.)

Note 7.26 (Where to store committed edits).

One thing I’m somewhat curious about is whether it is ever safe to assume that we can make changes to an attached scholium directly, without waiting for its owner’s input. It seems more likely that we would want to put the updated information on that scholium’s metadata article, which is its “world-writable” space (see Note 3.46). If we do this, that means that the routine we use to find a scholium’s about data when rendering a given article may have to look at the scholium’s metadata article. (The case in which the same person owns both articles might be one case in which we would wish to make changes directly.)

Note 7.27 (The role of metadata).

If we assume that changes aren’t made directly, then the owner of the scholium whose metadata article has been changed will have the opportunity to accept or decline the suggested changes. Note that if the author doesn’t accept the changes, then the rest of the world may still have the option to use about data from the metadata article instead of from the article itself. Also note that even if a commentator retracts remarks for some updated version of the article, they may continue to apply to earlier versions.

When a commentator accepts changes, most likely what will happen is that a new version of the scholium will be created, and the specific suggestions removed from its metadata article. However, if the suggestion is declined, most likely the suggestions will stay on the metadata article. (This situation seems a bit backwards.) Alternatively, if the suggestions are declined, the author might just delete them (if we make it so that anyone can make changes to the metadata article – which might be reasonable).

For more on this topic, see Section 7.7.

Note 7.28 (The possibility for conflicts in backpropagated ‘about’ data).

The potential for conflicts in suggested changes coming from edits to various different articles exists, in theory, but I think it is dispatched by the fact that each region is drawn from exactly one article. So, as long as each link is kept up-to-date with respect to any changes in the one article that it applies to, there won’t be any conflict.

Note 7.29 (Conflicts in a distributed environment).

Of course it is the case that if the same article is being changed or even just talked about in several different places, or if it is a scholium that applies to several articles that reside in different repositories, many different sorts of conflicts could arise. The task of assimilating changes from various locations should be given at least partial support in code.

Note 7.30 (Committing and reverting).

Note that “revert” is in some weak sense an opposite action to “commit”. (See also Note 7.25.)

7.5 Editing en masse

Note 7.31 (Enhanced control panel).

It might be good to add another column to the article listing control panel. Any changes to the article listing that take place in this section should be done somewhat delicately and artistically. Presumably this implies that the list of column-fields should be stored as a variable, so that it can be easily modified. Note that functions for mapping over lists of articles forms the theoretical basis of this section (see Note 12.21).

Note 7.32 (Adding metadata en masse).

It isn’t quite “editing”, but equally important would be a way to apply a given scholium to a set of articles (both severally and not severally), or add type data to a set of articles that you own, or whatever.

Note 7.33 (On ‘label-marked-articles’).

Apply the specified label to each marked article. New labels can be applied no problem.

(defun label-marked-articles (label)
  (interactive (list
                (intern (let ((completion-ignore-case t))
                           "Label: "
                           (label-to-list ’label))))))
  (let (article-names)
      (goto-line 2)
      (goto-char (line-beginning-position))
      (while (re-search-forward "^> ." nil t)
        (setq article-names
              (append article-names
                       (get-text-property (point) ’name))))))
    (dolist (name article-names)
      (label-article name label))))

7.6 Deletion

Note 7.34 (Deleting articles).

It is sometimes desirable to delete articles that are no longer worth looking at. We want to give the user this power, though in some cases it would be nice if the article wasn’t really deleted and was instead just depreciated.

There is a somewhat complex operation that should be executed when an article is actual deleted (and/or, conversely, a somewhat complex system we should put in place to deal, at render time, with articles that have been deleted without sufficient follow-through). Some of the steps in the deletion process should be: (1) deleting backlinks; (2) unindexing the article from whatever labels the article has been indexed in.

But right now, we just delete.

Note 7.35 (Steps to be taken when deleting an article).

If the article is a scholium attached to the current article, we should remove the associated text properties and overlays.

Since in theory we can’t modify the about data that point at the deleted article, we might want to broadcast a signal to these articles that say that their referent has been deleted. One way to do this would be to replace the article with a “Deleted Article” stub, and probably add it to a “deleted” label.

Of course, as with other issues having to do with propagation things are somewhat different if all the articles are owned by the same person (or otherwise have conventions associated with them whereby they can be kept in sync without problems).

Note 7.36 (Set current article to ‘nil’).

If the current article is deleted, then the ‘name-of-current-article’ variable should probably be set to nil. (Either that, or something from the temporal browser should be used to set the variable to the last-browsed page.)

(defvar delete-article-hook nil)

(defun delete-article (name)
  (interactive (list (read-article-name)))
  (let ((article (get-article name)))
    (if article
          (remhash name article-table)
          (when (equal name name-of-current-article)
            (setq name-of-current-article nil))
          (run-hooks ’delete-article-hook))
      (error "No article by that name found"))))
Note 7.37 (Deal with effects on temporal browser).

It is a little weird to revised history to make it look like deleted pages never existed. Seriously Orwellian. For now, that’s exactly what we do.

Note 7.38 (On ‘remove-appearances-in-history’).

This function will run inside the scope of ‘delete-article’.

(defun remove-appearances-in-history ()
  (setq sb-history (delete name sb-history)))

(add-hook ’delete-article-hook ’remove-appearances-in-history)
Note 7.39 (Deal with effects on articles that deleted article was attached to).

Presumably articles that the deleted article linked to will have to have their backlinks adjusted. That is to say, it would be somewhat odd to have backlinks to an article that doesn’t exist.

Note 7.40 (On ‘delete-scholium-associated-with-current-marked-region’).

Provides a quick way to delete the scholium attached to region containing point. If more than one scholium is so attached, we bring up a menu to select items for deletion.

This function assumes that the ‘scholia’ property is simply comprised of a list of the names of attached scholia; I’m not so sure that this is an accurate assumption (if not, then we should be able to easily extract such a list from the actual ‘scholia’ property).

This is just one of several functions that would benefit from the addition of progressive markup modifications.

(defun delete-scholium-associated-with-current-marked-region ()
  (let ((linked-scholia (scholia-named-at-point)))
     ((eq (length linked-scholia) 0)
      (message "No scholium attached here."))
     ((eq (length linked-scholia) 1)
      (delete-article (car linked-scholia)))
      (list-articles linked-scholia)
       "Inspect articles and select items for deletion with \"d\".")))))
Note 7.41 (Deletion in various strengths).

Probably there should be several different kinds of deletion (like in X windows, there is kill, destroy, minimize, wipe out to the fifth generation, etc.; also, compare the several different “strengths” derivative articles come in).

7.7 Responding to incoming metadata

Note 7.42 (How to deal with metadata).

This section gives support for article owners dealing with incoming metadata coming from editing events. An important example: scholia may see suggested changes to their about data when the article(s) that they are about change). Authors might want to incorporate such incoming changes into their document, or obliterate them, or revise them. Perhaps a scholium author will want their comments to apply to the version of the article it commented on, but not to the revised version. Or perhaps a scholium author would like to drop their content all together (to the extent that this is possible). We’ll do what we can to accomodate these various actions in this section.

7.8 Editing labels

Note 7.43 (To edit labels).

If you’re going to edit a label, you’d like to do things like insert new articles, delete old articles, rearrange the order of entries, and so on. Preliminary support for this sort of thing is given in this section.

Note 7.44 (On ‘article-menu-insert-new-article’).

This function lets you add new articles to a recently browsed label.

For now, we assume that we really are adding a new article (and not some existing one); and that this new article will take the quotidian form offered by ‘make-scholium’ (Note 4.10). These assumptions may be relaxed later.

The function one of ‘label-article’ and ‘label-article-insert-before’ to add the label to the new article (Note 3.74, Note 3.75).

(When creating this new article, I hope that enough ancillary metadata , type and so on, is added to allow it to be recognized as part of the relevant hierarchy later.)

(defun article-menu-insert-new-article ()
  (if currently-displayed-label
      (let ((new-entry (read-string "New entry: "))
            (old-entry (save-excursion
                         (goto-char (line-beginning-position))
                         (search-forward-regexp "[. >] ." nil t)
                         (setq name-of-current-article
                               (get-text-property (point) ’name)))))
        (setq new-scholium-name new-entry)
        (let ((line (line-number-at-pos)))
          (if old-entry
               new-entry old-entry currently-displayed-label)
            (label-article new-entry currently-displayed-label))
          (display-label currently-displayed-label)
          (goto-line line))
    (message "Listing doesn’t represent a label.")))

(define-key article-menu-map "i" ’article-menu-insert-new-article)
Note 7.45 (Committing edits to labels).

So far, the editing steps we do with labels are automatically entered into the backend, so, at this point, we don’t need a ‘commit-label’. If it turns out that we want one, then these routines will have to be rewritten somewhat, and ‘commit-edits’ will have to have ‘commit-label’ hooked into it.

7.9 Further notes on editing and deleting

Note 7.46 (Reparsing and propagating after editing).

Editing an article is one of the main reasons we’d ever need to do reparsing. When text moves around, scholia are likely to be attached to different regions. This means we need to change the about elements of attached articles for the collection to stay coherent. Proper reparsing behavior is necessary for various functions to behave properly; see for example Note 6.2. We probably will want to add the appropriate reparsing function to the corresponding after-editing/before-saving-changes hook.

Note 7.47 (Changing names).

One simple “edit” would be changing the name of an article; doing this of course requires the new name to replace the old name in every reference to the article.

Note 7.48 (Changes to an article affect attached articles).

However, changes to the text of a given article may affect any scholia that are about that article. (To say nothing of how the semantic contents of the change might affect these attached scholia, the regions may be wrong.) One approach to take would be to attempt to propagate the changed buffer positions to all attached scholia. Of course, we can’t necessarily assume that the owner of the attached scholia will accept these propagated changes.

Note 7.49 (We shall adopt a versioned model).

These considerations suggest that we should go with the “versioned” diachronic model. According to this model, changes can be propagated, but on a tentative basis. If another user doesn’t accept your changes, they will still have a perfectly good scholium attached to a previous version of your article.

Note 7.50 (Moving between versions).

It would probably be handy to have this use existing browser code, e.g. from the local browsing section (Section 6.2).

Note 7.51 (Different ownership models can do different things).

When permissions aren’t a concern, we can set the system up so that propagated changes are automatically accepted. In addition, we can set up versioning to resolve at different levels of detail depending on the application.

Note 7.52 (Changes outside the system).

One further consideration is the fact that articles may change outside of the scholium system (e.g. a file may be edited, a remotely-hosted web page may change, or a process may be killed). We assume that when a change like this is noticed, the scholium system will create a new version of the article automatically and propagate appropriate changes to other documents.

Note 7.53 (How to do automatic updates).

Noticing changes to files is a bit of a challenge; not insurmountable, but it does mean that we’ll need an internal representation of the file’s contents (i.e. one that we assume is not subject to change in the same way that the file is!). As it stands, the system doesn’t include this sort of version management apparatus. In order to mend things, whenever an article is created (or edited), a backup copy should be stored that can serve as a reference. At least, I guess this is how version management software works. Presumably it isn’t hard to add this sort of feature, but before plunging ahead, I think it would be worth looking into the algorithms and protocols used by extant version management tools.

Note 7.54 (When should auto-accept work?).

Although auto-accept is a nice-sounding thing, it may be best to always have changes approved by a person, since changes to the text could completely change its meaning – and threaten to make attached scholia ridiculous. Maybe auto-accept is most useful when one person is editing the document, but even then, there may be times when content changes so much that the attached scholium would become ridiculous or meaningless. In this case, maybe auto-accept is good, but the user should be able to roll back propagated changes easily?

Keeping track of the genesis of a given scholium is important. Maybe this procedural info will be marked up itself sometimes? Certainly there can be extra user-supplied info, outside of the bookkeeping info, that describes relationships between documents.

Note 7.55 (Changes to metadata in a semantically programmed environment).

We assume that changing type and about data also require changing the article version.

Note 7.56 (Using bookkeeping information to track versions).

As if it wasn’t obvious, the bookkeeping information will be used to keep track of different article versions. (Presumably including diffs between versions or what have you: it would be good to hook the GNU diff tools into this system. See also the comment about storing the “current” version of articles in full, to check against changes from the outside.)

8 Derivative works

8.1 Introduction to clusions

Note 8.1 (Derivative works in the scholium system).

Section 5 focuses on providing useful views into the document. The matter of assembling new articles which derive from other articles drawn from the document is essentially a separate problem; we will tackle this problem in this section. (Other sorts of derivative or derivative-like works, such as collections and indices are given their primary treatment elsewhere.)

Note 8.2 (Different kinds of derivatives).

Inclusion is probably the most straightforward way to assemble an an article that derives from another article. Material from a certain version of the source article is simply inserted into the current article. Inclusion is treated in Section 8.3. Transclusion is similar to inclusion, except that instead of simply inserting source text as it is right now, transclusion causes the source to be tracked as it changes, and its most up-to-date version to appear automatically in the derivative work when it’s rendered. Transclusion is treated in Section 8.2. Identification is like a two-way version of transclusion: instead of the source material being inherited by the derivative article, under identification, the source is shared between several articles. Thus, changes in one place will be mirror in each of the other places. This is developed in Section 8.4.

Note 8.3 (Clusions).

We refer to specific instructions specifying an inclusion, transclusion, or identification as a clusion. (By analogy to the morphisms of category theory.) In this lingo, we could also refer to identification as biclusion.

Both identification and transclusion are represented as links inside the text field of an article, and this property is definitional for clusions.

Identification, on the other hand, is currently implemented as a live, one-time operation, and so it isn’t strictly an example. However, once we have a proper versioning system working, it would be fairly easy to do inclusion as a proper clusion like the others (by making it point at a specific version of a document from which to draw text) – assuming that is what is desired. It may be that the user wants to simply quote from the other document, and doesn’t want to do clusion maintenance. We’ll have to give some thought to what way of operating is most desirable here.

I haven’t thought of any other major classes of clusions (at least, not yet), but just to point out that any clusion can be composed with straightforward text operations (e.g. reversing the characters in a cluded region) to obtain another clusion.

Mainatining clusions is part of a broader problem of maintaining consistency conditions, see Section 3.4.

Note 8.4 (Allowable sources).

Compare Note 3.32, which talks about the different allowable kinds of text, but note that, one should be able to clude things other than text. The set of objects that can be cluded is same as the set of objects that a scholium can be about, i.e., anything a link can point at.

Note 8.5 (Derivatives and versioning).

In order to get the relevant functions for building derivatives set up properly, we really should have version management in place.

Note 8.6 (List processing as a prototypical example).

The “,@” from LISP comes to mind when thinking about transcluding, whereas shared list structure is like identification. It would be interesting to spell the relationship out in detail.

One further thought on the subject is that we may sometimes want to transclude symbols and s-exps of LISP code – which brings the discussion full circle pretty quickly.

Note 8.7 (Boundaryless formulations).

The simplest cases to deal with appear to be those in which no boundary is mentioned. For example, cluding the whole of some article A into another article B. In practice, sometimes we can stick to a boundaryless treatment; other times, boundaries are unavoidable.

Note 8.8 (Clusion algorithms).

It would probably be good to put an overview of things like 8.21 here, so that the subsequent discussion of markup makes sense. (We could talk about the easy work-around for building this document, and contrast it with the more intensive operations that go on in the *Compilation* buffer (cf. Note 8.53).

Note 8.9 (Splitting text properties).

Text inserted in the middle of a quotation does not derive from the same source as the quoted text. Therefore, it typically should not take on the ‘derives-from’ markup of the surrounding text. For a discussion of inserting within a marked up region in general (i.e., not just in the cases having to do with derivatives), see Note 7.2.

Note 8.10 (Splitting clusions).

Notice that edits in either the source or the recipient article can cause a clusion to split, as illustrated in Figure 3.

xxabcd yyabcd
efgxxx efgyyy

A (×2) B
xxabcd yyabcd
xxxefg efgyyy

A (×2) B
xxabcd yyabcd
efgxxx yyyefg
Figure 3: Edits in either A or B can cause a clusion to split
Note 8.11 (Masking clusions).

We want to mask clusions in much the same way we mask links (Note 5.46). Indeed, clusions are essentially links, by definition (Note 8.3).

Note 8.12 (Format of text with derivatives).

The format of the text field for articles with derivatives is as follows:

(twd <string | clusion> ...)

Each clusion is a link with the corresponding instruction consed onto the front of it (cf. Note 3.11). The three instructions are ‘transclude’, ‘identify’, and ‘include’.

For example,

(twd "This Note can be used as part of an example.\n"
     (transclude "Format of text with derivatives"))


(twd "Part of this Note can be used as part of an example.\n"
     (transclude "Format of text with derivatives"
                 (passage 1 25)))

There is a small break in transparency here; presumably this won’t be a problem (Note 2.16).

Note 8.13 (Transclusion and freedom).

The right to make and circulate derivative works is important (i.e., when it is allowed!). One easy sort of derivative is a compilation of all of the sub-objects that have a given property. For example, one could put together a collection of all the first paragraphs drawn from the various sections of an article, or all the paragraphs that have been marked as “needs revision”, or “theoretically interesting”, etc. Transclusion makes it easy to build simple, useful, derivatives of this sort, and keep them up to date as the master document changes. (Sometimes one would want to use somewhat special instructions, e.g. in order to collect only the new items matching an otherwise-global search criterion.)

Note 8.14 (The dependence of the various derivatives on ‘commit-edits’).

I think we may have to do a commit after running ‘transclude-article’, since it involves changing the text field. By contrast, running ‘include-article’ causes new scholia to be stored automatically, but we left committing the text field to the user. So, I’m not sure; if there is a way to do transclusion without requiring a commit automatically, that might be kind of nice.

Note also that each of the different derivative styles will have things to look for whenever they commit again. (E.g. articles with inclusions should be inspected to see if perhaps all of the included text has been deleted.)

Note 8.15 (Derivatives outside of the scholium system).

We leave it up to the user to write a suitable function for quoting text from scholium system objects to non-scholium system objects (if the user has any of these!).

Note 8.16 (Technical difficulties with text properties).

The fundamental difficulty with text properties seems to be that using them comes at a non-trivial cost. So we don’t use them for everything (cf. Note 8.50).

When doing transclusion, rather than using a uniform text property, we need a text property that gives a unique ID to each character within the marked up region (Figure 4, Note 8.19).

I’ve been assuming that typical text properties designating the association of a region with a scholium will be “sticky” – if someone adds to a marked-up region, the added text will bear the new markup. At some point, we might have a non-sticky option. However, the text properties associated with transclusions should always be non-sticky, since text added in the middle of a quoted passage needs to be treated as an “island” in the middle of the transclusion (which actually gets split in two).

The problem arises in the fact that Emacs makes it hard and/or ugly to use an individual text property for each transclusion (see Note 5.40). The relevant experiment took place in CVS versions 41 through 43; version 44 was then identical to version 41.

The point is, that I’ve been maintaining all of the data about scholia on precisely one text property, which of course takes on different values as needed. However, the “stickiness” of each text property is controlled individually, and I can’t continue to use this trick.

I could, I suppose, use one sticky and one non-sticky text property, but I think that there may be a better approach involving some work on the text property system itself. This will need some more investigation.

Note 8.17 (Treating clusions).

The following subsections provide commands for building clusions interactively, and for rendering them properly.

8.2 Transclusion

Note 8.18 (On ‘transclude-article’).

First we check to see whether the article contains transclusions already, and if not, we put the text field into the format used for text with transclusions (see Note 8.12).

Next, we divide the text of the article in two; everything prior to point, and everything after point. (We have to divide the rendered text, fairly clearly; since there might be more than one different rendering, we hope that the different possible divisions are consistent, or at least that we know how to do all of the divisions.)

Of course, we also need to be able to specify the objects that are to be transcluded!

;; totally not working yet!
(defun transclude-article (article)
  (interactive (list (read-article-name)))
  (let ((current (get-article name-of-current-article))
    ;; it really doesn’t make sense to use something like this here --
    ;; we only deal with the internal format at commit time.
    (unless (and (listp current)
                 (eq (car current) ’twd))
           (setq newtext ‘(twd
                           ,(scholium-text current))))
    (let ((beg (point)))
      (insert (sch-plain-text (get-article article)))
      ;; this condition isn’t really strong enough
      (when name-of-current-article
         ‘(derives-from ,name-of-current-article ,article ,(genref))
         ‘((passage (,name-of-current-article
Note 8.19 (Transclusion markup design).

The idea is that when some object T is transcluded into the current buffer, im(T) is marked up in the following way. Character i in this image is given a text property that says “This is character i of n contiguous characters that have been transcluded from the object T.” Then, when parsing, if these text properties show up in new places (e.g. non-contiguously) or if some of the numbers fail to show up at all, we’ll know to initiate new transclusions or to change the definition of the current one being scanned.

The reason for doing things this way is similar to the issue discussed in the ‘‘Not Intervals’’ node of the Elisp manual1616 16 (info "(elisp)Not Intervals"). If the image of a transclusion was simply marked up with a property that said “This is the image of the object T under transclusion,” then elements of the region that bears this text property could be rearranged without anything registering at the system level (Figure 4). This would pretty much be a disaster.

The idea here is that we should be able to edit texts coming from transclusion and continue to have the transclusion basically work. In Note 8.9, we decided that text inserted into a quotation should not take on the “derives-from” property that the surrounding text bears. Thus, if a transcluded region is broken up, or pieces of it are deleted, we assume that new transclusions are being requested by the user.

These remarks apply to the representation of transclusions in rendered documents; the internal representation is discussed in Note 8.12.

-----# -----#
----## -#--##
---### ---###
--#### --##-#
-##### -#####
Figure 4: Rearrangement isn’t detected using uniform text properties
Note 8.20 (Changing boundaries in source documents).

In Note 8.19, we talked about adapting transclusion boundaries to changes in the derivative document. However, it should be noted that boundaries may shift in source documents as well: if the object being transcluded is “the region between character position 10 and character position 250 from article A” and a lengthy header is inserted, the idea behind the transclusion may be completely swept away. The way to be safest here is to specify objects without referring to their boundaries, e.g. “Article A” should be safe.

There may be a way of maintaining the boundaries recorded on the transclusions themselves in such a way that edits to the source article are reflected in updated information in the transclusion; i.e., if text is inserted into the region that is being transcluded, the boundaries could grow, for example. But doing this seems tricky (at least modestly tricker than maintaining backlinks, for example), and I think I will hold off on doing anything like this until I get some clue that it is actually feasible to do it right.

Note 8.21 (The transclusion algorithm).

(1) The first thing is to run a function to select an object to transclude.

(2) Then we render it to plain text and insert that text in the current article. Appropriate text properties indicating that it is a transclusion are added next. (If instead of adding a new transclusion, we simply wish to render an article that includes transclusions, we must render each transcluded item in this way.)

(3) We let the user go wild.

(4) If things have changed at commit time, we need to put new transclusion instructions together. This new information replaces the old. And, in any event, the rendered transclusions are converted to an internal format for storage (Note 8.12).

Note 8.22 (Simple selection makes for expediency).

In light of the comments in Note 8.20, for the time being we choose the simple method of only allowing the transclusion of whole articles. There don’t appear to be any major theoretical obstructions to the transclusion of passages, but let’s start with something that will clearly be manageable. (See Note 2.14.)

Note 8.23 (Derivative predicates for transclusions).

Some sort of ‘derives-from’ predicate may have to be maintained for transclusions, although here such a ‘derives-from’ predicate might be deduced from the transclusion text directly. (This is similar the approach we took with inclusions.)

8.3 Inclusion

Note 8.24 (On ‘include-article’).

It is simple enough to find an article and insert its text at point (compare ‘insert-buffer’).

(defun include-article (article)
  (interactive (list (read-article-name)))
  (let ((beg (point)))
    (insert (sch-plain-text (get-article article)))
    ;; this condition isn’t really strong enough
    (when name-of-current-article
       ‘(derives-from ,name-of-current-article ,article ,(genref))
       ‘((passage (,name-of-current-article
Note 8.25 (By default, don’t print ‘derives-from’ scholia).

Typically we don’t need to see ‘derives-from’ scholia, although it could be handy to write a special display mode in which these scholia are printed (and probably special semantics for how they are displayed).

(add-to-list ’non-printing-types ’derives-from)
Note 8.26 (Deleting ‘derives-from’ properties).

If upon reparsing, if no sections are actually marked up with a ‘derives-from’ property that was previously recorded, it can be assumed that the text that was quoted from that particular source article has been deleted. In which case, odds are that the formerly-derivative document doesn’t derive from that source any longer.

Note 8.27 (Add ‘include-article-formatting-minimally’).

It might be helpful to have a function that would do everything that the previous version does, but without marking the text up, or perhaps just running it through ‘format’. But I’m going to hold off on this until the use cases seem clearer to me.

Note 8.28 (Record version number of included article).

Since inclusions are static, we should probably make note of the specific version of the source article that the recipient derives from.

Note 8.29 (Including passages).

It might be useful to support optional “beg” and “end” arguments that would make it possible to quote only a substring of the article that is being included. This seems most likely to be useful in a setting in which we select the passage to include interactively. (Cf. Note 8.22.)

8.4 Identification

Note 8.30 (Identification from a design perspective).

Basically, identification is “like transclusion for editing”: we have direct write access to the original, as well as read access. Identification is more powerful than transclusion; and, obviously, one can only objects that one has permission to modify can be edited through identification.

Unlike inclusion and transclusion, identified copies should be marked up with uniform, sticky, text properties. (The user should be careful that text from the current buffer that wasn’t intended to be part of the identified copy doesn’t get stuck to the identification by mistake!)

Note 8.31 (The identification algorithm).

The object that we’re identifying with should be inserted and marked up with an ‘identifies-with’ text property at render time. Any markup of this text (e.g. processing of transclusions or identifications within the text) should be carried out just as if the source was being edited directly. At commit time, text bearing the ‘identifies-with’ property should be compared with the object it derives from. If it has been changed, a new version of that object should be committed. As for the derivative article, an internal representation similar to the one described in Note 8.12 should be stored.

Note 8.32 (Precedent for identification).

The emacs package ‘‘all.el’’ is abstractly similar1717 17, as is the idea of shared list structure (Note 8.6).

Note 8.33 (Identification use case).

Just for instance, one could use this style of transclusion to render two articles into the same buffer and cut and paste portions of each into the other, and have the changes mapped back appropriately.

Note 8.34 (On ‘insert-identification-at-point’).

This is similar to Note 8.24. For now, I’m just going to write it so that you can do identification with other articles; later, more general objects will be allowed (Note 8.4).

We don’t actually need to create any scholia, but we can use the scholia property. It should be exactly the same when we render articles that already had identifications recorded internally.

(defun insert-identification-at-point (article-name)
  (interactive (list (read-article-name)))
  (let ((beg (point)))
    (insert (sch-plain-text article-name))
     ;; there aren’t any links, so there’s no proper linkid
     ‘(identifies-with ,article-name))))
Note 8.35 (On ‘propagate-changes-to-identification-source’).

This runs within the context of ‘store-updates-from-identification-images’ (when it runs). Its purpose is to copy changes from the recipient article in an identification to the source article.

(defun propagate-changes-to-identification-source ()
  (when (not (equal (buffer-substring-no-properties
                     (second (car identifications))
                     (third (car identifications)))
                      (second (caar identifications)))))
    (let ((source (get-article (second (caar identifications)))))
      ;; compare ‘commit-edits’.  When we actually do
      ;; document versions, this should update the version
      ;; number and any other metadata about how this
      ;; version was created that we get interested in.
      (scholium (second (caar identifications))
                ;; this is bad, since it is storing
                ;; the rendered text, whereas we should of course
                ;; be storing the formatted-for-internal-storage
                ;; text.  But I suppose this is OK for testing
                ;; purposes (wherein we’ll only have one level
                ;; of identification).
                 (second (car identifications))
                 (third (car identifications)))
                (scholium-about old-contents)
                (scholium-type old-contents)
                (scholium-bookkeeping old-contents)))))
Note 8.36 (On ‘store-updates-from-identification-images’).

This function will run within the scope of ‘commit-edits’ via the ‘commit-edits-hook’ (see Note 7.18). We must first identify the identifications (as it were), which we do by examining the contents of the ‘adjusted-markup’ list. Then, once these are found, we form a suitable internal representation.

Everything between identifications should get stored as plain text, whereas the identifications themselves should be stored as tags.

In addition to adjusting the internal representation of the article with the identification commands, we should adjust the text of the source articles, if necessary. (For simplicity’s sake, we could just store updated text unconditionally for now.)

Concerning the use of ‘buffer-substring-no-properties’: zapping the properties is questionable, if we’re planning to commit these various derivative properties serially. Will have to think about this later. I think that since this thing is being used to build internal representations, zapping text properties might actually be OK. But I’m not sure.

For clarification on the use of ‘adjusted-text’, see comments on ‘commit-edits’.

Note that the only relevant option besides “less” is “equal” in the ‘if’ here.

(defun store-updates-from-identification-images ()
  (let ((identifications (remove-if
                          (lambda (elt)
                            (not (eq (caar elt) ’identifies-with)))
        (pt 1)
    (while identifications
      (if (< pt (second (car identifications)))
          (setq formatted-contents
                 (list (buffer-substring-no-properties
                        pt (second (car identifications)))
                       ‘(ident ,(second (caar identifications)))))
                pt (third (car identifications)))
        (setq formatted-contents
               (list ‘(ident ,(second (caar identifications)))))
              pt (third (car identifications))))
;      (propagate-changes-to-identification-source)
      (setq identifications (cdr identifications)))
    (when formatted-contents
      (when (< pt (point-max))
        (setq formatted-contents
               (list (buffer-substring-no-properties
                      pt (point-max))))))
      (setq adjusted-text (list ’twd formatted-contents)))))

(add-hook ’commit-edits-hook ’store-updates-from-identification-images)
Note 8.37 (Working identifications into the rendering pathway).

We’ll need to modify rendering to look at the “ident” objects in the document’s representation, and mark these up as identifications.

The way identifications and transclusions are expanded will be similar. (See section 8.5 for more on this.)

Note 8.38 (Visible identification markup).

It would be handy to optionally have overlays showing which regions are marked up as identifications, and also to have scholia that say what they are identified to. (This seems like a good example of a situation in which it would be be best if the scholia only showed up when the regions they applied to appeared onscreen.)

Note 8.39 (Cutting and pasting identifications in multi-user environment).

One should be careful that passages that have been marked up as identifications, which are copied and pasted into other articles which don’t have the same permission-set associated with them, are no longer treated as identifications. Perhaps some identifying information (‘name-of-current-article’ being a likely candidate) can be added to the text property added by ‘insert-identification-at-point’, and some additional checks put into the commit phase.

8.5 Rendering articles containing derivative portions

Note 8.40 (Cluded parts).

These variables will be modified by ‘unwind-derivatives’ to contain a list of the transcluded sections of a rendered document. Specifically, the items on the list are of the form

(<object> <beg> <end>)

where ‘object’ is a link to the thing being transcluded, and ‘beg’ and ‘end’ represent the beginning and end positions of the image of the object under this particular clusion, in the rendered version of the cluding article.

This information will subsequently be used as part of the markup routine, where it will allow us to add appropriate text properties indicating cluded regions.

(It may be advantageous to assume that we can always use a cached version of the article we’re cluding. This could be a help when the article itself is derivative.)

(defvar transcluded-parts nil)
(defvar identified-parts nil)
(defvar included-parts nil)
Note 8.41 (On ‘unwind-derivatives’).

This will run within the scope of ‘sch-plain-text’, and make recursive calls to ‘sch-plain-text’ as needed, to turn cluded texts into strings suitable for display. Specifically, a simple test has been added to ‘sch-plain-text-hook’ that causes ‘unwind-derivatives’ to run whenever rendering an article whose text field is a list that begins with the token ‘twd’ (see Note 8.43).

The boundaries of the cluded regions are recorded for subsequent use, as described in Note 8.40.

The contents of the “ *Unwinding Derivatives*” buffer needs to be cleared out before this function first runs for it to return the correct string. (Um, is this all going to work out properly if we enter into the fully recursive run of ‘unwind-derivatives’?) That is accomplished by modifications to ‘scholia-display-pre-update-hook’; see Note 8.42.

The need for (extensive) recursion in this function could be done away with if we cached a rendered version of every article that used clusions somewhere where it could be found by other articles that clude from it.

However, if we are going to have things like references from cluded documents appear in the assembled document, or (in general) if we want to have access to the scholia attached to the cluded regions (Note 8.61), we’ll either have to pre-render the derivative components, or come up with some scheme for mapping the positions of markup associated with these things into suitable positions in the assembled document.

Note that at present markup will be added by ‘add-inclusion-and-transclusion-markup’ (Note 8.44).

(defun unwind-derivatives ()
  (set-buffer (get-buffer-create " *Unwinding Derivatives*"))
  (dolist (elt (cdr text))
    (if (stringp elt)
        (insert elt)
      (let (object)
         ((eq (car elt) ’transclude)
          (setq object (cdr elt))
          (setq transcluded-parts (cons ‘(,object ,(point))
         ((eq (car elt) ’identify)
          (setq object (cdr elt))
          (setq identified-parts (cons ‘(,object ,(point))
          (setq object elt)))
        (insert (sch-plain-text object))
         ((eq (car elt) ’transclude)
          (setcdr (cdar transcluded-parts) ‘(,(point))))
         ((eq (car elt) ’ident)
          (setcdr (cdar identified-parts) ‘(,(point))))))))
  (setq ret (buffer-string)))
Note 8.42 (Preparation for rendering text with derivative components).

We need to zap ‘transcluded-parts’, ‘identified-parts’ and the contents of the buffer in which derivatives are to be unwound before ‘unwinding-derivatives’ runs.

(defun prep-for-rendering-text-with-derivative-components ()
  (setq transcluded-parts nil
        identified-parts nil
        included-parts nil)
    (set-buffer (get-buffer-create
                 " *Unwinding Derivatives*"))

(add-hook ’scholia-display-pre-update-hook
Note 8.43 (Adding ‘unwind-derivatives’ to the rendering pathway).

We make a call to ‘unwind-derivatives’ in ‘sch-plain-text’ when the appropriate criterion is satisifed (see Note 5.4). The relevant criterion is that we are rendering an object (i.e., something that gives us an article) whose text field is a list that begins with the token ‘twd’.

(The complex wording here has to do with the matter that we sometimes render objects that aren’t articles, e.g. a link to a certain passage. We will have to make sure that the criterion used here is correct for those cases as well as the easy case of rendering a whole article. See Note 5.5 for details.)

(add-hook ’sch-plain-text-hook
          ’(lambda ()
             (when (and (listp text) (eq (car text) ’twd))
Note 8.44 (On ‘add-inclusion-and-transclusion-markup’).

For identifications, this should add the same sort of faked-up markup that we added with ‘insert-identification-at-point’ (Note 8.34).

It might be kind of cute to put some “corners” in, showing where the included text begins and ends (see Note 2.30).

These modifications to ‘mark-things-up’ appear to be desired unconditionally (unlike e.g. the modifications to ‘sch-plain-text’ we just saw in Note 8.43). So we just put ‘add-inclusion-and-transclusion-markup’ directly on the ‘mark-things-up-hook’ (cf. Note 5.53).

(defun add-inclusion-and-transclusion-markup ()
    (dolist (elt identified-parts)
       (second elt)
       (third elt)
       ‘(identifies-with ,(first elt))))
    ;; this may end up having to be considerably more complicated
    (dolist (elt transcluded-parts)
       (second elt)
       (third elt)
       ‘(transclusion-of ,(first elt))))))

(add-hook ’mark-things-up-hook ’add-inclusion-and-transclusion-markup)
Note 8.45 (Should identifications be unwound recursively?).

I think they won’t need to be; if someone asks to identify a region with some other region that also contains identification, then you might expect that we’d need to follow through to identify the indicated portion of the new article with that other earlier source. That’s probably the right way to do things; however, I suppose we could do something simpler in the mean time, since if we don’t write documents that depend on documents with identifications in them, the case we’ve been discussing won’t come up! Besides, it would probably be instructive to have on hand a non-recursive function that does a similar job to the function written for transclusions.

(Note that labels can sometimes be used to limit complexity of certain operations, see Note 3.69.

8.6 Expanded and collapsed views

Note 8.46 (Expanded and collapsed views of derivative articles).

There should probably be at least two different views into an article that makes use of any of the derivative-making tools we’ve discussed in this section: one that shows all of the rendered text, and another which just shows tokens standing in for the rendered text. See Note 2.39, as well as Note 2.40. Collapsed views are essentially “graphical” representations of formats like the one listed in Note 8.12. In order to make these views available to the user, we’ll want to provide ‘expand-derivatives’ and ‘collapse-derivatives’ commands, as well as, probably, individual commands for individual derivative items and each of the three derivative types.

Note 8.47 (View is initially expanded).

New derivative pieces are inserted in their expanded form (see, e.g., Note 8.34).

Note 8.48 (Collapsing and expanding individual derivative items).

We could set things up to render everything as collapsed or expanded all at once, but another option is to have a function that collapses or expands individual items.

Note 8.49 (Identification through nontrivial maps).

In some cases it would be useful to have identifications go through non-trivial maps, for example, to transform a rendered piece of LaTeX text into the standard source form. In addition, it might be nice to store metadata about Notes in some other place than the note’s immediate neighborhood.

Note 8.50 (Alternative to identification for simple literate programming applications).

It may also be the case that we don’t really need to do identifications to get the ability to import and export code from the scholium system. As long as everything is appearing in Notes, we should be able to edit things in an outline fashion and then export that to the LaTeX document.

8.7 Quick compilations and other listing tricks

Note 8.51 (Uses for compilations).

What is presented in this section likely to just be a stop-gap measure until more general and powerful mechanisms for managing derivatives are available. But nevertheless, the basic ideas seem useful. For example, when viewing a certain article that is a member of a certain label, it might be handy to view everything in the label all at once in a compilation. And so on and so forth for various compiled views into a document.

Note 8.52 (On ‘find-names-in-listing’).

This can be used to find the names of listed articles.

(defun find-names-in-listing (just-marked)
  (let ((names (list t)))
      (set-buffer (get-buffer-create "*Generic List*"))
      (goto-char (point-min))
      (while (and (not (eobp))
                  (if just-marked
                      (search-forward-regexp "^>" nil t)
        (let* ((next-change (next-single-property-change
                             (point) ’name))
               (prop (when next-change
                       (get-text-property next-change ’name))))
          (if (not next-change)
              (goto-char (point-max))
            (goto-char next-change)
            (when prop
              (nconc names (list prop)))))))
    (cdr names)))
Note 8.53 (Making a compilation from a listing of articles).

In Note 1.3, we talk about making all the articles that match a given criterion into a new article. A quick way to get this functionality to the user is to turn an article listing into a compilation.

The version here is very preliminary; eventually we’ll be putting in identifications or something like that to make the various pieces of the compilation “hot”.

It be nice to have various display options, either features that screen certain kinds of content in or out at render time (e.g. printing of attached code snippets could be optionally be made automatic), or that add actionable features to the display (e.g. to enable the user to expand and collapse cluded articles, or to select from a cluded label).

We present a couple of variants here.

(defun listing-to-compilation (just-marked-items)
  (interactive "P")
  (let ((names (find-names-in-listing just-marked-items)))
    (pop-to-buffer (get-buffer-create "*Compilation*"))
    (dolist (name names)
      (let ((article (get-article name)))
        (insert (upcase
                 (propertize (format "%s" (scholium-name article))
                             ’face ’italic))
                 (format "%s" (scholium-text article))
    (goto-char (point-min))))

(defun listing-to-compilation-1 (just-marked-items)
  (interactive "P")
  (let ((names (find-names-in-listing just-marked-items)))
    (pop-to-buffer (get-buffer-create "*Compilation*"))
    (dolist (name names)
      (let ((article (get-article name)))
        (insert  "\\begin{notate}{"
                 (propertize (format "%s" (scholium-name article))
                             ’face ’italic)
                 (format "%s" (scholium-text article))
        (dolist (scholium (mapcar (lambda (backlink)
                                    (get-article (car backlink)))
                                  (get-backlinks name)))
          (when (typedata-includes (scholium-type scholium) ’code)
            (insert "\\b" "egin{lisp}\n"
                    (scholium-text scholium)
                    "\\e" "nd{lisp}\n\n")))))
    (goto-char (point-min))))
Note 8.54 (Finding code continuations with backlinks).

Establishing the existence of attached code snippets in ‘listing-to-compilation-1’ (Note 8.53) is currently done in a way that doesn’t take full advantage of the backlink-to-link correspondence we’ve put together. This sort of thing was one reason I thought we might want to use typed backlinks; but eventually I decided that simply putting backlinks in direct correspondence with links would be enough, see Note 3.57.

Note 8.55 (Additional compilation tricks).

Code snippets aren’t the only “bonus features” we might like to have appear in a compilation (cf. Note 8.53); for example, we might want to make an article appear together with the articles that immediately precede and follow it in some linear order. (This makes the compilation a bit more like browsing a printed document.)

It would also be handy to have an option for exporting a fully LaTeX-ready version of a compilation (compare Note 8.56).

Another fun trick that would be fairly cute, when this is all working quite a bit better, would be to have links that run internally within the compilation show up in a different color from links that point outside of the compilation. (We’ll want to develop some code for making the display jump to just the right point when following an internal link, or any link that points to a specific portion of a given article.) One might want to view one of these intra-linked articles in place in the compilation, or perhaps instead on its own, using the normal single-article display.

The sequence of things displayed in the compilation buffer is the sort of thing that might benefit from having its own history list; see Note 6.55.

Note 8.56 (Selectively populate existing outline).

It would be kind of interesting to combine ‘listing-to-compilation’ with ‘export-scholium-system’ (Note 12.38) in such a way as to add some particular subset of the Notes into the section hierarchy.

Note 8.57 (On ‘listing-to-label’).

It can be handy to turn an arbitrary listing into a more permanent label.

(defun listing-to-label (label-name)
  (interactive "MLabel: ")
  (let ((names (find-names-in-listing)))
    (scholium label-name
              (cdr names)

8.8 Further notes on derivative works

Note 8.58 (Derivatives and copyright).

Recall that for copyright purpose, it is expression that is supposed to be examined when deciding whether a given work derives from some other work. This somewhat elusive quality is going to be harder to capture than the property of being a derivative via direct quotation and subsequent revision.

Note 8.59 (Pedigrees).

When adding metadata to show what articles a given article derives from, we probably only want to bother with the parent, not with ancestors from earlier generations. More extensive information about an article’s pedigree can be found by chasing backlinks, as needed.

Note 8.60 (Aggressive management of derivatives).

A more “aggressive” solution to recording information about derivative works would be to mark up rendered articles with a text property that identifies them; this way, whenever the text is cut-and-pasted to a new buffer, the text property will be carried along, and the source of the text can be identified and noted when the recipient article is reparsed and committed.

Note 8.61 (Fancy derivatives).

It may be useful at times to carry over certain scholia attached to the region that is being quoted. There are probably a number of variations that people would be interested in trying.

Note 8.62 (Eventual improved support for virtual collections?).

In Note 8.1 we briefly contrasted derivative articles with collections and indices. It seems likely that we can come up with tools that will make it easier to build useful indices; subcollections (Section 3.5) seem to capture the basic idea, but there is room to grow there.

Note 8.63 (Derivative articles and forking).

One obvious way to fork an article is simple – to include it in full and then begin edits there. The ability to do this gives users quite a bit of freedom; however, it is worth noting that it can frequently be to a user’s benefit to get their changes inserted in (or around) the original source, instead of placing them in a separate derivative version. (This relates to Section 13.)

Note 8.64 (Tracking changes in parents).

Transcluding will make it so that changes to the parent are immediately reflected in the child. This is not so with inclusions; however, owners of articles that use inclusion may still want to be updated when the parent changes. See Section 10.1.

Note 8.65 (Tracking changes in children).

Nevertheless, there are certainly times when it is beneficial or expedient to create a modified version. To cater to users in this situation, we could also set up a derivative-tracking device, who’s function would be to update the original author about the fate and progress of any derived versions. See Section 10.1.

Note 8.66 (Overviews).

It would be nice to include markup associating an overview with the section of the document that was being overviewed.

Editing (or mixing) down is essentially similar thing done in a different spirit. Whereas I’m imagining that the edited-down version would replace the old version, the overview would likely exist in parallel with the original.

Note 8.67 (Transclusion with modifications).

In addition to the overviews of Note 8.66, we could be interested in other sorts of “transclusion with modifications”. For example, if the overview is created by transcluding only the first paragraph from every chapter, other modified versions could be created by replacing foo by bar everywhere.

Note 8.68 (Transclusion (typically) induces a linear structure).

If an article is built out of other transcluded articles, then it should be possible to abstract an order, so that one can say “next article” or “previous article” and have these things make sense. But of course, this order can easily be broken, say by allowing there to be two “next articles” (compare Literary Machines). Also note that nested transclusions (typically) induce a tree structure.

Note 8.69 (Transclusion-like actionable derivatives).

In order to keep the first example of a scholium given in section 3.2.2, “Remark on Scholium Definition”, up to date, it would be nice not to have to keep changing the numbers that identify the the beginning and end of the region being indicated by hand. This problem could be solved by putting a scholium in place of each number that renders as the appropriate number whenever the article is rendered. A standard scholium about the region in question would be maintained by reparsing (Section 7.3), but the actual numbers that designate the beginning and end of this region will move around. Designing the actionable scholium that would accomplish the job here will take a bit of doing.

Note 8.70 (Indivisible read-only text for collapsed labels).

Perhaps collapsed text (Note 2.39) should have a special read-only property (alternatively, changes to such text would constitute suggested changes to the name of the article that was transcluded; see Note 8.33).

Note 8.71 (Scholia Display as transclusion).

Although it isn’t technically put together that way (at least, not now), one could think of the Scholia Display as being an article that is put together by transcluding all of the scholia attached to the current main article.

Note 8.72 (Rendering deleted objects).

This is a general problem that affects both rendering straightforward articles which happen to be transcluded and rendering articles with transclusions that point at articles that have been deleted. Presumably, some kind of “404 File not found” error will be displayed at an appropriate point in the text.

Note 8.73 (Building transcluded articles by reparsing).

Particularly if transcluded portions of a document are not directly editable, it might make sense to build “text with transclusions” by first assembling the content of a buffer containing transclusions, and then parsing the text properties. If the transcluded portions were directly editable, we could get into a somewhat tricky situation (e.g. should shearing three characters off of the end of the image of some transcluded text also change the transclusion command so that those characters are not copied over in the future? – probably, although there are probably better ways of editing transclusion specifics).

This is essentially what I’ve decided to do; see Note Transclusion from a design perspective.

Note 8.74 (Rendering text with derivative pieces to specific places).

Right now, text that includes transclusions is set up to only render to the “Main Article Display” buffer (because the text field is not a buffer). We could set things up so that articles whose text contains transclusions could be associated with specific buffers where the rendered text would appear. This would require some changes to ‘scholia-overwhelm-display’ (see Note 5.22).

9 Saving and restoring

Note 9.1 (Why saving?).

Having created several different scholia, you may wish to save your work, to read back in later, or to send to a collaborator.

Note 9.2 (Reverting).

The idea of reverting to an earlier revision relates to the previous section, as well as, vaguely, to the notion of restoring a saved article. It isn’t clear which section it should go in, since it isn’t actually any of these things! It seems like a somewhat likely candidate to go into the previous section instead of this one.

Note 9.3 (Restoring and managing state).

Various details (e.g. an index counting new articles that have been added to the system) may need to be maintained between sessions.

Note 9.4 (Saving buffers to files).

When saving/quitting, articles that come from buffers may then get associated with files (assuming that the buffers end up getting saved to a file). Presumably we should offer to save buffers to files; alternatively, we could save them in some internal format that the scholium system knows how to restore (reminds me of Emacs filesets; see also Note 9.10).

Note 9.5 (Saving special scholia).

We should presumably offer the option of just saving only new or modified scholia, or those that are connected with the current buffer, or those that have a certain type, etc., rather than saving the whole library. In some cases, the collection to be will presumably only be found by predicate matching; other times, we will be able to maintain an index of the subcollection to be saved (e.g. we can keep track of all the articles that have been modified since the last save).

Note 9.6 (Autosave).

We might consider making some environment variable that would cause things to be saved to disk automatically.

(defun save-all-scholia (filename)
  (interactive (list
                (read-file-name "Filename: ")))
    (write-file filename)
    (kill-buffer (current-buffer))))
Note 9.7 (On ‘gather-scholia’).

It might be a good idea for this function to take an optional predicate or label, and gather only scholia that match that predicate. We could easily cause the gathering function to run only within a certain context as well.

This function doesn’t really have anything to do with saving, logically speaking, so perhaps it shouldn’t go in this section.

(defun gather-scholia ()
  (set-buffer (get-buffer-create "*Scholia*"))
  (delete-region (point-min) (point-max))
  (maphash (lambda (name val)
               (write-scholium (cons name val)))
  (display-buffer "*Scholia*"))
Note 9.8 (Improving the design of ‘write-scholium’).

One could probably make these things print a bit nicer, e.g. to make everything fit within 80 columns, but this seems to be good enough for the time being.

(defun write-scholium (article)
  (let ((nl "\n          "))
     (concat "(scholium " (maybe-quoted-format (scholium-name article))
             nl (maybe-quoted-format (scholium-text article))
             nl (maybe-quoted-format (scholium-about article))
             nl (format "’%S" (scholium-type article))
             nl (format "’%S" (scholium-bookkeeping article))

(defun maybe-quoted-format (obj)
  (if (and (not (null obj))
           (or (atom obj) (consp obj)))
      (format "’%S" obj)
    (format "%S" obj)))
Note 9.9 (Design of ‘read-scholia-file’).

This reads and evaluates all of the scholia that have been written out into the file stored at filepath.

(defun read-scholia-file (filepath)
  (interactive "fFile: ")
  (find-file-literally filepath)
  (kill-buffer (current-buffer)))

(defun read-scholia-buffer ()
  (while (condition-case nil
             (eval (read (current-buffer)))
           (error nil))))
Note 9.10 (Apparently excessive generality of ‘read-scholia-file’).

This system would actually read any elisp file (er, I’m not sure it would deal well with comments). There are built-in functions that accomplish the same thing (‘load-file’ and ‘eval-buffer’ come to mind). The idea here was that we might want to do something somewhat more complicated than simply evaluating the code found in the file. That may still happen.

(defvar search-directory-for-scholia t)

(add-hook ’find-file-hook ’search-directory-for-scholia)

(defun search-directory-for-scholia ()
  (list-directory default-directory t)
  (let ((sch-file
         (replace-regexp-in-string "\\..*" ".sch" (buffer-file-name))))
    (when (search-forward sch-file nil t)
      (read-scholia-file sch-file)
  (kill-buffer "*Directory*"))

10 Distributed authorship

Note 10.1 (Introducing distributed authoriship).

Thus far, there has only been one author. But on a multi-user system, several different authors would be adding text to the system. It is fairly trivial to set up a system-wide list of directories to browse for articles, enabling users of a shared filesystem to co-author a digital library together. We can do something very similar when the authors are not physically co-located, by using CVS or GNU Arch to collect the contributions of the various authors.

Note 10.2 (Masters).

Do we need to have one “master” copy of each article in the system, or can downstream users consider the version that they downloaded to be the de-facto master copy? Probably the Arch documentation has something to say about this.

Note 10.3 (Differences between the system here and the typical diff-based model).

I’m not completely sure what the advantages of the system here will be over the typical diff- and email-based system for maintaining code. One difference is that information will be relayed using a “pull” rather than a “push” model (see Note 10.7). Other aspects of the system may simply be more convenient ways of doing things that people already do; for example, loading and unloading sets of patches. (I’m sure there is already plenty of software for doing this conveniently.) On this note, compare filesets1818 18; contributed data coming from any given source would likely be loaded into a special subcollection.

Note 10.4 (Relationship of namespaces to distributed authorship).

One might consider reading articles from each distinct article source into its own namespace. On the other hand, it would be possible to simply tag the articles with some bookkeeping information to show where they had come from. The same complexity issues arise here as in other (possible) applications of the namespace principle.

Note 10.5 (Merging articles with name conflicts).

From time to time, authors will create scholia about similar things (e.g. Emacs functions) and someone will want to merge the results. We should make it straightforward to do this. (It may be that the best way to merge is to just use namespaces and always display the articles together.)

Note 10.6 (Scalability of search).

In general, some work will have to be done to keep search spaces fairly small, but an opt-in model for spaces to search will help with that (compare RSS). The algorithms associated with a global version of this system (like Xanadu) would be pretty intensive. We’ll have to think more about this if a deployed system ever gets big.

Note 10.7 (Draw).

People should subscribe or not depending on their preferences. Free software, if nothing else, teaches us to look at the demand side!

In distributed implementations, we typically won’t send updates as much as make them available for pickup. We certainly could send some sort of notification saying that new content is available. On the other hand, there’s no particular reason not to send content directly in some implementations. (For more on the topic of making updates available, see Note 7.24.)

Note 10.8 (FAIF not FAIB, and DRM).

To think about this another way: the Xanadu system is FAIF but not FAIB. The idea was to charge people on a pay-per-view (or pay-per-use) basis, with some fraction of royalties going back to each of the authors whose work made up part of the delivered text. (It would be interesting to try to connect – or to dissociate – this model from current DRM ideas.)

Note 10.9 (The p2p connection).

Assembling a document from various disparate sources in the database, reminds me of something that comes up in p2p filesharing, namely the way in which documents can be assembled from partial data drawn from various nodes. How should one go about the process of assembling the document from the sources available? We will need to think and learn more about the algorithms that would need to be employed in a system like the one described here, including its potential uses in distributed content systems.

Note 10.10 (Increasingly expensive local operations).

Assembling massive amounts of information, e.g., sorting the scholia about a given document, is going to be somewhat computationally intense (nlogn, right?). If we ever got into situations in which there were thousands of scholia about any given article, we might want to avoid doing sorting (perhaps by storing the scholia in sorted order some place or other). Also, by default, we should avoid sorting any scholia we don’t need to display (e.g. frequently, actionable scholia), because sorting them is just a waste of time.

Note 10.11 (Reasons things might not bubble up too bad).

My guess is that even a very popular article won’t have a whole lot of printing scholia attached to it. A number of people will want to make comments, sure, but others will prefer to make derived versions, or comments on the 1st generation of comments.

Note 10.12 (Are names always available?).

I’m not sure whether an article would ever appear that had been marked up with names or namespaces that we don’t have access to. Maybe. This seems more likely to be an issue in real life when we’re dealing with distributed documents, but it could potentially be an issue whenever there are multiple namespaces (or even one poorly maintained namespace; compare Note 8.72 – sometimes articles will be deleted).

Note 10.13 (Accepting updated ‘about’ information).

Why would an author want to incorporate updated about information into the original article? One reason is simply to check that this information is accurate. By allowing users to set updated about data automatically when editing a scholium’s target, a degree of flexibility is built into the system that would be impossible (and, indeed, probably wouldn’t make sense) otherwise. But when one’s work is being interpreted in a new way, and, indeed, in a new context, it may be worthwhile to check that you agree with this interpretation.

10.1 Bidirectional updates

Note 10.14 (Setup for bidirectional updating).

We’ll assume that one of the authors is using the scholium system, and the other isn’t. (The other possible cases have either been covered or are not part of our mandate.)

In order to make bidirectional updating work when some of the editing is going on outside of the scholium-based system, we would need to be able to read a stream of diffs coming from the other system and decide how to incorporate the modifications. And we also need to be able to provide them with a stream of diffs that they can use however they see fit. We’ll certainly have to think more about how to do this!

Note 10.15 (Using simulation to do bidirectional updating).

It seems to me that the most useful approach for someone working within the scholium system and trying to do bidi updates with a non-scholium system would be to run a simulated version of the non-scholium system inside the scholium system.

Note 10.16 (The simplest case).

The simplest case of bidirectional updating is identification, which is treated in Section 8.4. Even this case has some complexity to it in the distributed case; updates coming from multiple authors presumably need to be formatted into branching derivative versions. (Actually, this is an n-directional updating problem; Arch has had some things to say about this, but we can probably say more.)

10.2 Interoperability

Note 10.17 (Exporting).

It should be possible to reduce metadata and so on to some format that would work with external programs (e.g. database programs). I’m not at all sure how the use of text properties would be replaced in other programs, but I think that, in theory, the things we save (Section 9) should be sufficient and readable for any other interested program. I’d be interested to know the thoughts of others on building other interfaces to the scholium system.

10.3 A simple scheme for multiuser cooperation

Note 10.18 (Crosscut by users and dates).

A simple scheme for facilitating interactions between several users is as follows. Each user gets a directory to call their own. Each users scholia for the day are stored in a file in this directory. To catch up with the activities of all users, one simply loads all of the files that have appeared since the last load date.

To implement this scheme, we need a method for populating the data-store and another for scanning it to determine which files to load.

We’ll assume that only the scholia that bear a certain label (say “to publish”) will be published, and that once they have been published, this label is wiped out (maybe it is set to “last published”). In general, the “to publish” set could be determined by some somewhat complicated algorithm, not just by looking at a label.

Also, we should presumably maintain all of the imported scholia in a namespace (most likely, certainly some subcollection) in order to keep them straight. (For a first draft, we could just import everything to the main article table, given that namespaces don’t completely work yet.)

Presumably we should start by building the infrastructure in our CVS directory. (Some more general formulation that allows collaboration between different people writing to disparate locations online would be good to have eventually; to support this we’ll want some sort of protocol, maybe something like RSS.)

Note 10.19 (Networked blotters).

Keeping one another up to date on the important features of daily goings on is an important use of the multiuser system. The notion that each user would maintain some sort of “blotter”, possibly incorporating some auto-generated content, which others can read to stay in the loop.

Note 10.20 (Strategic usefulness of building and using collaboration tools early on).

While we could ask people to work on other things first (and to some extent it will be necessary for people to work on other things all along the way) it seems likely that using powerful collaboration tools will help people get things done on other projects or subprojects. When many people can share differing views on a topic, the discussions of this topic will develop a richness that wouldn’t be so likely to develop otherwise. A system that can take into account many views in a coherent and useful way is going to be a powerful system. (It would be good to do some research on the differing abilities of various extant systems to accomplish this.)

Collaboration is a challenging thing to deal with. You have greenhorn or yahoo types who want to charge off and do things on their own (I’m often one of them), and in the mean time, you don’t always have a central point for collaboration on some topic where you might hope one would be. It isn’t always clear when you have a certain vision whether there are others out there who would share that vision or not. Partly, that’s because when you have a vision it isn’t always completely clear what that vision is; but these things interact.

11 Bindings and environment variables

Note 11.1 (Room to improve on bindings).

This is just an attempt to make something usable, but it is probably nowhere near close to optimal. One thing that would be nice would be to have some prefix for browsing relative to all scholia versus some other prefix for browsing relative to only the current scholium. Thus, you could cycle between regions for the current scholium or regions for all scholia (or go to the first region or whatever) depending on what prefix you specified – but the final keystroke (“f” for forward or whatever) would be the same. Perhaps we could substitute in the first register (as opposed to the second) for changing the general style of browsing (e.g. from scholia to temporal or whatever), but keep second and third pretty much the same. It would be good to spell out the analogies in a somewhat detailed table.

 (lambda (elt) (global-set-key (eval (car elt)) (eval (cdr elt))))
 ’(([?\C-\;] . (make-keymap))
   ([?\C-\;?m] . (make-sparse-keymap))
   ([?\C-\;?d] . (make-sparse-keymap))
   ([?\C-\;?o] . (make-sparse-keymap))
   ([?\C-\;?s] . (make-sparse-keymap))

   ((kbd "C-; c") . ’commit-edits)
   ((kbd "C-; g") . ’gather-scholia)
   ((kbd "C-; f") . ’follow-reference-or-scholium)
   ((kbd "C-; n") . ’name-of-current-scholium)
   ((kbd "C-; r") . ’save-all-scholia)

   ((kbd "C-; v n") . ’back-to-normal)
   ((kbd "C-; v o") . ’back-to-other-view)

   ((kbd "C-; b b") . ’sb-back)
   ((kbd "C-; b f") . ’sb-forward)

   ((kbd "C-; l a") . ’article-menu-list-all-articles)
   ((kbd "C-; l l") . ’article-menu-list-labels)
   ((kbd "C-; l m") . ’article-menu-list-articles-matching-regexp)
   ((kbd "C-; l d") . ’article-menu-list-metadata-articles)
   ((kbd "C-; l p") . ’article-menu-list-plain-articles)

   ((kbd "C-; m a") . ’make-scholium-about-current-article)
   ((kbd "C-; m b") . ’make-scholium-about-current-buffer)
   ((kbd "C-; m l") . ’make-scholium-about-current-line)
   ((kbd "C-; m n") . ’make-new-undirected-article)
   ((kbd "C-; m i") . ’make-current-buffer-into-article)
   ((kbd "C-; m p") . ’make-scholium-about-part-of-current-article)
   ((kbd "C-; m P") .
   ((kbd "C-; m s") . ’make-scholium-about-current-scholium)

   ((kbd "C-; d a") . ’display-article)
   ((kbd "C-; d b") . ’display-scholia-about-current-buffer)
   ((kbd "C-; d l") . ’display-article-listing)
   ((kbd "C-; d p") .
   ((kbd "C-; d c") .
   ((kbd "C-; d r") . ’redisplay-article)

   ((kbd "C-; o y") . ’sch-turn-main-article-overlays-on)
   ((kbd "C-; o n") . ’sch-turn-main-article-overlays-off)

   ((kbd "C-; s n") .
   ((kbd "C-; s p") .
   ((kbd "C-; s b") . ’move-to-previous-region-with-scholium)
   ((kbd "C-; s f") . ’move-to-next-region-with-scholium)
   ((quote [S-tab]) . ’move-to-next-region-with-scholium)
   ((kbd "C-; s a") . ’move-to-first-region-for-current-scholium)))
Note 11.2 (Style of code for bindings).

I’ve heard that using ‘eval’ is typically bad style, but something had to be done to make these bindings work. What would have been better in terms of coding style?

Note 11.3 (Environment variables).

It would be handy to be able to have things set up so that e.g. every buffer is automatically an article, too. Various innovations along these lines should be added here.

Note 11.4 (Automatically loading scholia).

Along the lines of Note 11.3, this section would be a good place to put a variable that would cause scholia to be loaded for all buffers (which I think says slightly more, or anyway something slightly different from, the statement that every buffer would automatically be an article). Note that whenever loading scholia for a given buffer, there may be several different places to look to find those scholia: in the article table, but also in local “.sch” files, or other places.

12 Experiments

Note 12.1 (Experiments to try).

This section should describe our experiments modeling different CBPP systems that are out there in the world – and coming up with new ones that we like better.

In this section, we can describe some separate packages that provide more interesting or advanced display features, for example. Also maybe some packages that do things with code (e.g. interactive text-based games). Wiki portage would be one basic thing to try. Finally, of course, it would presumably be useful to get the HDM codebase into the scholium system.

  • Importing a wiki and getting the scholium system set up to do wiki-like things.

  • Using the scholium system to write a play, novel, screenplay, or an interactive fiction type of game (actionable scholia would probably be part of just about any game). Hipster the Gathering1919 19 would presumably be an easy game to code up. My forays into “fine literature” with the Penmaster’s Apprentice saga could also be typed up. These things are currently interesting examples of physical scholia (see Section 12.9).

  • Using the scholium system to maintain a text-based forum or set of fora, as found on PM or slashdot.

  • Using the scholium system to manage an evolving codebase, e.g. HDM content, say APM and APM-Ξ, or this document itself (making the system interoperable with LaTeX would be important).

  • Using the scholium system to manage TODO lists (like we used to do with TODL etc.).

  • Port GCIDE to Emacs and give instant access to definitions from within the browser. Make GCIDE scholiumific (supplying semantically close definitions or whatever?).

  • do something like mmm-mode but using scholia about the buffer instead of begin/end regexps?

  • Make a semantic network describing relationships between words and use it for writing poetry.

  • make a system for exporting to other formats (see Note 2.40)

  • Write text-generation software that will provide different ways of viewing the network’s structure.

Note 12.2 (Cross-referencing in LaTeX).

Note that the system of labels and references in LaTeX allow you to make “hypertextual” linear documents. This system could perhaps be extended (maybe by exploiting the scholium system). It would be cool to be able to cross-reference the relevant label in some other document and have a suitable reference show up here. This is a scholium-like thing: how to make it work in LaTeX? (Maybe \pmxref should be able to refer to anything that has a \label about it. Perhaps this would work already.) This also relates to the topic of Namespaces, see Section 3.5.

Note 12.3 (A comprehensive regression suite).

This would be a fairly simple experiment, which would nevertheless be very helpful for programming the system. Namely, create a short program that tries to do everything that you can do with the system.

Note 12.4 (Go moves).

It would be nice to have a browsable database of Go moves and games with user annotations: presumably the already-existing Go resources online include things like this. If the scholium system can help improve upon these resources, that would be cool. For example, perhaps gnugo’s move book could be simultaneously an instruction manual for human users.

Note 12.5 (Games built of scholia).

Note that the relationships of people to rooms (containment), people to their backpacks (carrying), and so forth – the basic elements of text-based games – are all interpretable as scholia-based. The “fantasy” tie-in of these sorts of games works reasonably well with the name Arxana. If we’re going to push development along this route, it would be good to do some reading on MUD-like systems.

Note 12.6 (Stress testing the system).

It would probably be worth doing some empirical tests to see what happens in a library full of a bunch of junk. How long does it take to find and display the one relevant scholium? How much are things helped by storing articles in namespaces?

Note 12.7 (Scholia about functions).

This function, or something like it, seems especially important if we’re going to use the scholium system to manage code. Every time we create a function, for example, it would be good to create a scholium attached to that function (perhaps a scholium object whose name is the function’s symbol name, though there could be the need to disambiguate between the function and variable versions of the same symbol) with a “defined-in-buffer” link about the buffer in which the given function definition is found; Emacs already has functionality like this in its help subsystem. This could be done by explicit command or just on-the-fly by some appropriately-constructed automatic function.

In addition, one could create scholia that say, for instance, which functions that function calls. Then, later, one could assemble a who-calls or who-is-called listing about a given function, for the user to browse. Although this can be done easily enough already (there is a who-calls.el), it is my sense that this approach would save on search. Browsing code at the “who calls” level seems potentially interesting; it would be worth investigating the prior art here more.

Note 12.8 (Barkov’s “Hamlet”).

The system could be used to handily create different versions of a play, e.g., to write a version of ‘‘Hamlet’’ manifesting the view that actually Horatio is the prince2020 20 or to splice in “Rosencrantz and Guildenstern are Dead” by Stoppard or “King of Shreds and Patches” by Elrord or whatever (assuming that doing so was legally copacetic; for personal use of course it would be). Similarly, our treatment could describe theories about how the play relates to things outside the play, or (being even more meta), how the interpretation of the play relates to the presentation, or (still more), where the interpretation comes from, and how I think my interpretation of these various things gets involved.

Note 12.9 (Using scholia on PlanetMath).

Since articles in the encyclopedia are supposed to describe known math, they aren’t really the right place to describe new insights, even if these new insights occur while reading that article. (In theory such insights could be discussed in the attached forum along with basic questions or whatever; but a catch-all forum or collection of catch-all fora and postings without metadata may not be the most conducive to research-oriented work.) My thought is, why not have an attached discussion not in the encyclopedia for talking about some particular advanced question. This area can itself use encyclopedia definitions as scholia to help orient casual readers. This way, treatments of established and “live” mathematics can develop together.

Note 12.10 (Implement a web browser that works with the scholium system).

Maybe I’ll write this before long? It seems like the sort of thing that could be very helpful; search through scholia on web pages you’ve browsed (and also download everything you browse and save it to disk, so that you can search through the web pages themselves too).

Through this sort of approach, you might imagine the web (or portions thereof) being ‘‘embedded’’ in a more advanced hypertext system, at least for some users. Something similar can be done for working with Project Gutenberg2121 21 or any collection of computer-readable text.

Note 12.11 (Use the scholium system to assist in literature review).

There are a number of different literatures that are relevant to the scholium system. From artificial intelligence, we have things like KM or SNePS that are available as free software, and which can accordingly be reviewed in their entirety. Or, at least in as much of their entirety as seems useful. It may be desirable to simulate or at least partially simulate these systems (which is partly what the preliminary literature review is supposed to determine, by looking at the basic theory of these systems).

In hypertext, there are systems like Noösphere which can be treated similarly. Indeed, I am planning a meetup with Noösphere’s author, Aaron Krowne, for purposes of mutual exchange over and potential cross-breeding between these two systems.

There are no doubt many other interesting literature things to study out there. Deciding which ones will be most profitable to look at is an interesting challenge.

Note 12.12 (Backlink tricks).

One obvious thing to do with backlinks is to compute the most-linked-to article, or, even better, to rank articles by their popularity according to in-links. This can be done by mapping a very simple function over the metadata label (or the ranking could be maintained on the fly).

12.1 Some extremely simple example to show people what’s going on

Note 12.13 (Simplest example).

It would be handy to create a very simple example of a buffer together with scholia.

Note 12.14 (Next-simplest example).

After that, it would be handy to make an example of a properly scholia-based document (showing some richer graph structure).

Note 12.15 (Various other simple examples?).

And while we’re at it, maybe later a simple example of transclusion or identification or whatever.

12.2 Use the scholium system to maintain a library of projects

Note 12.16 (Managing several projects).

We’ll do several different experiments, some of which will involve corpora that are (at least essentially) distinct. One high-level object to point at the “main pages” of these various projects might come in handy. There could be a number of other more advanced library-like features to add later.

Note 12.17 (Managing workflow).

In addition to simply presenting output of these projects, it would be helpful to have the system set up to manage workflow (tasks and so on). For now, see C.

Note 12.18 (Library of major works).

We have an overall structure for the digital library as a whole worked out already (Section 3.1), but not yet a specific list of major works. It is convenient to have an index of these (just like, later, it will be convenient to have many different indices of different kinds of documents with different features). So, we create a virtual library to index these works in. (Once this is set up properly, of course the imported version of the scholium system itself will be included.)

(scholium ’major-articles nil nil ’label ’system)

12.3 Use the scholium system to do literate programming

Note 12.19 (Importing LaTeX docs).

Note that importing a LaTeX document (this one, in particular) wouldn’t be so different from importing a wiki (see Section 12.5).

One of the issues is how we’re going to represent cross references (see Note 12.2). In compiled LaTeX, they typically appear as a number, whereas in source they appear as a tag. In the scholium system, reference markup should presumably be used, and I also suppose that we may as well use the name of the article being referred to directly. So, for example, this article would be linked to by text that read “see ‘Importing LaTeX docs’.” This is similar to the way references appear in Texinfo.

Outside references (footnotes) could be rendered in a different color (Note 4.30), and made browsable within the scholium system (Note 12.10).

On the implementation: ‘end’ might be followed by a lisp expression that should be attached to the note, but this should be fun for a trial run.

The format of the scholia representing the notes is kind of weird. We shouldn’t have junk in the text field of the article. It would probably make more sense for items of type "note" to be rendered specially (if one wished) and maybe to store the tag as a part of a formatted text field, or, more likely, part of the type data.

I think I’d be relatively comfortable adding identification properties to the regions of the larger document, to get them to inherit from the individual pieces. Later we could parse the section structure.

It would, generally speaking, be a good idea if lower levels in the hierarchy were about their parents in such a way that we could easily move “up”.

(add-hook ’scholia-display-post-update-hook ’text-mode)

(add-to-list ’modified-type-labels ’(note . note) t)
(add-to-list ’modified-type-labels ’(section . section) t)
(add-to-list ’modified-type-labels ’(subsection . subsection) t)
(add-to-list ’modified-type-labels ’(subsubsection . subsubsection) t)
Note 12.20 (On ‘map-label’).

This gives you a way to apply a function to every article that bears a given label.

(defun map-label (function label)
  (mapc function (scholium-text (get-article label))))
Note 12.21 (Example of mapping across a label).

Notice that we use ‘map-label’ (Note 12.20) in the APM-Ξ importing routine, to convert once-processed content to a more useful format. See Note 12.66.

Note 12.22 (On ‘swap-out-latex-references’).

Does the opposite of ‘swap-in-latex-references’ (Note 12.42): that is, this function makes \ref tags in the text look like hyperlinks.

(defun swap-out-latex-references ()
  (let ((tags-alist (scholium-text (get-article
    (map-label (lambda (name)
                   (let ((article (get-article name)))
                     (insert (scholium-text article))
                     (goto-char (point-min))
                     (while (re-search-forward
                             ;; I don’t think we use "xrefs" any more.
                             "\\\\\\(x\\)?ref{\\([^}]+\\)}" nil t)
                       (let ((target (cdr
                                      (assoc (match-string 2)
                         (when target
                           (replace-match target t t)
                           (let ((name-of-current-article name))
                              (match-beginning 0)
                              (+ (match-beginning 0) (length target))
                     ;; we don’t want these references to be fake
                     (scholium name
                               (scholium-about article)
Note 12.23 (On ‘import-scholium-system’).

For now, this is just ‘import-sections’, but it would be good if we could do something about sections that contain neither subsections nor notes, for example, the preface. Other nice features (e.g. creating “function type” scholia for functions, as in Note 12.44) should be added too, eventually. Sometimes work arounds can be applied for the time being.

Hint: I’ve found that for now one has to ‘(clrhash article-table)’ before running this function for things to work properly. I intend to address this issue soon.

(We might want to make this article use some kind of identification routine; alternatively, make it work with a “large structure” map of the document’s contents.)

(defun import-scholium-system ()
;  (label-article ’section ’major-articles)
Note 12.24 (Shorthand for editing the text of a scholium).

It would certainly be nice to have some short-hand from the "editing" section to make it so that we didn’t have to use the ‘scholium’ function every time we wanted to edit some article.

Note 12.25 (Autobacklink to label).

Also, it would be nice if the mere inclusion of “section” in the “major-articles” label meant that there would be a “parent” (back)link from “section” to “major-articles” (see Note 12.23 for the source of this example).

Such backlinks would let you navigate from things that bear a certain label to the label itself and thence to the peers within that label. (This certainly seems like a good idea!)

Note 12.26 (On ‘import-sections’).

For each section, import any notes and any subsections.

I’m going to try to get this to import front- and back-matter (before the first section and after the last note, respectively). This will make it easier to typeset everything with one command.

(defun import-sections ()
    (set-buffer "sbdm4cbpp.tex")
    (goto-char (point-min))
    (search-forward-regexp "^\\\\section{Prelude}")
    (goto-char (match-beginning 0))
    (scholium "Scholium system frontmatter"
              (buffer-substring-no-properties (point-min) (point))
    (while (re-search-forward
             "\\( +\\\\label{\\)?"
            nil t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (section-end (save-excursion
                             "^\\\\section{.*" nil t)))
             (notes-end (or (save-excursion
                               section-end t))
             (notes (let ((current-parent name))
             (subsections (let ((current-parent name))
        (when (not (equal tag ""))
          (scholium ’Tag-to-Name-converter
                     (scholium-text (get-article
                     ‘(,tag . ,name))))
        (scholium name
                  ’((section parent))
                  ’(section label))))
    (goto-char (point-max))
    (search-backward-regexp "^\\\\clearpage")
    (scholium "Scholium system backmatter"
              (buffer-substring-no-properties (point) (point-max))
Note 12.27 (On ‘import-subsections’).

For each subsection, import any notes and any subsubsections. notes are imported if they appear before the first subsubsection, if there is one, or before the end of the subsection, otherwise, or before the end of the section, if this is the last subsection

(defun import-subsections ()
  (let (subsections)
    (while (re-search-forward
             "\\( +\\\\label{\\)?"
            section-end t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (subsection-end (or (save-excursion
                                    section-end t))
             (notes-end (or (save-excursion
                               subsection-end t))
             (notes (let ((current-parent name))
             (subsubsections (let ((current-parent name))
        (when (not (equal tag ""))
          (scholium ’Tag-to-Name-converter
                     (scholium-text (get-article
                     ‘(,tag . ,name))))
        (scholium name
                  ‘((,current-parent parent))
                  ’(subsection label))
        (setq subsections
              (append subsections (list name)))))
Note 12.28 (On ‘import-subsubsections’).

For each subsubsection, import any notes that appear before the end of the subsubsection (if this is the last subsection, then before the end of the subsection, etc.).

(defun import-subsubsections ()
  (let (subsubsections)
    (while (re-search-forward
             "\\( +\\\\label{\\)?"
            subsection-end t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (notes-end (or (save-excursion
                               subsection-end t))
             (notes (let ((current-parent name))
        (when (not (equal tag ""))
          (scholium ’Tag-to-Name-converter
                     (scholium-text (get-article
                     ‘(,tag . ,name))))
        (scholium name
                  ‘((,current-parent parent))
                  ‘(subsubsection label))
        (setq subsubsections
              (append subsubsections (list name)))))
Note 12.29 (On ‘import-notes’).

This imports the notes at the current level in the section hierarchy.

(defun import-notes ()
  (let (notes)
        (re-search-forward (concat "\\\\begin{notate}"
                                   "\\( +\\\\label{\\)?"
                           notes-end t)
      (let* ((name (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (beg (progn (next-line 1)
             ;; no need to bound the search for the end, because we
             ;; assume that every "notate" environment is actually
             ;; closed
             (end (progn (search-forward-regexp
                         (match-beginning 0))))
        (when (not (equal tag ""))
          (scholium ’Tag-to-Name-converter
                     (scholium-text (get-article
                     ‘(,tag . ,name))))
        (scholium name
                  (buffer-substring-no-properties beg end)
                  ‘((,current-parent parent))
        (setq notes
              (append notes (list name)))
Note 12.30 (On ‘import-code-continuations’).

This runs within the scope of ‘import-notes’, to turn any Lisp chunks that follow a given Note into scholia attached to that note. (This won’t import the one stand-alone “verbatim” environment used in the code; we could change this function, or include the environment inside of a Note. Or we could just put the literate programming code into another file. Minor issue. Also, figure environments would be left out, except for a small trick, namely including them within surrounding Notes.) Notice that previous versions of these articles shouldn’t exist at import time, or Lisp sections will be doubled.

The code continuations and the code descriptions wind up being mutually about one another, which is kind of cute.

(defun import-code-continuations ()
  ;; ugly formatting = latex overrides!
  (while (looking-at (concat "\n\n\\\\b" "egin{lisp}"))
    (scholium ‘(code-continuation ,name)
              (let ((old-text (scholium-text
                                ‘(code-continuation ,name))))
                    (new-text (buffer-substring-no-properties
                               (progn (next-line 3)
                               (progn (search-forward-regexp
                                       (concat "\\\\e" "nd{lisp}"))
                                      (match-beginning 0)))))
                (if old-text
                    (concat old-text "\n" new-text)
              ‘(((article ,name)))
    ;; this should add an appropriate link to the article
    ;; that this is a code-continuation of.
    (let ((article-to-edit (get-article name)))
      (scholium (scholium-name article-to-edit)
                (scholium-text article-to-edit)
                 (scholium-about article-to-edit)
                 ‘(code-continuation (code-continuation ,name)))
                (scholium-bookkeeping article-to-edit)))))
Note 12.31 (A theoretical issue with code continuations).

I like to think of the notes as being scholia attached to the code, rather than the code as scholia attached to the notes. But the way the importing routine is set up, this relationship is reversed.

Note 12.32 (Rendering code continuations).

If we know what kind of code is stored in the code continuation articles, we may want to render it in the appropriate mode. (The use of connection coloration is disruptive to this kind of display, so we’d want to use some other mechanism for showing associations, if these need to be shown. See Note 2.29.)

Note 12.33 (Identification of notes is slow).

I tried identifying each note with its source region by running

 ‘(identifies-with ,name))

towards the end of ‘import-notes’, but it slowed everything down.

We may be able to get something similar by other means anyway (Note 8.50).

(defun browse-scholium-system ()
  (display-article ’section)
  (message "You’ve successfully imported the system!"))
Note 12.34 (Finding all articles with code continuations).

Cf. Note 6.42.

 (lambda (name)
   (let ((article (get-article name)))
     (and (member-if (lambda (link)
                       (typedata-includes (cdr link)
                     (scholium-about article))
          (typedata-includes (scholium-type article)

You can then run ‘listing-to-compilation’ to get a new document that drops all the interstitial chit chat found in this one.

Note 12.35 (Finding all articles without code continuations).

For our friends from the liberal arts,

 (lambda (name)
   (let ((article (get-article name)))
     (and (not (member-if (lambda (link)
                            (typedata-includes (cdr link)
                          (scholium-about article)))
          (typedata-includes (scholium-type article)

(Curiously enough, this document currently seems pretty balanced in that the code plus immediately important documentation takes up just about the same amount of space as the philosophical musings. Well, almost as much. And of course those verbatim environments are long.)

Note 12.36 (Finding all articles with emphasized parenthetical remarks).

Cf. Note 6.41. These are the articles that contain side remarks that should eventually be turned into scholia.

(progn (article-menu-list-articles-matching-regexp "\\\\emph{(")
       (listing-to-label "Comments"))
Note 12.37 (Exporting LaTeX docs).

This should give an inverse to importing LaTeX documents (Note 12.19). Exporting in one go is a cheap alternative to live updating via identifications (Section 8.4). Either would work, but exporting seems a little easier to get going quickly. (Incidentally, exporting is similar to producing a compilation; see Section 8.7.)

Note 12.38 (On ‘export-scholium-system’).

We use a similar sort of "recursive" style to that used in the import. First, loop through the sections. These must of course be exported in their own limited fashion. Then, for each, examine the contents. The items contained in each section are either subsections or notes. If the item really is a subsection, then do something similar to what we did with sections; otherwise, it is a note and we simply export it. The items at the penultimate level are either subsubsections or notes; and finally, in the end, the items are all notes.

Note that getting the “Tag-to-Name-converter” each time it is used in the child functions is really sort of excessive, since it could be found once in the parent. However, things seem to run fast enough.

(defun export-scholium-system ()
  (set-buffer (get-buffer-create "*Export*"))
  (insert (scholium-text (get-article "Scholium system frontmatter")))
  (dolist (sect (scholium-text (get-article ’section)))
    (export-section sect)
    (let ((contents (scholium-text (get-article sect))))
      (dolist (item contents)
        (let ((current-item (get-article item)))
          (if (typedata-includes (scholium-type current-item)
              (export-subsection (scholium-name current-item))
            (export-note current-item))))))
  (insert (scholium-text (get-article "Scholium system backmatter"))))
Note 12.39 (Child functions).

Notice the sort of interesting use of the words “child” and “parent” in Note 12.38; this is what comes of working on section listings most of the day. No reason that I can think of not to view code this way. Eventually we may want to parse Lisp functions and annotate them with hierarchies representing this sort of information.

Note 12.40 (Stable import/export).

The text, import, and export routines are all adapted to each other well enough to make it so that the text is stable under importing and subsequent exporting. We’ll have to do more extensive testing to be sure that it is also behaving properly with regard to intervening editing operations, but things are currently looking pretty good in this regard. (But be careful, and back up your work!)

Note 12.41 (On ‘export-note’).

This is for exporting notes. When the note has a code continuation it also exports that code.

To really work, this is going to have to replace references with \refs and \refs. Hopefully we’re storing the relevant information somewhere easily accessible at import time? We should at very least be able to do the opposite of what is done with ‘swap-out-latex-references’.

We essentially need to render the article to know what the relevant references are (and where they are). This suggests to me that if we don’t want to disrupt the user’s experience of the rendered articles appearing in the standard display, we should be able to “beam” a rendering to some other buffer and then pick it up from there. Also, if some references have been generated that don’t correspond to items with tags, we’ll have to accomodate that.

(defun export-note (note)
  (set-buffer (get-buffer "*Export*"))
  (let* ((name (scholium-name note))
         (tag (car (rassoc name (scholium-text
    (insert "\\begin{notate}{" name "}"
            (if tag
                (concat " \\label{" tag "}\n")
            (swap-in-latex-references note)
    (dolist (scholium (mapcar (lambda (backlink)
                                    (get-article (car backlink)))
                              (get-backlinks name)))
      ;; we were missing one function from a list of two in my last
      ;; test of exporting, should look into this.
      (when (typedata-includes (scholium-type scholium) ’code)
        (insert "\\b" "egin{lisp}\n"
                (scholium-text scholium)
                "\\e" "nd{lisp}\n\n")))))
Note 12.42 (On ‘swap-in-latex-references’).

Maybe it would be easier if each of the references had the name of the tag stored locally? This doesn’t seem like it would actually help. The point I guess is that we don’t want to detect the scholia structure all at once, but rather, zip through and make changes to each item we encounter, in order. I.e., the endpoints of the regions that we’ll be modifying are always in transition.

‘move-to-next-region-with-scholium’ will have to be set up to return ‘nil’ if there is no next scholium. It would also be nice if this returned the boundaries of the region to which the scholium is attached, when there is a scholium, since we’re going to want to do a replacement of the text in that region. (If it proves more reasonable to use a different function for this purpose, that would be fine, but this is roughly what it should do.) Note that ‘move-to-next-region-with-scholium’ currently works in terms of overlays, which isn’t really right.

After we find the scholia at the start and end of the marked region, we could check a complicated condition to see if there is a reference at the beginning and another copy of the same reference at the end. But for the time being we aren’t going to work with this general case of overlapping references, and we’ll just assume one reference per marked region and get on with things.

(defun swap-in-latex-references (note)
    (let ((tags-alist (scholium-text (get-article
      (display-article (scholium-name note))
      (set-buffer (get-buffer "Main Article Display"))
      (let ((next-region (move-to-next-region-with-scholium)))
        (while next-region
          (let ((scholium-property-at-start
                 (get-text-property (first next-region) ’scholia))
                 (get-text-property (second next-region) ’scholia)))
            (let* ((scholium-id (first scholium-property-at-start))
                   (scholium-name (first scholium-id))
                   (possible-reference (get-article scholium-name)))
              (when (typedata-includes
                     (scholium-type possible-reference)
                (let ((scholium-tag
                       (car (rassoc (reference-to-article
                  (if scholium-tag
                      (progn (delete-region (first next-region)
                                            (second next-region))
                             (goto-char (first next-region))
                             (insert "\\ref{" scholium-tag "}"))
                    (let ((new-tag (replace-regexp-in-string
                                    " " "-"
                                     (first next-region)
                                     (second next-region)))))
                      (delete-region (first next-region)
                                     (second next-region))
                      (goto-char (first next-region))
                      (insert "\\ref{" new-tag "}"))))))
            (setq next-region (move-to-next-region-with-scholium)))))
      (buffer-substring-no-properties (point-min)

(defun export-section (section-name)
  (set-buffer (get-buffer "*Export*"))
  (let* ((tag (car (rassoc section-name (scholium-text
    (insert "\\section{" section-name "}"
            (if tag
                (concat " \\label{" tag "}\n\n")

(defun export-subsection (subsection-name)
  (set-buffer (get-buffer "*Export*"))
  (let* ((tag (car (rassoc subsection-name (scholium-text
    (insert "\\subsection{" subsection-name "}"
            (if tag
                (concat " \\label{" tag "}\n\n")
  (let ((contents (scholium-text current-item)))
    (dolist (item contents)
      (let ((current-item (get-article item)))
        (if (typedata-includes (scholium-type current-item)
            (export-subsubsection (scholium-name current-item))
          (export-note current-item))))))

(defun export-subsubsection (subsubsection-name)
  (set-buffer (get-buffer "*Export*"))
  (let* ((tag (car (rassoc subsubsection-name (scholium-text
    (insert "\\subsubsection{" subsubsection-name "}"
            (if tag
                (concat " \\label{" tag "}\n\n")
  (let ((contents (scholium-text current-item)))

    (dolist (item contents)
      (let ((current-item (get-article item)))
        (export-note current-item)))))
Note 12.43 (Name conflicts).

As we talked about vis a vis namespaces (Note 2.37), it would be nice if we could deal with multiple different things having the same name, say a Section and a Note, or two Notes in different sections. A disambiguation page could be created whenever the ambiguity needed to be paid attention to.

Note 12.44 (Revise to support “function” type articles).

Instead of referring to Notes that describe functions, sometimes we may prefer to refer to “function” articles, which will include the function and the note that describes it.

Note 12.45 (Group functions sharing scope).

Objects with type functions could have “runs within the scope of this function” slots. This would facilitate propagating changes in notation.

Note 12.46 (Other features for function articles).

As top-level objects, functions could be fitted out with their own fora and FAQ’s. They could be displayed with the appropriate mode turned on by default.

A short document containing only the functions that one had been working on recently could be created easily; see Note 8.53 and Note 12.53.

12.4 On-the-fly scholium creation

Note 12.47 (Why on-the-fly?).

This section is about creating scholia automatically. For example, every time I type “scholium system”, maybe something happens – for instance, maybe the text gets marked up with a reference to some other object describing the scholium system. This is sort of like what happens in Noösphere, though there, auto-linking happens when the document is rendered.

Another similar thing is the auto-indexing feature present in Apple Computer, Inc.’s latest OS, wherein content is automatically indexed as it is being created. We could do something fairly similar with the scholium system (and some of this indexed information could be shared with other collaborators whenever there are multiple people using the scholium system together).

Another example use would be to display interesting feedback about things you had just typed – maybe related definitions or phrases, or some other cool stuff. Doing this on an automatic on the fly basis is a minor simplification (from the user’s point of view) over doing it only when the user issues an explicit command.

Obviously you don’t want to search every editspace. For example, displaying a definition as a scholium means that you would probably just want to search the dictionary namespace. This is not the same as creating a scholium, indeed, the dictionary may not “really” contain scholia at all; you can use the structure of the dictionary to easily retrieve information that you would like to superimpose on the current document. (This is a somewhat traditional sounding digital libraries sort of thing.)

Note 12.48 (Compare flyspell).

Some good ideas might be available in the flyspell package for Emacs.

Note 12.49 (Automatically adding files to the article list).

We can also set things up so that visited files are added to the article list automatically, etc.

Note 12.50 (Lexical autolinking).

One major application of this system would be to create Noösphere-like autolinking. Thus, part of this “experiment” is to reimplement autolinking in this model. Note, that once you have implemented linking within the model, then autolinking should probably be “easy” to add by reimplementing the Noösphere algorithm. One advantage of doing things this way is that links appear at edit time rather than render time (presumably anyway), which means that they can be tinkered with right away as needed.

I wonder if the autolinker could be used to make an index of terms automatically, too? Seems likely.

12.5 Simulating a wiki with the scholium system

Note 12.51 (Build scholia by first adding markup and then parsing).

The idea in this section is to assemble a collection of scholia that simulate the page-reference markup found in a wiki. The strategy we use is to read the wiki page and add text properties to the “alt text” indicating the link targets. We then parse the buffer and create scholia about the regions that bear text properties. The reason for going through this two-step process is that the position of text in the buffer will be shifting around as we get rid of wiki markup, so it is convenient to add scholium system markup in a format that in invariant under such shifting. To the extent that we need to specify type data, we overload the name data that we use for initial markup. For example, “(external-page <page-name> <URL>)” is the format we’d choose as the name of scholia representing references to external pages.

Note 12.52 (On ‘mark-up-wiki-with-reference-scholia’).

Find things in double brackets, possibly with an intervening “|”, making sure to find the minimal double bracket pair. What should happen (though the current draft is a bit different) is: a reference scholium should be created for each reference, and stuck into the article list in some appropriate place (presumably the namespace associated with the article in question, more specifically, on some specific reference subspace of that space). Then when someone goes to render the document in question, we grab the references out of their storage facility and mark up the document with text properties. (The current implementation is really just a regexp check…)

(defun mark-up-wiki-with-reference-scholia ()
    (goto-char (point-min))
    (while (re-search-forward
            ;; we could probably get all types of reference in
            ;; one regexp.  I’m not sure why the old
            ;; version for grabbing external references was deleted,
            ;; but it is featured in the Sept 4 printout.
            "\\[\\[\\([^]|\n]+\\)\\(|\\)?\\([^]]+?\\)?]]" nil t)
      (replace-match (propertize (or (match-string 3)
                                     (match-string 1))
Note 12.53 (Recent changes and recent referents).

It would be handy to have some code that would list the articles that have had new scholia written about them. This would help a user (e.g. the author of the articles under discussion) see what people are talking about. Some additional semantics could be provided, to list recent modified versions, which are similar to document changes, but arguably more relevant in a system with object ownership. Actual changes should be listed too.

Note 12.54 (Handling references).

(1) Do special markup; (2) What is the format of a reference precisely?; (3) Do they have backlinks?; (4) Notice that they will probably need to have a special markup property associated with them (instead of the typical ‘scholia’ property); (5) Note that we might want to record all the references associated with a given buffer on its metadata article, as well as on the full list of articles and a ‘reference’ label (unless we were to just make a reference namespace).

Note 12.55 (Strikethrough).

One of the most important things you can do with a wiki is to delete things that other people have written. Showing “deletion” as a sort of scholium would be a handy feature for a wiki modeling experiment.

Note 12.56 (Delete/insert patches).

Note that patches that delete or insert changes are something like “quoted” changes to files; i.e., as opposed to the “direct” changes that occur when editing.

12.6 Using the scholium system to do AI stuff

Note 12.57 (Take an example from a known system).

It would be a good idea to find at least one example from a “standard AI” system (KM’s chemistry-problem solving, for example) and import and otherwise emulate it in the system here. We’ll have to look around for a good example.

12.7 Using the scholium system for HDM things

Note 12.58 (What is HDM and how does it relate to the scholium system?).

The HDM (expanded: Hyperreal Dictionary of Mathematics) project is about taking the world’s mathematical knowledge, and getting it into a format that is as useful as possible.

The key point to make here is that HDM will be a free culture artifact containing lots and lots of mathematical information (definitions, theorems, proofs, expositions, examples, and excursions, etc.) that will need to be organized for use by both individual humans and networks of machines. The scholium system is part of the HDM project because of these organizational needs – and (in particular) it is being designed in such a way that it should be able to satisfy the “AI needs” of the project.

The revolution will be computerized – and it will be total.

Note 12.59 (Application of the scholium principle for HDM).

We should talk about encoding different types of objects in the system and showing their relationships using typed links; also about assembling reports via transclusion, and so on and so forth. We should also show how this system can be used as a stand-in for more “traditional” KRR systems (SNePs, KM), and how reasoning modules can be implemented for the system. But this is pretty advanced stuff, and it is going to be a little while until we get to it.

In the mean time, we have various fun things (a little music and even less Go), and we can think about the relationships to Dirk Gently and so on and so forth.

Note 12.60 (Template utilities as an example).

Ray was working on a set of template utilities that could be encoded as a scholium-based document.

Note 12.61 (Overview of Arxana as an example).

It might be nice to create a simplified version of the current document that would deal with the core functions and ideas only. This would presumably help people learn how to work with the system.

12.7.1 Managing the contents of a mathematical dictionary

Note 12.62 (Importing the APM-Xi).

See 12.67 (currently the main function in this section, although we may come up with some additional functions later on). Actually, we should probably also import the APM, however, I’m not completely sure I can get copyright waivers for the APM’s content. I hope so; I should investigate that. If I can get written permission, then I suppose we should put APM into the HDM’s archive along with APM-Ξ.

Note 12.63 (Large-scale structure of APM and APM-Ξ).

We have some words and phrases. These phrases come up in the APM in various places. Each phrase has a list of problems that it is associated with. It also has a list of definitions that it is associated with. For APM, we would want the relevant definitions to appear as attached scholia. For APM-Ξ, we presumably want a namespace for each phrase, into which we put each of the definitions as its own article, together with the specially-designated list of associated problems. Here it seems like we have the opportunity to use links with different types. (Maybe we would just attach the list of associated problems to the namespace as a whole.) To parse the document this far, the first thing to do is to break it into sections, then parse each of these sections into lists of problems.

Note 12.64 (Types for the APM-Ξ).

We keep track of the sections, entries, and stubs that belong to APM-Ξ as special types.

(add-to-list ’modified-type-labels ’(apmxi-section . apmxi-section) t)
(add-to-list ’modified-type-labels ’(apmxi-entry . apmxi-entry) t)
(add-to-list ’modified-type-labels ’(apmxi-stub . apmxi-stub) t)
Note 12.65 (On ‘make-apmxi-sections-into-articles’).

This function makes every section in the APM-Ξ into its own article. The list of these articles is recorded on the article ‘apmxi-section’.

(defun make-apmxi-sections-into-articles ()
    (find-file "../p2/Xi.tex")
    (goto-char (point-min))
    (while (re-search-forward "^\\\\section{{ \\(.+\\)}} " nil t)
      (let ((article-name (match-string-no-properties 1))
            (beg (point))
            (end (save-excursion (search-forward "\\section" nil t)
                                 (match-beginning 0))))
        (when end
          (scholium article-name
                    (buffer-substring-no-properties beg end)
Note 12.66 (On ‘chunk-out-apmxi-definitions’).

Apparently some of the terms in APM-Ξ were defined in multiple sections. At least, stubs appear in multiple places. This should presumably create multiple definitions when multiple definitions are given, or get rid of stub-like definitions, or something.

Note that the way this is set up, even sections that contain only stubs can also contain references to the real articles, when they exist – this means that the real articles may appear in several listings, which I think is as it should be.

(defun chunk-out-apmxi-definitions ()
  (map-label (lambda (name)
                 ;; this ensures that sexps are defined properly
                 (let ((topic (get-article name))
                   (insert (scholium-text topic))
                   (goto-char (point-min))
                   (while (re-search-forward "^(" nil t)
                     (let* ((beg (match-beginning 0))
                            (title-beg (match-end 0))
                            (title-end (progn (goto-char beg)
                                              (1- (point))))
                            (stub-p (looking-at " *$"))
                            (title (buffer-substring-no-properties
                            (end (save-excursion
                                   (search-forward-regexp "^$")))
                            (possible-previous (get-article title)))
                       (when (or (not possible-previous)
                                 (typedata-includes (scholium-type
                         (scholium title
                                   ‘((,name parent))
                                   (if stub-p
                                       ’(apmxi-stub apmxi-entry)
                       (setq contents (cons title contents))))
                   (scholium name
                             ’(apmxi-section label)))))
Note 12.67 (On ‘import-apmxi’).

This function puts into play all of the other functions developed above, to fully process the APM-Ξ and get it nicely situated in the scholium system.

(defun import-apmxi ()
Note 12.68 (Special reference types).

In the mathematics domain, would be handy to have “related to”, “depends on”, “proof of” and “prerequisite for understanding” reference sub-types. It is easy enough to create a subtype – just add the secondary category to the type variable along with the “reference” datum. Special types of references are one of the key semantic network-like features of the scholium system (see also Note 3.39). Maybe the reference-creating facility should provide an optional second argument for specifying reference sub-type.

Note 12.69 (Selecting all APM-Ξ stubs).

If you want to look at a list of all stubs in the APM-Ξ, run this code:

 (lambda (name)
   (when (typedata-includes (scholium-type (get-article name))

(See Note 6.42.)

Note 12.70 (Screening out APM-Ξ stubs).

If you want to screen out the stubs in the APM-Ξ, run

(display-difference-of-labels ’apmxi-entry ’apmxi-stub)

(see Note 5.100). Of course, eventually we’re going to want to do some more processing to make the stubs into useful things.

12.7.2 Facilitate collaboration between HDM authors

Note 12.71 (Use a simple scheme).

The collaboration scheme is described in Section 10.3. I might try to make that version “general” and in this section implement some “specifics”.

12.8 Model a physical space and its contents

Note 12.72 (Holtzermann Kitchen Companion).

I’ve been thinking that it would be handy to have a full inventory of the kitchen I share. The inventory, or model, really, could be shared with the kitchen-mates and it could help us remember where things go and what needs to happen around the place. More generally, the idea would be: to help improve communication about goings on in this shared space.

Note 12.73 (Typelock).

In order to talk about the contents of cupboards versus the refrigerator, for example, the user should be able to specify a bunch of ‘(contains A B)’ relationships. Both ‘contains’ and ‘A’ will stay fixed for reasonably long periods of time.

In addition, probably everything that is created in this experiment should have a kitchen-element type (and label). Thus, we want some sort of typelock at the level of object creation and link creation. (Indeed, we could probably set things up so that the links didn’t need to be specially-typed, but rather were just part of some specially-typed scholia.)

A sort of segue away from the main topic: it could be handy to optionally display all of the ‘contains’ items using the listing mechanism instead of the standard scholia display.

Note 12.74 (Case for namespaces).

Both the sink cabinet and the doorless cupboard contain a “drawerful of assorted utensils”. Of course, these aren’t the same drawers (nor are they the same utensils). We find the same sort of thing in earlier imports of the scholium system, in which several sections had “Further notes” subsections. As a work-around, all of these were given distinct names, but subsequently, especially when working with complexes of formulaic scholia (Note 3.40), that behavior wouldn’t be desirable.

12.9 Make more physical scholia

Note 12.75 (Extended rendering).

The scholium-based document idea can be extended outside of the computer. The examples that are produced this way can help illuminate and inspire out work. See Note 5.14.

Note 12.76 (Math paper as gallery tour).

We could put pictures from a math paper on the walls of a gallery, with the space divided up in such a way as to have each wing correspond to a section of the paper.

Note 12.77 (Scholia on architectural space).

A generalization of Note 12.76 is to put together spaces that incorporate scholia into their surfaces (graffiti, chalkboards), or into their basic design. Memory spaces (mnemonic architectures) are spaces with double (or more) meanings, i.e., multiple sets of scholia on the same space.

Note 12.78 (The HPDA).

This is a somewhat famous example (among a certain geek crowd) example of a scholium-based document system. I invented a game that goes along with it.

Note 12.79 (Printing things out).

Questions as to the impact and relevance of putting things on paper continue.

Note 12.80 (Node-based systems).

Systems like freemind2222 22 aren’t exactly “physical” in the sense of something you can handle, but they provide a different sort of view into a document, and views are physical things. (It seems to that freemind itself only provides hierarchical views, which isn’t particularly pleasant for us. I know that the KM people had a tool that looked good for more complicated views, but I’m also pretty sure they didn’t want to release this tool to the public.) Some sort of nice graphical tool could be used to make a schematic of the scholium system that people might find useful.

Note 12.81 (Outline management).

In Table 2, we illustrate line-numbers 1–5 at left, items a–e at right, together with labels αϵ. We envision this outline changing over time, and references such as \itemref{α} consistently producing the correct line number. I think LaTeX can already handle this sort of thing, I just forget how you do it.

1 a (α)
2  b (β)
3 c (γ)
4  d (δ)
5   e (ϵ)
Table 2: A scheme for outline management
Note 12.82 (Offline usefulness).

Printed documents are only useful offline; the question as to whether they are more useful offline than a non-networked computer remains. Sometimes they will have a certain pragmatic edge, e.g., they allow one to avoid lugging the computer from place to place. On the other hand, one typically then must type up one’s handwritten comments.

(The problem of integrating interesting documents into Arxana is present whether or not these documents are physical.)

Note 12.83 (Getting jobs done in different ways).

Many of the things we talk about in Arxana (and in the HDM project in general) have to do with finding new ways to get jobs done which we already can get done in some other way. The hope is that the new ways of working will allow us to be more efficient or have more fun or that they will otherwise be beneficial. (It isn’t completely clear that you really do get the same job done if you do it in a different way.)

13 Philosophy of the scholium system

Note 13.1 (Code).

The concerns mentioned in Note 1.1 are closely related to the questions and ideas of ‘‘Code and Other Laws of Cyberspace’’2323 23 Cf. Note 3.64.

I’d be interested to know whether pass-by-reference as a basis for document exchange doesn’t provide a final solution to the copyright problem. OK, almost final; you still need some high-tech system for managing the references and finally retrieving and operating on content.

Note 13.2 (An observation).

There is a lot of room in cyberspace.

Note 13.3 (Definitions of freedom).

In Note 2.2, I talk about the importance of user freedom. I should go into this more. For now, I’ll just say that for me “user freedom” in some sense includes the notion of “user convenience.” Why? Because what is freedom to do if it doesn’t go along with actual ability to do? Empty! I call the basic notion at work here the “liberties model of freedom.” (With all due apologies to The Dead Kennedys.)

(Note that the guaranteed freedom to do something that one might potentially be able to do at some point but can’t do just now isn’t empty, it’s more like “half full”.)

Note 13.4 (“Free culture” is ambiguous).

We should be careful when using this term to make sure it is really defined properly, something I don’t think anyone really has done yet. I’m not really cool with using terms that aren’t defined. But then I get caught up using the term. So, I’m not sure. I’m guessing that there is a false dichotomy here similar to the dichotomy between “text” and “hypertext” – in other words, something that could perhaps be formalized, but which as yet hasn’t been (see Note 13.25). My sense is that “free culture” may be as ambiguously overloaded a term as “intellectual property” (just for example). And yet, freedom seems to be an important part of a useful system. We proceed with caution.

Note 13.5 (“All information is superimposed.”).

Statements are superimposed on an author’s perceptions and inner life. Perceptions are superimposed on the base data of experience (and are a part of this data). Everything we identify as “information” is superimposed.

Note 13.6 (What, then is at the bottom?).

Is there a bottom, a “real” reality, even if it is an extra-informational one? Or is reality as we know it a mutually agreed-upon set of mutually-referential superimposed bits? (And for that matter, is there a top, and what would that look like?)

Note 13.7 (A possible interpretation of physical scholium systems).

The physical systems of Section 12.9 and elsewhere show that physical systems can “do” simulation. Compare also Alan Moore’s “Supreme”, a fictional character who is somewhat aware of the effects of being as such, although not quite to the point of being fully cognizant of the fact that he is fictional. Physical scholium systems are only partially aware of their computational possibilities (and, mutatis mutandi, computational systems don’t always fully take into account their physical manifestations, whether concrete or only barely imaginable).

Note 13.8 (Making things easier to learn).

If subject matter to be learned is embedded in a scholium system, questions and answers that people have asked already would tend to be easily accessible and clearly organized. It would also be easy to ask new questions in the context in which the right people will take notice.

Note 13.9 (The extensions of man).

What are the things you can do in a given medium? What can you point at (indicate)? What sorts of gestures can you make? See Note 14.6.

Note 13.10 (Designing a free code market).

A “consortium” of free software supporting companies would be similar to the scholium model of business and relates to FSB. It is important to know what is in demand and what the supply and costs are.

Note 13.11 (Scholia in “The star’s my destination”).

The modifications to the main character’s teeth are an example of actionable scholia. The tattoos are another obvious example, adding art on top of skin. This makes me think that perhaps the issue of scholia also comes up in the book by Dumas that this one draws so heavily upon. The SF book might also be considered to be a sort of scholium attached to Dumas’s “The Count of Monty Cristo” (cf. Note 12.8).

Note 13.12 (Anthropogenic change).

Changes effected by humans upon the world might be considered to be scholia. Changes effected upon ourselves (whether skin-deep or destiny-changing) could be as well. But this vision of anthropogenic change is also somewhat anthrocentric. Humans are just part of a much broader world. Our “will” and “agency” are brought into question in the context in which “everything is connected.” (Compare “Les Enfants Terrible” on that one.)

Note 13.13 (Document closure).

Wittgenstein said that the world is all that is the case. Baudrillard said that the world is that which can’t be exchanged for anything else. A document that contains references to things outside of itself is really an article in a larger document.

Note 13.14 (When is a document finished?).

The question as to what needs to go into a document before the document is finished has probably been thought about by many authors and editors in the past. Some things are bound to be off topic, whereas other things are bound to be completely critical. Some things are of only peripheral importance.

Note 13.15 (Different kinds of documents).

Different kinds of documents correspond to different sorts of context – spatial, temporal, ideational, etc. (see Note 5.17). Note 13.16, Note 13.17, Note 13.18, and Note 13.19 talk about several different kinds of documents. See also Note 5.68 for some further words about different kinds of context.

Note 13.16 (Document as container).

Each document or article contains statements. Sometimes these statements can be bonafide objects, for example, each section in this document contains Notes and certain other objects. “Frames” and “ontologies” fall into this category (and presumably so does the notion of “category”). This model seems particularly well-adapted to the notion of transclusion (and vice versa), and to the idea of a document as the expansion of an outline. Comments about the document as a whole should apply to the containing object (so that in some cases, comments about the document as a whole come from within, i.e., comments need to be contained somewhere; see Note 13.13). If we set the current document up in the scholium system on public display (and for public comment), we would probably want to have a special place for general comments about the article. This relates to the issue of namespaces and labels (Note 2.37).

Note 13.17 (Document as thread).

Sometimes a document takes you along a linear path through a more complicated space (see Note 8.68 for a description of one way this can work). Frequently authors don’t know where their writing is going when they start. Franz Kafka and Stephen King are notable proponents of this style of writing, which is almost the antithesis of writing from an outline. Whether stitching a specific pattern or exploring a strange labyrinth, the writer’s experience has several threadlike qualities. When there are many authors, then the metaphor is more obviously one of weaving many threads. Of course, individual threads and an overall threadlike quality writing only appear under certain circumstances. (I wonder what Dijkstra said about writing. One thing I know is that his thought about writing proofs was that each step should be made clear; the reader shouldn’t be left guessing.)

Note 13.18 (Document as process).

The view advanced in Note 13.17 admits, but does not necessarily imply, the view of the document as a process. A thread or body of woven threads could exist outside of the process proper; in this model, the author might follow the thread along to its conclusion (like a detective “unraveling” a mystery). It seems apparent that a document, whether it is written by one or many authors is a process, at least until it is “finished”. Even then, though the central text becomes fixed, the body of peripheral texts continues to grow, as the central text is reinterpreted in new contexts.

In order to evolve, documents typically need to interact with the world somehow. The schedule of interactions could be arbitrary, but when humans are involved some sort of phasic structure is often involved too. Laying down the law with some sort of schedule for work on this document might be something I could make myself do (whether or not keeping the schedule is also is not yet decided, and could be interesting to document either way). Personal schedules and organizational schedules are related. I really don’t know what to do (when things grind to a halt, but I assume it would be a good idea to do something.

Note 13.19 (Document as collection).

Maybe the evolution of the desktop that we hear so much about (or the evolution of email, a la Gmail) is going to carry over to documents as well. Instead of being organized into sections, documents could just be miscellaneous collections of notes, which then take on certain organization through search (or maybe some fancy self-organizing phenomena) as needed.

Note 13.20 (Document as ontology).

Documents typically come in an outline-oriented fashion. The idea of “document as ontology” just expands on the idea of “document as container” and posits some nested containers. I’m not actually sure what the difference between an “ontology” and a set of nested sets is; my sense is that ontologies will typically also specify interrelationships beyond just “is-a”. It would be worth looking into this more.

Note 13.21 (Document as simulasm).

Eventually when one has abstracted out the author a document takes on the appearance of an agentless simulation.

Note 13.22 (Unlimited refactoring).

The system seems to go one level beyond “refactoring” to something more like “multiple on-the-fly as-needed refactoring.” Code will take whatever shape you put it in temporarily; however, these shapes can also be saved. Maybe this is what refactoring is already about in sophisticated systems (on a wiki it means something much weaker). This would be worth investigating.

Note 13.23 (Assimilating data).

We learn to map data “up” (local) hierarchies. For example, when we make lists of things to do, we map observations up from a perceptual space to a potential action space. It is valuable when living and writing to have good heuristics for figuring out the best level for observations. (Some should be simple data, some should be containers for other sorts of data.) This also relates to the issue of text and code from Note 13.24.

If the document is a process (Note 13.18), then it may sometimes be a “process of understanding”, or a trace thereof. When we think about something, we may be thinking in terms of an internal “scholia-based document”.

As I’ve been working on this document, various ideas have occured to me, and they make their way in as marginalia, then as Notes, and perhaps more expanded or refined.

It would be nice to have a specific reference to something from “psychology” here – the On Intelligence book would work, even though it could be a better book. Note that the Minsky book contains similar (but more theoretical) notions.

The idea of “hooking” from Note 13.30 seems related – both with regard to expressing assimilated knowledge by supplying suitable hooks, and with regard to figuring things out by hooking them together or hooking them to things that we already know.

Note 13.24 (Text and code).

What items should be coded up and what should be written about? This question seems particularly on-topic for a literate program. (See Note 2.12.) Historically, many thinkers were not able to write (useful) programs, because there weren’t computers to run the programs on. But algorithms have been being designed for a long time. If Newton had kept calculus on a completely abstract and philosophical level, it wouldn’t have been useful for computation. On the other hand, I have very little idea how the phenomenologists could have gone about coding up their ideas. Even though now, work in knowledge representation apparently follows in their footsteps. Part of the issue is that there are different languages for different things. People frequently work on the “conceptual” level, and they need things to be narrated to them. Programs typically only “work”, if they come with some suitable narration, either intrinsic or extrinsic. The actionable-nonactionable dichotomy may not be so useful (compare Minsky on dumbbell theories). The notion that things ‘‘evolve’’ from nonactionable to actionable is probably a somewhat misleading, as appealing as it might be. Sometimes we need to ask: what programs must be written to understand this question? To paraphrase a correspondent of mine (Ray Puzio), text and code are like yin and yang (cf. ‘‘chaos linguistics’’2424 24 Understanding how they relate to one another in more detail would presumably be quite fruitful.

Note 13.25 (Hyperstuff).

Text is (or can be) hypertext; mathematics is just as “hyperreal” as the HDM. But I suppose that with some work towards formalizing a definition the “hyper” prefix might give us more than just hype about a quote-un-quote new technology. (As an example of how these categories run together, Note 8.1 contrasts hypertext-like indices with more text-like compilations.)

Accordingly, you can build an AI out of all sorts of miscellaneous components. Mathematics is already very AI-like, even though many of its agents are human beings. It isn’t completely clear what the differences there are between working with systems with more electronic components and systems with less. Is, or, how is, one set inherently more powerful, or more risky, or just better for different things?

Compare Note 5.14.

Note 13.26 (Not-so-holistic mathematics).

The QA section is typically split into two or three parts, each found in a separate library. Here in Minnesota, there is the math library, the math philosophy section in the humanities library, and the engineering library. It can be somewhat inconvenient and a bit offputting for people who want to look at math from all of these perspectives. On the other hand, it frequently makes sense to divide when resources can’t be copied. Note that digital libraries don’t need to be divided in a permanent way; cf. Aaron’s Emory paper – in both positive and negative senses – in particular, think about Google, today’s great integrator. Compare Leibnitz on the university and Minsky on the mind.

Note 13.27 (Philosophy of intelligent documents).

The main thing to look at here is the Dreyfus and Dreyfus article in The Artifical Intelligence Debate. The ideas in the current document (e.g. in Note 14.6) should give some ideas about why there is hope for AI, and why we can make something that people like to use as we make progress towards this goal.

Note 13.28 (Usefulness of bidirectional linking).

When a note contains a link to a section that contains more information on the topic of that note, this gives the author a way to have some of this topic spread into different contexts, but also to have a compressed treatment that reflects all the different ideas about the given topic in the document as a whole. Bidirectional links can be used to facilitate this bifurcated style of writing.

Note 13.29 (Being “meta”).

One way to be “meta” is to loop through some dimension that was previously held static. Just for example, we might switch from single-article rendering to multi-article rendering at some point.

Note 13.30 (Thought processes).

It is interesting how different pieces of information will attach themselves to a train of thought – like so many carriages. Or like the miscellaneous stops on a messenger race.

In such races, there is actually no “message” to deliver; rather, the racers must simply visit several predesignated stops, and sometimes perform a certain task at each – like drink a beer, for instance. Typically these stops can be visited in any order.

Like post facto routes, thought process seems to be exploring a large web-like space – and only finally getting pulled together into one narrative thread. Perhaps one unwinds an Ariadne’s thread as one explores; and perhaps at each important stop a hook is added, so that when we get to the “end” of the path, we pull the Ariadne’s thread, and the hooked-in items from hither and yon are present before us, ready in salute.

This business with the Ariadne’s thread is pretty amazing, but the hooks, by contrast, seem pointedly quotidian. They include such drole things as word-association, metaphor, alliteration, and various other tropes.

Some thoughts are fleeting and remain between the lines forever (“you should have seen the one that got away!”). Is this a problem? Perhaps less so when you are focused (Note 13.37).

Note 13.31 (Mnemonics).

The fact that humans can often improve their ability to remember things maybe means that we shouldn’t trivialize memory-hooks too much. Ultimately, however, it seems like we do run into “wetware limitations”.

Note 13.32 (Search).

Being able to search is a very nice feature of the computer interface. But of course, it is also possible to use thought processes (Note 13.30) to search. (If it wasn’t, we’d almost surely be utterly screwed.)

Note 13.33 (Experience).

Experiences give you things to point at or hook to. They are sort of like pre-computed cases – you can save time and be secure by knowing what to do in a given situation in advance. However, our experiences are almost always (or maybe “always”) only heuristically applicable.

Scholia, too, are sort of like pre-computed cases; i.e., they document the experiences of persons who have interacted with a given subject.

Note 13.34 (Establishing setting).

In filmmaking (and fiction writing), there is a tradition of images that are only secondarily part of the story per se, but which, rather, tend to say where the story takes place, and to lend it some realism (or similar quality of substance).

These images are sort of like attached scholia; they could be cut and the plot would remain. However, we often see films that have been “ruined” by overzealous cutting.

Note 13.35 (This document as an example of CBPP).

This document will hopefully be CBPP (“for real”, and not just via the Swift-Minsky societies of my mind). The transition from solo work to group work is probably almost always rocky, and sometimes even too off-putting for it to happen at all. However, there are certainly some steps that can be taken to ease the process along – it is possible that having a “cool” product helps (some would argue vehemently that the cool factor is unimportant to the extreme).

Hopefully by trying to make the code and documentation as accessible, starting with some discussions about the system, moving to discussions within the system, and from there soliciting code, etc., things will work out for the current current project.

Note 13.36 (Freedom, bookmaking, and book-buying).

In my view, part of the idea of paying for copies should be that the copy itself is a work of art. If the copy is simply something you could run off yourself on your laser printer, it might not really be worth buying. If it is a limited edition artwork – and if the price is right – then it could be very much worth buying. On the other hand, if the salient features can be simulated cheaply, then even the “unique” artwork may not be very valuable.

This relates to Section 12.9. The value of print versions of a document is an interesting psychological question. Writing marginal notes into a printed document seems to be a perfectly viable way of building meaning clusters, but it isn’t clear whether it has any specific psychological benefits over writing marginal notes into a sufficiently robust electronic system.

Note 13.37 (Staying focused).

In a scholia-based document, it should be somewhat obvious when you drift off subject or off task. Sometimes the connection between the “off subject” elements of the document and the “on subject” elements comes through a “meta” sort of connection. For example, I initially worte this Note as a reminder to myself to stay focused on the code I’m looking at today, and not to get distracted by too many peripheral or minor concerns. Even though this Note is peripheral with regard to the semantic content of the code, hopefully it will have a salutory “meta-level” effect on the coding process.

Note 13.38 (Hetrarchy).

If it isn’t obvious, a hetrarchy is an example of a scholium-based document in which the articles are people and the attachment relationships are given by who is working on whose project. It shouldn’t be too hard to map out these sorts of relationships in a given CBPP system (like PlanetMath, for example).

Note 13.39 (Hierarchy).

Of course, a hetrarchy (Note 13.38) can be viewed as the joining-together of several hierarchies. It seems helpful to be able to look at the local hierarchies associated with certain articles. Who cites who, and who do they cite? – and so on.

Note 13.40 (Scholia-based digital libraries).

Some of the relationships to Digital Libraries are as follows.

In a physical library, marginalia are considered to be graffiti. In a digital library, they can easily be ignored, if that is desired. They are more like sticky notes than they are like true marginalia, but even more potentiallly-ignorable than these handy-dandy things.

Still, the fact of the matter is that adding scholia to a document will always change the document’s meaning in some sense. However, in the case of private scholia, other people will be in the dark as to the nature of the change. There is a gradient here; some scholia. may be “so public” that everyone sees them, no matter how their filters are set. (E.g., the user interface to monolithic operating systems.)

In a physical library, loud conversation is considered to be disruptive. In a scholia-based document, it should be possible to mark conversations with metadata that will make them more or less private. (Conversations are an example of processes-as-scholia.) In a typical feedback-forum situation, user feedback isn’t actionable. But of course scholia can be made actionable – and they need not be favorable. For example, scholia can be used to vote such-and-such an item out of the collection.

Note 13.41 (The system).

“The system” is comprised exactly of the most-public articles. Every user “sees” the system (as filtered through their own customizations). Freedom of users to make (not-publically-visible) customizations ad infinitum is certainly important. But questions as to who can change the system itself and how it can be changed seem to be at least as important.

Note 13.42 (Ontology versus pre-ontology).

Most of the work done so far in this document has been at the level of pre-ontology; work on the associate ontology or ontologies can begin before long.

Note 13.43 (Various free and nonfree cultures).

I still don’t know what to say about the license-defined common versus a managed public domain common commons.

Note 13.44 (Teamwork).

Building a good team relates to the platform’s usability (in both directions). Is the whole just as good as the sum of its parts? What’s good/bad/indifferent about informal teams?

Note 13.45 (Places to store thoughts).

Places, or spaces, for thoughts can be helpful to have around. File X under “experiments”, Y under “community spirit” and so on. As to whether the metaphor of “file” works, that’s a good question; labels may be uniformly better to use. Furthermore, just about any sort of attachment relationship can be used to categorize things; compare Note 13.25.

Note 13.46 (Work ethic).

The achievement of a workable work ethic was supposed to be the big event in Penmaster’s Apprentice issue 2. With regard to this work, it is interesting to consider the interplay of different work styles: small and large, physical and computational. We can get very different sorts of meaning clusters simply by adjusting our scholia-creation patterns (at least in theory). The (judicious) use of summaries, critiques, new-found relationships, and so forth, give an overall feeling to the evolution of a document.

Note 13.47 (Interpretation).

What is meaningful? In relationships, in activities, and so forth? In literature, you want to figure out e.g. how to do character development in a “meaningful” way, to create something that people will want to read (and which they will find meaningful).

Note 13.48 (Vanity publishing).

One’s vanity can at least have the effect of encouraging one to write and perhaps to do research as well.

Note 13.49 (Role of philosophy in this document).

Some people have told me that they don’t see the point of writing so much. But code is allegedly supposed to come after design, and design is supposed to come after you understand the problem. I think on paper, and code things up as I understand them. By this token, perhaps eventually we’ll see a version of the document in which most of the philosophy has code associated with it.

One thing to realize (might as well do so at some point) is that this is more than just a program; it is a philosophy of documents, whence, maybe even a tentative metaphysics, since what can’t be regarded as a document, after all!, and as making some addition to the philosophy of (the society of) mind.

Note 13.50 (Philosophy of dictionaries).

Someone or other said something about defining via every use. Which implies that each definition is a more-than-octopodal sort of thing, reaching all over the place. Uses can often be partitioned, and showing this partition only can massively simplify things. It is interesting to think about the scholium nature of partitions and representatives of the partitional classes.

Note 13.51 (Control).

One tends to imagine that the things that one can control are relatively safe.

13.1 Mathematics of the scholium system

Note 13.52 (Document topology).

The conversation about identification of document regions has a very topological flavor, as does the idea (also mentioned in the paper) about stitching documents together out of articles attached along their margins.

Note 13.53 (Cyberspace geometry).

The “observation” mentioned in Note 13.2 could presumably be translated – at least partially – into a statement about the hyperbolic geometry of cyberspace. And I think people have been studying this stuff for a while. However, I think there is more to the statement than just that.

Note 13.54 (Region arithmetic).

The notion that regions can be copied, broken apart, deleted, and added on the fly gives a notion of *, /, -, and + for marked up regions (and this applies to both text and markup). I don’t know whether much can be made of these things, but they are the fundamental operations, so perhaps there is something here.

Note 13.55 (Clusion algebra).

Studying articles and clusions could be an interesting computational variant on the study of objects and morphisms.

Note 13.56 (Transclusion linearity).

The fact that transclusions shouldn’t be indicated by constant, but rather, linear, functions, seems cute (but I’m not sure what to make of it).

Note 13.57 (Scholium mathematics in general).

I’m particularly reminded of topology (including graph theory) and ergodic theory. “Information theory” seems related to the issues of updates and versioning; similarly, “knowledge theory” may be related to the issues of multi-directional updating. From there we could move on to game theory, I suppose, as the mathematical domain most closely related to culture building (nomics are what I’m thinking of here; these also relate to MUDs).

Economics, more generally, seems to be where things are headed, when we think about the social welfare situation associated with CBPP system policies (e.g. rejecting corrections on PlanetMath has no repurcussions; if an essay about Frankenstein is posted to the site and the author rejects all corrections which say “this is not math!”, the administrators will presumably have to get involved, and take some extra-protocol action to remove the article).

Other topics that should be taken up are the linguistic and “mathematical artificial intelligence” threads coming from the HDM project.

Note 13.58 (Network navigation).

If a scholium-based document is broken up into trees, the document’s underlying network can be linearized simply by ordering the roots of these trees. This relates to questions about literate programming and proof checking (how do you write a “structured proof”?).

13.2 Document analysis and the scholium system

Note 13.59 (Pitfalls of summarization).

It isn’t completely clear that Note 1.1 makes a great summary of the whole of the philosophy of this work. Any summary leaves things out or is otherwise vague.

Note 13.60 (Recapitulation).

An article A that appears after some article B in a given ordering which contains a summary of B. This relationship could be indicated by a link in one direction or another (likely to run from A to B).

Note 13.61 (Discourse analysis and holographic or monadological structures).

Long-range reference and design theory seem related to each other and to this work as a whole; cf. Section 13.1.

Note 13.62 (Unified semantics).

Nelson (I think) has advertised some kind of unified theory of semantics; it might be fun to take a look at that.

14 Conclusions

Note 14.1 (Original conclusion).

In a context with multiple authors talking about many intersubjectively important concepts, with ownership of the texts that they write but freedom with the content, treating the total document as a collection of scholia could provide important alternative perspectives on a given subject. The presence of on-screen scholia could encourage responsible maintainership and aid users in making specific comments and suggestions part of their critical reviews of existing entries.

Note 14.2 (Difficulty of putting things together).

Technological difficulties are posed by the fact that a document must be assembled out of several pieces for presentation purposes and the fact that standard web browsers and editors may not be very useful for browsing or creating these sorts of documents. (This isn’t always a difficulty; rather, it motivates the current development effort!) Compare the message on the outside of the Walker (Figure 1).

And: why didn’t we see something like this document before? Why haven’t the systems on the Hyperworlds server become “popular”? I’m not completely sure; there are probably a number of interrelated reasons.

Note 14.3 (Freedom in an online context).

Another very important thing to talk about is free versus non-free web services. I’m not sure if we mean to completely skewer non-free systems, i.e., systems that don’t have a collaborative free scholia-based compenent to them. But on the other hand, such systems really may be much less useful. The current model is basically: if you want to create some commentary on something else, you need to find your own platform/soapbox from which to make that commentary. Some systems, like PM’s attached fora, invite commentary and provide a place for it. But many, many, pages out there do not invite commentary or simply do not supply a place to put it. Another thing to notice about the scholia-based system is that it is very egalitarian. Just because something was “there first” doesn’t make it better. The commentary can become just as important as the main document, or more important. On the other hand, it would be very useful to develop some algorithms that can be used to sort out the useful commentary from the utter crap. Slashdot does a pretty good job of this, but I think there is still room for improvement. And, even though PM provides attached fora, they could do a significantly better job of exposing the content that is provided in those fora in the “default view” or some “secondary view” of the object that is being commented on. It might just take something simple, like expanding all of the threads, or possibly using some Slashdot-like rating system to expand notable threads. Both interactive and automatic ways of determining the usefulness of certain feedback can be used – and there are already some things like this in Gnus (I’m sure).

Note 14.4 (Knowledge communities).

It would also be good to talk about Ray’s comments about knowledge communities – need to take another look at those again. Also, referring to other Ray stuff, I wonder what conditions would be necessary for academics to accept more off-the-beaten-track publications (e.g. PlanetMath). Of course, if these things became popular they wouldn’t be off-the-beaten-track any more, really, but let’s not get circular. My sense is that making some sort of medium “respectable” has to do with the social contract thing. On the other hand, people do seem to get respect for out-there stuff if it is somehow novel, or useful, or something. (Not sure what the relevant quality here is, how much “uptake” has to do with it, how much “propaganda” (or “PR”) or what – if I knew maybe I’d try to exploit this knowledge!)

Note 14.5 (Social simplicity).

Sometimes complexity may not be totally avoidable within the code. (E.g. if there are 1 billion scholia about a given article, sorting them would take a long time.) Sometimes we can do clever things – but other times we can probably rely on the user groups to structure things in ways that won’t push things too far. It isn’t clear how much we can rely on them for this – but my guess is that things that the machine would whine about are frequently going things that people want to avoid for their own reasons.

Note 14.6 (Human and artificial intelligence).

The HDM project may be better off building something that smart people can hack than building smart code right away. Perhaps that’s why I’m giving precedence to the scholium system over things like parsers and the like. The “human element” in software development seems powerful. It is curious that I’ve envisioned the scholium system as being mainly about the AI component of HDM; but it may be that we can expect more from it, namely, a contribution to AI for HDM creation. Take a look at Note 13.24 for some words against thinking about things in terms of “evolution towards actionable form.” I think a better argument would be for co-evolution (which is good, because there has been plenty of thinking done on this issue). SNePs seems to have some things to say on this topic2525 25 Presumably so do others; still, by contrast with many other systems, the co-evolution view seems to be emphasized particularly strongly here.

Note 14.7 (Labels vs namespaces, transclusion vs inclusion, overlays vs text properties).

It is interesting to compare the “hard” to the “soft” approaches, both of which appear in this document in different roles (sometimes in interestingly similar relationships to one another).

Note 14.8 (Compatibility between versions of this system?).

All else equal, it would be nice for any future versions of this system to be compatible with this one. But we can make no guarantees that this will actually happen, although I do think that the system will be revised somewhat to continue to make improvements and add new features.

Note 14.9 (Future work).

It would be pleasant to have Arxana hooked up with a graphical node/edge display, such as fenfire2626 26 Building rich displays is a problem that we’ll want to consider from many angles (cf. Note 5.15). The usefulness of getting various tools worked out in plain-text modes shouldn’t be ignored!

Note 14.10 (