(Sorry for the terminology - it is my own. Suggestions for improvement are most welcome ...)
This subsection intends to describe the basic ideas in as simple a manner as I manage to do it. If it looks way to abstract, do skip ahead to the next section where this things are implemented in code.
Let us keep for the moment Tcl's linear flow of execution: everything is set up for sequential evaluation, all function calls nest nicely. The explanation of the ideas is simpler within this framework, although many of them look like they may not be worth the trouble. Bear with me for a while, we will describe the new possibilities opened by this code design soon enough!
We define a new EvalFrame structure that contains all the information required for:
We maintain a stack of such EvalFrames. When a function needs to eval a script, it must set up and queue the corresponding EvalFrame, indicate which function should evaluate the EvalFrame, and finally call a FlowManager to perform the evaluation.
The FlowManager is responsible for calling the script evaluation, and when that evaluation returns it calls the return function of the EvalFrame that requested the evaluation. The manager itself returns after executing the return function of the first EvalFrame it evaled - i.e., the one responsible for the creation of the manager. We say that that EvalFrame owns the manager.
What this structure permits is to eliminate from the C-stack the calling function before evaluating the script, with obvious gains when there are deeply nested evaluations. We will see in the next section another new possibility opened by the independence of the flow of execution and the C-stack.
We implement two ways of requesting an evaluation: the standard one (ST) and the new non-recursive (NR). The way this is implemented is: a standard evaluation calls a new manager to perform the evaluation and awaits the return of the manager; the NR call schedules the evaluation to be performed by the last not-yet-returned manager.
Let us consider again the previous example : A calls B, which in turn calls C (which calls F)and then D. We assume that the call of B from A is a standard call, and let the other calls be all either ST or NR. The stack effects are (M represents here a manager; we also show for comparison the original Tcl implementation OR):
------- C-STACK ------- FLOW OR NR ST --------------------------------------------- start A A A A calls B A:B A:M:B A:M:B B calls C A:B:C A:M:C A:M:B:M:C C calls F A:B:C:F A:M:F A:M:B:M:C:M:F F returns A:B:C A:M:C A:M:B:M:C C returns A:B A:M:B A:M:B B calls D A:B:D A:M:D A:M:B:M:D D returns A:B A:M:B A:M:B B returns A A A
The manager has now the possibility to maintain different EvalFrame stacks (muThreads), and assign execution time to one or the other. The muThreads provide a very simple framework for collaborative multitasking and/or co-routines.
It should be very simple to create a muThread framework and solve the coordination problems between tasks if we maintain the TclThread convention that to each interp corresponds just one muThread.
In this case, the creation of a new muThread is as lightweight as the creation of a new interp. I think it should also be easy to define safe muThreads using the existing infrastructure for safe interpreters.
The existing code defines only the very basic infrastructure for muThreads with round-robin scheduling. In order to exploit them, it will be necessary to write a muThread extension or add the corresponding functionality to the core.
The idea is to keep the Tcl library API as is, and to provide additional functions that provide the NR functionality. As an example, there will be a new Tcl_EvalObjNR that provides similar functionality to Tcl_EvalObjEx but does unwind the C-stack before running.
An NR-enabled caller (one that is prepared to be evicted from the C-stack) will call Tcl_EvalObjNR. A non-NR caller calls the old interface Tcl_EvalObjEx, which in turns actually calls Tcl_EvalObjNR in a new manager. This new manager protects the caller from eviction from the C-stack.
We will see that users of the old interface are in a sense " bad citizens" in terms of the new functionality, but they are also the main sufferers from bad citizenship:
We could reprogram Tcl on top of (Java, ML, CommonLisp, Oberon, ... ) instead of C. Some of these languages' standard implementations may provide an easier access to the needed functionality than C. So, why C?
The advantages of C are (AFAIU):
Given the
C flow conventions
this might seem an impossible task. POSIX however
provides a mechanism for non-local function exits, usually associated
with error handling: setjmp/longjmp
.
The main idea is that a function signals a spot in its code as
a possible non-local return target with setjmp
. If another function
then calls a longjmp to that target, the C-stack will be unwound
and execution will continue just as if the original
setjmp
function
were returning. Everything that went into the C-stack later than
the setjmp
is wiped out.
The main constraints for the use of setjmp/longjmp
are:
longjmp
to a target defined by a function that
has already returnedThis provides precisely what we need to implement the mechanism
described previously: a manager calls setjmp
and thus provides a
barrier to the unwinding of the C-stack. The call to evaluate is
then simply a longjmp
to the last manager in the C-stack, which resumes
evaling the EvalFrames in the stack.
We propose to use longjmp
to signal a request for NR evaluation.
What alternatives are there?
longjmp
. In any case, setjmp/longjmp
does more or less exactly what we
want and, assuming it really is widely available, there is no real
need for an alternative.