The language mix
The language mix
Abstract
During the third ConTEXt conference that ran in parallel
to EuroTEX 2009 in The Hague we had several sessions
where mkiv was discussed and a few upcoming features
were demonstrated. The next sections summarize some
of that. It’s hard to predict the future, especially be-
cause new possibilities show up once LuaTEX is opened
up more, so remarks about the future are not definitive.
1 TEX
From now on, if I refer to TEX in the perspective
of LuaTEX I mean “Good Old TEX”, the language
as well as the functionality. Although LuaTEX pro-
vides a couple of extensions it remains pretty close
to compatible to its ancestor, certainly from the
perspective of the end user.
As most ConTEXt users code their documents
in the TEX language, this will remain the focus of
mkiv. After all, there is no real reason to abandon
it. However, although ConTEXt already stimulates
users to use structure where possible and not to use
low level TEX commands in the document source,
we will add a few more structural variants. For
instance, we already introduced \startchapter and
\startitem in addition to \chapter and \item.
We even go further, by using key/value pairs for
defining section titles, bookmarks, running headers,
references, bookmarks and list entries at the start of
a chapter. And, as we carry around much more in-
formation in the (for TEX so typical) auxiliary data
files, we provide extensive control over rendering the
numbers of these elements when they are recalled
(like in tables of contents). So, if you really want
to use different texts for all references to a chapter
header, it can be done:
\startchapter
[label=emcsquare,
title={About $e=mc^2$},
bookmark={einstein},
list={About $e=mc^2$ (Einstein)},
reference={$e=mc^2$}]
... content ...
\stopchapter
VOORJAAR 2015 1
Under the hood, the mkiv code base is becoming
quite a mix and once we have a more clear picture
of where we’re heading, it might become even more
of a hybrid. Already for some time most of the font
handling is done by Lua, and a bit more logic and
management might move to Lua as well. However,
as we want to be downward compatible we cannot
go as far as we want (yet). This might change as
soon as more of the primitives have associated Lua
functions. Even then it will be a trade off: calling
Lua takes some time and it might not pay off at all.
Some of the more tricky components, like vertical
spacing, grid snapping, balancing columns, etc. are
already in the process of being Luafied and their
hybrid form might turn into complete Lua driven
solutions eventually. Again, the compatibility issue
forces us to follow a stepwise approach, but at the
cost of (quite some) extra development time. But
whatever happens, the TEX input language as well
as machinery will be there.
2 MetaPost
I never regret integrating MetaPost support in
ConTEXt and a dream came true when mplib
became part of LuaTEX. Apart from a few minor
changes in the way text integrates into MetaPost
graphics the user interface in mkiv is the same as
in mkii. Insofar as Lua is involved, this is hidden
from the user. We use Lua for managing runs and
conversion of the result to pdf. Currently generating
MetaPost code by Lua is limited to assisting in the
typesetting of chemical structure formulas which is
now part of the core.
When defining graphics we use the MetaPost
language and not some TEX-like variant of it.
Information can be passed to MetaPost using special
macros (like \MPcolor), but most relevant status
information is passed automatically anyway.
You should not be surprised if at some point we
can request information from TEX directly, because
after all this information is accessible. Think of
something w := texdimen(0) ; being expanded at
the MetaPost end instead of w := \the\dimen0 ;
being passed to MetaPost from the TEX end.
2 MAPS 50 Hans Hagen

3 Lua And, knowing Lua helps as well, since the following

What will the user see of Lua? First of all he or she is more efficient:
can use this scripting language to generate content.
But when making a format or by looking at the \startluacode
statistics printed at the end of a run, it will be clear local colored = {
that Lua is used all over the place. background = "color",
So how about Lua as a replacement for the TEX backgroundcolor = "bluegreen",
input language? Actually, it is already possible to foregroundcolor = "white",
make such “ConTEXt Lua Documents” using mkiv’s }
built in functions. Each ConTEXt command is also local basespec = {
available as a Lua function. framecolor = "bluered",
align= "middle",
\startluacode style = "type",
context.bTABLE { offset=".5ex",
framecolor = "blue", }
align= "middle", local bTR, eTR = context.bTR, context.eTR
style = "type", local bTD, eTD = context.bTD, context.eTD
offset=".5ex", context.bTABLE(basespec)
} for i=1,10 do
for i=1,10 do bTR()
context.bTR() for i=1,20 do
for i=1,20 do local r= math.random(99)
local r= math.random(99) bTD((r < 50 and colored) or nil)
if r < 50 then context("%#2i",r)
context.bTD { eTD()
background = "color", end
backgroundcolor = "blue" eTR()
} end
context(context.white("%#2i",r)) context.eTABLE()
else \stopluacode
context.bTD()
context("%#2i",r) Since in practice the speedup is negligible and
end the memory footprint is about the same, such
context.eTD() optimizations seldom make sense.
end At some point this interface will be extended, for
context.eTR() instance when we can use TEX’s main (scanning,
end parsing and processing) loop as a so-called corou-
context.eTABLE() tine and when we have opened up more of TEX’s
\stopluacode internals. Of course, instead of putting this in your
TEX source, you can as well keep the code at the
Of course it helps if you know ConTEXt a bit. For Lua end.
instance we can as well say:

if r < 50 then 42 36 59 8 45 50 18 47 62 31 56 9 31 21 12 42 97 25 53 92

context.bTD { 41 82 76 74 60 69 38 97 23 97 61 59 42 31 55 17 48 36 10 40
background = "color", 5 45 56 52 54 21 12 60 60 95 42 60 23 54 82 77 45 2 50 33
backgroundcolor = "blue", 34 38 10 77 46 99 37 10 84 39 99 54 22 72 68 64 79 93 4 28
foregroundcolor = "white", 26 8 67 7 46 95 69 60 64 9 59 45 61 72 24 42 42 78 23 38
} 72 22 57 96 2 29 17 47 27 70 95 58 89 95 75 27 93 7 12 3
else 90 84 90 42 97 74 60 10 81 26 39 10 31 33 75 18 23 96 6 31
context.bTD() 5 95 36 81 50 25 62 21 31 70 46 55 29 77 34 97 17 1 10 64
end 5 70 96 79 40 65 23 97 78 54 72 23 84 59 3 23 65 44 47 32
context("%#2i",r) 38 71 83 70 70 22 86 47 4 16 92 32 48 48 11 4 35 6 99 42
context.eTD()
Figure 1. The result of the displayed Lua code.
The language mix
The script that manages a ConTEXt run (also called
context) will process files with that consist of such
commands directly if they have a cld suffix or when
you provide the flag --forcecld.1
context yourfile.cld
But will this replace TEX as an input language?
This is quite unlikely because coding documents in
TEX is so convenient and there is not much to gain
here. Of course in a pure Lua based workflow (for
instance publishing information from databases) it
would be nice to code in Lua, but even then it’s
mostly syntactic sugar, as TEX has to do the job
anyway. However, eventually we will have a quite
mature Lua counterpart.
4 XML
This is not so much a programming language but
more a method of tagging your document content
(or data). As structure is rather dominant in xml, it
is quite handy for situations where we need different
output formats and multiple tools need to process
the same data. It’s also a standard, although this
does not mean that all documents you see are
properly structured. This in turn means that we
need some manipulative power in ConTEXt, and
that happens to be easier to do in mkiv than in
mkii.
In ConTEXt we have been supporting xml for
a long time, and in mkiv we made the switch
from stream based to tree based processing. The
current implementation is mostly driven by what
has been possible so far but as LuaTEX becomes
more mature, bits and pieces will be reimplemented
(or at least cleaned up and brought up to date with
developments in LuaTEX).
One could argue that it makes more sense to
use xslt for converting xml into something TEX,
but in most of the cases that I have to deal with
much effort goes into mapping structure onto a given
layout specification. Adding a bit of xml to TEX
mapping to that directly is quite convenient. The
total amount of code is probably smaller and it saves
a processing step.
We’re mostly dealing with education-related doc-
uments and these tend to have a more complex
structure than the final typeset result shows. Also,
readability of code is not served with such a split as
most mappings look messy anyway (or evolve that
1 Similar methods exist for processing xml files.
2 This example is inspired by one of our projects where the cleanup involves
sanitizing (highly invalid) html data that is embedded as a CDATA stream, a trick to prevent the xml file to be invalid.
VOORJAAR 2015 3
way) due to the way the content is organized or
elements get abused.
There is a dedicated manual for dealing with xml
in mkiv, so we only show a simple example here. The
documents to be processed are loaded in memory
and serialized using setups that are associated to
elements. We keep track of documents and nodes in
a way that permits multipass data handling (rather
usual in TEX). Say that we have a document that
contains questions. The following definitions will
flush the (root element) questions:
\startxmlsetups xml:mysetups
\xmlsetsetup{#1}{questions}{xml:questions}
\stopxmlsetups
\xmlregistersetup{xml:mysetups}
\startxmlsetups xml:questions
\xmlflush{#1}
\stopxmlsetups
\xmlprocessfile{main}{somefile.xml}{}
Here the #1 represents the current xml element. Of
course we need more associations in order to get
something meaningful. If we just serialize then we
have mappings like:
\xmlsetsetup{#1}{question|answer}{xml:*}
So, questions and answers are mapped onto their
own setup which flushes them, probably with some
numbering done at the spot.
In this mechanism Lua is sort of invisible but
quite busy as it is responsible for loading, filtering,
accessing and serializing the tree. In this case TEX
and Lua hand over control in rapid succession.
You can hook in your own functions, like:
\xmlfilter{#1}
{(wording|feedback|choice)/function(cleanup)}
In this case the function cleanup is applied to
elements with names that match one of the three
given.2
Of course, once you start mixing in Lua in this
way, you need to know how we deal with xml at
the Lua end. The following function show how we
calculate scores:
4 MAPS 50 Hans Hagen

\startluacode able to share the implementation with TEX input, it

function xml.functions.totalscore(root) also makes sense to stick to some hybrid approach.
local n = 0 In any case, more of the calculations and logic will
for e in xml.collected(root,"/outcome") do move to Lua, while TEX will deal with the content.
if xml.filter(e,"action[text()='add']") then A somewhat strange animal here is xsl-fo. We
local m = xml.filter do support it, but the mkii implementation was
(e,"xml:///score/text()") always somewhat limited and the code was quite
n = n + (tonumber(m or 0) or 0) complex. So, this needs a proper rewrite in mkiv,
end which will happen indeed. It’s mostly a nice exercise
end of hybrid technology but until now I never really
tex.write(n) needed it. Other bits and pieces of the current xml
end goodies might also get an upgrade.
\stopluacode There is already a bunch of functions and macros
to filter and manipulate xml content and currently
You can either use such a function in a filter or just the code involved is being cleaned up. What di-
use it as a TEX macro: rection we go also depends on users’ demands.
So, with respect to xml you can expect more
\startxmlsetups xml:question support, a better integration and an upgrade of
\blank some supported xml related standards.
\xmlfirst{#1}{wording}
\startitemize 5 Tools
\xmlfilter{#1} Some of the tools that ship with ConTEXt are also
{/answer/choice/command(xml:answer:choice)} examples of hybrid usage.
\stopitemize Take this:
\endgraf
score: \xmlfunction{#1}{totalscore} mtxrun --script server --auto
\blank
\stopxmlsetups On my machine this reports:

\startxmlsetups xml:answer:choice MTXrun | running at port: 31415

\startitem
\xmlflush{#1}
\stopitem
\stopxmlsetups
The filter variant is like this:
\xmlfilter{#1}{./function('totalscore')}
So you can take your choice and make your source
look more xml-ish, Lua-like or TEX-wise. A careful
reader might have noticed the peculiar xml:// in the
function code. When used inside mkiv, the serializer
defaults to TEX so results are piped back into TEX.
This prefix forced the regular serializer which keeps
the result at the Lua end.
Currently some of the xml related modules, like
mathml and handling of tables, are really a mix of
TEX code and Lua calls, but it makes sense to move
them completely to Lua. One reason is that their
input (formulas and table content) is restricted to
non-TEX anyway. On the other hand, in order to be
MTXrun | document root: c:/data/develop/context/
lua
MTXrun | main index file: unknown
MTXrun | scripts subpath: c:/data/develop/context
/lua
MTXrun | context services: http://localhost:31415
/mtx-server-ctx-startup.lua
The mtxrun script is a Lua script that acts as a con-
troller for other scripts, in this case mtx-server.lua
that is part of the regular distribution. As we use
LuaTEX as a Lua interpreter and since LuaTEX has
a socket library built in, it can act as a web server,
limited but quite right for our purpose.3
The web page that pops up when you enter the
given address lets you currently choose between the
ConTEXt help system and a font testing tool. In
figure ?? you seen an example of what the font
testing tool does.
Here we have LuaTEX running a simple web
server but it’s not aware of having TEX on board.
When you click on one of the buttons at the bottom

3 This application is not intentional but just a side effect.

The language mix
of the screen, the server will load and execute a
script related to the request and in this case that
script will create a TEX file and call LuaTEX with
ConTEXt to process that file. The result is piped
back to the browser.
You can use this tool to investigate fonts (their
bad and good habits) as well as to test the currently
available OpenType functionality in mkiv (bugs as
well as goodies).
So again we have a hybrid usage although in this
case the user is not confronted with Lua and/or TEX
at all. The same is true for the other goodie, shown
in figure ??. Actually, such a goodie has always been
part of the ConTEXt distribution but it has been
rewritten in Lua.
The ConTEXt user interface is defined in an
xml file, and this file is used for several purposes:
initializing the user interfaces at format generation
time, typesetting the formal command references
(for all relevant interface languages), for the wiki,
and for the mentioned help goodie.
Using the mix of languages permits us to provide
convenient processing of documents that otherwise
would demand more from the user than it does now.
For instance, imagine that we want to process a
series of documents in the so-called epub format.
Such a document is a zipped file that has a descrip-
tion and resources. As the content of this archive is
prescribed it’s quite easy to process it:
context --ctx=x-epub.ctx yourfile.epub
This is equivalent to:
texlua mtxrun.lua --script context --ctx=x-epub.
ctx yourfile.epub
So, here we have LuaTEX running a script that itself
(locates and) runs a script context. That script
loads a ConTEXt job description file (with suffix
ctx). This file tells what styles to load and might
have additional directives but none of that has to
bother the end user. In the automatically loaded
style we take care of reading the xml files from
the zipped file and eventually map the embedded
html like files onto style elements and produce a
VOORJAAR 2015 5
pdf file. So, we have Lua managing a run and mkiv
managing with help of Lua reading from zip files
and converting xml into something that TEX is
happy with. As there is no standard with respect
to the content itself, i.e. the rendering is driven by
whatever kind of structure is used and whatever the
css file is able to map it onto, in practice we need
an additional style for this class of documents. But
anyway it’s a good example of integration.
6 The future
Apart from these language related issues, what more
is on the agenda? To mention a few integration
related thoughts:
󰁁 At some point I want to explore the possibility
to limit processing to just one run, for instance
by doing trial runs without outputting anything
but still collecting multipass information. This
might save some runtime in demanding work-
flows especially when we keep extensive font
loading and image handling in mind.
󰁁 Related to this is the ability to run mkiv as a
service but that demands that we can reset the
state of LuaTEX and actually it might not be
worth the trouble at all given faster processors
and disks. Also, it might not save much runtime
on larger jobs.
󰁁 More interesting can be to continue experiment-
ing with isolating parts of ConTEXt in such a
way that one can construct a specialized sub-
set of functionality. Of course the main body of
code will always be loaded as one needs basic
typesetting anyway.
Of course we keep improving existing mechanisms
and improve solutions using a mix of TEX and Lua,
using each language (and system) for what it can do
best.
7 Notes
Hans Hagen
Pragma ADE, Hasselt
pragma (at) wxs dot nl