In a post on the Esperanto Language Stack Exchange I then heard of another conlang, Toki Pona. This is billed as a minimalist language: it only has about 120 words, and a very fixed and simplistic sentence structure. Unlike Esperanto, it is not really a proper language for everyday use, but more of a philosophical experiment. How does your language influence the way you think about the world? This is kind of related to the Sapir-Whorf Hypothesis. When you have to limit yourself to describing/paraphrasing everything with a limited vocabulary, you need to reflect more about what it is you’re talking about. For example, a friend is a jan pona (‘good person’), and a bad person is a jan ike. So how do you say ‘bad friend’? You are not able to say someone is both good and bad at the same time. So a friend cannot be a bad person, or vice versa.

Toki Pona arguably is a toy language only. There is just one word for fruit, vegetable, etc.: kili. If you want to say banana, you say kili jelo (“yellow fruit”). If you want to talk about lemons as well, you’re out of luck. It’s very context dependent, and definitely not useful for a scientific treatise, or even recipes. But it’s fine for basic stories, myths and legends, and so on. And, most importantly, it’s easy to learn. No morphology. Very limited syntax. Small vocabulary. The main difficulty is to express yourself given those limited means. But we grow only when challenged!

I’m interested in doing NLP with Toki Pona, as it is so limited. It should be possible to quickly get to the semantic or pragmatic levels, as morphology and syntax will be dealt with easily. Analysing and generating sentences should be extremely easy. More on that as it materialises.

If you’re interested in languages, and what to explore the way you express meaning, give Toki Pona a try. There are various on-line resources available, plus a text book Toki Pona – The Language of Good.

I’ll post more on this at a later time…

Now, the first thing you learn about natural language is that it is full of ambiguities, and so there will always be several alternatives available, several possible paths through the network which defines the grammar. The traditional solution is to dump all the alternatives on a stack, and look at them when the current path has been finished with. You can either go depth-first, where you complete the current path before you get the next one off the stack, or breadth-first, where you advance all paths by one step at a time, kind of pseudo-parallel.

Having to deal with a stack is tedious, as you need to keep track of the current configuration: which network are you at, what node, what position in the sentence, etc. But then, it occurred to me, there’s an easier way to do it (at least it’s easier in Erlang!): every time you come to a point where you have multiple alternatives, you spawn a new process and pursue all of them in parallel.

The only overhead you need is a loop which keeps track of all the processes currently running. This loop receives the results of successful paths, and gets notified of unsuccessful ones (where the process terminates without having found a valid structure). No need for a stack, and hopefully very efficient processing on multi-core machines as a free side-effect.

I’m still amazed how easy it was to implement. I wouldn’t have fancied doing that in Java or even C. For my test sentences I had about 8 to 10 processes running in parallel most of the time, but it depends on the size of the grammar and the length of the sentence really. What I liked about this was that it seemed the natural way to do in Erlang, where working with processes is just so easy.

And also, another nail in the coffin for the claim that you can’t use Erlang for handling texts easily!