K for Good Thought: [Part 2] k,q,kdb tutorial verbs, functions

Last time, we got to list things and reorder them a bit,  but that’s kind of boring on it’s own. This second tutorial is concerned with being able to play, transform and generally do things with nouns. In English, these types of words, action words, are normally called verbs and I’ll let the OED define it.

OED 

Verb:

o          NOUN- Grammar

A word used to describe an action, state, or occurrence, and forming the main part of the predicate of a sentence, such as hear, become, happen.

o          VERB- Grammar

Use (a word that is not conventionally used as a verb, typically a noun) as a verb.

Interestingly, though there are two definitions for verb, we will mostly concern ourselves with the first, but I shall leave a small footnote on how the second is related.

The first definition is fairly trivial, that is most speakers of a language understand that you need a category of words that do things, even if you don’t need the category itself. That is we tend to ‘place’, ‘buy’, ‘blend’ and generally do stuff to things. This doing stuff is what I mean by verb. In math, in computer science and in programming languages there is a concept of ‘function’ and this concept performs much of what a verb does in English. It allows you to manipulate the stuff. To take a crude example, verbs are like little machines that allow us to change something about the stuff. So for instance, I am running. Which is to say ‘I’ am now in transit. I think belaboring this point serves to confuse rather than enlighten. So I leave it here. Anyway, I will use the word verb instead of function or something else, as I find those other terms have been overloaded with meaning (people think of very specific examples) especially for programmers.

Verbs take an atom or a list and produce an atom or a list. This list can be a list of atoms or a list of lists or a list of lists of lists….

Whoa, almost got carried away there.

Anyway, the more important thing is to remember that a verb and noun produce a new noun. For those inclined to equations: verb (noun) => new noun

Which can be read as verb applied to noun yields new noun. (don’t worry there will be little else I borrow from chemistry.)

For example, the verb shake produces shaken noun.

Where shaken is an adjective.

Sometimes there is no adjective to describe the new noun.

For example, take the verb blend. If I blend a cat with a dog. I might get catdog or dogcat or a mush, in any case, we can talk of having blended the cat and dog. Often we don’t have a word that describes the new thing. The predicate phrase is the result of the verb. So for instance, Gena went to the zoo, ‘went to the zoo’ is the predicate phrase and is the description of ‘Gena’ since English lacks a special phrase that describes a person who has gone to the zoo. Other times, we do have a phrase “shake this drink” the drink is now mixed or shaken, but it is most definitely not stirred. The predicate phrase is always a description of a subject. Predicate phrases or verbs transform the subject and give us a new subject. One that is described by the predicate phrase.

Because K is a descendant of APL, there are certain conventions which K kept but simplified and enhanced. I will therefore first explain what APL did and then switch to K’s simplification and enhancement.

In APL, all verbs just as in English take a noun and make it do something, the result is always just a noun that has done that thing. So if I(noun) run (verb) then I am-running (new state of ‘I’). Just as in English some verbs need two nouns to make sense or have a different sense if used on their own. For instance, I winked, means something different though related to I winked at him. So for example in K the ‘%’ symbol is the verb divide. If we just say divide seven in K %7, that returns  0.1428571. Which is the reciprocal of 7, or one seventh. Where as if we 21%7 we get 3. This what APL calls monadic and dyadic verbs.

Verbs can be of two types: monadic and dydadic. Simply put, monadic verbs take a noun or a list of nouns on the right and produce a noun or a list (that list can contain lists, as always I will cease making this point and assume it from here on in unless it is not true. Then I will make special mention). Dyadic verbs take a noun or list on the left and the noun or list on the right and create a noun or list.

In the first case % behaves like a monadic verb reciprocate (take reciprocal or multiplicative inverse). That is the verb only referred to the noun following it. In the second case % behaved more like the verb divide, which takes a number on the left and divides it into the right (pun intended) number of equal pieces. The result is the size of each piece. Though this is intuitive for verbs that apply to one or two nouns at a time. It does not make much sense for verbs that apply to more nouns.

This is where K made a change from APL and created it’s own convention, from here on in it’s back to K and Q.

Going back our verb ‘blend’ suppose we wanted to blend a cat, dog, and a rabbit. Where do we put the noun rabbit. Each noun, is often called an argument.

Argument: The noun or list that a verb does its action to is often called an argument. The term is helpful, because we can refer to verbs as taking some number of arguments. For instance, Monadic take one argument, the one on the right of the verb. Dyadic take two arguments, the one on the left and then the one on the right. (Sometimes in some textbooks these are called parameters. I will try to stick with argument.)

K has adopted the following convention. If you have a verb that takes a specific number  of arguments, then you do so by putting the verb first and then putting a open square bracket([) the arguments separated by a semicolon(;) and a closing square bracket (]). This is very easy to see:

In K 10÷5 can be written at least these two ways:
10%5
%[10;5]

The nice thing about the second notation, is that it generalizes to verbs that apply to more than two arguments. You would simply write:
blend[dog;cat;rabbit]

Because this convention is simple, all verbs that a user defines automatically follow it and there is no way for a user to create dyadic verbs that take a left and right argument. Instead user defined verbs always apply to the arguments in the order notated in the brackets. That is instead of the noun being to the right and left of the verb. They are now right and left respectively inside the brackets. There is a small caveat: this convention does not allow you to define a verb that applies to more than eight arguments. The reason is that keeping track of eight arguments is difficult enough and the limit suggests to the writer that possibly the verb they are building should be composed of several verbs, there are ways to get around this limit, but think of it as a guideline.

q simplified this concept of verb even further. All functions in k that can either be monadic or dyadic are always dyadic in q and a word is chosen to represent the monadic case. If you want the reciprocal you are going to have to ask for it directly reciprocal[7].

Alright, we have to answer two basic questions: what are the verbs that we can use?, ie the vocabulary of verbs and two what do we do if a verb is not in the vocabulary? define new verbs.

For the first I will link to a site: http://code.kx.com/nwiki/JB:QforMortals2/built_in_functions

Suppose that a verb is not in the vocabulary. For instance, suppose we want to build into q the idea of cheering up. First we need to define some nouns. Let’s use the emoticons to signify a scale of happiness.

Very simply we might have:

Crying – “:’(”
Sad – “:(”
Annoyed – “:/”
Neutral – “:|”
happy – “:)”
very happy – “:))”
laughing -“:D”
tears of joy – “:’)”

Alright so each of these emoticons, will symbolize possible emotional states. Because list of characters are themselves lists. We need to create a list of lists, the easiest way is to use the enlist verb.

emotions: enlist(“:’(“)

This creates the first emoticon as a separate entry in the list of emotions, the rest can be added by appending. Of course you can do it in one shot but this way, you can use enlist and avoid all kinds of annoyances from KDB trying to interpret all of the symbolic characters incorrectly. Note the semicolons between elements to separate the list of characters. For clarification, I’m putting spaces between the semicolons, but that is not necessary.

emotions,: (“:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” )

one shot:
emotions:( “:'(“; “:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” )

So we have emotions. Let’s define a verb that cheers someone up by one level. Let’s call it lift.

Definitions in k and q use the colon notation. Just like lists. But we use curly braces to show that we are creating one unified action. We also can specify an argument name. I will use ‘ce’ which stands for current emotion.

lift:{[ce] emotions:( “:'(“; “:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” ); emotions[(emotions?ce)+1]}

That is the whole definition. As you can tell most of it is filled with defining the emotions. That’s this part:

lift:{[ce] emotions:( “:'(“; “:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” ); emotions[(emotions?ce)+1]}

The next bit uses the ? verb to search. It takes a list and returns the number of the first match. If there are no matches it returns the number of elements in the list. Which is one after the last element (remember everything is starts from zero so a list with ten items has an empty slot at 10).

lift:{[ce] emotions:( “:'(“; “:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” );
emotions[(emotions?ce)+1]}

The last bit increments the index by one and returns the emotion at that index. k,q automatically return the last statement in a verb definition.

lift:{[ce] emotions:( “:'(“; “:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” );
emotions[(emotions?ce)+1]}

For example lift “:(” returns

“:/”

What about if we reach the end or we search for an emotion not in the list? In both cases lift will simply return an empty string “”.

Well we all know you can lift someone’s spirits, but what about depressing them. It would seem we should be able to depress them and maybe by a certain quantity.

depress: {[ce;q] emotions:( “:'(“; “:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” ); emotions[(emotions?ce)-q]}

depress[“:|”;2]
returns “:(”

What we have done is add the argument q standing for quantity and now we can depress by a certain factor. Suppose we wanted to make a really simple version of depress that can only move an emotion down by one level. Let’s call it unlift.

unlift:{depress[;1]x}

Really short right?!

But let’s see what we did. First we took advantage of the fact that k,q has built in argument names for the first three arguments, they are x,y,z respectively. So I didn’t have to name it in the beginning explicitly like this

unlift: {[x] depress[;1] x}

Implicit Arguments: In k,q any verb you create can will automatically use x,  y,  z as the first 3 parameters. There are two equivalent ways to write a verb that converts the dimension to volume:

vol_3d:{[x;y;z] x*y*z}
vol_3d:{ x*y*z}

You can’t mix these two. So the following doesn’t work

vol_3d:{[h] x*y*h}

Now, to the really cool trick.  If you don’t fill in all the arguments, k will automatically create a new verb that is only missing the arguments you left out. So since I filled in that I would like to depress the emotional state by 1, and k doesn’t know which emotion I want to depress. It creates a new verb that depresses by 1. We then give that new verb a name and we now have a new verb. Note that if you get rid of the depress verb; k won’t be able to unlift. So technically, unlift has a dependency on depress.

This is known as function currying or projection in math and in some fancier programming languages, but don’t get scared by the name. It’s actually really simple. In APL, K, Q family this is almost always referred to as projection. The name comes from an analogy that most people are all familiar with. On a sunny day at high noon your shadow can follow you but can only move in two dimensions. It is projected onto the surface of the ground. The difference between your location and the location of the shadow, is that you can move in three directions {north,south} {west,east} and {jump(up,down)}. While your shadow can only move in two {north,south} {west,east}. The same idea is happening to a verb in K like addition (+), normally + adds two different numbers. But if we wanted to create verb named increment.
increment:+[1;]
This new verb always adds one to a number. Some of the flexibility is gone from the addition verb.

Finally (Footnote as promised) on the second definition of verb.

The second definition is the idea of taking a word that is not normally a verb and making it a verb. Well, remember lists. Lists had a very similar notation to how we use verbs.

For instance: emotions 3 or emotions [3] both return

“:|” the fourth element in the list.

In other words, lists are actually a special kind of verb! Specifically, they are verbs that transform a particular index number into whatever is at that place in the list. This applies even more broadly to a type of list in k,q called a map or dictionary. Here is a really simple example: (we use ! to define this structure)

emotionsdict: `crying`sad`annoyed`neutral`happy`vhappy`laughing`tofjoy! ( “:'(“; “:(” ; “:/” ; “:|” ; “:)” ; “:))” ; “:D” ; “:’)” )

crying  | “:'(“
sad     | “:(“
annoyed | “:/”
neutral | “:|”
happy   | “:)”
vhappy  | “:))”
laughing| “:D”
tofjoy  | “:’)”

and emotionsdict `crying returns
“:'(”

and emotionsdict?”:'(” returns
`crying

The point is that instead of just thinking of this as an emotion to emoticon dictionary we can also think of it as a verb that takes an emotion and displays it as emoticon. More on this here. Also our addition and multiplication tables behave like those functions. For instance add[2;3] returns 5.

This idea that all lists are really special verbs or that verbs are special types of lists is very freeing. Because the next time you want to make a list, first think is there a way for me to just make a verb.

For example, are the even numbers a list or a verb? They are both:

(0, 2, 4,6…) and {2*x}.

Which is easier?

K for Good Thought: [Part 1] k,q,kdb tutorial atoms, nouns

I am a new convert to the K/Q/J/APL cult. This makes me suited to write a tutorial for those who find this whole family of languages quite arcane.

The tutorial is called K for Good Thought, an homage to Ken Iverson’s notation as a tool for thought and the tutorials ‘learn x for great good’ meme. Leslie Lamport claims that until you have expressed your thoughts in writing you haven’t really thought. So there is something about being able to express yourself in K that allows for another way of thinking. Often the K solutions tend to be simpler and more general. Also, because K is a computer language interpreted by a machine that does exactly what you told it to do, it’s a much stricter reader of your thinking.

I should start by saying that this is a language that values brevity over almost everything. As Arthur Whitney said in an interview. If you can do the same thing and it’s shorter and not much slower he wants to hear about it. If it’s faster you better tell him.

Arthur Whitney is a man of few words and when a subject is very difficult and the explanation is dense and technical, few will march on. So with that, I aim to make this tutorial approachable for those who have had high school math and no programming experience. If  I venture out into jargon or CS concepts or high level math, I will try to explain the concepts first before diving into the syntax, though of course sometimes one will motivate the other. I believe that the less programming background you have the more likely the concepts will become natural to you since you won’t need to unlearn many bad habits.

There are many tutorials that give the history of these languages.  So I am not going to bother, except to say that apl was developed to allow programmers to work on the problem at hand and not on the low level implementation, though of course inevitably you will need to get your hands dirty with the inner workings of K as well, but those tend to be advanced topics. –quick credit to all the abridged and short materials by Arthur Whitney [AW] (K, Q, KDB)

Let us start at the beginning:

In the beginning, AW created atoms and they were good.

  • Atoms are indivisible.
  •  Atoms can be of different types.

    I promised not to use jargon, so when I do. You will find an explanation in a quote box like this:
    Types are things that allow us to distinguish between different kinds of data. That is, an Atom can be a number or letter or a timestamp. Not everything that you can think of is a type. For instance, perhaps you want to have an emotion as a type and you would use various emojis to represent them.
    [AW] has not seen it fit to add them to the types that K understands. However, you can create an emotional type. We will look at that later.

Some example atoms:

  • 5
  • 1b
  •  ‘string of letters’
  • `symbol
  • 2009.11.05D15:21:10.040666000

We cannot do much with atoms, just like we cannot really say much if we just had nouns. I guess we could classify different Nouns: person, place or thing.

K uses @: to interrogate and find out what something is. In q we can use ‘type’

Because K is terse it uses numbers to signal what type something is.

A negative sign in front means that the type is an atom.

  • @:5 = -7                                              An integer between negative 2^63 and positive 2^ 63
  • @:1b = -1                                            Boolean: Can be (1b) True or (0b) False
  • @: ‘s’ = -10                                           A charecter or letter
  • @:`symbol   = -11                                    A token (advanced) used to refer to one copy
  • @:2009.11.05D15:21:10.040666000 = -12  A Timestamp

Besides classifying we can make lists:

In English:

Animals: cat, dog, mouse

And in K we can do something similar.

  • integers: 1 2 3 4 5
  • animals: `cat `dog `mouse
  • alpha: (“a”;”b”;”c”)

Here something unexpected happens. While ‘animals’ is a list of symbols, ‘alpha’ is actually a list of characters which is what programmers call a string.

So alpha can also be written as “abc”. In fact, if you ask K to display alpha it will automatically display it as “abc” instead of “a”,”b”,”c”.

If we check what “abc” is

@:”abc” =  10

It’s positive. That’s because if the noun is actually a list then kdb will tell you what it’s a list of.

Oh by the way, I snuck in some notation. In English we often have a colon ‘:’ that tells us that a definition follows. Well K lets you do the same thing.

In K/Q all lists are semicolon(;) separated enclosed in parentheses. As a matter of convenience K /Q lets you define some common lists just by using spaces. So the following are equivalent.

(1;2;3) and 1 2 3  

So alpha is defined as a list of characters specifically a, b, c.

Lists in K are always ordered, which means that we can talk about the first or last element in a list.

In fact K has a couple of things things that are designed specifically for lists.

First lets introduce the count (!) in K or the til in q

This allows us to create lists of integers until that number starting from 0.

For example !10

0 1 2 3 4 5 6 7 8 9

til10: !10

til10

0 1 2 3 4 5 6 7 8 9

til10[1] /the second item on list; list start with zeroth item

1

til10[9] /the tenth item on the list; remember to start counting from 0

9

Let’s say we want just first 3 numbers

3#til10 /the pound sign or hashtag means ‘take’

0 1 2

-3#til10 /this takes the last 3 slots. The negative sign signifies that we are taking from the end of the list.

7 8 9

3_til10 /The underscore(_) is used to drop. if we only want what’s not in the first 3 slots

4 5 6 7 8 9

(3_til10) = (-7#til10)

1111111b

There is an equivalence between take and drop. Dropping the first 3 is equivalent to taking the last 7, if your list has 10 things. The equals (=) sign is used to compare things each slot from each list will be compared and return a 0 or 1. 1 if it is equal and 0 if the two elements in the slots are not equal.

In general we can say given a list with n items dropping the first x items is equivalent to taking the last n-x items. In our example, we have 10 items so dropping 3 is the same as taking the last 7.

What about reversing the order of a list?

In K it’s ‘|:’ in Q it is just called ‘reverse’.

reverse 7_til10

9 8 7

Finally, we can make lists of lists. In English we call these tables. In math we call them matrices.

m: (1 2 3; 4 5 6)

1 2 3
4 5 6

In K ‘+:’ is flip and in Q it is ‘flip’

in K: “+:m” is the same as “flip m” in Q

+:m
(1 4; 2 5; 3 6)
1 4
2 5
3 6

m[0] Returns the first noun, which is a list of numbers.

 1 2 3

Remembering the game battleship helps to see what is happening.

m[0][2]

3

BONUS SECTION

What if we wanted to make a list of lists of lists. We can! In math these are called tensors. In English these don’t really have a name, I think most people don’t have a use for them.

However, we all encounter these structures in the form of pages. Let me explain, If a line in a book is a list of words (or as k would have characters), then a page in a book is a list of lines and a collection of pages, a book, is a list of lists of lists. One way to reference a word in a book would be to tell you the page number (dimension 1), the line number (dimension 2), and the word number (dimension 3). To make this clear, lets do a pretty simple example. Let’s say that we wanted to write a math textbook and the first couple of pages were reference tables. The first table might be the addition table. The second table could be multiplication.

Let’s first build the addition table.

in K:

a:!10;add:(a+)’a

in Q:

a: til 10; add:(a+) each a;

add is now equal to

0 1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9 10
2 3 4 5 6 7 8 9 10 11
3 4 5 6 7 8 9 10 11 12
4 5 6 7 8 9 10 11 12 13
5 6 7 8 9 10 11 12 13 14
6 7 8 9 10 11 12 13 14 15
7 8 9 10 11 12 13 14 15 16
8 9 10 11 12 13 14 15 16 17
9 10 11 12 13 14 15 16 17 18

We can see that indexing add[3;2] will give us the cell containing 5, which is what we would expect. Cell add[2;3] will also equal 5, but I highlighted add[3;2], notice row column order. A good way to remember this is to think of a regular list as a vertical structure. If the list has lists inside, then to get to a particular row and column, we need to first go down the first list and then enter the second level list and go to the right. So the top level is always a row number.  Basically, the first index, indexes the first level list. A deeper list will be a deeper index. Page number, line number, word number.

Let’s now create the mltp table.

in K:

mltp:(a*)’a

in Q:

mltp:(a*) each a

mltp

0 0 0 0 0 0 0 0 0 0
0 1 2 3 4 5 6 7 8 9
0 2 4 6 8 10 12 14 16 18
0 3 6 9 12 15 18 21 24 27
0 4 8 12 16 20 24 28 32 36
0 5 10 15 20 25 30 35 40 45
0 6 12 18 24 30 36 42 48 54
0 7 14 21 28 35 42 49 56 63
0 8 16 24 32 40 48 56 64 72
0 9 18 27 36 45 54 63 72 81

So our reference tables can be a list of tables:

reftab:(add;mltp)

reftab[0] is the first table or the addition table.

reftab[1] is the second table or the multiplication table

I leave  creating a subtraction table in reftab[2] to the reader. (hint use the comma(,) to join a subtraction table to reftab. Like so reftab:reftab,subtr;)

You can make a division table as well, you will see how K/Q handles division by zero.

Now let’s say you wanted to play 3 dimensional battleship.

Each player could either attack the surface or the submarine  layer. Suppose in this really simple game you had two ships, each 2 units long. and there were only 6 squares. 2*3 board.

Here is a sample board:

Layer 0 or surface:

0 0 1

0 0 1

Layer 1 or submarine level

1 1 0

0 0 0

I have a submarine and a boat. Both are two units long. We need three coordinates to track down a single location.

First let’s make the surface layer

s: 0 0 1

sur: (s; s)

Next we can make the submarine layer

easy way

sub: (1 1 0; 0 0 0)

fancy way

sub: 2#|:+:sur,enlist 0 0 0 /Don’t worry I’ll explain.

sur, enlist 0 0 0 /gives us a three by three table. Note the comma used to append a list to a list.

(0 0 1;0 0 1;0 0 0)

+: sur, enlist 0 0 0 /flips the table on it’s side, sometimes called transpose

(0 0 0;0 0 0;1 1 0)

|: +: sur, enlist 0 0 0 /reverses the flipped table only reverses the top level list

(1 1 0;0 0 0;0 0 0)

2# |: +: sur, enlist 0 0 0 /takes the first 2 rows of the flipped table

(1 1 0;0 0 0)

Now we can make the whole board

b: (sur; sub)

((0 0 1;0 0 1);(1 1 0;0 0 0))

b[0]

(0 0 1;0 0 1)

b[0][1]

0 0 1

b[0][1][2]

1

And obviously we can make lists of lists of lists of lists…. and everything will work the same. But I’m stopping here.