Thursday, June 11, 2009

Using Implicits to Select Types

Scala 2.8's collection revision is doing interesting things. The most obvious one will be the methods revision. One thing that shouldn't be all that obvious, though, is how much code will be shared, and how that shared code will still look like it was specific to a class.

I'll quote here a couple of examples from the Scala 2.8 Collection's Whitepaper (pdf at the bottom):



scala> "abc" map (x => (x + 1).toChar)
res1: scala.runtime.RichString = bcd

scala> "abc" map (x => (x + 1))
res2: scala.collection.immutable.Vector[Int] = Vector(98, 99, 100)

scala> Map("a" -> 1, "b" -> 2) map { case (x, y) => (y, x) }
res3: scala.collection.immutable.Map[Int,java.lang.String] = Map(1 -> a, 2 -> b)

scala> Map("a" -> 1, "b" -> 2) map { case (x, y) => y }
res4: scala.collection.immutable.Iterable[Int] = List(1, 2)


Now, this "map" method is inherited from a common trait, and yet it knows when the result of a map over a RichString is a RichString, when the result of a map over a Map is a Map, and when it's not. How can it be?

The answer is implicit parameters. One thing about implicits, is that a more specific implicit has preference over a more generic one. Take a look at this example:



scala> abstract class X[T] { def id : Unit }
defined class X

scala> implicit def a[T] = new X[T] { def id = println("generic") }
a: [T]X[T]

scala> implicit def b = new X[Int] { def id = println("Int") }
b: X[Int]

scala> def f[T](a : T)(implicit g : X[T]) = g.id
f: [T](T)(implicit X[T])Unit

scala> f(5)
Int

scala> f('c')
generic


Both "a" and "b" have the correct type, X[T]. The second definition, though, will return X[Int], which is more specific than X[T], which the first definition returns. So, whenever X[Int] is required, "b" will take precedence over "a". Naturally, if some other type of X is required, "b" won't be used at all.

This is how map can do its trick. To compose the result, it uses methods from the class Builder[-Elem, +To]. Such a build will create a collection of type To from elements of type Elem. Now, this is not enough to solve map's problem. For example, I want a RichString as result, if I'm doing a map over a RichString, but not if I'm mapping over a List of Chars. See the two examples below:



scala> "abc" map (x => (x + 1).toChar)
res1: scala.runtime.RichString = bcd

scala> List('a', 'b', 'c') map (x => (x + 1).toChar)
res11: List[Char] = List(b, c, d)


So, for methods with more complex semantics, like map, we need something else which can produce the correct Builder. For that, the trait BuilderFactory[Elem,
+To, From] exists. BuilderFactory will produce a Builder for a method which takes a collection of type From, and uses elements of type Elem to produce a collection of type To.

Now, the more generic implicit for a BuilderFactory is BuilderFactory[A, Iterable[A], Iterable[_]], which should be pretty obvious. Other implicits exists, though, such as BuilderFactory[Char, RichString, RichString]. Now, look at map's declaration:



def map[B, That](p: Elem => B)(implicit bf: BuilderFactory[B, That, This]): That


So, when you do "abc".map( x => (x + 1).toChar), the compiler knows that "B" is Char (the result of the function being mapped). "This" is known to be RichString, from the definition of class RichString itself. Now, given that "B" = Char and "This" = RichString, then BuilderFactory[Char, RichString, RichString] is a more specific match than the other implicit definitions. And, because it gets used, "That" gets defined to be RichString also, thus defining the result type for that map.

The cleverness of it is to use implicits to define unknown type parameters as a function of known type parameters. This is can be a very useful pattern for those who want to take maximum advantage of static type checking.

Case Classes and Product

It's a well-known fact that by adding "case" before "class" in Scala, the compiler will generate some scaffolding for you like making all your paremeters "val", creating apply and unapply in a companion object. etc.

It's not a particular known fact that one of the things it does is extending the class with the trait Product. This trait makes it easy to access the elements of the class, like in the example below:



scala> trait recordType {
| self : Product =>
| def records : String = (
| for(parm <- self.productIterator)
| yield parm.asInstanceOf[AnyRef].getClass.getSimpleName+": "+parm
| ) mkString "\n"
| }
defined trait recordType

scala> case class X(i : Int, c : Char, s : String) extends recordType
defined class X

scala> val x = X(5, 'd', "this")
x: X = X(5,d,this)

scala> println(x.records)
Integer: 5
Character: d
String: this

Let me talk about the trait recordType a bit first. Its body first line declares that "self : Product =>". This has two purposes. First, it creates an alias for "this". Second, and most important, it tells the compiler that to be able to use this trait, a class must mix in the trait Product.

Anyway, the code works fine for simple classes, but it might be weird for collections:



scala> case class Y(a : Array[String], l : List[Int]) extends recordType
defined class Y

scala> val y = Y(Array("this", "example"), List(1,2,3))
y: Y = Y([Ljava.lang.String;@60a517,List(1, 2, 3))

scala> println(y.records)
String[]: [Ljava.lang.String;@60a517
$colon$colon: List(1, 2, 3)

Here, Array is mucked up because Java's Array are mucked up. As for the List, any non-empty List actually belongs to class "::", a subclass of List.

Sunday, June 7, 2009

Function.tupled, Function.curried

Scala's library hides small gems of which little is said. Let me speak of a personal favorite here, the object Function.

This object has a number of methods to construct a function from another function (or others, in the case of "chain").  They are variations of "curried", "tupled", "uncurried" and "untupled". As their names makes clear, they transform a function whose arguments are not curried or are not a tuple into a function whose arguments are curried, or which receives a tuple as an argument. Take, for instance, this definition of zipMap, which applies a function of two arguments between two lists:



scala> def zipMap[A,B,C](l1 : Seq[A], l2 : Seq[B])(f : (A,B) => C) =
| l1 zip l2 map (x => f(x._1, x._2))
zipMap: [A,B,C](l1: Seq[A],l2: Seq[B])(f: (A, B) => C)Sequence[C]

scala> zipMap (List(1, 3, 9), List(5, 2, 3)) ((x,y) => x max y)
res5: Sequence[Int] = List(5, 3, 9)


This is not a bad definition, but accessing the tuple arguments to pass to f is too much mechanics for my taste. So, instead, we could do this:



scala> def zipMap[A,B,C](l1 : Seq[A], l2 : Seq[B])(f : (A,B) => C) =
| l1 zip l2 map Function.tupled(f)
zipMap: [A,B,C](l1: Seq[A],l2: Seq[B])(f: (A, B) => C)Sequence[C]

scala> zipMap (List(1, 3, 9), List(5, 2, 3)) ((x,y) => x max y)
res4: Sequence[Int] = List(5, 3, 9)


Much better, don't you think? By the way, this was done on 2.8, where Seq has the method zip.

Friday, June 5, 2009

Scalable Algorithms 3

Back to matching shortcuts to menu options, I decided my best option would be a deterministic finite automaton. It would make excellent use of partial input, and take much less space than a set of all possible inputs accepted -- to make use of state machine lingo -- by each menu option.

Now, anyone who has ever worked with DFA before will tell you that the trick to build one is to first generate the regular expression which represents what you want to match. Yes, that's your good, old RegExp. Well, not fully... you can't have back references, for one thing. But most of what you think of as regexp will do.

So, what regular expression would match our input? Let's take a small menu option: "New Tab". We want to accept either word as well as its abbreviation, so "New", "Tab" and "NT" should be matched. Any prefix of either word should be taken as well, so we add "N", "Ne", "T" and "Ta" to it. Finally, we want to combine word prefixes, like "NTab" or "NeTa", you get the idea.

So, let's get the first word, and see how far we can take. We have "N", "Ne" and "New". So "N" is mandatory, and "e" is optional. That would be "Ne?" in regexp. Next, "w" is also optional, but it can only appear if "e" appears as well. That's "N(ew?)?" in regexp, or, to make regular use of parenthesis, "N(e(w)?)?". Now, since we may as well only match the second word, the whole first word is optional, which gives us "(N(e(w)?)?)?".

Naturally, each word would follow the same pattern, but what about the whole menu option? As it happens, you only have to concatenate the patterns: "(N(e(w)?)?)?(T(a(b)?)?)?". It should be pretty simply to create such a function.

In fact, we can do it using foldRight. As you know, foldRight combines things from right to left, using the result of the last operation and the next element. Now, it may not be immediately obvious what the operation is, so let's assume we already processed "w". That means we have "(w)?" as the last result, and "e" as the next letter, and the result we want to come up with is "(e(w)?)?". It's trivial to see, then, that the operation is "("+"e"+"(w)?"+")?".

Now that we know what the operation is, let's think about the initial "result" that fold takes as parameter. We want x in "("+"w"+x+")?". Since that must be equal to "(w)?", x = "".

Ok, so now we know what the regular expression for a menu option must look like, as well as how to build one from its string. We can now proceed to building a DFA.

But... we can match regular expressions to strings just as well as DFA. I might not take advantage of partial inputs, and I can't go the next step, which is creating one DFA which will accept all menu options and, as an added bonus, have the associated menu options at each state.

Still, how hard would it be to match against regexp? Well, let's try...



val wordsep = "\\W+"
def words(s : String) = s.split(wordsep)
def phraseRegex(s : String) =
"^" + words(s).map(_.foldRight("")("(" + _ + _ + ")?")).mkString
def regexList(l : List[String]) = l.map(s => phraseRegex(s) -> s)
def regexSearcher(l : List[String]) = {
val r = regexList(l)
(s : String) => r.filter(s matches _._1).map(_._2)
}


That's it. You could, and should improve it by pre-compiling the regexp expressions. It's a bit uglier, but for the sake of doing things right, I'll post it at the end. Anyway, this is fairly inexpensive in terms of memory usage, and regexp matches are very fast. I doubt it can beat a hash tree, but it should give it a run for its money. Still, the fact that you'll be matching every menu option against the input is problematic, and the hash tree doesn't suffer that problem. The setup time for this algorithm should be pretty fast too, even precompiling the regexp.

Anyway, it's short, it's obvious, and it's not too shabby. We'll next get into DFA, and do it by using Scala's standard library. That should be the worst of all algorithms, but a stepping stone to the last one I'll present. For now, here's the revised code.



val wordsep = "\\W+"
def words(s : String) = s.split(wordsep)
def phraseRegex(s : String) =
("^" + words(s).map(_.foldRight("")("(" + _ + _ + ")?")).mkString).r
def regexList(l : List[String]) = l.map(s => phraseRegex(s) -> s)
def regexSearcher(l : List[String]) = {
val r = regexList(l)
(s : String) => r.filter(_._1.findPrefixOf(s) == Some(s)).map(_._2)
}