scala, existential type and the old precog blog post

There was a nice scala article on the precog blog before precog was purchased. The blog seems to no longer be available.

I recently had a need to compose some REST services and needed a configuration object that represented the configuration properties for each REST layer. I used the precog existential type approach for this. Since the blog is no longer avaliable, I thought I would write down what I did and offer it up as a replacement for the original precog blog entry.

The approach is nothing more than what was described in a paper on the expression problem awhile ago here.

The basic idea is that you could use an existential type to define the configuration parameters needed for different layers in your "REST" service. While the traditional cake solutions may restrict your flexibility, a small variation may be helpful in certain circumstances. While I won't claim that all situations call for the approach that precog used in the original blog, I am reproducing it, to some degree here.

The Problem

The basic problem to solve is how to create a configuration object that is the union of all configuration information needed for each component in your application. While you could and probably should use typesafe's config library to read configuration information from flatfiles deployed with your application, you want a strongly typed representation of the configuration information to use in your layers.

For example if you have two layers with configuration parameters:

trait Layer1 {
  val l1param1: String
  val l1param2: Int
  ...
}
trait Layer2 {
  val l2param1: String
  val l2param2: String
  ...
}

you could envision creating a config object like

trait Layer1 {
  class Layer1Config(l1param1: String, l1param2: Int)
  val l1config: Layer1Config
  ...
}
trait Layer2 { 
  class Layer2Config(l2param1: String, l2param2: String)
  val l2config: Layer2Config
  ...
}

and so on to create your configuration objects. If you now mix these together into a REST service trait

trait REST extends Layer1 with Layer2 {
  val l1config: ...
  val l2config: ...
}

The fundamental problem is expressed as the "expression" problem: How do you extend the configuration information in the dimension of data variants as well as new processors. The precog blog, I seem to recall, was thinking about data extensibility. The other aspect of the "expression" problem is how to extend these things independently of other modules. At some point, for real-word sized problems the approach above (using separate config objects all scattered about) will not work especially as the trait start becoming mixed together. While this may not be apparent in this simple "configuration" example, its a real problem in software.

The Adaptable Type

The story goes something like this. Suppose you want each of the components to have a configuration object.

trait Configuration { 
  type Config
  def config: Config
}

Now, we want each layer to be forced to create a configuration object

trait Layer1 extends Configuration {
  type Config <: Layer1Config

  trait Layer1Config { 
     val l1param1: String
     val l2param2: int
   }
}

trait Layer2 extends Configuration { 
   type Config <: Layer2Config

  trait Layer2Config { 
     val l2param1: String
     val l2param2: String
  }
}

Now we want to combine these layers into a REST client that can be instantiated (hence we make it a class):

class REST extends Layer1Config with Layer2Config with (...more layers here...) {
   type Config = config.type

  object config extends Layer1Config with Layer2Config {
     val l1param1: String = ...
     val l1param2: Int = ...
     val l2param1: String = ...
     val l2param2: String = ...
   }
}

By defining an object in the REST layer that integrates the configuation information from the upper layers, we can finally declare, at the end of the world, the type for the Config object. A path dependent type is used. Since config is an object, its type is available to declare the type Config type required by the Config trait.

Essentially, each layer does not know about the other layer's configuration information but we can combine them together at exactly the point we need the configuration centralized into one place.

You may scratch your head because it may seem like its just easier to have each configuration object declared int the REST layer and in some ways you are right for some problems. It's also easy to see that if you use subclasses of the layer, such asLayer3 extends Layer2 then standard OO techniques for extending the Config object can be used. In other words, we can extend downward through a subclass, which is fairly common, and we can extend by combining together independent extensions, which is less common and harder to do in some languages.

That's it! I used this recently, but could not find the precog blog anymore so wrote this one. I use typesafe's config library to grab some of these parameters from configuration files but I also sometimes just type them in since recompiliation is easy and meets the needs for some deployments.

There are other ways to formulate your problem and other techniques but this one comes in handy once in awhile in a variety of areas.