The problem with automatic derivation

Before we start though, we should specify what we mean by auto-derivation. Auto-derivation is often confused for plain old derivation, so here are some examples to tell them apart.

For these examples, assume that there is a typeclass defined as:

The following is plain old manual instance definition. There is no derivation of any kind and we must write out exactly how JsonParser works for User.

Next is simple, manual derivation, NOT automatic. Notice that you have to call deriveInstance, but you don't have to specify how it is derived. Some macro system is likely used underneath.

And finally, we have automatic derivation:

Notice that for automatic derivation, we never defined the instance, and yet when we call parse, an instance is found. This is actually very problematic and we will explain why below.

How it works

For us to understand why it is problematic, we need to understand how it works. If an instance is required (like when we call JsonParser.parse[User]), the Scala compiler will look for implicit instances in the following order:

• Look at the local scope (locally defined instances, meaning instances defined in the same scope where we call parse)
• Look at the imports (instances imported, like import cats.implicits._)
• Look at the companion object of the type being requested and the companion object of the typeclass being requested (like the implicit vals we defined in the companion objects of User, and the companion object of JsonParser)

This list is simplified as there are other special cases around objects extending traits, but that’s outside of the scope of this post. Just know that a local instance will override (with no warning) an imported one, and an imported one will override one in a companion object, and there lies the issue.

How automatic derivations works

Automatic derivation libraries normally have a macro or some other system to automatically generate instances for types. There are two kinds of systems: automatic via companion types and automatic via an import: both of these systems are problematic.

Automatic derivation via companion object of the typeclass

This systems simply places the automatic derivation on the typeclass’ companion object, like so:

The main problem here is that this will make all instances defined in the User’s companion object conflict with it (with a compile error). Essentially forcing us to manually import the instances (since imports take precedence) whenever we want to manually define the instance of a type.

Automatic derivation via an import

Another popular system via imports. For this the user is required to import an auto package like so:

This system is both better and worse than the previous one. Better because you can just not import it, but worse because it can cause inconsistent behavior.

Remember that imports take precedence over companion objects, this means that if you create an instance on the companion object of User, JsonParser.parse[User] will use it. But if you purposely or accidentally import auto somewhere in you code, the behavior of JsonParser.parse[User] will change! This can lead to JsonParser.parse[User] doing different things in different areas of the code. And since imports take precedence, there will be no warning or error.

Some libraries that use this system also export an semi package, which is the same thing but without the implicit, allowing you better control over derivation.

What should we use?

Use manual derivation, always. If there is no option for manual derivation, then let the library’s authors know about this issue (and maybe make a PR of your own!). Automatic derivation seems great at first, even magical, but eventually the business will require something that cannot be derived and must be manually defined. For example, an email address which you want to verify during parsing.

Assuming we have a json object {"email": "hello@example.com"}. Our business requires that you separate the host name (example.com) from the user (hello) and you decide to do that during parsing.

With “Automatic derivation via companion object of the typeclass”, this couldn’t even be done, we would get a compile error. There are tricks we can do with objects and extensions of traits, but those tricks aren’t reliable.

With “Automatic derivation via an import” you would get inconsistent behaviour. Anywhere auto._ isn't imported will work as expected, but if auto._ is imported (purposely or accidentally), the behavior will change from callsite to callsite.

So to all library authors out there: please stop implementing automatic derivation in your libraries.

scalaz-deriving

There are libraries that will ease the tediousness of using manual derivation. In the DSS AdTech team, we have been using scalaz-deriving ( https://github.com/scalaz/scalaz-deriving). That library takes a configuration file which maps typeclasses to their derivation functions, allowing you to write something similar to what you would write in Haskell (where all of this originated):