Fabian Zeindl

Fabian Zeindl

← back to homepage

Type-safe, multiplatform JSON serialization for Kotlin, generated at compile-time

For discussion and comments please contact me.

Table of contents

Introduction

This project is available on Github: kotlin-json-stream-serializer

There is also a sample available: kotlin-json-stream-serializer-sample

For quite some time I used Gson for serialization in Kotlin and Java applications. It was the most brittle part of the code, for several reasons:

  • When Gson doesn’t find a no-arg constructor, sun.misc.Unsafe is used. This means initializers aren’t called and the deserialized objects could be broken.
  • Gson uses reflection to set and retrieve fields. Reflection is slow on Android.
  • Nulls in JSON can to break Kotlin’s null-safe typing.

To improve on that situation, I decided to write a KSP plugin that generates serialization code at compile-time.

Design Choices

Serialization from Streams using OKIO

My library kotlin-json-stream is used to serialize and deserialize in a stream-based fashion directly from and to OKIO buffers without in-memory structures. This skips many allocations and allows for infinitely large JSON documents as long as the target object fits into memory.

Additionally, the plugin is able to fail early and output useful error messages. If an unexpected JSON token is read from a buffer, deserialization stops immediately.

All necessary data needs to be in the constructor

I decided to instantiate deserialized objects with an existing constructor only (the one with the most arguments). This ensures that they are always initialized properly.

The only needed annotation is @Ser which goes on the top of a class:

@Ser
class MyClass(val userName: String,
              val notifications: List<String>)

This forces the separation of initial from generated data. It also removes the need to exclude lazy or @Transient properties.

The generated serializer is an extension function on JsonWriter:

fun JsonWriter.valueMyClass(obj: MyClass?) {
    if (obj == null) {
        nullValue()
        return
    }
    beginObject()
    name("userName").value(obj.userName)
    name("notifications").value(obj.notifications, JsonWriter::valueMyClass)
    endObject()
}

The field names and types are taken from the constructor.

The code assumes they are also accessible as members, which is easy to obtain.

Developer-friendly deserialisation

Deserialisation is more complex since it does a couple of things to help development:

  • Field names and enum-values are deserialized case-insensitively. This is debatable, but it prevents some flow-breaking bugs and was never an issue in production. Serialization keeps the original casing.

The deserializer:

fun JsonReader.nextMyClass(): MyClass {    
    var userName: String? = null
    var notifications: MutableList<String>? = null    beginObject()
    while (hasNext()) {
        when (nextName().toLowerCase()) {
            "username" -> userName = nextString()
            "notifications" -> notifications = nextList(JsonReader::nextString)
            else -> skipValue()
        }
    }
    endObject()    
    
    val obj = MyClass(
        userName = checkNotNull(userName) {
            "error parsing MyClass, field not found: userName: String"},
        notifications = checkNotNull(notifications) {
            "error parsing MyClass, field not found: notifications: List"}
    )    
    return obj
}

Nested objects and custom types

Since all methods are created as extensions on JsonWriter and JsonReader adding support for more fields is as simple as creating a new extension method or adding @Ser to the referenced class:

@Ser
class MyClass(val user: UserInfo,
              val time: ZonedDateTime)
			   
class UserInfo(name: String)

The generated code for this example won't compile because it misses:

  • JsonWriter.valueUserInfo(UserInfo)
  • JsonReader.nextUserInfo()
  • JsonWriter.valueUserInfo(ZonedDateTime)
  • JsonReader.nextZonedDateTime()

To make it work add @Ser to UserInfo and add the following code for ZonedDateTime.

fun JsonWriter.valueZonedDateTime(obj: ZonedDateTime) {
    value(obj.toString())
}

fun JsonReader.nextZonedDateTime(): ZonedDateTime {
    return ZonedDateTime.parse(nextString())
}

Abstract Types

Reading and writing abstract types is necessary and possible, consider this:

@Ser
class MyClass(val users: Set<User>)

@Ser
interface User

@Ser
data class Customer(val username: String, val password: String) : User

The plugin checks the type hierarchy and writes the object and type name into an array. I chose an array over an object, so I wouldn’t have to cache anything in case the type-name arrives after the data during deserialisation.

fun JsonWriter.valueUser(obj: User?) {
    if (obj == null) {
        nullValue()
        return
    }
    beginArray()
    when (obj) {
        is Customer -> value("package.Customer").value(obj)
        else -> throw IllegalStateException("type $obj not annotated")
    }
    endArray()
}

/* deserialize */
fun JsonReader.nextUser(): User {
    beginArray()
    val typeName = nextString()
    val obj: User = when (typeName) {
        "package.Customer" -> nextCustomer()
        else -> throw IllegalStateException("unknown type: $typeName")
    }
    endArray()    
    return obj
}

Graphs and circular objects

Using another annotation @ParentRef, the plugin can process circular graphs, like a leaf node having a reference to its root.

@Ser
class Root(var leaf: Leaf?)

@Ser
data class Leaf(@ParentRef val root: Root, val number: Int)

The deserializer for Leaf returns a function now:

fun JsonReader.nextLeaf(): (root: Root) -> Leaf {
    var number: Int? = null    
	
	beginObject()
    while (hasNext()) {
        when (nextName().toLowerCase()) {
            "number" -> number = nextInt()
            else -> skipValue()
        }
    }
    endObject()    
	
	return { root->
        Leaf(
            root = root,
            number = checkNotNull(number) {
                "error parsing Leaf, field not found: number: Int"
            }
        )
    }
}

For type-safe deserialization to work, the field in the parent object must be either nullable or a collection type:

fun JsonReader.nextRoot(): Root {    

	var leaf: ((Root) -> Leaf)? = null
    
    beginObject()
    while (hasNext()) {
        when (nextName().toLowerCase()) {
            "leaf" ->  {
                 leaf = nextOrNull(JsonReader::nextLeaf)
             }
            else -> skipValue()
        }
    }
    endObject()    
	
	/* construct parent */
    val obj = Root(
        leaf = null
    )    
	
	/* set child */
    obj.leaf = leaf?.invoke(obj)    
	
	return obj
}

Summary

This was a short overview of kotlin-json-stream-serializer. Go check out the sample at kotlin-json-stream-serializer-sample

The choice of compile-time-generation and Kotlin with its extensions methods and lambdas makes it possible to easily create convenient, type-safe, and fast serialization code. Assuming all necessary data to be passed in the constructor helps to keep everything safe and enforces separation of initial from generated data.

← Back to home