All About Proto DataStore

In this post, we will learn about Proto DataStore, one of two DataStore implementations. We will discuss how to create it, read and write data and how to handle exceptions, to better understand the scenarios that make Proto a great choice.

Proto DataStore uses typed objects backed by Protocol Buffers, to store smaller datasets while providing type safety. It removes the need for using key-value pairs, making it structurally different from its  predecessor and its sibling implementation, Preferences DataStore. However, that’s not all — DataStore brings many other improvements over . Feel free to quickly jump back to our first post in the series and take a look at the detailed comparison we’ve made there. Going forward, we will refer to  as just , unless specified otherwise.

Partner with aster.cloud
for your next big idea.
Let us know here.

To sum up:

  • Provides a fully asynchronous API for retrieving and saving data, using the power of Kotlin coroutines
  • Does not offer ready-to-use synchronous support — it directly avoids doing any work that blocks the UI thread
  • Relies on Flow’s inner error signalling mechanism, allowing you to safely catch and handle exceptions when reading or writing data
  • Handles data updates safely in an atomic read-modify-write operation, providing strong ACID guarantees
  • Allows easy and simple data migrations
  • Need full type safety and your data requires working with more complex classes, like enums or lists? This isn’t possible with Preferences, so choose Proto instead

Intro to Protocol Buffers

To use Proto DataStore, you need to get familiar with Protocol Buffers — a language-neutral, platform-neutral mechanism for serializing structured data. It is faster, smaller, simpler and less ambiguous than XML and easier to read than other similar data formats.

You define a schema of how you want your data to be structured and specify options such as which language to use for code generation. The compiler then generates classes according to your specifications. This allows you to easily write and read the structured data to and from a variety of data streams, share between different platforms, using a number of different languages, like Kotlin.

Example schema of some data in a  file:

// Copyright 2022 Google LLC.	
// SPDX-License-Identifier: Apache-2.0
   
message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

How to use the generated Kotlin code for constructing your data model:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */
   
val person: Person = 
  Person.newBuilder()
    .setId(1234)
    .setName("John Doe")
    .setEmail("[email protected]")
    .build()

Or you can try out the newly announced Kotlin DSL support for protocol buffers for a more idiomatic way of building your data model:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

val person = person {
    id = 1234
    name = "John Doe"
    email = "[email protected]"
}

Investing a bit more time into learning this new serialization mechanism is definitely worth it as it brings type safety, improved readability and overall code simplicity.

Proto DataStore dependency setup

Now let’s look at some code and learn how Proto works.

We will use the Proto DataStore codelab sample. If you’re interested in a more hands-on approach with implementation, we really encourage you to go through the Working with Proto DataStore codelab on your own.

This sample app displays a list of tasks and the user can choose to filter them by their completed status or sort by priority and deadline. We want to store their selection — a boolean for displaying completed tasks and a sort order enum in Proto.

Read More  Google I/O 2019 | Best Practices for Developing on Android TV

We will firstly add Proto dependencies and some of the basic protobuf settings to your module’s . If you’re interested in a more advanced customisation of the protobufs compilation, check out the Protobuf Plugin for Gradle notes:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

plugins {
    …
    id "com.google.protobuf" version "0.8.17"
}

dependencies {
    implementation  "androidx.datastore:datastore:1.0.0"
    // Starting from Protobuf 3.8.0, use the lite runtime library
    implementation  "com.google.protobuf:protobuf-javalite:3.18.0"
    ...
}

protobuf {
    // Configures the Protobuf compilation and the protoc executable
    protoc {
        // Downloads from the repositories
        artifact = "com.google.protobuf:protoc:3.14.0"
    }

    // Generates the java Protobuf-lite code for the Protobufs in this project
    generateProtoTasks {
        all().each { task ->
            task.builtins {
            // Configures the task output type
                java {
            // Java Lite has smaller code size and is recommended for Android
                    option 'lite'
                }
            }
        }
    }
 }

💡 Quick tip — if you want to minify your build, make sure to add an additional rule to your  file to prevent your fields from being deleted:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */
   
-keepclassmembers class * extends androidx.datastore.preferences.protobuf.GeneratedMessageLite {
    <fields>;
}

Protobuf setup for Proto DataStore

Our journey with Proto starts by defining the structure of your persisted data in a  file. Think of it as a readable schema for you and a blueprint for the compiler. We will name ours  and add it to the  directory.

Following the Protobuf language guide, in this file we will add a message for each data structure we want to serialize, then specify a name and a type for each field in the message. To help visualize this, let’s look at both a Kotlin data class and a corresponding protobuf schema.

 — Kotlin data class:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

data class UserPreferences(
    val showCompleted: Boolean,
    val sortOrder: SortOrder
)

enum class SortOrder {
    UNSPECIFIED,
    NONE,
    BY_DEADLINE,
    BY_PRIORITY,
    BY_DEADLINE_AND_PRIORITY
}

 —schema:

// Copyright 2022 Google LLC.	
// SPDX-License-Identifier: Apache-2.0

syntax = "proto3";
option java_package = "com.codelab.android.datastore";
option java_multiple_files = true;

message UserPreferences {

    bool show_completed = 1;
    SortOrder sort_order = 2;

    enum SortOrder {
        UNSPECIFIED = 0;
        NONE = 1;
        BY_DEADLINE = 2;
        BY_PRIORITY = 3;
        BY_DEADLINE_AND_PRIORITY = 4;
    }
}

If you haven’t used protobufs before, you might also be curious about the first few lines in the schema. Let’s break them down:

  •  — specifies that you’re using  syntax
  •  — file option that specifies package declaration for your generated classes, which helps prevent naming conflicts between different projects
  •  — file option that specifies whether only a single file with nested subclasses will be generated for this  (when set to false) or if separate files will be generated for each top-level message type (when set to true); it is false by default

Next is our message definition. A message is an aggregate containing a set of typed fields. Many standard simple data types are available as field types, including , , , double, and . You can also add further structure to your messages by using other message types as field types, like we did with .

The ,  markers on each element identify the unique “tag” that the field uses in the binary encoding — like an ID of sort. Once your message type is in use, these numbers should not be changed.

Read More  How To Install And Configure MySQL In Ubuntu

When you run the protocol buffer compiler on a  the compiler generates the code in your chosen language. In our specific case, when the compiler is run, this leads to the generation of the  class, found in your app’s … directory:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

public  final class UserPreferences extends GeneratedMessageLite<...>
    implements UserPreferencesOrBuilder {
        private UserPreferences() {}
        …
}

💡 Quick tip — You can also try out the newly announced Kotlin DSL support for protocol buffers to use a more idiomatic way of building your data model.

Now that we have , we need to specify the guidelines for how Proto should read and write them. We do this via the DataStore  that determines the final format of your data when stored and how to properly access it. This requires overriding:

  •  — what to return if no data is found
  • — how to transform the memory representation of our data object into a format fit for storage
  •  — inverse from the above, how to transform from a storage format into a corresponding memory representation
/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

object UserPreferencesSerializer : Serializer<UserPreferences> {
    
override val defaultValue: UserPreferences = UserPreferences.getDefaultInstance()

    override suspend fun readFrom(input: InputStream): UserPreferences {
        try {
            return UserPreferences.parseFrom(input)
        } catch (exception: InvalidProtocolBufferException) {
            throw CorruptionException("Cannot read proto.", exception)
        }
    }

    override suspend fun writeTo(t: UserPreferences, output: OutputStream) =
        t.writeTo(output)
}

To keep your code as safe as possible, handle the  to avoid unpleasant surprises when a file cannot be de-serialized due to format corruption.

💡 Quick tip — If at any point your AS is unable to find anything  related, clean and rebuild your project to initiate the generation of the protobuf classes.

Creating a Proto DataStore

You interact with Proto through an instance of .  is an interface that grants access to the persisted information, in our case in the form of the generated .

To create this instance, it is recommended to use the delegate  and pass mandatory  and  arguments:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

private val Context.userPreferencesStore: DataStore<UserPreferences> by dataStore(
    fileName = DATA_STORE_FILE_NAME,
    serializer = UserPreferencesSerializer,
    …
)

 is used to create a  used to store the data. This is why the  delegate is a Kotlin extension property whose receiver type must be an instance of , as this is needed for the File creation via . Avoid using this file in any way outside of Proto, as it would break the consistency of your data.

In the  delegate, you can pass one more optional argument — . This handler is invoked if a  is thrown by the serializer when the data cannot be de-serialized.  would then instruct Proto how to replace the corrupted data:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

corruptionHandler = ReplaceFileCorruptionHandler(
    produceNewData = { UserPreferences.getDefaultInstance() }
)

You shouldn’t create more than one instance of DataStore for a given file, as doing so can break all DataStore functionality. Therefore, you can add the delegate construction once at the top level of your Kotlin file and use it throughout your application, in order to pass it as a singleton. In later posts, we will see how to do this with dependency injection.

Read More  Using A Subflow In Dialogflow CX

Reading data

To read the stored data, in  we expose a from . This provides efficient access to the latest saved state and emits with every change. This is one of the biggest strengths of Proto — your ’s values already come in the shape of the generated . This means you don’t have to do any additional transformations from the saved data into a Kotlin data class model, like you would with  or :

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

// Don't
suspend fun fetchCachedPrefs(scope: CoroutineScope): StateFlow<UserPreferences> =
   userPreferencesStore.data.stateIn(scope)

// Do
suspend fun fetchInitialPreferences() = userPreferencesStore.data.first()

The Flow will always either emit a value or throw an exception when attempting to read from disk. We will look at exception handling in later sections. DataStore also ensures that work is always performed on  so your UI thread isn’t blocked.

🚨 Do not create any cache repositories to mirror the current state of your Proto data. Doing so would invalidate DataStore’s guarantee of data consistency. If you require a single snapshot of your data without subscribing to further Flow emissions, prefer using :

Writing data

For writing data, we will use a suspend  function.

Let’s break that down:

  •  interface — we’re currently using  as the concrete Proto implementation
  • — a suspend block used to apply the specified changes to our persisted data of type T

Again, you might notice a difference to  which relies on using  and , similar to  and , as the default data representation.

We can now use this to change our  boolean. Protocol buffers simplify this as well, removing the need for any manual transformation from and to data classes:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

suspend fun updateShowCompleted(showCompleted: Boolean) {
      userPreferencesStore.updateData { currentPreferences ->
            currentPreferences.toBuilder().setShowCompleted(completed).build()
        }
}

There’s a few steps to analyze:

  • — gets the  version of our  which “unlocks” it for changes
  •  — sets the new value
  •  — finishes the update process by converting it back to 

Updating data is done transactionally in an atomic read-modify-write operation. This means that the specific order of data processing operations, during which the data is locked for other threads, guarantees consistency and prevents race conditions. Only after the  and  coroutines complete successfully, the data will be persisted durably to disk and  Flow will be reflecting the update.

🚨 Keep in mind that this is the only way of making changes to the DataStore state. Keeping a  reference and mutating it manually after  completes will not change the persisted data in Proto, so you shouldn’t attempt to modify  outside of the  block.

If the writing operation fails for any reason, the transaction is aborted and an exception is thrown.

Migrate from SharedPreferences

If you’ve previously used  in your app and would like to safely transfer its data to Proto, you can use . It requires a context,  name and an instruction on how to transform your  key-value pairs to  within the  parameter. Pass this via the  parameter of the ​​ delegate to migrate easily:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

produceMigrations = { context ->
    listOf(
        SharedPreferencesMigration(
            context,
            USER_PREFERENCES_NAME
        ) { sharedPrefs: SharedPreferencesView, currentData: UserPreferences ->
            ...
                currentData.toBuilder().setSortOrder(
                    SortOrder.valueOf(
                        sharedPrefs.getString(SORT_ORDER_KEY, SortOrder.NONE.name)
                    )
                ).build()
                ...
            }
        )
}

In this example, we go through the process of building the  and setting its  to what was previously stored in the corresponding  key-value pair, or simply defaulting to NONE.

 will ensure that the is run before any potential data access to DataStore. This means your migration must have succeeded before DataStore emits any further values and before it begins making any new changes to the data. Once you’ve successfully migrated, it’s safe to stop using , as the keys are migrated only once and then removed from .

The  accepts a list of . We will see in later episodes how we can use this for other types of data migrations. If you don’t need to migrate, you can ignore this as it has a default  provided already.

Exception handling

One of the main advantages of DataStore over  is its neat mechanism for catching and handling exceptions. While  throws parsing errors as runtime exceptions, leaving room for unexpected, uncaught crashes, DataStore throws an  when an error occurs with reading/writing data.

We can safely handle this by using the  Flow operator and emitting :

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

val userPreferencesFlow: Flow<UserPreferences> = userPreferencesStore.data
    .catch { exception ->
        if (exception is IOException) {
            Log.e(TAG, "Error reading sort order preferences.", exception)
            emit(UserPreferences.getDefaultInstance())
        } else {
            throw exception
        }
    }

Or with a simple block on writing:

/* Copyright 2022 Google LLC.	
   SPDX-License-Identifier: Apache-2.0 */

suspend fun updateShowCompleted(completed: Boolean) {
    try {
        userPreferencesStore.updateData { currentPreferences ->
            currentPreferences.toBuilder().setShowCompleted(completed).build()
        }
    } catch (e: IOException) {
        // Handle error
    }
}

If a different type of exception is thrown, prefer re-throwing it.

To be continued

We’ve covered Protocol Buffers and DataStore’s Proto implementation — when and how to use it for reading and writing data, how to handle errors and how to migrate from . In the next and final post, we will go a step further and look at how DataStore fits in your app’s architecture, how to inject it with Hilt and of course, how to test it. See you soon!

By Simona Stojanovic
Source Medium

For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!
Our humans need coffee too! Your support is highly appreciated, thank you!

Related Topics
You May Also Like