Welcome to Textile ThreadDB private alpha-testing program. We’re excited to be sharing an early look at a new approach to an offline-first local database that syncs to the distributed web. In this document, we will provide detailed onboarding instructions, along with some background and motivation for this early preview release.
Getting help
The Textile/Threads developers/community are active on Slack and Twitter(@textileio), join us there for news, discussions, questions, and status updates. For this private alpha-testing program, we have created an invite-only channel within our public Slack workspace, so that participants can discuss things with one another, and send questions to the Textile team directly. During testing, if you think you've found a bug in ThreadDB, please file a GitHub issue in the main repo.
Background
See this 5-minute project overview video to get a feel for the underlying project structure, and how we’re conceptualizing the new ThreadDB in Javascript.
Let’s start with some of the motivations around our new approach to ThreadDB, and our shift from“Threads the database” to“Threads the database syncing mechanism”.
Local first. The concept of offline-first or local-first software isn’t a new one. There are some great discussions(and summaries) of the ideas out there already, and we encourage you to take a look. User ownership of data, interoperability, collaboration, offline-capable, etc are all ideals we appreciate.
Usability. This one doesn’t get discussed as much in the context of offline-first software, but it is actually way easier to test and develop apps that don’t actually require real-time access to any remote services. As you’ll see in some of our demo videos below, you can spend a good chunk of time testing, validating, and breaking things before you even touch a“live” remote API. This means less garbage data while developing, while at the same time leveraging and testing against the real APIs you’ll encounter once you move to production.
Flexibility. Why should a developer looking to leverage Threads in their workflow have to migrate to a new database implementation, when there are already lots of great databases out there? Wouldn’t it be great if you could just use the right database for the job, and then simply have that database sync to web3 automatically? We think so, and this is part of what we are exploring with this release of ThreadDB.
When thinking about the above constraints(plus a whole lot of technical considerations), we landed on a simple, but powerful framework for our new vision of ThreadDB for Javascript. This early release addresses the local-first database component by leveraging IndexedDB, via Dexie. This provides a lightweight, easy to use database, where all the complexity of remote sync is separated from the complexity of the database itself. The database features a MongoDB-like experience, with change tracking, custom UUIDs, and more. For those already familiar with Dexie, you can peel back the layers to have direct access to the Dexie database object for even more control over your local database experience. Check out the getting started section to get a feel for the basic APIs provided by Textile. There, you’ll see what is required to go from basic interactions with a local ThreadDB Database to pushing those local changes to a Remote.
Modular
ThreadDB for Javascript contains a fully-featured local-first database(Dexie) and a set of remote features that enable sync, archive, and recovery over the network. The Remote object itself simply defines several middlewares that intercept Dexie calls, store the list of updates in a local sandbox, and then provides tooling to push, pull, rebase, and stash these local changes with a remote Thread daemon.
The project design is purposefully modular, and is organized roughly into three components: the“local” store, the“middleware”, and the communication with a“remote”. Ideally, the“local” component is minimal, and provides only a light wrapper around a local database. In our case, that database is IndexedDB via Dexie, but in theory, almost any local database will eventually work. The middleware is similarly light-weight. Here, we provides a few sub-modules for hooking into the local database to extract the things we need to communicate with the remote module. It additionally provides a few middleware components that provide MongoDB style queries, track changes, custom UUIDs, schemas, and more. Those familiar with our existing Threads Client library will notice several useful similarities.
That brings us to the“remote” module. This is where the magic of syncing to the decentralized web happens. In practice, we’re leveraging existing Textile APIs to validate this proof-of-concept model. Eventually, as we optimize what is going on under the hood, we’ll roll out new APIs that are more efficient, and provide more useful functionality in an offline-first scenario. Essentially, we track changes on the local database, and provide low level APIs to push, pull, and resolve any conflicts that arise when syncing with a remote. In general, the remote is considered the“source of truth” in any real conflicts, and ThreadDB is designed to be used more as a federated system of peers, rather than pure peer-to-peer. As such, each local“remote” module connect to one remote daemon at a time, and relies on that daemon for network operations and syncing. By default, remotes are configured to work against Textile’s Hub daemon, but as a developer, you have easy access to setting this to other remote hosts.
Getting started
Check out this“getting started” video to get a feel for the basic APIs. In it, you’ll see what is required to go from basic interactions with a local ThreadDB Database to pushing those local changes to a Remote Daemon.
Before digging into the specifics of ThreadDB, you’ll likely want to familiarize yourself with existing Textile developer tooling. In particular, if you are planning to use ThreadDB against Textile’s Hub, you’ll want to make sure you have a developer account setup, and likely want to create some new keys for testing etc. For the purposes of testing and building toy examples, we recommend, you stick with development mode.
Additionally, for this early release, we’ve tried to keep the Database APIs relatively similar to our existing Thread Client library. That means ThreadDB supports Collections and Instances, very similarly to what we already have going on over here. We highly recommend you take a look at those existing docs before proceeding.
Assuming you have a developer account setup, and you’ve read the docs(seriously,read the docs), let’s get started. The following examples are taken directly from the above getting started video. So feel free to follow along, copy/paste the code snippets, and start testing things. The demo videos and most of our examples are in Typescript, but we provide a vanilla Javascript React demo below as well. Additionally, you can use any environment you are comfortable with, including browser apps via a bundler such as webpack or rollup, or directly in Node/Deno. For the brave, you can access the existing Jupyter Notebookdirectly.
Start by creating a new npm project. Just stick with the defaults here, we’re not going to need anything fancy:
mkdir demo-project
cd demo-project
npm init
# Defaults are fine
Now you can install the required Textile modules. For this first mini-demo, we’ll stick with the minimal setup(we’re including threads-client here for validation later):
npm i @textile/threaddb @textile/crypto @textile/threads-id @textile/threads-client
Usage
With those in place, we’re ready to start interacting with the Database. If you’re running this code in NodeJS, it will automatically include an IndexedDB polyfill to store data in SQLite, otherwise(in a browser) it should leverage the embedded IndexedDB database.
import { Database } from"@textile/threaddb"
import { schema, Person } from"./schema"; // Available with the notebook
Getting help
Background
Getting started
mkdir demo-projectcd demo-projectnpm init# Defaults are finenpm i @textile/threaddb @textile/crypto @textile/threads-id @textile/threads-clientimport { Database } from "@textile/threaddb"import { schema, Person } from "./schema"; // Available with the notebook