Overview: What is OTA?

● A system for applying large updates to field

databases that
– May be more efficient than writing the db using
other methods, and
– Allows an update to be suspended and resumed.
● An acronym for “Over The Air”
 Overview: How is OTA used
● The user creates an “OTA database” containing
the changes to apply to the “target database”
● Then somehow sends it to the device
● Where an application uses the OTA module API
to copy changes from the OTA to the target
database.
 Overview: OTA Deployment

System
Applications OTA Application

target.db
ota.db
state.db

Read-only access Read/write access Can be the same db

 Overview: OTA Database Example
Target Database Schema:

CREATE TABLE t1(a TEXT PRIMARY KEY, b INTEGER, c INTEGER);

Update in SQL:

INSERT INTO t1 VALUES('one', 1, 4);

UPDATE t1 SET b=5 WHERE a = 'two';
DELETE FROM t1 WHERE a = 'three';

OTA Database Schema

CREATE TABLE data_t1(a, b, c, ota_control);

OTA Database Contents:

a b c ota_control
'one' 1 4 0 If the target table
'three' NULL NULL 1 actually has a column
'two' 5 NULL '.x.' named “ota_control”,
you're out of luck
 Overview: Application Code
int apply_ota(){
sqlite3ota *pOta;
int rc;

/* Open an OTA handle */

pOta = sqlite3ota_open("target.db", "ota.db", "state.db");

/* Call sqlite3ota_step() on the OTA handle until it returns

** something other than SQLITE_OK. Either SQLITE_DONE to
** indicate that the update has been fully applied, or else
** an SQLite error code. */
do {
rc = sqlite3ota_step(pOta);
}while( rc==SQLITE_OK );

/* Clean up the OTA handle. sqlite3ota_close() always returns

** the same value as the last call to sqlite3ota_step(). */
rc = sqlite3ota_close(pOta);
assert( rc!=SQLITE_OK );
return rc==SQLITE_DONE ? SQLITE_OK : rc;
}
 What does this do again?
● Why are we using OTA instead of regular SQL?
– Avoids the “write amplification”/“cache
thrashing” problem.
– Regular SQL transactions are not resumable.

● If neither of these seem applicable...

 Write Amplification
● When writing X bytes of data to the database
results in (N * X) bytes being written to disk. For
inconveniently large values of N.
● In the worst cases, (N * X) bytes might be many
times larger than the target database (!!).
● The worst cases occur when the working set is
larger than the cache.
● For b-trees, the solution is to sort the keys before
they are inserted.
 Write Amplification: Illustration

B-Tree:

SQLite Cache:

ZipVFS

Operating System
 Write Amplification
● For a non-trivial schema:
CREATE TABLE t1(a PRIMARY KEY, b, c UNIQUE) WITHOUT ROWID;
CREATE INDEX t1b ON t1(b);

Each SQL operation writes up to 3 b-trees.

● So it is not possible to simply sort the SQL
statements:
INSERT INTO t1 VALUES(0, 'A', 'iii');
INSERT INTO t1 VALUES(1, 'C', 'i');
INSERT INTO t1 VALUES(2, 'B', 'ii');
...
 Write Amplification
● OTA makes all required updates to each b-tree
before moving on to the next.
● By making multiple passes of the data_xxx
table:
a b c ota_control
0 'A' 'iii' 0
1 'C' 'i' 0
2 'B' 'ii' 0

SELECT … FROM data_t1; -- For the PK b-tree

SELECT … FROM data_t1 ORDER BY c; -- For the UNIQUE b-tree
SELECT … FROM data_t1 ORDER BY b; -- For index t1b
 How UPDATE/DELETE ops work
● UPDATE and DELETE operations use the state
database.
● An OTA update containing a DELETE:
DELETE FROM t1 WHERE a=4;

a b c ota_control
4 NULL NULL 1

● The first pass can delete the entry from the PK

b-tree, but how about the other two?
● Values required to delete index entries are
saved in the state db as part of the first pass.
 Write Amplification Example
● POI Database Schema:
CREATE TABLE poiServiceLocationTable(
serviceLocationId IPK, poiId, catId, mortonCode, iconSetId
);
-- ...and 4 indexes…\

● 985,000 entries on 14356 pages of 8192 bytes

each. Zipvfs. File size of 56MB.
● 69500 inserts, 39352 deletes, 7075 updates.
● Directly: 28582 page writes, 40000 reads, 38
seconds.
● With OTA: 14343 page reads/writes, 9 seconds.
...So that was “write amplification”. Next is
“resumable transactions”...
 Resumable Transaction Support
● Means this scenario is supported:
– Applications reading from a database in state A.
– OTA Application begins applying an update (to
move the db to state B).
– The system is rebooted.
– Applications resume, still reading the database in
state A
– The OTA Application also picks up where it left off –
midway through applying the update.
 Application Code II
int apply_ota(int *pbFinished){
sqlite3ota *pOta;
int rc;

/* Open an OTA handle */

pOta = sqlite3ota_open("target.db", "ota.db", "state.db");

/* Call sqlite3ota_step() until either the entire update is

** applied or the application needs to shutdown. */
do {
rc = sqlite3ota_step(pOta);
if( <app needs to shutdown> ) break;
}while( rc==SQLITE_OK );

/* Clean up the OTA handle. sqlite3ota_close() always returns

** the same value as the last call to sqlite3ota_step(). */
rc = sqlite3ota_close(pOta);
*pbFinished = (rc==SQLITE_DONE);
return rc==SQLITE_DONE ? SQLITE_OK : rc;
}
 How SQLite Wal Mode Works
● How “wal mode” works:
– Writers append data to the *-wal file
– Readers read from both the db file and the *-wal file
– Eventually data is copied from the *-wal file into the
db file. This is called a checkpoint.

target.db

Database Applications

target.db-wal
How Resumable Transactions Work
● Target database must be in “rollback mode”
● OTA writes into the *-oal file, which is like a *-
wal file but nobody else knows it's there
● Once the entire update is in the *-oal file, OTA
moves it to *-wal
● Then runs a checkpoint (incrementally – which
regular clients cannot do).
How Resumable Transactions Work
Database Database
Applications Applications

target.db target.db
target.db-oal target.db-wal

OTA Application OTA Application

...And that was “resumable transactions”. Now
more about the OTA database schema...
 Details: PRIMARY KEY-less Tables
● Add an “ota_rowid” column to data_xxx table:
Target Database Schema:
CREATE TABLE t1(a TEXT, b INTEGER, c INTEGER);

Update in SQL:
INSERT INTO t1 VALUES('x', 'y', 'z');
INSERT INTO t1(rowid, a, b, c) VALUES(13, 'a', 'b', 'c');
DELETE FROM t1 WHERE rowid = 5;

OTA Database Schema

CREATE TABLE data_t1(ota_rowid, a, b, c, ota_control);

OTA Database Contents:

ota_rowid a b c ota_control
5 NULL NULL NULL 1
13 'a' 'b' 'c' 0
NULL 'x' 'y' 'z' 0
 Details: Virtual Tables
● Virtual tables must have a “rowid” column that
works like a PRIMARY KEY (most do)
● Add the “ota_rowid” column to the data_xxx
table
● Any hidden columns (i.e. languageid columns)
are optional
Target Database Schema:
CREATE VIRTUAL TABLE x1 USING fts4(txt, languageid=lid);
OTA Database Schema:
CREATE TABLE data_x1(ota_rowid, txt, lid, ota_control);
 Details: Limitations
● INSERT, UPDATE and DELETE operations only,
● INSERT may not use default values,
● UPDATE and DELETE must identify their target rows by
primary key,
● UPDATE statements may not modify primary key columns,
● No triggers are fired,
● No foreign key constraint handling is performed,
● CHECK constraints are not enforced
● Constraint handling (for UNIQUE and NOT NULL constraints) is
always “OR ROLLBACK”.
 And Finally...
● It does large updates to databases:
– With b-tree writes in sorted order to avoid “write
amplfication”, and
– With support for “resumable transactions”

● Sort the contents of the data_xxx tables by PK.

● Canonical documentation is comments in

“sqlite3ota.h” - part of devkit.