sqlite performance

Definitely do follow the advice on stackoverflow about wrapping batch queries (eg a single query with 1000 insert statements) in their own transaction with

begin transaction
..
end transaction

I got a 2-orders of magnitude speed increase just from trying that, it was the only optimisation I needed.