I just spotted the youtube video of my OpenSQL Camp (Portland 2009) talk on An Open Storage Engine API. I talked about some of technical issues for implementing storage engines across many SQL front ends, not just MySQL.
You can find this talk and other mostly technical material at http://www.tokutek.com/technology/.
Resources
RELATED POSTS
This is *very* interesting.
my ambitions would also be to get us a new storage engine api, logical, consistent, and easy to use from the ground up (although I haven’t looked at BDB API for quite a while)
how do you see the future of your project ?
Btw, the slides aren’t yet at http://tokutek.com/technology/ – can you put them online please ?
—
BDB XA – it’s not their api, it’s the standard api. See http://www.opengroup.org/pubs/catalog/c193.htm
I put the slides on the site. Sorry for the oversight.
For the future: I’d like to get this project done.
My usage is that the BDB XA API is a particular C binding for an implementation of the XA specification. So the API was developed by sleepycat, and XA itself is specified by The Open Group. I could be convinced that some other usage is better.
Hello,
I see the big boost in insert/search performance due to the fast inserts (fast merges) but how are you fighting against terribly slow deletes from the one huge index chunk? i believe it is not possible without much of the unused space left after the deletions followed up by some background job on compacting the indexes..
basically its a trade between inserts/searches and deletes
good luck with your startup
regards, mitja
Although this topic isn’t really directly relevant to my Open Storage Engine API talk, it’s still an interesting topic.
Fractal trees improve both insertions and deletions. See, for example, this post.
I don’t know what data structure you have in mind, but fractal trees do not employ a single huge index chunk (whatever that might mean), and they recover space after deletions.
In my experience there is no fundamental tradeoff between insertion performance and deletion performance.
If you have a problematic workload, possibly with a mix of insertions, deletions, and searches, I’d love to discuss whether fractal trees can help.
Mitja,
I am not sure why you think there is a tradeoff for deletions. TokuDB’s deletions are very fast. An benchmark run can be found at the bottom of the first page in here http://tokutek.com/2009/10/performance-brief/.
There is no tradeoff.
Can you please elaborate on what “deletes from the one huge index chunk” means?