repomd metadata as sqlite database

[Panu Matilainen]

Been a bit quiet here for the last months, but I'm not dead yet :)

Seth & others added an option to latest createrepo (released without 
announcements AFAICT) to generate sqlite databases of the repository 
metadata on the server in addition to the xml files. This directly 
benefits yum, as it doesn't need to locally generate the sqlite caches on 
each client. Reportedly this gives something like 28% performance gain for 
yum.

I've done a bit of initial experiments on using those sqlite db's for apt 
indexes and its more than clear that it is a big win in pretty much every 
way compared to using the original repodata xml files:
- Memory use goes down so much you wouldn't believe it
- Random-access things suddenly become not only sanely possible but also
   very fast

I'm planning to add this as a new repository type instead of extending the 
repomd type to use the database indexes when available dynamically:
- Existing more-or-less stable repomd code can be left alone.
- It allows us to get rid of the libxml2 dependency. Yes, there's still
   repomd.xml that needs to be parsed somehow. Whether we use expat or
   libxml2 or whatever to parse it, we can in any case get rid of the
   xmlReader requirement. Means we can get repomd(db) support for any old
   distro where sqlite3 is available, IIRC that includes even the ancient
   RHEL AS 2.1.

There is a "risk" that since the db format is originally yum internal 
details, it's subject to change. OTOH the db includes a schema version 
number, so it's at least possible to detect an unsupported version unlike 
with the xml repodata. So even in that sense this looks like a win-win 
situation over the xml-repodata.

I'm hoping to get a rough initial implementation into svn this weekend, or 
"real soon" anyway. Just posting this here as "pre-announcement" in case 
you folks have comments / futher ideas etc.

 	- Panu -