New repository configuration ponderings

Tue May 30 10:16:05 PDT 2006

On Tue, 2006-05-30 at 18:21 +0300, Panu Matilainen wrote:
> On Tue, 2006-05-30 at 11:37 +0200, Ralf Corsepius wrote:
> > On Tue, 2006-05-30 at 00:13 -0700, Panu Matilainen wrote:
> > > On Tue, 30 May 2006, Ralf Corsepius wrote:
> > > 
> > > > On Sun, 2006-05-28 at 20:47 +0300, Panu Matilainen wrote:
> > > >> On Sun, 2006-05-28 at 19:31 +0200, Ralf Corsepius wrote:
> > > >>> On Sun, 2006-05-28 at 20:17 +0300, Panu Matilainen wrote:
> > > >>>> On Sun, 2006-05-28 at 18:20 +0200, Ralf Corsepius wrote:
> > > >>>>> On Sun, 2006-05-28 at 17:33 +0300, Panu Matilainen wrote:
> > > >>>>>> I've been thinking about improving the repository configuration and
> > > >>>>>> related things for a while now. Just yesterday I realized we could
> > > >>>>>> easily use the existing main configuration system for this, instead of
> > > >>>>>> inventing yet-another-file-format and parser for it.
> > > >>>>>
> > > >>>>> Well, nice proposal, but ...
> > > >>>>>
> > > >>>>> ... wouldn't an xml config file format be more suitable?
> > > >>>>
> > > >>>> I *hate* XML configuration files from user POV.
> > > >>>
> > > >>>
> > > >>> If xml is suitable for something, then it's parsing config files.
> > > >>>
> > > >>> It avoids all the kind of semifunctional implementation nonsense apt's
> > > >>> and yum's config currently carries around, and would dramatically
> > > >>> simplify the syntax and a parser's implementation.
> > > >>
> > > >> Note the "user POV" in my comment above.
> > > > Pardon, complete rubbish, IMO.
> > > >
> > > > I don't see any substantially difference between
> > > > <apt>
> > > >  <cachedir>cache/fedora/</cachedir>
> > > > </apt>
> > > >
> > > > and
> > > >
> > > > Apt {
> > > >  Cache-dir { cache/fedora }
> > > > }
> > > 
> > > Yeah, I know it's not that big a difference, but XML still gives me the 
> > > creeps when encountered in a human editable file.
> > 
> > Why? Your favorite editor doesn't have an xml-syntax-highlighting mode
> > but it has an apt.conf-syntax highlighting mode?
> 
> I don't need syntax highlighting to make sense out of apt.conf. My
> bigger gripe with XML as human written language is the amount of extra
> crap you have to type - I just manually converted my apt.conf to mockup
> XML and absolutely hated it from the fift line or so.
> 
> > 
> > >  And I'm not alone with 
> > > my opinion in that.
> > And apt's configuration gives me creeps all over the place:
> > * Where's the grammar?
> 
> Run 'apt-config dump' - there's your grammar. The foo {blah
> "something";}; can be considered a shorthand for the dump syntax.
> 
> > * Which options do exist?
> > * When are which options used/set/initialized?
> 
> There are a different issue really.
Not quite. 

It depends on how one implements a parser, esp. if you'd implement a
batch parser or an interactive parser. If implementing a batch-parser,
you'd read a file once, transform it into an internal
data-representation once, and then work on this data-representation.
If implementing an interactive parser (aka. interpreter), you'd read in
sequentially read and input file and perform actions while reading.

config-files typically are static and read non-interactive, therefore
batch parsers are an appropriate means to implement them - I.e. their
internal data representation and grammar also can be static.
This is not the case for apt's configuration language.

>  Now, I'm not at all opposed to
> beating more sense into apt configuration system eventually (like
> forcing options to be registered before they can be used etc) but that's
> not solved by just switching to XML.
Right, this is the batch vs. interactive parser aspect.

> > > >>  From programming point of view
> > > >> I fully agree XML would be much nicer configuration file format. It's
> > > >> just that XML isn't "human editable" IMO (programmers don't count as
> > > >> humans here ;)
> > > >
> > > > c.f. above.
> > > >>> But given your attitude, I withdraw my comment above. All your proposal
> > > >>> does is adding yet another source of errors and of incompatibilities.
> > > >>
> > > >> Dunno, apt has a parser which does the job adequately and users are
> > > >> already familiar with the format.
> > > > Pardon again, IMO, a very short-sighted POV.
> > > >
> > > > apt currently has a swamp of key-values pairs, written in a proprietary
> > > > block oriented language, hardly with any grammar.
> > > 
> > > ...just like 99% of the existing open or closed source software out 
> > > there :)
> > Agreed, only very few applications have well designed configuration
> > languages.
> > 
> > >  It's just yet another bad configuration "language" among other 
> > > things, at least I'm not inventing a new one here.
> > > 
> > > That said, there *would* be significant benefits to using XML, I'm not 
> > > claiming anything else. What it would take IMO to switch to XML in apt 
> > > configuration is
> > > 1) convert *all* of apt configuration to XML
> > Agreed. If apt's configuration language grammar was well defined, this
> > would be trivial. As there isn't any formal grammar and as there isn't a
> > finite set of options, this probably isn't "that easy".
> 
> The grammar is quite clear IMO,
OK, then you probably will be able to formulate it in (E)BNF.

>  it's just there are two ways to
> accomplish the same thing which can be used interchangably and in any
> arbitrary order. The internal representation of the configuration data
> is a tree which should be quite well presentable with XML.
Internal representation == parse tree.

> What's not so trivial is the directives in apt.conf - #include and
> #clear. XML is fine for presenting the configuration state but
> "commands" such as those dont really fit in.
If using xml-entity expansion, you get #include automatically.

I don't know what you mean with "#clear". If you mean "restart" parsing
from the beginning, then we are talking about different things:
static configuration file parsing (batch parser) versus interactive
command language (interactive parser).

> > > 2) provide basic tools for manipulating that data without having to touch 
> > > the xml manually
> > IMO, this is not fair.
> > 
> > There isn't any tool to provide this feature for current apt's
> > configuration. So, all you are trying here, is to fight xml by raising
> > additional demands.
> 
> Erm, fair to whom?
YOU rejected xml because of point 2) above. This is unfair, because you
are trying to raise an additional barrier, which current apt also
doesn't fulfill.

Personal preference is completely different issue.

>  Others have expressed their concerns about XML as
> configuration format as well, it's certainly not just my opinion. If
> this was a democracy you would've lost already :) 
Let me shoot back in the same tone: If dictators had a clue, such
discussions would not pop up, because they'd implement reasonable
configuration file formats and reasonable config file grammars.

Wrt. this xml definitely isn't the ultimate solution to all config-file
parsing problems. It is simply a systematic and schematic approach,
providing a semi-consisted, look'n'feel and implementation.

Most people which are ranting against xml only rant at it because they
are used to using M$-ish config files (similar to yum's), or because
they don't want packages to ever change and to stick to "old time
practice forever", no matter how broken it is.

> You want XML configuration? Here's my suggestion:
> 1) convert apt's configuration class to use XML doc representation
> internally
> 2) add methods for reading xml configuration data in 
> 3) add 'dumpxml' command to apt-config to allow automatic conversion
> from old format to xml presentation
> 4) add tools for programmatically manipulating the configuration
> 5) eventually deprecate and finally remove the old format completely
> 
> 1-4 can be done without affecting users at all unless they wish to
> switch to the new format.
We'll see what I can do. My time is very limited these days, so I'll
probably not be able to look into this in near future.

> Oh and btw, all this has little to do with the original proposal of
> making repository configuration use apt configuration engine instead of
> using another cumbersome and very limited file format (sources.list).
Yes, that's another issue. I never understood why sources.list (and
vendors.list) applies a different file format than apt.conf, rsp. why
sources.list aren't treated as #include files to apt.conf.

> The configuration system can be used right now for storing the necessary
> data, if the config system is some day switched to XML the repository
> configuration will automatically follow as long as the configuration API
> is kept compatible.

Ralf