MakeMusic
SmartMusic Finale Garritan MusicXML

Library of Congress data challenge

Moderator: Michael Good

Re: Library of Congress data challenge

Postby James Sutton » Mon Mar 03, 2014 11:04 am

Good points all.

I am coming to the conclusion that the default-x/y and relative-x/y are not terribly useful for these reasons. Arguably if all elements specify these and if the score is laid out observing them all, then it may work, but in general one wants rather more flexibility in the layout than pdf, and so it's probably better to do as most (all?) do and discard these values. I suspect anything more sophisticated like the architecture outlined here would be very complicated, which would be a significant barrier to adoption.

A specification of the valid sequence of anything that can specify start/continue/stop would be really useful including a simple publicly available test program.

I agree that the 'single piece of proprietory software' is not a good solution. An archived copy of Word or whatever is not guaranteed to work on any future available hardware

James


L. Peter Deutsch wrote:
Scott Lis wrote:Maybe there are other alternatives. JSON is lightweight and extensible. However, it looks like many of the representational problems have been solved with XML, and there are many tools already available.

The real issue is not the choice of syntax, JSON vs. XML or whatever: it is semantics -- what does the data *mean* or *represent*?

In my opinion, MusicXML as it exists has at least four deep problems, one of which cannot be fixed and three of which I see no prospect of fixing. I have brought all of them up on this forum, and I believe Michael Good and I have agreed to disagree about whether they are serious, fixable, or even problems at all, but I would like to state them again for the record, since the issue at hand has such long-lived consequences.

The unfixable problem is the collision between MusicXML's desire to represent music semantically and the desire to represent printed scores accurately. Probably the most serious of these is its mixing of two different methods of specifying X,Y coordinates on the page: relative to the "default location" (i.e., whatever the engraving algorithms of the implementation choose), and absolute on the page. MusicXML files that mix these -- which are common, and include the files produced by MusicXML's favorite implementation in Finale -- inherently simply cannot be rendered even approximately correctly by any software other than their original producer, unless that software has reverse engineered the producer's engraving algorithms. For example, I believe I have seen real vocal score files produced by Finale in which the notes were positioned using the "default location" method but the lyrics or even some music symbols on the staff were positioned with absolute coordinates.

I believe the root of this problem is the failure to consider the graphic design of the page in terms of flow regions and fixed regions at an architectural level. While I am sure many score file representations have the same problem as MusicXML in this regard, I believe fixing it is not impossible in an archive format. Be this as it may, this problem is not fixable in MusicXML -- the current capabilities cannot be removed from MusicXML, MakeMusic has no incentive to rationalize the capabilities of Finale, and any acceptance of MusicXML as an archiving format must clearly accept current MusicXML output from Finale. (In the absence of a specification, I believe Finale's algorithms have become the de facto definition of what MusicXML files denote, which is a perversion of what it means to have an implementation-neutral standard.)

The three other problems are fixable in theory, but given what has actually happened in the 2+ years since MakeMusic acquired Recordare
(namely, nothing visible), I see no prospect of this happening.

The first is that there is no specification of what is and is not a valid MusicXML file -- that is, an unambiguous mechanical test that can be applied to a claimed MusicXML file to produce a yes-or-no answer. Conformance to the DTD is necessary but not sufficient. The classic example of the problem is unterminated constructs like beams and slurs
("start" element with no "stop" element), which are syntactically valid at the XML level, but whose meaning and legality are not defined at the MusicXML level. I believe there are multiple problems like this in MusicXML.

The second is that there is no adequate specification of what even syntactically valid MusicXML files denote in terms of printed output. The worst offender is the "backup" element, but I believe there are others. Michael has said many times that "music is too complex, and there are too many interacting features and conditions" for this to be possible. Michael and I are both experienced musicians with many years' experience in software technology, but unlike Michael, I also have deep professional experience in reading and writing standards, and with all due respect to Michael's extraordinary work, I simply disagree with him about this. IMO, to the extent that "too many interacting features and conditions" are the problem, it is a problem of design, not a problem of the domain. Finally, Michael has (in my opinion) actively encouraged the propagation of bad implementations of the current inadequate (but unquestionably valuable and useful) specification. The principle of "selective encoding" is used to condone MusicXML producers that do not output information that is available and representable. More seriously, and (in my opinion) without justification, I have documented Finale's implementation as disregarding information in MusicXML input that is clear, unambiguous, and clearly within Finale's capabilities, but this has not been considered a bug or significant deficiency. While this is not a problem with MusicXML per se, it is one that works against its solidity in practice. The Library of Congress currently accepts a fair number of poorly defined, proprietary formats for archiving, including Microsoft Word. While this is lamentable, it is not catastrophic, because those formats have a de facto definition -- the single piece of proprietary software that implements them, which itself can be archived. MusicXML has neither a solid definition (like PDF and MIDI) nor a single controlled implementation (like Word): there is simply no way to ensure that an archived MusicXML file will be rendered accurately now, let alone accurately or even identically at some future time, which makes a mockery of the basic mission of archiving. MusicXML files cannot even serve well for analysis in the absence of a complete definition of their semantics.

We all know the saying "The best is the enemy of the good." It seems likely that that principle will carry the day here, since MusicXML is so clearly the most widely supported existing openly documented score representation format and since it is good enough for so many purposes. I don't mind being on the losing side of the discussion, especially since I probably won't be on the planet 20 or 30 years from now when Finale will no longer be around and no one will be able to reconstruct how those archived files are supposed to look. I'm just not very happy with the outcome. L Peter Deutsch
James Sutton
 
Posts: 37
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Marc Sabatella » Mon Mar 03, 2014 11:12 am

L. Peter Deutsch wrote:
Scott Lis wrote:Maybe there are other alternatives. JSON is lightweight and extensible. However, it looks like many of the representational problems have been solved with XML, and there are many tools already available. The real issue is not the choice of syntax, JSON vs. XML or whatever: it is semantics -- what does the data *mean* or *represent*?

In my opinion, MusicXML as it exists has at least four deep problems, one of which cannot be fixed and three of which I see no prospect of fixing.

The words "problem" and "fix" imply some specific purpose, though. The
"problems" you mention in no whatsoever diminish the usefulness of the format for archival as an adjunct to PDF. It successfully provides a great deal more semantic information than PDF, in a format that is far more portable, far more open, and far more future-friendly than any format ever devised or likely to be devised. Given that there are *many* people who would welcome such a thing, just because you personally would not use it because you fear you won't be able to extract every last pixel-level detail of information from it in 30 years does not strike me as reason to "oppose" its use for the benefit of the very many who see enormous value in it. It's essentially zero cost, except maybe for the effort it would take to improve MusicXML support in the software used to create these scores, which is effort that should be spent anyhow.

L. Peter Deutsch wrote:We all know the saying "The best is the enemy of the good." It seems likely that that principle will carry the day here, since MusicXML is so clearly the most widely supported existing openly documented score representation format and since it is good enough for so many purposes.

Indeed, including this one. Sure, there is room for improvement. And I see no reason to assume it won't steadily improve in the future just as surely as it has thus far. So maybe we don't need to be in a huge hurry actually creating this archive while we settle on a few things. But I personally don't need to see MusicXML resulting in a pixel-perfect reproduction of a score. Again, that's what PDF is for in my book.

Marc
Marc Sabatella
 
Posts: 24
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Simon Giddings » Mon Mar 03, 2014 11:38 am

What I find interesting here, is that there would seem to be an argument over pixel based output as opposed to structural output.

MusicXML, because it uses XML, is structure based. This means that the responsibility is upon the application which outputs the information to be as accurate as the existing format permits. It is then upto the importing application to correctly interpret the data, ignoring anything which it does not understand - or does not care about.

As with nearly every standard - it gets better as it is used and people identify those points which need improvement. The fact that it does not answer everybody's needs today - or perhaps better put - every flavour of written music is not a barrier, but rather a motivation to continue collaborating with it's development. The added benefit of MusicXML is the ability to be backwards compatible. That which was created one year ago should still be "readable" by competent applications.

For my part, I am really pleased that this standard exists. It makes my life a whole lot easier as a developer not having to handle the headace of proprietary files which could (and often do) change format from one version to another.

Not everyone will be pleased or like this - but that's normal. For the rest of us, we look forward to greater and tighter implementation in standard software like MuseScore, Sibelius and Finale, so that our own software development can advance better.

Simon

Marc Sabatella wrote:
L. Peter Deutsch wrote:In my opinion, MusicXML as it exists has at least four deep problems, one of which cannot be fixed and three of which I see no prospect of fixing.

The words "problem" and "fix" imply some specific purpose, though. The
"problems" you mention in no whatsoever diminish the usefulness of the format for archival as an adjunct to PDF. It successfully provides a great deal more semantic information than PDF, in a format that is far more portable, far more open, and far more future-friendly than any format ever devised or likely to be devised. Given that there are
*many* people who would welcome such a thing, just because you personally would not use it because you fear you won't be able to extract every last pixel-level detail of information from it in 30 years does not strike me as reason to "oppose" its use for the benefit of the very many who see enormous value in it. It's essentially zero cost, except maybe for the effort it would take to improve MusicXML support in the software used to create these scores, which is effort that should be spent anyhow.

L. Peter Deutsch wrote:We all know the saying "The best is the enemy of the good." It seems likely that that principle will carry the day here, since MusicXML is so clearly the most widely supported existing openly documented score representation format and since it is good enough for so many purposes.

Indeed, including this one. Sure, there is room for improvement. And I see no reason to assume it won't steadily improve in the future just as surely as it has thus far. So maybe we don't need to be in a huge hurry actually creating this archive while we settle on a few things. But I personally don't need to see MusicXML resulting in a pixel-perfect reproduction of a score. Again, that's what PDF is for in my book.

Marc
Simon Giddings
 
Posts: 4
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby David Webber » Mon Mar 03, 2014 6:35 pm

L. Peter Deutsch wrote:In my opinion, MusicXML as it exists has at least four deep problems ... The unfixable problem is the collision between MusicXML's desire to represent music semantically and the desire to represent printed scores accurately. Probably the most serious of these is its mixing of two different methods of specifying X,Y coordinates on the page: relative to the "default location" (i.e., whatever the engraving algorithms of the implementation choose), and absolute on the page....

Actually I agree with you (i think) on this: I find the x-y position information difficult. But in fact my software uses it a lot less than it might, and sample files apparently omit an awful lot of positional information which they might have included. So I accept MusicXML as a good sematic representation, with occasional bits of helpful positional information (eg is this symbol above or below the staff). As far as mixing semantics and page geometry goes: if you think a desire to include both is too optimistic (and you may be right) then you have to accept MusicXML as a format primarily for semantics. The other possibility would be geometry only, and that can already be achieved with a bitmap image.

L. Peter Deutsch wrote:MusicXML files that mix these -- which are common, and include the files produced by MusicXML's favorite implementation in Finale -- inherently simply cannot be rendered even approximately correctly by any software other than their original producer, unless that software has reverse engineered the producer's engraving algorithms...

It *is* difficult :-) It certainly wasn't designed with Mozart in mind. :-(

L. Peter Deutsch wrote:I believe the root of this problem is the failure to consider the graphic design of the page in terms of flow regions and fixed regions at an architectural level....

Sounds complicated. However if we accept that a MusicXML file plus an image of the page might be a good starting point, then arbitrary notation software can import the music (semantically) and the user can edit the geometry to look like the original if he wishes. This is a valid compromise I think, and not a lot more than what you' do if you obtained a Word document formatted for a non-standard paper size and edited it to fit your printer.

L. Peter Deutsch wrote:... I believe Finale's algorithms have become the de facto definition of what MusicXML files denote, which is a perversion of what it means to have an implementation-neutral standard.)

I *hope* you're wrong.

L. Peter Deutsch wrote:...there is no specification of what is and is not a valid MusicXML file -- that is, an unambiguous mechanical test that can be applied to a claimed MusicXML file to produce a yes-or-no answer. Conformance to the DTD is necessary but not sufficient.

Yes. Unfortunately music is so irregular that this would be very difficult, so I'm not too surprised. Software B has to do its best to cope with what has been output from Software A. Worrying about
'validity' is just part of that.

L. Peter Deutsch wrote:The second is that there is no adequate specification of what even syntactically valid MusicXML files denote in terms of printed output. The worst offender is the "backup" element, but I believe there are others.

Actually I hate the back-up element as it makes life very difficult for Mozart which has no equivalent. (My quick glance at MEI revealed, I think, a <layer> element which would have ben a nicer alternative.) But we are where we are, and a backup element is clear I think - as long as the parallel notes all have 'voice' parameters attached.

L. Peter Deutsch wrote:Michael has said many times that "music is too complex, and there are too many interacting features and conditions" for this to be possible. Michael and I are both experienced musicians with many years' experience in software technology, but unlike Michael, I also have deep professional experience in reading and writing standards, and with all due respect to Michael's extraordinary work, I simply disagree with him about this. IMO, to the extent that "too many interacting features and conditions" are the problem, it is a problem of design, not a problem of the domain.

Suppose you were to define a spec which could label a MusicXML file valid or invalid. What would you do with it (except reassure those of us struggling to import MusicXML files that our problems were in some cases not our fault)?

L. Peter Deutsch wrote:Finally, Michael has (in my opinion) actively encouraged the propagation of bad implementations of the current inadequate (but unquestionably valuable and useful) specification. The principle of "selective encoding" is used to condone MusicXML producers that do not output information that is available and representable. More seriously, and (in my opinion) without justification, I have documented Finale's implementation as disregarding information in MusicXML input that is clear, unambiguous, and clearly within Finale's capabilities, but this has not been considered a bug or significant deficiency. While this is not a problem with MusicXML per se, it is one that works against its solidity in practice.

Sounds like Finale regards MusicXML as a semantic format too. Actually I regard it as a strength of MusicXML that some information can be omitted or ignored and it can still generate a useful import.

L. Peter Deutsch wrote:The Library of Congress currently accepts a fair number of poorly defined, proprietary formats for archiving, including Microsoft Word. While this is lamentable, it is not catastrophic, because those formats have a de facto definition -- the single piece of proprietary software that implements them, which itself can be archived.

I find that highly unsatisfactory. In the year 3000 we'll need an archived copy of Word 2007 and a machine to run it on?

L. Peter Deutsch wrote:MusicXML has neither a solid definition (like PDF and MIDI)

The spec looks solid enough to me - solid enough for me to write software which imports/exports it.

L. Peter Deutsch wrote:nor a single controlled implementation (like Word): there is simply no way to ensure that an archived MusicXML file will be rendered accurately now, let alone accurately or even identically at some future time, which makes a mockery of the basic mission of archiving. MusicXML files cannot even serve well for analysis in the absence of a complete definition of their semantics.

Depends what you mean by 'accurately'. If you want an exact copy, then archive it as a bitmap. No I'd prefer to have archived MusicXML files which my users could access, and import as a semantic representation of the music. An exact geometric copy is only desirable if you want only to print it the way it was, and a bitmap is perfect for that. If you want to start from the music and arrange it for other groups or whatever, you only really want the semantics.

L. Peter Deutsch wrote:We all know the saying "The best is the enemy of the good." It seems likely that that principle will carry the day here,

As it did with betamax.

L. Peter Deutsch wrote:since MusicXML is so clearly the most widely supported existing openly documented score representation format and since it is good enough for so many purposes.

Exactly.

L. Peter Deutsch wrote:I don't mind being on the losing side of the discussion, especially since I probably won't be on the planet 20 or 30 years from now when Finale will no longer be around and no one will be able to reconstruct how those archived files are supposed to look. I'm just not very happy with the outcome.

The point about a MusicXML representation is that it doesn't matter whether Finale will be around in 20 or 30 years, as long as MusicXML is popular enough that software which *is* around can read it.

[Anyway I wouldn't bet on 20 or 30 years: I started developing Mozart in the mid 80's as a hobby (and, to be honest, some of its current awkwardness to program stem from the limitations of the machines I started to design it for). I didn't think I'd still be developing it nearly 30 years later, but here I am. I, personally, probably won't be doing it in another 30 years but things are unpredictable. As I haven't managed to put Finale out of business yet <vbg> it's just possible they may keep going. But I'd be very upset if the Library of Congress stuff was in a format only their software could read, just as I wouldn't expect it to be in a format which only mine could read. If one is pragmatic, MusicXML looks like a good option.]

Dave

David Webber Mozart Music Software
http://www.mozart.co.uk/
David Webber
 
Posts: 148
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Andrew Hankinson » Mon Mar 03, 2014 8:52 pm

Hi David et al.

First off, a disclaimer: I’m on the team that develops MEI, so I’ve got some irons in the fire there, so to speak.

Second off, I think this is a great conversation to be having. The longevity and archive-ability of notation encoding formats is a pretty good conversation to be having. If I may put in a plug, there will be the second Music Encoding conference 20-23 May 2014 in Charlottesville, VA where you can meet any number of people interested in this exact topic.

I thought I might address a few of David’s earlier questions about MEI to me here, since I think it may contribute to the overall discussion. Hopefully I won’t take up too much bandwidth!

Eleanor Selfridge-Field wrote *the* book on music encoding formats,
"Beyond MIDI", current to about 1997 or so. There are many, many encoding systems discussed in there, some of which are still thriving in certain communities. The **kern format is pretty big for computational musicology and theory. There are also "silos" of encoded music in everything from DARMS to MuseData. Since "Beyond MIDI", several other formats have emerged, including MusicXML. IEEE1599 is one; the ill-fated NIFF another. Every time I turn around someone is inventing a new format! So that’s what I meant by "lots." Some are one-off languages for a specific corpus or software system (openly defined and re-implementable, nevertheless) while some have been trying to gain traction in the long run. MEI was started by Perry Roland at about the same time as MusicXML, but took a different route, emphasizing developing a complete representation of music notation over developing a system for notation interchange between software systems.

RelaxNG is the "New Generation" of schema languages. It’s meant as a replacement for DTDs and W3Schema, and offers a few more tools for specifying exactly how the elements and attributes in an XML spec are supposed to behave. For music, what that means (in my opinion) is that it lets you dig yourself into fewer corners! It’s not a proprietary schema language (it’s maintained as part 2 of ISO/IEC 19757) and works with all the big XML tools (oxygen and xmllint, to name a few). The MEI schema itself is also open source and maintained by a community of developers, with advice from academics, librarians, and practicing musicians. We also maintain the MEI Encoding Guidelines, the document that you found. It's a 700+-page document that details both best-practices as well as the technical spec for the encoding scheme.

I have noticed some questions in this thread about maintaining the graphical appearance or the structure of a notation file — a lot of times they seem to be at opposite ends of the spectrum when it comes to choosing one or the other. However, we’re currently working on developing systems, using MEI, that maintains both graphical and structural as two separate streams. It’s a bit too technical to get into here, but in some ways you might consider it to be the same as the separation of content and presentation in HTML. That’s a pretty big simplification, but if you want more details I would be happy to provide them.

If you’re interested in a bit more easy information about MEI, including how it compares to MusicXML, I was recently interviewed by Philip Rothman of the Sibelius Blog. I have been developing an open-source Sibelius->MEI plugin, which is our first foray into commercial software support. The interview is here: <http://www.sibeliusblog.com/news/mei-plug-in-released-for-sibelius/>

Otherwise, I would love to chat with anyone and everyone about MEI
(personally off-list, so we don’t take up too much bandwidth here, or on the MEI-L mailing list, http://www.music-encoding.org/support/mailingList).

Cheers,
-Andrew


Somebody wrote:
L. Peter Deutsch wrote:In my opinion, MusicXML as it exists has at least four deep problems ... The unfixable problem is the collision between MusicXML's desire to represent music semantically and the desire to represent printed scores accurately. Probably the most serious of these is its mixing of two different methods of specifying X,Y coordinates on the page: relative to the "default location" (i.e., whatever the engraving algorithms of the implementation choose), and absolute on the page....

Actually I agree with you (i think) on this: I find the x-y position information difficult. But in fact my software uses it a lot less than it might, and sample files apparently omit an awful lot of positional information which they might have included. So I accept MusicXML as a good sematic representation, with occasional bits of helpful positional information (eg is this symbol above or below the staff). As far as mixing semantics and page geometry goes: if you think a desire to include both is too optimistic (and you may be right) then you have to accept MusicXML as a format primarily for semantics. The other possibility would be geometry only, and that can already be achieved with a bitmap image.
L. Peter Deutsch wrote:MusicXML files that mix these -- which are common, and include the files produced by MusicXML's favorite implementation in Finale -- inherently simply cannot be rendered even approximately correctly by any software other than their original producer, unless that software has reverse engineered the producer's engraving algorithms...

It *is* difficult :-) It certainly wasn't designed with Mozart in mind. :-(
L. Peter Deutsch wrote:I believe the root of this problem is the failure to consider the graphic design of the page in terms of flow regions and fixed regions at an architectural level....

Sounds complicated. However if we accept that a MusicXML file plus an image of the page might be a good starting point, then arbitrary notation software can import the music (semantically) and the user can edit the geometry to look like the original if he wishes. This is a valid compromise I think, and not a lot more than what you' do if you obtained a Word document formatted for a non-standard paper size and edited it to fit your printer.
L. Peter Deutsch wrote:... I believe Finale's algorithms have become the de facto definition of what MusicXML files denote, which is a perversion of what it means to have an implementation-neutral standard.)

I *hope* you're wrong.

L. Peter Deutsch wrote:...there is no specification of what is and is not a valid MusicXML file -- that is, an unambiguous mechanical test that can be applied to a claimed MusicXML file to produce a yes-or-no answer. Conformance to the DTD is necessary but not sufficient.

Yes. Unfortunately music is so irregular that this would be very difficult, so I'm not too surprised. Software B has to do its best to cope with what has been output from Software A. Worrying about
'validity' is just part of that.
L. Peter Deutsch wrote:The second is that there is no adequate specification of what even syntactically valid MusicXML files denote in terms of printed output. The worst offender is the "backup" element, but I believe there are others. Actually I hate the back-up element as it makes life very difficult for Mozart which has no equivalent. (My quick glance at MEI revealed, I think, a <layer> element which would have ben a nicer alternative.) But we are where we are, and a backup element is clear I think - as long as the parallel notes all have 'voice' parameters attached. Michael has said many times that "music is too complex, and there are too many interacting features and conditions" for this to be possible. Michael and I are both experienced musicians with many years' experience in software technology, but unlike Michael, I also have deep professional experience in reading and writing standards, and with all due respect to Michael's extraordinary work, I simply disagree with him about this. IMO, to the extent that "too many interacting features and conditions" are the problem, it is a problem of design, not a problem of the domain.

Suppose you were to define a spec which could label a MusicXML file valid or invalid. What would you do with it (except reassure those of us struggling to import MusicXML files that our problems were in some cases not our fault)?
L. Peter Deutsch wrote:Finally, Michael has (in my opinion) actively encouraged the propagation of bad implementations of the current inadequate (but unquestionably valuable and useful) specification. The principle of "selective encoding" is used to condone MusicXML producers that do not output information that is available and representable. More seriously, and (in my opinion) without justification, I have documented Finale's implementation as disregarding information in MusicXML input that is clear, unambiguous, and clearly within Finale's capabilities, but this has not been considered a bug or significant deficiency. While this is not a problem with MusicXML per se, it is one that works against its solidity in practice. Sounds like Finale regards MusicXML as a semantic format too. Actually I regard it as a strength of MusicXML that some information can be omitted or ignored and it can still generate a useful import. The Library of Congress currently accepts a fair number of poorly defined, proprietary formats for archiving, including Microsoft Word. While this is lamentable, it is not catastrophic, because those formats have a de facto definition -- the single piece of proprietary software that implements them, which itself can be archived.

I find that highly unsatisfactory. In the year 3000 we'll need an archived copy of Word 2007 and a machine to run it on?
> MusicXML has neither a solid definition (like PDF and MIDI)

The spec looks solid enough to me - solid enough for me to write software which imports/exports it.
L. Peter Deutsch wrote:nor a single controlled implementation (like Word): there is simply no way to ensure that an archived MusicXML file will be rendered accurately now, let alone accurately or even identically at some future time, which makes a mockery of the basic mission of archiving. MusicXML files cannot even serve well for analysis in the absence of a complete definition of their semantics.

Depends what you mean by 'accurately'. If you want an exact copy, then archive it as a bitmap. No I'd prefer to have archived MusicXML files which my users could access, and import as a semantic representation of the music. An exact geometric copy is only desirable if you want only to print it the way it was, and a bitmap is perfect for that. If you want to start from the music and arrange it for other groups or whatever, you only really want the semantics.
L. Peter Deutsch wrote:We all know the saying "The best is the enemy of the good." It seems likely that that principle will carry the day here,

As it did with betamax.

L. Peter Deutsch wrote:since MusicXML is so clearly the most widely supported existing openly documented score representation format and since it is good enough for so many purposes.

Exactly.

L. Peter Deutsch wrote:I don't mind being on the losing side of the discussion, especially since I probably won't be on the planet 20 or 30 years from now when Finale will no longer be around and no one will be able to reconstruct how those archived files are supposed to look. I'm just not very happy with the outcome. The point about a MusicXML representation is that it doesn't matter whether Finale will be around in 20 or 30 years, as long as MusicXML is popular enough that software which *is* around can read it. [Anyway I wouldn't bet on 20 or 30 years: I started developing Mozart in the mid 80's as a hobby (and, to be honest, some of its current awkwardness to program stem from the limitations of the machines I started to design it for). I didn't think I'd still be developing it nearly 30 years later, but here I am. I, personally, probably won't be doing it in another 30 years but things are unpredictable. As I haven't managed to put Finale out of business yet <vbg> it's just possible they may keep going. But I'd be very upset if the Library of Congress stuff was in a format only their software could read, just as I wouldn't expect it to be in a format which only mine could read. If one is pragmatic, MusicXML looks like a good option.] Dave David Webber Mozart Music Software

http://www.mozart.co.uk/
Andrew Hankinson
 
Posts: 5
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Michael Good » Tue Mar 04, 2014 1:23 am

There is lots of good discussion here! I can't reply to every point raised but here are some highlights.

First of all, the Library of Congress is already aware of MusicXML. MusicXML is included in its digital preservation site for sustainability of digital formats:

<http://www.digitalpreservation.gov:8081/formats/fdd/fdd000358.shtml>

I hope that librarians and archivists, rather than politicians, will be driving what is considered to be an archival format. MusicXML has always been intended to be an archival, long-term storage format for music notation semantics, best used in conjunction with PDF files for visual representation and audio formats for sound. It is increasingly serving that function both at publishers and web sites, and we hope to keep making improvements to make things better.

Web sites continue to grow which offer music either directly in MusicXML format or in formats that are easily converted to MusicXML. You can see a list at http://www.musicxml.com/music-in-musicxml. If there are site that we're missing - or software that we're missing on the corresponding software page - please let me know.

While there are indeed many symbolic music formats, MusicXML is the only open format besides MIDI that is in widespread use. MEI is a specialized format designed exclusively for scholarly use. I believe that much if not all of what MEI users are looking for can be done in MusicXML. I think it would be better for musicologists to use the same format as other musicians, and fortunately many musicologists agree, but the MEI folks have taken another path.

As Andrew mentions, RELAX NG is another XML schema format. The compact format in particular tends to be more readable than either DTDs or W3C schemas. RELAX NG's main drawback has been lack of widespread software support. That's changed somewhat, especially recently, so it might be worth taking another look at whether RELAX NG should be a schema language for defining MusicXML 4.0.

James mentioned MIDI. While I agree that MusicXML has superseded it for notation, MIDI still has a central role to play in digital music performance. Standard MIDI Files still contain a level of performance information that MusicXML does not yet match, more so than differences between PDF and MusicXML for graphical detail. One exception is in instrument and sound definitions, where MusicXML provides a more comprehensive, higher-level instrument taxonomy than MIDI currently offers.

Peter makes an important point that MusicXML does not capture all of what is in notation formats like Finale and Sibelius. Specifically, when it comes to graphical details, MusicXML represents a final result. Programs like Finale and Sibelius tend not to represent this information statically within the file, but instead compute it dynamically. One could say that when it comes to graphical detail, notation programs often represent intent rather than final results.

Much of the difficulty of MusicXML exchange, especially the import of graphical details, is how to go from the final result in MusicXML to the implementation technique or intent representation specific to your application. The relative-x and relative-y attributes that Peter disparages can be seen as a rare attempt at trying to capture intent within MusicXML in an application-neutral way. That poses problems too. I think that general problem is inherently insoluble, given the diversity of notation applications. Fortunately, it need not be solved to achieve high accuracy. For despite these difficulties, there is plenty of software - not just Finale - that succeeds in doing things with MusicXML that Peter says cannot be done.

Since MakeMusic has acquired Recordare, the Dolet plug-ins have continued to improve, both for Finale and Sibelius. There's a new unified specification document online that is the first step towards a more comprehensive specification. We have held our first MusicXML community meetings at the NAMM and Musikmesse trade shows. Later this week we should be launching our new MusicXML forum to replace this mailing list. Most importantly, software support has continued to grow with over 170 applications supporting MusicXML - and the quality of the implementations has grown as well. It is this thriving software ecosystem that is MusicXML's best insurance for a role as a long-term archival format for music notation semantics.

The SMuFL initiative has really provided the first major expansion of capabilities that seems to warrant a new MusicXML 4.0 release. Daniel Spreadbury will be attending the MusicXML community meeting at Musikmesse to introduce SMuFL and contribute to a discussion of what "MusicXML support for SMuFL" might mean. I look forward to the discussion that we will have on this and other important topics. You can see the agenda for the meeting at:

<http://www.musicxml.com/musicxml-musikmesse-sxsw-interactive/>

Best regards,

Michael Good VP of R&D MakeMusic, Inc.
Michael Good
VP of MusicXML Technologies
MakeMusic, Inc.
User avatar
Michael Good
 
Posts: 2197
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Carsten Bönsel » Tue Mar 04, 2014 2:49 am

When working on Semantic Web issues for a couple months, I learned that separating the specification of syntax and semantics indeed is possible. RDF data can be represented with a bunch of different markups, regardless of the domain or ontology that is concerned. JSON-LD, Turtle, NTriples, RDF/XML are all of them official W3C standards that can be alternatively supported – in coexistence ( e.g. here: http://schema.rdfs.org/ ).

Seeing startups more and more investigating on web music apps where it seems to be necessary or at least easier using json representations
(https://github.com/saebekassebil/musicjson) I wonder if it might be a good idea considering this in future versions of MusicXML.

Best regards, Carsten Bönsel

Scott Lis wrote:Maybe there are other alternatives. JSON is lightweight and extensible.

--- Diese E-Mail ist frei von Viren und Malware, denn der avast! Antivirus Schutz ist aktiv. http://www.avast.com
Carsten Bönsel
 
Posts: 6
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Carsten Bönsel » Tue Mar 04, 2014 2:59 am

Wikifonia is offline for some weeks already. So to my opinion, it could be removed from your list.

Michael Good wrote:Web sites continue to grow which offer music either directly in MusicXML format or in formats that are easily converted to MusicXML. You can see a list athttp://www.musicxml.com/music-in-musicxml. If there are site that we're missing - or software that we're missing on the corresponding software page - please let me know.

--- Diese E-Mail ist frei von Viren und Malware, denn der avast! Antivirus Schutz ist aktiv. http://www.avast.com
Carsten Bönsel
 
Posts: 6
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Mogens Lundholm » Wed Mar 05, 2014 12:13 pm

Hello all

A very interresting and important discussion. My point of view is that music-xml should be the native format of music-writing programs and players and readers. I.e. you should be able to open and save directly - no import and export (MuseScore is terrible). If some thing is missing in Music Xml, then add it Like Snake wrote: "The desirable solution is to fix MusicXML so it is suitable as an archival format for music".

Davids comments are very intreresting - I also had my fun with the <backup>-command. And do not like the <tie>-command. But now these problems are solved, so this is past.

Midi could work - but unrealistic to collect people to define approvements. Just look at the attempt described in the book "Beyond Midi" to define a command for Clef.

I like the idea of a Music XML format checker: There must be a title, there must be names on the voices/instruments, verse lyrics must be numbered 1,2,3,4,5 etc, <backup> must be within song. Also there should be number of test-files that a program must pass.

The next statements are obvious - but nevertheless not according the any music program I know

1. Any program must ignore commands not known, 2. If the program can change a Music XML file, it must preserve
commands not known to the program 3. If the program has changed the Music XML data, then there shall
be a "Save"-command, that saves the file to same position, from
which the file was read 4. If the program has changed the Music-XML-data and the program is
terminated then a message window shall appear asking:
"The file has changed - do you want to save the file" 5. If the XML-Data has not been changed and the program is closed then there may not come any messages asking you to save the file 6. The message box "save as" may only appear if it is impossible the save
the file at the original position or the menu "save as" has explicitly been selected.

I see no alternative to Music XML.

Kind regards Mogens
Mogens Lundholm
 
Posts: 60
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Marc Sabatella » Wed Mar 05, 2014 12:27 pm

Mogens Lundholm wrote:A very interresting and important discussion. My point of view is that music-xml should be the native format of music-writing programs and players and readers. I.e. you should be able to open and save directly - no import and export

Nice dream, but this seems completely impractical for existing software. Neither Finale nor Sibelius nor MuseScore nor any other program that has been around long enough to have developed its own proprietary format is ever likely to do the almost complete rewrite that would be necessary to support this idea. Each program has its own data structures and the mapping to and from MusicXML is not likely to be straightforward (I
*know* it isn't for MuseScore; I can only assume this is true for the others). A program would have to be designed from the ground up to use data structures that map well to MusicXML in order for this to work. Not that it wouldn't be a bad idea to design such a program, and I'd like to see more MusicXML readers in particular (for different devices, to get better playback facilities, etc). But any new notation program designed around MusicXML would be at a huge competitive disadvantage compared to programs that have been around a while and that have control over their own formats.

Marc
Marc Sabatella
 
Posts: 24
Joined: March, 2014
Reputation: 0

PreviousNext

Who is online

Users browsing this forum: No registered users and 1 guest

cron