MakeMusic
SmartMusic Finale Garritan MusicXML

Library of Congress data challenge

Moderator: Michael Good

Library of Congress data challenge

Postby Scott Lis » Sun Mar 02, 2014 4:02 pm

Challenge.gov is "A partnership between the public and government to solve important challenges."

https://challenge.gov/

For example, the Library of Congress sponsored Legislative Data Mapping:

<https://challenge.gov/listings?utf8=%E2%9C%93&q=xml&sort=recent&type=&agency=>

Here is a description of the winners:

<http://www.lawtechnologynews.com/id=1393336019340?kw=Library%20of%20Congress%20Announced%20Data%20Challenge%20Winners&et=editorial&bu=Law%20Technology%20News&cn=20140225&src=EMC-Email&pt=Daily%20Alert>

The Library faces many challenges with archiving and holding its store of music scores. Obsolete formats, incomplete representations, inaccurate submissions, etc...

Isn't it time for Congress to initiate a challenge on using XML to preserve works of music? This could include works in the public domain and protected works that will ultimately end up in the public domain. Additionally, all works submitted could be used in research and possibly made available during infringement litigation.

A separate submission system could be used for works in the public domain. This could accept multiple versions of a work with the capability for rating the accuracy of the work by users designated as experts.

It is my understanding that in many cases a piano reduction is used to determine if there is substantial similarity between the disputed works in a copyright infringement case. An XML score that more accurately represents the works would probably be more fair.

Here a link to the current list of acceptable submission formats: <http://www.copyright.gov/eco/help-file-types.html>

Thanks,

-Scott
Scott Lis
 
Posts: 13
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby L. Peter Deutsch » Sun Mar 02, 2014 5:59 pm

Scott Lis wrote:Isn't it time for Congress to initiate a challenge on using XML to preserve works of music?

I'm assuming that by "XML" you mean specifically MusicXML.

MusicXML does not, and is not intended to, completely and accurately represent music in either its visual or its auditory form. It also does not, and is not intended to, capture all of the information that is encoded in any of the proprietary score formats such as Finale and Sibelius. It also does not, and as far as I can tell is not intended to, have a specification of standards-level quality. Indeed, I believe Michael Good has even said on occasion that it is only intended as a transport or interchange format, not a long-term storage format. For these reasons, I would strongly oppose its use as an archiving or preservation format.

L Peter Deutsch
L. Peter Deutsch
 
Posts: 205
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby David Webber » Mon Mar 03, 2014 6:03 am

L. Peter Deutsch wrote:MusicXML does not, and is not intended to, completely and accurately represent music in either its visual or its auditory form. It also does not, and is not intended to, capture all of the information that is encoded in any of the proprietary score formats such as Finale and Sibelius. It also does not, and as far as I can tell is not intended to, have a specification of standards-level quality. Indeed, I believe Michael Good has even said on occasion that it is only intended as a transport or interchange format, not a long-term storage format. For these reasons, I would strongly oppose its use as an archiving or preservation format.

MusicXML is a headache for my software (Mozart - http://www.mozart.co.uk). Specifically since I have been developing an export capability to go with the already existing import module, I've been testing by exporting MusicXML and then reimporting, and trying to get as much as possible back. At this point one comes hard up against the meaning of

"not intended to, capture all of the information that is encoded in any of the proprietary score formats such as..." ... Mozart.

Nevertheless, I would support the use of MusicXML as a means of creating libraries of notation. Think of it as a "transport or interchange" mechanism from what is "out there" to whatever software you're using.

At the moment, as far as any well-defined publicly-available format for notation libraries goes, there is MusicXML (or the less well defined - and now severely necrotic - NIFF). And that's it.

So in opposing the use of MusicXML as a storage format, you are, in essence, opposing storage of generally available music notation files. The answer to that is that you personally would not have to take any notice of their existence. But if such a library existed, those of us who write software (with proprietary notation formats) could point our users at it as a very valuable resource. And MusicXML - largely due to Michael's continued vigorous promotion of it since its inception - makes that possible.

[I've been a big fan of the US Library of Congress - ever since I discovered Alan Lomax's ground breaking sessions with Jelly Roll Morton!]

Dave

David Webber Mozart Music Software
http://www.mozart.co.uk/
David Webber
 
Posts: 148
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby snake » Mon Mar 03, 2014 6:37 am

Hey Guys,

The desirable solution is to fix MusicXML so it is suitable as an archival format for music.

MusicXML+PDF may be the best solution available to us now.

Notation software that only saves in the MusicXML format may accelerate this process.

Is there an app that only reads and writes MusicXML?

Snake...

L. Peter Deutsch wrote:MusicXML does not, and is not intended to, completely and accurately represent music in either its visual or its auditory form. It also does not, and is not intended to, capture all of the information that is encoded in any of the proprietary score formats such as Finale and Sibelius. It also does not, and as far as I can tell is not intended to, have a specification of standards-level quality. Indeed, I believe Michael Good has even said on occasion that it is only intended as a transport or interchange format, not a long-term storage format. For these reasons, I would strongly oppose its use as an archiving or preservation format. MusicXML is a headache for my software (Mozart - http://www.mozart.co.uk). Specifically since I have been developing an export capability to go with the already existing import module, I've been testing by exporting MusicXML and then reimporting, and trying to get as much as possible back. At this point one comes hard up against the meaning of "not intended to, capture all of the information that is encoded in any of the proprietary score formats such as..." ... Mozart. Nevertheless, I would support the use of MusicXML as a means of creating libraries of notation. Think of it as a "transport or interchange" mechanism from what is "out there" to whatever software you're using. At the moment, as far as any well-defined publicly-available format for notation libraries goes, there is MusicXML (or the less well defined - and now severely necrotic - NIFF). And that's it. So in opposing the use of MusicXML as a storage format, you are, in essence, opposing storage of generally available music notation files. The answer to that is that you personally would not have to take any notice of their existence. But if such a library existed, those of us who write software (with proprietary notation formats) could point our users at it as a very valuable resource. And MusicXML - largely due to Michael's continued vigorous promotion of it since its inception - makes that possible. [I've been a big fan of the US Library of Congress - ever since I discovered Alan Lomax's ground breaking sessions with Jelly Roll Morton!] Dave David Webber Mozart Music Software http://www.mozart.co.uk/
snake
 
Posts: 14
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Andrew Hankinson » Mon Mar 03, 2014 6:47 am

David Webber wrote:At the moment, as far as any well-defined publicly-available format for notation libraries goes, there is MusicXML (or the less well defined - and now severely necrotic - NIFF). And that's it.

That’s not quite true. There are tons of other music notation formats out there. If you’re looking for XML-based formats, the Music Encoding Initiative (MEI) is open source, community developed, and publicly available as well. <http://www.music-encoding.org/home;https://code.google.com/p/music-encoding/>

-Andrew
Andrew Hankinson
 
Posts: 5
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby James Sutton » Mon Mar 03, 2014 6:59 am

Hi Snake,

SeeScore for iPad only reads MusicXML. In time it will write MusicXML too, as that is effectively its native format. The main problem hitherto has been the poor quality of MusicXML files, since SeeScore tries to make sense of almost everything as long as it passes musicxml.xsd validation, so software which writes garbage values, produces unusable files for SeeScore, unlike other score rendering programs which tend to ignore the absurd. Things are improving however, and software is improving.

It seems to me that MusicXML (+PDF) or MIDI are the only viable alternatives now and for the forseeable future, and MusicXML is effectively a superset of MIDI, which is now obsolescent. We should embrace MusicXML. Any shortcomings can be fixed.

with best regards James


snake wrote:Hey Guys,

The desirable solution is to fix MusicXML so it is suitable as an archival format for music.

MusicXML+PDF may be the best solution available to us now.

Notation software that only saves in the MusicXML format may accelerate this process.

Is there an app that only reads and writes MusicXML?

Snake...

David Webber wrote:(or the less well defined - and now severely necrotic - NIFF). And that's it. So in opposing the use of MusicXML as a storage format, you are, in essence, opposing storage of generally available music notation files. The answer to that is that you personally would not have to take any notice of their existence. But if such a library existed, those of us who write software (with proprietary notation formats) could point our users at it as a very valuable resource. And MusicXML - largely due to Michael's continued vigorous promotion of it since its inception - makes that possible. [I've been a big fan of the US Library of Congress
- ever since I discovered Alan Lomax's ground breaking sessions with Jelly Roll Morton!] Dave David Webber Mozart Music Software
http://www.mozart.co.uk/
James Sutton
 
Posts: 37
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby James Sutton » Mon Mar 03, 2014 7:14 am

Hi Andrew,

Interesting. I had not come across MEI.

Thanks for the info James

On 3 Mar 2014, at 12:47, Andrew Hankinson wrote:

Andrew Hankinson wrote:
David Webber wrote:At the moment, as far as any well-defined publicly-available format for notation libraries goes, there is MusicXML (or the less well defined - and now severely necrotic - NIFF). And that's it.

That’s not quite true. There are tons of other music notation formats out there. If you’re looking for XML-based formats, the Music Encoding Initiative (MEI) is open source, community developed, and publicly available as well. <http://www.music-encoding.org/home;https://code.google.com/p/music-encoding/>
-Andrew
James Sutton
 
Posts: 37
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby Scott Lis » Mon Mar 03, 2014 8:20 am

Good points. The Congressional challenge is just that -- a challenge. It is a way to publicize, promote, and possibly solve an important issue. How do we preserve and archive sheet music in a way that best captures the most information? Simply taking a picture of a score is not enough. Capturing a specific performance may not even reflect what the composer intended.

Of course, Congress, your Senators and Representatives, can not promote a specific company, proprietary standard -- or represent a foreign interest. Those of us in the U.S. can make a request to members of Congress that a challenge be issued.

The considerable engineering work that Michael has done in putting MusicXML together can be leveraged. While it may not be the standard for archiving, the knowledge gained could help.

Maybe there are other alternatives. JSON is lightweight and extensible. However, it looks like many of the representational problems have been solved with XML, and there are many tools already available.

What are we losing if we wait another 10-20 years to start on this issue?

-Scott

James Sutton wrote:Hi Andrew,

Interesting. I had not come across MEI.

Thanks for the info James

On 3 Mar 2014, at 12:47, Andrew Hankinson >
Andrew Hankinson wrote:That’s not quite true. There are tons of other music notation formats out there. If you’re looking for XML-based formats, the Music Encoding Initiative (MEI) is open source, community developed, and publicly available as well. <http://www.music-encoding.org/home;https://code.google.com/p/music-encoding/>
-Andrew

Post your message to the list by sending it to .

To contact the list owner, send your message to
.

your email address, click here. <http://cgi.mail-list.com/u?ln=musicxml&nm=scott%40scottlis.to>
Scott Lis
 
Posts: 13
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby David Webber » Mon Mar 03, 2014 9:27 am

David Webber wrote:At the moment, as far as any well-defined publicly-available format for notation libraries goes, there is MusicXML (or the less well defined - and now severely necrotic - NIFF). And that's it.

Andrew Hankinson wrote:That’s not quite true. There are tons of other music notation formats out there.

Lots??? There's MusicXL, NIFF (which is essentially dead), abc (which is completely inappropriate) and...

> If you’re looking for XML-based formats,

[not necessarily]

Andrew Hankinson wrote:the Music Encoding Initiative (MEI) is open source, community developed, and publicly available as well. <http://www.music-encoding.org/home;https://code.google.com/p
/music-encoding/>

....Thanks. I was completely unaware of that one - no-one has asked me for it. But I'm interested and I've bookmarked the site. But a quick look gives me little idea of the format spec. There seem to be lots of
.rng files. I haven't met these (forgive me if this is dreadful ignorance) and so I checked the extension and found 5 possibilities

-Random Number Generator Data File (Psytek Ltd.)
-RELAX NG (OASIS RELAX NG TC)
-Unknown Apple II File (found on Golden Orchard Apple II CD Rom)
-Top Secret Crypto Gold RSA Key Ring File (TAN$TAAFL Software Company)
-Nokia Phone Ringtone File (Nokia)

none of which looked promising - but a haphazard visit to the MEI2012 page seems to indicate that RELAX might have something to do with it. So I looked further and found "RELAX NG is a schema language for XML. RELAX NG is comparable with XML. Where W3C develops the XML specs, OASIS develops the RELAX NG specs."

So it looks like a proprietary version of XML. Why do we need that? (I googled OASIS and found a rock group, a fashion chain, and a dating site. I googled "OASIS RELAX" and found various restaurants, hotels, and health spas, and well down the list <https://www.oasis-open.org/> which has lots of technical details).

The MEI site looks very neat but I'm not having much luck getting into the details - quickly at least. (But finally found a 'guidelines' PDF which to my mind, when I look at it, approximates a 'specification'.)

Anyway - back to the plot:

In what way does MEI improve on MusicXML? Is there a list of software which imports/exports it? Is there a documented .dtd file? (I believe that they're rather old fashioned these days, but I find MusicXML's commented .dtd files an extremely useful reference to the spec.)

Dave

David Webber Mozart Music Software
http://www.mozart.co.uk/
David Webber
 
Posts: 148
Joined: March, 2014
Reputation: 0

Re: Library of Congress data challenge

Postby L. Peter Deutsch » Mon Mar 03, 2014 10:33 am

Scott Lis wrote:Maybe there are other alternatives. JSON is lightweight and extensible. However, it looks like many of the representational problems have been solved with XML, and there are many tools already available.

The real issue is not the choice of syntax, JSON vs. XML or whatever: it is semantics -- what does the data *mean* or *represent*?

In my opinion, MusicXML as it exists has at least four deep problems, one of which cannot be fixed and three of which I see no prospect of fixing. I have brought all of them up on this forum, and I believe Michael Good and I have agreed to disagree about whether they are serious, fixable, or even problems at all, but I would like to state them again for the record, since the issue at hand has such long-lived consequences.

The unfixable problem is the collision between MusicXML's desire to represent music semantically and the desire to represent printed scores accurately. Probably the most serious of these is its mixing of two different methods of specifying X,Y coordinates on the page: relative to the "default location" (i.e., whatever the engraving algorithms of the implementation choose), and absolute on the page. MusicXML files that mix these -- which are common, and include the files produced by MusicXML's favorite implementation in Finale -- inherently simply cannot be rendered even approximately correctly by any software other than their original producer, unless that software has reverse engineered the producer's engraving algorithms. For example, I believe I have seen real vocal score files produced by Finale in which the notes were positioned using the
"default location" method but the lyrics or even some music symbols on the staff were positioned with absolute coordinates.

I believe the root of this problem is the failure to consider the graphic design of the page in terms of flow regions and fixed regions at an architectural level. While I am sure many score file representations have the same problem as MusicXML in this regard, I believe fixing it is not impossible in an archive format. Be this as it may, this problem is not fixable in MusicXML -- the current capabilities cannot be removed from MusicXML, MakeMusic has no incentive to rationalize the capabilities of Finale, and any acceptance of MusicXML as an archiving format must clearly accept current MusicXML output from Finale. (In the absence of a specification, I believe Finale's algorithms have become the de facto definition of what MusicXML files denote, which is a perversion of what it means to have an implementation-neutral standard.)

The three other problems are fixable in theory, but given what has actually happened in the 2+ years since MakeMusic acquired Recordare (namely, nothing visible), I see no prospect of this happening.

The first is that there is no specification of what is and is not a valid MusicXML file -- that is, an unambiguous mechanical test that can be applied to a claimed MusicXML file to produce a yes-or-no answer. Conformance to the DTD is necessary but not sufficient. The classic example of the problem is unterminated constructs like beams and slurs
("start" element with no "stop" element), which are syntactically valid at the XML level, but whose meaning and legality are not defined at the MusicXML level. I believe there are multiple problems like this in MusicXML.

The second is that there is no adequate specification of what even syntactically valid MusicXML files denote in terms of printed output. The worst offender is the "backup" element, but I believe there are others. Michael has said many times that "music is too complex, and there are too many interacting features and conditions" for this to be possible. Michael and I are both experienced musicians with many years' experience in software technology, but unlike Michael, I also have deep professional experience in reading and writing standards, and with all due respect to Michael's extraordinary work, I simply disagree with him about this. IMO, to the extent that "too many interacting features and conditions" are the problem, it is a problem of design, not a problem of the domain.

Finally, Michael has (in my opinion) actively encouraged the propagation of bad implementations of the current inadequate (but unquestionably valuable and useful) specification. The principle of "selective encoding" is used to condone MusicXML producers that do not output information that is available and representable. More seriously, and (in my opinion) without justification, I have documented Finale's implementation as disregarding information in MusicXML input that is clear, unambiguous, and clearly within Finale's capabilities, but this has not been considered a bug or significant deficiency. While this is not a problem with MusicXML per se, it is one that works against its solidity in practice.

The Library of Congress currently accepts a fair number of poorly defined, proprietary formats for archiving, including Microsoft Word. While this is lamentable, it is not catastrophic, because those formats have a de facto definition -- the single piece of proprietary software that implements them, which itself can be archived. MusicXML has neither a solid definition (like PDF and MIDI) nor a single controlled implementation (like Word): there is simply no way to ensure that an archived MusicXML file will be rendered accurately now, let alone accurately or even identically at some future time, which makes a mockery of the basic mission of archiving. MusicXML files cannot even serve well for analysis in the absence of a complete definition of their semantics.

We all know the saying "The best is the enemy of the good." It seems likely that that principle will carry the day here, since MusicXML is so clearly the most widely supported existing openly documented score representation format and since it is good enough for so many purposes. I don't mind being on the losing side of the discussion, especially since I probably won't be on the planet 20 or 30 years from now when Finale will no longer be around and no one will be able to reconstruct how those archived files are supposed to look. I'm just not very happy with the outcome.

L Peter Deutsch
L. Peter Deutsch
 
Posts: 205
Joined: March, 2014
Reputation: 0

Next

Who is online

Users browsing this forum: No registered users and 2 guests

cron