This seems to rather defeat the basic idea of XML… I also vote "broken by design", with comments.

Putting this another way, in order to edit a cell, I need to know (and check) everything that could possibly index that cell. By the way, this is already a known (and solved) problem in SQL where we might have a partial index on all values less than 3, and a row update operation adds 1 to a value to take it from 2 to 3… the row must automatically drop out of the partial index, although the row operation itself knows nothing about this index (indeed, the same client may be executing the same row operation on several different databases, each with a different partial index definition).

SQL solves the problem by forcing all data operations through an "engine" which has access to the full data and index specification. XML has no concept of index operations, only data structure, storage and retrieval, so there is no generic XML engine with the ability to help with such a problem. The only sensible answer (from the XML way of looking at things) is that all saved data is pure data only (no index allowed, no links allowed) and each application loading the data does its own internal index rebuild. Then at least you have a portable spec (but slow loading).

A broken change tracking list in ODF means it can only be removed.

There are other problem with ODF as well.
For instance that it can use a text to replace the value in a spreadsheet. That causes all kind of serious redundancy problems especially if this text representation is used in cells with derived values.
Cell text replacing the actual cell value also has scary consequenses in possibly hiding real cell content (with could be strings with macro statements) in a spreadsheet from view creating possbile security risk.

What are you smoking? There is no calculation chain in OpenOffice files

Correct. Never said there was.
But OpenOffice files have a feature call change tracking list.
If used it presents much the same problems as a calcchain list.
Any mutation of the spreadsheet also results in a mutation in the change tracking list. Not adding a mutation fucks up your change tracking list making the feature useless.

What are you smoking? There is no calculation chain in OpenOffice files.

Not convinced? Take a look at this :

- I started OpenOffice, created a new spreadsheet, then did basically the same than in my article : I put 10 in a cell, then 20 next to that, and then I create a SUM formula of the two cells. This is saved as BookSample.original.ods
- Then I replace the formula with constant value 40 and save as BookSample.updated.ods

Here is the visual diff between the files :

http://diffopc.arstdesign.com/diffopc_sample_odf_2/diffopc.html

The main part is "content.xml", that's where all worksheet tabs are stored. See for yourself, the formula removal only triggered the removal of the XML formula attribute in that cell.

Call me crazy, but that is a WONDERFUL design. Easy to edit, no hassles.
OpenOffice spreadsheets do it the right way.

If you keep making stuff up, you probably want to create your own Microsoft blog on MSDN. They are in full speed there, posting as much as they can.

But if you don't mind, I'd like to keep the discussion sane and factual.

Have a nice day.

So it requires programs to always track changes when a change tracking list is present in the spreadsheet or mutilate the spreadsheet by removing the change tracking list whihc cannot be rebuild.

Another part which is strange and backwards is the shared strings section. I see no need to introduce that level of indirection for storing simple strings. Sharing strings between cells is an unnecessary indirection which makes it troublesome to make simple changes in a spreadsheet, you need to be prepared to edit the contents of a simple cell in several places. This is an artifact from the early 1990's, to conserve small amounts of memory and disk space at the expense of added complexity. In a binary file it makes no big difference, but for an XML parser it makes all the difference. For example, it makes the document almost impossible to process using a SAX parser. You need a DOM parser, or at least you need to parse all the indexing parts using DOM and keep them around in memory as you parse the rest of the document. Generally speaking, the interdependency between different parts in most OOXML files is mostly too high for the format to allow any "easy editing" at all.

I agree with Stéphane Rodriguez, and I think we will see more reports of problems using OOXML without going through Microsoft's own APIs in the near future.

it's possible to make manual changes very easily, which
is what the Microsoft Office propaganda machine has
been telling us ever since it was announced.

It is very easy to change the a spreadsheet manually.
You chuck the calcchain.xml from the package. How easy is that.

What you do not seem to graps is that you actually created a file that has a calcchain that does no longer conform to the specifcation as it contains cell that no longer need calculation (or even worse) are likely to no longer contain cell that do require calculation.
Those are non OOXML conforming files. And then you go and experiment how MS Office handles those files and blame it on the format specification.

You wrote "Any serieus spreadsheet application would want code to (re)build a calcchain. So this is nog a real extra effort for full spreadsheet application."

Apparently, you still haven't read the article. Please at least read the introduction. The whole point of the article is to fact-check whether it's possible to make manual changes very easily, which is what the Microsoft Office propaganda machine has been telling us ever since it was announced.

Note that these are not random changes. I have read elsewhere some people saying that you should not be able to make random changes. That goes without saying. Those changes are very careful. And a number of flaws in the "spec" get in the way.

Also, you keep saying these are implementation flaws, well you are wrong again. These are "spec" flaws. Again, read the article, I take several references of said "spec" to explain what's wrong with it. Also, it should not be lost on you that the said "spec" was actually derived from the implementation. The irony is that the "spec" has so many typos and is so lacking (attributes are quickly described, not specified at a quality level you should expect for an international standard) that there is no way someone can write an Office competing product with this "spec" alone.
It's not really something new. We all know that, today's Office competitors have been built with much of reverse engineering (the old binary format "spec" fell short on the exact same accounts : many typos, many undocumented parts).

-Stephane Rodriguez

Given the past record of "backwards compatibility" these bugs are there to stay.

Minor implementations would not build a calcchain but either leave the spreadsheet without calcchain (which is fine for 99% of small to medium sized spreadsheets) or in the rare occasions you need a calcchain use a third party API for producing the calcchain as building a calcchain is a fairly standardized action. For instance simply opening and saving the file throug Excel related API's would rebuild the calcchain.

And if you are an expert why not create and sell a developer API for building a calcchain.

Therefore SpreadsheetML is a regression

As I see it the regression is in Excel that does not handle the calcchain inconsistency efficiently. That is the whole point with your entire article.
You put MS Office 2007 implementation as the standard of how Office Open XML works. However MS Office 2007 is an early implementation that is still filled with flaws or less than optimal solutions. For instance adding binary elements in spreadsheets because some Office products (like outlook) when embedding code cannot convert the Office Open XML code in their formats (yet).

So whem looking at MS Office you spot flaw but seems unable to let go of that and link all of them to Office open XML whereas I think it is actually suboptimal implementation solutions by Microsoft.

I welcome your comment but I would prefer you read the article first. Here are the couple points I'm making with the first example :

- making a simple change in a cell is not a "no-brainer" scenario. This simple fact contradicts one and a half year of Microsoft propaganda. If you want to do me a favor, please visit openxmldeveloper.org and see for yourself.
- to have to touch the calculation chain yourself is backwards, clearly a flaw in the spec. It's cells which should be governing the calculation chain, not the opposite. If you have experience in Excel BIFF, it is exactly how it works : the cells govern the calculation chain. Therefore SpreadsheetML is a regression. That's already a second point.
- I'm not entirely sure what you are advocating relating to the calculation chain. Can't you understand that rebuilding the calculation chain requires a lot of code that you have to write? Isn't the guy only willing to make a change in a cell. If the point was to make bigger changes, say replace a pivot table, then sure enough that would require more work, but here we are only talking about changing the value of a cell. I don't think it can get any more trivial than that. And yet, it's not simple.
- as for changes in the calculation chain, if you actually read the article, you would see that's what I do next, and that does not improve the situation at all. In fact it gets worse : "catastrophic failure" error. This actually is another flaw in the spec, it's just poor engineering (you can't have a calculation chain with no cell reference in it). And that's a third point.

Beyond this trivial change, the point is that Microsoft never intended that people out there would make changes. That's I think certainly prevalent in other points as well. You can easily draw the conclusion from that.

If you did not know, I sell some of the most advanced OOXML related products out there (diffopc+, xlsgen). So perhaps it's wiser for you to avoid saying I don't know what I am talking about.

-Stephane Rodriguez

That is however not true. A valid file should conform to the specification.
Actually the calcchain.xml part is explained correctly in Office Open XML. It described that is is a list that describes in what order to handle the formulas in the spreadsheet. What this guy does is removing a formula from the spreadsheet but leaving the reference to it in the calcchain.xml making the calcchain still listing a place that however is no longer a formula to calculate.

So this guy deliberatly breaks the calcchain.xml specification as it now list a cell that contains no formulas to calculate. That means the file is no longer conforming to OOXML and is actually corrupt according to the specification.

Prefebly an implementation coming accross a corrupt calcchain.xml should then just remove the calcchain.xml as it is an optinal part anyways.
So it is relativly easy to handle this corruption.
However MS Office does not but gives an error. Ugly but it is ugly handling of an invalid OOXML file !!!

It is ridicolous that this person cannot understand how to implement the specification and creates corrupt files but then complains about Office Open XML when MS Office cannot handle a corrupt file he created.

"Interestingly enough, we thought parts of a spreadsheet file were individually updatable as long as we did not touch elements that are, according to the ECMA 376 documentation, indexes to other parts."

The point here being that the specification is broken, since it is not possible to edit just a small part, without touching other parts, entangled in yet other parts (calcchain.xml)
Or how else would you go about editing the spreadsheet by a script or whatnot (not Excel) and then open it (as is claimed should work) by another program (of course capable of interpreting the spreadsheet). If we follow your line of thought, either to edit the calcchain.xml (which according to spec isn't needed) or remove it (and then of course the opening program would have to rebuild it)

So your solution, really, is either to only use Excel to modify the spreadsheets (great portability and openness there… or wait…) or to make the modifications and remove the calcchain.xml. Sure, with a script that would come automatic, but why, if the specification works, would it be necessary to modify (in this case remove) another file, if we only edited one local part?

I certainly see his point, don't you?

That is a MS Office 2007 implementation issue and not a Office Open XML issue.
The whole document is more a complaint about MS Office 2007 implementations of Office Open XML then about Office Open XML itself.

Since everything exposed from the COM object model is Variant-centric, I would not be surprised that, when making a VBA call such as Range("C3").Value, the internal IEEE is coerced into a string using some flavor of the Windows OLE API function call VarBstrFromR8. (yep, another Windows dependency)

We can figure out how it works internally. In fact, we can attach a debugger and find out all it does, but what really matters is that, at the end of the day, VBA Office developers, the target audience of those XML formats, are not going to get the same thing than what they are used to. Excel through VBA and the user interface really hides layers of complexity, which is a certainly a good thing. But the XML, all of a sudden, just expose the guts. And suddenly, this becomes a problem if you expect to use your VBA asset (or some other code derived from your VBA asset) to consume/edit this XML. Particularly if you are writing critical calculation functions for instance.

-Stephane Rodriguez

Unless VBA has access to the verbatim type-in value and keeps the value in string representation, these rounding errors are what you will get from VBA also. The fact that OOo Calc actually seems to keep the value which was typed in could be either due to Calc actually keeping the typed-in value verbatim, or that there is some extra, reasonable but nonstandard rounding taking place when saving. I would need to make closer investigations or take a look at the source to know for sure.

Perhaps VBA does such "smart rounding" when reading the values from MS Office? That would be a reasonable explanation for the discrepancy between VBA and the stored value in OOXML.

DrawingML+VML store content with their own locale.

Aha. I actually missed that in my first reading, sorry. Ouch. That's bad formatting
indeed. But there are plenty of other places where the OOXML spec doesn't hold
together as a single standard, so I can't say that I'm surprised.

The whole ECMA 376 spec looks certainly a bit rushed, just like everything
Microsoft does related to international issues.

I can fully agree on that. It's a rush job straight through. Most of it looks as if
it has received only a cursory review by ECMA. It is clearly immature, both in
terms of the technical content and the presentation of the document.

- rounding error : the point I was trying to make is that if you read values from direct access in the XML, you'll get erroneous values. That's not what happens if you are using VBA in Office. And for this reason, it qualifies as a regression. You are right about the rounding error itself though, thanks for noting that.
- US English locale : here again, the problem is that VBA Office developers get values, functions and just about everything else localized, not US English. This qualifies as a regression. Also, I wish the XML was at least uniformly US English, but as I allude to in the article, this is not quite the case either : for instance, DrawingML+VML store content with their own locale. The whole ECMA 376 spec looks certainly a bit rushed, just like everything Microsoft does related to international issues. I'll leave it as an exercise to readers to figure out if their market forces/ubiqituity should allow them to redefine what international standards are on the subject, or if we should stick with the proven international standards.

-Stephane Rodriguez

Having said that, I looked at what OpenOffice.org does in the same situation,
and Calc appears to store the entered values verbatim without making the
round-trip to "double" representation. At least, the XML contains the correct
values, exactly as entered. The actual values are stored in US English notation,
while the displayed values are stored using the current locale and display precision.

I also think that Stephane's criticism of Excel for using a US-centric number format
and not the locale when storing numeric values is uncalled for. Programmers and
XML processors can be expected to work with the programming language notation
for numeric values, in fact using anything else than that, like a locale-specific
setting, would seem unreasonable to me.

I did try to tell the author about this, but I could not find an e-mail address.
If anyone reading this knows how to contact Stephane, please do so.
I think it is important that we keep all OOXML criticism as accurate as possible.

Note that while Stephane might have jumped the gun at these two particular
issues, I think that the other findings are relevant and very revealing.

He does 13 case studies in which he explains how the format failed. Read it. His conclusions is that ECMA-376 was broken by design.

1) Self-exploding spreadsheets

2) Entered versus stored values

3) Optimization artefacts become a feature instead of an embarrasment

4) VML isn't XML

5) Open packaging parts minefield

6) International, but US English first and foremost

7) Many ways to get in trouble

8) Windows dates

9) All roads lead to Office 2007

10) A world of ZIP+OLE files

11) BIFF is gone…not!

12) Document backwards compatibility subject to neutrino radioactivity

13) ECMA 376 documents just do not exist