RealClimate

Comments

RSS feed for comments on this post.

  1. Good point about the independent results thing being more important.
    A lot of these people wanting code to check what groups of scientists have done, should be writing their own code and producing results to check against the results produced by the code they want to see.

    Of course one could assume that they aren’t interested in the results or producing any, which might explain why they don’t produce their own models and want to see the code.

    If they can do better, all will benefit. But will they publish it as an open source model?

    Comment by The Ville — 26 Oct 2010 @ 6:54 AM

  2. The whole business of code archival is actually a form of publication that used to be referred to as a methods paper. Ironically, even as the internet supposedly opened up vast new amounts of information space, it has been getting harder and harder to actually publish methods in detail. The general reaction in the last decade that methods are boring and should be minimized in papers. Why can’t we be a bit more creative with publication formats so that detailed methods can be included?

    [Response: I totally agree. The issue is all about the publication formats. There is a really interesting article on some of the pitfalls of publishing 'everything' in the Journal of Neuroscience, which we linked to above (here)--eric]

    .

    Comment by Mitch Lyle — 26 Oct 2010 @ 7:14 AM

  3. Thank you for the links and this excellent discussion of the science behind source code release. In terms of science I think we are in broad agreement: it is difficult, certainly in the current publishing and promotion regime, to use science as a justification for releasing your code. Science will be better the more code you release, but it’s not an easy argument to make.

    However, the real reason we started Climate Code Foundation is public perception. Science is being used to justify public policy, and the public’s perception of that justification (and hence, the policy) is clouded by not having access to source code. Public policy will be advanced by releasing source code.

    [Response: David, thanks for commenting. I think we should be clear that the public perception is being clouded by what pundits are saying about the importance of access to source code, not the actual alleged lack of it's availability. But don't get me wrong -- your work is important and will indeed serve science.-eric]

    Comment by David Jones — 26 Oct 2010 @ 7:40 AM

  4. As a software engineer writing math-ish code in an unrelated field, I can say from the outset that a clear description of the mathematics is far, far, far more important than the implementation itself.

    Consider a case where you’re in a code base trying to work out what’s done. Even something like reconstructing the symbolic version of a large-ish Jacobian from some badly written, cryptic code is the stuff of nightmares. I’d rather get the equations and re-write the code.

    Only having one or the other (code or description) to me is a false dichotomy. Time and time again, I come across papers that a slightly incomplete descriptions of an algorithm (for a trivial example, something has to be implemented with numerical integration… what method did they use? ). Having the code makes these sort of assumptions clearer and sometimes fills in the gaps missing in a paper.

    [Response: Very well stated points. Not to mention typos in the mathematics, that can lead to endless frustration until you figure them out! I recall a glaciology paper by a mentor of mine with a missing ^2 that took me forever to find. Of course, if I had had the code, I would have just used that, and never learned as much as I had to when working it out myself, and then I never would have known the equations were wrong, and perhaps would have propagated the error in future publications. None of these solutions are without pitfalls!--eric]

    Comment by Damien — 26 Oct 2010 @ 7:55 AM

  5. The point here is that in some fields, the code is the method. Not making code available feels a lot like when the cold fusion folks chose to not reveal the details of their methods. If several labs had independently claimed to have made cold fusion work, would that have meant that revealing the methods was unnecessary or that their claims were more believable?

    For the claims that are based on code to be tested, the code itself has to be checked. Something as simple as a minor typo can lead to a problem that creates a false result even if the basic analysis is correct. Checking through back-casting can’t always find these problems because future conditions put in the model are not the same as past conditions used for checking.

    I’m sure that IP problems are real, but for code-based scientific papers to be independently verified, the code itself has to be open to review. Maybe some kind of limited review could be implemented until the IP problems are resolved. Until something is worked out, we’re just repeating the cold fusion debacle, with skeptics trying to guess at what the code does and talking about how it is the wrong way to do things.

    Comment by vboring — 26 Oct 2010 @ 9:30 AM

  6. From the previous comments (and my own experience as well), it seems the reason having code open along with the publications is to clarify the assumptions and the methods that lead to the results. While I could replicate the rusults of a paper with a cryptic description of the method and the results, it would take a lot longer for me to do so (and I’d assume everyone would have this experience). You do raise some important objections to having everything open source immediately (especially for projects that weren’t started open source). While on the whole I think open source is the way to go, as you say, there are pitfalls.

    Thanks for a very good and informative discussion.

    Comment by sambo — 26 Oct 2010 @ 9:34 AM

  7. Thanks for saving me from having to write this. Seriously, I’ve had had a few hundred words lying around for a few weeks now, waiting for a wee bit of polish. But all it said, in essence, you’ve said, although I would have spent a bit more time on the difference between replication and repetition (probably without adding much signal).

    So, just asking for “code publication” is far from enough, like “data publication”, communities are going to have to work out “what is the publication unit”, and “how does one get credit for publishing (code)”?
    (Here I’m not talking about “putting code snippets on the internet”, that can happen or not, and wont make much difference to much, I’m talking about the publication of significant pieces of code, such asthe piece that the climate code foundation started with.)

    Comment by Bryan Lawrence — 26 Oct 2010 @ 9:35 AM

  8. This isn’t rocket science folks! Think about Linux and and their willingness to work together to create something really special! If climate science is as important as we say it is, there should be no hesitation to stand on each others work and build something that our children can be proud of rather than doubting us because of secrecy and the need for personal aggrandizement!

    [Response: Huh? Did you read what we wrote, or are you just responding to what you imagine we wrote?--eric]

    Comment by adrian smits — 26 Oct 2010 @ 9:40 AM

  9. The starting point in any computer project is always a very, very clear statement of the real world goals. Without that, people always seem to wander off in some direction or other, usually getting distracted by very intense technical details, feverishly solving complex or laborious problems, only to find out that no one really needed 90% of what was done and vast amounts of effort were spent on ultimately irrelevant details.

    And stating the real world goals is not always as obvious as it seems.

    So… what is the actual real world goal of this?

    There is a vague, implied purpose of improving the science… but that’s not specific enough. How? It’s already been stated that part of the scientific process is to reproduce individual efforts separately, without tainting the reproduction by using the exact same original (and potentially flawed) methods. This is a sensible approach. Sharing code will defeat this. It will be too easy for people to say “I don’t the have time (or a grad student to spare), so I’ll just re-use his code.”

    I will say that it seems to me as if the primary goal is to make the software available so that self styled amateur auditors can comb through it looking for mistakes. If this is in fact the case, then publishing the software will not improve the science. It could, in theory, help to identify errors that might otherwise go unnoticed, but experience to date has shown that it will instead be used by the unprincipled to mount an attack on the science by trying to discredit valid work by exaggerating the impact of minor problems.

    I’m not saying that makes it wrong to share the code, or that it’s a reason to “hide” the code. I’m just saying go in with your eyes open, and a clear, positive goal in mind.

    Alternately, a specific scenario like the sharing of model components (such as is described for AR5) is a more narrow and specific focus, and is liable to have very, very different requirements and implementation techniques than a problem defined with a broader scope. It’s a very different problem with a different solution.

    The goal is everything.

    I am not saying I’m against this at all. I’m a big fan of open source with everything. But I do think everyone needs to go in with their eyes wide open, and with very, very clear real world goals established and stated at the outset. That one step has a tremendous impact on what you actually do, how you do it, and how you address the pitfalls you encounter along the way to a solution.

    If you don’t know where you are going, you might wind up someplace else! — Yogi Berra

    Comment by Bob (Sphaerica) — 26 Oct 2010 @ 9:41 AM

  10. I read the linked announcement (J of Neuroscience re Supplemental Material). It seemed to me that the main critisism was that too often the supplemental material tempted researchers to put key components of the methods into the supplemental material when it should be in the article. Is this what everyone else understood?

    It would seem that ensuring the methods are clearer in the article would be a good thing instead of burying it in the code.

    Comment by sambo — 26 Oct 2010 @ 9:46 AM

  11. 8 (adrian smits),

    This isn’t rocket science folks! Think about Linux and and their willingness to work together to create something really special…

    This demonstrates a clear lack of appreciation of the problem. It’s equivalent to saying that it’s easy to drive from New York to San Fransisco, so it should be easy to drive from San Fransisco to Tokyo.

    The two scenarios are wildly different. In fact, the sharing of code in science is very far removed from any other scenario of sharing computer code.

    To give one example, Linux involved a single base of code (with offshoots that included commercial ventures), with one specific purpose, that of offering very limited and clearly pre-defined functions (as defined by myriad operating systems, and the definition of an operating system, and the industry standard for a worthy operating system, and the commercially available competitors such as Windows and Mac OS). A huge system developed around this which included project plans, teams, assignments, documentation methods, and so on. All of this involved people who were focused on Linux (or any other open source project) as the primary product of the effort.

    We are instead talking here about sharing thousands and thousands of individual run-once programs, written in a variety of computer languages, by people who are generating the code in support of other goals and as just one small part of their projects. The code is supplementary, and not even secondary in most cases… it’s much further down in the pecking order.

    No one in the world should be sitting back and saying “heck, it’s easy, what’s taking so long?”

    Comment by Bob (Sphaerica) — 26 Oct 2010 @ 10:09 AM

  12. I actually work pretty heavily in software and have a interest in documenting code. I recently posted in my blog about that. (www.cleancodeco.com/blog). I have even done small scientific models on my own.

    My thoughts would be that the software needs to be documented in a step by step mentality. I think it should fall back to the theory of a scientific lab report. In other words, another scientist can duplicate the results of the experiment/analysis. If the documentation is not specific enough, another scientist would not be able to duplicate the results.

    Spending a lot of time looking at other people’s source code, I have learned a very hard fact. Reading and understanding someone else’s software/code is extremely difficult and time consuming. My experience is it takes about an hour to understand 50 lines of code. With some climate models getting into the thousands or millions (?) of lines of code, it may be a lot of hours to understand every line of code.

    All climate models/analysis code follows one model. It has input, goes into a black box (the analysis code), and then produces output. I believe the black box code needs to be explained at a high level to go from input to output. The assumptions used should be in there. It should be expected to take as long as writing the software itself.

    A document explaining what happened would be much more valuable than open sourced code.

    Comment by Paul Wowk — 26 Oct 2010 @ 10:10 AM

  13. In 5, vboring said:

    For the claims that are based on code to be tested, the code itself has to be checked. Something as simple as a minor typo can lead to a problem that creates a false result even if the basic analysis is correct. Checking through back-casting can’t always find these problems because future conditions put in the model are not the same as past conditions used for checking.

    This sort of approach is exactly what I’m afraid of.

    First of all, for example, a main part of the validation of the GCM’s comes from how closely their short term predictions match the real world, and how complex, detailed modeling of micro-behaviors result in a visible, realistic macro-behavior. It doesn’t require that the code be read.

    Another part of the validation comes from the fact that multiple teams, working in isolation, create different models which independently come to similar conclusions. [They're never going to share the same missing semi-colon.] Sharing code will actually begin to defeat this sort of validation, not improve it.

    One is not going to validate such models by going through the code, line by line, looking for a misplaced semi-colon, especially as the models get more and more complex.

    But people will try. And those who are politically or financially motivated to do so will get traction by making irrefutable claims of supposed errors (irrefutable, because to most people computer code looks like magic spells, following Arthur C. Clarke’s maxim that “any sufficiently advanced technology is indistinguishable from magic”). Look at what people did with a tiny snippet of “Harry’s” code in the UEA hack?

    …the code itself has to be checked…

    No. The results need to be checked, not the code. No one ever, ever checks the code*. You check the results, and revise the code when invalid results are identified. [* The one exception is a project where code needs to be ported. In that case, the code is often read and translated, and in the process old bugs may be recognized by an insightful coder. But this is a side effect of the process, not an intended goal.]

    Something as simple as a minor typo can lead to a problem that creates a false result…

    This is a gross oversimplification of the realities involved in validating computer code.

    The reality is that industry expends a huge amount of effort performing QA on all software it creates for commercial purposes (or rather, they should, although foolish companies often run out of funds and short change that end of the project). It is never, ever, ever done by paying someone to go through the code, line by line, looking for mistakes. It might be done, in the scenario presented, if a lot of people downloaded the code, ran it in a variety of applications, and used very clever methods to look for inconsistencies or unexpected results in the output. But the complexity of that sort of effort (if the stated goal is to find mistakes in the code) should not be downplayed.

    This particular problem (i.e. validating code written for scientific endeavors) is no where near as easy as is implied, or likely to be a major result of any effort to share scientific code.

    Comment by Bob (Sphaerica) — 26 Oct 2010 @ 10:28 AM

  14. I very much like the point on independent replication. In one one of Scott Mandia’s recent articles

    http://profmandia.wordpress.com/2010/10/15/shooting-the-messenger-with-blanks/

    He made an acute point. And though he was referring to the Hockey Stick it applies here as well

    Consider the odds that various international scientists using quite different data and quite different data analysis techniques can all be wrong in the same way.  What are the odds that a hockey stick is always the shape of the wrong answer?

    It is through independent replication, especially when from multiple lines of evidence, that the implied, possible, potential and hypothetical transition into the likely, probable, or robust confidence that results from varied analysis.

    Corroborating results form so many sources, from so many different lines of evidence, form so many different source code constructs. . . what are the odds that they are all wrong AND that they largely agree in the results?

    I find it much more interesting that different models, form different scientists, that were not produced from open code, showed the same general conclusions.

    The funding issue is in form the same as the stove-piping issue that allowed increased risk prior to 9/11. Neither, in that case, has that problem been effectively solved. Without getting into the political philosophy argument, I do think that competition fosters innovation and that should not be ignored.

    It’s ironic that most denialists would likely argue that competition is good, would miss the fallacy of their argument when applied to things they prefer to whine about. Between rights and the what is best argument, it remains complex as well illustrated in the above article.

    An open source code is a wonderful thing and presents it’s own challenges. It is now proven (though their are still denialists in the form of commercial companies that for some reason don’t like open source, and have a tendency to spread disinformation about how good it is… hmmm… this pattern sounds vaguely familiar???).

    Discussion Balancing Economies (commenting fixed)
    October Leading Edge: The Cuccinelli ‘Witch Hunt”

    Fee & Dividend: Our best chanceLearn the IssueSign the Petition
    A Climate Minute: Natural CycleGreenhouse EffectClimate Science HistoryArctic Ice Melt

    Comment by John P. Reisman (OSS Foundation) — 26 Oct 2010 @ 10:35 AM

  15. vboring: “Not making code available feels a lot like when the cold fusion folks chose to not reveal the details of their methods. If several labs had independently claimed to have made cold fusion work, would that have meant that revealing the methods was unnecessary or that their claims were more believable?”

    You can buy books from your local bookstore which will tell you the methods used in all climate models starting from the 1950s to the present day!
    If you doubt the model, buy a book about climate models, learn the science and make your own model to see if you get the same results.

    The more the merrier!

    Comment by The Ville — 26 Oct 2010 @ 11:18 AM

  16. @Bob (Sphaerica): you say “It is never, ever, ever done by paying someone to go through the code, line by line, looking for mistakes”. I and my colleages have been paid to do just that. Entire books have been written about Software Inspection, and Tom Gilb’s is one of the best. We have worked on projects that produced very reliable code (millions of hours of running between failures).

    Comment by David Jones — 26 Oct 2010 @ 11:23 AM

  17. Bioinformatics researchers like me face a lot of the same issues about transparency in an interdisciplinary field. As a computer scientist, I think you should be a little less cavalier about the importance of sharing our work. Independent measurements are important, but independent assessments of the analysis is equally important. Especially when most of this work ends up buried in the supplement.

    Code can have mistakes, sometimes very subtle mistakes. Having the code released in a readable way provides the opportunity for another level of peer review. Having one open source analysis tool developed by 10 researchers is much better than ten independent perl scripts thrown together on the fly. When we can check each other’s work, the quality is a lot better.

    I strongly disagree with this statement “it is very rare that the code itself (many of which have been freely available for some time) is an issue for replication”. Sometimes, looking at the source code is the only way to determine what the methods were. Sure, having two models based on independent measurements agree on predictions is nice. But there also might be systematic biases in the approaches taken. For example, picking apart models for predicting gene regulation helped determine that a significant number of independent approaches were grossly over-fitting some parameters.

    Of course it’s not trivial, but pushing for transparency and coming up with better practices for handling source code is critical.

    Comment by Ryan Cunningham — 26 Oct 2010 @ 11:29 AM

  18. @The Ville:

    “If you doubt the model, buy a book about climate models, learn the science and make your own model to see if you get the same results.”

    Ok. Let’s say, I just made a model and it says the exact opposite of what the existing models say about the future, but does back-casting just as accurately as they do. I can state my assumptions and general method, but won’t show the details of my work.

    Let’s say ten other working groups then also produce similar results.

    Then what happens?

    Everyone can attack funding sources, intentions, and other irrelevant factors, but nobody can check anyone else’s actual methods or the details of their assumptions, so no science-based debate can actually result.

    This black box method is plainly anti-scientific.

    [Response: This is a strawman. In any paper you would need to demonstrate why your results were different - a different assumption, input or something, and that would have to be well enough described to allow replication - whether there was code or not. How useful is it to archive 100,000 lines of code without pointing out what the key change was? - gavin]

    Comment by vboring — 26 Oct 2010 @ 11:37 AM

  19. Of course, another big advantage of the “open source” approach is that it allows all those global-warming skeptics out there to help improve the science by scrutinizing the model code, submitting bug-reports and patches, assisting with q/a, unit-testing, documentation, etc…

    (**ducks and runs….**)

    Comment by caerbannog — 26 Oct 2010 @ 11:40 AM

  20. Regarding the last two paragraphs: Climate science would do well to learn from the Free/Open Source Software (FOSS) community’s approach to software development. The problem of organizing code for distributed teams of workers has largely been solved, especially through the use of modern distributed code repository tools, such as git.

    If a group of researchers are working on a paper, they could simply establish a code repository on a place such as github. The authors collectively work against that repository, and when they are ready to publish a paper, they simply tag the final “commit” that was used for the paper. The paper would identify the tag that was used.

    Subsequent development proceeds from there, and there is no need to re-archive everything for each further paper that might be published – just make changes and when the next paper is published, tag that subsequent commit.

    If it is desired to have several repositories for separate toolsets (for example, one for some analysis of some set of observational data, and another for a GCM), this is also possible. In fact, many FOSS projects proceed in just this way.

    There are also structures to allow various outsiders to participate. Different people can be given different levels of access to the repository, such as read-only or the right to make commits. If a climate team wanted to develop a true Free Software GCM, they could make the code available to the public, license it under the GPL, and allow members of the public to submit patches (to the project owners, generally not directly to the repository).

    Comment by Mitch Golden — 26 Oct 2010 @ 11:42 AM

  21. 16 (David Jones),

    I and my colleages have been paid to do just that.

    It’s done… about 1/1,000,000,000,000th of the time. And even then, you select segments of code to review, not the entire thing.

    Realistically, the effort involved is untenable.

    What would you estimate is the percent of code written world wide that undergoes actual code review?

    What sort of projects have that sort of money to spend on the effort?

    What would you estimate is the percent of the code that was reviewed on the projects on which you performed software inspection?

    Comment by Bob (Sphaerica) — 26 Oct 2010 @ 11:48 AM

  22. I really enjoyed this post. It shows that there is a lot more to open source science than is implied by those who angrily proclaim, \Release everything, even the emails you wrote to your colleagues about it!\

    I recently had the pleasure of interviewing Dr. Ben Santer, which touched on many of these topics about reproducibility and data sharing. You can read my article about it here. Lots of angry comments that necessitated deletion, though – why are deniers so hostile to this guy?? I didn’t think anybody actually took Frederick Seitz seriously, but apparently I was wrong…

    Comment by Kate — 26 Oct 2010 @ 11:50 AM

  23. One further point:

    I am fairly certain that the developers use some form of code management already. However, the older systems such as CVS or subversion do not offer anywhere near the level of sophistication of git, and are nowhere near as good at the kind of distributed development done in a FOSS project. And, more importantly here, they don’t offer an automatic way to make the codebase public, as publishing on github does.

    Please feel free to contact me at the embedded e-mail address if you’d like help setting it up!

    Comment by Mitch Golden — 26 Oct 2010 @ 11:51 AM

  24. Bob (Sphaerica):

    I do of course agree with your major thesis. However, you seem to go a little far in claiming that code isn’t checked, and doesn’t need to be checked. All code benefits from careful debugging, and line-by-line code review is a traditional part of software development. In particular, code review may identify degenerate cases that need special testing.

    There is nothing wrong at all with finding and correcting bugs no matter how small. I think the point you were trying to make is that the effect of any such bug must be negligible, since the code produces correct (“realistic”) results on a larger scale. I think that point got a little lost.

    The question that I keep asking is why, after all this time, have the self-appointed climate auditors not found dozens of minor bugs in the vast oceans of already available climate code? The bugs are there, they just aren’t looking hard enough. Some of the bugs may even have important implications for some fine-scale processes or little-understood emergent phenomena that are being studied using a particular model.

    Comment by Didactylos — 26 Oct 2010 @ 12:13 PM

  25. David Jones says: (millions of hours of running between failures)
    How long between failures? One million hours is more than a century.

    Comment by Hank Roberts — 26 Oct 2010 @ 12:50 PM

  26. @Gavin @18

    The scientifically irrefutable contrary black box is only a straw man if it can’t be done by making small changes to existing models that are within the generally accepted estimates. Considering the number and size of uncertainties, my assumption is that this is possible.

    And whether the contrary model is a straw man or not doesn’t change the primary point, which is that the details of the method are only revealed in the code. The cold fusion folks claimed that tiny changes in the method would prevent the experiment from working (how the platinum had been stored, etc), yet refused to reveal the exacting specifications for their methods. Maybe 10 working groups used ten bad fudge factors in order for their model to properly back-cast. Until the details of the method are revealed, skeptics are free to make any kind of wild unsupportable claims they want. And scientifically minded folks get to simply hope that uninspected work was done perfectly.

    100,000 lines of code sounds like a lot to archive. If each character is a byte and each line has 50 characters, that is 5 million bytes. Where will anyone find the space to archive 5 megabytes of uncompressed text? IP rights may be an issue, documentation may be an issue, archival space is obviously not. Speaking of straw men…

    [Response: Huh? The issue is not the size of the archive but the time needed to go through some one else's code. If you are given 100,000 lines of code and told to find the key assumptions, you'll be there a year (or more). Complex code absent understanding and some documentation is basically useless. Any science paper needs to have enough information so that others can replicate the results - your 'black box' paper simply does not (and cannot) exist. What cold fusion hucksters announce at a press conference is as far from a scientific paper as you can get. There have clearly been cases where replication hit roadblocks that would have been dealt with given code archiving - the MSU afffair for instance, or more recently Scafetta. But it is also true that the independent replication attempts in both cases (which might not have happened had the original code been available) were useful in and of themselves. There are no 'one size fits all' rules here. - gavin]

    Comment by vboring — 26 Oct 2010 @ 12:51 PM

  27. vboring wrote: “If several labs had independently claimed to have made cold fusion work, would that have meant that revealing the methods was unnecessary or that their claims were more believable?”

    This is a very strange hypothetical, but the answer is “yes.” If several labs were independently able to make cold fusion work, this would in fact be incredibly strong evidence that cold fusion is possible. This is true of any scientific result: the evidence that it’s correct gets stronger as more people are able to replicate it separately. If you and I both set out to solve the same problem via different methods and come to the same conclusion, this is good evidence that we are both correct. Of course, in the case of cold fusion, the non-replicability is exactly why we know there was something wrong.

    Comment by JBL — 26 Oct 2010 @ 12:53 PM

  28. 24 (Didactylos),

    Sort of.

    …claiming that code isn’t checked, and doesn’t need to be checked.

    I’m not claiming it doesn’t need to be checked. QA is hugely important. But it is very rarely done by just reading the programs.

    My concern, however, is that very, very few people have any inkling as to what computer software involves, either in writing, checking, documenting or maintaining it. All of those people out there who think that computer code is like a poem that you can read for grammatical errors and poor rhythm and weak rhymes are on the completely wrong track.

    The point of making the code public should not be so that auditors can read it looking for mistakes. Some can try, and every contribution is a contribution, no matter how it was attained… but nobody should think that that is either an objective or a likely outcome of such a project. There are other values to this, but that’s not one of them.

    The way that someone will find a mistake is by trying to implement the code themselves, provide their own inputs, make their own enhancements, and in so doing learn to understand how the code works and then, possibly, realize where a mistake has been made.

    Of course, if that is the case, the world might have been better served for them to put that same effort into starting from scratch, writing their own programs, and publishing those results if they contradict the original (or not, which becomes an affirmation instead of a refutation of the original). You know, kind of the way science has always worked. Sharing code makes it easier for people, and in so doing it makes people more likely to repeat previous errors without even realizing it.

    Which gets back to my original question: What is the actual goal of any concerted, systematic effort to share scientific code?

    Comment by Bob (Sphaerica) — 26 Oct 2010 @ 12:57 PM

  29. vboring, Well, if other groups are independently confirming your results, then you would have to look in detail at the models. That does not necessarily mean looking at the code. Presumably the “10 other groups” did not, so the issue is not there. It would have to be in the physics.

    I usually learn much more looking at physics than I do looking at code. Try it.

    Comment by Ray Ladbury — 26 Oct 2010 @ 12:58 PM

  30. Let me comment about Adrien Smits point about Linux.

    Linux system and application code is written by teams, often consisting of one person. It is in principle open and available, but for anything at all complicated, you have to in essence become part of the team in order to decipher the code. For example, on various occasions, I’ve tried to figure out exactly what the program gimp was doing, but just studying the code was of little use. I had to ask questions in forums and directly of the authors. And then I was only partly successful. To do better than that I would have to in effect become part of the gimp development team, and I don’t want to spend my time working to be able to do that.

    I can’t imagine trying to evaluate climate model code without becoming a climate modeller myself, and that would require spending all my time for at least three years first doing the equivalent of a graduate program in the subject, under the supervision of people already doing that work.

    Finally note that there is no group of skeptics, who aren’t experts in Linux, trying to show that Linux is a fraud. One of the reasons it works as well as it does is that Linux users trust the developers. So we don’t have to spend a lot of time debating the virtue of the system.

    [Response: Actually, Microsoft was sponsoring very similar attacks on Linux not too long ago, and oddly enough, it was spearheaded by the same 'think tanks' that also attack climate science. - gavin]

    Comment by Leonard Evens — 26 Oct 2010 @ 1:04 PM

  31. Leonard Evens wrote: “Finally note that there is no group of skeptics, who aren’t experts in Linux, trying to show that Linux is a fraud.”

    Actually, at least for a while there were some folks at Microsoft who tried to make that argument.

    Comment by SecularAnimist — 26 Oct 2010 @ 1:11 PM

  32. I guess having the code is not a requirement. Some papers do get published in my field that simply don’t include enough details to replicate the research, which can be infuriating. Having the code here would help, but arguably peer review should have caught these and told them to spend more time on their methods.

    More code would save people a lot dreary implementation work that might be useful once, but not 20 times. All that wasted time might have been good for the career of one scientist, but not for those of the 20 others, and not for science as a whole.

    Comment by kyn — 26 Oct 2010 @ 1:13 PM

  33. (P.S. — I’m not a programmer, it’s a serious question: can you count “millions of hours … between failures” if hundreds of copies average a year — 8760 clock hours if uninterrupted — between failures?

    I do know the story from Techweb 9 April 2001:

    “… The University of North Carolina has finally found a network server that, although missing for four years, hasn’t missed a packet in all that time….. by meticulously following cable until they literally ran into a wall. The server had been mistakenly sealed behind drywall ….”

    I assume you’re using an industry standard for time between failures, but–how?)

    Comment by Hank Roberts — 26 Oct 2010 @ 1:32 PM

  34. Yes, I find all these people worshipping at the altar of open source very strange indeed.

    Open source is terrible. And, like democracy, it is the worst possible solution – except for all the others that have been tried.

    We have to do the best with what we have, imperfect as it is.

    Comment by Didactylos — 26 Oct 2010 @ 1:36 PM

  35. If the point is that you need to document somewhere exactly how you did, whatever you did then, in my experience, it is a heck of a lot easier to document the code with many, many comment statements. If not you have to go through the whole documentation process twice. Moreover, that kind of documentation makes it easier for the next person to build on your results or, dare I suggest others are afflicted with my problem, to answer questions about what you did months later. Open code to me would just simplify documentation.

    Comment by Roger Caiazza — 26 Oct 2010 @ 1:36 PM

  36. @Gavin @26 – this is a very inefficient way to comment on anything. Comment systems with the ability to reply are much more effective.

    “If you are given 100,000 lines of code and told to find the key assumptions, you’ll be there a year (or more). Complex code absent understanding and some documentation is basically useless. Any science paper needs to have enough information so that others can replicate the results – your ‘black box’ paper simply does not (and cannot) exist.”

    So, give me 100,000 lines of code plus the key assumptions. The key assumptions are already published, so all I need is the code. People hack code that they have no documentation or even code for all the time. Given both the key assumptions and the code, determining the precise methods would be a fairly straightforward (though time consuming) task. In any case, it is a weird argument to say that the code shouldn’t be published because people might find it difficult to use. Philosophical arguments are often difficult to follow. Does that mean philosophers should just publish key assumptions and results and skip the details of their reasoning altogether?

    Essentially, you’re claiming that all of the necessary information is already contained in published sources and I’m saying that I don’t care what you think I do or don’t need to know. If the output from your work is being used to justify alterations to my life (and I honestly think they should be), then I should have the right to see the actual method used. Not the overview, not the key assumptions, not the basic physics. The actual method. The code and data.

    Doing otherwise creates an image of secrecy which makes the science seem more controversial than it is, which reduces the chances of climate policy being implemented effectively.

    [Response: Did you even read the post? Then how can you think I am arguing for secrecy? Nonetheless, 99% of anything that is archived will be completely ignored in terms of public policy (as it is now), and since archiving is never going to be 100%, someone will always be able to claim something isn't archived (even if it is - few people bother to check). Since most scientists (believe it or not) are not obsessed with the public policy outcomes of their work, this is not a sufficient incentive - it has to be useful for science too otherwise it isn't going happen. This is just an observation, not an aspiration. - gavin]

    Comment by vboring — 26 Oct 2010 @ 1:41 PM

  37. @Ray Ladbury @29

    I can only know the details of how the physics were implemented by looking at the code.

    When you’re talking about a 2 degree C signal in hundreds of years of data with several inputs that have order-of-magnitude error bars, the details matter.

    If one is looking for a way to agree with something, the overview argument is enough to convince them that the speaker is making a serious claim that they’ve really thought about. If one is looking for a way to disprove something, every detail is needed because it is thought that any one of them could be the undoing of the argument. This is why people who agree with AGW claims think that enough information has already been given and those who don’t agree think that every detail of the methods must be made public.

    Comment by vboring — 26 Oct 2010 @ 1:51 PM

  38. Pet Peeve: Commercial software that is a year old so they won’t support it and it won’t run on your new machine but it is still patented and copyrighted. The law favors the big corporation to an extreme extent.

    Another problem: I have enough trouble figuring out how to use data that are in NASA and NOAA web sites. I’m pretty sure the pundits wouldn’t recognize code if they saw it.

    Comment by Edward Greisch — 26 Oct 2010 @ 2:02 PM

  39. I think that perhaps a good in-between would be just simply to have the provision of the skeleton of the coding done. Methods sections in papers are often insufficient for understanding how things are implemented but the provision of say a skeleton would let those reading the code understand what was done, but still have to write the code themselves to do it.
    I agree just providing the code is not a good thing but step by step instructions on how something was done could be important because people seeing that can write their own code and perhaps streamline or improve things.

    Comment by Robert — 26 Oct 2010 @ 2:20 PM

  40. Hank Roberts quoted TechWeb 2001: “The University of North Carolina has finally found a network server that, although missing for four years, hasn’t missed a packet in all that time … by meticulously following cable until they literally ran into a wall. The server had been mistakenly sealed behind drywall”

    Betcha it was a NetWare server.

    Comment by SecularAnimist — 26 Oct 2010 @ 2:30 PM

  41. vboring,
    Good point. Any input with large enough error bars could cause the entire model to unravel. That is why some who include uncertainties in their calculations arrive at 21st century warming between 1 and 6C.

    Comment by Dan H. — 26 Oct 2010 @ 2:54 PM

  42. The point that at the end of the day climate science’s veracity rests on the physical evidence, and not code in models, is very important. If different groups, funded by different governments on different continents using different measuring devices and writing their own modelling/analysis code are all coming to the same conclusions it shows that climate change isn’t an artefact of a coding error in some computer simulation.

    However in many cases in the public debate this is too nuanced a point to be useful – it doesn’t fit into a soundbite. Therefore the code really does have to be completely made open, even stuff that isn’t really useful on its own, so that the answer to bullsh** questions like ‘Aren’t climate scientists hiding the code they use to produce their results?’ is a simple ‘No, all such code is freely available to anyone on the internet’.

    Comment by HappySkeptic — 26 Oct 2010 @ 3:14 PM

  43. @gavin @36

    “Did you even read the post? Then how can you think I am arguing for secrecy? Nonetheless, 99% of anything that is archived will be completely ignored in terms of public policy (as it is now), and since archiving is never going to be 100%, someone will always be able to claim something isn’t archived (even if it is – few people bother to check). Since most scientists (believe it or not) are not obsessed with the public policy outcomes of their work, this is not a sufficient incentive – it has to be useful for science too otherwise it isn’t going happen. This is just an observation, not an aspiration. – gavin”

    I am a bit of a hard liner when it comes to openness is science. As I see it, the work of any public employee who is paid to generate knowledge belongs to the public (with exceptions for national security, etc).

    Separately, for a scientific paper to be published, I think every detail should be present to allow for identical replication of the original result. Finding problems in the published approach would take a small fraction of the time compared to starting from zero trying to replicate results based on a described method, so the value to science is obvious.

    From this perspective,arguing for anything short of complete openness is arguing for secrecy and excuses for why it doesn’t matter or why some scientists don’t want to do it are uninteresting.

    [Response: 'identical replication'? Under all circumstances? Not possible. Different compilers, different math libraries mean that any reasonably complex code will generate bitwise different results (and far more than bitwise in a chaotic system) if run on a different computer, or even the same computer at a later time. Some codes are specifically written only for the computer they were designed for (the Earth Simulator for instance). This is not - in practice - identically reproducible. Or are we supposed to archive the entire OS and functions as well? No absolutist approach such as you advocate is possible, and in demanding it, you actually set back the work on getting anything done. - gavin]

    Comment by vboring — 26 Oct 2010 @ 3:14 PM

  44. Hello,

    I’ve ported many codes. The only documentation I read carefully when I have to go through the program line by line, are (1) the purpose of the subroutine (2) the list of parameters and assumed data structure (integer, floating, array) and (3) the list of output and their data structure. The problem with detailed documentation is that frequently the documentation is erroneous … the programmer may have written a program, perhaps modified the program and then forgot to change the documenation to reflect the change. With just the purpose of the subroutines and calling sequences for the input and output I can write a standalone program to call the subroutine and single step through the subroutine to see what it does.

    Sometimes I want to understand the code. It helps then if the subroutines are short and performs only one function. In my own coding I try to keep the length of a subroutine to 2 screenfulls of source code. If it is much longer, I try to break it into two subroutines.

    Being able to reproduce results is the bedrock of science and, IMHO, the code and data are necessary for reproduction. Suppose A presents the results depending on a computer run in a paper. A makes available the data and a detailed description of the algorithm to produce in the code. B tries to reproduce the code and using the same data, cannot get the result that result. It seems there are 3 possibilities: (a) A has a bug in his/her program (b) B has a bug in his/her program and (c) there was a misunderstanding in the specification of the program.

    klee12

    Comment by klee12 — 26 Oct 2010 @ 3:16 PM

  45. vboring:
    “Essentially, you’re claiming that all of the necessary information is already contained in published sources and I’m saying that I don’t care what you think I do or don’t need to know.”

    Or rather you aren’t interested in listening, your on a pre-programmed course and no one is going to change your mind. I think others can assess your attitude and make their own judgement.

    vboring:
    “If the output from your work is being used to justify alterations to my life (and I honestly think they should be), then I should have the right to see the actual method used.”

    Well I hope you apply that logic to all public policy!
    Do you apply it to policy based on ‘morals’ or religion or anecdotal evidence?
    I’m guessing you accept public policy when you personally feel happy about it, which has little to do with the reasons for introducing the policy.

    vboring:
    “Not the overview, not the key assumptions, not the basic physics. The actual method. The code and data. “

    Comment by The Ville — 26 Oct 2010 @ 3:19 PM

  46. Yet the need for more code archiving is clear.

    I am glad that you wrote that.

    You rightly critique the per paper approach to archiving, but that itself is an improvement over the previous practice.

    Comment by Septic Matthew — 26 Oct 2010 @ 3:44 PM

  47. I think the issue of source code availability is important but more so for the wider acceptance of scientific work than for the progression of scientific knowledge itself.

    Simply put: People want to replicate the work of scientists to ensure the results are reliable and not simply the word of a small group of insiders. The ins and outs of how the debate evolved is something we could all write about extensively but when a topic is controversial it’s a lot to ask of people not in the scientific community to trust in peer review etc when they could just have the code and data instead.

    I think the ability of others to independently analyse and implement GISTEMP has done a lot to dispel the idea that warming trends are purely an artifact of analysis. Naturally it hasn’t removed all criticism but they are those who will never be satisfied, the goal should be to satisfy those who are reasonable.

    [Response: Fair enough, but despite all code and all data being available, and it being independently coded, and it being independently validated, I am still seeing accusations of corruption, fraud and data mishandling. And these are still making news. It is precisely the fact that there are loudmouths who will never be satisfied and plenty of people around who don't check their claims, that the notion of perfect archiving suddenly improving public appreciation of the science is nonsense. People will continue to use stale talking points as long as this has political connotations. - gavin]

    Comment by sharper00 — 26 Oct 2010 @ 3:57 PM

  48. Gavin wrote: “… in demanding it, you actually set back the work on getting anything done.”

    Of course, with regard to climate science, for some people that’s a feature, not a bug.

    Comment by SecularAnimist — 26 Oct 2010 @ 4:06 PM

  49. “I am still seeing accusations of corruption, fraud and data mishandling.”

    Certainly and they’ll continue because not everyone is playing the same game. I think we’re all aware there are those who know nothing about climate and care even less but are only involved in order to dismiss the entire thing because it doesn’t match their preconceptions.

    Which is why I say above that the goal should be to satisfy those who are reasonable. When data and code is locked away reasonable people will ask why and become concerned about accusations behind such a thing. When data and code is open and independently implemented reasonable people scoff at claims it’s manipulated.

    I mentioned GISTEMP above, the independent reconstructions should put to bed the concerns any reasonable person might have that warming trends are simply inserted into the data. Of course that still leaves the daily rants about adjustments at location X but there’s basically nothing that can be done about those.

    “the notion of perfect archiving suddenly improving public appreciation of the science is nonsense.”

    I don’t believe it will anymore than problems of unarchived/unavailable code and data instantly damaged public perceptions of science. However over time freely available code will serve to increase confidence while unavailable code will do the opposite.

    Comment by sharper00 — 26 Oct 2010 @ 4:28 PM

  50. @hank: Yes, ‘millions of hours between failures’ means ‘hours of usage’. So if the software is being used in one million sites, and each hour on average one site experiences a failure, then they have a million hours between failures.

    It is possible to imagine cases where this metric is pointless (an obvious one would be some kind of input that will not occur by random chance/wide-scale usage but only at a specific date or time, such as the Y2.038K bug; if your software runs on millions of sites for years with few failures, that’s great and you’ll have a good average, but it’s still possible that every one of those million ‘reliable’ sites will fail drastically in 2038) but typically it’s probably a pretty good measure.

    I once worked on a system which monitored the same metric and managed a quality 1.5 hours between failures. :) Largely my fault, too. Since decommissioned.

    Excepting code for nuclear power stations and rockets and whatever, most code is not written to anything like the specifically-checked or million-hour standards. However it’s not true that outside that realm code is very rarely read. Lots of organisations, including the one I work at, have code-review policies where developers have new code casually reviewed either by peers or by senior developer before committing it.

    Even though our reliability requirement is along the lines of ‘eh it’s a website, if it breaks we’ll patch it’, rather than the ‘oops, if it breaks we might accidentally launch a nuclear strike’ level, the review process is still quite important for us.

    I would second the recommendation for a distributed VCS such as Git when used in this context. Archive quality code in a proper version control system which can be publicly referenced and you can easily point to the exact version used. For throwaway crappy code written just for that paper, or a few lines of Matlab or whatever it was, sure, go ahead and archive it with the paper.

    Incidentally the same process might apply to datasets depending on how they are stored and structured; in other words if you use a standard dataset there is no point archiving the whole thing with the paper, but if that dataset is stored in a version control system, you can simply reference the precise version that you used.

    Because distributed VCS are, er, distributed, you have security in the event that (for example) GitHub goes bust. Not a problem, everyone who uses it has a full copy; you can stick it onto some other host and all the history, tags, other references, etc should still work.

    Comment by sam marshall — 26 Oct 2010 @ 4:50 PM

  51. sharper00 said: “People want to replicate the work of scientists to ensure the results are reliable and not simply the word of a small group of insiders.”

    …only they don’t. When given the code, all the reams and reams of it, they ignore it.

    If you look, you will find that the few people who do take advantage of open climate code are not the people screaming for it and bashing away on their drum.

    Maybe you are going to counter by saying “it’s not about whether the code is actually checked, but whether it’s available to be checked if we want”. But that’s utterly bogus, as well. The data in many cases is already available, and nobody is checking it. And if nobody looks at it, why spend all the time and money making it available?

    No – the deniers are just paranoid. We can’t help them. Conspiracy theorists don’t turn in to normal people in my experience.

    So, vboring and sharper00: keep banging away at your drums and ignoring all the data you already have access to. There are far better reasons for publishing code than your hysterics, as Gavin and Eric have explained in a rather good post that you totally ignored. I’m still thinking about the implications myself.

    Comment by Didactylos — 26 Oct 2010 @ 4:58 PM

  52. klee12: Being able to reproduce results is the bedrock of science and, IMHO, the code and data are necessary for reproduction.

    BPL: The code is NOT necessary as long as the paper gives the algorithm. Sometimes even the data isn’t necessary if another researcher or team can gather their own. I don’t think you understand what “reproducibility of results” actually means. It doesn’t mean you duplicate someone else’s work line by line. It means you test the same principle and get the same result. In fact, it’s more impressive if you get the same result a different way.

    Comment by Barton Paul Levenson — 26 Oct 2010 @ 5:36 PM

  53. Modelers think they are doing a great job at internal climate variability….

    Maybe a warning like on cigarettes will do the trick (wouldn’t be too difficult to make it retrospective).

    —————————————————————————

    WARNING! WARNING! WARNING! WARNING!
    THIS PUBLICATION CONTAINS DATA FROM A COMPUTER SIMULATION. IT REMAINS POSSIBLE THAT COMPUTER SIMULATIONS HAVE GROSSLY UNDERESTIMATED OR OVERESTIMATE INTERNAL PROCESSES.

    [Response: Sure, and on data papers we could say "DATA MAY BE REVISED AT ANY TIME DUE TO POTENTIAL UNDISCOVERED CALIBRATION PROBLEMS AND FORTHCOMING WORK THAT MIGHT SUGGEST THAT THE MODEL-DATA COMPARISON WAS NOT COMMENSURABLE. WE RESERVE THE RIGHT TO REVISE ANY AND ALL INTERPRETATIONS WITH NO ADVANCE NOTICE." - how about that? - gavin]

    Comment by Isotopious — 26 Oct 2010 @ 5:39 PM

  54. I think well worth understanding is that in some fields such as mining exploration repetition is required, and replication is not good enough. If a mining company can do some exploration, publish the raw data, the general methods and the final answer, then the potential for fraud exists. A small adjustment of the results, (say by 1%) would not be detectable by replication, as replication is not expected to be exact, so a small difference is acceptable. When trying to sell a new mining operation overstating results by 1% could lead to millions of dollars more value, so a powerful motivation to fudge the results is present. Protecting against such potential fraud can only be done by insisting on exact repetition, and so in fields such as mining exploration repition becomes the necessary gold standard.

    Then you can get someone from such a field who knows that repition is the gold standard, and that lack of repitition can easily allow fraud. If they don’t understand the reason for such a rule, and how climate science is different, then they will insist in a full audit of the climate science books and suspect fraud if they can’t get it.

    [Response: Hmm. In point of fact, most companies, mining or otherwise, are able to hold on to virtually all the information they have, even in the face of audits. At least in the U.S., this is true even when they have government funding. Try getting information on the currently-hot topic (no pun intended) of active geothermal research. You'll find 'replication' is not going to be possible. That's because corporations have rights to intellectual property individuals apparently don't.--eric]

    Comment by Michael — 26 Oct 2010 @ 5:50 PM

  55. No problem Gavin. Catering for the lowest common denominator is a fact of life, put yourself in their shoes.

    Comment by Isotopious — 26 Oct 2010 @ 6:33 PM

  56. Didactylos(#51)

    I don’t think you can reasonably say that because there are very few looking at the code, the motives of those asking for more transparency has to be to deny and delay. It would seem that the population of those willing to put that much effort into that type of code review/replication would be extremely small even if not one person asking for transparency had alterior motives.

    Just looking at the GISSTEMP replication and there are at least 7 – 8 full replications of that work, while at least another one or two dozen bloggers posting about it in some for (I’m talking about the technical nitty gritty here).

    Considering the code for the GCM’s is so much larger and more complicated, not to mention the fact that they wouldn’t be able to run on a normal desktop computer (unless it’s an older version) I think that the number of bloggers digging through the code is actually quite high.

    One other note regarding comments in code. There are some developers that would say you’re code should be able to describe what it is doing without needing extensive documentation. This is idealized but the idea of using better names for variables, spending a bit more time developing a better structure to the program or using more appropriate tools for the task at hand can go a long way.

    Comment by sambo — 26 Oct 2010 @ 6:44 PM

  57. Dan H. says “Any input with large enough error bars could cause the entire model to unravel. That is why some who include uncertainties in their calculations arrive at 21st century warming between 1 and 6C.”

    WRONG!!! The range of estimates has to do with 1)sensitivity, and 2)emissions scenario. The range of estimates for sensitivity has to do with those that yield Earth-like behavior of the climate. Emissions are up to us.

    Comment by Ray Ladbury — 26 Oct 2010 @ 7:31 PM

  58. “Mathematicians progress through providing step by step, line by line documentation of every proof.”

    Thanks for the chuckle. Almost no mathematics is so rigourous; the proofs are attempts to communicate the reasoning to other mathematicians. A few months back there was an article in the Notices of the American Mathematical Society by somebody who actually does computerized formal proofs of major results in mathematics. There is a research community working on improved methods for formal proofs, typically of computer programs, applied via computer.

    A formal proof meeting all the criteria laid down by logicians is an awe inspiring, even fearsome, thing to behold; such belong only in the supplimental information. :-)

    [Even reCAPTCHA agrees: "YAHWEH doncerri".]

    Comment by David B. Benson — 26 Oct 2010 @ 7:45 PM

  59. I have said it before many times. Share data sources (though not datasets as in the same databases). Share methods. Be as frigging helpful as you can. DO NOT share code. Archive it so if there is a question you can go over it later. If you have a disagreement that can’t be resolved, then go over each others’ code line by line if you need to. However, keep a firewall between your code and that of your rivals.

    The goal is verification of the methods and results. Replication of the results is a way to do that. Running the same code on the same data and getting the same results (i.e. repetition) proves only that your machines are compatible.

    Comment by Ray Ladbury — 26 Oct 2010 @ 7:48 PM

  60. Sam Marshall sez:

    I would second the recommendation for a distributed VCS such as Git when used in this context. Archive quality code in a proper version control system which can be publicly referenced and you can easily point to the exact version used. For throwaway crappy code written just for that paper, or a few lines of Matlab or whatever it was, sure, go ahead and archive it with the paper.

    The source code for GISS Model E is available in a public CVS repository and cvsweb is set up so people can view it that way, if they prefer. There’s also a source code browser available on the GISS Model E website.

    So why do people think that suggestions like “use a VCS tool” is news to people involved in the more complex world of climate-related software development?

    Oh, it’s not git … it’s CVS … ummm, that just means they’ve been doing using version control software a LONG TIME.

    As with so many things in the so-called climate science debate … people tend to pontificate without first asking … have the people being criticized already thought of this stuff?

    CO2 as a GHG does not violate the second law of thermodynamics … CHECK!

    Source code to at least some climate science related software is under version control … CHECK!

    Gavin acts like a “mean [high school] girl” so all of climate science is a fraud … CHE… oops, no, not so much :)

    [Response: Actually we will be switching to git in a couple of weeks. - gavin]

    Comment by dhogaza — 26 Oct 2010 @ 8:20 PM

  61. dhogaza @60 — Writing in all caps is consideered to be the equivalent of shouting and so is considered to be quite rude.

    Comment by David B. Benson — 26 Oct 2010 @ 9:21 PM

  62. @60: If you read my original comment, the point I was making was that they were probably suing something like CVS but that git is substantially different, and far more suitable to solve the problem being posed: of organizing code by function and making it public.

    Gavin: glad to hear you are making the switch. Some questions: (1) where is the public repository being hosted, and will the public have access to it? Do you think it’s worthwhile for some relevant professional organization to run a repository, instead of github?

    Will you be making public every single commit in the development process? This is a level of transparency that goes beyond the requirements of making the results public, and would actually perhaps indicate the thought processes during development.

    Comment by Mitch Golden — 26 Oct 2010 @ 9:33 PM

  63. Of course we have different sorts of people making the requests. For some deniers, simply being able to claim they are hiding something, or forcing researchers to waste time and money satisfying a request is enough. And I do suspect we will have people of bad faith looking through the sources for things that can be made to sound bad taken out of context “hide the decline” sorta things.

    But to get more into the technicalities: I’m currently doing QA on a large commercial engineering code. Now the range of problems for our application is much broader than say analying temperature data, or running a GCM, so bombproof QA is an impossible task. But, one thing I’ve noted is that maybe half of reported bugs upon detailed inspection are actually user errors. So I think the issue could come up, if you greatly expand the user community for your codes the effort to filter out reported false negatives (reported bugs that aren’t really bugs), could expand greatly.

    Secondarily, often more important than the actual source code are the proceedure and methods used for verification and shakeout of bugs. Often these include throwaway code, that is used to generate and/or analyze specialized test cases. Reading source code looking for errors is pretty difficult. Also if “code” is being generated by automated or semiautomated means -say as output from a computer algebraic system, or data-table driven code generation, then it may be truly human-unreadable, and the proper “replication archive” would be the input to the code generator rather than the source code itself.

    Lastly, as Gavin suggested, old code was compiled and run (and hopefully QAed) on older computers with now out of date and no longer supported compilers and libraries. In my own work I can get exact source code from several years ago, but the ability to compile and link it, and run it under the original environment does not exist. So there are real limitations to what historical archiving can do.

    Comment by Thomas — 26 Oct 2010 @ 11:49 PM

  64. Gavin:

    Response: Actually we will be switching to git in a couple of weeks. – gavin

    Yes, and I’m using git in my current major consultancy contract, while older work I’m involved in still uses CVS, and another project I’ve had involvement in a decade ago (PostgreSQL) is now just about ready to switch to git rather than CVS.

    My point wasn’t that CVS was as good as git … my point was that the fact that you guys have been using CVS for years means you’ve been on top of shit for years, despite what detractors say (and the fact that you’re switching to git – if you haven’t checked out the improvements to the CVS->git stuff contributed by the PostgreSQL folk, i.e. in terms of preserving history etc, do so! – simply amplifies the point).

    dhogaza @60 — Writing in all caps is consideered to be the equivalent of shouting and so is considered to be quite rude.

    Which caps are you talking about, pray tell? CVS has traditionally been written “CVS” forever.

    “CHECK!” yeah I meant to emphasize that, you have a problem with it?

    CO2? that’s wrong?

    GHG? I’m rude?

    GISS? wrong? “E” (after “Model”) rude?

    VCS? another acronym?

    WTF are you talking about?

    Oh, sorry … I meant “wtf” because I hate to be rude …

    Comment by dhogaza — 27 Oct 2010 @ 12:28 AM

  65. @60: If you read my original comment, the point I was making was that they were probably suing something like CVS but that git is substantially different, and far more suitable to solve the problem being posed: of organizing code by function and making it public.

    Not really. It’s better, but not for the reasons you mention. CVS makes code public just as well as git.

    “substantially different”? My 40 years of software engineering experience argues against it. Incrementally better, yes, and easier to manage, yes. CVS is file level, SVN and git oriented towards file sets, but people using CVS have worked around this forever and for a very long time has been the open source standard for public repositories.

    Yes, we’re moving forward, and it’s a good thing. But this Holy Allah git rap is simply wrong.

    Comment by dhogaza — 27 Oct 2010 @ 12:49 AM

  66. Eric and Galvin,

    “However, while everyone is in favor of openness, transparency, motherhood and apple pie, there are some serious issues that need consideration before the open code revolution is going to really get going.”

    Here is my take on this situation….

    Scientists and Mathematicians need to build an abstract scientific open source framework. The framework itself is not a model nor does it contain models; instead, it contains various algorithms and tools that scientists and mathematicians could use to construct a model. If mathematicians and scientists from all backgrounds would be willing to collaborate, the framework could become very powerful, and it could have the ability to have a great impact on scientific and mathematical research.

    Such a framework would have to be built in C++. C++ contains enough tools that the framework could incorporate sophisticated 3d models with tools such as direct x. In addition, sophisticated network algorithms could be created so that scientists could collaborate computer resources for intensive calculations. Scientists could also release objects for their specific models that other scientists could plug into the framework to see the results along with creating unique models for independent investigation.

    But for the framework to be useful and powerful, people would have to collaborate. We are already past the point where we can be masters of every science; therefore, we could benefit greatly by having people who are masters of a specific field contribute to that area of the framework.

    Here is the great problem with the idea…
    Scientists and mathematicians compete for spots in history books and funding for research. The open source framework is a focus on the community more than the individual, and some people may not like the idea of a community approach to scientific research.

    Comment by E.L. — 27 Oct 2010 @ 1:21 AM

  67. Barton Paul Levenson @52 wrote

    “The code is NOT necessary as long as the paper gives the algorithm….It doesn’t mean you duplicate someone else’s work line by line. It means you test the same principle and get the same result. In fact, it’s more impressive if you get the same result a different way.”

    I think we are addressing different problems. A writes a paper and describes the algorithm. B thinks that the algorithm is fine but, but for whatever reason, thinks there is a bug in code. He codes up the algorithm and gets different results. How do we decide who is correct? The algorithm may be correct but its implementation may not be.

    There is always the chance that there are bugs in programs and we should recognize it, especially in scientific code. Especially insidious are round-off errors. You cannot find these errors without the code.

    In regard to CVS and such systems, I don’t think it is necessary if you have a single person working on the code. If a code is experimental and for private use it doesn’t need a formal version number. But when you release it to the public or release data that requires requires your code, you’ve got to have give the data and the code version numbers so you can keep track of all the stuff you’ve made public.

    If the project is so large that there are a number of programmers. contributing code to a project, you’ve need CVS or something like it to keep the versions by all the programmers synchronized. I don’t think in most cases a single programmer working on a single project needs CVS. The exception may occur when the programmer wants to release different versions for different platforms. However, I think portability is not a great problem with scientific codes. I could be wrong here.

    klee12

    Comment by klee12 — 27 Oct 2010 @ 1:28 AM

  68. In addition to my above comment, the framework would have to be centralized instead of scattered. One large problem with open source is that is has the tendency to scatter into multiple projects where people try to obtain control.

    Comment by E.L. — 27 Oct 2010 @ 1:29 AM

  69. @Hank Roberts @33 (ah, the perils of writing in “blog mode”). Yes I have written software that runs for millions of hours between failures. No that does not mean there is one instance running for millions of hours; it means there are thousands of instances all of which run for thousands of hours with no failure in any instance. This is a fairly standard way to try and measure MTBF. Under a suitable model, we estimate p, the probability of failure in any given hour; 1/p then gives the Mean Time Between Failure.

    Comment by David Jones — 27 Oct 2010 @ 1:57 AM

  70. @Didactylos

    “…only they don’t. When given the code, all the reams and reams of it, they ignore it.”

    No what you mean is that some don’t because some didn’t really care about the code being open in the first place. That’s not an argument against bothering to open it in the first place.

    “If you look, you will find that the few people who do take advantage of open climate code are not the people screaming for it and bashing away on their drum.”

    The kinds of people who engage in drum bashing are probably not the type to sit down and invest many hours in a sober analysis of source code.

    However consider this: What were the people who did perform that analysis doing before the code was released? We can’t say for sure of course but in the case of GISTEMP individuals with a range of opinions went and reconstructed it thus in some cases people who already thought it was ok reaffirmed their opinion but in other cases people who were undecided or tending towards the belief there was something “funny” going on learned that it was actually a fair and robust means of building a global temperature record.

    Does that convince people who don’t care much about evidence? By definition no.

    That said:

    “So, vboring and sharper00: keep banging away at your drums and ignoring all the data you already have access to. There are far better reasons for publishing code than your hysterics”

    This is all pure nonsense. I posses no drums and lack a sufficient sense of rhythm to use them if I did. You do the position you’re apparently advocating harm by reflexively assuming anyone who thinks having open data and code is a “good thing” must embody the very worst of those who request such things.

    It’s far from hysterical to make a case for how this enhances the credibility of the work involved and undermines the arguments of those who seek to oppose science purely because it produces answers they dislike.

    Comment by sharper00 — 27 Oct 2010 @ 4:21 AM

  71. ‘git’ is an unfortunate term to use. Here in the UK it has a whole different meaning:

    http://en.wikipedia.org/wiki/Git

    I imagine it causes much amusement in software circles here.

    Comment by The Ville — 27 Oct 2010 @ 5:45 AM

  72. Regarding the post below. Nobody spread “misinformation” or slander. If anyone did it was not Gail,Rpauli or myself. http://www.gcbl.org/land/green-infrastructure/trees-and-air-pollution
    http://archive.greenpeace.org/ozone/radiant/1rad.html
    http://www.eoearth.org/article/Impact_of_ozone_on_health_and_vegetation
    http://whqlibdoc.who.int/monograph/WHO_MONO_46_(p233).pdf
    http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VB5-4NF2HPD-1&_user=10&_coverDate=06%2F30%2F2007&_rdoc=1&_fmt=high&_orig=search&_origin=search&_sort=d&_docanchor=&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=4cea5a7e04b1add34eebad1f5622718a&searchtype=ahttp://www.amnh.org/nationalcenter/youngnaturalistawards/2006/megan.html%20%3C—
    http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B75CG-48XVFF8-F&_user=10&_coverDate=10%2F31%2F1972&_rdoc=1&_fmt=high&_orig=search&_origin=search&_sort=d&_docanchor=&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=91a8023282be90a0b29f2f4a632ee295&searchtype=a
    http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VB5-48XKNCG-1T&_user=10&_origUdi=B75CG-48XVFF8-F&_fmt=high&_coverDate=12%2F31%2F1988&_rdoc=1&_orig=article&_origin=article&_zone=related_art&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=e0213c0b4aa5b0635b522500f7e6ba82
    http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VRD-43YR2YH-2&_user=10&_coverDate=09%2F30%2F2001&_rdoc=1&_fmt=high&_orig=search&_origin=search&_sort=d&_docanchor=&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=e155629c5b8cbb07b70ca3318df683a0&searchtype=a
    http://www.nasa.gov/topics/earth/features/soybeans.html
    http://www.forestthreats.org/publications/su-srs-018/ozone-bioindicator-plants
    http://www.ars.usda.gov/research/publications/publications.htm?seq_no_115=239382 Studies by the USDA and NASA hardly qualify as misinformation. THese people know what they are talking about. And we ought to listen!

    Comment by Highschooler — 27 Oct 2010 @ 8:06 AM

  73. Klee12 says, “A writes a paper and describes the algorithm. B thinks that the algorithm is fine but, but for whatever reason, thinks there is a bug in code. He codes up the algorithm and gets different results. How do we decide who is correct?”

    Because then C will take her shot and should agree with either A or B. If not, then we can fairly attribute the problem to bugs in the code of all three. Then and only then, they might want to get together and compare lines of code. Keep in mind that most “bugs” do not alter the result in a significant fashion. The purpose of scientific computer models is not to give “answers” but to give insight.

    Comment by Ray Ladbury — 27 Oct 2010 @ 8:13 AM

  74. Does http://reproducibleresearch.net/ have anything useful to offer to this discussion?

    Comment by Bill Harris — 27 Oct 2010 @ 8:17 AM

  75. I think that some folks here have no idea of the magnitude of the project they are talking about. A good example of an open source scientific computing package is the GEANT 4 code used in applications from particle physics to space science. It is a great package, but the configuration control is critical, and each time there is a revision it takes a huge amount of work. What is more, SOMEBODY has to pay for that work. If you want to double the funding of climate science so they can hire people full time to manage the code, GREAT. But pony up first.

    Comment by Ray Ladbury — 27 Oct 2010 @ 8:19 AM

  76. Indeed The Ville, if you follow your link and then the Git software link you get Linus Torvalds reportedly saying “I’m an egotistical [sic] bastard, and I name all my projects after myself. First Linux, now git.”

    Comment by P. Lewis — 27 Oct 2010 @ 8:30 AM

  77. From a CS background but not a Climate background…

    Are we really talking about two types of code here? In my opinion the first bit of code is around the data and how it is brought together. I can see this type of code being released because that set of code is the implementation of a set of principles around the input date itself.

    I think that the model itself is what should be independently replicated starting with the same set of known data and the rules and processes around that set of known data. I would think that it would be a valid replication if you can take a known set of data combined with a hypothisis and either reproduce the results or not reproduce the results.

    Comment by Frank — 27 Oct 2010 @ 8:41 AM

  78. Thanks for the link, Gavin and Eric. I don’t really have time to comment this week but here are a few pointers. For more, see the Climate Code Foundation, both now and in future, as we write white papers and blog posts to cover more of this in more detail.

    First, my Nature piece was written to very strict word limits (“World View” pieces have to fit a single page in print), and the editor specifically wanted something which would address and challenge working scientists in many fields. So I hope nobody assumes that it’s all I have to say on the subject, or that I’m unaware of the range of current practices, both across science and specifically in climate science. For instance, I am well aware – and wrote in my submission to the Muir Russell inquiry – that science includes many medium-to-large software projects (e.g. 100K-10M lines of code), within which software development practices are often quite sophisticated. Similarly, code publication is already the practice of some scientists, and is even commonplace in some entire fields of science. I am also aware that the principle of “all the code, all the time” may be very hard to achieve.

    Having said that, the main point of my Nature article is certainly still valid, and applies across science. Software has become an indispensable part of the work of science, and therefore it should be published so that it can improve, in the same cooperative competitive way that has made science such a fabulously successful social enterprise over the last 300 years. Some of your competitors (probably almost none of them, in fact) will read your code. Some will find bugs in it, some will write letters criticising it, some will steal ideas from it and use them in their own research. That is how science works. Excluding source code from the principle is inconsistent and also hinders progress.

    Secondly, I am not advocating that code should be published so that other people can run it. In some cases there is some benefit in doing so, or (somewhat more often) in modifying and adapting it for further scientific research. But the fact that some commenters see this as the main or only purpose of publication is a sign of how poorly software is understood. It is a long-standing tenet of software engineering, practically an axiom, that your most important readers are human beings, not compilers. Science software should be published so that scientists can read it, not so that they (or anyone) can run it. This is one reason why open-source publication, although valuable, is not critical. By all means, if you are so inclined, use a restrictive license when publishing your code, and thus prevent others from modifying or developing it.

    Thirdly, I entirely agree that re-running some code is not replication in the scientific sense. Of course it is not, and I have never argued otherwise.

    Fourthly, and this is where we get specific to climate science: public trust in the science has been seriously damaged. The best response is (even more) openness and transparency. Given that several of the inquiries into the stolen emails made specific recommendations for code publication, any failure to publish code will undoubtedly now be used as ammunition to attack scientists, in the media and otherwise. I want to increase public understanding of the science – that’s the goal of the Foundation – by removing this source of FUD. Nobody who has spent much time in the climate blogosphere can imagine that I am intending to attack the science or scientists: I am trying to help them defuse these attacks.

    Lastly on the question of versioning code and versioning methods, raised in the original post. A given result, dataset, or chart, is produced at a specific time by a specific version of a body of code applied to specific input datasets, invoked in a particular way, on a particular platform with a particular configuration. All of that information needs to be recorded: overall this is configuration management and is a very well-understood area of software engineering, with many tools and a large body of knowledge. Obviously it incorporates version control (the CVS/Subversion/Perforce/git piece of the puzzle), although there is more to it than that. One aspect is that different results may be produced by different configurations (e.g. different versions of the code), and that – for instance – later code versions may produce “better” results (e.g. ones with smaller error bars). But to the extent that the published results are important or valuable, are useful, they should be tied to the particular configuration which produced them.

    I’m away on personal business until Friday 5th, and I’m unlikely to be able to contribute here much in the meantime. David Jones might. Anyone wanting to help the work of the Foundation is welcome to contribute.

    [Response: Nick, Thanks for commenting. No disagreement here, and hope it's clear that we weren't criticizing your Nature piece, which we really liked. We simply wanted to add some of the broader context that you -- understandably -- didn't have space for. I would like to re-emphasize though, that as far as there is a public perception of a 'lack of transparency' in science, it is not helped by simply calling for 'transparency', as if it is something new that has never occurred to any of us before. The fact is, climate science has been *particularly* open compared with most other sciences. Unless calls for transparency make clear we're talking about better methods at transparency (as you are doing) and 'even more transparency' (as you have said), then it sends a misleading signal.--best -- Eric]

    Comment by Nick Barnes — 27 Oct 2010 @ 8:50 AM

  79. From the OP:

    > a stable master archive of code, organised ‘by function’ (not ‘by paper’),
    > that was referenced by specific applications in individual papers

    Sounds great. Don’t tap me for funding, hosting, or code. But I can usually spare an acronym: How about SCROLL = Scientific Code Repository and On-Line Library?

    Under whose auspices should it be set up, to get researchers, institutions and publishers on board?

    And how wide a circle of disciplines and research fields would it make sense to include under one such umbrella? Climate science, Earth sciences, everything under the sun?

    Comment by CM — 27 Oct 2010 @ 9:04 AM

  80. Gavin/Eric,

    I just re-read this (I must have been in quite a fog yesterday). It really is an excellent piece, hitting all of the main points, although it really took a re-read to break out the separate issues more clearly in my mind. The article talks about many disparate things, all of which have to do with code sharing, but are really very separate problems.

    Is there one administrative or regulating body within the realm of climate science (or above that, embracing all sciences) that could actually tackle, structure, and enforce solutions to these disparate issues? I mean something along the lines of forming a committee, making recommendations, and then instituting and enforcing a field-wide policy.

    For example, I could see the establishment of a private, secure repository with the field-wide requirement (perhaps enforced by the journals themselves) that all papers submit their code — but that such a repository is private and particular pieces of code are not accessible to any individual without special permission, so that requests for code (for the purposes of confirmation/replication, or re-use) must go through a review-approval phase, through a regulating body, and the owner of the code is informed of every request.

    Authors could (of their own volition) choose to make their code open/public within that repository, so no review is necessary. The regulating body could also, at some point, decide that some piece of code is open for access without review, either because some reasonable amount of time has passed (sort of like the lapse of a patent), or because it is recognized that there is great value in doing so (i.e. they are getting a lot of valid, approved requests for that piece of code).

    At the same time, this body could implement certain standards for code submitted at various levels. For example, the most basic level is “just give us what you’ve got.” In other cases, they could require an established level of documentation and a defined organization of the code, such as complete separation of I/O from subject logic (I use the term “subject logic” instead of the programming/commercial industry terms “model” or “business logic”, but that’s really what I mean — I just don’t want the computer meaning of the word “model” to be confused with the concept of a program which models climate).

    Another function such a body could perform, if properly funded, would be the actual re-organization and cataloging of code (as requested, by function), and perhaps the value-added process of translating selected functions into a preferred computer language for ease of re-use. One could not really ask the original researcher to do this, but there’s a distinct value in doing so (in some cases).

    Comment by Bob (Sphaerica) — 27 Oct 2010 @ 9:16 AM

  81. 76 (CM),

    How about SCROLL = Scientific Code Repository and On-Line Library?

    I like it! SCROLL. Choosing a cool and oddly, surprisingly appropriate acronym is by far the most important part of any project!!! It’s all downhill from here…

    :)

    That’s actually an intriguing image. In the early middle ages, witches and warlocks might have looked into scrolls for spells, or an alchemist might have looked in ancient scrolls for arcane formulas, hoping to turn lead into gold. In the 21st century, scientists use their almost magical computer devices to access SCROLL, to get the computer programs needed to determine if the emissions from black gold are as dangerous as lead.

    Comment by Bob (Sphaerica) — 27 Oct 2010 @ 9:25 AM

  82. sharper00: if you were an honest advocate of open source, then wouldn’t you have actually read the post by Gavin and Eric at the top of this page, and addressed the points therein – instead of just blindly demanding… what, exactly? “Open code, open code, open code!” – with no thought to reality. So much code is open already, but the peanut gallery remains unsilenced. Do you think making absolutely every line of code would silence them? Of course not! They will go on to demand to see every single email written by climate scientists, and code in progress, and no end of unreasonable things.

    And those tiny handful of people convinced by open code: they are doing it wrong. If someone works away at GISTEMP and discovers that it has no major bugs, and that it really produces the graph that NASA already made – and then claim that global warming is real, and their suspicions unfounded – they are spectacularly missing the point*. Not that such a scenario ever happens. People replicating GISTEMP fall into two camps: those you dream about, who just want to show to the world that there is nothing to hide, and on the other hand there are those searching for problems. When they find none, they just go back to claiming there are problems with the raw data – there is an endless supply of data to nitpick.

    Okay, it is possible that I misunderstood you. There are plenty of people who are hysterical about the subject of open source software, and who have no interest in climate science at all. Maybe you fall into that category?

    If you don’t like being labelled, then say something pertinent about what Gavin has to say, instead of saying things that sound like talking-points.

    Please can we move on from the wrong reasons for opening up code, and talk about the right reasons?

    * Do you know why? Read the article again.

    Comment by Didactylos — 27 Oct 2010 @ 9:41 AM

  83. I like the proposal from Gavin and Eric. It reminds me strongly of efforts by the biological sciences to work together on gene annotation, through huge online databases. But they have just exchanged one problem for another: there are dozens of competing databases, and huge vested interests at work.

    Hopefully a code archive can avoid this fragmented approach, particularly if it limits its scope to climate science.

    I’m generally a little sceptical of current efforts to deify git, and use it instead of any other source control. But git might provide some concrete benefits to a project like this. Also, git tool support has reached a point where it can be used without too much fuss.

    What I would like to see is more organised archival of data. Data suffers from all the change tracking problems that code does, and the solutions should be similar. In addition to that, a few standards for data exchange would save an awful lot of wasted time futzing around with different ad-hoc file formats. Is there any movement in that area that I’m not aware of?

    Comment by Didactylos — 27 Oct 2010 @ 10:00 AM

  84. SecularAnimist, but as I’m sure you know, a bug is very often just an undocumented feature.

    Comment by Rod B — 27 Oct 2010 @ 11:06 AM

  85. I am just curiousif someone can answer some questions that popped into my head.
    As someone who may have to publish my code:
    * Do I have to support it if someone cannot understand/use it?
    * If yes, how long does this support need to be?
    * Will I have to guarantee it works on specific computer systems?
    * What if I use proprietary/commercial software?
    * Do I have to support / provide this software?
    * What if I use proprietary computers / equipment to generate results? (Do I have to provide facilities for you to use them?)
    * Will all code have to be open source?
    * If yes, which ones?
    * Do I have to document it?
    * Should the documentation be in english?
    * What standards do I need to use for coding & documenting?
    * What if those standards change (i.e. do I have to version code/documentation to be forward compatible)?
    * Where should I make the code available?
    * If it is my own website – how do I pay for it?
    * Can I charge for downloading / support / documentation / use?
    * If not, why not?

    Comment by Mak — 27 Oct 2010 @ 11:52 AM

  86. @Didactylos

    “sharper00: if you were an honest advocate of open source, then wouldn’t you have actually read the post by Gavin and Eric at the top of this page, and addressed the points therein”

    I already did and this only indicates you’re not reading what I’m saying carefully. Gavin’s points concern the relevance of open code to scientific progress and I agree with him. However I believe the drive to open up code and data is relevant to acceptance of scientific propositions and less so their progression.

    “And those tiny handful of people convinced by open code: they are doing it wrong.”

    I disagree completely. Their efforts have left no room for the rational to consider it plausible that warming trends are an artifact of adjustment. If your benchmark standard is “Well some people still say it is” then all scientific and educational efforts throughout human history have been a waste since some people still say the Earth is flat.

    “Okay, it is possible that I misunderstood you. “

    How wonderful of you to concede to the possibility that someone who disagrees with you on the openness of source code might be less than a drum beating hysterical bore!

    Comment by sharper00 — 27 Oct 2010 @ 12:02 PM

  87. Mak@85:

    * Do I have to support it if someone cannot understand/use it?
    No.
    * If yes, how long does this support need to be?
    N/A.
    * Will I have to guarantee it works on specific computer systems?
    No.
    * What if I use proprietary/commercial software?
    * Do I have to support / provide this software?
    No. Only publish your own code.
    * What if I use proprietary computers / equipment to generate results? (Do I have to provide facilities for you to use them?)
    No.
    * Will all code have to be open source?
    No.
    * If yes, which ones?
    N/A
    * Do I have to document it?
    No.
    * Should the documentation be in english?
    N/A.
    * What standards do I need to use for coding & documenting?
    It’s up to you.
    * What if those standards change (i.e. do I have to version code/documentation to be forward compatible)?
    No.
    * Where should I make the code available?
    Wherever you like.
    * If it is my own website – how do I pay for it?
    Use Googlecode or sourceforge.
    * Can I charge for downloading / support / documentation / use?
    No.
    * If not, why not?
    Do you charge for people reading your other scientific output?

    Comment by Nick Barnes — 27 Oct 2010 @ 1:17 PM

  88. sharper00 said: “Their efforts have left no room for the rational to consider it plausible that warming trends are an artifact of adjustment.”

    I’m confident that rational people don’t buy into the conspiracy theory that global warming is a massive hoax. But all this is to miss the actual point I was trying to make (and which I footnoted for you). The accuracy of a single study or data product says very little about the robustness of the result. It is not by exact repetition that science moves forward, and similarly such repetition should not sway the mind of any “rational” person. Replication is what we want: evidence that the results are robust despite the method or the source of the data. And that is exactly what we already have, with all the different temperature products from satellites and ground stations analysed in many different ways.

    Let me say it again: if anyone thinks that mere repetition of a method using exactly the same code adds significantly to the credibility of the paper, then they are wrong. I’m not saying such an exercise is valueless – it isn’t. But it doesn’t have the value that you claim.

    “the possibility that someone who disagrees with you on the openness of source code”

    The irony here is that I too think open source code is a good thing. But I’m not an absolutist about it, nor do I view the world through rosy-tinted glasses. Can’t we agree that open source is a good thing in general, then focus on real-world concerns, difficulties and costs?

    Comment by Didactylos — 27 Oct 2010 @ 1:21 PM

  89. Mak asked many questions (but sadly didn’t give any context). Answered inline:

    * Do I have to support it if someone cannot understand/use it?
    No. It’s nice if you can, though.

    * Will I have to guarantee it works on specific computer systems?
    No.

    * What if I use proprietary/commercial software?
    Whether you can make code available when you are not the owner of the code will depend on the license conditions. Generally, consult a lawyer if you are in doubt.

    * Do I have to support / provide this software?
    It might be sufficient simply to point people to where they can obtain the software, and presumably they will receive support from the same place.

    * What if I use proprietary computers / equipment to generate results? (Do I have to provide facilities for you to use them?)
    No.

    * Will all code have to be open source?
    No… but it will need either a license that allows other people to use it, or a declaration putting the code into the public domain. Choosing a license is a difficult issue, and many people will recommend that you choose an open source license.

    * If yes, which ones?
    The code you are releasing (which you own).

    * Do I have to document it?
    Yes! (No. But documentation is really appreciated.)

    * Should the documentation be in english?
    No. However, English documentation will make the code more useful to more people.

    * What standards do I need to use for coding & documenting?
    Whatever you think is best.

    * Where should I make the code available?
    For open source projects, there are many free hosting options.

    * Can I charge for downloading / support / documentation / use?
    If your chosen license allows this, then yes. Otherwise, no.

    Comment by Didactylos — 27 Oct 2010 @ 1:35 PM

  90. Highschooler, citing to actual sources for actual claims is the right approach.
    Often it will help the reader to quote up-to-date information from a cited source.

    For example, one of your links given above reports:

    “… through 2001…. There is not yet any evidence linking FHM ozone bioindicator response data to a specific tree health problem or a regional decline.”

    It’s good to provide information that fails to support your argument. You should do it explicitly, though, not just leaving it for others to search out. Otherwise readers who aren’t cynical enough might have thought your list of links was of links supporting the extreme assertions about ozone.
    Did you look through each of the links you posted for relevant information?l

    No one is arguing that ozone is a problem — one of many problems. People replying here are consistently asking for cites to actual sources of quotable facts, and for commenters not to go beyond those in making claims about what’s known happening.

    It’s way too easy to be cast as an alarmist — and there are plenty of people far out along all spokes of the political wheel who are alarmists.

    Make clear what’s known from citable sources, and what’s your personal fear.

    If you’d looked through that USDA monitoring site, you’d have found more current information — for example

    http://www.forestthreats.org/about/fhm/recent-publications/william-smith/Ozone%20Bioindicator%20Sampling%20and%20Estimation.pdf/view?searchterm=ozone%20bioindicator

    “the ozone biomonitoring data of the USDA Forest Service Forest Inventory and Analysis Program (FIA) are the only source of information available
    that documents plant injury from air pollution using consistent protocols….” followed by a discussion of the monitoring program and ending

    “Periodic recommendations to analysts will be made as results for QA analyses become available. There is also a companion ozone bioindicator user guide (Smith and others, in review) that analysts are encouraged to consult for additional guidance on interpreting ozone biomonitoring data and reporting on the issue of ozone and forest health for the FIA program.”

    So — they’re watching for evidence of the problem.

    Comment by Hank Roberts — 27 Oct 2010 @ 1:46 PM

  91. Bob #81, re: “SCROLL”, glad you liked it. I was thinking along more ancient lines, myself — say, Hypatia uploading a hydrometer design to the library of Alexandria (where my mind often ends up when it goes wandering).
    :)

    Comment by CM — 27 Oct 2010 @ 2:42 PM

  92. @Didactylos

    “I’m confident that rational people don’t buy into the conspiracy theory that global warming is a massive hoax.”

    Generally speaking sure but it’s not necessary to convince people of this, for those of a certain persuasion it’s only necessary to introduce sufficient doubt to get people to deprioritise the issue in favour of other things.

    If someone says to me “I heard on the internet that if you look at the raw data there’s no warming at all!” I can go into a long spiel about peer review and such which still requires people to basically trust a process they have no direct experience or knowledge of or I can say “Well look, all the data, method and source code is open. Here’s 4 different people who reconstructed it independently and they found exactly the same thing”.

    When AR5 hits the shelves I very much want to be able to say the same thing.

    ” But all this is to miss the actual point I was trying to make (and which I footnoted for you). “

    That’s because we’re talking past each other despite my repetition of my position that I’m talking about general acceptance not advancement of the field.

    “if anyone thinks that mere repetition of a method using exactly the same code adds significantly to the credibility of the paper, then they are wrong.”

    Well that person still gets a vote so a determination has to be made as to what constitutes the reasonable concerns of people outside the scientific process who may be asked to vote on policies based out the output of that process.

    I think the replication of scientific work by others will help that person have greater confidence in that work.

    “Can’t we agree that open source is a good thing in general, then focus on real-world concerns, difficulties and costs?”

    We certainly can. I work in the software business so I understand well the difficulties and benefits of open source software.

    Incidentally I’m not being “absolutist”. I understand there’s a balance between opening the code and other things. I also understand Gavin’s points concerning the near irrelevance of open code to making research more scientifically credible however as above I think AR5 will be strengthened if the code involved is open. People may nitpick for sure but by all means let them show their criticisms have a meaningful impact on the results.

    Comment by sharper00 — 27 Oct 2010 @ 2:43 PM

  93. sharper00 said “I understand there’s a balance between opening the code and other things.”

    If you had opened with that, I may have been a little less inclined to bracket you with vboring.

    I still disagree with you, though. Those swayed by “Well look, all the data, method and source code is open. Here’s 4 different people who reconstructed it independently and they found exactly the same thing” are in a tiny minority. The purveyors of doubt have plenty of other weapons in their toolkit.

    Comment by Didactylos — 27 Oct 2010 @ 3:14 PM

  94. Rat Kadbury wrote @ 73

    “Because then C will take her shot and should agree with either A or B. If not, then we can fairly attribute the problem to bugs in the code of all three. Then and only then, they might want to get together and compare lines of code.” I doubt this can work in practice, but even if a resolution comes it would be at great cost. Let me explain.

    First of all in any code project there are specifications (or requirements) that determine in great detail the input and the desired output. It was found (at least in large projects) that many problems came from delays in the projects came in misunderstandings of the specification. For example the specifications should be specific as to the how to handle out of range input or other such events. So besides finding the bugs in the code, there might be misinterpretation of the specifications.
    In the case of scientific programing there may be misinterpretations of the algorithm as well as the specifications.

    If B suspects an error in A’s code, B will probably spend a great deal of time trying to find it. If B finds an bug, bug it may be significant or not significant, and the A’s results stand. But if B finds an bug that is significant then A’s results may be questioned. A does not have to look at B’s attempted reconstruction of A’s code from supplied algorithm. Note B’s program will probably use different subroutines, may use different data structures, and use an expressive but exotic scripting language that is unsuitable for scientific programming and A may not be familiar with that language. Now introduce a third party, C. B will may have to look at A’s code and C’s code, A at B’s and C’s, and C at A and B’s. Assume there are 6 people trying to reproduce the results and 3 get the same results that A does, and 3 do not.

    I do not see the point of withholding the code. It’s written so why not publish it. One could argue that maybe I’ll turn into a project and earn millions. But that just isn’t so. Once A have described the algorithms involved in your code, if B can write codes that reproduces your results, anyone else can.

    There is a saying from software engineering (I think) that said the code is the documentation of the code. I would say the code is also the specification for the code. There is not ambiguity in the code about what the code does. What the code says overrides programs and specifications written on paper. Google has proprietary (secret) code; it’s secret because they do not publish the algorithms. When someone isolates a computer virus, that person can deduce the algorithm and specifications from the code.
    The code (and data for reproduction of results) is everything.

    One more comment. In some large projects the programmers get together and review everyone else’s code.

    BTW, Nick Barnes @78 has a very good post. He makes an different argument for publishing code.

    klee12

    Comment by klee12 — 27 Oct 2010 @ 3:49 PM

  95. klee12 said “First of all in any code project ”

    Scientific code is very rarely like that. This isn’t a case of two implementations of a W3C standard, or commercial software, or anything like that.

    Scientific code varies from the very old (decades old Fortran code that has been built on by many people over many years) to the very young – a quick and dirty script, or code designed to run once.

    And people who claim that the code is all the specification you need just make me sick. They’re just too lazy to create a stable standard to implement, thus blocking anyone else from meaningful interoperation – because code is anything but stable, and when the “standard” changes with each commit, you might as well give up right there. Yes, this situation does arise in the real world – all too frequently.

    All this “the code is everything” talk is nonsense, too. In a scientific discipline less reliant on computing, nobody demands that the *actual test-tube* be archived forever, and accessible to anyone who wants to see it. No, the method is published, and other scientists reproduce and build on it. If problems arise, they discuss it – informally, or through the literature.

    Code also isn’t self-documenting*. If you are writing documentation that mirrors the code in any way, then you are commenting incorrectly. Documentation is supposed to make it easier to understand the code, so that you don’t need to analyse every single part of the whole in detail before you have a clue what anything does.

    There are good reasons for archiving code. Repetition should not be one of them. If there is a significant bug, then anyone using the same code will have the same bug. The benefit from having extra eyes looking for the bug is far outweighed by the potential damage. No, having the code and data archived so that replicators can test against their own implementation – that’s useful.

    * Except in the abstract, completely useless sense.

    Comment by Didactylos — 27 Oct 2010 @ 4:25 PM

  96. The language Ada was supposedly self-documenting. I learned so many languages [about a dozen] prior to 1976, that I have confused them. Mathcad and Mathematica are supposedly just like the human way of doing it, except they aren’t. Some language environments are supposed to do the coding for you so that you can’t make a mistake, supposedly. But that environment only works on a computer I had 25 years ago. Formal methods are supposed to prevent errors as well. Java is supposed to work on all computers except it doesn’t work on an older computer after they update Java.

    All scientists should have masters degrees in computer science, but they do need to start working before they retire. Perhaps 2 or 3 semesters of computer science would be sufficient.

    I don’t think your code would do me any good if I had it. I don’t have your machine and I haven’t been trained to use your code. I have used other people’s code before. It works after you get a private 3 day course with the person who wrote it.

    Maybe there is some compromise that would work, but people keep on inventing new languages and new machines so fast that I doubt that archiving code will work any time soon. You aren’t going to write Macintosh certifiable software for your own research anyway. Your archived code won’t be used by the general public.

    It would be nice if the machine would program itself…….

    Comment by Edward Greisch — 28 Oct 2010 @ 1:20 AM

  97. Didactylos @ 95

    Oops in my post that Didactylos referred to I wrote

    First of all in any code project there are specifications (or requirements) that determine in great detail the input and the desired output.

    which was a mistake due to typing too fast. I should have written that “there should be specifications “. Sorry about that.

    Didactylos wrote

    And people who claim that the code is all the specification you need just make me sick.

    I should have elaborated on that. Specifications are, perhaps implicitly, drawn up before the code is written, for example the one generally decides how to handle erroneous input before rather than after the code is written. Or one might want to specify that all computations or be done in double precision. But often the specifications are not reflected in the code. One cannot claim the code is correct just because it runs and gives results. One has to be sure that the specifications are met. After the code is written the code determines whether the specifications are met. That is what I meant by the code is everything. And programmer B who wants to determine whether or not programmer A’s code satisfies the specifications needs to look at the code.

    Code also isn’t self-documenting*

    The code can differ from the documentation. A programmer writes some well documented code, later makes an enhancement, then forgets to change the documentation to reflect the change. In an earlier post I said when I am having a problem porting programs, all I want in the way of documentation is a statement of what the subroutine does, and the order and data structures of the input parameters and the output parameters. I can read the code. But even then I will check that that calling sequence is correct for the caller and callee. It is not sufficient to read the documentation and say yes that should work. One must look at the code. That is what I meant when I said the code is everything. If programmer B is checking the a program for bugs, B cannot depend on the documentation in the form of comments.

    I think we have lost sight of the point of this little subthread. I asserted that in science reproducibility is important. Publishing the data and code is sufficient for reproducibility. A description of the algorithm, even if complete and lucid, is not sufficient (in general) because (1) there might be errors in the (possibly implicit) specifications and (2) there might be errors in the code, even if the code is well documented. IMHO, detecting these types of errors is very difficult if not impossible without the code.

    Let me point out that even teams of very good and experienced programmers make errors. Every thime Microsoft issues a service pack or a Linux distribution issues a security warning about one if it’s programs, Microsoft or the Linux distribution is saying they’ve discovered at least one serious error in a software release. It is no shame to have write programs with bugs; all programmers do it.

    klee12

    Comment by klee12 — 28 Oct 2010 @ 2:20 AM

  98. Hello! In France today, you should have a look at this tchat about “real” climate http://www.lexpress.fr/actualite/environnement/posez-vos-questions-sur-les-climato-sceptiques_931798.html

    Comment by RealEarth — 28 Oct 2010 @ 3:48 AM

  99. P Lewis:
    “Indeed The Ville, if you follow your link and then the Git software link you get Linus Torvalds reportedly saying “I’m an egotistical [sic] bastard, and I name all my projects after myself. First Linux, now git.””

    heh, heh.

    So us Brits still have an influence on the world. :-)

    Comment by The Ville — 28 Oct 2010 @ 6:32 AM

  100. The best solution is the develop and describe the algorithm and the code together. An example is the non-trivial problem of computing potential magnetic fields in the solar corona from line-of-sight measurements http://www.leif.org/research/Calculation of Spherical Harmonics.pdf
    This works when the code is small and can be understood. How one deals with code in the millions of lines is a different problem. Perhaps the real problem is that code should not be allowed to grow into such monsters.

    Comment by Leif Svalgaard — 28 Oct 2010 @ 7:35 AM

  101. http://www.leif.org/research/Calculation%20of%20Spherical%20Harmonics.pdf

    Comment by Leif Svalgaard — 28 Oct 2010 @ 7:38 AM

  102. 100 (Leif),

    How one deals with code in the millions of lines is a different problem. Perhaps the real problem is that code should not be allowed to grow into such monsters.

    Which is to say… don’t bother to use computers to solve anything except trivial problems?

    Comment by Bob (Sphaerica) — 28 Oct 2010 @ 9:39 AM

  103. No, klee12: you still have it backwards.

    Publishing the data and code is sufficient for repetition. It tells you absolutely nothing about the intent of the code, or whether the code does what is intended.

    And your description of “all I want in the way of documentation is a statement of what the subroutine does, and the order and data structures of the input parameters and the output parameters” sounds a lot like undocumentation. Yes, stale documentation is also a concern. But if the specification document is normative, then the code doesn’t matter. Given any implementation, you can tell whether it is correct – whether by examining the code, or testing the results. (And “specification documents” really are pretty normative, since that is the part that gets peer reviewed and published.)

    Publishing the method, the intent, the specification – that is the vital part. The code is just gravy.

    “often the specifications are not reflected in the code. One cannot claim the code is correct just because it runs and gives results. One has to be sure that the specifications are met.”

    You are so very close to being right with this. But then you ruin it by saying:

    “programmer B who wants to determine whether or not programmer A’s code satisfies the specifications needs to look at the code”

    …and you’re back to being wrong. 180 degrees wrong. The code without the specifications tell you nothing. The specifications without the code can tell you enough, because you also have the results of the code. You can determine from the output whether the code meets the specification. And most importantly, by replicating those results by applying the method from first principles yourself, you provide very strong evidence that the method is correct, and not an artefact of buggy code.

    Please be aware that it isn’t often code that future scientists will rely on when they build on someone’s work. It is the methods, the published algorithms. So for useful replication, it is the reliability of the methods that must be tested, not just the repeatability of a single implementation.

    Also, consider this: code review processes usually find something like 65% of bugs. Now, fixing 65% of bugs is a useful thing to do, but it doesn’t say anything about whether the output is correct. If there is a significant problem in that elusive 35%, then you are stuck. But if the code does what it should under the prevailing conditions, then we can be confident that the undetected bugs either a) have such a small effect on the result as to be negligible, or b) only manifest under conditions that are currently out of range.

    [Response: Italicized emphasis mine --eric]

    Comment by Didactylos — 28 Oct 2010 @ 10:34 AM

  104. 103
    Didactylos @103

    I’m afraid we’re going to have to agree to disagree

    klee12

    Comment by klee12 — 28 Oct 2010 @ 12:29 PM

  105. Many years ago I majored in Computer Science at Dartmouth and wrote a lot of code. About a decade later I was cleaning up my garage and found printouts of my own work. Now, I was an undergraduate at the time and went to law school in the interim, so a certain lack of skill and loss of skill might be expected. But I was completely incapable of understanding my own raw code. The only way I could figure out what I was trying to do was to read my own comments. So let me add my voice to those who argue that the raw code is much less important than the specification which describes what the programmer was trying to do.

    Comment by Francis — 28 Oct 2010 @ 1:54 PM

  106. klee12 said “I’m afraid we’re going to have to agree to disagree”

    It would be far more valuable if you examined your assumptions and worked out why I and the RC scientists disagree with you.

    Our opinions aren’t so far apart. We all want clear specifications and full archival of code and data. I’m not confident you have fully grasped the difference between repetition and replication, and that is why your priorities are different.

    Comment by Didactylos — 28 Oct 2010 @ 2:42 PM

  107. As with Francis, I also have had trouble deciphering my own code, at least when the project was somewhat complex. Early on, I learned the necessity of providing adequate comments, but even with those, it was usually quite difficult to work out just what I was doing. In many cases, it might have been easier to rewrite the program from scratch.

    On the basis of such experience, it seems to me to make more sense to describe the approach and algorithms that are being used and to have others write their own programs to see if they come up with the same results from the same inputs. Robustness under that kind of test would to me to be much more convincing that checking through enormously many lines of code for mistakes.

    Comment by Leonard Evens — 28 Oct 2010 @ 4:29 PM

  108. Francis @105 wrote

    So let me add my voice to those who argue that the raw code is much less important than the specification which describes what the programmer was trying to do.

    Oops, I didn’t mean to say that documentation in the form of comments are bad. I meant, but did not express well, that for me they are not that very important. Commenting can be time consuming and I would much rather be programming. I just wanted to encourage people here not to feel they must go through the trouble of extensive documentation of code before publishing it.

    Let me explain why documentation is not important for me. Suppose I get a program where I suspect there is a bug. I assume I know something about what the program is supposed to do, i.e. solve partial differential equations rather than render graphics.

    I would not start reading code. I would look at the main program and look at the subroutines in the top level. Then

    1. I would try to compile the program with all optimizations turned off, and warning messages or debugging mode turned on. There is a saying in scientific computing “Do you want it to run fast or run correctly. Pick one”. In debugging mode, a compiler can insert code for runtime checks that references to array elements are within the bounds of the declared arrays. Other runtime checks may be inserted. Optimizations can do strange things that I won’t go into now. If I get a good compilation, I read the warning messages. If not I fix it up.

    2. I would eyeball the main program and look at the major subroutines called. I try to figure out what they do by looking at parameters going into the subroutines a the output parameters. I might insert my own debugging statements into the code. I might build my on test data and run the program and try to verify what is going on. The goal is to try to find the most like place there is an error.

    3. If I want to examine a subroutine more carefully and if it contains more subroutines I might repeat step 2 on the subroutine. Eventually I will want to examine a few subroutines. I might break them out of the code and build my own test harness (program) to run the subroutine. It is easier to vary the parameters in the harness than in the main program.

    4. At this point I read look at the code. I examine calling sequences, undeclared variables, global variables, and other things. Then if there is documentation I now read the documentation. I note which data structures are 32 bit and which are 64 bit. If there are specifications, I check to see if they are followed. After all that I read the code, I insert debugging statements, and run the test harness.

    5. If I still can’t figure out what the code is doing I use a debugger to to single step through the code. The debugger can execute a single line of code and then wait for a command. Commands may be to (1) continue to execute the next line of code (2) print the value of any variable (3) set a breakpoint so program runs until that line is reached. There are many other commands.

    That’s what I would do. Different programmers may use different approaches. Every program is different and this approach might not work for all programs. Sometimes I get things under control in a day, sometimes in a couple of weeks. But that’s a problem for the programmer checking the code, the the author.

    klee12

    Comment by klee12 — 28 Oct 2010 @ 5:42 PM

  109. Seems to Eli there are two cases here which are being inconveniently mixed. The first is one off, simple stuff, used once in some analysis and interred. There is absolutely no need to go beyond publishing a description of this.

    The second is production code (GISTEMP, RSS, UAH and the Community Climate Code, etc, which (or at least their cores) are constantly used by many or whose outputs are used by many. The code for those should be publicly available.

    One could make an argument for the first class of programs that they should be written in as high a level language as possible, MATHEMATICA and the like.

    Comment by Eli Rabett — 28 Oct 2010 @ 5:42 PM

  110. Once code gets “out there” it can take on a life of its own.

    I wrote a useful little program, just for me, for a very specific set of conditions. Of course I didn’t document it, no input controls no pretties. Coworkers liked it and used also, it spread.

    Later I ran into a pretty version. Overall a nasty program.

    Yes my very simple little program was at the heart of the nasty one. It was no longer obvious that it only catered for a specific set of circumstances and gave garbage for others.

    In truth I put in less effort on the original program than someone else did creating input controls, but I only made it for me.

    So yes I can see problems in open sourcing all code.

    Comment by Tony O'Brien — 28 Oct 2010 @ 6:21 PM

  111. Eric and Gavin,

    Thanks for bringing this out. It is amazing that scientist have to do extra work, with no further pay to satisfy those who just don’t believe the facts. To bad in the political game donations do not have to be accountable. Maybe a comment not appropriate in this discussion but one to ponder.
    Much appreciation to the article and discussion.

    Comment by Mark — 28 Oct 2010 @ 6:33 PM

  112. klee12: that’s a great approach if you just want to fix the bugs. But if the question is “does the software meet the client’s operational needs” – then maybe you would try a different approach.

    Oh, and here’s a secret: nearly all programmers dislike writing comments. (We try not to let anyone know this, though.) After a few experiences like Francis and Leonard’s, you too will really wish your comments were better. You might even start writing better comments. It’s still unlikely you will enjoy it. If you think it’s easier to interpret the code – wait until you have had to reinterpret the same code a few times; you will wish you had added some comments the first time you worked everything out. Either that, or you’re some kind of masochist ;-)

    Comment by Didactylos — 28 Oct 2010 @ 7:35 PM

  113. Now yer talkin’ my kind of language…

    I was originally a computer designer, and now a professional programmer for the past 30 years. I realise that I’m mainly echoing the points raised in this excellent article, and by the likes of Damien in comment #4, and Bob in comment #11. But they are points that need to be constantly reinforced nonetheless:

    1. The algorithm is the important thing, not the code
    In a published paper, I want to see the basic algorithms/methods you used along with any important constants and assumptions. Then I can develop my own code and run my own data (coming to that in a second) through it to see if I get similar results.

    2. GIGO (Garbage In, Garbage Out)
    It is much more important that many independent, reliable, data sets of a given type are gathered, than that everyone is trying to run their own home-brewed algorithm on a single data set to see how far they can push the limits of credibility :-)

    So I agree that an open source library of climate-related functions that are difficult to implement would be handy and time-saving, just like any high level language comes with a built-in library. But asking for entire coded bases to be open-sourced? Not useful, IMO.

    ETA: in fact, pseudo-code might be even more useful. Then someone could implement each function in any computer language.

    Comment by Steve Metzler — 28 Oct 2010 @ 7:42 PM

  114. klee12,
    I’m going to go out on a limb here and guess that you haven’t written any code for scientific computing. Specifications? You’re lucky if you get a well documented dataset. And the majority of the time, the issues you are looking for are not “bugs”. A decent scientific programmer or his working group will have found most of the significant bugs before the results are published. More often than not, a “bug” means the code either doesn’t work or yields results that are not consistent with known science. When was the last time you heard of a result being retracted because of a bug in the code?

    And I beg your pardon, but a third researcher trying to reproduce the results is precisely what would happen if you had two other researchers who disagreed. And a fourth, fifth and sixth for that matter. If a result significantly advances understanding of a field of study, you will have every other researcher in the field trying to reproduce the results and to see if it also advances their project.

    And why not release the code:
    1)it does not advance the science
    2)most scientific code is used within small groups and may not be sufficiently well documented for outsiders to use without wandering outside the validity of the models being used.
    3)if a researcher can’t reproduce the analysis from a description, then either the description was inadequate or the analyst trying to reproduce it is not sufficiently skilled to be mucking about in the field anyway.
    4)independent replication is more likely to lead to advances than simple repetition.

    I have found that it is possible to get access to code via certain advanced techniques–namely asking nicely. The advantage of this is that it usually also comes with help from the author of the code.

    Comment by Ray Ladbury — 28 Oct 2010 @ 7:51 PM

  115. Tony @110 had a good and easily overlooked point. The broader and less specialized the user community for a code, the greater is the need for investment in user-friendliness. Otherwise naive users can easily violate the logic and capabilities of the code, often getting nonsense results, or reporting bugs. So you can either make a large investment in bells whistles, user hand-holding, sophisticated algorithms to determine whether the solution is converging correctly etc, etc. This sort of coding can often cost more than the cost of the original code, and is not particularly enjoyable. So the temptation will be to just make the source code available on as asis basis. But, then you run a reputational risk when naive users get into trouble, and blame you for writing low quality code. So there is a danger that a requirement for openness which is motivated by a need for greater public confidence can end of having the exact opposite result.

    A good example of this, just happened to me with CAPTCHA. After typing in the comment, I hit SAYIT, without filling in the CAPTCHA, and had to completely retype this message (because of poor quality software).

    Comment by Thomas — 28 Oct 2010 @ 8:59 PM

  116. A quicky from the peanut gallery. I’m in full agreement with most of the commenters and the thrust of the article. Ray L.’s points in 114 sums up the rational pretty well (as have many others). Distributing raw code seems a colossal waste of time with the probability of 0.1% of anything productive coming of it. Though there probably ought to be a mechanism for a very select list of people getting a copy in some circumstances.

    The documentation, specs, algorithms ought to be freely available (sometimes under non-disclosures) to aid the perception and reality of openness in this critical highly-charged area. Though I recognize the significant dilemma and aggravation this can be sometimes.

    btw, I once knew a guy in old Big Blue who could debug (once in a while) even massive programs by reading machine code. Don’t know if this is uncommon, but as a neophyte at the time I was greatly impressed!

    Comment by Rod B — 28 Oct 2010 @ 9:02 PM

  117. I think a useful analogy would be the world of computer security.

    Encryption algorithms and security protocols have to be rigorously described and all assumptions documented. The algorithms involve highly complex and advanced mathematical theory. The protocols have to be designed around established cryptographic principals. Successfully writing code to implement the algorithms and protocols requires a significant understanding of cryptography. The algorithms and supporting reasoning are generally subject to public review (that is, by folks that could actually follow the reasoning).

    Code to implement the encryption algorithms often is freely available under open source licensing (for example, openssl, openssh). But organizations and individuals often write their own implementations for their own purposes. The code for encryption is often quite “simple” from a software engineering perspective. The main software engineering problem is performance – so reducing overhead with fewer function calls and extraneous variable checking is critical. This requires a higher degree of software engineering skill, talent and experience to avoid fubars and subtle defects that can be exploited by a hacker. Such code needs a high degree of review to avoid subtle defects. Only experts bother to review the code for obvious reasons. Again for those reasons, most folks use the free versions mainly because they are the most widely used, best reviewed and hence most trusted.

    The analogy being that climate model code is really only comprehensible if you understand the physical model that the code implements. The difference here is that the implementation is being done largely by smart climate scientists and not software engineers. All that means though is that the code has much less gold plate, is probably not quite bulletproof (like say a temperature data set with an unexpected control character might crash), is probably not written in a self-documenting fashion using variable naming conventions and style to maximize readability, is potentially inefficient [although fortran is generally regarded as very efficient] and is not taking advantage of the latest advances in software engineering theory. [NOTE: I've read through the some of the climate model code that is available so I'm not just guessing wildly here.]

    Could subtle bugs produce lousy results? Sure. But the vast majority of bugs, even the subtle ones, produce obviously wrong results. A bug in a decryption or encryption algorithm doesn’t change “Tom” to “Bob”, it changes “Tom” to “4#~”. When the climate model produces obviously wrong results, was it an error in the algorithm or the code or the data set or the computer configuration? Poorly written code can make that discovery process more time consuming than necessary and hence reducing the amount of time spent on climate science that could help us make sound policy decisions.

    This is where well-written, reviewed code is beneficial. A coder can quickly read the code and say “The only reason you could get that result is that the algorithm for pressure gradient was implemented wrong.” or “The temperature data set must be corrupt for Florida.”

    Documentation of algorithms, file formats, and interfaces are absolutely critical and need to be well-written and error free.

    But code “documentation” is largely useless or dangerous since it is rarely correct. Comments are particularly dangerous. The code needs to speak for itself.

    Seriously, I personally wrote, maintain and expand a codebase of 1+ million lines of relatively complex C++ and manage a team of Java developers with a codebase of 1.5+ million lines of code (by comparison Windows OS is reputed to be 40+ million lines). It really just comes down to the discipline of writing readable code. No amount of “documentation” will make poorly written code comprehensible. The time spent writing “documentation” would have been better spent making the code more readable.

    Bottom line, it seems to me that “archiving” climate modeling code should be done to help the climate scientists produce better code not for political purposes to satisfy demands for “transparency”.

    [Response: Thanks for you level headed comments and interesting perspective as software expert. I agree that in general, decisions ought to be made that best serve the science, not to satisfy political winds. I think Nick Barnes would reply that his project is serving both purposes -- hence ClearClimateCode.org and Open Climate Code.--eric]

    Comment by Robin D Johnson — 29 Oct 2010 @ 12:54 AM

  118. It can’t be emphasized too much that the code is not the world. The code in GCMs attempts to mimic a physical process, but as near as I can tell, it’s never the code of the GCMs that the naysayers and deniers have fits about. The code they have fits about are statistical routines: the code which attempts to find signal in noise. The naysayers never seem to want to kvetch about code that mimics a physical process. No, they moan that certain statistical procedures weren’t done to their satisfaction.

    Comment by Jeffrey Davis — 29 Oct 2010 @ 9:25 AM

  119. While I agree that papers published describing coded implementation of mathematical models can be tedious and may not be a good presentation of the model they do serve to announce the existence of an implementation and some of the thinking behind it. I also agree that new and different coded and tested approaches to the same or similar problem and data provides another view for comparison, but if an implementation produces particularly useful data then it is necessary to look closely at the code and ideas that produced said data. If a coded solution does produce an answer as expected it is possible in the name of tuning or calibrating to paint the hood rather than fixing the engine. The result looks great, but it doesn’t really run. How would anyone know that if the code and documentation is not refereed?

    Comment by RiparianJohn — 29 Oct 2010 @ 1:37 PM

  120. Ray Ladbury @114 wrote

    I’m going to go out on a limb here and guess that you haven’t written any code for scientific computing.

    Well you decide. I worked several years as a senior scientists at NASA’s Ames Research Center at Moffett Field, California. I worked with a division involved with computational fluid dynamics (CFD) but I was attached to a section that was involved with benchmarking and alogrithm development. The algorithms had to do with taking advantage of the architectures of the machines we had (Crays and early parallel processing). I worked with the guys who really developed code and I sometimes I was asked to help make their code run faster. In my benchmarking I did write some code to investigated some algorithms but I didn’t write CFD code. At that time I could read Fortran like a newspaper. I later worked with C, but never got to that level of expertise as I did in Fortran because C was too close to the machine level … more tricky things could happen than in Fortran. I had an interest in software engineering.

    The work at Ames may have been defense related. I needed security clearance. The codes, AFAIK, was not published. But it was not important for science even if it was published. It was engineering, it did not add to our scientific knowledge. It was successful if it could produce a plane that could fly faster and/or more efficiently or whatever.

    Before I go on, let me say where I’m coming from. On this thread, there are many subthreads. Nick Barnes started a good on @comment 78. He gave gave reasons to publish code. I’m suggesting another reason, name science requires that results can be checked, and science should publish their code since that is (IMHO) you essentially cannot check the results without the code. In particular it is difficult to resolve differences if someone, attempting to reproduce the results, gets a different result. Who is correct? I am arguing that giving a description of an algorithm may not, in software engineering terms, be an adequate set of specifications and that even if the code implements the algorithms, bugs may occur in the code.

    I also point that it does not take much effort publish the code and the data. No documentation, no support; just the whatever is necessary to produce the result. See post 85 by Nick Barnes.

    Now back to Ray Ladbury@114.

    Specifications? You’re lucky if you get a well documented dataset. Yes, I knew that and I think I wrote implicit specifications, which I left undefined. Generally description of methods do not specify the precision of the computation or the handling of run time errors. However these issues must be handled in the code so I invented a classification of errors with the name implicit specification error. I was trying to point out why two different implementation, both without coding bugs, may give two different results. By the way small round off errors can lead to large errors when inverting ill conditioned matrices. That’s why they invented double precision.

    When was the last time you heard of a result being retracted because of a bug in the code?
    I won’t define bugs but I know it when I see it. When Microsoft issues a service pack, they have found bugs. When an open source software issues a security warning to update a package, they have found a bug. If a teams of professional programs highly motivated to produce bug free programs still write code that is buggy, why do you think that you can write a rather large bug free programs? I certainly don’t think I can. There’s an saying from software engineering that says “there are bugs in every large program, you just haven’t found them yet”.

    klee12

    Comment by klee12 — 29 Oct 2010 @ 2:23 PM

  121. A lot of naysayers I’ve dealt with seem kind of hazy on the idea that there’s anything *besides* statistics in GCMs.

    Comment by Kevin McKinney — 29 Oct 2010 @ 2:59 PM

  122. A quick nod to Ray Ladbury #114, Rod B #116, Robin D Johnson #117, you guys (and others) have hit it well.

    So many times I’ve come into an old piece of code, read the comments, and discovered that the comments had little to do with the code anymore.

    I once knew a guy who was called into a problem in the software, looked at the output, and declared something to the effect that the only way this problem could happen was if register X of processor 13 of 16 was faulty. Awe ensued. It wasn’t even a problem in the millions of lines of multi-threaded code; it was a problem in the hardware.

    Sometimes readability and performance are at odds with each other. I once took a very simple function than was maybe 10 lines of code and rewrote it in to something 3 times longer that even my teammates looked at and said,
    “Why the heck did you do this?”
    “It now runs 6 times faster, and it is used a lot.”

    Just seconding that looking at code without a high degree of expertise in both code and the domain the code was written for is generally a waste of time, not only for the viewer, but also for all the people they ask for help.

    Modularizing, eliminating redundancy, and imposing version control over a pre-existing, disparate library of climate code would be a daunting task. My hat is off to David Jones and company.

    Lawyers at my company would come into my cube, rip off my arms, and beat me with them if I released code into the public domain. Then security would come and escort me to the door. Maybe I should hope that security gets there first.

    Comment by Chris G — 29 Oct 2010 @ 3:25 PM

  123. I know it is not the right thread, but that piece on beetles and forests below this one is marvelous. It’s the kind of thing that blogs do so well….

    Comment by Michael Turton — 29 Oct 2010 @ 8:12 PM

  124. klee12, you are still missing the point. Ray never implied that the code is bug-free, just that any bugs don’t affect the result.

    I believe it is relatively common for minor bugs to be found during replication. The number of times they have resulted in a retraction? Very small.

    Since this is exactly the point I made in a reply to you way back when, it seems you aren’t really taking in what is being said.

    “I am arguing that giving a description of an algorithm may not, in software engineering terms, be an adequate set of specifications”

    Then that is a flaw in the description. It’s not something that causes everything to collapse. The description gets fixed, we move on. And yes, this is a real situation that occurs during replication. Nobody is arguing that the “specification” is perfect – just that it is central. Nobody is arguing that any code is bug-free. Any non-trivial code inevitably contains bugs.

    This would be a lot easier if you didn’t argue against straw men.

    Comment by Didactylos — 29 Oct 2010 @ 9:13 PM

  125. h/t to Robin w/respect to writing clear code.

    I worked on the Unix kernel and the NT kernel for about 15 years and some fairly complex applications (virtual filesystems and ill designed application code) for the last 10 years. None had much in the way of any comments, but all were reasonably clearly written. I’ve been writing code for very complex systems for a long time. Comments are, well, worthless.

    Pretty much the only time I write comments is to explain where I do something “tricky” (oh, that’s a bad word) or to make clear assumptions on internal routines. Sometimes this is done through asserts (to catch things I never expect) or comments to indicate the conditions under which I expect a routine to be called, although it will work in those circumstances in which it is not appropriate to call.

    The other situation in which I will write comments is to make clear the assumptions which were valid at the time the code was written indicating where code will have to be changed if the assumptions change. The vast majority of the code I write is uncommented, but I do try and make it readable. This approach is the best that you can do.

    I’ve been working on complex systems which are mostly written by others for years — porting operating systems to new architectures for a long, long time. Readable code is your best friend. A good debugger is your next best friend. For the sort of stuff I am best at (OS work), being able to look at the dump provided by a logic analyzer is the next best thing (a good ICE which can read the symbolic output of your compiler or assembler is better). Lots of times you have to be able to read the output of your assembler or compiler and compare it with the LA dump to see what is wrong.

    However, you can do good clear programming even in assembler (however, it does take lots of comments especially for Intel architectures where you have a shortage of registers, an exception). I remember getting some questions I got on some cache control code I wrote for an adaptation of the MIPS R4600 to support a secondary cache. “Why don’t you follow the conventional calling conventions?”. I tried very hard to not pollute cache lines in the SC. Once I explained the reason (not polluting lines while trying to ascertain which lines were to be cleaned) the questions went away. This was fun code to write, because I used every trick in the book to speed it up. It was ugly code, but it worked. Have fun all you “programmers” out there pontificating.

    Notice I said nothing about comments. Currently I am doing server side web site programming and I comment much more, but only because I am making the assumption that people who might follow in my footsteps don’t know WTF they are doing and will take an approach to debugging like klee12. He obviously has never done Real Programming(tm).

    One of the things that I was proudest of, and which USL (Mashey knows who they are) did not accept was an interface which did auto configuration of the kernel at runtime for the System V ESMP kernel for the type of processor it was running on. This involved detection and loading at runtime of the proper driver for the MP architecture it was being run on.

    This involved abstract classes (written in C and assembler!) which made it very easy for users to install on a machine, and if the disk was moved to a new machine, it still worked — as long as it was an Intel machine, no way to tell which version of locore should be loaded. I realize now that I should have gotten a patent on that one, because we could have sued Microsoft a few years later, but didn’t realize that I had something there (too young to realize I had a good idea there).

    As has been pointed out in other posts what I like to call “pessimizing” should not be done (it makes the code less readable) because modern compilers are pretty good at optimizing and code movement which makes stuff faster. Writing fast code should wait until a routine is identified as a sinner and not prior to that.

    Writing real code is far removed from something like the Matlab scripts which Mike Mann currently releases with his papers (and he has been doing it for a long time). Two of the three most important climate models (CCSM and GISS) are available, in source, on the interwebs. Scripts, such as Mann and others release, are trivial and are well described in the papers. I don’t understand why this stuff should be subject to the same rigorous software engineering standards that so many of the people calling for “openness” seem to think that they should be subjected to. Oh, I’m sorry, Matlab isn’t open. Well, if you can’t afford a Matlab license (and there are open Matlab interpreters out there) get lost. You just aren’t serious.

    Comment by Rattus Norvegicus — 29 Oct 2010 @ 10:14 PM

  126. I find some of the discussion about code comments to be quite extraordinarily scary. I strongly suspect that many who claim that they don’t need comments are (at least in part) using it as an excuse for their own poor commenting practices.

    Now, what you do in the privacy of your own home is your business. If nobody else will see your code, then do what you like. The only person who will be puzzling over your code later is yourself.

    But for shared code, comments are important. If you think otherwise, then perhaps you don’t know how to write good, useful comments yourself, and it’s a fact that good, useful comments are rarely to be found in any code, open source or commercial.

    So for the love of all that is holy, please stop using the existence of useless comments as an excuse for perpetuating the problem. Write comments that will actually save you time, save your colleagues time, and add to the value of the code. Comments that say things that are *not* immediately apparent from the code itself.

    I’m not holding myself up as an example. My comments range from good to bad to awful.

    Rattus Norvegicus: We all admire Real Programmers, and we all pray we will never, ever have to wade through the undocumented monstrosity they pretend is code. Job security is nice, but….

    Comment by Didactylos — 30 Oct 2010 @ 8:54 AM

  127. Didactylos @124 wrote

    I believe it is relatively common for minor bugs to be found during replication. The number of times they have resulted in a retraction? Very small.

    The probability of a significant bug may be small, but the should still be exposed. I suspect that the probability of significant bug is at least as large as that of large applications released to the public and some of those applications have reported security bugs.

    Rattus Norvegicus @125 wrote

    people who might follow in my footsteps don’t know WTF they are doing and will take an approach to debugging like klee12.

    In my post @108 I wrote

    Let me explain why documentation is not important for me.

    I think it was understood that I was speaking of scientific code. I wrote not only to show why documentation is not important but offer an to scientists an way to approach porting and understanding code.

    Readable code is your best friend. A good debugger is your next best friend. For the sort of stuff I am best at (OS work), being able to look at the dump provided by a logic analyzer is the next best thing (a good ICE which can read the symbolic output of your compiler or assembler is better).

    I’m not sure this is relevant. Of course readable code is your best friend but this is something you have no control over when trying to understand codes written by other programmers. Before I use the debugger I try to narrow down the area of code that I might have problems with. I do this with print statements, looking at structure of code. I don’t think dumps and ICE are relevant or useful to the average scientific programmer.

    Currently I am doing server side web site programming and I comment much more, but only because I am making the assumption that people who might follow in my footsteps don’t know WTF they are doing and will take an approach to debugging like klee12.

    I have used my approach and found it successful. What approach would you take with large scientific programs?

    He obviously has never done Real Programming(tm).

    Not sure what you consider to be Real Programming, but its not relevant to my posts here.

    klee12

    Comment by klee12 — 30 Oct 2010 @ 1:52 PM

  128. Didactylos,

    I think we are in violent agreement. Code should be clear. Comments should make things which are not apparent from the code clear (like I said, tricks, assumptions, conditions in which the code should be called…). Redundant comments are pretty useless and can be worse than useless if poorly written or not maintained, and I’ve seen my share (and probably written some) of those too.

    Comment by Rattus Norvegicus — 30 Oct 2010 @ 2:21 PM

  129. klee12, I think security bugs are a different class of bugs than normal software bugs. I would best that most security bugs, are bugs of omission, not considering the possibility of some mode of attack. Security versus hackers is more like an arms race, the software developer has to outsmart the attackers. Normal scientific software is designed ofr “friendly” users. Now I don’t for a minute believe most scientific software is bug free, a lot of bugs cause too small an effect of the most common solutions to me noticed. Often it is when some input tries to use the code in an unanticipated way, that they are revealed. Also there are a lot of bugs which result in using the wrong data somewhere in the code. Usually as long as the bogus values are innocuous enough, the small errors in the final solution are not noticed. But, a fresh compile may scramble the order of memory usage, and suddenly the code crashes on input it used to appear to process fine. Over time, more and more such bugs are flushed out. Ideally we would have really good tools available to flush out such things, but in practice that is rarely doable.

    Comment by Thomas — 30 Oct 2010 @ 5:54 PM

  130. Rattus Norvegicus: your code doesn’t look like line noise? I’m beginning to doubt your Real Programmer credentials.

    klee12: I take it you’re not familiar with the story of Mel?

    Comment by Didactylos — 30 Oct 2010 @ 7:54 PM

  131. Guys, you’re way OT here. Yeah, I know it’s a natural thing to want to expound on your field of expertise when given the slightest opportunity to do so. My natural inclination as well.

    But please, this level of discussion belongs on slashdot. Or go read the latest comments on TDWTF if you want to bitch about the subtleties of code. There be primarily climatologists here, and code is not even secondary to what their core concerns are. It’s arguably not even tertiary. Which is what the original article is all about, actually.

    Next, someone’s going to launch a diatribe about how bad Java is. And I just couldn’t take that.

    /concern troll

    Comment by Steve Metzler — 30 Oct 2010 @ 9:19 PM

  132. Re #126

    I stand by my statement that comments are useless and dangerous. All code is dangerous to change without understanding the entire context. Modifying non-trivial code is essentially equivalent to writing that code from scratch plus understanding all the code that depends on the code that will be changed. This is why most large software projects are done using an object oriented language like C++ or Java or C#. The ability to extend, encapsulate and handle exceptions is critical to the long term health of large codebases. This is a well-worn area of software engineering.

    Comment by Robin D Johnson — 30 Oct 2010 @ 10:40 PM

  133. No it doesn’t look like line noise, but like Mel, I started out writing in machine code. The first programming class I ever took was in high school and involved the use of a Wang “desktop” programmable calculator. They called it a desktop because it was, well, the size of a desktop. We had an optical card reader and they way you wrote programs for this thing was to use a #2 pencil to write the 8-bit opcodes in binary. Had a bug? Dump the program you loaded and look at the assembly output, then examine the logic, find the error and go back and write a new card, or sometimes the remainder of the deck to fix the bug, because erasing the opcode and replacing it with the correct one often caused problems for the card reader. I was pretty proud of the fact that I could get the thing to play tic-tac-toe with some alacrity. Who needs an assembler! This was in the early 70′s and, thankfully, things have gotten better.

    I have written code which needed an oscilloscope to debug (it supplied a software generated synthetic clock signal to a chip which couldn’t use the bus clock) and in the mid to late 90′s that counted as real programming. Having to use an LA to debug code during the bring up of a new computer also counts, in my mind at least since you had to read hex, know the opcodes and understand what the contents of memory should be. I do have to say that once the last bug was out of the system and you saw “login:” on the screen, typed “root”, then “password” and hit return and got a “#” prompt gave you a feeling like nothing I have ever experienced in any other type of programming. It was fun!

    I have also written lots of code in assembler for the MIPS chips which made heavy use of the delay slot to load something useful for the target instruction, a feature of the chip that the compilers usually ignored at that time. Of course, almost every line of these routines was commented and subroutine block comments contained the pseudo code (c’ish in this case) which laid out the logic of the routine. God knows it is hard enough to read your own assembly, much less someone else’s.

    So no, in the sense of Mel, I am not a real programmer. I have never written self modifying code or code which was optimized for the spin of a drum. I would hope that I never have to write code like that, although I understand that writing for the Alpha might have entailed a more modern version of such nightmares.

    But this is all neither here nor there with respect to the subject of the post. The most complex climate code seems to be reasonably well written. Steve Easterbrook has done a lot of work on this. What I fail to see is the need to publish scripts which hold to the highest standards of software engineering practice when most of the code you use every day honors these standards in the breach. a couple of hundred lines of script just aren’t a big thing, and if you can’t read and understand them with under a day of work, I have to wonder why you are asking to see the code in the first place, because you Just. Can’t. Read. Code. and having the code won’t help you understand the paper one whit.

    So while publishing the code is good or at least good publicity, it is of limited scientific value. And while I sympathize with the idea of an open repository of library code, for the most part this isn’t necessary given that most statistical analysis code is written in Matlab or R these days so this doesn’t seem to be really necessary. Finally, the Nature news article linked in this post contains a cautionary tale about sharing code. The word “retraction” comes to mind.

    Comment by Rattus Norvegicus — 30 Oct 2010 @ 11:14 PM

  134. Steve: you are quite right.

    Robin: you managed to be absolutely wrong and absolutely right in the same comment. Well done.

    Rattus: I worship at the feet of Dijkstra, and quiche tastes wonderful when made right. But assembly is important too, and I have little respect for programmers with no clue what goes on under the hood.

    I got the impression that any “open repository” would have a much larger scope than just bits and bobs of script. I got the impression from Gavin that the primary purpose would be archival – tagged versions that can be referenced in papers with zero ambiguity. Merely having everything in one place can be an enormous help – look at CPAN.

    Our dear hosts: is this OT for Unforced Variations, too, or should we just stop now?

    Comment by Didactylos — 31 Oct 2010 @ 10:46 AM

  135. Robin D Johnson (#134) says:

    I stand by my statement that comments are useless and dangerous.

    Sorry, I vehemently disagree. If a person writes page after page of code without any comments, to me that shows a callous disregard for those that have to maintain the stuff later on. I’m a stickler for good commenting, and it even helps when you come across a problem in your own code a few months (or even years) later. Well written comments tell a third party what you are trying to accomplish. Useful pseudo code, if you wish.

    OTOH, if the comments don’t agree with the code (you change the code but don’t update the comments), then that is bad. Shame on you for not being *professional*.

    As for the rest of your post after that opening salvo… I agree with what you say 100%, Hah! Bet you didn’t expect that :-)

    Comment by Steve Metzler — 31 Oct 2010 @ 7:10 PM

  136. Didactylos @130


    I take it you’re not familiar with the story of Mel?

    Oh my gosh … I had forgotten about Real Programmers.

    Sorry if I got a little testy

    klee12

    Comment by klee12 — 31 Oct 2010 @ 9:16 PM

  137. Thomas wrote

    I think security bugs are a different class of bugs than normal software bugs.

    Well, OK. I mentioned them because I am most concerned about those kind of bugs, but there were others. In the early days, the Blue Screen of Death on Microsoft products would indicate a bug. If you define a crash (rather than a graceful exit tries to preserve data) of an application certainly many open source applications have bugs. I guess I am just more pessimistic about significant bugs in climate science than others. When I worked a Ames Research Center we tested the results of our code by building physical models of planes and putting in the wind tunnel (now they build the plane and fly it with instruments). AFAIK there really isn’t a test like that in climate science.

    On the subject of a climate science repository, at the Ames Research Center we used a numerical analysis library called NAG. It was very well documented and you could buy a book (IIRC, Numerical Recipes) that described the algorithms. A repository made up of voluntary contributions has the following defects, IMHO, that supported packages are not supposed to have.

    1. uncertain quality control. Someone has to screen for bad code.

    2. Uncertain standards, especially on exceptions and documentation. On divide by zero, do you get a core dump or an informative message.

    3. Uncertain support. If I had a problem who do I call. What if he/she has died.

    klee12

    Comment by klee12 — 31 Oct 2010 @ 10:57 PM

  138. klee 137: AFAIK there really isn’t a test like that in climate science.

    BPL: Look again.

    http://BartonPaulLevenson.com/ModelsReliable.html

    Comment by Barton Paul Levenson — 1 Nov 2010 @ 8:09 AM

  139. OMG, I finally got a chance to read the story of Mel that Didactylos linked to in #130. That is so precious.

    I’m going to have to turn in my Real Programmer Badge™ now. I’m not worthy. Oh wait… I used to put bootstrap instructions into a PDP-11 using the 16 binary switches on the front panel. I’m still in there with a chance.

    Comment by Steve Metzler — 1 Nov 2010 @ 1:10 PM

  140. Re #135 – C’mon. This isn’t SlashDot – which I avoid like the plague because I can’t stand Flame Wars. Accusing me of being unprofessional is pretty silly – especially since it seems you didn’t read my previous post #117 for appropriate context nor have you seen the 2.5 million lines of code my team has produced – so how would you know? Well-written code DOES speak for itself. I’ve worked with tons of open source software (openssl, Apache, curl, STL, etc) and routinely have to read OS code headers/source (sockets, pthreads, file i/o, etc) to understand what they are doing and their comments were useless (and it wasn’t particularly well-written either but that’s another story).

    Comment by Robin D Johnson — 1 Nov 2010 @ 3:21 PM

  141. Barton Paul Levenson wrote

    klee 137: AFAIK there really isn’t a test like that in climate science.

    BPL: Look again…

    Well it depends on what type of test one is talking about. In klee@137 I wrote

    When I worked a Ames Research Center we tested the results of our code by building physical models of planes and putting in the wind tunnel (now they build the plane and fly it with instruments). AFAIK there really isn’t a test like that in climate science.

    I was talking about controlled tests that were reproducible (if you had a wind tunnel). I was talking about a high level of certainty, like Newton’s law (under certain conditions ignoring relativity). Building models which correctly predict the near future does not cut it for this level of certainty. Maybe the sun irradiance was was exceptionally strong or weak, maybe there were exceptionally long range internal cycles of something that worked for a hundred years and which may reverse. You can’t rule out other factors, some of which may be yet unknown, that might have caused agreement between prediction and result. In view of the possibility of confounding effects, the level of certainty is not has high for me as it is for others scientists. How certain you feel is subjective of course. In the case of airplane design, the number of factors (pressure, temperature, whatever) is small and can be taken into account in the CFD models. Because of that my certainty about CFD applied to airplane design is much greater than my certainty about climate science models. But that doesn’t mean climate models should be ignored; IMHO, their results should still influence real world decisions.

    Economics is another area where my level of certainty is fairly low. Economists have elaborate models that seem reasonable and usually they make correct predictions. And the models influence my real world decisions (i.e. my investing). But I don’t have as high level of faith in them as CFD models about airplanes.

    klee12

    Comment by klee12 — 1 Nov 2010 @ 4:28 PM

  142. Klee12,
    OK, so let’s say you are living on Mt. Merapi, and the government seismologists tell you it’s going to erupt. Do you leave? After all, you can’t do repeatable experiments.

    Or you want to know if it’s safe to dump toxic substances in a region that may flow into an aquifer. Again, no repeatable experiments.

    And the same for much of biology, medicine, ecology, astrophysics, cosmology… Do you think that maybe your lack of confidence in these fields might have to do with your lack of understanding rather than with whether they lack sufficient evidence for confidence in their results?

    Comment by Ray Ladbury — 1 Nov 2010 @ 5:23 PM

  143. But code “documentation” is largely useless or dangerous since it is rarely correct. Comments are particularly dangerous. The code needs to speak for itself.

    Comments *are* useful to a point. I have seen beautiful code without comments and ugly code exquisitely commented. For example:

    /* See Farrell p358 Equation 10.36 */

    Is extremely helpful for anyone reading the code later on.

    Where the comments do not match the code, it is an excellent indicator that both are wrong.

    Comment by Damien — 1 Nov 2010 @ 10:13 PM

  144. I particularly liked the Zeeya Merali article, which I saw before this one. I’ve already added a comment over there. The important take-away message is that for all the real flaws in climate science (nobody’s perfect), it is way ahead of biology (where I started doing research a couple of years back from a computer science background) and probably most other areas of applied science in terms of data and code quality. Sure, we all (that is those who actually studied computer science) would prefer that James Hansen didn’t use FORTRAN, but at least his code is published and available to anyone to check.

    Everyone who makes claims about the state of climate science as if it’s anomalously bad should be pointed at the Merali article.

    Comment by Philip Machanick — 2 Nov 2010 @ 1:46 AM

  145. klee 141: I was talking about controlled tests that were reproducible (if you had a wind tunnel). I was talking about a high level of certainty, like Newton’s law (under certain conditions ignoring relativity). Building models which correctly predict the near future does not cut it for this level of certainty.

    BPL: Which part of “the models correctly made seventeen predictions which panned out” did you not understand? By any reasonable standard, as opposed to your standard, those models are vindicated. Period.

    Comment by Barton Paul Levenson — 2 Nov 2010 @ 4:42 AM

  146. #120 klee12

    If I might attempt a redirect for the sake of context. And mind you I’m not a programmer, though I have done a bit of schema design for software.

    Yes, bugs exist.

    But isn’t it really about the results?

    The main issue that is tossed about in the denial of human caused global warming is that the models are wrong and we are not warming, or that it is not human caused.

    Now, let’s set a side the models and start from scratch for a second.

    - We see that the Arctic ice seems to be losing volume.
    - We see the glaciers seem to be losing ice mass at what looks, from eyeballing, to be an accelerated rate.
    - We see that the frequency of stronger hurricanes seems to be increasing.
    - We see the sea level rate seems to be rising and recently accelerated its rise rate.
    - We see a lot of things that are changing in the biosystems.

    Why?

    Now, from the basic understanding of how GHG’s work, they block infrared. SO we might assume that more GHG’s means more warming, and less would mean less warming.

    There’s also that thing about interglacial cycles and Milankovitch.

    Well, we know that we have increased atmospheric concentrations of GHG’s and that information is empirical.

    Now, models.

    Lot’s of different disciplines have looked at global warming. They have used different methods, and models, and code. They all seem to be showing generally the same results.

    I will go back to what Scott Mandia said:

    Consider the odds that various international scientists using quite different data and quite different data analysis techniques can all be wrong in the same way.  What are the odds that a hockey stick is always the shape of the wrong answer?

    So while the models are likely wrong due to a number of reasons that would take too long to illustrate, they are right enough in showing replicable results from multiple disciplines.

    Making models better is great. Open code is great. But what is really important is that we recognize that we are warming, it is human caused, and we need to take meaningful action if we wish to avoid unpleasant results in the overall economy.

    Economics: Balancing Economies
    October Leading Edge: The Cuccinelli ‘Witch Hunt”

    Fee & Dividend: Our best chanceLearn the IssueSign the Petition
    A Climate Minute: Natural CycleGreenhouse EffectClimate Science HistoryArctic Ice Melt

    Comment by John P. Reisman (OSS Foundation) — 2 Nov 2010 @ 5:35 AM

  147. Is it possible to point to any studies which have been performed using open / shared code and data sets that demonstrate that the majority of climate change simulations are incorrect in their forward projections over the coming decades and that the conclusions of the last UN Assessment Report are thereby unreliable or untenable?

    In those cases where researchers have demonstrated that the shared code is based on less-than-adequate precepts, have those researchers provided an open critique of the code and made openly available their new models and simulation systems? Do those new models and simulations explain the temperature data sets?

    Where is the ‘best’ simulator currently provided openly by the climate change skeptical research community?

    Comment by MX — 2 Nov 2010 @ 6:12 AM

  148. Gavin, Eric,

    You made a strong case as to why simple repetition isn’t half as important as independent replication, at least to the science.

    But to the suspicious public (and those susceptible to suspicion) simple repetition apparently is very important. Like it or dislike it, that seems to be the current reality, and I think that is why also many supporters of science are calling for this kind of openness that allows simple repetition.

    Hey, at least the time that people spend repeating what you did, they don’t spend that time yelling at you!

    Comment by Bart Verheggen — 2 Nov 2010 @ 9:02 AM

  149. Bart,
    Those calling for repetition base their calls on paranoid delusions of a vast scientific conspiracy. They will not be satisfied regardless of the evidence. Scientists should not cater to idiots.

    Outside efforts to make the code work–and even to improve it–are great. Scientists should stick to doing the science rather than changing the mental diapers of the denialists.

    Comment by Ray Ladbury — 2 Nov 2010 @ 9:51 AM

  150. Bart:

    Hey, at least the time that people spend repeating what you did, they don’t spend that time yelling at you!

    I have not been following the antics of denialists very closely, but have any actually repeated or independently done a serious analysis of the data?

    Comment by Richard Simons — 2 Nov 2010 @ 9:54 AM

  151. Ray Ladbury wrote @142 wrote

    so let’s say you are living on Mt. Merapi, and the government seismologists tell you it’s going to erupt. Do you leave? After all, you can’t do repeatable experiments.

    I would leave even if I thought Mt. Merapi wold not erupt for the same reason I bought life insurance when my son was born. It’s the precautionary principle.

    I try always to make a distinction between the science and my decisions in the real world. Within science I have a high bar for being certain of a theory or fact. It has to be reproducible and controlled… there can be no other factor which might possibly affect the outcome. Newton’s law of gravitation meets (taking in account relativity) meets my bar for high certainty; the models from climate science do not. By the way, it is logically consistent not to have a high certainty for climate models but still advocate action regarding climate change. We all have to make decisions with some uncertainty. Decisions based on Newton’s law (e.g. sending man to moon, assuming all engines fire correctly, etc.) have, for me, no uncertainties. Decisions based solely on climate models have uncertainty. The amount of uncertainty depends on your faith in the models.

    John Reisman @146 wrote

    The main issue that is tossed about in the denial of human caused global warming is that the models are wrong and we are not warming, or that it is not human caused.

    Now, let’s set a side the models and start from scratch for a second.

    Yes I agree there is more evidence for climate change than just what the models predict. But I was just concerned with computer models.

    Let me review this how this subthread evolved.I suggested that every climate scientists publish the their programs because science has to be reproducible. If the results of a paper depends on computer code, the only way to see if the code is correct is to examine the code. Describing the methods is not enough because if another scientist got different results, who’s code do you trust? In mathematics one must publish the reasoning behind a new theorem.

    The thread morphed into a discussion regarding comments in code. I don’t think comments in the code were very important.

    So far, so good. I know something about computers and I think I was on firm ground. Then I made a statement about the relative certainty of the results of CFD and climate science models. That statement is more about the epistemology or philosophy of science than about computer science. Now I feel I’m being invited into discussing climate science. I’m not a climate scientists and I’m don’t want to go there. I think I can add something, or at least present a view for you consideration, on computer science that may be informative to some on this board.

    I don’t know much about climate science and there’s nothing much I can add on that subject. My view, what whatever it’s worth, is that climate is warming, that humans are adding to the warming and that we should reduce CO2 emissions due to the precautionary principle.

    klee12

    Comment by klee12 — 2 Nov 2010 @ 1:33 PM

  152. Robin said in reply to Steve: “Accusing me of being unprofessional”

    Robin, he didn’t accuse you of anything. He said leaving outdated comments is unprofessional, which it is.

    Further, this discussion wasn’t just about bad comments – it was about good comments. Which from your description, you have never encountered. Which is sad, because it implies that you and your team write worthless comments (by your own evaluation).

    We agree that readable code is important. But if you can’t enhance the readability, and, in particular, decrease the time needed to understand code with judicious use of comments, then you lack an important skill.

    Why do you insist all comments are “useless and dangerous”? It’s just unreasonable. Yes, we know there are bad comments out there, just as there is bad code. That doesn’t change the value of good comments any more than it changes the value of good code. It’s not a choice – we can demand both.

    Comment by Didactylos — 2 Nov 2010 @ 3:06 PM

  153. klee12 said “I was talking about controlled tests that were reproducible”

    The problem isn’t that climate science doesn’t have reproducible, definite tests. The problem is that climate isn’t reproducible and definite. In particular, the weather isn’t consistent at all. It is chaotic.

    This means that in climate science, evaluating the results of climate models needs a different benchmark than evaluating a wind tunnel test. Other branches of science have other standards, developed to match whatever peculiarities the particular field has. Particle physicists see gazillions of collisions, so they require a ludicrously high confidence to compensate. Biologists often have tiny samples, so they do everything they can to extract statistically significant results.

    And climate scientists deal with chaos, so they expect a wide range in their model output. In fact, if climate models produced predictable weather patterns all the time, that would indicate the model is broken.

    Note also that climate isn’t weather. Weather models will diverge if you run them beyond reasonable time-periods. Climate models only show useful results when run over long time periods.

    People with experience of other kinds of model often make assumptions about climate models. I recommend you check your assumptions against the FAQ on climate models (see part II also).

    Comment by Didactylos — 2 Nov 2010 @ 3:27 PM

  154. Klee12 says: “Within science I have a high bar for being certain of a theory or fact. It has to be reproducible and controlled… there can be no other factor which might possibly affect the outcome. Newton’s law of gravitation meets (taking in account relativity) meets my bar for high certainty; the models from climate science do not.”

    OK, so let me get this straight: You are saying that biology, geology, astronomy, astrophysics, geophysics, meteorology, and for that matter about just about any science you encounter outside of freshman lab is not science? Dude, do you really want to take a position that is that silly?

    FWIW, I’ve actually walked through the crater of Mt. Merapi–albeit in one of its quieter phases. It’s a pretty amazing place. You’re missing out.

    Comment by Ray Ladbury — 2 Nov 2010 @ 7:09 PM

  155. klee12 #151: Newton’s Laws do not guarantee reproducible results without uncertainties. Why do you think a space probe has to carry extra fuel to correct its course? More on my blog.

    You may find it instructive to study the Muir Russell report on the CRU email theft if you think that publishing methods and not code renders results irreproducible.

    Comment by Philip Machanick — 2 Nov 2010 @ 7:33 PM

  156. I know its a bit off topic, but I thought I’d enter a defense of Fortran. The language name is short for Formula Translation, which is a big part of what scientists do when programming. Beyond that the accidental way the lanquage evolved leads to the possibility of very efficient usage of modern computers. Our particular application is mostly written in Fortran. Because the new vector like constructs available in modern computers are pretty restricted, performance is greatly enhancd if the compiler can make certain assumptions about low level address alignment, i.e.
    “does every array in this loop has the same alignment with respect to cahche line boundaries”. With Fortran common blocks asuring that is a snap. It is hard for me to imagine the compiler would be abe to examine C code and figure that out. So you can leave a lot of performance, which I think for large climate simulations limits how much you can accomplish, you are going to throw quite a bit of that away by using other more modern lanquages. I also find C much more prone to subtle errors. Twice this week I’ve wasted a couple of hours tracking down C code performing weirdly and wrongly, and in all cases, its been a misplaced equals sign, like missing == in an if test. The human brain (at least mine) just glosses over these sorts of typos, but then eventually you gotta figure out why your code is acting insane. All languages have their traps. Fortran got a bad name because of all the code written in the early years, before anyone had heard of software engineering. But, the modern evolutions of it are actually quite good.

    Comment by Thomas — 2 Nov 2010 @ 9:43 PM

  157. I agree emphatically with Thomas. Modern Fortran (95 and later ANSI standards) is a nicely structured, extremely useful, very readable programming language. And nothing out there but assembly language beats it for crazy speed if you’re running a simulation. They’ve had 53 years to optimize its performance. I can’t think of any other language that’s been going that strong, that long.

    For those who want to try it, the “Personal Edition” of Salford Fortran-95 is free to download. That’s what I used until I turned into a Visual Basic 2008 freak. I retain it in case I have to translate a program into Fortran for professionals.

    Comment by Barton Paul Levenson — 3 Nov 2010 @ 5:06 AM

  158. #151 klee12

    I agree certainly with the precautionary principle you ascribe. However, it seems that when you combined the computer models, the error bars, combined with the observations and the maths and physics combined provide a rather stronger metric than merely the precautionary principle, for reasoned action.

    Besides, the precautionary principle is not enough for those that approach this from the perspective of belief and political bias.

    Since we are dealing with tremendous inertia’s of energy and related feedbacks, we need to apply the brakes now and as firmly as possible without degradative disruption of the socio-economic system.

    From this one can reasonably conclude that meaningful policy action is needed now, rather than alter and that the confidence levels of the correlation of multiple disciplines of science concur with the general results, scientific consensus is strong and that our economy depends on said meaningful action.

    The combined lines of evidence, models and error bars included have resulted in what is now claimed as an unequivocal result in that humans are in fact influencing the climate. So the precautionary principle is not really even needed although not unimportant.

    That being said, I would like your opinion regarding Fee and Dividend policy as an effective means by which to exercise the precautionary principle, and adding your perspective to the mix of even possibly signing it?

    http://www.climatelobby.com/
    http://www.climatelobby.com/fee-and-dividend/q-a/

    BTW, for those that may have actually tried to click on the top two links below, I did fix them :)

    Economics: Balancing Economies
    October Leading Edge: The Cuccinelli ‘Witch Hunt”

    Fee & Dividend: Our best chanceLearn the IssueSign the Petition
    A Climate Minute: Natural CycleGreenhouse EffectClimate Science HistoryArctic Ice Melt

    Comment by John P. Reisman (OSS Foundation) — 3 Nov 2010 @ 8:24 AM

  159. Philip Machanick (#155)

    I don’t think his comment was related to the small amount of error in those calculculations (newton’s law). It was related to how much you can believe the results represent reality. Newton’s Law has be validated much more that climate models. This is a fact and there is no way you can argue that. Besides, Newton’s Law is included in the climate models so any uncertainty there shows up in the climate models as well.

    IMHO, the main difference between the climate models and Newton’s Law is that there are inherent uncertainties in the climate system (ie. even with perfect information they won’t go away) while most uncertainties in Newton’s Law results from imperfect information. I do think most of these uncertainties are well represented in the models, but there may be some that aren’t understood fully.

    In summary, I have greater confidence in Newton’s Law than Climate Science (shouldn’t this be self evident?) but that doesn’t mean the GCM models are useless. The results of the models should be qualified though, when they are communicated to the political actors (in a form they will understand). Generally, I think scientists have been ernest in trying to communicate this information but haven’t been able to communicate this information across the political spectrum (clearly!).

    Comment by sambo — 3 Nov 2010 @ 10:19 AM

  160. Re #152

    ROFL… Like in the last 30 years, I’ve never looked at code written by someone else…

    I don’t bother writing comments since my code is perfectly readable. The professional software engineers I know, agree with me on comments so I know I am not alone. And just to be clear my teams have produced high-volume, reliable SOA applications that generate millions of dollars in revenue so we’re \serious\. My experience with comments comes from having to review and fix reams of code written by others. Even \good/valid\ comments can become obsolete (ie useless and dangerous) because of changes in library or application code being called by the commented code. The called code is usually completely unaware of the caller’s code – so how can it possibly be kept up-to-date over months and years of development? Fundamentally, comments are never actually \tested\ or debugged in any real sense. So, their content is inherently suspect and even \quality\ comments degrade over time. Believing a comment instead of following the code back properly through the layers is often a fool’s errand. The bottom line is that knowledge of the underlying purpose of the code exists external to the code (and the comments). Without that knowledge – the code and comments will not be comprehensible.

    Re #156

    Fortran is perfectly fine and considered very efficient for doing calculations. Where Fortran falls down is \housekeeping\ – memory management, threads, sockets, system resources, shared resources, object allocation/deallocation important to building large complex applications.

    BTW C++ is perfectly capable of doing cache alignments [and other fancy performance enhancements - including allowing inline assembler] and most C++ compilers produce highly efficient code – when the code is written with that goal in mind.

    Comment by Robin D Johnson — 3 Nov 2010 @ 11:14 AM

  161. Sambo@159 “IMHO, the main difference between the climate models and Newton’s Law is that there are inherent uncertainties in the climate system (ie. even with perfect information they won’t go away) while most uncertainties in Newton’s Law results from imperfect information.”

    Ever hear of the three-body problem?

    Comment by Ray Ladbury — 3 Nov 2010 @ 11:38 AM

  162. Thomas:

    I almost fell over laughing* when you started talking about “modern” languages like C. The wheel has turned. Today’s modern languages usually make issues such as address alignment moot, since they will handle structure padding and things of that nature quietly, by default. C is indeed prone to some very subtle errors, largely thanks to pointers. And pointers are evil – useful but evil. The “if” test example you quote wouldn’t be an issue in a language such as C#, which doesn’t allow that sort of implicit casting. Language designers have learned a lot (mostly by stealing from other languages).

    You did raise an interesting question in my mind: is there room for a new scientific computing language? One with all the pros of Fortran, but that also benefits from the decades of language evolution since then? I don’t know enough about Matlab to know if that comes close to fitting the bill, and for all R’s benefits, it may be newish but it is neither modern nor suitable for general use.

    I keep hearing how Fortran is great for scientific computing, but it really seems to lack any specia-lised features. I think it gets used purely due to legacy code.

    * I didn’t.

    PS: Dear moderators: the value of a spam filter that can’t distinguish between a drug name embedded in another word and real spam? That can be fooled merely by adding a hyphen in the offending word? Zero. You have turned on reCAPTCHA. This is a 100% moderated blog. Perhaps you should consider turning the useless spam filter off?

    Comment by Didactylos — 3 Nov 2010 @ 12:21 PM

  163. Robin:

    “in the last 30 years, I’ve never looked at code written by someone else”

    “My experience with comments comes from having to review and fix reams of code written by others”

    I’m sorry, I can’t reconcile these comments. Are you saying you retired a long time ago? Software engineering has moved on.

    “changes in library or application code being called by the commented code.”

    This is really important. Comments can capture assumptions made about library code that aren’t explicitly verified by the calling code. But if the library code changes in ways that break calling code, then you have a more serious error of process than a few outdated comments!

    “Fundamentally, comments are never actually \tested\ or debugged in any real sense.”

    Why should they be? They should be checked at the same time as the code, but their function is to help. They aren’t normative. Nobody should assume the comments are perfect, any more than they should assume the code is perfect.

    “Believing a comment”

    See above.

    If as I suspect, you left programming 30 years ago, then perhaps you just aren’t aware of the sort of standards expected from today’s programming graduates. We look at code from way back when, and wonder how you ever got anything done.

    Comment by Didactylos — 3 Nov 2010 @ 12:36 PM

  164. Ray Ladbury (#161)

    In the three-body problem, most of the uncertainty comes from small perturbations to the ideal solution. Often, this comes from assumptions that are inexact (point mass, rounding error is small). If this information were perfect, I believe it would be exact solutions (and I’m quite confident of that!). Moreover, the discussion was about decision’s under these uncertainties (practical space probes). The lack of analytical solution to the three body problem says very little about the practical aspects I was commenting on. Yes, it needs to be considered, however in most cases it has relatively minor affects (if not, you are not planning your mission very well).

    Comment by sambo — 3 Nov 2010 @ 12:37 PM

  165. Re #163

    ROFL… Are you kidding me? Did you simply misread what I said or deliberately misquoting?

    “Like in the last 30 years, I’ve never looked at code written by someone else…”

    The usual interpretation of that is “Yes. I have looked at other people’s code.”

    As for…

    “If as I suspect, you left programming 30 years ago, then perhaps you just aren’t aware of the sort of standards expected from today’s programming graduates. We look at code from way back when, and wonder how you ever got anything done.”

    Again you failed to read or comprehend my previous posts. How about I suspect you don’t even write production code? Not that I think that matters particularly.

    But I DO design and write production code, for Unix and Windows, yesterday and today. Over a million lines of robust C++ and counting (40,000 in just the last two months alone – lots of new stuff recently…). You know, fully object oriented, using exceptions, templates, multi-threading, blah, blah, blah…

    And seriously if you wanna comment your code – be my guest. I don’t care. But I don’t think Climate Scientists should be “expected” to comment their code since the industry sure as the world doesn’t think its useful. The general consensus of practicing engineers is “useless and even dangerous to rely on comments for anything serious”.

    Comment by Robin D Johnson — 3 Nov 2010 @ 3:40 PM

  166. Didactylos wrote @ 153
    The problem isn’t that climate science doesn’t have reproducible, definite tests. The problem is that climate isn’t reproducible and definite.

    Yes I agree, but I draw the conclusion that models can not give a high level of certainty. This is reflected, in part, by the acknowledged long term predictions has a high level uncertainty. If I send a space probe around the solar system I have confidence I can predict its path for a hundred years, baring exogenous effects like solar wind, unforseen meteoroids, and so on.

    Ray Ladbury @ 154 wrote


    You are saying that biology, geology, astronomy, astrophysics, geophysics, meteorology, and for that matter about just about any science you encounter outside of freshman lab is not science

    Not really. Long, long ago, biology was just the classification of various organism into genra, species, or whatever. No predictions, nothing to reproduce. Then Mendel made some observations and experiments about heredity of certain traits and the experiments were reproducible. Today we can isolate genes, and create new strains of rice or corn, which is reproducible.

    I suppose I could go on. Geology has observations (I think) but I don’t think you do large scale experiments so there is nothing to reproduce. You work with the theory that seems to fit the work the best and I’ll call that science. And I’ll act assuming the latest theory is the correct one. But it won’t have the level of certainty (for me) as Newton’s laws.

    Philip Machanick @155 wrote

    Why do you think a space probe has to carry extra fuel to correct its course?
    There may be many reasons (1) On lift off there may be a slight error in the trajectory or speed (2) solar wind (which really exerts a very small force) may push it slightly off course. Maybe very slight force over years might make you miss a target millions of miles away by thoudsands of miles. (3) friction or the gravitational effects when flying through or near asteroid belt between Mars and Jupiter.

    You may find it instructive to study the Muir Russell report on the CRU email theft if you think that publishing methods and not code renders results irreproducible.I stated earlier if someone tried to reproduce the results and got a different result, it is very difficult, if not impossible, to see where the discrepancy arose. The code may differ from the method described (error in specification) or in the code (bug. Re the Muir report, I briefly took a look at the table of Findings and couldn’t find where they discussed models. A page number would help.

    John Reisman wrote

    Besides, the precautionary principle is not enough for those that approach this from the perspective of belief and political bias.

    I separate the the science aspect from the public policy aspect. The scientific part is concerned with nature and only nature and should be as objective as possible. The public policy part involves making decisions in the face of uncertainty and different people of good will may have different opinions. Scientists should take part in the public policy debate but they should be careful not to oversell their position. I cringe when I hear scientists say “the science is settled”. They do not specify what is settled and when unconvinced scientists examine the evidence some may, in good faith, not be entirely convinced that the “science is settled” and resent what they perceive is an attempt to mislead. I think that Al Gore is not held in high regard, and I think that in the long run his efforts have hurt rather than help his cause. Scientists should not oversell the science because it may lose credibility and hinder their cause.

    My position there is that the science is not settled, and from a public policy point of view I invoke the precautionary principle to advocate action.

    On the Fee and Dividend policy, that seems much better than Cap and Trade, but it is still too unwieldy. I don’t see it happening. This open is not based on science, it is not the efficacy, it is based on my reading of the political situation which may be wrong.

    Comment by klee12 — 3 Nov 2010 @ 3:47 PM

  167. Did 162: I keep hearing how Fortran is great for scientific computing, but it really seems to lack any specia-lised features. I think it gets used purely due to legacy code.

    BPL: What part of “faster than anything else but assembly language for numerical simulation” did you miss?

    Comment by Barton Paul Levenson — 3 Nov 2010 @ 3:50 PM

  168. sambo 164: In the three-body problem, most of the uncertainty comes from small perturbations to the ideal solution. Often, this comes from assumptions that are inexact (point mass, rounding error is small). If this information were perfect, I believe it would be exact solutions (and I’m quite confident of that!).

    BPL: Then I take it you’ve never actually taken a course in celestial mechanics, or read Laplace.

    Comment by Barton Paul Levenson — 3 Nov 2010 @ 3:51 PM

  169. BPL said: “What part of “faster than anything else but assembly language for numerical simulation” did you miss?”

    That’s your opinion, and I pretty much just ignored it. Mostly because you didn’t bother providing anything to back the claim, but ultimately because you are wrong. Any benchmark is limited in utility, but there is absolutely no way your bizarre claim is justified based on all the benchmarks I’ve seen.

    I did not base my opinion on the fact that you think Visual Basic is a good programming language. Instead, I checked those benchmarks again.

    Comment by Didactylos — 3 Nov 2010 @ 5:07 PM

  170. Robin: you wrote 40,000 lines of code in 2 months? Yay for you. I hope I never see it, and I *really* hope you don’t work anywhere important.

    I’m optimistic that you just don’t know how to measure SLOC, and that you don’t really produce high volumes of dreck and imagine that all the effort equates to productivity!

    Either way, your comments about comments reflect poorly on the software industry.

    I do apologize for misunderstanding your attempt at humour.

    “The general consensus of practicing engineers” – no. No. That’s your wildly misinformed opinion, not shared by everyone.

    Comment by Didactylos — 3 Nov 2010 @ 5:33 PM

  171. BPL (#168)

    You’re being presumptious. I have taken a class in celestial mechanics. I stand by what I said (it is an opinion statement in any case). Argue against that if you wish, but I’d rather you not ressort to false assumptions about me.

    BTW, to what are you refering in terms of Laplace. As far as I know he didn’t write much that would argue against what I stated.

    Comment by sambo — 3 Nov 2010 @ 5:45 PM

  172. klee12: you have some very odd ideas about science. It is waaaaay beyond my ability to correct your ideas. Perhaps try reading a few popular science books, or something?

    The Muir Russell report discusses reproducing results using completely different methods and data. That is far, far more valuable from the point of view of the result. It may not help advance any single method, but it shows that minor differences in method are irrelevant.

    And you really need to do some research on the n-body problem. Don’t get distracted by details such as launch imperfections. The key point is that useful positional data is the result of a complex numerical simulation, and not a neat equation.

    Comment by Didactylos — 3 Nov 2010 @ 5:53 PM

  173. I’m optimistic that you just don’t know how to measure SLOC, and that you don’t really produce high volumes of dreck and imagine that all the effort equates to productivity!

    Well, at least he can’t be accused of counting comment lines as lines of code! /snark

    Comment by Steve Metzler — 3 Nov 2010 @ 6:11 PM

  174. Sorry Robin, one more thing that really needs to be said regarding the commenting issue: the people that write system libraries. Bet you’re glad *they* wrote comments, huh? Or do you always procure the source for every library method and read *every line of source code in every library routine* that you use to make sure it has no bugs in it before you use it? Yeah, didn’t think so. That would be really productive. Not.

    You would also need to do that just to figure out how to even *call* every freakin’ library routine if no one bothered to document the parameters. And that applies to *everything* your esteemed co-workers write that you need to call.

    Now do you realise how ridiculous the position you are taking looks? Enough said.

    Comment by Steve Metzler — 3 Nov 2010 @ 6:29 PM

  175. #166 klee12

    I would agree that the science is not settled and in fact never will be settled.

    However, there is enough certainty to in climate science to take action without the precautionary principle. IN other words the science is understood enough in key areas to see that certain conclusions can be drawn with sufficient confidence on the robust results of the science to take action without inordinate amounts of guessing or exercising the precautionary principle.

    In other words, the science is settled enough in understanding the major forcing components, error bars included, multiple disciplines included, same general conclusions included that we really don’t need to guess about the fact that virtual certainty exists and humans are impacting the climate.

    My point is that all this science, while uncertainties do exist in certain areas and models are not perfect allows reasonable basis to invoke meaningful action on sufficient magnitude to counter the future impacts of the increased emissions at various levels of impact as estimated by those imperfect models.

    I think we can agree that the political situation is far from reasonable comprehension at this time. but that does not diminish the potential of Fee and Dividend, it merely illustrates a different problem.

    In addition to #172 Didactylos

    Don’t let the perfect be the enemy of the good. A perfect example was Steve McIntyres statistical analysis of the Hockey Stick. While it may have added in method, it weakened the model. And, the result was that the Hockey Stick is still a Hockey Stick and the McIntyre fix proved statistically insignificant.

    Economics: Balancing Economies
    October Leading Edge: The Cuccinelli ‘Witch Hunt”

    Fee & Dividend: Our best chanceLearn the IssueSign the Petition
    A Climate Minute: Natural CycleGreenhouse EffectClimate Science HistoryArctic Ice Melt

    Comment by John P. Reisman (OSS Foundation) — 3 Nov 2010 @ 7:49 PM

  176. Re: 170 LOL Whatever… And that is 40,000+ high quality lines of code in two months. With design documentation, QA and in production used by customers.

    We do numerous releases a year with rigorous SDLC processes – considered a model in the company (we wrote the processes). My team and our code generate millions of dollars in revenue every year. No unplanned downtime, 24/7/365. And no stupid comments. We’ve been doing the same for years. What have you done?

    Comment by Robin D Johnson — 3 Nov 2010 @ 8:18 PM

  177. klee12 #166: I didn’t find it hard to find the discussion in the Muir Russell report on reproducing results independently but it starts on p 150.

    sambo et al.: a space probe has to navigate under much more complex conditions than the 3-body problem. We have over a million asteroids for a start, of which a small fraction have been mapped. You can theoretically model the entire solar system with differential equations and solve them exactly but in practice this is impossible. If your space probe goes nowhere near the asteroid belt this specific correction may be minor, but there are many unmapped bodies in space. The climate system is subject to many more variables but the same principle applies. In theory you could apply classical physics to the entire system rendering it totally deterministic but in practice, it’s impossible. I direct this argument specifically at people who claim climate science isn’t “proper” science because it can’t yield exact answers. The same is true of any science of the real world that works with large-scale phenomena like oil geology.

    Comment by Philip Machanick — 4 Nov 2010 @ 4:10 AM

  178. BPL: What part of “faster than anything else but assembly language for numerical simulation” did you miss?

    Did: That’s your opinion, and I pretty much just ignored it. Mostly because you didn’t bother providing anything to back the claim, but ultimately because you are wrong. Any benchmark is limited in utility, but there is absolutely no way your bizarre claim is justified based on all the benchmarks I’ve seen.

    BPL: When I run a certain radiative-convective model written in Fortran-95, it runs in 33 seconds. In VB 2008, it runs in 72 seconds. In Borland C/C++ it runs in 58 seconds. Fortran beats everything.

    The “benchmark” programs assume every program is doing everything you can do with a computer. Real programs don’t. That’s why I specified “numerical simulation.” Numerical simulation doesn’t have to get on the web, move pictures around in a GUI, send signals, coordinate coroutines, or dereference pointers. I’m talking about crunching numbers, about fast, highly efficient routines for arithmetic and transcendental functions. For that, Fortran beats anything but assembler.

    Comment by Barton Paul Levenson — 4 Nov 2010 @ 7:34 AM

  179. sambo 171,

    The 3-body problem is only perfectly soluble for a perfect case. Maybe you did take celestial mechanics, but if so you must have skipped numerical analysis. Any real-world solution to the n-body problem is an approximation, always and everywhere. See Laplace 1826.

    Comment by Barton Paul Levenson — 4 Nov 2010 @ 7:35 AM

  180. klee12@166

    OK, so I’m not seeing a difference between the situation in geophysics, for example, and climate science. And you do know that Newton’s laws don’t work so well under nonclassical conditions, don’t you? I wouldn’t use them in the vicinity of a black hole or in a degenerate gas, for instance.

    Sambo, the 3-body problem is not soluble in the general case in the real world. That reality, she’s a bitch, ain’t she?

    Comment by Ray Ladbury — 4 Nov 2010 @ 9:48 AM

  181. BPL (#179)

    I suggest you stop being presumptious. Read what I stated in 164. I said there isn’t an analytical solution, and the perturbations are due to the numerical methods and imperfect information. I think Laplace 1826 would support that statement.

    In any case for the practical engineering decisions, you rarely need the three body solutions. You typically restrict the problem to a two body solution since you don’t need the three body one.

    If you reply again please don’t hypothesis what I have and haven’t understood from my degree. It’s rather anoying and you’re making a lot of false assumptions.

    Comment by sambo — 4 Nov 2010 @ 9:53 AM

  182. Oh, BPL…. I know from experience that you are immune to reason, so I’m not surprised to see you base your beliefs on a single test that you ran yourself.

    But just to give you a clue where you are going wrong: you are confusing “benchmark” with those general purpose PC-benchmarks that the computer magazines invented for consumer testing.

    Why do you have to be so absolute about everything? The algorithm is the important factor, and for some implementations Fortran will beat C, and in other cases C will beat Fortran. Fortran code is only “nearly as good as assembly”* if an expert spends some considerable time making it so. But the same is equally true of C, or any language that allows that level of control. For a while, Fortran had a significant advantage in that the available Fortran numerical computing libraries were already highly optimised. But that advantage has gone long ago.

    * What you meant, of course, is assembly hand-crafted and tuned by an expert. It’s perfectly easy to write inefficient code in assembly, you know.

    Comment by Didactylos — 4 Nov 2010 @ 10:27 AM

  183. Did 182: I know from experience that you are immune to reason

    BPL: Your kind of reason, anyway. I think most people are.

    Comment by Barton Paul Levenson — 4 Nov 2010 @ 11:50 AM

  184. Didactylos wrote @172

    The Muir Russell report discusses reproducing results using completely different methods and data. Ah, but I was talking about verifying that the method used to obtain the result is correct, i.e. meets specs and no bugs. I think we will have to agree to disagree on this point.

    And you really need to do some research on the n-body problem. I’ve retired gave away my books but IIRC, Newton’s law (ignore relativity) says the trajectory of a probe satisfies some ordinary (not partial) differential equations. The equations cannot be solved analytically but it can be solved numerically and maybe iteratively. It is an initial value problem but there are errors in measurements if the initial values. Let t=0 at the time the engine on the probe separates. The solution to ordinary differential equations can be calculated. But the initial position and velocity vector at t=0 has errors. The force solar winds and effect of the asteroid belt cannot be calculated with great precision. Hence the need for some error correction.There is no discrepancy with the theory. However, I think the corrections (margin of error) is very small.

    John Reisman @175 wrote

    IN other words the science is understood enough in key areas to see that certain conclusions can be drawn with sufficient confidence on the robust results of the science to take action without inordinate amounts of guessing or exercising the precautionary principle.

    Ah but who understands the key areas. The average voter without a degree in climate science does not understand. He/she takes much on faith. The problem is which faith.

    A climate scientist may evaluate the literature understands and makes judgements based. His certainty may be fairly high, but I don’t think it should be as high as Newton’s law. And based on his/her knowledge the scientist advocates takes a position.

    I have degrees in physics and math to morphed into a non-degreed computer scientist. In my training in physics I was exposed to some of the history of physics and mathematics. In particular the history of the quantum theory and non-euclidean geometry made an impression on me and helped form my epistemological view on knowledge. In regard to climate science, I decided to investigate about a year ago. I have looked at some of the papers and didn’t understand much of what they said, though I understood the abstract of most of them. So I would rate myself somewhere between the average voter and a climate scientist. But still I form my opinion based on what I know. Climate scientists who have more knowledge will form their opinions on what they know and it may be different from my opinion.

    I guess what I am saying is that different people of good will may reach different opinions of climate change. The reasons for the different opinions is reflected in part by their knowledge of the subject.

    Philip Machanick wrote @177

    it starts on p 150
    I don’t have time now to read and comment, but tomorrow maybe

    Ray Ladbury @180 wrote And you do know that Newton’s laws don’t work so well under nonclassical conditions, don’t you? Yes, I knew some special relativity, never got into general relativity. You all my previous statements you can replace my relevant references to Newton’s law by special or general relativity. There may be singularities in general relativity but those, I think, are predicted by the theory.

    klee12

    Comment by klee12 — 4 Nov 2010 @ 12:31 PM

  185. klee12 said “However, I think the corrections (margin of error) is very small.”

    Imperceptible differences in initial conditions can have huge effects on the later trajectory. This is really the very definition of chaos. Did you look at the animation I linked to? The secret of rocketry is that in nearly all cases, small gravitational fields can be ignored, and the problem reduced to a simple one. I don’t think this is true when considering longer trips. Even a trip to the moon involves three significant gravitational forces (a 4-body problem).

    BPL: You are right. Most people don’t bother with reason, instead preferring to rely on their instincts, their hopes, and their fears. This is really what denialism is all about. I’m surprised to see you grouping yourself with the deniers, though (unless you were oh so subtly trying to insult me).

    Comment by Didactylos — 4 Nov 2010 @ 3:46 PM

  186. Thanks Dr. Schmidt and Dr. Steig.

    In the past three years I think we have come a lot closer to the position that people like Nick Barnes and I take.

    In general, I think the central argument I would make is this. I do not push for open code because it is scientifically interesting. I do not think that repetition is better than replication. I push for open code because it is beneficial to scientific advance. Absent any cogent argument, absent any factual argument that open code is detrimental to science, I think the arguments for the benefits win.

    Because of the release of GISS code Nick and others can now answer some annoying silly questions that Dr. Ruedy should not have to bother himself with. Because of that release people can now engage in “what if” scenarios that the word constraints in journals do not permit. For example, in the forthcoming H2010, hansen will extend the use of nightlights ( 1995 DMSP lights based on F12) to the entire station inventory. because the code is released and up and running in individuals hands we will be able to do the following.

    A. re run his analysis using the improved station location data that WMO will publish on nov 8.
    B. re run his analysis using the Nightlights data from F16 ( the F12 data has been deprecated according to the PI and he doesnt want anybody using it )
    C. Run the code with the Beta V3 data.

    and other things as well.

    That is a large part of what people like me have wanted to do. Take the code behind a paper and do things that we find interesting without re writing “accepted science” from scratch. It took a long time to get here. But I think you can agree that what CCC have been able to do is a benefit.

    So, the point is this. Replication is more scientifically interesting that repetition. No one ever argued otherwise. What many have wanted is the code required to produce the results, so that more investigations can be done with the same code. Is it possible to write code from the descriptions in papers. Sometimes yes, sometimes no. But that has never been the important question. The question has been “is it better to release the code or not?”

    In the particular areas I’m interested in, I’ve never seen a piece of code that was better off unreleased.

    Anyway, despite all the harsh words over the years, I do want to thank you and Dr. Steig ( and Dr. Benestad as well) for your efforts in freeing up more code. and with that. peace out

    Comment by steven mosher — 4 Nov 2010 @ 4:17 PM

  187. P. Machaninck wrote @177

    I didn’t find it hard to find the discussion in the Muir Russell report on reproducing results independently but it starts on p 150.

    OK, I got it.

    It seems that the Muir commissioned replicated the CRUTEM analysis using the data and algorithm and methods listed on following page(s). This replication does not mean verification in my mind, especially if someone else tried and failed to replicate the results.

    I took at quick glance at the “Code to Process the Data” which I assume constitutes an methods and alogrithms description. The second bullet item had phrase “ we required good data …. It doesn’t define what good is. I know you want assume that a weather station reporting -100 degrees whatever near the equator is probably in error, but, still, how do you define good? In the next bullet item choose the most valid monthly normals How precisely did you choose. And so on.

    I realize that the methods and algorithms of the developers of CRUTEM may have been much more precise than the Muir Committee, but this does illustrate the type of errors, which I call specification errors, which perhaps the scientific community, does not take as seriously as they code.

    Ordinary language is not precise a computer program is. On large software projects people work for a long time on drawing up specifications (aka requirements) and break up the project into modules, and assign teams of programmers to work on each module which have precise specifications. The team programs the module up, checks there are no coding bugs, and then the whole program with all the modules are tested together. Frequently, (dare I say always) there are still errors due to misunderstanding of the specifications … something has to be changed. Often the delay in a project is due to delays in the testing part of the project when you put everything together and it doesn’t all fit. I think it is very difficult to detect these types of errors without the code. Replication doesn’t detect speicfication errors if one groups of programmers assumes one interpretation of the specification and another group assumes another. The correct interpretation is revealed unambiguously in the code of the original author.

    There may also be coding bugs that should corrected, no matter how insignificant, if they exist.

    Didactylos wrote @185
    Imperceptible differences in initial conditions can have huge effects on the later trajectory. Correct. I think what the scientists in charge of the probe did was calculate the ideal trajectory, follow the actual trajectory, and if the probe started to deviate from the ideal, give the probe a nudge to get back on the ideal trajectory. There was probably a nudge early on to correct for errors in initial conditions and smaller ones later to correct for unforseen events like solar wind, etc.

    klee12

    Comment by klee12 — 5 Nov 2010 @ 1:51 PM

  188. klee12: I feel you are going in circles. “It doesn’t define what good is”, “How precisely did you choose” – the whole point the report was making is that these issues don’t have a significant effect on the result. Hansen’s various treatments of the UHI effect or Menne et al’s analysis of the surfacestation.org data show that even going to ridiculous extremes in excluding suspect data, the result is not significantly altered.

    Your comments about specifications are absolutely correct for commercial software, and they should probably apply to large and complex scientific undertakings, too. But there’s no way they apply to the typical scientist’s work, when they are either working alone or in a very small group. Yes, specifications are still important. But the “specification” is the important part, the published part. The code validates the specification, not the other way around. Hence, an error in the validating code should not be propagated to an independent validation. That’s what replication is all about.

    You are right that ambiguities still need resolving, but science has mostly managed to resolve ambiguities without publishing every last byte of code. Besides, even if you do have the code, you still don’t know whether the code or the description is correct. Code may be precise and unambiguous, but it is easy to be precise, unambiguous and wrong.

    Comment by Didactylos — 5 Nov 2010 @ 4:35 PM

  189. Didactylos wrote

    the whole point the report was making is that these issues don’t have a significant effect on the result. I shouild have been more clear. I was not looking at the methods as someone who is trying to replicate the code, but as someone who thinks there is something wrong with it. He/She writes the code with one assumption of “good” and the original author has a different interpretation. Their code gives different results. This was meant as an example of what could happen and could be easily avoided by publishing the code. Even if one assumes in this example the errors will be insignificant, there are other cases where the differences are not insignificant. By the way, how do you define “insignifcant”. When I port code, if the differences between my ported code differs from the results reported for by the code I am trying to port cannot explained by round off or similar machine dependent factors I don’t feel comfortable.

    Your comments about specifications are absolutely correct for commercial software,… But there’s no way they apply to the typical scientist’s work, when they are either working alone or in a very small group.

    I agree with that statement, but my point (which I did not emphasize to enough) was that what I call implicit specifications are important and can cause others not to be able to reproduce your results using just methods and algorithms. Evidence for the importance is the efforts to spell them out on large projects and even when immense efforts are expended to write the specifications errors often occur. The code is so much easier than spelling the specifications out. You don’t need to write out the specifications; the specifications are implicitly very obvious in the program … “good” may mean within 3 standard deviations of an average and it’s in the code. Again I’m looking at this from the point of view of a person who thinks the author made a mistake and want to check.

    klee12

    Comment by klee12 — 5 Nov 2010 @ 10:29 PM

  190. klee12 said “By the way, how do you define “insignifcant”.”

    I try to leave that to the statisticians – but speaking informally, I mean “does not alter the conclusions”.

    “Even if one assumes in this example the errors will be insignificant”

    Nobody is assuming anything. In this particular example, the errors ARE insignificant. In other examples, they may not be – but in that case, the very first attempt at replication should identify the problem. Remember that scientists do share code and talk to one another, even if the whole codebase can’t be published.

    “looking at the methods as someone … who thinks there is something wrong with it.”

    Exactly. And it is this assumption of bad faith that really irritates scientists. Then they open up the code, and people STILL call them frauds. Where’s the upside? No. Scientists should be allowed to open up code for their own reasons, not for yours.

    I’m going to ignore your tiresome repetition of “code = perfect specification”, since it doesn’t hold water and it has been explained to you dozens of times now.

    Comment by Didactylos — 6 Nov 2010 @ 10:03 AM

  191. Did 185: I’m surprised to see you grouping yourself with the deniers, though

    BPL: Wake up, and it will all be over. You’ll be back in your bed, ready to go to school.

    Comment by Barton Paul Levenson — 6 Nov 2010 @ 4:56 PM

  192. klee12, OK, let me get this straight. With a BS in physics and a single year of trying to read papers you didn’t understand, you feel you are competent to pronounce judgment on a century and a half of science done by an entire community.

    Dude, I’ve got a PhD in physics and have been actively doing research in physics and it took me 3 years of fairly concerted effort before I felt I really understood the basics of climate science.

    Here’s a place to start: Google “Dunning-Kruger”.

    Comment by Ray Ladbury — 6 Nov 2010 @ 6:28 PM

  193. Ray Ladbury wrote “it took me 3 years of fairly concerted effort before I felt I really understood the basics of climate science.” Me too and I started with the advantage of 50 years, now, as a amateur (hobby) geologist.

    Comment by David B. Benson — 6 Nov 2010 @ 8:40 PM

  194. David Benson,
    I wouldn’t call myself a geologist. I’m a rockhound–nearly every surface in my house would wind up coated with rocks if my wife let me.

    Comment by Ray Ladbury — 6 Nov 2010 @ 9:19 PM

  195. Ray Ladbury @194 — I prefer to leave the rocks where they are and go visiting. I’ve also read a middling amount of the literature, which is what did help in picking up some aspects of climatology.

    Comment by David B. Benson — 6 Nov 2010 @ 9:54 PM

  196. Didactylos @ 190

    Let’s just agree to disagree

    Ray Ladbury @192 wrote
    With a BS in physics and a single year of trying to read papers you didn’t understand
    you feel you are competent to pronounce judgment on a century and a half of science done by an entire community.
    Nope, never said that. I said @184 So I would rate myself somewhere between the average voter and a climate scientist., and I stand by that statement. In @151 I wrote

    Now I feel I’m being invited into discussing climate science. I’m not a climate scientists and I’m don’t want to go there….I don’t know much about climate science and there’s nothing much I can add on that subject. My view, what whatever it’s worth, is that climate is warming, that humans are adding to the warming and that we should reduce CO2 emissions due to the precautionary principle.. You might look at the context also to understand what I was getting at.

    I don’t think paper degrees are relevant in science. What is important is does argument reasonable. If you think I made a mistake, let’s discuss it in a civil manner. If we reach agreement good; if not that’s OK too. I did pontificate a bit on computer science, on which I felt I was on firm ground but I am sure I didn’t express a definitive position on climate science.

    klee12

    Comment by klee12 — 6 Nov 2010 @ 10:24 PM

  197. Klee12,
    At this stage, what I can do is recommend resources to further your education. I can also recommend that forays into philosophy of science (e.g.climate change is less certain than Newton’s laws) wait until you’ve actually done some science so that you see how the process works. Newton’s Laws work great in everyday life. However, the Newtonian model of space and time is falsified, and therefore so are Newton’s equations. The consensus model of Earth’s climate is alive and well and quite unlikely to be “falsified”

    If you have not read Spencer Weart’s history, I recommend you do so. It will give you some idea of how we got where we are. Beyond that, ask questions. This is a complicated field. It takes work to understand it.

    Comment by Ray Ladbury — 7 Nov 2010 @ 7:27 AM

  198. #192 Ray Ladbury

    The only degree I have is a degree of insanity and I still don’t understand much (ok, I did have two years of engineering and electronics, but nothing can prepare you for the real world of climate science ;)

    But what I don’t know, does not prevent me from at least a reasonable understanding of the basics.

    There’s that GHG blocks IR thing
    CO2 and spectra wavelengths
    the H20 thing
    albedo
    land use
    carbon sink
    ocean acidification
    lat shift
    oceans warming
    bunches of letter combination’s (ao, aao, nao, pdo, enso, etc.)
    path divergence and attribution
    CCN stuff (ugh)
    atmospheric lifetimes of different thingys
    seasonal shift
    inter/intra/multi-decadal
    columns, and models, and layers oh my!

    oh, and the um, the, heck what was it, oh yeah, the feedback things, fast and slow…

    Of course, I’m still learning :)

    Economics: Balancing Economies
    The Leading Edge: The Cuccinelli ‘Witch Hunt”

    Fee & Dividend: Learn the IssueSign the Petition
    A Climate Minute: Natural CycleGreenhouse EffectClimate Science HistoryArctic Ice Melt

    Comment by John P. Reisman (OSS Foundation) — 7 Nov 2010 @ 2:15 PM

  199. Hi All

    I find all the talk of making the code available interesting, however as a Systens Project Manager, I invariably find the actual code to be far less interesting that the journey that was undertaken in building the code.

    Are there places where the following is available? Or are you considering making the following available? Its not clear from the discussion..

    Functional Specifications
    High level and detailed designs
    Test scope by phase of testing
    Test conditions for unit, assembly, etc
    Expected results from test runs vs the actuals?
    Defect trend reporting by test phase
    Systems Test closure reports
    Acceptance testing: scope, conditions, expected vs actual, defect reports, and closure

    With all due respect to my colleagues who write the code, in my opinion, making the above available would have a far greater impact than merely the code which is simply one (albeit important!) output of a specification?

    Gerry

    Comment by GerryC — 12 Nov 2010 @ 4:17 AM

  200. It is interesting to see how similar problems are addressed by scientists in widely different disciplines.

    The open/closed code argument spans many disciplines and subdisciplines. Although specific code is relevant to the interpretation of behaviors of the code under different, sometimes slightly so, initial conditions or context, it is not the most important issue with respect to the value of the science that employs it. Nonetheless, it is often the case that different perspectives as to what the relevant issues are may welll be decided by subtle issues such as ill-conditioning, aberrant behavior under certain rare but specific circumstances, etc., such that having the code in hand, if necessary is important.

    Consequently, much of the debate here could be easily addressed not by archiving every line of code or endless debating what should or should not be archived, but rather by providing a registry, where authors could provide either sufficient metadata or code as the need may be to settle substantive differences in interpretation.

    The community need only decide what constitute adequate or sufficient metadata to extract suitable information, and sometimes code, from the registry. Relevant metadata might include descriptors say, original publication, authors, source language, licence issues, method of distribution, website for more information, etc.

    One might view a useful registry as simply a multigraph, with nodes representing the particular issues/formulas/phenomenona and the edges as the implementations (aka code or algorithm). Since there may be a large number of possible ways to code for the same problem, the nodes may have many parallel edges. For a variety of reasons, some edges would be more interesting/useful than others since they may do a etter job at addressing ill-conditioning, computational-efficiency, convergence, etc. Where two papers utilize the same algorithms and/or code they would in principle share the same nodes and edges.

    It may be that the graph may need to evolve/change to either include more or different nodes, depending up need/scope/context for such a registry to remain current or useful. Probably, a very simple vocabulary to differentiate nodes would provide a flexible starting point for such a registry. This would require some standardization for the metadata, so that searches could be usefully constrained and specific algorithms/research could be systematically categorized.

    Authors (registrants) would be only required to meet minimally sufficient requirements in metadata usage to classify their work. Those who register or extract from the registry would be responsible for proofing/usage/interpretation/distribution/etc. beyond the minimal metadata requirements. Consequently, management overhead could be kept to a bare minimum. After all, scientists should be willing to stand on the merits of what they register.

    At that point, one has the framework that can be pointed to address concerns as to precisely what was done and how it was done. The public, as well as the research community, then has an effectively open record and the community has a resource that can be tailored to better share useful methods/implementations/code/algorithms that can be refined as the need arises and resources become available. Such a registry would also have the useful benefit of providing a snapshot as to what methods/code/issues are most used, sources of controversy, etc. Importantly, it would go far to address the public and political perceptions that somehow something is being hidden (there may of course, be perfectly good reasons, proprietary code, copyright issues, patent rights, contractual obligations, etc) that may make totally open source a goal rather than something always achieved.

    Nonetheless, a registry would facilitate clarification and ongoing research, as well as debunk the too often bandied about notion that scientists have something to hide, while leaving the onus on those who do the work to have used appropriate methods/procedures as well as on those who would criticize them.

    Comment by S. G. Poss — 14 Nov 2010 @ 8:50 PM

Sorry, the comment form is closed at this time.

Close this window.

0.708 Powered by WordPress