Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable. That task just requires the scholar to write something doable over one or a few machines. But then how can that same person test his/her model inside of one of the larger climate models?

I have been doing some research on this, but have not found much. I found some descriptions of how these larger scale models are built from their development documentation. The code is available as open source, but something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough--though I don't have any person experience doing that with a climate model. I have also spoken with some climate folks at Caltech who are working on developing their own large scale climate models, but they could only explain the challenges that they faced with building a flexible meshing scheme, etc., The Caltech folks did not tell me about their experiences using the established large scale climate models.

Hence, I figured I would ask the SE community.

UPDATE: As per some feedback below, I just wanted to explain why this question is posted on ES instead of the Academics.SE. The thought was that Academics.SE is a much more general site across all of academia and many people there don't know about the nuances of numerical computing and the computational setup that goes along with it. Hence, I posted the question on ES where the audience is more familiar with running these types of simulations. I recognize that this question is "soft" however it seemed relevant to others conducting ES and Atmospheric research--especially those who want to do research on this topic and come from other disciplines.

Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable... But then how can that same person test his/her model inside of one of the larger climate models?

Yes, new parameterizations are often developed separately from the full climate model before being added to it. But the longer a parameterization is developed in isolation the more likely it is that it will be conceptually or technically incompatible with the full model. The trick is knowing early on that you want to couple it into the larger model later and to design your parameterization accordingly. In my experience, however modular we aspire to make these models, it can still be quite a pain to couple in parameterization code that’s had a well-established, independent life outside of the climate model. In general though, this is where the shared code repositories and an active community are really useful.

Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.

I remember years ago, as a student, a computer scientist friend of mine avoided doing an industry placement at a climate modelling center because he had such a low opinion of their software. "It's so basic and boring", he said, "They just use Fortran!" But those same things that are off-putting to a computing student are beneficial to the largely self-taught programmers (i.e., physical scientists) who are working with these models. Fortran is a fairly straightforward and safe language to learn and use, with relatively few concepts and gotchas. Compare that with OOP paradigms, which are hard to use well without significant training.

But those are more comments on the portability of the programmers than of the programs. As Simon mentions, the difficult bits of getting a climate model running on new hardware (big or small) tend to be handled by the support staff of that hardware rather than the academic researchers themselves.