Debugging tips and FAQs

Q: why is CARDAMOM_RUN_MDF generating different results on different machines Possibilities: the DALEC model, MCMC or cost function may all behave (for some reason or other) diffetently across your local machine and/or two ore more HPC environments.

Test 1. Forward model mirroring test

To determine whether DALEC forward model or cost function behave differently, take the following steps.

Step 1.

Local run: /path/to/CARDAMOM_RUN_MODEL.exe /path/to/input.nc /path/to/parameters.nc /path/to/output_local.nc Remote run: /path/to/CARDAMOM_RUN_MODEL.exe /path/to/input.nc /path/to/parameters.nc /path/to/output_remote.nc

In brief, same input.nc file, same parameters.nc file, only possible difference is therefore the output.nc files. Make sure you have identical code (branch, commit) and executables in both your local and remote environments.

Step 2. Now compare the contens (e.g. pseudocode below)

PL=ncread(‘output_local.nc’,’POOLS’); PR=ncread(‘output_remote.nc’,’POOLS’); DIFF = sum(PL(:) - PR(:));

Outcome 1. If the POOLS or FLUXES values are different: there is a difference in how the forward model behaves on the different machines

Step 3. If you find diff in pools, here are some ideas on how to pin down where the error happens

  • Look at the first timestep zero , e.g. POOLS(:,t,:), have any differences between PL and PR shown up yet?

  • What about subsequent timesteps? Timestep 1 and 2?

  • Is it one or more pools that diverge by several % points? Which one diverges first?

  • If you do identify a difference (for example Csom in remote is looking unphysical) then skjs

  • Try

Outcome 2. If the POOLS or FLUXES values are the same, but the cost function is different: there is a difference in the following:

Q: why is CARDAMOM_RUN_MODEL writing out different results every time on the same (local) machine?

Likely cause: a defined variable “VAR” (e.g. “double VAR;”) also needs to be initialized (e.g. “double VAR=0;”) if “VAR” is then used in subsequent operations, otherwise the value of “VAR” is incorrectly set by the (seemingly random) previous memory contents

Issue: CARDAMOM can’t find a “starting solution”, all solutions are -inf.

Solution:

MCMC search for EDC=1 starting solution failing

image