Kb_Lastrsrc

Emulab FAQ: Testbed Operations: What does "Inconsistent lastrsrc in stats record" mean?

Users may occasionally get the following error when trying to swap or modify an experiment:

Inconsistent lastrsrc in stats record for [Experiment: xxx/yyy]!
*** ERROR: swapexp: Preswap failed!
Cleaning up and exiting with status -1 ...

This typically happens to experiments when boss crashes during an experiment swap or modify. It can also happen if there is a software error that causes the swap/modify to terminate early. The abnormal termination leaves the experiment's statistics records in a inconsistent state that must be fixed up by hand. Note that a future software release will include a script to do this, but for now:

First find the experiment index for the experiment. This is on the Show Experiment page, or you can get it with mysql:

mysql> select idx from experiments where pid='xxx' and eid='yyy'

Then get the current resource entry ids from the experiment_stats table using the index your retrieved in the previous statement:

mysql> select rsrcidx,lastrsrc from experiment_stats where exptidx='$idx';

Then some fields from the experiment_resources table for the current resource entry:

mysql> select byswapin,byswapmod,swapin_time from experiment_resources where idx='$rsrcidx';

If byswapin is non-zero, the experiment was in the process of swapping in when it was interrupted:

mysql> update experiment_stats set lastrsrc=NULL where exptidx='$idx';
mysql> update experiments set state='active',expt_locked=NULL,canceled=0 where idx='$idx';

Now, you must swap the experiment out via the web interface. Failure to do so will mess things up much worse! If the experiment belongs to another user, then go into red-dot mode and use the "Force Swap Out" option (in the admin section of the menu) on the Show Experiment page. You are done.

Else if byswapmod is non-zero and swapin_time is zero, the experiment was in the middle of a modify while inactive:

mysql> update experiment_stats set rsrcidx='$lastrsrc',lastrsrc=NULL where exptidx='$idx';
mysql> update experiments set state='swapped',expt_locked=NULL,canceled=0 where idx='$idx';
mysql> delete from experiment_resources where idx='$rsrcidx';

You are done.

Else if byswapmod is non-zero and swapin_time is non-zero, the experiment was in the middle of a modify while it was active:

mysql> update experiment_stats set rsrcidx='$lastrsrc',lastrsrc=NULL where exptidx='$idx';
mysql> update experiments set state='active,expt_locked=NULL,canceled=0 where idx='$idx';
mysql> delete from experiment_resources where idx='$rsrcidx';

Else if you get here, something is really whacked and we will have to help you. Send us email.