Hi Giuseppe,
Can you please edit the file src/alps/osiris/xdrdump.C
There around line 344 you should find:
// destructor closes the stream and file OXDRFileDump::~OXDRFileDump() { xdr_destroy(&xdr_); if(file_) std::fclose(file_); }
can you please insert a call to flush() before the xdr_destroy? It should look like this:
// destructor closes the stream and file OXDRFileDump::~OXDRFileDump() { flush(); xdr_destroy(&xdr_); if(file_) std::fclose(file_); }
Could you please try this and tell me whether it helps?
Matthias
On 9 May 2010, at 17:34, Giuseppe Carleo wrote:
Hi,
I have realized that there's a "problem" with the filesystem of the cluster. The binary checkpoints files are corrupted because only part of them is written on the hard-disk before the job ends. In fact, while on other machines the dimension of the checkpoints files is increased whenever a call to dump<<something is performed, on this machine the files are not increased in size and only rarely something (but NOT everything) is written on the files.
Probably, for efficiency reasons, the writing on the files is postponed later and the data are kept in memory until the next flush.
Now, I was wondering if I could circumvent this problem calling a flush()-like function of the alps::ODump , but I think this function is not implemented. Is it correct?
Thanks again,
Giuseppe
Hi Giuseppe,
It seems that something is written incorrectly on that machine. What happens if you compile and link your code (statically) on another Linux machine and then run it on the bug cluster?
Matthias
On May 5, 2010, at 1:38 PM, Giuseppe Carleo wrote:
It seems that the checkpoint files generated on the cluster have some problems :
If I use checkpoint files generated on another machine, than the simulation on the cluster is correctly resumed. At the end of this resumed simulation, if I try to resume it again I get a message error like before.
If instead I use the checkpoint files of the cluster to resume a simulation on another machine, then I get the same error message as before, plus an extra error message "vector::_M_fill_insert" .
Probably there's some issue with the binary format... don't know.
Giuseppe
Can you read the checkpoint files on other machines, or can you read checkpoint files on other machines on that one?
On May 5, 2010, at 11:09 AM, Giuseppe Carleo wrote:
Hi Matthias,
thank you for your quick answer.
The machine I'm talking about is this one http://www.top500.org/system/9881 , so it is basically a linux cluster with Intel processors.
A typical error message is :
parsing task files ... Loading information about run 1 from file /scratch/cont003/carleo/beta40/N180.task1.out.run1 failed to read array of type double from an IXDRDump Cannot open simulation file /scratch/cont003/carleo/beta40/N180.task1.out.xml.
This issue happens for all the checkpoints, and the checkpoints files exist (i.e., in the previous case both /scratch/cont003/carleo/beta40/N180.task1.out.xml and /scratch/cont003/carleo/beta40/N180.task1.out.run1 exist) and they are not truncated (i.e. at least the .out.xml files correctly end with </MCRUN></SIMULATION> ).
Moreover, the checkpoints file are indeed accessible and have the right permissions (-rw-r-----)... uhm, strange.
Giuseppe
Hi Giuseppe
What type of machine are you using ALPS on? I cannot immediately tell you what the problem might be. Do all files actually exist locally or might the checkpoints not be accessible? Or maybe the file was truncated by the process being killed? Does this happen to all checkpoints or just some?
Matthias
On May 5, 2010, at 10:38 AM, Giuseppe Carleo wrote:
> Hello everybody, > > > I am currently using the ALPS (v. 1.35) scheduler in my QMC code, and everything works pretty well. > > Nonetheless, I've experienced an error when trying to restart my simulations on a HPC machine : > > "failed to read array of type double from an IXDRDump" > > which doesn't allow me to restart any simulation... > > On the other hand, on other machines the simulations are correctly restarted. > I therefore assume that the way I use to restart simulations is correct, i.e. I invoke something like mpirun ./myprogram.o simulation_name.out.xml , > and that the dumping of the internal variables is done correctly in my code. > > I think the error message should be related to some machine-specific issue. > > Do you have suggestions for this problem? > > > Thank you in advance, > > > Giuseppe >