ccx out of core?

prop_design

hi guys,

just wondering if you are using the pardiso ccx solver from the ccx website, can it solve out of core? i can solve about 700,000 node model with 15.7gb of memory. if the model gets larger than that then there wouldn't be enough memory, unless it can go out of core. i can run a test, but i thought i would ask first.

thanks,

anthony

MikeMcMullen

Don't quite know about out of core but I have useda Pcie 4.0 SSD for a Windows page file. Probably Ok for offloading the few GB of largely unused other stuff in Memory (some chuncks of Mecway too) but doubtfully cost effective if swapping while solving matrices. Victor mentioned something about a swap file for Calculix (probably not in the version provided), and a large problem option in Mecway. I have not gotten beyond 1.25 million nodes and have not tried Mecway 14 for large problems since I am running mostly non-linear (Calculix solver). Note I have 64 GB of memory so your milage may vary. To the best of my knowledge Calculix is capable of out of core and multi node solutions, but probably requires a special compilation.

prop_design

thanks mike,

doing a google search brought up two old threads from this forum. seems due to licensing you would have to compile it yourself to get the out of core functionality.

MikeMcMullen

Yeah, and I use windows. My last update was a compile with the latest MKL paradiso which required a Linux shell (I used msys2-Shell), a makefile, all the pieces, and a lot of intel files direct from intel. My first compile in many a year. A little time consuming and I know nothing except in a general sense about what is in the makefile, but the compile attempts tell you what is missing, which is usually one of the few dozen intel files (the ones with the liscensing issues why precompiled can't be distributed) or not found. Intel updates their high performance computing MKL files, seemingly at least once a year, and the new ones usually work, but are named slightly differently to distinguish them from older versions. After moving some of these files to where the makefile expected them, and renaming those with changed names it compiled fine and runs fine. It is not quite as fast as Pastix for the problems I run, and lacks documentation, and took about 4-6 hours to chase down, install, and compile. I don't know about out of core, but it doesn;t have some errors due to array size limits that I have encountered with other precompiled versions of Calculix.

Note Problem size is not just nodes but also also their interconnectivity as sparce matrixes usually use a banding technique. I hear contact and other connectivity variations have a big effect too, but so far have not used these. I did find that using hex20, or a model that is thicker WRT number of elements seems to have more effect on problem size than just the number of nodes.

MikeMcMullen

Mecway 14 using intermal solver may be a good choice for linear large problems. I just reran one of my problem sets, and then changed all the elements from Hex8 to Hex20 drastically increasing the problem size. Using 4 cores:

Hex8, 391680 elements, 493545 nodes, max Mem 14.4 GB, 05:28 run time.

Hex 20, 391680 elements, 1871515 nodes, max mem 47.1GB, 01:33:32 run time. The latter also used something under 100GB of temp files, almost all writes.

Doubled the spans,783360 elements, 3743010 nodes, max mem 56GB, ?:??:?? run time The latter also used about 136GB of temp file space, a mix of writes and reads, plus some pagefile use. Reads and writes up to 1 GB/sec.

Maybe later tonight I will try to run a problem bigger yet, Note when running Calculix from Mecway Mecway uses significant memory simultaneous with Calculix.

MikeMcMullen

Last run mentioned was 3:11:33 run time. Trying a slightly larger version of the problem 4588000 nodes failed with a P-2 message at the start of solving the Matrix at 46:00, decreased to 4083375 nodes, but with probably a significantly wider bandwidth due to a finer mesh. About 9 hrs into run at the moment. Running smoothly with enough free mem to check mail etc.

prop_design

thanks for the info and updates. i don't think i want to get into compiling it myself though. i'm just using the pardiso version from the ccx website. it calls the intel mkl pardiso version. i can live without out of core. it would be nice to have though. but not really interested in compiling it to get the feature.

prop_design

i found this online. perhaps it will help someone.

https://software.intel.com/content/www/us/en/develop/articles/how-to-use-ooc-pardiso.html

it explains some of the intel pardiso settings to get out of core engaged. however, i don't know what ccx requires.

MikeMcMullen

OK, Mecway 14 Static 3D,

Out of core of a 1 span prestressed girder bridge with Hex20 elements.

Mostly 8 elements thick for analysis of rail impact loads from a 36000 kg truck.

929180 elements, 4083375 nodes.

Run using 4 cpu's of a Ryzen 3700X with 64 GB of ram, a pagefile in NVME PCI 4.0 SSD, OOC out of core files on a Pcie 3.0 SSD, and a swapfile, all used at one point or other. RAM only running at 2133mhz. Temperatures mostly reasonable.

Never fully used the speed capability of the SSD's.

RAM usage got up to 99% and then pagefile started to be used, mostly to write.

Run time about 15:00:00 (computer hibernated some why not very precise).

Similar to a problem mentioned previously with about 10% more nodes that failed during solve.

I used computer simultaneously for e-mail and browsing when ram usage was not near peak.