Propeller Hub

SteveMelton

800K nodes with constraints and loads. I like Mecway.

Sergio

Beautifull model! I preffer to remove small finishing radius/chamfers/features to keep the model/work agile.

prop_design

very nice

JohnM

Nice. If you haven't already tried it, now might be the time to give the PARDISO solver a go. I have found it runs in about 55% of the time for SPOOLES, and seems only limited by the machine memory.

Sergio

@JohnM , how do you run CCX with Pardiso? I'm struggling now with a big model that with spools refuse to solve, even if I have enough resources on my machine. I have downloaded the CCX 2.15 executables from CCX site, and for running with Pardiso there is a note about downloading some Intel DLLs

prop_design

i'd be curious to try this as well. the pardiso site says not to use the intel mkl version as it's out of date. the new pardiso appears to be much faster. the ccx manual seems to indicate you have to compile pardiso yourself. so i may not try it, if it's that involved. there are some downloads for pardiso on their website. not really sure how to get it all working though.

JohnM

PARDISO is nicely hidden in plain site on the Calculix website. On www.dhondt.de, look for Windows executable..... files
Download the Intel MKL from here https://software.intel.com/en-us/mkl/choose-download/windows
Run with ccx_pardiso solver, it will complain about not having some DLLs. I didn't know how to link them so I simply copied them into the same directory as the ccx_pardiso_dynamic.exe, and things work fine. When you search your C: drive for the DLLs, you might find you have multiple copies from different applications (for example, if you run ANSYS). I had to swap one or two to get it right, but no big deal.

Sergio

Great @JohnM , but can you share mkl_intel_lp64.dll ? I didn´t found on my pc after installing MKL

prop_design

hi john,

thanks. i got it running. i wanted to add some notes for anyone else trying this. i went with a slightly different approach, that doesn't requiring copying files to other directories. I already had intel fortran installed. but i didn't have the mkl installed. so i went to the download site john mentioned and downloaded the mkl. i only installed the version for 64 bit windows fortran. but that seemed to be enough. i moved everything in the mecway ccx folder to a backup folder. i then copied the ccx pardiso exe to the mecway ccx folder. i pointed mecway to that exectuable, using the mecway calculix options. i created three paths in the user environment variables. these point to the files that john copied to the mecway ccx folder. the paths are as follows:

C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.3.056\compilers_and_libraries_2019\windows\redist\intel64_win\mkl

C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.3.203\windows\redist\intel64_win\compiler

C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.3.203\windows\mkl\lib\intel64_win

i attached some pics of the steps as well.

most importantly i got a massive speed up. a non-linear solve i did was 57.7% faster. it used one core and two threads. this was the same as the regular ccx that mecway came with. the two threads seems to be meaningless since only one actual core was used in both cases. but still, it was way faster. so thanks a lot john. total time difference was 21.56 min vs 9.12 min for 51,918 nodes. i have 8gb of memory and an amd a9-9400 laptop apu.

p.s. the pardiso exe is in a download on the www.dhondt.de website. it says it's for bconverged. so it's misleading. it's in a zip file with a link that says 'files' it's listed twice as well. so pretty confusing to find.

prop_design

here is a pic of what the download page looks like.

prop_design

it's shocking how fast it is. it's flying through modal analysis that used to take forever. now i'm getting solutions in 1-2 mins. it would be nice if it was the newest version of pardiso. but even the intel mkl version is way way faster than spooles and arpack.

note to victor;

this would be worth licensing from them, if it was reasonably priced. something where you could include the latest exe in the ccx folder.

prop_design

i figured something else out too. if you add OMP_NUM_THREADS to your system environment variables you can run PARDISO as multi-core. I have 2 cores so I set mine to 2. previously, when i was reading the ccx manual i thought that was something you had to do to compile the code. but it's not. just set the system variable and it works. it didn't seem to work for me when i put it in my user variables. i attached a pic.

prop_design

so as a point of comparison when i use two cores the same model gains another 18.52% speedup. so the total speedup from default ccx using 1 core is 65.54%. that's a heck of a lot. and that was just one model. i don't have comparison data for the rest, but all the composite models and modal analysis models run extremely fast. it seems like even more of a speedup compared to the solid model i have comparison data for. again this was for a nonlinear stress with 51,918 nodes. their website says the new version of PARDISO is much faster in multi-threaded performance than the old MKL version. so if you are serious about multi-core speed definitely check out the new version of PARDISO. total run-time summary:

1 core ccx with spooles; 21.56 min
1 core ccx with PARDISO MKL version; 9.12 min
2 cores ccx with PARDISO MKL version; 7.43 min

this model has contact which seems to slow it down. the models i don't have contact on seem to run even faster. the model was the bolt assembly i posted in a different thread. mecway.com/forum/discussion/737/bolt-assembly-example#latest

JohnM

This is why forums work, you get back more than you put in
We run multi-core, although the biggest speedup is in the first 4, and by 10 it's not worth it. Note, this plot is based on SPOOLES, NOT PARDISO, but I suspect a similar behavior.

Sergio

This post should be renamed "CCX with Pardiso for Windows"

Have tested a big model that spooles refuse to solve (even with 32 GB of ram) and with Pardiso was solved easily. Also try another model that took 01:17:07 (hh:mm:ss) to solve with Spooles and now 00:37:37 with Pardiso (half of time!).

Hengre

That sounds very good.

But how to get it running, I just download Intel MKL and tried to get the required dll and lib. I am not really familiar with such libraries. In fact I found many of them but almost older release. The mkl_intel_lp64.dll i did not find. Where to get the DLLs and LIB shown in JohnM's Post above...

Regards

prop_design

look at my posts above. i show pics of where they are. or at least where they are for me. see if you find them there. then add paths in your windows system variables. no need to copy them.

Victor

Thanks very much JohnM and prop_design for revealing this! This is amazing news and I'll include it with the next version of Mecway if the licenses permit, or at least try to ease the installation if they don't.

Sergio

Check this about Intel® Math Kernel Library licencing, I'm not an expert on the matter, but looks like it could be freely distributed.

https://software.intel.com/en-us/mkl/license-faq

prop_design

just to be clear. i wasn't talking about distributing the old intel mkl version of pardiso. i was asking to look into distributing the current version of pardiso. it shows a huge multi-threaded speedup compared to the old intel mkl version. i thought this would be worth looking into for people who have a lot of cores. we can already get the intel mkl version ourselves, after john pointed it out to us.

the pardiso website doesn't say how much it costs or if you have to compile it yourself. so i don't know if this is something victor can do or not. but it's worth a look in my opinion, giving the massive speed up and multi-threaded potential. as a worst case fallback position, distributing the intel mkl version could be an option. but it's probably better for people just to do that themselves.

Sergio

I'm a little confused, the Intel MKL is old? Is there any other newer? Even faster??? On the other side, somebody knows what is using Abaqus???

prop_design

hi sergio. yes it's many years old. i don't know why they aren't using the new version. more info is on the pardiso website; https://www.pardiso-project.org/ it is kind of hard to spot but if you search for intel mkl it will highlight it for you. i don't know what abaqus is using. but a lot of fea providers use the same sources. i wouldn't be surprised if pardiso is one of the available solver types. this is the blurb from the pardiso site:

Important: Please note that the Intel MKL version of PARDISO is based on our version from 2006 and that a lot of new features and improvements of PARDISO are not available in the Intel MKL library.

there is also a chart showing how much faster the new version is. they show a large speedup for a 16 core processor.

Victor

That's a good point prop_design. Though something that got me suspicious - the benchmarks are all on circuit simulation problems. It's especially suspect considering the huge difference between the circuit 1 and circuit 2 graphs. With that much variability, it's a complete guess how they'd compare on an FEA problem.

Apparently for circuit simulation, "The matrices have very peculiar characteristics as compared to sparse matrices arising in other scientific applications. The matrices are extremely sparse and remain so when factorized." https://link.springer.com/chapter/10.1007/978-3-642-22453-9_1 FEA matrices become sort of dense when factorized, so it's possible that the speedups largely take advantage of those special differences.

Just for clarity, I don't mean to say that "real" Pardiso isn't faster than MKL, just that the gains might not be as spectacular as they look. Density in the matrix is bad for parallelization, so that might hurt FEA performance at higher core counts. I'll investigate properly later though.

VMH

JohnM and others, can you put those file in a zipped folder and post them here for us to download please? Thanks!

JohnM

VMH, I'm not sure we're supposed to share that way, but if you install the Intel MKL and then look for those files (I've screenshot so you can see the date stamp and file sizes). You will set this up quick, promise.

VMH

JohnM,
Should we download the full version or a custom version of MKL? Im installing the full version of Intel MKL 2019 Update 3 for Windows right now. Thanks

VMH

I must be downloading and looking at the wrong one. I installed it (screenshot) and found none of the dlls...

disla

MKL 2019 Update 3 for Windows just for FORTRAN is enough not the full collection with C++ . Visual studio integration is not required.
Do not move files . Add to path according to picture 5.png of prop_designprop_design post to locate the files.
Reboot was needed for me.

Thanks JohnM and prop_design. Big improvement for the community.

Hengre

Thanks to all. The forum is a great thing. I finally managed to get the paradiso-solver running after struggling around with the DLL's

I hope Victor is able to include it with the next version of Mecway if the licenses permits...

prop_design

@disla strange that you had to reboot after you added the path variables. i'm using windows 10 home edition. after you add path variables you just have to close that window and then they take affect. no reboot required for me. but you do have to close the window. otherwise they aren't in affect yet.

to those having issues finding the dll files from intel mkl. they are in a picture above. but i'll post the paths here again:

C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.3.056\compilers_and_libraries_2019\windows\redist\intel64_win\mkl

C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.3.203\windows\redist\intel64_win\compiler

as far as choosing c or fortran when installing the mkl. i only chose fortran because i program in that already. if you only program in c you would want to go with that. if you do both then choose both. if you don't program at all i imagine either one would suffice. i'm not sure if you can choose none or not.

VMH

prop_design,

Thanks for your help but it didnt work for me for some reason (see attached)