Professional Documents
Culture Documents
ANSYS, Inc.
Southpointe
275 Technology Drive
Canonsburg, PA 15317
ansysinfo@ansys.com
http://www.ansys.com
(T) 724-746-3304
(F) 724-514-9494
Release 15.0
November 2013
ANSYS, Inc. is
certified to ISO
9001:2008.
Disclaimer Notice
THIS ANSYS SOFTWARE PRODUCT AND PROGRAM DOCUMENTATION INCLUDE TRADE SECRETS AND ARE CONFIDENTIAL AND PROPRIETARY PRODUCTS OF ANSYS, INC., ITS SUBSIDIARIES, OR LICENSORS. The software products
and documentation are furnished by ANSYS, Inc., its subsidiaries, or affiliates under a software license agreement
that contains provisions concerning non-disclosure, copying, length and nature of use, compliance with exporting
laws, warranties, disclaimers, limitations of liability, and remedies, and other provisions. The software products
and documentation may be used, disclosed, transferred, or copied only in accordance with the terms and conditions
of that software license agreement.
ANSYS, Inc. is certified to ISO 9001:2008.
Third-Party Software
See the legal information in the product help files for the complete Legal Notice for ANSYS proprietary software
and third-party software. If you are unable to access the Legal Notice, please contact ANSYS, Inc.
Published in the U.S.A.
Table of Contents
1. Introduction ............................................................................................................................................ 1
2. Parallelization Methodology .................................................................................................................. 3
2.1. Parallelization of Structured Part Calculations .................................................................................... 3
2.2. Parallelization of SPH Calculations ..................................................................................................... 3
2.3. Parallelization of Unstructured Calculations ....................................................................................... 4
2.4. Parallelization of Contact Interactions ............................................................................................... 4
2.5. Parallelization of Euler-Lagrange Coupling Interactions ..................................................................... 5
3. Establishing a Parallel Processing Environment ..................................................................................... 7
3.1. Windows Systems using Platform MPI ............................................................................................... 9
3.1.1. Running Autodyn with MPIRUN ................................................................................................ 9
3.2. Linux Systems using Platform MPI ................................................................................................... 10
3.2.1. Host Configuration Set to Run Autodyn in Parallel ................................................................... 10
4. Creating Domain Decomposition Sets .................................................................................................. 13
4.1. Decomposing Structured Calculations ............................................................................................. 13
4.2. Decomposing SPH Calculations ....................................................................................................... 27
4.3. Decomposing Unstructured Calculations ........................................................................................ 30
4.4. Decomposing Euler-Lagrange Coupled Problems ............................................................................ 35
4.5. Fully Automatic Decomposition on Windows ................................................................................... 38
A. Overview of Parallelized Features and Known Limitations ........................................................................ 41
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
iii
iv
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Note
Although the joins between unstructured parts have been parallelized it is strongly advised
to avoid the use of unstructured parts in combination with joins in parallel analyses.
In Autodyn a new option is available that will merge joined unstructured nodes that reside
at the same physical location in the model into one single unstructured node. The option is
available under the Join menu and will increase robustness in many applications involving
joins.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Introduction
Coupled calculations have been parallelized between the Euler Ideal Gas and Euler multi-material solver
and structured and unstructured Lagrange and Shell solvers.
Interaction calculations have been parallelized between Lagrange, ALE, Shell and SPH solvers, using the
Gap and Trajectory contact algorithm. Parallel simulations using Trajectory contact are subject to the
following restrictions:
The contact method must be set to penalty; the Decomposition Response method is not available in
parallel.
Trajectory contact cannot be run in parallel when an SPH part is present in the model.
Bonded connections are available as a beta option in parallel.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Parallelization Methodology
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Parallelization Methodology
To further enhance the efficiency of coupled calculations the sub-domains of the FE structure will be
placed on the same processor as the Euler sub-domains located in the same geometric space. This decreases the necessary inter-processor communication for the coupling calculations.
Table 2.1: 8 CPU Automatic Decomposition of a Casing Fragmentation Analysis
Note
The multi-material Euler part should not be joined to other Euler Parts.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Whether you intend to run a calculation in serial mode or parallel mode, Autodyn is always started by
activating a single process on one machine. This is done by starting the autodyn process.
If a serial calculation is being performed, this is the only process that is run. It performs all aspects of
the problem (for example, setup, execution, I/O, post processing).
If a parallel calculation is being performed, this process is considered to be the master process and
additional slave processes are started on each CPU used by the parallel system (including the CPU
running the master process). These slave processes are spawned automatically by the master process,
which issues the required system commands to start the adslave process once on each CPU. This
parallel processing environment is shown schematically in the figure above.
In what follows, we assume that the configuration we are implementing comprises of n hosts, each
having m CPUs, and that the hostname of the nth host is hostn. Host1 is special, in that it runs the
master process (autodyn) and is usually the system on which the program files are stored.
When running Autodyn in parallel on a system of networked computers, the program executables and
libraries must be accessible on each of the computers. They can either be installed on all computers in
the system or installed on one computer (usually the one running the master process) and shared with
the others using NFS mounting or Windows Sharing. We recommend sharing as it simplifies the handling
of the files. Setup is further simplified if the shared mount points/folders are the same on each host
(including the master).
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Note
Please refer to the MPI documentation found in your MPI installation (C:\Program Files
(x86)\IBM\Platform-MPI\doc\pcmpi.win.09.01.00.00.rn.pdf for a default installation) for the description of command options such as: -cache, -pwcheck, -pass, and so on.
Please be aware that for the Windows 7 (and higher) OS, you must disable User Account Control
(UAC) on all of the machines being used for the parallel solves. To do so:
1. In the Windows 7 Search box on the Start menu, type UAC and press Enter.
2. Move the UAC settings slider to Never notify, click OK, and then reboot the machine.
The mixture of 32-bit and 64-bit Windows platforms in distributed runs is currently not supported. Both master and slave processes must be on two or more 32-bit machines, or on two or
more 64-bit machines.
When Platform MPI is installed using the ANSYS installer, it will install Platform MPI version 9.1. The installer will set MPI_ROOT to point to this version and will add MPI_ROOT\bin to the path environment
variable.
The above applfile will launch 1 master process, 2 slaves on machine1, and 2 additional slaves on
machine2. If using Windows XP 32-bit, winx64 should be replaced by intel.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
The MPI_WORKDIR text should be modified if the users installation of Autodyn is not in the default
location.
Once the batch file is executed, the Autodyn program starts along with the number of slaves specified
in applfile. The user can now create their model or load an existing one as usual.
/usr/ansys_inc/
nfs
defaults
10
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
(1) Defines the default path for the Autodyn master and slave executables (autodyn.e150 and adslave.e150) to be \\host1\autodyn , where autodyn is a shared directory on host1. When
the master and slave executables reside in the directory /usr/ansys_inc/v150/autodyn/bin/linx64, the share \\host1\autodyn should point to that directory.
(2) Defines the configuration set name that is to be used (office).
(3) Indicates that the following lines, until the next #@PPCFG statement, defines configuration set office.
(There is only one configuration set defined in this example.)
(4) Adds host1 to configuration set office. It has a relative speed of 1000 and uses the default path
for the Autodyn executable (since no ep parameter is defined).
(5) Specifies host1 has 128 Mb of memory, 1 CPU, and is to have 1 task (slave process) started on it.
(6) Adds host2 to configuration set office. It has a relative speed of 500 and /autodyn as the path
for the Autodyn master and slave executables.
(7) Specifies host2 has 256 Mb of memory, 2 CPUs, and is to have 2 tasks (slave processes) started on
it. The operating system will automatically allocate 1 task to each CPU.
The parallel.cfg file is an external file that is read whenever a project is executed in Autodyn. Host
configuration sets are therefore not project dependent (unlike the domain decomposition sets described
in the next chapter). When you create or modify a host configuration set while working on a particular
project, that set becomes available for all projects.
When an Autodyn analysis is started using the command script autodyn150, the host configuration
set defined in the parallel.cfg file is read, applfile is generated automatically and mpirun is
started (similar as is done on Windows operating systems).
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
11
12
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
13
The problem consists of a single Lagrange part (49x100x10 cells) filled with two materials. The steel has
an initial z-velocity of 0.06 cm/microsecond, while the aluminum is initially at rest.
If you run this problem for 100 cycles and then view a velocity vector plot, you will see the screen
above, showing the impact of the steel on the aluminum.
This model already contains Decomposition Set information. The current decomposition is shown in
the Decomposition Set text box.
14
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Presently the current decomposition set is set to None. The current decomposition set can be changed
by using the drop-down list activated by pressing the list button on the right of the text box. The image
below shows a close up of the Parallel panel.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
15
Choose the decomposition set denoted by TWO. Further buttons allowing the modification of the decomposition set will appear.
16
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Select the check box Plot parallel decomposition. The decomposition of the model will now be displayed
(you may need to manually refresh the screen if automatic refresh is not activated):
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
17
This image shows the decomposition of the model into its constitutive sub-domains. The scale illustrates
which sub-domain is assigned to which task. Here the bottom sub-domain is assigned to task 1 and
the top sub-domain assigned to task 2. This is the optimum decomposition for efficient parallel computing over two tasks. The model has been decomposed in the J direction. As the J direction contains
the most elements, this decomposition will result in the smallest amount of information being passed
between tasks.
There are 490 elements on each sub-domain boundary; 49 in the I direction and 10 in the K direction.
If we had decomposed the part in the I direction, there would have been 1000 elements on each subdomain boundary; 100 in the J direction and 10 in the K direction. Obviously there is a computational
overhead associated with the amount of information passed between tasks and therefore each element
on the sub-domain boundaries increases the amount of communicated information and therefore
computational expense.
Before proceeding click Delete to discard the TWO decomposition. The instructions to enable you to
replace it are detailed below.
Click Add in the Decomposition Set section of the Parallel panel.
18
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Additional buttons and information will then appear in the Parallel panel:
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
19
Type TWO in the Selected decomposition text box. This will be the name of the decomposition set
that will be defined. This name is also displayed in the Current text box:
20
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Click Subdomains to activate a popup window in which the decomposition information can be entered.
The following window will appear:
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
21
As we wish to decompose the part into two sub-domains in the J direction, enter 2 in the Number of
J-sections text box. Autodyn will automatically evaluate the optimum dividing plane for the decomposition and enter it in the Dividing plane #1 text box:
22
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
23
The sub-domains can be assigned manually or automatically. If the Automatic option button is selected
then the sub-domains will be assigned at run time. If the model is a very simple one then this course
of action could be taken. Here we will manually assign the sub-domains to the available tasks in order
to become familiar with the process.
We wish to assign the two J-sections of the model to two different tasks. To do this, enter 1 into the
to J-Section text box and click Assign Now:
24
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
These actions have assigned the first J-section sub-domain to the first task.
To assign the second J-section sub-domain to the second task enter 2 into the From J-section, to Jsection, and Assign to task text boxes and click Assign Now.
If you have automatic refresh activated the image of the model on screen should have been updated
to show the two sub-domains on two different tasks:
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
25
We could have balanced the task assignments automatically and viewed the results instantly by selecting
the Balance automatically option button in the Task Assignment window. This results in a change
to the configuration of the window:
26
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
This is an iron Taylor Test problem where the deformation of a cylinder is studied after impacting a wall
at 221 ms-1. The model consists of a single SPH object made up of 18540 SPH nodes. Running the
model to 0.06 ms will produce the following results:
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
27
To decompose the model, click the Parallel button on the Setup toolbar and activate parallel processing
by selecting the appropriate check box.
Following the same method as when decomposing the Lagrange calculation; click Add in the Decomposition Sets section of the Parallel Processing panel. Again enter TWO as the name of the decomposition and click Sub-domains, which will result in the following window being displayed.
28
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
By default, the Automatic option button will be selected. By leaving the default setting, the sub-domains
would be assigned to tasks at runtime. As we wish to view the decomposition, select the Manual option
button, which will result in further options being displayed.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
29
The number of tasks will automatically be placed in the Number of tasks text box and therefore we
only have to click Balance Now in order to assign the SPH nodes to the different tasks. Autodyn will
attempt to assign the nodes in the most efficient configuration possible for parallel processing. After
doing this, click the check button to change the focus back to the main Autodyn window. Select the
Plot parallel decomposition check box and the decomposition will be displayed.
The model can be now efficiently run as a parallel computation.
30
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
This makes two unstructured parts called VOLUME 1 and VOLUME 2. Rename these parts to TARG and
CYLINDER in the parts menu.
Go to the Parallel panel and tick the box to activate parallel processing. Add a decomposition set and
call it TWO:
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
31
You will see that there are two sections within the Decomposition Sets area of the menu (Manual and
Automatic). For unstructured parts, the second section should be used. Clicking Set Decompositions
activates a window in which the information regarding the decomposition can be entered:
32
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
33
34
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
The model consists of an Euler Ideal Gas part (50x20x40) filled with air and two structural parts that
model the soil (29.000 Tetra elements) and vehicle (1650 shell elements). The Euler part is filled with
high pressurized gas in the region of the soil part where the mine is located and the vehicle will be hit
by the blast wave and eroded nodes of the ejected soil. The blast wave interaction is calculated through
Euler-Lagrange Interaction and the interaction of the ejected soil and vehicle is modeled through Lagrange-Lagrange interaction and erosion.
If this problem is run for 2 ms and a combined material/velocity vector plot is selected, you will see the
following view:
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
35
Go to the Parallel Processing panel and select the box to activate parallel processing. Add a decomposition set and call it TWO:
Euler in combination with unstructured parts can automatically be decomposed. Click Set Decomposition.
This activates a window in which the information regarding the decomposition can be entered:
36
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
The unstructured parts within the model and structured Euler Ideal Gas part will be shown in the Parts
to be decomposed window and all are highlighted by default. To decompose the model over two tasks
(task 1 and task 2) simply enter 2 into the Number of Tasks box and click Decompose Now:
Click the check button. Select Plot parallel decomposition in the Parallel Processing window to see
the following automatic decomposition:
The decomposition algorithm automatically produces a decomposition configuration with good load
balancing qualities and minimal inter-processor communication.
To further enhance the efficiency of the coupled calculation the sub-domains of the FE structure are
placed on the same processor as the Euler Ideal Gas sub-domains located in the same geometric space.
This decreases the necessary inter-processor communication for the coupling calculations, as can be
seen below.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
37
Again the sub-domains of the FE structure are placed on the same processor as the Euler Ideal Gas subdomains located in the same geometric space to decrease the necessary inter-processor communication
for the coupling calculations, as can be seen above.
38
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
39
40
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
Solvers
Available
in Parallel?
Structured
Yes
Unstructured
Yes
SPH
Yes
NBS Tet
Yes
Rigid Bodies
Yes
Yes
Limitation
Connections
Joins
Gap
Yes
Trajectory
Yes
Bonded/Breakable
Yes
Spotwelds
Yes
Reinforcement
Beams
Yes
Structured/Unstructured
Yes
SPH-Structured
Yes
Yes
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.
41
Available
in Parallel?
Euler Multi-Mater- Yes
ial
Euler/LagrangeInteraction
Boundary
Conditions
Yes
Limitation
Yes
Flow/Transmit
Analytical Blast
No
42
Autodyn Materials
Yes
Sesame Library
No
Release 15.0 - SAS IP, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates.