This piece of software is a very versatile field generation code, with lot of features that gives the advanced users all the power through the various input files.
This also has the provision for extending / adding new programs and algorithms. The whole MPI part could be used as wrapper around different versions of EM fields (emflds?.for) and various derivatives (derivs.for). This is done by considering the whole setup as a black-box and writing customized emflds and derivs subroutines by following the interfaces provided in this program.
Differences between two versions
Customizing the Makefile
Various input files description
setup
int_values and double_values
data
maxdef.h
Various output files
Important sections of the code:
stream.c
fileoperation.c
getnewjob.c
myodestep.c
loadcommon.for
emflds?.for
Differences
between the Versions:
The main difference between the two versions is the fact of libraries
included linux supports the library g2c which enables input and output
from within the Fortran subroutines. The Sun OS doesn't support the g2c
library so the second version is written making use of F77 library and
working around the shortcomings of Input / Output from within Fortran code
as well as Sine, Cosine function calls.
Customizing the Makefile:
The main variables to be set inside the MAKEFILE are
MPICC
CLINK
CC
F77
This should be pointing to the right versions of MPI compiler, linker (same mpicc compiler in most case), c compiler and the fortran compiler.
CFLAGS
FFLAGS
CLINKFAGS
are all the various options that need to be passed to the compiler while compiling, linking.
CMPI_LIB
CMPI_INC
should point to the directories which have the MPI library and include files.
File setup:
This is the main input file which sets up all the
important parameters needed for the execution of the code.
The format of this file is as hard coded follows: (It needs to be the exact same format no more extra spaces or characters)
NFS=1
output_node_0=0
output_filename=/tmp/output
derivative_type=4
field_type=9
num_fieldlines=1
NZ=10000
NX=1
NY=1
nits=100000
lenx=100000
leny=100000
lenz=100000
dt=0.0001
racc=1e-15
Det_t=2.5e-1
dbob=0.9999
b0=1.0
num_integers=0
num_doubles=0
NFS=1 - Integer value = 0 or 1
1 means NFS is supported every node could read the
same file
0 means that there is no NFS support (i.e.) only
node 0 could read the file
This value is used to choose whether all the
nodes read the input files or
node 0 reads the file and broadcasts the data to
every other node.
output_node_0=0 - Integer value = 0 or 1
1 means all the output files are written into
a single file in node 0.
0 means every node writes the data in a local file for all the jobs that particular node has completed.
output_filename - Character String
If output_node_0 = 0 it will create the
file named <output_filename>+my_rank
on nodes but for node and write all
the output to that file.
If output_node_0 = 1 it will create the file named <output_filename> in node 0 and write all the output to that file.
derivative_type=1 - Integer value = 1 or 2
This is used to select the type of derivative
function to be executed
These are implemented in derivs.for
derivative_type = 1 -> field line
derivative_type = 2 -> charge particles
field_type=1 - Integer value = 1 or 2
This is used to select the type of field calculation
to be done
This parameter chooses which one to be executed
from the various emflds?.for
field_type = 1 -> 1d
filed_type = 2 -> 2d
num_fieldlines=1 - Integer
Number of field lines whose data is
present in the "data" file. All the memory allocation is done based on
this number so it has to be accurate for the proper function of this code
else SEGMENTATION / BUS ERROR would be occurring almost instantaneously.
This number needs to be accurate as the loop inside the data read function
is based on this value.
NZ=512 - Integer, Number of points on the Z axis
NX=1 - Integer, Number of points on the x axis
NY=1 - Integer, Number of points on the y axis
nits=10000 - Integer, Number of iterations
lenx=10000 - Integer, Length in X direction
leny=10000 - Integer, Length in Y direction
lenz=10000 - Integer, Length in Z direction
dt=0.000025 - floating point number, initial guess dt
After Reading in the dt value the
following logic is applied to
determine the actual value of dt.
/* Deciding on the correct value for dt */
if (dt > Det_t || dt == 0.0 )
/* Using default value */
dt = 0.00001 * Det_t;
else
if (dt < 0.0 )
/* Using the input
value of dt as a
fraction of Det_t */
dt = Det_t * abs(dt);
racc=1e-9 - floating point number with exponential
Det_t=0.25e0 - floating point number with exponential
dbob=0.9 - floating point value
b0=1.0 - floating point value
num_integers=5 - Integer, lets the program know that it has to read 5 values from the file int_values.
num_doubles=4 - Integer, lets the program know that it has to read 4 values from the file double_values.
Files int_values and double_values:
These are the two input files to input integer and double values so
that it could be accessed as part of the integer_input and double_input
arrays inside the c functions and as
integer intinput(100)
common /intinput/ intinput
double precision dbleinput(100)
common /dbleinput/ dbleinput
inside the fortran subroutines. The values from integer_input and double_input are copied into intinput and dbleinput inside the fortran subroutine loadcommon.for.
The number of values in these two files has to be specified as num_integers, num_doubles in the setup file
There needs to be a string following the value, which could be used to comment about the values. In case there is no comment just put in couple of '\'s so that it qualifies as a string. The string is just read and discarded its only for the comments by the user for their own understanding.
the typical file would be:
::::::::::::::
int_values
::::::::::::::
-3897694 \\
-100
\\
-200
\\
1
\\
256
\\
::::::::::::::
double_values
::::::::::::::
1.0
\\
0.5
\\
1.0
\\
1.666666667
File data:
data - it SHOULD contain <num_fieldlines> (from the setup
file) rows of entries for the job array.
The structure of the job array is as follows:
The JOB array has nine elements, the dimension of the length array is
setup using the variable "length" (length = 9;) in the code.
JOB array is defined as an array of type DOUBLE (so all the entries
in the data file should end with .0 in case it doesn't have any decimal
values) with dimension = length.
The structure of JOB array is (job[10])
job[0] = job_number
job[1] = job_color
job[2] = current_time
job[3] = job_status
job[4] = main_x0
job[5] = main_y0
job[6] = main_z0
job[7] = vx0
job[8] = vy0
job[9] = vz0
Of these:
job_number is generated automatically to be consistent inside the file reading function.
job_status for all the entries is set to 0 in the file reading function and the rest of the values have to given as input in the data file.
current_time is assigned inside myodestep just before starting computation for that timestep.
i.e.
job_color main_x0 main_y0 main_z0 vx0 vy0 vzo
so the data file SHOULD contain <num_fieldlines> rows of 7 values mentioned above.
It is strongly recommended that you write a piece of code which puts in the data in the required format rather than typing in the values using an editor.
Typical file:
integer NXMAX
parameter ( NXMAX
= 65536 )
integer NYMAX
parameter ( NYMAX
= 65536 )
integer NZMAX
parameter ( NZMAX
= 65536 )
integer NNMAX
parameter ( NNMAX
= (NXMAX+2)*NYMAX )
- This file is included only at the point where the common block is defined. The compiler needs to know the dimension of the array that is common block before at compile time to statically allocate that much of memory. The disadvantage of setting up large values as default is that a lot of memory is wasted even though its never used.
So the following condition should be true
NXMAX >= NX
NYMAX >= NY
NZMAX >= NZ
there would be segmentation faults / incorrect values.
Whenever maxdef.h is modified the program needs to be recompiled. Otherwise you could just modify the values of NX, NY, NZ in the setup file and the various other files and those could be run without recompiling the program.
progress_file:
This file keeps the record of what the system is
currently doing so that the user could find out whether or not the program
has completed. This show the current state of the system.
Data output files:
The output for the streamlines that are computed
are written into the various files based on the input filenames provided
as explained earlier.
stream.c - This is the main program which sets up the MPI environment, defines all the global variables, reads the input files and divides the code the code for the parent and child nodes.
Important function fileoperation.c
fileoperation.c - Contains all the functions which deal with the file reading / writing.
x0_file_read() -> copies the entries from the data file into the job_input_array taking care of the job_number, job_status fields
void c_file_write__( int *dimension, int *n, char *fname, double *data)
-> this file writes into the file named 'fname' the values
from the array data [n][dimension] i.e. 'n' rows of 'dimension' values.
In case of dimensions greater than 2 make sure that the matrix is transposed before this function is called else the output is going to be the transpose of whatever was passed. this is due to the difference in how array is accessed in C and FORTRAN.
void error_printf__(char *display_string)
-> This prints out any error message which is called from
within the Fortran subroutine as call error_printf('Error Message')
Important function getnewjob.c
getnewjob.c - This copies for the values for next 'job' from the job_input_array (which has all the jobs) into the job array.
When all the valid values are read (<num_fieldlines> set of values) from the job_input_array the further requested job arrays are initialized with job_number = -1 and all the field values are set to 0.
The job_number of -1 is checked by the child nodes to quit from the infinite loop.
Important function myodestep.c
myodestep.c - This provides the big delta t increments to the field line calculation. The values at the big delta t are stored into the job_result array to be written to the file together after the whole job has been completed.
When this subroutine completes its execution it sends back to the main program how many entries it has put into the job_result array which is then used as the index to write to the output file.
The main function call is
odestep_( &job[4], &t, &tDet_t, &dt, &nStts );
which does all the number crunching within the FORTRAN subroutine.
1st parameter is the job input array the output is over written on the
job array
t is the initial time step
tDet_t is the final time step
dt is = .00001 * Det_t
nStts is the internal variable to pass around error messages
Important function
loadcommon.for
loadcommon.for:
All the common blocks are loaded through loadcommon.for
and how to modify / customize this software is shown below.
The following is the definition and initialization of the common blocks to be used within all the Fortran subroutines.
The various common blocks are as follows:
The common block which we felt absolutely necessary have been put into
common blocks as follows.
/box/, /constant/, /grids/, /debug/.
The other two blocks /intinput/ and /dbleinput/ each with length 100 that are there to be used by the programmers for customizing their own emflds example of how to use these common blocks is shown in the emflds.for in this program.
c definition of all common blocks for use within
c fortran subroutines
integer NX, NY, NZ
double precision lenx, leny, lenz
common /box/ lenx, leny, lenz, NX, NY,
NZ
double precision eps, b0, dbob
common /constant/ eps, b0, dbob
integer field_type, derivative_type
common /setup/ field_type, derivative_type
double precision delx, dely, delz
common /grids/ delx, dely, delz
c
logical debug
common /debug/ debug
double precision timet1, timet2, currentdeltat
double precision guessdeltat, totalnits,
dt
common /times/ timet1, timet2, currentdeltat,
& guessdeltat, totalnits, dt
integer intinput(100)
common /intinput/ intinput
double precision dbleinput(100)
common /dbleinput/ dbleinput
integer ikind
common /ikind/ ikind
integer i
ikind = 0
NX = NX1
NY = NY1
NZ = NZ1
dbob = dbob1
b0 = b01
lenx = dble(lenx1)
leny = dble(leny1)
lenz = dble(lenz1)
field_type = field_type1
derivative_type = derivative_type1
eps = racc
delx = lenx / dble(NX)
dely = leny / dble(NY)
delz = lenz / dble(NZ)
do i = 1, num_integers
intinput(i) = integer_input(i)
end do
do i = 1, num_doubles
dbleinput(i) = double_input(i)
end do
Important functions emflds1.for
The way to assign the values from the array into variables. The values in initinput and dbleinput array are the same order as int_values and double_values files. The important point here is that the programmer should be able to match these values there is no error checking involved so segmentation faults are possible. There shouldn't be any problem if one puts in enough values and access then within valid limits.
The following line in the emflds1.for needs to be modified to make the modification consistent with other parts of the code.
c This is the place to add the rest of the parameter
c that is needed to be loaded into the common
block
c which is read in from int_values and double_values
iseed_1 = intinput(1)
iseed_2 = intinput(2)
iseed_3 = intinput(3)
nkmin = intinput(4)
nkmax = intinput(5)
eslab = dbleinput(1)
fracx = dbleinput(2)
lambda = dbleinput(3)
alpha = dbleinput(4)