Topic 1.

Direct Solution Methods

1.1. Costs of Basic Storage

Question:

⚫ To evaluate the CSR with varying sparsity and to state the motivation to use CSR.

⚫ To deduce the scalability of ordinary and CSR matrix storage and to predict the matrix size
for which the storage consumption would hit 16GB.

Expected Outcomes

⚫ The CSR method is expected to be heavy on storage consumption for dense matrices, while
storage efficient for sparse matrices.

⚫ The “storage consumption vs matrix size” curves (for both CSR and ordinary matrices) are
expected to exhibit quadric shapes.

Experiment

⚫ Proceeded curve fitting for the “Full Storage Memory” curve in figure 1 (b).

⚫ Calculated the ratios between full and CSR storage consumptions to ensure the curve pattern
of other curves in figure 1(b).

⚫ Calculated matrix size for storage consumption to reach 16GB.

Data Presentation

Figure 1. storage consumption comparison with size fixed (left); Storage comparison with size varied (right).

For curve fitting section, the size was filled with 1 and the sparsity with 100% to determine quadratic
coefficient for full storage memory curve (the blue one in figure 1 (b)). Mean values and standard

1
deviations for the CSR curves were calculated to validate the corresponding curve shapes.
Subsequently, the quadratic coefficients and predicted matrix size N were computed.

Table 1. Datasets for curve fitting (sampled for N>4000)

Sparsity Full Matrix 1% 10% 100%
Mean Ratio ± SD 1±0.00 67.14±0.09 6.84±0.01 0.89±0.00
Quadratic coefficient 7.63×10-6 0.11×10-6 1.12×10-6 8.57×10-6
Size N for 16GB 2048 137492 14008 1823

Observation & Conclusion

⚫ In reference to Figure 1. storage consumption comparison CSR storage is preferred for sparse
matrix manipulation but is heavy on storage consumption for dense matrix cases. The scalability
aligned with this observation.

⚫ As expected, all the curves fit the form 𝑎𝑥 2 for which 𝑎 values shown in Table 1. Datasets for
curve fitting (sampled for N>4000) at 2nd row. The predicted N value for 16GB was computed
referring to the approximated equations.

1.2. Cost of Working storage and Runtime Scalability

Questions:

⚫ Evaluate the effect of sparsity on decomposition (SVD, QR and DenseLU) performances.

⚫ Deduce the effect of matrix size on decomposition performances.

Expected Outcomes:

⚫ The time complexity for all the given decomposition strategy is O(n3) so there should be a
cubic increasing tendency between working time and the matrix size.

Data Presentation:

The memory use values remain constant for different sparsity and were recorded in Table 2. The
additional memory values for various decomposition methods (N=1000) The curve fitting results for figure 2
below were also shown in Table 2. The additional memory values for various decomposition methods (N=1000)

Table 2. The additional memory values for various decomposition methods (N=1000)
Decomposition Strategy SVD QR Dense LU
Memory use (MB) 38.2 0.252 0.00381
Time Complexity O(N3) O(N3) O(N3)
Space Complexity O(N2) O(N) O(N)

2
 Figure 2. Working time vs sparsity plot (a), and run time & memory consumption vs matrix size plots (b, c, d)

Observations & Conclusions:

⚫ Referring to Error! Reference source not found. the working times exhibit negligible
fluctuations in response to the varying sparsity, and the memory use values remain constant
(demonstrates in Table 2. The additional memory values for various decomposition methods). Since
the provided program also yield mutable working time values for fixed sparsity, it’s
reasonable to assert that these fluctuations were not ascribed to sparsity alteration. In
conclusion, there is no difference in performance between sparse and dense matrix with the
same dimension.

⚫ In reference to Table 2, the run time increases in cubic tendency. For SVD, the storage
presents cubic growth pattern while for the other 2 decomposition methods, the storage
consumption scales linearly with matrix size.

1.3. Impact of data reordering

Questions

⚫ Find out whether reordering impose essential effects on the introduced decomposition
performance.

⚫ State whether reordering is worthy to be proceeded.

Experiment

⚫ 4 sparsity values ranging from 25% to 100% are used, matrix size remain fixed at 10%.

3
 ⚫ No notable discrepancies observed for both run time and memory use, so no data recorded.

Observation & Conclusion

⚫ Conclusion: Neither reordering method have effect on storage consumption. Both Reverse
Cuthill-Mckee and Metis library have no essential effect on run time for the 3 decomposition
methods.

⚫ Evidence: There is no change in storage consumption. And the fluctuation pattern in run
time was resembles as mentioned in section 1.2.

⚫ Reordering is valuable for optimizing computation algorithms and its negative impacts on
resource consumption is hardly observable, thus is worthy to be conducted.

Topic 2. Iterative Solution Methods

2.1. Convergence of the basic method

Question

⚫ To explore the impacts of sparsity and matrix size on iteration method performances.

⚫ To explore and compare the characteristics of the iteration methods.

Experiment

⚫ In linear scale, plotted figure 3.

⚫ In logarithmic scale, plotted figure 4 and 5.

Data Presentation

In figure 5 below, the matrix size range for various iterative methods are consistent (all are N = 100,
400, 1600). But not for figure 3 and 4 because one curve may otherwise dominant the plot and make
the data trend on the other curve hardly observable (even for logarithmic scale y-axis).

4
Figure 3. Convergence vs Iteration (Sparsity = 1%)

Figure 4. Convergence vs Iteration (Sparsity = 10%)

5
 Figure 5. Convergence vs Iteration (Sparsity = 100%)

Since the evaluation times are not constant despite keeping all the parameters fixed, their values are
not recorded. The efficiencies of iterative methods are quantified by the final residual value in the
same iteration turns and the iteration taken to reach the residual of 10-6.

Observation & Conclusion

⚫ Observation 1: In reference to figure 3, the Deepest Descent algorithm curve presents

growing residual for larger iteration number when N > 177. The other 3 algorithms present
discernable pattern of convergence.

⚫ Observation 2: In reference to figure 4 and 5,

◼ For Deepest Descent algorithm, the residual exhibit exponential increase during the
iteration process.

◼ For Minimal Residual, the residual curve presents slower rate of convergence when
substantial number of iterations are employed.

◼ For Sparse GMRES, when applied to sparse matrices, it can achieve a rapid reduction
of residual to the order of 10-6 within 45 iterations. And it still managed to reduce the
residual to approximately 10-1 for dense matrices within 200 iterations.

◼ Sparse BiCGSTab is the swiftest algorithm for residual minimization, but there are
potential of sudden increase in residual magnitude particularly for dense matrices.

⚫ Conclusion: In conclusion, the Sparse GMRES is the greatest iterative method to solve
6
 dense matrices while the Sparse BiCGSTab is the optimal choice for practical engineering
contexts where most matrices are highly sparse matrices.

2.2. The impacts of data reordering

Experiment

⚫ Found out the proper matrix size and sparsity to make the iterative methods convergence at
desired iteration number, and recorded data.

Data Presentation

Table 3. Steepest Descent (N = 177, sparsity = 1%)

Reordering Memory Use (MB) Final Residual (10-7) Iteration
None 0 9.40 121
Metis Library 0 8.81 107
Reverse Cuthill McKee 0 9.31 122

Table 4. Minimal Residual (N = 150, sparsity = 1%)

Reordering Memory Use (MB) Final Residual (10-7) Iteration
None 0 9.52 102
Metis Library 0 9.34 127
Reverse Cuthill McKee 0 9.62 88

Table 5. Sparse GMRES (N = 1600, sparsity = 10%)

Reordering Memory Use (MB) Final Residual (10-7) Iteration
None 39 8.37 35
Metis Library 39 9.46 36
Reverse Cuthill McKee 39 8.27 36

Table 6. Sparse BiCGSTab (N = 6400, sparsity = 50%)

Reordering Memory Use (MB) Final Residual (10-7) Iteration
None 625 8.03 51
Metis Library 625 7.32 48
Reverse Cuthill McKee 625 9.79 51

Observation & Conclusion

⚫ There is no observable impact on total memory use by data reordering.

⚫ In general, the Metis Library tend to decrease both final residual and iteration numbers
7
 (except for Sparse GMRES)

⚫ The Reverse Cuthill McKee tend to decrease one attribute while increase another (except for
Sparse BiCGSTab, in which the iteration number kept unchanged).

2.3. Impacts of Preconditioning

Question

⚫ To explore the impact of Incomplete LU Thresholding on accuracy, memory use and

evaluation time.

⚫ To explore the impacts of combination of ILUT and reordering on performances.

Experiment

⚫ Sparse GMRES is selected to test whether it is workable in solving large scale sparse matrix
with preconditioning and reordering employed.

⚫ Selected 4 matrices (N = 1000, 2000; Sparsity = 1%, 10%), printed the convergence value,
run time and memory use in linear scale.

⚫ Employed Metis Library, recorded the corresponding run time and memory use.

Data Presentation

Figure 6. ILUT Preconditioning performance against threshold values.

8
 Figure 7. ILUT Preconditioning performance with Metis Library Reordering Technique.

Observation & Conclusion

⚫ Referring to figure 6, when the threshold is less than 0.1, both run time and memory use
present decreasing tendency while the convergence value exhibits contrast trend, indicating
the trade off between resource consumption and computational accuracy.

⚫ In reference to figure 7, the preconditioning process becomes faster and consumes less
memory when reordering employed. The convergence is still below 10-6, so the reordering
makes large scale matrix solving workable.