The Jenkins Controller is preparing for shutdown. No new builds can be started.
Skipping 303 KB.. Full LogMetric Check mismatch on component 2 of node 29. truth = 1.666660680987628496E+01, test = 1.302834332283042507E+01, relerror = 2.182965932147568622E-01 Metric Check mismatch on component 0 of node 30. truth = 2.196193323763444369E+03, test = 1.716771803144894648E+03, relerror = 2.182965932147551691E-01 Metric Check mismatch on component 1 of node 30. truth = 1.829353730461022224E+03, test = 1.430012043316678955E+03, relerror = 2.182965932147544197E-01 Metric Check mismatch on component 2 of node 30. truth = 1.999879855653390223E+03, test = 1.563312896325438714E+03, relerror = 2.182965932147552246E-01 Metric Check mismatch on component 0 of node 31. truth = 1.190094890997557400E+02, test = 9.303012306905034734E+01, relerror = 2.182965932147566401E-01 Metric Check mismatch on component 1 of node 31. truth = 6.470058435267209518E+01, test = 5.057666720947989347E+01, relerror = 2.182965932147549748E-01 Metric Check mismatch on component 2 of node 31. truth = 8.021489950777682054E+01, test = 6.270426022016508938E+01, relerror = 2.182965932147565291E-01 Metric Check mismatch on component 0 of node 32. truth = 4.505863150243867494E+03, test = 3.522248575053691638E+03, relerror = 2.182965932147629406E-01 Metric Check mismatch on component 1 of node 32. truth = -2.250093819124796482E+03, test = -1.758906003996258278E+03, relerror = 2.182965932147629962E-01 Metric Check mismatch on component 2 of node 32. truth = 1.149413205656357150E+03, test = 8.985002186655149217E+02, relerror = 2.182965932147627464E-01 Metric Check mismatch on component 0 of node 33. truth = 2.268927371394629233E+01, test = 1.773628255967466316E+01, relerror = 2.182965932147577226E-01 Metric Check mismatch on component 1 of node 33. truth = -1.591363406390970248E+01, test = -1.243974196209189387E+01, relerror = 2.182965932147577504E-01 Metric Check mismatch on component 2 of node 33. truth = 2.206118162317572029E+01, test = 1.724530083254444435E+01, relerror = 2.182965932147575838E-01 Metric Check mismatch on component 0 of node 34. truth = 1.666660551160403259E+01, test = 1.302834230796654147E+01, relerror = 2.182965932147593047E-01 Metric Check mismatch on component 1 of node 34. truth = 6.799028609804064871E+00, test = 5.314823827114138588E+00, relerror = 2.182965932147619414E-01 Metric Check mismatch on component 2 of node 34. truth = 1.761217837752572990E+01, test = 1.376749983862121240E+01, relerror = 2.182965932147595267E-01 Metric Check mismatch on component 0 of node 35. truth = 4.142460838077022345E+01, test = 3.238175749599254516E+01, relerror = 2.182965932147586108E-01 Metric Check mismatch on component 1 of node 35. truth = 1.678456102524437910E+01, test = 1.312054853482829309E+01, relerror = 2.182965932147599153E-01 Metric Check mismatch on component 2 of node 35. truth = 4.142460866722004198E+01, test = 3.238175771991138419E+01, relerror = 2.182965932147576671E-01 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 3.966u 0.078s 0:04.11 98.0% 0+0k 56+6992io 0pf+0w Thu Mar 24 13:21:03 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CutCell_Circle_Interface_Scalar2d_UniAdapt_Test.job Start 802: CutCell_Interface_StarShape_Jump_TEST 265/327 Test #787: CutCell_rae2822_RANS_M0p3_Re6p5e6_UniAdapt_Test .........***Failed 84.87 sec OUT/CutCell_rae2822_RANS_M0p3_Re6p5e6_UniAdapt_Test > ERROR: Test failed for CutCell_rae2822_RANS_M0p3_Re6p5e6_UniAdapt_Test.job in column 1 > New: 1337 Orig: 1586 > > Truth file: 1586 3.216506876937e-03 1.966878997711e-03 3.158847149292E-03 > New file: 1337 1.687939965323e-02 3.253716561139e-03 5.952021974424E-03 > > Truth file: trueOUT/CutCell_rae2822_RANS_M0p3_Re6p5e6_UniAdapt_Test.hist.TRUE > New file: OUT/CutCell_rae2822_RANS_M0p3_Re6p5e6_UniAdapt_Test.hist Start 805: CutCell_rae2822_CHT_TEST 266/327 Test #796: CutCell_rae2822_RANS_M0p3_Re6p5e6_TEST ..................***Failed 65.44 sec OUT/CutCell_rae2822_RANS_M0p3_Re6p5e6_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 7155 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CutCell_rae2822_RANS_M0p3_Re6p5e6_TEST.job Metric Check mismatch on component 2 of node 450. truth = 1.481568698279667115E+04, test = 1.117919421845355646E+04, relerror = 2.454488116930150787E-01 Metric Check mismatch on component 0 of node 451. truth = 4.391065429150638977E+01, test = 5.221470027379621115E+01, relerror = 1.891123262969932051E-01 Metric Check mismatch on component 1 of node 451. truth = 6.025438781920693998E+01, test = 6.212205797727632017E+01, relerror = 3.099641745051526232E-02 Metric Check mismatch on component 2 of node 451. truth = 4.747791285535375891E+02, test = 3.829049706649583413E+02, relerror = 1.935092601237192456E-01 Metric Check mismatch on component 0 of node 452. truth = 1.404287390810646002E+03, test = 1.125538001267934760E+03, relerror = 1.984988196624046797E-01 Metric Check mismatch on component 1 of node 452. truth = -1.502407119831560067E+03, test = -9.666101460770804579E+02, relerror = 3.566256886579116103E-01 Metric Check mismatch on component 2 of node 452. truth = 1.614029320680671481E+04, test = 1.338686144876547951E+04, relerror = 1.705936641151012678E-01 Metric Check mismatch on component 0 of node 453. truth = 1.829669544989493943E+03, test = 1.491565768847509844E+03, relerror = 1.847895304744363032E-01 Metric Check mismatch on component 1 of node 453. truth = -2.257239398764870657E+03, test = -1.679812630648089907E+03, relerror = 2.558110444256557203E-01 Metric Check mismatch on component 2 of node 453. truth = 3.996506289138044667E+04, test = 3.137986228993423720E+04, relerror = 2.148176427190820748E-01 Metric Check mismatch on component 0 of node 454. truth = 8.706288688959704132E+02, test = 6.695633399832864825E+02, relerror = 2.309428691098328601E-01 Metric Check mismatch on component 1 of node 454. truth = 1.045956703259466167E+03, test = 8.858950044122545933E+02, relerror = 1.530289909213438382E-01 Metric Check mismatch on component 2 of node 454. truth = 1.053840855487599947E+04, test = 8.946993166141188340E+03, relerror = 1.510109785977581665E-01 Metric Check mismatch on component 0 of node 455. truth = 6.977735353274540557E+02, test = 5.779302654231482848E+02, relerror = 1.717509533348312301E-01 Metric Check mismatch on component 1 of node 455. truth = 7.172883300415437589E+02, test = 7.328527321847562916E+02, relerror = 2.169894795627174897E-02 Metric Check mismatch on component 2 of node 455. truth = 7.447014472272594503E+03, test = 7.518803895883793302E+03, relerror = 9.640027406753638381E-03 Metric Check mismatch on component 0 of node 456. truth = 7.510346888196268083E+02, test = 6.561173050851623429E+02, relerror = 1.263821567065598062E-01 Metric Check mismatch on component 1 of node 456. truth = -1.505364326921988720E+02, test = -1.424232464668901059E+02, relerror = 5.389516730410211925E-02 Metric Check mismatch on component 2 of node 456. truth = 2.656683270995679322E+03, test = 2.397401898568798970E+03, relerror = 9.759589156057212056E-02 Metric Check mismatch on component 0 of node 457. truth = 1.105692422249220499E+03, test = 9.906727562948997274E+02, relerror = 1.040250106085972709E-01 Metric Check mismatch on component 1 of node 457. truth = 3.701924270624466118E+01, test = 7.038119831083534450E+01, relerror = 9.012057828768863521E-01 Metric Check mismatch on component 2 of node 457. truth = 3.104687015411306220E+03, test = 3.183713081078746654E+03, relerror = 2.545379462572691306E-02 Metric Check mismatch on component 0 of node 458. truth = 7.063371558283984086E+03, test = 5.126301109073507178E+03, relerror = 2.742416186415485568E-01 Metric Check mismatch on component 1 of node 458. truth = 6.085003449151140558E+03, test = 4.148741872248964683E+03, relerror = 3.182022151807136212E-01 Metric Check mismatch on component 2 of node 458. truth = 2.819347573364639175E+04, test = 2.146338021888741787E+04, relerror = 2.387110967920569871E-01 Metric Check mismatch on component 0 of node 459. truth = 1.645180029624599001E+03, test = 1.229680042682792873E+03, relerror = 2.525559388394812421E-01 Metric Check mismatch on component 1 of node 459. truth = -2.394715312270318918E+03, test = -1.751932759861352451E+03, relerror = 2.684171054134923406E-01 Metric Check mismatch on component 2 of node 459. truth = 4.310084231396203540E+04, test = 3.387598320145981415E+04, relerror = 2.140296712835686732E-01 Metric Check mismatch on component 0 of node 460. truth = 1.841196594039171487E+03, test = 1.856426585517460580E+03, relerror = 8.271789947687180344E-03 Metric Check mismatch on component 1 of node 460. truth = 1.853254894865375263E+03, test = 1.295052040630945839E+03, relerror = 3.012013381327011685E-01 Metric Check mismatch on component 2 of node 460. truth = 1.129564515474064683E+04, test = 8.571042258994577423E+03, relerror = 2.412082584413149933E-01 Metric Check mismatch on component 0 of node 461. truth = 6.476315182588679818E+03, test = 4.969251360315380225E+03, relerror = 2.327039033438307314E-01 Metric Check mismatch on component 1 of node 461. truth = 5.335597066480411740E+03, test = 3.705076313775488416E+03, relerror = 3.055929322227633471E-01 Metric Check mismatch on component 2 of node 461. truth = 5.937826214581970271E+04, test = 4.316518045496968261E+04, relerror = 2.730474268686800543E-01 Metric Check mismatch on component 0 of node 462. truth = 2.520826276199800759E+03, test = 2.163965758622390695E+03, relerror = 1.415648991549647295E-01 Metric Check mismatch on component 1 of node 462. truth = 2.848467996848785788E+03, test = 2.864668560944451201E+03, relerror = 5.687465723184475033E-03 Metric Check mismatch on component 2 of node 462. truth = 1.019814344216213613E+04, test = 9.898886689937618939E+03, relerror = 2.934423838238060106E-02 Metric Check mismatch on component 0 of node 466. truth = 2.373544357570443708E+02, test = 3.800348021781063039E+02, relerror = 6.011278700816417020E-01 Metric Check mismatch on component 1 of node 466. truth = 7.370271817566855077E+01, test = -5.060512034177548912E-01, relerror = 1.006866113163039644E+00 Metric Check mismatch on component 2 of node 466. truth = 3.750470774711446893E+02, test = 3.602998413132917221E+02, relerror = 3.932102672893801609E-02 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3798 Call : PXAdjustGridHP(pxa, &pg1, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 60.333u 3.887s 1:04.22 99.9% 0+0k 32+62176io 0pf+0w Thu Mar 24 13:21:43 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CutCell_rae2822_RANS_M0p3_Re6p5e6_TEST.job Start 808: CutCell_bump_Euler_Test 267/327 Test #805: CutCell_rae2822_CHT_TEST ................................***Failed 89.84 sec OUT/CutCell_rae2822_CHT_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 8118 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CutCell_rae2822_CHT_TEST.job truth = 6.923966466148887093E+02, test = 5.892523909558784680E+02, relerror = 1.489670063586820414E-01 Metric Check mismatch on component 1 of node 291. truth = 4.580455732768210169E+02, test = 3.898114727565196631E+02, relerror = 1.489679291782259052E-01 Metric Check mismatch on component 2 of node 291. truth = 5.977638879841014386E+03, test = 5.087178015222934846E+03, relerror = 1.489653159914141323E-01 Metric Check mismatch on component 0 of node 292. truth = 9.283799126907690606E+02, test = 7.900827586242182861E+02, relerror = 1.489661206323576459E-01 Metric Check mismatch on component 1 of node 292. truth = 2.326327089361353501E+03, test = 1.979783149354057969E+03, relerror = 1.489661284486147874E-01 Metric Check mismatch on component 2 of node 292. truth = 1.173772990814394871E+04, test = 9.989206364085412133E+03, relerror = 1.489660741678307532E-01 Metric Check mismatch on component 0 of node 293. truth = 1.086965280667019215E+03, test = 9.250443255289837907E+02, relerror = 1.489660783265056854E-01 Metric Check mismatch on component 1 of node 293. truth = 4.126775351159791398E+02, test = 3.512025761037307348E+02, relerror = 1.489660904245040662E-01 Metric Check mismatch on component 2 of node 293. truth = 1.201575716102169054E+03, test = 1.022581714225809833E+03, relerror = 1.489660613789730581E-01 Metric Check mismatch on component 0 of node 294. truth = 5.507142148917448594E+02, test = 4.686761298379346954E+02, relerror = 1.489667105650007151E-01 Metric Check mismatch on component 1 of node 294. truth = -6.603385426259691826E+01, test = -5.619664483015228740E+01, relerror = 1.489722134547074306E-01 Metric Check mismatch on component 2 of node 294. truth = 3.938397215818411610E+02, test = 3.351711235399233715E+02, relerror = 1.489656701113787063E-01 Metric Check mismatch on component 0 of node 295. truth = 5.609220082469718136E+02, test = 4.773636787379039674E+02, relerror = 1.489660385589246239E-01 Metric Check mismatch on component 1 of node 295. truth = -6.092091478901846813E+01, test = -5.184586634985979714E+01, relerror = 1.489644151041955300E-01 Metric Check mismatch on component 2 of node 295. truth = 2.274198515207751370E+03, test = 1.935420065048167316E+03, relerror = 1.489660853677217955E-01 Metric Check mismatch on component 0 of node 296. truth = 3.851725904346887091E+03, test = 3.277949445345663662E+03, relerror = 1.489660669658982584E-01 Metric Check mismatch on component 1 of node 296. truth = 1.879614898392194618E+03, test = 1.599616059916280392E+03, relerror = 1.489660667807127248E-01 Metric Check mismatch on component 2 of node 296. truth = 1.497390817333951873E+03, test = 1.274330397132738653E+03, relerror = 1.489660665866536504E-01 Metric Check mismatch on component 0 of node 297. truth = 5.645588882446033949E+02, test = 4.804621472434005796E+02, relerror = 1.489600868081040197E-01 Metric Check mismatch on component 1 of node 297. truth = -1.525186047701971574E+02, test = -1.298060147767892829E+02, relerror = 1.489168487190752344E-01 Metric Check mismatch on component 2 of node 297. truth = 5.276966214567617044E+02, test = 4.490886317520173634E+02, relerror = 1.489643604079532724E-01 Metric Check mismatch on component 0 of node 298. truth = 7.612860821336396384E+02, test = 6.478806060673133516E+02, relerror = 1.489656500070082323E-01 Metric Check mismatch on component 1 of node 298. truth = -2.453603964979413377E+02, test = -2.088095042712410248E+02, relerror = 1.489681821043478405E-01 Metric Check mismatch on component 2 of node 298. truth = 1.172836287049378598E+03, test = 9.981226550543661915E+02, relerror = 1.489667687845478705E-01 Metric Check mismatch on component 0 of node 299. truth = 2.685564071442015575E+02, test = 2.285507955905314077E+02, relerror = 1.489653960562151436E-01 Metric Check mismatch on component 1 of node 299. truth = 1.981345273908617060E+02, test = 1.686198577032966739E+02, relerror = 1.489627783518073512E-01 Metric Check mismatch on component 2 of node 299. truth = 2.413359808450362380E+03, test = 2.053850432818783247E+03, relerror = 1.489663391147725280E-01 Metric Check mismatch on component 0 of node 300. truth = 7.038604076714614166E+02, test = 5.990098003527604078E+02, relerror = 1.489650592303833343E-01 Metric Check mismatch on component 1 of node 300. truth = 1.503985479985478264E+02, test = 1.279945232144763878E+02, relerror = 1.489643688866448545E-01 Metric Check mismatch on component 2 of node 300. truth = 1.408179286878325911E+02, test = 1.198408029705605173E+02, relerror = 1.489662993394434698E-01 Metric Check mismatch on component 0 of node 301. truth = 2.684115283096211897E+03, test = 2.284273168944292593E+03, relerror = 1.489660733538571424E-01 Metric Check mismatch on component 1 of node 301. truth = -1.968336320985236171E+03, test = -1.675120950255179423E+03, relerror = 1.489660926356782267E-01 Metric Check mismatch on component 2 of node 301. truth = 8.755213499629764556E+03, test = 7.450983763864089269E+03, relerror = 1.489660687110403170E-01 Metric Check mismatch on component 0 of node 302. truth = 6.721621163655602231E+02, test = 5.720327703954446861E+02, relerror = 1.489660656740426359E-01 Metric Check mismatch on component 1 of node 302. truth = -1.676282992018614948E+02, test = -1.426573725188601713E+02, relerror = 1.489660564588250746E-01 Metric Check mismatch on component 2 of node 302. truth = 3.221531262944619129E+02, test = 2.741632427013101960E+02, relerror = 1.489660651291860594E-01 Metric Check mismatch on component 0 of node 303. truth = 2.260663711003672233E+03, test = 1.923900566174517962E+03, relerror = 1.489664929772507862E-01 Metric Check mismatch on component 1 of node 303. truth = -2.923085061414728443E+03, test = -2.487643983223550549E+03, relerror = 1.489662698971993271E-01 Metric Check mismatch on component 2 of node 303. truth = 5.431385379172764260E+03, test = 4.622294041545675100E+03, relerror = 1.489659232669509337E-01 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 87.689u 0.816s 1:28.53 99.9% 0+0k 72+69312io 0pf+0w Thu Mar 24 13:22:41 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CutCell_rae2822_CHT_TEST.job Start 811: CHT_NS2d_Scalar2d_FullEmb_FlatSlab_TEST 268/327 Test #784: RAE2822_RANS_Transonic_UniAdapt_TEST ....................***Failed 184.09 sec OUT/RAE2822_RANS_Transonic_UniAdapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 925 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for RAE2822_RANS_Transonic_UniAdapt_TEST.job truth = 1.016980858155724127E+04, test = 9.106784786083439030E+03, relerror = 1.045274143509029446E-01 Metric Check mismatch on component 1 of node 283. truth = -3.199515036938310459E+03, test = -2.865078193815863870E+03, relerror = 1.045273546963782635E-01 Metric Check mismatch on component 2 of node 283. truth = 3.901293326843109389E+04, test = 3.493501293981586059E+04, relerror = 1.045273960959774567E-01 Metric Check mismatch on component 0 of node 284. truth = 1.703744502781036545E+03, test = 1.525656516394298023E+03, relerror = 1.045274019056519343E-01 Metric Check mismatch on component 1 of node 284. truth = 1.169526156020887356E+03, test = 1.047278625471318492E+03, relerror = 1.045274019056531140E-01 Metric Check mismatch on component 2 of node 284. truth = 2.608047459003791118E+03, test = 2.335435034067487322E+03, relerror = 1.045274019056520315E-01 Metric Check mismatch on component 0 of node 285. truth = 2.662666776849584949E+03, test = 2.384345136098308558E+03, relerror = 1.045274020659021780E-01 Metric Check mismatch on component 1 of node 285. truth = -3.289488802432673538E+02, test = -2.945647083789492058E+02, relerror = 1.045274020659077568E-01 Metric Check mismatch on component 2 of node 285. truth = 3.086410075481748208E+03, test = 2.763795648581612568E+03, relerror = 1.045274020659033021E-01 Metric Check mismatch on component 0 of node 286. truth = 1.456283951659389686E+03, test = 1.304062373718586059E+03, relerror = 1.045274019310258318E-01 Metric Check mismatch on component 1 of node 286. truth = -3.046909443271192686E+02, test = -2.728423915246954152E+02, relerror = 1.045274019310233476E-01 Metric Check mismatch on component 2 of node 286. truth = 3.422919832183384187E+03, test = 3.065130915107071360E+03, relerror = 1.045274019310260538E-01 Metric Check mismatch on component 0 of node 287. truth = 2.095801918012887654E+03, test = 1.876733188133211570E+03, relerror = 1.045274021351138838E-01 Metric Check mismatch on component 1 of node 287. truth = -2.177298943393021545E+03, test = -1.949711541168623171E+03, relerror = 1.045274021351127181E-01 Metric Check mismatch on component 2 of node 287. truth = 5.458322334493050221E+03, test = 4.887778080852429412E+03, relerror = 1.045274021351124405E-01 Metric Check mismatch on component 0 of node 288. truth = 7.951354940921319212E+02, test = 7.120220434601983470E+02, relerror = 1.045274060200653832E-01 Metric Check mismatch on component 1 of node 288. truth = 1.334612702008321889E+02, test = 1.195109089273824736E+02, relerror = 1.045274127277317638E-01 Metric Check mismatch on component 2 of node 288. truth = 5.437932748108758005E+02, test = 4.869519756614443509E+02, relerror = 1.045274036704483128E-01 Metric Check mismatch on component 0 of node 289. truth = 1.230630569509427914E+04, test = 1.101995955208422492E+04, relerror = 1.045274004141499985E-01 Metric Check mismatch on component 1 of node 289. truth = -4.804797756577087057E+03, test = -4.302564737566360236E+03, relerror = 1.045274004141466678E-01 Metric Check mismatch on component 2 of node 289. truth = 1.148479463058114925E+04, test = 1.028431890355611358E+04, relerror = 1.045274004141499985E-01 Metric Check mismatch on component 0 of node 290. truth = 3.859727670585236046E+04, test = 3.456280364863135765E+04, relerror = 1.045274019710636937E-01 Metric Check mismatch on component 1 of node 290. truth = -3.191322341866142233E+04, test = -2.857741708618663688E+04, relerror = 1.045274019710636798E-01 Metric Check mismatch on component 2 of node 290. truth = 9.235316950514665223E+04, test = 8.269973263298034726E+04, relerror = 1.045274019710643321E-01 Metric Check mismatch on component 0 of node 291. truth = 1.035534627261818741E+03, test = 9.272928829084405606E+02, relerror = 1.045274020817567040E-01 Metric Check mismatch on component 1 of node 291. truth = 1.911690443825952457E+02, test = 1.711866409483115490E+02, relerror = 1.045274013835211235E-01 Metric Check mismatch on component 2 of node 291. truth = 8.232463975169185915E+02, test = 7.371945905378930775E+02, relerror = 1.045274018065254357E-01 Metric Check mismatch on component 0 of node 292. truth = 7.106963952146115844E+03, test = 6.364091473365720958E+03, relerror = 1.045274021062210373E-01 Metric Check mismatch on component 1 of node 292. truth = -7.926211176842155510E+03, test = -7.097704913981536265E+03, relerror = 1.045274021062230357E-01 Metric Check mismatch on component 2 of node 292. truth = 1.764225408999219144E+04, test = 1.579815510266742785E+04, relerror = 1.045274021062225778E-01 Metric Check mismatch on component 0 of node 293. truth = 8.075052147367312045E+02, test = 7.230987890929613968E+02, relerror = 1.045274062673249849E-01 Metric Check mismatch on component 1 of node 293. truth = 4.977556139270888025E+02, test = 4.457265114609535885E+02, relerror = 1.045274046346696406E-01 Metric Check mismatch on component 2 of node 293. truth = 7.618403944042436251E+02, test = 6.822071948770793597E+02, relerror = 1.045274050996431309E-01 Metric Check mismatch on component 0 of node 294. truth = 7.623982598586401309E+02, test = 6.827067504064756349E+02, relerror = 1.045274020784628666E-01 Metric Check mismatch on component 1 of node 294. truth = -5.624258860787680305E+02, test = -5.036369693452764977E+02, relerror = 1.045274020784635743E-01 Metric Check mismatch on component 2 of node 294. truth = 1.232447720360787798E+03, test = 1.103623161953949875E+03, relerror = 1.045274020784635882E-01 Metric Check mismatch on component 0 of node 295. truth = 7.986590270767153470E+02, test = 7.151772745230189230E+02, relerror = 1.045274012105763956E-01 Metric Check mismatch on component 1 of node 295. truth = 3.321461291188525706E+02, test = 2.974277574199058449E+02, relerror = 1.045274012105779360E-01 Metric Check mismatch on component 2 of node 295. truth = 7.270823152590056679E+02, test = 6.510822903788121039E+02, relerror = 1.045274012105773392E-01 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 182.860u 0.207s 3:02.99 100.0% 0+0k 40+12880io 0pf+0w Thu Mar 24 13:22:43 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for RAE2822_RANS_Transonic_UniAdapt_TEST.job Start 814: naca_Euler2d_Newton_ILU_PARALLEL_TEST_2 269/327 Test #814: naca_Euler2d_Newton_ILU_PARALLEL_TEST_2 ................. Passed 2.68 sec Start 816: naca_Euler2d_Newton_ILUK_PARALLEL_TEST_2 270/327 Test #816: naca_Euler2d_Newton_ILUK_PARALLEL_TEST_2 ................ Passed 2.66 sec Start 818: naca_Euler2d_Newton_ILUT_PARALLEL_TEST_2 271/327 Test #818: naca_Euler2d_Newton_ILUT_PARALLEL_TEST_2 ................ Passed 2.91 sec Start 820: naca_Euler2d_Newton_UMFPACK_PARALLEL_TEST_2 272/327 Test #820: naca_Euler2d_Newton_UMFPACK_PARALLEL_TEST_2 ............. Passed 3.81 sec Start 822: naca_Euler2d_LeanJacobi_PARALLEL_TEST_2 273/327 Test #822: naca_Euler2d_LeanJacobi_PARALLEL_TEST_2 ................. Passed 5.02 sec Start 824: ScalarSin3d_HDG_BDDC_ILUK_MG_TEST_4 274/327 Test #824: ScalarSin3d_HDG_BDDC_ILUK_MG_TEST_4 ..................... Passed 4.11 sec Start 826: ScalarSin3d_HDG_BDDC_UMFPACK_TEST_4 275/327 Test #826: ScalarSin3d_HDG_BDDC_UMFPACK_TEST_4 ..................... Passed 6.22 sec Start 828: naca_Euler2d_Newton_ILUK_ASM_PARALLEL_TEST_4 276/327 Test #828: naca_Euler2d_Newton_ILUK_ASM_PARALLEL_TEST_4 ............ Passed 2.61 sec Start 830: naca_Euler2d_Newton_ILU_ASM_PARALLEL_TEST_4 277/327 Test #830: naca_Euler2d_Newton_ILU_ASM_PARALLEL_TEST_4 ............. Passed 2.57 sec Start 832: naca_Euler2d_Newton_ILUK_ASM_MG_PARALLEL_TEST_4 278/327 Test #832: naca_Euler2d_Newton_ILUK_ASM_MG_PARALLEL_TEST_4 ......... Passed 2.61 sec Start 834: naca_Euler2d_Newton_ILU_ASM_MG_PARALLEL_TEST_4 279/327 Test #834: naca_Euler2d_Newton_ILU_ASM_MG_PARALLEL_TEST_4 .......... Passed 2.63 sec Start 836: naca_Euler2d_Newton_Jacobi_Right_TEST_2 280/327 Test #836: naca_Euler2d_Newton_Jacobi_Right_TEST_2 ................. Passed 4.51 sec Start 838: naca_Euler2d_Newton_Jacobi_TEST_2 281/327 Test #838: naca_Euler2d_Newton_Jacobi_TEST_2 ....................... Passed 4.24 sec Start 840: square_Scalar2d_GMRES_Jacobi_TEST_2 282/327 Test #840: square_Scalar2d_GMRES_Jacobi_TEST_2 ..................... Passed 2.61 sec Start 842: square_Scalar2d_GMRES_ILU_TEST_3 283/327 Test #842: square_Scalar2d_GMRES_ILU_TEST_3 ........................ Passed 2.61 sec Start 844: MixedMesh_Scalar2d_Newton_GMRES_Jacobi_TEST_2 284/327 Test #802: CutCell_Interface_StarShape_Jump_TEST ...................***Failed 157.43 sec OUT/CutCell_Interface_StarShape_Jump_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 8040 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CutCell_Interface_StarShape_Jump_TEST.job truth = 6.584132465019368396E+01, test = 6.436148563461921412E+01, relerror = 2.247583904844959249E-02 Metric Check mismatch on component 1 of node 243. truth = 9.394388767120011252E+00, test = 9.183241997231659681E+00, relerror = 2.247583904844952657E-02 Metric Check mismatch on component 2 of node 243. truth = 6.630375722621892010E+01, test = 6.481352465049496914E+01, relerror = 2.247583904844925942E-02 Metric Check mismatch on component 0 of node 244. truth = 4.597034824024769506E+01, test = 4.493712609219905119E+01, relerror = 2.247583904844216787E-02 Metric Check mismatch on component 1 of node 244. truth = 6.441625173938767546E+00, test = 6.296844243319102752E+00, relerror = 2.247583904841481128E-02 Metric Check mismatch on component 2 of node 244. truth = 4.225559915170791925E+01, test = 4.130586910627850017E+01, relerror = 2.247583904844554017E-02 Metric Check mismatch on component 0 of node 245. truth = 2.850420339634770173E+01, test = 2.786354750860713381E+01, relerror = 2.247583904844912758E-02 Metric Check mismatch on component 1 of node 245. truth = 1.508632124444206024E+00, test = 1.474724351631878205E+00, relerror = 2.247583904844910677E-02 Metric Check mismatch on component 2 of node 245. truth = 2.972085642892453095E+01, test = 2.905285524344596126E+01, relerror = 2.247583904844904085E-02 Metric Check mismatch on component 0 of node 246. truth = 6.959155743929187565E+01, test = 6.802742879515540153E+01, relerror = 2.247583904844980759E-02 Metric Check mismatch on component 1 of node 246. truth = -1.529932397501967412E+00, test = -1.495545883180831481E+00, relerror = 2.247583904836665189E-02 Metric Check mismatch on component 2 of node 246. truth = 7.231841810931612713E+01, test = 7.069300098365275176E+01, relerror = 2.247583904844826369E-02 Metric Check mismatch on component 0 of node 247. truth = 1.091399738396868599E+02, test = 1.066869613539141284E+02, relerror = 2.247583904844895758E-02 Metric Check mismatch on component 1 of node 247. truth = 1.336347848775831970E+01, test = 1.306312309614005152E+01, relerror = 2.247583904844911717E-02 Metric Check mismatch on component 2 of node 247. truth = 1.236022246047313189E+02, test = 1.208241608984851041E+02, relerror = 2.247583904844924901E-02 Metric Check mismatch on component 0 of node 248. truth = 9.241758471253928064E+01, test = 9.034042195329382707E+01, relerror = 2.247583904844921432E-02 Metric Check mismatch on component 1 of node 248. truth = 1.300645661387609842E+01, test = 1.271412558843198326E+01, relerror = 2.247583904844907554E-02 Metric Check mismatch on component 2 of node 248. truth = 9.037430991459993379E+01, test = 8.834307147084471978E+01, relerror = 2.247583904844919003E-02 Metric Check mismatch on component 0 of node 249. truth = 5.756575628698693947E+01, test = 5.627191761397836700E+01, relerror = 2.247583904844922473E-02 Metric Check mismatch on component 1 of node 249. truth = 4.036660715327581528E+00, test = 3.945933378796681268E+00, relerror = 2.247583904844913105E-02 Metric Check mismatch on component 2 of node 249. truth = 5.400228316278203522E+01, test = 5.278853653816658920E+01, relerror = 2.247583904844880839E-02 Metric Check mismatch on component 0 of node 250. truth = 3.326252417672798600E+01, test = 3.251492103698672054E+01, relerror = 2.247583904844851002E-02 Metric Check mismatch on component 1 of node 250. truth = -3.615749168309293271E+00, test = -3.534482171962776587E+00, relerror = 2.247583904845831468E-02 Metric Check mismatch on component 2 of node 250. truth = 4.171265361957341611E+01, test = 4.077512673053612247E+01, relerror = 2.247583904845039046E-02 Metric Check mismatch on component 0 of node 251. truth = 4.380893875053337894E+01, test = 4.282429609429301109E+01, relerror = 2.247583904844943636E-02 Metric Check mismatch on component 1 of node 251. truth = 4.871268773879767977E+00, test = 4.761782920956299847E+00, relerror = 2.247583904845125088E-02 Metric Check mismatch on component 2 of node 251. truth = 5.157548647912202000E+01, test = 5.041628414617181875E+01, relerror = 2.247583904844894370E-02 Metric Check mismatch on component 0 of node 252. truth = 5.010629955230249521E+01, test = 4.898011842825158624E+01, relerror = 2.247583904844871819E-02 Metric Check mismatch on component 1 of node 252. truth = 7.536740832851002381E+00, test = 7.367346258941975812E+00, relerror = 2.247583904844820124E-02 Metric Check mismatch on component 2 of node 252. truth = 5.361016736159979246E+01, test = 5.240523386862005140E+01, relerror = 2.247583904844919350E-02 Metric Check mismatch on component 0 of node 253. truth = 5.599475341064467671E+01, test = 5.473622434542942727E+01, relerror = 2.247583904844915881E-02 Metric Check mismatch on component 1 of node 253. truth = 8.260170664868367041E+00, test = 8.074516398492063374E+00, relerror = 2.247583904844927677E-02 Metric Check mismatch on component 2 of node 253. truth = 5.644489650651475188E+01, test = 5.517625009752795506E+01, relerror = 2.247583904844917616E-02 Metric Check mismatch on component 0 of node 254. truth = 5.200318848738306343E+01, test = 5.083437319293449974E+01, relerror = 2.247583904844872860E-02 Metric Check mismatch on component 1 of node 254. truth = 7.741561620542787736E+00, test = 7.567563527575813964E+00, relerror = 2.247583904844952657E-02 Metric Check mismatch on component 2 of node 254. truth = 5.171483705205827164E+01, test = 5.055250269805943475E+01, relerror = 2.247583904844916228E-02 Metric Check mismatch on component 0 of node 255. truth = 5.641808225387939046E+01, test = 5.515003851771904664E+01, relerror = 2.247583904844889166E-02 Metric Check mismatch on component 1 of node 255. truth = 7.336192573767081448E+00, test = 7.171305490250741599E+00, relerror = 2.247583904843865679E-02 Metric Check mismatch on component 2 of node 255. truth = 5.576593952769384543E+01, test = 5.451255324648387557E+01, relerror = 2.247583904844869390E-02 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 155.856u 0.453s 2:36.39 99.9% 0+0k 120+38704io 0pf+0w Thu Mar 24 13:23:40 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CutCell_Interface_StarShape_Jump_TEST.job Start 846: naca_Euler2d_Newton_Osher_TEST_2 285/327 Test #844: MixedMesh_Scalar2d_Newton_GMRES_Jacobi_TEST_2 ........... Passed 3.29 sec Start 848: naca_NavierStokes2d_Newton_GMRES_ILU_TEST_3 286/327 Test #848: naca_NavierStokes2d_Newton_GMRES_ILU_TEST_3 ............. Passed 2.81 sec Start 850: FlatPlate_RANS_SA2d_DIC_TEST_2 287/327 Test #846: naca_Euler2d_Newton_Osher_TEST_2 ........................ Passed 3.81 sec Start 852: FlatPlate_RANS_SA2d_ADC_TEST_2 288/327 Test #852: FlatPlate_RANS_SA2d_ADC_TEST_2 .......................... Passed 9.56 sec Start 854: FlatPlate_RANS_SA2d_DC_TEST_2 289/327 Test #811: CHT_NS2d_Scalar2d_FullEmb_FlatSlab_TEST .................***Failed 84.80 sec OUT/CHT_FlatSlab -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 8323 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CHT_NS2d_Scalar2d_FullEmb_FlatSlab_TEST.job truth = 7.075849433664431842E+01, test = 6.196875272242140653E+01, relerror = 1.242217163695481796E-01 Metric Check mismatch on component 1 of node 68. truth = -7.506853294945969424E+01, test = -6.574337845940000591E+01, relerror = 1.242218826407317789E-01 Metric Check mismatch on component 2 of node 68. truth = 1.452384638730865447E+02, test = 1.271968226344972237E+02, relerror = 1.242208211066912343E-01 Metric Check mismatch on component 0 of node 69. truth = 4.032385859269803063E+01, test = 3.531463737329690389E+01, relerror = 1.242247491738851656E-01 Metric Check mismatch on component 1 of node 69. truth = -3.229320458808154370E+01, test = -2.828140860782714583E+01, relerror = 1.242303460256475034E-01 Metric Check mismatch on component 2 of node 69. truth = 7.214935573672460123E+01, test = 6.318672182906170320E+01, relerror = 1.242233394344898489E-01 Metric Check mismatch on component 0 of node 70. truth = 2.751694626136122324E+01, test = 2.409885380919017805E+01, relerror = 1.242177245870725866E-01 Metric Check mismatch on component 1 of node 70. truth = -1.853366556464891701E+01, test = -1.623161582814375947E+01, relerror = 1.242090901271081238E-01 Metric Check mismatch on component 2 of node 70. truth = 4.200805671838050870E+01, test = 3.678996179944495992E+01, relerror = 1.242165271751879407E-01 Metric Check mismatch on component 0 of node 71. truth = 2.320662150826279202E+01, test = 2.032388851048617440E+01, relerror = 1.242202789729737883E-01 Metric Check mismatch on component 1 of node 71. truth = -1.339042959948020517E+01, test = -1.172709664144831976E+01, relerror = 1.242180428696965239E-01 Metric Check mismatch on component 2 of node 71. truth = 3.125906923735702136E+01, test = 2.737608198603483345E+01, relerror = 1.242195415940829106E-01 Metric Check mismatch on component 0 of node 72. truth = 8.366650620524980653E+00, test = 7.327346598650084530E+00, relerror = 1.242198424451101402E-01 Metric Check mismatch on component 1 of node 72. truth = -3.869003595131863538E-01, test = -3.388423125412983739E-01, relerror = 4.805804697188797991E-02 Metric Check mismatch on component 2 of node 72. truth = 8.460175980090038905E+00, test = 7.409243000587975914E+00, relerror = 1.242211724644146503E-01 Metric Check mismatch on component 0 of node 73. truth = 1.029709639290322443E+01, test = 9.017967632685218504E+00, relerror = 1.242222769808762339E-01 Metric Check mismatch on component 1 of node 73. truth = -4.420399217210248533E-01, test = -3.871366515425885280E-01, relerror = 5.490327017843632529E-02 Metric Check mismatch on component 2 of node 73. truth = 1.055997521559474350E+01, test = 9.248228699673386899E+00, relerror = 1.242187116106293204E-01 Metric Check mismatch on component 0 of node 74. truth = 1.161812795513866270E+01, test = 1.017494619311765369E+01, relerror = 1.242180984400928501E-01 Metric Check mismatch on component 1 of node 74. truth = -1.456271290016669440E+00, test = -1.275375838951766871E+00, relerror = 1.242182361933619611E-01 Metric Check mismatch on component 2 of node 74. truth = 1.392856032504513131E+01, test = 1.219831432998502940E+01, relerror = 1.242228884164663649E-01 Metric Check mismatch on component 0 of node 75. truth = 1.785371707597906976E+01, test = 1.563596887829343807E+01, relerror = 1.242177294648326852E-01 Metric Check mismatch on component 1 of node 75. truth = -8.965041346820447288E+00, test = -7.851406933523561449E+00, relerror = 1.242196628230664845E-01 Metric Check mismatch on component 2 of node 75. truth = 2.687248920025155741E+01, test = 2.353430719480636668E+01, relerror = 1.242230290081925703E-01 Metric Check mismatch on component 0 of node 76. truth = 3.770341940581739948E+01, test = 3.301988008616584835E+01, relerror = 1.242205453367688645E-01 Metric Check mismatch on component 1 of node 76. truth = -3.275826487525576880E+01, test = -2.868901127880150881E+01, relerror = 1.242206695607985384E-01 Metric Check mismatch on component 2 of node 76. truth = 6.794124407200884264E+01, test = 5.950154034291174554E+01, relerror = 1.242206239284066316E-01 Metric Check mismatch on component 0 of node 77. truth = 4.729217076389594610E+01, test = 4.141748405891809170E+01, relerror = 1.242211260359979141E-01 Metric Check mismatch on component 1 of node 77. truth = -4.668890734879736470E+01, test = -4.088914767107353043E+01, relerror = 1.242213623547827051E-01 Metric Check mismatch on component 2 of node 77. truth = 9.715208249305833021E+01, test = 8.508378116268283975E+01, relerror = 1.242207168460623540E-01 Metric Check mismatch on component 0 of node 78. truth = 4.077338396430964451E+01, test = 3.570845877432388704E+01, relerror = 1.242213595618961164E-01 Metric Check mismatch on component 1 of node 78. truth = -3.682785550707534128E+01, test = -3.225305569613419721E+01, relerror = 1.242211838824619219E-01 Metric Check mismatch on component 2 of node 78. truth = 7.808251672726365200E+01, test = 6.838308697940745162E+01, relerror = 1.242202499918845793E-01 Metric Check mismatch on component 0 of node 79. truth = 3.665055637923574494E+01, test = 3.209780638961461818E+01, relerror = 1.242204877468237462E-01 Metric Check mismatch on component 1 of node 79. truth = -2.623950116882685535E+01, test = -2.298004223300155502E+01, relerror = 1.242195465094288603E-01 Metric Check mismatch on component 2 of node 79. truth = 5.260652958443004934E+01, test = 4.607175377264313454E+01, relerror = 1.242198613633888477E-01 Metric Check mismatch on component 0 of node 80. truth = 2.893732170787283664E+01, test = 2.534270357890557790E+01, relerror = 1.242208302916053414E-01 Metric Check mismatch on component 1 of node 80. truth = -1.777616219570509060E+01, test = -1.556797345345368377E+01, relerror = 1.242219056026012541E-01 Metric Check mismatch on component 2 of node 80. truth = 3.812071368770486401E+01, test = 3.338532019270168405E+01, relerror = 1.242210083944596810E-01 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 83.475u 0.205s 1:23.76 99.8% 0+0k 64+8056io 0pf+0w Thu Mar 24 13:24:06 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for CHT_NS2d_Scalar2d_FullEmb_FlatSlab_TEST.job 290/327 Test #854: FlatPlate_RANS_SA2d_DC_TEST_2 ........................... Passed 12.74 sec Start 856: bump_Euler3d_GMRES_TEST_2 Start 858: Sphere_Q23D_TEST_2 291/327 Test #858: Sphere_Q23D_TEST_2 ...................................... Passed 10.36 sec Start 860: channel_NavierStokes3d_TEST_2 292/327 Test #856: bump_Euler3d_GMRES_TEST_2 ............................... Passed 10.96 sec Start 862: naca_InviscidNE2d_PerfectGas_TEST_3 293/327 Test #850: FlatPlate_RANS_SA2d_DIC_TEST_2 .......................... Passed 38.57 sec Start 864: naca_InviscidNE2d_RealGas_TEST_2 294/327 Test #862: naca_InviscidNE2d_PerfectGas_TEST_3 ..................... Passed 6.51 sec Start 866: diamond_Euler2d_Shock_TEST_2 295/327 Test #860: channel_NavierStokes3d_TEST_2 ........................... Passed 9.62 sec Start 868: diamond_Euler2d_PDEShock_TEST_3 296/327 Test #864: naca_InviscidNE2d_RealGas_TEST_2 ........................ Passed 4.34 sec Start 870: diamond_Euler2d_PDEShock_Jump_TEST_3 297/327 Test #866: diamond_Euler2d_Shock_TEST_2 ............................ Passed 6.84 sec Start 872: naca_NavierStokes2d_AuxVariable_FluxCoeff_TEST_2 298/327 Test #870: diamond_Euler2d_PDEShock_Jump_TEST_3 .................... Passed 8.84 sec 299/327 Test #872: naca_NavierStokes2d_AuxVariable_FluxCoeff_TEST_2 ........ Passed 4.63 sec Start 874: naca_NavierStokes2d_AuxVariable_Gradient_TEST_2 Start 876: 2d_Periodic_TEST1_2 300/327 Test #868: diamond_Euler2d_PDEShock_TEST_3 ......................... Passed 9.67 sec Start 878: 2d_Periodic_TEST2_3 301/327 Test #876: 2d_Periodic_TEST1_2 ..................................... Passed 2.91 sec 302/327 Test #878: 2d_Periodic_TEST2_3 ..................................... Passed 2.61 sec Start 880: turbineblade_Periodic_TEST_4 Start 882: 3d_Periodic_TEST_2 303/327 Test #874: naca_NavierStokes2d_AuxVariable_Gradient_TEST_2 ......... Passed 3.42 sec Start 884: naca_Euler2d_Adjoint_TEST_3 304/327 Test #882: 3d_Periodic_TEST_2 ...................................... Passed 2.64 sec Start 886: ConvectingVortex_Euler2d_BDF1_TEST_3 305/327 Test #884: naca_Euler2d_Adjoint_TEST_3 ............................. Passed 2.84 sec Start 888: ConvectingVortex_Euler2d_BDF2_TEST_3 306/327 Test #880: turbineblade_Periodic_TEST_4 ............................ Passed 5.05 sec Start 890: ConvectingVortex_Euler2d_Trap2_TEST_3 307/327 Test #886: ConvectingVortex_Euler2d_BDF1_TEST_3 .................... Passed 4.01 sec Start 892: ConvectingVortex_Euler2d_IRK2_TEST_3 308/327 Test #888: ConvectingVortex_Euler2d_BDF2_TEST_3 .................... Passed 4.11 sec Start 894: ConvectingVortex_Euler2d_IRK4_TEST_3 309/327 Test #890: ConvectingVortex_Euler2d_Trap2_TEST_3 ................... Passed 4.31 sec Start 896: naca_Euler2d_Adapt_TEST_2 310/327 Test #892: ConvectingVortex_Euler2d_IRK2_TEST_3 .................... Passed 6.31 sec Start 898: naca_Euler2d_CurvedAdapt_TEST_3 311/327 Test #896: naca_Euler2d_Adapt_TEST_2 ...............................***Failed 3.81 sec OUT/naca_Euler2d_Adapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 18206 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_Euler2d_Adapt_TEST.job Metric Check mismatch on component 0 of node 92. truth = 5.689760039767215866E-02, test = 5.000983924996215552E-02, relerror = 6.887761147710003140E-03 Metric Check mismatch on component 1 of node 92. truth = -9.143337169865531008E-03, test = -8.036486933632387350E-03, relerror = 1.106850236233143658E-03 Metric Check mismatch on component 2 of node 92. truth = 4.369826141831986005E-02, test = 3.840835138708454527E-02, relerror = 5.289910031235314780E-03 Metric Check mismatch on component 0 of node 93. truth = 4.121537682444183019E+02, test = 3.622603335222468104E+02, relerror = 1.210553889503282188E-01 Metric Check mismatch on component 1 of node 93. truth = -4.716979741650904856E+01, test = -4.145963924348419027E+01, relerror = 1.210553889516249870E-01 Metric Check mismatch on component 2 of node 93. truth = 6.891655245585733383E+02, test = 6.057383239318891128E+02, relerror = 1.210553889504575875E-01 Metric Check mismatch on component 0 of node 94. truth = 3.435291896711543946E-02, test = 3.019431300024527295E-02, relerror = 4.158605966870166509E-03 Metric Check mismatch on component 2 of node 94. truth = 3.450489139108037923E-02, test = 3.032788834377299347E-02, relerror = 4.177003047307385752E-03 Metric Check mismatch on component 0 of node 95. truth = 4.065367802664344304E-02, test = 3.573233121436918125E-02, relerror = 4.921346812274261795E-03 Metric Check mismatch on component 2 of node 95. truth = 3.481510963363079053E-02, test = 3.060055299358729192E-02, relerror = 4.214556640043498614E-03 Metric Check mismatch on component 0 of node 96. truth = 1.289939208753848376E+01, test = 1.133785116040770191E+01, relerror = 1.210553890085499101E-01 Metric Check mismatch on component 1 of node 96. truth = 1.702887848043856778E+01, test = 1.496744097292649833E+01, relerror = 1.210553889312256381E-01 Metric Check mismatch on component 2 of node 96. truth = 2.253794440457299117E+02, test = 1.980960477875357242E+02, relerror = 1.210553889406983108E-01 Metric Check mismatch on component 0 of node 97. truth = 8.376505230464267981E+01, test = 7.362484132367740131E+01, relerror = 1.210553888760988323E-01 Metric Check mismatch on component 1 of node 97. truth = -3.157799608873859398E+01, test = -2.775530950358960069E+01, relerror = 1.210553885182171913E-01 Metric Check mismatch on component 2 of node 97. truth = 6.964819526053740617E+01, test = 6.121690587968956976E+01, relerror = 1.210553891498319379E-01 Metric Check mismatch on component 0 of node 98. truth = 2.815830942825028771E+01, test = 2.474959432707164098E+01, relerror = 1.210553889914568887E-01 Metric Check mismatch on component 1 of node 98. truth = -1.732482246226381761E+01, test = -1.522755934029296654E+01, relerror = 1.210553889679977513E-01 Metric Check mismatch on component 2 of node 98. truth = 8.685631067386209736E+01, test = 7.634188620320047391E+01, relerror = 1.210553889416553092E-01 Metric Check mismatch on component 0 of node 99. truth = 7.608251180019109938E+01, test = 6.687231374178591636E+01, relerror = 1.210553889518412168E-01 Metric Check mismatch on component 1 of node 99. truth = 2.756212041611194152E+01, test = 2.422557720823317950E+01, relerror = 1.210553889724799437E-01 Metric Check mismatch on component 2 of node 99. truth = 2.130298020217572628E+02, test = 1.872413964801778548E+02, relerror = 1.210553889495028235E-01 Metric Check mismatch on component 0 of node 100. truth = 3.490130910464708802E-01, test = 3.067631755612701405E-01, relerror = 4.224991548520073970E-02 Metric Check mismatch on component 1 of node 100. truth = 1.398365610200322795E-01, test = 1.229085917384669380E-01, relerror = 1.692796928156534153E-02 Metric Check mismatch on component 2 of node 100. truth = 5.749591390118199419E-01, test = 5.053572368062596576E-01, relerror = 6.960190220556028429E-02 Metric Check mismatch on component 0 of node 101. truth = 2.233047886963625928E+01, test = 1.962725406446519472E+01, relerror = 1.210553889575005510E-01 Metric Check mismatch on component 1 of node 101. truth = 4.224331677916233652E+00, test = 3.712953563621476771E+00, relerror = 1.210553889430879271E-01 Metric Check mismatch on component 2 of node 101. truth = 8.325471988235808851E+01, test = 7.317628738557701240E+01, relerror = 1.210553889439813513E-01 Metric Check mismatch on component 0 of node 102. truth = 1.583526644376091497E+01, test = 1.391832210533130798E+01, relerror = 1.210553889470474820E-01 Metric Check mismatch on component 1 of node 102. truth = 4.467801398296932014E+00, test = 3.926949961960815916E+00, relerror = 1.210553890202643451E-01 Metric Check mismatch on component 2 of node 102. truth = 1.293377963529775911E+02, test = 1.136807591084585880E+02, relerror = 1.210553889582992731E-01 Metric Check mismatch on component 0 of node 103. truth = 1.218064512447361665E+02, test = 1.070611239126308192E+02, relerror = 1.210553889504482478E-01 Metric Check mismatch on component 1 of node 103. truth = -4.268689773185799652E+01, test = -3.751941872375902420E+01, relerror = 1.210553889523433291E-01 Metric Check mismatch on component 2 of node 103. truth = 2.134507664232185675E+02, test = 1.876114008720065272E+02, relerror = 1.210553889508138442E-01 Metric Check mismatch on component 0 of node 104. truth = 1.793471082986419773E+00, test = 1.576361743465899190E+00, relerror = 1.210553889494545843E-01 Metric Check mismatch on component 1 of node 104. truth = 6.155777033164292616E-02, test = 5.410587050112002083E-02, relerror = 7.451899830522905332E-03 Metric Check mismatch on component 2 of node 104. truth = 3.880596486232950504E+00, test = 3.410829369228809860E+00, relerror = 1.210553889513419357E-01 Metric Check mismatch on component 0 of node 105. truth = 1.358111164741753463E+01, test = 1.193704489366517052E+01, relerror = 1.210553890163317686E-01 Metric Check mismatch on component 1 of node 105. truth = 1.024808771184361200E+00, test = 9.007501474017723941E-01, relerror = 1.210553883523220603E-01 Metric Check mismatch on component 2 of node 105. truth = 8.761372550606778020E+01, test = 7.700761189106809468E+01, relerror = 1.210553889100988850E-01 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3798 Call : PXAdjustGridHP(pxa, &pg1, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 3.198u 0.095s 0:02.72 120.5% 0+0k 0+4392io 5pf+0w Thu Mar 24 13:24:52 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_Euler2d_Adapt_TEST.job Start 900: Scalar2d_Adapt_TEST_4 312/327 Test #894: ConvectingVortex_Euler2d_IRK4_TEST_3 .................... Passed 6.82 sec Start 902: Scalar2d_CircleBoundaryLayer_Adapt_TEST_2 313/327 Test #900: Scalar2d_Adapt_TEST_4 ...................................***Failed 2.15 sec OUT/Scalar2d_Adapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 18481 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_Adapt_TEST.job Writing out OUT/Scalar2d_Adapt_TEST_P1_a00_Dual_0.pxa . . .Done. ************************** * Adaptation Iteration 0 * ************************** Incoming DOF = 24 Warning : Running HP-Adaptation (Not Well Tested) Adaptation Mode : DWR Adapting on Heat: AdaptTargetError = 0.000000E+00, GlobalError = 5.056814E-03 AnisotropyMethod = Hessian MetricTransferMode = Average meshDofReq = 1000.000000, meshDofCurr = 58.820259 ratio = 1.700094510441129e+01 Scaling the continuous metric Scaling the element metric **************************************************************************************** **************************************************************************************** CHECKING Truth Metric File: TrueMetric/Scalar2d_Adapt_TEST_Metric_a00.attach Metric Check mismatch on component 0 of node 0. truth = 4.243739987757014376E+01, test = 4.226464839859501410E+01, relerror = 4.070736649123399276E-03 Metric Check mismatch on component 1 of node 0. truth = -2.127339956490372774E+00, test = -2.118680115764294936E+00, relerror = 4.070736649145915986E-03 Metric Check mismatch on component 2 of node 0. truth = 5.360168246293591920E+01, test = 5.338348412967962986E+01, relerror = 4.070736649118568938E-03 Metric Check mismatch on component 0 of node 1. truth = 4.578326069419801314E+01, test = 4.559688909697377568E+01, relerror = 4.070736649123285651E-03 Metric Check mismatch on component 1 of node 1. truth = -1.041211499088002146E+00, test = -1.036973001279194895E+00, relerror = 4.070736649105156924E-03 Metric Check mismatch on component 2 of node 1. truth = 8.219497777649543480E+01, test = 8.186038366808793398E+01, relerror = 4.070736649109256075E-03 Metric Check mismatch on component 0 of node 2. truth = 5.698843517813379123E+01, test = 5.675645026647792690E+01, relerror = 4.070736649124134798E-03 Metric Check mismatch on component 1 of node 2. truth = -6.855472742836819400E+00, test = -6.827565918695640335E+00, relerror = 4.070736649101039557E-03 Metric Check mismatch on component 2 of node 2. truth = 8.918274039805585574E+01, test = 8.881970094824892215E+01, relerror = 4.070736649115658939E-03 Metric Check mismatch on component 0 of node 3. truth = 3.078020123180149170E+01, test = 3.065490313857992177E+01, relerror = 4.070736649119578547E-03 Metric Check mismatch on component 1 of node 3. truth = -1.271207910937258045E+00, test = -1.266033158305573503E+00, relerror = 4.070736649105032891E-03 Metric Check mismatch on component 2 of node 3. truth = 3.535139989565922747E+01, test = 3.520749365650647178E+01, relerror = 4.070736649114306722E-03 Metric Check mismatch on component 0 of node 4. truth = 3.648475588620901533E+01, test = 3.633623605328882178E+01, relerror = 4.070736649120160547E-03 Metric Check mismatch on component 1 of node 4. truth = -1.819185680604555655E+00, test = -1.811780254782922261E+00, relerror = 4.070736649143151704E-03 Metric Check mismatch on component 2 of node 4. truth = 5.815196916625858137E+01, test = 5.791524781415503753E+01, relerror = 4.070736649119291450E-03 Metric Check mismatch on component 0 of node 5. truth = 4.329810999744660904E+01, test = 4.312185479424240953E+01, relerror = 4.070736649119181295E-03 Metric Check mismatch on component 1 of node 5. truth = -4.240392944327044944E+00, test = -4.223131421361903115E+00, relerror = 4.070736649119967993E-03 Metric Check mismatch on component 2 of node 5. truth = 8.751930616413717701E+01, test = 8.716303811702987048E+01, relerror = 4.070736649113137519E-03 Metric Check mismatch on component 0 of node 6. truth = 3.046664761875388550E+01, test = 3.034262591971641498E+01, relerror = 4.070736649119490944E-03 Metric Check mismatch on component 1 of node 6. truth = -1.042810277911444095E+00, test = -1.038565271895167674E+00, relerror = 4.070736649027262635E-03 Metric Check mismatch on component 2 of node 6. truth = 3.946969712429973498E+01, test = 3.930902638168701202E+01, relerror = 4.070736649098964828E-03 Metric Check mismatch on component 0 of node 7. truth = 3.041698789048398410E+01, test = 3.029316834312237816E+01, relerror = 4.070736649119130121E-03 Metric Check mismatch on component 1 of node 7. truth = -8.910238460457734488E-01, test = -8.873967226204585446E-01, relerror = 3.627123425314904281E-03 Metric Check mismatch on component 2 of node 7. truth = 4.289788604460342469E+01, test = 4.272326004771208829E+01, relerror = 4.070736649115240871E-03 Metric Check mismatch on component 0 of node 8. truth = 3.846679269588950945E+01, test = 3.831020451308839370E+01, relerror = 4.070736649116277368E-03 Metric Check mismatch on component 1 of node 8. truth = -5.424665397319711424E+00, test = -5.402583013077506102E+00, relerror = 4.070736649142649502E-03 Metric Check mismatch on component 2 of node 8. truth = 8.550534213948071738E+01, test = 8.515727240953728483E+01, relerror = 4.070736649128229613E-03 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3798 Call : PXAdjustGridHP(pxa, &pg1, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 0.240u 0.092s 0:01.12 29.4% 0+0k 0+5096io 11pf+0w Thu Mar 24 13:24:54 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_Adapt_TEST.job Start 904: poissonLshape_TEST_3 314/327 Test #898: naca_Euler2d_CurvedAdapt_TEST_3 .........................***Failed 3.46 sec OUT/naca_Euler2d_CurvedAdapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 18456 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_Euler2d_CurvedAdapt_TEST.job Metric Check mismatch on component 0 of node 92. truth = 5.689760039754101356E-02, test = 5.000983924992373486E-02, relerror = 6.887761147617278701E-03 Metric Check mismatch on component 1 of node 92. truth = -9.143337169690946703E-03, test = -8.036486931980226303E-03, relerror = 1.106850237710720400E-03 Metric Check mismatch on component 2 of node 92. truth = 4.369826141832010985E-02, test = 3.840835138538040150E-02, relerror = 5.289910032939708351E-03 Metric Check mismatch on component 0 of node 93. truth = 4.121537682462873704E+02, test = 3.622603335215040943E+02, relerror = 1.210553889561161583E-01 Metric Check mismatch on component 1 of node 93. truth = -4.716979741671573834E+01, test = -4.145963924342588314E+01, relerror = 1.210553889567124730E-01 Metric Check mismatch on component 2 of node 93. truth = 6.891655245615404510E+02, test = 6.057383239305561347E+02, relerror = 1.210553889561759577E-01 Metric Check mismatch on component 0 of node 94. truth = 3.435291896719223220E-02, test = 3.019431300005339172E-02, relerror = 4.158605967138840481E-03 Metric Check mismatch on component 2 of node 94. truth = 3.450489139103744829E-02, test = 3.032788834337210235E-02, relerror = 4.177003047665345942E-03 Metric Check mismatch on component 0 of node 95. truth = 4.065367802497073246E-02, test = 3.573233121968964060E-02, relerror = 4.921346805281091863E-03 Metric Check mismatch on component 2 of node 95. truth = 3.481510963303171419E-02, test = 3.060055299524190586E-02, relerror = 4.214556637789808324E-03 Metric Check mismatch on component 0 of node 96. truth = 1.289939208705553142E+01, test = 1.133785116129202564E+01, relerror = 1.210553889070868222E-01 Metric Check mismatch on component 1 of node 96. truth = 1.702887848104132829E+01, test = 1.496744097248898520E+01, relerror = 1.210553889880295053E-01 Metric Check mismatch on component 2 of node 96. truth = 2.253794440508562218E+02, test = 1.980960477828564876E+02, relerror = 1.210553889814517114E-01 Metric Check mismatch on component 0 of node 97. truth = 8.376505230487983056E+01, test = 7.362484132181201346E+01, relerror = 1.210553889008565420E-01 Metric Check mismatch on component 1 of node 97. truth = -3.157799608769623489E+01, test = -2.775530950175987144E+01, relerror = 1.210553885471472441E-01 Metric Check mismatch on component 2 of node 97. truth = 6.964819526139045536E+01, test = 6.121690588148966583E+01, relerror = 1.210553891347516814E-01 Metric Check mismatch on component 0 of node 98. truth = 2.815830942953095573E+01, test = 2.474959432727730402E+01, relerror = 1.210553890241283514E-01 Metric Check mismatch on component 1 of node 98. truth = -1.732482246261697867E+01, test = -1.522755934035914294E+01, relerror = 1.210553889820950163E-01 Metric Check mismatch on component 2 of node 98. truth = 8.685631067355359392E+01, test = 7.634188620290075278E+01, relerror = 1.210553889419841711E-01 Metric Check mismatch on component 0 of node 99. truth = 7.608251180040910810E+01, test = 6.687231374204982615E+01, relerror = 1.210553889508910463E-01 Metric Check mismatch on component 1 of node 99. truth = 2.756212041631181364E+01, test = 2.422557720858727137E+01, relerror = 1.210553889660067467E-01 Metric Check mismatch on component 2 of node 99. truth = 2.130298020229691929E+02, test = 1.872413964790154637E+02, relerror = 1.210553889599596256E-01 Metric Check mismatch on component 0 of node 100. truth = 3.490130910476799131E-01, test = 3.067631755591508913E-01, relerror = 4.224991548852902179E-02 Metric Check mismatch on component 1 of node 100. truth = 1.398365610215821508E-01, test = 1.229085917281554363E-01, relerror = 1.692796929342671453E-02 Metric Check mismatch on component 2 of node 100. truth = 5.749591390136781222E-01, test = 5.053572368123925296E-01, relerror = 6.960190220128559258E-02 Metric Check mismatch on component 0 of node 101. truth = 2.233047886974289398E+01, test = 1.962725406462456235E+01, relerror = 1.210553889545609996E-01 Metric Check mismatch on component 1 of node 101. truth = 4.224331677929030526E+00, test = 3.712953563598484052E+00, relerror = 1.210553889511934572E-01 Metric Check mismatch on component 2 of node 101. truth = 8.325471988271065982E+01, test = 7.317628738506753905E+01, relerror = 1.210553889538230066E-01 Metric Check mismatch on component 0 of node 102. truth = 1.583526644351239732E+01, test = 1.391832210519494062E+01, relerror = 1.210553889418650025E-01 Metric Check mismatch on component 1 of node 102. truth = 4.467801398668708401E+00, test = 3.926949963038491642E+00, relerror = 1.210553888521940097E-01 Metric Check mismatch on component 2 of node 102. truth = 1.293377963556231975E+02, test = 1.136807591108235158E+02, relerror = 1.210553889579931985E-01 Metric Check mismatch on component 0 of node 103. truth = 1.218064512452652224E+02, test = 1.070611239124373952E+02, relerror = 1.210553889558538265E-01 Metric Check mismatch on component 1 of node 103. truth = -4.268689773201904103E+01, test = -3.751941872373822662E+01, relerror = 1.210553889561465368E-01 Metric Check mismatch on component 2 of node 103. truth = 2.134507664241417046E+02, test = 1.876114008716602655E+02, relerror = 1.210553889562373392E-01 Metric Check mismatch on component 0 of node 104. truth = 1.793471082995315324E+00, test = 1.576361743461732301E+00, relerror = 1.210553889561374885E-01 Metric Check mismatch on component 1 of node 104. truth = 6.155777033175219987E-02, test = 5.410587050141717896E-02, relerror = 7.451899830335020902E-03 Metric Check mismatch on component 2 of node 104. truth = 3.880596486248734323E+00, test = 3.410829369222609486E+00, relerror = 1.210553889565147145E-01 Metric Check mismatch on component 0 of node 105. truth = 1.358111164679784189E+01, test = 1.193704489331207519E+01, relerror = 1.210553890022253998E-01 Metric Check mismatch on component 1 of node 105. truth = 1.024808771588735290E+00, test = 9.007501476183745748E-01, relerror = 1.210553884877817177E-01 Metric Check mismatch on component 2 of node 105. truth = 8.761372550879706012E+01, test = 7.700761189199882040E+01, relerror = 1.210553889268560779E-01 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3798 Call : PXAdjustGridHP(pxa, &pg1, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 3.431u 0.060s 0:02.30 151.7% 0+0k 0+6568io 7pf+0w Thu Mar 24 13:24:55 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_Euler2d_CurvedAdapt_TEST.job Start 906: AdDiffBL2_DofControl_TEST_4 315/327 Test #904: poissonLshape_TEST_3 ....................................***Failed 2.60 sec OUT/poissonLshape_TEST Numerical error: tolerance = .00000000010000000000 or 0.0000000000001 > ERROR: Test failed for poissonLshape_TEST.job in column 2 > New: .00081368255155350000 Orig: .00081357889605370000 > > Truth file: 440 8.135788960537e-04 -8.135788960537e-04 1.836054410566E+00 > New file: 440 8.136825515535e-04 -8.136825515535e-04 1.836053796902E+00 Start 908: hemisphere_Euler3d_Adapt_TEST_2 316/327 Test #902: Scalar2d_CircleBoundaryLayer_Adapt_TEST_2 ...............***Failed 4.55 sec OUT/Scalar2d_CircleBoundaryLayer_Adapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 18723 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_CircleBoundaryLayer_Adapt_TEST.job Metric Check mismatch on component 2 of node 904. truth = 2.010307275008552779E+02, test = 1.860116813278202983E+02, relerror = 7.471020156841984283E-02 Metric Check mismatch on component 0 of node 905. truth = 4.521577831788422941E+02, test = 4.183770047693852234E+02, relerror = 7.471015576015360704E-02 Metric Check mismatch on component 1 of node 905. truth = 3.098229747984217131E+00, test = 2.866755205553153107E+00, relerror = 7.471187137805611334E-02 Metric Check mismatch on component 2 of node 905. truth = 2.006781492232387620E+02, test = 1.856854418660686008E+02, relerror = 7.471021341985741027E-02 Metric Check mismatch on component 0 of node 906. truth = 4.461947143090154668E+02, test = 4.128594305931546273E+02, relerror = 7.471017169597002350E-02 Metric Check mismatch on component 1 of node 906. truth = 9.516271002362161724E+00, test = 8.805310961835701278E+00, relerror = 7.470994051661448976E-02 Metric Check mismatch on component 2 of node 906. truth = 2.000534477845063179E+02, test = 1.851074251582884642E+02, relerror = 7.471014767172330651E-02 Metric Check mismatch on component 0 of node 907. truth = 4.411159235711978681E+02, test = 4.081600758723228068E+02, relerror = 7.471017466807873542E-02 Metric Check mismatch on component 1 of node 907. truth = 1.561902466803447354E+01, test = 1.445213141565625570E+01, relerror = 7.470973874356917621E-02 Metric Check mismatch on component 2 of node 907. truth = 1.991753821806738927E+02, test = 1.842949679998403099E+02, relerror = 7.471010733312120899E-02 Metric Check mismatch on component 0 of node 908. truth = 4.367905588287156320E+02, test = 4.041578673536662336E+02, relerror = 7.471015756969710819E-02 Metric Check mismatch on component 1 of node 908. truth = 2.153975732859295178E+01, test = 1.993051690759036632E+01, relerror = 7.471023913841402797E-02 Metric Check mismatch on component 2 of node 908. truth = 1.980748757793956543E+02, test = 1.832766698690966223E+02, relerror = 7.471016125629389992E-02 Metric Check mismatch on component 0 of node 909. truth = 4.330975100276222065E+02, test = 4.007407371772258671E+02, relerror = 7.471013363326120538E-02 Metric Check mismatch on component 1 of node 909. truth = 2.747016065457902911E+01, test = 2.541784219772912934E+01, relerror = 7.471082832956772724E-02 Metric Check mismatch on component 2 of node 909. truth = 1.967968395777772059E+02, test = 1.820940977225272945E+02, relerror = 7.471025391868224563E-02 Metric Check mismatch on component 0 of node 910. truth = 4.299209749220522099E+02, test = 3.978015119944151365E+02, relerror = 7.471015558954891000E-02 Metric Check mismatch on component 1 of node 910. truth = 3.369877263228464415E+01, test = 3.118112592204034428E+01, relerror = 7.471033849560156093E-02 Metric Check mismatch on component 2 of node 910. truth = 1.954121548749100441E+02, test = 1.808128742391740218E+02, relerror = 7.471019724991782851E-02 Metric Check mismatch on component 0 of node 911. truth = 4.271399376712579397E+02, test = 3.952282115170899033E+02, relerror = 7.471023741809044993E-02 Metric Check mismatch on component 1 of node 911. truth = 4.069562017453012714E+01, test = 3.765526940900928565E+01, relerror = 7.470953268390498336E-02 Metric Check mismatch on component 2 of node 911. truth = 1.940590417388196727E+02, test = 1.795608774392568137E+02, relerror = 7.471006849078261347E-02 Metric Check mismatch on component 0 of node 912. truth = 4.246299874528556302E+02, test = 3.929058042557945782E+02, relerror = 7.471018094449398406E-02 Metric Check mismatch on component 1 of node 912. truth = 4.932869549030304057E+01, test = 4.564330199622704498E+01, relerror = 7.471094577800997671E-02 Metric Check mismatch on component 2 of node 912. truth = 1.930574621326344982E+02, test = 1.786340776180422836E+02, relerror = 7.471031865467624156E-02 Metric Check mismatch on component 0 of node 913. truth = 4.224533643108545675E+02, test = 3.908918619113861155E+02, relerror = 7.471002734456742844E-02 Metric Check mismatch on component 1 of node 913. truth = 6.147713169194055638E+01, test = 5.688404134597752915E+01, relerror = 7.471217702508989311E-02 Metric Check mismatch on component 2 of node 913. truth = 1.931818070399272358E+02, test = 1.787490513615417171E+02, relerror = 7.471073958534059367E-02 Metric Check mismatch on component 0 of node 914. truth = 4.205892531904534621E+02, test = 3.891670283005513511E+02, relerror = 7.471000424177108579E-02 Metric Check mismatch on component 1 of node 914. truth = 7.602137649678884657E+01, test = 7.034192923446416046E+01, relerror = 7.470855598838817280E-02 Metric Check mismatch on component 2 of node 914. truth = 1.663977273153208216E+02, test = 1.539661066545415622E+02, relerror = 7.471027916878668351E-02 Metric Check mismatch on component 0 of node 915. truth = 4.283664565966642499E+02, test = 3.963632372329361147E+02, relerror = 7.470990987014025642E-02 Metric Check mismatch on component 1 of node 915. truth = 1.024946805414455895E+02, test = 9.483740553136316009E+01, relerror = 7.470899923421948918E-02 Metric Check mismatch on component 2 of node 915. truth = 1.228850590473537494E+02, test = 1.137043353287371303E+02, relerror = 7.470984503558587841E-02 Metric Check mismatch on component 0 of node 916. truth = 4.198036272767603805E+02, test = 3.884400400606759831E+02, relerror = 7.471013868922002366E-02 Metric Check mismatch on component 1 of node 916. truth = 7.346930675907962893E+01, test = 6.798042736899540728E+01, relerror = 7.470982961746654161E-02 Metric Check mismatch on component 2 of node 916. truth = 9.156259053720209806E+01, test = 8.472195341923460887E+01, relerror = 7.470995608395464282E-02 Metric Check mismatch on component 0 of node 917. truth = 4.430250885708222768E+02, test = 4.099266085698448023E+02, relerror = 7.471017072137288439E-02 Metric Check mismatch on component 1 of node 917. truth = 1.397366636360839465E+01, test = 1.292969110794710907E+01, relerror = 7.471018904387966009E-02 Metric Check mismatch on component 2 of node 917. truth = 6.226258582006595788E+01, test = 5.761093720810505658E+01, relerror = 7.471017386595225906E-02 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3798 Call : PXAdjustGridHP(pxa, &pg1, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 4.709u 0.097s 0:03.42 140.0% 0+0k 0+6808io 5pf+0w Thu Mar 24 13:24:58 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_CircleBoundaryLayer_Adapt_TEST.job Start 910: bumpQ2_Euler3d_Jacobi_TEST_2 317/327 Test #906: AdDiffBL2_DofControl_TEST_4 .............................***Failed 2.52 sec OUT/AdDiffBL2_DofControl_TEST Numerical error: tolerance = 1.0 or 0.0000000000001 > ERROR: Test failed for AdDiffBL2_DofControl_TEST.job in column 1 > New: 108 Orig: 132 > > Truth file: 132 1.427802167237e-04 8.958505572551e-05 2.020425490045E-04 > New file: 108 1.694143416847e-04 1.221501663602e-04 2.331102721432E-04 Start 912: box_Euler2d_TEST_3 318/327 Test #908: hemisphere_Euler3d_Adapt_TEST_2 .........................***Failed 3.93 sec OUT/hemisphere_Euler3d_Adapt -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 18902 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for hemisphere_Euler3d_Adapt_TEST.job Metric Check mismatch on component 5 of node 151. truth = 9.550913477420202469E+00, test = 9.529285522470322789E+00, relerror = 2.264490721333767970E-03 Metric Check mismatch on component 0 of node 152. truth = 7.953857219441792914E+00, test = 7.935845783569552836E+00, relerror = 2.264490721333835190E-03 Metric Check mismatch on component 3 of node 152. truth = 7.953857219441792914E+00, test = 7.935845783569552836E+00, relerror = 2.264490721333835190E-03 Metric Check mismatch on component 5 of node 152. truth = 7.953857219441792914E+00, test = 7.935845783569552836E+00, relerror = 2.264490721333835190E-03 Metric Check mismatch on component 0 of node 153. truth = 8.423281109220500440E+00, test = 8.404206667305484046E+00, relerror = 2.264490721333834757E-03 Metric Check mismatch on component 3 of node 153. truth = 8.423281109220500440E+00, test = 8.404206667305484046E+00, relerror = 2.264490721333834757E-03 Metric Check mismatch on component 5 of node 153. truth = 8.423281109220500440E+00, test = 8.404206667305484046E+00, relerror = 2.264490721333834757E-03 Metric Check mismatch on component 0 of node 154. truth = 8.495404281075957797E+00, test = 8.476166516907481352E+00, relerror = 2.264490721333858609E-03 Metric Check mismatch on component 3 of node 154. truth = 8.495404281075957797E+00, test = 8.476166516907481352E+00, relerror = 2.264490721333858609E-03 Metric Check mismatch on component 5 of node 154. truth = 8.495404281075957797E+00, test = 8.476166516907481352E+00, relerror = 2.264490721333858609E-03 Metric Check mismatch on component 0 of node 155. truth = 1.029623579498961128E+01, test = 1.027292006456719164E+01, relerror = 2.264490721333869451E-03 Metric Check mismatch on component 3 of node 155. truth = 1.029623579498961128E+01, test = 1.027292006456719164E+01, relerror = 2.264490721333869451E-03 Metric Check mismatch on component 5 of node 155. truth = 1.029623579498961128E+01, test = 1.027292006456719164E+01, relerror = 2.264490721333869451E-03 Metric Check mismatch on component 0 of node 156. truth = 1.004087415856934307E+01, test = 1.001813669220318204E+01, relerror = 2.264490721333842563E-03 Metric Check mismatch on component 3 of node 156. truth = 1.004087415856934307E+01, test = 1.001813669220318204E+01, relerror = 2.264490721333842563E-03 Metric Check mismatch on component 5 of node 156. truth = 1.004087415856934307E+01, test = 1.001813669220318204E+01, relerror = 2.264490721333842563E-03 Metric Check mismatch on component 0 of node 157. truth = 1.051059448010229502E+01, test = 1.048679333642640188E+01, relerror = 2.264490721333727637E-03 Metric Check mismatch on component 3 of node 157. truth = 1.051059448010229502E+01, test = 1.048679333642640188E+01, relerror = 2.264490721333727637E-03 Metric Check mismatch on component 5 of node 157. truth = 1.051059448010229502E+01, test = 1.048679333642640188E+01, relerror = 2.264490721333727637E-03 Metric Check mismatch on component 0 of node 158. truth = 1.080689836078201260E+01, test = 1.078242623971762448E+01, relerror = 2.264490721333781414E-03 Metric Check mismatch on component 3 of node 158. truth = 1.080689836078201260E+01, test = 1.078242623971762448E+01, relerror = 2.264490721333781414E-03 Metric Check mismatch on component 5 of node 158. truth = 1.080689836078201260E+01, test = 1.078242623971762448E+01, relerror = 2.264490721333781414E-03 Metric Check mismatch on component 0 of node 159. truth = 8.772519072112407557E+00, test = 8.752653784070885123E+00, relerror = 2.264490721333810037E-03 Metric Check mismatch on component 3 of node 159. truth = 8.772519072112407557E+00, test = 8.752653784070885123E+00, relerror = 2.264490721333810037E-03 Metric Check mismatch on component 5 of node 159. truth = 8.772519072112407557E+00, test = 8.752653784070885123E+00, relerror = 2.264490721333810037E-03 Metric Check mismatch on component 0 of node 160. truth = 1.109068096409009563E+01, test = 1.106556621995363976E+01, relerror = 2.264490721333839961E-03 Metric Check mismatch on component 3 of node 160. truth = 1.109068096409009563E+01, test = 1.106556621995363976E+01, relerror = 2.264490721333839961E-03 Metric Check mismatch on component 5 of node 160. truth = 1.109068096409009563E+01, test = 1.106556621995363976E+01, relerror = 2.264490721333839961E-03 Metric Check mismatch on component 0 of node 161. truth = 9.374433318417125349E+00, test = 9.353205001149806463E+00, relerror = 2.264490721333894171E-03 Metric Check mismatch on component 3 of node 161. truth = 9.374433318417125349E+00, test = 9.353205001149806463E+00, relerror = 2.264490721333894171E-03 Metric Check mismatch on component 5 of node 161. truth = 9.374433318417125349E+00, test = 9.353205001149806463E+00, relerror = 2.264490721333894171E-03 Metric Check mismatch on component 0 of node 162. truth = 8.680402927508577449E+00, test = 8.660746235621795464E+00, relerror = 2.264490721333806567E-03 Metric Check mismatch on component 3 of node 162. truth = 8.680402927508577449E+00, test = 8.660746235621795464E+00, relerror = 2.264490721333806567E-03 Metric Check mismatch on component 5 of node 162. truth = 8.680402927508577449E+00, test = 8.660746235621795464E+00, relerror = 2.264490721333806567E-03 Metric Check mismatch on component 0 of node 163. truth = 8.812231441716857461E+00, test = 8.792276225382844146E+00, relerror = 2.264490721333744985E-03 Metric Check mismatch on component 3 of node 163. truth = 8.812231441716857461E+00, test = 8.792276225382844146E+00, relerror = 2.264490721333744985E-03 Metric Check mismatch on component 5 of node 163. truth = 8.812231441716857461E+00, test = 8.792276225382844146E+00, relerror = 2.264490721333744985E-03 Metric Check mismatch on component 0 of node 164. truth = 1.060230339349769402E+01, test = 1.057829457583835087E+01, relerror = 2.264490721333966162E-03 Metric Check mismatch on component 3 of node 164. truth = 1.060230339349769402E+01, test = 1.057829457583835087E+01, relerror = 2.264490721333966162E-03 Metric Check mismatch on component 5 of node 164. truth = 1.060230339349769402E+01, test = 1.057829457583835087E+01, relerror = 2.264490721333966162E-03 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3798 Call : PXAdjustGridHP(pxa, &pg1, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 3.155u 0.074s 0:02.64 121.9% 0+0k 0+3736io 6pf+0w Thu Mar 24 13:25:01 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for hemisphere_Euler3d_Adapt_TEST.job Start 914: Scalar2d_UniAdapt_TEST_2 319/327 Test #910: bumpQ2_Euler3d_Jacobi_TEST_2 ............................ Passed 4.04 sec Start 916: Scalar2d_UniAdapt_Unsteady1_TEST_2 320/327 Test #914: Scalar2d_UniAdapt_TEST_2 ................................***Failed 2.51 sec OUT/Scalar2d_UniAdapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 19011 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_UniAdapt_TEST.job $ * * * * $ $ * * * * $ $ ****** * $ $ * * * $ $ * * * $ $ * * * $ $ $ $$$$$$$$$$$$$$$$$$$$$$$ Done. Writing out OUT/Scalar2d_UniAdapt_TEST_P3_a00_Dual_0.pxa . . .Done. ************************** * Adaptation Iteration 0 * ************************** Writing out OUT/Scalar2d_UniAdapt_TEST_a00_PreMetricOpt_AdaptInfo_0.pxa . . .Done. meshDofReq = 1000.000000, meshDofCurr = 235.925435 ratio = 4.238627339017738e+00 Scaling the continuous metric Scaling the element metric Writing out OUT/Scalar2d_UniAdapt_TEST_a00_AdaptInfo_0.pxa . . .Done. **************************************************************************************** **************************************************************************************** CHECKING Truth Metric File: TrueMetric/Scalar2d_UniAdapt_TEST_Metric_a00.attach Metric Check mismatch on component 0 of node 0. truth = 7.711827223060084613E+00, test = 7.694127420941067719E+00, relerror = 2.295150242226710619E-03 Metric Check mismatch on component 1 of node 0. truth = -9.194972254069227802E+00, test = -9.173868411273039314E+00, relerror = 2.295150242226016296E-03 Metric Check mismatch on component 2 of node 0. truth = 4.207174901887707108E+01, test = 4.197518803392468811E+01, relerror = 2.295150242245865436E-03 Metric Check mismatch on component 0 of node 1. truth = 7.005367655993610221E+00, test = 6.989289284721012407E+00, relerror = 2.295150242234835197E-03 Metric Check mismatch on component 1 of node 1. truth = -5.675765265687729766E+00, test = -5.662738531663335628E+00, relerror = 2.295150242231818946E-03 Metric Check mismatch on component 2 of node 1. truth = 4.075983990378809807E+01, test = 4.066628994736037583E+01, relerror = 2.295150242212506270E-03 Metric Check mismatch on component 0 of node 2. truth = 9.282081631249120335E+00, test = 9.260777859344749174E+00, relerror = 2.295150242231196614E-03 Metric Check mismatch on component 1 of node 2. truth = -8.940651703303574394E+00, test = -8.920131564381271971E+00, relerror = 2.295150242204404677E-03 Metric Check mismatch on component 2 of node 2. truth = 4.408435948426343742E+01, test = 4.398317925591776145E+01, relerror = 2.295150242157737146E-03 Metric Check mismatch on component 0 of node 3. truth = 5.604761424269736558E+00, test = 5.591897652917652728E+00, relerror = 2.295150565442648574E-03 Metric Check mismatch on component 1 of node 3. truth = -6.999072273813476741E+00, test = -6.983008342227929433E+00, relerror = 2.295151551106187210E-03 Metric Check mismatch on component 2 of node 3. truth = 4.127691350232788636E+01, test = 4.118217675976080727E+01, relerror = 2.295150836840943471E-03 Metric Check mismatch on component 0 of node 4. truth = 5.802185966502596770E+00, test = 5.788869077976921851E+00, relerror = 2.295150242090910828E-03 Metric Check mismatch on component 1 of node 4. truth = -6.650322284011406460E+00, test = -6.635058795211555172E+00, relerror = 2.295150242048796946E-03 Metric Check mismatch on component 2 of node 4. truth = 4.231953924997981176E+01, test = 4.222240954921185363E+01, relerror = 2.295150242402614615E-03 Metric Check mismatch on component 0 of node 5. truth = 5.985549539542634356E+00, test = 5.971811804067044704E+00, relerror = 2.295150242234796599E-03 Metric Check mismatch on component 1 of node 5. truth = -6.177374381804296100E+00, test = -6.163196379495562205E+00, relerror = 2.295150242228440572E-03 Metric Check mismatch on component 2 of node 5. truth = 4.464202734367807324E+01, test = 4.453956718380749891E+01, relerror = 2.295150242209689946E-03 Metric Check mismatch on component 0 of node 6. truth = 4.188343010605143135E+00, test = 4.178730134593651613E+00, relerror = 2.295150131484247243E-03 Metric Check mismatch on component 2 of node 6. truth = 3.851510952355631190E+01, test = 3.842671150109329403E+01, relerror = 2.295151787351209886E-03 Metric Check mismatch on component 0 of node 7. truth = 4.767936984075936913E+00, test = 4.756993851478303448E+00, relerror = 2.295150425473655581E-03 Metric Check mismatch on component 1 of node 7. truth = -3.953731460917814466E+00, test = -3.944657045518841887E+00, relerror = 2.295152184378767852E-03 Metric Check mismatch on component 2 of node 7. truth = 3.940694092162517848E+01, test = 3.931649604944973930E+01, relerror = 2.295150804913307155E-03 Metric Check mismatch on component 0 of node 8. truth = 4.772977671633313435E+00, test = 4.762022970774284936E+00, relerror = 2.295150242192480188E-03 Metric Check mismatch on component 1 of node 8. truth = -4.849348281002122008E+00, test = -4.838218298121716998E+00, relerror = 2.295150241942405488E-03 Metric Check mismatch on component 2 of node 8. truth = 4.459975648022846428E+01, test = 4.449739333834298805E+01, relerror = 2.295150242151095324E-03 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 0.173u 0.072s 0:01.17 20.5% 0+0k 0+3512io 4pf+0w Thu Mar 24 13:25:03 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_UniAdapt_TEST.job Start 918: Scalar2d_UniAdapt_Unsteady2_TEST_2 321/327 Test #916: Scalar2d_UniAdapt_Unsteady1_TEST_2 ......................***Failed 2.41 sec OUT/Scalar2d_UniAdapt_Unsteady1_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 19186 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_UniAdapt_Unsteady1_TEST.job $ * * * $ $ * * * $ $ * * * $ $ $ $$$$$$$$$$$$$$$$$$$$$$$ Done. Integrated conservative output error = 3.2266673141E-03 Integrated unconservative output error = -2.2596124574E-03 Writing out OUT/Scalar2d_UniAdapt_Unsteady1_TEST_P3_a00_Dual_0.pxa . . .Done. ************************** * Adaptation Iteration 0 * ************************** Writing out OUT/Scalar2d_UniAdapt_Unsteady1_TEST_a00_PreMetricOpt_AdaptInfo_0.pxa . . .Done. meshDofReq = 1000.000000, meshDofCurr = 235.925445 ratio = 4.238627167228016e+00 Scaling the continuous metric Scaling the element metric Writing out OUT/Scalar2d_UniAdapt_Unsteady1_TEST_a00_AdaptInfo_0.pxa . . .Done. **************************************************************************************** **************************************************************************************** CHECKING Truth Metric File: TrueMetric/Scalar2d_UniAdapt_TEST_Metric_a00.attach Metric Check mismatch on component 0 of node 0. truth = 7.711827223060084613E+00, test = 7.694127109374045759E+00, relerror = 2.295190643419962179E-03 Metric Check mismatch on component 1 of node 0. truth = -9.194972254069227802E+00, test = -9.173868040198675189E+00, relerror = 2.295190598450469639E-03 Metric Check mismatch on component 2 of node 0. truth = 4.207174901887707108E+01, test = 4.197518633129724464E+01, relerror = 2.295190711859874770E-03 Metric Check mismatch on component 0 of node 1. truth = 7.005367655993610221E+00, test = 6.989289001465682993E+00, relerror = 2.295190676276747613E-03 Metric Check mismatch on component 1 of node 1. truth = -5.675765265687729766E+00, test = -5.662738302351963071E+00, relerror = 2.295190644074006272E-03 Metric Check mismatch on component 2 of node 1. truth = 4.075983990378809807E+01, test = 4.066628829876557916E+01, relerror = 2.295190688759906218E-03 Metric Check mismatch on component 0 of node 2. truth = 9.282081631249120335E+00, test = 9.260777483988663406E+00, relerror = 2.295190681014293039E-03 Metric Check mismatch on component 1 of node 2. truth = -8.940651703303574394E+00, test = -8.920131202497280754E+00, relerror = 2.295190718447437730E-03 Metric Check mismatch on component 2 of node 2. truth = 4.408435948426343742E+01, test = 4.398317747421855017E+01, relerror = 2.295190657834229268E-03 Metric Check mismatch on component 0 of node 3. truth = 5.604761424269736558E+00, test = 5.591898225021599167E+00, relerror = 2.295048490813037835E-03 Metric Check mismatch on component 1 of node 3. truth = -6.999072273813476741E+00, test = -6.983012299353957175E+00, relerror = 2.294586172456998269E-03 Metric Check mismatch on component 2 of node 3. truth = 4.127691350232788636E+01, test = 4.118218710028873630E+01, relerror = 2.294900320824806716E-03 Metric Check mismatch on component 0 of node 4. truth = 5.802185966502596770E+00, test = 5.788868836588734190E+00, relerror = 2.295191845064178601E-03 Metric Check mismatch on component 1 of node 4. truth = -6.650322284011406460E+00, test = -6.635058594854731595E+00, relerror = 2.295180369434361104E-03 Metric Check mismatch on component 2 of node 4. truth = 4.231953924997981176E+01, test = 4.222240835152958738E+01, relerror = 2.295178543331393047E-03 Metric Check mismatch on component 0 of node 5. truth = 5.985549539542634356E+00, test = 5.971811560107523675E+00, relerror = 2.295191000317144125E-03 Metric Check mismatch on component 1 of node 5. truth = -6.177374381804296100E+00, test = -6.163196124480595728E+00, relerror = 2.295191524325123928E-03 Metric Check mismatch on component 2 of node 5. truth = 4.464202734367807324E+01, test = 4.453956538499225815E+01, relerror = 2.295190536420051511E-03 Metric Check mismatch on component 0 of node 6. truth = 4.188343010605143135E+00, test = 4.178729748372575159E+00, relerror = 2.295242344819085466E-03 Metric Check mismatch on component 2 of node 6. truth = 3.851510952355631190E+01, test = 3.842673776531868413E+01, relerror = 2.294469867301779809E-03 Metric Check mismatch on component 0 of node 7. truth = 4.767936984075936913E+00, test = 4.756994049981761563E+00, relerror = 2.295108792486730883E-03 Metric Check mismatch on component 1 of node 7. truth = -3.953731460917814466E+00, test = -3.944660576661010776E+00, relerror = 2.294259068039483651E-03 Metric Check mismatch on component 2 of node 7. truth = 3.940694092162517848E+01, test = 3.931650576671247421E+01, relerror = 2.294904217319658751E-03 Metric Check mismatch on component 0 of node 8. truth = 4.772977671633313435E+00, test = 4.762022814212421906E+00, relerror = 2.295183043909521537E-03 Metric Check mismatch on component 1 of node 8. truth = -4.849348281002122008E+00, test = -4.838218360315401867E+00, relerror = 2.295137416778844554E-03 Metric Check mismatch on component 2 of node 8. truth = 4.459975648022846428E+01, test = 4.449739224171178620E+01, relerror = 2.295174830428888500E-03 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 0.212u 0.048s 0:01.18 21.1% 0+0k 0+3048io 5pf+0w Thu Mar 24 13:25:04 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_UniAdapt_Unsteady1_TEST.job Start 920: naca_Euler2d_UniAdapt_TEST_3 322/327 Test #918: Scalar2d_UniAdapt_Unsteady2_TEST_2 ......................***Failed 2.51 sec OUT/Scalar2d_UniAdapt_Unsteady2_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 19214 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_UniAdapt_Unsteady2_TEST.job $ * * * $ $ * * * $ $ * * * $ $ $ $$$$$$$$$$$$$$$$$$$$$$$ Done. Integrated conservative output error = 3.2266673137E-03 Integrated unconservative output error = -2.2596124571E-03 Writing out OUT/Scalar2d_UniAdapt_Unsteady2_TEST_P3_a00_Dual_0.pxa . . .Done. ************************** * Adaptation Iteration 0 * ************************** Writing out OUT/Scalar2d_UniAdapt_Unsteady2_TEST_a00_PreMetricOpt_AdaptInfo_0.pxa . . .Done. meshDofReq = 1000.000000, meshDofCurr = 235.925435 ratio = 4.238627339017688e+00 Scaling the continuous metric Scaling the element metric Writing out OUT/Scalar2d_UniAdapt_Unsteady2_TEST_a00_AdaptInfo_0.pxa . . .Done. **************************************************************************************** **************************************************************************************** CHECKING Truth Metric File: TrueMetric/Scalar2d_UniAdapt_TEST_Metric_a00.attach Metric Check mismatch on component 0 of node 0. truth = 7.711827223060084613E+00, test = 7.694127420940957585E+00, relerror = 2.295150242240991730E-03 Metric Check mismatch on component 1 of node 0. truth = -9.194972254069227802E+00, test = -9.173868411272907863E+00, relerror = 2.295150242240312152E-03 Metric Check mismatch on component 2 of node 0. truth = 4.207174901887707108E+01, test = 4.197518803392431153E+01, relerror = 2.295150242254816609E-03 Metric Check mismatch on component 0 of node 1. truth = 7.005367655993610221E+00, test = 6.989289284720930695E+00, relerror = 2.295150242246499477E-03 Metric Check mismatch on component 1 of node 1. truth = -5.675765265687729766E+00, test = -5.662738531663274344E+00, relerror = 2.295150242242616732E-03 Metric Check mismatch on component 2 of node 1. truth = 4.075983990378809807E+01, test = 4.066628994735991398E+01, relerror = 2.295150242223837483E-03 Metric Check mismatch on component 0 of node 2. truth = 9.282081631249120335E+00, test = 9.260777859344639040E+00, relerror = 2.295150242243062123E-03 Metric Check mismatch on component 1 of node 2. truth = -8.940651703303574394E+00, test = -8.920131564381165390E+00, relerror = 2.295150242216325697E-03 Metric Check mismatch on component 2 of node 2. truth = 4.408435948426343742E+01, test = 4.398317925591723565E+01, relerror = 2.295150242169664671E-03 Metric Check mismatch on component 0 of node 3. truth = 5.604761424269736558E+00, test = 5.591897652917329431E+00, relerror = 2.295150565500331166E-03 Metric Check mismatch on component 1 of node 3. truth = -6.999072273813476741E+00, test = -6.983008342227759790E+00, relerror = 2.295151551130424766E-03 Metric Check mismatch on component 2 of node 3. truth = 4.127691350232788636E+01, test = 4.118217675976399050E+01, relerror = 2.295150836763824604E-03 Metric Check mismatch on component 0 of node 4. truth = 5.802185966502596770E+00, test = 5.788869077976679378E+00, relerror = 2.295150242132700316E-03 Metric Check mismatch on component 1 of node 4. truth = -6.650322284011406460E+00, test = -6.635058795211284277E+00, relerror = 2.295150242089530855E-03 Metric Check mismatch on component 2 of node 4. truth = 4.231953924997981176E+01, test = 4.222240954921317524E+01, relerror = 2.295150242371385256E-03 Metric Check mismatch on component 0 of node 5. truth = 5.985549539542634356E+00, test = 5.971811804066909701E+00, relerror = 2.295150242257351474E-03 Metric Check mismatch on component 1 of node 5. truth = -6.177374381804296100E+00, test = -6.163196379495318844E+00, relerror = 2.295150242267836142E-03 Metric Check mismatch on component 2 of node 5. truth = 4.464202734367807324E+01, test = 4.453956718380714364E+01, relerror = 2.295150242217647990E-03 Metric Check mismatch on component 0 of node 6. truth = 4.188343010605143135E+00, test = 4.178730134593592105E+00, relerror = 2.295150131498455062E-03 Metric Check mismatch on component 2 of node 6. truth = 3.851510952355631190E+01, test = 3.842671150109406142E+01, relerror = 2.295151787331285286E-03 Metric Check mismatch on component 0 of node 7. truth = 4.767936984075936913E+00, test = 4.756993851478049429E+00, relerror = 2.295150425526931975E-03 Metric Check mismatch on component 1 of node 7. truth = -3.953731460917814466E+00, test = -3.944657045518674021E+00, relerror = 2.295152184421225643E-03 Metric Check mismatch on component 2 of node 7. truth = 3.940694092162517848E+01, test = 3.931649604945301490E+01, relerror = 2.295150804830184844E-03 Metric Check mismatch on component 0 of node 8. truth = 4.772977671633313435E+00, test = 4.762022970774171249E+00, relerror = 2.295150242216298809E-03 Metric Check mismatch on component 1 of node 8. truth = -4.849348281002122008E+00, test = -4.838218298121534033E+00, relerror = 2.295150241980135290E-03 Metric Check mismatch on component 2 of node 8. truth = 4.459975648022846428E+01, test = 4.449739333834304489E+01, relerror = 2.295150242149820736E-03 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 0.199u 0.052s 0:01.18 20.3% 0+0k 0+3176io 5pf+0w Thu Mar 24 13:25:06 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for Scalar2d_UniAdapt_Unsteady2_TEST.job Start 922: naca_NavierStokes2d_UniAdapt_TEST_4 323/327 Test #920: naca_Euler2d_UniAdapt_TEST_3 ............................***Failed 3.37 sec OUT/naca_Euler2d_UniAdapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 19236 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_Euler2d_UniAdapt_TEST.job truth = 7.985298362366814728E-02, test = 7.212874816333832151E-02, relerror = 7.724235460329825775E-03 Metric Check mismatch on component 2 of node 92. truth = 8.765999554819137174E-02, test = 7.918058230450611412E-02, relerror = 8.479413243685257617E-03 Metric Check mismatch on component 0 of node 93. truth = 4.606881178689954481E+02, test = 4.161254310872638484E+02, relerror = 9.673070577089154520E-02 Metric Check mismatch on component 1 of node 93. truth = 1.628972631640661461E+02, test = 1.471400959299383828E+02, relerror = 9.673070577163429828E-02 Metric Check mismatch on component 2 of node 93. truth = 6.061240404558727732E+02, test = 5.474932342378741623E+02, relerror = 9.673070577088761779E-02 Metric Check mismatch on component 0 of node 94. truth = 7.496998831872647706E-02, test = 6.771808842827668762E-02, relerror = 7.251899890449789443E-03 Metric Check mismatch on component 2 of node 94. truth = 9.048761304945414363E-02, test = 8.173468238717954137E-02, relerror = 8.752930662274602258E-03 Metric Check mismatch on component 0 of node 95. truth = 8.536845745875788760E-02, test = 7.711070620882849724E-02, relerror = 8.257751249929390358E-03 Metric Check mismatch on component 2 of node 95. truth = 8.190910815463811612E-02, test = 7.398598239184563108E-02, relerror = 7.923125762792485038E-03 Metric Check mismatch on component 0 of node 96. truth = 4.531550328203036315E+01, test = 4.093210269637659593E+01, relerror = 9.673070512695779166E-02 Metric Check mismatch on component 1 of node 96. truth = 3.843791383786818283E+00, test = 3.471978746727218734E+00, relerror = 9.673070152243745667E-02 Metric Check mismatch on component 2 of node 96. truth = 1.379123223145958264E+02, test = 1.245719661759925145E+02, relerror = 9.673070480368124724E-02 Metric Check mismatch on component 0 of node 97. truth = 1.068722860981634994E+02, test = 9.653445436709846206E+01, relerror = 9.673070642064877933E-02 Metric Check mismatch on component 1 of node 97. truth = -1.705903407041111208E+01, test = -1.540890185397205059E+01, relerror = 9.673069469397538311E-02 Metric Check mismatch on component 2 of node 97. truth = 1.069349644886111577E+02, test = 9.659107009512418074E+01, relerror = 9.673070396529311055E-02 Metric Check mismatch on component 0 of node 98. truth = 5.949477331650299305E+01, test = 5.373980192685726109E+01, relerror = 9.673070538533148910E-02 Metric Check mismatch on component 1 of node 98. truth = 1.948704855765216903E+00, test = 1.760205266684240266E+00, relerror = 9.673070220115845275E-02 Metric Check mismatch on component 2 of node 98. truth = 7.455801561809482791E+01, test = 6.734596612383703018E+01, relerror = 9.673070607457895376E-02 Metric Check mismatch on component 0 of node 99. truth = 9.931651792990759020E+01, test = 8.970956105632708955E+01, relerror = 9.673070576598938319E-02 Metric Check mismatch on component 1 of node 99. truth = 1.211817659113387968E+01, test = 1.094597679196315809E+01, relerror = 9.673070782186386773E-02 Metric Check mismatch on component 2 of node 99. truth = 1.960034028127628005E+02, test = 1.770438553431648074E+02, relerror = 9.673070567917424478E-02 Metric Check mismatch on component 0 of node 100. truth = 6.777398968691479286E-01, test = 6.121816383164805364E-01, relerror = 6.555825855266739222E-02 Metric Check mismatch on component 1 of node 100. truth = 8.229158808098210875E-02, test = 7.433146468627035985E-02, relerror = 7.960123394711748901E-03 Metric Check mismatch on component 2 of node 100. truth = 7.397670318342941531E-01, test = 6.682088447380474250E-01, relerror = 7.155818709624672813E-02 Metric Check mismatch on component 0 of node 101. truth = 4.906440628605547971E+01, test = 4.431837164166719134E+01, relerror = 9.673070569157486698E-02 Metric Check mismatch on component 1 of node 101. truth = -3.961157717943862178E+00, test = -3.577992135576447019E+00, relerror = 9.673070593268545680E-02 Metric Check mismatch on component 2 of node 101. truth = 4.986612824656184273E+01, test = 4.504254246546835816E+01, relerror = 9.673070580582041056E-02 Metric Check mismatch on component 0 of node 102. truth = 5.113330320520434924E+01, test = 4.618714269814776685E+01, relerror = 9.673070576346340377E-02 Metric Check mismatch on component 1 of node 102. truth = 4.437038559660153147E+00, test = 4.007840693073529081E+00, relerror = 9.673070468413907941E-02 Metric Check mismatch on component 2 of node 102. truth = 1.826776942589864348E+02, test = 1.650071520381949028E+02, relerror = 9.673070536865650826E-02 Metric Check mismatch on component 0 of node 103. truth = 1.752938520059630036E+02, test = 1.583375539843617332E+02, relerror = 9.673070576955811184E-02 Metric Check mismatch on component 1 of node 103. truth = -8.092023027220204057E+00, test = -7.309275928227537911E+00, relerror = 9.673070582716294941E-02 Metric Check mismatch on component 2 of node 103. truth = 1.796388397005998172E+02, test = 1.622622479523017773E+02, relerror = 9.673070577197687148E-02 Metric Check mismatch on component 0 of node 104. truth = 2.242212568974180975E+00, test = 2.025321764688707837E+00, relerror = 9.673070577099714129E-02 Metric Check mismatch on component 1 of node 104. truth = 1.715867243339433135E-01, test = 1.549890193886584999E-01, relerror = 1.659770494528481355E-02 Metric Check mismatch on component 2 of node 104. truth = 2.218986706081596871E+00, test = 2.004342555906923895E+00, relerror = 9.673070577051941232E-02 Metric Check mismatch on component 0 of node 105. truth = 4.183811148446809369E+01, test = 3.779108143604350545E+01, relerror = 9.673070568510246103E-02 Metric Check mismatch on component 1 of node 105. truth = -1.337306172962486883E+01, test = -1.207947603666904968E+01, relerror = 9.673070528719572569E-02 Metric Check mismatch on component 2 of node 105. truth = 6.846547743771076000E+01, test = 6.184276352277562694E+01, relerror = 9.673070520775109427E-02 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 3.723u 0.121s 0:02.32 165.5% 0+0k 0+8416io 6pf+0w Thu Mar 24 13:25:08 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_Euler2d_UniAdapt_TEST.job Start 924: diamond_Euler2d_UniAdapt_TEST_2 324/327 Test #912: box_Euler2d_TEST_3 ...................................... Passed 10.62 sec 325/327 Test #922: naca_NavierStokes2d_UniAdapt_TEST_4 .....................***Failed 7.57 sec OUT/naca_NavierStokes2d_UniAdapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 19261 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_NavierStokes2d_UniAdapt_TEST.job truth = 1.686472649754225672E-01, test = 1.533214631006002138E-01, relerror = 1.532580187482235345E-02 Metric Check mismatch on component 0 of node 93. truth = 4.677796495522857754E+02, test = 4.252702247385055898E+02, relerror = 9.087489131788047125E-02 Metric Check mismatch on component 1 of node 93. truth = 2.117827672408571402E+02, test = 1.925370312838255415E+02, relerror = 9.087489132269073455E-02 Metric Check mismatch on component 2 of node 93. truth = 1.015134596221067795E+03, test = 9.228843501212710407E+02, relerror = 9.087489131313897239E-02 Metric Check mismatch on component 0 of node 94. truth = 1.509589101107965958E-01, test = 1.372405355624078671E-01, relerror = 1.371837454838872872E-02 Metric Check mismatch on component 1 of node 94. truth = -1.219869690263815191E-02, test = -1.109014164538430804E-02, relerror = 1.108555257253843879E-03 Metric Check mismatch on component 2 of node 94. truth = 1.991518288219722987E-01, test = 1.810539280205614165E-01, relerror = 1.809790080141088220E-02 Metric Check mismatch on component 0 of node 95. truth = 1.495038087521665393E-01, test = 1.359176663805774510E-01, relerror = 1.358614237158908833E-02 Metric Check mismatch on component 2 of node 95. truth = 1.726849007632697552E-01, test = 1.569921791749112616E-01, relerror = 1.569272158835849362E-02 Metric Check mismatch on component 0 of node 96. truth = 1.394150364507602546E+02, test = 1.267457101650311415E+02, relerror = 9.087489131922846242E-02 Metric Check mismatch on component 1 of node 96. truth = -4.635756550986517510E+01, test = -4.214482678151132689E+01, relerror = 9.087489133693510412E-02 Metric Check mismatch on component 2 of node 96. truth = 4.906355841517121803E+02, test = 4.460491287671105169E+02, relerror = 9.087489131406913112E-02 Metric Check mismatch on component 0 of node 97. truth = 1.691525626972837415E+02, test = 1.537808419464861061E+02, relerror = 9.087489131516701679E-02 Metric Check mismatch on component 1 of node 97. truth = -6.124692670853858090E+01, test = -5.568111890039930501E+01, relerror = 9.087489131700926537E-02 Metric Check mismatch on component 2 of node 97. truth = 5.478430640288748918E+02, test = 4.980578851279671539E+02, relerror = 9.087489131428290456E-02 Metric Check mismatch on component 0 of node 98. truth = 2.002594279385535572E+02, test = 1.820608741899818881E+02, relerror = 9.087489131425866007E-02 Metric Check mismatch on component 1 of node 98. truth = -1.206277575970063651E+02, test = -1.096657232359377474E+02, relerror = 9.087489131390985575E-02 Metric Check mismatch on component 2 of node 98. truth = 4.541467351335419949E+02, test = 4.128761999376506537E+02, relerror = 9.087489131405013243E-02 Metric Check mismatch on component 0 of node 99. truth = 2.002672718571093071E+02, test = 1.820680052933318223E+02, relerror = 9.087489131405684928E-02 Metric Check mismatch on component 1 of node 99. truth = 9.518021050299493879E+01, test = 8.653071921822346724E+01, relerror = 9.087489131471616910E-02 Metric Check mismatch on component 2 of node 99. truth = 6.892976291020887629E+02, test = 6.266577819731253385E+02, relerror = 9.087489131590516245E-02 Metric Check mismatch on component 0 of node 100. truth = 9.742106136372452596E-01, test = 8.856793300042814510E-01, relerror = 8.853128363296380865E-02 Metric Check mismatch on component 1 of node 100. truth = 2.349238910648120127E-01, test = 2.135752079969326089E-01, relerror = 2.134868306787940373E-02 Metric Check mismatch on component 2 of node 100. truth = 1.757792204754540899E+00, test = 1.598053029194772323E+00, relerror = 9.087489131405873666E-02 Metric Check mismatch on component 0 of node 101. truth = 1.950975219460661947E+02, test = 1.773680558435266903E+02, relerror = 9.087489131430757927E-02 Metric Check mismatch on component 1 of node 101. truth = 5.899141546321893514E+01, test = 5.363057699462665795E+01, relerror = 9.087489131252923791E-02 Metric Check mismatch on component 2 of node 101. truth = 2.605903429638104285E+02, test = 2.369092238692988417E+02, relerror = 9.087489131475723347E-02 Metric Check mismatch on component 0 of node 102. truth = 1.107395141689109153E+02, test = 1.006760728546707639E+02, relerror = 9.087489131377612939E-02 Metric Check mismatch on component 1 of node 102. truth = -1.240646602392492781E+01, test = -1.127902977204868051E+01, relerror = 9.087489134311672878E-02 Metric Check mismatch on component 2 of node 102. truth = 5.560620948694776189E+02, test = 5.055300124366461887E+02, relerror = 9.087489130992221220E-02 Metric Check mismatch on component 0 of node 103. truth = 3.021659487174943592E+02, test = 2.747066509692043041E+02, relerror = 9.087489131332505965E-02 Metric Check mismatch on component 1 of node 103. truth = -8.414263095525949154E+01, test = -7.649617851187940687E+01, relerror = 9.087489131931082709E-02 Metric Check mismatch on component 2 of node 103. truth = 5.723034438466573874E+02, test = 5.202954305873770409E+02, relerror = 9.087489131590013869E-02 Metric Check mismatch on component 0 of node 104. truth = 1.963204012367812723E+00, test = 1.784798061120999924E+00, relerror = 9.087489131179905810E-02 Metric Check mismatch on component 1 of node 104. truth = 3.199897277324385558E-01, test = 2.909106960026123234E-01, relerror = 2.907903172982623241E-02 Metric Check mismatch on component 2 of node 104. truth = 2.326944754425695816E+00, test = 2.115483902775885383E+00, relerror = 9.087489131300853507E-02 Metric Check mismatch on component 0 of node 105. truth = 7.111633432505460917E+01, test = 6.465364517396521649E+01, relerror = 9.087489129501656604E-02 Metric Check mismatch on component 1 of node 105. truth = -4.159180198857170296E+01, test = -3.781215151027621602E+01, relerror = 9.087489114643390598E-02 Metric Check mismatch on component 2 of node 105. truth = 2.555361371842895153E+02, test = 2.323143185172100118E+02, relerror = 9.087489121091399391E-02 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 21.312u 0.362s 0:06.52 332.3% 0+0k 0+9360io 7pf+0w Thu Mar 24 13:25:14 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for naca_NavierStokes2d_UniAdapt_TEST.job 326/327 Test #924: diamond_Euler2d_UniAdapt_TEST_2 .........................***Failed 17.82 sec OUT/diamond_Euler2d_UniAdapt_TEST -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 19290 on node reynolds exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for diamond_Euler2d_UniAdapt_TEST.job truth = 1.193696708204909243E+02, test = 1.174863359195899193E+02, relerror = 1.577733177913491144E-02 Metric Check mismatch on component 1 of node 607. truth = -5.046603983252437331E+01, test = -4.966982037851763465E+01, relerror = 1.577733177893603581E-02 Metric Check mismatch on component 2 of node 607. truth = 2.217231445236181457E+02, test = 2.182249449084700359E+02, relerror = 1.577733178313046533E-02 Metric Check mismatch on component 0 of node 608. truth = 8.499264393367904802E+01, test = 8.365168679296452581E+01, relerror = 1.577733176250982125E-02 Metric Check mismatch on component 1 of node 608. truth = 1.171887608195876851E+01, test = 1.153398348435548115E+01, relerror = 1.577733191393071097E-02 Metric Check mismatch on component 2 of node 608. truth = 1.126402739569529388E+02, test = 1.108631109809656863E+02, relerror = 1.577733179756310097E-02 Metric Check mismatch on component 0 of node 609. truth = 2.475091475436166775E+02, test = 2.436041136037719070E+02, relerror = 1.577733178187531657E-02 Metric Check mismatch on component 1 of node 609. truth = -1.659214017326280199E+01, test = -1.633036047287377102E+01, relerror = 1.577733177609436099E-02 Metric Check mismatch on component 2 of node 609. truth = 2.344325301336891414E+02, test = 2.307338103253792383E+02, relerror = 1.577733178156054059E-02 Metric Check mismatch on component 0 of node 610. truth = 7.805174364724015845E+01, test = 7.682029539078619962E+01, relerror = 1.577733179183757675E-02 Metric Check mismatch on component 1 of node 610. truth = -1.594031327093176564E+01, test = -1.568881765992177790E+01, relerror = 1.577733177105163434E-02 Metric Check mismatch on component 2 of node 610. truth = 1.177632282542359832E+02, test = 1.159052387308866372E+02, relerror = 1.577733177743888271E-02 Metric Check mismatch on component 0 of node 611. truth = 9.149078463810596418E+01, test = 9.004730417380649499E+01, relerror = 1.577733178274939516E-02 Metric Check mismatch on component 1 of node 611. truth = -3.057915943501598122E+01, test = -3.009670189068166479E+01, relerror = 1.577733179224860213E-02 Metric Check mismatch on component 2 of node 611. truth = 2.289361199431200475E+02, test = 2.253241188219484457E+02, relerror = 1.577733178176084911E-02 Metric Check mismatch on component 0 of node 612. truth = 1.633157134724444575E+02, test = 1.607390272758124468E+02, relerror = 1.577733178177471302E-02 Metric Check mismatch on component 1 of node 612. truth = -9.091322313219885132E+01, test = -8.947885504742275486E+01, relerror = 1.577733178253235349E-02 Metric Check mismatch on component 2 of node 612. truth = 2.536424762840613596E+02, test = 2.496406747817239307E+02, relerror = 1.577733178198315045E-02 Metric Check mismatch on component 0 of node 613. truth = 6.613002148202569686E+01, test = 6.508666619221189364E+01, relerror = 1.577733178413362122E-02 Metric Check mismatch on component 1 of node 613. truth = 1.697620428075968135E+01, test = 1.670836507080748490E+01, relerror = 1.577733193607697842E-02 Metric Check mismatch on component 2 of node 613. truth = 9.517446569542507007E+01, test = 9.367286657064629196E+01, relerror = 1.577733180645487371E-02 Metric Check mismatch on component 0 of node 614. truth = 7.726476289486389248E+01, test = 7.604573109517954776E+01, relerror = 1.577733178762370445E-02 Metric Check mismatch on component 1 of node 614. truth = 1.726150050003888126E+01, test = 1.698916007944531970E+01, relerror = 1.577733179064868055E-02 Metric Check mismatch on component 2 of node 614. truth = 1.415569622883501211E+02, test = 1.393235711290283518E+02, relerror = 1.577733177667639888E-02 Metric Check mismatch on component 0 of node 615. truth = 7.303414926219728898E+01, test = 7.188186525724496789E+01, relerror = 1.577733179057850751E-02 Metric Check mismatch on component 1 of node 615. truth = 3.084952100541391928E+01, test = 3.036279787500603078E+01, relerror = 1.577733185297986590E-02 Metric Check mismatch on component 2 of node 615. truth = 1.000634318850690079E+02, test = 9.848469791933057138E+01, relerror = 1.577733179841104075E-02 Metric Check mismatch on component 0 of node 616. truth = 1.034342730992886885E+02, test = 1.018023562550383758E+02, relerror = 1.577733178135067721E-02 Metric Check mismatch on component 1 of node 616. truth = -6.529771250179541653E+01, test = -6.426748882708457700E+01, relerror = 1.577733178145425408E-02 Metric Check mismatch on component 2 of node 616. truth = 2.582949481072737399E+02, test = 2.542197430133660419E+02, relerror = 1.577733178201845554E-02 Metric Check mismatch on component 0 of node 617. truth = 6.262640694799255670E+01, test = 6.163832934487243875E+01, relerror = 1.577733182011633803E-02 Metric Check mismatch on component 1 of node 617. truth = 5.595021800975747972E+00, test = 5.506747284013835397E+00, relerror = 1.577733208233753071E-02 Metric Check mismatch on component 2 of node 617. truth = 8.854344178477069249E+01, test = 8.714646252714794628E+01, relerror = 1.577733177594892178E-02 Metric Check mismatch on component 0 of node 618. truth = 7.638313066121988015E+01, test = 7.517800866619306532E+01, relerror = 1.577733178248298326E-02 Metric Check mismatch on component 1 of node 618. truth = 1.317822778078905976E+01, test = 1.297031050898597826E+01, relerror = 1.577733176733967999E-02 Metric Check mismatch on component 2 of node 618. truth = 1.639142544003848911E+02, test = 1.613281248252218063E+02, relerror = 1.577733178010302592E-02 Metric Check mismatch on component 0 of node 619. truth = 6.387554618380661964E+01, test = 6.286776049918143627E+01, relerror = 1.577733177772297143E-02 Metric Check mismatch on component 1 of node 619. truth = 2.613227798032726668E+01, test = 2.571998035957479090E+01, relerror = 1.577733181404464627E-02 Metric Check mismatch on component 2 of node 619. truth = 1.201680988766791955E+02, test = 1.182721669088682859E+02, relerror = 1.577733180048544390E-02 Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/Grid/PXMeshMetric.c Line : 250 Call : PX_CODE_FLOW_ERROR Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 2907 Call : PXCheckTruthMetricFile(pg_meshing, CheckTruthMetricFileRoot, aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3321 Call : PXAdjustGridH(pg0,pxa->Parameter,&pgH, aiter, &EfficiencyIndex) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdaptDirect.c Line : 8351 Call : PXAdjustGridHP(pxa, pg1,aiter) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXHPAdapt.c Line : 3717 Call : PXAdaptHPDirect(pxa,aiter,DoneAdapt, &pg1, adaptModeHP) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 790 Call : PXAdaptHP(pxa, aiter, PrimalSolveConvergedFlag, DoneAdapt) Error -9 (Code Flow) has occured. File : /home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds/src/PXRunSolver.c Line : 1026 Call : PXAdaptIteration( pxa, aiter, ConvergedFlag, &DoneAdapt ) 30.965u 0.360s 0:16.72 187.3% 0+0k 0+23680io 5pf+0w Thu Mar 24 13:25:26 EDT 2016 > ERROR: ../platforms/coverage_release_gnu49/bin/ProjectX exited with error code 247 for diamond_Euler2d_UniAdapt_TEST.job 327/327 Test #808: CutCell_bump_Euler_Test ................................. Passed 755.21 sec 87% tests passed, 42 tests failed out of 327 Label Time Summary: PXRegTest = 3014.61 sec PXUnit = 866.55 sec Total Test time (real) = 1388.93 sec The following tests FAILED: 10 - PXHPAdapt_PXU (Failed) 136 - PXMeshMetric_PXU (Failed) 292 - PXMetric_PXU (Failed) 673 - Wedge_PDEShock_Adapt_TEST (Failed) 727 - naca_Euler2d_Adapt_TEST (Failed) 733 - naca_Euler2d_AdaptNLOPTVertWPenalty_TEST (Failed) 736 - naca_Euler2d_CurvedAdapt_TEST (Failed) 739 - Scalar2d_Adapt_TEST (Failed) 742 - Scalar2d_CircleBoundaryLayer_Adapt_TEST (Failed) 745 - poissonLshape_TEST (Failed) 748 - PoissonLshape_CutCell_Adapt_TEST (Failed) 751 - AdDiffBL2_DofControl_TEST (Failed) 754 - hemisphere_Euler3d_Adapt_TEST (Failed) 763 - Scalar2d_UniAdapt_TEST (Failed) 766 - Scalar2d_UniAdapt_Unsteady1_TEST (Failed) 769 - Scalar2d_UniAdapt_Unsteady2_TEST (Failed) 772 - naca_Euler2d_UniAdapt_TEST (Failed) 775 - naca_NavierStokes2d_UniAdapt_TEST (Failed) 778 - diamond_Euler2d_UniAdapt_TEST (Failed) 781 - RAE2822_RANS_Subsonic_UniAdapt_TEST (Failed) 784 - RAE2822_RANS_Transonic_UniAdapt_TEST (Failed) 787 - CutCell_rae2822_RANS_M0p3_Re6p5e6_UniAdapt_Test (Failed) 790 - RAE2822_RANS_Subsonic_NoMaxQuad_UniAdapt_TEST (Failed) 793 - CutCell_rae2822_Euler_M0p2_TEST (Failed) 796 - CutCell_rae2822_RANS_M0p3_Re6p5e6_TEST (Failed) 799 - CutCell_Circle_Interface_Scalar2d_UniAdapt_Test (Failed) 802 - CutCell_Interface_StarShape_Jump_TEST (Failed) 805 - CutCell_rae2822_CHT_TEST (Failed) 811 - CHT_NS2d_Scalar2d_FullEmb_FlatSlab_TEST (Failed) 896 - naca_Euler2d_Adapt_TEST_2 (Failed) 898 - naca_Euler2d_CurvedAdapt_TEST_3 (Failed) 900 - Scalar2d_Adapt_TEST_4 (Failed) 902 - Scalar2d_CircleBoundaryLayer_Adapt_TEST_2 (Failed) 904 - poissonLshape_TEST_3 (Failed) 906 - AdDiffBL2_DofControl_TEST_4 (Failed) 908 - hemisphere_Euler3d_Adapt_TEST_2 (Failed) 914 - Scalar2d_UniAdapt_TEST_2 (Failed) 916 - Scalar2d_UniAdapt_Unsteady1_TEST_2 (Failed) 918 - Scalar2d_UniAdapt_Unsteady2_TEST_2 (Failed) 920 - naca_Euler2d_UniAdapt_TEST_3 (Failed) 922 - naca_NavierStokes2d_UniAdapt_TEST_4 (Failed) 924 - diamond_Euler2d_UniAdapt_TEST_2 (Failed) Errors while running CTest make[4]: *** [test/CMakeFiles/commit_check] Error 8 make[3]: *** [test/CMakeFiles/commit_check.dir/all] Error 2 make[2]: *** [test/CMakeFiles/commit_check.dir/rule] Error 2 make[1]: *** [commit_check] Error 2 make: *** [default] Error 2 Build step 'Execute shell' marked build as failure [WARNINGS] Skipping publisher since build result is FAILURE Skipping Cobertura coverage report as build was not UNSTABLE or better ... [xUnit] [INFO] - Starting to record. [xUnit] [INFO] - Processing CTest-Version N/A (default) [xUnit] [INFO] - [CTest-Version N/A (default)] - 1 test report file(s) were found with the pattern 'platforms/coverage_release_gnu49/Testing/**/Test.xml' relative to '/home1/jenkins/workspace/ProjectX_Commit/PX_ARC/coverage_release_gnu49/buildnode/reynolds' for the testing framework 'CTest-Version N/A (default)'. [xUnit] [INFO] - Check 'Failed Tests' threshold. [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value. [xUnit] [INFO] - Setting the build status to FAILURE [xUnit] [INFO] - Stopping recording. Warning: this build has no associated authentication, so build permissions may be lacking, and downstream projects which cannot even be seen by an anonymous user will be silently skipped Finished: FAILURE