Article Number: 000053970, ® Print VPLEX: Logrotate job failed due to corrupt logrotate.status file and VPLEX director dials home with the message, "The filesystem mounted at '/var/log' is XX% full" ‘Summary: Logrotate job failed due to corrupt logrotate. status file and VPLEX director dials home with the message, “The filesystern mounted at ‘/var/log' is XX% full” Article Content ‘Symptoms elow is @ VPLEX Call Home message that is reported due toa erector’s'/var/ag'fle system showing a high utilization which exceeded the threshold ‘Sample outout af call home message: symptoncede: oxéa4h9002 Status: Failed Component iD: dixactox-1-2-8 aah root#i28.221.252.68 << Denotes drector-1-2-B (Custer, engine-1-2,Ditector 8) 2) Once onthe diector the disk filesystem (éf) command was run to check the usage ofthe file system reported in the Call Home message, inthis case Yvar/log. The “hi option displays the sizes in human readable values, Sample output 16dixector-2-i-bi- # ae -h fee 3032 2.06 945m 970M S08 / dew 185 200K 186 1% /aow tapte 286 4.0K 18614 /dev/atm (see) s3a1 126 29m BBM 258 /boot Pou) 2835 1204 7.6 112M 79 /var/opt/zephye/flashDie fsev/sds6 «1007 6924 EAM 9E8 /vax/Log < Use has exceeded the threshold in percentage Po! 837 6.06 1.56 4.26 268 /cores tapes 2.06 132 1.06 18 /emp 3) Change directories (ed) to the /varlog deectory as follows: Samole output duzector~ ~ 4 ea fvar/iog 3.1) The disk usage (du) command was used to check the disk usage of flles inthe partition ‘Narllogh NOTE: In the below sample output you see the following options were usec, the “option is {or reporting only the sum ofthe usage, the “K option lists the usage in 1024-bytes. The ‘sort option sorts the output according to the numerical value, smallest to largest, and the ‘ail option just shows that last numberof lines indicated by the value entere, inthis case 20, forthe last 20 lines, andthe asterisk "isa wildcard that says show all les you are looking for, files that look to be large in size. For those noted the stens further dawn were used ta check each file noted with a large filesize, 10M to 20MB,, Sample output Gixector-1-2-b:/var/log 4 du -#k # |aort -m [east -20 712 vpspiitter-1og.27.82 716 vpspiitter-10g.17.¢2 16 upspiitter.teg.25.92 716 vpspiitter-1og.29.62 2212 /eleg-ng- tog cimonlog.txt cimorlog.txt-t securizylog.taz¢ securityleg.txt1 fL1_plip_server.teg 1 FL_bitp serv! acter. log.00 113784 ecomeri.ieg < logrotate job falled, filesize is 113MB 384436 netw.log /dov/nsi1 Note: the Yotate’ value sted above is how many times a log will be rotated before it starts to lover write the previous rotated logs cause Inttcduction: Logrotate is designed to ease administration of systems that generate large numbers of logfiles. It allows automatic rotation, compression, removal, and mailing of log files. Eaci log file may be handled dally, weekly, monthly, or when it grows too large. Normally, logrotate is run as a dally cron job. It will not modify log multiple times in one day unless the criterium for that log is. based on the logs size and logrotate is being run multiple times each day or unless the for force option is used, cau Ifyou find the size ofthe large log fle exceeds the fle size defined Inthe logrotate Fle, It means the logrotate process may have fale for some reason, Inthe sample output below, where you see ‘error Iisted, this reports corrupt Togrotate status fl, which isthe cause of ths failure Sample outout -2-bi~ $ logrotate -v /etc/logrotate.cont error: bad line ¢ in state file /var/lib/logrotate.status < why logtote filed ‘The below command was used to check the logrotate status’ lle to see what the reason was forthe error. 36Gixector-I-2-bi- # eat /var/1ib/Logrotate statue "/uar/log/nafu_alarna.log" 2017-10-23 ar/log/wenp” 2017-10-22 < abnormal line, shoulé read "/var/og/wimp* Resolution Permanent Fix ‘The logrotate issue is permanently fixed in GeoSynchrony 6.5 SP2 PS and 6.1 and later. NOTE: For VS2s, GeoSynchrony code versions 4.x through 5.4 Service Pack 1 Patch 4 are End of Service Life (EOSL) and no longer supported. If you are running your VPLE) local field representative and discuss the planning for the upgrade ofthe VPLEX to atleast the target code 6.0 SPI P7, which has many fies, enhancements and vulnerability fixes not avallabe inthe EOSL code versions and that you will benefit from. You may also choose to upgrade tothe latest code release. fon one ofthese £ SL code versions it is strongly recommended that you contact your Also to make you aware GeoSynchrony 5.5.xis at End of Life (EOL) and will be going EOSL on 28 February 2021. Ifyou are running (on any version of §.5x you have until 28 February 2021 to plan and upgrade the VELEX to atleast target code 6.0 SPI P7 or the latest GA code release. you are running on GeaSynchrony 5.5 SP2 PS you willnt be abe to upgrade to 6.0 SP1 P7, you will need to upgrade to 6.2 Instead. This fe because 6.5 SP2 PS and the 6.0.x code releases were released in parallel and there ate fixes, enhancements and vulnerability fixes in 5.5 SP2.P5 that were notin 6.0 SP1 P7 yet where added in 6.1.x and later releases. Workaround: For those code versions the fix isnot available for, to fc the corrupted Togrotate. status file or the affected director (in the example ‘mentioned above, the affected director i drector-I-2, follow the steps below. 1 Login into the management server with the service account credentials. NOTE: I this is a VPLEXMeteo, be sure you access the management server forthe director where the issue was reported as shown in the Issue section above In the sample output mentioned in issue" section above the director reporting the issue is director--2-8 of cluster a8 follows: 21Next ssh 10 root ofthe affected director the report came from using that directors internal Primary IP ‘Address, n this case director--2-8 primary IP address is 128.221.252.68, > gob zoote126.221.252.68 et Login: Thy Feb § 13152237 2016 f£om 126.221.252.33 dizectort-2-b~ 3) Please follow the below steps to replace and remove the corupted logrotate status file 3.1) From the affected director context, Change Directory (ed) to the vars folder as follows: ixector-l-2-pi~ # ed /var/1ib/ 3.2) To ensure the files there type the 'g- command on the logrotate status’ file as 46t-2-p i ‘ib # is “1 logrotate. status 3.3) Make a backup copy ofthe orginal Togrotate status’ file using the copy (cp) command, NOTE: The-p option preserves the specified attributes, the -r option performs ‘a recursive copy of the file, L-2-bi/vac/Lib 4 op -pr logrotate.status logrotate. status.ori 3.4) Run the’ Ig-"commane again on the logrotate. status’ fle using the asterisk wildcard, to ensue the orginal and backup files are present as follows: 1-P-b:/var/lib # te -1 Legsotate.statuer - 1 xoot zeot 1831 Dee 28 05:16 logratate.status ~ 1 root root 1531 Dac 28 05:16 logrotate.statws.ori 3.5) Please remove the current/orgial logrotate status! le as fllows: NOTE: -roptionis for recursively removing the contents ofthe fle the {option forces the command trun without prompts, 1-2-b:/vas/Lib # xm af logrotate. status 4) Please follow the below stops to re-create the logrotate statue. 43) inthis step you wil re-create the orotate status’ le trough syslog, 4) From the affected etector context, Change Directory (cd) tothe Yustsbi folder. Gurector-i-2+b:/var/lib # ed /usr/abin bi /usr/sbin # b] Manually sun the logrotate command which will e-veate the logrotate status file Gizeotor-1-2-b:/uss/sbin 4 logrotate =f /ete/legrotate.4/* error: /ete/logrotate.d/syslog, innew:1? dyplicate log ent: for var/ lagi ror in /vac/log/uarn /var/loe/messages /v Log/tix s/log/alinessages fvar/log/locainessages /¥: + skipping grrori f/ets/logrotave.d) log-rpmnew:28 duplicate log entry for /var/log/mail error: found ersor in /var/log/mail AoginatlAnto /vaz/leg/natl.warn /var/log/aaller ) skipping error; /ete/logrotate.d/sysiog.rpanew:44 duplicate leg entry for /var/1o9/news/news.eri error: found exzor in /var/Log/news/s fear /log/news/news notice » skipping error: MV At /var/Log/newe/ og/rabbitma” has insecure permissions. Tt must be owned and be writable By soot only to avoid security problens. Set the "eu" directive in the config file to cell 8 £9 error: "/vaz/log/sabbltng” nas Insecure permissions. TE must be owned and be weltable by logrotate which user/group should be u eavion, root only to avoid security problens. Set the "si Gizective in the config file to tell Reload sysloy serv: ©) Verity shat logrotate status files availble under /var/tby folder, showing the curent date on which itwas recreated, Gsroctor-I-2-b:/var/1sb # le -1 logrotate. status 566) Manually verity whether the log rotation is working now by running the logrotate ‘command with the verbose, '»v' option, against the Yete/logrotate conf file Fno errs are reported the logrotate is now working agin oirectos cl-si/var/iin # logrotate -v /ate/logrotace. cont reading config fle /ete/Loe compress _prag is now /ust/oin/arip? uncompress prou Le now /usz/bin/bunelez including /ete/logratate.d nt fvas/log/nypp/nistory 10885760 byt enpty log files are not rotated, old lage a considering log /var/Los/zypp/nistory log /var/log/aypp/nistory does not oxist rotating pattern: /var/log/zypp-refresh.1og 10485760 bytes (89 rotations) ing log /vat/Loy/2ypp~ Noslaype-re8 og doo ~~ skipping 5) Confirm ifthe use percentage ofthe /var/log/ partition on the affected director is reduced sing the f+ commane again, ‘Sample output: dizector-2-L-pi~ # df“ Filesystem Size Used Avail Usot Mounted on /ee0/ 2d 2.06 945K 970M So 1ec 200K 1861s 166 4.0K 186 1¥ feev/sda5 ize 7.64 tue 78 /eov/oda6 108M 114M B4SM ze feev/ada 6.09 1.8G 4.26 26 tof 1.06 132K 1.06 1 [NOTE: I the issue sil persists even after following above resolution steps;please open a Live Chat with VPLEX Customer Support and follow the steps displayed Article Properties Affected Product PLEX Last Published Date 16 Mar 2021 Version 3 Article Type Solution a6