DNS error: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP

I saw a lot of errors like below in my /var/log/syslog:

Jun 10 11:06:39 zabbix systemd-resolved[30232]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jun 10 11:06:48 zabbix systemd-resolved[30232]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jun 10 11:06:51 zabbix systemd-resolved[30232]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jun 10 11:06:51 zabbix systemd-resolved[30232]: message repeated 4 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Jun 10 11:07:02 zabbix systemd-resolved[30232]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

I checked my /etc/resolv.conf and it symlinked to a file which was not exist.

ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Apr 27  2018 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

So I removed it and symlinked it to /run/systemd/resolve/resolv.conf which is systemd's resolv.conf file because I am using systemd's resolver.

So,

ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Then restart the relevant service: systemctl restart systemd-resolved

I checked again and the errors was gone.

That's all there is to it.

How to upgrade Ubuntu 19.04 to 19.10

Steps to upgrade from 19.04 to 19.10

1. sudo cp /etc/apt/sources.list.bak /etc/apt/sources.list
2. sudo sed -i -e 's|cosmic|eoan|g' /etc/apt/sources.list
3. sudo apt update
4. sudo apt upgrade


Happy upgrading!

How to upgrade ubuntu 18.10 to 19.04

Since 18.10 no longer supported you can upgrade manually to 19.04

Steps to upgrade:

1. mkdir /tmp/upgrade
2. cd /tmp/upgrade
3. wget http://old-releases.ubuntu.com/ubuntu/dists/disco-updates/

   main/dist-upgrader-all/current/disco.tar.gz
4. tar -xvf disco.tar.gz
5. sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
6. /etc/apt/sources.list change all links prefix to ->

    http://old-releases.ubuntu.com/
7. sudo python3 dist-upgrade.py or run do-release-upgrade

If you run do-release-upgrade, answer yes to all the questions asked.

Happy upgrading!

zfs and megacli

Recently I had a task to replace a disk in my server that was throwing many smart errors to my email. It was actually an indicator that the disk is failing. Maybe not as critical as it seemed because it is a mirrored disk.  But i didn't like it when it happened.

The error I got in my email as below:

This message was generated by the smartd daemon running on:

   host name:  hq-hv-06
   DNS domain: mydomain.com.my

The following warning/error was logged by the smartd daemon:

Device: /dev/bus/1 [megaraid_disk_03] [SAT], 1 Offline uncorrectable sectors

Device info:
WDC WD2002FAEX-007BA0, S/N:WD-WCAY00005116, WWN:5-0014ee-206439d3b, FW:05.01D05, 2.00 TB

For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
The original message about this issue was sent at Sun Jan 13 02:43:43 2019 +08
Another message will be sent in 24 hours if the problem persists.

To replace the disk, since I use mega RAID card, I have to use megacli utility for removal of the old disk and to replace with the new disk

These commands should be run before disk removal:

1.  Offline the disk: megacli -PDOffline -Physdrv[252:3] -a0
2. Mark the disk as missing: megacli -PDMarkMissing -Physdrv[252:3] -a0
3. Prepare for removal: megacli -PDPrpRmv -Physdrv[252:3] -a0

Then, you can remove the disk from the slot and insert the new disk. When you remove the disk, the zpool status shows DEGRADED state

Let the server detects the disk and check the info of the disk by running :

megacli -PDInfo -PhysDrv[252:3] -a0

The you run this command to create virtual disk:

megacli -cfgLDAdd -r0 [252:3] -a0

and you can also use megaclisas-status command :

megaclisas-status

The output is as below: 

But, If you get this error:

The current operation is not allowed because the controller has data in cache for offline or missing virtual drives

Run this command the find what disk cause the cache:


megacli -GetPreservedCacheList -a0

The output is:

Adapter #0

Virtual Drive(Target ID 03): Missing.

Exit Code: 0x00

then run this command to discard the cache:

megacli -DiscardPreservedCache -L"03" -a0

The output:

Adapter #0

Virtual Drive(Target ID 03): Preserved Cache Data Cleared.

Exit Code: 0x00

Then you can run the command above to create a virtual disk.

Then run megaclisas-status again to check whether the virtual disk is created

If it is successfully created, for zfs, run command to replace the disk:

zpool replace rpool /dev/sdd 

(if the disk is sdd) and if it is successful, it will start the resilvering process

 

Check the status: zpool status rpool

When the resilvering process finish, check zfs zpool status

That's all there is to it