Important!
This article is THE most-viewed on The Z-Issue, and is sometimes read tens of thousands of times per day. If it has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!Recently I ran into a problem with RHEL 6 (and any derivatives, like CentOS 6 or Scientific Linux 6) where having two NICs (network interfaces) in the same subnet resulted in strange behaviour. In RHEL ≤5 (or CentOS ≤5), one could have two interfaces with IPs in the same subnet and there weren’t any problems (besides the obvious question of why one would set it up this way instead of just bonding the interfaces). However, in RHEL 6 (or CentOS 6), having two interfaces with IPs in the same subnet results in the primary one pinging but the secondary one not responding.
The cause of this problem is that the rp_filter settings changed between these kernels (2.6.18 in RHEL 5 and 2.6.32 in RHEL 6). In RHEL 5, the rp_filter setting was a boolean where 1 meant that source validation was done by reversed path (as in RFC1812), and 0 meant no source validation. However, in RHEL 6, this setting changed to an integer with the following settings:
*****
0 – No source validation
1 – Strict Reverse Path validation (RFC3704) – Packets are checked against the FIB (Forwarding Information Base), and only the best ones succeed
2 – Loose Reverse Path validation (RFC3704) – Packets are checked against the FIB, but only non-reachable BY ANY INTERFACE will fail
*****
So, though the default setting is still 1, it now has a different meaning. In order to get these two network interfaces with IPs in the same subnet to both respond, I needed to make two changes in /etc/sysctl.conf:
- Change
net.ipv4.conf.default.rp_filterfrom ‘1’ to ‘2’ - Add the line
net.ipv4.conf.all.rp_filter = 2
To better illustrate the changes, here are the differences:
DEFAULT SETTINGS:
# grep '.rp_filter' /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 1
REQUIRED SETTINGS:
# grep '.rp_filter' /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
====================
Update – 19 April 2017: (Big thanks to Henry Butterworth for making me aware of this caveat)
In some distributions, changing the global settings for default and/or all is not sufficient. In those cases, you also need to specify the rp_filter setting for each ethernet interface individually. For example, if you have eth0 and eth1 to configure, you will need the following lines in your /etc/sysctl.conf:
net.ipv4.conf.eth0.rp_filter = 2
net.ipv4.conf.eth1.rp_filter = 2
====================
In order to make these changes effective immediately, you can reload the configuration with:
# sysctl -p
Ultimately, the new defaults make it so that the kernel discards packets when the route for outbound traffic differs from the route of incoming traffic. Changing these settings as mentioned above will make the kernel handle those packets like it did before 2.6.32. That way, having two or more interfaces with IPs in the same subnet will function as intended. Also, these changes aren’t limited to just RHEL 6 and derivatives, but also to any distribution with ≥kernel-2.6.32 in which the defaults were not changed.
Cheers,
Zach
83 comments
Skip to comment form
Thank you. Great post, good information. Saved me after 2 days of troubleshooting a new deployment on RHEL 7.
Author
Really glad that the post helped you after the installation. 🙂
Cheers,
Zach
You are my Hero with many thanks !!!
Author
Glad that the article helped you! 🙂
Cheers,
Zach
Hi Zach
I changed the rp_filer what you described. But now i can ping from 2 nics subnets suppose 172.20.20.0 network interface_ip 172.20.20.82 and 2ns network 172.20.24.0 interface _ip 172.20.24.82. But I could ping from 3rd network to 2nd interface(172.20.24.82) though I change the rp_filter for all, default, eth0,eth1.
Linux ver centos 6.9
Kindly advise
Thanks
Muhamamd
Author
Hello Muhammad,
I apologise, but I don’t understand what you’re trying to accomplish. Are you wanting to have three NICs within the same subnet?
Cheers,
Zach
Great post, we actually got hit by this issue few years back in a support ticket from a RH employee himself that configured 2 nics on same subnet, eventually we found the problem together. Now i see this article.
Using this issue on our technical support eng. candidates when recruiting, whoever finds this article passes 🙂
Author
Glad that the article helped you solve your problem! 🙂
Cheers,
Zach
Exactly, it’s my problem on HPE DL580 Gen10. But Oracle Linux 7 does’t work with your solutions. Pls help me!
Author
Hello Duc,
I don’t support the enterprise Linux distributions (and this post only covered RHEL because of CentOS). You might want to see the following page from Oracle:
https://support.oracle.com/knowledge/Oracle%20Linux%20and%20Virtualization/1252266_1.html
Hope that helps.
Cheers,
Zach
Nice article !! i have a different issue could you please help me with below scenario .
1) Hadoop Cluster was configured with below details .Everything was working fine .
IP address and Host name was picking up as per the requirement .3.3.3. is Cluster network
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
3.3.3.1 linux01.linux.com linux01
3.3.3.2 linux02.linux.com linux02
3.3.3.3 linux03.linux.com linux03
3.3.3.4 linux04.linux.com linux04
3.3.3.5 linux05.linux.com linux05
2) Application team configured the Application as below
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.5.110 linux01.linux.com linux01
192.168.5.111 linux02.linux.com linux02
192.168.5.112 linux03.linux.com linux03
192.168.5.113 linux04.linux.com linux04
192.168.5.114 linux05.linux.com linux05
3.3.3.1 linux01.linux.com linux01
3.3.3.2 linux02.linux.com linux02
3.3.3.3 linux03.linux.com linux03
3.3.3.4 linux04.linux.com linux04
3.3.3.5 linux05.linux.com linux05
3) When i restart Hadoop cluster instead of picking 3.3.3. series it is picking up 192.168.5 and same entries are foud in the cluster .but requirement is to use 3.3.3… series
Author
Hello Vijay,
This is a completely separate issue, not related to the one in this post. If you would like assistance with it, please email me and we can work out a price for my services.
Cheers,
Zach
Thank you so much, Zach. I was scratching my head the whole day to find out what is the issue.
And finally, your post was just a great relief. It worked 🙂
Author
You’re welcome, Saroj. I’m glad that the article helped you solve your problem.
Cheers,
Zach
Thank you very much Zach.
I was also stuck with the same issue. You made my day easier.
I wanted my RHEL 6 machine to be connected to both a USB Modem and the Local network. It was not working concurrently.
Now both are working concurrently fine after modifying /etc/sysctl.conf
Only I have to reload the conf.
Thanks.
Author
You’re welcome, VP. I’m glad that the article helped you.
Cheers,
Zach
Thank you so much Zach for this artichle. You helped me a lot to sort out this problem. I was trying to figure out this problem for last two days.
Author
Hello Kamran,
You’re welcome. I’m glad that my article helped you fix your problem!
Cheers,
Zach
Hi Zach!
Thanks a lot for posting this!!
Jorge
Author
You’re welcome, Jorge. I’m glad that you found it useful! 🙂
Cheers,
Zach
Thankyou you are a legend. I reinstalled a VM twice as the 2nd network adapter was added after the original install. Reinstall with both present and same problem again. Found your article and BOOM fixed.
Thanks again
Adam
Author
Hi Adam,
Glad that the article helped you! 🙂
Cheers,
Zach
I have following settings on my dual nic system with RHEL6:
eth0: 192.1.1.20 which is connected to another system at 192.1.1.105 via D-Link Switch
eth1: 192.1.1.25 which is connected to another system at 192.1.1.104 via Cisco Switch
when both the interfaces are up and working the problem is when i am trying to ping 192.1.1.104 from my system, it is not pinging. But when i put eth0 down, it starts pinging.
In my system rp_filter is already set to 0, i tried with rp_filter=2 also but, no success.
Please guide solution why this is happening.
Author
Hello Prateek,
Since it appears that all the hosts are in the same private subnet, you shouldn’t be running into a firewall problem. That being said, you will need to see if you’re pinging out of a particular interface (eth0 or eth1). It is likely that you have the default gateway set for one of the two interfaces. You may need static routing here, but I’m not sure without seeing it. This post explains the situation with pinging the source host with two interfaces within the same subnet (so the host with 192.1.1.20 and 192.1.1.25 in your situation). As such, the issue you’re seeing is different.
Cheers,
Zach
Hi Zach,
I had mapped 500GB Lun from storage to my rhel 6.5 host and created the partition using below steps:
[root@CTSSG-DC-DB2 ~]# pvcreate /dev/sdb
Physical volume “/dev/sdb” successfully created
[root@CTSSG-DC-DB2 ~]# vgcreate myvg /dev/sdb
Volume group “myvg” successfully created
[root@CTSSG-DC-DB2 ~]# lvcreate -L 200G -n mylv myvg
Logical volume “mylv” created
[root@CTSSG-DC-DB2 ~]# mkfs.ext4 /dev/myvg/mylv/
[root@CTSSG-DC-DB2 ~]# mkdir -pv /myfs
mkdir: created directory `/myfs’
[root@CTSSG-DC-DB2 ~]# mount /dev/myvg/mylv /myfs
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_albctssgdcdb2-lv_root 50G 2.1G 45G 5% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 485M 39M 421M 9% /boot
/dev/mapper/vg_albctssgdcdb2-lv_home 485G 6.9G 454G 2% /home
/dev/mapper/myvg-mylv 197G 188M 187G 1% /myfs
but after restart when I am trying to mount created filesystem /myfs I am getting below error
[root@CTSSG-DC-DB2 ~]# mount /dev/myvg/mylv /myfs
mount: you must specify the filesystem type
Please help with permanent solution..
Author
Hello again MS,
As I mentioned in my previous comment, your other Linux questions are outside the scope of this article. If you would like to hire me to help you with some of these tasks, please send me an email or reply to this comment, and I will keep it private & reply to your email.
Cheers,
Zach
Thanks Zach I will apply and update you.
I need some article to build 2 node failover cluster and Active/Active cluster please help me on that.
Rgds,
MS
Author
You’re welcome. HA clustering is really outside the scope of this article, and I don’t generally offer individualised support outside of this blog. If you would like to hire me to help you with a particular task, please send me an email or reply to this comment (I will keep it private, and reply to you via email).
Cheers,
Zach
Lines are not visible please repeat it again
Author
Hello again, MS,
I’m not sure what you mean by the lines are not visible. Which lines? If you’re talking about the code lines, you could highlight them to darken the background. Otherwise, here they are in plain text:
***
# grep ‘.rp_filter’ /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
***
and reloading with:
sysctl -p
Cheers,
Zach
Hi sir, in my centOS 7.1 it doesn’t see “net.ipv4.conf.default.rp_filter”and ” net.ipv4.conf.all.rp_filter ” in /etc/sysctl.conf , so what happen and how can I do next?
Author
Hello Meng,
This thread might be able to help:
https://community.spiceworks.com/topic/1025992-centos-7-0-upgraded-to-7-1-issue
Cheers,
Zach
Hi Zach,
I had 1 Linux 6.5 server with 4 NIC , NIC 1 and NIC 2 is connected to iSCSI switch and NIC 3 and NIC 4 is connected to Local LAN switch. All the NIC is configured on same subnet.
The problem is I cant reach to both the targets at same point in time either local gateway or iSCSI target is reachable at same point in time.
Even I had modified sysctl.conf file as recommended by you but it did not work for me.
Please help to get this fixed I am completely new to Linux.
Rgds,
MS
Author
Hello MS,
Have you tried setting no source validation?
# grep '.rp_filter' /etc/sysctl.confnet.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
and then issuing the reload with
sysctl -p? That may be your first step when using four NICs including the iSCSI connections. Also, and more importantly, you definitely should look into separating your iSCSI traffic from your Ethernet traffic. See this article for some reasons why.Cheers,
Zach
Dear zach
Thank you so much for your response 🙂 and your help i am very glad and happy with your response.
Author
You’re very welcome, Mohammad. I’m glad that your problem is fixed. I’m quoting another reply of yours so that people can see the link that you provided.
“Dear zach
i got a good and working article for my problem kindly share with other people if they have same problem
https://blogs.oracle.com/networking/entry/advance_routing_for_multi_homed “
Dear zach,
with your giving manual i can access both network (local and global) from my local lan but i can’t access both network from internet or another global IP kindly suggests me a best possible solution for my Scenario i am waiting for your reply.
Author
Hello Mohammad,
It sounds like the problem that you’re having is more involved. I have sent you an email, and we can communicate that way.
Cheers,
Zach
Dear sir,
According to your link static routing was not working kindly share me some reliable solution for centos 7 i am waiting for your reply
Dear sir
i explain you my Scenario i have two nic with different network one is global and one is local with your giving setting
both network are accessible with each and my problem was solved but when i restart my centos 7 version my network can ‘t accessible to each other kindly give me a permanent solution for this situation and guide me if centos 7 is supported with this command.
Author
Hello again Mohammad,
If you have one public and one private, then the issue presented here in this article isn’t applicable to you–it is for two NICs with IPs in the same subnet. For your situation, you will need to set up static routes for each of the interfaces (using the respective gateways). For more information about doing that within CentOS7, you should consult this article:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Using_the_Command_Line_Interface.html#sec-Static-Routes_and_the_Default_Gateway
Hope that helps.
Cheers,
Zach
Dear sir ,
After reboot my machine both nic are notworking properly only one network card is accessible what can i do for it.
Author
Hello Mohammad,
I would need more information about your configuration. If you are hesitant to post it here, that’s understandable. Basically, though, if both NICs are working properly, but only one NIC is accessible, it sounds like it could be either a routing table problem, or the issue that was specified in this article. If they were working before a reboot, then you need to follow the steps to make the configuration change persistent by putting them in
/etc/sysctl.conf.Hope that helps.
Cheers,
Zach
Hi Zach,
Thanks for the article, since I have a little different scenario I would really appreciate if you can assist.
I have a CentOS 7 with 2 NIC’s, one is configured as private LAN IP Address 192.168.0.51 with different gateway and the 2nd one is using a Public WAN IP 202.61.50.205 with different gateway.
Currently I can only access/ping to one IP address which ever is selected.
How can I enable both NIC’s so that both the internal and external users can access the system.
Thanks
AB
Author
Hello Arif,
Having two NICs (one with a public subnet and one with a private subnet) really shouldn’t be impacted by this problem. You would need to set the default gateway for the public subnet, and then set up static routes for the private interface. That’s really outside of the scope of this article, but if you need more assistance, I can try to help. An article that you might want to read is:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s1-networkscripts-static-routes.html
It’s specifically for RHEL7, but CentOS7 is a derivative thereof, so the same instructions should be applicable.
Cheers,
Zach
I want to thank the solution employed.
After migrate the Redhat version 5.2 to 6.6, we realize that windows stations did not open connections to the database server, starting from a different vlan.
There is even a documentation of redhat about it (https://access.redhat.com/solutions/53031), as well as an open RFC on this setting.
Thank you!
Otoniel Araújo
Author
Hello Otoniel,
Glad that the article helped you with the upgrade to RHEL 6.6. It was alarming to me that RedHat didn’t really do much in the way of notifying customers about the change. It was a rather large change, and caused network disruption, so I would have appreciated notification from them.
Anyway, glad that my article helped fix your problem!
Cheers,
Zach
Hello, I did these changes but i couldnt solve this problem, when i bring up eth1 one of them internet connection is losing 🙁 Help me, i can give my ssh
Author
It may be a different problem. You may have an incorrect routing table for the interface, or a chain in iptables that is preventing the secondary interface from being used. Without some additional information, there’s not much that I can recommend.
Cheers,
Zach
This article was very helpful. we struggled for a long to make multiple NIC cards work together and finally got our issue resolved. Thanks a lot.
Author
You’re very welcome; glad that it helped!
Cheers,
Zach
Thanks for posting this! Just replaced a RHEL 5 server with a RHEL 6 one and had this exact issue with multiple nics on the server. Only one would respond to pings if you were on anything beyond the local subnet.
Author
You’re very welcome, Mark. I’m glad that the article helped you with the problem.
Cheers,
Zach
Hi Zach..
I have 2 interfaces on the same subnet, and I can ping them both, BUT, if I run a tcpdump on the first interface, then ping the second interface from another machine, all the traffic is going through the first interface! Can you see if you’ve got the same issue?
Jason.
Author
Hi Jason,
I can’t seem to reproduce this problem. Are you sure that you’re running the tcpdump on only the first interface (e.g. using -i $INTERFACE) and not using specific host options that would cause all traffic to pass through the tcpdump filter?
Cheers,
Zach
Hi Zach,
Thank you for sharing this descriptions. I’ve been trying to resolve this problem for a week now. The tricky thing was that the problem appeared after restarting my virtual machine and disappeared over night without a trace in logs. The only clue were martian source entries on the host machine.
In my case I also needed to add :
net.ipv4.conf.all.accept_source_route=1
net.ipv4.conf.default.accept_source_route=1
and configure source routing for the two interfaces.
Now it works like it should. Thank you!
Cheers,
Kris
Author
Hi Kris,
I’m glad that my blog post helped you fix the problem. It was definitely a stumper for me for a while too.
Cheers,
Zach
Hello Zach,
you are really superb.. you saved me lot of time, Thank you so much man..
You R D man…Keep it up..
Author
Hi Mari,
Glad that I was able to save you some time. This is one of the most annoying problems I’ve found in RHEL and derivatives as of late.
Cheers,
Zach
Hi Zach,
Thank you for documenting this I have been pulling my hair out for days now!
Same subnet connections for iscsi only one would ping and as for iscsiadm discovery just hanging.
Quick well documented change (thank you!) and all now working as expected.
Cheers
Paul
Author
Hi Paul,
Very glad that the documentation helped you. It was truly unbelievable to me that RedHat didn’t send out notification about this change. I also spent FAR too long trying to get things working again.
Cheers,
Zach
Thanks for the post, you saved me hours of going mental about iscsi interfaces not working properly.
This set up is useful if you are to configure multiple interfaces on iscsi network.
Author
You’re very welcome. I’m glad that I was able to save you some time when troubleshooting this problem. I know that it caused me quite the headache! 🙂
Cheers,
Zach
Great article,
I was becoming crazy for this issue.
Thanks,
Giulio
Author
Glad that the article helped you solve the problem.
Cheers,
Zach
BINGO!!! Saved me a bunch of headaches. I started to suspect infrastructure routers/switches (it’s new). But also wondered if it was a RHEL change from 5 to 6.
Author
Glad that I could help. 🙂
Zach,
You saved my life. Thank you so very much for this article.
I was stuck and god only know what not I tried to get it working.
Thanks,
Krishna
Author
You’re very welcome, Krishna; glad to help!
Cheers,
Zach
Hi Zach,
Simple and apt instructions which worked.
Thanks,
Santhosh
Author
You’re very welcome, Santhosh. I’m glad that the instructions were helpful to you.
Cheers,
Zach
Thanks!! I’ve been pulling my hair out for days. I thought it was caused by GRE tunnels that I’ve created.
Author
You’re very welcome, Al; glad that I could help. I fought with it for far too long as well.
Cheers,
Zach
Thanks! Just had the same issue. You saved me alot of time and hassle.
Author
Very glad that I could help, Tom. It was definitely an annoying issue.
Cheers,
Zach
Hi Zach,
Thank you for posting this excellent article. You helped me resolve an issue I’d been stuck on for over a week.
Cheers,
Robert
Author
Hi Robert,
Very glad that it helped you fix your problem too! I appreciate you taking the time to comment indicating that the article helped you. 🙂
Cheers,
Zach
Great post, thanks for sharing!
It sounds like this could impact home users whose wired and wireless connections are in the same subnet and, for some reason, wish to use the wireless interface.
An unusual case might be a technician using his wireless access to troubleshoot an Ethernet switch in the same subnet, although this very case is an argument to segment the network into separate VLANs.
Author
Very good points, John. My biggest gripe is that they didn’t just switch the meanings of ‘1’ and ‘2’. If they had done so, the default would have still worked.
Cheers,
Zach
Hi, thanks for this, first time inlstlaing CentOS (always been Ubuntu up til now, wanted to try something else). Ran into this problem. installed 3.6 as above via network. 30Mb line. Not too slow at all
Author
Glad that the information was helpful for you!
Definitely got owned by this setting up iscsi. Great article.
Author
Hi Andy,
Really glad that you liked the article. I fought with this one for far too long, so I thought that it might spare someone else the pain. 🙂
Cheers,
Zach