«

»

Mar 21 2014

Linux – RHEL 6 / CentOS 6 two NICs in the same subnet, but secondary doesn’t ping

Recently I ran into a problem with RHEL 6 (and any derivatives, like CentOS 6 or Scientific Linux 6) where having two NICs (network interfaces) in the same subnet resulted in strange behaviour. In RHEL ≤5 (or CentOS ≤5), one could have two interfaces with IPs in the same subnet and there weren’t any problems (besides the obvious question of why one would set it up this way instead of just bonding the interfaces). However, in RHEL 6 (or CentOS 6), having two interfaces with IPs in the same subnet results in the primary one pinging but the secondary one not responding.

The cause of this problem is that the rp_filter settings changed between these kernels (2.6.18 in RHEL 5 and 2.6.32 in RHEL 6). In RHEL 5, the rp_filter setting was a boolean where 1 meant that source validation was done by reversed path (as in RFC1812), and 0 meant no source validation. However, in RHEL 6, this setting changed to an integer with the following settings:

*****
0 – No source validation

1 – Strict Reverse Path validation (RFC3704) – Packets are checked against the FIB (Forwarding Information Base), and only the best ones succeed

2 – Loose Reverse Path validation (RFC3704) – Packets are checked against the FIB, but only non-reachable BY ANY INTERFACE will fail
*****

So, though the default setting is still 1, it now has a different meaning. In order to get these two network interfaces with IPs in the same subnet to both respond, I needed to make two changes in /etc/sysctl.conf:

  • Change net.ipv4.conf.default.rp_filter from ‘1’ to ‘2’
  • Add the line net.ipv4.conf.all.rp_filter = 2

To better illustrate the changes, here are the differences:

DEFAULT SETTINGS:
# grep '.rp_filter' /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 1

REQUIRED SETTINGS:
# grep '.rp_filter' /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2

In order to make these changes effective immediately, you can reload the configuration with:

# sysctl -p

Ultimately, the new defaults make it so that the kernel discards packets when the route for outbound traffic differs from the route of incoming traffic. Changing these settings as mentioned above will make the kernel handle those packets like it did before 2.6.32. That way, having two or more interfaces with IPs in the same subnet will function as intended. Also, these changes aren’t limited to just RHEL 6 and derivatives, but also to any distribution with ≥kernel-2.6.32 in which the defaults were not changed.

Cheers,
Zach

61 comments

Skip to comment form

  1. Andy

    Definitely got owned by this setting up iscsi. Great article.

    1. Zach

      Hi Andy,

      Really glad that you liked the article. I fought with this one for far too long, so I thought that it might spare someone else the pain. 🙂

      Cheers,
      Zach

  2. John

    Great post, thanks for sharing!

    It sounds like this could impact home users whose wired and wireless connections are in the same subnet and, for some reason, wish to use the wireless interface.

    An unusual case might be a technician using his wireless access to troubleshoot an Ethernet switch in the same subnet, although this very case is an argument to segment the network into separate VLANs.

    1. Zach

      Very good points, John. My biggest gripe is that they didn’t just switch the meanings of ‘1’ and ‘2’. If they had done so, the default would have still worked.

      Cheers,
      Zach

    2. Pradeep

      Hi, thanks for this, first time inlstlaing CentOS (always been Ubuntu up til now, wanted to try something else). Ran into this problem. installed 3.6 as above via network. 30Mb line. Not too slow at all

      1. Zach

        Glad that the information was helpful for you!

  3. Robert Peebles

    Hi Zach,

    Thank you for posting this excellent article. You helped me resolve an issue I’d been stuck on for over a week.

    Cheers,
    Robert

    1. Zach

      Hi Robert,

      Very glad that it helped you fix your problem too! I appreciate you taking the time to comment indicating that the article helped you. 🙂

      Cheers,
      Zach

  4. Tom Szontagh

    Thanks! Just had the same issue. You saved me alot of time and hassle.

    1. Zach

      Very glad that I could help, Tom. It was definitely an annoying issue.

      Cheers,
      Zach

  5. Al

    Thanks!! I’ve been pulling my hair out for days. I thought it was caused by GRE tunnels that I’ve created.

    1. Zach

      You’re very welcome, Al; glad that I could help. I fought with it for far too long as well.

      Cheers,
      Zach

  6. Santhosh Kumar

    Hi Zach,
    Simple and apt instructions which worked.
    Thanks,
    Santhosh

    1. Zach

      You’re very welcome, Santhosh. I’m glad that the instructions were helpful to you.

      Cheers,
      Zach

  7. Krishna

    Zach,

    You saved my life. Thank you so very much for this article.
    I was stuck and god only know what not I tried to get it working.

    Thanks,
    Krishna

    1. Zach

      You’re very welcome, Krishna; glad to help!

      Cheers,
      Zach

  8. Urban Farm Dweller

    BINGO!!! Saved me a bunch of headaches. I started to suspect infrastructure routers/switches (it’s new). But also wondered if it was a RHEL change from 5 to 6.

    1. Zach

      Glad that I could help. 🙂

  9. Giulio Bernardini

    Great article,
    I was becoming crazy for this issue.

    Thanks,
    Giulio

    1. Zach

      Glad that the article helped you solve the problem.

      Cheers,
      Zach

  10. Redwan

    Thanks for the post, you saved me hours of going mental about iscsi interfaces not working properly.
    This set up is useful if you are to configure multiple interfaces on iscsi network.

    1. Zach

      You’re very welcome. I’m glad that I was able to save you some time when troubleshooting this problem. I know that it caused me quite the headache! 🙂

      Cheers,
      Zach

  11. Paul

    Hi Zach,

    Thank you for documenting this I have been pulling my hair out for days now!

    Same subnet connections for iscsi only one would ping and as for iscsiadm discovery just hanging.

    Quick well documented change (thank you!) and all now working as expected.

    Cheers

    Paul

    1. Zach

      Hi Paul,

      Very glad that the documentation helped you. It was truly unbelievable to me that RedHat didn’t send out notification about this change. I also spent FAR too long trying to get things working again.

      Cheers,
      Zach

  12. Mari

    Hello Zach,

    you are really superb.. you saved me lot of time, Thank you so much man..

    You R D man…Keep it up..

    1. Zach

      Hi Mari,

      Glad that I was able to save you some time. This is one of the most annoying problems I’ve found in RHEL and derivatives as of late.

      Cheers,
      Zach

  13. Kris

    Hi Zach,

    Thank you for sharing this descriptions. I’ve been trying to resolve this problem for a week now. The tricky thing was that the problem appeared after restarting my virtual machine and disappeared over night without a trace in logs. The only clue were martian source entries on the host machine.
    In my case I also needed to add :
    net.ipv4.conf.all.accept_source_route=1
    net.ipv4.conf.default.accept_source_route=1
    and configure source routing for the two interfaces.

    Now it works like it should. Thank you!

    Cheers,
    Kris

    1. Zach

      Hi Kris,

      I’m glad that my blog post helped you fix the problem. It was definitely a stumper for me for a while too.

      Cheers,
      Zach

  14. Jason

    Hi Zach..
    I have 2 interfaces on the same subnet, and I can ping them both, BUT, if I run a tcpdump on the first interface, then ping the second interface from another machine, all the traffic is going through the first interface! Can you see if you’ve got the same issue?

    Jason.

    1. Zach

      Hi Jason,

      I can’t seem to reproduce this problem. Are you sure that you’re running the tcpdump on only the first interface (e.g. using -i $INTERFACE) and not using specific host options that would cause all traffic to pass through the tcpdump filter?

      Cheers,
      Zach

  15. Mark

    Thanks for posting this! Just replaced a RHEL 5 server with a RHEL 6 one and had this exact issue with multiple nics on the server. Only one would respond to pings if you were on anything beyond the local subnet.

    1. Zach

      You’re very welcome, Mark. I’m glad that the article helped you with the problem.

      Cheers,
      Zach

  16. shashidhar

    This article was very helpful. we struggled for a long to make multiple NIC cards work together and finally got our issue resolved. Thanks a lot.

    1. Zach

      You’re very welcome; glad that it helped!

      Cheers,
      Zach

  17. Ulvi

    Hello, I did these changes but i couldnt solve this problem, when i bring up eth1 one of them internet connection is losing 🙁 Help me, i can give my ssh

    1. Zach

      It may be a different problem. You may have an incorrect routing table for the interface, or a chain in iptables that is preventing the secondary interface from being used. Without some additional information, there’s not much that I can recommend.

      Cheers,
      Zach

  18. Otoniel Araújo

    I want to thank the solution employed.

    After migrate the Redhat version 5.2 to 6.6, we realize that windows stations did not open connections to the database server, starting from a different vlan.

    There is even a documentation of redhat about it (https://access.redhat.com/solutions/53031), as well as an open RFC on this setting.

    Thank you!
    Otoniel Araújo

    1. Zach

      Hello Otoniel,

      Glad that the article helped you with the upgrade to RHEL 6.6. It was alarming to me that RedHat didn’t really do much in the way of notifying customers about the change. It was a rather large change, and caused network disruption, so I would have appreciated notification from them.

      Anyway, glad that my article helped fix your problem!

      Cheers,
      Zach

  19. Arif

    Hi Zach,

    Thanks for the article, since I have a little different scenario I would really appreciate if you can assist.
    I have a CentOS 7 with 2 NIC’s, one is configured as private LAN IP Address 192.168.0.51 with different gateway and the 2nd one is using a Public WAN IP 202.61.50.205 with different gateway.

    Currently I can only access/ping to one IP address which ever is selected.

    How can I enable both NIC’s so that both the internal and external users can access the system.

    Thanks
    AB

    1. Zach

      Hello Arif,

      Having two NICs (one with a public subnet and one with a private subnet) really shouldn’t be impacted by this problem. You would need to set the default gateway for the public subnet, and then set up static routes for the private interface. That’s really outside of the scope of this article, but if you need more assistance, I can try to help. An article that you might want to read is:

      https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s1-networkscripts-static-routes.html

      It’s specifically for RHEL7, but CentOS7 is a derivative thereof, so the same instructions should be applicable.

      Cheers,
      Zach

  20. Mohammad Rizwan Ismail

    Dear sir ,
    After reboot my machine both nic are notworking properly only one network card is accessible what can i do for it.

    1. Zach

      Hello Mohammad,

      I would need more information about your configuration. If you are hesitant to post it here, that’s understandable. Basically, though, if both NICs are working properly, but only one NIC is accessible, it sounds like it could be either a routing table problem, or the issue that was specified in this article. If they were working before a reboot, then you need to follow the steps to make the configuration change persistent by putting them in /etc/sysctl.conf.

      Hope that helps.

      Cheers,
      Zach

  21. Mohammad Rizwan Ismail

    Dear sir
    i explain you my Scenario i have two nic with different network one is global and one is local with your giving setting
    both network are accessible with each and my problem was solved but when i restart my centos 7 version my network can ‘t accessible to each other kindly give me a permanent solution for this situation and guide me if centos 7 is supported with this command.

    1. Zach

      Hello again Mohammad,

      If you have one public and one private, then the issue presented here in this article isn’t applicable to you–it is for two NICs with IPs in the same subnet. For your situation, you will need to set up static routes for each of the interfaces (using the respective gateways). For more information about doing that within CentOS7, you should consult this article:

      https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Using_the_Command_Line_Interface.html#sec-Static-Routes_and_the_Default_Gateway

      Hope that helps.

      Cheers,
      Zach

  22. Mohammad RIzwan

    Dear sir,
    According to your link static routing was not working kindly share me some reliable solution for centos 7 i am waiting for your reply

  23. Mohammad RIzwan

    Dear zach,
    with your giving manual i can access both network (local and global) from my local lan but i can’t access both network from internet or another global IP kindly suggests me a best possible solution for my Scenario i am waiting for your reply.

    1. Zach

      Hello Mohammad,

      It sounds like the problem that you’re having is more involved. I have sent you an email, and we can communicate that way.

      Cheers,
      Zach

  24. Mohammad RIzwan

    Dear zach
    Thank you so much for your response 🙂 and your help i am very glad and happy with your response.

    1. Zach

      You’re very welcome, Mohammad. I’m glad that your problem is fixed. I’m quoting another reply of yours so that people can see the link that you provided.

      “Dear zach
      i got a good and working article for my problem kindly share with other people if they have same problem
      https://blogs.oracle.com/networking/entry/advance_routing_for_multi_homed

  25. MS

    Hi Zach,
    I had 1 Linux 6.5 server with 4 NIC , NIC 1 and NIC 2 is connected to iSCSI switch and NIC 3 and NIC 4 is connected to Local LAN switch. All the NIC is configured on same subnet.

    The problem is I cant reach to both the targets at same point in time either local gateway or iSCSI target is reachable at same point in time.

    Even I had modified sysctl.conf file as recommended by you but it did not work for me.

    Please help to get this fixed I am completely new to Linux.

    Rgds,
    MS

    1. Zach

      Hello MS,

      Have you tried setting no source validation?

      # grep '.rp_filter' /etc/sysctl.conf
      net.ipv4.conf.default.rp_filter = 0
      net.ipv4.conf.all.rp_filter = 0

      and then issuing the reload with sysctl -p? That may be your first step when using four NICs including the iSCSI connections. Also, and more importantly, you definitely should look into separating your iSCSI traffic from your Ethernet traffic. See this article for some reasons why.

      Cheers,
      Zach

  26. MS

    Lines are not visible please repeat it again

    1. Zach

      Hello again, MS,

      I’m not sure what you mean by the lines are not visible. Which lines? If you’re talking about the code lines, you could highlight them to darken the background. Otherwise, here they are in plain text:

      ***
      # grep ‘.rp_filter’ /etc/sysctl.conf
      net.ipv4.conf.default.rp_filter = 0
      net.ipv4.conf.all.rp_filter = 0
      ***

      and reloading with:

      sysctl -p

      Cheers,
      Zach

  27. MS

    Thanks Zach I will apply and update you.

    I need some article to build 2 node failover cluster and Active/Active cluster please help me on that.

    Rgds,
    MS

    1. Zach

      You’re welcome. HA clustering is really outside the scope of this article, and I don’t generally offer individualised support outside of this blog. If you would like to hire me to help you with a particular task, please send me an email or reply to this comment (I will keep it private, and reply to you via email).

      Cheers,
      Zach

  28. MS

    Hi Zach,

    I had mapped 500GB Lun from storage to my rhel 6.5 host and created the partition using below steps:
    [root@CTSSG-DC-DB2 ~]# pvcreate /dev/sdb
    Physical volume “/dev/sdb” successfully created
    [root@CTSSG-DC-DB2 ~]# vgcreate myvg /dev/sdb
    Volume group “myvg” successfully created
    [root@CTSSG-DC-DB2 ~]# lvcreate -L 200G -n mylv myvg
    Logical volume “mylv” created
    [root@CTSSG-DC-DB2 ~]# mkfs.ext4 /dev/myvg/mylv/
    [root@CTSSG-DC-DB2 ~]# mkdir -pv /myfs
    mkdir: created directory `/myfs’
    [root@CTSSG-DC-DB2 ~]# mount /dev/myvg/mylv /myfs
    df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/vg_albctssgdcdb2-lv_root 50G 2.1G 45G 5% /
    tmpfs 16G 0 16G 0% /dev/shm
    /dev/sda1 485M 39M 421M 9% /boot
    /dev/mapper/vg_albctssgdcdb2-lv_home 485G 6.9G 454G 2% /home
    /dev/mapper/myvg-mylv 197G 188M 187G 1% /myfs

    but after restart when I am trying to mount created filesystem /myfs I am getting below error

    [root@CTSSG-DC-DB2 ~]# mount /dev/myvg/mylv /myfs
    mount: you must specify the filesystem type

    Please help with permanent solution..

    1. Zach

      Hello again MS,

      As I mentioned in my previous comment, your other Linux questions are outside the scope of this article. If you would like to hire me to help you with some of these tasks, please send me an email or reply to this comment, and I will keep it private & reply to your email.

      Cheers,
      Zach

  29. Prateek

    I have following settings on my dual nic system with RHEL6:
    eth0: 192.1.1.20 which is connected to another system at 192.1.1.105 via D-Link Switch
    eth1: 192.1.1.25 which is connected to another system at 192.1.1.104 via Cisco Switch
    when both the interfaces are up and working the problem is when i am trying to ping 192.1.1.104 from my system, it is not pinging. But when i put eth0 down, it starts pinging.
    In my system rp_filter is already set to 0, i tried with rp_filter=2 also but, no success.
    Please guide solution why this is happening.

    1. Zach

      Hello Prateek,

      Since it appears that all the hosts are in the same private subnet, you shouldn’t be running into a firewall problem. That being said, you will need to see if you’re pinging out of a particular interface (eth0 or eth1). It is likely that you have the default gateway set for one of the two interfaces. You may need static routing here, but I’m not sure without seeing it. This post explains the situation with pinging the source host with two interfaces within the same subnet (so the host with 192.1.1.20 and 192.1.1.25 in your situation). As such, the issue you’re seeing is different.

      Cheers,
      Zach

  30. Adam

    Thankyou you are a legend. I reinstalled a VM twice as the 2nd network adapter was added after the original install. Reinstall with both present and same problem again. Found your article and BOOM fixed.

    Thanks again

    Adam

    1. Zach

      Hi Adam,

      Glad that the article helped you! 🙂

      Cheers,
      Zach

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>


*