«

»

Mar 21 2014

Linux – RHEL 6 / CentOS 6 two NICs in the same subnet, but secondary doesn’t ping

Recently I ran into a problem with RHEL 6 (and any derivatives, like CentOS 6 or Scientific Linux 6) where having two NICs (network interfaces) in the same subnet resulted in strange behaviour. In RHEL ≤5 (or CentOS ≤5), one could have two interfaces with IPs in the same subnet and there weren’t any problems (besides the obvious question of why one would set it up this way instead of just bonding the interfaces). However, in RHEL 6 (or CentOS 6), having two interfaces with IPs in the same subnet results in the primary one pinging but the secondary one not responding.

The cause of this problem is that the rp_filter settings changed between these kernels (2.6.18 in RHEL 5 and 2.6.32 in RHEL 6). In RHEL 5, the rp_filter setting was a boolean where 1 meant that source validation was done by reversed path (as in RFC1812), and 0 meant no source validation. However, in RHEL 6, this setting changed to an integer with the following settings:

*****
0 – No source validation

1 – Strict Reverse Path validation (RFC3704) – Packets are checked against the FIB (Forwarding Information Base), and only the best ones succeed

2 – Loose Reverse Path validation (RFC3704) – Packets are checked against the FIB, but only non-reachable BY ANY INTERFACE will fail
*****

So, though the default setting is still 1, it now has a different meaning. In order to get these two network interfaces with IPs in the same subnet to both respond, I needed to make two changes in /etc/sysctl.conf:

  • Change net.ipv4.conf.default.rp_filter from ‘1’ to ‘2’
  • Add the line net.ipv4.conf.all.rp_filter = 2

To better illustrate the changes, here are the differences:

DEFAULT SETTINGS:
# grep '.rp_filter' /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 1

REQUIRED SETTINGS:
# grep '.rp_filter' /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2

In order to make these changes effective immediately, you can reload the configuration with:

# sysctl -p

Ultimately, the new defaults make it so that the kernel discards packets when the route for outbound traffic differs from the route of incoming traffic. Changing these settings as mentioned above will make the kernel handle those packets like it did before 2.6.32. That way, having two or more interfaces with IPs in the same subnet will function as intended. Also, these changes aren’t limited to just RHEL 6 and derivatives, but also to any distribution with ≥kernel-2.6.32 in which the defaults were not changed.

Cheers,
Zach

20 comments

Skip to comment form

  1. Andy

    Definitely got owned by this setting up iscsi. Great article.

    1. Zach

      Hi Andy,

      Really glad that you liked the article. I fought with this one for far too long, so I thought that it might spare someone else the pain. :)

      Cheers,
      Zach

  2. John

    Great post, thanks for sharing!

    It sounds like this could impact home users whose wired and wireless connections are in the same subnet and, for some reason, wish to use the wireless interface.

    An unusual case might be a technician using his wireless access to troubleshoot an Ethernet switch in the same subnet, although this very case is an argument to segment the network into separate VLANs.

    1. Zach

      Very good points, John. My biggest gripe is that they didn’t just switch the meanings of ‘1’ and ‘2’. If they had done so, the default would have still worked.

      Cheers,
      Zach

    2. Pradeep

      Hi, thanks for this, first time inlstlaing CentOS (always been Ubuntu up til now, wanted to try something else). Ran into this problem. installed 3.6 as above via network. 30Mb line. Not too slow at all

      1. Zach

        Glad that the information was helpful for you!

  3. Robert Peebles

    Hi Zach,

    Thank you for posting this excellent article. You helped me resolve an issue I’d been stuck on for over a week.

    Cheers,
    Robert

    1. Zach

      Hi Robert,

      Very glad that it helped you fix your problem too! I appreciate you taking the time to comment indicating that the article helped you. :)

      Cheers,
      Zach

  4. Tom Szontagh

    Thanks! Just had the same issue. You saved me alot of time and hassle.

    1. Zach

      Very glad that I could help, Tom. It was definitely an annoying issue.

      Cheers,
      Zach

  5. Al

    Thanks!! I’ve been pulling my hair out for days. I thought it was caused by GRE tunnels that I’ve created.

    1. Zach

      You’re very welcome, Al; glad that I could help. I fought with it for far too long as well.

      Cheers,
      Zach

  6. Santhosh Kumar

    Hi Zach,
    Simple and apt instructions which worked.
    Thanks,
    Santhosh

    1. Zach

      You’re very welcome, Santhosh. I’m glad that the instructions were helpful to you.

      Cheers,
      Zach

  7. Krishna

    Zach,

    You saved my life. Thank you so very much for this article.
    I was stuck and god only know what not I tried to get it working.

    Thanks,
    Krishna

    1. Zach

      You’re very welcome, Krishna; glad to help!

      Cheers,
      Zach

  8. Urban Farm Dweller

    BINGO!!! Saved me a bunch of headaches. I started to suspect infrastructure routers/switches (it’s new). But also wondered if it was a RHEL change from 5 to 6.

    1. Zach

      Glad that I could help. :)

  9. Giulio Bernardini

    Great article,
    I was becoming crazy for this issue.

    Thanks,
    Giulio

    1. Zach

      Glad that the article helped you solve the problem.

      Cheers,
      Zach

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>