I have 2 Linux VMs and they refuse to connect [to each other] by port 80 (HTTP)

Richard E 0

Hello all, I'm new here and I have a problem.

I feel that I am missing something, but I continue to be blocked.

I am trying to make one of the vms into a internal mini-repo so the other vm can pull packages from.

I have two (2) VM in Azure and in a single resource group (both Alma9 Linux)

These VM are on the 10.0.0.0/16 VNET "LuTest"
I have the default subnet: default 10.0.0.0/24

LuServer: Virtual machine LUTEST Linux Standard_B2s 20.79.###.### 10.0.0.6

LuMGT: Virtual machine LUTEST Linux Standard_B2s 20.79.###.### 10.0.0.4

Two vNICs

luserver6_z1 Network interface 10.0.0.6 default

lustremgt313_z1 Network interface 10.0.0.4 default

[I can ping each other, I can ssh to each, and to each other. I can update from the internet]

{But I can't telnet 80 from each other]

[rdxxxx@LuMGT ~]$ telnet 10.0.0.6 80

Trying 10.0.0.6...

telnet: connect to address 10.0.0.6: No route to host

[Do I need to create a route for 2 VM on the same segment?]

I have a network secruity group.

LuServer-nsg is applied to the default subnet.

I added these rules to open port 80 as only 22 is open (generally).

Rule Title Port Protocol Source Dest.

330 AllowAnyHTTPInbound_any 80 TCP Any Any Allow

1050 AllowAnyCustom80Outbound 80 TCP Any Any Allow

I have not set up any firewalls in Azure, on on the vms themselves [iptables, ufw]

LuServer-nsg Effective rules.

Inbound rules

NamePrioritySourceSource PortsDestinationDestination PortsProtocol****AccessSSH3000.0.0.0/0,0.0.0.0/00-655350.0.0.0/0,0.0.0.0/022-22TCPAllowSSH3000.0.0.0/0,0.0.0.0/00-655350.0.0.0/0,0.0.0.0/022-22TCPAllowAllowAnyHTTPInbound_any3300.0.0.0/0,0.0.0.0/00-655350.0.0.0/0,0.0.0.0/080-80TCPAllowAllowVnetInBound65000Virtual network (2 prefixes)0-65535Virtual network (2 prefixes)0-65535AllAllowAllowAzureLoadBalancerInBound65001Azure load balancer (2 prefixes)0-655350.0.0.0/0,0.0.0.0/00-65535AllAllowDenyAllInBound655000.0.0.0/0,0.0.0.0/00-655350.0.0.0/0,0.0.0.0/00-65535AllDenyOutbound rules

NamePrioritySourceSource PortsDestinationDestination PortsProtocol****AccessAllowAnyHTTPOutbound3200.0.0.0/0,0.0.0.0/00-655350.0.0.0/0,0.0.0.0/080-80TCPAllowAllowAnyHTTPOutbound3200.0.0.0/0,0.0.0.0/00-655350.0.0.0/0,0.0.0.0/080-80TCPAllowAllowVnetOutBound65000Virtual network (2 prefixes)0-65535Virtual network (2 prefixes)0-65535AllAllowAllowInternetOutBound650010.0.0.0/0,0.0.0.0/00-65535Internet (282 prefixes)0-65535AllAllowDenyAllOutBound655000.0.0.0/0,0.0.0.0/00-655350.0.0.0/0,0.0.0.0/00-65535AllDenyI feel I am missing something, but I don't know what.

So what is blocking 10.0.0.4 to telnet 80 to 10.0.0.6?

hossein jalilian 4,040 Reputation points

2024-05-06T22:45:11.9866667+00:00
Thanks for posting your question in the Microsoft Q&A forum.

Verify that the Apache HTTP server or any other web server is running on server and listening on port 80:
sudo systemctl status httpd

Try temporarily disabling the NSG associated with the subnet or the network interfaces of the VMs. This will help you determine if the NSG rules are causing the problem.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful
Richard E 0 Reputation points

2024-05-07T21:16:21.5733333+00:00

I disabled the network security on the NICs. But that did not help.
I don't know why httpd is needed. (I am using nginx for my repo) I an trying to connect by "raw" http by telnet 10.0.0.6 80.
At this point I just want it to connect, nothing more.
When that happens I can then connect my dnf/yum and see if I need to enable or change nginx settings.
Richard E 0 Reputation points

2024-05-14T08:41:48.3466667+00:00

After a bit of time to find the IP verify control panel, (not intuitive to find). I have the links to the docs not controls.
Anyway:

My first test IP verify:
And the other:

I did ping both systems and themselves.

And

I will look more.

Also I installed tcpdump without issues with dnf.

Both system can reach the internet and Alma repositories.
Richard E 0 Reputation points

2024-05-14T08:57:25.1166667+00:00

< place holder, until a proper answer is found >
KapilAnanth-MSFT 36,316 Reputation points Microsoft Employee

2024-05-14T09:02:53.9666667+00:00
@Richard E ,

Thanks for sharing the results.

This indicates the platform is not blocking the traffic between these 2 VMs.

ICMP Tests are succeeding

You confirmed us that SSH works.

This indicates there should be something in the OS (third party firewall) or the application itself not listening/responding on port 80.

I believe tcpdump should give us more info on this.

Cheers,

Kapil
Richard E 0 Reputation points

2024-05-14T09:16:01.2066667+00:00

network NSG.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Richard E 0 Reputation points

2024-05-14T09:35:34.03+00:00

SSH to 10.0.0.6 from 10.0.0.4

SSH to 10.0.04 from 10.0.0.6

As the other VM is requesting the connection fingerprint, I believe that SSH is reaching.
Richard E 0 Reputation points

2024-05-14T11:46:15.42+00:00

Now with TcpDump

0 packets captured, 0 packets received by filter, 0 packets dropped by kernel

Other system:

Other direction:

and same thing:
KapilAnanth-MSFT 36,316 Reputation points Microsoft Employee

2024-05-15T05:41:07.18+00:00
@Richard E ,

I am afraid the tcpdump filters are incorrect.

I don't suggest using the filters "host" or "src port".

As source port would always be random, not 80 or 8080 - you will not see the original TCP SYN Packet or any packet that the VMwhichListensOn80 "receives".

You will only see the reply packets the VMwhichListensOn80 "sends"

Since you will never have source as 10.0.0.6 and sourcePort as 80, the tcpdump captured 0 packets.

Just run "tcpdump" and capture all the packets in both the VMs simultaneously.

While reading the captured packets, you can apply filters.

Only filter with the IP (either 10.0.0.6 or 10.0.0.4) so as you can see all the packets between these two, not only port 80 packets.

Also, you should

1.Ping

Telnet on Port 22

Telent on Port 80

So that we would know Ping and SSH works and only 80 is not working.

Cheers,

Kapil
Richard E 0 Reputation points

2024-05-15T11:33:58.13+00:00
Even more data: grab_lustreMgt.txt

I ran through the tests.

I test both systems with poing, ssh, and telnet.

Here are my results.

[rdeseke@LustreMGT 10.0.0.4 ~]$ sudo tcpdump -c 100 -x > grab_Lserver

dropped privs to tcpdump

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

100 packets captured

120 packets received by filter

0 packets dropped by kernel

[rdeseke@LustreMGT ~]$ less grab_Lserver

[rdeseke@LustreMGT ~]$ grep "ssh" grab_Lserver | wc -l

6

[rdeseke@LustreMGT ~]$ grep "ssh" grab_Lserver

09:42:44.766284 IP lustremgt.internal.cloudapp.net.ssh > ip5f5bd6c7.dynamic.kabel-deutschland.de.52600: Flags [P.], seq 3442478310:3442478370, ack 1688191849, win 501, options [nop,nop,TS val 3673515907 ecr 1546603568], length 60

09:42:44.766339 IP lustremgt.internal.cloudapp.net.ssh > ip5f5bd6c7.dynamic.kabel-deutschland.de.52600: Flags [P.], seq 60:192, ack 1, win 501, options [nop,nop,TS val 3673515907 ecr 1546603568], length 132

09:42:44.766408 IP lustremgt.internal.cloudapp.net.ssh > ip5f5bd6c7.dynamic.kabel-deutschland.de.52600: Flags [P.], seq 192:292, ack 1, win 501, options [nop,nop,TS val 3673515908 ecr 1546603568], length 100

09:42:44.784646 IP ip5f5bd6c7.dynamic.kabel-deutschland.de.52600 > lustremgt.internal.cloudapp.net.ssh: Flags [.], ack 60, win 501, options [nop,nop,TS val 1546603616 ecr 3673515907], length 0

09:42:44.784647 IP ip5f5bd6c7.dynamic.kabel-deutschland.de.52600 > lustremgt.internal.cloudapp.net.ssh: Flags [.], ack 192, win 501, options [nop,nop,TS val 1546603616 ecr 3673515907], length 0

09:42:44.789052 IP ip5f5bd6c7.dynamic.kabel-deutschland.de.52600 > lustremgt.internal.cloudapp.net.ssh: Flags [.], ack 292, win 501, options [nop,nop,TS val 1546603616 ecr 3673515908], length 0

[rdeseke@LustreMGT ~]$ grep "ICMP"grab_Lserver

< No Data>

ping is not working for 10.0.0.6 now:

[rdeseke@LusterServer ~]$ ping -n 5 10.0.0.4 PING 10.0.0.4 (10.0.0.4) 56(124) bytes of data.

^C

--- 10.0.0.4 ping statistics ---

40 packets transmitted, 0 received, 100% packet loss, time 39931ms

There is HTTP data but I don't think it is from Telnet

[rdeseke@LustreMGT ~]$ grep "HTTP" grab_Lserver 10:54:56.944122 IP lustremgt.internal.cloudapp.net.39286 > 169.254.169.254.http: Flags [P.], seq 1:122, ack 1, win 502, options [nop,nop,TS val 518731817 ecr 3604556394], length 121: HTTP: GET /metadata/instance?format=text&api-version=2017-04-02 HTTP/1.1 10:54:56.951810 IP 169.254.169.254.http > lustremgt.internal.cloudapp.net.39286: Flags [P.], seq 1:165, ack 122, win 49153, options [nop,nop,TS val 3604556402 ecr 518731817], length 164: HTTP: HTTP/1.1 200 OK 10:54:56.957917 IP lustremgt.internal.cloudapp.net.39286 > 169.254.169.254.http: Flags [P.], seq 122:262, ack 165, win 501, options [nop,nop,TS val 518731831 ecr 3604556402], length 140: HTTP: GET /metadata/instance/network/interface/?format=text&api-version=2017-04-02 HTTP/1.1

==== Other system ====

Noted above

[rdeseke@LusterServer ~]$ ping -n 5 10.0.0.4

PING 10.0.0.4 (10.0.0.4) 56(124) bytes of data.

^C

--- 10.0.0.4 ping statistics ---

40 packets transmitted, 0 received, 100% packet loss, time 39931ms

[rdeseke@LusterServer 10.0.0.6 ~]$ sudo tcpdump -c 400 -x > grab_lustreMgt

[rdeseke@LusterServer ~]$ grep "ssh" grab_lustreMgt

10:33:51.253841 IP lusterserver.internal.cloudapp.net.ssh > ip5f5bd6c7.dynamic.kabel-deutschland.de.52604: Flags [P.], seq 1017226651:1017226711, ack 2704431584, win 501, options [nop,nop,TS val 716344696 ecr 1791943164], length 60

10:33:51.253867 IP lusterserver.internal.cloudapp.net.ssh > ip5f5bd6c7.dynamic.kabel-deutschland.de.52604: Flags [P.], seq 60:248, ack 1, win 501, options [nop,nop,TS val 716344696 ecr 1791943164], length 188

...

10:34:28.179512 IP lusterserver.internal.cloudapp.net.ssh > lustremgt.internal.cloudapp.net.32950: Flags [F.], seq 1474, ack 1463, win 501, options [nop,nop,TS val 664481130 ecr 2266264333], length 0

10:34:28.179948 IP lustremgt.internal.cloudapp.net.32950 > lusterserver.internal.cloudapp.net.ssh: Flags [.], ack 1475, win 501, options [nop,nop,TS val 2266264335 ecr 664481130], length 0

[rdeseke@LusterServer 10.0.0.6 ~]$ grep "ICMP" grab_lustreMgt

10:34:00.345757 IP lustremgt.internal.cloudapp.net > lusterserver.internal.cloudapp.net: ICMP echo request, id 1, seq 1, length 64

10:34:00.345824 IP lusterserver.internal.cloudapp.net > lustremgt.internal.cloudapp.net: ICMP echo reply, id 1, seq 1, length 64

10:34:01.347015 IP lustremgt.internal.cloudapp.net > lusterserver.internal.cloudapp.net: ICMP echo request, id 1,

...

10:34:06.388765 IP lustremgt.internal.cloudapp.net > lusterserver.internal.cloudapp.net: ICMP echo request, id 1, seq 7, length 64

10:34:06.388821 IP lusterserver.internal.cloudapp.net > lustremgt.internal.cloudapp.net: ICMP echo reply, id 1, seq 7, length 64

10:34:48.490563 IP lusterserver.internal.cloudapp.net > lustremgt.internal.cloudapp.net: ICMP host lusterserver.internal.cloudapp.net unreachable - admin prohibited filter, length 68

Total count 15

HTTP data full listing

[rdeseke@LusterServer ~]$ grep "HTTP" grab_lustreMgt 10:33:54.983228 IP lusterserver.internal.cloudapp.net.37750 > 168.63.129.16.http: Flags [P.], seq 1:201, ack 1, win 502, options [nop,nop,TS val 1982430815 ecr 3603294442], length 200: HTTP: GET /machine/?comp=goalstate HTTP/1.1 10:33:54.985289 IP 168.63.129.16.http > lusterserver.internal.cloudapp.net.37750: Flags [FP.], seq 1:2278, ack 201, win 49153, options [nop,nop,TS val 3603294446 ecr 1982430815], length 2277: HTTP: HTTP/1.1 200 OK 10:33:54.992149 IP lusterserver.internal.cloudapp.net.37762 > 168.63.129.16.http: Flags [P.], seq 1:188, ack 1, win 502, options [nop,nop,TS val 1982430824 ecr 3603294452], length 187: HTTP: POST /HealthService HTTP/1.1 10:33:54.992175 IP lusterserver.internal.cloudapp.net.37762 > 168.63.129.16.http: Flags [P.], seq 188:380, ack 1, win 502,
Richard E 0 Reputation points

2024-05-15T11:55:29.1533333+00:00

Other system grab.
KapilAnanth-MSFT 36,316 Reputation points Microsoft Employee

2024-05-15T13:02:16.7633333+00:00

@Richard E ,

To troubleshoot further, I believe we will need a specialized 1:1 session, where a support engineer can have a screen share session to pinpoint the issue.

If you have a support plan you may file a support ticket, please check the private messages, we will try and help you get a one-time free technical support.

Cheers,

Kapil

3 answers

KapilAnanth-MSFT 36,316 Reputation points Microsoft Employee

2024-05-08T11:19:18.95+00:00
@Richard E ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Looking at your scenario, this clearly indicates there is an OS Level filtering firewall blocking the traffic.

I see SSH from one VM to the other and vice versa succeeds

I also see ICMP Ping from one VM to the other and vice versa succeeds

This means, routing should be fine between the two VMs

Wrt, "[Do I need to create a route for 2 VM on the same segment?]"

No

As long as the VMs are in the same VNET, we do not have to add any manual routes.

As next steps,

For Platform troubleshooting, you can consider using

IP flow verify or

NSG diagnostics

Make sure you run the test with the VM which listens at 80, and Direction as Inbound.

Local Port becomes 80

Remote port becomes 1234 and the remote IP becomes 10.0.0.4

Similarly, run the same test but this time use the VM which is expected to initiate the connection and Direction as Outbound

Local Port becomes 1234

Remote port becomes 80 and the remote IP becomes 10.0.0.6

Please share a screenshot of the same.

In case the above succeeds,

This means the issue is at the OS level

As next steps, you have to collect tcpdump at both the servers simultaneously.

Make sure you start the tcpdump in both the servers,

First ping the 10.0.0.6 from 10.0.0.4

Second telnet the 10.0.0.6 from 10.0.0.4 for port 22 (SSH)

Third telnet the 10.0.0.6 from 10.0.0.4 for port 80 (HTTP)

Then stop the tcpdump

See if the first two (ICMP and SSH) packets are captured

If you see packets destined to 80 at 10.0.0.6 but the 10.0.0.6 is not responding, it means there is an OS/Application issue

If you don't see any packets at all (for port 80) in 10.0.0.6, do let us know.

Cheers,

Kapil
Please sign in to rate this answer.
Richard E 0 Reputation points

2024-05-08T19:58:37.8733333+00:00

I checked the vms internal firewall and nothing

And ufw in not installed.

I'm a bit confused, your solution is to route the connection through a different port?
So: (using port 8080 as an example)
vm01:80 --> 8080 | 8080 --> vm02:80

And:
vm02:80 --> 8080 | 8080 --> vm01:80

Should I define the service as "HTTP" for the extra rules?

KapilAnanth-MSFT 36,316 Reputation points Microsoft Employee

2024-05-09T05:10:24.9566667+00:00

@Richard E ,

Thanks for stating that there is no firewall.

wrt, "your solution is to route the connection through a different port?"

No

I am not sure which VM is the source and which VM is the destination, so let's call it VMSource, VMDestination and VMDestination is the one that listens on port 80 (which does not work) but SSH and ICMP works.

My suggestion was to check connection from VMSource:RandomPort and VMDestination:Port80 in Azure platform using IP flow verify or NSG diagnostics.

To do this, you have to run IP Flow verify on both VMSource and VMDestination separately.

One for Outbound (from Source)

One for Inbound (from Destination)

So, with VMSource selected,

Run this and share us the results.

With VMDestination selected,

Run this and share the results with us

If both the results state that traffic is allowed in both the scenarios, you have to go for tcpdump in both the VMs simultaneously.

Cheers,

Kapil

Richard E 0 Reputation points

2024-05-15T13:20:12.79+00:00

I found the/a issue. There is a 3d firewall that I did not know about that seems to have been the problem. Alma9 uses firewalld as part of its' install.

[rdeseke@LusterServer ~]$ sudo systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; preset: enabled) Active: active (running) since Wed 2024-05-15 08:25:42 UTC; 4h 24min ago Docs: man:firewalld(1) Main PID: 723 (firewalld) Tasks: 2 (limit: 22608) Memory: 44.1M CPU: 733ms CGroup: /system.slice/firewalld.service └─723 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

While there is nothing setup

[rdeseke@LusterServer ~]$ sudo firewall-cmd --list-all

public (active)

target: default

icmp-block-inversion: no

interfaces: eth0

sources:

services: cockpit dhcpv6-client ssh

ports:

protocols:

forward: yes

masquerade: no

forward-ports:

source-ports:

icmp-blocks:

rich rules:

[rdeseke@LusterServer ~]$ sudo firewall-cmd --get-zones

block dmz drop external home internal nm-shared public trusted work

I stopped it and....

[rdeseke@LusterServer ~]$ sudo systemctl stop firewalld

[rdeseke@LusterServer ~]$ telnet 10.0.0.4 80

Trying 10.0.0.4...

Connected to 10.0.0.4.

Escape character is '^]'.

^CConnection closed by foreign host.

From:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 60:45:bd:18:c4:80 brd ff:ff:ff:ff:ff:ff

inet 10.0.0.6/24 brd 10.0.0.255 scope global noprefixroute eth0

I still have an issue with 10.0.0.4 / LustreMGT

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 00:22:48:e6:1e:c7 brd ff:ff:ff:ff:ff:ff

inet 10.0.0.4/24 brd 10.0.0.255 scope global noprefixroute eth0

valid_lft forever preferred_lft forever

inet6 fe80::222:48ff:fee6:1ec7/64 scope link

valid_lft forever preferred_lft forever

Stopped

[rdeseke@LustreMGT ~]$ sudo systemctl stop firewalld

[rdeseke@LustreMGT ~]$ telnet 10.0.0.6 80

Trying 10.0.0.6...

telnet: connect to address 10.0.0.6: Connection refused

This, I think is a NSG issue more than the firewall issue (now).
So the initial issue was a firewall. Just one I did not know about.

KapilAnanth-MSFT 36,316 Reputation points Microsoft Employee

2024-05-15T13:47:30.39+00:00

@Richard E ,

Wonderful, glad we were able to see good progress here.

Wrt communication to 10.0.0.6, make sure there is a service listening on Port 80 in 10.0.0.6

If you doubt this is because of NSG, you can leverage the IP flow verify or NSG diagnostics as mentioned earlier.

Just reverse the direction of the tests.

Thanks,

Kapil

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.
Sign in to comment
Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more
Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Share via

I have 2 Linux VMs and they refuse to connect [to each other] by port 80 (HTTP)

3 answers