Networking: Slow Inbound Network Speed on Google Cloud VMs

[Laha-dev]

Hi!

We're experiencing issues with inbound network speed that's only about one-tenth of the outgoing speed, and this happens across all types of virtual machines.

We've run iperf3 tests with different Linux distros and firewall appliances, and the results are the same for all of them. Below is the test result from a performance test using an external 500 Mbps fiber connection in the same GCE zone where the virtual machines are running:

Changing the number of vCPUs doesn't seem to matter, nor does the interface type or network tier (whether standard or premium). MTU settings also have no impact.

Have we run into an unknown/hidden soft cap that's throttling inbound traffic?

Any insights on where the bottleneck might be and how to troubleshoot network issues on Google Cloud are all welcome!

---

Test Configuration:

• Machine type: e2-highcpu-4
• Network tier: premium
• Interface type: gVNIC

Test resuls:

# Inbound (GCP VM <= External test server)

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 8.50 MBytes 8697 KBytes/sec 23 42.0 KBytes
[ 5] 1.00-2.00 sec 5.25 MBytes 5367 KBytes/sec 16 49.2 KBytes
[ 5] 2.00-3.00 sec 5.62 MBytes 5768 KBytes/sec 17 28.8 KBytes
[ 5] 3.00-4.00 sec 4.50 MBytes 4612 KBytes/sec 13 45.6 KBytes
[ 5] 4.00-5.00 sec 7.50 MBytes 7673 KBytes/sec 9 66.0 KBytes
[ 5] 5.00-6.00 sec 7.50 MBytes 7655 KBytes/sec 5 92.3 KBytes
[ 5] 6.00-7.00 sec 5.38 MBytes 5522 KBytes/sec 22 22.8 KBytes
[ 5] 7.00-8.00 sec 3.50 MBytes 3584 KBytes/sec 18 33.6 KBytes
[ 5] 8.00-9.00 sec 6.38 MBytes 6528 KBytes/sec 6 39.6 KBytes
[ 5] 9.00-10.00 sec 5.75 MBytes 5887 KBytes/sec 9 54.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 59.9 MBytes 6130 KBytes/sec 138 sender
[ 5] 0.00-10.04 sec 59.6 MBytes 6077 KBytes/sec receiver

# Outbound (GCP VM => External test server)

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 55.2 MBytes 56518 KBytes/sec
[ 5] 1.00-2.00 sec 55.2 MBytes 56575 KBytes/sec
[ 5] 2.00-3.00 sec 55.5 MBytes 56889 KBytes/sec
[ 5] 3.00-4.00 sec 55.1 MBytes 56412 KBytes/sec
[ 5] 4.00-5.00 sec 55.6 MBytes 56876 KBytes/sec
[ 5] 5.00-6.00 sec 55.6 MBytes 57067 KBytes/sec
[ 5] 6.00-7.00 sec 55.6 MBytes 56969 KBytes/sec
[ 5] 7.00-8.00 sec 55.4 MBytes 56673 KBytes/sec
[ 5] 8.00-9.00 sec 55.5 MBytes 56856 KBytes/sec
[ 5] 9.00-10.00 sec 55.5 MBytes 56811 KBytes/sec
- - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 558 MBytes 56886 KBytes/sec 596 sender
[ 5] 0.00-10.00 sec 554 MBytes 56765 KBytes/sec receiver

[diannemcm]

Hi @Laha-dev,

Welcome to Google Cloud Community!

The high retransmissions and fluctuating speeds suggest congestion. Some cloud providers limit inbound bandwidth to certain VM sizes. If your test server is outside GCP, its ISP might be throttling inbound connections. Try different test servers or use a GCP-to-GCP iPerf test. Run a traceroute (traceroute -T <server-ip>) to check for high-latency hops.

It appears that a more thorough investigation for your project is necessary. It is recommended to reach out directly to the Google Cloud Support team so they can provide in-depth assistance needed for this issue.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[Laha-dev]

Thanks for your feedback!

If I understand your response correctly, there is no soft cap on inbound traffic controlled by GCP, so any potential limitations might be due to external factors that we need to investigate further?

We'll perform some additional internal tests, and if we can't determine the cause, we'll reach out to Google Cloud Support.

Br!