How to connect to Memorystore redis CLI?

skatefriday · 05-11-2023 11:44 AM

I have a redis instance running in MemoryStore that I believe has some garbage data in it that is causing problems within an AppInstance application.

I would like to be able to connect to the underlying redis cli so that I can examine the data, however I'm unable to figure out how to do this with the existing documentation.

This page https://cloud.google.com/memorystore/docs/redis/connect-redis-instance says to create a VM instance that users the Redis instance's authorized network by following these instructions https://cloud.google.com/compute/docs/create-linux-vm-instance and yet those instructions do not mention the authorized network part of things. Which makes them incomplete in relation to the subject matter at hand.

I have gcloud installed, so I thought, maybe I'd try following these instructions https://cloud.google.com/memorystore/docs/redis/connect-redis-instance

However, gcloud tells me that it can't find the redis instance.

$ gcloud redis instances describe --region=us-central1 10.188.71.203
Python
ERROR: (gcloud.redis.instances.describe) NOT_FOUND: Resource 'projects/...elided.../locations/us-central1/instances/10.188.71.203' was not found
- '@type': type.googleapis.com/google.rpc.ResourceInfo
  resourceName: projects/...elided.../locations/us-central1/instances/10.188.71.203

And yet in the console that definitely exists. The authorized network for that instance is a VPC, which I find largely mystifying when trying to manage.

At any rate, I can't even SSH to my Compute Engine VM instance. My login is the Owner of the project, I think I have all the right permissions, and I have SSH ingress opened on the firewall rules, but

I did add an ingress firewall rule for the ip range specified, but ok, so try without Cloud IAP...

And ingress traffic for that ip range is open. The VPC firewall rules are as such...

All I want is a CLI onto that Redis instance. I'd think this would be a common thing and pretty basic, but I'm at a complete loss.

ms4446

It looks like the firewall you created is only allow SSH traffic to devices on ml-redis-vpc-dev network. If you are looking to SSH from you computer over the internet you need to create a rule that allows you source IP address.

For example:

gcloud compute firewall-rules create allow-ssh-ingress-from-iap \
--direction=INGRESS \
--action=allow \
--rules=tcp:22 \
--source-ranges=35.235.240.0/20

Or from

Cloud Console:

Open the Firewall Rules page in the Google Cloud Console.
Click "Create firewall rule."
Configure the following settings:
- Name: allow-ingress-from-iap
- Direction of traffic: Ingress
- Target: All instances in the network
- Source filter: IP ranges
- Source IP ranges: 35.235.240.0/20
- Protocols and ports: Select TCP and enter 22 to allow SSH.
Click "Create."

skatefriday

Isn't that what the first entry in the screenshot above is?

Unobscuring the columns...

Is what I have here different from what you are describing? The inability to connect as described above in the original post screenshots were taken with the ingress rule you suggest.

ms4446

Sorry for the confusion. Replace 35.235.240.0/20 with your actual source IP.

(I was just using this IP address as an example)

skatefriday

The documentation here is confusing.

https://cloud.google.com/iap/docs/using-tcp-forwarding

It states,

To allow IAP to connect to your VM instances, create a firewall rule that:

applies to all VM instances that you want to be accessible by using IAP.
allows ingress traffic from the IP range `35.235.240.0/20`. This range contains all IP addresses that IAP uses for TCP forwarding.
allows connections to all ports that you want to be accessible by using IAP TCP forwarding, for example, port `22` for SSH and port `3389` for RDP.

I read that as IAP specifically uses 35.235.240.0/20 as a set of IP addresses that it uses for TCP forwarding. As in this was something internal to IAP and you just use those settings. Not necessarily, "You must specify the IP address range you want IAP to forward."

At any rate, I added my WAN IP address on the local machine I am trying to connect to as w.x.y.z/32, and still no ability to connect. I added this rule first for the default network, and then when that didn't work, added the rule for the ml-redis-vpc-dev network.

The instance I am trying to connect to is in that VPC network if that makes any difference.

It has to be possible to obtain a shell on a VM instance, right? What am I doing wrong here?

ms4446

Here are some troubleshooting steps you can follow:

Check network tags:
- Ensure that the Compute Engine VM instance you are trying to connect to has the appropriate network tags associated with it.
- Verify that the firewall rule is targeting the instances with the correct network tags.
Verify network configuration:
- Check the VPC network configuration to ensure that it allows traffic between subnets or networks.
- Verify that the instance you are trying to connect to is associated with the correct network and subnet.
Check for other firewall rules:
- Confirm that there are no conflicting or more restrictive firewall rules that could be blocking the SSH traffic.
- Ensure that there are no other firewall rules that override the rule allowing access from your IP address.
Verify instance connectivity:
- Check if the instance is running and accessible within the network.
- Ensure that the instance's network interfaces are configured correctly.
- Verify that the instance's network configuration does not have any issues, such as incorrect IP address assignment or subnet routing.
Check for network-level restrictions:
- Check if there are any network-level restrictions or security policies in place that could be blocking the SSH traffic.
- Review any network-level firewalls or security groups that might affect the connectivity.
Check for OS-level firewall:
- Ensure that the firewall settings on the instance itself (OS-level firewall) are not blocking SSH connections.
- If applicable, check the firewall rules within the operating system running on the instance.

ms4446

As alternative to a dedicated VM, you can connect to a Redis instance from the Google Cloud Shell. To connect to a Redis instance from the Cloud Shell, you'll need to perform the following steps:

Open the Google Cloud Console by visiting the following URL: https://console.cloud.google.com/
Click on the "Activate Cloud Shell" button at the top right corner of the console. This will open the Cloud Shell in a new window at the bottom of the console.

Once you have the Cloud Shell open, you can use the redis-cli command to connect to your Redis instance. Here's an example of how you can connect to a Redis instance using the Cloud Shell:

Obtain the connection details for your Redis instance, including the hostname or IP address and the port number.
In the Cloud Shell, execute the following command, replacing HOSTNAME with the Redis instance's hostname or IP address, and PORT with the port number:

redis-cli -h HOSTNAME -p PORT

For example, if your Redis instance's hostname is my-redis-instance and the port is 6379, the command would look like:

redis-cli -h my-redis-instance -p 6379
Press Enter to execute the command. If the connection is successful, you'll see the Redis command-line interface, where you can interact with the Redis instance.

Make sure that your Redis instance is configured to allow connections from the IP range associated with the Cloud Shell. If you encounter any connection issues, check your Redis instance's firewall rules and network configuration.

Note that the Cloud Shell has a timeout period, and if your Redis commands require a long-running session, if you run into this issue than connecting from a VM will be only option.

skatefriday

From a Cloud Shell...

redis-cli -h 10.188.71.203 -p 6379
Could not connect to Redis at 10.188.71.203:6379: Connection timed out

This is what my Memorystore for Redis console looks like.

@ms4446 wrote:
Make sure that your Redis instance is configured to allow connections from the IP range associated with the Cloud Shell. If you encounter any connection issues, check your Redis instance's firewall rules and network configuration

I'm not sure I understand how to do this. I added the ip address of the Cloud Shell VM to the allow-ingress firewall rule. See below. I can't figure out what it means to configure the redis instance. I can not see, nor connect to the redis instance in any fashion and I don't know from that description above, which Cloud Console page allows you to configure what you reference.

ms4446

You need make sure that your Redis instance is configured to allow connections from the IP range associated with the Cloud Shell. If you encounter any connection issues, check your Redis instance's firewall rules and network configuration.

Redis uses Port 6379

skatefriday

I apologize profusely for my denseness in this matter, but this response confuses the heck out of me.

I have no idea how to connect to the VM that is running the redis instance. Which means I have no idea how make sure my Redis instance is configured to allow connections from the IP range associated with the Cloud Shell.

Frankly I don't even know what you mean by "IP range" unless you mean a CIDR description. Although both my local machine and the Cloud Shell are single IP addresses, not ranges. I assume you mean the IP assigned the to VM running the Cloud Shell as shown by ifconfig. Maybe you are referring to something else? Does that screenshot from the VPC console in my last message not show that it allows connections from 3 different IP address ranges? (Well the last two I realize are a range of 1). Is that not applying to the instance described in the Memorystore for Redis page?

But beyond that, as shown above, the only view I have onto the Redis instance is what is given by the Memorystore for Redis. Is this what you are referring to?

This?

I guess I'm just not following how the text descriptions of things I need to look at map to the Cloud Console pages.

ms4446

I'm sorry to hear about the issues you're encountering. So let's start from the beginning.

There are a few steps involved in connecting to a Memorystore instance, and it can be tricky if you haven't done it before. Let's walk through it step by step.

First, to connect to a Memorystore instance, you will need to create a Compute Engine VM instance in the same authorized network as your Memorystore instance. When you create the VM instance, make sure it's using the same network and subnet as your Memorystore instance.

Create a VM instance:
- Go to the VM instances page in the Google Cloud Console.
- Click "Create instance."
- Under Networking, click on the "default" network.
- In the dropdown, select the network that your Memorystore instance is using.
- Make sure to also select the correct subnet.

Once you've done that, you should be able to SSH into your VM instance.

SSH into the VM instance:
- From the VM instances page in the Google Cloud Console, click on the SSH button next to your VM instance. This should open a new browser window with an SSH session.

If you're still having trouble SSH-ing into the VM instance, it might be a firewall rule issue. Make sure you have a firewall rule that allows ingress traffic on TCP port 22 for the network that your VM instance is in.

Connect to the Memorystore instance:
- Once you're SSH'd into your VM instance, you can connect to the Memorystore instance using the Redis CLI. You should be able to do this with the following command, replacing [HOST_NAME] and [PORT] with your Memorystore instance's host name and port:

redis-cli -h [HOST_NAME] -p [PORT]

As for the gcloud issue, it seems like you're using the IP address of your Redis instance as the instance id in the gcloud command. The instance id is actually the name you gave your Redis instance when you created it. The correct command should be something like:

gcloud redis instances describe [INSTANCE_ID] --region=[REGION]

skatefriday

Thank you for being so patient, and I am sorry to be such trouble. Let's take a step back and focus just just on connecting to the Compute Engine VM.

This is what happens when I click the SSH button on the VM instance page.

I am logged in as Owner of the project, and do believe that I have the required permission.

My VM is in this this network...

Which is the network that Memorystore instance is using. And that's the subnet for the Memorystore instance also. Although while I can see the network being used on the Memorystore page, I can't find where it will display the subnet, but when setting up the VM, when selecting the ml-redis-vpc-dev network, the only option for subnet is, as shown, serverless-subnet.

My firewall rules show "allow-ingress-from-iap-to-ml-redis" allowing the ip address specified in the error message above, along with the WAN ip address of my local laptop I'm sitting in front of, and a 172 address, which to be honest, at this point in time, I don't remember where that came from.

I also have a similar setup for the default network. Taking shots in the dark...

I believe that I've satisfied all of the bullet points, and followed all of the steps you enumerated in your last message, and I think this screen shots show in detail what I've done. And yet, still I can't SSH to the Compute Engine VM.

Is any of the documentation of what I'm doing that I've provided here, incorrect, or missing a step?

Thank you for your continued patience.

ms4446

No worries. A few additional things to check:

User permissions: Your user account should have the iap.tunnelInstances.accessViaIAP permission. This permission allows your user account to establish a connection to the VM instance via the IAP.

You can verify and update the IAM permissions from the Google Cloud Console. Navigate to IAM & Admin and check the permissions assigned to your user account. If the required permission is not listed, you will need to add it.

You can check and update the firewall rules from the Google Cloud Console. Navigate to VPC Network -> Firewall and check if there's a rule allowing the necessary traffic. If not, you will need to create a new rule.
HTTPS connections to the IAP: Ensure that your environment can make proper HTTPS connections to the IAP. The hostname for the IAP is https://tunnel.cloudproxy.app.

One way to test this is by trying to access the IAP hostname using a web browser or a tool like curl from the command line. If you're not able to establish a connection, it could be due to network restrictions on your end.

skatefriday

Additionally, trying with IAP as it suggests might work in the error message above results in...

And yet the firewall rule this message is suggesting I need, already exists...

I am able to ssh to AWS instances from the local machine, so I don't think it's my local machine blocking ssh traffic.

skatefriday

@ms4446 wrote:
User permissions: Your user account should have the iap.tunnelInstances.accessViaIAP permission. This permission allows your user account to establish a connection to the VM instance via the IAP.

Verified. See screenshot.

@ms4446 wrote:
You can check and update the firewall rules from the Google Cloud Console. Navigate to VPC Network -> Firewall and check if there's a rule allowing the necessary traffic. If not, you will need to create a new rule.

Also verified. See screenshot in message posted here https://www.googlecloudcommunity.com/gc/Databases/How-to-connect-to-Memorystore-redis-CLI/m-p/553576...

curl returned 400. No idea what that means.

$ curl https://tunnel.cloudproxy.app/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1552  100  1552    0     0  11593      0 --:--:-- --:--:-- --:--:-- 11669<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (bad path)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>That’s an error.</ins>
  <p>Your client has issued a malformed or illegal request.  <ins>That’s all we know.</ins>

Still can't ssh to the VM.

ms4446

Ok, the curl command response (Error 400, bad path) suggests that the service is reachable (i.e., network connectivity is not an issue). So this is not the issue

Given that you've already confirmed the necessary User permissions and the firewall rule for Identity-Aware Proxy (IAP), the new error message suggests that there might be a broader issue with SSH access to the VM.

Here are a few additional things you could try:

Check the VM's SSH configuration: Ensure that the SSH daemon is running on the VM, and that its configuration allows connections on port 22. The configuration file is typically located at /etc/ssh/sshd_config. You will need to connect to VM's serial console to access its command line (since obviously you can't SSH from the Cloud Console).

You can connect to a VM's serial console using the Google Cloud Console by: (https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console)
1. Go to the Google Cloud Console in your web browser.
2. Select your project if it's not already selected.
3. Navigate to the VM instances page by clicking on the hamburger menu in the upper-left corner, then click on "Compute Engine" and then "VM instances".
4. Click on the name of the VM instance you want to connect to.
5. In the VM instance details page, scroll down to the "Remote access" section and click on "Connect to serial console".
Verify the VM's network settings: Make sure that the VM's network interface is correctly configured and that it has a valid IP address. In the Google Cloud Console, you can find this information in the VM instance's details page.
Check the VM's status: In the Google Cloud Console, check that the VM instance is running and doesn't have any reported issues.

skatefriday

curl https://tunnel.cloudproxy.app/ returns 400 for an acquaintance of mine as well. We are sitting about 300 miles apart on completely different networks.

skatefriday

Seems completely hosed.

May 15 22:09:46 tmp-instance google_guest_agent[546]: CRITICAL main.go:298 error registering service: failed to shutdown within timeout 15s
May 15 22:10:01 tmp-instance OSConfigAgent[781]: 2023-05-15T22:10:01.8181Z OSConfigAgent Critical main.go:112: Error parsing metadata, agent cannot start: network error when requesting metadata, make sure your instance has an active network and can reach the metadata server: Get http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json&wait_for_change=true&last_etag=0&...: dial tcp 169.254.169.254:80: connect: network is unreachable
May 15 22:10:01 tmp-instance systemd[1]: google-osconfig-agent.service: Main process exited, code=exited, status=1/FAILURE
May 15 22:10:01 tmp-instance systemd[1]: google-osconfig-agent.service: Failed with result 'exit-code'.
May 15 22:10:03 tmp-instance systemd[1]: google-osconfig-agent.service: Scheduled restart job, restart counter is at 2.
May 15 22:10:03 tmp-instance systemd[1]: Stopped Google OSConfig Agent.
May 15 22:10:03 tmp-instance systemd[1]: Started Google OSConfig Agent.

And I can't login, presumably because it can't find the metadata, presumably because the network is unreachable.

How is it that the GCP web infrastructure can get a user tied into knots like this?

ms4446

The error messages you provided are from the Google OSConfig Agent, which is a service that runs on VM instances and provides features like operating system patch management, package installation, and configuration management. This error suggests that the instance doesn't have network connectivity, as it's unable to reach Google's metadata server. Most likely the same reason why you are unable to SSH into the VM. Here are a few things you can check:

Instance network interface: In the Google Cloud Console, go to the VM instance's details page and check the network interface settings. Make sure that the network, subnet, and external IP settings are correct.
Instance startup script or system logs: Check the instance's startup script (if there is one) and system logs for any errors related to network configuration. You can access the system logs from the serial console, and the startup script from the instance's details page in the Google Cloud Console.

If these steps don't help, you might want to consider stopping the instance, taking a snapshot of its disk for troubleshooting purposes, and then creating a new instance with the same configuration to see if the issue persists. If the new instance works correctly, there might be a problem with the system state of the original instance.