Creation of successful GCP Instance template and groups (from existing bitnami Google Launchpad VM)

[v14]

Hello,
So I used the Bitnami Google Cloud Launchpad to create a VM in a project. The VM is launched successfully and I can ssh into it and everything. So I followed the procedure the create an instance template from an existing VM and the template was created successfully. Now when I try to create an instance group from this template, the group gets created but its never successful(the green check button never comes, and in the "errors section" there is an 'INVALID USAGE errortype and an 'Instance 'myproject-group-1-4w9l' creation failed: External IP address: 34.66.139.222 is already in-use' error message).

I need help resolving this issue.

How I created the template from an existing VM:
gcloud compute instance-templates create myproject-template --source-instance=bitnami-lampstack-6658 --source-instance-zone=us-central1-f

The goal is to autoscaled and loadbalance the app running on this VM

[glen_yu]

Your answer is in the error you're getting.

See warning message in documentation , but because the VM your instance is based off of has an external IP, that becomes part of your template and since you can't create a template that will will scale out with the same external IP, you get your error.  

When you create your instance template, make sure it only has private addresses as it will typically be fronted by a load balancer of some sort anyway.

Hope that helps.

[v14]

Thanks for the response. But how would i go about making sure that it only has private addresses? how can i modify the command for creating the instance template to make sure that it only has private addresses?

Looking forward to hearing from you. I am new to GCP

[WaiKeat]

You can click into the individual VM instance, then click edit around top-left, then scroll down to 'network interface' and click the pencil icon next to it, finally scroll down to 'external IP' there and in the dropdown list choose 'none'

[v14]

This works. But I now have a problem accessing my files using filezilla, Now which Ip address should I use to transfer files to an from my VM using filezilla?. Filezilla is not letting me access my VM using the IP address I generated for my load balancer even though I am using the same ppk key

[glen_yu]

Ah...ok....I see.  So you need to be able to transfer files over to those servers (i.e. HTML files) so it can be served correct?  

An HTTP(S) load balancer won't help you because they're specific in the ports they connect to and type of data it serves so you won't be able to ssh via LB. 

There are 2 very common use patterns I would recommend:

Option 1:

-------------

For this, you will actually need another server with a public IP commonly referred to as a bastion server .  In any case, because your BItnami servers have no public IP, you can't get to them via their private IP, hence you need a bastion server (typically on the same network or has access to the network you want to reach to).   You can then ssh to the bastion server and thru it, access your internal/Bitnami servers.  

For example:

- 3 Bitnami servers with private IPs: 10.0.0.3, 10.0.0.4, 10.0.0.5

- 1 bastion server with private IP: 10.0.0.6 and some public IP

- ssh to bastion via public IP (you should be able to transfer files via filezilla to the bastion via public as well)

- from bastion server, you can scp those files to your Bitnami servers, you can also ssh from bastion to Bitnami servers to unzip, move files, etc. as needed

Essentially your bastion server is going to be your doorway into your environment, which is why you should make sure you have it secure with proper firewall rules that ideally only allow your own public IP as the source IP to port 22.

Option 2:

------------------

I personally would recommend this more, but here you create a cloud storage bucket and transfer your files there.  Then you can ssh into your Bitnami servers via the the GCP console (Compute Engine --> VM Instances) and you can click the SSH button that should be available to you and you should be able to get a terminal window.  From there you can use the gsutil command (which should already be installed on the VM) to copy contents from your GCS bucket to the VM.

For example:

- gsutil cp gs://my-bucket/*.zip .

Hope that helps.

[v14]

Thanks for being thorough and prompt. 

So, bitnami VMs actually have a public IP when you create them. Please take a look at the photo attached. When I paste that IP in my browser, I see the default page (basically index,html page that’s on the VM). With the ppk file and this IP that bitnami provides here, I am able to use putty ssh client to connect to ssh into the server and access phpmyadmin, install ssl certificates using the command line etc. I am also able to transfer files (html,js,php) from my computer into my VM using filezilla. So this system doesn’t scale at this point. But the php database driven web application works and can easily be updated.

Since the goal now is to add auto scaling and load balancing before launching to the public, I started with creating an instance template from an existing VM that bitnami has created in my GCP project accessible through the console. This VM has the same external IP like shown in the photo. When I set public IP as none, as suggested on this post, this external IP disappears and I can’t use Putty or filezilla like before since they all require this IP address to grant access.

On the other hand, I successfully followed all the steps for auto scaling and load balancing and everything worked.  But only that I have no way to access my files, database etc, and I can’t change the default page shown when I visit the dns/ip address shown on my load balancer.

Would your suggestion still be the same or there is an easier way that can help me scale this system while leaving me with the same control I had before I decided to try auto scaling and load balancing?

Note: To create the VM in bitnami we use the Google Cloud Launch pad and this doesn’t allow you to specify whether the IP for your VM should be public or private, public is the default like you see in the photo

[glen_yu · Accepted solution]

Ok, so the Bitnami server that you create following the instructions that you did is suitable for a lone server setup/test environment.  A public IP is provided to the server and that's how you and others would access the server (whether it's SSH or HTTP).  You could have also created it without a public IP and created an LB that points to it.  This would be the more secure option, but if you're just serving content from a single server or using it as a test, one might argue that you don't need an LB (I mean, why pay for an LB if load just goes to a single server anyway, right?)

In practice -- and it's what you're experiencing now,  your backend will need to be able to scale and in order to do that more autonomously, you will need to use instance templates, but as you have seen, you can't create instance templates off of VMs that already have a public IP and it wouldn't make sense anyway as each VM's content would only be accessible by its public IP.  Hence the LB, so you have 1 public IP that will forward traffic to your backend whether you have 1 or 100.  As a result, you won't need (and shouldn't have) public IPs for each individual VM because now you just have extra avenues for attackers enter your environment with.  And of course removing the public IPs to the individual VMs poses other problems from a management/update point of view.  Security shouldn't be convenient but you should really still side with security.  The 2 options I previously mentioned are common methods of getting access to your env or updating its content without exposing yourself too much.

There's actually a 3rd option that I neglected to mention, but that is to update your Bitnami server with your webpage content BEFORE creating an instance template -- this way, the instance will already come with the content preloaded.  If you ever need to update the content, you can either use the 2 options I mentioned above -- or better yet, update your instance template and do a rolling update on the backend VMs, replacing each VM created from an old template with one creating from a new template.

If you're new to GCP or cloud computing in general, I recommend doing some Qwiklabs hands-on courses to build up your foundational knowledge.  For your current use case, I recommend Qwiklabs Network & HTTP LBs lab.  If you don't have an account, you can sign up for one for free and I think they give you some credits to start.  It's been a while since I created my account but if you don't get any free credits to start, feel free to DM me the e-mail you registered Qwiklabs with and I'll send you some of mine.

As for your comment about Bitnami not giving you options to select IPs, etc. I think that's just part of the template that they provide you and so options are limited.  These type of templates are good in some cases to get started, but in most cases it's almost always better to roll your own from scratch off of a blank VM.