What should I use for multithreaded compile server?

[janstrakowski]

Hi,

My PC has not that bad CPU but I’m currently working in some bigger C projects and don’t really wanna buy threadripper to compile the stuff on my own machine. I’m looking for a service that I can setup a compile pod. Send an archive with source code. Run some arbitrary build script. Download the binaries. And finally terminate that I’m not billed for example threadripper node 24/7 but only when I use it.

Maybe some docker, idk. But must have a lot of cores for a short period of time.

[ChieKo]

Hi @janstrakowski,

Welcome to Google Cloud Community!

Google Cloud offers several services that perfectly fit your needs for compiling large C projects without needing a powerful local machine. Here's how you could accomplish this using Google Kubernetes Engine (GKE) and Docker:

1. Setting up a GKE Cluster:

• Create a Project: If you don't already have one, create a Google Cloud project.
• Enable APIs: Enable the Kubernetes Engine API.
• Create a Cluster: Create a GKE cluster. Crucially, choose a machine type with a high core count. You can choose a preemptible machine type to significantly reduce costs (these machines can be reclaimed by Google at any time, but you'll only pay for the time you use them). The preemptible option is ideal for your "short period of use" requirement. Specify the number of nodes needed (start with one, you can scale up later).
• Enable Cloud Build (Optional but Recommended): Cloud Build offers a more managed way to build your code, especially if you want to leverage build caching and other features. However, the approach outlined below (using a simple Dockerfile and kubectl apply) is perfectly acceptable for simpler builds.

2. Creating Your Docker Image : Create a Dockerfile in the root directory of your C project. 

3. Deploying to GKE :

• Build the Docker Image: Build your Docker image using "docker build -t my-c-project ".
• Push to Container Registry: Push the image to Google Container Registry. You'll need to follow the instructions to authenticate with the registry. 
• Deploy to GKE (using kubectl): Create a Kubernetes deployment file (e.g., deployment.yaml): Apply the deployment using “kubectl apply -f deployment.yaml”. The sleep 300 command keeps the pod running for 5 minutes (300 seconds), allowing sufficient time for the build to complete. Adjust this value as needed.

4. Downloading the Binaries :

• Get the Pod's IP Address: Use kubectl get pods to find the IP address of your newly created pod.
• Copy Files: Use scp (or a similar tool like rsync) to securely copy the output.tar.gz file from the pod to your local machine. You'll need to set up SSH access to the pod, which might involve configuring a service account and setting up keys. Google Cloud documentation provides instructions on how to do this.

5. Terminating the Pod: After downloading the binaries, delete the pod using “kubectl delete pod <pod-name>”. This ensures you are only billed for the time the pod was running.

Important Cost Considerations:

• Preemptible VMs: Using preemptible VMs significantly reduces costs.
• Spot Instances: Google offers something similar to Spot Instances as well (although not explicitly called that) through their preemptible VM instances.
• Manage Resources: Monitor your resource usage carefully to control costs.

This detailed approach uses standard tools and avoids unnecessary complexity. Remember to replace placeholders like “<your-project-id>” with your actual values. This approach provides a scalable and cost-effective solution for compiling large C projects on Google Cloud. The entire process is easily repeatable for future builds. Remember to clean up your resources (clusters, images, etc.) when you're finished to avoid unnecessary charges.

I hope the above information is helpful.