GNU make CPU utilization in cloud environments

[Tamás Fülöp]

Hi everyone,

I am experiencing problems with the GNU make 3.81 in Amazon-like cloud environment. The "make.exe" does not utilizes the virtual CPUs, the utilization sits around 1% and that makes the building process really slow.
I have made the tests in the following environment:
OS: Windows 7 x64 Enterprise Edition SP1
System:Processor: QEMO Virtual CPU version (cpu64-rhel6) 2.80 GHzRAM: 8GB

Is there anyone, who experiences the same issue in cloud computing?

Thank you in advance!
Best regards,

Tamas Fulop

[Paul Smith]

On Wed, 2016-02-17 at 10:35 +0100, Tamás Fülöp wrote:
> I am experiencing problems with the GNU make 3.81 in Amazon-like cloud
> environment. The "make.exe" does not utilizes the virtual CPUs, the
> utilization sits around 1% and that makes the building process really
> slow.

You haven't given us enough information to respond. First, are you
using the "-jN" flag (where N is some value related to the number of CPU
cores you have)? Are you using any other flags with your make?

Second, is your makefile environment recursive (with make recipes
invoking sub-makes), or is it non-recursive (one make process loads all
the makefiles)? If it's recursive, are you sure you're using the
$(MAKE) variable to start your sub-makes?


In versions of GNU make prior to 4.0, the "job server" feature of GNU
make was not supported on Windows. This means that if your makefile is
very recursive you will get very little parallelism in those versions
regardless of how large your value is for "-j".

If you want more parallelism you'll have to either (a) upgrade to a
newer version of GNU make, or (b) rework your makefiles so they're non
-recursive.

[Tamás Fülöp]

Hello everyone,

I have played with the "-j" switch, but it did not affect the performance. I have tried with "-j 2", "-j" and without "-j" options, none of these improved the performance. I have also tried to set the priority level of the make.exe to "High" and "RealTime", but the CPU utilization still sits under 1%. Both 3.81 and 4.1.1 version of GNU make were tried and I am experiencing the same results with both versions.
The original bare make command which I have been using:

make -j -f Makefile
This command utilizes at least 80% of the CPU on a local machine, but not in the cloud environment.
Reworking the make files is not an option, because we have a tons of them. :(
Thank you for the quick response!

Best regards,

Tamas Fulop
> Subject: Re: GNU make CPU utilization in cloud environments
> From: ***@gnu.org
> To: ***@windowslive.com; make-***@gnu.org
> Date: Wed, 17 Feb 2016 08:25:55 -0500
>
> On Wed, 2016-02-17 at 10:35 +0100, Tamás Fülöp wrote:
> > I am experiencing problems with the GNU make 3.81 in Amazon-like cloud
> > environment. The "make.exe" does not utilizes the virtual CPUs, the
> > utilization sits around 1% and that makes the building process really
> > slow.
>
> You haven't given us enough information to respond. First, are you
> using the "-jN" flag (where N is some value related to the number of CPU
> cores you have)? Are you using any other flags with your make?
>
> Second, is your makefile environment recursive (with make recipes
> invoking sub-makes), or is it non-recursive (one make process loads all
> the makefiles)? If it's recursive, are you sure you're using the
> $(MAKE) variable to start your sub-makes?
>
>
> In versions of GNU make prior to 4.0, the "job server" feature of GNU
> make was not supported on Windows. This means that if your makefile is
> very recursive you will get very little parallelism in those versions
> regardless of how large your value is for "-j".
>
> If you want more parallelism you'll have to either (a) upgrade to a
> newer version of GNU make, or (b) rework your makefiles so they're non
> -recursive.

[Paul Smith]

On Wed, 2016-02-17 at 14:51 +0100, Tamás Fülöp wrote:
> This command utilizes at least 80% of the CPU on a local machine, but
> not in the cloud environment.

Wait, I think we were not clear what you meant by "cloud environment".

GNU make supports _parallel_ jobs: that is, running multiple recipes at
the same time on the same host.

GNU make does not support, at all, _distributed_ jobs: that is, running
recipes on multiple different hosts.

If by "cloud environment" you mean a set of separate servers running in
the cloud, there is no facility for distributing build jobs across
those in GNU make, and there has never been. If you want to improve
build times you'll need to provision a single host with more CPU
capacity.

There are add-on tools you can use if you would like distributed
builds, such as distcc: https://github.com/distcc/distcc However, as
far as I'm aware they only work on Linux not Windows.

Maybe someone else has pointers to distributed build capabilities that
will work for you.

[Tamás Fülöp]

It is just a virtual machine in the cloud, so I do not plan to create a distributed jobs, only parrallel ones on the same host. I would like to use only 1 job / 1 host architecture.

You are right about the CPU capacity, but the strage thing is the following. The CPU utilization is below 1% with the running make command, but file operations (like zipping files) occupies the whole CPU, so it could not be a cloud issue.

Thank you for the answer!


Best regards,

Tamas

> Subject: Re: GNU make CPU utilization in cloud environments
> From: ***@gnu.org
> To: ***@windowslive.com; make-***@gnu.org
> Date: Wed, 17 Feb 2016 10:12:10 -0500
>
> On Wed, 2016-02-17 at 14:51 +0100, Tamás Fülöp wrote:
> > This command utilizes at least 80% of the CPU on a local machine, but
> > not in the cloud environment.
>
> Wait, I think we were not clear what you meant by "cloud environment".
>
> GNU make supports _parallel_ jobs: that is, running multiple recipes at
> the same time on the same host.
>
> GNU make does not support, at all, _distributed_ jobs: that is, running
> recipes on multiple different hosts.
>
> If by "cloud environment" you mean a set of separate servers running in
> the cloud, there is no facility for distributing build jobs across
> those in GNU make, and there has never been. If you want to improve
> build times you'll need to provision a single host with more CPU
> capacity.
>
> There are add-on tools you can use if you would like distributed
> builds, such as distcc: https://github.com/distcc/distcc However, as
> far as I'm aware they only work on Linux not Windows.
>
> Maybe someone else has pointers to distributed build capabilities that
> will work for you.