MAKEPP_SPEEDUP(1) Makepp MAKEPP_SPEEDUP(1)NAMEmakepp_speedup-- How to make makepp faster
DESCRIPTION
So you think makepp is slow? It has gotten noticeably faster, but
granted, it's still slow, especially if you come from GNU make. This
is because it conscientiously checks all those things where gmake gives
you a headache, ignoring lots of dependencies (the "I think I need to
gmake clean to get rid of a mysterious bug" syndrome). If you suspect
some Perl code you added to your makefiles might be at fault, take a
look at perl_performance.
But there are a few things you can do to squeeze out more speed. Some
of the things are labeled unsafe, in the sence that you're asking
makepp not to check or do certain things, which you think are not
needed. If these things would have been necessary, the build may not
be correct. Luckily this problem will be temporary, however. It will
get corrected as soon as you let makepp do all checks.
You can combine several of these tips to increase the time gain even
more.
Safe Methods
Use makeppreplay
The stand-alone utility makeppreplay, mppr repeats things that makepp
has already done.
Use a Faster Perl
Within version 5.8, all are roughly the same, only 5.8.7 is a bit
faster. Within version 5.6, each newer subversion is noticeably
faster. And, at least for makepp, 5.6.2 is 10% faster than 5.8.7. If
you don't need any newer Perl features in your Makeppfiles, I suggest
getting a 5.6.2 with Digest::MD5 added.
Tuning your Perl can also help, like not compiling it for 64 bits,
which makepp doesn't need. For example ActiveState's build
(<http://www.activestate.com/activeperl>) of 5.8.7 for Linux is faster
than the Perl 5.8.7 that comes with SuSE Linux 10.0.
Include as Little as Possible
Each additional file you include is doubly penalizing. On the one
hand, the compiler must look for and at all those files. You don't
notice this so much, because it's just a little extra per compiler
call. On the other hand makepp must look too, to find dependencies and
figure out whether they incur a rebuild. Then it can seem to stall,
while it is digesting a lot of dependencies at once.
An absolutely deadly variant is the project master include file, which
in turn conveniently includes anything you might need. The result is
that any header file change leads to a full build. Even without a
change, makepp must think about all those headers again, for every
source you compile. Just a tiny effort, since this is cached, but
thousands of files can make this staggering.
It may be cumbersome to figure out the minimal set of includes, and to
cleanup those no longer needed, but it really pays off. If anybody
knows a tool that can identify which files get included unnecessarily,
I'd be glad to mention it here!
Build as Little as You Need
If you have a default target which makes several programs, then makepp
will have to check all their dependencies, right down to the smallest
header file. But maybe you want to test your change with only one of
those programs.
Then you would call makepp with an explicit target. The less modules
or headers all those programs have in common, the greater the benefit
of not letting makepp check them all.
Say your top level Makeppfile has this rule:
$(phony all): proggie1 proggie2 $(only_phony_targets */**/all)
Then you would call things like
$ makepp proggie2
$ makepp proggie1 dir/subdir/proggie27
Use preferred makefile names
Makepp looks for makefiles (unless you specify them explicitly on the
command line or with "load-makefile") in the order RootMakeppfile,
RootMakeppfile.mk, Makeppfile and Makeppfile.mk, followed by the
classical makefile names. (The .mk variants are for purely suffix-
based systems.)
So, if you use RootMakeppfile at the root of your build tree, and
Makeppfile everywhere else, the files will be found slightly faster.
Makepp will also have a slightly smaller memory consumption (caching
the fact that the other names don't exist), which also means speed
through less memory management.
Likewise if you have a statement
include standard
there will first be an attempt to find standard.makepp, so you might as
well use that name.
Have as few rules as you need
Makepp keeps track not only of existent files, but also of any it
learns to create. (That's why it offers reliable wildcards like *.o.)
The price for this power is a lot of management. So, if you tell it
how to create a .o from a .c, that's fine, because it will happen for
most if not all candidates.
But if you tell it how to link any suffixless executable from a like
named .o, that's expensive, because it will probably only happen for a
small part of them (those that contain a main function), but the basis
will get laid for all. You have to weigh the comfort of a linker
pattern rule, against the efficiency of individual linker rules.
If you don't use any of them, you should also turn off the builtin
rules with:
makepp_no_builtin = 1
If you do use them, but, for the reasons explained above, not the
builtin linker rules, you should turn those off with:
makepp_no_builtin_linker = 1
Put makepp extensions into a module
Makepp offers very convenient possibilities of being extended through
Perl. But if you write some functions, commands or statements in a
file and include that from dozens of makefiles, you will get dozens of
copies of them all in memory. And they will be read dozens of times by
the makepp parser, which is a bit slower than Perl's.
In this situation it is better to put your own functions into a module.
Use Repositories and/or a Build Cache
If you have several developers working on the same machine or if you
change to and fro between sets of build options, this is for you.
Repositories allow you to offer a central reference where you only need
to build what is locally different. A build cache simply collects all
produced files, and reuses them as appropriate, with less planning
needed. The latter page also describes the differences.
Use Sandboxes
If your build is so big that makepp is having a hard time digesting all
the information and if you can find a way of splitting it up into
smaller independent parts, sandboxes might give you better parallelism
than the "--jobs" option.
Don't log what you do
Makepp's logging feature is very powerful for tracking down bugs in the
build system, or for analyzing your dependencies. Whenever you don't
do these things, you can save quite a bit of formatting and I/O with
"--no-log --no-scan-log".
Almost Safe Methods
Get a Headstart
The option "--stop-after-loading" (or just "--stop") allows makepp to
start its work while you are still editing. It will suspend itself
when it gets to the point analyzing the dependencies. You decide when
you're ready to let it go on. On our huge project this saves half a
minute, and that's only when we have a CPU to ourselves.
This method has two potential drawbacks:
· Makeppfiles have been read by the time makepp stops. If you edit a
Makeppfile or something from which it would have to be rebuilt,
after starting makepp, this will go unnoticed till the next time.
But this should rarely be necessary, since makepp greatly reduces
the need for Makeppfile changes.
· If a target depends on a wildcard, and that would match more than
when the Makeppfile was read, makepp will not notice:
proggie: *.o
$(LD) $(inputs) -o $(output)
If you add another source file, or a file from which makepp knows
how to generate a source, then "*.o" should match the object that
produces. But, if this file was added after starting makepp, it
will not, because the wildcard was expanded too early.
In both of these cases you should kill the prestarted makepp and start
it anew.
You can do something like the following in your Shell's $ENV file or
.profile to save typing (csh users replace '=' with ' '):
alias mpps='makepp --stop'
Gulliver's Travels
The option "--gullible" tells makepp to believe that a rule changes
what it says it will, neither less nor more. Not performing these
checks can save a few percent of makepp's CPU time. And the Disk I/O
savings is especially welcome on network file systems. If you do
nightly full builds in an empty directory with the "--repository"
option, but without the "--gullible" option, you can be fairly sure
that your rule set is consistent. Then this option shouldn't hurt in
your daytime work.
Potentially Unsafe Methods
These methods are unsafe if you give makepp the wrong hints. But
everything will again be fine, however, as soon as you let makepp do
all the checks, by not passing it any limiting options. For this
reason I suggest using these hints to get quick intermediate builds,
and use lunchtime and nights to let makepp do its job thoroughly.
Build as Little as Needed
This is the same tip of using explicit targets discussed under "Build
as Little as You Need" above. But it becomes more dangerous, if you do
it because you are sure that your change will not affect any of the
other programs. Then they will not be built, even though it might have
been necessary.
Know Where Not to Build
The option "--dont-build" is very powerful for speeding makepp up a
lot. If you know one or more directories, which you are sure are
unaffected by any change you made since the last time, you can issue
"--dont-build" options for them. This can save makepp a lot of
dependency analysis. But it will not build anything in those
directories, even if it should have.
Know Where to Build
This is the same as "Know where not to build", but instead of an
exclusion list, you supply an inclusion list. The trick is that a
"--do-build" option, with a "--dont-build=/" option or under a
"RootMakeppfile(.mk)" directory without a "--dont-build" option on a
higher level directory means: build nothing except what I tell you to.
This is what users of traditional makes are looking for when they want
to build just one directory:
$ makepp --do-build=dir/subdir
or, if you don't have a "RootMakeppfile(.mk)":
$ makepp --dont-build=/ --do-build=dir/subdir
The difference is that any default target in the top level Makeppfile,
i.e. link commands are also executed this way. If you don't want that,
you must give an explicit target, which is automatically also marked
for "--do-build":
$ makepp --do-build=dir1/subdir dir2/proggie
Know What to Build
An extreme variant is asking makepp not to build anything but what you
tell it to. This is not so dangerous if you changed no include files,
only modules, and you know which programs they go into.
Say you have only changed "src/a.cpp" and "src/b.cpp" and these are
linked directly into one program. Dot is the current directory
including all subdirectories.
$ makepp --dont-build=. src/a.o src/b.o proggie1
Or equivalently, because a "--do-build" option, without a
"--dont-build" option on a higher level directory implies
"--dont-build" for the root of the build tree:
$ makepp --do-build=src/a.o src/b.o proggie1
You can do something like the following in your Shell's $ENV file or
.profile to save typing (csh users replace '=' with ' '):
alias mppb='makepp --do-build'
alias mppsb='makepp --stop --do-build'
Then the last example becomes:
$ mppb src/a.o src/b.o proggie1
Build on a RAM disk
Modern computers, especially servers, typically have a high mean time
between failure. If this is the case for you, and you have lots of RAM
to spare, you can save the time you wait for I/O. You should edit on a
real disk, or replicate your edits there quickly. But the build
results are reproducible, so they can reside in RAM. If you don't want
to risk rebuilding, you can always replicate to disk after each build
or at night. You should not do this during the build, as you might
catch partially written files, just as if the machine had crashed.
If you have a system and/or storage unit with good caching and RAID,
the gain might not be so big.
AUTHOR
Daniel Pfeiffer <occitan@esperanto.org>
perl v5.20.3 2012-02-07 MAKEPP_SPEEDUP(1)
