%
% $Header$
%
\begin{onlystandalone}
\documentstyle[11pt,literate]{article}
\begin{document}
\title{The ``make world'' configuration system---a guide for the brave}
\author{The GRASP Team (scribe: Will Partain), \\
Department of Computing Science, \\
University of Glasgow, \\
Glasgow, G12 8QQ, UK. \\
\\
Email: glasgow-haskell-\{bugs,request\}\@dcs.glasgow.ac.uk}
\maketitle
\begin{rawlatex}
\tableofcontents
\clearpage
\end{rawlatex}
\end{onlystandalone}

%************************************************************************
%*									*
\section[make-world-tutorial]{An introduction to the ``make world'' configuration subsystem}
%*									*
%************************************************************************

{\em [MEGA-NOTE: This subsystem needs SUBSTANTIAL REVISION; however,
the required time hasn't sprung/isn't springing free.  Also, there is
no clear winner emerging in the How-to-Configure-Free-Software
sweepstakes.  (Ideas welcome.)  This documentation is only intended as
a ``survival guide'' in the interim.]}

Our sources are manipulated with a variant of the ``imake'' system
that comes with the X Window System.\footnote{Actually, our variant is
a variant of Paul DuBois's ``imake for multiple projects,'' from the
Wisconsin Regional Primate Research Center.  We are grateful to him
for making his work available to the world.}  (We call our version of
the main program ``jmake,'' so things don't get mixed up.)  Our
everyday term for the system is ``make world,'' because the system's
mission is to (re-)build {\em everything} correctly when you type
\tr{make world}.

%************************************************************************
%*									*
\subsection[make-world-goals]{Goals of ``make world''}
%*									*
%************************************************************************

The purpose of the ``make world'' subsystem is to help with these
configuration tasks:
\begin{itemize}
\item
Manipulating the system as a whole.  We want to be able to do:
\begin{verbatim}
% cd ghc-<version>
% make <target>
\end{verbatim}

and for the make \tr{<target>} to be built {\em throughout} the system
(e.g., \tr{make depend}, to build all make dependencies in the
distribution).  The code to do this recursive make-ery must live in
one place and must {\em not} rely on (zillions of) hand-built
Makefiles to work.

\item
Porting the system to a new machine platform.  All machine dependencies
must be gathered together in one place and, again, must not be
distributed through zillions of Makefiles (just waiting to go wrong).

\item
Customising the system for a particular site.  Same rationale.

\item
Customising the \Haskell{} compilation system that you build.  Example
possibilities (not all relevant at this time): choosing one of several
parsers, choosing which optimisation passes to build in, choosing to
omit/include some debugging code, choosing which code generator(s) to
build in, choosing which C compiler to use in compiling
\Haskell{}-generated C files, what C compiler options to use in that
case, and on and on....

\item
Customising the installation of the compilation system.  Where the
binaries go, where the library files go, ...

\item
Supporting several builds/customisations ongoing simultaneously.

\item
Supporting several people working from a master copy of the source.
\end{itemize}

%************************************************************************
%*									*
\subsection[make-world-concepts]{Basic concepts of ``make world''}
%*									*
%************************************************************************

The ``make world'' subsystem builds an ordinary \tr{Makefile} for each
directory in a system, from a set of system-wide configuration files
(in the \tr{mkworld/} directory) and a single directory-specific file,
called the \tr{Jmakefile}.  Things the ``make world'' system needs to know
from you:
\begin{description}
\item[Machine architecture of the host you're building with:]
It normally figures this out for itself.

\item[What project you're configuring for:]  For the new Glasgow
Haskell compiler, use \tr{ghc}.

\item[What setup of the project you're configuring for:]  Unless you
have reason to do otherwise, use the \tr{std} setup.

\item[The absolute pathname of the top of the source tree.]
\end{description}

Armed with this information, the ``make world'' system will create
\tr{Makefiles} in all the subdirectories of a system.  Each of the
Makefiles will be, in effect, a {\em database of configuration facts}.
Obviously, the program \tr{make} can then use these facts.  You can
also use the utility program \tr{msub} to substitute these facts into
other source files.

%************************************************************************
%*									*
\subsection[make-world-operation]{Basic operation of ``make world''}
%*									*
%************************************************************************

Just like its ``imake'' ancestor for the X Window System, ``make
world'' is a {\em low-tech} solution to the tasks mentioned above.
The files involved are:
\begin{description}
\item[Platform-, site-, project-, and setup-dependent configuration files:]
{\em All} such information lives in {\em one} directory, notably the
top-level \tr{mkworld} directory.

\item[Configuration file for a particular directory:]
This is what the one-per-directory \tr{Jmakefile}s are.
\end{description}

All of the above files are a mixture of \tr{cpp} (C pre-processor)
directives and plain-vanilla make stuff (very plain vanilla---so it
will work with any make). The \tr{jmake} program simply feeds the
master template file (\tr{mkworld/Jmake.tmpl}) into \tr{cpp}, which
will then \tr{#include} all of the appropriate platform-, site-,
etc., dependent files, as well as the \tr{Jmakefile} for the directory
in question.  \tr{cpp} expands all of the \tr{#define}d macros, and
what you are left with is an ordinary (but large and confusing)
\tr{Makefile}!

Here is a small, possibly instructive example, from the \tr{Jmakefile}
for this document:
\begin{verbatim}
INeedLitStuffHere(all depend)

LitDocRootTargetWithNamedOutput(root,lit,root-standalone)
\end{verbatim}

The exact meanings of these \tr{cpp} macro-invocations are not
important;\footnote{They say that (1)~the literate stuff must be available
before doing \tr{make all} or \tr{make depend}, and (2)~we want to
build a literate document with provision for both standalone
processing and otherwise.} what is important is what you {\em don't}
see: hard-wired filenames about where to find the literate programs or
other tools, hardwired install directories, presuppositions about how
your man pages are arranged, what kind of hardware you have,
idiosyncratic (and probably inappropriate) \tr{make} suffix rules,
etc., etc.

After expanding the master \tr{Jmake.tmpl} template with all of the
appropriate configuration files in \tr{mkworld/}, the above fragment
blows out into (in the real \tr{Makefile}):\footnote{Some long lines
truncated.}
\begin{verbatim}
all depend:: $(INFO) $(MAKEINFO) $(POSTMAKEINFO)

$(INFO) $(MAKEINFO) $(POSTMAKEINFO):
	@echo "checking $@ over in" $(INFOSRC) "first..."; \
	cd $(INFOSRC); $(MAKE) all; \
	echo "okay, continuing in $(CURRENT_DIR)"

all depend:: $(LITSRC)/lit2pgm $(LITSRC)/lit2texi $(LITSRC)/lit2latex [...]

$(LITSRC)/lit2pgm $(LITSRC)/lit2texi $(LITSRC)/lit2latex [...]
	@(cd $(LITSRC); if [ -f lit2pgm ]; then \
	echo "checking $@ in $(LITSRC) first..."; $(MAKE) all; else \
	echo "bootstrapping $@ with lit2pgm.BOOT in $(LITSRC) first..."; \
	$(MAKE) LIT2PGM="lit2pgm.BOOT -Alit2pgm" lit-deatify all; fi; \
	echo "okay, continuing in $(CURRENT_DIR)")

docs:: root.info root.dvi

clean::
	$(RM) root.info root.dvi root.texi root.tex

depend::
	$(MKDEPENDLIT) $(MKDEPENDLITFLAGS) root.lit

root.info:: root.texi
	$(RM) $@
	$(MAKEINFO) $(MAKEINFOFLAGS) root.texi \
	&& $(POSTMAKEINFO) root.info

root.dvi:: root.tex
	$(RM) $@
	$(LTX) root.tex

root.texi: root-standalone.itxi
	$(RM) $@
	$(LIT2TEXI) -S $(LIT2TEXIFLAGS) -N root.info -o root.texi root-standalone.itxi

root-standalone.itxi: root.lit
	$(RM) $@
	$(LIT2TEXI) -S -c $(LIT2TEXIFLAGS) -o root-standalone.itxi root.lit

root.tex: root-standalone.itex
	$(RM) $@
	$(LIT2LATEX) -S $(LIT2LATEXFLAGS) -o root.tex root-standalone.itex

root-standalone.itex: root.lit
	$(RM) $@
	$(LIT2LATEX) -S -c $(LIT2LATEXFLAGS) -o root-standalone.itex root.lit
\end{verbatim}

%************************************************************************
%*									*
\subsection[make-world-more-info]{More information about Imake (``make world'' ancestor)}
%*									*
%************************************************************************

{\em As I say, the ``make world'' system is a clear candidate for
revision, so we aren't over-exerting ourselves to make this version
better.}

I got the ``imake'' code that I turned into ``jmake'' (a.k.a. ``make
world'') from Paul DuBois, at the Wisconsin Regional Primate Research
Center.  The principles of operation are {\em exactly} the same, so I
have put a copy of every known ``imake'' document in the
\tr{mkworld/doc/} subdirectory.

%************************************************************************
%*									*
\subsection[config-file-org]{Organisation of the \tr{mkworld} configuration files}
%*									*
%************************************************************************

The main Makefile template in \tr{mkworld/} is \tr{Jmake.tmpl}.  It
first determines the machine type, then starts slurping these files
(all in \pl{mkworld/}) in this order (assume project \tr{project} and
setup \tr{setup}):
\begin{verbatim}
# find out things about the machine
<a platform-specific file>

# now things about your site
site-project-setup.odef
site-project.odef
site.def

# things all projects can use but that your project/setup might override
AllProjects-project-setup.otmpl
AllProjects-project.otmpl
AllProjects.tmpl

# things just for your project
Project-project-setup.otmpl
Project-project.tmpl

# the rules, or macro definitions, actually used in Jmakefiles
# (may be overidden [uncommon])
Jmake-project-setup.orules
Jmake-project.orules
Jmake.rules

# finally, the directory-specific what-you-want-to-happen info
Jmakefile
\end{verbatim}

Notice how you can always use a \tr{*-project-setup.*} file to
override a \tr{*-project.*} file, which you can use to override the
general defaults...

%************************************************************************
%*									*
\subsection[std-make-world-actions]{Standard actions in ``make world''}
%*									*
%************************************************************************

The expected normal cycle of \tr{make}ing with a ``make world''-ified
project (once you're sure all your configuration files are set up
correctly!):
\begin{description}
\item[\tr{make Makefile}:]
Make the very topmost Makefile.  It's crucial to get this right!
\item[\tr{make Makefiles}:]
Make all the other Makefiles (recursively).
\item[\tr{make depend}:]
Make all the automatically-generated dependencies (with
\tr{mkdependHS}, \tr{mkdependC}, \tr{mkdependlit} etc.).
\item[\tr{make all}:]
Build the system(s).
\item[\tr{make tags}:]
Make some Emacs-friendly TAGS files, if you believe in them.
\item[\tr{make runtests}:]
Run any test suite that you may have provided.
\item[\tr{make docs}:]
Make any documentation that goes with the system(s).
\item[\tr{make install}:]
Install the system(s).
\item[\tr{make install_docs}:]
Install the documentation.
\item[\tr{make clean}:]
Clean up after yourself.
\item[\tr{make veryclean}:]
If you really want to get rid of all machine-generated files...
\end{description}

Another generally-supported make target is: \tr{make whoami} (echos
the project and setup strings).

%************************************************************************
%*									*
\subsection[when-to-make-what]{When to use what \tr{make} target}
%*									*
%************************************************************************

Several of the standard ``make world'' targets depend upon you to
re-run them at the right time.

\begin{itemize}
\item
The normal thing you do 90\% of the time is just type:
\tr{make} (or, equivalently, \tr{make all}).

\item
If you change a \tr{Jmakefile} (or the configuration files in the
\tr{mkworld/} directory), then you need to re-\tr{make Makefile}.

\item
If you changed the configuration files in some globally-important way,
then you should re-\tr{make Makefiles}, {\em after} the
\tr{make Makefile}.

\item
If you change anything from which make dependencies are automatically
generated (e.g., import declarations in \Haskell{} modules or
\tr{#include} directive in C programs), then you need to
re-\tr{make depend} before continuing with a regular \tr{make}.
\end{itemize}

Rather than worry too much about these constraints, we re-run these
things every night:
\begin{verbatim}
% (make Makefile && make Makefiles && make depend) >& overnight.log
\end{verbatim}

%************************************************************************
%*									*
\section[day-to-day-make-worlding]{Day-to-day use of ``make world'' (valuable tips!)}
%*									*
%************************************************************************

%************************************************************************
%*									*
\subsection[bad-Makefile]{When you mess up your \tr{Makefile}}
%*									*
%************************************************************************

The NUMBER ONE \tr{make}ing RULE: If, in running \tr{make Makefile} or
\tr{make depend} or {\em anything}, you get a message indicating your
\tr{Makefile} is rubbish, {\em immediately} (a)~save a copy to look
at, and (b)~restore a viable \tr{Makefile} with:
\begin{verbatim}
mv Makefile.bak Makefile
\end{verbatim}
That way, at least you will have a working (if out-of-date)
\tr{Makefile} while you consider your options!

%************************************************************************
%*									*
\subsection[new-dir-in-mkworld]{Adding a directory to a ``make world'' source tree}
%*									*
%************************************************************************

\begin{verbatim}
% mkdir new_dir

% emacs ./Jmakefile
    # Add "new_dir" to the list of SUBDIRS (in parent directory).

% emacs new_dir/Jmakefile
    # Create a Jmakefile for the new subdir; probably copy one from elsewhere.
    # An empty Jmakefile works, and is sometimes useful.

% make Makefile
    # Tells current-directory's Makefile that it has a new subdirectory.

% make SUBDIRS=new_dir Makefiles
    # makes the Makefile (from the new Jmakefile) in "new_dir"
\end{verbatim}

%************************************************************************
%*									*
\subsection[feeding-flags-with-mkworld]{Feeding command flags into ``make world''}
%*									*
%************************************************************************

NOT DONE YET.

%************************************************************************
%*									*
\subsection[overnight-mkworlding]{Nice ``mkworld'' things to do overnight}
%*									*
%************************************************************************

Out-of-sync ``make world'' stuff spells eventual trouble.  What we do
every night (at a minimum) is:
\begin{verbatim}
% cd top-of-your-world
% make Makefile
% make Makefiles
% make depend
% make all
\end{verbatim}

%************************************************************************
%*									*
\subsection[fastmake-script]{Using the \tr{fastmake} script (to cheat)}
%*									*
%************************************************************************

One rather great pain about the make-dependencies generated by
\tr{makedependHS} for the compiler is that there are zillions of them,
and quite innocuous changes to the source modules can spawn {\em
massive} recompilation.  If you keep your wits about you, judicious
use of the \tr{fastmake} script (provided in \tr{utils/scripts}) can
help a lot.

Typing \tr{fastmake -help} gets you the documentation for the script;
it also describes the only two uses that I would recommended:
(1)~Recompiling when interfaces are known to be stable [very common], and
(2)~Compiling a module that's just had functions {\em added} then
tweaking the \tr{.hi} file's timestamp so the change ``doesn't show''.

%************************************************************************
%*									*
\subsection[using-sun4s-and-sun3s]{Using both Sun4s and Sun3s to compile}
%*									*
%************************************************************************

[This is happily irrelevant in a Sun4-only compiling-with-HBC world.
It is most useful if compiling-for-Sun3s (prototype Glasgow compiler),
but you'd really rather use a Sun4 to do it [to go faster].]

It is a pain to build the compilation system for Sun3s,
even though most of us have Sun4s.  This is really only a
problem for the compiler proper (\tr{compiler/hsc}).  Here's what we
do.  (We assume you have a copy of the Sun4-to-Sun3 version of the
prototype compiler.)

The basic idea, after doing \tr{cd compiler}, is to do \tr{(l|fast)make asms}
on a Sun4 (to make all the \tr{.s} assembler files) and then finish
the job on a Sun3 with \tr{make hsc} (or \tr{make all}).

The problem is that the rest of the distribution has been built for a
Sun3, so when \tr{make} reaches out to run some other part of the
distribution (e.g., \tr{lit-deatify}, run by \tr{lit2pgm}), you'll be
trying to run a Sun3 binary on a Sun4.  No go.

There are two solutions.  Both depend on {\em installing} at least the
literate-programming stuff, and perhaps the ``make world'' stuff, too.
(\tr{cd} to those subdirectories, and see if \tr{make install} does
what you want.)

The first solution is to use the \tr{lmake} script, provided in
\tr{utils/scripts/lmake.sh}, when you are compiling with a Sun4.
This trivial script merely overrides \tr{make}'s default ideas about
where to find some programs.

The second solution is to reconfigure your whole system to use
``installed utilities,'' a mysterious and murky `feature' of the
make-world system.  Cd to the root directory of your source tree and
proceed just as described in the installation document,
with {\em one change}: Instead of the magic line:
\begin{verbatim}
make -f Makefile.BOOT BOOT_DEFINES="-P ghc -S std -DTopDirPwd=/h/me/ghc-0.06"
\end{verbatim}

use this other even more magic line:

\begin{verbatim}
mkworld/jmkmf -P ghc -S std -DTopDirPwd=/h/me/ghc-0.06
\end{verbatim}
Then \tr{make Makefiles}, \tr{make depend}, etc., as before.  After
all this, you should be able to \tr{make asms} on either a Sun3 or
Sun4, without problems.

%************************************************************************
%*									*
\section[sharing-w-link-trees]{Sharing source code using symbolic-link trees}
%*									*
%************************************************************************

If you want to build the system for several architectures, setups, or
whatever, but not duplicate the sources, you can make a {\em link tree}
of the distribution---that is, a ``clone'' of the distribution
directory structure but with each file represented by a symbolic link
to the real source file.

If you make changes to the real source file, then you simply have to
re-\tr{make} in the link tree to rebuild for that arch/setup.
Contrariwise, things that happen in the link tree should not affect
the master sources.

This is what we do at Glasgow, where several people are hacking on
bits of the compiler at any given time.  These notes were written
mainly to help us remember what we did last time!

%************************************************************************
%*									*
\subsection[link-tree-principles]{The objectives of working this way}
%*									*
%************************************************************************

What we want to have on our disks:
\begin{enumerate}
\item
A single common more-or-less-stable source tree that everyone can
share (assumed to be about a half dozen people).  ``Read-only.''

\item
A handful of common ``builds'' of the common source for various
machines, which people can share.  These ``build trees'', also
``read-only,'' link into the common source tree (above).

Out of the whole distribution, one person is likely to be interested
in a tiny part of it.  There is no reason they should have a full set
of object files (which is probably huge).  By linking into a common
build tree, all those objects can be shared.

\item
Separate link trees for individuals, that they can play with and use
to prepare wadges of ``stable code'' --- to be fed back into the
``common source tree'' periodically.
\end{enumerate}

What we want to do with them:

%************************************************************************
%*									*
\subsection[basic-link-trees]{Creating link trees of object files for everyone to share}
%*									*
%************************************************************************

Let's assume that the basic source tree is in \tr{~grasp/ghc}.

Now, if everyone simply links to this and \tr{makes} thereafter, each
link tree will end up with a full set of objects, most of which will
be of no interest to the link-tree owner.

So, we do a basic build for each architecture, and then link {\em into
those object-full trees}!  Here goes:
\begin{verbatim}
% mkdir ghc-full-hbc-sun4	# full setup, compile w/ hbc, for sun4
% cd ghc-full-hbc-sun4
% lndir ~grasp/ghc		# make the links

% make -f Makefile.BOOT BOOT_DEFINES="-P ghc -S full -DTopDirPwd=/users/fp/grasp/ghc-full-hbc-sun4"
\end{verbatim}

%************************************************************************
%*									*
\subsection[create-link-tree]{Creating a link-tree version of the compiler}
%*									*
%************************************************************************

Suppose the source is in \tr{~grasp/ghc}.  We want to make a link tree
called \tr{./ghc-sun4}, which will be used to do a build for Sun4s.
Here's how to set up the basic structure:
\begin{verbatim}
% mkdir ghc-fred
% cd ghc-fred
% lndir ~grasp/ghc
\end{verbatim}

If you wish to be a bit tidier:
\begin{verbatim}
% cd ghc-fred
% zap-junk-links
\end{verbatim}

Now fiddle the configuration stuff, as described in the installation
document.  make your Makefiles and dependencies, ..., also as
described there.  It will be something like:
\begin{verbatim}
% make -f Makefile.BOOT BOOT_DEFINES="-P ghc -S fred-setup -DTopDirPwd=/u/h/ghc-fred"
\end{verbatim}

ToDo: we need to provide emacs-ery to ensure copying works right.

We often build not-quite-full link trees: usually just a link tree of
the \tr{compiler/} directory.  See the example not too far below
about building a new setup.

%************************************************************************
%*									*
\subsection[chopping-down-link-trees]{Finding the good stuff in a link tree}
%*									*
%************************************************************************

(NOT TESTED RECENTLY.)

Suppose, as in the previous section, you had a link tree
\tr{./ghc-sun4}, linked to \tr{~grasp/ghc}.  Suppose that, for some
reason, you have made changes in the \tr{./ghc-sun4} tree, and you
want to find those changes, so that they might be applied to the
(presumably) master source in \tr{~grasp/ghc}.
\begin{verbatim}
make veryclean
# remove the links
find . -type l -print | xargs rm
# remove the generated Makefiles
find . -name Makefile -print | xargs rm
# rm the generated .prl files and lit2stuff in ./literate ???
# take the diffs
diff -rc2 ~grasp/ghc .
# Remove the fluff from the diffs, mangle, etc., etc.
# Watch particularly for files that exist _only_ in ghc-sun4!
# ...
# now apply them if you wish
cd ~grasp/ghc
patch -p0 < <the-diffs-file>
\end{verbatim}

%************************************************************************
%*									*
\subsection[new-setup]{Building a new setup (configuration) of the system}
%*									*
%************************************************************************

NOT DONE YET.  (Not expected to be hard.)

This is what I did when I created a special setup for trying to
bootstrap the compiler (compile it with itself).

First, I created a directory for it:
\begin{verbatim}
% mkdir ~grasp/ghc-boot
\end{verbatim}

I then made symbolic links to the master copy of all parts of the
compilation, except the compiler itself.  There, I made a whole
symlink-tree.
\begin{verbatim}
% cd ~grasp/ghc-boot
% ln -s ~grasp/ghc/* .
% rm compiler		# put in a real directory for this one piece
% mkdir compiler

% cd compiler
% lndir ~grasp/ghc/compiler # make the link tree -- takes time!

% mkdir tests		# lndir doesn't mess w/ "tests" subdirs
% touch tests/Jmakefile
\end{verbatim}

This is going to be a setup called \tr{boot} (still for the project
\tr{ghc}).  The ``make world'' system requires that the following
files exist (they can be empty):
\begin{verbatim}
ghc% ll mkworld/*boot*
-rw-rw-r--  1 partain  grasp   0 Aug  6 20:36 mkworld/AllProjects-ghc-boot.otmpl
-rw-rw-r--  1 partain  grasp   0 Aug  6 20:35 mkworld/Jmake-ghc-boot.orules
-rw-rw-r--  1 partain  grasp   0 Aug  6 20:36 mkworld/Project-ghc-boot.otmpl
-rw-rw-r--  1 partain  grasp 133 Aug  6 20:50 mkworld/site-ghc-boot.odef
\end{verbatim}

Now, we need to tell the ``make world'' system about this special setup and
to tell it about the new ``top of the world'' (\tr{~grasp/ghc-boot}), I do:
\begin{verbatim}
% cd ghc-boot/compiler
% rm Makefile
% ../mkworld/jmake -P ghc -S boot -I../mkworld -DTopDirPwd=/users/fp/grasp/ghc-boot -DTopDir=../. -DCurDir=./compiler
../mkworld/site-ghc.odef: 9: TopDirPwd redefined
\end{verbatim}
(It's brute force, but it works!  The crucial bits are: \tr{-S boot},
and \tr{-DTopDirPwd=/users/fp/grasp/ghc-boot}.)

Test that your \tr{Makefile} stuff is OK by...
\begin{verbatim}
% make Makefile
% make Makefiles
% make clean		# tidy up some of the junk:
			# won't work until the Makefiles are properly made
\end{verbatim}

I had to fiddle some of the ``make world'' flags mentioned above even
after I'd made my Makefiles.  No problem---after the fiddles, just do:
\begin{verbatim}
% make Makefile
% make Makefiles
% make depend
\end{verbatim}
(All three should have been successfully at some point before getting
down to business.)

In my case, getting down to business was something like...
\begin{verbatim}
% fastmake -k hcs
\end{verbatim}

\begin{onlystandalone}
\printindex
\end{document}
\end{onlystandalone}