Introduction
The information presented here comes from work with the build and packaging system used by previous developers at Sun of the so-called Companion CD. We use techniques that are compatible with other build systems at Sun such as ON and SFW. Some changes will be made and methods streamlined as more users work with the system.
Setting up the required software
This document assumes that you have already installed the CCD source in a directory on your build machine either by downloading the tarball or checking out the svn repository. You must also have installed a very recent set of CCD packages. You should probably also have confirmed that you can do a standard nightly build of the CCD with no changes to see if your environment is set up fully.
The first build steps
If you are going to work on individual programs, the you will need to study the following steps which will set up your environment. Some of this repeats what is in the main build web pages.
Remember also that you need to put the CDDL license paragraphs into README, Makefile, install-sfw and files that are generated by hand. Files like DISTFILES.sfw, patches, and others might not need the CDDL or might now work properly if the CDDL is added.
su - gk
bldenv -f freeware-build
cd $SRC
make rootdirs
make tools
make headers
This is all needed because install scripts do chown and chgrp to root, bin, and other uids when installing things in the proto area. This cannot be done by non-root users - yet.
The Building Steps
It is recommended that you browse the entire usr/src tree before you start adding to it or changing it. Study the Makefiles and where files are placed. The best way to learn the system is to see how the current packages are done.
To see what happens in a typically build, just go into one of the source directories, say cmd/aalib, and run
make -f Makefile.sfw clobber
make -f Makefile.sfw all
make -f Makefile.sfw install
This will create aalib/SFWaalib files and directories in the proto area.
If you are planing to add a new program, we suggest that you do test builds outside the repository before you put them into the work area, but that is optional. Here is what it might be done. The csope is just an example and will not be part of the CCD.
1. Create a directory to place new source for testing. We will call it /home/me/src. The new source code in this example is cscope-15.5.tar.gz. It is probably best to try to build outside the svn working copy first so as to not accidentally commit something to the repository you have not fully tested.
2. Copy cscope-15.5.tar.gz into /home/me/src.
3. In the /home/me/src directory, uncompress and untar the source.
gunzip cscope-15.5.tar.gz
tar xvf cscope-15.5.tar
4. Change to the source directory created and determine the configure or other compile options by reading the documentation and from
../configure --help
5. Do a test configure like
./configure --prefix=/tmp/cscope --(other options)
the prefix is chosen as a directory in /tmp, but could be elsewhere. If the configure step fails, find the problem and patch the code where needed. Keep details of the patches you need to make. This will be used later.
6. Run "gmake" to see if the source compiles properly. Fix any problems and keep running gmake until a clean build is made. Note that there are several versions of the make program. gmake is GNU make and is used for compiling programs. make (without the g in front) is the parallel make that is used to run more than one make process. This make is used to do the full package building. A distributed make called dmake is also used.
7. If there are any tests included in the source, run "gmake test" or "gmake check". Fix any problems.
8. Run
make install
This will put files in /tmp/cscope.
There are cases where a program will try to put things in places other than /tmp/cscope, like /etc or /var. The CCD installs only in /opt/sfw. Builds should install in in directories like /opt/sfw/etc or /opt/sfw/var. They should never install in /etc or /var. Modify your configure options to make sure this happens.
Keep the /tmp/cscope directory and files therein for later use.
9. It is now time to put the source into the usr/src directories. There are several options you can take at this point. First, go to the build directories like lib or cmd in $SRC. Your first option is to find one of the programs that is similar to your current program. For example, a GNU based program will probably have similar build structure to another. Create a directory in say usr/src/cmd, here called cscope.
10. Copy the source cscope-15.5.tar.gz into the new cscope directory and go to that directory.
11. Create a README.sfw file and put any details about configure options and patches in it.
12. Create a Makefile.sfw file.
The following is an example, but there are others that can be studied if the simple one here does not handle more complex configures and compiles.
--------------------------------------------------------------------------
VER=cscope-15.5
all: $(VER)/config.status
(cd $(VER); env \
CC=$(SRC)/tools/gcc \
MAKE=/usr/sfw/bin/gmake \
/usr/sfw/bin/gmake)
@find . -name core -exec rm -f {} \;
ROOTSRC= $(ROOTSHSFWSRC)/$(VER)
DISTFILES:sh= cat ./DISTFILES.sfw
DISTDIRS:sh= cat ./DISTDIRS.sfw
EXFILES:sh= cat ./EXFILES.sfw
ROOTEXFILES= $(EXFILES:%=$(ROOTSRC)/%)
include ../Makefile.cmd
# List of src files to set executable is in EXFILES.sfw
install: all .WAIT install_src
$(SH) ./install-sfw
$(ROOTEXFILES) := FILEMODE= 0555
install_src: $(ROOTSRC) .WAIT $(ROOTDISTDIRS) .WAIT $(ROOTDISTFILES)
$(FIXSRC) $(ROOTSRC)
$(VER)/config.status: $(VER)/configure
(cd $(VER); env MAKE=/usr/sfw/bin/gmake \
CC=$(SRC)/tools/gcc \
LD_OPTIONS="-L${ROOT}/opt/sfw/lib -L/usr/lib
-R/opt/sfw/lib:/usr/lib" \
./configure --prefix=/opt/sfw)
$(VER)/configure: $(VER).tar.gz
/usr/bin/gzip -dc $(VER).tar.gz | /usr/sfw/bin/gtar xpf -
touch $(VER)/configure
find $(VER) -type d -exec /usr/bin/chmod 755 "{}" \;
find $(VER) -type f -exec /usr/bin/chmod ugo+r "{}" \;
clean:
-rm -rf $(VER)
include ../Makefile.targ
.SUFFIXES:
$(ROOTSRC)/%: $(VER)/%
$(INS.file)
FRC:
---------------------------------------------------------------------------
The VER variable needs to be set to the source file name. The gzip -dc line may need to be changed if the file is a .bz2 or .Z file instead. The configure line will need to be edited to include any options chosen in the test compile.
13. Now run
make -f Makefile.sfw cscope-15.5/configure
this creates the cscope-15.5 directory and does any patches (see later for more details on patching) that might be needed to the source code. Next do
cd cscope-15.5
makedist
mv *.sfw ..
cd ..
This will create three .sfw files called DISTDIRS.sfw, DISTFILES.sfw, and EXFILES.sfw and put them in the main cscope directory. These files are used later to create the source packages.
If you don't need to patch any source files, you can skip down to step 19. You should also consider submitting any patches to the original authors of the code.
14. Patching files may be a two step process. Files may need to be patched before "configure" is run or after "configure" is run, but before make is run. So, more than one patch file may be needed and therefore more than one patch step in Makefile.sfw.
For files that need to be patched before configure, say the configure file itself, one might do the following -
Go into the cscope-15.5 directory and find any files that need to be patched before "configure" is run. Suppose a configure file itself needs to be patched. Go to the subdirectory containing the configure file and do
cp configure configure.orig
Then edit the configure file to correct any problems.
Do this for all such files that need to be, or can be, fixed before configure is run.
15. Go back to the main cscope directory (the directory where Makefile.sfw is located) and run the command.
makepatch cscope-15.5
This will create a file called patch in the cscope directory. Rename the file
mv patch patch1
for example.
16. Now edit the Makefile.sfw to add the line
/usr/bin/gpatch -p0 < patch1
after the "touch $(VER)/configure" line. This will cause the patches to be applied after the source is uncompressed but before configure is run.
17. It is also possible that patches will need to be installed after configure is run. Configure may create some files that need to be patched. Follow steps similar to the ones above, Run
make -f Makefile.sfw clobber
make -f Makefile.sfw cscope-15.5/config.status
This will reinstall the source, getting rid of the .orig files and also do any patches. Configure will be run also.
Now go into the cscope-15.5 directory and look for any files created by configure that you may need to patch - you will know this from your test build earlier. Copy those files to .orig files and go back to the main cscope directory (where Makefile.sfw is located) and again run
makepatch cscope-15.5
Now move the new patch file as in
mv patch patch2
Now edit the Makefile.sfw file and put
/usr/bin/gpatch -p0 < patch2
before the /usr/sfw/bin/gmake line in the top all: section. This will patch files before gmake is run.
18. Now in the cscope directory, again run
make -f Makefile.sfw clobber
make -f Makefile.sfw cscope-15.5/configure
Then run
cd cscope-15.5
makedist
mv *.sfw ..
cd ..
to create the final set of DISTFILES.sfw, DISTDIRS.sfw, and EXFILES.sfw files that will be used to create the source part of the packages later.
Makedist is run twice in this example because we need to run make -f Makefile.sfw a number of times. The first makedist was run so that we have the files Makefile.sfw needs. The second is done so that all the patched source files and so forth are included in the source listings.
19. You can now run
make -f Makefile.sfw clobber
make -f Makefile.sfw all
to do a build and look for errors. If there are some, go back and fix them and redo the above steps as needed. If there are none, we now must create the file installation steps. The files are now going to be installed in /opt/sfw eventually, but first go into the proto area to be used to build packages.
20. Next, we want to get information on what files, directories, and links are created by the make install process. We have done this once in the test build and we will use that information now. A simple shell script like the one below called create could be written, but you may have your own means of doing the same thing.
--------------------------------------------------------------------
#!/bin/sh
rm -rf /tmp/files
rm -rf /tmp/links
rm -rf /tmp/copylist
rm -rf /tmp/finallist
rm -rf /tmp/directories
rm -rf /tmp/finaldirs
FILELIST=/tmp/files
LINKLIST=/tmp/links
COPYLIST=/tmp/copylist
DIRSLIST=/tmp/directories
find $1 -type f > /tmp/files
for FILENAME in `cat $FILELIST`
do
size=`ls -l $FILENAME | awk '{print $5 "c"}'`
name=`/usr/bin/basename $FILENAME`
result=`find . -name $name -print`
echo $result $FILENAME >> /tmp/copylist
done
find $1 -type l > /tmp/links
for LINKNAME in `cat $LINKLIST`
do
size=`ls -l $LINKNAME | awk '{print $5 "c"}'`
name=`/usr/bin/basename $LINKNAME`
result=`find . -name $name -print`
echo $result $LINKNAME >> /tmp/copylist
done
sed -e 's:'$1':${ROOT}/opt/sfw:g' < $COPY
LIST > /tmp/finallist
find $1 -type d > /tmp/directories
sed -e 's:'$1':${ROOT}/opt/sfw:g' < $DIRSLIST > /tmp/finaldirs
---------------------------------------------------------------------
This script can be placed somewhere in your PATH like /opt/onbld/bin. We now use it by going to any clean source directory cscope-15.5 in the cmd/cscope directory and run it as
create /tmp/cscope
This will create two /tmp files, finallist and finaldirs. Here is what they contain
finaldirs:
${ROOT}/opt/sfw
${ROOT}/opt/sfw/man
${ROOT}/opt/sfw/man/man1
${ROOT}/opt/sfw/bin
finallist:
./cscope-15.5/doc/cscope.1 ${ROOT}/opt/sfw/man/man1/cscope.1
./cscope-15.5/src/cscope ./cscope-15.5/contrib/webcscope/cscope
${ROOT}/opt/sfw/bin/cscope
./cscope-15.5/contrib/ocs ${ROOT}/opt/sfw/bin/ocs
The finaldirs file contains all the directories that will be used by the software install. The finallist file contains all the files that were created in those directories and where they may come from in the source directory where they were built. We note however that in the second line the script has found two cscope files in the source directory that may match with the final cscope executable. We might use the size of the file in the script to eliminate on of thes files, but, that may not work.
The "make install" step may change the file size by using strip, or just the fact that we compiled to a different final install directory might change the size of the installed files. So, we may have to do some hand editing to chose which file to install from the source directory and what we might have to do to it. Sometimes, there may not be a source file that matches any names because the installation causes the file name to change. The lines may have to be edited to enter what file was moved.
Hence, we need to be careful when automating steps like this because a lot of software that builds using libtool doesn't put stuff in an obvious place. It hides it in a .lib directory.
A create script is written in order to generate the entries in the install-sfw file (below) in the main cscope directory. Here is what we might want to generate - there are other install-sfw structures that you might prefer in the CCD source directories. It is a good idea to study a number of install-sfw files or even create your own setup so long as the script puts the directories and files in the proto area properly.
--------------------------------------------------------------------------------
#!/bin/sh
VERS=cscope-15.5
PREFIX=${ROOT}/opt/sfw
. ${SRC}/tools/install.subr
mkdir -p ${PREFIX}/bin
mkdir -p ${PREFIX}/man/man1
cd ${VERS}
_install N ./doc/cscope.1 ${ROOT}/opt/sfw/man/man1/cscope.1 root bin 444
_install N ./src/cscope ${ROOT}/opt/sfw/bin/cscope root bin 555
_install N ./contrib/ocs ${ROOT}/opt/sfw/bin/ocs root bin 555
exit 0
--------------------------------------------------------------
The VERS line is like the one in Makefile.sfw.
The PREFIX line indicates that the files will go into the proto area $(ROOT)/opt/sfw.
The
. ${SRC}/tools/install.subr
points to the location of some perl scripts which help the movement of files.
The mkdir lines get their information from the /tmp/finaldirs file. We have to be sure we the correct directories have been created so the files can install.
We next change directory to the main cscope source build directory.
the _install command tells us that we are installing a normal file and, in the first line, the man page cscope.1 is copied to the /man/man1 directory in the proto area. It is owned by root:bin with read-only permissions.
A study of the install.subr file will show the other options like N, such as L, for links.
21. Now that we have the install-sfw file in the cscope directory, we have
DISTDIRS.sfw
EXFILES.sfw
README.sfw
cscope-15.5
install-sfw
DISTFILES.sfw
Makefile.sfw
cscope-15.5.tar.gz
22. Next we run the actual build with
make -f Makefile.sfw install
(if you just want to test the building without installing the files, do
makefile -f Makefile.sfw all)
This will start the uncompressing, untar, patching, configure, make, and make install.
We can then go to $ROOT/opt/sfw and see that the files have installed the correct places.
23. In the $SRC/cmd or $SRC/lib directories, the main Makefiles need to be edited to include, in this case, the cscope directory. It is typical to put this into the list at the end of the list unless the software might be needed by some other program.
As stated before, we recommend studying the Makefiles, install-sfw files, and other files that are already done to help design your builds. Note that there are many differences since builds were done by a variety of people. As time goes on we will update these to be as consistent as possible.
Packaging
Packaging takes place in the $SRC/pkgdefs directory. Each program requires two directories. In the cscope case, these will be called SFWcscp and SFWcscpS. The first is for packaging the actual compiled files and the second is for packaging the sources. The initial commands are
cd $SRC/pkgdefs
mkdir SFWcscp
mkdir SFWcscpS
Each directory must contain a few common files.
Makefile depend prototype_com prototype_sparc copyright pkginfo.tmpl prototype_i386
You can look at the the other directories to find out what structure of each file is.
There are four files that must be created or edited, depend, copyright, pkginfo.tmpl, and prototype_com. pkginfo.tmpl is easy file to create. Simply copy one of the other pkginfo.tmpl files and edit it to include the new details. For cscope, the the SFWcscp directory, we have for pkginfo.tmpl
# # CDDL HEADER START # # The contents of this file are subject to the terms of the # Common Development and Distribution License (the "License"). # You may not use this file except in compliance with the License. # # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE # or http://www.opensolaris.org/os/licensing. # See the License for the specific language governing permissions # and limitations under the License. # # When distributing Covered Code, include this CDDL HEADER in each # file and include the License file at usr/src/OPENSOLARIS.LICENSE. # If applicable, add the following below this CDDL HEADER, with the # fields enclosed by brackets "[]" replaced with your own identifying # information: Portions Copyright [yyyy] [name of copyright owner] # # CDDL HEADER END # # PKG="SFWgdb" NAME="gdb - GNU source level debugger" ARCH="ISA" VERSION="6.2.1,REV=0.0.0" SUNW_PRODNAME="GDB" SUNW_PRODVERS="RELEASE/6.2.1" SUNW_PKGTYPE="" MAXINST="1000" CATEGORY="system" DESC="GDB - GNU source level debugger" VENDOR="www.gnu.org" HOTLINE="Please contact the owners of this software" EMAIL="" CLASSES="none" BASEDIR=/opt SUNW_PKGVERS="1.0" #VSTOCK="" #ISTATES=" " #RSTATES=' ' #ULIMIT=" " #ORDER=" " #PSTAMP=" " #INTONLY=" "
So, for a new program, it is fairly obvious what entries in the above need to be changed - they are PKG, NAME, ARCH, VERSION, SUNW_PRODNAME, SUNW_PRODVERS, DESC, and VENDOR.
The prototype_com file in SFWcscp is
# # CDDL HEADER START # # The contents of this file are subject to the terms of the # Common Development and Distribution License (the "License"). # You may not use this file except in compliance with the License. # # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE # or http://www.opensolaris.org/os/licensing. # See the License for the specific language governing permissions # and limitations under the License. # # When distributing Covered Code, include this CDDL HEADER in each # file and include the License file at usr/src/OPENSOLARIS.LICENSE. # If applicable, add the following below this CDDL HEADER, with the # fields enclosed by brackets "[]" replaced with your own identifying # information: Portions Copyright [yyyy] [name of copyright owner] # # CDDL HEADER END # # # This required package information file contains a list of package contents. # The 'pkgmk' command uses this file to identify the contents of a package # and their location on the development machine when building the package. # Can be created via a text editor or through use of the 'pkgproto' command. #search# where to find pkg objects #include # include another 'prototype' file #default # default used if not specified on entry #= # puts parameter in pkg environment # packaging files i pkginfo i copyright i depend # source locations relative to the prototype file # # SFWcscp d none sfw 0755 root sys d none sfw/bin 0755 root bin f none sfw/bin/cscope 0755 root bin f none sfw/bin/ocs 0755 root bin d none sfw/man 0775 root bin d none sfw/man/man1 0775 root bin f none sfw/man/man1/cscope.1 0644 root bin
This file is created using the file listing in /tmp/cscope above. To create the prototype_com, follow the steps
cd /tmp
rm -rf sfw
mkdir sfw
cd /tmp/cscope
cp -r * /tmp/sfw
cd /tmp
pkgproto sfw > /tmp/cscopelist
The /tmp/cscopelist file is
d none sfw 0775 root other d none sfw/bin 0775 root other f none sfw/bin/cscope 0755 root other f none sfw/bin/ocs 0755 root other d none sfw/man 0775 root other d none sfw/man/man1 0775 root other f none sfw/man/man1/cscope.1 0644 root other
Then take the beginning of a prototype_com file
# # CDDL HEADER START # # The contents of this file are subject to the terms of the # Common Development and Distribution License (the "License"). # You may not use this file except in compliance with the License. # # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE # or http://www.opensolaris.org/os/licensing. # See the License for the specific language governing permissions # and limitations under the License. # # When distributing Covered Code, include this CDDL HEADER in each # file and include the License file at usr/src/OPENSOLARIS.LICENSE. # If applicable, add the following below this CDDL HEADER, with the # fields enclosed by brackets "[]" replaced with your own identifying # information: Portions Copyright [yyyy] [name of copyright owner] # # CDDL HEADER END # # # This required package information file contains a list of package contents. # The 'pkgmk' command uses this file to identify the contents of a package # and their location on the development machine when building the package. # Can be created via a text editor or through use of the 'pkgproto' command. #search# where to find pkg objects #include # include another 'prototype' file #default # default used if not specified on entry #= # puts parameter in pkg environment # packaging files i pkginfo i copyright i depend # source locations relative to the prototype file # # SFWcscp
where we have added the line
# SFWcscp
and then we append the /tmp/cscopelist file and edit the ownerships to give the final file. It is typical for the ownership of the /opt/sfw directory to be root:sys, while the subdirectories and files therein are root:bin. See other pkgdefs files for more examples.
The other two files that are needed are copyright and depend. The copyright file should contain a list of any licenses or other ownership statements from the source code. See one of the many such files in the pkgdefs directories. The depend file contains any packages that need to be on the system in order for the packaged software to run. You can copy one of the depend files from one of the other directories to edit. Most of the SUNW packages in depend are standard ones that need to be installed for most all packages. The SFW packages are put there to notify the end user what other SFW packages are required.
Once these files are created, we run (in the SFWcscp directory)
make clobber
The packaging is then tested with
cd $SRC/pkgdefs
make awk_pkginfo
cd SFWcscp
make clobber
make all install
If problems exist, repair the files - remembering to run make clobber and make all install again.
We now must do the package for the source files. This is done in the SFWcscpS directory in $SRC/pkgdefs. The technique for creating the pkginfo and prototype files is identical to the above steps except the pkgproto step. For example, we now do
cd $ROOT/opt
pkgproto sfw/src/cscope-15.5 > /tmp/cscopesrc
and then append /tmp/cscopesrc to
# Copyright (c) 2000 Sun Microsystems, Inc. # All Rights Reserved # # @(#)prototype_com 1.2 01/04/30 # # This required package information file contains a list of package contents. # The 'pkgmk' command uses this file to identify the contents of a package # and their location on the development machine when building the package. # Can be created via a text editor or through use of the 'pkgproto' command. #search# where to find pkg objects #include # include another 'prototype' file #default # default used if not specified on entry #= # puts parameter in pkg environment # packaging files i pkginfo i copyright i depend # source locations relative to the prototype file # # SFWcscpS # d none sfw 755 root sys d none sfw/src 755 root sys
to get the prototype_com file. Again we run the test steps as above
24,. Now, go to $SRC/pkgdefs and edit the Makefile there to include the two lines
SFWcscp
SFWcscpS
to the list of other SFW directories there.
It is now a good idea to test the packages. In the packages directory, typically $PKGARCHIVE, run something like
pkgadd -d . SFWcscp
or
pkgadd -d . SFWcscpS
and see what happens. If you get attribute conflicts or other error messages, check your files yet again and correct any problems and rerun the steps.
Nightly Build
Once you have done successful builds, you can test how your build works with the whole build system. First, log out as gk (or you own build user name), and then log in again. You then run
nightly freeware-build
and then check the appropriate log files in the log directory. The file called nightly.log has all the output of the build. The mail_msg file will contain error message clues that can then be found in the nightly.log file. Depending on how your freeware-build configuration file is set up initially, the mail_msg file will be emailed to you. On my machines the nightly build takes from one and a half to two hours.
Commiting files to the SVN repository
If you believe that you have correct builds and have done all of the paperwork required by the contributor rules, you can then send the files back to the repository. But, before you do this, you might want to check for updates to your working repository to see if any changes have been made that might be relevant to your builds. Consider running the svn status command to check for changes to the main repository. You might want to do an svn update and a rebuild of your new software. Then, if things are OK, do an svn commit,
For the cscope example above you would first make sure your usr/src/cmd/cscope like directory is clean by running the make -f Makefile.sfw clobber command, then
cd $SRC
cd cmd
svn add cscope
to add cscope to the local work repository. The run something like
svn commit -m "Adding cscope to the repository"
to send the files in. When changes are sent to the repository they will be checked with a nightly build. If problems are seen, the repository will be reverted to a previous release and you will be asked to fix any problems.