Choosing tools for PHP application automation

Summary

This little post narative the journey that I had been through of using various building tools for PHP application. In this post, I will provide example build files for each tool for achieving similar target.

Phing

Moving from C/C++ world into PHP many years ago. I was using make and CMake for my project automation. In PHP world, I am constantly looking for something that is inherit and versatile too. No supprise, Phing jumped into my eyes as a written in PHP buillding tool. Then I started use it.

I stealed maven's idea of lifecycle and divide the phing config files into separate ones so that I can reuse them. They are

  • compile.xml
  • deploy.xml
  • package.xml
  • aws.xml1)

And in my main build.xml, I will import them as below

   <fileset id="sources" dir="${app.basedir}" expandsymboliclinks="false">
    <include name="tools/**" />
    <include name="web/application/**" />
    <exclude name="web/application/configs/application.local.ini" />
    <exclude name="web/application/configs/db.php" />
    ...
  </fileset>
 
  <!--classpath for importing customized tasks-->
  <includepath classpath="${app.basedir}/config/phing/lifecycle" />
 
  <import file="config/phing/lifecycle/deploy.xml" />
  <import file="config/phing/lifecycle/compile.xml" />
  <import file="config/phing/lifecycle/package.xml" />
  <import file="config/phing/lifecycle/aws.xml" />
 
  <target name="main" depends="package" >
    <echo>Hello World</echo>  
  </target>
  ...

Then my phing -l produce

Main targets:
-------------------------------------------------------------------------------
 all                    Bld: shortcut of package and deploy
 aws.fireup             AWS: finish the last meter of release
 aws.launch             AWS: launch latest ami.
 aws.leg1               AWS: launch latest ami and deploy to it
 aws.leg2               AWS: from ec2 to Lc. params[instance|ami]
 aws.release            AWS: safe button release
 cleanfilecache         Dev: help purge the files in cache dirs
 compile                Bld: whole compile lifecycle
 compile.compile        Bld: whole compile lifecycle
 compile.less           Bld: generate less file
 deploy                 Bld: deploy to candidate server. params:[host]
 deploy.cleanfilecache  Dev: help purge the files in cache dirs
 deploy.dumpdb          Dev: create a dev db dump file. params[dumpfile]
 deploy.restoredb       Dev: Refill the database with fixtures
 deploy.upgradedb       Dev: create database upgrade SQL files
 dumpdb                 Dev: create a dev db dump file. params[dumpfile]
 fireup                 AWS: finish the last meter of release
 launch                 AWS: launch latest ami.
 leg1                   AWS: launch latest ami and deploy to it
 leg2                   AWS: from ec2 to Lc. params[instance|ami]
 less                   Bld: generate less file
 package                Creating tar package file
 package.all            Bld: shortcut of package and deploy
 package.deploy         Bld: deploy to candidate server. params:[host]
 package.package        Bld: make package tarball
 release                AWS: safe button release
 restoredb              Dev: Refill the database with fixtures
 upgradedb              Dev: create database upgrade SQL files

Subtargets:
-------------------------------------------------------------------------------
 -movefiles
 -prepare
 -valid
 -validate
 aws.-valid
 clean
 compile.-validate
 compile.versioning
 deploy.-validate
 main
 maketar
 package.-movefiles
 package.-prepare
 package.-validate
 package.clean
 package.maketar

Just skip a lots of details and focus on import feature for now. Importing those files into build.xml enables a quick and full fledged build configuration. In practics, I shared them within a lots of projects and had no problem. If I need to overwrite some 'task' ( 'target' in phing's terminology), I just need to define it in 'build.xml' with the same 'task' name. Variables (property) are available when it's not defined within tasks.

Besides that, there are some small tricks:

  • Difference between 'Main targets' and 'Subtargets' is if you provide 'description' or not.
  • task name starting with '-' is to make it private because you cannot run it from CLI, for example phing -validate

It looks good. However, as I used it long and trying to do things neat. I will miss the well defined namespace, conditional control over tasks and some feature like closure. Although creating a customized task in phing is pretty easy, it's still breaks the DRY principle. Firstly most of the tasks are very light and hardly reusable. Creating a customized task seems like a waste but there is no other better way. Secondly I end up with copying a lots of scalefolding code to make the task class.

More itchy thing is that what task apply do. I had defined a set of assets files, js and css. I would like to collect a timestamp for all of them, which I used as a version number to beat the browser cache in frequent releases.

Initially I was using a 'foreach' task and calling another task which collect timestamp of one file at a time. This was before Phing introduce 'apply' task. The problem was that 'foreach' produces too much noise which I cannot suppress selectively. I argued with myself, “OK, that's just some extra logging. I can filter them by sed”. But I cannot produce any peace in my mind. Although the build efficiency isn't a big concern, the 'code'2) is too itchy.

To my excitment, I discovered 'apply' task. which deal with the exact senario as above. Then I altered my 'code' as below

        <apply executable="export" dir="${assetDir}" output="${versionFile}.tmp" append="true" relative="true" >
          <arg line="res=$(echo" />
          <srcfile/>
          <arg line="); echo $res=$(stat -c %Y $res)" />
          <fileset refid="resources" />
        </apply>

Yeah! I have got rid of those noises. But this time, it's even more itchy. Too quirky.

Ant

Later I got a chance to start a project which wasn't in hurry. I decided to see if Ant, which Phing borrows the idea, is any better. I wasn't do this simply because I didn't know much about JAVA and emotionally didn't like it. Too heavy. But Ant, to my supprise, is way better and more mature. Not only I can implement what I have got so far using Phing, but also provide better support for conditions. The build xml file is shorten a lot. All tasks can have a condition meta.

In example below, I try to compile a hash file3) recording all files within a release package file. Tool md5deep is way quicker and better than using md5sum. However not every environment will have that ready. Thanks to unless:set and if:true meta value, the configuration is more neat and reading friendly than Phing's peer.

    <target name="package" depends="pre.package">
        <tstamp>
            <format property="rls.stamp" pattern="yyyy-MM-dd_H-m" />
        </tstamp>
        <exec executable="composer" dir="${dir.buildsrc}" failonerror="true" logError="true">
            <arg value="install" />
            <arg value="--no-dev" />
            <arg value="--prefer-dist" />
            <arg value="--optimize-autoloader" />
        </exec>
        <delete file="${dir.buildsrc}/checksum" quiet="true" failonerror="false" />
        <!-- prefer md5deep over md5sum on creating signatures-->
        <available file="md5deep" filepath="${env.PATH}" property="hasMd5deep"/>
        <apply executable="md5sum" output="${dir.buildsrc}/checksum" append="true" unless:set="hasMd5deep">
            <srcfile />
            <fileset dir="${dir.buildsrc}" />
        </apply>
        <exec executable="md5deep" dir="${dir.buildsrc}" output="${dir.buildsrc}/checksum" if:true="${hasMd5deep}">
            <arg value="-lr" />
            <arg value="." />
        </exec>
        <tar destfile="${dir.build}/${ant.project.name}.${rls.stamp}.tgz" 
            basedir="${dir.buildsrc}" longfile="gnu" compression="gzip" />
    </target>

Bueaty isn't it? And lots of effort has been put into using Ant to do PHP jobs, such as https://github.com/shrikeh/ant-phptools/. But it still have the apply task type. And it only support of constructing a command line or equivalent for applying that to a fileset. How much can you do within a command line? So itchy point is still itchy until I read about this Node.js as a build script , while I constantly evaluate other tools4).

The problem with Ant is that XML is a markup language and when you try to do things that requires a programming language (logic, loops, etc..)

It is very inspiring. The merits of build tools are that they simplify the dependency resolving as well as providing a lot of common tasks or commands. But using a XML file looses the good part of programming language such as closure. I agree with the author that lots of existing tools share the same problem.

Then I met gradle.

Gradle

Before having the idea planted, I probably won't take a second look at gradle simply because it was JAVA, which I have very limited knowledge. Since my good experience with Ant and Gradle looks like a full-fledged implemetation embracing that idea. I kind of forcing myself using it.

What a bueaty!

  • It supports ant tasks, which means it doesn't need to create the wheel again.
  • It support groovy. Closure, loop are handy and everywhere
  • Java class library is well used.
  • ……

Here is an example that I am going to compress js and css files.

task compress(dependsOn: prepareFiles) << {
    description = 'Crompress js or css files'
 
    //set the public dir for assets
    buildPublic.from("$buildDir/src/public")
 
    def css = buildPublic.matching {
        include '**/*.css'
        exclude '**/*.min.css'
        exclude '**/bootstrap*.css'
    }
 
    def js = buildPublic.matching {
        include '**/*.js'
        exclude '**/html5shiv*.js'
        exclude '**/bootstrap*.js'
        exclude '**/*.min.js'
    }
    def assets = ['js': js, 'css': css]
 
    def yuicompressor = file('src/vendor/bin/yuicompressor.jar')
 
    assets.each {type, asset ->
        asset.each { File f ->
            javaexec {
                main = '-jar'
                args = [yuicompressor, '-v','--type', 
                    type, '-o', ".$type\$:.min.$type", f]
            }
        }
    }
}

End

I definitly recommend Gradle for PHP applications.

1) This came later for automated deployment to AWS environment
2) They are descriptive xml instead of programming code
3) This is for verifying release integrity later on
4) Unfortunately I have not got my head around rake yet