Separate builder code from Module::Build

[plicease]

I've hinted at this for some time. The idea is to separate out the detection and package builder code from the Module::Build specific stuff. The original rationale that @jberger used for using Module::Build was sound, and as a user and Perl programmer (rather than a maintainer, or sys-admin, the oft cited individuals that prefer EUMM) I still prefer MB to EUMM. That being said there is a big advantage is supporting EUMM in that it is in the Perl core and thus would reduce the dependencies on anyone using Alien::Base. More importantly, we have the opportunity to make AB more easily support any installer system.

I've started Alien::Builder here:

https://github.com/plicease/Alien-Builder

and an example use case Alien::Libmicrohttpd here:

https://github.com/plicease/Alien-Libmicrohttpd

Alien::Builder is a partial replacement for Alien::Base::ModuleBuild, the other part being the installer system. In the case of EUMM it is Alien::Builder::MM which integrates nicely (or as nicely as possible) with EUMM. The POD includes summary for how I envision this to work. There are a few advantages to Alien::Builder over Alien::Base::ModuleBuild:

1. Alien::Builder is designed to be subclassed. This allows one-off extensions for Alien::foo because libfoo because perhaps the download page has some broken HTML or something. It also allows more generally useful plugins, like say git support (I envision a Alien::Builder::Download::Plugin::Git). Some of those generally useful plugins can get merged into the core of Alien::Builder/Alien::Base as they prove themselves.
2. The modular nature of the Alien::Builder class makes it easier to test and maintain.
3. More reasonable defaults. As Alien::Base has grown, we've determined that certain strategies are more reliable, such as alien_isolate_dynamic and should be the default. Also %p has been deprecated. I plan on using DESTDIR as an alternative to the blib scheme.
4. Alien::Builder can be used in a context without any installer. It might be useful, from Perl for example to install a tool into an arbitrary directory.
5. The Retriever interface allows for tarballs nested within multiple directory hierarchy
6. The Downloader interface allows to easily add new protocols (for example, git).

The way I see this proceeding is thus:

1. There are a few minor changes that need to be made to lib/Alien/Base.pm in order for it to work with an alien module built with Alien::Builder. I will submit a PR for this in the near future.
2. Release Alien::Builder as a separate distribution with an experimental status.
3. Write or rewrite some Aliens using the new Alien::Builder interface and watch the cpan testers results.
4. Merge Alien::Builder and Alien::Base into one dists.
5. Spin off Alien::Base::ModuleBuild into its own distribution.

Steps 4 and 5 could be one step.

I intend on writing a series of posts to blogs.perl.org to educate the community before any breaking changes are made. I will also be giving a talk at DCBPW (http://dcbpw.org/dcbpw2016/) about all of this (possibly Pittsburg too, if that happens).

I intend on keeping any breakage to a minimum. Please comment here if you have concerns, ideas. Right now it is relatively easy to make design changes to Alien::Builder.

I intend on donating Alien::Builder and some key plugins to the Perl5-Alien org once they become stable.

[plicease]

Alien:HTSlib https://rt.cpan.org/Ticket/Display.html?id=112608

[plicease]

Alien::Gfsm https://rt.cpan.org/Ticket/Display.html?id=112610 Alien::Moot https://rt.cpan.org/Ticket/Display.html?id=112611 Alien-DDC-Concordance https://rt.cpan.org/Ticket/Display.html?id=112613 Alien-GfsmXL https://rt.cpan.org/Ticket/Display.html?id=112614

[karenetheridge]

👍, this is a very comprehensive documentation of the current and changes!

[zmughal]

Very interesting approach. I like where it is going. I'll try to read over the code at some point.

[genio]

👍

[plicease]

I have made a lot of progress on Alien-Build (reboot of Alien-Builder):

https://github.com/plicease/Alien-Build

To do some testing, I have also created a branch with just Alien::Base and Alien::Base::PkgConfig in it:

https://github.com/Perl5-Alien/Alien-Base/tree/just-alien-base

There would also be an Alien-Base-ModuleBuild dist for legacy AB modules that would be everything that is not in the just-alien-base branch. Eventually I'd like to merge Alien-Base and Alien-Build, since that will ultimately have a much lighter set of prereqs than the current Alien-Base (including AB::MB).

[plicease]

I have some working examples for the Alien-Build recipe system that I am calling alienfile:

https://github.com/plicease/Alien-Build/tree/master/example

The format is described in the POD for alienfile:

https://github.com/plicease/Alien-Build/blob/master/lib/alienfile.pm

xz uses a fairly standard pkg-config / autoconf recipe which is easy in both AB::MB and in alienfile. Here it is without comments (but I urge you to read the commented version if you wish to get some insight into why it works):

use alienfile;

plugin 'PkgConfig' => 'liblzma';

share {
  plugin Download => (
    url     => 'http://tukaani.org/xz/',
    version => qr/^xz-([0-9\.]+)\.tar.gz$/,
  );
  plugin Extract => 'tar.gz';
  plugin 'Build::Autoconf' => ();
};

What I find compelling about this recipe system is that the bzip2 Makefile style recipe isn't really that much more complicated. Whereas with AB::MB it helps a lot to understand the internals if you want to know how to get things working with a non pkg-config / Autoconf style package. (and as someone who does know the internals of AB::MB even then it can be tricky at times).

The design of this recipe system is heavily influenced by my experience in crafting Alien modules using Alien::Base over the years, and the design addresses pretty much all of the open issues on this repository. For those that are interested I encourage feedback.

The big remaining step is to provide a thin layer between Alien-Build and Extutils::MakeMaker and to release a couple of modules that use it to get some reliability metrics from capntesters.

[jberger]

I think you know my broad opinions, since they have never been tied to Module::Build. I've always favored a system that (1) makes creating an Alien:: library as "config-only" as possible, (2) to make it relatively easy to get a the current version of the native library even without a CPAN release of the Alien:: library, (3) if installing the native library via the Alien:: installer (ie, the system doesn't already provide the library) then the system is not affected (and neither are unrelated perl installations).

That said, as we discussed in person, the current system has plenty of warts that arose from my original design decisions / relative inability at the time. If you can continue down my original goals while building a better system, more reusable and broadly applicable, I'll be overjoyed.

[plicease]

@jberger, I agree with the 3 principals that you mention and I believe Alien-Build recipe system follows those principals quite closely (or is unrelated to Alien::Build since it delegates runtime work to Alien::Base). If you disagree please let me know!

If it wasn't clear, the idea is to have a recipe, which I am calling alienfile which describes how to probe for system libs, or build from the internet. That file goes with your distribution, along side your Makefile.PL or Build.PL. There will be a thin layer like Alien::Build::MM and/or Alien::Build::MB to drive the recipe during install. (Thin layer does not yet exists, but the work that I did with Alien::Builder and Alien::pkgconf suggests that isn't really the hard part). At run-time, we'd still be using Alien::Base.

Also I want to make it ultra clear that I have always felt that using Module::Build was the correct design decision at the time! At the same time, I feel there is benefit of not tying the installer subsystem strongly with the Perl installer (EUMM, MB, Dist::Build, what have you), and the fact that EUMM is part of the Perl Core and MB is not, it is helpful to provide EUMM support out of the box.

[jberger]

I think I agree entirely (though I'll admit I haven't had the time to read your code, but I like your example). My previous comment was a wordy way of saying "I agree" but in doing so restating my overarching original goals.

[plicease]

I think that we more or less have consensus, to go ahead with the split next week. AB::MB will be a prereq of AB until 1 October 2017. Here is the public notice:

http://blogs.perl.org/users/graham_ollis/2017/03/alienbuild-vs-alienbasemodulebuild.html

[plicease]

We are a step closer, today I released Alien::Base and Alien::Base::ModuleBuild (ABMB) 0.040 as separate distributions. You can test the future when ABMB is no longer a prereq of AB by setting ALIEN_BASE_NO_ABMB=1. I have used this in travis to uncover a few prereq bugs (thankfully they have all been of the sort that only happen in CI, and would not happen in actual usage).

[devel-chm]

Congrats and thanks! As someone who never quite grokked Module::Build, I'm looking forward to a more build-agnostic AB technology.

[plicease]

I think the next step once AB::MB is no longer a prereq of AB is to reunify one or more of:

Alien-Base
Alien-Build
Alien-Base-Wrapper
Test-Alien

further reading from IRC:
https://irclog.perlgeek.de/native/2017-07-07#i_14841877

[plicease]

I've merged Alien-Base-Wrapper and Test-Alien into Alien-Build. The only remaining piece is to merge Alien-Base into Alien-Build. I am going to do this an cut a dev release probably Wednesday.

This was discussed on IRC, but the consensus seemed to be that the name for the merged dist should be Alien-Build. I like both names, and see advantages for both. In particular, for now, I think Alien-Base may still have better name recognition. On the other hand Alien-Build is doing a lot of the heavy lifting and is the majority of the code in any combined dist. Would-be Alien authors will be interacting more with Alien::Build + alienfile than Alien::Base. There may also be a "shiny" advantage in calling it Alien-Build.

I have a branch in Alien-Build which merges Alien-Base:

https://github.com/Perl5-Alien/Alien-Build/tree/alien-base

I've tried to keep as much of the change history in there, and to include the appropriate copyright and author attributions. If anyone does have any concerns this is a time to take a look at the above branch and voice them, either here or by opening an issue on Alien-Build.