Building custom OpenWRT packages: an (hopefully complete) guide

Disclaimer: in order to keep this post as short as possible, I’m going to be slightly technical. Should you have trouble following this guide, please, leave a comment below: I’ll be proud to help you in an ad-hoc thread


1. The context

If you are reading these notes, chances are quite high that you ALREADY rely on OpenWRT for some of your needs (Wireless; Embedded; etc.).

In my case, I own a nice TP-Link WDR4300 that loooong time ago was “upgraded” to OpenWRT. For various reasons, last week I decided to run on it a software (pmacct, indeed) that was NOT included in the official package repository.

Such a scenario was exactly what I was waiting for…. to start discovering the “development” side of OpenWRT. Previously (…up to a couple of weeks ago!) I was “only” sort of… (advanced) OpenWRT user 🙂

So… here is what’s happened when, lately, I decided to became an OpenWRT developer, by creating my first “package”: a real, working, “pmacct” package to be deployed on my OpenWRT box 🙂

This POST is going to describe the whole story 🙂

2. The problems

2.1. Architecture issues

I already knew that building a software to run on my WDR4300 were going to be not an easy task. The main problem I already knew was related to “architecture” issues. My WDR4300 is powered by a MIPS-based CPU:

So, I needed to build an executable to run on a “MIPS 74Kc V4.12” CPU. Obviously, it’s close to impossible to build the whole PMACCT on such a device (I mean: having a GCC compiler and all the related tool in such a small device… is definitely challenging!). Hence I needed to use my PC. Unfortunately my PC is running on a very common x86 platform. So:

Problem 1:
how to build a MIPS-executable using a compiler running on an x86 platform?
Answer to such a question is quite simple: cross-compilation.

Basically, the GCC compiler can surely act as a cross-compiler, being able to run on several platforms (including x86) and creating executables for other, different, platforms (aka: targets) including several ones related to MIPS “ecosystem”.

Unfortunately “cross-compilation” capabilities needs to be explicitly included inside GCC so… this means that you need a specific version (better: a specific “build”) of GCC. A “build” purposely built for your cross-compilation requirements.

This is exactly where the Official OpenWRT SDK come to help, as it’s described with: “The SDK is a pre-compiled toolchain designed to cross compile packages for a specific target“.

So I promptly downloaded the SDK version provided for the WDR4300 platform (ar71xx/generic) and got ready to use it:

To start experimenting with the SDK, as told by official instruction, I defined:

  • the STAGING_DIR environment variable: I guess it’s required by the cross-compiler itself, in order for it to know where exactly its own tools (includes, libraries, tools, etc.) are located around the file-system;
  • the PATH: by prepending the cross-compiler “bin” directory to the existing PATH. This is required so that the cross-compiler can launch its own tool (compiler, linker, etc.).

In order to make some test, I decided to focus on the classic helloworld.c

using both che x86 compiler and the MIPS one.

Everything went OK with the x86 version:

As you can see in line 3, the compiler properly built the x86-64 version, that actually succesfully run on my Linux box (lines 5 and 6).

Then I tried the MIPS variant:

As you can see, the cross-compiler correctly built a MIPS executable and, as such, it CANNOT run on my (x86) machine (see the error message in line 5).

As soon as I moved such a MIPS-binary to my WDR4300…. Bingo!… it worked!

 

Problem 1: SOLVED!
we built a MIPS-executable from a “.c” file, with a cross-compiler running on an x86 platform!
And so? Can we shout “Uau! Finished!” ?
Unfortunately not. Up to now… we are simply able to build a MIPS binary from a single “.c” source file. Nice result…. but not enough!

2.2. cross-compiling a COMPLEX application (autoconf issues)

As you might guess, PMACCT is built around much more than a single “.c” file. Actually a whole cycle of “configure” and “make” is required to build it. And so?

Problem 2:
How to cross-compile a WHOLE, autoconf/configure based application (…instead of a single .c file)?
This is where things start getting interesting…. as the official documentation, unfortunately, is lacking lots of detail: the official Hello World Tutorial  is really well-written; but unfortunately it doesn’t go much beyond the simple “helloworld.c” example. The good news is that… I’ve been succesfull, thanks exactly to the SDK.

In order to get the whole picture, some preliminary issues need to be fixed/clarified. Let’s proceed…

3. OpenWRT SDK: some (important) to-be-known facts

I struggled quite a lot in understanding the “insights” of the SDK. It was easy to understand “the main figure” (aka: the SDK as a tool to build “packages” and/or “kernel-modules” WITHOUT requiring to build “the whole world”). But, unfortunately, behind such a “main figure”, lots of details were hidden.

Indeed, when you know about SDK…. when you get “comfortable” with it… when you have already and successfully used it a couple of times…. than everything sounds very easy. But I guess that if you’re reading these notes, chances are high that it’s your “first time”, so you –like me– are struggling a bit. In such a scenario, my goal is that following notes will result very helpful. Hopefully.

So, here follow a numbered-list of items that can quickly help you in “getting the whole picture”.

3.1. “Packages”, “Package repositories” and “OpenWRT feeds”

I’m sure you already know about “OpenWRT packages”. It’s amazing that, once connected via SSH, you simply launch an opkg install tcpdump and in a few seconds you get the “tcpdump” package downloaded and installed on your box.

Such a behaviour is definitely possibile:

  • thanks to “packages“: where packages are the way to distribute software components (ex: tcpdump, iftop, lsblk, etc.) within the OpenWRT whole ecosystem;
  • thanks to “package repositories“: sort of…. repositories…. holding a (quite large, actually) set of packages

I’m also sure that you know as well, that just before start working with the opkg package manager, you need to UPDATE the list of available packages. Actually, you need an opkg update . Such a command will connect to several remote sites and download a “list” of packages than can be retrieved from such sites. Well, you got the point:

  • a “feed” is a directive that specify WHERE packages are stored and, as such, can be retrieved by the opkg package manager

In my stock OpenWRT box, I found the following six pre-configured feeds:

Every line (aka: every feed) specify a format (src/gz), an identifier (openwrt_core, openwrt_base, openwrt_luci, etc.) and a URL

In order to keep things under control, multiple feeds have been created so to “group”/”classify” the various package repositories.

3.2. “OpenWRT SDK” and related prerequisite (aka: feeds update)

Right after the download/unzip of the SDK, if you don’t perform other actions, basically the SDK is “useless”. Better: it can be used but only in the way we used it here above; to compile stand-alone “.c” application, without any chances to deal with openwrt packages or complex autoconf projects.

In order to deal with packages (rebuilding existing ones or building new ones) and/or complex application (going trough complex “configure & make” cycles), you need to “PREPARE” the SDK.

I’m sure you already guessed one of such actions: updating the feeds, so that the SDK for itself will know WHERE to download packages sources, should it need some of then during the rebuilding and/or compiling activities.

3.2.1 SDK: updating feeds

Once the SDK has been downloaded and unzipped, “feeds” need to be checked-in with a ./scripts/feeds update -a command:

Please note that:

  • the above is REQUIRED by the SDK, but only to know from WHERE to download the packages it might need during building activities;
  • default feeds are defined in the feeds.conf.default file (this is going to be an important point for later discussion; so keep it in mind).

Right after feeds-update, SDK is STILL useless. But in this stage… it’s useless simply ’cause it has nothing to do!

So to step further we need to tell the SDK what to do (eg.: rebuild an already existing package or building a new package from scratch).

3.2.2 SDK: preparing packages to work with

Let’s start trying to rebuild an existing package. Let’s experiment with the tcpdump package.

In order to tell the SDK that we want to rebuild the tcpdump package, we simply issue a ./scripts/feeds install tcpdump . Such a command simply:

  • “download” the sources from a proper repository and put them somewhere inside the SDK folder structure;
  • “download” some additional files (among of which, unfortunately, a file named Makefile…that is a really different thing with respect to the Makefiles included in the tcpdump sources, related to the autoconf cicle);
  • “download” every required packages (tcpdump requires libpcap, so the libpcap packages is “installed” as well)
  • let the SDK to be able to “select” the building of the tcpdump tool.

Here is the output:

3.2.3 SDK: defining exactly what to (re)build

Now, in this stage, everything is in place and finally we can tell the SDK what we want (rebuild tcpdump). How? By firing the make menuconfig command, that will provide us the well-known curses-based interface, where we can define what we need.

To put in a different way:

  1. to rebuild a package (like tcpdump) you need to “mark” such a rebuilding process, via the “make menuconfig” application;
  2. to find references of the package your’re rebuilding “inside” the “make menuconfig” menus, you need to “install” the package sources, from the related “feed”;
  3. in order for the package to be “feed-installable”, you need to “update” the repository/feeds (feeds update);
  4. in order to be able to properly use the “scripts/feeds” utility, you need to download the SDK (specific to your platform);
  5. in order for the SDK to work properly, you need to define the STAGING_DIR environment variable AND update the ensure to point to SDK binaries, via properly prepending the SDK bin PATH to your pre-existing PATH variable.

Very logic. And simple, actually. Don’t you?

Well…. It took, to me, around 10 hours of hard work to really “catch” such an ordered list 🙂

Back to “make menuconfig”, right after launching it, you get the main page:

we start telling the SDK that we DON’T need to build all target/kernel/modules and, also, we DON’T need key-signing issues. To do so, just enter the “Global build settings”submenu and DESELECT all the four items (see below):

My guess is that if we don’t DESELECT such options, the SDK will start rebuilding the OpenWRT kernel and modules for ALL targets (and this is going to require a looooong time. So long, that we don’t want to wait! Not to mention that, in our case, is also useless! Remember: we want to rebuild ONLY tcpdump! Nothing more!)

Back to the main menu, we enter the “Network” submenu and… guess what?

we’ll find the tcpdump (and tcpdump-mini) options!

This is exactly the results of the “feeds install” command that we issued before.

Let’s tell the SDK that we want to build it, by pressing the “M” key (“M” stand for “Module”) on “tcpdump”. Than “SAVE”. Then let’s confirm “.config” as the filename to save, and then simply exit until getting back to prompt.

As soon as you will get back to prompt, you’ll see something similar to this:

As you can see, just before the prompt we have a nice “End of the configuration” message, kindly informing us that we can proceed with “make”. So…

3.2.4 SDK: launching the REAL rebuilding (aka: compiling the sources of the package)

….let’s issue a “make” (actually, as I have an 8-core CPU, I’m going to launch a make -j8 ) and take the chance of a small walk, as the building process will require some time.

After 3 minutes (in my case, remember, I used a parallel approach thanks to -j8) this came out:

No errors. Cool!

3.2.5 SDK: All done!

So… everything seems succesfully finished. Result should be under the [...]/bin/packages folder tree. Let’s check:

Got it! We have exactly two packages:

  • tcpdump_4.9.2-1_mips_24kc.ipk
  • libpcap_1.8.1-1_mips_24kc.ipk

The first is the one we required. The second has been automatically built by the SDK as it’s a REQUIRED DEPENDENCY for tcpdump. So we need it as well.

Problem 2: SOLVED!
we rebuilt a MIPS-based OpenWRT package (autoconf based), thanks to the SDK running on an x86 platform!
 

Can we shout: “All done!”. Unfortunately… NOT yet 🙂

Why? Because the PMACCT tool is available only as a “.tar.gz” and is NOT packaged inside some of the existing OpenWRT repositories. As such it CANNOT be “feeds/installed” and…. the whole process, above, CANNOT be applied.

4. “OpenWRT SDK” how to build a not-already-packaged (and autoconf based) application

Ok. Let’s start again by writing down our problem:

Problem 3:
We have the SDK. We are able to rebuild an existing package with the SDK. But we don’t have “a package”. We have a (big) “.tar.gz”containing sources that can be compiled with “./configure ; make”. HOW CAN WE BUILD AN OPENWRT PACKAGE from such .tar.gz?
Well… The good news is that most of the steps described above needs to be applied also in our specific scenario. So, if you haven’t already done it, please proceed in (see above for details):

  • download the SDK
  • fix the STAGING_DIR and PATH variables
  • update the feeds
  • ensure that the “make menuconfig” can be succesfully executed (you should ensure than when launched, the curses-based interface will pop-up without firing any errors)

Now, in this stage, you have all the required SDK components in place. What is missing is “the package” that the SDK is expecting for the rebuilding activity.

The optimal way to solve this problem is really simple: define a new “feed” inside of which we’re going to put our new “pmacct” package.

4.1. Defining our own “custom” feed

In order to define our own “package repository” (our own “openwrt feed”) we simply create an empty folder on our local Linux box and define such a folder at the top of the [...]/feeds.conf file.

Please note that the feeds.conf file is NOT included in the standard SDK tar.gz. This, ’cause normally the SDK rely on the “feeds.conf.default” file. But in our case, “defaults” are not enough…. as we need to add our own repository. So we simply create the new “feeds.conf” file and add, inside, what we need for our own purposes.

In my case, as “pmacct” requires both “libpcap” and “libpthread”, which are included inside the “packages” feed, in decided to define a “feeds.conf” file containing my brand-new repository as well as the “base” and “packages” one. Here it is:

As you can see, my own repository is going to be defined inside the SOABIT_OWRT_REPO folder.

So I simply created such a folder and… everything should be ready to start hosting my own “packages”.

4.2. Defining a “package” inside our own repository

To add our “custom” package inside our brand-new “custom” repository, it’s really simple. As mentioned in the official documentation, we simply need to create an ad-hoc folder for our new package and create, inside of it, a “package manifest file”. Actually, a “Makefile” specifiying how to deal with such a package.

Creating the mail folder and related “pmacct” folder was really easy, obviously:

The REALLY BIG PROBLEM was figuring out HOW TO CREATE THE MAKEFILE for my PMACCT package.

As I already stated above, the official documentation is quite oversimplied as the “Makefile” it provide is for the “simple” helloworld.c file and… as you know, now, our “pmacct” is quite a different beast.

The good news is that:

  • after several hours of deep searches allaround the web;
  • after several reverse-engineering of “Makefile” included in official, existing, packages (remember the “./scripts/feeds install tcpdump” command? Well… among other things, it will “install” also the “Makefile” created by tcpdump package-mantainer to build the related package! Exactly what we need to… copy!);
  • after LOTS of trial-and-error

I finally was able to create the Makefile of my PMACCT package.

4.3. Defining a Makefile (package manifest) for our package

Once more:
Following Makefile HAS NOTHING TO DO with the Makefile included in the pmacct sources.
They are two VERY DIFFERENT things!
Keep this in mind!

Here it is:

I’m NOT going to describe above file in detail…. not only ’cause it’s HEAVILY based on other Makefiles I found, but also ’cause it’s REALLY DIFFICULT to find proper documentation of the various directives. Anyway, here is what I think it’s important:

  • Lines 16+15 and 11+12: they are used to calculate the URL where the SDK will DOWNLOAD the source file (in this case, a “.tar.gz”). It’s also possible to host the “.tar.gz” locally, but I found easier (and better) to rely on the “real” official source. Obviously I double-checked that the calculated URL is working properly;
  • Lines 17: this is the SHA256 checksum of the downloaded “.tgz”. Keep in mind that as soon as you change something (among the sources) you need to update also such a checksum. If you forget this… the build process will NOT start and… it’s not immediately clear why it’s failing;
  • Lines 21: this is the folder where the SDK will uncompress the TGZ and start the real build
  • Lines 30-34: before building pmacct for OpenWRT I tried building it for a “standard” Linux environment. As pmacct has plenty of “configure” options, due to typical OpenWRT constraint (low resources) I disabled lots of unneeded options. Once I was certain that the building process can succeed in a normal linux-based environment, I simply reported the “configure” options here;
  • Lines 30-31: pmacct rely on libpcap and… I learnt from this very exercise that it’s definitely savvy to compile pmacct against a related, freshly compiled, libpcap. These two lines say exactly this 🙂
  • Lines 51-57: they say where the “pmacct” module can be “actived” inside the “make menuconfig” interface and, also, that pmacct require also libpcap and libpthread packages (as dependencies)
  • Lines 60-64: they tell the SDK that the package it’s going to build should include the four binaries pmacct, pmacctd, sfacctd and nfacctd and that those binaries should be installed in the /usr/sbin folder

That’s it!

Quite simple, BTW (once you know!)

4.4. Building our package

Once the Makefile-manifest is ready, we simply:

  • “refresh” our feeds (so to be sure that our new repository… and related new package… are recognized by the SDK);
  • “install” the pmacct file (so that the SDK will be ready to “build” it)

Here it is:

Now we fire the make menuconfig  tool and we should find “pmacct” under the Network menu:

Now we simply press “M” to activate the PMACCT package and then “SAVE” the “.config” config file and Exit.

Once back to the prompt, we launch the building with a make and few minutes later, here is the result:

As you can see, despite some minor warnings, everything has been built. Actually, even the “libpcap” package has been built (remember? libpcap is a “dependency” for our pmacct).

And so…. let’s check if we have the REAL packages:

Yes!!! Here it is! A freshly compiled pmacct_1.6.2-1_mips_24kc.ipk package!!!!

And so….

Problem 2: SOLVED!
We built an OpenWRT package starting from an autoconf-based “.tar.gz”!

5. Final check

After transferring the ipk file to the OpenWRT box, it has been a matter of
opkg install /tmp/pmacct_1.6.2-1_mips_24kc.ipk
and then, everything worked correctly:

It worked!

Now… it’s only a matter to properly configure the various tool but… this is definitily a topic for a completely different POST!

Enjoy! (…and, please,  leave comments below!)

2 Comments

  1. Stephane Bourdeaud

    Excellent post. I’m struggling to even get helloworld to work on mine. I’m using a Linksys WRT32x which is using an ARMv7 Processor rev 1.
    I’m using the following sdk: https://downloads.lede-project.org/releases/18.06.1/targets/mvebu/cortexa9/ which I think is right.

    I compiled the helloworld.arm successfully on my Linux box (the x86 version worked fine), but when I run it on my router, it says:

    -ash: /tmp/helloworld.arm: not found

    I checked the file is there and that is has the execute bit… Any ideas on what I’m doing wrong?

    • ladreune

      Hi Stephane, thanks for commenting.

      Unfortunately it’s not easy to “guess” the exact problem you’re facing. Anyway I’d start checking two things: 1) that your executable is really an executable built for the Armada385/ARM platform; 2) that you build a “static” binary, so that it (the binary) do **NOT** require additional files (aka: dynamic libraries).

      As for 1), it should be enough to launch an “file <path_to_executable>”.

      As for 2) it should be enough to launch an “ldd <path_to_executable>”

      Those two commands should be launched on your Linux-box.

      If you want, you can cut-and-paste here related output so that, eventually, I can step further!

      Cheers,
      DV

Leave a Reply

Your email address will not be published. Required fields are marked *