Disclaimer: in order to keep this post as short as possible, I’m going to be slightly technical. Should you have trouble following this guide, please, leave a comment below: I’ll be proud to help you in an ad-hoc thread
1. The context
If you are reading these notes, chances are quite high that you ALREADY rely on OpenWRT for some of your needs (Wireless; Embedded; etc.).
In my case, I own a nice TP-Link WDR4300 that loooong time ago was “upgraded” to OpenWRT. For various reasons, last week I decided to run on it a software (pmacct, indeed) that was NOT included in the official package repository.
Such a scenario was exactly what I was waiting for…. to start discovering the “development” side of OpenWRT. Previously (…up to a couple of weeks ago!) I was “only” sort of… (advanced) OpenWRT user 🙂
So… here is what’s happened when, lately, I decided to became an OpenWRT developer, by creating my first “package”: a real, working, “pmacct” package to be deployed on my OpenWRT box 🙂
This POST is going to describe the whole story 🙂
2. The problems
2.1. Architecture issues
I already knew that building a software to run on my WDR4300 were going to be not an easy task. The main problem I already knew was related to “architecture” issues. My WDR4300 is powered by a MIPS-based CPU:
root@WRT:~# cat /proc/cpuinfo system type : Atheros AR9344 rev 2 machine : TP-LINK TL-WDR3600/4300/4310 [...] cpu model : MIPS 74Kc V4.12 [...]
So, I needed to build an executable to run on a “MIPS 74Kc V4.12” CPU. Obviously, it’s close to impossible to build the whole PMACCT on such a device (I mean: having a GCC compiler and all the related tool in such a small device… is definitely challenging!). Hence I needed to use my PC. Unfortunately my PC is running on a very common x86 platform. So:
how to build a MIPS-executable using a compiler running on an x86 platform?
Basically, the GCC compiler can surely act as a cross-compiler, being able to run on several platforms (including x86) and creating executables for other, different, platforms (aka: targets) including several ones related to MIPS “ecosystem”.
Unfortunately “cross-compilation” capabilities needs to be explicitly included inside GCC so… this means that you need a specific version (better: a specific “build”) of GCC. A “build” purposely built for your cross-compilation requirements.
This is exactly where the Official OpenWRT SDK come to help, as it’s described with: “The SDK is a pre-compiled toolchain designed to cross compile packages for a specific target“.
So I promptly downloaded the SDK version provided for the WDR4300 platform (ar71xx/generic) and got ready to use it:
$ wget https://downloads.openwrt.org/[...]/ar71xx/generic/openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64.tar.xz $ unxz openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64.tar.xz $ tar xf openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64.tar
To start experimenting with the SDK, as told by official instruction, I defined:
- the STAGING_DIR environment variable: I guess it’s required by the cross-compiler itself, in order for it to know where exactly its own tools (includes, libraries, tools, etc.) are located around the file-system;
- the PATH: by prepending the cross-compiler “bin” directory to the existing PATH. This is required so that the cross-compiler can launch its own tool (compiler, linker, etc.).
$ export STAGING_DIR=[...]/openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64/staging_dir $ export PATH=[...]/openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64/staging_dir/toolchain-mips_24kc_gcc-7.3.0_musl/bin:$PATH
In order to make some test, I decided to focus on the classic helloworld.c
#include <stdio.h>
int main(void)
{
printf("\nHello, world!\n\n");
return 0;
}
using both che x86 compiler and the MIPS one.
Everything went OK with the x86 version:
$ gcc -o /tmp/helloworld.x86 helloworld.c $ file /tmp/helloworld.x86 /tmp/helloworld.x86: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=c5849de6b550632e7660f74cda5e46aca3550154, not stripped $ /tmp/helloworld.x86 Hello, world!
As you can see in line 3, the compiler properly built the x86-64 version, that actually succesfully run on my Linux box (lines 5 and 6).
Then I tried the MIPS variant:
$ mips-openwrt-linux-gcc -o /tmp/helloworld.mips helloworld.c $ file /tmp/helloworld.mips /tmp/helloworld.mips: ELF 32-bit MSB executable, MIPS, MIPS32 rel2 version 1, dynamically linked (uses shared libs), with unknown capability 0x41000000 = 0xf676e75, with unknown capability 0x10000 = 0x70403, not stripped $ /tmp/helloworld.mips -bash: /tmp/helloworld.mips: cannot execute binary file
As you can see, the cross-compiler correctly built a MIPS executable and, as such, it CANNOT run on my (x86) machine (see the error message in line 5).
As soon as I moved such a MIPS-binary to my WDR4300…. Bingo!… it worked!
$ scp /tmp/helloworld.mips root@192.168.2.1:/tmp
root@192.168.2.1's password:
/tmp/helloworld.mips 100% 7392 7.2KB/s 00:00
$ ssh root@192.168.2.1
root@192.168.2.1's password:
BusyBox v1.27.2 () built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt SNAPSHOT, r6487-becf58e
-----------------------------------------------------
root@WRT:~# /tmp/helloworld.mips
Hello, world!
root@WRT:~#
we built a MIPS-executable from a “.c” file, with a cross-compiler running on an x86 platform!
Unfortunately not. Up to now… we are simply able to build a MIPS binary from a single “.c” source file. Nice result…. but not enough!
2.2. cross-compiling a COMPLEX application (autoconf issues)
As you might guess, PMACCT is built around much more than a single “.c” file. Actually a whole cycle of “configure” and “make” is required to build it. And so?
How to cross-compile a WHOLE, autoconf/configure based application (…instead of a single .c file)?
In order to get the whole picture, some preliminary issues need to be fixed/clarified. Let’s proceed…
3. OpenWRT SDK: some (important) to-be-known facts
I struggled quite a lot in understanding the “insights” of the SDK. It was easy to understand “the main figure” (aka: the SDK as a tool to build “packages” and/or “kernel-modules” WITHOUT requiring to build “the whole world”). But, unfortunately, behind such a “main figure”, lots of details were hidden.
Indeed, when you know about SDK…. when you get “comfortable” with it… when you have already and successfully used it a couple of times…. than everything sounds very easy. But I guess that if you’re reading these notes, chances are high that it’s your “first time”, so you –like me– are struggling a bit. In such a scenario, my goal is that following notes will result very helpful. Hopefully.
So, here follow a numbered-list of items that can quickly help you in “getting the whole picture”.
3.1. “Packages”, “Package repositories” and “OpenWRT feeds”
I’m sure you already know about “OpenWRT packages”. It’s amazing that, once connected via SSH, you simply launch an opkg install tcpdump and in a few seconds you get the “tcpdump” package downloaded and installed on your box.
Such a behaviour is definitely possibile:
- thanks to “packages“: where packages are the way to distribute software components (ex: tcpdump, iftop, lsblk, etc.) within the OpenWRT whole ecosystem;
- thanks to “package repositories“: sort of…. repositories…. holding a (quite large, actually) set of packages
I’m also sure that you know as well, that just before start working with the opkg package manager, you need to UPDATE the list of available packages. Actually, you need an opkg update . Such a command will connect to several remote sites and download a “list” of packages than can be retrieved from such sites. Well, you got the point:
- a “feed” is a directive that specify WHERE packages are stored and, as such, can be retrieved by the opkg package manager
In my stock OpenWRT box, I found the following six pre-configured feeds:
root@WRT:~# cat /etc/opkg/distfeeds.conf src/gz openwrt_core http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages src/gz openwrt_base http://downloads.lede-project.org/snapshots/packages/mips_24kc/base src/gz openwrt_luci http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci src/gz openwrt_packages http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages src/gz openwrt_routing http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing src/gz openwrt_telephony http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony root@WRT:~#
Every line (aka: every feed) specify a format (src/gz), an identifier (openwrt_core, openwrt_base, openwrt_luci, etc.) and a URL
In order to keep things under control, multiple feeds have been created so to “group”/”classify” the various package repositories.
3.2. “OpenWRT SDK” and related prerequisite (aka: feeds update)
Right after the download/unzip of the SDK, if you don’t perform other actions, basically the SDK is “useless”. Better: it can be used but only in the way we used it here above; to compile stand-alone “.c” application, without any chances to deal with openwrt packages or complex autoconf projects.
In order to deal with packages (rebuilding existing ones or building new ones) and/or complex application (going trough complex “configure & make” cycles), you need to “PREPARE” the SDK.
I’m sure you already guessed one of such actions: updating the feeds, so that the SDK for itself will know WHERE to download packages sources, should it need some of then during the rebuilding and/or compiling activities.
3.2.1 SDK: updating feeds
Once the SDK has been downloaded and unzipped, “feeds” need to be checked-in with a ./scripts/feeds update -a command:
[verzulli@pc-damiano openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64]$ ./scripts/feeds update -a Updating feed 'base' from 'https://git.openwrt.org/openwrt/openwrt.git' ... Cloning into './feeds/base'... remote: Counting objects: 9035, done. remote: Compressing objects: 100% (7608/7608), done. remote: Total 9035 (delta 1827), reused 4724 (delta 763) Ricezione degli oggetti: 100% (9035/9035), 10.89 MiB | 2.18 MiB/s, done. Risoluzione dei delta: 100% (1827/1827), done. Create index file './feeds/base.index' Collecting package info: feeds/base/package/firmware/lantiq/dsl-vrx200-firmware-Collecting package info: done Collecting target info: done Updating feed 'packages' from 'https://git.openwrt.org/feed/packages.git' ... Cloning into './feeds/packages'... remote: Counting objects: 4189, done. remote: Compressing objects: 100% (3535/3535), done. remote: Total 4189 (delta 180), reused 3277 (delta 112) Ricezione degli oggetti: 100% (4189/4189), 2.54 MiB | 4.83 MiB/s, done. Risoluzione dei delta: 100% (180/180), done. Create index file './feeds/packages.index' Collecting package info: done Collecting target info: done Updating feed 'luci' from 'https://git.openwrt.org/project/luci.git' ... Cloning into './feeds/luci'... remote: Counting objects: 3983, done. remote: Compressing objects: 100% (2264/2264), done. remote: Total 3983 (delta 822), reused 2627 (delta 395) Ricezione degli oggetti: 100% (3983/3983), 3.75 MiB | 3.07 MiB/s, done. Risoluzione dei delta: 100% (822/822), done. Create index file './feeds/luci.index' Collecting package info: done Collecting target info: done Updating feed 'routing' from 'https://git.openwrt.org/feed/routing.git' ... Cloning into './feeds/routing'... remote: Counting objects: 386, done. remote: Compressing objects: 100% (320/320), done. remote: Total 386 (delta 27), reused 147 (delta 3) Ricezione degli oggetti: 100% (386/386), 256.71 KiB | 0 bytes/s, done. Risoluzione dei delta: 100% (27/27), done. Create index file './feeds/routing.index' Collecting package info: done Collecting target info: done Updating feed 'telephony' from 'https://git.openwrt.org/feed/telephony.git' ... Cloning into './feeds/telephony'... remote: Counting objects: 250, done. remote: Compressing objects: 100% (238/238), done. remote: Total 250 (delta 21), reused 29 (delta 0) Ricezione degli oggetti: 100% (250/250), 164.16 KiB | 0 bytes/s, done. Risoluzione dei delta: 100% (21/21), done. Create index file './feeds/telephony.index' Collecting package info: done Collecting target info: done [verzulli@pc-damiano openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64]$
Please note that:
- the above is REQUIRED by the SDK, but only to know from WHERE to download the packages it might need during building activities;
- default feeds are defined in the feeds.conf.default file (this is going to be an important point for later discussion; so keep it in mind).
Right after feeds-update, SDK is STILL useless. But in this stage… it’s useless simply ’cause it has nothing to do!
So to step further we need to tell the SDK what to do (eg.: rebuild an already existing package or building a new package from scratch).
3.2.2 SDK: preparing packages to work with
Let’s start trying to rebuild an existing package. Let’s experiment with the tcpdump package.
In order to tell the SDK that we want to rebuild the tcpdump package, we simply issue a ./scripts/feeds install tcpdump . Such a command simply:
- “download” the sources from a proper repository and put them somewhere inside the SDK folder structure;
- “download” some additional files (among of which, unfortunately, a file named Makefile…that is a really different thing with respect to the Makefiles included in the tcpdump sources, related to the autoconf cicle);
- “download” every required packages (tcpdump requires libpcap, so the libpcap packages is “installed” as well)
- let the SDK to be able to “select” the building of the tcpdump tool.
Here is the output:
[verzulli@pc-damiano openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64]$ ./scripts/feeds install tcpdump Collecting package info: done Collecting target info: done Installing package 'tcpdump' from base Installing package 'libpcap' from base [verzulli@pc-damiano openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64]$
3.2.3 SDK: defining exactly what to (re)build
Now, in this stage, everything is in place and finally we can tell the SDK what we want (rebuild tcpdump). How? By firing the make menuconfig command, that will provide us the well-known curses-based interface, where we can define what we need.
To put in a different way:
- to rebuild a package (like tcpdump) you need to “mark” such a rebuilding process, via the “make menuconfig” application;
- to find references of the package your’re rebuilding “inside” the “make menuconfig” menus, you need to “install” the package sources, from the related “feed”;
- in order for the package to be “feed-installable”, you need to “update” the repository/feeds (feeds update);
- in order to be able to properly use the “scripts/feeds” utility, you need to download the SDK (specific to your platform);
- in order for the SDK to work properly, you need to define the STAGING_DIR environment variable AND update the ensure to point to SDK binaries, via properly prepending the SDK bin PATH to your pre-existing PATH variable.
Very logic. And simple, actually. Don’t you?
Well…. It took, to me, around 10 hours of hard work to really “catch” such an ordered list 🙂
Back to “make menuconfig”, right after launching it, you get the main page:
we start telling the SDK that we DON’T need to build all target/kernel/modules and, also, we DON’T need key-signing issues. To do so, just enter the “Global build settings”submenu and DESELECT all the four items (see below):
My guess is that if we don’t DESELECT such options, the SDK will start rebuilding the OpenWRT kernel and modules for ALL targets (and this is going to require a looooong time. So long, that we don’t want to wait! Not to mention that, in our case, is also useless! Remember: we want to rebuild ONLY tcpdump! Nothing more!)
Back to the main menu, we enter the “Network” submenu and… guess what?
we’ll find the tcpdump (and tcpdump-mini) options!
This is exactly the results of the “feeds install” command that we issued before.
Let’s tell the SDK that we want to build it, by pressing the “M” key (“M” stand for “Module”) on “tcpdump”. Than “SAVE”. Then let’s confirm “.config” as the filename to save, and then simply exit until getting back to prompt.
As soon as you will get back to prompt, you’ll see something similar to this:
As you can see, just before the prompt we have a nice “End of the configuration” message, kindly informing us that we can proceed with “make”. So…
3.2.4 SDK: launching the REAL rebuilding (aka: compiling the sources of the package)
….let’s issue a “make” (actually, as I have an 8-core CPU, I’m going to launch a make -j8 ) and take the chance of a small walk, as the building process will require some time.
After 3 minutes (in my case, remember, I used a parallel approach thanks to -j8) this came out:
No errors. Cool!
3.2.5 SDK: All done!
So… everything seems succesfully finished. Result should be under the […]/bin/packages folder tree. Let’s check:
[verzulli@pc-damiano openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64]$ find bin/packages -name 'tcpdump*' bin/packages/mips_24kc/base/tcpdump_4.9.2-1_mips_24kc.ipk [verzulli@pc-damiano openwrt-sdk-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64]$ ls -l bin/packages/mips_24kc/base/ totale 404 -rw-r--r-- 1 verzulli verzulli 84927 30 mar 00.15 libpcap_1.8.1-1_mips_24kc.ipk -rw-r--r-- 1 verzulli verzulli 692 30 mar 00.17 Packages -rw-r--r-- 1 verzulli verzulli 403 30 mar 00.17 Packages.gz -rw-r--r-- 1 verzulli verzulli 863 30 mar 00.17 Packages.manifest -rw-r--r-- 1 verzulli verzulli 313937 30 mar 00.15 tcpdump_4.9.2-1_mips_24kc.ipk
Got it! We have exactly two packages:
- tcpdump_4.9.2-1_mips_24kc.ipk
- libpcap_1.8.1-1_mips_24kc.ipk
The first is the one we required. The second has been automatically built by the SDK as it’s a REQUIRED DEPENDENCY for tcpdump. So we need it as well.
we rebuilt a MIPS-based OpenWRT package (autoconf based), thanks to the SDK running on an x86 platform!
Can we shout: “All done!”. Unfortunately… NOT yet 🙂
Why? Because the PMACCT tool is available only as a “.tar.gz” and is NOT packaged inside some of the existing OpenWRT repositories. As such it CANNOT be “feeds/installed” and…. the whole process, above, CANNOT be applied.
4. “OpenWRT SDK” how to build a not-already-packaged (and autoconf based) application
Ok. Let’s start again by writing down our problem:
We have the SDK. We are able to rebuild an existing package with the SDK. But we don’t have “a package”. We have a (big) “.tar.gz”containing sources that can be compiled with “./configure ; make”. HOW CAN WE BUILD AN OPENWRT PACKAGE from such .tar.gz?
- download the SDK
- fix the STAGING_DIR and PATH variables
- update the feeds
- ensure that the “make menuconfig” can be succesfully executed (you should ensure than when launched, the curses-based interface will pop-up without firing any errors)
Now, in this stage, you have all the required SDK components in place. What is missing is “the package” that the SDK is expecting for the rebuilding activity.
The optimal way to solve this problem is really simple: define a new “feed” inside of which we’re going to put our new “pmacct” package.
4.1. Defining our own “custom” feed
In order to define our own “package repository” (our own “openwrt feed”) we simply create an empty folder on our local Linux box and define such a folder at the top of the […]/feeds.conf file.
Please note that the feeds.conf file is NOT included in the standard SDK tar.gz. This, ’cause normally the SDK rely on the “feeds.conf.default” file. But in our case, “defaults” are not enough…. as we need to add our own repository. So we simply create the new “feeds.conf” file and add, inside, what we need for our own purposes.
In my case, as “pmacct” requires both “libpcap” and “libpthread”, which are included inside the “packages” feed, in decided to define a “feeds.conf” file containing my brand-new repository as well as the “base” and “packages” one. Here it is:
$ cat feeds.conf src-link soabitrepo /usr/local/src/SOABIT_OWRT_REPO src-git base https://git.lede-project.org/source.git;v17.01.4 src-git packages https://git.lede-project.org/feed/packages.git^cd5c448758f30868770b9ebf8b656c1a4211a240 $
As you can see, my own repository is going to be defined inside the SOABIT_OWRT_REPO folder.
So I simply created such a folder and… everything should be ready to start hosting my own “packages”.
4.2. Defining a “package” inside our own repository
To add our “custom” package inside our brand-new “custom” repository, it’s really simple. As mentioned in the official documentation, we simply need to create an ad-hoc folder for our new package and create, inside of it, a “package manifest file”. Actually, a “Makefile” specifiying how to deal with such a package.
Creating the mail folder and related “pmacct” folder was really easy, obviously:
$ ls -l /usr/local/src/SOABIT_OWRT_REPO totale 0 drwxrwxr-x 2 verzulli verzulli 131 26 mar 00.01 pmacct $
The REALLY BIG PROBLEM was figuring out HOW TO CREATE THE MAKEFILE for my PMACCT package.
As I already stated above, the official documentation is quite oversimplied as the “Makefile” it provide is for the “simple” helloworld.c file and… as you know, now, our “pmacct” is quite a different beast.
The good news is that:
- after several hours of deep searches allaround the web;
- after several reverse-engineering of “Makefile” included in official, existing, packages (remember the “./scripts/feeds install tcpdump” command? Well… among other things, it will “install” also the “Makefile” created by tcpdump package-mantainer to build the related package! Exactly what we need to… copy!);
- after LOTS of trial-and-error
I finally was able to create the Makefile of my PMACCT package.
4.3. Defining a Makefile (package manifest) for our package
Once more:
Following Makefile HAS NOTHING TO DO with the Makefile included in the pmacct sources.
They are two VERY DIFFERENT things!
Keep this in mind!
Here it is:
[verzulli@pc-damiano ~]$ cat /usr/local/src/SOABIT_OWRT_REPO/pmacct/Makefile
#
# Copyright (C) 2006 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#
include $(TOPDIR)/rules.mk
PKG_NAME:=pmacct
PKG_VERSION:=1.6.2
PKG_RELEASE:=1
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=http://www.pmacct.net/
PKG_HASH:=e6ede7f500fb1771b5cdfb63dfa016e34c19b8aa2d2f672bd4c63016a5d6bbe2
PKG_MAINTAINER:=Damiano Verzulli <damiano@verzulli.it>
PKG_LICENSE:=GPL-2.0
PKG_BUILD_DIR:=$(BUILD_DIR)/$(PKG_NAME)-$(PKG_VERSION)
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
include $(INCLUDE_DIR)/package.mk
CONFIGURE_ARGS += \
--with-pcap-includes="$(STAGING_DIR)/usr/include" \
--with-pcap-libs="$(STAGING_DIR)/usr/lib" \
--enable-bgp-bins=no \
--enable-bmp-bins=no \
--enable-st-bins=no
TARGET_CFLAGS += -ffunction-sections -fdata-sections
TARGET_LDFLAGS += -Wl,--gc-sections
CONFIGURE_VARS += \
BUILD_CC="$(TARGET_CC)" \
HOSTCC="$(HOSTCC)" \
td_cv_buggygetaddrinfo="no" \
ac_cv_linux_vers=$(LINUX_VERSION) \
ac_cv_header_rpc_rpcent_h=no \
ac_cv_lib_rpc_main=no \
ac_cv_path_PCAP_CONFIG=""
MAKE_FLAGS := CCOPT="$(TARGET_CFLAGS)" INCLS="-I. $(TARGET_CPPFLAGS)"
define Package/pmacct
SECTION:=net
CATEGORY:=Network
DEPENDS:=+libpcap +libpthread
TITLE:=a powerful IP accounting set of tools
URL:=http://www.pmacct.net/
endef
define Package/pmacct/install
$(INSTALL_DIR) $(1)/usr/sbin
$(INSTALL_BIN) $(PKG_BUILD_DIR)/src/pmacct $(1)/usr/sbin/
$(INSTALL_BIN) $(PKG_BUILD_DIR)/src/pmacctd $(1)/usr/sbin/
$(INSTALL_BIN) $(PKG_BUILD_DIR)/src/sfacctd $(1)/usr/sbin/
$(INSTALL_BIN) $(PKG_BUILD_DIR)/src/nfacctd $(1)/usr/sbin/
endef
$(eval $(call BuildPackage,pmacct))
I’m NOT going to describe above file in detail…. not only ’cause it’s HEAVILY based on other Makefiles I found, but also ’cause it’s REALLY DIFFICULT to find proper documentation of the various directives. Anyway, here is what I think it’s important:
- Lines 16+15 and 11+12: they are used to calculate the URL where the SDK will DOWNLOAD the source file (in this case, a “.tar.gz”). It’s also possible to host the “.tar.gz” locally, but I found easier (and better) to rely on the “real” official source. Obviously I double-checked that the calculated URL is working properly;
- Lines 17: this is the SHA256 checksum of the downloaded “.tgz”. Keep in mind that as soon as you change something (among the sources) you need to update also such a checksum. If you forget this… the build process will NOT start and… it’s not immediately clear why it’s failing;
- Lines 21: this is the folder where the SDK will uncompress the TGZ and start the real build
- Lines 30-34: before building pmacct for OpenWRT I tried building it for a “standard” Linux environment. As pmacct has plenty of “configure” options, due to typical OpenWRT constraint (low resources) I disabled lots of unneeded options. Once I was certain that the building process can succeed in a normal linux-based environment, I simply reported the “configure” options here;
- Lines 30-31: pmacct rely on libpcap and… I learnt from this very exercise that it’s definitely savvy to compile pmacct against a related, freshly compiled, libpcap. These two lines say exactly this 🙂
- Lines 51-57: they say where the “pmacct” module can be “actived” inside the “make menuconfig” interface and, also, that pmacct require also libpcap and libpthread packages (as dependencies)
- Lines 60-64: they tell the SDK that the package it’s going to build should include the four binaries pmacct, pmacctd, sfacctd and nfacctd and that those binaries should be installed in the /usr/sbin folder
That’s it!
Quite simple, BTW (once you know!)
4.4. Building our package
Once the Makefile-manifest is ready, we simply:
- “refresh” our feeds (so to be sure that our new repository… and related new package… are recognized by the SDK);
- “install” the pmacct file (so that the SDK will be ready to “build” it)
Here it is:
[verzulli@pc-damiano SDK]$ ./scripts/feeds update -a Updating feed 'soabitrepo' from '/usr/local/src/SOABIT_OWRT_REPO' ... Create index file './feeds/soabitrepo.index' Updating feed 'base' from 'https://git.lede-project.org/source.git;v17.01.4' ... Already up-to-date. Create index file './feeds/base.index' Updating feed 'packages' from 'https://git.lede-project.org/feed/packages.git^cd5c448758f30868770b9ebf8b656c1a4211a240' ... Create index file './feeds/packages.index' [verzulli@pc-damiano SDK]$ [verzulli@pc-damiano SDK]$ [verzulli@pc-damiano SDK]$ ./scripts/feeds install pmacct [verzulli@pc-damiano SDK]$
Now we fire the make menuconfig tool and we should find “pmacct” under the Network menu:
Now we simply press “M” to activate the PMACCT package and then “SAVE” the “.config” config file and Exit.
Once back to the prompt, we launch the building with a make and few minutes later, here is the result:
As you can see, despite some minor warnings, everything has been built. Actually, even the “libpcap” package has been built (remember? libpcap is a “dependency” for our pmacct).
And so…. let’s check if we have the REAL packages:
Yes!!! Here it is! A freshly compiled pmacct_1.6.2-1_mips_24kc.ipk package!!!!
And so….
We built an OpenWRT package starting from an autoconf-based “.tar.gz”!
5. Final check
After transferring the ipk file to the OpenWRT box, it has been a matter of
opkg install /tmp/pmacct_1.6.2-1_mips_24kc.ipk
and then, everything worked correctly:
It worked!
Now… it’s only a matter to properly configure the various tool but… this is definitily a topic for a completely different POST!
Enjoy! (…and, please, leave comments below!)
Excellent post. I’m struggling to even get helloworld to work on mine. I’m using a Linksys WRT32x which is using an ARMv7 Processor rev 1.
I’m using the following sdk: https://downloads.lede-project.org/releases/18.06.1/targets/mvebu/cortexa9/ which I think is right.
I compiled the helloworld.arm successfully on my Linux box (the x86 version worked fine), but when I run it on my router, it says:
-ash: /tmp/helloworld.arm: not found
I checked the file is there and that is has the execute bit… Any ideas on what I’m doing wrong?
Hi Stephane, thanks for commenting.
Unfortunately it’s not easy to “guess” the exact problem you’re facing. Anyway I’d start checking two things: 1) that your executable is really an executable built for the Armada385/ARM platform; 2) that you build a “static” binary, so that it (the binary) do **NOT** require additional files (aka: dynamic libraries).
As for 1), it should be enough to launch an “file <path_to_executable>”.
As for 2) it should be enough to launch an “ldd <path_to_executable>”
Those two commands should be launched on your Linux-box.
If you want, you can cut-and-paste here related output so that, eventually, I can step further!
Cheers,
DV
Thanks very much for your post!
It’s enlightening but it would be better with some more clarification where I struggled about .
The final file structure for our own repo should be like this:
/usr/local/OWN_REPO/pmacct/Makefile
I had put Makefile in /usr/local/OWN_REPO/ and there are errors all the time!
It may seem stupid but it’s worth mentioning if anyone has the same problem.
Cheers!
Hi Robert, thanks for commenting!
As for your own repo, are you referring to the “feeds.conf” file I described in paragraph 4.1?
As you can see above, the first line of the “feeds.conf” is exactly referencing my own repository:
`src-link soabitrepo /usr/local/src/SOABIT_OWRT_REPO`
and it refers to the whole folder (/usr/local/src/SOABIT_OWRT_REPO) and not to the Makefile.
What exactly should be better described? If you let me know, I’d be glad to further explicit the whole point.
Again, thanks for commenting!
Hi buddy,
I am just starting out with OpenWrt using an RUT230 from Teltonika and this has been a very great eye opener.
My aim is to be able to program my own software and build it in as a package into OpenWrt. I would like to write a package that utilizes Modbus TCP to read registers from a modbus device connected via ethernet and send the data to Azure IoT Hub.
I do not want to reinvent the wheel. Could you be aware of any such a solution? In your opinion and experience, can it work?
I know you may not have enough time, but I would really like to connect to you and see if we can put our heads together in this (If you allow me.)
Hi Brian. I’m quite sure that something around ModBus/TCP have already been built for OpenWRT but… unfortunately I don’t have details. Let me add that I also had some “contact” with ModBus (and ModBus/TCP) also regarding Arduino and other “smaller” (with respect to OpenWRT) environment.
Having said that, let me add that I’m surely interested in any kind of software development, provided that it will be conducted under the Open Source umbrella. Also, as a final note, for sure I’ll have to find some time, for such an activity, to be “cutted” from other tasks. Definitely, not much time but… for sure, higher than zero hours per week 🙂
Feel free to drop me an e-mail: [at] it
This is a terrific tutorial for a complicated, error-prone subject. Thank you very much!
With your help (and not a lot else) I was able to compile existing packages and run the hello-world on my WR710n. Excellent!
Now I’m just struggling to compile an up-to-date package from Master rather than the outdated package in the stable version. Probably a simple thing but beyond my intuition.
Hi UB, thanks for commenting and for appreciating my writing 🙂
I’d be pleased to help you furthermore. Feel free to add details regarding your problem: I’ll to my best to help you 🙂
Thanks again,
DV
Excellent!! Needed that jump start.
Thanks for this post!
Very very helpful.
I may be late on this but remember the linux command: file
it will give you a lot of information about your executeable(?) file
Hi there, thanks for a very informative post about building your own package from source code.
This is exactly what I’m trying to do with Slamtec RPLIDAR Public SDK for C++ at “https://github.com/Slamtec/rplidar_sdk”
I’ve been able to build helloworld example as a package by following the official documentation but the Lidar source is MUCH more complicated and beyond my ability.
I’ve been able to ‘make’ and run the lidar examples in my linux box so i know the source files work, but I have no idea how to reconfigure the ‘feeds’ Makefile to compile the lidar code. All I require is the simple_grabber app really.
Would you be able to help me create a functional feed to make this app? Any suggestions would be greatly appreciated. Thanks.
Hello, I have been following your guide to try to get the c++ library “Pistache” to work. I am able to build the library with the toolchain and cmake(-gui), I get no errors.
If i do $file pistache/src/libpistache.so.0.0.001 it says:
ELF 32-bit MSB shared object, MIPS, MIPS32 rel2
But I do not understand how to package this. When I follow the guide on pistaches webpage to prepare the package I get no tar.gz.
Or is it possible, since I have the .so file to copy it into the staging directory and use it from there?
I feel very close but really lost.
Thank you so much Damiano! For someone who has been working on packages in openwrt and similar sdk’s, it is nice to have a reference to go to. I agree, the openwrt wiki is good, but often too basic.
Right now I was packaging a autoconf-based source, and I was missing the PKG_FIXUP:=autoreconf. The compiler stated that I was missing a Makefile or target to make. So this line helped. This is documented in the openwrt docs, but was ambiguous for me to understand.
brother can you compile https://github.com/rosywrt/nft-qos
this package for linksys wrt610n v2 ??
I’m strugling with it but still can’t get it to work…
Thanks, hit me up using my email when you’re done
Dear Rohit,
it looks to me that the nft-qos is aready packaged in the official kernel, as reported in your link.
You can check the official package, here:
https://github.com/openwrt/packages/tree/master/net/nft-qos
Due to this, you shoud be able to “recompile” it, by working directly on the “official OpenWRT distribution” and… *WITHOUT* the need to go through the lenghty, complex and error-prone process I’ve reported in my post.
HTH
Bye,
DV
Thanks for this great post. Very helpful.
But I try this by myself and have not the same results.
$ mipsel-openwrt-linux-gcc -o hello.mipsel hello.c
/openwrt/openwrt-sdk-18.06.4-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64/staging_dir/toolchain-mipsel_24kc_gcc-7.3.0_musl/bin/mipsel-openwrt-linux-gcc: line 5: /openwrt/openwrt-sdk-18.06.4-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64/staging_dir/toolchain-mipsel_24kc_gcc-7.3.0_musl/bin/../../host/lib/ld-linux-x86-64.so.2: cannot execute binary file: Exec format error
/openwrt/openwrt-sdk-18.06.4-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64/staging_dir/toolchain-mipsel_24kc_gcc-7.3.0_musl/bin/mipsel-openwrt-linux-gcc: line 5: /openwrt/openwrt-sdk-18.06.4-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64/staging_dir/toolchain-mipsel_24kc_gcc-7.3.0_musl/bin/../../host/lib/ld-linux-x86-64.so.2: Success
Can anyone help me?
Thanks
Ok, I figured out this: The error only occurs in 32bit linux machines.
I made the same test on a 64bit linux machine and it worked fine.
Compiled the hello.c only.
Let’s see the next steps.
Hi Sergio! Sorry for my late reply! Glad to see that you solved your issue 🙂
Bye,
DV
Hello ladreune.
Maybe You can help me now.
I have this:
$ make target=mipsel
make: *** No rule to make target ‘mipsel/sha1.o’, needed by ‘link’. Stop.
I can’t figure out what is the problem.
Regards.
This sounds like a wrong (bugged?) “Makefile”. It looks like in the Makefile you have a “target” defined like “mipsel” and, within such a definition, you have some “rules” to build sort of “sha1” library (and, hence, mipsel/sha1.o object file).
But it seems that the very same Makefile does _NOT_ provide any rule to “build” (aka: compile) the “sha1.c”.
Frankly, I really don’t know how to help. Sorry. The only think that I can say is that you should _DEEPLY_ investigate the Makefile and related definition.
Good luck! 🙂
Hi,
I don’t know if you’re still active.. I hope you can help me with this one
My script have utilize the libcurl library, when compiling it in my local machine I need to put in something like this:
agent git:(master) ✗ mips-openwrt-linux-gcc -o x.mips main.c -lcurl
main.c:3:10: fatal error: curl/curl.h: No such file or directory
#include
^~~~~~~~~~~~~
compilation terminated.
But as you can see it cannot detect the library. Do you have a solution for this?
Thanks!