Libpnd hub

libpnd - the Pandora Library

libpnd is a basic collection of functions and tools to make working Pandora-specific operations easier; to wit, it is hoped multiple applications will make use of this library rather than re-implement similar functionality and lead to problems down the road. (Instead we can run into problems together and thus clobberize them.)

libpnd thus is a collection of ...

- handy system code tha hpoefully is reusable - io functions, cpu clock setting, etc.

- applications for supporting the pandora unique features, such as pndnotifyd for auto-discovery of PXML and .pnd applicatons

- libraries for support of the .pnd and PXML systems; ie: for locating, executable, mounting and unmounting PXML-directory and .pnd applications, loading PXML and handling overrides

- hopefully reusable code for support of these things; ie: a rudimentary but useful config-file parser, rudimentary singly-linked-list container, etc.

libpnd will/does also include utility functions for setting the CPU clock, reading gpio inputs without depending on SDL, and other handy bits.

You can, nay, are needed - contribute to libpnd!

Design considerations and philosophy

While much of libpnd is obvious, some decisions had to be made and it may help to know the guiding pricinple behind it; the Pandora device will be a handheld _computer_, but libpnd is designed to facilitate certain operations with goals towards..

- simplicity - just work right, configuration not generally needed

- obviousness - work as one woudl expect for this sort of device; ie: the user assumes somethign will work a certain way, and it does

- like other guys - as an extension of the prior point, we have tried to follow unix spirit and provide reusable small tools to do the work sothe bits can be leveraged, but further tried to fllow the spirit of other consoles. So while the device is a powerful little computer, it is probably going to be used by gaming and retro enthusiasts and so we colour some of our decisions that way. (ie: Consider that a PS2/xbox/PSP/etc will have save-game or data management as a separate application, for instance. If thats the best multi-million dollar companies can come up with over decades, well, its not so bad for us to fall back on!)

- not over-defined; libpnd and PXML try to define a bunch of things, but we also do not _over define_; we try to lead by providing examples and styles, and while we have designed ahead for many things, we have also not nailed a lot of that down too early. It is better to be simple and shipped, then complex and a mess.. adoptible rather than scary.

The library is of course broken into parts:

include     - include these files to make use of the lib
lib         - the code that produces the lib
test        - mini tools to test various pieces of the lib in isolation
bin         - test tools - tools for testing the lib as it is developed, not anticipated to be used by many
testdata/   - for testing /etc/pandora; will contain 'conf' dir, for example
apps/       - actual applications people may use, such as pndnotifyd

For now though, here are a couple notes about the design.

- it is straight C code, to be maximumly re-usable (perhaps C++ and other language bindings will tie back to it). I have gone out of my way to limit external references so that bindings are easier to make.

- contrary to the previous comment, we've used a bit of C++ for the tinyxml parser .. hopefully this is not an issue.

- I have tried to keep the API relatively simple and am faking some OO like data hiding to try to keep dev noses out of structures that are subject to change. ie: functions for list management and node access are provided, don't dig into the void* cast structs, or you could get burnt.

- the API should be pretty stable; so far very few user-code changes have ever been needed.. lib internal changes should be binary compatible.

- As a result of that decision, 'handles' are used; a handle is a 'black box' type, something you cannot just use 'handle h; h -> foo = 5' type code for. Instead, internally the library will cast the handle to the actual useful type (which may involve pointer arith etc). This way the user can be returned a handle, and pass the handle to various access functions to get useful data/operations.. but the user is not encouraged to look inside the data objects himself

- As another implication, access functions are available aplenty; it is a little unwieldy perhaps, but its all about keeping the API good without the user peeking inside and risking being broken down the road. (By example, consider.. you use ftell() on a FILE object; you don't go looking in FILE for the 'position' member, since it might be platform specific, etc.)

- The lib works pretty well, but there is room for improvement internally; it won't hurt my feelings for you to suggest improvements, and I'm sure we would all be grateful for any patches you provide. We've _started_ this library, but by no means wish to carry it alone!

Opportunities for improvement

Certainly nothing is perfect, especially when chasing moving targets. Some aspects of the library are less efficient, or missing, or buggy, and a few things are noted here should someone have time. The API was designed to be pretty stable so the internal code could be altered without breaking the API interface.

- better iso lang handling .. due to shift from original PXML spec to new PXML spec with better lang support I've not updated some of the internal code, API uses etc to handle it. (even .desktop files could include all iso-lang titles, and let the WM pick through as the spec suggests.)

Obtaining the library

libpnd is in the Official Pandora GIT repo: [1]

To obtain yourself a copy, use something like:

git clone git://git.openpandora.org/pandora-libraries.git

Thats all you normally need. For copy-paste from the private wiki sake, heres what I had before:

Without going into how to set up a GIT, the summary that I (as a registered dev in the repo) used was:

mkdir libpnd
cd libpnd
git init
git remote add libpnd ssh://skeezixgit@git.openpandora.org/srv/git/pandora-libraries.git
git pull libpnd master

Building the library

Standard 'make' should be sufficient here; pull down the directory from the GIT, cd into it, and run make. Thats it.

Note that the make defalut target is 'everything', which will emit:

libpnd.a - the static linked verison of the lib
libpnd.so.1.0.1 - the dynlinked version
libpnd.so.1 - the stub for the shared lib; ie: apps should link against libpnd.so.1 and not the 1.0.1 over-specific version
bin/* - the 'test' dir and the 'apps' dir builds will end up in bin

There are a few additional make targets, however.

make clean - wipes out the object files, the bin files, and *~* (emacs backup files), empties some of the deploy stuff, etc. Do this before commiting to GIT or we will beat you with sticks.
make deploy - populates the deploy/ directory, which is what is used inthe shipping images; ie: testdata/conf includes test configs, while deploy/etc/pandora/conf includes the configs we ship with. This make will copy over some of the libs and binaries and such.
make pnd - create a few sample pnd files for testing the lib on an x86 machine

Assumptions about the filesystem and Configuration

Note that for libpnd to work, it makes certain assumptions about the filesystem and config files. Since libpnd was developed with the images it should work for the Pandora .. but just dropping libpnd onto another Linux distribution and it won't likely 'work out of the box'. Certain config files will need to be set, and certain conventions followed.

Searchpaths

libpnd was designed so that most everything is configured in files, and that these files should be easily hackable and easily overriden, without compromising the machine. It should be feasible to add a config file to an SD card, have it take over if available when required, and then 'undo' those configs simplky by removal of the SD card and rebooting.

To facilitate this design goal, all config files are 'sought' using a search-path system.

Developers should be mindful of using the libpnd APIs so this behaviour is inherited to interested applications; developers could also reuse the logic for searching for files, if they find it useful.

Searchpaths are colon-delimited list of paths, searched left to right.

ex: foo:bar:/my/baz means to look for a given file in 'foo', then in 'bar', then in '/my/baz'.

The standard config files have a default searchpath, and those config files will define further searchpaths for scripts and such the system uses. This makes the system highly flexible -- default scripts are included, but can be overriden by dropping the right named script on an SD card, or by altering a config file.

See the example config files below for example searchpaths -- ie: how pndnotifyd searches for pnd_run, etc.

Notification paths

The 'apps' config includes a searchpath that specifies the sub-paths to be 'watched'. Various watch mechanisms exist for variousw filesystem types, but over the course of Pandora deveopment (various kernels, various k-modules, various DM and WM's, I found the included system most stable, though not most current. After a final device image is settled on, it might be time to alter the notification system, but it should not impact the libpnd API.)

Anyway, watches are placed upon the paths defined, and if anything in those paths changes, the auto-discovery code is kicked off to seek our PXML.xml and .pnd type applications in the search paths.

Note that searchpath and notifypaths are different (and usually very similar, but have the option to be very different.)

Config files

libpnd includes a generic simple config file parser and assumes the presence of a few config files. It also includes defaults so if a config is missing or broken things should work.

In general, Pandora specific config files should be located in:

/etc/pandora/conf

The config files are:

conf -- this file is generally _not_ included with the Pandora and left to default; its job is to define the basic default searchpath for config files, so should you wish to override things entirely, you may need to alter this file. ie: If you wish to switch to test config files on SD or in /tmp, instead of using any other conf files .. you can create this guy. This provides a mechanism to short circuit the entire conf system.

apps -- this config file defines auto-discovery and other application bundle parameters

desktop -- this config defines "dotdesktop" .desktop and FreeDesktop integration paramters -- example is where to spit out icons or .desktop files from discovered PXML.xml or pnd applications

categories -- this config maps PXML.xml spec based categories to FreeDesktop standard categories; in this way a PXML based applicatoin can show up in the appropriate categories of a WM that does not use libpnd

conf File

No default is included with the distribution.

The default search path is reasonable and assumes we want the official /etc/pandora/conf near the end, with checks to the SD cards for overrides first. (Likely most people will not override, so the canonicle location will be used 99.9% of the time.)

#define PND_CONF_SEARCHPATH "/media/mmcblk0p1/pandora/conf:/media/mmcblk1p1/pandora/conf:/etc/pandora/conf:./testdata/conf"

The key for override is:

#define PND_CONF_KEY        "conf.searchpath" /* if not found, use PND_CONF_SEARCHPATH */

Which suggests using something like this.

[conf]
searchpath   /foo/bar:/baz/bing

desktop File

# Open Pandora
# Desktop configuration

[dotfiles]
#(~/Desktop for xfce, /usr/share/applications for WMs that actually follow spec)
dotdesktoppath  ~/Desktop/      # path for pndnotifyd to spit .desktop files into (run as root)
iconpath        /tmp    # path for pndnotifyd to drop icons into (can be same as .desktop if WM permits)

[launcher]
# if hupscript is commented out entirely, pndnotifyd will not try to find/run the hup
# if it is uncommented, pndnotifyd will attempt to invoke the hupscript after emitting .desktop files
# (the hupscript exists to hup the WMs to redisplay .desktop apps)
#hupscript      pnd_hup.sh

apps File

# Open Pandora
# Application configuration

[autodiscovery]
# searchpath is a list of paths (colon separated) in which to look for PXML.xml or .pnd-file applications
searchpath      /media/*/pandora/apps:/usr/pandora/apps
# notifypath is a list of paths to monitor; if anything in those paths changes, the searchpath is rescanned
# note that for each path chunk, all current subdirs of that path are also watched)
notifypath      /media:/media/*/pandora/apps:/usr/pandora/apps:./testdata/app?

# PXMLs may be overridden .. ie: overrides are a subset of PXML, where the values are copied over the full PXML
[overrides]
# searchpath to look for PXML overrides (which are named by the unique-id)
searchpath      ~/pxml-overrides

# [pnd] defines where to locate the pnd support scripts, so the user may override pnd_run.sh without clobbering built in
[pnd]
searchpath      /media/*/pandora/scripts:/usr/pandora/scripts
runscript       pnd_run.sh

categories File

# Open Pandora
# dotdesktop configuration

# this config file maps 'PXML' categories to free-desktop standard categories
# ie: category 'Foo' could map to more standard 'Utility', thus making .desktop file
# emitting a more useful thing

# the standard listing of categories is:
# http://standards.freedesktop.org/menu-spec/latest/apa.html

# note that 'map' section in the config is _required_ for a match to be found; this
# is done to separate categories from (future) top-level directives

default Application;Utility;Network;

[map]
Development     Development
Education       Education
Games   Game
Graphics        Graphics
Internet        Network
Multimedia      AudioVideo
Office  Office
Settings        Settings
System  System
Utilities       Utility

SD card layout

libpnd concerns itself with things in the 'pandora' root folder on SD. We agonized over this decision, but settled on this to avoid cases where someone with a 16GB SDHC card puts 50-million files with ".pnd" on them, and expects reasonable performance. Instead, applications for instance would have to be in:

/pandora/apps

ie: the searchpath for autodiscovery includes /media/*/pandora/apps

Also in the /pandora/ folder is a magic 'appdata' folder:

/pandora/appdata

When a .pndfile or PXML.xml dir get mounted, any writes back to the mounted area are put into appdata, with a subdir by the unique-id. ie: With the hypothetical MAME.pnd, it might update its config files after a user changes something, or create new files, or store hiscores or whatever. If it doesn't write them to NAND, and doesn't want ot write them to /mame/some/dir, but instead record it to itself for whatever reason, it would go to /pandora/appdata/MAMEs-ID/ instead. The pnd or PXML.xml app area is never modified during a run, only the appdata. (But when an application reads its own path, it will pick up the appdata version of the file instead of the actual file. Then magic of union-type filesystems!)

Example:

If Myapp with unique-id foo.123 is mounted, then any write-backs are directed to..

/pandora/appdata/foo.123/

On the SD that hosts the application.

NAND layout

In addition to seeking applications in SD cards, /usr/pandora/apps is included in the default searchpath, so that 'built in' apps can be listed, or users can drop them there.

It might be a cool addition to put a user homedir into the searchpath as well, so NAND but user-homedir (for on-device browser downloads, say.)

Command line tools and testing

A number of scripts and tools are included or built with libpnd, to facilitate testing of the lib as its being developed and to assist in making .pnd files and PXML.xml files.

PXML.xml and .pnd files

On most platforms, applications are distributed as an executable and optional datafiles and likely an installer. The installers job is to integrate the application into the filesystem, as the transport mechanism itself is not executable and is considered temporary. In Windows an application might be installed into Program Files, while in Unix an application might be dropped into /usr/local/bin .. data files in various other places.

For the Pandora platform we wanted a very easy to use workflow -- download and run -- and also have a very different operating environment. With multiple SD cards available for expansion (and that space being highly desirable due to a smaller internal store) it is desirable to install applications to SD, and yet SD cards are by definition something users will wish to carry multiple of. With two slots, it could be quite a chore to install an application to SD1, its data to SD2, and then saqy have only one of them plugged in at a time, or switch which slot they're in, or any number of scenarios.

The PXML.xml and .pnd-file approach we came up with attempts to address these issues.

The goals:

- for the user, an application can be a single file download

- furthermore, there is no installation whatsoever, beyond dropping that single file into the right place on the SD card

- using the application should be easy -- stick in the SD card, and tap/launch on the icon.

- for uninstallation, delete the single file. Thats it, no fuss.

- for the developer, creating that single file should be easy to do

- for the system, performance should be high

File format - PXML.xml

See the PXML specification

Alternatively the spec is included in the docs/ directory of the libpnd distribution, though the wiki version linked above should be considered cannon.

A note on history; the PXML spec was originally created by EvilDragon, and went through a number of revisions as suggestions came in; as these things are wont to do, the target was ever moving and the scope and meaning of PXML changed so at some point a more strategic approach was needed. dflemstr took it upon himself to help recode some of the parse and re-evaluate many of the PXML decisions and hence was born the PXML-version-2 spec.

File format - .pnd files

The pnd file format is pretty straight forward, though most will interact with it through the included (or community supplied) tools.

Basically its just an ISO file with a PXML and an optional PNG icon appended. (cat appfolder.iso app.pxml icon.png > app.pnd)

The ISO portion of the file can include anything the developer wishes, as long as it includes a PXML.xml file and an executable somewhere in there. It can also include images, audio files, data files of any kind. It can be small or large.

Yes, the PXML.xml is included in the ISO portion (in the root directory), and also appended to the end of the .pnd file (for _convenience_ .. it is very easy and fast to locate in this position; we can backseek into the ISO to find the PXML, but if its a 500MB ISO it could take awhile. Hence, we append it and can find it in one seek.)

This simple format makes it possible to easily create .pnd files with the default utility's of most operating systems. Further, it is a fairly fail-safe system -- the PXML is appended, but is also included within the ISO image part of the file so if the appended-PXML is ever lost (unpack and repack poorly, say) the system will still work (the PXML inside the ISO can be found easily, even without mounting the ISO _as_ an ISO, since it is a plaintext uncompressed image format.)

Manually creating a .pnd (or even a PXML) is a pain so there will be a few helper scripts or maybe even a gui app out when we launch.

Included with libpnd are some scripts to create PXML and pndfiles, though it is expected people will create handier utilities. See the tool section of this wiki for some details.

Question from the gp32x forum regarding confusion of PXML location -- remember that in general mot developers will keep a directory that represents the .pnd or PXML-dir application. Their makefile/build-process will drop the executable(s) into that directory, which inludes a PXML.xml description of the application, and possibly an icon (and whatever all else they wish.) To build the .pnd file, a script (say) is used which compresses all this into the pnd format (including appending the PXML.) So yes, the PXML is in two places in the .pnd file and one place in the source directory. You'll get used to it ;)

Rationale --

Several iterations of pnd files have been considered, from zipfiles to cramfs to other formats, to putting in a key/value set of information tags in the file, to table-of-contents indexes, to any number of options to help performance, and so on. In the end we went for a good overall and simple solution and dropped getting fancy -- PXML.xml can be extended to define the type of the file if it cannot be magic'd, so we could using other filesystem types in the future (and this is why people should use libpnd rather than rolling their own solution.) For now however, pnd files are ISO files with the PXML.xml appended and an optional icon. This keeps the PXML and icon easy to locate, and yet a easy file format.

PND generation workflow

- create a directory to represent the application on SD card; ie: ./mame-app

- create a PXML.xml fil and drop it into the dir (./mame-app/PXML.xml); genpxml script can be used, or do it by hand

- drop the executable and data files into this dir (say, ./mame-app/mame.bin, and ./mame-app/artwork/foo, etc.) <- note, with a PXML.xml and executabe, it can now be discovered and run without even bundling into .pnd!

- invoke pnd-make.sh to produce mame4all-pandora-1.0-arm.pnd or whatever you want to call it (foobearspanky.pnd, we don't care.)

- you're done, distribute it as you see fit

Integration with FreeDesktop .desktop systems

Icons

There are some rules for Icons in PXML.xml and pnd/directory bundles

- An Icon should be in the root of the directory or .pnd bundle

- An Icon must be specified in the PXML.xml if you expect it to work

- As the PXML.xml is parsed, the icon is sought; if the icon is specified, but not found, it is assumed to be a system default icon and the filename will be placed into the .desktop Icon= line verbatim (ex: Icon=foo.png), and the window manager presumably will know how to locate it. If the icon is specified, and is found, then it will be copied into the configured IconPath, and the full path to it will be used for the Icon= line in the .desktop file ex: Icon=/tmp/myicon.png). If no icon is specified in the PXML file, then no Icon= line will be entered into the .desktop and the WM will presumably apply a default icon based on category or whatever.